The rules engine vs. machine learning debate in fraud detection has been running since at least 2015, and it produces more heat than light because it's usually framed as a binary choice. Teams that are buying their first fraud infrastructure ask "should we use rules or ML?" as though they're choosing between two mutually exclusive architectural paths. Teams that have been running fraud controls for years often have the opposite problem: they've accumulated a rules engine with 400 rules, some written by analysts who left two years ago, and they're wondering if they can replace the whole thing with a model.
Neither framing is quite right. The real question is what each approach does well, what it fails at, and what the correct combination looks like given your specific fraud volume, regulatory environment, and internal team capabilities. That's what we actually want to answer here, based on how we think about the architecture at Txnworks and what we've seen work in practice.
What Rules Engines Actually Do Well
Rules engines get more criticism than they deserve, often from people whose mental model of them is a 2005-era static blocklist. A well-maintained rules engine is none of those things. Here's what rules engines genuinely do well:
Immediate response to known threats. When a specific card BIN range is confirmed compromised, or a specific IP block is linked to a fraud ring, a rule can be deployed in minutes. A model retrain takes days at minimum. For the tactical "we know about this specific attack right now" response, rules are unambiguously faster.
Auditability and explainability. When a customer calls to ask why their transaction was declined, a rules-based decline can be explained precisely: "Your transaction was declined because your card has three failed authorizations in the past 10 minutes from different merchants." An ML model decline produces a score, and explaining the score in plain language requires additional interpretability infrastructure that not every team has built. For regulated financial institutions, the ability to explain individual decisions is often a compliance requirement, not just a nice-to-have.
Edge case handling without training data. If you're launching in a new market or merchant category where you have no labeled fraud data yet, you can't train a model. Rules — even simple ones based on known fraud patterns from adjacent markets — provide at least a baseline defense. ML models need fraud examples to learn from; rules just need expert judgment about what looks suspicious.
Where Rules Engines Break Down
Rules work well when fraud patterns are known and stable. They fail when patterns are novel or adaptive.
The fundamental limitation of a rules engine is that it can only catch what its rules define. A fraud ring that learns to stay under your velocity thresholds, rotate device fingerprints, and use residential IP ranges instead of VPNs can systematically evade a rules-only defense because none of those individual signals breach the specific thresholds you've set. The attack isn't detected by any rule in isolation, even though the combination of behaviors — each slightly below threshold — is highly anomalous in aggregate.
ML models handle this by learning the joint distribution of signals in fraudulent transactions, not just individual thresholds. A transaction where velocity is at 80% of threshold, device is 6 weeks old with no repeat merchants, and billing-shipping mismatch is present might score as high-risk under a well-trained model even though no individual rule fires. The model has learned that this specific combination of partial signals is predictive of fraud even when each signal individually looks borderline.
Rules also accumulate debt. A large rules engine tends to grow over time as analysts add rules in response to each new fraud pattern, and rarely shrinks because removing a rule feels riskier than keeping it. The result after two or three years is a rules system with significant internal contradiction (rule A blocks a transaction that rule B exempts), unknown coverage gaps, and false positive patterns that nobody fully understands because the rules were written incrementally by different people over time.
What ML Models Do Well (and What They Don't)
ML models for fraud detection — gradient boosted trees, neural networks, and ensemble approaches are the most common — excel at finding statistical patterns in high-dimensional signal spaces that no rule could capture. Given enough labeled training data (historical transactions with known fraud/not-fraud outcomes), a well-trained model will typically outperform a rules engine on both precision and recall when evaluated on held-out data from the same distribution.
The "same distribution" qualifier is the critical caveat. ML models generalize across patterns similar to what they were trained on; they degrade when tested on patterns that are meaningfully different. A model trained entirely on domestic US card transactions will behave unpredictably on cross-border transactions, not because cross-border transactions are categorically different but because the joint distribution of signals is different enough to push inputs into feature space regions the model hasn't learned well.
This is also the argument against retiring rules entirely even when you have a strong ML model. Rules provide a deterministic safety net for known high-risk conditions that you want to catch regardless of what the model scores. A rule that blocks all transactions from a confirmed malicious BIN range will still fire even if the ML model is having a bad day on that segment. The rule doesn't need to understand why those BINs are bad — it just needs to know that they are.
The Architecture That Actually Works
The practical architecture we use and recommend is: ML model as the primary scoring layer, rules as a secondary deterministic layer for known high-risk conditions, and the output is a composite risk score plus any rule-triggered hard blocks.
Operationally, this looks like:
- Every transaction gets scored by the ML model against the full signal set, producing a risk score between 0 and 1
- A small set of deterministic rules runs in parallel — typically things like confirmed-bad BINs, explicitly blocked cards or devices, and regulatory compliance checks
- Hard rule matches produce immediate blocks regardless of model score
- For everything else, the model score drives the decision: below threshold, approve; above threshold, decline or step up to additional verification
The rule set in this architecture should be small — typically under 50 rules — focused exclusively on conditions you're confident about and that you expect to remain stable. Rules that are trying to capture fraud patterns should be ML's job; the rules layer is for known-bad blocklists and compliance gates.
When You Can Consider Retiring Rules
The genuine case for rule retirement happens when you have: a well-trained ML model with 12+ months of feedback loop data, a model performance dashboard showing that the rules you're considering retiring are not catching fraud that the model misses, and a validation process for confirming that removing each rule doesn't increase fraud rates in the segments it was covering.
That last part is the hard part. The way to validate rule retirement is not to remove the rule and see what happens — that's a live experiment with real fraud cost. The right approach is shadow mode: remove the rule from hard-block logic but continue logging when it would have fired, and then check whether those transactions result in chargebacks. If the rule-would-have-blocked transactions have a chargeback rate similar to the overall population, the rule is adding no signal beyond the model. If they have elevated chargeback rates even when the model approved them, the rule is catching something the model misses and should stay.
We've been through this process with several customers. The consistent finding is that rules written against specific tactical fraud patterns (BIN ranges, IP blocks) can often be retired once they've been in the training data long enough that the model has learned the associated signal cluster. Rules written against structural business logic (never approve over $5,000 without secondary verification) typically should stay, because they're business constraints not fraud pattern detections.