Fraud Models Don't Fail, They Starve

Most fraud teams start with rules, and rules work. "Block a first transfer above this amount." "Review a login from a new device followed by a payout within ten minutes." Rules are fast, explainable and easy to tune.

Then the fraud changes. Fraudsters learn your thresholds and split payments just under them. A mule network spreads one theft across forty accounts. A takeover happens slowly, over days, so no single event looks wrong. The rules that caught last quarter's fraud miss this quarter's, and the team responds the only way it can: more rules, each one catching a little more fraud and blocking a few more honest customers.

So teams reach for machine learning, and many are disappointed. The model goes live, scores every payment, and a few months later it is no better than the day it shipped. The usual diagnosis is a weak model. More often the model is fine. It is starving. It has never been told which of its decisions were right.

A fraud model learns from outcomes: this payment was later charged back, that one settled, this alert was real fraud, that one was a false alarm. Those outcomes are called labels, and in most fintechs they are scattered across support tickets, bank emails and analysts' spreadsheets, or never recorded at all. This article is about the system that collects them, the feedback loop, and why it matters more than which model you choose.

Why labels are scarce in African payments

In card-heavy markets, labels arrive almost on their own: a cardholder disputes a charge, the network raises a chargeback, and the fraud label lands in the merchant's system with a reason code. Much of the fraud-detection literature quietly assumes that world.

Much of African fintech runs on different rails, and the labels do not arrive by themselves.

• Instant bank transfers are push payments. The victim authorised the transfer, often after being deceived, so there is no card-style chargeback. Recovery, when it happens, runs through dispute and reversal requests between institutions, and the outcome may never reach the fintech that scored the transaction.
• Mobile money and agent networks have their own complaint channels. A fraud case can be confirmed by an agent, a call centre or the operator, each recording it somewhere different.
• Outcomes arrive late. A mule account may sit quiet for weeks before it is reported. A model retrained on last week's data treats those transactions as genuine, because nobody has said otherwise yet.
• Analysts' verdicts get lost. The most valuable labels a fraud team produces are its own calls: "this alert was real", "this one was a false alarm". In many teams they live in a spreadsheet, a Slack thread or an analyst's memory, never joined back to the transaction that was scored.

The result is a model trained once, on a historical dataset, and then left to drift while the fraud around it moves on.

The feedback loop

A feedback loop turns every decision into a lesson. The model scores a payment, the real outcome comes back later, that outcome is stored as a label on the same transaction, and the model is retrained on what actually happened.

The gate in the middle is the part teams skip. A retrained model is not automatically a better one: it may have learned from a bad week of labels, or overfitted to one fraud ring. It replaces the current model only after it beats it on transactions neither model trained on.

Three sources of truth, and which one wins

Labels come from three places, and they will sometimes disagree. Decide in advance which one wins, or the training data ends up depending on whoever wrote last.

Two consequences follow. An analyst should never be able to overwrite a chargeback. And a transaction with no outcome yet is unlabelled, not genuine: treating silence as innocence is how late-arriving fraud teaches a model the wrong lesson.

Six mistakes that poison a feedback loop

A feedback loop that collects the wrong labels is worse than none: it trains the model, confidently, on noise. These are the mistakes we see most often.

1. Recording the alert, not the transaction. An analyst closes an alert as "resolved", but the payment it was about carries no outcome. Labels must attach to the scored transaction, or the model never sees them.
2. "Resolved" that means three different things. Real fraud, a false alarm and "we looked and moved on" all closed with the same status. Give analysts distinct verdicts, confirmed fraud and false positive, and treat everything else as unlabelled.
3. Letting the last writer win. An analyst's guess overwrites a chargeback that arrived a week earlier. Rank your sources (the table above) and enforce the ranking in code.
4. Mixing test and live traffic. Sandbox transactions, demo accounts and load tests leak into the training set. Keep the mode on every transaction, label and alert, and train on live only.
5. Training on the label window you have not finished waiting for. If fraud is typically reported within thirty days, the most recent thirty days are not "mostly genuine"; they are mostly unknown. Leave them out of training until the window has closed.
6. Promoting a model because it is new. Every retrained model must beat the current one on held-out data it never saw, on the measures that matter (fraud caught, honest customers flagged), before it touches a live decision. Keep the old model running until it loses.

A checklist for your team

Whatever engine you use, your own or ours, you can check where your feedback loop stands this week.

• Every scored transaction has an ID that your settlement, reversal and dispute records also carry.
• Analysts close alerts as confirmed fraud or false positive, not just resolved.
• Each verdict is stored on the transaction, with who made it and when.
• You have written down which source wins when two disagree.
• Test and live traffic are kept apart in every table that feeds training.
• You know your typical reporting delay, and recent transactions inside it are left out of training.
• A new model goes live only after it beats the current one on held-out data.
• You can say how many labelled outcomes you collected last month.

If you tick fewer than half, the model is not your bottleneck. Fix the loop first, and the model you already have will start getting better.

Where ArcheaOne fits

We built ArcheaOne, a real-time fraud detection platform for African fintechs, around this loop from day one. A fraud engine that cannot learn from its own decisions is only a rules engine with extra steps.

• A clear decision with plain-language reasons for every payment, login and account event, in real time.
• Outcomes that flow back from your payment records and your analysts’ everyday calls, without spreadsheets in between.
• Detection that gets sharper on your own traffic, without your team building a data pipeline.

Want to see the loop running on your own traffic? Request access to ArcheaOne.