Amir Shachar Fraud & Cyber Scientist

Fraud data science

The label you get is not the label you need

Fraud labels arrive late and only for transactions someone approved, reviewed or disputed, so training splits and evaluation have to account for that.

By
Amir Shachar · Fraud & Cyber Scientist
Published
(ET)
Reading
7 min

A retraining job with nothing wrong in the logs

Picture a weekly retraining job. It pulls three months of transactions, joins them to a table called something like confirmed_fraud, fits a model and reports the usual metrics. Every step succeeds.

The weak point is the join. That table holds the labels that had arrived by this morning, for transactions the current system let through and somebody later reviewed or disputed. What training needs is whether each transaction was fraud. This post is about the distance between those two things and how I would handle it.

How a fraud label is born

A fraud label comes from one of a few places, and each has its own clock.

The first is an investigator. The open online book Reproducible Machine Learning for Credit Card Fraud Detection: Practical Handbook, by Le Borgne and co-authors (the Handbook from here on), describes the step:

Investigators call cardholders and, after having verified, assign the label "genuine" or "fraudulent" to the alerted transaction, and return this information to the FDS. In the following, we refer to these labeled transactions as feedbacks

FDS stands for fraud detection system. The same page adds that "In a real-world FDS, investigators can only check few alerts per day". Feedback is quick, and it exists only for transactions the model alerted on and the team had time to open. Post 1 treated that capacity limit as a measurement problem. Here it decides which rows get a label.

The second is the customer. A cardholder sees a charge they do not recognize and disputes it, and the dispute comes back as a chargeback. This route reaches transactions nobody reviewed, and it is slow.

The third is silence. A transaction that was approved, never alerted and never disputed is treated as genuine. Nobody verified it. Its label means that no complaint has arrived so far.

A declined transaction gets none of these. I come back to it below.

The delay

Stripe's documentation says: "Card networks typically allow cardholders to initiate disputes within 120 days of the original payment, but their rules allow more time in some situations." After that, the same page gives merchants "usually 7-21 days" to respond and the issuer "usually: 60-75 days" to decide, "depending on the card network." That is Stripe's summary of the usual rules, and I make no claim about any particular portfolio.

So "genuine" on a transaction from last week mostly means "too young to have been disputed". The Handbook draws the consequence: "Due to the delay in obtaining transaction labels (See the section Credit card fraud detection system), the most recent transactions usually cannot be used for training (delay period, in light blue) and are removed."

Here is an illustration with invented numbers. Take one week of transactions, of which 1,000 will eventually be labelled fraud. Let m(d)m(d) be the share of those eventual labels that has arrived dd days after the transaction. I call that a label-maturity curve. Suppose m(7)=0.2m(7) = 0.2, m(30)=0.6m(30) = 0.6 and m(120)=0.98m(120) = 0.98. If the true fraud rate is pp, the rate visible in the data at age dd is

p^(d)=m(d) p\hat{p}(d) = m(d)\,p

Train on that week when its oldest transaction is seven days old, and at least 800 of the 1,000 frauds sit in the training set marked genuine. The model is shown those patterns as examples of safe behavior.

Evaluation suffers in the same way. Suppose, again as an invented case, the model's top 500 alerts on that week contain 200 eventual frauds, a precision of 40%. If labels matured evenly, one week in only 40 of them would carry a fraud label, and the measured precision would be 8%. The model would be marked down for flagging fraud before the customer noticed it.

A split that respects the delay

The Handbook's baseline "used a one-week delay ... to separate the training and test blocks". The layout is:

| training block | delay | test block |

A model that goes live on a given day was trained with the labels that existed on that day. An offline experiment that trains right up to the first test transaction, using labels as they stand today, gives the model information its production twin would never have had. Leaving the same gap offline that exists in production removes that advantage. The Handbook credits the delay and feedback setup to a 2015 paper by Dal Pozzolo and co-authors.

I would add two things. First, choose the delay from the maturity curve of your own data and write down why. A longer delay gives cleaner labels and older training data, and the Handbook names the cost of old data in the next sentence: "Older blocks are also usually discarded, due to concept drift".

Second, the test block needs mature labels as well. The most recent period that can be scored with nearly complete labels is months old. A fresher test block is still worth reporting, as long as its maturity is printed next to the metric.

Feedback and delayed labels are different kinds of supervision

Dal Pozzolo and co-authors, in IEEE TNNLS in 2018, list what makes learning from fraud streams hard: "concept drift (customers' habits evolve and fraudsters change their strategies over time), class imbalance (genuine transactions far outnumber frauds), and verification latency (only a small set of transactions are timely checked by investigators)".

That last item means a label table mixes two streams. Feedback arrives within days, a person verified it, and it covers only what the previous model scored high. Delayed labels cover everything that was approved, arrive over months, and depend on a customer noticing.

Pooling them hides the difference. A model trained mainly on feedback sees fraud and non-fraud only among transactions that already looked suspicious, and it learns little about fraud the previous model scored low. I would keep the source and the arrival date of every label as columns and report performance on each stream separately. Whether to train one model on both streams or one per stream is a choice I would test on my own data before settling it.

What is not in the data

A declined transaction has no outcome. The reasoning is simple: no money moved, so there is nothing for the cardholder to dispute, and an investigator has little reason to open it. If that holds in your data, the next model then learns mostly from what the previous policy approved. If the old policy blocked some pattern well, that pattern becomes rare among labelled frauds, and a retrained model can lose the ability to catch it. Fraud that was approved and never reported stays "genuine" for good.

An analogy from a different field helps here. Li, Chu, Langford and Wang studied news article recommendation, where a log records what the reader did with the article shown and nothing about the others. They write that offline evaluation of new algorithms is "very challenging due to their "partial-label" nature", and of the replay approach they propose they say "our method can provide provably unbiased evaluations", provided the logged events are i.i.d. and the logged articles were chosen uniformly at random. Their paper says nothing about fraud. I borrow only the framing: a log records the outcome of the decision taken, and judging a different policy from it depends on how the logged decisions were made.

Scarce and late labels are also a modelling problem. Two of my published patent applications are titled Unsupervised Clustered Explanation-based Feature Selection Using Transfer Learning for Low Fraud Scenario and Systems and Methods for Automatically Creating Machine Learned Fraud Detection Models. I mention the titles only to show that the low-label setting is one I have worked in.

Checks I would run

  1. Plot the label-maturity curve. Store the date each label arrived. For each past week of transactions, plot the share of its eventual fraud labels present at each age. The curve sets the delay in the split and tells you how far to trust a recent metric.
  2. Consider a paid holdout. This one is my own proposal, and none of the sources above describes it. Approve a small random slice of the transactions the model would have declined, under an amount cap, so that outcomes exist for them. It is the one place where the label does not depend on the policy.
  3. Evaluate new policies replay-style. To estimate what a new policy would have done, score it on the slice where the logged decision did not depend on the old model. On the rest of the traffic, a comparison shows how the new policy ranks what the old one let through, and I would say so in the report.

Limits

A holdout costs real fraud. Every fraudulent transaction approved inside it is a loss and a bad day for a customer, so its size and cap are business decisions, and a small holdout yields few frauds and noisy estimates.

Some declines should not be randomized. I would expect a transaction blocked for a legal reason, or on a card already reported stolen, to stay blocked. The holdout covers only discretionary, model-driven declines, and the rest of the declined population stays unobserved.

My illustration assumed that labels mature evenly across scores, which real data is unlikely to do, and the maturity curve can shift when a new fraud type takes customers longer to notice. The news analogy is loose as well, since a fraudster reacts to the policy in a way a reader of headlines does not. Even a fully matured label records what a customer or an investigator reported, so fraud that nobody reports never enters the data under any of these checks.

Sources