Alert budgets, not AUC: judging a fraud model in the units that matter

A great number and a queue full of junk

Picture a model review. The slide says the new fraud model has an AUC of 0.97, up from 0.95. A month later the investigators report that their queue feels the same as before. Most of what they open is still genuine customers.

AUC summarizes how a model ranks transactions across the whole range of scores, and investigators only ever see the very top of it. I argue here for measuring a fraud model where it is used: in the alerts a team has time to open, and in the money each decision moves.

Why rare events make familiar metrics misleading

Start with how rare fraud is. The ULB credit card dataset, a public benchmark, describes itself as having "492 frauds out of 284,807 transactions", so that frauds "account for 0.172% of all transactions." A model that approves everything is 99.828% accurate on that data and catches no fraud at all.

AUC is the probability that a randomly chosen fraud scores higher than a randomly chosen genuine transaction, an average over every possible threshold, including ones that would flag half of all traffic. The ROC curve behind it plots the true positive rate against the false positive rate, and when fraud is rare a tiny false positive rate is many people.

As an illustration with invented numbers, suppose a portfolio sees 1,000,000 transactions a day and 1,000 of them are fraud. A threshold that catches 80% of the fraud at a 1% false positive rate sits near the top-left corner of the ROC plot. It also raises

0.01 \times 999{,}000 = 9{,}990

false alerts alongside 800 true ones. Precision is 800 out of 10,790, about 7.4%, and somebody has to open 10,790 cases every day.

Saito and Rehmsmeier, writing for a bioinformatics audience in PLOS ONE in 2015, showed that "the visual interpretability of ROC plots in the context of imbalanced datasets can be deceptive". They recommend precision-recall plots, since precision is the fraction of positive predictions that are truly positive, which is what a reviewer experiences. A precision-recall curve still spans every threshold, though, and a team operates at one of them.

Rarity is also a modelling problem. The abstract of one of my patents, Method for Extreme Class Imbalance Within Fraud Detection (US 12,541,721), describes "a computerized-method for building ensemble of supervised and unsupervised Machine Learning (ML) models for fraud-predictions, for a client having an extremely-imbalanced-dataset".

The real constraint: investigators can only check so many alerts

A fraud team has a fixed number of investigators, and each case takes time. The open online book Reproducible Machine Learning for Credit Card Fraud Detection: Practical Handbook, by Le Borgne and co-authors (the Handbook from here on), states the consequence: "The number of alerts that may be checked during a given period is therefore necessarily limited. Precision top-k metrics aim at quantifying the performance of an FDS in this setting". FDS stands for fraud detection system.

Dal Pozzolo and co-authors, writing in IEEE TNNLS in 2018 about "a real-world data stream containing more than 75 million transactions, authorized over a time window of three years", describe a gap between much published research and working systems: "This lack of realism concerns two main aspects: 1) the way and timing with which supervised information is provided and 2) the measures used to assess fraud-detection performance."

Go back to the invented portfolio and suppose the team can review 500 alerts a day. Even if every one of them were a false alarm, the false positive rate would be 500 out of 999,000, about 0.05%. The team's whole working range sits inside the leftmost 0.05% of the ROC plot's horizontal axis, and nearly all of the area under the curve comes from thresholds the team will never run.

Precision at k, and card precision at k

Precision at $k$ measures that sliver. Rank the day's transactions by score, take the top $k$ , and count how many are fraud:

P@k = \frac{\text{frauds among the top } k \text{ alerts}}{k}

In the Handbook's words, "The k parameter quantifies the maximum number of alerts that can be checked by investigators in a day." I would compute it per day and average over the test period.

Now compare two hypothetical models on the invented portfolio with $k = 500$ . Model A puts 150 frauds in its top 500 and spreads the other 850 through the rest of the top 5% of traffic. Model B puts 250 frauds in its top 500 and spreads the other 750 evenly through the rest of the top half. AUC is the average, over frauds, of the share of genuine transactions ranked below each one: essentially all of them for a fraud in the top 500, about 97.5% for one spread through the top 5%, and about 75% for one spread through the top half. So

\text{AUC}_A \approx 0.15 \times 1.00 + 0.85 \times 0.975 \approx 0.98 \qquad \text{AUC}_B \approx 0.25 \times 1.00 + 0.75 \times 0.75 \approx 0.81

On the slide, model A wins by a wide margin. In the queue, investigators find 250 frauds a day with model B and 150 with model A, so B is the better model for a team that opens 500 alerts. AUC credits model A for ranking frauds well at depths nobody reviews.

The Handbook also defines card precision at $k$ , which counts cards instead of transactions, because "investigators check all the recent transactions when contacting cardholders." Ten alerts on one card are one investigation and should count once.

From precision to money

Counting frauds treats a stolen 30 dollar purchase and a stolen 2,000 dollar transfer as equal. The Handbook's chapter on cost-sensitive learning defines the field as addressing "classification problems where the misclassification costs are not equal" and lists "finance (e.g., fraud detection)" among its applications.

A cost matrix writes this down as one cost for each pairing of decision and truth. In the simplest version a correct decision costs nothing, a missed fraud costs the amount $a$ , and a wrongly declined genuine transaction costs a figure $c$ for lost margin and customer friction. With fraud probability $p$ , declining is the cheaper action when

p \cdot a > (1 - p) \cdot c \quad\Longleftrightarrow\quad p > \frac{c}{a + c}

The threshold depends on the amount. Take $c$ to be 20 dollars, an assumption for illustration. A 2,000 dollar transfer should then be declined above a probability of about 1%, and a 30 dollar purchase only above 40%. At $p = 0.03$ , approving the transfer has an expected loss of $0.03 \times 2{,}000 = 60$ dollars and declining it has an expected cost of $0.97 \times 20 = 19.40$ dollars, so declining is cheaper even though the transfer is 97% likely to be genuine. The 30 dollar purchase with the same score goes the other way, with 0.90 dollars of expected loss against 19.40.

Charles Elkan made the general argument in "The Foundations of Cost-Sensitive Learning" in 2001: "it can be rational not to approve a large credit card transaction even if the transaction is most likely legitimate."

For the review queue, I would rank alerts by expected loss, $p \cdot a$ , and report the fraud value inside the top $k$ next to the fraud count. The two can disagree: model B's 250 frauds could all be small purchases while model A's 150 include the large transfers.

What the model owes the queue: calibration, ties, and capacity changes

The cost rule multiplies $p$ by money, so $p$ has to behave like a probability. AUC cannot tell you whether it does, because any order-preserving rescaling of the scores leaves AUC unchanged. Rebalanced training data is one way to break it. Suppose, as an illustration, that the training sample kept every fraud and one genuine transaction in ten. The odds of fraud in that sample are ten times the odds in live traffic, and the scores inherit the inflation. A raw score of 0.25 means odds of 1 to 3 in the sample and 1 to 30 in production, a probability of about 0.032. With $c$ at 20 dollars, the raw score declines a 100 dollar purchase and the corrected score approves it.

Coarse scores can give several hundred transactions the same value right at the cut, and precision at $k$ then depends on an arbitrary ordering. I would state how ties were broken, or report the expected value under random ordering.

Budgets move with staffing, so I would report precision at several values of $k$ around the budget, for example 250, 500 and 1,000. In the invented example, models A and B tie at about 6,900 alerts a day, and beyond that A finds more fraud because it holds all 1,000 frauds in the top 5%.

What this does not buy

The amount at risk is easy to read off a transaction. The cost of declining a good customer is hard to measure, and the threshold moves with it: on the 30 dollar purchase, the break-even probability is about 14% if $c$ is 5 dollars and 62.5% if $c$ is 50. I would show results under a low and a high assumption for $c$ . The simple matrix also treats a missed fraud as a total loss of the amount and a declined fraud as free.

Precision at $k$ is also silent about everything below the cut. In the example, the better model for the queue still leaves 750 frauds a day unreviewed, and their labels arrive late or never, which is the first of the two gaps Dal Pozzolo and co-authors name. Reporting the share of fraud count and fraud value captured in the top $k$ keeps that tail visible.

Finally, precision at $k$ depends on the fraud rate. On a day with 100 frauds, a perfect model scores 20% at $k = 500$ , so the number does not transfer between portfolios or between quiet and busy weeks. AUC is unaffected by the fraud rate, which is a fair reason to keep it and the precision-recall curve in the report as secondary checks under the alert budget and the money figures.