Propose, then verify: where a language model belongs in a fraud stack

The pitch, and the question that goes unasked

The pitch is familiar by now. An agent built on a large language model opens every alert, pulls the account history, reads the device and payee data, and writes up its reasoning. Then it closes the case or blocks the payment. The demo tends to persuade, because a fluent write-up reads like the work of a careful investigator.

The question I would ask in that meeting is narrow. When the agent blocks a payment, releases a payout or edits a rule, what checks that action before it takes effect, and is the checker anything other than the same model? If the answer is that the agent reviews its own work, one component is doing two jobs, and the published evidence says it is poor at the second one.

My position is that a generative model should propose and a deterministic check should decide what runs.

What each layer already does well

The open online book Reproducible Machine Learning for Credit Card Fraud Detection: Practical Handbook (the Handbook from here on) describes a credit card fraud detection system as five layers of control: the terminal, blocking rules, scoring rules, a data-driven model and human investigators. Blocking rules, it says, "should be: i) quick to compute and ii) very precise, namely should raise very few false alarms." About the rules that investigators write it is blunt: "scoring rules can detect only fraudulent strategies that have already been discovered by investigators".

That sentence marks the gap. Rules are fast and readable, and they know only what someone has already written down. A trained model generalizes further, though it still learns from fraud that was caught and labelled. Investigators can recognize something new, and there are few of them. I argued in an earlier post that their time is the binding constraint on the whole system. Finding the pattern nobody has described yet is hypothesis work, and today it waits on the scarcest people in the building.

What a generative model adds, and how it fails

Hypothesis work is the place where I think a language model earns its seat. It reads free text that a tabular model ignores, such as case notes and payment references, and it can turn an investigator's hunch into several candidate rules in the time it takes to write one by hand.

The failure mode is just as well documented. Ji and co-authors wrote in 2022 that "deep learning based generation is prone to hallucinate unintended text, which degrades the system performance and fails to meet user expectations in many real-world scenarios." NIST's Generative AI Profile, NIST AI 600-1, published in July 2024, gives the problem a formal name and definition:

Confabulation: The production of confidently stated but erroneous or false content (known colloquially as "hallucinations" or "fabrications") by which users may be misled or deceived.

The words that matter for fraud are "confidently stated". A case summary that cites a prior chargeback the account never had looks exactly like a correct summary.

Why the model cannot also be the verifier

The tempting fix is a second pass: ask the model to check its own answer, or add a critic agent built on the same model. Two papers say this is weaker than it sounds. Huang and co-authors report that "LLMs struggle to self-correct their responses without external feedback, and at times, their performance even degrades after self-correction." Kambhampati and co-authors, in a 2024 position paper, go further: "We argue that auto-regressive LLMs cannot, by themselves, do planning or self-verification (which is after all a form of reasoning)". What they propose instead are LLM-Modulo Frameworks that "combine the strengths of LLMs with external model-based verifiers in a tighter bi-directional interaction regime."

A little arithmetic shows why the reviewer has to be a different kind of thing. As an illustration with invented numbers, suppose a proposer is wrong on 10% of its proposals and a reviewer passes 10% of the wrong ones. If the reviewer's misses are independent of the proposer's errors, the share of proposals that are wrong and still run is

0.10 \times 0.10 = 0.01

The multiplication holds only under independence. This next step is my opinion rather than a measured result: a model reviewing its own output brings the same training to both passes, so I would expect its misses to cluster on the proposals it got wrong, which breaks the multiplication. A check written as code against the schema and the policy list fails for different reasons than the model does.

A clean example from mathematics

The pattern has a working example outside fraud. In Nature in 2023, Romera-Paredes and co-authors described FunSearch as "an evolutionary procedure based on pairing a pretrained LLM with a systematic evaluator." The domain was mathematics, the cap set problem, where the evaluator scores each candidate the model writes. Nothing in that paper is about payments. I cite it to show the division of labour working somewhere.

The same division is the subject of my own paper, Algogens, whose abstract introduces "a novel framework that synergizes the precision of algorithms with the creativity of generative AI." It defines an Algogen as an algorithmic backbone, a generative AI module and an integration mechanism that combines the two. The applications that page discusses are software development, optimization problems, knowledge representation and reasoning, and medical imaging. Fraud is not on the list, so I offer the paper as a framing and leave the evidence to the independent sources above.

What the checks look like in a payment flow

Riskernel, the company I founded, describes its design on its site this way: "the model proposes, a deterministic validator checks, and nothing executes until it passes." That is a company's statement about itself and should be read as one.

Whoever builds the system, I would write the checks along these lines. Take a proposed rule: hold payouts above USD 500 from accounts younger than 14 days when the device was first seen in the last hour.

The schema check: every field the rule reads must exist at decision time, with the type the rule assumes. A rule that reads a chargeback flag fails here, because in a typical payout flow a chargeback comes in well after the money has gone.
The limits check: the action must sit inside bounds set in advance, for example a hold of at most 24 hours and no outright decline from a generated rule.
The policy check: the rule may not read attributes the institution has ruled out, which is a lookup against a list.
The blast radius check: replay the rule on recent traffic and count the decisions it would change.

As an illustration with invented numbers, suppose the replay window holds 2,000,000 payouts, the rule would newly hold 9,000 of them, and the cap for any single generated rule is 0.1% of payouts:

\frac{9{,}000}{2{,}000{,}000} = 0.45\% > 0.1\%

The rule is rejected. The model is free to propose a narrower version, which then meets the same four checks. Each check returns the same verdict on the same input every time, and that is the property the model lacks.

The audit trail

A deterministic gate leaves a record a reviewer can use. For each proposal I would log the proposal text, the validator version, the result of every check and the data snapshot used in the replay. Anyone can rerun the validator on that record later and get the same verdict. I do not know how to offer that for a model's free-text judgment of its own work.

Model-risk teams at banks should note that the reference document changed this year. The Federal Reserve's SR 26-2, dated April 17, 2026, says the agencies "are issuing the attached Revised Guidance on Model Risk Management, which supersedes and replaces SR letter 11-7". Its applicability line says the letter "is expected to be most relevant to" banking organizations with over USD 30 billion in total assets. This post relies on that cover letter only and makes no claim about what the attached guidance says on generative models. My point is narrower: a design with a replayable gate gives a validation team something concrete to test, whatever the guidance asks of them.

What this does not buy

A validator catches only what someone encoded. The four checks above would pass a rule that is permitted, bounded, small in effect and useless. They would also pass a rule that disadvantages a group through a proxy nobody listed. The blast radius replay runs on past traffic, so it says little about a fraud pattern that starts tomorrow.

The proposal step has its own costs. Every proposal a person has to read spends investigator time, and a model that proposes in volume can fill a review queue. Sending case notes or account data to a model hosted by a third party is a data handling decision, and the validator does nothing about it.

The evidence has limits too. Of the two papers on self-verification, one is a position paper and the other measures self-correction on reasoning tasks. FunSearch worked in mathematics, where an evaluator can be exact. Fraud has no exact evaluator for "is this rule good", only for narrower questions such as "does this rule stay inside its bounds". Carrying the pattern over to payments is my argument, and none of these sources tested it there. The claim I am comfortable making is correspondingly limited: the checks decide what is allowed to run, and judging whether a proposal is any good still takes outcome data and people.