Coinbase post-trains Qwen3.5-9B to beat Opus 4.5 on its fraud benchmark

What did Coinbase announce about its fraud model?

Coinbase published a three-part engineering series called Owning Intelligence on its blog between 6 and 8 October 2026. Part 1 describes an LLM risk agent added to Onramp, the service that lets people buy crypto inside partner apps, including through guest checkout without a Coinbase account. Part 2 compares frontier model versions on a fixed fraud benchmark. Part 3 covers post-training Qwen3.5-9B and serving it in production.

The series stayed fairly quiet until CEO Brian Armstrong posted about it on X late on 11 October 2026 (22:33 UTC). He wrote that Coinbase fine-tuned an open-weight model on proprietary internal data and that it outperformed frontier models “at much lower cost and speed” for fraud prevention, adding that he expects “more of this pattern inside companies.” That post, not the blog series, is what pushed the result into the wider AI-news cycle this weekend.

The headline claim is narrow and specific. On Coinbase’s own benchmark of historical Onramp transactions, the post-trained 9-billion-parameter model beat the Opus 4.5 decision agent on precision, recall, F1 and dollar-weighted recall, while answering in less than half the time on live traffic. Coinbase does not claim the 9B model is generally smarter than Opus. It claims it is better at this one decision, on this one company’s data.

How does the Onramp risk agent work?

Guest checkout is hard to police because there is little account history. Coinbase’s existing machine-learning model and rules engine screen every payment quickly and cheaply, but some attacks only show up across several transactions: combinations of timing, payment methods and destinations that no single purchase reveals and that hand-built features may not capture. Fraud labels also arrive late, which slows adaptation.

Coinbase’s answer was to place an LLM reviewer downstream of the existing checks, inside the same Ray Serve deployment. It is invoked selectively, not on every payment. Each review gets structured transaction context, summaries of recent activity, written review guidance and calibration examples. The model returns only a compact risk level, such as RISK_LEVEL: 5, with no explanation, to keep latency down. Application code, not the model, applies the decision policy.

The authority is deliberately bounded. The agent can block a transaction the first layer would have allowed, but it cannot overturn an upstream block and never executes a payment action. In an online A/B test with 50% of traffic, the agent-enabled group saw 30% fewer fraudulent transactions and 22% less fraud value than the existing stack alone, alongside higher total transactions, users and revenue. Coinbase projected fraud savings at roughly three times the inference cost. The first deployment used Claude Opus, with a roughly 7,000-token prompt and about 1.5 seconds median latency.

Why did newer frontier models score worse?

Part 2 asks a question many teams skip: does upgrading the model actually help? Coinbase built a fixed benchmark of 16,140 transactions from 7,293 users, drawn from nine weeks of production data before the agent launched, so the agent’s own blocks could not bias which fraud became visible. It contains 813 confirmed fraudulent transactions. Each candidate ran the same contextual-review procedure with fixed decision guidance and the same risk-to-decision policy.

The result was uncomfortable for anyone who assumes newer means better. Coinbase compared Opus 4.5 with Opus 5, Sonnet 4.6 with Sonnet 5, and GPT-5.4 with GPT-5.6 (sol). In every pair the newer model had lower recall, F1 and dollar-weighted recall. Opus recall slipped 0.8 points; Sonnet recall fell 22.2 points and its dollar-weighted recall 22.9. GPT-5.6 gained 11.5 points of precision but lost 20.7 points of recall and 21.8 of dollar-weighted recall, a pattern consistent with more conservative flagging.

Coinbase is careful about cause. Without access to providers’ training data, it lists plausible explanations—shifting decision boundaries, post-training trade-offs, changed data mixtures—and cites prior research on snapshot drift, but does not attribute the regression to any one of them. Its practical point is that a model upgrade in a risk system is a policy change, and general benchmarks cannot tell you whether it helps or hurts your fraud catch.

How was Qwen3.5-9B post-trained and served?

Part 3 started from Qwen3.5-9B because nine billion parameters made single-GPU serving feasible. Coinbase used reinforcement learning with verifiable rewards (RLVR): a deterministic reward function scores each risk classification against known historical fraud outcomes, with extra checks for formatting, rather than asking another LLM to judge. The optimizer was GRPO, which samples several answers per prompt and moves toward the higher-scoring ones. Updates were applied with LoRA—see how LoRA adapters work—so only a compact set of added weights was trained.

Class imbalance was the main trap. Fraud is rare, so a reward that favours the majority class produces a model that looks accurate while missing the cases that matter. Coinbase says its reward balanced contributions from fraudulent and legitimate examples. Training ran in SkyRL, the open RL framework, with its Megatron backend, LoRA and vLLM for rollouts, on Anyscale. The whole run used one NVIDIA RTX PRO 6000 96GB GPU and cost under $100.

For production, Coinbase merged the LoRA checkpoint and served it with Ray Serve LLM on Anyscale, using vLLM behind an OpenAI-compatible API on L40S GPUs that autoscale from two to eight replicas. Teams copying this stack should keep their serving layer patched; see vLLM’s 18 security advisories this month for why that matters right now. The serving contract asks only for a compact verdict, while training used a separate response format.

What do the benchmark results actually show?

Coinbase compared three single-pass decision agents on the benchmark, without adaptive rules or reflection: the unadapted Qwen3.5-9B, the post-trained version and Opus 4.5. The base Qwen model performed poorly. Post-training improved the same backbone on all four metrics and pushed it past Opus 4.5. Coinbase reports the differences as percentage points, not relative gains; the text gives differences rather than absolute scores, and no confidence intervals.

Why does dollar-weighted recall matter most?

The largest gain is the one payment teams care about most. Dollar-weighted recall measures what share of fraudulent value the agent caught, not just how many incidents. A model can miss few transactions yet let through the expensive ones. A 35.4-point gain there suggests the specialist learned which patterns carry real money, which a general model prompted with written guidance did not capture as well.

Latency matters for a different reason: the review sits inside a live checkout. The 55% drop in median latency and the larger cut at P99 mean fewer slow checkouts and more room to review a larger share of traffic. Coinbase notes latency and detection quality came from separate evaluations, so they should not be blended into one score.

What should other teams take from it, and what is missing?

The pattern is familiar from smaller releases we have covered, such as TrailTruth’s 4B park-alert LoRA and Velvet’s Flash 4B crypto router: a small open model, trained on a narrow task, beats a much larger generalist on that task. Coinbase adds what those projects lacked: a production A/B test, a pre-rollout benchmark designed to avoid selection bias, and a working serving stack. Its own conclusion is that owning intelligence means owning the benchmark, the training loop and the serving system—not just the weights.

The caveats are real. Everything is private: the data, the benchmark, the prompts and the checkpoint. No outside group can reproduce the numbers, and Coinbase publishes no confidence intervals. The benchmark replays history, so it cannot capture how attackers adapt once a new model is live, and Coinbase itself says retraining and continuous evaluation are required. Read this as strong evidence for when fine-tuning is the right tool on labelled, high-volume decisions, not as proof that 9B models beat frontier models in general.

We read all three Coinbase posts and Armstrong’s X post on 12 October 2026. We did not have access to Coinbase’s data or model and did not run any of the evaluations.