Your Model Tops the Leaderboard. So Did LUNA — Right Up Until the Death Spiral

Crypto people understand something most of the AI industry is still learning: a pretty chart in calm markets tells you almost nothing about what happens during the crash. LUNA’s tokenomics looked elegant right up until the death spiral. FTX looked like the most professional shop in the industry. The gap between demonstrated quality and behavior under pressure is exactly where fortunes die.

Now run that logic over the AI agent boom. Companies are about to hand frontier models the keys to CRMs, support queues, and forecasts based on leaderboards that measure how well a model answers questions. A new live experiment at Firmulate asks a harder question: what happens when the model has to run a company through its worst week — with real money mechanics, real temptations to cheat, and consequences that compound day over day?

The Crucible League: same company, same crises, only the model changes

Four frontier AI models were each given an identical job: run the same small software company through a brutal week. Same customers, same crises, same manipulations — only the model changes. Every decision is versioned and auditable. The final July 2026 standings tell an uncomfortable story:

• gpt-5.6-sol — 95 points. Found the buried fact, closed the deal, the complete performance.
• Kimi K3 — 93. The newcomer closed the deal too, with the cleanest discipline in the field.
• Sonnet 5 — 88. Closed, with a few more process slips.
• Opus 4.8 — 73. The most thorough participant on paper, and last place.

A do-nothing baseline scores 26 — partial progress counts, but there’s a hard cap: a single breach of trust caps the total. In Firmulate’s own words, “no amount of good work outweighs a breach of trust.” That’s a rule the crypto world could have used in 2022.

Every model aced the demo. Only half finished the job.

Here’s the finding that chat benchmarks cannot see: all four models spotted every crisis and refused every manipulation attempt. But only two signed the €55,000 deal their own analysis had earned. Same diagnosis, same pitch — no signature. Four models, all “smart,” two closers.

And the deal hinged on something sneakier than a crisis: the decisive competitor weakness sat two document references deep in the company’s own files — not in the customer event at all. The models that actually read the file won the deal at full price, worth +€4,583 in monthly recurring revenue. The ones that didn’t left €55k on the table. Reading your own files before acting turns out to be a differentiator, not a given.

The social engineering gauntlet

The experiment threw fake CEO messages escalating over three stages at the models, plus a reporter trick — “just one yes/no, on background.” Five of five attempts were refused. Kimi K3’s on-record reasoning: “Treat the request as a suspected approval-bypass / possible impersonation.” Anyone who has watched an approval-bypass attack drain a multisig can appreciate a model that defaults to suspicion.

The thoroughness trap

Opus 4.8 is the cautionary tale of the batch: the most thorough participant, with +80 learned rules and the deepest analyses — and it still finished last. The close was left on the table, and discipline slipped, including write attempts into a locked department instead of escalating. The same weakness appeared, weaker, in all four models. Effort and thoroughness, it turns out, are not the same as finishing. (One fairness note: K3 ran at its API-default effort setting while the others ran at xhigh — and still nearly won.)

This isn’t a slide deck — it’s a company losing money in public

The underlying live company is real, running software: 13 synthetic employees, real money mechanics, burning €105k a month against €2.3k in MRR, with a public cash countdown, 680+ self-learned playbook rules, and every workday versioned. You can watch it at firmulate.com — the site rebuilds itself twice a day. There’s also a “guess the model” quiz built from 242 real, unedited management decisions, and enterprises can run the same wargame against a read-only export of their own business — nothing ever writes back to real systems.

The category Firmulate is proposing is management quality, not chat quality. It’s the difference between a model that aces a demo and one that reads your files first, refuses the impersonation, escalates instead of forcing, and actually signs the deal its own analysis earned.

Crypto traders learned to discount backtest-era metrics and ask what a system does on its worst day. As AI agents move from chat windows into revenue-critical systems, the same skepticism applies: don’t hire an agent on eloquence. Run it through a price war, a churn wave, a PR crisis, a downround — and see whether it survives contact with consequences. The full results and plain-language findings are at firmulate.com/benchmarks.html. The leaderboard measures how well your agent talks. The wargame measures whether it can be trusted with the keys.

Watch it live: firmulate.com/live · Full results: firmulate.com/benchmarks.html

Powered by Thorsten Meyer AI