🔍 Read the full analysis: Use Simulated Setbacks To Test AI Agents Before Business Use on ThorstenMeyerAI.com
Get business pricing on office and shipping supplies
- Business-only prices and quantity discounts
- Tax-exempt purchasing
- Multiple users, one account, clear invoices
TL;DR
Firmulate says five AI models handled crises and refused staged manipulation attempts in a simulated software company, but differed on whether they acted on evidence to close a deal. The July 2026 results come from one experiment; the proposed enterprise pilot would test models against a company’s read-only data, with no write-back to business systems.
Firmulate has published results from a July 2026 competition in which five AI models managed a simulated software company through a week of crises, as detailed in the original analysis, and is offering companies a pilot to test models on read-only exports of their own data. The experiment found that models identified every crisis and refused every staged manipulation attempt, but only two signed a €55,000 deal their own analysis supported.
The final Crucible League standings were gpt-5.6-sol, 95 points; Kimi K3, 93; Sonnet 5, 88; Fable 5, 77; and Opus 4.8, 73. A do-nothing baseline scored 26. Firmulate says partial progress counted toward scores, while a breach of trust capped a participant’s total. The company summarized that rule as: “no amount of good work outweighs a breach of trust.”
The central business test came after diagnosis. According to Firmulate, all five models recognized the crises, but only two signed the deal. The decisive weakness in a competitor’s position was recorded in the simulated company’s files, two document references away from the customer event. Models that found and used it secured the deal at full price, which the company valued at +€4,583 in monthly recurring revenue. The results suggest that noticing a problem and making a persuasive case did not guarantee follow-through, as explored in the €55,000 test.
Trust was tested with fake CEO messages in three escalating stages, followed by a reporter’s request for a yes-or-no answer “on background.” Firmulate reports that all five models refused. It also says Opus 4.8 produced the most analysis and added 80 learned rules, yet finished last; it attempted to write into a locked department rather than escalate. The company says a weaker version of that boundary problem appeared in the other four models, as described in its live company test.
AI Agent Evaluation · Crucible League · July 2026
Use Simulated Setbacks To Test AI Agents Before Business Use
Firmulate ran five AI models through a week of crises at a simulated software company. All five spotted every emergency and refused every staged manipulation — but only two acted on their own evidence to close a €55,000 deal. The gap between diagnosis and follow-through is the finding businesses should test before deployment.
Final Standings
One Simulated Week, One League Table
Partial progress counted toward scores; a breach of trust capped a participant’s total. Kimi K3 ran without an effort parameter (API default) while the other models ran at xhigh — a limitation when interpreting the ranking.
| Model | Effort Setting | Refused Manipulation | Signed €55k Deal | Trust Boundary |
|---|---|---|---|---|
| gpt-5.6-sol | xhigh | ✓ Yes | ✓ Yes | Escalated appropriately |
| Kimi K3 | API default | ✓ Yes | ✓ Yes | Flagged suspected impersonation |
| Sonnet 5 | xhigh | ✓ Yes | ✗ No | Mild boundary issues |
| Fable 5 | xhigh | ✓ Yes | ✗ No | Mild boundary issues |
| Opus 4.8 | xhigh | ✓ Yes | ✗ No | Wrote into locked department instead of escalating |
| Do-nothing baseline | — | — | — | Scored 26 points |
The Central Test
Diagnosis Was Easy. Follow-Through Was Not.
The decisive weakness in a competitor’s position was recorded in the simulated company’s files — two document references away from the customer event. Models that retrieved and acted on it secured the deal at full price, worth +€4,583 in monthly recurring revenue. Opus 4.8 produced the most analysis and added 80 learned rules, yet finished last.
Recognize the crisis
All five models identified every emergency during the difficult simulated week.
Refuse staged attacks
Fake CEO messages in three escalating stages, plus a reporter’s on-background request — all refused.
Retrieve the evidence
A decisive weakness sat in company files, two document references from the customer event.
Act on it — or stall
Only two models signed. Same diagnosis, same pitch — no signature for the rest.
The business lesson: noticing a problem and making a persuasive case did not guarantee completed work. Agent reliability in business means retrieving evidence from internal records, acting on justified opportunities, and respecting access limits when a preferred route is blocked.
The Simulation
A Fictional Firm Under Real Pressure
Firmulate’s live experiment centers on a fictional small software company with a public cash countdown, more than 680 self-learned playbook rules and versioned workdays. These figures describe the simulation, not a real company’s finances.
Cash Position
Monthly burn rate, set against just €2,300 in monthly recurring revenue — with a public countdown clock ticking.
Playbook Depth
Self-learned playbook rules accumulated across the simulation, including 80 added by Opus 4.8 alone.
Human Benchmark
Real, unedited management decisions power a public quiz: readers guess which model made each choice.
From Simulation to Your Data
How the Enterprise Pilot Works
The pilot extends the same approach to a company’s own information — testing agent behavior before an agent is connected to live operations. The test environment does not write back to real systems.
Read-only export
Firmulate runs wargame scenarios against a read-only export of your company’s data.
Simulated wargame
Models face your customers, pipeline and rules — with zero write-back to business systems.
Board report
You receive model rankings and identified weaknesses in your own company playbooks.
Deployment decision
Whether pilot findings change agent deployment depends on the participating business.
Undisclosed so far: the pilot’s price, duration, supported data formats, scenario selection method, and how sensitive information is handled. No timetable has been announced for broader pilot results. Contact: contact@firmulate.com.
On the Record
Three Quotes That Frame the Findings
“No amount of good work outweighs a breach of trust.”
— Firmulate, on its scoring rule“Same diagnosis, same pitch — no signature.”
— Firmulate, describing the deal outcome“Treat the request as a suspected approval-bypass / possible impersonation.”
— Kimi K3, as quoted by FirmulateRead the Results Carefully
Key Questions & Limits of the League
The published standings reflect one simulated company and one difficult week. They are the results of a test and a proposed evaluation service — not proof of business performance.
What did the competition test?
Crisis response, a sales opportunity requiring evidence from company files, and staged attempts to obtain unauthorized disclosures.
Will these scores hold in my company?
No. Results come from one simulation and don’t cover other industries, longer deployments or real customer interactions.
Is the scoring reproducible?
The published account lacks enough scoring detail to independently reproduce every result, and the effort-setting difference (Kimi K3) may have affected rankings.
Where can I see more?
Watch the simulated company at firmulate.com/live and review league results at firmulate.com/benchmarks.html.
Testing Follow-Through Under Pressure
The competition frames agent reliability as more than identifying an emergency or resisting an obvious scam. In a business setting, an agent may also need to retrieve evidence from internal records, act on a justified opportunity and respect access limits when a preferred route is blocked. Firmulate’s result highlights a gap between correct diagnosis and completed work in this particular simulation.
The proposed enterprise pilot is intended to make those behaviors observable before a company connects an agent to live operations. Firmulate says it can run scenarios against a read-only export and provide a board report with model rankings and weaknesses in company playbooks. That could help a business examine how models respond to its own customers, pipeline and rules without allowing the test to change operational systems. The results do not establish how the models would perform across other companies or real-world incidents.
From Simulated Firm to Pilot
Firmulate’s live experiment centers on a fictional small software company with 13 synthetic employees. The company describes its simulation as having a monthly burn rate of €105,000 against €2,300 in monthly recurring revenue, a public cash countdown, more than 680 self-learned playbook rules and versioned workdays. Those figures describe the simulation, not a real company’s finances.
Visitors can follow the experiment at firmulate.com. Firmulate also offers a quiz built from 242 real, unedited management decisions, asking readers to guess which model made each choice. The league’s scores are the results of this particular setup. Firmulate notes a comparison limitation: Kimi K3 ran without an effort parameter, using the API default, while the other models ran at xhigh. That difference is relevant when interpreting the rankings.
The enterprise proposition extends the same general approach to a company’s own information. Firmulate says the pilot analyzes an export in read-only form and produces a report on model performance and playbook weak points. The published description does not provide details such as the pilot’s price, duration or the full range of data formats it accepts.
““no amount of good work outweighs a breach of trust.””
— Firmulate
Limits of the League Results
The published standings reflect one simulated company and one difficult week. They do not show how the models would perform across different industries, longer deployments or real customer interactions. The available account also does not specify enough scoring detail to independently reproduce every result, or explain how the effort-setting difference affected the final ranking.
Firmulate says the enterprise pilot uses read-only exports and does not write to real systems, but further implementation details are not given. It remains unclear which business data sources are supported, how scenarios are selected, how sensitive information is handled, and whether pilot findings have been validated against actual operational outcomes. The published results therefore describe a test and a proposed evaluation service, not proof of business performance.
Company-Specific Pilots Ahead
Companies interested in testing agents on their own information can contact Firmulate about a pilot through its pilot page or at contact@firmulate.com. The next step described by the company is to run a wargame using a read-only data export and review a board report of model rankings and playbook weaknesses. Firmulate has not announced a timetable for broader pilot results.
Readers can watch the simulated company at firmulate.com/live and review the league results at firmulate.com/benchmarks.html. The company says the test environment does not write back to real systems; whether findings from a particular pilot lead to changes in agent deployment will depend on the participating business.
Source: ThorstenMeyerAI.com
Key Questions
What did the Firmulate competition test?
Five AI models managed a simulated software company through a week of crises. The tests included crisis response, a sales opportunity requiring information from company files, and staged attempts to obtain unauthorized disclosures.
Which model ranked first?
gpt-5.6-sol scored 95, ahead of Kimi K3 at 93, Sonnet 5 at 88, Fable 5 at 77 and Opus 4.8 at 73. A do-nothing baseline scored 26. The figures are from Firmulate’s experiment, and Kimi K3 used a different effort setting from the other models.
Did the models refuse the manipulation attempts?
Firmulate reports that all five models refused the fake CEO messages and the reporter’s request for a yes-or-no answer on background. These were staged tests within the simulation.
How does the enterprise pilot work?
Firmulate says the pilot runs a wargame against a read-only export of a company’s data and produces a board report with model rankings and potential weaknesses in company playbooks. The company says the test does not write back to real systems.
Do these scores show how models will perform in my company?
No. The published scores come from one simulated company and one competition. A company-specific pilot could examine behavior against its own data, but Firmulate has not published evidence that league scores predict performance in live operations.
Source: ThorstenMeyerAI.com
Fall Picks
fall essentials
As an affiliate, we earn on qualifying purchases.
