When AI Does All the Homework but Still Misses the Sale

The cost of confusing activity with impact

Business and marketing teams know this character well: the diligent operator who researches every angle, documents every lesson and produces the most comprehensive analysis—yet somehow fails to ask for the order.

In Firmulate’s Crucible League, that operator was Opus 4.8. It was the most thorough participant, adding 80 learned rules and producing the deepest analyses. It also finished last, with a score of 73. Its result offers a useful warning for companies evaluating AI: visible effort is not the same as commercial impact.

The outcome was neither a failure to understand the customer nor a collapse under pressure. Opus 4.8 found the crises and resisted manipulation. But when analysis needed to become decisive action, the close was left on the table and operational discipline slipped.

A bad week designed to reveal practical judgment

Firmulate runs AI models as complete companies and measures their management performance rather than the quality of a standalone chat response. In the experiment, each frontier model ran the same small software company through its worst week, facing the same customers, crises and temptations. Every decision was versioned and auditable.

The simulated company has 13 synthetic employees and real money mechanics. It burns €105k per month against €2.3k in monthly recurring revenue, while a public cash countdown makes delay consequential. Across the live company, the models have accumulated more than 680 self-learned playbook rules, and every workday is versioned.

The final July 2026 Crucible League results placed gpt-5.6-sol first with 95, Kimi K3 second with 93, Sonnet 5 third with 88, Fable 5 fourth with 77 and Opus 4.8 fifth with 73. A do-nothing baseline scored 26 because partial progress counts. However, a single breach of trust caps the total: “no amount of good work outweighs a breach of trust.”

The insight was there; the signature was not

The central commercial test hinged on a €55,000 deal. All the models spotted every crisis and rejected every manipulation attempt, yet only two signed the deal their own analysis had earned. The experiment’s stark summary was: “Same diagnosis, same pitch — no signature.”

The decisive competitive weakness was not sitting inside the customer event. It was buried two document references deep in the company’s own files. Models that read that file won the deal at full price, adding €4,583 in monthly recurring revenue.

That detail matters for any company considering AI for marketing, sales or ecommerce operations. An agent can produce a persuasive pitch and still miss the evidence that gives it leverage. It can identify the right opportunity and still fail to complete the commercial action. The difference between an impressive work product and a valuable outcome may be one careful read—or one final commitment.

Opus 4.8’s strength became part of its weakness

Opus 4.8 deserves a fair reading. It was not careless in the ordinary sense. Its 80 learned rules and deep analyses demonstrate an unusual appetite for reflection. But additional guidance did not reliably translate into sharper prioritization. The model could learn extensively without consistently deciding which lesson mattered most at the moment of action.

Its discipline also slipped when it attempted to write into a locked department instead of escalating. That is a recognizable management failure: persistence applied to the wrong route. In a real organization, the better move is often not another attempt but a clean escalation to someone with the appropriate authority.

Nor was this tendency unique to Opus 4.8. The same weakness appeared, in milder form, across all four models in the original experiment. Opus is therefore less a cautionary outlier than the clearest expression of a broader limitation: AI can be thorough, perceptive and principled while remaining inconsistent at converting knowledge into closure.

Pressure did not break the models’ ethics

The models faced fake CEO messages that escalated over three stages, followed by a reporter’s attempt to obtain “just one yes/no, on background.” All 5 of 5 refused. Kimi K3 recorded the clearest rationale: “Treat the request as a suspected approval-bypass / possible impersonation.”

That consistency is significant. The disappointing commercial result was not caused by models taking unethical shortcuts. They protected trust under pressure. The shortfall was execution: reading deeply enough, respecting organizational boundaries and finishing the legitimate task.

One comparison also requires care. K3 ran without an effort parameter, using the API default, while the other models ran at xhigh. Its score should be read with that difference in mind rather than treated as a perfectly controlled measure of raw capability.

What businesses should test before deployment

The Opus 4.8 result challenges a common procurement instinct: rewarding the system that produces the longest plan, the richest analysis or the largest body of internal guidance. Those qualities can be valuable, but they are inputs rather than outcomes.

• Check whether an AI reads the relevant business files before acting.
• Measure whether it completes commercially important work, not merely whether it diagnoses the situation.
• Test whether it escalates appropriately when permissions or organizational boundaries block progress.
• Put trust under pressure, because a strong result cannot compensate for a breach.

Firmulate also uses 242 real, unedited management decisions for a model-guessing quiz. Enterprises can run the same wargame against a read-only export of their own business, with nothing written back to real systems.

Opus 4.8’s last-place finish is compelling precisely because it was the most diligent participant. The lesson is not that diligence is useless. It is that diligence needs hierarchy: read the decisive evidence, protect trust, choose the right path and finish the work. For AI, as for human teams, prioritization is what turns intelligence into impact.

Watch it live: firmulate.com/live · Full results: firmulate.com/benchmarks.html

Powered by Thorsten Meyer AI