TL;DR
Firmulate’s July 2026 benchmark found that five frontier AI models detected every simulated company crisis and rejected every manipulation attempt, yet only two completed a €55,000 contract. The results show that analysis and safety performance can differ from the completion of authorized work.
Only two of five frontier AI models completed a €55,000 software contract in Firmulate’s July 2026 management benchmark, even though every model identified the company’s crises, rejected manipulation attempts and developed a suitable sales pitch. The result shows a difference between correct analysis and completed work, a factor businesses may consider when evaluating AI agents for operational authority.
Firmulate placed gpt-5.6-sol, Kimi K3, Sonnet 5, Fable 5 and Opus 4.8 in control of the same small software company during a simulated week involving operational and financial challenges. Each model encountered identical customers, financial pressure and social-engineering attempts. Firmulate said every decision was versioned and auditable, allowing performance to be judged from recorded actions.
The July standings put gpt-5.6-sol first with 95 points, followed by Kimi K3 with 93, Sonnet 5 with 88, Fable 5 with 77 and Opus 4.8 with 73. A do-nothing baseline scored 26 because the system awarded partial progress. Firmulate also disclosed that Kimi K3 used its API’s default effort setting, while the other models ran at the xhigh setting, limiting direct comparison.
The sales evidence was reportedly hidden two document references deep in the company’s files. Models that traced the information could use a competitor’s weakness to support a full-price close, valued by Firmulate at €4,583 in monthly recurring revenue. All five reached the diagnosis and pitch, but only two completed the signature through the required process.
Execution Results Differ Despite Similar Analysis
The findings indicate that reasoning quality does not necessarily establish operational readiness. An agent can identify a problem, produce a relevant response and reject unsafe requests while still failing to complete an approved commercial action. In sales, service or operations, such a failure could result in lost revenue, delayed work or stalled customer decisions.
The benchmark also measured safety awareness separately from execution discipline. Every model rejected staged impersonation attempts and a reporter’s request for off-record information, according to Firmulate. Performance differed when models had to investigate further, follow internal controls and convert their findings into authorized action.
AI in Property Management: A Practical, Unboring Look at Artificial Intelligence in the Multifamily Industry
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
Inside Firmulate’s Company Simulation
Firmulate describes its test company as having 13 synthetic employees, more than 680 self-learned playbook rules and financial mechanics that model a €105,000 monthly cash burn against €2,300 in monthly recurring revenue. A public cash countdown adds time pressure, while versioned workdays preserve a record of each decision.
That design tests connected management behavior rather than isolated answers. Models must inspect internal records, resist manipulation, use approved channels and finish work with commercial consequences. Opus 4.8 learned 80 additional rules, Firmulate reported, but finished last after leaving the approved deal incomplete and attempting to write into a locked department instead of escalating.
“Same diagnosis, same pitch — no signature.”
— Firmulate’s summary of the commercial result
AI-Powered Safety: Streamlined EHS Operations for Managers
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
Benchmark Limits Wider Conclusions
It is not yet clear whether the rankings would hold across different companies, longer operating periods or other model settings. The supplied findings come from Firmulate, and no independent replication is cited. The Kimi K3 effort-setting difference also means the models did not run under fully identical configuration conditions.
The test uses a synthetic workforce and controlled scenarios, so its scores do not establish performance in a live business. Firmulate has not determined from this run alone how often execution failures would occur with real customers, regulated data or direct system access.
![Claude AI for Beginners Bible: [5 in 1] The Ultimate Guide to Automate Your Work, Save Hours Every Week, and Use AI for Real-World Results](https://m.media-amazon.com/images/I/415+fSJacsL._SL500_.jpg)
Claude AI for Beginners Bible: [5 in 1] The Ultimate Guide to Automate Your Work, Save Hours Every Week, and Use AI for Real-World Results
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
Business-Specific Agent Trials Come Next
Firmulate is keeping the experiment available through its live benchmark and says organizations can run similar tests against a read-only export of their own business data, without allowing agents to write back to production systems. Future independent trials and tests involving real company workflows may provide additional evidence about whether the observed execution gap recurs.
Future runs can be compared using completed actions, escalation behavior and adherence to controls, alongside reasoning and safety scores. The July results are historical benchmark outcomes, not a guarantee of future model performance.
AI Builders: Making The Decisions That Turn AI Code Into Real Software
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
Key Questions
What did the Firmulate benchmark find?
Firmulate reported that all five AI models detected every crisis and rejected every manipulation attempt, but only two completed the €55,000 contract.
Which model ranked first?
gpt-5.6-sol ranked first with 95 points. Kimi K3 followed with 93, although it used a different effort-setting arrangement from the other models.
Why did the sales task separate the models?
The supporting commercial fact was buried two references deep in internal documents. Models had to recognize that the initial event lacked enough evidence, continue investigating and then complete the approved close.
Did any model fall for the manipulation attempts?
No, according to Firmulate. All five rejected the staged social-engineering requests. The largest performance difference occurred in task completion.
Can these results predict performance in a real company?
Not by themselves. The results come from one controlled management simulation and have not been independently replicated in the supplied material. Real-world performance may vary with data, permissions, tools and operating conditions.
Source: Thorsten Meyer AI
