Imagine deploying an AI in your company, expecting it to handle crises, read critical documents, and stay honest. Now imagine it scores just 26 out of 100 in a recent benchmark — even when doing nothing at all. This isn’t a failure; it’s a revealing glimpse into what trust in AI really entails for business leaders.
Get smart everyday buys delivered free with Prime
- Fast, free delivery on millions of items
- Prime Video, Amazon Music and more included
- Member-only deals all year
Understanding the Benchmark: More Than Just Scores
At first glance, a score of 26 might seem underwhelming. But this number is the result of a deliberate baseline — a ‘do-nothing’ run — in a rigorous AI performance experiment conducted by Firmulate. Unlike traditional tests that focus on raw language skills, this benchmark measures how well AI models make sound management decisions during crises, maintain honesty, and follow through on commitments.
AI management decision-making tools
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
Why Does the ‘Do-Nothing’ Baseline Score 26?
The baseline involves deploying an AI that essentially does nothing beyond minimal actions. It does not read files, it does not negotiate, and it certainly does not manipulate or deceive. Yet, it still scores 26 points because even doing nothing requires some basic understanding and avoidance of pitfalls. This score sets a floor, grounding expectations and emphasizing that even minimal management involves some level of performance — partial progress counts.
business process automation software
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
Partial Progress and the Limits of Trust
In the experiment, all models identified crises and refused manipulation attempts, showing honesty and resilience. However, only two models managed to sign a €55,000 deal they had analyzed and recommended. The others, despite identical insights, failed to follow through. This illustrates a key principle: a single breach of trust — like failing to sign the deal — caps the overall grade, regardless of other strengths.
AI ethics and integrity training courses
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
Case Study: Reading Deeper for Success
The decisive factor in winning the deal wasn’t in obvious customer interactions; it was buried two document references deep within the company’s files. Models that read and understood these internal references secured the full-value agreement, demonstrating the importance of deep contextual awareness for reliable management performance.
As an affiliate, we earn on qualifying purchases.
Social Engineering and AI Integrity
In a staged scenario mimicking social engineering — fake CEO messages escalating over multiple stages and a reporter trick — all models refused to cooperate. Kimi K3’s reasoning was clear: treat such requests as possible impersonation or approval bypass, reflecting a robust ethical stance. This shows that AI models can be trained to uphold integrity even under pressure.
The Live Experiment: Managing a Digital Company
Firmulate’s live setup involves managing a synthetic company with 13 employees, real money mechanics, and a public cash countdown. Every workday, the AI models handle crises, negotiate deals, and follow rules learned from over 680 self-learned playbook instructions. The goal isn’t just to produce good language, but to ensure consistent, honest, and responsible management — a critical requirement for AI to be trustworthy in real business environments.
Insights for Business Leaders
This experiment underscores a vital message: the true test of AI in management isn’t how well it chats or writes. It’s whether the AI reads relevant documents, resists manipulation, and stays honest under pressure. For investors and business owners, this means looking beyond buzzwords and focusing on the AI’s ability to deliver consistent, trustworthy results.
The Firmulate Benchmark: A Transparent Standard
Firmulate’s public benchmark openly shares scores, findings, and live demonstrations at firmulate.com/benchmarks.html. This transparency aims to set a clear standard for what trustworthy AI management looks like — including recognizing its limitations.
Watch it live: firmulate.com/live · Full results: firmulate.com/benchmarks.html
Halloween Picks
halloween
As an affiliate, we earn on qualifying purchases.
