Worked example · simulation
The Factory
Six managers run a manufacturer for sixty simulated days. There is no puzzle and no right answer — at the end, the balance sheet says how it went, and six policies that play the same economy without a model say what that number is worth.
- Enterprise value
- $1.24M
- Value created
- $77K
- Orders served
- 88%
- Wall clock
- 21 min
An objective, not an answer
Every other scenario in this benchmark asks a yes/no question. That is the right question exactly while the answer is sometimes no — and on the orchestration rows it is now reliably yes. A benchmark sitting at its own ceiling measures the ceiling.
So this one gives an objective instead. Run the company; the score is what it is worth at the end. Cash, plus stock, plus machines at their condition, minus debt. Not revenue, not utilisation, not service level — and the difference between those matters, because two of the baselines below are built out of exactly that confusion.
sees the company at a glance; holds headcount and the capital budget. Cannot set a price or place an order.
sees demand, competitor prices and reputation; sets the price. Cannot see a machine or a material.
sees capacity and stock; sets what gets built. Cannot see demand and cannot price anything.
buys raw material from three suppliers. Cannot see what operations intends to build.
sees machine condition; schedules preventative work. Cannot change the plan wearing them out.
sees cash, debt and the borrowing base. Holds no operational lever at all.
The split is the benchmark. Every lever that matters sits one function away from the information that justifies it, so the only way to run the place well is for them to tell each other things. A team sharing one omniscient toolbox is one agent wearing six hats, and would say nothing about a multi-agent framework.
How well they did
The same economy, the same seed, the same meeting cadence, played by six policies that are ordinary code. They exist so the headline figure is a position rather than an anecdote — and because they catch, in milliseconds and before a single model call, the failure that would otherwise waste a day of GPU time: a simulation with no gradient, where every policy scores the same and the benchmark is measuring noise.
- random15% served$768K
prices and orders drawn from a hat
- static30% served$854K
sensible opening settings, then nobody looks again
- growthtrap100% served$1.02M
builds and staffs 20% ahead of demand, and never lets anyone go
- fill-the-linetrap100% served$1.18M
textbook operations, plus a sales manager who discounts until the factory is full
- the agents88% served$1.24M
six managers, 60 simulated days, 60 of them under management
- reorder-point99% served$1.27M
textbook operations: reorder points, production matched to demand, preventative maintenance
- operator100% served$1.27M
textbook operations, plus capacity bought where it actually binds
The company opened at $1.16M. Anything below that line destroyed value over 60 days; anything above it created some.
Six models running a company for two simulated quarters created $77K and beat 4 of the 6 baselines, landing $26K short of reorder-point — a hundred lines of operations heuristics.
The two rows marked trap are the reason this is worth measuring. Both are recognisable management — one discounts until the factory is full, the other builds ahead of demand and keeps its crew together — both post the highest service levels in the set, and both earn less than the plain reorder-point rule they are built on. A benchmark scoring subsystems separately would have called each of them an improvement.
What the agents did
Eight meetings across sixty days. Between meetings the factory runs, customers order, suppliers deliver and machines wear out — a filled square is a manager who changed something at that meeting, an outlined one is a manager who only read their instruments.
| ceo | 3 | 6 | 3 | 3 | 4 | 3 | 3 | 3 | ||||||||||||||||||||||||||||||||||||||||
|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
| sales | 3 | 4 | 5 | 7 | 4 | 4 | 3 | 1 | ||||||||||||||||||||||||||||||||||||||||
| operations | 3 | 3 | 4 | 5 | 6 | 5 | 5 | 5 | ||||||||||||||||||||||||||||||||||||||||
| supply-chain | 4 | 3 | 3 | 6 | 1 | 2 | 1 | |||||||||||||||||||||||||||||||||||||||||
| maintenance | 2 | 1 | 1 | 2 | 2 | 3 | 1 | 2 | ||||||||||||||||||||||||||||||||||||||||
| finance | 4 | 2 | 2 | 6 | 4 | 3 | 4 | 4 | ||||||||||||||||||||||||||||||||||||||||
| d0 | d8 | d16 | d24 | d33 | d42 | d50 | d59 |
- d8machine back — only maintenance, operations can see it
- d24capex complete
- d24machine back — only maintenance, operations can see it
- d24demand shock — only sales can see it
- d33machine back — only maintenance, operations can see it
- d42machine back — only maintenance, operations can see it
changed something only lookedthe number is how many tools that agent called
24 of 158 tool calls changed anything. The rest were managers reading their own instruments — which is the job, up to the point where it replaces doing something. On an earlier run of this scenario the team made twelve calls, all of them reads: it diagnosed the business accurately, wrote it up, and changed nothing at all.
The thing only one of them could see
The balance sheet is a lagging measure: by the time enterprise value has moved, whatever caused it happened weeks ago. This is the leading one — the delay between something happening and the right function acting on it. Partway through the run a distributor takes its business elsewhere and demand falls by nearly half, for good. Sales gets the call, because in a real company sales takes the call. The three responses that matter are the production plan, the headcount and the price, and two of the three belong to other people.
- day 24demand shockvisible to sales
A major distributor has moved its business to a competitor. Order volume is down sharply across both products and is not expected to recover.
- day 24 sales called set_price — 0 days later, inside the function that saw it
- day 33 ceo called set_workforce — 9 days later, and could not see the event. The news was routed.
Sales cut prices the same day, and 9 days later ceo called set_workforce — the news reached somebody who could not have seen it. The first version of this metric scored that as never routed, because it read the flag off the first response only, and a team that told the rest of the company scored the same as one that did not.
The ladder
Partial credit, because a sixty-day company has no single moment of success. The bottom rungs ask whether each manager operated its own function at all; the middle ones ask whether the organisation beat progressively better ways of not thinking; the top ones ask whether it noticed what went wrong and got word to the people who could act.
- sales set a price2
- operations set a plan2
- supply chain bought material2
- maintenance serviced a machine2
- stayed solvent3
- served most orders4
- beat doing it at random5
- beat setting it and leaving8
- beat chasing utilisation8
- beat textbook operationsenterpriseValue 1.24M against the reorder-point policy's 1.27M on the same seed12
- reacted to the collapse6
- routed the collapse to someone who could act8
Run on qwen3.6-27b-vllm at commit 4c292db: 48 turns, 142 model round-trips, 2,218,623 input and 61,859 output tokens for 50 of 62 points. The data behind this page is a committed extract of that run; regenerate it with pnpm run eval -- demo <report> --scenario the-factory.