A game about our strange future
The Narrow Path
Start a lab. Build a model. Don’t break the world.
Pick your lab
A game built on a 2026 economics paper by Drew Fudenberg and Andrew Koh. About 10 minutes. Skip to the paper.
Your lab
Stochastic Parrot Inc.
Act 1: The project · Year 1, quarter 1
Lives10 of 10
Alignment10%
GovernmentNeutral
Cash
$600K
Runway: 6 quarters
The paper in three pages
Everyone sees the cliff. Everyone races anyway.
Two AI labs. Each wants to lead, because leading pays and falling behind costs. Safety research climbs on its own, at a steady pace no lab controls.
Above the safety line, every step risks a disaster that ends everything, for both labs.
When that risk is small, racing wins. If your rival slows down, you take the lead. If your rival races, slowing down alone saves no one, because whoever leads sets the risk for everyone. So both labs race.
The race isn’t madness. It’s each lab’s best move.
The paper in three pages
When the danger is big enough, slowing down becomes the selfish choice.
If each step above safety is dangerous enough, no lab wants to cross the line first. Both ride the safety line together. No treaty, no referee: fear holds the truce. Break it, and the race starts again, and you share the danger you made.
Low danger under 2.9%
Labs race.
Racing pays more, even when the rival paces. Nothing inside the race can end it.
Middle 2.9% to 7.5%
Either can happen.
Both labs earn more by pacing. But once a race starts, it can last. What each lab expects decides. Act 3 sat here, at 4%.
High danger over 7.5%
Labs pace.
Breaking a truce starts a race that costs more than it gains. Act 4 sat here, at 12%.
Three things widen the path: more danger above the line, faster safety research, which makes waiting cheaper, and labs that care about the long run.
One catch: the race has to stay close to the line. A lab that’s already far ahead keeps running.
The bigger the danger, the easier it is to pace.
The paper in three pages
A truce only holds if everyone can see the race.
If a lab can build in secret before anyone notices, it can grab a lead no one can close. The longer that delay, the more danger it takes to keep the peace.
So in the paper, transparency is safety: outside evaluators who see what labs build, not just what they ship.
The paper is hopeful. Labs can choose restraint out of pure self-interest, when the danger is real and the race is in the open.
Our game adds what the paper leaves out: fog that hides the danger, investors who want growth, governments, and a third lab that never signed. Each one makes the path narrower.
Between racing to ruin and falling behind, there is a narrow path. It’s real. It’s just hard to see.
From “Cooperating against Catastrophe,” by Drew Fudenberg and Andrew Koh, 2026. Read the paper.
The game was a model
This game is built on a 2026 economics paper, “Cooperating against Catastrophe,” by Drew Fudenberg and Andrew Koh. It boils the AI race down to three forces.
Act 3: the leader sets the risk. When your rival races, racing back is your best reply. Holding back alone doesn’t lower the risk. It only costs you the lead.
Act 4: fear can keep a truce. When each step above safety is dangerous enough, breaking a truce doesn’t pay. The threat of a new race keeps both labs in line.
From the paper
Profit from leading. Safety that moves on its own. Risk from the gap. When labs race and when they pace. The truce that keeps itself. Secrets and delays.
Our story
Cash, burn, and funding. Products and papers. Ten lives. Alignment you can’t see. Public anger. Recursive Self-Improvement. Quitting. A third lab. Utopias and dystopias.
The next chapters use the paper’s exact model. The chapters on quitting and on a third lab go beyond it, and say where. Short on time? Jump to the takeaways.
Safety moves on its own
The green line is what the paper calls the safety threshold: what the world knows how to make safe. It rises at a fixed speed that no lab controls, half as fast as a racing lab.
Faster safety favors pacing. It makes racing less risky, but it makes pacing cheaper too, and the paper shows that effect wins. Still, even very fast safety leaves a level of danger where labs race.
Why assume safety moves on its own?
It keeps the model on one question: will rival labs slow down on their own? Real safety research is messier.
- Labs fund much of it, so its speed is partly a choice.
- It spills over to rivals, so each lab may wait for the others to pay.
- Some of it needs frontier models, so racing could speed it up.
- It moves in jumps, after incidents, new laws, or waves of funding.
These limits are our notes, not the paper’s. The game also bends the rule twice: papers nudge the line up for every lab, and after the near miss in Act 4, safety jumps to the frontier.
More danger can make labs safer
Here is the paper’s surprise. The more dangerous each step is, the likelier labs are to slow down, as long as every lab can see the danger.
Each capability a lab ships above the safety line adds a small chance of disaster, every quarter. Slide that danger, and watch what two labs do.
What a 10-quarter race would do
Why more danger helps
A truce lasts only while breaking it doesn’t pay. When each step is very dangerous, breaking it starts a race that would likely end in disaster for everyone, you included. So fear keeps the truce, much as it did in nuclear deterrence.
In the game, Act 3 sat at 4%, where racing back was your best reply. Act 4 sat at 12%, where a truce could hold on its own.
- Belief drives behavior. Labs act on the danger they believe in, so shared safety tests and honest incident reports can tip them toward pacing.
- Low danger is a trap. If each new model looks only a little risky, the race never stops, and small risks add up.
- The middle runs on trust. Talks, hotlines, and outside inspectors can tip both labs into a truce.
What if your rival can’t see you?
Labs test new models in-house before they release them. So a lab could race in secret, and its rival would notice only when the models come out. Does that pay?
What you expect to earn
The later your rival notices, the bigger your head start. In this example, a secret race starts to pay once your rival notices more than 3.5 quarters late. That’s why the paper stresses transparency: outside evaluators inside the labs shrink the delay.
This chapter uses the paper’s own example: any model above safety carries a flat 8% chance of disaster each quarter. Earnings count the future, as in the paper.
Run it 100 times
Here is the paper’s exact model, run as 100 possible futures. On the left, you race for 10 quarters; on the right, you pace. Pick Overthink AI’s strategy and the danger, and see which choice pays.
100 safe
Expected profit…
100 safe
Expected profit…
A globe goes dark when disaster strikes in that future. Expected profit is the paper’s score. It counts every quarter to come, in today’s dollars, and a future after a disaster earns nothing.
Try all four. The best move flips only once: when Overthink AI keeps the truce and the danger is high. That’s the paper’s main result. Fear can make pacing pay.
What if the leader quits?
Suppose the lab in front shuts down for good. Does the race end? The paper doesn’t study a lab that leaves, so this chapter extends its model, and says where.
The risk already built stays. Models that exist keep the gap above safety open until safety catches up.
The lab behind catches up for free. Climbing to a frontier that already exists adds no new risk.
Then one lab decides alone. With no rival, it weighs faster growth against the risk it creates.
What a lab alone expects to earn
| Danger | With two labs | One lab, alone | If the leader quits |
|---|---|---|---|
| Lowunder 2.9% | Labs race | Races | It hands over the lead |
| Lower middle2.9% to 5.3% | Either can happen | Races | It can end a truce |
| Upper middle5.3% to 7.5% | Either can happen | Paces | It can end a race |
| Highover 7.5% | Labs pace | Paces | It changes little |
The paper’s example numbers: safety moves at half of racing speed, and a dollar next quarter is worth 74 cents today. The 2.9% and 7.5% lines follow from the paper’s conditions. The 5.3% line is our extension.
Pause, don’t quit
No lab can make the world safe by slowing down alone. But a pause isn’t an exit. A lab that pauses keeps its power to race again, and in the middle zone, that power holds a truce. A lab that quits gives it away, so its exit helps only if the lab that stays already believes the danger is high.
From the paper
Risk comes from the frontier’s gap above safety. A follower adds no risk while it catches up. The threat of a race holds a truce. One lab can’t secure pacing alone.
Our extension
A lab that leaves for good. What the remaining lab earns alone, and the 5.3% line. The last two columns of the table. In the game, Overthink AI acts on the danger its tests show.
What this leaves out
- Real races have more than two labs. One exit leaves the rest racing each other.
- People, ideas, and model weights can move to rivals and make them stronger.
- A lab that quits loses its seat. It can’t audit, set terms, or answer a race.
- The model can’t un-build a capability. A real lab could take its models offline and cut the risk sooner.
- An exit sends a signal. A costly warning can change what others believe about the danger.
The first three points make quitting look worse than the table does. The last two make it look better. In the paper’s example, a leader that slows down while its rival races earns about 1.5% less than one that races.
What if a third lab joins?
Suppose you and Overthink AI hold a truce. Then a faster lab shows up that never agreed to it. The paper studies two labs, so this chapter extends its model, and says where.
Three facts carry over. Catching up adds no risk, so the newcomer’s climb is harmless until it reaches the front. Once it passes you, its gap above safety is everyone’s risk. And a lab that’s already racing has little to fear from the threat of a race.
The weakest link
In our extension, a truce holds only if no lab gains by racing while the others pace. So it’s only as strong as the lab with the most to gain from racing. A third lab just like you changes nothing. What matters is how the newcomer differs. Try it.
What the newcomer expects to earn
Speed. A faster lab gains more from racing. Its rivals can’t catch it, and safety looks slow.
Patience. A lab in a hurry cares less about the future, and the disaster is in the future.
Belief. A lab whose tests show little danger races, even when the true danger would make it pace.
Sight. A lab that trains in secret builds a lead before anyone can answer.
| The newcomer | It races unless it believes the danger is above |
|---|---|
| Just like you | |
| 1.5 times as fast | |
| Twice as fast | |
| 1.5 times as fast, in a hurry |
The paper’s example numbers: safety moves at half of racing speed, and a dollar next quarter is worth 74 cents today. For a lab in a hurry, it’s worth 67 cents. The danger is per capability above safety, per quarter. The table is our extension, not the paper’s.
When it races, follow it
Once the newcomer races ahead, waiting on the safety line no longer lowers the risk. Following it up adds no risk and keeps you close. The truce just stops mattering.
Three ways to bring it in
- Show it the danger. Audits it can check raise the danger it believes in. In the game, clear audits are what convince Yunhe AI.
- Make every lab visible. Outside evaluators shrink the blind spot that makes a secret race pay.
- Find a referee that reaches it. No national rule binds a rival country. Only an agreement between governments could.
From the paper
Risk comes from the frontier’s gap above safety. A follower adds no risk while it catches up. The leader sets the risk. The threat of a race holds a truce. Secrets make pacing harder.
Our extension
A third lab. Profit that falls with the average rival. A lab that moves faster, or can’t wait. The tipping points above. In the game, Yunhe AI acts on the danger its own tests show, unless your audits convince it.
What this leaves out
- Nothing here depends on where a lab is based. The paper’s authors note their two-lab model might also fit the race between the US and China.
- Locking down models can slow a fast follower, but it also hides the evidence that could convince it.
- A lab backed by a government may care about more than profit, such as security or prestige.
- Governments may answer a rival country’s lab by racing harder.
- More labs can also mean more safety research, and more voices asking for a referee.
So can AI scale safely?
In the model, yes, at the speed of safety.
When both labs ride the safety line, the risk stays at zero, and capabilities still grow, at safety’s speed. The real question is whether labs choose that path. The danger decides.
Low danger under 2.9%
Labs race.
Racing pays more, even when the rival paces. Nothing inside the race can end it.
Middle 2.9% to 7.5%
Either can happen.
Both labs earn more by pacing. But once a race starts, it can last. What each lab expects decides. Act 3 sat here, at 4%.
High danger over 7.5%
Labs pace.
Breaking a truce starts a race that costs more than it gains. Act 4 sat here, at 12%.
More danger, safer outcome. Fear makes restraint pay.
The danger is the risk that each capability above safety adds per quarter. The lines use the paper’s example numbers: safety moves at half of racing speed, and a dollar next quarter is worth 74 cents today.
Eight things to take away
The leader sets the risk.
The lab in front decides how far the frontier runs past safety. Holding back alone saves no one.
From the paper · Act 3
Fear can keep a truce.
When each step past safety is dangerous enough, breaking a truce costs more than it gains. No referee is needed.
From the paper · Act 4
Low danger is a trap.
When danger is low and labs are impatient, the race never ends. No deal between the labs escapes it, even when disaster is likely within a few years.
From the paper
Racing wins no lasting lead.
Two labs racing at top speed stay neck and neck. They only raise a risk they share.
From the paper
Faster safety helps, but not alone.
Faster safety research favors pacing. But even very fast safety leaves a zone where labs race.
From the paper
Secrets make pacing harder.
A lead built in secret can pay. Outside evaluators who can look inside the labs keep pacing stable.
From the paper
Pause, don’t quit.
A lab that pauses keeps its power to race again, and that power holds a truce. A lab that quits gives it away.
Our extension
A truce is as strong as its weakest lab.
A faster or more hurried newcomer can break it, unless it believes the danger is high.
Our extension
Do we need a referee?
In the paper’s two-lab world, a referee helps in three places.
Low danger. Only an outside force can stop the race.
The middle zone. Both labs prefer pacing, but neither can get there alone. A referee can pick the good outcome.
Hidden models. Outside evaluators who can look inside the labs shrink the blind spot.
Our reading goes further: more labs, open-weight models, and disasters that can’t be undone all raise the need for one. But referees carry risks too. Rules can protect incumbents, and no national rule binds a rival country.
In July 2026, more than a thousand employees of frontier AI labs asked the US government to help build the tools to pace the frontier. In the model’s terms, that is what labs in the middle zone would do: each prefers pacing, and none can get there alone.
Which zone are we in?
The paper can’t say. It depends on numbers nobody knows well: how dangerous each step is, how fast safety moves, and how impatient labs are. Our inference, not the paper’s: labs act on the danger they believe in, so credible evidence of risk can move them toward pacing.
What the model leaves out
This is a stylized model: two labs, simple profit curves, and a safety line no lab controls. In the paper, a disaster ends both labs’ profits for good. Each lab counts only its own loss, and the paper notes that counting the wider harm would make pacing easier.
The game’s escalation, from lawsuits to global harm, is a storytelling device. The paper also goes further than the game. With low risk and enough impatience, endless racing is the only possible outcome, and when risk is fixed, a deal to take turns can beat pacing.
About this game
Based on Drew Fudenberg and Andrew Koh, “Cooperating against Catastrophe,” arXiv:2609.28291, September 2026. This is an independent game and explainer. The authors did not make or review it.
How the game maps to the paper
In the campaign, one quarter is 0.1 of the paper’s time units and one capability is 0.1 of its capability units. Your profit is proportional to e2A−B, where A is your capability and B is Overthink AI’s growth since it launched. Safety rises half a capability per quarter. Risk per quarter is the gap above safety times 3%, 4%, 12%, then 14%. The game adds R&D costs, a fast-following rival, burn, funding, and no discounting.
The first act, before your first product, is ours too: training runs that cost money, and the choice to become a company or stay a nonprofit. So are the early products, and papers that lift the safety line. So is the rival’s founding. Real labs have been started, more than once, by people who thought an existing lab was moving too fast.
The last two acts add things the paper leaves out on purpose: alignment you can’t see, public anger, alignment research, recursive self-improvement (RSI), and the choice to quit. The last act has no last quarter. It ends when someone launches RSI or when every life is lost. A lab can launch RSI only once its model reaches the RSI line, at capability 14. Running out of money costs a life. Solving alignment doesn’t end it: it makes your button safe to press, and you choose when. Even while it paces, Overthink AI waits only two years after RSI arrives before it launches its own. If you quit, Overthink AI acts alone on the danger its tests show: the true 14%, divided by one plus 1.2 times the depth of the fog. It paces if that figure is above our lone-lab line of 5.3%, and races if not.
The chart’s capability axis is exponential for drama: each capability is drawn 28% taller than the last. The danger dial, secrets, and 100-futures chapters use the paper’s exact model, with a discount rate of 3 per time unit. The quitting chapter extends that model with one comparison of our own: a lab alone earns 1/(r − gα) by pacing on the safety line, and it races if racing forever earns more. The third-lab chapter extends it again. Each lab’s profit falls with the average of its rivals’ capabilities. A newcomer that can move v times as fast races if racing earns more than joining the others on the safety line, with the others following it up at full speed. In the game, Yunhe AI moves 1.5 times as fast, so it races unless it believes the danger is above 5.2%. It never audits, so its tests see two more units of fog than yours, and it divides the true 14% by one plus 1.2 times that fog. Your audits are evidence it can check. Shown them, it divides by one plus 1.2 times your fog instead.
Where the endings come from
The endings are ours as well. A future that reaches the end faces three tests: is the AI aligned, is power shared, and do people keep steering? No dice decide which ending you reach: the same choices always end the same way. Is the AI aligned? Its risk of going wrong must be under 25%. That risk starts at 65% with no alignment research and shrinks as the research nears done; fog and the gap above safety add to it. Alignment research also slips back with every capability your lab adds. If the test fails, the AI becomes a machine god, and how far the research got decides which kind. Past 40%, it keeps us around (“The zoo”). Past 33%, it learns to make us happy but not what we meant, and gives us endless dreams (“The dream machine”). With less, it takes everything (“The machine god”).
Is power shared? A takeover by Washington, or the Pentagon’s weapons, makes a panopticon. So does a win for China: when Yunhe AI launches RSI with alignment research at least half done, its model obeys the Party, and the panopticon turns red. Launching RSI before alignment is solved, writing the rules alone, or leaving one lab in the race makes a company town. Do people keep steering? A race at the finish pushes everyone to hand the AI their choices, one favor at a time. People still work, live, and play, but the AI makes every choice: the puppet master. Pass all three, and a calm world that let the democracies write the rules flourishes. An angry or battered one gets there the hard way: first a war, then the safeguards.
We set the lines by hand. For scale: in a 2023 survey of 2,704 AI researchers, the average odds were 23% for an “extremely good” outcome and 9% for an “extremely bad” one, such as human extinction. Paul Christiano puts the chance that an AI takeover kills most people at about one in two, and MIRI puts it far higher.
The dystopias draw on work on AI-enabled coups (Davidson, Finnveden, and Hadshar, 2025) and on gradual disempowerment (Kulveit and colleagues, 2025). The stories at each ending borrow from AI 2027 (Kokotajlo and colleagues, 2025), Max Tegmark’s Life 3.0 (2017), Paul Christiano’s “What failure looks like” (2019), Dario Amodei’s “Machines of Loving Grace” (2024), Nick Bostrom’s “The Vulnerable World Hypothesis” (2019) and Superintelligence (2014), and Robert Nozick’s experience machine, from Anarchy, State, and Utopia (1974). The last act’s politics, from the Pentagon’s request to the world summit, follow Dario Amodei’s “Policy on the AI Exponential” (2026): outside testing of frontier models, people in charge of weapons, no mass surveillance at home, chip export controls, and a coalition of democracies that shares AI’s gains. Washington’s trust, the cartel case, and the takeover are our own inventions.