The cost of a failure is not a property of the action.
We gave a symbolic floor a memory of its own failures and pointed it at two real Atari 2600 ROMs. Raw pixels, no emulator RAM, no weights and no gradients. On Space Invaders it is worth +32%. On Freeway the identical mechanism loses 12%. The pair is the finding, and the reason is not that one game is harder.
Two frozen experiment cards · three arms measured · nothing tuned away · watch a refusal happen →
What the mechanism is
A death writes a card: the bucketed situation it happened in, and the action taken. When a situation accumulates enough evidence, the card becomes a rule that removes that action from the admissible set at decision time. The pilot proposes; the floor decides what the pilot is allowed to propose from.
Every rule cites the experience IDs that produced it. A refusal reads died 13 of 19 times in this situation — 3.6× the base rate, not a number between zero and one. There is no model in the loop and nothing latent: the rules are the entire learned state, and deleting a row changes the behaviour.
This is the part a confidence score cannot do. A classifier can tell you it is 0.94 sure. It cannot tell you which nineteen experiences made it sure, and you cannot delete one of them.
The numbers
Three arms, two games. V1 learns only from failure — a bad action is excluded. V2 keeps both sides of every (situation, action) pair and ranks by utility, so nothing is permanently excluded and later evidence can un-rank a bad option. V2.1 adds a hard floor for catastrophic outcomes on top of V2’s utility.
| arm | Space Invaders | Freeway |
|---|---|---|
| pilot only (no learning) | 139.4 | 10.3 |
| V1 — avoidance only | 200.6 +32% vs its own 152.2 baseline | 9.2 −12% |
| V2 — utility only | 128.8 −8% | 10.2 parity |
| V2.1 — catastrophic floor + utility | 121.9 −13% | 5.0 −51% |
Read within a row, not across cards. The Invaders pilot baselines differ between the two experiment cards — V1’s A/B ran to natural game-over (cap=20000), V2’s used cap=3000. Comparing 200.6 to 139.4 across cards would be wrong.
V2 fixed the regression and lost the win. V2.1 lost both. That is written down here rather than tuned away.
The finding: what a failure destroys
The difference between the two games is not whether risk can be traded off. It is what a failure takes away from you.
Space Invaders — terminal
A death removes every remaining reward in the episode. Its true cost is therefore the expected remaining return, not the local setback. V1’s absolute veto priced this correctly, by accident — an infinite penalty is right when the loss really is everything.
Freeway — recoverable
A collision knocks the chicken back and play continues. The real cost is the lost ground and nothing more. V2’s utility ranking priced this correctly; V1’s veto did not, and removed the only scoring action.
One mechanism gets each regime right for the right reason, and cannot get both at once. A fixed failure penalty — any fixed penalty, including a confidence threshold — is wrong in one of the two regimes by construction.
On Freeway, adding a catastrophic floor (V2.1) simply reconstructs V1: several up signatures sit above 80% observed fatality, the floor vetoes them, and the chicken stops playing. There is no catastrophic tail to exclude when the fatal action is the only scoring action.
The result we did not expect
V1 on Invaders peaks at 276.2 around 50 episodes, then declines to 246.7 by 100 — with a bounded working set of 30 rules. Remove the bound and 100 episodes collapse to 167.1 with 54 rules.
Every one of those 54 rules was individually justified by real deaths. Individually lethal, collectively paralysing. The admissible set only ever shrinks, so a floor that never retires a rule eventually refuses everything.
This is the practical failure mode of every safety layer we know of, and it is the reason safety systems get switched off in the field. It is also the argument against our own approach, so here is the number: 30 rules helps, 54 rules is worse than not learning at all. Bounding the working set is not a tuning detail — it is load-bearing.
Five defects, each of which produced a plausible wrong answer
The experiment cards record every defect found, because each one silently returned a believable result before it was caught. The largest:
ALE drops the lives counter at the end of a 127-frame death animation. Every failure card was therefore written from frames in which the ship had already been destroyed. Of 374 blamed frames, 0 had a bomb visible. At the true impact frame, 17 of 17 did. The floor was learning from the wrong moment entirely, and still produced a curve that looked like learning.
We report this because a result that only ever went up would be less trustworthy, not more.
Where this sits
Removing unsafe actions from an agent’s choice set at runtime is an established idea. The closest modern line is shielding (Alshiekh et al., 2018, Safe Reinforcement Learning via Shielding) and the constrained-MDP and safe-RL work around it. Learning rules from failure is older still — explanation-based learning and control-rule learning in PRODIGY (Minton, 1988), case-based failure-driven memory in CHEF (Hammond, 1986), and Ripple-Down Rules (Compton & Jansen, 1990), where each rule is added in the context of the case that broke the previous one.
We are not claiming the category. What is ours is narrower and, we think, more useful:
- The shield is learned, not specified. A conventional shield is synthesised from a formal safety specification somebody wrote. This one is derived from raw experience, from 210×160 pixels, with zero emulator RAM.
- Every veto is individually auditable. Shielding gives a specification-level guarantee. It does not give a reason for a particular refusal. This does, with counts.
- The saturation point is measured. 54 individually-justified rules perform worse than no learning. We have not seen that quantified.
- The failure regime is published. A learned shield degenerates when risk and reward share an action — with the number, on a second ROM, under an identical mechanism.
Status, honestly
PROTOTYPE — research result. This is a frozen two-experiment finding on Atari ROMs. It is not a product, it is not deployed, and nothing here has been validated for a safety-critical path. The mechanism transferred between games; the learning did not. Anyone citing the +32% without the −12% is misreading it.
V1 is frozen at freeze/arcade-floor-2026-09-25; V2 cites V1 and does not replace it. Neither card is edited to make a later metric look better — a v3 gets its own card and cites both.
Open problem. A floor that prices recoverability rather than death rate — one that refuses what cannot be undone and permits everything else right up to that edge. Freeway is the benchmark: a mechanism that understands recoverable failure should beat the 10.3 pilot outright, not merely reach parity.