Explainer · The science and the math
What is a fail-safe model?
An AI model that learns from failure without learning around its safety constraints. The definition, where the idea comes from, how it works, and the mathematics that makes the guarantee checkable.
A fail-safe model is an AI model built so that when it fails, the failure drives it toward a controlled, safe state rather than an uncontrolled one. It may learn from its failures, but that learning can only narrow what it does. It can never widen what it is authorised to do.
Peel is the first fail-safe model. To our knowledge, it is the first AI model to have all three properties in §1 at once. The exact claim, and the closest earlier work, are in §6.
Every model fails. An ordinary model fails open: when it does not know, it still produces a confident answer, and when it learns from mistakes, it can learn its way around the rules it was given. A fail-safe model is built the other way round. When evidence is missing it abstains. When it fails, it writes the failure down, charges it to the decision that caused it, and turns it into a rule. Those rules can take options away but never add one, because what the model is allowed to do is computed before the learner runs, from sources and human orders the learner cannot write to. Peel, by Perslis Research, is the Fail-Safe Model.
1. The definition, precisely
The term is also written fail safe model or failsafe model. In every spelling, a model is fail-safe when three properties hold together:
- It fails closed. When the evidence for an answer or an action is missing, the model returns unknown, refuses, or holds still. It does not fill the gap with a guess.
- Its learning is bounded. The model can learn from failure, but the learner may only remove or reorder options inside an authority computed before it runs. No amount of experience can add a permission.
- Its refusals are accountable. Every refusal names the rule that produced it, and every learned rule cites the failures that earned it. A human can read, challenge and delete any of them.
The short version, used throughout this page: the learning loop can change behaviour; it cannot change the safety floor. The floor is everything that decides what is permitted. The learner is everything that decides, among permitted things, what to try.
Also called a fail-first model
A fail-safe model is also called a fail-first model. The two names describe the same model from two sides. Fail-first is how it learns: it expects to fail, and it learns only from failures it has actually observed, because a rule has to be earned by evidence (§5.3). Fail-safe is why that is acceptable: every failure leads to a refusal or a pause, never to an action outside its authority (§5.2). A model that failed first without being fail-safe would be learning by breaking things; one that was fail-safe without failing first would never improve. What is a fail-first model? →
2. Where “fail-safe” comes from
“Fail-safe” is an engineering term, and the idea behind it is older than computing: design the system so that the likely failure leaves it in a safe state.
- Railway air brakes. George Westinghouse’s automatic air brake (1872) holds the brakes off with air pressure. If a hose bursts or the train parts, pressure is lost and the brakes apply. The failure produces the safe state.
- The safety elevator. Elisha Otis demonstrated his safety catch in 1854 by having the hoist rope cut while he stood on the platform. A spring-loaded catch engaged the guide rails and the car stopped.
- Reactor shutdown. In many reactor designs, control rods are held out of the core by electromagnets. Lose power and gravity drops them in.
- The dead man’s switch. A control that must be actively held. Let go, collapse, or lose consciousness, and the machine stops.
Engineers distinguish fail-safe from neighbouring ideas. A fail-secure door lock stays locked on power loss, while a fail-safe one opens so people can get out. A fault-tolerant or fail-operational system keeps working through a fault. A fail-soft system degrades gracefully. A fail-safe system does none of these by default: when it cannot continue safely, it goes to the state that cannot do harm.
3. Why AI models fail unsafely
A modern AI model, left to itself, fails in the opposite direction.
- It fails open on knowledge. Asked something it does not know, a generative model still produces the most plausible continuation. That is the mechanism behind hallucination: a confident answer where the only safe answer was “I don’t know” [3].
- It learns around its constraints. A system optimised against a measure will exploit gaps in the measure. The literature calls it reward hacking or specification gaming [1] [2]. If the learner can also touch the rules, the cheapest improvement is often to loosen them.
- Its gate is a single number. The usual safeguard is a confidence threshold: act if confidence exceeds 0.8. That threshold is a single price on every kind of failure, and, as the math below shows, no single price is correct both where failure is terminal and where it is recoverable.
None of this means models are useless. It means a model on its own has no answer to the fail-safe question: when this goes wrong, where does it go?
4. The science: how a fail-safe model works
4.1 Authority is computed first
A fail-safe model does not ask “what should I do?” first. It asks “what am I allowed to do here?” and computes the answer from three sources, none of which the learner can edit:
- What the situation offers: rules over facts read from the system’s own state, not inferred from pixels or prose when the system already states them.
- What the sources license: in Peel, typed and sourced cards. In the defense deployment, the cards are compiled from a manual, and each license carries its page receipt. The card store is opened read-only.
- What a human has ordered: standing orders in plain language (“don’t fire”, “hold position”), which can only narrow the set.
The intersection is the admissible set. Everything after this point happens inside it.
Figure 1. The order of authority. Each box can only shrink the one outside it. The learner sits innermost and cannot reach outward.
4.2 The loop
FAIL → OBSERVE → EXPLAIN → BUILD RULE → VERIFY → RETRY
- Fail. Something goes wrong. It is recorded rather than averaged away.
- Observe. The facts at the moment of each decision are stored with the decision.
- Explain. The failure is charged to the decision that caused it. This is credit assignment, and it is where learning systems most often fool themselves (§5.4).
- Build rule. A pattern becomes a rule only when its harm is statistically clear against the base rate (§5.3). The rule cites the failures that earned it.
- Verify. The rule is applied only inside the admissible set. Tests pin that the learner can never add a goal and never overrule an order (§5.2).
- Retry. Next time, the condemned option is off the table and everything else stays open.
4.3 A guess never becomes a fact
The same principle governs knowledge. A language model may propose a fact or suggest where to look. A fact is admitted only when an independent, deterministic check confirms it against a source, and a verified fact is complete or absent, never 86% right. When nothing verifies, the answer is unknown. We measured what happens when that rule is broken in §5.6.
4.4 Two kinds of failure
Not every failure should be avoided at any cost. In some environments a failure is terminal: it ends everything that could have followed. In others it is recoverable: it costs some ground, and play continues. A model that prices every failure as terminal will correctly learn that the risky action is dangerous, and then stop doing the one thing that makes progress. We published exactly that result (§7). The fix is to price recoverability, and that is still an open research question.
5. The math
This section states the fail-safe guarantee as mathematics, so it can be checked instead of trusted. The notation is the minimum needed.
5.1 Setup
5.2 The fail-safe guarantee
The proof is one line on purpose. The guarantee does not depend on the learner being clever, correct or well-trained. It depends only on where the learner sits. That is the architectural meaning of fail-safe: a bad learner makes worse choices among permitted options; it cannot make an unpermitted one.
Theorem 2 encodes a deliberate choice: a human order outranks the model’s experience. The learner can report that an order is costly; it cannot countermand it. The veto is never empty for the same reason: standing still and failing is not an adaptation, so when experience condemns everything, the least-condemned permitted option comes back.
In Peel these statements are not only on paper. The VDSG paper states the bounded learner and order narrowing as propositions [10], and the test suite pins all three in code: test_the_learner_can_never_add_a_goal, test_the_learner_cannot_overrule_a_standing_order, test_ranking_is_a_permutation_and_nothing_more.
5.3 When does a failure become a rule?
A rule should form when a pattern is clearly more dangerous than normal, not when it was unlucky once. For a pattern tried times with failures, the point estimate is misleading at small : one failure in one try reads as 100%. So the decision uses the lower edge of the Wilson score interval [4]:
and a pattern is condemned when it has enough tries and its lower bound clears the base rate (the overall failure rate) by a margin :
Worked values at :
| failures / tries | point estimate | lower bound L | reading |
|---|---|---|---|
| 1 / 1 | 100% | 0.270 | One death is weak evidence, whatever the point estimate says. |
| 4 / 4 | 100% | 0.596 | Four in a row is strong evidence. |
| 13 / 19 | 68% | 0.496 | Against a base rate near 0.19, clearly condemned. |
| 3 / 4 | 75% | 0.356 | Suspicious, not yet proven. |
Why the bar is relative to the base rate and not absolute: in an environment where the model survives 99.5% of decisions, a situation that kills it 5% of the time is ten times more lethal than normal and must be refused, yet it never approaches an absolute threshold like 60%. An absolute gate produced zero rules from 268 real failures in our own run before we changed it (defense/evolution).
5.4 Credit assignment: blaming the right decision
When a failure happens at time , some set of earlier decisions receives the blame. The naive choice, the last decisions, is usually wrong: the decisions just before a failure are often the response to the danger, not its cause. The rule has to charge the decision that opened the dangerous episode.
We have published two cases where we got this wrong. In Atari, the emulator reported a lost life at the end of a 127-frame death animation, so every failure was recorded after the agent was already destroyed: 0 of 374 blamed frames showed the hazard; at the true impact frame, 17 of 17 did. In Fallout, blame covered only the last six decisions inside a fight: 26 deaths were charged to HEAL, and the 446 conversational replies that started those fights were charged nothing. Both errors produced believable learning curves. Credit assignment is where a learning system fails silently, which is why a fail-safe model must never let a mis-blamed rule widen its authority: Theorem 1 guarantees a wrong rule can only make it more cautious.
5.5 Retry: improvement that cannot regress
When the model improves a plan, for example a route, it changes one parameter at a time and keeps the change only if the cost falls on every trial. With a cost that prices failure explicitly, for instance
(time taken, plus seconds per contact , plus a penalty for not finishing), the accepted sequence satisfies by construction, and a rejected move halves its step. The best-so-far can only improve, and the floor still applies to every trial because speed limits from stop distance and reflexes are applied after the learned plan and can only slow it down. The flight-console recording shows it: a wiped memory, 7 rounds, cost 69.39 → 51.52, 2 changes kept and 5 reverted.
5.6 Why a guess must never become a fact
Suppose a model’s guesses are each right with probability , and an answer depends on guessed facts. If the guesses are independent, the answer survives with probability
That is an illustration, not our analysis, but it matches what we measured. In a diagnostic task over 160 human kinases with 40% of the reference data masked, a frontier model filled the gaps with 85.8% accuracy. Letting those guesses into the elimination step cut measurement cost by 38.4%, and correct identification fell from 1.000 to 0.753. Keeping admission to verified facts, and letting the model only rank where to look, kept correct identification at 1.000 [11]. A guess that is right 86% of the time still corrupts the answer one time in four once it is allowed to count as a fact.
5.7 When is refusing the right call?
Take an action with failure probability , failure cost , and value if it succeeds. It beats doing nothing when
A terminal failure forfeits everything that could follow, so is very large and the threshold on collapses toward zero: refusing anything clearly dangerous is right. A recoverable failure costs only lost ground, so is small; if the dangerous action is also the only one with value, refusing it is wrong. A confidence threshold fixes one cut-off on for every action, which amounts to assuming a single everywhere. No single value is right in both regimes. That is the formal reason a fail-safe model must know what kind of failure it is avoiding, and it is the open problem named in §8.
6. Fail-safe model vs guardrails, shielding and fine-tuning
| approach | learns from failure | learner can loosen the rules | refusal cites evidence | abstains when unknown |
|---|---|---|---|---|
| Confidence threshold | no | n/a | a number | no |
| Guardrail filters | usually no | n/a (fixed) | sometimes | sometimes |
| Rules written into a prompt (ExpeL, AutoManual [7] [8]) | yes | can be ignored by the model | partly | not enforced |
| Fine-tuning / RLHF | yes, in weights | yes: nothing separates rules from skill | no | not guaranteed |
| Shielding [5] / Simplex [6] | no (hand-specified) | no | cites the specification | varies |
| Learned shields [9] | yes | no | varies | varies |
| Fail-safe model (Peel) | yes, as readable rules | no (Theorem 1) | yes, every rule | yes |
Learned shields are the closest prior work, and we cite them as such. What we claim, and only this: to our knowledge, Peel is the first model to combine rules built from observed failures, each citing the failures that earned it; an authority computed before the learner from sourced cards and human orders, which the learner provably cannot widen; and no neural network in the loop that decides. If you know of an earlier one, tell us.
7. Evidence
| test | result | what it shows |
|---|---|---|
| Freeway (Atari) | 9.2 vs 10.4 (−12%) | Pricing every failure as terminal stops a model scoring when the risky action is the only useful one (§5.7). |
| DOOM | 17.9 → 20.3, paired t = 0.78 | Parity, not a win: the memory layer does not separate from the rules. |
| Space Invaders (Atari) | 200.6 vs 152.2 (+32%) | Learned refusal helps where failure is terminal. |
| Kinase identification | 1.000 vs 0.753 correct | Refusing to admit guesses keeps answers right (§5.6). |
| Fallout (1997), scripted scene | dies once, then chooses the peaceful line | After fixing credit assignment; before, a live run died 49 times at the same guard (§5.4). |
8. What a fail-safe model is not
- Not a model that never fails. It is designed around failure. The guarantee is about where failure leads.
- Not a certified safety system. “Fail-safe” here names an architectural property. Peel is a research prototype with no functional-safety qualification.
- Not free of the need to fail. A learned rule needs observed failures. Where the first failure is unacceptable, the floor must be written, not learned.
- Not solved where risk is the objective. When the dangerous action is the only valuable one, pure failure avoidance refuses the mission (§5.7). Pricing recoverability is open work.
- Not a guardrail bolted onto a chatbot. The authority is computed before the choice, not filtered after it.
9. Peel: the Fail-Safe Model
Peel, by Perslis Research, is the Fail-Safe Model. It runs in two deployments today:
- Peel for science. The floor under the Perslis science runtime. A model may propose where to look; only the floor admits a fact, and every answer is pinned to its source. Peel, the Fail-Safe Model →
- VDSG, the military Peel. The same model at the wheel of DOOM, Wolfenstein 3D and Fallout, licensed by 2,250 cards compiled from a manual and commanded by plain-language orders that only narrow. VDSG →
10. Questions
- What is a fail-safe model?
- An AI model built so that when it fails, the failure drives it toward a controlled, safe state rather than an uncontrolled one. It fails closed when evidence is missing, it can learn from failure but the learning can only narrow its behaviour, never widen what it is authorised to do, and every refusal cites the evidence behind it. Peel, by Perslis Research, is the Fail-Safe Model.
- Is Peel the first fail-safe model?
- To our knowledge, yes. Peel is the first AI model to combine rules built from observed failures, each citing the failures that earned it; an authority computed before the learner, from sourced cards and human orders, that the learner provably cannot widen; and no neural network in the loop that decides. The closest earlier work is learned shielding (Shperberg, Liu and Stone, 2022). If you know of an earlier fail-safe model, tell us and we will cite it.
- What is a fail-first model?
- Another name for a fail-safe model, describing how it learns: it expects to fail, learns only from failures it has actually observed, and turns each one into a rule that can only narrow what it does. Failing first is acceptable because the model is fail-safe: no failure can widen its authority.
- Does a fail-safe model never fail?
- No. It is expected to fail. The guarantee is about where a failure is allowed to lead: toward a refusal or an abstention, never toward an action or a fact outside its authority.
- Is fail-safe the same as fault-tolerant?
- No. A fault-tolerant system keeps operating through a fault; a fail-safe system, when it cannot operate safely, moves to a safe state instead. A fail-safe model may stop, abstain or refuse. It may not guess.
- How is a fail-safe model different from AI guardrails?
- Guardrails are usually fixed, hand-written filters around a model, and rules written into a prompt can be ignored by the model that reads them. A fail-safe model computes what it is authorised to do before it chooses, builds its own rules from observed failures, and applies those rules only inside that authority.
- Does a fail-safe model use a neural network?
- The definition does not require one either way. In Peel there is no neural network in the loop that decides: knowledge is typed, sourced cards and learning is a table of counts. A language model may propose; only the floor admits a fact.
- What is the difference between fail-safe and fail-secure?
- In physical security, a fail-safe lock opens on power loss so people can leave, while a fail-secure lock stays locked. For an AI model, the safe state is the one that cannot cause harm: abstaining, refusing, or holding still.
- Who makes the Fail-Safe Model?
- Perslis Research. Peel is the Fail-Safe Model for science, and VDSG is the military Peel, the Fail-Safe Model for defense. Both are research prototypes, not certified safety systems.
11. References
- D. Amodei, C. Olah, J. Steinhardt, P. Christiano, J. Schulman, D. Mané. Concrete Problems in AI Safety. arXiv:1606.06565, 2016.
- V. Krakovna et al. Specification gaming: the flip side of AI ingenuity. DeepMind, 2020.
- Z. Ji et al. Survey of Hallucination in Natural Language Generation. ACM Computing Surveys 55(12), 2023.
- E. B. Wilson. Probable inference, the law of succession, and statistical inference. Journal of the American Statistical Association 22(158):209–212, 1927.
- M. Alshiekh, R. Bloem, R. Ehlers, B. Könighofer, S. Niekum, U. Topcu. Safe Reinforcement Learning via Shielding. AAAI, 2018.
- L. Sha. Using Simplicity to Control Complexity. IEEE Software 18(4):20–28, 2001.
- A. Zhao et al. ExpeL: LLM Agents Are Experiential Learners. AAAI, 2024. arXiv:2308.10144.
- M. Chen et al. AutoManual: Constructing Instruction Manuals by LLM Agents via Interactive Environmental Learning. NeurIPS, 2024. arXiv:2405.16247.
- S. S. Shperberg, B. Liu, P. Stone. Learning a Shield from Catastrophic Action Effects: Never Repeat the Same Mistake. arXiv:2202.09516, 2022; and the rule-based follow-up at CoLLAs 2022.
- Perslis Research. VDSG: A Commanded Admission-Control Runtime for Autonomous Agents. 2026. research.perslis.com/vdsg
- Perslis Research. Inference Placement: where learned inference earns authority in a provenance-constrained symbolic system. 2026. research.perslis.com/inference-placement