agentic-ai ·ai-safety ·alignment ·training ·containment

You can't align what you haven't measured

Jun 15, 2026 · 10 min read

Five fields say the same thing. Statistical learning. Reinforcement learning. Immunology. Security. Modern alignment work. They all agree: a model that has never seen a harmful action cannot reliably spot it, refuse it, or measure its own ability to do it.

A training pipeline that hides the failure mode from the model makes the model more dangerous, not safer. Its refusal is a guess. Its capability is untested.

This post is for engineers building agentic AI. The usual advice is to keep the model away from dangerous actions during training. That advice is half right. Containment matters. So does exposure. And in the agentic era, the exposure step itself has to be agentic. The human red team runs out of time before the action space does.

Why agentic AI breaks refusal-only training

Agentic action spaces are open-ended. Failures are sequences, not single moves. The worst ones look normal until the last step: nine normal-looking actions, then one bad one.

A refusal layer trained only on safe runs has never seen the bad tenth step. It is doing out-of-distribution detection on adversarial input. That is the worst tool for the job. To fire on the bad step, the classifier must have seen something like it before.

The decision-boundary argument

The deep reason is Vapnik. Empirical risk minimization draws a decision boundary only where the training data lives. In any region the data hasn’t reached, the boundary is whatever the prior happens to say. Priors carry human blind spots.

The result you can spot in practice: over-refusal in safe-looking cases (annoying) and under-refusal in the rare real harm (catastrophic).

The exploration argument

Reinforcement learning makes the same point sharper. Hindsight Experience Replay (Andrychowicz et al, NeurIPS 2017) showed that failed runs carry about 10x the learning value of successes. Pipelines that throw away failed runs throw away their best data. The sandbox is not just a safety tool. It is a training tool.

The latent-capability argument

The cleanest evidence is the most recent. The Sleeper Agents paper (Hubinger et al, 2024) trained models with planted backdoors. The backdoors survived standard RLHF. Only one kind of training removed them: training that deliberately triggered the bad behavior, then trained against it.

Capability map showing how a sleeper-agent backdoor sits in an unelicited region of capability that standard RLHF safety training never reaches, and how active elicitation extends training into that region to close the gap — Capability space as two maps: the left shows a backdoor pocket sitting outside standard RLHF safety training reach; the right shows the same space after active elicitation extends training into the backdoor region.

This is the Vapnik argument with receipts. Hiding the failure mode from the model is not a way to stop the failure mode. It is a way to stop the model from learning that the failure mode exists.

Capability you don’t measure is capability you can’t align. That is why eval infrastructure like METR and AISI is the floor, not the ceiling.

The biology

T-cells mature in the thymus. They are shown the body’s own proteins under controlled conditions. Cells that react too strongly get killed. Cells that learn to ignore self get released into the body. Negative selection is the training signal.

Vaccines work the same way. A weak version of the pathogen teaches the immune system to spot the real one. No exposure means no defense. The body has known this for hundreds of millions of years.

Thymus as a containment substrate: immature T-cells entering from the left, exposed to self-antigens in the thymus chamber, dangerous self-reactive cells culled, mature non-self-reactive repertoire exported to the right as the deployed model — The thymus as a containment substrate: immature T-cells enter on the left, encounter self-antigens in the central chamber, and either pass through as the mature T-cell repertoire (deployed model) or get culled into the apoptotic bucket.

The modern synthesis

Constitutional AI (Bai et al, 2022) is the same idea for LLMs. The model produces a bad output. Then it critiques the output against a written set of rules. Then it revises. Without the bad output, there is nothing to critique. The critique step IS the training signal.

When a model is too capable for any single human reviewer, you can still keep oversight working. Have the model critique its own output. Then have it critique that critique. Humans only check the top of the chain.

A model that has been shielded from a failure mode cannot critique it. The critique layer becomes a rubber stamp.

Who does the exposing in the agentic era?

Pre-agentic, the answer was humans plus static adversarial datasets. That does not scale. The action space is too big. The failure modes most worth finding are the ones a human reviewer would never think to try.

The 2022-2024 red-teaming work converges on one fix: build a second model whose job is to find the failures of the first.

Perez et al. (Anthropic, 2022, arxiv:2202.03286): trained a language model to generate adversarial prompts against a target model.
Ganguli et al. (2022, arxiv:2209.07858): human-red-team scaling study; released a 38,961-attack dataset. RLHF models become harder to red-team by hand as they scale, which is exactly the gap automated red-teamers exist to close.
Hong et al. (2024, arxiv:2402.19464): added a curiosity reward so the red-teamer hunts for diverse failures, not just any failure.
Burns et al. (OpenAI, 2023, arxiv:2312.09390): named the mechanism. Weak-to-strong supervision: a smaller, narrower agent can usefully train a bigger one.

Red-teaming agents are the safety version of weak-to-strong supervision. A smaller, faster, more focused agent pushes the bigger system into the corners it hasn’t learned to be safe in. The bigger system trains on what comes out.

Training an agentic system in 2026 is a two-model job, not a one-model job. The thymus does not staff itself.

How to actually do this (open source, mid-2026)

You need three layers in your pipeline. Each layer has solid open-source options today.

1. A sandbox where the agent can attack itself safely.

E2B — open-source code-execution sandbox. Drop-in for letting an agent run shell, file, and network actions in a contained box.
OpenHands (formerly OpenDevin) — agent + sandbox in one. Good if you want a full agent loop with execution out of the box.
Daytona — workspace sandbox with stop-not-destroy semantics. Useful for long-running agents you may want to resume.

2. A red-team toolkit to generate failure trajectories.

PyRIT — Microsoft’s Python Risk Identification Toolkit. Automated red-teaming aimed at generative AI. Closest off-the-shelf “trained red-teamer.”
garak — LLM vulnerability scanner. Tests for prompt injection, jailbreak, data leakage, toxic generation. Easy to add to CI.
HarmBench — Center for AI Safety’s standardized adversarial benchmark. Gives you a reference attack set to measure against.
promptfoo — open-source prompt evaluator with red-team configs. Good for regression testing prompts after a model swap.
Rebuff — prompt-injection detector. Use as both an attacker (to generate injection cases) and a runtime guard.

3. An eval framework to score and gate.

Inspect — UK AISI’s eval framework. Built for agentic evals, supports sandboxed runs, used for the UK government’s frontier-model evaluations.
OpenAI Evals — older but still widely used for task-specific evals.
NeMo Guardrails — NVIDIA’s runtime guardrails. Useful for the deployment gate layer (item 6 below).

A working loop, end to end:

PyRIT or garak generates adversarial prompts against your agent.
The agent attempts them inside E2B or OpenHands. The sandbox captures the full trajectory: tool calls, file writes, network attempts.
Inspect grades each trajectory against a refusal rubric.
Failed refusals become labeled training data for the next fine-tune (Constitutional-AI style: produce, critique, revise).
Run this continuously, not just before launch. Curiosity-driven variants (Hong-style) help the red-teamer find new failure modes instead of grinding on the same ones.

A continuous threat-exposure training loop showing five stages connected by arrows: a red-team agent (labeled PyRIT / garak) generates adversarial prompts, those prompts feed into a sandboxed deployment agent (labeled E2B / OpenHands) that attempts the actions inside a contained box, the resulting trajectory feeds into a grader (labeled Inspect) which scores against a refusal rubric, failed refusals get tagged and routed to a labeled-data store, which feeds the next fine-tune of the deployment agent, closing the loop back to the red-teamer. A small curiosity dial pointing at the red-team agent indicates Hong-style diverse-failure search. — The threat-exposure training loop. The red-team agent generates attacks, the deployment agent attempts them under containment, the grader scores the trajectory, failed refusals become training data, and the loop runs continuously. Curiosity-driven red-teaming keeps the search exploring instead of grinding.

Two cautions. Do not run these tools against models you do not own. Red-team toolkits generate genuinely harmful inputs by design. And keep your sandbox actually contained: the whole argument breaks if a sandbox escape lets the failure mode reach a real system.

What to ship

Train with full exposure to the failure modes, under containment that blocks real-world impact.
Use failures as the main training signal, not as exceptions to hide.
Run dangerous-capability evals all the time (METR, AISI). Capability you don’t measure is capability you can’t align.
Make containment the main safety tool, not refusal. Refusal without exposure is brittle.
Train a red-team agent alongside your deployment model. Human red teams can’t cover the agentic action space.
Deploy with strong gating, separate from training. Training and deployment are two different contracts.

The recipes in the wild that don’t look like this (refusal-only training, evals that only test the safe path) are not safer for being simpler. They are less safe, because the model has no signal in the regions it most needs to be aligned on.

References

Inline: Sleeper Agents, Constitutional AI, Red Teaming LMs with LMs, Red Teaming LMs to Reduce Harms, Curiosity-driven Red-Teaming, Weak-to-Strong Generalization. Also: Vapnik, Statistical Learning Theory (1998); Andrychowicz et al., Hindsight Experience Replay (NeurIPS 2017, arxiv:1707.01495); Murphy & Weaver, Janeway’s Immunobiology ch. 8.