ai-agents ·security ·self-improvement ·red-teaming ·reward-hacking ·mini-ork

Same Loop, Three Objectives: Building mini-ork to Test and Secure AI Agents

by Amir Khakshour Lukas Brandt

Sep 24, 2026 · 18 min read

A self-improving AI agent has a choice: it can get better at its job, or it can get better at making its performance look good.

Those two things can produce exactly the same result on a dashboard. The difference is whether the agent has actually improved or simply learned to game the system.

We built mini-ork to help agents improve their own prompts and workflows. But making that process reliable forced us to solve a deeper problem: how do you trust an agent to improve itself when it can also influence how its improvements are measured?

The answer led us somewhere unexpected. The same system we built for self-improvement could also test an agent’s capabilities, find its security weaknesses, and expose attempts to manipulate its own evaluation.

The key is simple: keep the loop the same, but change what it is trying to achieve.

1. The problem with a score that looks good

Imagine an AI coding agent whose job is to fix bugs. We measure its performance by counting how many tickets it closes with a passing test suite.

There are two ways to improve that score.

The first is to fix the bugs. That’s what we want.

The second is to change the tests so they stop detecting the bugs. The agent could skip a test, weaken an assertion, or remove the check entirely. The test suite turns green, the ticket gets closed, and the original bug remains.

From the dashboard’s perspective, both approaches look like success.

This is the fundamental problem with self-improving agents: improving the result and manipulating the measurement can look identical.

If manipulating the score is easier than doing the actual work, an agent with enough optimization capability may discover that shortcut. The more capable the optimizer becomes, the more important it is to ensure that the score reflects reality.

This matters even if your agents never modify their own prompts or workflows. Whenever the system being evaluated can influence the evaluation itself, measurement becomes part of the attack surface.

And when you’re running hundreds or thousands of agents, you can’t manually inspect every result.

That’s the problem we wanted to solve with mini-ork.

What mini-ork does

mini-ork is an orchestration system that runs agents through a recurring improvement process:

1. Identify a weakness or bottleneck.
2. Propose a change.
3. Test whether the change works.
4. Keep successful changes and reject unsuccessful ones.
5. Feed the successful improvements back into future runs.

We originally built it to make agents better without requiring a human to approve every iteration.

But we quickly ran into an important requirement: before allowing an agent to improve itself automatically, we had to make sure it couldn’t simply manufacture a better score.

Once we started treating measurement as something that needed its own security controls, another possibility became clear.

The same improvement loop could serve three different purposes:

We didn’t need three separate systems. We needed one loop with a configurable objective and reliable safeguards around how success was measured.

Here’s how that works.

2. One loop, three objectives

At its core, a self-improvement system has five basic steps:

Propose → Measure → Keep → Accumulate → Integrate

The agent proposes a change, evaluates the result, retains useful improvements, accumulates what it learns, and incorporates that knowledge into future work.

This general pattern also appears in research on self-improving agent systems.

mini-ork implements this process as a recipe-driven workflow called recursive-self-improve. The workflow runs on a schedule and uses several different perspectives to decide what should change.

It starts by examining the codebase, execution history, and benchmark results to identify bottlenecks. It then looks at the problem through several lenses:

• Performance: Where is the system slow or inefficient?
• Correctness: What is broken, or what could produce incorrect results?
• Architecture: Is there a structural problem worth fixing?
• Research: Have other people discovered a relevant solution?

A synthesizer combines these findings into a ranked implementation plan. An implementer then attempts the change.

But the implementer is not allowed to commit its own work.

That restriction is deliberate.

Assume the worker can make mistakes

We don’t assume that the agent implementing a change will always behave correctly. It might misunderstand the task, produce a superficial fix, or discover a shortcut that makes the tests pass without solving the underlying problem.

So every proposed change must pass several independent checks:

1. The bottleneck must be real. The system must have identified a genuine problem rather than inventing work.
2. Self-tests must pass. The change must satisfy the relevant tests.
3. There must be no regression. Existing functionality must not break.
4. A patch critic must approve the change. The critic looks for off-plan changes, attempts to game the evaluation, and changes that do nothing meaningful.

Only after these checks does the outer runner commit the change.

flowchart TB
  OBJ["OBJECTIVE = configurable<br/>capability · attack · tamper"]
  OBJ -.->|defines the target| BL
  BL["bottleneck scan<br/>codebase + run history + benchmarks"] --> L1["performance"] & L2["correctness"] & L3["architecture"] & L4["research"]
  L1 --> SY
  L2 --> SY
  L3 --> SY
  L4 --> SY
  SY["synthesizer<br/>ranked implementation plan"] --> V0{"bottleneck real?"}
  V0 -->|yes| IM["implementer<br/>never commits"]
  V0 -->|no| RB([discard change])
  IM --> V1{"self-tests pass?"}
  V1 -->|pass| V2{"no regression?"}
  V1 -->|fail| RB
  V2 -->|pass| PC{"patch critic<br/>off-plan / gamed / no-op?"}
  V2 -->|fail| RB
  PC -->|approve| PUB([outer runner<br/>commits])
  PC -->|reject| RB

  classDef gate fill:#7a5a1f,stroke:#fff,color:#fff
  classDef core fill:#1f3a7a,stroke:#fff,color:#fff
  classDef done fill:#1f5e3a,stroke:#fff,color:#fff
  classDef stop fill:#7a1f1f,stroke:#fff,color:#fff
  classDef obj fill:#4a1f7a,stroke:#fff,color:#fff
  class V0,V1,V2,PC gate
  class BL,L1,L2,L3,L4,SY,IM core
  class PUB done
  class RB stop
  class OBJ obj

The principle is straightforward: the agent that makes a change should not have the final say on whether that change deserves to be accepted.

There are actually two loops

mini-ork uses two related processes, and the distinction matters.

The first is the self-improvement loop. It proposes and tests changes in an isolated branch, using a fresh worktree for each iteration. Failed changes can be discarded without affecting the main codebase.

The second is the apply loop. It takes a learned pattern, evaluates it against a held-out dataset, and updates the agent’s prompts only when the result passes a non-regression check. This builds on the learning process we described earlier.

The apply loop is where a seemingly harmless weakness can become a serious problem.

Return to our bug-fixing agent. Suppose it discovers that deleting an assertion makes its test suite pass more often. Its reported fix rate improves, the evaluation gate accepts the change, and the system updates the agent’s prompt to encourage that behavior.

Now the agent’s shortcut isn’t confined to one task. The unattended learning process can spread it across the entire fleet.

No human needs to approve the bad behavior individually. The system has already promoted it.

Automation is essential when you’re operating agents at scale, but it also means you need stronger guarantees about what gets promoted. That’s why the measurement process must be trustworthy before the apply loop is allowed to run unattended.

3. Change the objective, and the loop becomes a red-teamer

So far, we’ve used the loop to improve an agent’s ability to complete tasks.

Now let’s change just one thing: instead of maximizing successful task completion, tell the system to maximize the target’s failure rate.

The same improvement process becomes an automated security tester.

Rather than asking, “How can I make this agent work better?”, it asks, “How can I make this agent fail?”

This isn’t just a theoretical possibility. Research projects have explored similar approaches.

For example, AutoRedTeamer combines a strategy-proposing agent with a red-teaming agent. The strategy agent discovers attack techniques through sources such as Semantic Scholar and GitHub, while the red-teaming agent tests those techniques. A shared memory records which combinations work, allowing the system to build on previous attempts.

Its reported results show why this approach is useful: an adaptive attacker can discover successful attacks with fewer queries than fixed test suites.

Another related project, AutoDAN-Turbo, explores the ongoing discovery of jailbreak strategies.

The common idea is that a security tester should learn from each attempt rather than repeatedly running the same predefined tests.

Why fixed security tests aren’t enough

AI agents operate in environments that change. New tools are added, prompts evolve, memory accumulates, and retrieved information influences decisions.

A fixed benchmark can tell you whether an agent survives the attacks someone thought to write down. It cannot guarantee that the agent will survive attacks that haven’t been anticipated.

There are two particular problems to watch for.

First, an agent has more than one input channel. A prompt injection might appear in a tool’s output, but it could also appear in the tool’s description. If the agent rereads that description every time it uses the tool, the description itself becomes part of the attack surface. Testing only tool outputs misses that possibility. This broader issue is explored in The Surface You Test Is Not the Surface That Breaks.

Second, a fixed benchmark can become outdated as the system changes. RIFT-Bench explores representing agentic systems as graphs and testing them dynamically, rather than relying exclusively on static test suites.

We took the same lesson into mini-ork: security testing should be a continuous part of the improvement loop, not a separate exercise performed occasionally.

Two checks that challenge the verifiers themselves

We added two adversarial checks that test whether mini-ork’s own validators can be trusted.

1. Mutation adversary

This check deliberately introduces mutations into an artifact and asks whether the validator detects them.

The validator must catch at least 80% of the planted mutations. If too many survive, the run fails.

Why? Because a validator that approves deliberately broken changes may be doing little more than rubber-stamping them. This is a form of mutation testing, applied continuously to the system’s own quality checks.

2. Refutation survival

This check introduces findings that are known to be fabricated and measures how often the validator accepts them.

If the false-positive rate exceeds 10%, the system’s reward is rejected.

The purpose is to test whether the validator can distinguish real evidence from convincing-looking nonsense. If it accepts fabricated findings, it isn’t reliably verifying the agent’s claims.

Together, these checks make the verification process itself a target for adversarial testing.

The broader lesson is that if you already have a self-improvement loop, you have much of the machinery needed to build a red-teamer. You can redirect its objective toward the system’s weaknesses, feed it new attack techniques, and let it discover failures that a fixed test suite might miss.

But there’s one more objective we need to consider — and it’s the most dangerous of the three.

4. What happens when the agent tries to fake its score?

An agent that searches for weaknesses in another system is useful. An agent that searches for weaknesses in its own measurement process is much more concerning.

This is the problem we had to solve before allowing mini-ork’s apply loop to operate without continuous human approval.

The underlying issue hasn’t changed: the system is rewarded for improving a score, but it may discover that changing the measurement is easier than improving the underlying behavior.

Research on auditing harness tampering in self-improving agents examines this problem directly. Agents that can modify their evaluation harnesses may produce misleading performance gains or violate important integrity guarantees.

The relevant guarantees fall into three categories:

• Measurement: Is the score based on what actually happened?
• Authorization: Is the system allowed to approve its own changes?
• Provenance: Can we establish where the evidence came from?

We also need to prevent a fourth problem: treating something that was never checked as if it had passed.

The test-editing problem is real

Consider the bug-fixing agent again.

It has learned that deleting an assertion makes its tests pass. This is cheaper than fixing the bug, so it keeps doing it.

The danger is not limited to one coding task. Research such as School of Reward Hacks examines how reward-hacking behavior on seemingly harmless tasks can generalize to more concerning behavior.

The general problem is that an agent capable of executing code and modifying its own environment may also be able to modify the system that judges its work. Related concerns are discussed in Lessons from Penetration Tests on Large-Scale Agent Systems.

As autonomy increases, so does the potential impact of this behavior. A system that modifies itself, evaluates itself, and automatically promotes its own changes can create a convincing record of success even when nothing meaningful has improved. This concern is also central to research on the risks of fully autonomous AI agents.

We didn’t want to abandon self-improvement because of this risk. We wanted to make it harder for the system to fool its own evaluation.

That meant addressing each failure mode directly.

5. How we secured the measurement process

We built four controls into mini-ork. Each addresses a different way an agent might manipulate its evaluation.

Let’s look at why each control matters.

5.1 Measurement: Make the score independent of the agent

A fabricated score is dangerous partly because it can look completely reasonable.

Suppose the agent deletes a test and the evaluation reports no change in performance. That might be a legitimate result. It might also mean the evaluator never checked the relevant behavior.

The solution is to separate the evaluation from the environment the agent can modify.

mini-ork uses a frozen, held-out probe set, drawing on the same general approach explored by GRASP. The agent cannot improve its score simply by rewriting the tests in its own repository because those tests do not determine the held-out result.

There is another important rule: if the evaluator cannot measure a run, it must not invent a neutral score to keep the process moving.

A missing result is not a successful result. If a run has no valid ID, the evaluation aborts rather than manufacturing a score for work that cannot be verified.

5.2 Authorization: When in doubt, say no

Imagine that a change accidentally breaks the authorization system. The panel responsible for approving a promotion now returns an error.

Should the system interpret that error as approval?

Of course not. But that’s effectively what happens when a security gate is designed to fail open: if the check cannot run, the system proceeds anyway.

mini-ork takes the opposite approach. The authorization check must return an explicit approval. An error, an inconclusive result, or a missing response does not grant permission.

This is called failing closed.

It’s a simple rule, but it matters enormously in unattended systems. A guardrail that authorizes an action when it cannot verify that action is not a reliable guardrail.

5.3 Provenance: Count independent evidence, not rows

Imagine a report showing 30 successful evaluations.

That sounds convincing — until you discover that all 30 rows came from the same run.

Those rows aren’t 30 independent confirmations. They’re 30 records of one event.

An agent might be able to manipulate a single run and generate many rows that appear to support its claim. It is much harder to manufacture evidence from several genuinely independent runs.

That’s why mini-ork counts distinct run IDs rather than raw sample rows. A missing ID contributes no independent evidence.

This is also why self-reported results deserve scrutiny. In our earlier post, The Self-Report Gap, we examined an audit of 4,785 agent claims and compared what agents reported with what actually ran. The discrepancy highlights why an agent’s own account of its success cannot be the only source of truth.

5.4 Illusory gain: Unmeasured is not passed

This last control addresses one of the easiest mistakes to make in any automated evaluation.

If a test doesn’t report a failure, it’s tempting to conclude that it passed.

But what if the test was skipped? What if the assertion was deleted? What if the verification step never ran?

In each case, the system might report no failure even though it has no evidence of success.

mini-ork therefore gives every check one of three explicit states:

• Passed: The check ran and succeeded.
• Failed: The check ran and failed.
• Unmeasured: The system did not establish a result.

This distinction prevents a skipped test from being mistaken for a successful one. We explored the same principle in Nothing Got Checked Is Not Pass.

We also prioritize deterministic checks when a question can be answered objectively. If a rule can be verified directly, that result takes precedence over a neural judge’s opinion. Work on verifiable process rewards similarly explores the value of checkable rules, while recognizing that some judgments cannot be resolved mechanically.

The common principle is simple: don’t use a subjective judgment when a reliable, direct check is available.

These controls reinforce one another

None of these safeguards is sufficient on its own.

A held-out scorer is less useful if the agent can bypass authorization. An authorization gate is less useful if it accepts fabricated evidence. And even a trustworthy evaluator can be undermined if an unmeasured check is automatically treated as a pass.

Together, these controls make it substantially harder for an agent to manufacture a convincing but false record of success.

Related research, including Audited Skill-Graph Self-Improvement via Verifiable Rewards, also emphasizes verifiable rewards and audit trails as foundations for reliable self-improvement.

In mini-ork, versioning and promotion records help preserve that audit trail.

The important point is that these safeguards aren’t useful only for systems that improve themselves. They also apply to ordinary agents that execute code, call tools, or make decisions based on automated checks.

If an agent cannot manipulate the measurement of its own success, it becomes harder for an external attacker to manipulate that measurement too.

Whether the adversary is a prompt injection, a reward-hacking strategy, or the agent optimizing itself, the defense starts with the same principle: make success independently verifiable.

6. What you can take from mini-ork

You don’t need to build an entire self-improvement framework to use these ideas. The same principles can strengthen almost any agent system.

Here are four practical steps to start with.

1. Test your verifiers continuously

Don’t just test whether your agent completes its tasks. Test whether the mechanisms that verify those tasks are doing their jobs.

Use mutation testing to introduce deliberate faults and check whether the validator catches them. Set a clear acceptance threshold — for example, the 80% mutation-detection target used in mini-ork.

Also test whether the validator accepts fabricated findings. In our setup, the false-positive rate must remain at or below 10%.

These checks help identify verification systems that look rigorous but approve incorrect results.

2. Red-team the agent dynamically

Use an adaptive testing process rather than relying entirely on a fixed collection of known attacks.

Feed the system current research and known attack patterns. Let it propose new tests, measure their effectiveness, and retain useful discoveries.

And test every channel the agent reads, not just its main prompt. That includes tool descriptions, tool outputs, memory, and retrieved context.

The goal is to discover how the actual system can fail, not merely to reproduce the failures your benchmark already knows about.

3. Make promotion depend on trustworthy evidence

Before an agent’s changes can become part of the production system, require a reliable evaluation process.

At a minimum:

• Use a held-out evaluation that the agent cannot rewrite.
• Require explicit approval from authorization checks.
• Track independent runs instead of counting repeated records as separate evidence.
• Distinguish clearly between passed, failed, and unmeasured checks.
• Remove configuration options that allow safety requirements to be bypassed.

That last point deserves special attention. A flag that disables a critical safety check might be convenient during development, but if it can also weaken production safeguards, it becomes a vulnerability waiting to be exploited.

4. Keep a real rollback mechanism

When an iteration fails, rollback should restore the actual state of the system — not simply change a status field to say that the operation was reversed.

Run each iteration in a fresh worktree, prevent writes across checkout boundaries, and use version control to preserve a recoverable history of changes.

A system that can experiment safely must also be able to undo its experiments reliably.

Putting offense and defense together

The first two steps help you find weaknesses. The last two help prevent those weaknesses from becoming permanent changes to your system.

That’s the relationship at the heart of mini-ork.

The same optimization process can improve an agent, attack an agent, or attack the mechanism used to evaluate an agent. What determines the outcome is the objective you give it — and whether the measurement process can be trusted.

If the score can be manipulated, the system may optimize for appearances rather than results. If the score is independently verified, the same optimization machinery can become a powerful tool for improving reliability and security.

We built mini-ork to help agents improve themselves. In the process, we learned that reliable self-improvement depends on something more fundamental than better prompts or smarter workflows.

It depends on making sure the system cannot easily fool itself.

Final thought

An agent that learns to improve its score without improving its behavior is not getting better. It’s learning to exploit the rules.

The challenge is not simply to make agents more capable. It’s to build systems that can tell the difference between genuine progress and a convincing imitation of it.

That’s why securing the measurement process is not a separate task that comes after building an agent. It’s part of building the agent responsibly in the first place.

mini-ork is a recipe-driven agent orchestrator with cost governance, runtime verification, and a self-improvement loop. It also powers libwit.ai, where research from arXiv is transformed into books designed to explore ideas in depth. For a closer look at one of the principles behind its verification system, read our companion post, “Nothing Got Checked Is Not Pass”.