Deep DiveAugust 2026~40 min

The Evolution of the Agent Harness

A four-year story: each harness layer fixed the failure the last one exposed. From a prompt-only loop (2022) to Model + Harness as the unit of capability (2026).

Agent = Model + Harness. Read this deck as that growth path, not a model-quality timeline.

Deep dive · ~40 min

Agenda

01The arc: failure → layer 02What a harness is 032022: Loop lives in the prompt 042023: Tools become a contract 052024: Coding agent as product 06MCP + code-as-action 072025: Context, scale, isolation 08Across runs: sessions & control 09Context as a variable (RLM) 10Anatomy of a modern harness 11Evals & Model+Harness 12Self-edit, co-train, thesis

Section 01 · Arc

Four years of harness growth

Story rule for this deck: each stage answers “what broke last time?” and ships a new layer around the same ReAct skeleton.

2022 Prompt loop CoT → MRKL → ReAct

2023 Tool contract Function calling, frameworks

2024 Product surface CLIs, MCP, code-as-action

2025 Long-run machinery Context, isolation, RLM

2026 Harness as the unit Benches, self-edit, co-train next

Same circle the whole talk: each year adds a ring. The model stays in the center.

2022+ prompt loop

2023+ tool contract

2024+ product surface

2025+ context & isolation

2026+ self-edit & benches

Section 02 · Definition

What is an agent harness?

The software around an LLM that turns it into an agent: prompts, tools, the control loop, context and memory, permissions, and (often) sessions or skills. Distinct from the model’s own reasoning.

Agent = Model + Harness

Job of a harness: situate the model — prompts, tools, loop, context, permissions, and what may be edited. Watch this circle gain a ring on each chapter: same model, thicker harness.

Core model Loop ReAct Tools contract Product CLIs / MCP Outer context, isolation, skills, self-edit

Section 03 · 2022

The prompting era: the loop lives in text

Chapter start: before there is a “harness,” everything is text. No architecture change yet. Reasoning, routing, and action are all steered by what you put in the prompt.

Jan 2022 Chain-of-Thought Reasoning made visible via exemplars

May 2022 MRKL LLM as one component among external modules

Oct 2022 ReAct Reason → act → observe becomes the skeleton

Section 03 · Foundations

Two preconditions for the harness

Chain-of-Thought (Jan 2022)

Wei et al.: few worked examples that spell out intermediate steps sharply improve complex reasoning. Gains show up on zero/few-shot math and reasoning evals (e.g. grade-school math) that reward intermediate work, not only a final token.

Premise every later harness builds on: a model’s internal state can be steered by text alone. Evals already set the target.

MRKL (May 2022)

AI21’s neuro-symbolic framing: the language model is one component routed to discrete modules (calculators, APIs, databases) as needed.

Named the limitation harnesses exist to fix: LLMs are fluent but can’t do arithmetic or fact-lookup reliably. Direct ancestor of tool calling and MCP.

Section 03 · ReAct

ReAct fused thought and action

Yao et al. (Oct 2022) interleaved reasoning traces with actions so a model could think, act on an external environment, observe the result, and think again. Eval pressure: on ALFWorld (environment success, not a string match) it beat imitation and RL baselines by 34 absolute points with 1-2 examples.

1Reason

2Act

3Observe

4Repeat

↻ still the skeleton of every harness today

Reasoning traces help the model induce, track, and update action plans; actions let it interface with external sources to gather information.ReAct, 2022: the loop, in one sentence

Section 04 · Tools

Making tool use reliable

Next failure: free-text ReAct actions are brittle to parse. 2023 moves tool decisions into models, then into a structured contract the harness can dispatch.

Feb 2023 Toolformer Model learns when and how to call APIs

Jun 2023 Function calling Structured JSON contract replaces free-text parse

Nov 2023 Assistants API Managed threads, sandboxed Code Interpreter, retrieval

Path: free-text ReAct actions → Toolformer (in-model) → OpenAI function calling (typed contract).

Section 04 · Contract

Function calling gave the loop a contract

Developers describe functions; the model emits a structured JSON object naming the function and its arguments. Fragile text parsing became a machine-readable interface.

By DevDay (Nov 2023): parallel calls in one message, plus JSON mode that guaranteed valid output.

Why it mattered

Dispatch, not guesswork

The harness no longer had to guess intent from prose. It received a typed call it could dispatch directly. Major frameworks reorganized their tool layers around this contract within months.

Before Harness guessed tool intent by parsing free-text “Action: …” lines

After Model emits a typed function name + arguments; harness only dispatches

Section 04 · Nov 2023

Assistants: context becomes a platform job

OpenAI’s Assistants API is an early hosted harness surface: persistent threads, server-side context handling, Code Interpreter, and Retrieval. Context management stops being only the app’s problem.

Threads

Managed history

Long-running conversation state offloaded to the platform, not stuffed into one prompt by hand.

Tools

Hosted execution

Sandboxed Code Interpreter and retrieval as product features of the loop, not bespoke glue.

Why it matters

Harness, not demo

Marks the shift from “script a ReAct loop” to “ship an agent runtime customers sit inside.”

Section 04 · Frameworks

The framework explosion, and its limits

Productization

LangChain

Packaged ReAct as AgentExecutor. Early Action Agents: input → tool → observation → history → repeat.

Autonomy

AutoGPT

Self-planning goals, unattended runs. Exposed task drift, looping, and token burn from no memory discipline.

Separation

Plan-and-Execute

Planner vs executor to fight prompt bloat. Precursor to subagent isolation and offloading intermediate steps.

The lesson that shaped the next two years: raw autonomy without context management and structure is unreliable. SWE-bench (Oct 2023) already scores model + scaffold on real GitHub issues.

Section 05 · 2024-25

The coding harness becomes the product

Next failure: demos and notebook agents are not daily engineering. Frameworks abstract the loop; coding products make it installable: repo, shell, git, permissions, project files as default context.

Lineage

From scaffold to product

SWE-agent and agent-computer interfaces (2024) make “how the agent touches the machine” first-class. Aider stays git-native. Cursor, Claude Code, Codex CLI, OpenHands, Cline, OpenCode, pi become daily harnesses.

What they add

Batteries for real work

Repo maps, edit tools, test loops, PR workflows, permission modes, sessions on disk. The harness, not a notebook script, is what developers install.

Concurrent

Multi-agent frameworks

AutoGen / Crew-style systems parallel the coding CLI wave: multi-role orchestration before “subagents” go mainstream in coding products.

Eval link

SWE-bench era

SWE-bench (Oct 2023) and SWE-bench Verified (Aug 2024) already score model+scaffold. Terminal-Bench 1.0 (May 2025) and 2.0 (Nov 2025). Agent quality is measured inside a harness long before 2026.

Section 06 · MCP · Nov 2024

MCP standardized tool integration

Next failure: every agent–tool pair needs a custom connector (N×M). MCP (25 Nov 2024) is a client–server bus over JSON-RPC 2.0 so any compliant agent can use any compliant tool (USB-C for AI).

Anthropic deliberately reused Language Server Protocol ideas. Adoption: OpenAI Mar 2025, Google DeepMind Apr 2025; donated to the Agentic AI Foundation (Linux Foundation) Dec 2025.

Before: N×M connectors

Every agent × every tool

MCP

USB-C

After: one standard bus

Agent AAgent B

JSON-RPC bus

Tool XTool YTool Z

Section 06 · Action surface · late 2024

When JSON tools are too rigid: act in code

Fixed function schemas compose poorly for multi-step data work. The next move is not more tools: it is letting the model write the action as code.

Failure

JSON tool calls hit a ceiling

Each step is one named function + args. Filtering, looping, and intermediate transforms either burn tokens through the model or force a combinatorial tool zoo.

smolagents · late 2024

Code as the action language

Hugging Face CodeAgent: ReAct steps are Python snippets, not JSON tool calls. Optional sandboxed exec. The harness runs the code and returns results to the loop.

What changed

Action surface, not the whole harness

Code-as-action upgrades how the agent acts on tools and data. It does not yet rewrite prompts, skills, or subagent specs. That is a later chapter.

What it leaves open

History is still a window

Long runs still pack transcript and tool I/O into the prompt. Compaction and offload come next. Treating context itself as a programmable variable (RLM) arrives in late 2025.

Section 07 · Context

Every long run overflows a finite window

Next failure: long product runs fill a finite window. History, tool I/O, and intermediate steps cause overflow and context rot. Managing this becomes the defining 2025 harness job.

History

Tool I/O

Room

Overflow

Transcript Tool results Working room Context rot zone

Failure mode

Context rot

Degraded performance as the window fills: not a hard crash, a soft loss of focus and fidelity.

Insidious failure

Goal drift

After lossy summarization, the agent forgets the user’s intent. Summaries drop details unpredictably.

Section 07 · Compaction

Offloading and compaction became standard

Answer to overflow: push big tool results and stale history out of the window, keep a short working set, and leave the full transcript recoverable on disk.

Pattern from production systems (e.g. LangChain Deep Agents SDK) at rising thresholds:

~20k tokens Offload results Write to disk; keep path + preview

~85% window Offload inputs Truncate old args to pointers

Still full Summarize Intent, artifacts, next steps; full transcript on disk

Lossy summary + recoverable filesystem is the standard recovery pair.

Section 07 · Progressive disclosure · Nov 2025

Too many tools → code APIs

Next failure: hundreds of tool schemas and every intermediate result through the model burn context. After code-as-action and long runs, progressive disclosure becomes standard (Nov 2025 “code execution with MCP”).

Before All schemas in Hundreds of tool defs in the prompt

Pattern Code APIs MCP servers as a filesystem

On demand Discover + filter Only needed tools reach the model

Result 150k → 2k 98.7% token save in one documented example

Section 07 · Subagents

Subagents keep the main context clean

Next failure: one loop does everything and bloats itself. Isolation: the main agent hands side work to a subagent with its own prompt, tools, and permissions, and keeps only a summary.

Main agent (clean context)

Explore Read-only · denied Write/Edit

Plan Isolated thinking · returns plan

Custom Markdown-defined · least privilege

Only a summary returns to the main context. Descendant of 2023 plan/execute with true isolation.

Section 07 · Orchestration

From loop to operating system

Once many workers share a task, a bare ReAct loop is not enough. High-end systems add an explicit control plane:

Planning Goal decomposition, todos, prioritization

Policy Enforcement of permissions and approval gates

State / knowledge Durable memory, artifacts, session stores

Observability Quality checks, traces, retries, call limits

Protocols MCP for tools · emerging A2A for peer coordination

Section 07 · Sandboxing

Instructions aren’t security; the OS is

Next failure: prompt “don’t touch X” is not security once the agent has a real shell. Sandboxes are not new (Code Interpreter 2023); local coding agents made OS enforcement urgent.

A regex banning git loses to bash -c or a renamed binary; poisoned AGENTS.md made the risk concrete. Serious products wired Seatbelt / bubblewrap (or a remote box) into the agent runtime.

macOS

Seatbelt

OS-level profiles restricting filesystem and network.

Linux

Bubblewrap

Namespaces and isolation around the agent process.

Windows

WSL

Contained environments for agent shell access.

Subagents get independent least-privilege tool scopes; high-impact actions gated by human approval. Sandboxing is defense-in-depth: it can’t stop a script the agent tricks you into running outside it.

Section 08 · Across runs

Sessions, skills, and project judgment

Next failure: capability dies when the process exits. The harness stops being single-run: sessions, project judgment, and skills persist as files the next session loads.

Sessions

JSONL / resume

Append-only transcripts: rewind, resume, recover mid-task without starting from zero.

Project files

AGENTS.md / CLAUDE.md

Team judgment loaded at boot: style, architecture, “done” criteria. Converges on portable formats (e.g. AGENTS.md under AAIF-era standards).

Skills

Procedures on disk

Folders of instructions + scripts, progressive disclosure by description. Agents can author skills; humans version them in git.

Memory

What to keep

Auto-saved learnings and cross-session notes: not full chat dump, but curated harness state.

Section 08 · Control surface

Hooks, plan mode, verification loops

Modern coding harnesses add deterministic control the model cannot ignore, plus plan-before-act and test/review loops.

Hooks

Lifecycle automation

Pre/Post tool use, stop, session start: shell or policy that always runs. Permissions and hygiene that prompts alone cannot guarantee.

Plan mode

Read-only think first

Separate planning from edits (Claude Code plan mode, Cline-style plan/act). Reduces thrash on large changes.

Close the loop

Verify in the harness

Run tests, lint, agent self-review, multi-reviewer subagents, Ralph-style re-inject-until-done. Outcome checks become part of the loop, not a human afterthought.

Section 09 · Late 2025 · RLM

Context stops being only a window

Compaction and offload keep long runs alive but are lossy if everything must re-enter the prompt. Late 2025 answer: treat context as a variable the model can program over.

Failure mode

Window + compaction ceiling

History and tool I/O either burn tokens or get summarized away. Files on disk help, but the model still cannot query, slice, and transform its own past as code without reloading it into the window.

RLM · Oct–Dec 2025

Recursive Language Models

Zhang et al. (blog Oct 2025; arXiv 2512.24601, 31 Dec 2025): put the long prompt in an external environment (a REPL variable). The model writes code to inspect it and can call sub-LMs recursively over slices.

Idea

Context as a variable

Not “stuff more into the window.” Hold full history and data as programmable state. Compact the active prompt when needed; keep the rest addressable in the REPL.

What comes next

Product + self-edit in 2026

Prime Agent productizes RLM (IPython as the main surface, subagents as rlm(...)). Self-editing harnesses are a later move.

Section 10 · Vocabulary · late 2025

Framework vs runtime vs harness

By late 2025 the stack was thick enough to need names. Not a 2022 invention: a late label for a split that grew with the field. Without it, “we built an agent” meant three different jobs.

Layer	Job	Fixed which failure	Examples
Framework	Abstractions to assemble agents	Everyone reimplements ReAct by hand	LangChain
Runtime	Durable multi-step execution	Long runs die, drift, or lose state	LangGraph-style durable runs
Harness	Batteries-included model environment	A library graph is not a daily coding agent	Claude Code, Codex, Deep Agents, Prime Agent

Frameworks build. Runtimes keep runs alive. The harness is what the model sits in each turn, and what this talk evolves. Self-editing later rewrites the harness, not the framework API.

Section 10 · Product loop

Still ReAct; everything else is scaffolding

Checkpoint: after all the layers above, the product loop is still ReAct. Claude Code frames it as gather context, take action, verify results. Everything else is scaffolding around that skeleton.

Execution

Environment

Tools, persistent filesystem, code execution. Isolation is optional enforcement, not the harness.

Context

Management

Summarization, offloading, cross-session memory, on-demand skills.

Planning

Delegation

Todos and subagents in isolated contexts; plan mode before edits.

Safety

Control

Permissions, hooks the model cannot skip, human approval, verification loops.

Section 10 · Assembled

What a harness contains

Model Reasons, acts, observes

Harness Always around the model

System prompts

Defaults and instruction structure each turn

Tools, skills, MCP

Capability surface and progressive disclosure

Orchestration loop

Reason → act → observe (still ReAct)

Context machinery

Compaction, offload; context as variable (RLM)

Sessions & project files

JSONL, AGENTS.md, cross-run state

Subagents

Isolated contexts; return a summary

Hooks & plan mode

Deterministic lifecycle; plan-before-act

Permissions

Least privilege, human approval

Verification

Tests, review agents, stop conditions

OS sandbox (Seatbelt / bubblewrap / remote box) is harness enforcement when a shell would ignore prompts. Not every product ships it.

Section 11 · Evals co-evolved

Evals were there from the start

Harness progress was always eval-driven. What changed is not that evals appeared, but what they ask for and how much of the system they can see.

Prompt era Zero-shot / few-shot Score the model under a fixed prompt. CoT wins because evals reward intermediate steps.

Tool era Call correctness Right API, right args. Toolformer and function-call suites.

Agent envs Task completion ALFWorld (ReAct), SWE-bench (2023), Terminal-Bench (2025): environment outcome, not string match alone.

Today Scope keeps growing Real deliverables and traces; then Model+Harness as the unit (Harness-Bench 2026).

Still the point

Validate the output

Oracles, unit tests, rubrics, security gates: evals check whether the work is done. Scope expanded so checks match real agent work.

Scope grew

What they measure

Text answer → tool args → multi-step env success → process quality → which harness produced the run.

Section 11 · Unit of capability

Same model, different harness, different score

Agent leaderboards never measured a naked model. What matured is treating the harness as an explicit experimental and product axis.

Fixed harness

Isolate the model

SWE-bench Verified often freezes a scaffold (e.g. mini-SWE-agent) so model ranks are comparable. Still model + harness — just held constant.

Variable harness

Harness-Bench (May 2026)

106 sandboxed workflow tasks; thousands of trajectories. Fix task/budget/evaluator; vary harness native behavior. Large gaps across harnesses under the same model pool.

Claim

Report the pair

Capability is a model-harness configuration, not a model name alone. Climbing the board means climbing the wrapper.

Infra

Taskset × harness × runtime

Prime Intellect Environments Hub and hosted evals (and peers) separate environment, harness, and runtime for RL and public leaderboards.

Section 12 · Self-editing · 2026

Harnesses that rewrite themselves

After RLM made context programmable, the next failure was static harness state. Three tiers: static (human-owned), accumulating (skills/memory grow), self-editing (automated refine of harness state or code).

Product

Prime Agent (Aug 2026)

Ships RLM (REPL) and Continual Harness: CRUD on prompts/memory/skills/subagents; /refine from trajectory. Claimed ARC-AGI-3 95.5% Best@1 with Opus 5 (protocol-bound). No model trained on it yet.

Product

Hermes Agent

Autonomous skill creation/refine from complex tasks; cross-session memory. Accumulating/self-improving skill library; optional external optimizers (e.g. DSPy-style) with review gates.

Research

Self-Harness, AHE, Compass

Weakness mining → propose → validate; observability-driven evolution; component-wise optimization (e.g. SWE-bench Verified gains under paper protocols).

Research

Meta-Harness & more

Search over harness code; Continual Harness paper (embodied, mid-episode CRUD); The Last Harness You’ll Ever Build (meta-evolution). 2026 is crowded with self-edit loops.

Section 12 · Papers

Self-editing is a research program

arXiv 2605.09998

Continual Harness

Karten et al.: reset-free mid-episode CRUD on harness state (prompt, subagents, skills, memory). Abstraction Prime Agent productizes for coding.

arXiv 2606.09498

Self-Harness

Agent improves its own harness via weakness mining → bounded proposal → validation (no stronger outer agent required).

arXiv 2605.27922

Harness-Bench

Diagnostic: harness as primary axis under fixed tasks. Capability is a configuration, not a model name alone.

Also

Compass, AHE, Meta-Harness

Component-wise evolution; observability-driven coding harnesses; searching harness code for Pareto improvements. Updating ability ≠ benefit across model tiers.

Section 12 · Next

Next: train models for a harness

Self-editing improves the wrapper around a fixed model. The open frontier is model-harness co-learning: weights that expect a given loop, tool surface, and context regime.

Today

Self-edit the scaffold

Refine skills, prompts, subagents from traces without changing weights.

Gap

No co-training yet

Prime Agent states no model is trained around its RLM/Continual Harness interface; features under-used until then.

Train (or distill) models for a harness while the harness adapts. Large remaining gains are expected.

Section 13 · Pattern

Each layer fixed the failure of the last

The whole story on one slide. Read top-left → bottom: failure, then the layer that fixed it.

1 · 2022 · Prompting Opaque, ungrounded reasoningCoT + MRKL; ReAct loop (eval: few-shot + ALFWorld)

2 · 2023 · Tools Brittle free-text actionsFunction calling; Assistants host context

3 · 2023 · Frameworks Everyone reimplements the loopProductize ReAct; hit drift and bloat

4 · 2024 · Coding product Demos ≠ daily engineeringCLIs/IDEs; SWE-bench already scores scaffolds

5 · Nov 2024 · MCP N×M connectorsStandard tool bus (USB-C for AI)

6 · Late 2024 · Code-as-action JSON tools too rigid to composesmolagents: Python as action language

7 · 2025 · Context Window overflow / rotOffload, compact, recover on disk

8 · 2025 · Tool scale Schema load burns tokensProgressive disclosure; MCP as code APIs

9 · 2025 · Isolation One bloated loop; prompt “security”Subagents, permissions; OS sandbox optional

10 · 2025 · Across time Capability dies with the sessionSessions, skills, AGENTS.md, hooks, plan mode

11 · Late 2025 · Context as variable Can’t program over history/dataRLM: context is a variable in a REPL

12 · 2026 · Self-editing Hand-tuned skills go staleContinual Harness / Prime refine; Hermes; Self-Harness. Evals co-evolved the whole arc. Next: model-harness co-training

Evals are not a late layer: zero/few-shot → tool correctness → env success → Model+Harness as the unit. They made each failure above measurable.

Section 13 · Thesis

Intelligence moved outward

The model still thinks, acts, and observes. Durable capability lives outside the weights, and is measured as Model + Harness.

Still in the model

Unchanged job

ReasonPlan, handle exceptions, choose next step
ActEmit tool calls, code, language
ObserveFold results into the next thought

Now in the harness

Where agents actually win

Durable stateFilesystem, sessions, recoverability
Context as a variableRLM / REPL over history and data
Tool surfaceMCP, progressive disclosure, code-as-action
Self-edit & controlSkills, hooks, refine loops, verification

JSON-only tools Code-as-action + context as variable

Everything in the window Offload + programmable state

Model-only leaderboards Report Model+Harness

Static skills forever Self-editing harness

Same ReAct skeleton at the center. The product surface is the harness. Same model, different harness, different agent, and a different leaderboard row.

Section 13 · Sources

Primary sources

ReAct / CoT / Toolformer
OpenAI function calling & Assistants API
Anthropic MCP; code execution with MCP
SWE-bench / Verified; Terminal-Bench
Harness-Bench (arXiv 2605.27922)
smolagents CodeAgent
RLM (2512.24601); Continual Harness (2605.09998)
Prime Agent; Hermes Agent skills
Self-Harness, AHE, HarnessCompass, Meta-Harness
LangChain anatomy of a harness
Claude Code / Codex product docs (hooks, plan, skills)
SWE-agent / agent-computer interfaces
Prime Intellect Environments Hub / hosted evals

Vendor bench numbers are protocol-bound. Always report model + harness together.