You wrote a clear prompt. You curated the context. The model still failed.

Picture three failures that teams report all the time. A support agent retrieves the right refund policy, then calls the billing API with malformed arguments and quietly returns garbage. A coding agent writes correct code, then submits it without running the tests. A research agent finds the right documents, then retries a broken endpoint all night and runs up a bill nobody approved.

None of these are prompt failures. None are context failures. They are system failures. The model knew what to do. The infrastructure around it broke.

That infrastructure is what harness engineering is about.

Where Harness Engineering Sits

Think of agent reliability as three nested layers:

┌─────────────────────────────────────────────────────┐
│  Layer 3: SYSTEM LEVEL — Harness Engineering        │
│  Agent perceives own limits, recovers from tool     │
│  failures, verifies work, scopes sub-tasks          │
│                                                     │
│  ┌─────────────────────────────────────────────┐    │
│  │  Layer 2: SESSION LEVEL — Context Engineering│    │
│  │  What populates the context window each step │    │
│  │  Memory, retrieval, compaction, tool outputs │    │
│  │                                              │    │
│  │  ┌──────────────────────────────────────┐    │    │
│  │  │  Layer 1: MESSAGE LEVEL — Prompt Eng │    │    │
│  │  │  Wording, structure, examples,       │    │    │
│  │  │  output format constraints           │    │    │
│  │  └──────────────────────────────────────┘    │    │
│  └─────────────────────────────────────────────┘    │
└─────────────────────────────────────────────────────┘

Layer 1 controls what you ask. Layer 2 controls what the model knows. Layer 3 controls whether the system survives production.

This post is about Layer 3.

What Harness Engineering Is

Harness engineering is the practice of designing, building, and operating the infrastructure that constrains, informs, verifies, and corrects AI agents in production.

The core question: does the system still work when everything around the model goes wrong?

Where the term came from

The name is recent. In early February 2026, Mitchell Hashimoto (co-founder of HashiCorp, creator of Terraform) described a habit in his post My AI Adoption Journey: whenever an agent makes a mistake, you engineer the environment so it can't make that same mistake again. He called this "engineering the harness." About a week later, OpenAI published "Harness engineering: leveraging Codex in an agent-first world" [INSERT OPENAI POST URL], and the term spread quickly from there.

The metaphor is horse tack: gear that directs a powerful animal's strength toward useful work without letting it run wild.

The shorthand you will see everywhere:

Agent = Model + Harness

The model is the intelligence. The harness is everything else:

  • Context assembly: what the agent knows
  • Tool orchestration: what the agent can do
  • Verification loops: whether the agent checked its own work
  • Cost controls: how much the agent can spend
  • Observability: what you can see when it breaks

Harness engineering borrows from three older disciplines: circuit breakers from distributed systems, evaluation pipelines from MLOps, and observability from SRE. It adapts them for non-deterministic, multi-step, tool-using systems.

Why Agents Fail Without It

Small failures compound

Agent tasks are chains. A 20-step workflow where each step succeeds 95% of the time completes only about 36% of the time (0.95²⁰ ≈ 0.36). Every step looks fine on its own. The workflow still fails most days.

Tool calls are a common weak link. One observability vendor's July 2026 write-up puts tool-call failure at 3–15% in production, depending on model size and task complexity. Treat that as a rough vendor estimate, not a benchmark. The pattern matters more than the number: APIs return errors, responses arrive with missing fields, timeouts fire under load. Without a harness that catches these, the agent carries corrupted data into every later step.

Benchmarks show the same gap

Mercor's APEX-Agents benchmark (January 2026) tested agents on 480 long-horizon tasks written by investment banking analysts, management consultants, and corporate lawyers. The best model at the time, Gemini 3 Flash, reached 24.0% on first attempt (Pass@1), with GPT-5.2 at 23.0% and Claude Opus 4.5 at 18.4%. Note the scope: this is professional-services work, not software engineering, and it is a snapshot of models from early 2026. It shows that long, cross-tool tasks are hard. It does not, by itself, prove the harness was the cause.

What failure looks like in practice

Agent failed
      │
      ├── Tool returned 500 error → no retry logic → agent proceeded with corrupted data
      │
      ├── Agent wrote solution → never ran tests → submitted broken code
      │
      ├── Retry loop fired hundreds of times overnight → no cost envelope → surprise bill
      │
      └── Agent edited same file 12 times → no loop detection → doom loop

Here is a composite scenario (an illustration, not a specific incident). A document-processing agent runs fine for weeks. One night an upstream source starts returning malformed responses. The agent's retry policy correctly spots the bad data and re-fetches it, hundreds of times, and each retry pays for a full re-planning step. By morning the spend is many times the daily norm.

The verification loop worked. The cost envelope didn't exist.

The Five Core Components

Most production-grade harnesses share five components. Many teams build two or three and learn about the rest through incidents.

1. Context Engineering (a subset of the harness)

Context engineering is what the agent knows at each step. Inside a harness it becomes one component among five instead of the whole story.

The challenge is precision. Too little context and the agent lacks what it needs. Too much and it drowns in irrelevant data.

Manus made the cost side of this concrete in their context engineering write-up. Agents re-send a growing context on every step, so they treat KV-cache hit rate as a key production metric. With Claude Sonnet pricing, cached input tokens cost $0.30 per million versus $3.00 uncached, a 10x price difference. Keeping the prompt prefix stable is a harness decision that shows up directly on the bill.

2. Tool Orchestration

Tool orchestration is what the agent can do: the external systems it touches, how it touches them, and what happens when that fails. It includes:

  • Input validation: is the agent calling the tool with the right parameters?
  • Output parsing: did the tool return usable data?
  • Error handling: what happens when the tool is unavailable or returns garbage?
  • Timeout management: how long do we wait before declaring failure?

Each tool is also a decision point where the agent can choose wrong. Vercel's team found this the hard way with their internal text-to-SQL agent, d0. They cut it from 17 specialized tools down to two (a bash tool and a SQL executor). On a five-query benchmark, success went from 4/5 to 5/5, average runtime dropped from 274.8s to 77.4s (3.5x faster), and token use fell 37%. Their write-up is upfront about the caveats: it was a small test, and it only worked because their semantic layer was already well documented.

3. Verification Loops

Verification loops check the agent's work at each step before it moves on. This is the component most teams skip, and it is where a lot of silent failures get caught.

Schema-based verification checks that a tool call returned the expected format and required fields. Semantic verification uses a second LLM call to judge whether the output makes sense for the task.

File: harness/run_agent.py (illustrative pseudocode, not a drop-in library)

def run_agent_with_verification(task, tools, cost_ceiling, max_steps=50):
    context = assemble_context(task)
    total_tokens = 0

    for _ in range(max_steps):
        if task.is_complete():
            return TaskResult(status="complete", output=context.final_output)

        # Agent decides next action (planning calls cost tokens too)
        action, plan_tokens = agent.plan(context, tools)
        total_tokens += plan_tokens

        # Execute the action
        result = execute_tool(action)
        total_tokens += result.tokens_used

        # Verify the result before proceeding
        verification = verify_output(result, action.expected_schema)
        if not verification.passed:
            if verification.retry_recommended:
                result = retry_with_backoff(action, max_retries=3)
                total_tokens += result.tokens_used
                verification = verify_output(result, action.expected_schema)
            if not verification.passed:
                return TaskResult(status="failed", reason=verification.reason)

        # Check cost envelope
        if total_tokens > cost_ceiling:
            return TaskResult(status="budget_exceeded", partial=context)

        # Update context for next step
        context = update_context(context, result)

    return TaskResult(status="step_limit_reached", partial=context)

Schema checks are deterministic and cheap. Semantic checks cost an extra LLM call per verified step. Without either, silent failures travel down the whole chain. With them, failures are caught at the step where they happen and can be retried, rerouted, or escalated.

One honest caveat: verification is not magic. A July 2026 paper on a production enterprise agent found that most of the system's gains came from scaffolding, routing, and specialist models, and the verification step alone added a smaller amount, concentrated in the hardest tasks. Verification is necessary, but measure it in your own system before assuming it does the heavy lifting.

4. Cost Envelope Management

A cost envelope is a per-task budget ceiling the harness enforces no matter what the agent or retry policy wants. If the next step would push cumulative token spend past the ceiling, the harness ends the task with a structured failure response.

Most teams skip this until their first surprise bill.

The less obvious benefit: cost envelopes are reliability signals, not only financial controls. A task that hits its ceiling is behaving abnormally, usually because of a bad upstream response, context drift, or a broken tool integration.

5. Observability and Evaluation

Observability means structured execution traces: what the agent did, why, and what happened at each step. Evaluation means an automated pipeline that keeps measuring the agent against defined criteria.

Without traces, debugging is guesswork. Which step failed? What was in context? What did the tool return? Without evaluation, you find out about regressions from user complaints.

Observability stack:

  Agent step → Structured log → Trace collector → Dashboard
       │                                      │
       │         ┌────────────────────────────┘
       │         │
       ▼         ▼
  Per-step fields:          Evaluation pipeline:
    - tool_called             - representative tasks (20-50)
    - input_params            - expected outcomes
    - output_raw              - scheduled cadence (daily/weekly)
    - verification_result     - task completion rate tracking
    - tokens_consumed         - output quality scores
    - latency_ms              - cost per task trending
    - error_class (if any)

Two Control Mechanisms: Guides and Sensors

Birgitta Böckeler's article Harness Engineering for Coding Agent Users on martinfowler.com gives the clearest mental model for the controls in a harness, borrowed from control theory. The framing below is hers, in my words.

Guides (feedforward) steer the agent before it acts, so the first attempt is more likely to be good.

  • AGENTS.md: project conventions, coding standards, architecture rules
  • Skills: task-specific instruction sets (how to write tests, how to review)
  • Tool descriptions: clear, unambiguous statements of what each tool does
  • Reference docs: API specs, database schemas, deployment configs

Sensors (feedback) watch what the agent did and let it correct itself. They work best when their output is written for an LLM to read, such as a linter message that says how to fix the problem.

  • Linters: deterministic style and structure checks
  • Test suites: correctness verification
  • Type checkers: contract enforcement
  • Review agents: a second LLM that evaluates the first one's output
  • Runtime monitors: latency, error rates, SLO tracking

Either kind can be computational (deterministic and fast: tests, type checkers, linters) or inferential (semantic and slower, but non-deterministic: LLM-as-judge, review agents).

                    Guides                   Sensors
                  (before act)              (after act)
                  ┌──────────────┐      ┌──────────────┐
  Computational   │ LSP, linters │      │ Tests, types  │
                  │ AGENTS.md    │      │ Pre-commit    │
                  │ Tool schemas │      │ hooks         │
                  └──────────────┘      └──────────────┘

                  ┌──────────────┐      ┌──────────────┐
  Inferential     │ Skills       │      │ Review agent  │
                  │ Few-shot     │      │ LLM-as-judge  │
                  │ examples     │      │ Code review   │
                  └──────────────┘      └──────────────┘

You need both halves. With sensors only, the agent repeats the same mistakes. With guides only, it follows rules but never learns whether they worked.

Architecture Patterns

Three patterns cover most production deployments, in increasing complexity. Start with the simplest one that meets your requirements.

Pattern 1: Single Agent + Verification Loop

One agent, one verification step between actions. It is the right starting point for most teams.

┌──────────────────────────────────────────────────┐
│                                                  │
│  Task → Context Assembly → Agent Plan → Execute  │
│                                     │            │
│                              ┌──────▼──────┐     │
│                              │  Verify     │     │
│                              │  output?    │     │
│                              └──────┬──────┘     │
│                                pass │  fail      │
│                                     │    │       │
│                              ┌──────▼────┐       │
│                              │ Cost check│       │
│                              └──────┬────┘       │
│                                ok   │  exceeded  │
│                                     │    │       │
│                              ┌──────▼────┐       │
│                              │ Update    │       │
│                              │ context   │       │
│                              └──────┬────┘       │
│                                     │            │
│                              Loop back to Plan   │
└──────────────────────────────────────────────────┘

The verification loop catches tool failures. The cost envelope stops runaways. The context update keeps the agent focused.

Pattern 2: Two-Agent Supervisor

A primary agent executes the task. A supervisor agent reviews each step and can approve, request a revision, or override.

The supervisor sees the same context plus the primary agent's output. The cost is roughly one extra LLM call per supervised step. Use it when bad output costs more than verification: financial agents, customer-facing agents, or coding agents that touch production systems.

Pattern 3: Multi-Agent + Shared Harness

Several specialized agents coordinate through shared infrastructure: context management, tool access, verification, cost tracking, and observability that every agent uses.

This makes sense when one task needs different capabilities (research, analysis, writing, code generation). The tradeoff is real: orchestration complexity jumps, and agent-to-agent coordination becomes a new failure mode. Don't adopt it until you have outgrown a single agent with verification.

The Real Numbers

These are public, mostly self-reported case studies. Read them as signals, not controlled experiments.

Team Harness investment Reported result
OpenAI (Codex) Agent-legible repo docs, custom linters, structural tests, mechanical architecture enforcement About 1 million lines of code and roughly 1,500 merged PRs in about five months, with no hand-written code. The team started with 3 engineers and grew to 7
LangChain Harness-only changes (self-verification, context injection, middleware), same model Terminal Bench 2.0: 52.8% → 66.5% (+13.7 points), model fixed at gpt-5.2-codex
Vercel (d0) Tools cut from 17 to 2, sandboxed file-system access Success 4/5 → 5/5, 3.5x faster, 37% fewer tokens (5-query test)
Stripe (Minions) Blueprints that wrap agent steps in deterministic gates, pre-warmed dev environments, capped CI retries 1,300+ merged PRs per week, all human-reviewed [INSERT STRIPE MINIONS URL]
Manus KV-cache-friendly context design Cached input tokens are 10x cheaper than uncached (a pricing ratio, not a measured savings figure)

Two patterns stand out.

  1. Harness changes move results a lot. LangChain's 13.7-point gain came with the model held constant. A Stanford/MIT/KRAFTON paper, Meta-Harness, opens by noting that harness choices can cause up to a 6x performance gap on the same benchmark.
  2. Cost and quality can improve together. Fewer tools meant fewer wrong turns, which meant fewer wasted tokens in Vercel's case.

A caution before you generalize: these are different tasks, different benchmarks, and mostly first-party numbers. They support "the harness matters a lot," not a precise ratio between harness and model.

Debugging: Which Layer Failed?

When the output is wrong, walk the layers:

Output is wrong
      │
      ├── Model HAD the right information → fix the prompt (Layer 1)
      │
      ├── Model was MISSING or DROWNED in wrong info → fix context (Layer 2)
      │
      └── Model KNEW the right thing, system still failed → fix the harness (Layer 3)
            │
            ├── Tool call failed silently → add verification loop
            │
            ├── Agent looped 15 times on same file → add loop detection
            │
            ├── Task ran for 6 hours → add cost envelope
            │
            ├── Wrong tool selected → improve tool descriptions
            │
            └── Output passed schema but was semantically wrong → add supervisor agent

A concrete example:

Scenario: Coding agent submits broken code

Check 1: Was the prompt clear? (Layer 1)
  - Found: prompt says "fix the bug and submit"
  - Diagnosis: prompt did not say "run tests before submitting"
  - Fix: add "verify your work" instruction

Check 2: Did the model have context? (Layer 2)
  - Found: model had access to test files
  - But test output was not in context after execution
  - Diagnosis: tool output discarded before next planning step
  - Fix: preserve test results in context

Check 3: Did the harness catch it? (Layer 3)
  - Found: no verification loop between "execute" and "submit"
  - Found: no test suite ran as a pre-submit hook
  - Diagnosis: harness gap, no feedback sensor between action and commit
  - Fix: add a pre-completion check that runs tests before exit

LangChain's team used the same idea in their coding agent: a PreCompletionChecklistMiddleware that intercepts the agent before it exits and forces a verification pass. Prompt instructions can be forgotten under context pressure. A hook can't.

Practical Checklist

You need harness engineering when:

  • The agent calls external tools or APIs
  • Workflows have multiple steps with dependencies
  • Nobody reviews every output
  • You're running in production, not a demo
  • Cost matters

Start here:

  1. Add verification loops after every tool call. Schema-based first (cheap and deterministic), semantic second if needed.
  2. Instrument observability. Structured traces per step: tool called, input, output, verification result, tokens, latency. You can't improve what you can't see.
  3. Set cost envelopes. Find your median task cost and set the ceiling at roughly 3x that (a rule of thumb, tune it to your workload). It takes an afternoon and prevents the overnight surprise.
  4. Build an evaluation pipeline. 20 to 50 representative tasks, run daily or weekly. Track completion rate, quality, and cost over time.
  5. Add loop detection. Track per-file edit counts. After N edits to the same file, inject a "reconsider your approach" message.

Advanced:

  • Add a supervisor agent for high-stakes outputs
  • Write custom linters for project-specific patterns
  • Write guides: AGENTS.md and task-specific skills
  • Run mutation testing to check that your tests actually catch bugs
  • Mine traces for failure patterns and iterate on the harness

Design Your Harness for Deletion

Models improve. A failure mode that needs harness intervention today may vanish next year.

So build components you can remove. Loop detection exists because today's models get stuck in doom loops; when they stop, delete it. A verification step exists because today's models submit untested code; when they reliably test first, simplify it.

Vercel's d0 story is this principle in action. Scaffolding built around old model limits had become a liability once the model improved, and deleting it made the agent better. If your harness grows with every model release, something is off.

The Bottom Line

Prompt engineering is what you ask. Context engineering is what the model knows. Harness engineering is whether the system survives production.

Clear instructions are necessary. Curated context is necessary. Neither is sufficient. The teams shipping reliable agents built verification loops, cost envelopes, observability, and feedback sensors around every step.

You can't prompt your way to production reliability, and you can't context-engineer your way there either. Build a harness, measure where it breaks, fix the weakest component, and repeat.

Sources