Self-Improving AI Harnesses: The Future of Agent Intelligence

Key Insights

Harnesses are critical infrastructure: Once dismissed as "just prompt engineering," harnesses now deliver measurable performance improvements—an 18% boost between implementations and the difference between getting RAG to work or not.
Static harnesses are evolving: The field is transitioning from fixed system prompts to self-improving harnesses that can modify their own architecture, system prompts, and tools based on performance feedback.
Test-time compute matters: Models now generate test-time experience at scale, but most systems don't effectively adapt to new domains—harnesses solve this by enabling continuous learning from each interaction.
Local AI is becoming viable: OpenJarvis demonstrates that on-device LLMs can rival cloud-based systems at a fraction of the cost (800x lower running costs) while preserving privacy.
Enterprise adoption is accelerating: QM at Y Combinator shows how harnesses enable diverse workflows—from email automation to coding to database queries—with minimal configuration.

What Are AI Harnesses and Why They Matter

An AI harness is the layer between a language model and the external world. It's more than just a prompt—it's the infrastructure that adds persistent state, tools, compute resources, and feedback loops to make an LLM truly functional.

For years, harness development was dismissed as sub-research-level work. Reddit posts claimed "prompt engineering doesn't belong at top-tier ML conferences," and researchers argued it was purely a product problem. Yet harnesses now enable breakthrough results: Claude Opus scored 30% on ARC-AGI with no harness optimization, but rose to 95% with proper harness design. This isn't marginal improvement—it's fundamental.

The evolution from simple prompts to sophisticated harnesses mirrors computing history. A raw LLM is like a Turing machine with fixed tape; a harness transforms it into a von Neumann architecture with read/write operations on external memory, making it exponentially more powerful.

From Static to Self-Improving Harnesses

The Static Harness Era (2019–2025)

Early harnesses were simple loops: take input, process tokens, return output. Key milestones improved this foundation:

In-Context Learning (2020): Few-shot examples within context improved problem-solving over raw token generation.
Chain of Thought (2023): Models learned to "smear" reasoning across tokens, showing intermediate steps like "5 - 3 = 2" instead of just "2."
Tools & Memory (2023): Systems gained access to external tools (Python execution, web search, file operations) and memory banks for context management.
Skills & Subagents (2023): Multi-step routines (skills) composed tool calls; subagents handled parallel tasks.

By 2025, static harnesses had reached sophistication, but they couldn't adapt. The system prompt was fixed; the architecture didn't evolve.

Self-Improving Harnesses (2026+)

The latest frontier involves harnesses that modify themselves. Two approaches are emerging:

System Prompt Optimization: Tools like DSPy (January 2023) use genetic programming on a small training set to discover optimal prompts, giving CRUD (Create, Read, Update, Delete) operations over the system prompt itself.
Harness Code Modification: Darwin Gödel Machines (March 2026) let systems change not just the prompt but the harness code itself. An archive stores different agent configurations; the system samples, evaluates performance, and adds improved versions back into the archive—meta-learning at the harness level.
Continual Harness (May 2026): The ultimate vision where the main harness's purpose is producing and evolving other harnesses, with CRUD operations over meta-prompts, system prompts for all agents, and agent count itself.

Prime Agent: Self-Improving RLM Harness

Prime Agent, developed by Seth Karten and collaborators, demonstrates self-improving principles in production. It uses the Recursive Language Model (RLM) architecture, where everything—tools, memories, subagents—runs in an IPython shell that agents can programmatically manipulate.

Core Design Principles

Agentic Context Management: Context is organized in cache layers:

L1 (Fastest): Model weights
Active Context: Input tokens, in-context examples
L2 (RAM): Live REPL state and subagent sessions
L3 (Disk): File system and persistent storage

As context window limitations appear, compaction lets the agent summarize its own history. Beyond that, CRUD operations update and delete skills, memories, and prompts over time—preventing both RAM and disk crashes while refining what matters.

Persistent Subagents: Subagents spawn as independent sessions, run tasks, report back, then remain idle in RAM. The parent can message them later without losing context. This saves tokens compared to re-explaining tasks.

Agentic Expressibility: The harness exposes features that models need but can't discover alone:

Calling compact() to manage context
Running Python REPLs to manipulate data
Spawning subagents for parallel work
Accessing feedback mechanisms

Remove any of these, and capability drops. Models have natively figured out ReAct-style loops, but they still need infrastructure to express these capabilities.

Results on Long-Horizon Tasks

Prime Agent was tested on ARC-AGI (a fluid intelligence benchmark):

General harness, no optimization: 20% with Gemini 2.0 Flash
With community system prompt + REPL access: 78% with GPT-4o
Opus 4 + full harness: 95.5%
AVO (Nvidia): 100%

The REPL enabled coding, image analysis, and scenario testing—core to the breakthrough.

For auto-research (NanoGPT speedrun on 8×8 H200 GPUs over a week), models exhibited striking behavior: they ran CPU-based auxiliary experiments to gather hyperparameter data rather than burning expensive H200 hours. This "out-of-loop experimentation" emerged naturally from harness expressibility.

On Factorio (a 7-day multi-agent run), Prime Agent deployed 633 agents across 23 million output tokens. Subagents divided factory tasks (research, gathering, building), and the refinement system prevented getting stuck by leveraging past decisions. The system made continuous technological progress without human intervention.

OpenJarvis: Personal AI on Your Laptop

OpenJarvis, from Stanford researchers Avanika Narayan, Zaliha Mir Hosseini, and Christopher Rey, tackles a different problem: can we run sophisticated personal AI entirely on-device?

The Cloud Problem

Current personal AI (OpenClaw, Hermes Agent) relies on cloud LLMs, creating three pain points:

Cost: Thousands of dollars annually in API fees
Privacy: Personal data sent to cloud vendors
Energy: Orders of magnitude more consumption than local inference

The Local Solution

Modern local LLMs are competitive. Qwen 3.8 27B matches Claude 3.5 Opus (August 2025 SOTA), with the gap closing as Apple Silicon and Nvidia accelerators improve.

OpenJarvis defines five optimization primitives:

Intelligence: Choice of LLM (Qwen, Gemma, etc.)
Engine: Inference runtime (Ollama, vLLM, SGLang)
Agentic Logic: Reasoning and tool composition
Tools & Memory: MCP protocol for standardized interfaces
Learning: Prompt-based (DSPy, JetPAR) or weight-based (LoRA, SFT, GRPO)

A novel approach: cloud LLMs optimize local stacks. Claude or GPT-4 diagnoses issues in the local harness and proposes changes—but only during setup, not at inference. This one-time optimization phase avoids recurring cloud costs.

Results

On coding, research, and personal tasks, OpenJarvis achieved parity with cloud systems while reducing costs 800x. Latency also improved. Even with today's on-device LLMs, the gap to cloud is closing rapidly.

QM: YC's Open-Source Work Agent

QM, built by Josh Regan and team at Y Combinator, is a harness deployed across the entire organization. It evolved through several generations:

Evolution of YC Agents

January 2025 – "General Agent": Simple prompt + tools + loop. One-size-fits-all, but surprisingly effective for data queries.
June 2025 – Cloud Code + Codex: Ran in VMs, hooked to Slack. Solved bugs, spun up dev environments, ran CI pipelines.
January 2026 – OpenClaw era: YC partners used it as a personal assistant with its own computer. Highly customizable but resource-intensive.
April 2026 – Hermes fleet: 50+ agents in VMs provided flexibility but became hard to manage ("whack-a-mole" situation).
May 2026 – QM: Centralized architecture solving the fleet problem while preserving personalization and capability growth.

QM's Design

Brain Outside the Sandbox: Unlike Hermes and OpenClaw (brain + computer tightly coupled), QM pulls the system's core logic out of sandboxes. All agent conversations are centralized in Postgres; the agent accesses aggregated context across the entire system. Sandboxes become dynamic resources the agent allocates as needed—not its primary living space.

Three Core Tools:

Remote sandbox execution
Reading/writing object storage
Publishing internal web apps

Everything else (memory, cron jobs) is temporary scaffolding.

"Unhobbling" Capabilities: QM exposes features that unlock hidden model intelligence:

Choice of model provider (switches providers when one refuses tasks)
Dynamic sandbox selection (heavyweight machines for coding, lightweight for simple queries)
No lock-in across vendors

Use Cases in Production:

Email triage and automation
Legal and finance workflows
Document editing and database pulls
Spinning up live internal web apps
Event planning

Operational Insights

Budget-based persistence: Agents often give up too easily. Setting token budgets or wall-clock time limits forces deeper problem-solving, improving research output.

Permission systems matter: Agents struggle with implicit social context. What information is appropriate to share with whom? YC's fine-grained permissions help, but many organizations lack this infrastructure. Sharing knowledge with agents is bottlenecked by permission depth.

From review to trust: Early on, humans reviewed every agent action closely. As trust built, "rubber-stamping" approvals began—similar to how Claude Code users evolved from scrutinizing tool use to trusting the system.

The Broader Landscape: Why This Matters Now

Three converging trends:

Models Keep Improving: Test-time compute and inference-time scaling are reshaping what's possible. Models still don't adapt quickly to new domains without harness support.
Harness Sophistication Drives Gains: 18% improvements, breakthroughs on ARC-AGI, auto-research behavior—major leaps come from harness design, not just model training.
Personalization + Privacy = Local: The gap between on-device and cloud LLMs is shrinking as hardware accelerators improve. Expect a shift toward local inference for daily tasks.

Conclusion

AI harnesses have graduated from a dismissed footnote to critical infrastructure. The transition from static to self-improving harnesses represents a fundamental shift: systems that adapt, learn, and evolve their own architecture based on performance feedback.

Prime Agent shows how persistent subagents and agentic context management unlock long-horizon reasoning. OpenJarvis demonstrates that sophisticated personal AI can run locally, preserving privacy while cutting costs 800x. QM proves that enterprise harnesses can handle diverse workflows at scale while remaining thin and adaptable.

If you're building AI systems today, invest in harness design. The expressibility you expose, the feedback loops you build, and the self-improvement mechanisms you enable will determine whether your agents stumble or soar.

Original source: Self-Improving Harnesses, Local Personal AI And YC's Agent For Work | YC Paper Club

powered by osmu.app

(Ycombinator) Self-Improving AI Harnesses: The Future of Agent Intelligence