How to Build Agents That Improve Over Time: Model, Harness, Context | Interrupt London

Video thumbnail: How to Build Agents That Improve Over Time: Model, Harness, Context | Interrupt London
Oct 27, 202620m 20s video lengthLangChain

The Signal

LangChain posits that agents should follow an iterative improvement loop consisting of three levers: model, harness, and context. While the industry aims for fully automated self-improvement, the current reality relies on instrumenting feedback loops—using traces and evals—to guide human-led updates, as no universal correctness judge yet exists to automate this reliably.

The Case

Agent Architecture and Improvement

  • Agents are defined as a triad of model (the engine), harness (the code/logic/execution loop), and context (the knowledge/memory).0:42
  • Model updates are generally capital-intensive and human-initiated; Supervised Fine-Tuning (SFT) is currently the practical standard for narrower tasks, whereas Reinforcement Learning (RL) offers a higher ceiling for complex operations despite lacking mature managed tooling.3:52
  • Context is the easiest to update because it is structured as text, yet the primary bottleneck remains the domain-specific difficulty of deciding what information to remember, what to discard, and how to format it for retrieval.7:15

The Feedback Foundation

  • Traces—the exact sequence of inputs, tool calls, and outputs—serve as the foundational data type for debugging and iteration; traditional logs fail to capture the semantic and logical failures common in agentic systems.10:44
  • Evals act as the new specification layer, forcing developers to define "correct" behavior through executable criteria, realistic inputs, and safe simulation environments that avoid production state changes.12:04
  • Proactive red teaming, such as probing for inconsistencies by sending semantically similar queries, is currently necessary to catch rare errors that reactive monitoring might miss until they impact users.16:42

The 1 Minute Signal Take

Improvement loops are currently limited by the need for human-in-the-loop oversight and the lack of a universal judge for correctness. Teams should stop treating agents as static software and start building infrastructure to manage traces and evals, as these are the only mechanisms that can prevent regression in an evolving system.

Pro Analysis

Why It Matters

This framework moves AI development from a 'prompt engineering' hobby into a formal software engineering discipline. By decomposing agents into model, harness, and context, developers gain actionable levers rather than hoping for better outcomes through random prompt tweaks.

Strategic Implications

  • Obsolescence of Static PRDs: Teams that rely on written specifications instead of executable evals will likely suffer from 'drift' and unpredictable agent behavior.
  • Data as Moat: The real competitive advantage is shifting from model weights to the quality of an organization's proprietary traces and eval datasets.

Evidence & Hype Audit

The talk is grounded in practical product architecture (e.g., SmithDB, LangSmith). While the speaker positions their own products prominently, the underlying methodology—observability, iterative testing, and modular design—is sound and aligns with standard software reliability engineering (SRE) principles.

Counterarguments

Critics might argue that the 'model + harness + context' framework is overly prescriptive or specific to LangChain’s ecosystem, potentially ignoring emerging architectures where the model effectively replaces the harness entirely.

Who Should Care

  • AI Engineers: To shift focus toward observability and evaluation infrastructure.
  • Product Managers: To reframe requirements as 'eval datasets' rather than feature lists.
  • Engineering Leaders: To prioritize tooling that captures and acts on high-fidelity trace data.

What to Do Next

  • Audit your current agent logs to see if they contain enough signal for automated fine-tuning.
  • Build a small suite of 5-10 core evals that represent your agent's most important success criteria.
  • Implement a 'similar-query' red-teaming test to catch basic consistency errors.
  • Set up a mock environment to simulate agent tool-use before deploying changes.
  • Map your current agent architecture to the model/harness/context model to identify your weakest improvement lever.
Time saved:17m 22s

Share this

Tags

Written by: 1 Minute Signal Editorial Team