Why it Matters
Model routing represents the transition from 'prototype' AI to 'production-grade' systems. As agents move from labs into h...
How to Build a Model Router in the Harness
LangChain
The Signal
LangChain’s open-source team found that routing agent tasks to different models based on complexity can reduce costs by 64% without a measurable loss in quality. While effective, this efficiency hinges on accurate task classification and the ability to maintain a 'performance' tier for complex work that smaller models cannot handle.
The Case
- Sydney, a product manager at LangChain, used LangSmith traces to analyze the Open SWE coding agent's workflow, identifying specific task types like bug fixes and feature requests to define model-routing criteria.
- The routing system categorized tasks into three tiers: GLM-5.3-Flash for fast, simple requests; GPT-5.6 Sol for balanced work; and GPT-6 Astra for high-performance tasks.
- In an experiment with ~500 threads, the router directed 34% of traffic to the fast model, 56% to the balanced model, and 10% to the performance tier, resulting in a 64% reduction in median cost per thread.
- The team initially restricted routing to a single decision at the thread's first message to avoid complexity, deferring mid-thread rerouting due to potential cache-miss penalties.
- A follow-up test, which forced all tasks to the 'fast' model, was terminated within a day because users reported performance issues, confirming that a one-size-fits-all fast model cannot meet every task requirement.
Optimization Strategy
- Reported results showed no statistically significant change in PR merge rates, but the team emphasizes that routing success requires rigorous outcome tracking—such as PR merge rate and user feedback—before implementation.
- Future optimizations include evaluating routing for subagents, refining benchmarks with metrics that prioritize PR review quality, and exploring richer signals to improve decision-making accuracy.
The 1 Minute Signal Take
Model routing is a powerful lever for cost control, but it is not a set-it-and-forget-it optimization; it requires granular task-space analysis and a safety net for complex queries. The primary takeaway is that while smaller models can handle the majority of agent traffic, you must maintain a performance tier to prevent degradation in user-facing outcomes.
Pro Analysis
Full analysis always available on Pro.
Time saved:
Tags
Written by: 1 Minute Signal Editorial Team