Why It Matters
As agentic workflows move from novelty to core product surfaces, the 'evaluation problem' becomes the primary bottleneck ...
Inside Clay's Eval Stack: 300M Agent Runs, One LangSmith Pipeline
LangChain
The Signal
Clay is shifting from a UI-based research platform to an agent-driven ecosystem where UI, CLI, and public API interfaces provide identical tool access. Driven by massive scale—over 300 million monthly Claygent runs and 100,000 weekly Sculptor messages—the company is building a unified data foundation to enable agents to operate autonomously while solving the persistent, unsolved challenge of production and eval drift.
The Case
Agent Architecture and Scale
- Clay has evolved into an agent-centric platform with two primary tools: Claygent, a research agent launched in 2023, and Sculptor, a workflow-engineering agent launched in 2025.
- The sheer volume of activity makes manual inspection of agent traces impossible, forcing the company to treat agents as first-class users with shadow builds, guardrails, and separated compute resources to prevent production regressions.
- Clay is aggressively pursuing tool parity, ensuring that any capability accessible via the web UI is mirrored in the recently launched CLI and public API, creating a flywheel where external agent failures directly improve internal tool reliability.
Evaluation and Data Philosophy
- The company is consolidating disparate data sources—including LangChain traces, Snowflake analytics, Postgres, and ClickHouse—into a unified data lake to allow agents to act on first-party and third-party data seamlessly.
- Traditional "golden" tests are viewed as too brittle for complex workflows; Clay now prioritizes structured checks and deterministic multi-turn evals over agent-generated simulated users, which they found too noisy and costly to maintain.
- The hardest technical challenge identified is the feedback loop between production and offline evals, as companies frequently struggle with judge bias, overfitting to small test sets, and the reality that production use cases often drift from testing assumptions.
The 1 Minute Signal Take
The move toward unified interfaces and production-informed evaluation harnesses suggests that for high-scale agent products, reliability is no longer a post-hoc QA step but a core architectural requirement. Clay’s experience indicates that until production signals are explicitly linked back to offline testing, eval drift will remain the primary bottleneck for agentic self-improvement.
Pro Analysis
Full analysis always available on Pro.
Time saved:
Tags
Written by: 1 Minute Signal Editorial Team