Why It Matters
As firms transition from simple chatbots to agentic workflows, the complexity of debugging increases exponentially. Morni...
How Morningstar Stopped Flying Blind on Production Agents with LangSmith
LangChain
The Signal
Morningstar—a financial services firm known for its investment research—has moved beyond internal AI experimentation into live, externally facing products. By deploying agentic assistants within their advisory platforms, the team encountered significant debugging hurdles, forcing a transition from anecdotal tracking to a formal observability stack to maintain product reliability and performance.
The Case
The Problem
- Before adopting specialized tooling, the AI accelerator team felt they were "flying blind" because they lacked a common language for observability and tracing.
- Debugging required cumbersome, limited Splunk queries that failed to capture the full sequence of agent activity: the specific prompt, model, tool choice, and final tool output.
The Solution
- Morningstar integrated LangSmith—an observability and evaluation platform—to gain deeper visibility into agentic workflows like fetching analyst research, visualizing proprietary data, or surfacing ETF holdings.
- By using pre-built infrastructure like online evaluators and a prompt playground, the team shifted prompt management from an individual effort to a shared, collaborative process.
- The team now tracks specific operational metrics, including time to resolution and the speed at which they can identify a trace, to quantify their debugging efficiency.
The 1 Minute Signal Take
While the speaker reports significant improvements in performance metrics after a few months of using the new stack, they provided no numerical evidence or baseline data to support the scale of these gains. The primary value here is the clear identification of the specific observability gaps that emerge when moving agentic systems from prototype to customer-facing software.
Pro Analysis
Full analysis always available on Pro.
Time saved:
Tags
Written by: 1 Minute Signal Editorial Team