Why It Matters
This talk represents the 'second wave' of enterprise LLM adoption: moving from conversational interfaces to reliable agen...
Observing And Testing CX Agents | Interrupt 26
The Signal
Cisco's customer experience organization, which manages roughly 16 million annual support interactions, has moved from reliance on manual ticket queues to a production-grade agentic workflow. The core tension lies in maintaining human accountability for fixes while automating the high-volume triage of production feedback—a challenge they address by treating observability as infrastructure.
The Case
Architectural Loop
- Production signals including thumbs-downs, errors, and traces are captured via LangSmith and fed into an AI-assisted loop that performs triage, diagnosis, and clustering.
- The system automatically drafts potential fixes and opens Jira issues through an MCP-based integration, allowing an agent to operate without manual queue intervention.
- Cisco enforces a strict separation of concerns for agent roles: AI agents are empowered to read, cluster, and suggest fixes, but human experts must approve all write-stage actions and pull request merges.
Observability and Evals
- The team treats model evaluations as core code infrastructure rather than side experiments, keeping them versioned inside the repo with both component and end-to-end coverage.
- Observability is treated as a potential bottleneck, leading the team to instrument their own monitoring stack with Splunk to maintain visibility into up to 153,000 concurrent requests.
- By using MCP as an integration layer, the company maintains a decoupled architecture where backend agent tools can be swapped out without requiring front-end changes.
Semantic Routing
- For high-ambiguity enterprise customer requests, the system employs semantic routing that combines real-time environment data, historical case patterns, and specialized agent expertise.
- Guardrails are applied to routing prompts specifically to filter out profanity or emotional bias often generated during network outages.
The 1 Minute Signal Take
The transition from chatbot demos to production-agentic workflows relies on the disciplined treatment of evals as mission-critical regression testing. Cisco's framework suggests that scaling support doesn't mean removing humans, but rather restricting their involvement to high-leverage decision points while automating the high-volume noise of production triage.
Pro Analysis
Full analysis always available on Pro.
Time saved: