Strategic Significance
The shift from 'compliance as a gate' to 'compliance as code' is a fundamental maturity milestone for agentic sys...
Make Legal Write Your Evals: Building Jade, Chime’s Financial Copilot | Interrupt 26
LangChain
The Signal
Philipp Comans, a software engineer at the financial firm Chime, presents a workflow to align agentic AI systems with regulatory compliance. By translating legal risk definitions into a structured, shared taxonomy, Chime enables continuous co-development between engineers and legal partners, aiming to replace late-stage gatekeeping with evidence-based validation. The central tension lies in whether structured LLM-evals can bridge the expertise gap between legal experts and engineers to safely deploy regulated financial agents.
The Case
- Traditional product workflows historically isolate compliance partners at the bookends of development, which frequently causes late-stage blockers and weeks of rework.
- Chime decomposes high-level legal risks—such as the Investment Advisers Act's ban on personalized recommendations—into a structured taxonomy of domains, categories, and concrete, testable risks.
- These structured risk definitions serve as a common language, allowing compliance partners to author requirements that engineers can then turn into executable datasets and automated evaluations.
- While Chime uses Giskard, an open-source red teaming framework, to bootstrap adversarial test questions early in the cycle, the team maintains that synthetic data does not replace the necessity of real-world user data.
- The evaluation process functions as an alignment surface where individual inputs and outputs are scored by an LLM-as-judge, with successful aggregate metrics like a 93.9% pass rate informing launch readiness.
- The system features a feedback flywheel where human legal review of eval results can simultaneously update the agent's prompt, the dataset generator, the judge's strictness, and the foundational risk definitions.
The 1 Minute Signal Take
Chime's approach demonstrates that scaling regulated AI is less about testing frequency and more about creating a shared artifact that forces alignment between legal experts and engineers. By treating compliance as a translation problem rather than a gatekeeping hurdle, teams can build trust through continuous evidence rather than last-minute signoff.
Pro Analysis
Full analysis always available on Pro.
Time saved:
Written by: 1 Minute Signal Editorial Team