The Shift in Evaluation

• Agent evaluation is evolving because modern "Deep Agents"—systems with planning, sub-agents, and file-system read/write access—can now mutate environments, making text-only grading an incomplete metric for performance.0:32
• Harbor, an open-source framework from LangChain engineer Nick, provides a standard for reproducible agent evaluation by requiring three consistent inputs: an agent, a contained sandbox, and a dataset of tasks.3:53
• Each task is stored in a self-contained subfolder featuring a task.toml configuration, instruction.md guide, environment image, and tests folder, ensuring every run starts from a clean, verifiable state.5:08

Validation and Observability

• Unlike subjective human review, Harbor uses deterministic verification; PyTest scripts automatically score agent performance by checking objective criteria like file existence, document formatting, citations, and content accuracy.6:35
• Integration with LangSmith enables large-scale experiment tracking, allowing developers to view metrics such as binary rewards (1/0), latency, and token consumption across multiple tasks.9:21
• While the framework demonstrates a clear mechanism for reproducible testing, independent verification—specifically whether the demo agents truly meet all correctness criteria beyond the initial claims—remains unproven.10:05