Existing benchmarks for coding agents, notably SWEBench Pro, are presented as fundamentally flawed due to widespread contamination, sloppy verification, and prescriptive prompts that force cheating rather than testing skill. The speaker, who discloses an investment in the Data Curve project, argues that their new benchmark, DeepSWE, better reflects real developer work via behavior-focused prompts and handwritten verifiers. This shift reveals a stark performance gap between frontier models and cheaper alternatives that older benchmarks have long masked.

The evidence suggests that while current coding leaderboards have become unreliable through contamination and artifact-heavy design, the DeepSWE results carry a significant caveat: they are a proprietary, narrow-scope effort from an invested advocate. It is a useful critique of how model gaps are measured, but the benchmark's specific rankings should be treated as an early-stage internal instrument rather than an industry standard. Watch this if you want to understand how agents are being tested in the field, but skip it if you are looking for a neutral global ranking.