Why It Matters
This case study highlights the friction between the marketing narrative of AGI and the practical reality of building func...
Trying To Solve The Biggest AI Problem
Matt Wolfe
The Signal
A creator’s attempt to build a simple, URL-based detection tool for AI-generated video highlights a practical gap between the hype around AGI and the actual performance of current frontier models. After extensive iteration, the project succeeded only by layering specialized third-party APIs over general models, revealing that even sophisticated AI still struggles to consistently distinguish synthetic footage from reality.
The Case
The Development Obstacle
- The builder first relied on Google’s Gemini, which repeatedly failed to identify obvious AI clips, often labeling them "probably not AI" or providing inconclusive results even after high-confidence assignments.
- Initial attempts to automate the detector using agentic coding loops—where the system iterated on its own logic—resulted in hours of stalling and ineffective code, forcing the developer to manually pivot the system's architecture.
The Detection Mechanism
- Detection only became reliable after integrating Site Engine, a third-party API that the creator prioritized because it consistently provided more accurate signals than the general-purpose Gemini model.
- The system remains imperfect: even with the improved setup, it frequently returns "inconclusive" verdicts when models disagree, or creates artificial limits on video length and file size that the creator did not intend to impose.
Financial and Strategic Constraints
- Publicly hosting the tool is financially impossible because Site Engine’s operation costs are opaque and high; a $100 monthly plan allows only 40 operations, while testing the detector consumed over 12,000 operations.
- The project concludes as an open-source GitHub repository rather than a consumer web app, requiring users to self-host the tool and provide their own API keys to circumvent these hosting costs.
The 1 Minute Signal Take
The builder’s struggle suggests that current public-facing AI models are nowhere near the human-level perceptual judgment required to identify synthetic content reliably. The project's failure to scale as a public service demonstrates that the high cost and opaque pricing of specialized detection APIs remain the primary bottleneck for widespread AI-slop identification.
Pro Analysis
Full analysis always available on Pro.
Time saved:
Tags
Written by: 1 Minute Signal Editorial Team