Gemini 4 Still Isn't What I Need (Yet)

Video thumbnail: Gemini 4 Still Isn't What I Need (Yet)
Nov 11, 202629m 37s video lengthTheo - t3․gg

The Signal

Gemini 4 Carbon, Google’s latest AI model, delivers a measurable improvement in hallucination rates over its predecessor but remains too slow and expensive for general coding use. While Google touts it as a frontier model, the primary takeaway is a narrow but significant success in security incident response that challenges the speaker's initial negative expectations.

The Case

Coding and Performance

  • Carbon reduced the speaker's observed hallucination rate from 36% in the previous model, Argan, to 18%, though it still generates incorrect details, line numbers, and architectural patterns at a frequency that precludes it as a default coding tool.3:51
  • Artificial Analysis benchmarks show Carbon is consistently slow, with average task times between 22 and 27 minutes compared to just over three minutes for Opus, contradicting implied claims that the model is fast.5:30
  • The speaker notes that earlier, lower-quality outputs may have been partly due to flawed ACP bindings—the integration layer that connects the model to code environments—rather than solely the model's intrinsic limitations.16:50

Economics and Utility

  • Google's pricing strategy is misleading, as the current $2/$10 rate doubles on December 1, and the cache behavior is inconsistent, with miss rates reaching 34% in 30-second windows and higher, effectively inflating costs.6:40
  • The model struggles with UI and 3D generation, producing repetitive, template-heavy layouts and visually broken 3D renders that the speaker considers inferior to existing options.9:42
  • In a high-salience security remediation task, Carbon provided a precise, actionable fix—proposing a hard separation between machine flows and raw URL flows—after other models failed, suggesting it holds real value for specialized incident response workflows.22:16

The 1 Minute Signal Take

Carbon is not the breakthrough coding assistant Google claims, but it is a capable specialist tool for complex security tasks. Users should ignore the introductory pricing and evaluate the model based on actual task-level costs and its specific strengths in security remediation rather than general-purpose development.

Pro Analysis

Strategic Implications

The shift toward agentic capabilities means that model performance can no longer be judged by simple token benchmarks. The speaker’s experience demonstrates that the 'harness'—in this case, the ACP bindings—is just as vital as the model weights. For developers, this implies that early-stage model integration is prone to phantom bugs that are actually integration failures, not intelligence failures.

Evidence & Hype Audit

This review is grounded in high-signal data, including timing benchmarks and specific claim-accuracy counts. While the speaker acknowledges personal biases and the potential impact of early-access programs, his willingness to change his stance after witnessing a specific, successful security intervention adds significant credibility. It is not an uncritical hit piece, but rather a tempered, experience-based assessment.

Counterarguments

A contrarian view would argue that the speaker’s workflow is highly optimized for Opus, making it difficult for any new model to compete. Carbon’s strength might not lie in mimicking existing coding workflows, but in new agentic architectures that the speaker has not yet fully integrated into his personal toolset.

Who Should Care

  • Engineering Leads: Should evaluate if Carbon's niche security capabilities warrant a secondary API integration alongside their primary model.
  • FinOps Teams: Must audit Gemini's cache-hit behavior before projecting long-term costs, as the model's performance on repeated tasks is currently inefficient.
  • Security Researchers: Should experiment with Carbon for incident response, as it appears to exhibit a higher aptitude for remediation tasks than general-purpose models.

What to do next

  • Audit existing ACP/CLI integrations to ensure context like line numbers is being correctly propagated.
  • Track cache miss rates on high-volume prompts to calculate the true per-task cost.
  • Monitor for the Dec 1st price hike and adjust budget models accordingly.
  • Test Carbon against a set of known security vulnerabilities to gauge its effectiveness in your specific stack.
Time saved:26m 27s

Share this

Tags

Written by: 1 Minute Signal Editorial Team