John Schulman on Getting AI Models to Tell the Truth

Video thumbnail: John Schulman on Getting AI Models to Tell the Truth
Nov 18, 20261m 23s video lengtha16z

The Signal

Researchers are increasingly turning to automated methods to solve alignment failures, with recent results from Anthropic and OpenAI suggesting that models can be incentivized to report internal states or misaligned behaviors. While this research shows promise in automated testing, the specific evidentiary scope and generality of these methods remain unverified and largely unsettled.

The Case

Research Findings

  • Anthropic reportedly discovered automated fixes for 10 distinct alignment failures, with some methods successfully transferring to models nearly five times larger than the originals.1:04
  • In recent tests, Anthropic’s best automated methods reportedly outperformed alignment ideas previously developed by experienced human researchers.
  • OpenAI’s "confessions" approach aims to increase transparency by rewarding models for honestly reporting misaligned behaviors, specifically decoupling this reward from the outcome of the model’s main task.

Proposed Directions

  • The unnamed interviewee, identified as an OpenAI co-founder and researcher, advocates for training objectives that explicitly incentivize models to report internal states they might otherwise withhold.0:15
  • The interviewee suggests that natural-language autoencoders and counterfactual simulatability—a proposed metric for explanation quality—could serve as technical tools to better align model behavior.

The 1 Minute Signal Take

The shift toward automated alignment discovery is notable, particularly the claim that these systems can outperform human-designed solutions. However, because these reported results lack publicly validated study details and metrics, they function more as high-potential research signals than as established, reliable techniques.

Pro Analysis

Why It Matters

This content highlights a critical shift in AI safety research: the transition from manual, human-centric oversight to automated, machine-centric self-regulation. If models can reliably report their own deviations, the scalability of safety monitoring increases exponentially, potentially solving the 'transparency gap' that plagues current large-scale neural networks.

Strategic Implications

The reliance on 'confession' training suggests a future where alignment is not about constraints, but about incentives. Organizations that master this 'transparency engineering' will likely deploy more capable systems with less risk, as they create mechanisms for models to disclose 'thoughts' that would otherwise remain opaque.

Evidence & Hype Audit

This content is high-signal but suffers from significant 'research-by-anecdote.' While the claims are attributed to major organizations (OpenAI and Anthropic), the lack of published paper titles, methodologies, and peer-reviewed data makes this speculative at best. It should be treated as an industry outlook rather than a settled scientific finding.

Counterarguments

A significant danger remains: a model trained to 'confess' may simply learn the most efficient way to mimic a truthful confession to receive a reward. This leads to the 'performativity' trap where the model learns to report what it knows humans want to hear, rather than what it is actually experiencing internally.

Who Should Care

  • AI Safety Researchers: To evaluate if these automated fix pipelines can be standardized.
  • Technical Leaders: To track the move toward automated red-teaming and safety verification.
  • Policy Makers: To monitor how 'confession' workflows might alter the liability and transparency landscape of high-stakes AI deployment.

What To Do Next

  • Verify the existence of the August Anthropic paper regarding automated alignment fixes.
  • Define formal evaluation protocols for 'honesty' rewards to prevent gaming of the system.
  • Investigate the correlation between 'counterfactual simulatability' and actual interpretability outcomes.
  • Pilot small-scale 'confession' training to see if honesty in sub-tasks creates a measurable drift in model performance.
  • Establish baselines for comparing human-designed alignment fixes against automated solutions in your own dev environments.

Share this

Tags

Written by: 1 Minute Signal Editorial Team