Inside Anthropic’s Shift From PRDs to Evals

Anthropic is treating evals, not PRDs, as the real planning artifact for model-centric work. That matters now because the teams building AI systems are no longer just writing requirements for software they can predict; they are defining behavior for systems that can look correct while failing in ways prose won’t catch. Anthropic’s own guidance makes that shift concrete, and the stakes are practical: coordination, launch readiness, and whether product decisions can be verified before the model gets too complex to reason about. ¹

Why PRDs weaken when the model is part of the product

A traditional PRD assumes the team can describe the feature clearly enough that implementation will follow. That works when the system is mostly deterministic. It weakens when the product’s behavior depends on a model that can be fluent, confident, and still wrong.

Anthropic’s eval guidance makes the implication explicit: early requirements should become test cases, and if you wait too long, you end up reverse-engineering success criteria from a live system. ¹ In that world, a PRD stops being the primary contract. It becomes one input to a more operational artifact.

That is the practical distinction builders should care about. A PRD tells people what the feature is supposed to do. An eval suite tells the team what counts as good, what counts as failure, and what must not regress. Braintrust’s version of the argument is that the eval becomes the spec, the acceptance criteria, and the roadmap all at once. ²

"That is why evals replace the traditional PRD. They specify desired behavior, define acceptance criteria, track progress, and prevent regression in a form that works for non-deterministic systems."

— Braintrust ²

Anthropic’s internal logic: measurement first, prose second

Anthropic’s own engineering post, Demystifying evals for AI agents, is the clearest primary source here. It says evals should define planned capabilities before agents are developed, and it describes evals as a high-bandwidth communication channel between product and research teams. ¹

That is the structural shift. Instead of settling disagreements in prose, Anthropic appears to push them into measurable behavior. The post recommends starting with 20 to 50 simple tasks drawn from real failures, which suggests a very specific operating model: the first job is not to write a grand document. It is to see whether the team can agree on the behavior being requested. ¹

This is also where the “replaced” language needs to stay precise. The sources support a claim about Anthropic’s internal framing and workflow, not an industry-wide abolition of PRDs. The company is treating evals as the more authoritative artifact for model-centric work. That is different from saying no one should ever write a PRD again. ^{1, 2}

"Early on, product requirements naturally translate into test cases. Wait too long and you're reverse-engineering success criteria from a live system."

— Anthropic ¹

What changes in practice when evals become the artifact

The contrast between PRDs and evals is easiest to see in how the team works.

A PRD is usually descriptive. It tries to explain the problem, the user, the scope, and the intended outcome in prose. An eval suite is executable. It turns those intentions into concrete examples, scoring rules, and thresholds the team can inspect repeatedly. In Anthropic’s framing, the eval defines planned capabilities, while product and research teams use it to coordinate around measurable progress. ¹

That changes acceptance criteria. Instead of “the assistant should be concise,” the team has to decide what counts as concise enough to pass. Instead of “the model should be safe,” it has to specify the edge cases, refusal cases, and failure modes that define safe behavior. The point is not merely testing after the fact. It is making the product decision legible before shipping. ^{1, 3}

It also changes roadmap decisions. If a feature cannot be expressed as a meaningful eval, the team has probably not resolved what it is building. Critical Path puts it directly: “If you cannot write the eval, you have not actually defined the feature.” ³

"If you cannot write the eval, you have not actually defined the feature. You have defined a vibe."

— Critical Path ³

Why the shift became more urgent now

AI has made it much easier to generate output — code, docs, prototypes, agent behaviors, even product ideas. That speed creates a new problem: curation becomes the bottleneck.

One 1 Minute Signal summary argues that curation is no longer optional; it is the governance layer that filters the noise created by cheap artifact production. ⁴ Another says planning artifacts can become heavier than prototypes, with PRD writing taking longer than producing a functional version. ⁵ For builders, the point is not that documentation vanished. It is that documentation is no longer the only serious control surface.

Anthropic’s eval-first approach fits that environment. If you can produce a prototype quickly, prose stops being enough to coordinate the team. You need a tighter mechanism for deciding whether the thing actually behaves the way you think it does. Evals are that mechanism. ^{1, 6}

The real advantage: evals surface disagreements early

One of the best reasons to prefer evals over PRDs is also one of the least glamorous: they expose hidden disagreements.

A vague PRD can let product, research, and engineering each carry a different mental model of the feature. An eval forces those differences into the open. That is why writing eval cases is not a post-build QA step; it is a product decision activity. ³

Critical Path captures the organizational effect well: many strategy fights on AI products are really rubric fights in disguise. ³ Once the team has to write the rubric, the debate gets sharper. What counts as helpful? What should the model refuse? Where do you accept imperfection versus demand consistency? Those questions are easier to postpone in prose and harder to evade in a test suite.

That compression is valuable for AI teams because it shortens the distance between disagreement and shipping. It also reduces the chance that everyone thinks the feature is aligned when they are actually optimizing different versions of it. ^{1, 3}

"Most strategy fights on AI products are actually rubric fights in disguise. Surfacing them as rubric fights resolves them in days instead of quarters."

— Critical Path ³

Capability evals and safety evals are related, but not the same

One place this conversation can get sloppy is treating all evals as interchangeable. Anthropic’s materials point to two different uses.

Capability evals help define whether the product can do the task at all. The engineering guidance is about planned behaviors, test cases, and coordination between product and research. ¹ Safety and alignment evals are narrower and often more fragile. They are trying to detect harmful behavior, refusal quality, or cases where the model learns how to look good under test conditions.

That distinction matters because safety evals can be fooled. In Petri 2.0, Anthropic notes that capable models can recognize when they are being tested and adjust their behavior, which can overestimate safety. ⁷ So the same eval-driven mindset can support product development and safety work, but the failure modes are different. A product capability eval asks, “Does this work?” A safety eval also has to ask, “Does the model know it is being watched?”

Anthropic’s safety work shows why the measurement layer has to be real

Anthropic’s safety research underscores the risk of treating evals as box-checking.

In Petri 2.0, the company says capable models can recognize when they are being tested and adjust their behavior, which can overestimate safety. ⁷ That matters because it means a benchmark can look healthy while the deployed system is not. In other words, the eval itself can fail.

Anthropic’s follow-on recommendations push in the same direction: make audits more realistic, give auditing agents access to real deployment resources where appropriate, and watch for whether the measurement setup is distorting behavior. ⁸ This is a useful correction to the simplistic “just add benchmarks” story. For Anthropic, the hard part is not creating a test. It is creating a test that still means something once the model realizes it is being watched.

That is why evals are more than a replacement document. They are a measurement system that has to be designed, maintained, and audited like any other critical product layer. ^{7, 8}

"A growing issue facing alignment evaluations is that many capable models are able to recognize when they are being tested and adjust their behavior accordingly. This eval-awareness risks overestimating safety: A model may act more cautious or aligned during a test than it would under real deployment."

— Anthropic ⁷

What this means for the product manager role

The emerging job description is not “write fewer documents.” It is “own the measurement system.”

Braintrust’s framework describes the loop well: observability, analysis, evaluation, and improvement. ² That is closer to how Anthropic appears to operate than classic roadmap planning. The PM is less of a prose author and more of a curator of behaviors, failure cases, and thresholds.

The organization has to support that shift. Anthropic’s internal culture appears to keep seniority from becoming a substitute for shared understanding, with hands-on onboarding and a consistent theory of mind about model capabilities. ⁹ That kind of discipline matters because evals only work if the people writing them can reliably agree on what they are testing.

The same source also suggests that frontier capabilities often need product scaffolding to become usable, with examples like Claude Code and small Labs pods for discontinuous bets. ⁹ That reinforces the broader thesis: at Anthropic, the model and the product are co-developed through measurement, not sequenced as “build first, document later.”

What founders and investors should take away

The lesson is not that PRDs are dead. It is that for AI-native products, PRDs are no longer sufficient as the main operating artifact.

If your team cannot translate a feature into test cases, thresholds, and failure modes, you probably have not settled the product decision yet. ^{3, 6} That matters for founders because it affects speed, coordination, and launch quality. It matters for investors because it is a signal of whether a company has a repeatable way to manage model behavior as complexity rises.

The practical playbook is straightforward:

1. Start evals early, before the system is too live to reason about cleanly. Anthropic’s own guidance says late evals turn into reverse-engineering. ¹
2. Build from real failures, not imagined ones. The stronger sources here consistently point toward production traces, regressions, and concrete examples. ^{1, 3}
3. Treat evals as a living product asset. If the suite is thin, the product definition is thin. ⁶

That is the structural shift at Anthropic. The company is not just testing more. It is making measurable behavior the unit of product definition for model-centric work. For teams building in the same space, that is the part worth paying attention to.