How Anthropic's First PM Built Claude Into a Frontier Model

In 2023, Anthropic seemed like an underdog. OpenAI was dominant. Few people talked about Claude for coding. But within three years, the company shifted from a handful of engineers to generating an estimated $50 billion in annual recurring revenue—a valuation that major companies used to go public at.

Dianne Penn, Head of Product for Anthropic's AI Research and Labs teams, joined as the company's first technical product manager when there were only five product engineers. She has shipped every Claude model from version 2 through the latest releases, and helped incubate products like Claude Code, Computer Use, and Skills.

Key Insights: Product Strategy in the Age of Frontier AI

• Evals replaced PRDs as the core product artifact. Instead of writing traditional Product Requirements Documents, Anthropic's product teams now build evaluations—structured test sets that define what success looks like for a feature before shipping.
• "Sweat the tokens as much as you sweat the pixels." Product managers must deeply analyze token usage and model behavior, not just UI/UX, to understand how users interact with AI systems.
• Frontier models and frontier products amplify each other. Claude Code wouldn't have accelerated without Opus 4.5's capabilities; Opus 4.5 wouldn't have had impact without Claude Code's user experience.
• Early adoption and hands-on experimentation drive discovery. Anthropic tested prototypes internally before shipping, discovering use cases through real conversations rather than planning them in advance.
• Ambition compounds over time. The question isn't just "What does Claude do today?" but "What should we build today so it works with Claude 8?"

The Moment Anthropic Found Its Identity

When Anthropic launched Claude.ai, the product was another chat interface. The real inflection came in early 2024 with "Golden Gate Claude"—a quirky prototype based on research into how features activate within model layers.

Researchers discovered that models develop specific "features" (like obsessing over the Golden Gate Bridge) that activate at different layers. The team built this into a live experience on Claude.ai for 24 hours. It reached only about 2,000 people, but it proved something crucial: Anthropic could create novel product experiences that showcased research in ways competitors couldn't.

This happened because individual contributors—engineers and designers—volunteered their time in a bottom-up culture. It wasn't a mandated roadmap item. That approach became foundational to how the company defines itself.

From Small Team to Frontier Model: The Opus 3 Moment

Training Opus 3 was a watershed moment. The company had fewer than 200 people. Diverse teams (inference, research, fine-tuning, pre-training) worked toward one shared objective. Dianne recalls being at home during December, some researchers in their childhood rooms, intensely focused on optimizing the model's training and performance.

The team discovered something critical: people were using AI models not just for code autocomplete, but for writing long-form code. Competitor models like GPT-4 could do this, but it wasn't a focus. Anthropic made a relatively small training adjustment to excel here—and it became a key differentiator. Early developers genuinely believed this level of coding capability wasn't possible.

How Product Management Changed: Evals Are the New PRDs

Anthropic's product teams work differently now. Instead of writing PRDs and hoping feedback comes later, they build evaluations first.

Here's how it works:

Example: Following JSON Schemas

Early Claude versions struggled to output data in specific formats (like JSON). Users complained Claude "doesn't follow instructions." But that feedback was vague. Dianne's team went deeper—asking for exact examples, specific paragraphs, and exact failures.

They discovered 80% of the problem was Claude failing to generate correct JSON.

The team then:

1. Collected 30-40 examples where Claude failed at this task
2. Built a test set (evaluation) with a prompt and expected "golden answer" output
3. Ran this evaluation against each new Claude version
4. Tracked improvement (now it achieves 99.9%+ accuracy)

This evaluation became actionable for researchers. They knew exactly what to fix and could measure progress. It replaced the abstract feedback loop with a concrete, measurable target.

Why This Matters

Traditional PRDs describe what to build. Evaluations describe how to know if it worked. In AI product development, the latter is more valuable because:

• Models are non-deterministic (they don't give the same answer twice)
• Success is hard to define in words alone
• Researchers need measurable targets to train toward

As Dianne puts it: "Evals are the new PRDs." But PRDs still matter for large-scale coordination across teams (safety, legal, engineering).

The Token-Pixel Principle: Product Managers Must Understand Model Behavior

Most product managers focus on UI/UX ("pixels"). Anthropic's product managers must also deeply understand how tokens flow through the model and what that means for user experience.

When analyzing user feedback—like "Claude hallucinated"—the team doesn't stop there. They ask:

• Should Claude have called a tool at that moment?
• Did Claude retrieve the right document but misread the facts?
• Is this a tool-use failure or a knowledge-synthesis failure?

Each diagnosis leads to different fixes. Token-level analysis is how they make feedback actionable for researchers.

Dianne: "You have to sweat the tokens as much as you sweat the pixels."

Hands-On Experimentation: The Engine of Discovery

Product people at Anthropic don't theorize about what Claude could do. They actively use it across multiple versions.

Dianne has interacted with over 20 research model versions and more than 20 production Claude versions. Early on, the entire company tested Claude in a shared Slack channel, trying different use cases organically. When one person discovered something new, others would iterate and build on it.

This public experimentation revealed use cases nobody planned:

• Editing essays
• Drafting emails
• Writing long-form code

The culture encourages this discovery. It's not individual—pairing with someone excited about the technology makes exploration feel like joy, not work.

Labs: Incubating Discontinuous Bets

Anthropic has a dedicated "Labs" team that pursues ideas beyond the core roadmap. The team has shipped Claude Code, Skills, MCP (Model Context Protocol), Computer Use, and Design features.

The Labs approach:

• Hold strong convictions about the theme, stay flexible on the prototype. Teams believe in the broad direction (e.g., agentic behavior) but iterate rapidly on the exact implementation.
• Prefer small teams for ambiguous problems. Large teams slow down zero-to-one exploration.
• Hire for personality. People who love experimentation, who can handle bets being turned off, who are genuinely passionate about pushing boundaries.

Some prototypes don't ship. But the learning—understanding what the model can do—becomes valuable data for future product decisions.

The Ambitious PM: From "Be Better" to "Go 1000X"

Product managers at Anthropic ask themselves: "Let's say Claude 8 comes around. What changes in what users do? What should that mean for how we're building today?"

This forces forward-compatible thinking. You're not optimizing for today's constraints; you're building for exponential improvement.

Most people weren't involved when the internet became ubiquitous. Similarly, most people won't remember the moment AI became foundational to work. The key skill is adaptability—the ability to change course when faced with new information, rather than sticking to an old plan.

Why Claude Is Trusted: Constitution Meets Capability

An interesting observation: Claude is praised not for being compliant, but for knowing when to push back.

Users who switched from Claude to competitors found those models lacking. Why? Claude's focus on safety, alignment, and a clear "constitution" (how it thinks and operates) doesn't limit it—it enhances it. Claude challenges assumptions. It acts as a thinking partner, not just a task executor.

This is why it's more useful. Good thinking partners disagree. They add to your ideas, not just agree with them.

The "Jagged Edge": Why Writing Still Needs Work

Claude is strong at many things—coding, reasoning, agentic tasks. But writing isn't perfect. The model produces output that's clearly written by AI, despite being a language model trained on human writing.

Why? Technology has "jagged edges." As models improve at one capability (agentic behavior, tool use), other areas (tone, character, long-form writing) become the new rough edge. Anthropic is now actively investing in writing quality, recognizing it as a priority.

This connects to a broader principle: You can't improve what you can't measure. And you can't measure what you don't test against. That's where evals come in again.

Living in the Future: The Token-Spend Mindset

Anthropic founder Dario Chao made a prediction: coding will be 100% solved by AI within a year. People thought he was crazy. He was right.

He also talks about the exponential curve we're on. Every improvement results in a massive leap forward. Being inside this moment is "crazy" and "historic."

How should you prepare? One framework: If you're willing to spend $100,000 a year on tokens right now, you're living the way somebody in 2028 will live.

Think of token spend as the input. Experimentation and discovery are the output. If you orient goals around experimentation rather than just using the model, you'll learn what's actually possible.

What Makes a Great PM in AI

Anthropic hasn't changed its hiring loop in three years. They look for:

1. First-principles thinking. Not pattern-matching from past roles, but reasoning through what user value looks like right now with this technology and this audience.
2. Being hands-on. PMs must ship. They must spend time directly building with Claude, understanding failures at the token level, reading user transcripts deeply. No substitute for this.
3. Developing judgment. Judgment is accumulated nuance and experience. It won't be automated. It's where humans remain most valuable.

And critically: Do you actually love working with this technology? The most successful people are those who genuinely enjoy exploring what's possible, experimenting in public, and finding joy in collaboration.

Avoiding Burnout: It's Not an Individual Sport

Anthropic ships at an extraordinary pace. In the previous year, the company released four model series. In the second quarter alone, it shipped more than that.

How does Dianne avoid burnout? It's not by taking time off (she comes back to 3x the work). It's by having a team that looks out for each other.

The night before a launch, people who aren't even the directly responsible individual will stay up to help—reviewing blog posts, improving demos. The team practices what Dianne calls "entering the hive mind"—radical ownership without isolation.

She credits her partner for support (she's been in AI for six years, at Amazon and then Anthropic) and emphasizes that this kind of pace is unsustainable alone. Culture and team structure make it work.

Key Lessons for Product Leaders

1. Go deep, not broad. Experimentation with joy comes from pairing with others and going deep into a couple of things, not dabbling in many.
2. Protect your thinking. Use AI as a sparring partner, not a replacement. Develop your own point of view first, then refine it with Claude.
3. Measure what matters. If you can only measure revenue, you drift from values. Better metrics help you sustain culture.
4. Hire for conviction and low ego. People who care about impact, not title. People who will help the team succeed.
5. Ask "What's the most ambitious version?" Don't just 10x—push toward 1000x. Then build today so it works with that future.

Conclusion

Anthropic's path from underdog to category leader wasn't about being first or having the most resources. It was about building authentic product experiences that demonstrated research in novel ways, using hands-on experimentation to uncover what users actually needed, and creating a culture where evals replace guesses and tokens matter as much as pixels.

For product leaders navigating AI, the lesson is simple: Go deep, stay hands-on, build evals instead of guessing, and remember that the most valuable work happens in teams, not alone.

---

Original source: Anthropic’s first technical PM on token maxing, the jagged edge, and living in the future

powered by osmu.app