Discover how Jay Kreps scaled Confluent from an open-source project to a public company. Learn his framework for CEO success, product marketing, and building...
From Engineer to CEO: How Jay Kreps Built Confluent into a Billion-Dollar Company
Key Insights
- The CEO paradox: Operating in a "fog of partial understanding" where you can't know everything but must be directionally correct
- Product marketing matters more than you think: A single 25-page blog post did more for Kafka adoption than years of engineering
- Two critical lenses: "What can we do?" versus "What do we HAVE to do?" – the latter drives real breakthroughs
- Tenacity over optimization: The willingness to grind through difficult problems, even when 50% of the company thinks you're wrong, separates winners from everyone else
- Decentralization + accountability: The secret to scaling without losing the speed and excellence of a small company
The Unexpected Jump: From Software Engineer to CEO
When Jay Kreps transitioned from being a software engineer at LinkedIn to becoming CEO of Confluent, the shift shocked him. The skill sets required are almost exactly opposite. Engineering is about knowable problems with right and wrong answers. The CEO role, especially in early-stage companies, involves massive decisions with unknowable aspects—decisions that will fundamentally shape the company's future, but where you won't know if you're right until years later.
"The CEO job generally, you operate much more in a kind of fog of partial understanding," Kreps explains. As organizations grow larger, it becomes mathematically impossible to understand everything about everything. You must learn to be "roughly directionally right," understanding 80% of what your executives know about their functions. This isn't a weakness—it's a necessity. The key is knowing what matters most and having just enough context about other areas to make sound judgment calls.
The learning curve proved steeper than expected. Kreps had to develop competency across marketing, finance, sales, product strategy, and investor relations—disciplines far removed from his engineering background. But this multi-disciplinary requirement, while exhausting, became one of his greatest assets. It forced him to think systematically about how all parts of the business connect, which proved invaluable when building Confluent's go-to-market strategy.
How One Blog Post Changed Everything: The Power of Product Marketing
Before Confluent became a company, it was an open-source project called Apache Kafka. Despite being heavily used internally at LinkedIn and extensively developed, the project languished for about a year after its open-source release. The technical team had created something genuinely powerful, but the world didn't understand it.
The problem wasn't the technology—it was communication. Kafka was difficult to explain. It required lengthy discussions about real-time data streaming and use cases that most engineers hadn't encountered. Unlike their previous open-source database (which was simple: "a database with fewer features but greater scalability"), Kafka demanded conceptual work. Early promotion through talks and community engagement produced modest results.
Then Kreps made a decision that would become transformative: he would spend weeks writing a comprehensive blog post explaining not just what Kafka was, but why it mattered. "If we ultimately can't express why this is exciting, then I think it's probably not going to be that successful of an open source project," he realized.
The resulting post was unusual—approximately 25 pages long, filled with graphics and illustrations that were both technically rigorous and entertaining. It presented Kafka not as an isolated tool but as a fundamental shift in how organizations should think about data. The post captured how a real-time perspective on data architecture was becoming essential as companies became increasingly interconnected.
The response was immediate and dramatic. The blog post generated far more adoption than years of engineering improvements or traditional outreach. This taught Kreps a crucial lesson: product marketing isn't about slogans or spin—it's about building a complete pyramid of understanding, with the compelling headline sitting atop a foundation of evidence, customer examples, and detailed explanation.
This structure applies universally. The top of the pyramid is the memorable takeaway—your slogan or core message. But it can't exist in isolation. Beneath it must sit the full body of work: data, customer testimonials, long-form arguments, proof of concept. Most failed marketing either lacks a clear top (can't boil down the message) or has a slogan with no supporting foundation (all flash, no substance).
The Leap: From Open Source to Enterprise Company
By 2011-2012, Confluent had momentum. Major companies across industries—media companies, banks, retailers, insurers—were interested in using Kafka. This broad appeal signaled something important: they were solving a fundamental problem that transcended vertical markets.
But the path forward wasn't obvious. Building a company around open source wasn't fashionable in 2014. Red Hat had succeeded with Linux, but other open-source ventures—particularly around Hadoop—had stumbled. Enterprise software wasn't the hot sector; the marquee companies of the era were Google and Facebook, not IBM or Oracle successors. Additionally, cloud was emerging but seemed destined to be AWS-dominated with little room for competitors.
The team spent nine months debating company structure. They considered vertical solutions, different go-to-market models, and various product packaging. But ultimately, after conversations with users who expressed genuine enthusiasm about paying for a productized version, they committed: they would build a company directly around Kafka itself.
Critically, they didn't try to perform perfect market analysis. "A top-down, McKinsey-style market analysis sometimes works less well than just deeply engaging with a problem you're convinced holds significant value," Kreps notes. Their conviction was straightforward: data infrastructure was genuinely valuable to companies, even if the category wasn't obviously "cool." Companies would pay because the problem was real.
When Kreps and his co-founders announced their intention to leave LinkedIn, they expected to struggle with fundraising. But LinkedIn itself expressed interest in investing—the first time they'd done so with departing employees. This created urgency: they had to move quickly to secure a co-investor for a priced round. The result was what began as a seed round that evolved into a Series A, with Benchmark as a co-investor bringing Eric Eshleman and other valuable early supporters.
Building Two Products Simultaneously: The Courage to Do What Must Be Done
One of Confluent's most pivotal decisions came early: simultaneously building both an on-premise software product and a managed cloud service. This wasn't a choice the company wanted to make. It was a choice the market imposed.
The initial software product was pragmatic—it was what they could ship first, and it addressed the existing market for companies running infrastructure in their own data centers. Large enterprises loved it; they were closing deals for $100k+ within the first 12-24 months, despite the product being nowhere near feature-complete.
But Kreps recognized an existential threat: the cloud was becoming inevitable. Amazon had already released Kinesis, a competing service. Other companies might build around Kafka's open-source foundation. If Confluent didn't own the cloud market for Kafka, they might lose strategic positioning entirely.
Here's where Kreps introduces a critical framework that distinguishes successful founders from those who plateau: the distinction between "What can we do?" and "What do we HAVE to do?"
Most organizations focus on the former—asking what's feasible given current team size, budget, and timelines. This lens is natural but limiting. The second lens is more powerful. It asks: what is strategically essential for success? What will the company actually need, regardless of how difficult it seems?
For Confluent, it was clear they had to build a cloud service. The market was moving that direction. Customers who would define their future were increasingly cloud-based. The fact that half the company thought it was a terrible idea, that investors questioned the distraction, that the initial economics looked poor—none of that changed the fundamental reality that they ** had to** do it.
Once that conviction crystallized, the organization found ways. It wasn't pretty. The engineering team was stretched. Cloud product development took resources from the core software business. Some major early customers left when the cloud offering didn't meet their needs. The product was difficult to sell, didn't generate meaningful revenue for years, and looked like an "embarrassing failure" on the company's financials.
But they pushed through. "We just kind of bludgeoned our way through, like, none of it was pretty," Kreps recalls. By the third year of selling, the cloud product was "on the scoreboard" but represented such a small percentage of revenue that it seemed insignificant. Yet it was growing. As product functionality improved and the market shifted toward cloud consumption, growth accelerated. There was no single "crossing the chasm" moment—just relentless focus on solving the hundred small problems required to deliver on the cloud promise.
The lesson: things that seem impossible often become possible once you acknowledge they're essential. Kreps points to examples from sport—the four-minute mile, sub-two-hour marathons. For years, these seemed physiologically impossible. Then someone accomplished them, and suddenly everyone realized they needed to as well. Constraints shifted from physical to psychological. Once people knew these feats were necessary to compete, they found ways to achieve them.
Tenacity: The Underrated Competitive Advantage
When asked what allowed him to succeed as a first-time founder and CEO, Kreps identifies two key attributes: curiosity and tenacity. Curiosity drove him to deeply understand every aspect of the business—customers, markets, competitors, each functional discipline. But he emphasizes that tenacity might be the more important ingredient.
Tenacity isn't optimism, though optimism helps. It's the willingness to grind through problems well past the point where they're painful. It's the capacity to face existential threats that other leaders want to ignore, to lean into those problems rather than away from them, and to keep working even when progress feels impossibly slow.
"Not giving up on something for an extended period of time, even if it's not enough on its own, actually gets you pretty far," Kreps notes. In competitive situations, companies that keep improving while others tap out eventually win. Maybe the path doesn't look pretty initially, but if you eventually get there, the journey doesn't matter.
This requires a subtle balance. You must be extraordinarily committed to the big, strategic imperatives—cloud, market positioning, product-market fit in core areas. But you must remain flexible about tactics and details. The approach to solving a problem should evolve constantly. What should never change is the conviction that you will solve it.
At Confluent, this played out in product strategy. For years, they believed they needed to expand into data processing—not just data streaming, but actual processing and transformation of data in motion. Early products in this space didn't gain traction. But Kreps maintained conviction that this was strategically necessary. They kept iterating, kept investing, kept learning from failures. Eventually, in recent years, this area "really started to take off" and became a significant part of the business.
But this doesn't mean pursuing every idea forever. The distinction is between strategic imperatives (which you pursue relentlessly, changing tactics as needed) and exploratory bets (which you can abandon if the approach simply isn't working). The cloud service was a strategic imperative—they had to have it. Data processing became a strategic imperative—Confluent needs to own that to have true strategic importance. Lesser product bets can be abandoned without philosophical crisis.
Debugging Problems: The CEO's Most Difficult Task
As companies scale, diagnosing problems becomes exponentially harder. A top-level metric is missing its target. But why? Is it poor sales execution? Weak product? Insufficient market demand? Lack of product-market fit? Market timing? Wrong positioning?
The temptation is to blame the most visible actor. Sales team isn't hitting numbers? Fire the VP of Sales. But this reasoning often misses the real cause. The problem might be that the product genuinely doesn't solve the problem customers thought it did. Or the go-to-market story is confusing, so customers don't understand the value. Or the ideal customer is the wrong segment entirely. Or the sales team simply hasn't learned how to explain something that's genuinely complex.
Kreps has found that the CEO must personally spend significant time on this diagnostic work, particularly when issues are important and cross-functional. You can't rely on each function's self-assessment. The product team will often conclude the problem is insufficient features. The sales team might blame product being incomplete. The marketing team might blame messaging. Each has a lens shaped by their role.
But the CEO can ask everyone "Why isn't this working?" collect all the answers, and synthesize them into a coherent hypothesis. This requires intellectual flexibility and willingness to consider that the problem might be in an unexpected place. It also requires a culture where escalation is encouraged—where people feel comfortable surfacing uncomfortable truths about what's not working.
Kreps implements this by telling teams: "Neither I nor anybody on the management team wants you to do something stupid. So if you think you are doing something that is stupid, don't just do it. Come talk to us." This sounds obvious but is remarkably rare in larger organizations where people default to "I was told to do X, so I'll do X" rather than exercising judgment.
The Paradox of Scaling: Maintaining Excellence While Growing
As Confluent grew from a startup to a public company, Kreps faced a consistent challenge: how to preserve the speed, quality, and ownership mindset of a small company while operating at the scale and complexity of a large organization.
Small companies naturally operate like fine-dining restaurants. Everything can be customized, tailored, optimized for the specific outcome. Decisions are made quickly because the context is shared. Accountability is automatic because people are close to the work and results.
Large companies naturally become systems. Like Chipotle—standardized, efficient, consistent, but not exceptional. The natural tendency, and the most comfortable thing, is to build processes that work "good enough" across all situations. This creates efficiency but erodes excellence.
Kreps has learned that preserving pockets of excellence, even if it creates some inconsistency, is actually strategically important. One example: hiring. Teams with leaders who behave as if they must personally fight for every hire—as if there's no recruiting department—massively outperform teams that rely on standard hiring processes. This seems inefficient. But the leaders who are willing to spend extraordinary time finding the right people, who think about hiring as something critical to their domain, who build relationships with potential candidates—they end up with better teams.
In large tech companies with professional recruiting infrastructure, the system actively discourages this. There's a pipeline, a process, a set of incentives designed so that success doesn't depend on individual leaders' ability to find talent. This makes the system more scalable but also more mediocre.
As Confluent scaled, Kreps intentionally resisted over-centralization. The instinct is to create shared functions: one product strategy team, one marketing team, one finance team. This creates leverage and consistency. But it also creates bottlenecks and dilutes ownership.
Instead, Confluent moved toward a structure where distinct product areas operate with significant autonomy. Each area has its own revenue target, dedicated marketing, sales support, and growth strategy. They operate within a larger system that handles some shared functions, but the psychology is one of ownership: "This is your area. You need to figure out what's working, what's not, why customers are using or not using something, and how to unlock growth."
This structure requires the broader management team to actively help unblock issues that require coordination across areas. It requires leaders to think about end-to-end success, not just executing a narrow task. And it requires the CEO to constantly reinforce that the mindset is "figure out what needs to happen" rather than "implement what you were told to implement."
The Core Message About Company Building
When reflecting on what's broadly applicable from Confluent's journey, Kreps emphasizes several principles that he believes are underappreciated in startup advice:
First, embrace the "must do" mentality: Don't optimize for what's feasible given current constraints. Ask what's strategically essential, then reorganize constraints to achieve it. This doesn't mean pursuing everything—it means having clear conviction about what actually matters.
Second, invest in product marketing as a core founder responsibility: Technical founders often want to skip this, believing the product will speak for itself. But explaining why something is exciting, building the pyramid of understanding, articulating the strategic importance—these are among the highest-leverage activities a founder can do. Not as a marketing department responsibility, but as the founder's own work.
Third, be tenacious about the big things, flexible about the details: Your conviction about strategic imperatives should be unshakeable. But your approach should constantly evolve based on learning. The willingness to try different tactics, different teams, different market segments—while maintaining conviction about what must eventually work—is the recipe for breakthrough success.
Fourth, maintain small-company excellence as you scale: This is genuinely difficult and requires constant attention. It means resisting the natural drift toward standardization and process. It means empowering distinct units to own their outcomes. It means having leaders willing to operate outside normal business parameters to achieve excellence.
Fifth, personally engage in diagnostic work: The CEO must spend substantial time understanding why things aren't working. This can't be delegated. It requires connecting insights from across the organization into coherent hypotheses about what to change.
Conclusion
Jay Kreps' journey from LinkedIn software engineer to Confluent CEO demonstrates that success isn't about having all the answers—it's about asking better questions, maintaining conviction about strategic imperatives while remaining flexible about tactics, and having the tenacity to grind through problems that initially seem impossible. The most powerful tool Confluent deployed wasn't advanced engineering—it was a 25-page blog post that helped the world understand why real-time data streaming mattered. As you build your own company, remember that the distinction between "what can we do" and "what we HAVE to do" often separates winners from everyone else. Once you know something is essential, you'll find ways to achieve it that seemed impossible before.
Original source: Building Confluent: From open source side project to public company | Jay Kreps (Co-founder and CEO)
powered by osmu.app