Discover why expert data and validation environments are critical to AI progress. Learn how companies scale supervision and build better models beyond archit...
Why Data, Not Algorithms, Is AI's Real Bottleneck in 2026
Key Insights
- Data quality matters more than model architecture — Tesla dedicates 75% of effort to data vs. 25% to models, the opposite of typical PhD research
- Expert supervision is the true constraint — Limited domain experts (doctors, lawyers, engineers) create a bottleneck that GPU power and architectural innovation cannot solve
- Data is a product, not a commodity — UI changes or schema shifts render historical datasets obsolete, requiring continuous curation
- Data 2.0 requires sophisticated validation systems — Moving beyond simple labeling to expert-designed environments, test specifications, and nuanced reward models
- Language transfer is asymmetric and scale-dependent — Multilingual data benefits vary by model size and language pair; transfer relationships must be measured, not assumed
The Shift: From "Data is Commodity" to $100B Market Cap
When Focal Systems was founded in 2016, conventional wisdom held that "data is commodity." The machine learning community believed large datasets like ImageNet were sufficient; additional data seemed unnecessary. Major venture capitalists predicted data businesses would have "terminal value of zero."
The reality proved starkly different. Over the past decade, the data sector has generated over $100 billion in market cap, becoming one of Y Combinator's most successful investment categories.
The evidence is clear: in production AI systems, data allocation differs radically from academic research. While PhD work typically dedicates 95% effort to models and algorithms and only 5% to data, companies like Tesla reverse this ratio, investing 75% in data and 25% in models. In deep learning interviews, candidates who suggest improving an F1 score by tweaking model architecture (e.g., swapping ReLU for GeLU) miss the point entirely. The correct approach is meticulous data analysis—categorizing false positives and negatives to identify root causes. In an "in-stock/out-of-stock" detection system, poor performance may stem from fog on a refrigerator or obstructed views, not architectural flaws.
Andrej Karpathy's early ImageNet work reinforced this: manually analyzing images to establish human baselines revealed that even humans struggle with fine distinctions—particularly distinguishing dog breeds, which constitute a large portion of ImageNet. This underscores that data quality is paramount, even with advanced architectures.
The Expert Supervision Bottleneck
Expert data and RL environments represent the true constraint on AI progress. The sector distinguishes between two task categories:
Automatically verifiable tasks: unit tests, math problems, games, finite element analysis
Not automatically verifiable tasks: design, usability, code style, medical diagnoses, legal judgments
The latter categories rely on subjective human expertise where even experts may disagree. There are only so many cardiologists, MRI specialists, or senior software engineers in the world—and their time is finite.
This bottleneck persists because datasets and RL environments are products, not .zip files. They demand ongoing craftsmanship. A simple UI change by Salesforce can render years of accumulated user interaction data obsolete, necessitating fresh data collection. This perpetual need for updated, expert-curated data invalidates the belief that data businesses lack long-term value.
The inability of AI models to outperform random chance in predicting S&P 500 returns exemplifies this constraint: achieving human-level or superior AI capabilities in trading, accounting, medicine, and law requires specialized expert data that currently doesn't exist at scale.
Data 1.0 vs. Data 2.0: Scaling Expert Judgment
Data 1.0 relied on manual labeling and basic quality assurance—simple preference labels ("good"/"bad") requiring ~30 seconds of human judgment per sample. This approach has fundamental limitations:
- Not scalable: costs grow linearly with data points
- Noise sensitivity: requires redundant labeling (KxN re-annotation) to measure quality without ground truth
- Inflexible: schema or task definition changes force re-labeling from scratch
- Unattributed: labels are claims without source justification
In specialized domains (medicine, law, engineering), manual labeling is often impractical. Asking an MD-PhD to hand-label thousands of MRI scans is unrealistic.
Data 2.0 introduces sophisticated validation environments. Rather than simple labels, it encodes expert knowledge as software. Data programming—expressing expert heuristics as deterministic functions—enables:
- Scalable reuse through software patterns
- Adaptation as specifications evolve
- Auditability, discussion, and collaboration
A validation agent approach exemplifies this. Expert engineers define high-level validation specs (user stories, functional requirements) without over-specifying solutions. An agent then implements deterministic test scripts that:
- Execute the generated code against the codebase
- Measure both correctness and code quality ("taste")
- Assess whether solutions reflect senior engineer standards (architecture, conciseness, maintainability)
This methodology, pioneered in Senior SWE-bench, required extensive calibration. Rather than hand-grading every code sample, a small team of senior engineers scaled their judgment through carefully designed reward metrics and automated testing frameworks. The resulting benchmark revealed that multiple frontier models (Claude 3.5 Sonnet, Opus, Llama) now tie for top performance—indicating the Pareto frontier is advancing across the board.
Data's Role in Multilingual and Domain-Specific Models
Language transfer effects demonstrate that data interactions are complex and scale-dependent. Thai represents only 0.6% of English tokens in common crawl corpora, creating severe data scarcity for Thai-specific models. A purely monolingual approach leads to overfitting due to repeated epochs. The solution is strategic mixing: substantial Thai data supplemented by synergistic languages like Indonesian, Malay, and even linguistically distant languages like English or Croatian—provided they cover similar topics.
Critically, language transfer is asymmetric. If Indonesian helps Thai performance, the reverse may not be true. Transfer matrices must be empirically measured rather than inferred from linguistic theory. Additionally, transfer relationships shift with model scale: smaller models show more negative transfer (language interference), while larger models better accommodate multiple languages and reveal stronger synergies.
These principles extend beyond languages to domains, data quality tiers, and other data source categories—any scenario where understanding how distinct datasets interact is necessary for optimal model development.
Conclusion
Data is not a commodity to be downloaded and forgotten. It is the critical bottleneck in modern AI progress—more constraining than GPU availability, architectural innovation, or computational efficiency. The frontier is moving from Data 1.0 (simple labeling) to Data 2.0 (expert-validated environments and sophisticated reward models) because building human-level AI in specialized domains requires encoding, scaling, and validating expert knowledge itself. As AI capabilities expand, the data challenges compound: environments grow more complex, outputs become less verifiable, and agent autonomy increases. Success depends not on throwing resources at labeling, but on thoughtful system design that empowers domain experts—engineers, doctors, lawyers—to scale their judgment effectively.
Original source: Going In Deep On Data | YC Paper Club
powered by osmu.app