Brendan Foody
Why experts writing AI evals is creating the fastest-growing companies in history, covering AI product work, product design, and strategic decision-making.
Episode
Why experts writing AI evals is creating the fastest-growing companies in history | Brendan Foody (CEO of Mercor)
Summary
CEO of Mercor — which grew from $1M to $400M annual revenue run rate in 16 months — discusses how they identified the opportunity to be the marketplace connecting expert professionals to AI labs that need high-quality human evaluation data. The core insight: model improvement is bottlenecked by humans who can measure model capabilities, and frontier labs will pay whatever it takes for domain experts to write evaluations.
Key Takeaways
'If the model is the product, evals are the PRD' — evals define what success looks like for a model the way a product requirements document defines success for software.
The market size for evals is bounded by the set of tasks where humans can still do something models can't. That boundary is shrinking, but the hardest tasks will require human evals for years.
The fastest-growing companies aren't built on novel technology — they're built on identifying structural bottlenecks in industries with unlimited demand.
Models are only as good as their evals. Before building fancy fine-tuning pipelines, ask whether you have robust evals to measure what you're actually trying to improve.
The future of work is an 'RL environment machine': every economic activity will eventually become a context for generating training data and reward signals for models.
Notable Quotes
“Well, let's see. I use it a lot to write documents, as you would expect. I also talk to get advice on problems. I find it helpful to just reason through almost as a thought partner because, yeah, I don't know. I find I think better sometimes when I'm talking something through, but I can't talk through everything with colleagues or people around me.”
“So the key question is how long there's going to be things in the economy that humans can do that AI can't do? And I think there's certainly a bucket of people that say we're going to have superintelligence within three years and humans won't play a role in the economy. And that's one school of thought.”
“Tying to the point around initiative and that you can just do things, I encourage everyone, especially with AI and it being so much easier to build, just take the initiative to go out and build products and talk with customers and take that leap of faith because I think that that is in so many ways, the largest barrier to more innovation, the economy in any way that we can support that.”