Frankle: enterprise AI is stuck on how to measure it, not on how smart the model is

Empiricism won: measure the behaviour, do not prove it

He goes back to 2018, when the argument in deep learning was whether there would be a clean theory or whether the field would have to measure. His lottery-ticket paper's review scores were two 9s and a 5, the 5 for having 'no proof', and a faculty interviewer asked whether he would do anything 'principled', meaning theoretical and provable. He found that insulting because empirical work is principled too. He says the paper's findings 'didn't really stand the test of time'; what lasted was empiricism as a lens, which lets you ask big questions theory cannot answer. Today nobody writes a proof that a deep-learning model is good, which is why evaluation companies exist at all.

“empiricism is principled.”Jonathan Frankle

The bottleneck is saying what you want and knowing whether it was done

He draws the line at Sonnet 3.5: after it, models are good enough much of the time for many complex enterprise and knowledge tasks. Sit even the smartest person at a desk and say 'do work' and that is an underspecified task, so the hard part is describing the task and knowing whether it was done, which is an evaluation problem. He says even a model ten times smarter than Mythos would fail if the task and the standard of good are not set. He adds that 'the world is fractal in complexity': finance is thousands of small tasks, each different in every company because of its history, and perhaps three people in the world understand any one of them; so a 'finance benchmark' or a 'biology benchmark' he finds absurd and disrespectful to the people who do the work.

Prototype on the strongest model, train your own only once committed

His analogy is software: write the prototype in Python and move to Rust only once it is worth it. Agents are the same: take a foundation model you like, write a little prompt, make one or two evaluation examples and see whether the task is possible at all; high cost does not matter at this stage. Once people like it and it will last, move up: automatic prompt optimisation such as GEPA run against your own evaluation set; then see whether a smaller model works; only last a custom-trained model, which he says can cut cost by 5x or more, and says Cursor and Harvey have both mentioned recently. Evaluation grows out of vibe checks: everyone has a few questions they ask every new model, and you start there and add cases from production incidents, the way Waymo issues a recall and adds it to its suite.

Public benchmarks advance the science, not your business

He says he has never met anyone who truly likes a public benchmark, yet one like ImageNet advanced the whole field. Their bias comes from what academia can build and what it thinks matters: toward coding, though software engineering is a small part of the economy (he says about 1%); reinforcement-learning environments lean to coding too, since 'a Docker container is enough to hold an environment' is a coding view, while many Databricks environments are tens of petabytes, and letting an agent change one and revert it at once without breaking production is a new systems problem. What academia lacks is scale: working on 16 H100s is not the same as training models inside a company. So Databricks mostly uses benchmarks annotated on its own data, and also publishes its own, such as OfficeQA, a parsing and reasoning benchmark built on US Treasury documents.

Cost, permissions, security: the three things enterprises ask first

He says enterprise customers now ask first about cost, then governance (what an agent should be able to access), then security, and jokes that the era of token maxing is over and the era of value maxing is here. Governance is hard because a person has permission to hundreds of documents they have never opened, and an agent will find them, in ways a human would not predict; he gave an agent access to Jira and a Google document and narrowly avoided a complicated situation because he had not thought about what the two together would expose. So he argues evaluation cannot only ask whether the task was done; it also has to ask about the process: what permissions were used, what was accessed, whether it should have been. He says no one has really figured out how to measure this.