The eternal pivots of AI companies

On 12 October, Andrej Karpathy (an OpenAI co-founder, now at Anthropic) dusted off one of his posts from April and split the world into four groups. Around 6 billion people have barely touched an LLM (large language model). Between 1 and 2 billion use a free chatbot, mostly as a better search engine or a writing aid. About 20 million professionals run agents on real projects. And roughly 5,000 people have access to the internal frontier systems, of which he wrote that "the external world has seen the preview".

The second group is the one that stayed with me. For one or two billion people, AI at the end of 2026 is a text box and a send button; the same text box that went on the web in November 2022. Everything around it changed, several times. The box did not.

And I think this says something about how we measure progress. We count it in models; version numbers, benchmark tables, a launch event every few months. But when I look back at the last three years, most of the progress I can point to did not happen inside the model. It happened in what the software does with the model. And every time one of those levers stopped giving, the companies pivoted to the next one.

The Never Ending Story

Let me count the pivots. I get four, and the order matters.

The first was the prompt. GPT-4 came out in March 2023, and for that whole year (and a good part of the next) the way to get more out of a model was to write better input for it. Prompt engineering became a job title (Anthropic advertised one, if I remember correctly, at up to $335,000 a year), with courses, cheat sheets and "act as a senior engineer" incantations. It reminded me of the early 2000s, when people compiled Gentoo with -O3 -funroll-loops and swore the desktop felt faster :)

The model was a closed box, so we tuned what went into it. In January 2024 I was pasting Python into ChatGPT and asking it to find the type errors that pyright missed, and a bit of prompt engineering was what made the difference. As a job title, it ended the way the compiler-flags era ended; the models learned to read a sloppy prompt, and the trick moved inside.

I should declare an interest here. I build Promptivo, a tool that scores prompts, and in April I wrote about why prompts fail. The engineering ended; the prompts did not. There are more of them than ever. They are just not written by people anymore.

When an agent hands a task to a sub-agent, the brief is a prompt that a model wrote. When the context fills up, the model summarises its own session, and that summary is the prompt for everything that follows. Even the CLAUDE.md file, the one place where a person is supposed to talk to the harness, is often generated by the agent first (/init) and touched up by hand after (and there is a skill whose job is to write skills).

So the prompt engineer of 2023 is still at work. It is the model, writing for itself thousands of times a day, and I suspect that very few people read what it writes.

The second was thinking before answering. Chain of thought started as a prompt trick in 2022 (the famous "let's think step by step"), and in September 2024 OpenAI's o1 trained it into the model with RL (reinforcement learning). DeepSeek R1 showed in January 2025 that it could be done cheaply. (The month after o1-preview came out I wrote about Apple's GSM-Symbolic paper; change the names and the numbers in a grade-school maths problem and every model lost accuracy, o1-preview included, 17.5% of it on the hardest variant.)

Around the same time, MoE (mixture of experts) architectures made each token cheaper by waking up only part of the network (Mixtral, in December 2023, was the first popular open one), and a long list of inference tricks did the same from the outside; quantisation, speculative decoding, caching the prompt (I wrote about BitNet running models on CPUs in 2024, and last month I tried to compile a model's weights into a program, only to find that the wall that decides the speed is the memory bus, not the compiler). All of these have the same shape. They spend the compute at the moment you ask, not at the moment you train.

Why did the labs need that? Because the old lever was getting expensive. Ilya Sutskever said it at NeurIPS (the Neural Information Processing Systems conference) in December 2024; pre-training as we know it will end, because there is only one internet to train on. (I had written something close to it that summer, in my own words; first about models that will end up feeding on their own output, and a month later that we need a new paradigm, not faster matrix multiplications.) GPT-4.5 arrived in February 2025, very large, very expensive and a modest step, and OpenAI took it off the API (application programming interface) a few months later.

The third was the harness. MCP (Model Context Protocol) in November 2024 (a year later I called it more marketing than engineering, useful though), Claude Code in February 2025, Codex CLI (command-line interface) in April. The model was put in a loop with tools; read a file, run a command, read the error, try again. That is the year the same model started to look smarter every few months, sometimes without being retrained at all.

The fourth is the one we are in; agents. In February Karpathy retired his own "vibe coding" and proposed "agentic engineering" in its place (you do not write the code anymore, and orchestrating what does is a discipline of its own). Then the agents left the terminal for the desktop, and then they started running in groups.

Agents are not new, of course. I studied intelligent agents at university in the late 90s, and in February 2025 I wrote that agentic AI was mostly an AI in a workflow system (and asked where the custom GPTs of 2023 had gone, which were a pivot of their own). In April I went through four studies of what agents actually deliver, and in September a 135-page survey put a model in every box of the org chart.

Ok, the models got better through all of this, and I am not taking that away from anyone.

But look at what they are trained for now. The reinforcement learning happens inside the harness; the model is rewarded for tool calls that work, patches that pass the tests, proofs that the checker accepts. Karpathy makes the same point in his post; capability is concentrated where the results are digital and can be verified automatically, because that is where a reward can be computed. OpenAI even shipped a model (GPT-5-Codex, September 2025) presented as optimised for agentic coding inside one specific tool. So the model is no longer trained mainly to answer a person. It is trained to produce good input for a harness.

Deeper Blue

So what is a harness, if we open the lid?

It is a loop with three parts; a generator that proposes, an executor that runs the proposal, and a checker that says whether it worked. The model is the generator. The compiler, the test suite, the shell or a proof checker is the checker. And the harness decides the rest; what to try next, what to keep, when to stop, and what goes back into the context for the next round.

I watched this loop work on something of my own in September. I handed Claude Code a 74 KB DOS executable from 1997, a text adventure a friend and I wrote as students, whose source is long gone, and asked it to make it run in a browser. It did, in one afternoon. The model was good, but what made the afternoon possible was the checker at the end of the loop; does the game play?

We have seen this machine before. Deep Blue beat Kasparov in May 1997 (the upgraded version, unofficially called Deeper Blue) by looking at around 200 million positions a second. Its evaluation function was a heuristic, hand-tuned and not very deep; the search is what made it dangerous. Today the model is the evaluation function, a far better one, and the harness is the search.

And the search is where the cheap gains are. In February, LangChain took a fixed model (gpt-5.2-codex), changed only the harness (the system prompt, the tools and the middleware around the model calls) and moved their agent on Terminal-Bench 2.0 from 52.8% to 66.5%, from outside the top 30 into the top 5. These are their own numbers, I have not rerun them. But look at what they changed. A checklist that stops the agent before it declares victory and makes it verify against the task. A counter that notices when the agent keeps editing the same file and tells it to rethink. A warning when the time budget runs low. And maximum reasoning for planning and checking, with less for the typing in between (maximum everywhere scored 53.9%, because it ran out of time).

None of this is AI research. It is what a decent senior engineer tells a junior in the first week (and every piece of it reaches the model as a prompt, written by the harness at the moment it is needed). And this is why every three months the same model solves harder problems; somebody wrote the next page of that advice into the loop.

Infinite monkeys, finite budget

The mathematics is where this shows the most. In September OpenAI announced its answer to the Navier-Stokes Millennium Problem, and I have written about it three times already; first about the call to slow down that followed it, then about how the run actually worked, and then about the Lean translation that nobody signed. The mechanism, from their own account; around ten thousand agents running at once, in groups, each group prompted with a different version of the problem because nobody knew which way the answer went, most of them going nowhere, and a consolidation step that fed the useful partial results to the others. 88 hours, 2.7 million messages, around 130 billion output tokens. Then Lean checked the result.

That is search. Is the proof correct? The checker says yes, for the statement it was given (whether that statement is the right one is a separate story, and the third post tells it). Is it the best proof, or an elegant one, or one that teaches us something new about fluids? Nobody can tell from the run. It is a very large heuristic, pointed at a problem, running until something passes the checker.

And Navier-Stokes was not a one-off. OpenAI later opened its openai/math repository; an internal model was posed about 4,000 open problems, and what came back is 722 manuscripts in 372 families (around three hours of thinking each, on average, according to the README). Earlier this month I took one of them, a polynomial-time 2-approximation for the shortest common superstring, implemented it and measured it against the old algorithms. The proof holds, and a bound that people had chipped at since 1988 was closed by a model. And on average the textbook Greedy algorithm still wins; on genome reads Greedy found the optimum 98% of the time and the new algorithm 74%, at ten times the cost on small inputs. Correct, yes. Better in practice, no. That is exactly the line between what a checker can tell you and what it cannot.

We have been here before, many times. ENIAC (the Electronic Numerical Integrator and Computer) computed 2,037 digits of π in 1949, in about 70 hours. GIMPS (the Great Internet Mersenne Prime Search) has been running on volunteers' machines since 1996. In October 2024 Luke Durant, a former NVIDIA engineer, found the largest known prime, 2^136,279,841 − 1, with 41 million digits, on thousands of rented cloud GPUs (graphics processing units) in 24 data-centre regions, at a reported cost of around $2 million of his own money. The Lucas-Lehmer test was the checker. Nobody said that the GPUs understood primes. We said that a person with a lot of GPUs found one.

Of course, brute force is useful, and I am the last person to argue against it (in the first of those posts I wrote that the GPUs should be left to be as brute force as they can be). What changed is that the guesser got good enough to guess at proofs, and not only at numbers.

Behind the curtain

Which brings me back to Karpathy's smallest group, the 5,000. What they see, according to him, is swarms of thousands of agents working together for weeks on very large projects.

I have no inside view, so this is my read and only that. I do not think the internal models are a different species. I think they are the same family of models, with a harness and a budget that no product can offer; as much compute as they want, as many agents as they want, no rate limits, and none of the restrictions that the public versions carry. The Navier-Stokes run was exactly that; a model whose training had started days before, on a fleet nobody outside could rent, with people steering the consolidation.

The third group is the one I can speak for, because I sit in it. From there the uplift is real, and it is mostly a harness uplift; the same kind of agent that brought my old game back is now copying whole application suites, and this month I wrote about what that does to the price of software.

So the gap in his funnel is, I think, mostly a gap in harness and budget. The billion or two meet the model through a chat box with a message limit. The 5,000 meet it through ten thousand agents and an electricity bill. And when the tricks that work inside get written up and shipped, the public model "jumps" three months later.

Epilogue

So, where are we? The harnesses are maturing. The tricks are converging (plan, verify, detect the loop, split the work into sub-agents, compact the context) and they are cheap to copy; LangChain's whole improvement fits in a system prompt and a handful of middleware hooks. When everybody holds the same lever, it stops being an advantage. That is why the next pivot is coming, and why the companies are searching so hard for the next application rather than the next model. I do not know what it is. I am not sure they do either. If history is any guide, it will be an old idea with a new name; it happened with the network computer, and it happened with agents.

What I can see is who pays. Follow the bill through the four pivots. The prompt was paid with our time. The reasoning was paid with thinking tokens, billed as output. The harness was paid with loops that read the same files ten times. And the agents are paid with fleets that only a lab can afford (130 billion tokens for one theorem). Each pivot moved more of the cost out of training and into running, and each one moved further away from the text box where most people still meet AI. And none of them moved the cost of owning what comes out; reading it, running it, answering for it when it breaks (I wrote about that twice this month, for the process and for the people).

Karpathy wrote that the external world has seen the preview. I agree with him, more than he might like. For three years the preview was the harness, and the model was in it all along, a little better each time, in a supporting role. The next pivot will be paid for, like all the others, by whoever holds the token bill.