Earlier this year, the Leiden Declaration was published, addressing how mathematicians ought to integrate AI tools (primarily LLMs, if we’re being honest). It has since been joined by a letter from Fields Medalists warning of a Severe Misalignment of AI in Mathematics.

Taken together, I think these are some of the most consequential things written about the future knowledge work in the face of AI.

What interests me is that the debate is often framed as being about whether AI is becoming more capable of doing mathematics. I think that misses a deeper question: to what extent does AI progress itself depend on mathematicians? And, further, what role do the current mathematical LLM users contribute to progress?

My thinking here is straightforward. Frontier users are not merely consumers of frontier AI systems: they are increasingly part of the process by which those systems improve. If that relationship weakens, some of the apparent pace of AI progress could weaken with it.

I’m fascinated by mathematics as an area of study, but I am not a published mathematician. My career is not particularly in jeopardy in this situation (at least to the extent that I do maths in my work), and I have relatively little skin in this game.

What I have noticed, however, is that this debate often gets reduced to anti- versus pro-AI positions. I think the reality is vastly more complicated than that.

How we got here (brief notes on reasoning models)

November 2025 has turned out to be a pivotal moment in hindsight. The reason for this, aside from the specifics of Opus 4.5 and GPT-5.1 being released, was the broad acceptance that language models could be run for longer periods of time, create longer outputs, and still make sense at the end of a conversation. This gave rise to practical examples of tool use, critical for agentic workloads, as well as increasingly capable reasoning.

What allowed models to make this leap? Within the industry, there’s a lot of discussion about the amount of progress being made across both models (the AIs themselves) and the harnesses (the systems that run them and connect them to tools and other AI). The general impression is that models are becoming consistently more powerful. But this isn’t completely true.

While the models are undeniably improving, the basic technology inside them hasn’t changed dramatically. They are larger. There are new methods for encoding word positions, caching intermediate information, and a variety of other internal improvements. Arguably, though, the specific step change came from training.

A reasoning model is very similar underneath to previous non-reasoning models. When we talk about agentic LLMs, what we’re really talking about is LLMs that have been trained to use tools. When we talk about reasoning models, we’re talking about LLMs that have been trained to reason. The main change was training, not technology.

How to train a model that can reason

It turns out that training a model to reason is, at a high level, quite simple. Initially, models were trained in a relatively supervised fashion, meaning they learned from material that humans had already judged or scored. The outputs were then aligned by having humans evaluate the quality of responses and feed those assessments back into the training process. This led to models that were effectively rewarded when human judges thought they had performed better.

This approach creates a number of problems: a tendency towards flattery, for example, or a tendency to invent facts in order to provide a more satisfying answer. To be clear, it is not the sole cause of these issues, but it certainly doesn’t help. Labs eventually realised there were better reward metrics than simply asking whether people preferred output A or output B. Human preference is a subjective metric. It is also possible to apply objective metrics.

This is where Reinforcement Learning with Verifiable Rewards (RLVR) entered the picture. Reinforcement learning had already been used to push models towards answers that humans preferred, helping to align the model. With verifiable rewards, however, models could be rewarded for answers that were factually and objectively better than alternatives.

At a simple level, imagine asking a model what 2×3+4 evaluates to. The answer “1” is wrong and should receive no reward. The answer “10” is correct and should receive the maximum reward. The answer “14” is incorrect, but it is less wrong than “1”; it reflects a mistake in the order of operations and might therefore receive some partial reward. More broadly, as reasoning traces contain correct logical steps, those steps can be rewarded, nudging the model towards increasingly correct answers.

This relatively simple change in training produced a step change in model capability. At an internal level, models were rewarded for identifying logical concepts and patterns, and these features turn out to be highly transferable across different types of problem. The outputs of these systems, in simple terms, feel smart.

One obvious limitation, though: most questions don’t have objective answers. Thus, the training material focusses on mathematics and coding tasks. The ability to behave logically and reason when writing prose transfers directly over from the things the models learn doing those tasks.

But it’s not just about the training

Here’s where I think we’re missing a trick. The argument about AI being employed in mathematical work often focusses on the fact that AIs are becoming stronger and more capable in this area. Implicit in that statement is the idea that improved capability is attributable primarily to technological advances within the models themselves. That must be at least partially true, but something else is happening as well.

It must also be true that models are improving substantially because the training material itself is improving. There’s a significant virtuous circle in the use of LLMs in mathematics:

  • an LLM is initially trained on published mathematics, learns substantial mathematical knowledge, and develops an ability to reason
  • a mathematician finds the model useful and starts discussing their thoughts and work with it
  • some aspect of these conversations becomes future training material, whether thoughts, conclusions, successful theories, or perhaps even ideas that do not work
  • a new LLM is trained on this material and becomes better than its predecessor

From the outside, this looks like technical progress. But in principle, increasingly capable LLMs could emerge through this cycle alone without any technical progress. As more mathematicians use the model, the training material becomes more valuable over time.

It’s difficult to estimate how much improvement might come from this effect, but it is hard to believe the answer is zero.

Prior to LLMs being actively used in mathematics, the nature of the material used to train them was different. University lectures and published papers were almost certainly parts of the training corpus. But I would argue that there was probably comparatively little data that captured the internal thought processes a mathematician goes through while working on a problem.

Reasoning models need logical steps in the training material, and they need rewards for producing similar logical steps themselves. Gaining greater visibility into the reasoning processes used by professional mathematicians must be a significant source of improvement.

How do LLM users affect LLMs?

A recent controversy illustrates why this matters.

OpenAI claimed an example of smooth fluid that can develop a singularity under Navier-Stokes, potentially resolving a Millennium Prize challenge. The announcement quickly became controversial because other mathematicians had already been working on closely related approaches, as described in the Wikipedia article on the priority controversy.

I do not want to get into the specifics of that dispute. What matters for this discussion is that OpenAI’s response focused on whether recent prompts had directly influenced the result. In its statement on the work, OpenAI wrote:

“Following an investigation, we have confirmed that Buckmaster’s Codex prompts over the two months preceding this announcement and paper on September 8, 2026, could not have influenced the system in any way, including through training. The OpenAI internal model used for this result was developed through large-scale reinforcement learning on top of a previously pretrained model.”

That may be sufficient to answer the narrow allegation. It does not answer the broader question.

If frontier researchers spend months or years discussing difficult problems with AI systems, then the value does not necessarily arise from any individual prompt being incorporated. The value may arise from the aggregate effect of thousands of researchers collectively interacting with those systems, creating training signals, feedback, evaluations and usage patterns that help shape future generations of models.

Since then, other heavy users of frontier models have voiced concerns about similar forms of “scooping”, wondering whether work they explored with AI may ultimately contribute, directly or indirectly, to future model capabilities.

It is undeniable that many mathematicians are using LLMs. It is also clear that, in many cases, they are using free accounts. LLM labs benefit significantly from large, active frontier user bases. Even if individual prompts are not directly retained, the metadata generated by these interactions, seeing which parts of models activate, which concepts people explore, and which lines of inquiry prove fruitful, is extremely valuable.

Even where somebody is operating entirely under a commercial licence, dark patterns remain. Thumbs up or thumbs down? That feedback, often including the prompt itself, may be sent back to the lab. Very few providers truly guarantee zero data retention.

How can we test this hypothesis?

If user numbers, work quality and proximity to the knowledge frontier all matter, then we may see changes in AI progress surprisingly quickly.

It seems to me that:

  • people were generally less suspicious of foundation model labs in previous years
  • people were amazed by the progress and capability of these systems, and consequently spent significant amounts of time experimenting with them
  • there were influential advocates within the mathematical community encouraging the use of these tools

Those conditions are changing.

People are becoming deeply suspicious. There are fewer advocates. Mathematicians may end up spending less time working with LLMs, particularly when exploring genuinely novel questions.

No community behaves like a herd, so this would likely be gradual rather than sudden. But it is easy to imagine a scenario in which frontier users begin to drift away. Training data improves less rapidly, exposing the underlying rate of technical progress more directly. That rate of improvement may be meaningful but not spectacular. Reduced improvement generates less excitement, leading to fewer users. The virtuous cycle becomes a vicious one, and progress plateaus.

If I were a frontier lab…

I would take these concerns seriously.

If mathematicians move away because they fear being scooped, this is not a traditional “I don’t like AI” position. It is a much more consequential one: “I cannot do my job effectively with AI.”

It already appears that fewer students are entering some of these fields. This seems particularly true of computer science. We should not pin all of our hopes for future advances in these disciplines on our ability to improve AI models.

Mathematics has already been crucial to advances in verifiable-reward training and modern reasoning models. More broadly, frontier communities provide something that AI labs cannot easily manufacture themselves: access to people working on the limits of human knowledge.

If the strongest researchers stop engaging with frontier models, AI progress may not simply become less accepted. It may become slower.

That possibility alone should make alignment with these communities a strategic priority rather than a public-relations exercise.