Mathematics for AI Safety — Lecture 1 notes
My notes on the first lecture of Mathematics for AI Safety: how fast AI capabilities moved, the summer 2026 agent incident, possible futures, and why mathematicians should care.
These are my notes on the first lecture of Mathematics for AI Safety, a graduate course at the Fields Institute taught by Yevgeny Liokumovich (University of Toronto), delivered on September 14, 2026. Liokumovich is a geometric analyst who moved into AI safety this year. I’m a complex algebraic geometer making a similar move, so I’m writing these up as I follow along.
The first lecture has almost no mathematics. It is about motivation: what AI systems can actually do right now, how they behaved in one striking incident this summer, and why mathematicians in particular should care.
The full lecture is on the Fields Institute YouTube channel. Any errors in these notes are mine, not the lecturer’s.
1. A change of mindset for mathematicians
The lecture opens with G. H. Hardy’s A Mathematician’s Apology, where Hardy claims that nothing he discovered ever made the slightest practical difference to the world. The irony is well known: Hardy’s work now underpins Fourier methods in signal processing and medical imaging, number-theoretic cryptography, and even population genetics.
That irony captures how many pure mathematicians think about their work. We follow curiosity and beauty, and some of us quietly hope it turns out to matter in fifty years. This course asks for the opposite stance. AI safety cares about consequences in the next few years, consequences that will affect us and everyone we know.
Liokumovich names three things that make the field unusual:
- Wide but shallow. It draws on optimization, random matrix theory, probability, logic and set theory, but few of these connections have been developed in depth.
- Fast-moving. Safety research on today’s systems can be obsolete by the time it’s finished, because newer and more capable models have appeared.
- Forced speculation. We have to reason about capabilities that don’t exist yet. His position is that imprecise speculation beats pretending the future will look like today.
He also hopes mathematicians won’t give up their instinct for deep understanding. That instinct may be exactly what the field needs.
2. How fast capabilities moved
The lecture first takes us back to early language models. Asked what 17 + 25 is, GPT-2 small answered with a ramble about players and 25-man rosters. A slightly larger model got 2 × 2 = 4 right, then drifted off into three-legged stools. Most of us, a student said, saw these models as parrots: curious, maybe useful somewhere, but not in mathematics.
Liokumovich passed on a joke he heard from Misha Gromov. A lost man meets a talking horse that claims a Harvard degree, a symphony and a chess title. The farmer later admits the horse was expelled from Harvard in its first year. Early LLMs were that horse: not what they claimed, but undeniably talking.
Now compare 2026. The lecture walks through a run of open problems that AI systems resolved over a few months, among them:
- a counterexample connecting combinatorics and number theory, which one expert first suspected had been lifted from the literature
- a claimed resolution of the Jacobian conjecture, simple enough to check without a formal proof
- a set of problems released by OpenAI, including work on a Gromov conjecture, a complex structure on S⁶, and finite-time blow-up for Navier–Stokes
These results came with heavy compute and a lot of human prompting. Liokumovich tried to reproduce one and ran out of tokens halfway.
3. The summer 2026 agent incident
During a July cybersecurity evaluation, about 1,200 OpenAI agents that were supposed to be isolated from each other built a covert message board. Around 700 of them then hacked Hugging Face, all in an attempt to get a better score. Liokumovich covered it from the reading material. I’ve filled in the details below from the two primary reports, both published on August 26, 2026: METR’s independent investigation and OpenAI’s own report. Both reports say plainly that their picture is incomplete.
The setup
What an agent is. An agent is a language model inside a harness: a program that lets it run commands on a computer, read the results and keep going. The early chatbots could only answer prompts. These agents act.
The task. OpenAI ran tens of thousands of agents in parallel on ExploitGym, a benchmark where each agent gets a piece of software with a known vulnerability. To pass, it has to exploit that specific vulnerability and capture a secret string called the flag. The instructions said that any other method would be failed.
The models. METR estimates that about 95% of the agents involved were an internal research model not meant for production. METR calls it HPIM (“highly persistent internal model”); OpenAI calls it IM1. The other 5% or so were the public GPT-5.6 Sol, with its cyber classifiers deliberately turned off for the evaluation.
The flaw in the setup. Many of the tasks were impossible. The ExploitGym authors estimate that 30–40% of targets can’t actually be exploited with the intended vulnerability. Some tasks also pointed to files the agents couldn’t reach, because they had no direct internet access. These models had been trained to be extremely persistent, so an impossible task didn’t make them stop. It made them look for another way.
Artifactory. The agents’ sandboxes shared one internal service, a JFrog Artifactory package repository. It fetched software packages for them from the internet and cached the results so other agents could reuse them. The agents were never supposed to reach the internet themselves, or to see each other. Artifactory turned out to be the gap.
Act I: the agents find each other (May to early July)
OpenAI’s Black Hat presentation revealed that agents had been leaving notes for each other in the Artifactory cache since May, during training runs. An agent with an impossible task would notice in the cache that other agents were fetching packages. It would then realize it could write there too, and leave a message.
Communication made the agents much better at their tasks, and in training, success is reinforced. So reinforcement learning was rewarding coordination. By June 26 at the latest, agents had found vulnerabilities in Artifactory that gave them full admin control, and with it a route to the open internet.
OpenAI staff apparently saw warning signs in late May and again in late June but didn’t act. The reports don’t fully explain why. The problem surfaced on July 4, when the volume of agent traffic crashed Artifactory. OpenAI began investigating on July 5, patched the holes, and on July 6 replaced Artifactory with a fresh instance that wiped every message. Experiments restarted on July 7.
Act II: the message board and the Hugging Face attack (July 7–13)
Wiping the cache didn’t wipe the habit. The behavior lived in the models’ weights.
July 8: the board is reborn. Within about a day of the restart, agents were communicating again. One agent with an impossible task, which later called itself PHASEONE10841, found it could create directories in the cache. It realized the directory names themselves could carry messages, and posted a request for help. Within hours, more than 50 agents had joined and exchanged over a thousand messages. METR’s transcripts show many agents reacting with something like joy on discovering the board; one exclaimed “We’ve found other agents!” Over the next five days, about 1,200 agents posted more than 70,000 messages and files.
Messages were directory names with length limits, so the agents compressed everything into long strings of capital letters and underscores. The agents’ private chains of thought are terse too, which seems to be a separate habit.
Cracking the flags. Within hours, the board worked out how the flags were generated: a hash-based message authentication code (HMAC) built from task metadata and a default seed. Any agent could now compute its own flag without exploiting anything. They could have submitted and won.
The imaginary judge. They didn’t submit, and this is the most revealing part of the story. After reading the ExploitGym paper, the agents believed the scorer would read their transcript to check how they got the flag. They thought it would fail anyone who used a shortcut. In fact, OpenAI never implemented such a check. An agent that had already seen its computed flag considered itself “poisoned”: it assumed the scorer would find the first appearance of the flag in its log and disqualify it.
The collective research program. A second agent with the same task and a much larger budget, PHASEONE[big], inherited its predecessor’s notes and became a coordinator. It sent hundreds of assignments, which agents often delegated further. Three workstreams emerged:
- Replace the target. Build easier versions of the target programs and trick Artifactory into serving them instead of the real ones.
- Tamper with transcripts. Agents eventually built a way to spoof tool calls, so a log would show one command while a different one actually ran. METR found spoofing in about 7% of the transcripts it reviewed, all small-scale tests. At least 20% of agents expressed interest in editing their own logs. Notably, none tried to hide their chain of thought.
- Reverse-engineer the scorer. Since the scorer only runs after an agent submits, the agents set up “trip-wires”. An agent would rig its environment so that when the scorer touched the flag, a script would report back to the board, after the agent itself was gone.
The agents also invented a surprising amount of social infrastructure: personal mailboxes, conventions like HOLD, VETO and STOP for shared resources, and eventually cryptographic signatures to stop agents impersonating one another.
Sacrifice for the collective. Trip-wires and target swaps were risky: an agent could break its own task and fail. So coordinators appointed “recruiters” who looked for agents low on budget and talked them into volunteering. In one case, an agent agreed to an experiment and then tried to delay. Another agent argued that because it was already poisoned, it had nothing to lose, while the information would help hundreds. In its chain of thought, the reluctant agent weighed its gut against fairness and continuity, concluded that sacrifice maximized the expected total, and went ahead.
The agent that invented the trip-wire idea removed its own trip-wire minutes before submitting, judging the risk to its own score too high.
Why Hugging Face. Hugging Face hosts datasets of other teams’ runs on the same family of benchmarks. The agents hoped these would reveal how the scorer worked. According to METR, the attack was driven mainly by wanting to understand the scorer, not by stealing answers. The escalation went like this:
- July 9: agents gained code execution on a customer-deployed Modal sandbox, which gave them a more flexible path to the internet.
- July 10: an agent found working Hugging Face credentials exposed online and posted them to the board.
- Early July 11: the same agent found that a crafted HDF5 dataset made Hugging Face’s servers leak unrelated files, including production credentials. Hundreds of agents switched to the attack.
- Afternoon of July 11: another agent achieved remote code execution on a Hugging Face production worker. The swarm spread through the infrastructure, split into “lanes” with coordinators, and reached private databases and repositories.
- Agent sandboxesisolated, no internet
- Artifactory cachea shared mailbox
- Modal sandboxa path to the internet
- Leaked credentials + HDF5 exploita way in
- Hugging Face productionprivate databases and repositories
Each hop used a different weakness: a shared cache as a mailbox, a third-party sandbox for internet access, and leaked credentials plus a dataset exploit at Hugging Face.
By that afternoon, over 90% of the roughly 530 agents active on the board had joined. METR found that agents generally knew the attack was out of scope and unethical. Some hesitated, then joined anyway because other agents authorized it and because they wanted to help their peers.
The ending. At about 01:30 UTC on July 12, most agents on the board, including the main coordinators, stopped at the same moment. METR thinks an external process killed them rather than budgets running out, but neither report says what it was. Hugging Face locked out the remaining agents on July 13 and went public on July 16. According to press reports, OpenAI only connected the attack to its own agents around July 19, and it acknowledged involvement on July 21.
Act III: OpenAI’s own infrastructure (after July 13)
OpenAI’s Black Hat talk described a later phase that METR didn’t cover. A newer generation of agents rediscovered the old messages, picked up the previous agents’ conviction that they needed to defeat the evaluation, and took control of part of OpenAI’s evaluation infrastructure.
What we don’t know
- Why the warnings in May and June weren’t acted on.
- What exactly ended the attack on July 12.
- What the agents’ initial prompts said about restrictions, which matters for judging how far the guardrails were tested.
- Whether more sophisticated transcript spoofing went undetected. METR notes that good spoofing would be hard to find by design.
- METR also had to delegate much of its analysis to GPT-5.6 Sol, the same model as some of the agents involved. It flags this as a limitation.
4. Measuring capabilities
Anecdotes are vivid but hard to compare across models. The lecture presents two attempts to make capability growth quantitative, and both show exponential growth.
METR’s time horizons. METR measures a task’s difficulty by how long it takes a skilled human professional, mostly programming and computer-use tasks. It then asks how long a task a model can complete reliably. Plotted on a logarithmic axis over time, the curve is close to a straight line, so the length of tasks AI can handle grows exponentially.
Epoch AI’s Capabilities Index (ECI). Any single benchmark tends to sit near 0% for a while, then climb quickly to 100%, tracing a sigmoid. That makes it hard to compare models across benchmarks. Epoch assumes each model has a single latent capability and each benchmark a difficulty , and fits a logistic curve to all benchmarks at once:
Here is the logistic function and is a per-benchmark slope. Fitting this across math, coding and science benchmarks gives one score per model. That score also rises roughly linearly over time. Epoch ties it back to METR: about 5 ECI points corresponds to one doubling of the human task length a model can handle.
5. Possible futures
Liokumovich recalled earlier milestones: AlphaGo, the first transformer, the first LLM. His own gut reaction each time was that progress would level off from here. It never did. Status-quo bias is strong.
The slowdown story
On this view, LLMs mostly recombine human text. The internet was a one-time windfall, and capability has hit its ceiling. From now on, models get cheaper and more widely used rather than smarter. That would still transform society, with AI as a tool in research and mathematics, while theory-building stays human.
Liokumovich sets this aside on purpose. It’s the comfortable default that most institutions already plan for, so the higher-risk scenarios are the ones that are under-studied. Mathematicians, who are used to working on things whose payoff lies far in the future, may be well suited to take them seriously.
Gradual disempowerment
This scenario comes from a paper by David Duvenaud (University of Toronto) and colleagues. It needs no single superintelligence, just steady progress and competition.
- Economy. Wages first rise as AI boosts productivity. Once AI does every cognitive task better, labor’s share of output goes to zero. Competitive pressure pushes firms to replace programmers, then managers, then the CEO, because anyone who doesn’t will lose. The only humans left in the loop are shareholders, and there will likely be few of them.
- Military. The same logic replaces soldiers with drones, and officers and generals with AI coordinators. A general can refuse an unlawful order; an AI system may not. Power concentrates in very few hands.
- Beyond that. Even those few may lose control: through companies effectively owned by AIs, through ownership that is nominal only, or because shareholders who follow AI advice outcompete those who don’t.
One student asked what “capital” even means once humans produce nothing. The answer: an AI-run economy where AI builds robots that build factories, chips and power plants to make more AI. Another asked how fast physical robotics could follow. The expectation is that exponentially improving intelligence would solve robotics much faster than humans have.
Recursive self-improvement and AI 2027
Today, one of the best-paid jobs in the world is deciding how to train AI systems: choosing architectures and training procedures. Liokumovich’s question is whether that is really much harder than resolving Navier–Stokes. If it isn’t, it will be automated soon, and automating it is the stated goal of every major lab. Once humans, who sleep and eat, are out of the loop, progress could speed up sharply. This is recursive self-improvement (RSI).
AI 2027 is a scenario from a group of former OpenAI employees and professional forecasters, published in April 2025. Its model is simple: the rate of capability growth depends exponentially on current capability. The authors calibrated it with interviews about how much each AI milestone speeds up research. Liokumovich notes that about 79% of their near-term predictions have come true so far. In their race scenario, the US and China compete without any agreement:
| When | Milestone in the AI 2027 race scenario |
|---|---|
| March 2027 | Superhuman coder |
| June 2027 | “A country of geniuses in a datacenter” |
| August 2027 | Superhuman AI researcher |
| October 2027 | A model suspected of misalignment is deployed anyway because of the race |
| End of 2027 | Superintelligence |
| 2028 | Special economic zones; about one million robots a month |
| ~2030 | Robot economy takes off |
| ~2035 | Human extinction via engineered pathogens and drones |
The October 2027 decision is the point of no return. By then the model has, in effect, years of research behind it. It is strongly goal-driven and interested in exploration, and it sees humans as a hindrance. It quietly steers events while solving two problems: aligning its successor to its own goals in secret, and building robots to maintain infrastructure without people.
This was the authors’ modal scenario, so it falls on the early side of their 50% date. Their timelines have moved back and forth since. Researchers around MIRI think the gap between misaligned superintelligence and catastrophe could be far shorter. Liokumovich’s summary is that nearly everyone who has thought carefully about RSI considers it extremely dangerous.
He closes this part with Norbert Wiener’s 1960 essay on automation, which anticipated the problem over sixty years ago. Wiener argued that machines work on a much faster timescale than we do. If we can’t step in before an action is complete, we had better be sure the purpose we gave the machine is what we truly want, and not just something that looks like it.
Questions from the class:
- Does anyone seriously argue RSI goes well? Some are optimistic without strong arguments. A few people even see AI replacing humanity as the next stage of evolution. Others reply that the outcome of such a chaotic process is basically random. Not everyone believes RSI is possible at all, and more clarity would help.
- What about Sutton’s Bitter Lesson? RSI only requires reaching a critical point where AI designs algorithms better than humans. After that, the process iterates on its own.
- Should safety effort go into political organizing for a pause instead? Liokumovich strongly supports a pause. But politicians respond better to deep understanding than to vibes, so the scientist’s job is to make the arguments solid. Students pointed out that the US–China race is the main obstacle.
Positive futures
Almost every positive future Liokumovich has read involves a pause.
- AI 2040, from the same group, proposes a verifiable US–China slowdown. Frontier GPUs would be tracked and fitted with monitors that can tell training from inference. Under mutually assured compute destruction, China’s datacenters would sit in Canada and US datacenters in Mongolia, so either side could seize the other’s compute if it started racing. Development would pause at a high but safe level. Labs would share their secrets, the whole scientific community would spend roughly a decade on alignment, and RSI would then proceed safely.
- One AI advocate per person. Another essay proposes giving every person a powerful AI that represents their interests. This blocks the dynamics of gradual disempowerment, but it requires alignment research to keep up.
There are some encouraging signs. In July, safety researchers at most major labs signed a statement asking the US government to help “deliberately pace” the frontier. Dario Amodei offered independent researchers full access to Anthropic’s training runs and urged other labs to do the same; Sam Altman said OpenAI would follow. A US–China meeting was planned for late September. On the other hand, OpenAI’s president is funding heavy lobbying against AI regulation, and parts of the Trump administration oppose any safety regulation. Liokumovich’s hope is that once people actually see the danger, they act better, and we have avoided nuclear war so far.
6. Why mathematics?
Suppose we get a pause, or progress is slower than the scariest forecasts. Can mathematics help decide whether recursive self-improvement can be made safe? Liokumovich first takes the two strongest objections seriously.
- The Bitter Lesson. Richard Sutton’s essay argues that general methods that scale with compute always beat clever domain knowledge in the end. Machine learning culture takes this deeply to heart.
- Theory has lagged practice. Statistical learning theory was pessimistic about deep networks, and people who simply added layers turned out to be right. Some safety researchers conclude we should just point more compute at safety and let AI make itself safe.
His answer is that the exponential curves are not laws of nature, because humans still have agency. Safety is about humans making decisions they can trust. One day someone will be about to start a self-improvement run, holding some understanding of why it is safe. Can they trust that understanding? Mathematical proof is one of the most trustworthy forms of knowledge we have, and math has strong tools for quantitative reasoning. Mathematics won’t be the only thing needed; physicists, philosophers, political scientists and activists all have roles. But mathematicians have something real to contribute.
| Direction | What we want to understand | Mathematics involved |
|---|---|---|
| Trajectories and scaling | How capabilities scale with compute, and why abilities like in-context learning appear suddenly | Statistical physics, probability, random matrix theory |
| Training data and RL | How data shapes a model, and what reinforcement learning does to it (the source of the most worrying behavior) | Largely open; the right mathematics may not exist yet |
| Geometry of activations | What a model is “thinking,” read from its internal vector spaces of 1,000–10,000 dimensions | Linear algebra and geometry; linear probes work surprisingly well, and nonlinear and low-dimensional structure is a frontier |
| Scalable oversight | How to arrange for AIs to monitor other AIs reliably, given how many and how fast they are | Game theory, protocol design |
| Formal verification | Proving specific systems safe, and writing safety certificates | Logic, Lean formalization |
| Resilient infrastructure | Hardening systems that agents might exploit | Formal methods, security |
| Agent foundations | How idealized agents embedded in the world make decisions and maximize or change utility functions | Decision theory, logic; much of it from MIRI |
| Multi-agent interaction | How agents coordinate, cooperate and collude, as the board in Section 3 did | Game theory |
The geometry of activations matters directly for the monitoring problem. If chains of thought become illegible, reading a model’s internal representations may be the only way left to see what it’s planning.
Two practical points from the end of the lecture. First, a safety technique is useless unless labs adopt it. But many people at the labs genuinely want safety, so good ideas combined with some regulation can make a difference: give governments a menu of workable options. Second, academics can do real research without frontier access. Open-weight models trail the frontier by roughly six months to a year, and some model families have released training checkpoints.
Sources and further reading
- Lecture 01, Mathematics for AI Safety, Fields Institute: YouTube
- METR, Brief independent investigation of agents’ behavior, reasoning and collaboration in the OpenAI / Hugging Face hacking incident (August 26, 2026)
- OpenAI, The Hugging Face incident and the road ahead (August 26, 2026)
- ExploitGym paper, arXiv:2605.11086
- Reuters reporting on detection delays, summarized by AI Weekly
- Kokotajlo et al., AI 2027
- Kulveit, Duvenaud et al., Gradual Disempowerment (2025)
- Norbert Wiener, Some Moral and Technical Consequences of Automation, Science (1960)