By Jim Shimabukuro (assisted by Claude)
Editor
What the newest numbers from Anthropic, OpenAI, and a Princeton research team say about whether artificial intelligence has begun to improve itself
On September 19, the Associated Press asked the leading artificial intelligence companies a question that until recently belonged to science fiction conventions. Are your machines about to start improving themselves? Nobody dodged. Anthropic said its model Claude now leads 26 percent of the company’s own AI research and development work. OpenAI said it had reached its goal of building an automated research intern and is aiming at a full automated AI researcher by March 2028. Elon Musk said that at xAI “humans are gradually getting less and less in the loop,” and that “every successive model is built by the one before it,” with a fully autonomous process arriving by the end of 2026 and no later than 2027 (Huamani & Ortutay, 2026).
Anthony Aguirre read the same numbers and drew a different conclusion. Aguirre is a physics professor at the University of California, Santa Cruz, and the chief executive of the Future of Life Institute. “You can see in these plots from Anthropic over time, more and more of research is being done by the AI and it’s becoming closer and closer to fully autonomous,” he told the AP. “And the result of that success, ultimately is something that is, I think, extremely scary. I think this is probably the worst idea in the history of humanity to do this. And yes, they’re doing it” (Huamani & Ortutay, 2026).
Which raises a question that a lot of people who use these tools every day have been circling. Millions of us now hand work to agents: systems that read a goal, plan a sequence of steps, run code, check the result, notice the error, and try again. Watching an agent catch its own mistake and correct course looks exactly like learning. If a machine can learn from its mistakes, and the mistakes it is correcting are mistakes in building AI, hasn’t recursive self-improvement already quietly arrived? Is agentic AI simply automated RSI under a friendlier name?
The short version of what follows: part of the loop is already running, and the part that is running is narrower and more supervised than the phrase “self-improving AI” suggests. The interesting work is in saying exactly which part.
Irving John Good’s 3 conditions
The concept has a birth date. In 1965 the British statistician Irving John Good, who had worked with Alan Turing at Bletchley Park, published an essay in the journal Advances in Computers. “Let an ultraintelligent machine be defined as a machine that can far surpass all the intellectual activities of any man however clever,” he wrote. Then came the sentence that launched sixty years of argument: because designing machines is itself an intellectual activity, such a machine “could design even better machines; there would then unquestionably be an ‘intelligence explosion,’ and the intelligence of man would be left far behind” (Good, 1965).
Good’s claim has three moving parts, and all three matter. The machine has to improve something that determines its own future capability, not just its output on today’s task. The improvement has to stick without a person curating it. And each round has to make the next round faster, so the curve bends rather than climbs steadily. Good also attached a condition that gets quoted far less often than the rest: the first ultraintelligent machine is the last invention humanity need make, “provided that the machine is docile enough to tell us how to keep it under control.”
Measured against those three parts, “learning from mistakes” is a much weaker property than it sounds. A student who learns from mistakes is not thereby rewriting the curriculum. The question is what layer of the system the learning touches.
Recursive Self-Improvement (RSI)
Lilian Weng, who ran safety systems at OpenAI before leaving to co-found her own company, published a technical essay in July 2026 that is unusually clear on this point. She defines the surrounding software that turns a model into an agent as a harness: “the system surrounding a base model that orchestrates execution and decides how the model thinks and plans, calls tools and acts, perceives and manages context, stores artifacts, and evaluates results.” Recursive self-improvement, in her framing, happens when “an AI uses its current intelligence to improve the cognitive machinery that produces its intelligence” (Weng, 2026).
Cognitive machinery is deliberately broader than model weights. It includes the training pipeline and the deployment system. That breadth is what makes the reader’s question worth taking seriously rather than dismissing. Still, the three places an agent can improve behave very differently.
The first is inside a single task. An agent writes code, the test fails, the agent reads the error and rewrites the code. This is the behavior that feels most like learning, and it is the most fragile. A Google DeepMind team led by Jie Huang tested whether large language models could review and revise their own reasoning without any outside signal, and found that performance typically got worse rather than better; their paper carries the flat title “Large Language Models Cannot Self-Correct Reasoning Yet” (Huang et al., 2024). Models have improved substantially since that study, but the mechanism it identified has not changed. A model has no privileged channel telling it that it was wrong. Correction inside a task runs on external signal: a failing test, a compiler, a type checker, a human saying no. Take away the signal and the correction loses its footing. When the session ends, whatever was worked out disappears with the context window.
The second place is across tasks, through memory. Here something genuinely cumulative happens. In April 2026 Google researchers Jun Yan and Chen-Yu Lee described ReasoningBank, a system in which an agent judges its own completed attempt, extracts the reasoning pattern that worked, and, when the attempt failed, distills the failure into what the authors call preventative lessons. Those items are retrieved before the next task begins. On the WebArena benchmark the approach raised success rates by 8.3 percentage points, on SWE-Bench-Verified by 4.6, and it cut roughly three steps from the average task (Yan & Lee, 2026).
That is learning from mistakes in the ordinary sense, and it persists across sessions. The model’s parameters are byte-for-byte identical before and after. What changed is the file the agent reads before it starts. Delete the file and the agent is exactly what it was. This layer improves an agent’s performance; it leaves the underlying intelligence untouched.
The third place is the weights and the systems that produce them, and this is where Good’s loop actually lives. Researchers at MIT demonstrated one version in 2025 with a method called SEAL, in which a model generates what the authors term self-edits — restructured training material, hyperparameter choices, calls to data-augmentation tools — and then “through supervised finetuning (SFT), these self-edits result in persistent weight updates, enabling lasting adaptation” (Zweiger et al., 2025). The model is not editing itself in real time. It is writing the lesson plan that its next version will be trained on, and a reinforcement learning loop rewards it for lesson plans that work.
Google DeepMind’s AlphaEvolve
The clearest existing example is not a chatbot at all. In May 2025 Google DeepMind described AlphaEvolve, a Gemini-powered coding agent that proposes algorithm changes, evaluates them automatically, and keeps what survives. Its results are specific and auditable. A scheduling heuristic it found for Borg, Google’s data center orchestration system, has been running in production for over a year and “continuously recovers, on average, 0.7% of Google’s worldwide compute resources.” It rewrote a matrix multiplication kernel used in Gemini’s own training, producing “a 23% speedup for this vital kernel” and a one percent reduction in Gemini’s total training time. It proposed a Verilog change removing unnecessary bits from an arithmetic circuit, and that change was adopted into an upcoming Tensor Processing Unit (Google DeepMind, 2025).
Read that sequence again with Good’s three conditions in hand. A Gemini-powered system improved the kernels, the scheduler, and the chip design used to train Gemini. The improvement persisted in production. The freed compute and the faster training make the next round cheaper. All three conditions are satisfied. The loop is closed, it has been closed for over a year, and the world did not end, because the domain is narrow and every proposal is checked by an automatic evaluator that can tell in seconds whether the new code is faster and still correct.
That verifiability is the quiet load-bearing element. A kernel either runs faster and produces identical output or it does not. The reason AlphaEvolve is safe to run at scale is the same reason it cannot be pointed at the hard parts of research: there is no unit test for whether a research direction is worth a month of a team’s time.
Anthropic’s “AI leads 26 percent”
Anthropic’s figure comes from a measurement exercise published by Marina Favaro and Phillie Wright of the Anthropic Institute, built by cataloguing every kind of AI research and engineering work done inside the company, rating how automated each kind currently is, and aggregating the ratings. The scale runs through six levels, from no AI involvement through minimal involvement, AI assists, AI collaborates, AI leads, and finally AI is autonomous. “AI leads” is defined as work where “AI can complete most of the task end-to-end from a high-level prompt, while the human supervises.” “AI is autonomous” means the AI “operates fully autonomously, with no human in the loop” (Favaro & Wright, 2026).
In August 2026, 26 percent of the company’s AI research and development work sat at the “AI leads” level. In February it had been under one percent. More than 90 percent of the work now sits at “AI collaborates” or above. And the top rung is empty: “Claude is not operating fully autonomously for any measured subset of AI R&D work” (Favaro & Wright, 2026).
The scale of the operation is worth stating plainly. Anthropic reported roughly 30,000 agents doing research and engineering work at the company at any one time. Over August, the company analyzed more than a billion decisions those agents made. An automated monitor blocked 0.002 percent of them, about one in 47,000, with roughly 50 high-priority flags a week escalated to a person and human review of blocked actions inside a week. Around 6 percent of the compute spent on AI research went to safety work, and about 12 percent of the compute spent on AI-driven AI research went to safety work (Favaro & Wright, 2026).
Anthropic also published the limits of its own instrument, which is the part most coverage skipped. The ratings come from a frozen basket of roughly 15,000 tasks catalogued in July 2026, and the company notes the index “does not, on its own, tell us whether new kinds of work are appearing that humans have shifted onto.” The ratings depend on a judge model, which can disagree with human raters on borderline cases, particularly on exactly where “AI collaborates” ends and “AI leads” begins. Human and model ratings landed within one level of each other 97 percent of the time (Favaro & Wright, 2026).
Anthropic has published related internal numbers through other channels. Claude wrote roughly 80 percent of the code produced at the company, with code output up eightfold. In testing last November, Claude chose the better of two research directions 51 percent of the time compared with human researchers; by April that figure had reached 64 percent. OpenAI reported that the number of experiments each of its researchers runs had doubled by July 2026 (Booth, 2026).
Where AI agents fall short
In late July 2026 a team led by Peter Kirgis and Sayash Kapoor at Princeton, with roughly twenty co-authors including Arvind Narayanan, Helen Toner, and Rishi Bommasani, ran what they called a shadow evaluation. They took two unpublished papers accepted to NeurIPS 2026, extracted the central research question from each, and handed those questions to a frontier agent. The agent had six days and thousands of dollars of compute per attempt.
The agents “completed all of the engineering without human help, yet could not make substantial progress towards answering the research questions.” Both papers the agents produced were shown to the authors of the original work and rejected by them. The team catalogued five recurring failure modes: poor judgment about what meets a publishable standard, inflexibility when a research design turns out to be flawed, weak recovery from approaches that are not working, limited awareness of resource constraints, and drift away from the assigned task (Kirgis et al., 2026).
The third of those failure modes is the one that speaks to the reader’s question. Weak recovery from approaches that are not working is a failure to learn from mistakes, measured over six days on exactly the kind of work that recursive self-improvement would have to automate. Kapoor put it bluntly to MIT Technology Review: “The agents were unambiguously bad at carrying out the research itself.” He also identified why this is hard to train away: “It’s harder to create environments to train these models when the task itself is open-ended” (Kim, 2026).
Jack Clark, an Anthropic co-founder who has spent much of 2026 arguing publicly that recursive self-improvement should be treated as a live engineering question rather than a rhetorical one, has described the same gap from inside a frontier lab: “There’s a certain absence of valuable, intuitive creativity in today’s AI systems” (Kim, 2026).
Anthropic’s Claude Mythos
The most-cited external measure is METR’s time horizon: the length of task, measured by how long a human expert takes, that a model can complete with a given success rate. METR defines the 50 percent time horizon as “the length of task in our suite (measured by how long it takes a human expert) such that we’d predict with 50% confidence that the model could complete the task.” The trend has been steeply exponential, and in May 2026, when METR added a preview of Anthropic’s Claude Mythos, it attached a caution to the chart: “Measurements above 16 hrs are unreliable with our current task suite” (METR, 2026). The benchmark, in other words, is running out of runway before the models are.
Gary Marcus, the most persistent critic of straight-line extrapolation from these charts, has pressed on what the 50 percent standard actually licenses. “A graph that demands only 50% success does not address reliable performance. At all,” he wrote, adding that the measure “pertains only to software-development tasks. Not general intelligence” and that “babies don’t keep doubling forever, and nor will AI progress” (Marcus, 2026). His objection and Kapoor’s converge: the domains where agents look strongest are the domains where success can be checked cheaply.
Companies define danger by speed
Here is the finding that answers the reader’s question most directly, and it comes from the labs’ own safety documents rather than from their press statements. None of the thresholds that would trigger emergency safeguards are written in terms of weights, memory, scaffolding, or agency. They are written in terms of rate.
Anthropic’s Responsible Scaling Policy, version 3.0, dated February 24, 2026, sets a threshold for “AI systems that can fully automate, or otherwise dramatically accelerate, the work of large, top-tier teams of human researchers in domains where fast progress could cause threats to international security and/or rapid disruptions to the global balance of power — for example, energy, robotics, weapons development and AI itself.” The working operationalization is a system that could “compress two years of 2018–2024 AI progress into a single year” (Anthropic, 2026).
Google DeepMind’s Frontier Safety Framework, version 3.1, dated April 17, 2026, uses two machine learning research thresholds. The acceleration level is reached when a model “has been used to accelerate AI development, resulting in AI progress substantially accelerating from historical rates.” The automation level is reached when a model “can fully automate the work of any team of researchers at Google focused on improving AI capabilities, with approximately comparable all-inclusive costs” (Google DeepMind, 2026).
Both are stopwatches. Neither asks how the speedup was achieved. Under these definitions, an agent that learns from its mistakes counts as recursive self-improvement precisely to the degree that it moves the calendar, and not at all otherwise. Better memory files that shave three steps off a benchmark task do not move the calendar. A kernel rewrite that cuts one percent from Gemini’s training time moves it slightly. Twenty-six percent of a frontier lab’s research being led rather than merely assisted by its own model is the number that will eventually settle the argument, once there are enough readings to see whether the underlying pace of capability gains actually changed.
OpenAI published its own position on September 6, 2026, and it is candid. “We do not yet know how to safely get all the way to aligned, full RSI,” the company wrote. “People still set our research priorities, judge which ideas and results to pursue, and decide whether to scale, pause, or deploy systems.” It added that “we cannot assume that progress in alignment and safety will keep pace, and more capable systems can become harder to monitor,” and that “whether and how to proceed must depend on our ability to preserve human control and on informed democratic choices about the benefits and risks.” The company went further than its competitors on one point: “We believe that we and other companies should be required to publicly track our progress toward RSI.” The same document records a concrete pause, with the company halting reinforcement learning training on deployment-bound models following an infrastructure compromise (OpenAI, 2026).
Four brakes that slow the loop
John Thickstun, an assistant professor of computer science at Cornell, offered the AP the most deflationary observation in the story, and it is correct. “We have already, for years, been using these models in supportive roles for creating the next version of these models. So people use the past generation of models to write code for the AI systems that then create the next generation” (Huamani & Ortutay, 2026). The loop is old. What changed is its share of the work and the speed at which each turn completes.
Aguirre’s concern is specifically about that speed. “The really important thing here is that as AI is doing more of it, it gets faster, because AI operates just much, much more quickly than the humans do” (Huamani & Ortutay, 2026). Thirty thousand agents working simultaneously is a research organization that never sleeps and never needs to be convinced.
Four things are slowing the loop today. Compute is finite and expensive, and training runs cannot be hurried by thinking harder. Experiments take wall-clock time that no amount of intelligence compresses. Verification is cheap for code and math and expensive or impossible for research judgment, which is why AlphaEvolve works and shadow evaluations fail. And taste — knowing which question is worth six days — is the capacity the Princeton study found missing and the one Clark described as absent.
Weng’s essay names the structural safeguard that keeps the third brake from being released by accident. A system that evolves its own harness must not also grade itself: “the evaluator and permission control should likely sit outside the loop that evolves harness, with held-out tests, trace audits, and human review at decision points that matter” (Weng, 2026). She also notes a practical floor. Weaker models given self-improvement loops made themselves worse. Self-improvement requires a model already good enough to tell improvement from noise.
Agentic AI is not automated RSI
Agentic AI is the delivery mechanism for recursive self-improvement, and in one narrow domain the delivery has already happened. An agent that corrects itself inside a task is running on external feedback and forgets everything afterward. An agent that writes lessons to a memory file gets steadily better at its job while the intelligence underneath it stays fixed. An agent that rewrites the kernels used to train its successor is doing what Good described, on a small and heavily verified patch of ground.
The question worth arguing about is coverage. How much of the work that produces a better model can the loop absorb? In August 2026 the published answer from the company with the most aggressive measurement program was 26 percent led, zero percent autonomous, and a set of open-ended research tasks that agents could not finish at all. Those numbers will move. They have moved enormously in seven months.
For a reader who wants to track this without relying on anyone’s forecast, five readings are worth following. Whether the “autonomous” column in Anthropic’s index ever leaves zero. Whether OpenAI follows through on public RSI tracking and what it reports. Whether METR can extend its task suite past the 16-hour reliability ceiling and whether the exponential holds when it does. Whether the share of compute going to safety rises or falls from 6 and 12 percent as automation increases. And whether anyone replicates the Princeton shadow evaluation with an agent that produces a paper the original authors would accept.
Good’s 1965 condition still hangs over all of it. He thought the first ultraintelligent machine would be humanity’s last necessary invention, provided it was docile enough to explain how to keep it under control. In October 2025 the Future of Life Institute published a statement, signed by hundreds of scientists, policymakers, religious leaders, and public figures: “We call for a prohibition on the development of superintelligence, not lifted before there is 1) broad scientific consensus that it will be done safely and controllably, and 2) strong public buy-in.” Aguirre’s framing of why: “Nobody developing these AI systems has been asking humanity if this is OK. We did – and they think it’s unacceptable” (Future of Life Institute, 2025).
Twenty-six percent is the first figure of its kind that a frontier laboratory has committed to publishing on a schedule. Seven months earlier it was under one percent. The next reading is the one to watch.
References
Anthropic. (2026, February 24). Anthropic’s responsible scaling policy (Version 3.0). https://www.anthropic.com/responsible-scaling-policy/rsp-v3-0
Booth, H. (2026, August 7). What happens when AI starts building AI? Inside recursive self-improvement. TIME. https://time.com/article/2026/08/07/ai-recursive-self-improvement-anthropic-openai/
Favaro, M., & Wright, P. (2026, September). Measurements for understanding the pace of AI development inside frontier labs. The Anthropic Institute. https://www.anthropic.com/institute/measuring-pace-of-ai-development
Future of Life Institute. (2025, October). Prominent scientists, faith leaders, policymakers and artists call for a prohibition on superintelligence, as poll shows Americans don’t want it. https://futureoflife.org/press-release/prominent-scientists-faith-leaders-policymakers-and-artists-call-for-a-prohibition-on-superintelligence/
Good, I. J. (1965). Speculations concerning the first ultraintelligent machine. Advances in Computers, 6, 31–88. https://www.historyofinformation.com/detail.php?id=2142
Google DeepMind. (2025, May 14). AlphaEvolve: A Gemini-powered coding agent for designing advanced algorithms. https://deepmind.google/blog/alphaevolve-a-gemini-powered-coding-agent-for-designing-advanced-algorithms/
Google DeepMind. (2026, April 17). Frontier safety framework (Version 3.1). https://storage.googleapis.com/deepmind-media/DeepMind.com/Blog/strengthening-our-frontier-safety-framework/frontier-safety-framework_3-1.pdf
Huamani, K., & Ortutay, B. (2026, September 19). Will AI models achieve the ability to improve autonomously? Leading labs say the scenario is near. The Associated Press. https://abcnews.com/US/wireStory/ai-models-achieve-ability-improve-autonomously-leading-labs-136582737
Huang, J., Chen, X., Mishra, S., Zheng, H. S., Yu, A. W., Song, X., & Zhou, D. (2024). Large language models cannot self-correct reasoning yet. International Conference on Learning Representations. https://arxiv.org/abs/2310.01798
Kim, M. (2026, August 18). AI’s recursive self-improvement might not come so quickly after all. MIT Technology Review. https://www.technologyreview.com/2026/08/18/1142188/ai-recursive-self-improvement/
Kirgis, P., Kapoor, S., Schwartz, A., Rabanser, S., Africa, D., Voudouris, K., Nguyen, V., Pilditch, T., Dubois, M., Coppock, H., Ududec, C., Nadgir, N., Orona, M., Bayer, T., Chan-Sew, D., Ling, Y., Shetty, A., Toner, H., Hadfield, G., Lazar, S., Newman, S., Tekofsky, S., Bommasani, R., & Narayanan, A. (2026, July 29). Can AI agents conduct open-ended AI research? Early evidence from two case studies. arXiv. https://arxiv.org/abs/2607.27191
Marcus, G. (2026). Misplaced panic over AI progress. Marcus on AI. https://garymarcus.substack.com/p/misplaced-panic-over-ai-progress
METR. (2026). Task-completion time horizons of frontier AI models. https://metr.org/time-horizons/
OpenAI. (2026, September 6). Research acceleration: The view inside OpenAI. https://openai.com/index/research-acceleration-view-inside-openai/
Weng, L. (2026, July 4). Harness engineering for self-improvement. Lil’Log. https://lilianweng.github.io/posts/2026-07-04-harness/
Yan, J., & Lee, C.-Y. (2026, April 21). ReasoningBank: Enabling agents to learn from experience. Google Research. https://research.google/blog/reasoningbank-enabling-agents-to-learn-from-experience/
Zweiger, A., Pari, J., Guo, H., Akyürek, E., Kim, Y., & Agrawal, P. (2025, June 12). Self-adapting language models. arXiv. https://arxiv.org/abs/2506.10943
###
Filed under: Uncategorized |
Leave a Reply