By Jim Shimabukuro (assisted by Claude)
Editor
An AI agent that learns on the job, an AI team that picks its own research, and AGI in the report’s tenth year. One month after the State of AI Report, here is what the evidence says about each.
On October 8, Nathan Benaich, founder of the London venture firm Air Street Capital, closed the 2026 State of AI Report with nine predictions for the next twelve months (Benaich, 2026a). Most of them name an event that either happens or does not: a card network writing dispute rules for purchases made by AI agents, or a U.S. state requiring businesses to accept cancellations filed by a consumer’s agent. Three concern the pace of AI itself. The second prediction says an agent will halve its failure rate on new tasks after a month of customer work, without a model upgrade. The sixth says an autonomous AI team will beat human-led model research on equal time and compute while setting its own agenda. The ninth is two words long: “AGI 2027.”
Benaich grades his forecasts in public. Of the ten he made in 2025, his team scored two as hits, five as partial outcomes, and three as misses. A forecast about an AI-driven scientific discovery, for example, received partial credit because “the significance and extent of end-to-end autonomy were not broadly agreed” (Benaich, 2026a). The scorecard shows a forecaster who sets thresholds that are hard to clear in twelve months, and whose calls most often end up as partial outcomes.
The three predictions examined here also depend on one another. The first tests whether an AI system can improve from experience after it is deployed, which Benaich calls “a practical measure of continual learning.” The second tests whether AI can decide which research is worth doing, which he describes as a test of whether models can show “better research taste than the best researchers” (Benaich, 2026b). Those are two of the gaps that Google DeepMind’s chief executive, Demis Hassabis, has pointed to when explaining why AGI is not here yet. At Davos in January, he listed continual learning among the missing capabilities and said “maybe we need one or two more breakthroughs before we’ll get to AGI” (Kahn, 2026). A year earlier he had said current systems cannot yet “invent their own hypotheses or conjectures about science” (Kantrowitz, 2025). If agents cannot learn reliably on the job and cannot direct research, a general intelligence by the end of 2027 is hard to assemble.
This review weighs the case for and against each prediction as it stands in early November 2026. The first has the strongest support. Laboratory studies already show models with fixed weights improving from feedback, though so far by less than half and mostly on tasks they have seen before. The second is unlikely to be met under the conditions Benaich set, because the 2026 studies that put AI agents on open-ended research find them capable at engineering and weak at choosing and judging questions. The third is the weakest as written, since the report never says what AGI means. It becomes more defensible under the narrower meaning Benaich gave it in an interview: an AI that outperforms him at much of his own knowledge work. For readers who want to follow these trajectories, the order is useful. The first two predictions can be measured in 2027, and their results will say more about the third than any argument over the definition of a word.
An agent that learns from its customers
A deployed agent consists of a language model and the software around it, often called the harness, which supplies instructions, connects the model to tools, and stores notes. The model’s weights, the parameters fixed during training, can stay the same while the harness changes what the agent knows and how it behaves. Benaich’s second prediction asks whether the agent itself can make those changes, by drawing lessons from a month of real work, and cut its failures in half on tasks it has not seen.
The case that the harness alone can move results is already established. Zhang et al. (2026) ran the same GLM-5.1 model on a 100-task subset of SWE-bench Verified, a widely used set of software repair problems, under different harness configurations. Average success rose from 52.5% with a minimal harness to 65.5% with a full one, which cuts the failure rate from 47.5% to 34.5%, a reduction of about 27%. Benaich cites this result in the report itself (Benaich, 2026b). Engineers made those changes, however. The prediction requires the agent to earn its gains through its own experience.
The closest test of that comes from Tablan et al. (2026), who placed agents in a simulated bank’s customer service, where the agent handles multi-turn conversations with a simulated customer and can search about 700 policy documents. After each conversation, the system turned feedback into natural-language rules and stored them for later use. When the feedback included a verified correction, Claude Sonnet 5’s single-attempt success rose from 24.8% to 39.7%. That lowers the failure rate from about 75% to about 60%, a cut of roughly one-fifth. Halving it would have required success above 62%. The 97 tasks were also repeated across trials, so the agent was improving on the same problems it had already been corrected on.
Singh et al. (2026) tested the harder condition, improvement on interactions held out from the learning process. Agents running frozen models could edit their own notes, skills, and code between rounds of chess, Go, Hex, and NetHack. The authors report that some agents “achieve substantial and persistent gains, while others remain near or below their initial performance,” and that the gains “often transfer only partially” when conditions change. The mechanism Benaich is counting on therefore exists, and it produces uneven results that depend heavily on which model is doing the learning.
Customer work adds two complications that the laboratory studies avoid. The first is that “failure” is hard to measure in a live service. In a randomized field experiment on Alibaba’s Taobao platform, deploying an AI agent shortened customer chats but “substantially lowers ratings for AI-eligible chats.” Human intervention preserved service quality when the system escalated a technical problem and was less effective when a customer had already expressed frustration (Y. Wang et al., 2026). A company could lower an agent’s apparent failure rate by closing chats faster or by routing difficult cases to staff, and neither would mean the agent had learned anything.
The second complication is that memory can store mistakes along with lessons. R. Wang et al. (2026) found that “errors in stored abstractions can propagate to subsequent decisions through retrieval and reuse,” and that an agent drawing on a flawed rule may infer a new rule that repeats the error. Their remedy was to discard the agent’s synthesized guidance after each task while keeping the raw records. That is the opposite of what Benaich’s prediction needs, which is a growing store of lessons that keeps getting more useful over a month.
Taken together, the evidence makes a qualifying result in 2027 plausible, most likely in a narrow workflow with a steady stream of similar, checkable cases, such as billing disputes, IT tickets, or code repair. A convincing demonstration would run two copies of the same model version, one allowed to keep its lessons and one without them, and test both on fresh cases of comparable difficulty. It would also disclose how many corrections came from staff. The likeliest source of a claim is a vendor reporting on its own product, and the report is, as one reviewer noted, “written by a venture investor who backs some of the companies it covers” (Dominguez, 2026). An independent evaluation would carry more weight than a company announcement.
If agents improve between model releases, part of the value of an AI system shifts from the model a firm buys to the record of corrections the firm builds up. A college advising office that teaches its agent a transfer-credit exception owns that improvement. It also owns the risk that the agent applies the exception to students it was never meant to cover. Organizations would need to know what their agents have learned, where each rule came from, and how to remove one that is wrong.
An AI team that sets its own research agenda
Learning from customers still relies on someone else’s judgment: a supervisor marks a refund as wrong, and the agent records the correction. The sixth prediction removes that supervisor. Benaich specifies that the AI team “would need to choose its own research agenda as well as execute experiments,” and that “completing experiments chosen by humans would not satisfy all of those conditions” (Benaich, 2026a). The team must then beat human-led research on equal time and compute.
The case for this prediction rests mostly on reports from inside the frontier labs. Anthropic says Claude “leads” 26% of the company’s measured AI research and development work as of August 2026, up from under 1% in February. In Anthropic’s scheme, leading means completing most of a task from a high-level prompt while a person supervises, and the company states that Claude “is not operating fully autonomously for any measured subset of AI R&D work” (Favaro & Wright, 2026). In a second study, Anthropic selected 129 moments in past research sessions where a researcher’s next step had room for improvement. A separate Claude model, able to see how each session turned out, judged whether the human’s step or an AI-proposed step was better. Opus 4.5 beat the human choice 51% of the time; Mythos Preview, in April 2026, beat it 64% of the time. When the human’s move was already strong, the models’ suggestions were judged better only about 20% of the time (Favaro & Clark, 2026). In his October 29 interview with Matt Turck, Benaich described the 64% figure as human graders preferring the model’s research direction (Turck, 2026). Anthropic’s own account says the judge was a Claude model with the benefit of hindsight, which makes the result a narrower one.
An outside assessment points in the same direction. On November 5, the evaluation group METR reported that benchmark scores of Anthropic’s models have accelerated 1.9 times since October 2025, with a 95% confidence interval of 1.5 to 2.8 times. METR guessed that “perhaps ≈80%” of the current acceleration is due to AI, with substantial uncertainty. It also reported being told that research is accelerated “primarily via AI assistance rather than fully autonomous contributions,” and that Anthropic “redacted a majority of the evidence we found most informative” (Model Evaluation and Threat Research [METR], 2026). Outside the labs, the AI Scientist system described in Nature proposed ideas, ran machine-learning experiments, and wrote complete papers. One of three submissions scored above the acceptance threshold for an ICLR workshop with a 70% acceptance rate, and the authors’ own review found that none met the bar for the main conference (Lu et al., 2026).
That evidence shows AI doing a growing share of research under human direction. The studies that hand the direction to the AI are less encouraging. In July, a team of 24 researchers including Sayash Kapoor, Arvind Narayanan, Helen Toner, and Gillian Hadfield gave frontier agents the central research question of two unpublished NeurIPS 2026 submissions, along with six days, $3,000 in API credits, and GPU time. The original authors graded the results. “The agents completed all of the engineering without human help, yet could not make substantial progress towards answering the research questions,” and both papers “were unambiguously rejected.” The failures included “poor judgment about the bar for publishable research,” “ineffective backtracking from dead ends,” and poor resource awareness; both runs ended with less than half of the API budget spent. Before the runs, the coauthors had put the median chance of at least a weak accept at 30% (Kirgis et al., 2026). The authors caution that the study rests on five runs and that the reviewers knew the papers were written by AI.
A November benchmark measured the same weakness at larger scale. IdeaTaste compared more than 90,000 research proposals from six language models with what research communities went on to publish. The agents reached “about 26% of the human level on citation matching and only 18% on idea matching,” and richer context or larger groups of agents did not “substantially close this gap” (Zhong et al., 2026). Even on a well-defined optimization task, speeding up the training of the small NanoGPT model, METR researchers concluded that “autonomous optimization does not have dramatic effects on AI R&D progress,” with agents cost-competitive with human researchers only at budgets below about $3,000 (Cunningham et al., 2026).
Benaich’s own remarks support the skeptical reading. Describing an analysis of how OpenAI’s Codex spends its tokens, he told Turck that “very, very few tokens were used for what question should I explore and why should I explore it.” He called research taste “one of the hallmarks of intelligence” and acknowledged that it is unclear “how do you objectively measure taste?” (Turck, 2026). His written report concludes that “sustained, fully autonomous recursive improvement remains to be demonstrated” (Benaich, 2026b).
The prediction also leaves its contest undefined. It does not say who would judge, what counts as compute for an agent team that spends heavily on planning and coordination, or whether the human-led team may use AI tools, which every leading lab’s researchers now do. A narrow contest with a fixed objective, such as improving a model’s accuracy under a set training budget, could produce an AI win in 2027. It would not meet Benaich’s requirement that the AI choose the agenda. Under his stated conditions, the 2026 evidence points to a miss or, at most, a partial outcome.
If the prediction does come true, the consequences extend well beyond one contest. A lab whose AI teams can choose productive research questions is limited mainly by compute, and each generation of models could help design the next. The AI Futures Project, authors of the AI 2027 scenario, has adjusted its forecasting model “to take into account that training is needed to apply software improvements, which reduces the chance of very fast takeoffs” (Lifland et al., 2026b). Even a slower loop would shorten the time outside evaluators have to examine one model before its successor arrives. METR’s report on Anthropic, which was limited by redactions, shows that independent oversight of AI-driven research is already constrained.
AGI in the report’s tenth year
The first two predictions come with tests that can be described in advance. The third does not. The report’s only explanation is that “AI progress is moving so quickly that we expect AGI in 2027, the tenth anniversary of the State of AI Report” (Benaich, 2026a). Reviewers noticed the gap. The Chinese AI publication Machine Heart observed that the item appears in the report “without giving any description” (translated from Chinese) (Machine Heart, 2026). The newsletter writer Ruben Dominguez wrote that “‘AGI 2027’ is the one I doubt most, because the slide leaves it undefined and gives no way to check it” (Dominguez, 2026).
Benaich supplied a working definition in his interview with Turck. Asked why he expects AGI next year, he said that in areas such as legal, financial, and investment work, “the model could potentially be better than what I could do in any compressed amount of time,” and that “that would be approximately AGI.” He also described having AI conduct about a dozen customer interviews for his own research, saying that “automated customer research has been very powerful for me” (Turck, 2026). By that standard, AGI arrives when an AI can outperform a skilled investor at much of his own knowledge work within a limited time.
The case for a near date draws on several 2026 results. In September, the ARC Prize Foundation tested OpenAI’s GPT-6 Astra on ARC-AGI-3, a suite of unfamiliar game-like environments in which a system must work out the rules and goals through trial. With the foundation’s standard harness at maximum reasoning effort, Astra scored 62.7%. With OpenAI’s adapter, which preserves the model’s reasoning state between requests, it scored 98.6% and used fewer actions than the median human tester on 96% of levels; a run at high effort reached 99.9% for about $19,000 (Kamradt, 2026). Anthropic’s chief executive, Dario Amodei, told the Davos audience in January that AI would reach Nobel-level scientific research in multiple fields within about two years (Kahn, 2026), a view consistent with the company’s 2025 statement to the White House that “we expect powerful AI systems will emerge in late 2026 or early 2027” (as quoted in Greenblatt, 2025). METR’s finding that AI is speeding up Anthropic’s own research adds a mechanism by which progress could compound.
The ARC Prize Foundation, however, drew a limit around its own result: “we are not claiming that it is AGI.” It noted that ARC-AGI-3 “has a tightly bounded scope and format,” with deterministic mechanics and closed-ended goals (Kamradt, 2026). Professional work is neither. In October, the DAYJOB benchmark gave 30 model configurations 130 long assignments in healthcare and finance, each graded against an expert rubric. The best configuration, built on Claude Opus 5.5, passed 24.7% of healthcare attempts and 23.9% of finance attempts when every criterion had to be met; the median configuration passed 0.6% and 2.5%. Relaxing the bar to 90% of criteria raised the leader’s pass rates to 62.4% and 58.3% (Finley et al., 2026). One task shows why the strict standard is the relevant one: an equity price quoted in South African cents had been entered as rand, so a position worth about $260,000 appeared to be worth $26 million. An analyst who misses that error has failed the assignment, however well the rest of the report is written. These are the kinds of financial tasks Benaich has in mind, and the best systems currently complete about one in four without error.
The forecasters most associated with 2027 as a date have also moved away from it. The AI Futures Project, whose AI 2027 scenario made the year famous, wrote in January that “it has never been the case that we were confident AGI would arrive in 2027” (Lifland et al., 2026a). In its August update, Daniel Kokotajlo, whose medians are the earliest in the group, put his median for an automated coder at November 2027 and for “top-expert-dominating AI,” the group’s AGI-like milestone, at November 2028. His colleague Eli Lifland placed those milestones in January 2030 and July 2032 (Lifland et al., 2026b). Hassabis told Davos that today’s systems are “nowhere near” human-level AGI and gave roughly even odds of reaching it within the decade (Kahn, 2026). Before 2026 began, the AI safety researcher Ryan Greenblatt estimated the chance that Anthropic’s early-2027 forecast would hold at “around 6%” (Greenblatt, 2025).
The definition problem also shapes how any 2027 claim would be judged. Google DeepMind researchers Ryan Burnell and Oran Kelly (2026) have proposed measuring AI systems on ten cognitive abilities, including learning, reasoning, executive functions such as planning, and social cognition, and placing each system’s performance against the distribution of human performance on each one. The Santa Fe Institute’s Melanie Mitchell has argued for evaluations that test novel variations of familiar problems and that separate performance from competence. She compares “performance without competence” to a student who has memorized solutions, and notes that current systems can produce a coherent story and then “fail in weird ways” when questioned about it (Strogatz & Levin, 2026). Under frameworks like these, a system that excels at ARC-AGI-3 and fails three of four DAYJOB assignments would score high on some abilities and low on others, and would not count as general.
On the evidence available in November 2026, AGI in the broad sense of expert-level performance across most cognitive work is unlikely to arrive in 2027, and even the forecasters most sympathetic to short timelines now place it later. Under Benaich’s personal definition, a claim in 2027 is plausible for parts of his own work, such as expert interviews, research synthesis, and first drafts of investment memos. That gap between definitions is the prediction’s main weakness. It can be counted as met by narrowing the meaning of AGI, and Benaich’s own scorecard suggests that a prediction this loosely specified will end up graded as a partial outcome.
The effects that matter to most readers will arrive without any declaration. Using payroll data through June 2026, Brynjolfsson et al. (2026) found that employment of 22- to 25-year-olds in the occupations most exposed to AI now stands “19% below where it would be had it kept pace with that of their less-exposed peers,” driven mainly by reduced hiring of young workers. The authors call these “early, descriptive indicators” and do not estimate a causal effect. The pattern still poses a question for universities and employers that does not depend on the AGI debate: if software takes over the first-draft work through which junior analysts, paralegals, and programmers learn their trades, those workers will need another way to build the judgment they will later be asked to supply.
What 2027 will show
The three predictions offer a sequence of checkpoints. The first to watch is an independent evaluation of an agent that improves over a month of real work, measured on fresh cases against a copy without memory. The second is a blinded contest, or an expanded version of the shadow evaluations run this year, in which an AI team chooses its own research question and its results are judged by people who do not know who produced them. The third is movement on demanding professional benchmarks such as DAYJOB, where the best systems now complete about a quarter of assignments without error.
If the first two checkpoints are reached in 2027, the two gaps Hassabis identified in January will have narrowed, and the case for a near-term AGI will rest on demonstrated abilities. If they are not reached, Benaich’s ninth prediction will depend on how he chooses to define its two words. Either way, the prediction about agents learning on the job is the one most likely to affect ordinary organizations next year. A customer-service desk, a hospital billing office, or a university registrar could soon run an agent that behaves differently in December than it did in January, and each will need ways to see what it has learned.
References
Benaich, N. (2026a, October 8). Our AI predictions for 2027. State of AI Report 2026. https://www.stateof.ai/2026-explore/predictions
Benaich, N. (2026b, October 8). The State of AI Report 2026. State of AI. https://nathanbenaich.substack.com/p/state-of-ai-2026
Brynjolfsson, E., Chandar, B., & Chen, R. (2026, August). Canaries in the coal mine? Six facts about the recent employment effects of artificial intelligence [Working paper]. Stanford Digital Economy Lab. https://digitaleconomy.stanford.edu/app/uploads/2026/08/Canaries_August2026.pdf
Burnell, R., & Kelly, O. (2026, March 17). Measuring progress toward AGI: A cognitive framework. Google. https://blog.google/innovation-and-ai/models-and-research/google-deepmind/measuring-agi-cognitive-framework/
Cunningham, T., Shetty, M., Cheng, V., & Rush, N. (2026, July 21). Expenditure horizon: Measuring optimization ability, with an application to NanoGPT. METR. https://metr.org/blog/2026-07-21-expenditure-horizon/
Dominguez, R. (2026, October 29). State of AI 2026: AI is 13x cheaper. Where’s the payoff? The AI Corner. https://www.the-ai-corner.com/p/state-of-ai-report-2026-findings
Favaro, M., & Clark, J. (2026). When AI builds itself. Anthropic. https://www.anthropic.com/institute/recursive-self-improvement
Favaro, M., & Wright, P. (2026). Measurements for understanding the pace of AI development inside frontier labs. Anthropic. https://www.anthropic.com/institute/measuring-pace-of-ai-development
Finley, S., Panavas, L., Mikkelson, T., Hinton, C., Ganss, S., Monton, B., Kendall, E., Spradlin, M., Bye, L., O’Brien, M., Ylvisaker, L., Ray, D., Garre, S., Mehta, S., & Chen, E. (2026). DAYJOB: A benchmark for long-horizon professional work [Preprint]. arXiv. https://arxiv.org/abs/2610.01306
Greenblatt, R. (2025, November 3). What’s up with Anthropic predicting AGI by early 2027? LessWrong. https://www.lesswrong.com/posts/gabPgK9e83QrmcvbK/what-s-up-with-anthropic-predicting-agi-by-early-2027-1
Kahn, J. (2026, January 23). AI luminaries at Davos clash over how close human-level intelligence really is. Fortune. https://fortune.com/2026/01/23/deepmind-demis-hassabis-anthropic-dario-amodei-yann-lecun-ai-davos/
Kamradt, G. (2026, September 3). OpenAI’s GPT-6 Astra on ARC-AGI-3. ARC Prize. https://arcprize.org/blog/astra
Kantrowitz, A. (2025, January 23). Google DeepMind CEO Demis Hassabis: The path to AGI, LLM creativity, and Google smart glasses. Big Technology. https://www.bigtechnology.com/p/google-deepmind-ceo-demis-hassabis
Kirgis, P., Kapoor, S., Schwartz, A., Rabanser, S., Africa, D., Voudouris, K., Nguyen, V., Pilditch, T., Dubois, M., Coppock, H., Ududec, C., Nadgir, N., Orona, M., Bayer, T., Chan-Sew, D., Ling, Y., Shetty, A., Toner, H., Hadfield, G., . . . Narayanan, A. (2026). Can AI agents conduct open-ended AI research? Early evidence from two case studies [Preprint]. arXiv. https://arxiv.org/abs/2607.27191
Lifland, E., Kokotajlo, D., & Halstead, B. (2026a, January 27). Clarifying how our AI timelines forecasts have changed since AI 2027. LessWrong. https://www.lesswrong.com/posts/qPco9BX5kmKCDzzW9/clarifying-how-our-ai-timelines-forecasts-have-changed-since
Lifland, E., Kokotajlo, D., & Halstead, B. (2026b, August 16). Q2.5 2026 timelines update: Uplift and revenue. AI Futures Project. https://blog.aifutures.org/p/q25-2026-timelines-update-uplift
Lu, C., Lu, C., Lange, R. T., Yamada, Y., Hu, S., Foerster, J., Ha, D., & Clune, J. (2026). Towards end-to-end automation of AI research. Nature, 651(8107), 914–919. https://pmc.ncbi.nlm.nih.gov/articles/PMC13017497/
Machine Heart. (2026, November 7). State of AI 2026: AGI 2027 zhen de hui lai ma? [State of AI 2026: Will AGI 2027 really arrive?]. Sina Tech. https://finance.sina.com.cn/tech/roll/2026-11-07/doc-inixwtha4775388.shtml
Model Evaluation and Threat Research. (2026, November 5). Independent assessment of AI R&D acceleration at Anthropic in October 2026. https://metr.org/blog/2026-11-05-anthropic-ai-rd-acceleration-assessment/
Singh, H., Bakhtin, A., Shao, R., Synnaeve, G., Kulikov, I., Fergus, R., Arora, S., Keutzer, K., Weston, J., Mahajan, A., & Goyal, A. (2026). Agent plasticity: Measuring self-improvement through experience [Preprint]. arXiv. https://arxiv.org/abs/2610.08902
Strogatz, S., & Levin, J. (Hosts). (2026, August 20). Are we thinking correctly about AI intelligence? [Audio podcast episode]. In The Joy of Why. Quanta Magazine. https://www.quantamagazine.org/are-we-thinking-correctly-about-ai-intelligence-20260820/
Tablan, V., Taylor, S., & Bernhem, K. (2026). Learning on the job: Continual learning from deployment feedback for frozen-weights agents [Preprint]. arXiv. https://arxiv.org/abs/2607.22157
Turck, M. (Host). (2026, October 29). State of AI 2026: The biggest stories of the year (and AGI in 2027?) | Nathan Benaich [Audio podcast episode]. In The MAD Podcast with Matt Turck. https://www.mattturck.com/podcast/state-of-ai-2026-the-biggest-stories-of-the-year-and-agi-in-2027-nathan-benaich
Wang, R., Panaganti, K., & Zhou, D. (2026). More memory, worse agents: Error reproduction and anti-persistence in LLM agents [Preprint]. arXiv. https://arxiv.org/abs/2610.27406
Wang, Y., Zhu, C., Feng, T., Lu, L. X., & Jia, B. (2026). Agentic AI and human-in-the-loop interventions: Field experimental evidence from Alibaba’s customer service operations [Preprint]. arXiv. https://arxiv.org/abs/2605.14830
Zhang, Y., Wang, J., Ge, Y., Xu, W., Hamm, J., & Reddy, C. K. (2026). Stop comparing LLM agents without disclosing the harness [Preprint]. arXiv. https://arxiv.org/abs/2605.23950
Zhong, S., Xu, C., Ning, J., Liu, Z., Kumar, A., & Xiong, C. (2026). IdeaTaste: Do LLMs share the research taste of human researchers? [Preprint]. arXiv. https://arxiv.org/abs/2611.03424
#######################################
#######################################
By Jim Shimabukuro (assisted by ChatGPT)
Editor
Suppose a customer-service AI agent applies a refund policy to the wrong kind of order. A human supervisor corrects it. A month later, the agent encounters a different order with the same complication and applies the right rule on its own. If lessons like that halve the error rate across new cases, the company can improve the service between model releases. Nathan Benaich expects an agent to reach that mark within a year without upgrading its underlying model (Benaich, 2026).
That is one of nine predictions in the October 8, 2026, State of AI Report. A second prediction calls for an autonomous AI team to set its own agenda and beat human-led model research with equal time and computing resources. A third puts the arrival of artificial general intelligence, or AGI, in 2027. Benaich, the founder of the venture firm Air Street Capital, has a practice of revisiting his forecasts. His team credited two of the previous edition’s ten calls as hits, five as partial outcomes, and three as misses. A forecast about an important scientific discovery received partial credit because the significance and extent of the AI’s autonomy were disputed (Benaich, 2026).
The first of these three calls is the strongest. Experiments already show how an agent can improve with its model held fixed (Singh et al., 2026). A field trial could isolate the contribution of a month’s experience. An AI team could also win a carefully scoped research contest in 2027. Giving the team responsibility for deciding what to investigate makes that a substantially harder prediction. Benaich’s AGI date is most plausible in the sense he gave it in a follow-up interview, where he described an AI able to outperform him on a range of his own knowledge work (Turck, 2026). The case for that date weakens as the definition of general intelligence broadens. The distinction matters because employers, universities, and the public would encounter the consequences of better agents and faster model development well before they agreed on a definition of AGI.
What an agent can learn from a month of work
A deployed agent includes a language model and the software that supplies its instructions, connects it to tools, and checks what it does. This surrounding software is often called the harness. It can give the model a place to retain useful corrections, a way to retrieve them when a similar issue comes up, or a test that catches an error before an action goes through. The model’s weights, the parameters set during training, can stay fixed while the system behaves differently from one day to the next (Tablan et al., 2026).
Zhang et al. (2026) tested the same GLM-5.1 model on 100 software repair tasks drawn from a benchmark called SWE-bench Verified. With better instructions, tools, and feedback around it, the agent’s average success rose from 52.5% to 65.5% across two runs per setup. That is a 13-percentage-point gain in success, or a reduction of about 27% in failures. It came from changing the setup of a coding agent under controlled conditions. The next question is how much of that improvement an agent can earn through its own work after deployment.
Tablan et al. (2026) moved closer to that question in a simulated banking service. Their agents could already search the bank’s policy documents. In one condition, the agent also received a verdict after each conversation. In another, it received a verified correction after each failure. The system could turn that feedback into a rule and consult it in later conversations. For Claude Sonnet 5, verified corrections raised the single-trial success rate from 24.8% with static document retrieval to 39.7%. The study repeated its 97 simulated tasks with memory enabled across trials. That design shows how past mistakes can change future behavior and leaves open how much transfers to cases the agent has never handled.
An October study by Singh et al. (2026) tested transfer more directly. Agents using frozen models could update notes, code, and other materials between attempts, then were tested on interactions kept out of the learning process. In games including chess, Go, and NetHack, some agents made substantial gains on fresh instances of familiar tasks. Others improved little or fell below their starting performance, and gains often carried over only partly when the environment changed. For a customer-service agent, that difference could be the difference between handling an unfamiliar return under a known policy and navigating a new policy with different exceptions. Benaich’s forecast is easier to imagine when a company sees a steady stream of new cases governed by rules it can check.
In a randomized field experiment on Alibaba’s Taobao platform, workers supervised an agent handling eligible customer chats. Average chats got shorter, while customer ratings on AI-eligible chats fell. Human intervention preserved service quality more effectively when the agent escalated a technical problem than when a customer had already expressed frustration (Y. Wang et al., 2026). An agent could close a ticket more quickly yet leave the customer with the original problem. A company could also improve its apparent success rate by sending difficult cases to staff. A study of Benaich’s prediction would distinguish a correct resolution by the agent from a successful handoff and track both.
Consider an agent that mishandles 30 of every 100 comparable cases before it starts learning. To qualify, it would have to mishandle 15 or fewer per 100 on a fresh set after its month of work. One credible test would run the same model version with the same access and time budget in two copies, one allowed to retain lessons and one without that record. Both would face enough cases to distinguish a gain from chance variation, using samples of comparable difficulty kept separate from the month’s work. The evaluator would disclose which changes the agent made from feedback and which came from staff, along with how often humans had to intervene. The controls matter because changing the tools, the mix of cases, or the number of retries could produce a gain of its own (Singh et al., 2026). The forecast leaves room for human corrections. The persuasive result would trace the gains to what the agent learned from those corrections and from other outcomes.
A college’s advising agent could apply an adviser’s correction about a transfer-credit exception to another student’s case. The same memory could also turn a one-time exception into a standing rule or preserve information that belonged only in the original record. In an October preprint, R. Wang et al. (2026) showed how an erroneous rule in an agent’s memory can shape later behavior and be written back as if it were a lesson confirmed by experience. For a college or business, the value of the improvement would depend in part on whether staff could trace a learned rule to its source and correct it across the system. A halving of failures in one well-measured workflow would be a substantial result on its own.
Who chooses the research question
Given a request to speed up a training run, agents can try many changes and keep those that produce a measurable gain. An autonomous research team would also have to decide whether training speed was the problem worth tackling. It might find that the same model learns more from better data, or that an apparent improvement disappears when it is tested on a different dataset. Choosing what to investigate, and deciding when a line of work has stopped paying off, are part of the job Benaich wants to see agents take on (Benaich, 2026).
In a study published in Nature, Lu et al. (2026) gave their AI Scientist system room to propose ideas, run machine-learning experiments, and write papers. One of three manuscripts submitted for blind review cleared the threshold for a workshop at the International Conference on Learning Representations. The workshop’s acceptance rate was 70%, and the authors withdrew the submissions under a protocol agreed to in advance. The authors’ own review placed all three papers below the standard of the main conference. The study shows that an AI system can take a computational project far enough for some outside reviewers to find the result worthwhile. An equal-resource contest with a human-led team would also test the quality of the system’s research choices.
In his October 29 interview with Matt Turck, Benaich recalled human graders preferring a frontier model’s research direction 64% of the time (Turck, 2026). Anthropic’s account of that figure describes a different design. Its team selected 129 moments in past research sessions where a person had taken an unproductive turn. A separate Claude model, which could see how each session eventually ended, rated the original next step against one proposed by an AI. It preferred Mythos Preview’s proposal in 64% of these cases, up from 51% for Opus 4.5 (Anthropic, 2026a). This is evidence that newer models can suggest useful next steps at difficult points. It measures decisions at selected detours, with hindsight available to the judge. A prospective trial would give each team the same starting information and see where its decisions led.
Anthropic reported that Claude led 26% of the model R&D work in its internal index in August, up from under 1% in February. In its rating system, leading meant doing most of a defined task from a high-level prompt with a person supervising. All the work covered by the index still involved people at that point (Anthropic, 2026b).
A November 5 assessment by the nonprofit Model Evaluation and Threat Research (METR, 2026) offers an outside view. Based on access to Anthropic’s internal data and staff, METR estimated that the lab’s scores on its capability index were rising at 1.9 times their rate before October 2025, with a 95% confidence interval of 1.5 to 2.8 times. The evaluators judged that AI probably accounted for most of the acceleration as of October, with substantial uncertainty over how much. Interviews indicated that most of AI’s contribution came from assisting staff. Anthropic also exercised its right to redact much of the evidence METR found most informative. The estimate concerns one lab and progress on well-specified tasks. It gives reason to take accelerated development seriously while leaving the independent judgment of an AI research team to be tested.
Even a direct comparison would depend on what the two sides could spend. In a July analysis of a contest to speed up the training of NanoGPT, a small language model, Cunningham et al. (2026) found preliminary evidence that agents could be cost-competitive at low budgets. They distinguished the expense of running an agent from the expense of running experiments and from researchers’ wages. Benaich calls for equal time and compute, so the head-to-head would need to count the computation agents use to plan and coordinate as well as the hardware devoted to training experiments. The teams would also need comparable access to data and prior results. Allowing the human-led team to use AI tools under the same rules would make the test more demanding and better reflect how the leading labs actually work (Anthropic, 2026b).
Several agents could investigate competing explanations in parallel, with another checking their methods. Their conclusions can still rest on the same flawed assumption. In their October review, Feng et al. (2026) identify coordination among agents and correct tool use as conditions for reliable scientific work. An independent test of the finished result matters more than agreement among the agents.
A convincing contest could give both teams a broad objective, such as improving a model’s performance on unfamiliar data with a fixed training budget. Each would choose its own approach. Independent reviewers could evaluate results on reserved test data, examine the work logs for decisions made by people, and try to reproduce the result. This would address a problem that Feng et al. (2026) identify in their review of agentic science. A polished account of a result is easier to assess when reviewers can follow the sequence of tool calls, decisions, and experiments that produced it. The autonomous team could work within human-set safety boundaries, with a record of any intervention that redirected the research.
Such a win would show that an AI team could allocate part of a research budget and make a better set of scientific decisions under stated conditions. A stronger trajectory, in which the improvements help build a successor that in turn does better research, would take evidence over several rounds. A lab still has to train a model to incorporate some of the software improvements an agent discovers (Lifland et al., 2026). If those cycles speed up, labs could try more ideas and lower the cost of some experiments. Outside evaluators could also have less time to examine a model before its successor is in use, even while human scientists remain in charge.
What would count as AGI in 2027
The report puts AGI in 2027 on the strength of the pace of progress and leaves the threshold open (Benaich, 2026). In his October 29 conversation with Turck, Benaich supplied a more concrete one. He could imagine a system doing better than he could, in the time available to him, at work involving investing, finance, legal material, and unfamiliar subjects. He also described a due-diligence project in which agents helped find and interview experts and synthesize their answers. That example comes from Benaich’s own work. It explains why he sees a near-term prospect for delegation across activities that used to require his direct involvement (Turck, 2026).
In September, the ARC Prize Foundation tested OpenAI’s GPT-6 Astra on its semi-private ARC-AGI-3 suite. The model had to infer the rules and objectives of unfamiliar game-like environments through interaction. At maximum reasoning effort it scored 62.7% with the foundation’s standard setup and 98.6% with an adapter that preserved its intermediate reasoning between requests. With that adapter at maximum effort, Astra used fewer moves than the median successful human player on 96% of levels. A separate run with the adapter scored 99.9% and cost about $19,000 across the suite. The result shows that a system can infer the rules of unfamiliar problems and act on them efficiently. The gap between the two setups shows how much of that ability a test can miss if the system has a less effective way to carry forward what it has learned (Kamradt, 2026).
Consider what changes when the task starts with an underspecified request from a colleague. An October preprint tested 30 model configurations on 130 professional assignments in healthcare and finance. In DAYJOB, agents often had to inspect several files and challenge a premise before they could deliver usable work. The best configuration, using Claude Opus 5.5, satisfied every criterion in about a quarter of attempts in each domain. With a threshold of 90% of criteria met, the same configuration passed 62.4% of healthcare attempts and 58.3% of finance attempts. The study used an AI judge applying expert-written rubrics, and the score depends strongly on how strict a pass is. That is meaningful because a single missed item can change the decision. In one assignment, an equity price quoted in South African cents had been entered as rand, overstating the value of a holding by a factor of 100. Spotting the error and deciding whether to hold a client report were both part of the work (Finley et al., 2026).
Benaich’s personal threshold and a claim of broad human-level competence would call for different tests. The former could be evaluated with fresh assignments from his business, judged by people who could check the work and who knew how much time and help each side had. The broader claim would require many kinds of work and a stated human comparison. Google DeepMind researchers Ryan Burnell and Oran Kelly (2026) propose measuring a profile of abilities, including learning, planning, and social cognition, against the distribution of human performance. Melanie Mitchell (2026), writing in AI Magazine, argues that an evaluation should also check whether a system holds up when a problem is rephrased, or when familiar cues are changed in ways that leave the underlying problem intact. Those proposals help explain why the evidence from ARC and DAYJOB can point in different directions. They ask different things of the systems.
Faster AI research is the strongest reason to take a near date seriously. A lab that can run substantially more useful experiments might reach capabilities that today’s models lack. Turning that possibility into a year, however, takes assumptions about where progress will come from and how quickly it will spread across kinds of work. A separate group, the AI Futures Project, illustrates the sensitivity. In an August forecasting update, members of the team behind AI 2027 considered several ways of estimating when AI could automate coding at a leading lab. Daniel Kokotajlo noted that if development proceeded at 75% of the pace envisaged in the original scenario, his extrapolation put that milestone in mid-2027. At 60% of the original pace, it moved to early 2028. The milestone in that calculation is coding automation. A forecast for reliable judgment across professions would require its own assumptions and evidence (Lifland et al., 2026).
The effects on work can arrive in pieces. For a small firm, an agent capable of preparing a reliable first pass on a complex document could make professional services cheaper to deliver. In occupations where junior staff learn by preparing that first pass, the same change could alter how they acquire judgment. In an August working paper using U.S. payroll data through June 2026, Brynjolfsson et al. (2026) found that employment among 22- to 25-year-olds in occupations more exposed to AI was 19% below where it would have been had it kept pace with employment among their less-exposed peers. Reduced hiring accounted for most of the gap. The authors describe the result as an early pattern in employment and note differences that predated generative AI. For educators and employers, it raises a concrete question about where beginners will get practice if more of the assignments that once trained them go to software.
A system that learns one organization’s procedures and then enters a different kind of workplace, learns its rules, and handles exceptions as reliably as experienced staff would make a stronger case for generality than any single professional demonstration. By then, employers and educators could already face decisions about which work to delegate, when a person should step in, and how people who are new to a profession will learn to make the decisions they are expected to oversee.
References
Anthropic. (2026a, June 4). When AI builds itself. https://www.anthropic.com/institute/recursive-self-improvement
Anthropic. (2026b, September 17). Measurements for understanding the pace of AI development inside frontier labs. https://www.anthropic.com/institute/measuring-pace-of-ai-development
Benaich, N. (2026, October 8). Our AI predictions for 2027. State of AI Report 2026. https://www.stateof.ai/2026-explore/predictions
Brynjolfsson, E., Chandar, B., & Chen, R. (2026, August 12). Canaries in the coal mine? Six facts about the recent employment effects of artificial intelligence [Working paper]. Stanford Digital Economy Lab. https://digitaleconomy.stanford.edu/app/uploads/2026/08/Canaries_August2026.pdf
Burnell, R., & Kelly, O. (2026, March 17). Measuring progress toward AGI: A cognitive framework. Google. https://blog.google/innovation-and-ai/models-and-research/google-deepmind/measuring-agi-cognitive-framework/
Cunningham, T., Shetty, M., Cheng, V., & Rush, N. (2026, July 21). Expenditure horizon: Measuring optimization ability, with an application to NanoGPT. METR. https://metr.org/blog/2026-07-21-expenditure-horizon/
Feng, B., Walther, P., Zhang, L., Coyle, R. A., Kim, S., Mandal, A., Zhang, Z., Papidocha, S. M., Kelmanson, A., Segura, D. M., Thakur, S., Pal, S. C., Bernales, V., Gao, W., Yang, X., Cong, L., Schwaller, P., Aspuru-Guzik, A., & Zheng, Z. (2026). Agentic AI systems for scientific discovery. Nature Reviews Methods Primers, 6(1), Article 81. https://bpb-us-e2.wpmucdn.com/sites.wustl.edu/dist/0/4841/files/2026/10/agentic-ai.pdf
Finley, S., Panavas, L., Mikkelson, T., Hinton, C., Ganss, S., Monton, B., Kendall, E., Spradlin, M., Bye, L., O’Brien, M., Ylvisaker, L., Ray, D., Garre, S., Mehta, S., & Chen, E. (2026). DAYJOB: A benchmark for long-horizon professional work [Preprint]. arXiv. https://arxiv.org/abs/2610.01306
Kamradt, G. (2026, September 3). OpenAI’s GPT-6 Astra on ARC-AGI-3. ARC Prize. https://arcprize.org/blog/astra
Lifland, E., Kokotajlo, D., & Halstead, B. (2026, August 16). Q2.5 2026 timelines update: Uplift and revenue. AI Futures Project. https://blog.aifutures.org/p/q25-2026-timelines-update-uplift
Lu, C., Lu, C., Lange, R. T., Yamada, Y., Hu, S., Foerster, J., Ha, D., & Clune, J. (2026). Towards end-to-end automation of AI research. Nature, 651(8107), 914–919. https://pmc.ncbi.nlm.nih.gov/articles/PMC13017497/
Mitchell, M. (2026). Six principles for evaluating cognitive capabilities in AI models. AI Magazine, 47(2), Article e70061. https://doi.org/10.1002/aaai.70061
Model Evaluation and Threat Research. (2026, November 5). Independent assessment of AI R&D acceleration at Anthropic in October 2026. https://metr.org/blog/2026-11-05-anthropic-ai-rd-acceleration-assessment/
Singh, H., Bakhtin, A., Shao, R., Synnaeve, G., Kulikov, I., Fergus, R., Arora, S., Keutzer, K., Weston, J., Mahajan, A., & Goyal, A. (2026). Agent plasticity: Measuring self-improvement through experience [Preprint]. arXiv. https://arxiv.org/abs/2610.08902
Tablan, V., Taylor, S., & Bernhem, K. (2026). Learning on the job: Continual learning from deployment feedback for frozen-weights agents [Preprint]. arXiv. https://arxiv.org/abs/2607.22157
Turck, M. (Host). (2026, October 29). State of AI 2026: The biggest stories of the year (and AGI in 2027?) | Nathan Benaich [Audio podcast episode]. In The MAD Podcast with Matt Turck. https://www.mattturck.com/podcast/state-of-ai-2026-the-biggest-stories-of-the-year-and-agi-in-2027-nathan-benaich
Wang, R., Panaganti, K., & Zhou, D. (2026). More memory, worse agents: Error reproduction and anti-persistence in LLM agents [Preprint]. arXiv. https://arxiv.org/abs/2610.27406
Wang, Y., Zhu, C., Feng, T., Lu, L. X., & Jia, B. (2026). Agentic AI and human-in-the-loop interventions: Field experimental evidence from Alibaba’s customer service operations [Preprint]. arXiv. https://arxiv.org/abs/2605.14830
Zhang, Y., Wang, J., Ge, Y., Xu, W., Hamm, J., & Reddy, C. K. (2026). Stop comparing LLM agents without disclosing the harness [Preprint]. arXiv. https://arxiv.org/html/2605.23950v1
###
Filed under: Uncategorized |
Leave a Reply