By Jim Shimabukuro (assisted by Claude)
Editor
This update to ETC Journal’s November 2025 review of Kestin et al. revisits what the study established, what it left open, and what a year of larger field trials has added.
On September 28, 2026, a post on X opened with: “Researchers proved AI has deleted every reason universities exist” (Superman [@thesupermannx], 2026). The research behind it was not new. It was the Harvard physics experiment by Gregory Kestin, Kelly Miller, and colleagues, published in Scientific Reports in June 2025 and reviewed in ETC Journal on November 10, 2025 (Kestin et al., 2025; Shimabukuro, 2025). Within a day, technology and education sites were publishing correctives. Inside AI wrote that “the viral narrative oversimplifies what the experiment actually measured” (Andreou, 2026).
Two days after the post, a research team at the University of Maryland dated a working paper on a randomized trial with 2,379 undergraduates. In comparable sections of the same courses, access to an AI tutor built into the course website “reduced final grades by 0.37 standard deviations” (Liu et al., 2026). The Maryland authors cited the Harvard study by name and described their own results as “contrasting with gains from deliberately structured AI tutoring” (Liu et al., 2026).
In a single week, then, the same 2025 study was offered as proof that universities are finished and used as the yardstick for a new negative result. That prompted a second look at ETCJ’s November 2025 review. This update re-examines the Harvard paper, notes where the November review was accurate and where it overstated or missed something, and brings readers up to date on the larger trials that appeared in 2026, most of them published in August, September, and October.
The experiment ran in fall 2023 in Physical Sciences 2, which the paper describes as Harvard’s largest physics class, with 233 students enrolled. Of those, 194 were eligible for the study (Kestin et al., 2025). Over two consecutive weeks the course covered surface tension and then fluid flow. In the first week, half the students learned the material from a custom AI tutor called PS2 Pal while the other half attended an active-learning class taught by one of the course’s instructors. In the second week the groups switched. Each student took a short pre-test before each lesson and a post-test after it.
The tutor ran on a large language model of the GPT-4 class and was tightly scripted. Its instructions told it to keep replies brief, to move one step at a time, and to get students to attempt a problem before it helped. It was given the instructors’ own worked solutions so it would not invent wrong answers, and Kestin fed it the same feedback notes he used in class (Barshay, 2024). This design work is the paper’s main contribution, and ETCJ’s November review described it accurately.
The results were large. The median post-test score was 4.5 after the AI lesson and 3.5 after the classroom lesson, against a pooled pre-test median of 2.75, which means the AI group’s median gain was more than double the classroom group’s (Kestin et al., 2025). If the two methods had been equally effective, the chance of seeing a gap that large was smaller than one in 100 million. The median student spent 49 minutes with the tutor. Students reported more engagement after the AI lesson (4.1 versus 3.6 on a five-point scale) and more motivation (3.4 versus 3.1). The authors summed it up in one line: “We find that students learn significantly more in less time when using the AI tutor” (Kestin et al., 2025).
The ETCJ review called the study “a watershed moment in educational technology research” and “the most favorable evidence currently available for AI tutoring’s potential” (Shimabukuro, 2025). The second phrase still holds. The first overstated what a two-week experiment in one course can show, and a closer reading of the paper turns up several points the review did not raise.
The first point concerns the sample. The paper lists its eligibility rules as consent, “participation in both in-class and AI-tutored instruction, and completion of all pre-tests and post-tests” (Kestin et al., 2025). In a crossover design where all 194 eligible students did both lessons and all tests, each condition should have 194 post-test scores. The paper instead reports post-test medians for 142 scores in the AI condition and 174 in the classroom condition, and it does not explain the difference (Kestin et al., 2025). The shortfall is larger on the AI side: 52 missing scores against 20. If students who struggled with the tutor were less likely to have a usable AI post-test, the AI group’s median would be pushed up. There may be a simple explanation, such as a data-cleaning step the paper does not describe, but none could be found in the published text. The ETCJ review repeated the figure of 194 without noticing the gap. The authors could settle the question quickly, and anyone citing the study’s effect size should keep it in mind until they do.
The second point concerns the setting. The paper describes students using the tutor “at home, on their own” (Kestin et al., 2025). The Hechinger Report’s 2024 account of the preprint said something different: “In this experiment, the ‘at-home’ sessions with PS2 Pal were scheduled and proctored over Zoom” (Barshay, 2024). On either account, the AI lesson was a required course activity at a set time, and students knew a test was coming. A student opening an optional chatbot late at night before a problem set is due is in a different situation.
The third point concerns time. The widely repeated comparison of 49 minutes against 60 sets a measured number beside an assumed one. The class met for 75 minutes, and the paper subtracts the 15 minutes spent on tests to arrive at 60 minutes of instruction (Kestin et al., 2025). On the AI side, 70 percent of students finished in under an hour and 30 percent took longer. The tutor was faster for most students, but the time saving is an estimate.
The fourth point concerns what was measured. Post-tests came immediately after each lesson. The paper reports no delayed test, so it cannot say how much of the extra learning lasted a week or a semester; retention appears only as a subject for future work (Kestin et al., 2025). The tests also had a ceiling. The paper’s standard regression gives an effect size of 0.63 standard deviations, which the authors call “an underestimation due to ceiling effect,” and a quantile regression gives a range of 0.73 to 1.3 (Kestin et al., 2025). The ETCJ review reported only the higher range. Both figures are large by the standards of education research, and readers should see both.
The fifth point concerns scope. The study covered two introductory topics. In 2024, “Kestin said he’s seeing promising signs that it’s helpful for basic but not advanced problems” (Barshay, 2024). The paper itself is careful about its claim: “we do not presume that structured AI tutoring will always outperform in-class active learning in all contexts,” and “an AI tutor should not replace in-person teaching” (Kestin et al., 2025). Those sentences deserve as much attention as the title.
These points leave the study’s strengths intact. Students were randomly assigned, each served as their own comparison, and the AI was tested against active learning led by skilled instructors, a far harder benchmark than a lecture. On the measures used, the AI won. What the closer reading changes is the size of the claim. The study shows that a carefully engineered tutor, used as a scheduled assignment by well-prepared students, produced more immediate learning on two introductory physics topics than a good class did.
At the time of the November 2025 ETCJ review, much of the comparison evidence came from older computer tutoring systems. A 2025 systematic review of school studies found their effects “generally positive but are found to be mitigated when compared to non-intelligent tutoring systems” (Létourneau et al., 2025). The most cited warning about generative AI came from a high school math experiment in Turkey. Students given plain GPT-4 during practice raised their practice grades by 48 percent, then scored 17 percent lower than students who never had access once the tool was taken away. A version with teacher-designed hints and safeguards raised practice grades by 127 percent and largely avoided that penalty (Bastani et al., 2025).
Two studies released in late 2025 pointed toward combinations of human and machine. In a trial with 165 students in five UK secondary schools, Google DeepMind and the math platform Eedi had an AI model draft messages that expert tutors reviewed before sending. Tutors approved 76.4 percent of the drafts with little or no editing, and students tutored this way solved new problems on later topics 66.2 percent of the time, against 60.7 percent for students tutored by humans alone (LearnLM Team & Eedi, 2025). A Stanford trial of Tutor CoPilot, a tool that suggests responses to human tutors, found that “students in the Tutor CoPilot condition were four percentage points more likely to achieve topic mastery,” with the largest gains among students whose tutors were less experienced (Moon, 2026).
The 2026 trials have been larger, closer to ordinary school conditions, and more sobering. Most appeared between August and October.
In August, Philip Oreopoulos of the University of Toronto and colleagues released a randomized experiment with more than 6,000 middle school math students in Hamilton County, Tennessee (Oreopoulos et al., 2026). Students practiced fractions for one 50-minute class period, either with conventional computer-based exercises or with an AI tutor called NUMI that withheld answers and walked them through their mistakes. Half of each group also had to answer three questions correctly in a row after an error before moving on. Students using the AI “progressed more slowly and attempted fewer questions” but were more accurate when they did answer (Oreopoulos et al., 2026). On a test a week later, students who had both the AI and the three-in-a-row rule scored about three percentage points higher than the conventional group. The Hechinger Report noted that the gain was concentrated on material like what students had practiced: “In this study, AI didn’t produce a deeper or more transferable understanding of fractions” (Barshay & Fasiang, 2026). Oreopoulos was careful in describing the result. “We’re not saying this initial experiment shows AI tutoring will be a game changer that transforms learning,” he told his university’s news office, adding, “In most cases, a good human tutor is better” (Witten, 2026).
Oreopoulos and Nina Low also released a two-year trial in 18 Tennessee middle schools, randomized by school, in which students used Khan Academy and its AI tutor, Khanmigo, during daily remedial math periods (Oreopoulos & Low, 2026). Being assigned to the program raised math achievement by “1.3 national percentile ranks per term,” about 0.06 to 0.08 standard deviations over a school year, and the authors wrote that “these gains resemble those from Khan Academy practice without AI assistance” (Oreopoulos & Low, 2026). Nearly all students tried Khanmigo. The median student, however, messaged it on about a third of practice days and in only 17 percent of the sessions where they had made a mistake. The researchers’ summary, quoted by the Hechinger Report in September, was plain: “We observe how students actually used the AI tutor, and the answer is: not much” (Barshay, 2026). Their paper identifies the problem: “The binding constraint appears to be engagement” (Oreopoulos & Low, 2026). Sal Khan told the Hechinger Report that the findings matched what Khan Academy had seen internally, and the company has since changed the tool so that it offers help automatically after a wrong answer (Barshay, 2026).
Then came Maryland. In fall 2025, the university randomly assigned 30 instructors, within groups of similar courses, either to give their students a “GPT-4o–based virtual study assistant” built into the campus learning management system or to teach as usual (Liu et al., 2026). The assistant had a tutor mode that guided students toward answers and a direct mode that explained more fully. In same-course comparisons, access lowered final grades by 0.37 standard deviations and reduced participation in course-site activities by 0.90 standard deviations, and “estimated academic losses were greater among first-generation students” (Liu et al., 2026). Yet “only 14.63% of treated students ever used the VSA,” and 73.8 percent of the 3,460 requests students sent were for direct answers (Liu et al., 2026).
That low uptake has drawn pushback. Writing on October 15, education researcher Brendan Bartanen argued that “the most important finding is one not highlighted by the authors: very few people used this tool when it was offered” (Bartanen, 2026). He noted that the grade measure relied on course-site scores that instructors maintain inconsistently (“there is no mechanism that enforces Canvas scores being correct”), that the drop in participation could reflect control instructors simply assigning more online activities, and that the first-generation finding was one of dozens of subgroup comparisons. His verdict: “As implemented, this was not a study of the effects of AI tutoring” (Bartanen, 2026). Neither the Maryland paper nor the two Toronto papers has yet been peer reviewed.
Read together, these studies point to the one condition the Harvard experiment held fixed. In Physical Sciences 2, using the tutor was the assignment. Students sat down at a set time, worked through a defined lesson, and took a test. In Tennessee and Maryland, AI help was available, and students had to choose it, usually at the moment they were confused or behind. Most did not, and those who did often asked for the answer. The Harvard and Maryland results therefore measure different things. Harvard measured what a well-built tutor can teach when students use it as designed. Maryland measured what happens when a general assistant is made available and students decide whether to use it. The NUMI experiment falls between them: when the tutor’s help was built into required practice, students slowed down, worked through their mistakes, and did slightly better.
Schools and companies have not waited for these results. Alpha School, a private chain where “students spend two hours per day on core subjects with an AI tutor,” charges up to $75,000 a year, enrolls more than 1,200 students, and told Fortune it plans to expand to 50 campuses this year (Osmonbekov, 2026). The Fortune article cited no independent evaluation of Alpha’s results. In March, Google announced that “Gemini for Education is now integrated in all public universities in Malaysia,” a rollout covering 20 universities and roughly 600,000 students (Basrin & Henry, 2026). In January, a Brookings Institution study based on interviews and focus groups in 50 countries concluded that “the risks of using generative artificial intelligence to educate children and teens currently overshadow the benefits” (Turner, 2026). On October 5, the World Assessment Council advised that large rollouts measure what students can do without the tool, because “adoption numbers show how many students have an assistant, not how many have a tutor” (World Assessment Council, 2026).
The most encouraging evidence from lower-income countries aligns with the Harvard study. A World Bank program in Nigeria used generative AI as a tutor in a six-week English program in June and July 2024. The bank reported learning gains of about 0.3 standard deviations, larger than 80 percent of the interventions in its database of randomized trials in developing countries (De Simone et al., 2025). Like PS2 Pal, the Nigerian program was a structured activity with scheduled sessions, and the bank found that each additional day of attendance was linked to better results (De Simone et al., 2025).
Harvard’s own follow-up work has not yet been published. In October 2025, the Harvard Gazette reported that “Kestin added that follow-up research is in progress focused on understanding the long-term impacts of tutor bots,” and that “a section of Math 21A was offered with the AI tutor” (Lamb, 2025). In the same interview, Kestin described the risk his own design was meant to avoid. When students are simply handed ChatGPT for homework, he said, “it turns out the students often do worse on the test because they’re using the AI to think for them” (Lamb, 2025). As of mid-October 2026, ETCJ has found no published results from the retention study or the Math 21A section.
Conclusion: The ETCJ review described the experiment fairly and listed its main limits, including the two-week span, the focus on mid-level skills, and the Harvard-only sample. However, it should be revised in five places.
The phrase “watershed moment” should go. The study is the strongest single demonstration of what a purpose-built generative AI tutor can do in a controlled course setting, and it should be described that way. The effect size should be reported as the paper reports it: 0.63 standard deviations by standard regression and 0.73 to 1.3 by quantile regression. The unexplained gap between 194 eligible students and the 142 and 174 post-test scores should be noted until the authors account for it. The AI condition should be described as a scheduled course assignment, and the 49-minute figure as measured time set against an assumed 60. Finally, the question the review closed on, how to design AI tutors and where they fit beside human teaching, needs a third part: how to get students to use them as tutors. The 2026 trials show that good design and easy access do not by themselves produce use.
The research agenda proposed by ETCJ is still valid, with three additions from this year’s work. Studies should test students again weeks or months later, as the Toronto team did after one week. They should report how many students used the tool and what share of requests sought direct answers, as the Maryland team did. And replications should be run by researchers who did not build the tutor, at other institutions, with students less prepared than Harvard’s.
For teachers and administrators weighing an AI tutor this year, the evidence supports a short, specific list of practices. Make the tutor part of required work. Choose tools that coach students after mistakes and make direct answers harder to get. Track how many students use the tool and how they use it. Test students without it. The Harvard authors themselves recommended using the tutor alongside in-person teaching, so that class time can go to work that benefits from a room full of people (Kestin et al., 2025). That is a narrower claim than the one that circulated on X in September, and it is the one the evidence supports.
References
Andreou, S. (2026, September 29). Harvard AI tutor study sparks viral ‘universities are obsolete’ claims. Inside AI. https://insideai.news/news/artificial-intelligence/harvard-ai-tutor-study/13208/
Barshay, J. (2024, September 16). An AI tutor helped Harvard students learn more physics in less time. The Hechinger Report. https://hechingerreport.org/proof-points-ai-tutor-harvard-physics/
Barshay, J. (2026, September 21). Students didn’t get answers from Khanmigo. They didn’t want its questions, either. The Hechinger Report. https://hechingerreport.org/proof-points-khanmigo-math-ai-tutor/
Barshay, J., & Fasiang, K. (2026, August 17). Slow math: Kids may learn more when AI makes them review mistakes. The Hechinger Report. https://hechingerreport.org/proof-points-ai-mastery-learning/
Bartanen, B. (2026, October 15). AI tutoring lowers student performance and engagement. Or does it? Substack. https://brendanbartanen.substack.com/p/ai-tutoring-lowers-student-performance
Basrin, O., & Henry, D. A. A. (2026, March 8). Enabling Gemini for Education across all Malaysian public universities. Google. https://blog.google/company-news/inside-google/around-the-globe/google-asia/gemini-malaysia-universities/
Bastani, H., Bastani, O., Sungu, A., Ge, H., Kabakcı, Ö., & Mariman, R. (2025). Generative AI without guardrails can harm learning: Evidence from high school mathematics. Proceedings of the National Academy of Sciences, 122(26), e2422633122. https://doi.org/10.1073/pnas.2422633122
De Simone, M. E., Tiberti, F., Mosuro, W., Manolio, F., Barron, M., & Dikoru, E. (2025, January 9). From chalkboards to chatbots: Transforming learning in Nigeria, one prompt at a time. World Bank Blogs. https://blogs.worldbank.org/en/education/From-chalkboards-to-chatbots-Transforming-learning-in-Nigeria
Kestin, G., Miller, K., Klales, A., Milbourne, T., & Ponti, G. (2025). AI tutoring outperforms in-class active learning: An RCT introducing a novel research-based design in an authentic educational setting. Scientific Reports, 15, 17458. https://doi.org/10.1038/s41598-025-97652-6
Lamb, A. (2025, October 21). What if AI could help students learn, not just do assignments for them? Harvard Gazette. https://news.harvard.edu/gazette/story/2025/10/what-if-ai-could-help-students-learn-not-just-do-assignments-for-them/
LearnLM Team & Eedi. (2025). AI tutoring can safely and effectively support students: An exploratory RCT in UK classrooms (arXiv:2512.23633). arXiv. https://arxiv.org/abs/2512.23633
Létourneau, A., Deslandes Martineau, M., Charland, P., Karran, J. A., Boasen, J., & Léger, P. M. (2025). A systematic review of AI-driven intelligent tutoring systems (ITS) in K-12 education. npj Science of Learning, 10, 29. https://pmc.ncbi.nlm.nih.gov/articles/PMC12078640/
Liu, J., Sweet, T., Chen, M. H., Engelberg, J., Masters, M., Clark, M., Persaud, A., Lancaster, A., Hollingsworth, J., & Rice, J. K. (2026). The effects of course-integrated AI tutoring on student performance and engagement: A randomized university trial (EdWorkingPaper No. 26-1598). Annenberg Institute at Brown University. https://edworkingpapers.com/ai26-1598
Moon, T. (2026, February 17). Research notes: Two emerging strategies for using AI in tutoring. FutureEd. https://www.future-ed.org/research-notes-two-emerging-strategies-for-using-ai-in-tutoring/
Oreopoulos, P., Liut, M., Sungu, A., & Low, N. (2026). Making AI tutoring productive: Evidence from a mastery-based math practice experiment (NBER Working Paper No. 35621). National Bureau of Economic Research. https://www.nber.org/papers/w35621
Oreopoulos, P., & Low, N. (2026). One click away: AI tutoring with Khanmigo in a two-year school experiment (EdWorkingPaper No. 26-1551). Annenberg Institute at Brown University. https://edworkingpapers.com/ai26-1551
Osmonbekov, M. (2026, September 10). Wealthy parents are buying their kids $75,000 AI schooling with guardrails as schools grapple with a literacy crisis. Fortune. https://fortune.com/2026/09/10/alpha-school-wealthy-parents-ai-school/
Shimabukuro, J. (2025, November 10). Review of Kestin et al.’s June 2025 Harvard study on AI tutoring. ETC Journal. https://etcjournal.com/2025/11/10/review-of-kestin-et-al-s-june-2025-harvard-study-on-ai-tutoring/
Superman [@thesupermannx]. (2026, September 28). Researchers proved AI has deleted every reason universities exist [Post]. X. https://x.com/thesupermannx/status/2104519992532496810
Turner, C. (2026, January 14). The risks of AI in schools outweigh the benefits, report says. NPR, via GBH. https://www.wgbh.org/news/2026-01-14/the-risks-of-ai-in-schools-outweigh-the-benefits-report-says
Witten, M. (2026, September 2). Can an AI tutor help students improve at math? UTM study investigates. University of Toronto Mississauga. https://www.utm.utoronto.ca/main-news/can-ai-tutor-help-students-improve-math-utm-study-investigates
World Assessment Council. (2026, October 5). AI tutors have beaten good teaching and cut grades. Design decides which. https://worldassessmentcouncil.org/spotlight/ai-tutors-beat-good-teaching-and-cut-grades-design-decides/
###
Filed under: Uncategorized |
Leave a Reply