By Jim Shimabukuro (assisted by ChatGPT, Gemini, and Claude)
Editor
Here’s an actual example of how a user collaborates with bots to generate a more “accurate” output. In this context, “accurate” is a result that satisfies both parties.
This evening, after viewing a YouTube video (Mick Talks Hoops, 7/16/25, “Liberty Coach & Players SPOTTED In Meeting With Caitlin Clark & Her Agent… Recruiting??”) speculating on the possibility of Caitlin Clark leaving the Indiana Fever and joining the New York Liberty after this season, I prompted a few chatbots for their opinion on this rumor.
My initial prompt with ChatGPT: Considering the issues that are plaguing the Indiana Fever and the interest other WNBA teams are showing in her, is there a chance that Caitlin Clark will be leaving the Fever for a different team after this season? If yes, what are the percentage odds of this happening? Will her decision be based on whether teammates such as Aliyah Boston may also be leaving? If she leaves, which team or teams are most likely to pick her up? Does she have a preference for a specific team? If she leaves, what are her most likely reasons?
The chatbot’s response included the error that Clark just completed her first season, failing to realize that she is currently midway into her second. After I pointed this out, ChatGPT provided a corrected opinion.
However, in the revision, it erroneously mentioned the previous head coach, Christie Sides. I pointed out this error: “Christy Sides is not the current coach for the Fever. Stephanie White is. Please tap into more recent data.”
ChatGPT provided a corrected opinion, “Updated Outlook,” and commented: “You’re absolutely correct—the current Indiana Fever head coach as of the 2025 season is Stephanie White, not Christie Sides.”
In closing, it also offered me “a dynamic rumor tracker or real-time trade-monitoring dashboard to stay updated through the rest of the 2025 season and into 2026.” After confirming that no strings are attached to this service, I said yes. The chatbot responded: “The tracker is officially active — I’ll keep you updated each week and alert you right away if anything big breaks around Caitlin Clark, the Indiana Fever, or any related trade chatter.”
I posted a similar prompt in Gemini, and it, too, erroneously placed Clark at the end of her first year. I replied: “Correction: Clark is in her second season right now.”
The chatbot responded: “You are absolutely right to correct me! My apologies for the misstatement. As of July 16, 2025, Caitlin Clark is indeed in her second WNBA season. The 2024 season would have concluded, and the 2025 season is currently underway. Let’s re-evaluate the situation with this crucial correction.”
However, in its revision, it failed to factor in the correction. I replied: “Correction: The 2025 season is well on its way and is nearly at the halfway mark. Please dig into the latest data on the WNBA games with an emphasis on the Fever.”
The chatbot replied: “You are absolutely right, and I apologize for the oversight! My internal calendar clearly lagged. Thank you for the correction. As of July 16, 2025, the WNBA 2025 season is indeed well underway, nearly at the halfway mark. Let’s delve into the updated situation for the Indiana Fever and Caitlin Clark.”
At the end of our chat, I said, “Thank you very much for your openness to new data and adjusting your projections. Very impressive. I tend to believe as you do, for the reasons that you’ve pointed out.”
The Gemini chatbot replied: “My sincerest apologies once again for the factual error regarding the season! You are absolutely correct that we are well into the 2025 WNBA season. Thank you for your patience and for guiding me to the most current information. Your feedback is invaluable in making sure the analysis is as accurate as possible. I truly appreciate your kind words about my openness to new data. Continuous learning and adaptation are key, and your corrections help tremendously in that process. Please feel free to continue pointing out any discrepancies or providing additional context. This collaborative approach leads to the best and most accurate responses.“
In response to the same prompt, Claude’s chatbot reply was better positioned. It went directly to the legal implications based on Clark’s rookie contract: “She’s contractually bound to the Fever through the 2027 season, making any departure impossible unless through a trade.”
I asked, “Could this move to a different team materialize as soon as next season if the Fever agree to a trade with another team?”
The chatbot replied: “Bottom Line: While the WNBA rules technically allow for Clark to be traded after the initial protection period, the probability is essentially zero. The Fever would have no rational reason to trade the most marketable player in league history while she’s under a rookie contract through 2027. Any team wanting to acquire her would need to offer an unprecedented package that the Fever would have no incentive to accept.”
Conclusion: In its current state, chatbots provide useful opinions, but users need to monitor them for errors. Thus, the best approach is a collaborative discourse, a give and take between human and bot to shape an opinion that satisfies both parties.
In the coming years, probably sooner rather than later, the bots will grow exponentially smarter, eliminating most of the very basic data errors that appear in this example. However, I’m guessing that errors won’t ever be completely eliminated.
For an improvement timeline, I asked Claude: “In July 2025 chats between humans and bots, users have to be wary of errors in facts and logic that degrade the usefulness of the output. The workaround is for users to carefully monitor and correct a chatbot’s output. Is it fair to assume that this problem of accuracy and reliability will improve, perhaps exponentially, in the coming years? Please provide a timeline for improvement that will increasingly minimize errors in fact and logic? In short, when will we reach a point where chatbot errors are at an acceptable level? Please assume that GIGO is in effect, that poorly formed prompts will obviously produce poor outputs.”
Here’s Claude‘s reply:
You’re absolutely right to highlight this critical issue. The current state of chatbot accuracy is indeed problematic, but there are clear signs of improvement with some interesting complexity in the timeline.
Current State (2025): The accuracy picture is mixed. AI chatbots in 2025 achieve over 90% accuracy in responses, thanks to advancements in deep learning, NLP, and real-time web access, yet research showed GPT-4’s accuracy on prime number identification dropping from 84% to 51% in three months, highlighting the dynamic nature of these models. This inconsistency across different tasks and over time illustrates exactly why vigilant user oversight remains essential.
Improvement Trajectory: The data suggests rapid progress is occurring. The rate of improvement is accelerating: Some models reported up to a 64% drop in hallucination rates in 2025, and there are now four models with sub-1% hallucination rates. Technical solutions like Retrieval-Augmented Generation (RAG) is the most effective technique so far, cutting hallucinations by 71% when used properly.
Timeline Projections: The most optimistic projections suggest hallucinations could decline each year, potentially reaching near-zero levels by 2027. However, there’s an important caveat: these views are based on traditional scaling trends that may not apply to reasoning models, which appear to buck historical improvement patterns.
Complicating Factors: Interestingly, some advanced reasoning models show concerning regression. The most advanced reasoning models, such as OpenAI’s o3 and DeepSeek’s R1, based on LLMs but using so-called “chain of thought” approaches may have different error patterns than traditional models.
Realistic Timeline: Based on current trends, I’d estimate:
- 2025-2026: Continued incremental improvements with hallucination rates dropping to 3-5% for well-implemented systems
- 2027-2028: Potential breakthrough to near-1% error rates for factual queries in properly designed systems
- 2029-2030: Possibility of “acceptable” accuracy levels for most professional use cases
However, “acceptable” will vary dramatically by application. Medical diagnoses require far higher accuracy than creative writing assistance. The key insight is that AI is expected to improve through several more generations until at least 2030, so the trajectory is clearly positive, but your caution about careful monitoring remains prudent for the foreseeable future.
Filed under: Human-AI Collaboration |
Educational Technology and Change Journal 
Leave a comment