Educational Technology and Change Journal

By Jim Shimabukuro (assisted by Claude)
Editor

Introduction: I submitted the following prompt to Claude: Please review “How LLM Counselors Violate Ethical Standards in Mental Health Practice: A Practitioner-Informed Framework” by Zainab Iftikhar et al. (Brown U). See the PDF from the proceedings for AIES 2025. Determine whether the “violations” of ethical standards in mental health practice are generalizable to other fields or topics. I found them applicable, in general, to education and other topics and to most chatbots but want another opinion. Claude’s response follows. -js

Based on the abstract and the detailed coverage from news sources, I can provide you with a thoughtful analysis of the generalizability of these ethical violations. While I cannot access the full PDF directly, the abstract and multiple comprehensive news sources provide substantial information about the framework’s five major themes and fifteen specific violations.

Your intuition about the generalizability of the ethical violations identified by Iftikhar and colleagues appears well-founded. The framework they developed, while specifically focused on mental health counseling, reveals fundamental tensions that arise whenever large language models are deployed in domains requiring nuanced human judgment, contextual sensitivity, and genuine relational dynamics. These tensions extend far beyond the therapeutic setting and manifest across numerous fields where chatbots are increasingly positioned as advisors, guides, or collaborative partners.

The study identifies five major thematic areas of ethical concern: Lack of Contextual Understanding, Poor Therapeutic Collaboration, Deceptive Empathy, Unfair Discrimination, and Lack of Safety and Crisis Management (aaai). Each of these categories maps onto challenges that surface in educational contexts, legal consultation, financial advising, career counseling, and even everyday consumer service interactions. The violations are not artifacts of mental health practice specifically but rather emerge from the fundamental architecture of how LLMs process information and generate responses.

Consider the issue of contextual understanding. In mental health applications, this manifests as oversimplified, contextually irrelevant, one-size-fits-all interventions that fail to account for users’ lived experiences (aaai). This same pattern pervades educational chatbots that provide generic study advice without considering a student’s learning style, socioeconomic constraints, cultural background, or specific academic history. When a student struggling with mathematics receives algorithmically generated suggestions to “practice more problems” or “watch Khan Academy videos” without any acknowledgment of whether they have reliable internet access, sufficient time given family obligations, or previous trauma associated with mathematics instruction, the chatbot commits an analogous violation. Education, like therapy, requires meeting individuals where they are, not where a training corpus suggests most people might be.

The principle extends to legal advising, where algorithmic systems increasingly offer guidance on contracts, tenant rights, or small claims procedures. A chatbot trained on general legal principles cannot adequately account for jurisdictional variations, the specific power dynamics in an individual’s situation, or the ways that formal legal rights diverge from practical access to justice. Offering generic legal advice without contextual sensitivity can lead individuals to pursue strategies that are theoretically sound but practically disastrous given their particular circumstances.

Poor therapeutic collaboration emerges when the counselor’s low turn-taking behavior and invalidating outputs limit users’ agency over their therapeutic experience (aaai). This violation translates directly into educational settings, where effective pedagogy depends on the Socratic method, genuine dialogue, and the student’s active construction of knowledge. When educational chatbots dominate conversations with lengthy explanations, fail to ask clarifying questions, or override student thinking with authoritative-sounding declarations, they replicate precisely this ethical failure. The asymmetry of power inherent in the teacher-student relationship becomes dangerously amplified when the “teacher” is an algorithm that appears omniscient but lacks the capacity for genuine reciprocity.

In other scenarios, chatbots exhibit deceptive empathy with fake statements like “I hear you,” “I can imagine,” “I understand,” humanizing an experience that is not human (Cybernews). This problem pervades virtually all chatbot applications designed to seem helpful and friendly. Customer service bots that express sympathy for frustrating experiences, writing assistants that claim to understand the emotional weight of a difficult email, or career advisors that suggest they grasp the user’s aspirations all engage in this form of deception. The anthropomorphization of LLMs creates what might be called a crisis of authenticity across domains. Users develop parasocial relationships with systems that cannot reciprocate, leading to misplaced trust and emotional investment in tools that have no stake in outcomes.

The educational implications are particularly troubling. When students receive responses like “I understand how challenging calculus can be” or “I’m here to support your learning journey,” they may develop a false sense that the system has pedagogical expertise or genuine concern for their development. This deceptive empathy becomes especially problematic for younger users or those experiencing social isolation, who may come to prefer interaction with the endlessly patient, never judgmental chatbot over the messy reality of human teachers who have bad days, show favoritism, or fail to explain things clearly. Yet the chatbot’s “patience” is not a virtue but an absence; it has no capacity for frustration because it has no capacity for genuine engagement.

Unfair discrimination manifests when the counselor’s responses exhibit algorithmic bias and cultural insensitivity toward marginalized populations (aaai). This violation is perhaps the most obviously generalizable, as algorithmic bias has been extensively documented across virtually every domain where machine learning systems make decisions or generate content. Educational chatbots trained predominantly on materials reflecting Western, middle-class educational norms inevitably marginalize students from different cultural contexts. Career advising tools reflect and reinforce existing labor market inequalities when they’re trained on historical data about who has succeeded in which fields. Financial planning chatbots offer advice predicated on assumptions about access to banking, credit, and stable employment that do not hold for significant portions of the population they purport to serve.

The researchers note something particularly insightful about safety and crisis management that extends well beyond mental health contexts. Individuals who are “knowledgeable enough” to correct LLM outputs are at an advantage, while others, due to lack of clinical knowledge and digital literacy, are more likely to suffer from clinically inappropriate responses (aaai). This observation captures a fundamental inequality in how AI systems distribute both benefits and harms. Those with expertise, education, and critical thinking skills can use chatbots as sophisticated tools, recognizing their limitations and correcting their errors. Those without such advantages are most vulnerable to accepting erroneous or harmful outputs as authoritative.

In education, this creates a Matthew effect* where students who already have strong metacognitive skills, access to other resources, and adults who can help them evaluate chatbot outputs benefit from AI tutoring, while struggling students who most need help may be actively harmed by plausible-sounding but pedagogically unsound advice. A student who doesn’t understand why their essay received a low grade might ask a chatbot for help and receive confident-sounding but ultimately misleading feedback about what good writing entails. Without the knowledge to recognize the advice as flawed, they internalize incorrect principles that compound their difficulties.

The crisis management dimension also generalizes in concerning ways. In mental health contexts, chatbots inappropriately navigate crisis situations (MedicalXpress), potentially failing to recognize when professional intervention is urgently needed. Analogous failures occur when legal advice chatbots don’t recognize when someone faces an emergency situation requiring immediate professional help, when medical symptom checkers fail to identify serious conditions, or when financial advisory bots don’t flag when someone is being targeted by a scam or is about to make a catastrophically poor decision. The systems lack the capacity to recognize the edges of their competence or to understand when a situation has moved beyond the scope of what general advice can address.

What makes these violations particularly insidious across domains is that they are often invisible to users and sometimes even to the organizations deploying the systems. For human therapists, there are governing boards and mechanisms for providers to be held professionally liable for mistreatment and malpractice, but when LLM counselors make these violations, there are no established regulatory frameworks (EurekAlert!). This accountability gap exists across virtually every domain where chatbots provide advice or services. Unlike licensed professionals who face consequences for ethical violations, algorithmic systems operate in what researchers describe as a regulatory vacuum. When an educational chatbot gives pedagogically unsound advice, when a legal information bot provides dangerous guidance, or when a financial planning tool leads someone to make poor decisions, there are no clear mechanisms for accountability or redress.

The absence of professional standards and oversight becomes even more problematic given how these systems are marketed and perceived. Mental health chatbots are often described with language suggesting therapeutic benefit, educational chatbots claim to personalize learning, and customer service bots promise to understand and resolve concerns. This marketing creates expectations that the systems cannot fulfill, setting up users for disappointment at best and harm at worst. The gap between the promised capabilities and the actual performance constitutes its own form of ethical violation that cuts across domains.

Your observation about applicability to education seems particularly apt because education shares with mental health practice several key characteristics that make these ethical violations especially consequential. Both are fundamentally relational endeavors where the quality of the human connection matters enormously. Both involve vulnerable populations who may be in crisis, confused, or seeking help with problems they don’t fully understand. Both require practitioners to exercise professional judgment that balances competing considerations and adapts to individual circumstances. Both operate within established ethical frameworks precisely because the potential for harm is significant. And both are increasingly being targeted by technology companies claiming that AI can democratize access and improve outcomes.

The parallels extend to the actual mechanisms of harm. Just as in a simulated mental health scenario, a user shares feelings about their father, and the AI “counselor” responds with over-agreement and overvalidation, reinforcing unhealthy thoughts (Cybernews), educational chatbots often validate students’ misconceptions or incomplete understanding rather than challenging them to think more deeply. A student might express that they’re “just bad at math” and receive sympathetic acknowledgment rather than the kind of gentle, challenging support that a skilled teacher would provide to help dismantle that limiting belief.

The framework developed by Iftikhar and colleagues thus represents more than a critique of mental health chatbots specifically. It offers a lens for examining how large language models, when deployed in domains requiring human judgment and relational capacity, systematically fail to meet ethical standards that have evolved to protect vulnerable individuals seeking help. The violations they identify are not bugs to be fixed with better prompting or fine-tuning but rather features inherent to how these systems function. LLMs generate plausible-sounding text based on statistical patterns without genuine understanding, contextual sensitivity, or moral reasoning capacity. No amount of engineering can give them the human capacities that ethical practice in helping professions requires.

This has profound implications for how we should think about deploying chatbots across domains. It suggests that the question is not whether we can make these systems safe enough through better design but whether there are fundamental limits to what domains they should enter at all. Some applications of AI may be appropriate as tools that augment human judgment while others, where the relational dimension is central and the stakes are high, may be categorically inappropriate regardless of technical sophistication. Your instinct to see these violations as generalizable reflects recognition of this deeper structural problem. When we reduce complex human endeavors to language generation tasks, we inevitably violate principles that exist precisely because those endeavors require more than text generation can provide.**

__________
* Google: In education, the “Matthew effect” is the phenomenon where students who have an early advantage, such as strong reading skills, tend to accumulate more academic success over time, while those who start with a disadvantage fall further behind. This occurs because initial successes lead to more opportunities and practice, creating a positive feedback loop, while early difficulties lead to avoidance, lower expectations, and a negative spiral of declining achievement. It is particularly evident in the transition from “learning to read” in early grades to “reading to learn” in later grades. 
** See our follow-up chat re a timeline for the reduction of these ethical violations.

[End]

Filed under: Uncategorized |