Just a few years ago, conversation flow design was at the heart of chatbot research (Cho et al., 2025). Designers developed detailed guidelines to structure dialogues, crafted messaging frameworks for seamless interactions, and carefully designed output messages to align with chatbot personas. Then as transformer-based large language models (LLMs) arrived, the rigid and predefined conversation structures that worked for rule-based systems couldn’t accommodate LLMs’ dynamic, context-aware response. Research priorities shifted from designing fixed dialogue trees to exploring prompt engineering and interaction patterns (Cho et al., 2025). The expertise built over years around conversation flow design was fundamental but it needed rapid reframing; designers had to rethink how to guide conversations without prescribing every turn, how to maintain coherence without rigid structures, and how to evaluate interactions that varied with each user (Subramonyam et al., 2024). This narrative about change isn’t just applicable to chatbots. It’s applicable to Generative AI’s (GenAI) unique temporal challenge and it raises a critical question for anyone designing with or for AI – are our design methods keeping up, or do we need new ones?

Trigger Warning/Disclaimer: This blog post mentions suicide.

Governments, startup founders, academics, mental health professionals and others wrestle over who gets to define the future of AI mental health care.
Amidst a lack of regulatory oversight regarding AI-based mental health chatbots, some states in the US have taken steps to ban these systems in order to protect the public. Full bans are in place in Illinois and Nevada, and although Utah has not banned it outright, it still imposes strong restrictions and requirements around transparency, advertising, data use and human professional involvement. Bans as a political strategy and policy risk unintended consequences on a population-wide scale (Oliver et al., 2019).

Large language models (LLMs) are trained on large amounts of text from the internet, books, forums and other sources in a process called pre-training. This gives them great versatility, but also comes with a hidden challenge: human language data contains biases, misinformation and unsafe patterns, such as hate speech, toxic or discriminatory content. When models learn from such data, they not only gain useful knowledge but also inherit these problems. On top of this, LLMs tend to be statistically overconfident (Guo et al., 2017; Minderer et al., 2021), meaning they assign higher probabilities to their predictions, due to the way that they interpret data (Xu et al., 2024). They often present information with certainty, even when the output is false. This combination of biased training data and overconfidence can lead to hallucinations, biased answers or unsafe outputs, such as toxic content or instructions for harmful behavior.