Have you ever wished ChatGPT a good morning? Asked it to “please” do something for you? Or thanked it when it did? Now ask yourself why. Maybe you wanted to be polite? Maybe it was a reflex? Or maybe you expected it to provide better results if you were friendly?

As AI systems (Large Language Models (LLMs) in particular) become more fluent, responsive, and seemingly empathetic, many users report a subtle shift in how they interact with them. ChatGPT remembers things from your past conversations, meets your energy if you are polite, and remembers your name. At times, it feels less like using a tool and more like speaking to a person.

Imagine this: You’re in a courtroom and have been convicted of a crime. Before announcing the sentence, the judge consults a risk assessment tool – an AI system designed to promote consistency and reduce human bias in sentencing. The algorithm generates a score meant to help the judge weigh rehabilitation potential versus public risk. It’s intended to be impartial. Would you trust the tool to fairly assess your case? Would the judge?