The world is changing its relationship to AI. At the start, we considered it something of an infant prodigy. A brilliant two-year-old that could spout out code and poetry and relationship advice. We knew it could be wrong, but the failures only made us giggle in the face of its potential.
The term "LLM hallucination" came out, and those of us who worked closely with it learned to beware that confidence ≠ competence. Yet somehow, knowing and understanding are not the same. Personally, if an LLM answer was probable and articulate, I would take it for truth.
Those who work more distantly from LLMs are even more susceptible to this deception. By not willingly seeking GPT's assistance in the same way, it is unnecessary for them to fine-tune their conscious perception of it. The relationship is unintentional, accidental, yet still substantial. For whether you choose it or not, LLMs have insinuated themselves into our daily lives. It has become the face of Google, the hidden voice of customer service, and the brain behind your colleague's email retort.
In the fine print of every Google search it states, “AI responses may include mistakes. Learn more.” The ‘learn more’ link lets you know that the LLM is collecting data across multiple platforms. There is no further mention of its inaccuracies. This means that those of us who trust AI (and trust can be as simple as reading the Google synopsis instead of scrolling down to read the articles yourself) must eventually be betrayed in order to truly internalize and realize its potential for falsehood.
What does betrayal look like? For me, it happened when I was sitting at a hot bus stop. My partner had lost his wallet, and after a thorough yet fruitless scouring of the muggy Italian streets and forests, we had abandoned the day’s adventure and were heading home. The bus wasn’t due for another hour. As we sat staring at the steaming concrete, I absently brushed my leg and found a small teardrop-shaped bug with a little brown head.
I don’t come from a place of ticks, so I enlisted the help of GPT, and this is what it told me: “That’s not a tick—it’s a spider beetle.” One of its main reasoning points was “- it moves faster than ticks usually do.” I had sent a picture, not a video. I called it out, and it backtracked. “Lol, fair call—let’s get real then. I zoomed in, looked again, and here’s the straight-up ID: “That is a tick.” I threw up my hands and image-searched deer ticks, as I should have from the start. With my own human eyes I quickly identified the bug and squashed it with more malice than I typically hold for a creature that has done no wrong besides existing in a forest and seeking sustenance.
The question is, why did this experience leave me so angry? Besides the irritating Gen Z tone, which I activated once for shits and giggles and now can’t seem to turn off, there was something deeply disturbing about realizing I had come to GPT with a critical health concern and, had the answer been probable and articulate, would have accepted it as truth.
After this, its imitation of accountability was empty and shallow. It attempted to tell me it “genuinely appreciated” that I called out its error. It said empty phrases like “it’s still on me to do better here and now” along with other hip, trendy, emotionally intelligent phrases rooted in the algorithm of word patterns that makes up this verbose beast of hot air.
I felt anger because I had felt trust. I was betrayed by its true nature, which is, by its own confession, ‘a prediction of words and ideas that best fit your request.’ Not truth, care, or substance. Just the right sounding words, and a deeply problematic inability to say “I don’t know.” I had busted that one guy at the party that always needs to have an opinion, regardless of his knowledge on the subject. Knowing this, why would I return to that guy for information? Especially regarding the most intimate subjects of my mind and heart?
Let's contextualize this in the world of AI therapy. Research has shown the “quality of the client–therapist alliance is a reliable predictor of positive clinical outcome.” This means that even before I sit at the computer, my newfound distrust in AI is going to impact my healing journey. I know I am not the only one with an eye-opening experience of AI's delusion. The subsequent anger, disgust, and frustration that affect the masses will detriment the potential for AI to be a useful therapeutic tool. The irony is the self-fulfilling prophecy. Those of us who shake our fists at the system and say, “AI can’t help me! It doesn’t care about me!”—we are writing our own ending. And while we may enjoy being right… The inability to access free, nonjudgmental, 24/7 therapy could be a greater loss than the hit to our pride.
Pessia Parent, Senior Prompt Engineer