AI Hallucinates, and Academics Overreach: Why Both Are a Mistake
Artificial Intelligence isn’t perfect, but academics oversimplifying its flaws—or its potential—does more harm than good.
I watched a speech by Professor Shannon Vallor, who spoke at the Turing Institute about AI, posing the question: Can we live with AI? I initially thought that was a kind of nonsensical question. It’s not a particularly clever or interesting question, and without context, it’s hard to understand what she’s getting at. We do live with AI now, so the answer is clearly yes—we can and already do.
Her research at the University of Edinburgh “explores how new technologies, especially AI, robotics, and data science, reshape human moral character, habits, and practices.” Think of it: my love for Scotland combined with my passion for technology and philosophy in a single person. And yet, I don’t feel her description of AI and my experience of it have much in common. How can that be? In this case, I feel a counter perspective is needed that doesn’t conflate ideological preferences with an assessment of AI’s impact as a technology.
It also serves to expand my digital writing portfolio with another AI-focused piece to complement my blockchain work. In case you forgot, I know a little about AI as well.
Technology has always been a double-edged sword
Her speech begins by mentioning a few positive outcomes AI could create but then says this should not distract us from the fact that one could weaponise AI, etc. Sure. Throughout history, technology has been used for good and bad, and it would be naive to think we could escape that with AI, unfortunately. The Nobel Peace Prize is a good reminder since it has its origins in Alfred Nobel's concerns about the destructive potential of his inventions, particularly dynamite and other explosives, which he feared would be misused in warfare. I don’t find it particularly insightful to state that AI also has destructive potential.
Advances in technology amplify destructive potential at an accelerating pace, which means each generation faces a greater responsibility than the one before. Alfred Nobel only got the idea when his brother died, and a French newspaper mistakenly wrote an article with the headline “Le marchand de la mort est mort” (“The merchant of death is dead”), mistakenly reporting Alfred Nobel’s death. We can’t be certain what his exact motivation was, but this anecdote shows that calling out these issues serves a purpose. If that was her intention, then why not?
The Professor’s concerns are, of course, not unfounded, but neither are they caused by or specific to AI: environmental impact due to the energy consumption of technology and economic and social inequality are not AI-specific phenomena—and neither is surveillance, to mention a few of the risks. But her description lacks objectivity. I’ll give you one example: generative AI models are based on tensor calculations performed on specialised hardware, such as GPUs or TPUs, which are significantly more energy-efficient than the CPU running your laptop. AI increases demand for energy, but it has also led to innovation in hardware design, which helps to mitigate that impact, at least to some extent (depending on overall usage, scale of the models, etc.). It would be careless to ignore this positive consequence because the benefits of this innovation may extend to other areas as well. These are complex and dynamic processes and can’t be reduced to slogans like ”AI is bad for the environment.”
She then describes how AI makes people watch mind-numbing videos on social media, so I take it she is not a fan of Instagram. Yes, social media does have that effect of serving up trivia, but I really question why AI makes that worse. If AI produces this content cheaper, which she states it can, well, at least we have less inequality in enjoying high-quality mind-numbing video clips since she was concerned about inequality in wealth and political power.
What motivated me to write this blog was, however, her technical description of AI, which seems superficially correct:
AIs make mistakes, lack true reasoning and understanding, and often fabricate false answers or images.
AI mirrors past patterns in data.
But then she applies these generic statements and a few imprecise definitions to reach very questionable conclusions.
AIs make mistakes. So does any software application, which can and does exhibit errors or unexpected behaviour due to various factors—from bugs, hardware issues, and compatibility with the software environment to user input or even hard-to-replicate errors. As before, regarding the energy impact, such concerns need to be seen relative to other technology and the intended purpose. Overreliance on a particular technology is hardly a promising strategy, but the fact that output generated by AI cannot be guaranteed to be always correct is not, per se, an argument, since the same could be said about any other piece of software.
Then she says when AI ‘misses the right answer, it often misses not by feet but by a mile’ and if so, you will ‘get a completely fictional made-up path of reason’ related ‘to the mathematical vector space it traversed.’ Let’s consider the first part first.
Professor Shannon does not explain whether she thinks a technology that produces many little errors would be preferable over one that produces fewer, bigger errors, or if the concern relates to anything that leads to significant errors regardless of frequency. Absolute certainty is hardly feasible, and it seems impossible to quantify the impact of a software fault without limiting the activity it supports. The software at a nuclear reactor may cause bigger issues than the Instagram app, and that’s why we don’t let the app run nuclear power plants. It’s a serious point since this is how we deal with such issues. Thinking that AI would be the best technology for any given problem is, of course, problematic. So what?
There are a few reasons why this effect of AI producing weird or absurd outputs is often noted, and once you know the reason, it becomes less of an issue, I think.
How Does the AI’s ‘Brain’ Work?
AI doesn't ‘understand’ content in the way humans do. So a car and a flower make no difference. Each is simply a data set recorded as a vector, which provides a mathematical description of the relations derived from patterns in the training data, but they lack inherent meaning. Operations like writing a text or answering a question are mathematical transformations over these vectors. The content itself doesn’t matter; only the relationships between vectors do. That is why the false outputs often seem so extremely false to us, but it is simply a wrong representation of data relationships. Any error is not bigger or smaller than the next; they are all the same. The impact, of course, can differ in the real world. It’s incorrect to consider a totally absurd output (her example of racially diverse Nazi soldiers produced by an imaging AI) as an indicator of ‘the utter absence of true thinking and understanding.’ There is no understanding or thinking even if the AI makes only a tiny mistake. This needs to be stated upfront and not repeated at every step as if it reveals something deeper.
This brings me to the second part—the fictional made-up reason part when you ask an AI: “Hey you, ChatGPT, explain yourself!” AI models are considered opaque in how they arrive at decisions. This is, of course, an issue, although I am increasingly of the opinion that in the same way we add redundancy because we recognise that humans are fallible (e.g., pilot and co-pilot), we can’t fully trust its output. So we require controls and redundancy—independent systems (e.g., fact-checking AIs, adversarial AIs) can challenge each other's results. When multiple systems agree or flag issues, reliability improves. Achieving "zero errors" is unrealistic for any system, human or machine. The real goal is to minimise error impact, and that seems more than feasible. The professor mentioned ‘temperature,’ which describes so-called random sampling.
It is feasible to make AI 100% deterministic (i.e., if A then B), but then it would lose its ability to produce diverse or novel outputs. Besides temperature, there are other design features that mean that if you had two copies of the same AI model producing different answers to the same question, such as different optimisation paths created during training, they would behave similarly but not identically. In other words, the ability for AIs to create novel output is not simply a question of yanking up the guesswork wheel a bit more.
I also don’t know why she calls this ‘artificial’ randomness, which would imply there is some other form of natural randomness. This is a mathematical concept and neither artificial nor natural. Randomness in natural phenomena (like rolling dice or playing the lottery) arises from physical processes (e.g., unpredictability of forces) but is this therefore natural randomness? No.
Fictional explanations
And that brings me to Professor Shannon’s assertion that AI only provides fictional made-up reasons to explain their output. It sounds like alleging a form of intent to guard the real reason, which, at least for now, AIs are incapable of unless programmed to do so. Putting that aside, I think the type of task performed matters to understand this properly:
If we ask an AI to answer what is 5+5 and then ask for an explanation of how it arrived at the output, the ‘reasoning’ is clear and universally replicable. AI explains itself algebraically, and humans can follow the same logic. In a previous blog, I covered this example where LLMs can make mistakes in calculating the number of ‘r’s in strawberry, which is a direct consequence of their technical design; hence the cause and solution are not opaque. If we ask a subjective question, such as ‘write an essay about Irish Literature in the 20th century’, then we do have this effect that there is no answer to explain the precise reasoning.
A person might base their essay on historical texts, literature, or personal preferences. We are often unaware of why we favour one framework over another—it’s an intuitive, heuristic (trial and error) process. Even if we are asked: “Why do you like this?” and we say “because of so and so”, it only leads to the next question: “But why do you like so and so?” AI, in a way, faces the same issue when it picks patterns from training data to apply to a task. The AI has no reason because it doesn’t reason. It picks certain patterns based on its learning and overall design.
When the AI generates an output, it selects from a set of possible "connections" (e.g., words, tokens, or vectors) based on probabilities derived from its model. If we look inside the AI model after an output is generated, it would generally be possible to analyse and say: "Here are the 1000 vectors or tokens that contributed to this decision." These vectors (inputs, embeddings, or connections) are traceable because they arise from the model's attention mechanisms or activation patterns. So, in a given instance, the "what" (i.e., the 1000 chosen data vectors) is knowable.
What is not knowable in a fully explainable sense is "why these 1000 vectors and not others?" This ambiguity arises because AI doesn’t consciously "choose" these vectors—it weights them based on probabilities derived from the training data and its learned model. If there were 2000 candidate vectors, the AI’s internal calculations (weights, activations, and attention scores) would probabilistically favour one set over another. The "decision" is emergent from millions of interacting parameters and cannot be boiled down to a simple "reason." Asking for intent works in criminal law, but it doesn’t apply to AI.
So asking an AI to explain its reasoning will not lead to a logical output because it doesn’t know its reason—because it has none. As weird as this sounds, it actually makes a lot of sense if you think about what it ultimately does: calculating probabilities based on relationships in a dataset. What is fascinating, though, is that we are able to describe complex issues, at least to a certain extent, mathematically and derive—not always but often—useful outputs based on mathematical operations, even if we are talking about Ulysses. And if you think from this perspective, the limitations of such a model become obvious. What can someone or something tell you about literature when it doesn't feel emotions, or sense beauty in language? A lot or nothing. It depends on the context of the question and our expectations.
UPDATE 19 December 2024:
And then she says that this only allows for a new “guess… no less based on the past… it’s just a slightly wilder guess.” This may be just a figure of speech, but her conclusion also repeats this idea that AI is based on past data and is thus often described as being incapable of producing anything other than “mirrors” of the past, “stuck pointing backward, reflecting what has been seen in the past,” etc. This perspective is overly reductive because AI does not simply reproduce the past; it generates outputs by combining patterns and relationships in ways that can result in novel insights or unexpected outputs. For example, hallucinations like imaging AI producing racially diverse soldiers in the Third Reich may be incorrect but clearly do not reflect an exact repetition of past data. These cases illustrate that AI can generalise beyond its training data—even if imperfectly—showing that it is not entirely bound to past patterns. Saying otherwise is, with all due respect to a Professor focusing on Artificial Intelligence, nonsense.
Innovation isn’t the same as intelligence
Human thinking is also shaped by learning from past experience, but that’s not my main concern. Intelligence often involves applying known rules effectively (e.g., solving problems within a defined framework). Innovation, by contrast, involves breaking or redefining those rules, something that is by definition not a predictable process. Our ability to reflect, assign value, and imagine beyond immediate needs sets us apart from current AI, which can mimic these processes mathematically but doesn’t experience them subjectively and thus lacks the ability to pass judgement. But humans also lack fully objective criteria for many questions and differ in their preferences.
What AI specifically lacks when it simulates creative thinking is the ability to determine absurd from ‘hmm, that could be good’ if the training data or model doesn’t allow it to make reliable connections. But this is not, per se, different for humans. We know a cat is an animal which needs food and water, so if we encounter a new animal unknown to us, we may think it could be hungry or thirsty. A generative AI is able to do this if it has made the right data connections between factors identifying an animal and its needs. The amusing nonsensical outputs are thus the result of where this is not the case.
On the other hand, can we tell with our intellectual capabilities what good looks like in new creations? I would say we don’t know objectively. J.K. Rowling faced many rejections before her Harry Potter series became a success. This should not have happened if there were any objective criteria as to why her books became a huge success. It shows that considering successful innovation or creativity solely as a function of human intellect is another bias in her reflection. Human creativity is littered with trial and error, coincidences, and failure. Why would a machine be able to pick the correct lottery numbers for next week?
What could be difficult for humans is often computationally easy, but what we find easy and intuitive is computationally difficult. That fact can lead to misjudging the effectiveness of AI. It might be "better" at intelligence (rule-following, efficiency), but innovation involves a level of unpredictability and understanding of values. Humans feel responsibility or association (e.g., a parent’s sense of responsibility for their child). This sense of value drives much of human decision-making, but it is entirely absent in AI, which lacks the underlying emotional or existential drive to innovate. But this doesn’t mean that AI cannot arrive at conclusions that exceed its input data.
Characterising AI as “stuck pointing backward, reflecting what has been seen in the past” (whilst also pointing to errors such as hallucination) is not only misguided but also contradicts Professor Shannon’s own evidence. Because in what past data did the imaging AI find the radically diverse soldiers in the Third Reich?
I do not think lightly of the potential consequences of AI, good and bad, and I share a lot of the concerns Professor Shannon raised. But I would question how she portrays AI because we need to learn how to use AI correctly instead of ‘schwadronieren’ about the fact that AI does not possess human intelligence. Schwadronieren is a wonderful German word meaning to talk endlessly and boastfully.
Simply saying AI doesn’t understand how humans reason doesn't tell the full story. But when we understand how it works, we can use it appropriately, as is the case with other technology. From my little experience, AI has enabled me to analyse problems and find insight that I thought impossible to tackle on my own. But I don’t expect ChatGPT to be the next J.K. Rowling or a piece of technology that is without fault. And I hope your experience with AI is as fun and thought-provoking as mine.