Artificial Intelligence (AI) has been a game-changer, simplifying tasks like medical diagnoses, coding solutions, and much more. However, new research suggests that AI, much like humans, might experience cognitive decline over time.
A study published in BMJ in December 2024 raises questions about the longevity and reliability of AI models, especially in the medical field. The research highlights that prominent AI technologies, including large language models (LLMs) and chatbots, show signs of cognitive decline over time, mirroring the effects of human aging.
AI’s Cognitive Decline: A Closer Look
The study, which focused on evaluating the cognitive abilities of several leading LLMs, found that older versions of these models performed worse on tests compared to their newer counterparts. The models examined included:
- ChatGPT 4 and 4o
- Claude 3.5 ‘Sonnet’ (developed by Anthropic)
- Gemini 1.0 and 1.5 (developed by Alphabet)
Researchers used the Montreal Cognitive Assessment (MoCA) test to measure attention, memory, language, spatial skills, and executive function. Surprisingly, the results showed that only ChatGPT 4o passed with a score of 26/30, while ChatGPT 4 and Claude scored slightly lower at 25. Gemini 1.0, however, scored just 16, indicating more significant cognitive decline.
What is the MoCA Test?
The MoCA test is a widely used tool to detect cognitive impairments in humans, particularly in older adults. Adapted for AI, it evaluates several cognitive functions, including attention and executive skills. In human subjects, a score of 26/30 or higher is considered normal, but most of the AI models fell below this threshold, signaling early cognitive impairment.
AI’s Struggles in Visuospatial and Executive Tasks
One key finding was that all the chatbots struggled with tasks requiring visuospatial skills and executive function, such as the trail-making exercise and clock-drawing test. Additionally, Gemini 1.0 failed a recall task, struggling to remember a five-word sequence.
Implications for AI in Medicine
These cognitive impairments raise concerns, particularly in the context of AI tools being used for medical diagnoses. While AI has made great strides, the cognitive limitations seen in models like ChatGPT and Gemini challenge the assumption that AI will replace human doctors. This may affect the trustworthiness of AI in critical areas, such as medical diagnostics, where precision is crucial.
Interestingly, the study suggests that neurologists may not need to worry about being replaced by AI anytime soon. Instead, they might find themselves treating AI models experiencing cognitive decline in the future.
Looking Ahead: The Future of AI in Medicine
Despite the study’s findings, researchers remain optimistic that AI models will improve over time, with future updates possibly enhancing their performance. However, the study emphasizes that fundamental differences between human and machine cognition will likely persist.
It’s important to note that any anthropomorphized language used in the study was purely metaphorical. AI, unlike humans, doesn’t experience neurodegenerative diseases in the same way.