By Patrizia Catellani
One of the most enduring human needs is the desire to be understood. From ancient philosophy to modern psychology, this longing for recognition has shaped the way we interact with other people – and now increasingly with machines.
The more artificial intelligence advances, the more the line between conversation and understanding blurs. Can a machine not only answer our questions, but also understand who we are? This question is at the heart of the growing interest in large language models (LLMs) such as ChatGPT – particularly their potential to recognize personality traits based solely on the way we write.
In a recent study conducted by Marco Piastra (University of Pavia) and Patrizia Catellani (PsyLab – Catholic University of the Sacred Heart – Milan), this idea was put to the test. Without any extra training or examples, can ChatGPT-4 accurately infer a person’s personality using the Big Five model, which is based on the idea that personality can be described by five major dimensions: Openness, Conscientiousness, Extraversion, Agreeableness, and Neuroticism? The results reveal some surprising strengths and some critical weaknesses.
More than just a chatbot
Large language models like ChatGPT are already embedded in everyday life, helping users with tasks from composing emails to programming. But their ability to “understand” users – and not just respond – could open up new opportunities in education, mental health, and personalised AI assistants.
To make this leap, however, AI must reliably interpret personality without being reprogrammed for each new user or situation. This is where this study comes in: It tests whether ChatGPT-4 is capable of assessing a psychological profile without special training.
Reading personality from text
To evaluate the capabilities of ChatGPT-4, the researchers relied on two datasets. The first, a classic of computational linguistics, comprised over 2,400 essays by students. The second came from the X social feeds of nearly 300 users. In both cases, the users had completed personality tests based on the Big Five model.
ChatGPT-4 was given a “zero-shot” prompt – no examples, no fine-tuning – and asked to generate scores for each trait on an eleven-point scale, plus a confidence rating. These were then compared with the original self-assessments.
The model performed moderately well: ChatGPT-4’s ability to assess the Big Five personality traits is remarkable, especially considering that no explicit training was conducted for this particular task. But there was a catch: ChatGPT often did not seem to know when it didn’t have enough text to work with. It confidently evaluated features based on too little – or too general – input, casting doubt on its self-awareness as a tool.
A look at the potential
ChatGPT-4’s predictions aligned with self-assessments at a similar level to that found in studies with humans. This is impressive, considering that there is no task-specific training. It suggests that models like ChatGPT are able to pick up on subtle psychological signals.
However, ChatGPT’s confidence scores are another matter. The model often assigned high confidence to estimates based on too little data. Basically, it didn’t know what it didn’t know. This blind spot can be a risk in any area where accuracy and transparency are important.
Without a reliable method for judging when a text is “enough” for analysis, its practical applications remain limited. This is particularly true in sensitive areas such as recruitment, therapy, or education, where false certainty could have real consequences.
Where do we go from here?
The research also raises questions about how we can test these models. Comparing AI-generated results with self-assessments is useful, but not foolproof – people’s perceptions of their own traits are not always accurate. Future studies should incorporate evaluations by trained human judges to improve benchmarks.
And although this study focused on zero-shot performance, there are ways to improve the results using techniques such as prompt chaining or RAG. However, this would require a broader study design and more complex infrastructure.
For now, ChatGPT-4 is very promising. It can assess personality traits from texts with some degree of accuracy – a feat that was unthinkable just a few years ago. But until it can reliably judge when it’s working with enough information – and admit when it is not – it remains more of a research tool than a psychological oracle.
Patrizia Catellani is a professor of Social Psychology at the Catholic University of Sacred Heart, Milan, where she also directs the Psychology, Law and Policy Lab (PsyLab). She teaches Political Psychology and Psychology of Food and Lifestyles. She has published over 130 works and actively contributes to several international research boards and advisory panels.
Originally published under Creative Commons by 360info™.
