Beating the Bot

AI developers are known for making bold statements about their products. In 2023, OpenAI’s CEO Sam Altman proposed that AI could be creative. A year later, his then-Chief Technology Officer Mira Murati asserted AI could even take over some creative jobs. To Professor Dawei Wang of HKU Business School, those statements were provocative. “I was very curious to know how AI compares to humans in terms of creativity,” he said.

The answer came in a collaboration with Professor Haipeng Shen, Patrick S C Poon Professor in Analytics and Innovation and Chair of Business Analytics and Innovation, in a paper for Nature Human Behaviour, with a cover illustration painted by former fine art student Professor Wang. The journal’s editors told them this was one of the few large-scale studies they had seen comparing human and AI performance.

The team zeroed in on one measure of creativity – divergent or out-of-the box thinking – by asking people and AI platforms to create 10 words as different from each other as possible. Data was collected from 10,000 humans in Asia, North America, Europe and Australia, and 200,000 prompted responses from various AI platforms.

The results, on the surface, were surprising. Both humans and AI scored an average of about 79 out of 100. “We did not expect this,” Professor Shen said, “but once we saw it, it made sense. Because large language models are regression models trained on human data, they learn from us. On average, they should score the same as us.”

Humans score on diversity

Interestingly, the results between humans living on different continents and between different AI platforms were consistent. But deeper investigations revealed marked differences between people and machines. Humans displayed huge variation in their scores, ranging from around 35 to 95, while AI scores were within a much more limited range of 70 to 85.

“This is a quite powerful finding because it shows that humans have great diversity, but AI does not. We also found that humans can generate many more unique words than AI,” Professor Wang said. In a robustness check, the professors randomly drew 1,000 words from human and AI responses and found people used an average 577 unique words, while AI only used 181 unique words.

There was also much greater diversity of word choice between people, whereas AI models tended to produce similar results, such as variations of ‘cloud-apple-computer’, because they are all trained on the same pool of words.

“One of the messages we want to convey is that human diversity is what separates us from machines. We should not over-rely on AI to help us, or we could lose our uniqueness,” Professor Shen said.

Nonetheless, while AI does not have the ability to innovate and create new knowledge – at least not yet – it still has a useful role to play in supporting human creativity, they said. It performs particularly well in summarising knowledge.

This points to the importance of human-AI collaboration, although the development of this is still in its early days. One idea is for humans to generate initial ideas, then give AI tasks to improve on them. Users would need strong soft skills, such as leadership, team management, communication, and storytelling, which Professor Shen believes points a path forward for future educators.

Seeking hope

Professor Shen and Professor Wang have embarked on that path through an Executive Master of Business Administration course on AI and creativity. Professor Wang developed the platform Beat the Bot that lets students test themselves on divergent thinking. Students each input 10 words, then select an AI to compare scores. Subsequently, they work in teams and collaborate with AI to test out words and improve the divergency of their list. This has led to overall higher scores and engaged discussion around the class, he said.

A teaching package based around Beat the Bot has been developed that includes discussion questions and reflection points and has been used with students in other programmes and with others outside HKU.

The teaching simulation recently won HKU’s Teaching Innovation Award. Going forward, the professors want to test AI and humans on convergent thinking – the ability to find common ground between different things.

“The goal of Beat the Bot is to give people hope and make them confident of their own human abilities,” Professor Wang said. “We believe the correct way forward is for humans and AI to work together.”

Professor Shen cautioned, though, that while those who perform below average could improve with the right AI tool, those already performing well should not rely too heavily on AI because that may diminish their uniqueness. He also expressed concern about the importance of making AI tools accessible. “What happens to people who do not have the resources to access AI tools? What can they do to survive? This is a much bigger societal problem,” he said.