There sure are a lot of frivolous uses for AI these days, but one use that remains grounded in need (and real results) is the original ChatGPT functionality that started it all. Yes, the simple talk-and-respond chatbot remains the best public use case for AI in general, owed in part to arguments about public education and access to information.
Well, now it seems that even that might be overblown. A new paper has collected results from seven experiments with over 10,000 total participants and has come to the conclusion that learning with a chatbot leads to lower recall and shallower learning when compared to regular web search.
Now, it might seem odd to use regular web search as our “good” side of the coin here, since academia has spent decades vilifying it as the path to a paltry understanding. But it turns out that the proactive nature of web search at least prompts the learner to keep their mind engaged in the process of evaluating potential sources.
The study showed a clear pattern: Learning through a chatbot not only led to less retained information, but a lessened ability to explain that information after reading. Those given explanations written by chatbot-learners and search-learners consistently chose the search-born answers as more informative and helpful.
The researchers don’t spend much time speculating as to what it is about chatbots that seems ill-suited to education, but it’s not hard to figure out.
Chatbots provide an easy answer to almost any question, curated and requiring nothing more than rote memorization. In comparison, more traditional forms of learning, from searching the web to searching a library card index, involve sifting and organizing your thoughts, by necessity.
Credit: Picture Alliance/Getty Images
To surface a story from the entire internet, you have to have an ordered understanding of exactly what info you’re looking for and where that info might be. With ChatGPT, you just need to name the thing you want to know, then get down to the business of copying or memorizing.
Leave aside how untrustworthy these chatbots can be; even when they’re telling the truth, they can stultify the mind by removing the process that surfaces knowledge. Not only does this impact your mindset throughout, but it also forces you to read more than the minimum amount: If you’re searching for 10 pieces of information, you’ll probably need to read 50-100 pieces to find the 10 you’re looking for, where if you ask a chatbot for these 10 in a list, you’ll get those 10 and no others.
I actually think that chatbots deliver far too much information for many queries. A potential solution to the education problem would be to have chatbots deliver much shorter, more direct responses with more diverse suggestions for follow-up questions. ChatGPT’s “Study Mode” works a bit like this, but the user must activate it manually.
Counter-intuitively, making chatbot responses less sophisticated might be the key to making them better at making us smarter.