Relying on large language models (LLMs) to summarize information may diminish knowledge acquisition, according to a recent study involving over 10,000 participants.
Marketing professors Jin Ho Yun and Shiri Melumad co-authored a paper detailing this finding across seven studies. Participants were tasked with learning a topic, such as vegetable gardening, through either an LLM like ChatGPT or a standard Google search. Researchers placed no restrictions on tool usage duration or interaction for participants.
Participants subsequently wrote advice for a friend based on their learned information. Data consistently showed those who used LLMs for learning perceived they learned less and invested less effort in advice creation. Their advice was shorter, less factual, and more generic.
An independent sample of readers found the LLM-derived advice less informative, less helpful, and were less likely to adopt it. These differences persisted across various contexts.
One experiment controlled for potential variations in information eclecticism by exposing participants to identical facts from both Google and ChatGPT searches. Another experiment held the search platform constant—Google—while varying whether participants learned from standard Google results or Google’s AI Overview feature. Even with facts and platform standardized, learning from synthesized LLM responses resulted in shallower knowledge compared to gathering, interpreting, and synthesizing information via standard web links.
The study attributes this diminished learning to reduced active engagement. Google searches involve more “friction,” requiring navigation, reading, interpretation, and synthesis of various web links, which fosters deeper mental representation. LLMs perform this process for the user, shifting learning from active to passive.
Researchers do not advocate for avoiding LLMs given their benefits in other contexts. Instead, they suggest users become more strategic by understanding where LLMs are beneficial or harmful to their goals. For quick, factual answers, LLMs are suitable. However, for developing deep, generalizable knowledge, relying solely on LLM syntheses is less effective.
Further experimentation involved a specialized GPT model providing real-time web links alongside synthesized responses. Participants receiving an LLM summary were not motivated to explore original sources, leading to shallower knowledge compared to those using standard Google. Future research will explore generative AI tools that introduce “healthy frictions” to encourage active learning beyond easily synthesized answers, particularly in secondary education.
