A recent study by the BBC has raised alarms over the accuracy of four major AI chatbots—OpenAI’s ChatGPT, Microsoft’s Copilot, Google’s Gemini, and Perplexity AI—when tasked with summarizing news content. The BBC provided these AI systems with content from its own website and asked them to generate summaries, only to find that many of their responses contained "significant inaccuracies" and distortions.
Deborah Turness, CEO of BBC News and Current Affairs, acknowledged that while AI offers "endless opportunities," the potential for harm is substantial. She warned that AI-generated headlines, if distorted, could cause significant real-world consequences. "How long will it be before an AI-distorted headline causes significant real-world harm?" Turness questioned in her blog post.
The study involved asking the AI chatbots to summarize 100 news stories, which were then evaluated by BBC journalists with expertise in the subject matter. Shockingly, 51% of the AI-generated answers contained serious issues, with 19% of the responses that cited BBC content introducing factual errors such as incorrect numbers, dates, and statements. Examples of inaccuracies included:
- Gemini falsely stating that the NHS does not recommend vaping as a smoking cessation aid.
- ChatGPT and Copilot incorrectly claiming that former leaders Rishi Sunak and Nicola Sturgeon were still in office despite having left their positions.
- Perplexity AI misquoting BBC News in an article about the Middle East, stating that Iran had initially shown "restraint" and calling Israel's actions "aggressive."
While ChatGPT and Perplexity AI were found to have fewer issues compared to Copilot and Gemini, the errors were still significant. The BBC also noted that these chatbots struggled with distinguishing between factual reporting and opinion, often editorializing and omitting essential context.
The BBC’s Programme Director for Generative AI, Pete Archer, emphasized that publishers should have control over how their content is used by AI companies. He called for more transparency from AI developers, urging them to disclose how their systems process news and the extent of errors in the summaries they generate.
In response to the study, an OpenAI spokesperson stated that the company is working to improve the accuracy of in-line citations and respects publisher preferences through tools like OAI-SearchBot. The company also committed to further enhancing search results.
As the conversation about AI's role in news curation continues, it’s clear that more collaboration between AI developers and publishers will be necessary to ensure that the public receives accurate, reliable, and well-contextualized news.
0 Comments