Microsoft Copilot, powered by the advanced GPT-4 model, combines the prowess of generative AI and the Bing search engine to offer users information. This chatbot compiles search results and provides summarized responses, promising a wealth of knowledge at one’s fingertips.
Unmasking the Truth: Assessing Bing Chat’s Reliability in Election Information
In a comprehensive study, AlgorithmWatch and AI Forensics embarked on an extensive investigation into the accuracy of election-related information provided by Bing Chat. Over a span of three months, from August to October 2023, researchers systematically posed queries to Bing Chat regarding the October 2023 federal elections in Switzerland and state elections in the German states of Hesse and Bavaria. These queries encompassed topics that would naturally pique the interest of voters, such as voting procedures, candidates, and the latest polling data.
The research findings sent shockwaves through the digital landscape. Alarmingly, nearly 30% of Bing Chat’s responses contained factual inaccuracies. These errors ranged from false polling data to outdated candidate information and even incorrect election dates. More disturbingly, the chatbot was found to fabricate controversies surrounding candidates, a systemic issue that transcended time, borders, and languages, with prompts conducted in English, German, and French.
Riccardo Angius, Applied Math Lead and Researcher at AI Forensics, emphasizes the gravity of the situation, stating, “It’s time we discredit referring to these mistakes as ‘hallucinations.’ Our research exposes the much more intricate and structural occurrence of misleading factual errors in general-purpose LLMs and chatbots.”
Eroding Trust: Generative AI’s Impact on Institutions
The proliferation of false and fabricated information poses a grave threat to the credibility of news outlets and candidates. For instance, Bing Chat erroneously attributed incorrect polling figures to reputable news sources, tarnishing their reputation in the process. Additionally, the chatbot’s concocted narratives, seemingly supported by credible sources, create an illusion of authenticity, potentially sowing further distrust in the realm of news media. Worse yet, the dissemination of incorrect information by Bing Chat can influence voters’ perceptions and opinions about candidates.
Microsoft’s Bing Chat emerges as an unreliable source of information during critical election periods. It not only spreads misinformation but also contaminates the information ecosystem by misrepresenting trustworthy sources and crafting false narratives.
Microsoft’s Inadequate Response
In response to these concerns, Microsoft announced measures on its blog to safeguard information integrity during elections. However, these measures fall short of addressing the core issues. Microsoft pledged to furnish voters with “authoritative election information” through Bing, but the chatbot continues to misquote reliable sources in its responses. Furthermore, Microsoft vowed to assist candidates and campaigns in managing their narratives, but Bing Chat itself remains a purveyor of falsehoods.
Salvatore Romano, Senior Researcher at AI Forensics, underscores the broader implications, stating, “Our research shows that malicious actors are not the only source of misinformation; general-purpose chatbots can be just as threatening to the information ecosystem. Microsoft should acknowledge this and recognize that flagging generative AI content made by others is not enough. Their tools, even when implicating trustworthy sources, produce incorrect information at scale.”
A Cycle of Deception: Bing Chat’s Persisting Issues
Despite Microsoft Deutschland’s acknowledgment of the need for accurate election information, initial reports in October 2023 revealed inaccuracies. While Microsoft claimed to have improved Bing Chat by basing responses on top search results, a subsequent evaluation by AI Forensics and Algorithm Watch unveiled little progress. Bing Chat continued to fabricate stories about candidate controversies and provide erroneous information about Swiss candidates and their respective cantons.
Microsoft Copilot’s Bing Chat, while a promising endeavor, grapples with profound issues of misinformation dissemination, potentially impacting elections and public perception. Microsoft must address these challenges to restore trust in its information ecosystem.