People do this all the time: if they don’t know an answer, they just make something up that sounds right. It turns out AI has the same bad habit.
A Study Put AI Search to the Test, and It Did Not Go Well
Researchers at the Tow Center for Digital Journalism (part of Columbia Journalism Review) wanted to see how well AI-powered search tools could find real news articles. They tested eight AI search engines, including ChatGPT Search, Google’s Gemini, and Microsoft’s Copilot.
Here’s what they did:
- They picked 200 news articles from well-known publishers.
- They asked the AI tools to find the article’s title, publisher, publication date, and web link.
- They compared the AI’s answers to see how accurate they were.
The results that came back didn’t fill the researchers with a lot of confidence.
AI Search Tools Are Overconfident and Wrong
More than 60 percent of the answers were incorrect, but the AI acted like it was completely sure of itself. Some answers were worse than others. Here are some of the results:
- Perplexity was wrong 37 percent of the time.
- Grok-2 messed up 94 percent of the time.
- Microsoft’s Copilot was the only one that refused to answer when it didn’t know (which was actually better).
Some AI tools even made up fake website links or credited the wrong source. That’s like someone giving you directions to a place that doesn’t exist.
AI Was Worse With Blocked Content
Here’s the weirdest part: when publishers blocked AI crawlers from reading their content, the AI was somehow better at finding the right information than when it had free rein of the underlying information.
Even in cases where publishers had partnerships with AI companies, misattribution was still a problem. Some AI platforms linked to republished versions of articles on different websites, making it harder to track down the real source. Obviously, this is problematic for people using the platform to search for facts.
The Big Takeaway: Just Don’t Trust AI Search
The researchers expected AI to either find the right answer or admit it didn’t know. Instead, it just made things up. This matters because AI search tools are becoming more common, and if they’re unreliable, they could mislead people and harm publishers. The last thing society needs is more misinformation out there from supposedly reputable platforms.
AI can be useful, but when it comes to searching for facts, be skeptical and double-check the answers yourself. If you would like to talk to one of our knowledgeable experts about integrating artificial intelligence into your business, give us a call today at (281) 916-1101.
Comments