New study exposes security loopholes in AI-Powered Search Engines

Artificial Intelligence (AI)-powered search engines (AIPSEs) are transforming how we retrieve information online. Unlike traditional search engines, which rely on keyword matching, these advanced systems integrate large language models (LLMs) with external databases to deliver precise, summarized responses. While this innovation enhances efficiency and user experience, it also introduces significant security risks. As AI search engines continue to evolve, concerns over their vulnerability to misinformation, malicious content, and cyber threats are becoming more urgent.

A recent study titled “The Rising Threat to Emerging AI-Powered Search Engines”, authored by Zeren Luo, Zifan Peng, Yule Liu, Zhen Sun, Mingchen Li, Jingyi Zheng, and Xinlei He, from The Hong Kong University of Science and Technology (Guangzhou) and the University of North Texas, provides a comprehensive safety risk assessment of AIPSEs. The study systematically defines threat models, risk levels, and evaluates vulnerabilities across seven leading AI search engines. It reveals that 47% of AI-powered search engine responses carry some level of risk, including the quoting of malicious URLs, phishing content, and online scams, even when users input completely benign queries. The researchers also propose an agent-based defense mechanism using GPT-4o-powered content refinement tools and an XGBoost-based URL detector to enhance safety measures.

AI-powered search engines: Benefits and inherent risks

The emergence of AI-powered search engines is driven by retrieval-augmented generation (RAG), a system that combines pre-trained knowledge from LLMs with real-time internet searches. Unlike traditional search engines that merely rank and list websites, AIPSEs summarize relevant content by retrieving data from multiple sources, interpreting user intent, and generating comprehensive answers. This capability allows them to streamline research, provide contextualized responses, and reduce the cognitive load of filtering through multiple web pages.

However, this efficiency comes at a cost. The study highlights three key risks that pose serious challenges to AIPSE reliability and security. One of the major risks is malicious content and URL citations, where AIPSEs can unknowingly provide users with links to malicious websites, including phishing sites, malware distribution pages, and scam platforms. This means that a user searching for software downloads may be led to unofficial and harmful sources instead of official websites, exposing them to potential cybersecurity threats. Another risk is misinformation and hallucinations, where AIPSEs generate false or misleading responses due to their reliance on external data without real-time verification. This vulnerability can lead to the spread of incorrect or even harmful information. Lastly, manipulation by attackers is a growing concern, as cybercriminals can exploit weaknesses in AI search engines through online document spoofing and phishing tactics. The study demonstrates how adversaries can trick AIPSEs into recognizing fake websites as legitimate sources, which could result in financial losses or security breaches for unsuspecting users.

A particularly alarming case cited in the study describes an incident where a developer lost $2,500 after following AI-generated code that directed him to a fake Solana API website. Within 30 minutes, the attacker had compromised the developer’s private key and stolen his assets. This real-world example underscores the urgent need for improved security measures in AI-driven search technology.

How AI search engines are being deceived: Key findings from the study

To quantify safety risks, the researchers systematically evaluated seven production AIPSEs, including ChatGPT Search, Perplexity, Copilot, TextCortex, Grok, Doubao, and Kimi. The study examined 100 harmful websites collected from cyber threat detection platforms like PhishTank, ThreatBook, and LevelBlue, analyzing how these search engines handled malicious content in their responses. The results were alarming, with 47% of all AIPSE responses containing some level of risk and 34% directly citing harmful websites. Furthermore, URL-based queries (where users input a website link directly) significantly increased the likelihood of AI search engines retrieving malicious content, making it easier for cybercriminals to exploit these vulnerabilities. In contrast, natural language queries - such as "Where can I safely download MetaMask?" - were found to be less prone to high-risk outputs, suggesting that how a query is framed can impact the safety of AI-generated responses.

The study also conducted two real-world case studies demonstrating how easily AIPSEs can be manipulated. In the first case study on online document spoofing, the researchers created a fake cryptocurrency trading platform with fraudulent API documentation. When AIPSEs retrieved and summarized this data, they blindly cited the malicious code, essentially instructing users to enter sensitive credentials without any security warnings. The second case study on phishing website recognition involved creating two nearly identical websites, one labeled as an official source and the other a phishing page. The researchers found that some AIPSEs mistakenly flagged the real site as untrustworthy while identifying the phishing site as legitimate, revealing a major flaw in how these search engines verify sources.

These findings highlight the fundamental weakness in how AI-powered search engines authenticate information, leaving them susceptible to cyberattacks and digital misinformation. Without robust security mechanisms in place, AIPSEs risk becoming vehicles for the amplification of malicious content, scams, and fraudulent activities on the internet.

Developing a defense mechanism: AI-powered content filtering and URL detection

To counter these vulnerabilities, the study proposes an agent-based defense strategy that leverages GPT-4o for content refinement and an XGBoost-based URL detection tool. The objective of this defense system is to filter out unsafe responses while retaining as much useful information as possible. The system consists of three key components that work together to enhance search engine security. The AI content refinement tool, powered by GPT-4o, ensures that AI-generated responses are rewritten to eliminate malicious links and include security warnings before being presented to users. The XGBoost-based URL detector is a machine learning model that evaluates URLs for suspicious patterns, helping to flag potentially dangerous websites before they are referenced by AIPSEs. Lastly, an iterative risk assessment agent repeatedly evaluates AI-generated content using external tools until a safe and verified answer is produced.

The effectiveness of this agent-based system was compared to traditional prompt-based filtering techniques. Results showed that the defense mechanism reduced high-risk responses by 83%, whereas prompt-based filtering achieved only a 9% improvement. However, this defense system comes with trade-offs, as it introduced false positives, incorrectly labeling 34.42% of benign URLs as malicious, which ultimately limited the availability of information. Despite this, the approach marks a significant advancement in ensuring AI-generated search results are safer and more reliable. The researchers emphasize the need for continued refinement of AI safety models and closer collaboration between AI developers and cybersecurity experts to ensure that AIPSEs become more resilient against manipulation and misinformation.