According to Cloudflare, Perplexity’s crawling behaviour appears to involve attempts to circumvent network blocks by obscuring its identity.
The company alleges that when Perplexity’s declared user agent is blocked, it modifies its user agent and changes its source ASNs to hide its crawling activity. Furthermore, Cloudflare claims that Perplexity is ignoring or sometimes failing to even retrieve robots.txt files, which are used by websites to control crawler access.
Cloudflare said that the internet is built on trust and that crawlers are expected to be transparent, serve a clear purpose, and follow website directives. Due to what it describes as Perplexity’s incompatible behaviour, Cloudflare has delisted it as a verified bot and implemented new rules to block the stealth crawling activity.
Bot operators best practice
Cloudflare set out clear preferences on how good crawlers should behave. It said that all well-intentioned crawlers acting in good faith, should:
Be transparent. Identify themselves honestly, using a unique user-agent, a declared list of IP ranges or Web Bot Auth integration, and provide contact information if something goes wrong.
Be well-behaved netizens. Don’t flood sites with excessive traffic, scrape sensitive data, or use stealth tactics to try and dodge detection.
Serve a clear purpose. Whether it’s powering a voice assistant, checking product prices, or making a website more accessible, every bot has a reason to be there. The purpose should be clearly and precisely defined and easy for site owners to look up publicly.
Separate bots for separate activities. Perform each activity from a unique bot. This makes it easy for site owners to decide which activities they want to allow. Don’t force site owners to make an all-or-nothing decision.
Follow the rules. That means checking for and respecting website signals like robots.txt, staying within rate limits, and never bypassing security protections.
Cloudflare’s statement said: “All the undeclared crawling activity that Cloudflare observed from Perplexity’s hidden User Agent was scored by the company’s bot management system as a bot and was unable to pass managed challenges. Any bot management customer who has an existing block rule in place is already protected.”
It concluded: “Customers who don’t want to block traffic can set up rules to challenge requests, giving real humans an opportunity to proceed. Customers with existing challenge rules are already protected.”
