What Is PerplexityBot?
PerplexityBot is the official web crawler operated by Perplexity AI, the company behind one of the fastest-growing AI search engines. It scans and indexes web pages so that Perplexity can answer questions with current, conversational responses that include citations and links back to the original sources.
Unlike training crawlers such as GPTBot or ClaudeBot, PerplexityBot is a search-and-retrieval AI crawler: it gathers content to surface and cite in live answers, not to train a foundation model. It identifies itself with the user agent string "PerplexityBot" in server logs.
How PerplexityBot Works
- Search, not training: it retrieves and cites pages for Perplexity's answer engine rather than teaching a large language model.
- On-demand crawling: alongside a scheduled crawl, it is heavily triggered by live user queries, so it tends to fetch the most up-to-date version of a page.
- Two agents: "PerplexityBot" handles indexing, while "Perplexity-User" is the on-demand agent that fetches a page in response to a specific user request.
- robots.txt aware: it reads robots.txt directives, and an llms.txt file can point it towards your most authoritative pages.
PerplexityBot vs Training Crawlers
AI crawlers do two broad jobs: surfacing content for live answers, and collecting content to train models. PerplexityBot sits firmly in the first group.
| Crawler | Operated by | Primary purpose |
|---|---|---|
| PerplexityBot | Perplexity AI | Search and cite in live answers |
| GPTBot | OpenAI | Train and improve models |
| ClaudeBot | Anthropic | Train and improve models |
| Google-Extended | Control use in Gemini training |
How to Allow or Block PerplexityBot
You control PerplexityBot through your robots.txt file. For most South African businesses pursuing AI search visibility, allowing it is the right call, because it is the path to being cited in Perplexity answers.
To allow PerplexityBot (recommended for visibility):
User-agent: PerplexityBot Allow: /
To block it:
User-agent: PerplexityBot Disallow: /
Add a matching rule for the on-demand Perplexity-User agent to control live fetches too. Remember that robots.txt is a directive, not an enforced barrier: publishers who want a hard block usually combine it with server-side or firewall rules.
The Publisher Compliance Debate
PerplexityBot has drawn scrutiny from some publishers who reported their content appearing in Perplexity answers even where they had tried to restrict crawling. The compliance behaviour of AI crawlers is an active debate in digital publishing. The practical takeaway for site owners: decide deliberately whether you want AI-search citations, set your robots.txt accordingly, and add server-level controls if you need a firm block.
Why PerplexityBot Matters for South African Businesses
Perplexity's citation model surfaces sources at the moment of the query, so being crawled can place your page in front of users as a cited answer, outside Google's ranking algorithm. A South African legal firm, accountant or healthcare provider that publishes clear, authoritative, factually dense content has a direct route to citations when local or international users ask related questions.
Structuring content as self-contained content chunks with answer-first openings raises the quotability of individual paragraphs, making it easier for Perplexity to extract and cite specific passages. Allowing PerplexityBot, publishing an llms.txt, and writing extraction-friendly content together build AI search visibility as a fast-growing referral channel.
PerplexityBot and business visibility
Allowing PerplexityBot is part of being visible in Perplexity, an AI answer engine that retrieves web pages to ground its responses and cites the sources it uses. Because Perplexity shows citations and links, being among the sources it draws on can send referral traffic as well as build visibility, which is why many businesses choose to allow it. As with other AI crawlers, permitting access does not guarantee citation; whether Perplexity uses your content depends on it being indexed, relevant, clear and trustworthy for the questions users ask. Blocking PerplexityBot in robots.txt opts your content out of that system, a deliberate trade-off some publishers make to protect content. For most businesses seeking AI-mediated discovery, allowing it and making content genuinely quotable is the more useful choice.
Frequently Asked Questions
What is PerplexityBot?
PerplexityBot is the official web crawler of Perplexity AI. It reads and indexes web pages so Perplexity can answer questions with cited, conversational responses. It is a search-and-retrieval crawler, not a model-training crawler, and identifies itself as "PerplexityBot" in server logs.
Does PerplexityBot respect robots.txt?
PerplexityBot reads robots.txt directives, so you can request that it allow or skip parts of your site. However, robots.txt is a directive rather than an enforced block, and some publishers have raised compliance concerns. For a firm block, combine robots.txt with server-side or firewall rules.
How do I block PerplexityBot?
Add a User-agent: PerplexityBot block with Disallow: / to your robots.txt, plus a matching rule for the on-demand Perplexity-User agent. If you need to guarantee the block, add server-level or firewall rules as well, since robots.txt relies on the crawler's compliance.
Should South African businesses allow PerplexityBot?
Generally yes, especially if your website publishes authoritative, factual, or locally specific content. Allowing PerplexityBot increases the chance your pages are cited as sources when South African users search Perplexity AI for information relevant to your industry or location. Blocking it removes the possibility of citation-based traffic from Perplexity's growing user base.