Knowledge and RAG
SentientOne's website crawler
For website owners: what SentientOne-Crawler is, what it reads, how fast, and how to allow or block it with robots.txt.
Reference2 min readUpdated 4 November 2026
SentientOne-Crawler reads public websites for SentientOne customers: a customer adds a website to an agent's knowledge, and the crawler fetches its pages so the agent can answer from them. It crawls only when a customer asks (adding or re-crawling a website), never on its own schedule, and it reads only what any visitor could.
How to recognise it
SentientOne-Crawler/1.0 (+https://www.sentientone.ai/help/knowledge/website-crawler)Its requests come from SentientOne's API on Google Cloud. The product token for robots.txt is SentientOne-Crawler.
What it reads
/robots.txtfirst, then the sitemaps it lists (or/sitemap.xml), then the start page and the links on it, as deep as the customer chose (up to 10 links) and at most 200 pages a crawl.- Only the site the customer entered (
www.and the bare domain count as one), and only below the address they gave. - Web pages and plain-text files, without running JavaScript. It skips PDFs, images and other files, and never submits forms or signs in.
- Each page's main text, for the customer's agent. Pages that need a sign-in, and anything robots.txt asks it to leave alone, are not read.
How fast
- At most 4 requests at a time to a site, and one crawl of at most 4 minutes.
- If your robots.txt sets a
Crawl-delay, it reads one page at a time with that gap, up to 5 seconds (a longer delay counts as 5). - A
429 Too Many Requestsor503with a shortRetry-Afteris waited out once; otherwise that page is left. - Each response is read up to 5 MB, and each request is given up after 20 seconds.
Allow or block it
The crawler follows robots.txt as RFC 9309 describes it: the rules in a group for SentientOne-Crawler, or, if there's none, the rules for *. Allow and Disallow take * and $, and the longest matching rule wins. A robots.txt that redirects is followed, up to 5 redirects and to another host too; more than 5 counts as no robots.txt. Crawl-delay is honoured up to 5 seconds: a longer delay counts as 5.
User-agent: SentientOne-CrawlerDisallow: /User-agent: SentientOne-CrawlerDisallow: /account/Crawl-delay: 2If robots.txt can't be read because the site answers with a server error, the crawler doesn't crawl the site at all; a missing robots.txt (a 404) means no rules.
Questions
Email support@sentientone.ai with your site's address and roughly when you saw the requests.
Still stuck?
Email us with what you tried and any request id. A person replies.