Crawler documentation

KinyaBot

We operate KinyaBot, the Kinya web crawler. What follows is its complete operational record: what it requests, how it fetches, how it backs off, the dedicated egress address that identifies it, and how to verify its traffic.

Identity record

User-Agent
KinyaBot/1.0 (+https://www.kinya.ai/bot)
robots.txt token
KinyaBot
Dedicated egress
2.28.52.200/32
Operated by
Kinya
Legal name
Rhino Rank LTD
Company number
10849738
Registered address
5 Knights Court Archers Way, Battlefield Enterprise Park, Shrewsbury, Shropshire, England, SY1 3GA
Jurisdiction
United Kingdom
Contact
[email protected]

What KinyaBot does

KinyaBot requests a page for one of two reasons. They are not the same thing: the first has a customer behind it and the second does not.

  1. Analysis for a Kinya customer

    Our customer owns the site, or has asked us to analyse it. KinyaBot collects the page source and technical SEO evidence we need to set that site up, then revisits it so the analysis stays current as the site changes.

    Published as customer-site-analysis in the machine-readable identity record.

  2. Testing the crawler itself

    We run the crawler against real public pages to check its own quality. These runs are bounded, have no customer behind them, and are limited to public paths that the applicable robots.txt rules do not block for KinyaBot or for crawlers generally.

    Published as crawler-quality-testing in the machine-readable identity record.

Do not read either case as “every KinyaBot crawl is authorised by the site owner”. Crawler-quality testing has no customer behind it. It runs only on public paths that the site’s own robots.txt leaves open to KinyaBot or to crawlers generally, and a Disallow ends it in exactly the same way it ends the other case.

Where a provider’s taxonomy has an SEO or search-optimisation category, that is the category KinyaBot belongs in.

How KinyaBot fetches a page

Most of the time KinyaBot makes a plain HTTP request and reads the HTML your server returns. That is the default, and for most pages it is all that happens.

Some pages do not put their content in that first response. Where KinyaBot’s own evidence for a page suggests the HTML it received is incomplete, it fetches that page again in a real browser with JavaScript enabled, compares the two results, and keeps whichever one actually contains the page. That decision is made per route pattern on your site, from observed evidence — never applied blanket-fashion across a whole site — so once KinyaBot has established that a section of your site renders over plain HTTP, it stops using the browser for it.

  • Both paths send the identical User-Agent, KinyaBot/1.0 (+https://www.kinya.ai/bot). There is no second token, no version suffix for the browser path, and no separate string for test runs.
  • KinyaBot never presents itself as Chrome, as any other browser, or as another operator’s crawler.
  • The browser sandbox is enabled, and is never disabled.

In short: HTTP first, a real browser only where the HTTP response is demonstrably incomplete, and the identical User-Agent on both paths.

What it requests, and what it does not do

What it requests

  • The page source your server returns, and the rendered source on the routes it renders.
  • The response status and the response headers.
  • Your robots.txt.
  • Your sitemaps and sitemap index files.
  • Public files at the root of the site.

What it does not do

  • It does not log in, and does not enter an authenticated area.
  • It does not submit forms.
  • It does not start a checkout or any other transaction.
  • It does not request a path your robots.txt disallows for it.
  • It does not solve a CAPTCHA, bypass a challenge, or use rotating or residential proxies.

KinyaBot does not target or harvest personal data, and nothing it does is aimed at collecting it. It does read the page your server returns to any visitor — so if a public page you publish contains personal data, that data is in the page source KinyaBot reads, exactly as it is for every other reader of that page. If you do not want a page read, disallow it in robots.txt and KinyaBot will not request it.

Behaviour and controls

  • robots.txt first. KinyaBot fetches robots.txt before anything else on an origin and applies RFC 9309: the KinyaBot token is matched case-insensitively, groups that match it are merged, the User-agent: * group applies when none does, the longest matching path rule wins, an Allow wins an otherwise equal tie, and /robots.txt itself is always allowed.
  • Crawl-delay is ignored. KinyaBot follows the standardized robots path rules above. RFC 9309 does not define Crawl-delay, so KinyaBot treats that nonstandard extension like an unknown directive.
  • Retry-After is honoured. On a 429 or a 503, in both delta-seconds and HTTP-date form. The identical request is never retried sooner than the header asks.
  • Requests are bounded. A crawl runs to a bounded number of requests, under per-origin concurrency and request-rate ceilings that are also aggregated globally across all of our customers.
  • A challenge is terminal. If your site challenges KinyaBot, KinyaBot stops and reports the challenge. It does not solve CAPTCHAs, does not bypass or evade a challenge, does not rotate residential or datacentre proxies, and does not attempt a different route into a blocked origin.
  • A redirect off the origin is not followed blindly. KinyaBot does not continue onto an origin it was not authorised or robots-permitted to crawl.

Control it from robots.txt

Add one of these groups to the robots.txt at the root of your domain. Each is a complete, valid group and can be pasted as it stands. A change applies from the next time KinyaBot reads that file.

  • Allow KinyaBot everywhere

    # Allow KinyaBot everywhere
User-agent: KinyaBot
Allow: /

  • Block KinyaBot entirely

    # Block KinyaBot entirely
User-agent: KinyaBot
Disallow: /

  • Keep KinyaBot out of one area

    # Allow KinyaBot, except these paths
User-agent: KinyaBot
Disallow: /cart/
Disallow: /search
Allow: /

If your robots.txt has no KinyaBot group, KinyaBot applies the rules under User-agent: *. Site owners often assume the opposite. For controls in a CDN, WAF, host or plugin console instead, see the allow-list instructions.

Verifying that traffic is really KinyaBot

A User-Agent is a claim, not proof. Anything on the internet can send the string KinyaBot/1.0 (+https://www.kinya.ai/bot), and a share of reports about a crawler turn out to be somebody else using its name. If you are deciding whether to allow traffic — rather than merely to label or rate-limit it — a User-Agent match is not enough on its own.

This is KinyaBot’s complete current dedicated egress list: 2.28.52.200/32. Customers and providers may match that exact /32 in addition to the exact User-Agent. Publishing the address proves origin by network address; it does not authorize us to crawl your site. Robots, rate bounds and customer authorization remain controlling. Address changes will be published here and on https://www.kinya.ai/bot/identity.json before new production traffic uses them.

The current limits of verification:

  • No reverse-DNS verification. KinyaBot publishes no crawler DNS suffix and no PTR record, so a forward-confirmed reverse-DNS lookup cannot identify KinyaBot.
  • No request signing. KinyaBot does not sign requests and publishes no public key directory, so there is no cryptographic way to verify KinyaBot generally. A Shopify merchant may privately configure Shopify-issued signature headers for requests to that exact storefront only.
  • No verified-bot listing. KinyaBot is not listed, registered or approved by any CDN, host or security-vendor bot programme.

If we add reverse-DNS verification, request signing or a verified-bot listing, we will publish it here and on the canonical identity page before it appears in your traffic.

Contact, and stopping a crawl

Email us at [email protected] to ask for KinyaBot to be allowed, blocked, or kept out of part of a site you run — or to report a crawl that caused you a problem.

What is useful in the message:

  • The hostname affected.
  • Roughly when the requests arrived, and the time zone.
  • The matching log lines, if you have them.
  • What you want to happen: KinyaBot allowed, blocked, or limited to part of the site.

Please do not send credentials, private content, or raw logs containing personal data. The matching lines are enough.

You do not have to wait for a reply to stop a crawl. This group in your robots.txt stops KinyaBot from the next time it reads that file, and needs nothing from us:

User-agent: KinyaBot
Disallow: /

What KinyaBot collects

KinyaBot stores the page source it fetched, the rendered source where a page was rendered, the response status and headers, and the robots.txt, sitemaps and public root files it read. That is the material the technical SEO and content analysis is produced from.

Its requests carry nothing that identifies which customer, if any, asked for the crawl. Where a customer authorised a crawl, we do not name them without their consent.

Identity change log

Every change to KinyaBot’s published identity — the User-Agent, any published addresses, any signing keys, the declared purposes — is recorded here, and is published before it appears in your logs. A version number is never reused.

  1. 21 August 2026

    The identity page and the machine-readable identity record were first published, carrying the User-Agent in the identity record at the top of this page. No egress addresses, reverse DNS, request signatures or verified-bot listings existed.

  2. 24 August 2026

    KinyaBot’s complete current dedicated egress list was published as the single exact CIDR 2.28.52.200/32, in the machine-readable identity record and on the human identity pages. Reverse DNS, request signatures and verified-bot listings do not exist. Address changes are recorded here before new production traffic uses them.