Home / S2OResearchBotWeb crawler information

S2OResearchBot

S2OResearchBot is the web research agent operated by S2O. It reads a small number of public web pages published by regulators, government portals and companies, so that we can prepare business research about financial-services organisations.

User-Agent
Mozilla/5.0 (compatible; S2OResearchBot/1.0; +https://www.s2osales.com/bot)
robots.txt token
S2OResearchBot

What it reads

  • Public notices, circulars, orders and press releases from regulators and government bodies.
  • Public tender and procurement notices.
  • Disclosures that companies publish for investors and customers, such as credit ratings, stock-exchange intimations, annual reports and grievance-redressal information.

It records short factual notes with a link back to the original page, for example that a notice was published on a given date. It does not republish documents.

What it never does

  • It never logs in, creates accounts, submits forms or solves CAPTCHAs.
  • It always identifies itself with the User-Agent above. It never presents itself as an ordinary web browser, and never rotates addresses or identities to get around a block.
  • It does not collect personal contact details. It records a person's name only where an organisation publishes it for an official role.

How it behaves on your site

  • It follows robots.txt as defined in RFC 9309, using the token S2OResearchBot. Where no group names it, the rules for * apply.
  • It follows explicit prohibitions of automated access in a site's terms of use.
  • It makes one request at a time per site, at least 5 seconds apart (10 seconds for government sites), and honours Crawl-delay and Retry-After.
  • It stops when a site answers 401, 403 or 429, or shows a login or challenge page.

How to limit or block it

To block S2OResearchBot from your whole site, add this to your robots.txt:

User-agent: S2OResearchBot
Disallow: /

To block only part of your site:

User-agent: S2OResearchBot
Disallow: /private-section/

Changes take effect within 24 hours. You can also email ayush@s2osales.com to ask us to stop visiting your site or to slow down. We reply within two business days.

Operated by S2O · s2osales.com · Last updated 17 September 2026