AgentReadyBot
AgentReadyBot is the crawler behind AgentReady. It fetches a small, fixed set of public URLs on the domain being scanned, then analyzes the response without executing JavaScript, submitting forms, or storing page content.
User-Agent
AgentReadyBot/1.0 (+https://agenticai.vin/bot)
What it requests
| URL | Why |
|---|---|
| / (homepage) | Primary HTML used for all structural, actionability, and trust checks. |
| /robots.txt | Read to honor crawl rules and detect whether AI crawlers are blocked. |
| /llms.txt and /llms-full.txt | Read to score the Agent Instructions category. |
| /sitemap.xml | A HEAD request to confirm it exists. |
| http://your-domain/ (HEAD) | A single HEAD request to verify HTTP redirects to HTTPS. |
Allowing AgentReadyBot
AgentReadyBot already respects robots.txt. To explicitly allow it, no changes are needed — but if your robots.txt blocks all user agents from your homepage, add:
User-agent: AgentReadyBot Allow: /
Blocking AgentReadyBot
Add this to your robots.txt. Scans of your domain will stop immediately and the report will explain why.
User-agent: AgentReadyBot Disallow: /
Behavior
- Maximum 5 pages fetched per scan, capped at 2 MB per response and 8 seconds per request.
- No JavaScript execution, no form submissions, no cookies, no writes of any kind.
- Scans are rate limited and cached for 24 hours per domain to minimize repeat fetches.
- Private, loopback, and internal network addresses are never requested.