MCP tool
robots
Fetches and parses a site's robots.txt into user-agent groups (allow/disallow rules), crawl-delay and sitemap URLs.
The robots tool is provided by Mat Web & Data Toolkit (x402, Cloudflare). Install that server in your AI client and the assistant can call it directly.
Servers exposing robots
From the publisher:Fetches and parses a site's robots.txt into user-agent groups (allow/disallow rules), crawl-delay and sitemap URLs. Costs 0.01 USDC on Base via x402, or 12 sats Cashu ecash (NUT-24), or 12 sats over Lightning (L402).
Install Mat Web & Data Toolkit (x402, Cloudflare) →Parameter Type Description url*string
Tool lists are read from the live servers (or, for packaged servers, from their documentation) each morning, so this reflects what they exposed at the last refresh. Search all tools.