Crawler information
MarvedoBot
MarvedoBot is Marvedo’s SEO analysis crawler. It reads a site only when the site’s owner adds it to Marvedo.
User agent
Requests use this user agent:
Mozilla/5.0 (compatible; MarvedoBot/1.0; +https://marvedo.com/bot)robots.txt
MarvedoBot obeys robots.txt rules written for the MarvedoBot and Marvedo-SiteAudit tokens. For example, to block the whole site:
User-agent: MarvedoBot
Disallow: /Crawl rate
It is slow by default: about one request every 2 seconds. It honours Crawl-delay in robots.txt and backs off when the server answers 429 or 503.
IP address
Requests come from the IP address 34.14.118.103 (Google Cloud, europe-west1).
If the site owner’s hosting firewall blocks this address, on paid plans the pages can also be read through a page-rendering service (Firecrawl) so the analysis can finish. robots.txt rules are obeyed on this path too; requests come from that service’s addresses and are limited to at most 40 pages per site in the first analysis.
Stored data
MarvedoBot does not store the content (page bodies) of the pages it reads.
Contact
Questions and blocking requests: hello@marvedo.com