Back to HomeRecomaze

RecomazeBot

Web Crawler for AI Readiness Audits

RecomazeBot is the web crawler used by Recomaze AI Readiness Audit to analyze websites and evaluate their optimization for AI assistants like ChatGPT, Perplexity, and Google AI.

User Agent

RecomazeBot/1.0 (+https://audit.recomaze.ai/bot)

What RecomazeBot Does

  • ✓Analyzes publicly accessible pages for AI optimization factors
  • ✓Checks schema markup, structured data, and metadata
  • ✓Evaluates content quality and AI readability
  • ✓Measures page performance and technical SEO factors
  • ✓Checks robots.txt compliance for AI crawlers

Crawl Behavior

Frequency

On-demand only. We crawl when a user requests an audit of their website.

Pages per visit

Limited to 10-20 pages per audit to minimize server load.

Rate limiting

Respectful delays between requests. We don't overload servers.

Robots.txt

We respect robots.txt directives for RecomazeBot.

What We Don't Do

  • • We don't store or republish your content
  • • We don't access password-protected areas
  • • We don't submit forms or modify anything on your site
  • • We don't crawl continuously or index your entire site
  • • We don't share crawl data with third parties

How to Allow RecomazeBot (If You're Getting Errors)

If your AI readiness audit is failing or showing 0 products/pages, your site might be blocking our crawler with Cloudflare, a firewall, or other bot protection. Here's how to allow access:

Cloudflare Users

  1. Log in to your Cloudflare dashboard
  2. Go to Security → WAF
  3. Click Create rule under "Custom rules"
  4. Name it "Allow RecomazeBot"
  5. Set condition: User Agent contains RecomazeBot
  6. Set action: Skip → Select all security features
  7. Save and deploy

Alternative: Disable "Bot Fight Mode" temporarily in Security → Bots

robots.txt (Explicit Allow)

Add this to your robots.txt file:

User-agent: RecomazeBot
Allow: /
Crawl-delay: 1

Other Firewalls (Sucuri, Wordfence, etc.)

Whitelist our User-Agent string RecomazeBot/1.0 in your firewall or security plugin settings. Look for:

  • Whitelist / Allowlist settings
  • Bot management or User-Agent rules
  • Trusted bots / Verified bots section

Common Issues

  • 403 Forbidden errors: Your firewall is blocking us. Follow the Cloudflare or firewall instructions above.
  • 0 products found: Check that your product pages are public and not password-protected.
  • Audit times out: Your site might be very slow or blocking our crawler. Try whitelisting our bot.

How to Block RecomazeBot

If you don't want RecomazeBot to crawl your website, add this to your robots.txt file:

User-agent: RecomazeBot
Disallow: /

Note: If you block RecomazeBot, users won't be able to run AI readiness audits on your website through our service.

Questions or Concerns?

If you have any questions about RecomazeBot or notice any issues, please contact us at: [email protected]