1. Permission: Does your site allow it?
We read your robots.txt, meta robots tags and X-Robots-Tag headers for each major AI crawler, and show exactly which rule affects which bot.
Can AI actually read your website?
Your robots.txt might say yes while your firewall says no. Gainwide checks whether ChatGPT, Claude, Perplexity and other AI crawlers can reach your pages and read your content.
Example
- Permissionrobots.txt allows GPTBot, ClaudeBot, PerplexityBotPass
- AccessClaudeBot receives 403 from a firewall ruleBlocked
- ReadabilityMain content appears only after JavaScript runsPartial
Robots.txt is only half the story
Most AI crawler checkers stop at robots.txt. But plenty of sites that allow AI crawlers on paper still block them in practice – through a CDN rule, a firewall setting or a bot challenge page.
And even when a crawler gets in, it may find an empty page. Most AI crawlers don’t run JavaScript, so content that only appears after scripts load is invisible to them.
Three checks, one clear answer
2. Access: Can the crawler actually get in?
We request your pages the way AI crawlers do and flag 403 errors, challenge pages and timeouts that robots.txt can’t show you.
3. Readability: Can it read what it finds?
We compare your raw HTML with the fully rendered page. If your main content only appears after JavaScript runs, most AI crawlers will never see it.
We test using each crawler’s user agent. Some firewalls also verify a crawler’s IP address, which no outside tool can fully reproduce. When that may affect your result, we’ll say so.
Built for people who want to be found
- Site owners and indie makers
- Online store owners on Shopify, WooCommerce or custom platforms
- SEO and marketing teams
- Developers shipping JavaScript-heavy sites
Frequently asked questions
What is an AI crawler?
An automated program that AI companies use to fetch web pages – either to train their models or to answer people’s questions with current information. Examples include OpenAI’s GPTBot and OAI-SearchBot, Anthropic’s ClaudeBot and Perplexity’s PerplexityBot.
Why does it matter whether AI can read my site?
More and more people ask ChatGPT, Claude or Perplexity instead of searching. If these tools can’t read your pages, they can’t mention or link to them.
Isn’t checking robots.txt enough?
No. Robots.txt only says what you allow. Your CDN, firewall or bot protection can still block a crawler, and JavaScript-rendered content can still be invisible. Gainwide checks all three.
Should I allow every AI crawler?
Not necessarily. Crawlers that collect training data and crawlers that power AI search work differently. Many site owners block the first and allow the second. Gainwide shows you what each bot can do, so you can decide.
When will Gainwide launch?
We’re building it now. Join the waitlist and we’ll email you as soon as it’s ready.
Will it be free?
Yes, a basic check will be free. We may add paid features later for monitoring and full-site scans.