crawlcheck.dev

# CrawlCheck

> Free toolkit for Generative Engine Optimization (GEO). Audit your robots.txt against 61 AI crawlers, generate an AI-friendly policy in seconds, and scan your site live as the crawlers that power ChatGPT, Perplexity and Claude. The core principle: a page that is never crawled is a page that is never cited.

CrawlCheck helps website owners control what AI crawlers can see, and helps AI answer engines find, fetch, extract and cite their pages. All tools are free, run client-side where possible, and require no signup. The crawler database is built from the open [ai.robots.txt](https://github.com/ai-robots-txt/ai.robots.txt) community list and vendor documentation.

## Tools

- [AI Crawler Auditor](https://crawlcheck.dev/): Paste your robots.txt and get a per-crawler verdict for 61 AI crawlers — who is allowed, who is blocked, what it costs you. Includes a robots.txt Generator and an FAQ/HowTo/Article JSON-LD Schema Generator.
- [AI Visibility Scanner](https://crawlcheck.dev/scanner/): Live-fetch your site as 8 major AI crawlers (GPTBot, ClaudeBot, PerplexityBot and more). Real requests, per-bot robots.txt verdicts, cloaking check, one 0-100 visibility score. 3 free scans a day.

## Directory

- [AI Crawler Directory](https://crawlcheck.dev/crawlers/): All 61 known AI crawlers profiled — operator, category (training / search / assistant / agent), whether each respects robots.txt, and copy-paste block/allow rules.

## Guides

- [What is Generative Engine Optimization (GEO)?](https://crawlcheck.dev/guides/what-is-geo/): GEO is the practice of making your site readable, extractable and citable by AI answer engines. What it covers, why it matters, and the first five steps to take.
- [GEO vs SEO: What Changes and What Doesn't](https://crawlcheck.dev/guides/geo-vs-seo/): SEO is not dead, but AI answer engines added a second set of rules. A practical comparison of goals, crawl access, content format and how to measure success.
- [How AI Crawlers Work (And What They Want From Your Site)](https://crawlcheck.dev/guides/how-ai-crawlers-work/): AI crawlers train models, build search indexes, or fetch pages on request. Meet the major bots, learn how they fetch pages, and why access is step zero.
- [robots.txt for AI Crawlers: Every Rule That Matters](https://crawlcheck.dev/guides/robots-txt-for-ai-crawlers/): A 30-second robots.txt refresher, what each AI bot costs you when blocked, three copy-paste policies and the syntax mistakes that silently break your rules.
- [llms.txt: The Proposed Standard for AI-Friendly Sites](https://crawlcheck.dev/guides/llms-txt/): llms.txt is a proposed markdown file at your site root that helps AI models find your best pages. Full format spec, a copy-paste example, and the tradeoffs.
- [How to Measure AI Visibility: Metrics That Actually Matter](https://crawlcheck.dev/guides/measure-ai-visibility/): AI visibility has no industry-standard score yet. Track citation share, referral traffic, bot access rate and answer accuracy instead — here is how to do it.
- [How to Get Cited by ChatGPT, Perplexity and Copilot](https://crawlcheck.dev/guides/optimize-for-chatgpt/): ChatGPT, Perplexity and Copilot cite pages through a fetch-retrieve-extract pipeline. What each stage rewards, how the engines differ, and a 10-point checklist.
- [Content Structure for AI: Pages That Get Extracted and Cited](https://crawlcheck.dev/guides/content-structure-for-ai/): AI answer engines extract pages in chunks, not wholes. Structure rules that make content quotable: question headings, answer-first paragraphs, tables, FAQs.
- [Structured Data for AI: The Schema.org Markup Answer Engines Read](https://crawlcheck.dev/guides/schema-markup-for-ai/): JSON-LD explained in 30 seconds, the schema types answer engines actually use, a copy-paste FAQPage example, and how to validate your markup for free.
- [10 GEO Mistakes That Keep Your Site Out of AI Answers](https://crawlcheck.dev/guides/geo-mistakes/): Ten fixable mistakes that keep sites out of AI answers: blanket crawler bans, JS-only content, prose walls, stale pages, schema spam and more. Each with a fix.

## Policies

- [About](https://crawlcheck.dev/about/): What CrawlCheck is, where the crawler data comes from.
- [Contact](https://crawlcheck.dev/contact/): Corrections, bug reports, privacy requests.
- [Privacy](https://crawlcheck.dev/privacy/): Client-side tools, no cookies, no tracking.
- [Terms](https://crawlcheck.dev/terms/)
- [Refund Policy](https://crawlcheck.dev/refund-policy/)