LLMS Central - The Robots.txt for AI

โ˜๏ธ Cloudflare Worker Setup Guide

Track AI bots on ANY website - WordPress, Shopify, Wix, static sites, and more. No code changes needed on your actual site!

๐ŸŒ

Universal Solution

Works for ANY platform - WordPress, Shopify, Wix, Squarespace, static sites, custom apps

โšก

10-15 Min Setup

Simple step-by-step process. No coding required. Free Cloudflare account.

๐Ÿ”’

Server-Side Tracking

Catches ALL AI bots (GPTBot, Claude, etc.) that don't execute JavaScript

1

Sign Up for Cloudflare (Free)

  1. 1. Go to: https://dash.cloudflare.com/sign-up
  2. 2. Create a free account
  3. 3. Verify your email

๐Ÿ’ฐ Cost: FREE (Free plan is sufficient for bot tracking)

2

Add Your Website to Cloudflare

  1. 1. Click "Add a Site" in Cloudflare dashboard
  2. 2. Enter your domain (e.g., example.com)
  3. 3. Click "Add site"
  4. 4. Select "Free" plan
  5. 5. Click "Continue"

Cloudflare will scan your DNS records (takes ~1 minute).

3

โš ๏ธ Verify DNS Records (IMPORTANT!)

Cloudflare auto-imports your DNS records, but you MUST verify them before switching nameservers!

  1. 1. After adding your site, Cloudflare shows your DNS Records
  2. 2. Review all records - Cloudflare usually gets 80-90% of them
  3. 3. Compare with your current DNS (check your domain registrar)
  4. 4. Add any missing records manually (click "Add record")
  5. 5. Pay attention to:
    • โ€ข A records (your website IP)
    • โ€ข CNAME records (subdomains, CDN)
    • โ€ข MX records (email - CRITICAL!)
    • โ€ข TXT records (verification, SPF, DKIM)

๐Ÿšจ Missing records = broken functionality! If you skip this step, your email or website might break after switching nameservers.

๐Ÿ’ก Tip: Most registrars (GoDaddy, Namecheap) let you export DNS records. You can then import them into Cloudflare's DNS page to save time. Cloudflare also auto-scans and imports most records when you add your site.

4

Update Your Nameservers

Cloudflare will show you 2 nameservers like:

ava.ns.cloudflare.com
bob.ns.cloudflare.com
  1. 1. Go to your domain registrar (GoDaddy, Namecheap, etc.)
  2. 2. Find "Nameservers" or "DNS Settings"
  3. 3. Replace existing nameservers with Cloudflare's nameservers
  4. 4. Save changes

โฐ Wait 5-60 minutes for DNS to propagate. Cloudflare will email you when it's active.

5

Create the Worker

  1. 1. In your domain's Cloudflare dashboard, click "Workers Routes" (left sidebar)
  2. 2. Click "Manage Workers" button (top right)
  3. 3. Click "Get started" on "Start with Hello World!"
  4. 4. Name your worker: llms-central-bot-tracker
  5. 5. Click "Deploy"

๐Ÿ’ก Tip: The "Hello World" template creates a basic worker that you'll customize in the next step.

6

Add the Tracking Code

  1. 1. Click "Edit Code" button
  2. 2. Delete all existing code in the editor
  3. 3. Copy the code below and paste it
  4. 4. Click "Deploy"
/**
 * LLMS Central AI Bot Tracker - Cloudflare Worker
 * 
 * This worker intercepts all requests to your website and tracks AI bot visits
 * before forwarding the request to your origin server.
 */

// AI bot patterns to detect
const AI_BOT_PATTERNS = [
  'gptbot', 'chatgpt', 'claude', 'anthropic', 'google-extended',
  'googlebot', 'bingbot', 'applebot', 'facebookexternalhit',
  'meta-externalagent', 'perplexitybot', 'youbot', 'bytespider',
  'diffbot', 'grokbot', 'x-bot', 'grok', 'x.ai', 'xai',
  'cohere', 'amazonbot', 'yandexbot', 'baiduspider'
];

const LLMS_CENTRAL_API = 'https://llmscentral.com/api/bot-tracker';

export default {
  async fetch(request, env, ctx) {
    const userAgent = request.headers.get('user-agent') || '';
    const userAgentLower = userAgent.toLowerCase();
    
    const isBot = AI_BOT_PATTERNS.some(pattern => userAgentLower.includes(pattern));
    
    if (isBot) {
      const domain = env.DOMAIN || new URL(request.url).hostname;
      const page = new URL(request.url).pathname;
      const clientIP = request.headers.get('cf-connecting-ip') || 'Unknown';
      
      // Fire and forget - track bot visit
      ctx.waitUntil(
        fetch(LLMS_CENTRAL_API, {
          method: 'POST',
          headers: {
            'Content-Type': 'application/json',
            'User-Agent': userAgent,
            'X-Forwarded-For': clientIP
          },
          body: JSON.stringify({ domain, page })
        }).catch(err => console.error('Tracking error:', err))
      );
    }
    
    return fetch(request);
  }
};
7

Configure Environment Variable

  1. 1. Go to Settings tab
  2. 2. Click "Variables"
  3. 3. Under "Variables and Secrets", click "Add"
  4. 4. Type: Keep as Text
  5. 5. Variable name: DOMAIN
  6. 6. Value: Your domain (e.g., example.com)
  7. 7. Click "Save" or "Add variable"
8

Add Worker Route

  1. 1. Go back to your website in Cloudflare (not Workers section)
  2. 2. Click on your domain
  3. 3. Go to "Workers Routes" (under Workers & Pages)
  4. 4. Click "Add route"
  5. 5. Route: *example.com/* (replace with your domain)
  6. 6. Worker: Select llms-central-bot-tracker
  7. 7. Click "Save"
9

โš ๏ธ Disable "Block AI Bots" (CRITICAL!)

If you don't do this step, the worker won't be able to track AI bots!

  1. 1. In your domain's Cloudflare dashboard, click "Security" (left sidebar)
  2. 2. Click "Settings"
  3. 3. Find "Block AI bots" section
  4. 4. Click the edit icon (pencil) next to "Block AI Bots scope"
  5. 5. Click the "Do Not Block (allow crawlers)" feature
  6. 6. Click "Save"

๐Ÿ’ก Why? Cloudflare's "Block AI bots" feature blocks bots BEFORE the worker can track them. You need to disable it so the worker can intercept and track bot visits first.

โœ… Test Your Installation

Quick Test:

  1. 1. Visit your website normally - it should work exactly as before
  2. 2. Go to: https://llmscentral.com/dashboard?tab=bots
  3. 3. Look for your domain
  4. 4. You should start seeing bot visits within 24 hours

๐Ÿงช One-Click Test

Test your Cloudflare Worker setup instantly - we'll simulate a GPTBot visit to your domain:

Advanced: Command Line Test

For PowerShell (Windows):

Invoke-WebRequest -Uri "https://yourdomain.com" -UserAgent "Mozilla/5.0 (compatible; GPTBot/1.0)"

For Terminal (Mac/Linux):

curl -A "Mozilla/5.0 (compatible; GPTBot/1.0)" https://yourdomain.com

Replace yourdomain.com with your actual domain.

๐Ÿ”„ How It Works

AI Bot visits your site
โ†“
Request hits Cloudflare first
โ†“
Worker detects bot (server-side)
โ†“
Sends tracking data to llmscentral.com
โ†“
Forwards request to your website
โ†“
Bot gets your page (normal experience)

Your website works exactly the same. Bots are tracked invisibly.

๐Ÿค– AI Bots We Track

๐Ÿค–
GPTBot
OpenAI
๐Ÿง 
Claude
Anthropic
๐Ÿ”
Google-Extended
Google AI
๐Ÿ’ก
Perplexity
Perplexity
โœ–๏ธ
Grokbot
X.AI
๐Ÿ”ฎ
Cohere
Cohere
๐Ÿค–
Googlebot
Google Search
๐Ÿฆ…
Bingbot
Bing
๐ŸŽ
Applebot
Apple
๐Ÿ‘ฅ
FacebookBot
Meta
๐ŸŽต
Bytespider
TikTok
๐Ÿ”ฌ
Diffbot
Diffbot

โ“ Frequently Asked Questions

Will this slow down my website?

No! The worker adds less than 1ms of latency. It runs at Cloudflare's edge (closer to your visitors) and tracking is non-blocking (fire-and-forget).

What if Cloudflare or your API goes down?

The worker is designed to fail gracefully. If anything goes wrong, it simply forwards the request to your website normally. Your site will never break because of the tracker.

How much does Cloudflare cost?

The Free plan includes 100,000 requests per day, which is more than enough for bot tracking. You also get free SSL, DDoS protection, and CDN benefits!

Can I use this with WordPress?

Yes! But if you're using WordPress, we recommend our WordPress plugin instead - it's easier to install (just click "Install" in your WordPress admin).

Need Help?

Our support team is here to help you get set up!