FocusWeb AI Crawler Monitor – Track, Verify & Block AI Bots

by upi

Description

FocusWeb AI Crawler Monitor gives you the full loop for dealing with AI bots on your WordPress site: see which AI crawlers read your content, verify that each visit really comes from the vendor it claims (spoofed bot User-Agents are rampant), decide per bot what happens next — monitor, disallow via robots.txt, block, or block only the fakes — and measure how much real referral traffic AI answer engines (ChatGPT, Perplexity, and more) send you in return. Think of it as AI SEO / answer engine optimization (AEO / GEO) visibility and control for site owners.

Why verification matters

Anyone can send a request with “GPTBot” or “ClaudeBot” in the User-Agent header — and scrapers do exactly that to hide behind reputable names. Independent audits have found that on some sites, effectively all “ClaudeBot” traffic came from non-vendor IP addresses. This plugin checks every claimed bot visit against the vendor’s published IP ranges (refreshed daily) or documented reverse-DNS rules (processed hourly in the background, never in your visitors’ request path), and labels each visit Verified, Spoofed, or Unverifiable. Your analytics become trustworthy, and blocking becomes safe.

Headline feature: Block only fakes

The unique action no comparable plugin offers cleanly: per bot, refuse (403) only the visits that claim its User-Agent but fail verification. Impostor scrapers hiding behind “GPTBot” get blocked; the real GPTBot stays welcome. You protect your server and your content without giving up AI visibility.

Quick start:

Install and activate the plugin — no configuration required, logging and verification start immediately.
Wait for real traffic (or test with a curl request spoofing a bot User-Agent — it will show up flagged as Spoofed).
Check AI Crawlers > Dashboard for crawler visits with verification badges, and AI Crawlers > AI Referrals for human click-throughs from AI answer engines.
When you are ready to act, open AI Crawlers > Crawl Control and choose per bot: Monitor, Disallow (robots.txt), Block (403), or Block only fakes. Everything defaults to Monitor — nothing changes until you opt in.
Optionally generate a suggested llms.txt file from AI Crawlers > llms.txt Generator.

Features:

Verified Bot Detection: every logged visit is checked against vendor-published IP ranges (OpenAI, Perplexity) and reverse-DNS rules (Google, Apple, Amazon, ByteDance), with Verified / Spoofed / Unverifiable badges, summary cards, filters, and CSV export. Range data refreshes daily; reverse-DNS runs in an hourly background batch — zero added latency for visitors.
Crawl Control: per-bot actions with plain-language explanations and one-click presets (“Disallow all training crawlers”, “Block fakes everywhere”). Bots are categorized as AI training crawlers, AI search crawlers, or on-demand fetchers, so the referral-traffic tradeoff of each decision is clear — including hints computed from your own AI referral data.
Block only fakes: 403 spoofed impostors while the genuine crawler keeps its access.
robots.txt manager: Disallow rules served through the WordPress virtual robots.txt (robots_txt filter — no filesystem writes), with a live preview of the final merged file and conflict detection for physical robots.txt files and SEO plugins (Yoast SEO, Rank Math, All in One SEO).
robots.txt compliance flag: a disallowed bot that keeps visiting is labeled “Ignores robots.txt” — your log data becomes the evidence for escalating to a hard block.
Safety rails: everything defaults to Monitor; updates never change behavior until you opt in; blocking never applies to logged-in users; block responses send no-cache headers so page caches and CDNs never cache a 403 for humans.
AI Referral Traffic tracking (AEO / answer-engine-optimization visibility): detects human click-throughs from AI answer engines (ChatGPT, Perplexity, Microsoft Copilot, Google Gemini, Claude, Meta AI, You.com) via the Referer header or matching utm_source parameters, captures the source query when the platform exposes one, and shows a crawl-to-referral conversion insight alongside a per-day-per-source chart.
Dashboard with summary cards (including verified share, spoofed visits, and blocked requests), a Chart.js time-series chart, a most-crawled URL table, and a paginated raw log with CSV export.
Lightweight init-hook based detection, no server log file access required. Custom database tables with indexes for fast queries.
Configurable log retention with a daily cleanup cron job, optional IP anonymization (GDPR-friendly, enabled by default), and excludable URL path patterns.
Built-in llms.txt generator: produces suggested content from your site title, tagline, pages, and posts for manual placement at your site root (see FAQ for llms.txt’s actual adoption status).
Developer-friendly: extend the bot registry, verification rules, and block decisions via the aicm_known_bots, aicm_verification_config, and aicm_block_decision filters.
Fully functional on shared/managed hosting — no filesystem writes outside the database, no special server permissions needed.

How to use this data to optimize for AI:

A high number of crawler visits with few AI referrals is normal today — most AI crawling feeds training or indexing, not real-time answers. The Dashboard’s most-crawled-URL table shows which pages get the most attention; keep those accurate, current, and well-structured.
The AI Referrals page’s Crawl-to-Referral Gap panel puts a number on that relationship so you can track whether it improves.
Before blocking an AI search crawler, check the Crawl Control page’s per-bot referral hints — they show how many visitors that platform actually sent you in the last 30 days.
Check the Captured Queries table for the (best-effort) questions that led people to your site, and favor clear headings, concise direct answers, and FAQ-style content.

Roadmap (planned direction, not a commitment to specific versions or dates):

Correlating individual crawler visits with later referral visits to the same page.
Per-post/per-page llms.txt summaries, instead of one sitewide file.
Additional AI referral sources and bots as new platforms emerge, and additional vendor verification methods as they are published.
A combined “AI visibility” trend view comparing crawl volume and referral traffic over longer ranges.

Suggestions are welcome via the support forum.

Installation

Upload the focusweb-ai-crawler-monitor folder to /wp-content/plugins/.
Activate the plugin through the “Plugins” menu in WordPress.
On activation, the plugin creates its log tables and schedules its cron events (daily log cleanup, daily vendor IP-range refresh, hourly reverse-DNS verification batch).
Go to AI Crawlers > Dashboard for logged crawler visits with verification badges, AI Crawlers > Crawl Control to set per-bot actions and manage robots.txt rules, AI Crawlers > AI Referrals for human click-throughs from AI answer engines, AI Crawlers > Settings for tracked bots/sources, verification, retention, and privacy options, and AI Crawlers > llms.txt Generator to produce a suggested llms.txt file.

Screenshots

Crawl Control -- per-bot Monitor / Disallow (robots.txt) / Block (403) / Block only fakes actions, category presets, verification methods, referral-tradeoff hints, and the merged robots.txt preview with SEO-plugin conflict warnings.
Dashboard -- summary cards including verified traffic share, spoofed visits, and blocked requests, a 30-day per-bot visit chart, most-crawled URLs, and a paginated raw log with Verified/Spoofed badges and CSV export.
AI Referrals -- human click-throughs from AI answer engines like ChatGPT and Perplexity, a crawl-to-referral conversion insight, a per-day-per-source chart, and captured queries where the platform exposes one.
Settings -- tracked bots, verified bot detection toggle, tracked AI referral sources, log retention, IP anonymization, and excluded URL paths.

Faq

Can this plugin block AI crawlers?

Yes — per bot, and only when you opt in. On the Crawl Control page each bot can be set to Monitor (default, log only), Disallow (adds a robots.txt rule), Block (403 every matching request), or Block only fakes (403 only visits that claim the bot’s User-Agent but fail verification). Updating the plugin never changes behavior: every bot stays on Monitor until you choose otherwise.

How does bot verification work?

Where the vendor publishes official IP ranges (OpenAI for GPTBot and ChatGPT-User, Perplexity for PerplexityBot), each visit’s IP is checked against those ranges — instantly, using a locally cached copy refreshed daily. Where the vendor documents reverse-DNS rules instead (Google, Apple, Amazon, ByteDance), visits are logged as Pending and checked in an hourly background batch (reverse DNS plus forward confirmation), so DNS lookups never slow down your visitors. Bots whose vendors publish no method (e.g. Anthropic’s ClaudeBot, Common Crawl’s CCBot as of today) are honestly labeled Unverifiable rather than guessed at.

What exactly does “Block only fakes” do?

It blocks requests that pretend to be a known bot but fail verification — for example a scraper sending “GPTBot” from a random datacenter IP — while the genuine crawler from the vendor’s verified infrastructure is untouched. It never adds a robots.txt rule, because the real bot remains welcome. For bots with no published verification method it never blocks anything (there is no reliable way to tell real from fake, so the plugin refuses to guess).

Will blocking hurt my traffic from ChatGPT or Perplexity?

It can, depending on which bots you block. Blocking AI training crawlers (GPTBot, CCBot, Bytespider…) stops future model training with little effect on current traffic. Blocking AI search crawlers (PerplexityBot, and OpenAI’s search crawling) may reduce your citations in AI answers and the human referral visits they generate. The Crawl Control page shows, per bot, how many referral visits the related platform sent you in the last 30 days, so you can decide with your own numbers.

Will it block Google or hurt my normal SEO?

No. The registry targets AI-specific crawlers only. Google-Extended is Google’s AI-training opt-out token — controlling it does not affect Googlebot, ordinary crawling, indexing, or rankings. As an extra safety rail, blocking never applies to logged-in users.

Does it conflict with Yoast SEO, Rank Math, or All in One SEO robots.txt features?

The plugin adds its rules through the standard WordPress robots_txt filter (the virtual robots.txt), alongside rules from other plugins. The Crawl Control page shows a live preview of the final merged robots.txt and warns you about the cases that need attention — in particular a physical robots.txt file at your site root, which overrides the virtual one entirely.

Does it need access to my server’s raw access logs?

No. Detection happens via a WordPress init hook reading the User-Agent header on each request, so it works identically on shared and managed hosting.

Will it log every request from these bots?

The plugin skips wp-admin, wp-login.php, admin-ajax.php, REST API requests, and static asset requests (images, CSS, JS, fonts, etc.) to keep the log focused on actual content visits. You can add further exclusions (e.g. /checkout/*) on the Settings page.

Does it store personal data?

It stores the requesting IP address and User-Agent string for each logged visit. IP anonymization (masking the last octet/segment) is enabled by default. Verification checks the real IP at request time, but with anonymization enabled only the anonymized address is stored in the log; addresses queued for background reverse-DNS checks are held temporarily and deleted as soon as the check completes. You can also configure automatic log deletion after a set number of days.

Does the plugin slow my site down?

No. Range verification is pure in-memory math against a cached list. Reverse-DNS lookups and vendor range downloads only ever run in background cron jobs, never while a visitor (or bot) is waiting for a response.

Can I add more bots to track, or change verification/blocking behavior programmatically?

Yes. Three filters are provided: aicm_known_bots (add bots with a label, User-Agent pattern, and category), aicm_verification_config (add or override per-bot verification methods — IP-range URL or rDNS suffixes), and aicm_block_decision (final say over any block, receiving the bot key, IP, and verification verdict).

Does the llms.txt generator write files to my server?

No. It only generates suggested content in a textarea, with a button to download it as llms.txt. You place the file at your site root manually, avoiding filesystem permission issues on managed hosting.

Is llms.txt an officially supported standard?

Not yet. llms.txt is a community-proposed convention (introduced in 2024) suggesting a simple Markdown file at your site root to help AI systems find your key content. As of today, no major AI crawler or assistant (OpenAI, Anthropic, Google, Perplexity, etc.) has publicly confirmed that it reads or prioritizes llms.txt in production. Publishing one is a reasonable, low-effort bet on where things may be heading — not a guaranteed way to influence how AI systems treat your site. This plugin only generates suggested content for you to review and place manually; it never assumes llms.txt is authoritative or required.

Reviews

Excellent plugin

By roev on July 9, 2026

It helped me a lot. Its nice that such a high-quality plugin is free

Changelog

2.0.0

NEW — Verified Bot Detection: every logged crawler visit is checked against vendor-published IP ranges (OpenAI, Perplexity; refreshed daily) or documented reverse-DNS rules (Google, Apple, Amazon, ByteDance; processed in an hourly background batch) and labeled Verified, Spoofed, or Unverifiable. Verdict badges, a verification filter, verified-share and spoofed-visits summary cards, and a Verification column in the CSV export. DNS lookups never run in the visitor request path.
NEW — Crawl Control page: choose per bot between Monitor (default), Disallow via robots.txt, Block (403), and Block only fakes. Includes plain-language explanations of every action, bot categories (AI training / AI search / on-demand fetcher), one-click presets, and per-bot referral-tradeoff hints computed from your own AI Referrals data.
NEW — “Block only fakes”: 403 visits that claim a bot’s User-Agent but fail verification, while the genuine crawler keeps its access. Never blocks when no vendor verification method exists, and never blocks logged-in users.
NEW — robots.txt manager: Disallow rules served via the WordPress virtual robots.txt (no filesystem writes), live preview of the final merged robots.txt, and conflict detection for physical robots.txt files and SEO plugins (Yoast SEO, Rank Math, All in One SEO).
NEW — robots.txt compliance flag: bots that keep visiting more than 48 hours after being disallowed are labeled “Ignores robots.txt” on the Crawl Control page.
NEW — Blocked-requests tracking: blocked visits are logged with a 403 status (never silently dropped) and surfaced in a Blocked summary card and a Status column in the raw log.
NEW — developer filters: aicm_known_bots, aicm_verification_config, and aicm_block_decision.
Database: the log table gains an indexed verification column; existing installs are migrated automatically on update, no action needed.
Removed the Settings-page auto-update toggle (added in 1.4.0) to keep the plugin fully clear of WordPress update routines per Plugin Check guidance. Use the native “Enable auto-updates” link on the Plugins screen instead — any preference already set there is untouched.
Settings saves now merge with existing options, and the display name gained “Track, Verify & Block AI Bots”. Slug, settings, and data are unchanged — updates are seamless, and behavior stays monitoring-only until you opt in on the Crawl Control page.

1.4.0

Added an opt-in auto-update toggle to the Settings page. It reads and writes WordPress core’s own auto_update_plugins list directly, so it always stays in sync with the native “Enable auto-updates” link on the Plugins screen — off by default, same as core.
Added a “Crawl-to-Referral Gap” insight and a per-day-per-source chart to the AI Referrals page, showing how much AI crawling actually converts into real referral traffic.
Expanded the readme with guidance on acting on this data, an honest note on llms.txt’s actual (unconfirmed) adoption status by AI platforms, and a roadmap section.

1.2.4

Directory listing optimization: swapped the oversaturated seo tag for analytics, added a Quick Start section, and worked in AEO (answer engine optimization) and “track” terminology to better match how site owners search for this plugin on WordPress.org. No functional changes.

1.2.3

Added plugin directory assets: icon, banner, and screenshots for the WordPress.org listing. No functional changes.

1.2.2

Plugin Check follow-up: annotated remaining PluginCheck.Security.DirectDB.UnescapedDBParameter warnings in class-logger.php with justified ignore comments (all queries use fixed internal table-name identifiers passed through $wpdb->prepare() for any real parameters — no functional change).

1.2.1

Fixes from WordPress.org plugin review: updated bundled Chart.js from 4.4.4 to the latest stable 4.5.1, and replaced the raw inline script block on the llms.txt Generator page with a properly enqueued wp_enqueue_script() file (assets/llms-generator.js).

1.2.0

Added AI Referral Traffic tracking: detects human click-throughs from AI answer engines (ChatGPT, Perplexity, Microsoft Copilot, Google Gemini, Claude, Meta AI, You.com) via the Referer header or matching utm_source parameters, with a best-effort captured query where the referring platform exposes one (most reliably on Perplexity). Adds a new “AI Referrals” dashboard (summary cards, captured-queries table, paginated raw log, CSV export) and per-source tracking toggles on the Settings page. Existing sites will get the new database table automatically on next page load after updating.

1.1.0

Renamed to “FocusWeb AI Crawler Monitor” and re-slugged to focusweb-ai-crawler-monitor to comply with WordPress.org’s plugin naming policy (unique/brand-prefixed name required for directory submission). No functional changes.

1.0.1

Compliance fixes for the WordPress.org Plugin Check tool: removed placeholder Plugin URI/Domain Path headers, dropped the discouraged manual load_plugin_textdomain() call, justified/annotated direct-DB and nonce-verification warnings, prefixed template-local variables, and updated readme metadata (tags, tested-up-to, short description).

1.0.0

Initial release: bot detection and logging, admin dashboard with summary cards and Chart.js visualization, most-crawled URL and raw log tables with CSV export, settings page (tracked bots, retention, IP anonymization, excluded paths), and llms.txt generator.