PufferSights – AI Crawler Insights

PufferSights monitors 135+ known AI crawlers and agents, hashes IP addresses, groups traffic by bot, provider, crawl purpose, content type, and response status, tracks human referrals from AI surfaces, and can publish a dynamic llms.txt content map for public site content.

The dashboard summarizes:

• HTTP traffic by bot.
• Crawl purpose.
• Content type.
• Response status.
• AI referrals and crawl-to-refer ratio.
• Top crawled content.
• Tracked agent count.
• Dynamic llms.txt content map.
• robots.txt audit and policy snippets.

The crawler registry is based on current public operator documentation and industry references for OpenAI, Anthropic, Perplexity, Google, Apple, Common Crawl, Meta, ByteDance, Microsoft, Amazon, and related AI crawler operators.

The plugin does not contact any external service. All analytics data is stored in your own WordPress database.

robots.txt publishing is off by default. The plugin can generate and optionally publish policies for:

• Monitor only.
• Block training crawlers.
• Allow AI search/user-action bots while blocking training crawlers.
• Block all known AI bots.

llms.txt publishing is on by default and can be disabled in the PufferSights settings. The generated /llms.txt file lists selected published public pages and posts in Markdown so AI assistants can find the site’s main public content more easily.

Important Notes

Crawler detection is not bot verification. User agents and signature headers can be spoofed. Known browser agents are recognized by documented signing identities together with Signature and Signature-Input headers. Both a single quoted identity and a single dictionary entry are supported. The plugin does not authenticate signatures or fetch public keys. Header matches are labeled as unverified; unsigned requests and custom user agents may be unrecognized. Raw IP addresses are not stored; the plugin stores a salted hash for rough uniqueness.

robots.txt is voluntary. Use a WAF, CDN, or server-level controls when technical enforcement is required.

Google-Extended and Applebot-Extended are robots.txt control tokens, so they appear in robots.txt audits and policy snippets but are excluded from traffic detection and the tracked-agent count. Blocking Google-Extended restricts both Gemini training and grounding, but does not affect Google Search inclusion. The training policy includes this restriction. The former search-friendly training choice is retained as a compatible saved value but is no longer a separate setting.

ChatGPT Cloud browser has no documented robots.txt token. PufferSights does not generate a rule for it or treat a wildcard robots.txt rule as a supported control for that identity. Use server or CDN controls to restrict its access.

The registry was reviewed on October 3, 2026 and includes 135 trackable identities plus two robots.txt-only controls. Additions include Kimi’s training, search, and user-action crawlers; Kernel Search and Kernel Agent; Meta Muse; Grok Agent; RyeBot; and other documented AI crawlers and browser agents. OnirocoCrawler retains ChathiveCrawler as a legacy alias. Cloudflare AI Search External has separate attribution; older requests remain under Cloudflare AI Search. Known Amazon AgentCore regional signing identities share one entry. ADP research is not classified as AI model training. Documentation and identification limits are available in the dashboard Help tab.

Reflectionbot, CloudflareBrowserRenderingCrawler, and Cloudflare-AI-Search are recognized. Amazonbot is included in training restrictions; Amzn-SearchBot and Amzn-User remain separate. Googlebot, Applebot, and Bingbot are labeled Training & Search, while other ambiguous crawlers retain Mixed Purpose. Google-GeminiNotebook includes Google-NotebookLM as a legacy alias. Exa uses ExaSearchBot; the unconfirmed ExaBot match has been removed. Older ExaBot records retain their original data and may contain inaccurate attribution. Historical charts group by stable crawler IDs and show current names. Legacy Anthropic and Omgili identifiers are labeled in the dashboard Help tab and bot summaries.

Bing currently uses a page-level NOARCHIVE directive for AI training opt-out. This plugin does not add it. NOARCHIVE can also exclude content from Bing AI answers; an existing NOCACHE directive can change its effect. Bing robots.txt training opt-out support is planned for early 2027 and must not be assumed to work today.

The settings offer Monitor only, Request no AI training, and Restrict AI crawlers. The latter targets known AI-specific controls while preserving ordinary search crawlers. When enabled, Content Signals also publish a general ai-train=no preference without adding global Allow or Disallow restrictions. Content Signals are an extension, and crawler support varies. Experimental Content Signals use values and the IETF Content-Usage drafts are not enabled automatically.

AI referrals count recorded requests, not unique visitors or verified humans. Ordinary Bing search and OpenAI corporate-site referrals are excluded from AI referral reports, including older records; the stored data is preserved. The crawl-to-referral ratio compares HTML crawler requests with HTML AI referral requests. It is an aggregate across tracked crawlers, including shared search crawlers, and is not a per-platform attribution metric.

Only requests that execute WordPress can be captured. Static files, requests answered entirely by a page cache or CDN, and requests blocked before WordPress loads are outside these analytics. The plugin does not record every visitor to estimate an AI share of all site traffic.

llms.txt is a content map, not an access-control policy. It does not replace robots.txt and does not force AI systems to use or cite your content.

Privacy

PufferSights stores local analytics for public, logged-out requests only. It does not track wp-admin pages, logged-in users, AJAX requests, or WP-Cron requests.

The plugin stores:

• Request time and date.
• Event type, such as AI crawler request or AI referral.
• HTTP method.
• Request path without query string.
• HTTP response status.
• MIME/content group.
• Matched crawler or AI referral provider.
• User-agent string and user-agent hash.
• Salted one-way hash of the request IP address.
• Referrer origin only, such as https://chatgpt.com, without referrer path or query string.

The plugin does not store raw IP addresses, cookies, browser local storage, or complete referrer URLs. It does not send analytics, telemetry, crawler records, or site data to third-party services.

If llms.txt publishing is enabled, the plugin serves a Markdown overview of selected published public posts and pages at /llms.txt. Drafts, private posts, and password-protected posts are not included.

Administrators can disable tracking, clear captured events, and configure retention from the PufferSights admin page. The default retention period is 90 days. On uninstall, the plugin removes its custom analytics table, saved options, and scheduled cleanup hook.

The plugin also adds suggested disclosure text to WordPress’ Privacy Policy Guide.

1. Upload the puffersights-ai-crawler-insights folder to wp-content/plugins.
2. Activate PufferSights – AI Crawler Insights.
3. Open the PufferSights menu in wp-admin.

1. 

   Dashboard summary cards and HTTP traffic by bot.

2. 

   Crawl purpose and AI referral summary metrics.

3. 

   Request trend heatmap by bot across recent days.

4. 

   Content type breakdown for crawler requests.

5. 

   Response status breakdown for crawler requests.

6. 

   AI referral providers and visit counts.

7. 

   Top crawled content with request and bot counts.

8. 

   Recent event log showing crawler and referral activity.

9. 

   Settings screen for tracking, retention, robots.txt policy, and llms.txt.

0.1.3

• Expand the registry to 135 trackable identities, with documented Kimi, Kernel, Meta Muse, Grok, Rye, and other crawler and browser-agent additions.
• Extend unverified signing-identity recognition, group known AgentCore regions, retain Oniroco’s Chathive alias, and separate Cloudflare AI Search External attribution.
• Refresh operator references and purpose notes without treating ADP research or advertiser landing-page checks as AI training.
• Add Reflectionbot and Cloudflare crawler and AI Search identities; include Amazonbot in training restrictions.
• Distinguish Training & Search crawlers while preserving historical event records.
• Correct the HTML crawl-to-referral ratio and exclude ambiguous search and corporate-site referrals from AI reports.
• Simplify training policy choices, clarify Bing and enforcement limitations, and extend optional no-training Content Signals to allowed crawlers.
• Bring crawler and content reports into a responsive Overview grid, with daily requests by purpose and period comparisons.
• Combine robots.txt and llms.txt controls with tracking and data tools on a cleaner Settings page.
• Use native WordPress tabs and a compact date range dropdown, with stable tab switching and accessible keyboard focus.
• Preserve the selected tab and date range when filtering reports or saving settings.
• Fix recent-event date filtering and calculate report percentages using all events in the selected period.
• Keep tracking and policy settings independent when saving either section.
• Hide period comparisons when retention does not cover the full previous period.
• Correct robots.txt reachability updates and safely refresh older audit caches.

0.1.2

• Protect ordinary search crawlers and preserve existing site restrictions when publishing AI crawler policies.
• Correct robots.txt group parsing, audit results, and public policy publication status.
• Refresh robots.txt checks without delaying the dashboard; cache results briefly and retain HTTPS certificate verification.
• Use the origin-root robots.txt URL on sites installed in a subdirectory.

0.1.1

• Add MistralAI-Training, bedrockbot, amazon-kendra, amazon-QBusiness, Mozilla-Tabstack, AIWebIndex, AIWebIndex-Agent, and Webzio.
• Update Google Gemini Notebook and Exa identifiers and robots.txt rules.
• Recognize ChatGPT Cloud browser signature-header claims without implying verification; remove its unsupported robots.txt rule.
• Clarify Google-Extended’s effect on Gemini grounding and label legacy crawler identifiers.
• Keep renamed crawlers together in historical charts and exclude robots.txt-only controls from the tracked-agent count.
• Prevent Applebot-Extended policy tokens from being counted as Applebot traffic.

0.1.0

Initial release.