FAIR

Federated and Independent Repositories

LLMs.txt – AI Website Content Crawling

Plugin Banner

LLMs.txt – AI Website Content Crawling

by Magazine3

Download
Description

LLMs.txt – AI Website Content Crawling helps you generate a custom llms.txt file to control how AI language models (LLMs) and search engines crawl your site content.

Key Features

  • Automatically generate a valid llms.txt file.
  • Choose which post types are included for AI access.
  • Compatible with SEO and privacy-conscious setups.
  • Easy to use with minimal configuration.

This plugin is ideal for publishers, bloggers, and businesses looking to control their content visibility to AI systems and bots.

  1. Upload the plugin folder to the /wp-content/plugins/ directory.
  2. Activate the plugin through the ‘Plugins’ menu in WordPress.
  3. Go to Settings > LLMs.txt to configure which post types to include.
  4. Visit https://yoursite.com/llms.txt to see the output.
What is `llms.txt`?

It is a proposed standard for signaling content access preferences to large language models and AI crawlers, similar to robots.txt.

Does this replace `robots.txt`?

No. llms.txt complements robots.txt and focuses specifically on AI crawling policies.

Can I control which content types are included?

Yes, the plugin allows you to select which post types (e.g., posts, pages, custom types) should be listed.

1.0

  • Initial release.
Back to top