FAIR

Federated and Independent Repositories

Sali LLMs.txt Generator

Plugin Banner

Sali LLMs.txt Generator

by Ghasali Muhammad Elba

Download
Description

Sali LLMs.txt Generator is a lightweight, high-performance plugin that automatically compiles and serves structured markdown index directories for LLM agents, scrapers, and web crawlers.

By providing a standardized /llms.txt directory index at your root domain, search crawlers can easily list, parse, and consume your content. The plugin also builds /llms-full.txt which concatenates full page markdown for accelerated ingestion.

Unlike generic text generators, Sali LLMs.txt Generator features native deep integration with WooCommerce. It reads product specifications and converts catalog information (SKU, price, stock status, categories, and descriptions) into clean, standard-compliant markdown for seamless AI catalog indexing.

Engineered for Performance & Compatibility

This plugin is built from the ground up with a focus on e-commerce catalog complexity, resource efficiency, and compatibility, ensuring it uses minimal server resources:

  • Transient Database Caching: Directory and content pages are cached for 12 hours using WordPress Transients. When an AI crawler requests the directory, the cached markdown is served instantly without querying the database, resulting in near-zero CPU and RAM overhead.
  • Smart Cache Purging: Caches are automatically flushed whenever you create, edit, trash, or delete posts, pages, or WooCommerce products, keeping your indexes updated without manual work.
  • Size Protection Limits: Caps query counts per post type to prevent memory exhaustion on larger websites.
  • Conflict-Free Architecture: All classes, functions, options, and hooks use isolated namespace prefixes (like sali_llms_), ensuring complete compatibility with core WordPress features, SEO plugins, and third-party tools.

Key Features

  • WooCommerce Catalog Integration: Translates products, prices, SKUs, stock status, and descriptions into clean markdown.
  • Dynamic Search Retrieval API: Real-time search query endpoint (/llms-search.txt?q=...) to allow conversational AI agents to search your catalog and posts instantly.
  • Selective AI Bot Blocker: A configuration grid to block specific crawlers with a 403 Forbidden header and inject custom Disallow directives into the dynamic robots.txt output.
  • AI Crawler Analytics Tracker: An admin dashboard tracking cumulative and daily crawling logs of major AI agents.
  • Server-Agnostic File Generation: A single-click “Generate llms” button that writes physical files to the site root using standard WordPress Filesystem APIs, resolving Nginx 404 issues without server edits.
  • Robots.txt Auto-Integration: Automatically appends the directory link to your site’s dynamic robots.txt file for native AI discovery.
  • Dynamic Custom Post Types: Scans all registered public post types (Posts, Pages, WooCommerce Products, Portfolios, etc.) dynamically for index inclusion.
  • Granular Exclusions: Exclude specific items by Post IDs or entire post/product categories by comma-separated slugs.
  • Custom AI Instructions: Prepend a system prompt to guide crawlers on how to consume your site’s content.
  1. Upload the sali-llms-txt-generator folder to the /wp-content/plugins/ directory, or search for “Sali LLMs.txt Generator” in your WordPress plugins panel and install.
  2. Activate the plugin through the ‘Plugins’ menu in WordPress.
  3. Access configuration and documentation at the Sali LLMs menu in your WordPress dashboard.
  4. Visit your settings tab and click Generate llms to write physical files, or visit yourdomain.com/llms.txt to view your dynamic directory index.
How do I configure which content is indexed?

You can select which Custom Post Types to include in the settings dashboard. You can also exclude specific IDs or category slugs.

How does it protect server CPU and RAM?

All generation queries are cached for 12 hours using WordPress Transients. This serves crawls instantly from memory/cache without reloading the database or bloating server RAM. The cache is automatically cleared when content changes.

Why am I getting a 404 error when visiting /llms.txt?

On Apache/LiteSpeed servers, navigate to Settings > Permalinks and click Save Changes to flush rules. On Nginx servers, simply go to our plugin settings tab and click Generate llms to write physical files directly to the root folder, allowing Nginx to serve them statically without configuration changes.

What is the AI Bot Blocker?

It allows you to selectively restrict specific AI crawlers (like GPTBot, ClaudeBot, Perplexity, etc.) from indexing your directory. When blocked, the crawlers receive a 403 Forbidden header, and the plugin automatically injects Disallow directives for those bots into your site’s robots.txt.

What is the AI Search API?

The /llms-search.txt?q=your-query endpoint allows Custom GPT Actions or other conversational bots to search your WooCommerce catalog and articles in real-time, returning matches in clean markdown format.

What does the Analytics Dashboard show?

It logs cumulative hits, last active times, and daily crawler requests (for the last 30 days) of major AI crawlers, helping you monitor which engines are indexing your website.

1.0.0

  • Initial release of Sali LLMs.txt Generator.
  • Implemented standard routing for /llms.txt, /llms-full.txt, and /llms-search.txt.
  • Implemented WooCommerce product metadata compilation (Price, SKU, Stock, Categories).
  • Created admin settings dashboard with interactive tabbed layout.
  • Added “Generate llms” manual file writer to bypass Nginx 404 issues out-of-the-box.
  • Added AI Bot Blocker, AI Crawler Analytics Tracker, and dynamic robots.txt injection.
  • Added 12-hour transient caching and automatic purge hooks on post/product modifications.
Back to top