FAIR

Federated and Independent Repositories

Bottle

Plugin Banner

Bottle

by benrowland

Download
Description

Bottle is a bot throttling plugin. It consists of a set of tools that can be used to reduce server load when excessive bot requests are made to a website.

It works by performing a variety of checks on each HTTP request made to WordPress, and throttling sessions or IPs when thresholds are exceeded. These include:

  • Too many sessions per IP. Since some bots don’t bother with cookies, Bottle sets a JWT session cookie for each request. If a single IP address creates too many new sessions within a given timeframe, it is likely a bot, and will be throttled.

  • Too many requests per session. Bottle calculates a request rate for each session. Bots that exceed that rate will be throttled.

  • Too many requests per IP. Bottle calculates a request rate per IP. For bots that don’t process cookies (and therefore sessions), this can throttle abusive IPs.

  • Overall request rate. Bottle can throttle specific sections of a website that are being abused by bots. (For instance, the courses catalog.) This is useful when a large website is being scraped by a bot network with rotating IPs.

  • Excessive 404s. If bots are scanning a website, or are misconfigured, excessive 404s can be time-consuming for WordPress to process. Bottle can either throttle all 404 requests, or can block 404s for static files (i.e. images) only.

When a client is throttled, Bottle can outright throttle it with a 429 response, or can serve a JavaScript-based cryptographic puzzle. If a client is legitimate, it will automatically compute the puzzle’s solution and submit it to the server for verification. (No user input is required.) After it has proven its human-ness, Bottle will permit that user entry to the site. At the moment, some bots, but not all, will solve these puzzles.

When a throttle condition is reached, Bottle terminates WordPress processing, which substantially reduces server load when under attack. Throttles end automatically at the end of their respective measurement windows.

Bottle is not a firewall and will not block or sanitize malicious attacks. However, since such attacks are typically performed in large quantities, Bottle can be effective at slowing such attacks and reducing server load, keeping the website operational for legitimate users.

Overall, Bottle is a set of tools designed to reduce server load with minimal disruption to real users.

Install and activate the plugin. It will create a mu-plugin automatically.

Important: We recommend the use of an object cache such as Redis because Bottle uses transients as one method of measuring bot activity. The use of Redis dramatically improves the performance of the get_transient and set_transient functions, and not using it will reduce the performance of Bottle.

After installation, all Bottle settings are initially disabled. Go to Settings -> Bottle Settings and configure the plugin as desired.

We recommend that you enable the “Request Logging Enabled” setting temporarily while configuring Bottle. This will show you a live view of all requests that come into WordPress, including the status of each test Bottle runs on each request. This can help you identify which ones need to be configured or adjusted.

  1. Bottle offers a range of configurable tools, stats, and logging.

    Bottle offers a range of configurable tools, stats, and logging.

  2. A live requests log shows all requests and helps you configure each of the throttle tools.

    A live requests log shows all requests and helps you configure each of the throttle tools.

  3. The throttle log shows which requests were throttled and why. It also shows when the PoW is solved and allows a client to bypass the throttle.

    The throttle log shows which requests were throttled and why. It also shows when the PoW is solved and allows a client to bypass the throttle.

  4. The paths log shows which URLs triggered the throttles, and the number of throttles per URL.

    The paths log shows which URLs triggered the throttles, and the number of throttles per URL.

  5. The IPs log shows which IPs triggered the throttles, and the number of throttles per IP.

    The IPs log shows which IPs triggered the throttles, and the number of throttles per IP.

  6. Session IP Throttle is a tool that can throttle IPs that create too many new sessions too quickly.

    Session IP Throttle is a tool that can throttle IPs that create too many new sessions too quickly.

  7. Session Rate Throttle is a tool that can throttle sessions that request too many pages too quickly.

    Session Rate Throttle is a tool that can throttle sessions that request too many pages too quickly.

  8. Request Rate per IP is a tool that throttles IPs that request too many pages too quickly.

    Request Rate per IP is a tool that throttles IPs that request too many pages too quickly.

  9. Overall Page Matching Throttle is a tool that throttles a specific section of a website that is being abused by bots. (For instance, the courses catalog.) This is useful when a large website is being scraped by a bot network with rotating IPs.

    Overall Page Matching Throttle is a tool that throttles a specific section of a website that is being abused by bots. (For instance, the courses catalog.) This is useful when a large website is being scraped by a bot network with rotating IPs.

  10. 404s is a tool that can block IPs that make requests for too many non-existant static files. Or, it can throttle IPs that request too many non-existant pages.

    404s is a tool that can block IPs that make requests for too many non-existant static files. Or, it can throttle IPs that request too many non-existant pages.

  11. Exclude is a tool that lets you configure overrides for the throttle tools (URL begins with, URL contains, and IP address).

    Exclude is a tool that lets you configure overrides for the throttle tools (URL begins with, URL contains, and IP address).

  12. Proof-of-Work is an cryptographic puzzle displayed to users instead of an outright 429 HTTP code (throttle). Many bots do not process JavaScript and will not solve puzzles. The length is the complexity of the puzzle. A larger number generally takes longer to solve, so this is a measure of how long a delay your legitimate users will face when throttled.

    Proof-of-Work is an cryptographic puzzle displayed to users instead of an outright 429 HTTP code (throttle). Many bots do not process JavaScript and will not solve puzzles. The length is the complexity of the puzzle. A larger number generally takes longer to solve, so this is a measure of how long a delay your legitimate users will face when throttled.

  13. Logging options includes log location and rotation settings. We recommend using the Request Logging tool only for setup or debugging and leave it disabled the rest of the time.

    Logging options includes log location and rotation settings. We recommend using the Request Logging tool only for setup or debugging and leave it disabled the rest of the time.

  14. The Proof-of-Work presented to throttled clients.

    The Proof-of-Work presented to throttled clients.

Does proof-of-work actually work?

Yes. It’s a somewhat controversial method, and we cannot claim credit for it; ours is a rather simple implementation inspired by Anubis. Is it ethical to offload the work to your users? We’re not sure. Are there PoW-solving browser extensions that intercept and offload to GPU for faster processing? Sure. Does it stop all bots? Nope. We have found that some bots will solve this. However, at this time, and for the types of traffic we see on our sites, it significantly reduces server load while allowing users access to key sections of websites that would otherwise be blocked.

Does Bottle work with Varnish?

Yes, with proper configuration. We have provided Varnish VCL modifications that allow Varnish to continue caching pages while passing the JWT session cookie back and forth. When a page is cached by Varnish, it will never reach Bottle. This throttle is effective on the backend only. However, when a very large website is being scanned or attacked, and many of the pages are not cached, Bottle can be an effective deterrent without reducing the effectiveness of Varnish caching.

Why did we create Bottle?

We are a web software vendor that hosts many websites and applications for clients.

We observed many WordPress sites were being routinely attacked, and these attacks caused slowdowns and outages. Recently, with the uptick in AI bot traffic, sites began experiencing bot network attacks, in which clients rapidly rotate their IP addresses. One traffic pattern we identified was one in which large content sections of a website (such as /courses/*) were being scanned. Another involved dynamically contructed URLs (for instance, the following urls are all the same: /content/a-b-c /content/a-c-b /content/b-a-c /content/b-c-a /content/c-a-b /content/c-b-a). Such URls can be easily identified with regular expressions that could block this type of traffic, and the idea for Bottle was born.

Today, we run it on all of our sites. We find it protects them while minimizing user disruption. Your experience may be different. We think of Bottle as a toolkit we can configure based on the specific sites and situation.

0.2.2

  • Initial release
Back to top