Blocking Aggressive Scrapers at the EdgeIn Limiting expensive to render nginx endpoints , I describe how to use a few nginx limit_req module to substantially limit the amount of aggressive scraping traffic to my Gitea instance without impacting “normal” “human” behavior.
There’s three layered rate-limiters in here that are applied to only certain URIs:
One does a per-IP limit excluding my Tailscale network and some ASNs I connect from. Each IP can make one costly request per minute, otherwise receive a 503.
One tries to map certain cloud providers in to a single rate-limit key and gives each of these providers 1 RPM on these endpoints. Each group of cloud IPs can make one request per minute, otherwise receive…
External feed Read More at the Source: https://cce.whatthefuck.computer/updates#20250320T130459.421338