Blog

How to build a reliable proxy rotation and retry system with Scrapy

Build a Scrapy rotating proxy middleware that benches failing IPs, retries every failure through a fresh proxy, and catches soft blocks Scrapy misses.

November 18, 2026 · 10 min read

How to build a reliable proxy rotation and retry system with Scrapy

How to build a reliable proxy rotation and retry system with Scrapy

Scrapy can route traffic through proxies and retry failed requests out of the box. The catch is that these two features don't talk to each other. A request that failed because of a bad IP gets retried through that same IP, and many blocks never trigger a retry at all.

This guide builds a small Scrapy rotating proxy middleware that fixes both problems. It picks a proxy for every request, benches proxies that fail and retries each failure through a different IP.

How Scrapy handles proxies and retries by default

Two built-in downloader middlewares do the work. HttpProxyMiddleware reads the proxy URL from request.meta["proxy"] and builds the Proxy-Authorization header from any credentials in that URL. RetryMiddleware re-sends requests that fail with certain status codes or network errors.

Out of the box, the retry rules are narrow:

Setting

Default

RETRY_TIMES

2 (on top of the first attempt)

RETRY_HTTP_CODES

408, 429, 500, 502, 503, 504, 522, 524

That leaves two gaps.

First, a retried request is a copy of the original, meta included. So it goes back out through the same proxy that just failed.

Second, 403 isn't on the list. Neither is a "200 OK" page that holds a CAPTCHA or nothing at all. Scrapy treats those as successes and passes them straight to your parse callback.

The Scrapy downloader middleware docs cover each setting. Below, we replace that logic with one that knows about proxies.

Building a Scrapy rotating proxy middleware

The middleware needs three things: a pool of proxies, a way to pick a healthy one and a bench where failing proxies sit out a cooldown.

Start with the settings:

# settings.py

PROXY_LIST = [
   "http://user:[email protected]:8000",
   "http://user:[email protected]:8000",
   "http://user:[email protected]:8000",
]
PROXY_COOLDOWN = 300  # seconds a failing proxy sits out


DOWNLOADER_MIDDLEWARES = {
   "scrapy.downloadermiddlewares.retry.RetryMiddleware": None,
   "myproject.middlewares.RotatingProxyMiddleware": 610,
}

Setting RetryMiddleware to None hands every retry decision to the new middleware, so every retry gets a new IP. The order number matters too. Scrapy runs process_request() in increasing order, so our middleware (610) sets the proxy before HttpProxyMiddleware (750) reads it.

Keep the credentials inside each proxy URL. When the proxy changes on a retry, HttpProxyMiddleware replaces the auth header with the new URL's credentials. The header always matches the proxy.

Now the middleware itself:

# middlewares.py
import random
import time
from urllib.parse import urlparse

from scrapy.downloadermiddlewares.retry import get_retry_request
from scrapy.exceptions import IgnoreRequest

BAN_CODES = {403, 429}

class RotatingProxyMiddleware:
   def __init__(self, crawler):
       s = crawler.settings
       self.crawler = crawler
       self.proxies = s.getlist("PROXY_LIST")
       self.cooldown = s.getint("PROXY_COOLDOWN", 300)
       self.retry_codes = {int(c) for c in s.getlist("RETRY_HTTP_CODES")}
| BAN_CODES
       self.benched = {}  # proxy URL -> time it can come back

   @classmethod
   def from_crawler(cls, crawler):
       return cls(crawler)
   def pick(self, exclude=None):
        now = time.monotonic()

       ready = [p for p in self.proxies
                if p != exclude and self.benched.get(p, 0) <= now]
        return random.choice(ready or self.proxies)

   def process_request(self, request, spider=None):
        request.meta.setdefault("proxy", self.pick())

setdefault only assigns a proxy when none is set. That gives you an escape hatch: set meta["proxy"] yourself to pin a request to one IP. That's what you want for logins, carts and multi-page sessions. Our guide on sticky vs rotating proxy sessions explains when each one fits.

Using one rotating gateway instead of a list? Fill the pool with that gateway and different session IDs in the username.

Prefer something prebuilt? The scrapy-rotating-proxies package adds ban detection and exponential backoff. Its last release was in May 2019, so test it with your Scrapy version first.

Retrying failed requests with a fresh proxy

Next, add these methods to the same class. They decide what counts as a failure and what happens next.

    BLOCK_MARKERS = (b"captcha", b"access denied")  # tune per target site

   def process_response(self, request, response, spider=None):
       if response.status == 407:
           self.crawler.spider.logger.error("Proxy auth failed, check credentials")
           return response
       if response.status in self.retry_codes or self.is_soft_block(response):
           return self.retry(request, f"status_{response.status}") or response
       if request.meta.get("proxy"):
           self.crawler.stats.inc_value(f"proxy/{self.label(request.meta['proxy'])}/ok")
       return response

   def process_exception(self, request, exception, spider=None):
       if isinstance(exception, IgnoreRequest):
           return None
       return self.retry(request, type(exception).__name__)

   def is_soft_block(self, response):
       if response.status != 200:
           return False
       head = response.body[:5000].lower()
        return not head or any(m in head for m in self.BLOCK_MARKERS)

   def retry(self, request, reason):
       if request.meta.get("dont_retry"):
           return None
       bad = request.meta.get("proxy")
       if bad:
           self.benched[bad] = time.monotonic() + self.cooldown
           self.crawler.stats.inc_value(f"proxy/{self.label(bad)}/failed")
       new = get_retry_request(request, spider=self.crawler.spider, reason=reason)
       if new is not None and bad:
           new.meta["proxy"] = self.pick(exclude=bad)
       return new

   @staticmethod
   def label(proxy):
       return urlparse(proxy).netloc.rsplit("@", 1)[-1]  # host:port, no password

Here's what happens on each failure:

  • The proxy that failed goes on the bench for PROXY_COOLDOWN seconds.

  • get_retry_request() builds the retry copy. It respects RETRY_TIMES (or a per-request max_retry_times) and returns None once retries run out.

  • The copy gets a new proxy. Never the one that just failed.

is_soft_block() catches what Scrapy misses: 200 responses that come back empty or show a block page. The markers are only examples, so swap in a string that only appears on your target's block page. Behind Cloudflare, rotation alone rarely beats a challenge page. Our guide on avoiding Cloudflare blocks with proxies covers the rest.

Not every error is the target's fault. A 407 means the proxy rejected your credentials. Every IP on the same account will fail the same way, so the code logs it instead of burning retries. Tunnel errors and proxy-side 502s also point to the proxy. Our article on fixing proxy errors 407, 502, 504 and tunnel failures walks through each one.

Tuning settings so retries don't make things worse

A retry system can turn a small block into a big one. If a site starts sending 429s and you hit it again at once from ten fresh IPs, you've told it exactly what you are.

These starting values keep retries calm:

Setting

Scrapy default

Suggested start

RETRY_TIMES

2

3–5

DOWNLOAD_TIMEOUT

180 s

20–30 s

CONCURRENT_REQUESTS_PER_DOMAIN

1 in new projects (8 otherwise)

1–4

DOWNLOAD_DELAY

1 s in new projects (0 otherwise)

1–2 s

AUTOTHROTTLE_ENABLED

False

True

A higher RETRY_TIMES makes sense, because each retry uses a different IP. The 180-second timeout is far too generous. One stalled proxy can hold a request slot for three minutes. If timeouts pile up, measure your proxy latency before you blame the target.

AutoThrottle is worth turning on. Error responses can raise its delay but never lower it, so the spider slows itself down as soon as blocks start.

Monitoring proxy health and retry costs

Once the spider runs, the numbers tell you which proxies are pulling their weight. Scrapy's stats collector already tracks retries, because get_retry_request() writes them:

  • retry/count: total retries

  • retry/max_reached: requests that ran out of retries

  • retry/reason_count/<reason>: retries grouped by cause

The middleware adds proxy/<host>/ok and proxy/<host>/failed for each proxy, with the password stripped so it stays out of your logs. Drop any proxy that fails far more often than the rest. The Scrapy stats docs show how to read these values from code.

Watch the retry count closely if you pay per GB. Every retry downloads the page again. If one in four requests needs a second try, you pay for that traffic too. Our guide to cutting proxy costs in web scraping shows where else to trim the bill.

The cheapest retry is the one you never need. Test new proxies for speed, leaks and errors before they go into PROXY_LIST, and the bench will stay mostly empty.