How to build a reliable proxy rotation and retry system with Scrapy
Scrapy can route traffic through proxies and retry failed requests out of the box. The catch is that these two features don't talk to each other. A request that failed because of a bad IP gets retried through that same IP, and many blocks never trigger a retry at all.
This guide builds a small Scrapy rotating proxy middleware that fixes both problems. It picks a proxy for every request, benches proxies that fail and retries each failure through a different IP.
How Scrapy handles proxies and retries by default
Two built-in downloader middlewares do the work. HttpProxyMiddleware reads the proxy URL from request.meta["proxy"] and builds the Proxy-Authorization header from any credentials in that URL. RetryMiddleware re-sends requests that fail with certain status codes or network errors.
Out of the box, the retry rules are narrow:
|
Setting |
Default |
|---|---|
|
RETRY_TIMES |
2 (on top of the first attempt) |
|
RETRY_HTTP_CODES |
408, 429, 500, 502, 503, 504, 522, 524 |
That leaves two gaps.
First, a retried request is a copy of the original, meta included. So it goes back out through the same proxy that just failed.
Second, 403 isn't on the list. Neither is a "200 OK" page that holds a CAPTCHA or nothing at all. Scrapy treats those as successes and passes them straight to your parse callback.
The Scrapy downloader middleware docs cover each setting. Below, we replace that logic with one that knows about proxies.
Building a Scrapy rotating proxy middleware
The middleware needs three things: a pool of proxies, a way to pick a healthy one and a bench where failing proxies sit out a cooldown.
Start with the settings:
# settings.py
PROXY_LIST = [
"http://user:[email protected]:8000",
"http://user:[email protected]:8000",
"http://user:[email protected]:8000",
]
PROXY_COOLDOWN = 300 # seconds a failing proxy sits out
DOWNLOADER_MIDDLEWARES = {
"scrapy.downloadermiddlewares.retry.RetryMiddleware": None,
"myproject.middlewares.RotatingProxyMiddleware": 610,
}
Setting RetryMiddleware to None hands every retry decision to the new middleware, so every retry gets a new IP. The order number matters too. Scrapy runs process_request() in increasing order, so our middleware (610) sets the proxy before HttpProxyMiddleware (750) reads it.
Keep the credentials inside each proxy URL. When the proxy changes on a retry, HttpProxyMiddleware replaces the auth header with the new URL's credentials. The header always matches the proxy.
Now the middleware itself:
# middlewares.py
import random
import time
from urllib.parse import urlparse
from scrapy.downloadermiddlewares.retry import get_retry_request
from scrapy.exceptions import IgnoreRequest
BAN_CODES = {403, 429}
class RotatingProxyMiddleware:
def __init__(self, crawler):
s = crawler.settings
self.crawler = crawler
self.proxies = s.getlist("PROXY_LIST")
self.cooldown = s.getint("PROXY_COOLDOWN", 300)
self.retry_codes = {int(c) for c in s.getlist("RETRY_HTTP_CODES")}
| BAN_CODES
self.benched = {} # proxy URL -> time it can come back
@classmethod
def from_crawler(cls, crawler):
return cls(crawler)
def pick(self, exclude=None):
now = time.monotonic()
ready = [p for p in self.proxies
if p != exclude and self.benched.get(p, 0) <= now]
return random.choice(ready or self.proxies)
def process_request(self, request, spider=None):
request.meta.setdefault("proxy", self.pick())
setdefault only assigns a proxy when none is set. That gives you an escape hatch: set meta["proxy"] yourself to pin a request to one IP. That's what you want for logins, carts and multi-page sessions. Our guide on sticky vs rotating proxy sessions explains when each one fits.
Using one rotating gateway instead of a list? Fill the pool with that gateway and different session IDs in the username.
Prefer something prebuilt? The scrapy-rotating-proxies package adds ban detection and exponential backoff. Its last release was in May 2019, so test it with your Scrapy version first.
Retrying failed requests with a fresh proxy
Next, add these methods to the same class. They decide what counts as a failure and what happens next.
BLOCK_MARKERS = (b"captcha", b"access denied") # tune per target site
def process_response(self, request, response, spider=None):
if response.status == 407:
self.crawler.spider.logger.error("Proxy auth failed, check credentials")
return response
if response.status in self.retry_codes or self.is_soft_block(response):
return self.retry(request, f"status_{response.status}") or response
if request.meta.get("proxy"):
self.crawler.stats.inc_value(f"proxy/{self.label(request.meta['proxy'])}/ok")
return response
def process_exception(self, request, exception, spider=None):
if isinstance(exception, IgnoreRequest):
return None
return self.retry(request, type(exception).__name__)
def is_soft_block(self, response):
if response.status != 200:
return False
head = response.body[:5000].lower()
return not head or any(m in head for m in self.BLOCK_MARKERS)
def retry(self, request, reason):
if request.meta.get("dont_retry"):
return None
bad = request.meta.get("proxy")
if bad:
self.benched[bad] = time.monotonic() + self.cooldown
self.crawler.stats.inc_value(f"proxy/{self.label(bad)}/failed")
new = get_retry_request(request, spider=self.crawler.spider, reason=reason)
if new is not None and bad:
new.meta["proxy"] = self.pick(exclude=bad)
return new
@staticmethod
def label(proxy):
return urlparse(proxy).netloc.rsplit("@", 1)[-1] # host:port, no password
Here's what happens on each failure:
-
The proxy that failed goes on the bench for PROXY_COOLDOWN seconds.
-
get_retry_request() builds the retry copy. It respects RETRY_TIMES (or a per-request max_retry_times) and returns None once retries run out.
-
The copy gets a new proxy. Never the one that just failed.
is_soft_block() catches what Scrapy misses: 200 responses that come back empty or show a block page. The markers are only examples, so swap in a string that only appears on your target's block page. Behind Cloudflare, rotation alone rarely beats a challenge page. Our guide on avoiding Cloudflare blocks with proxies covers the rest.
Not every error is the target's fault. A 407 means the proxy rejected your credentials. Every IP on the same account will fail the same way, so the code logs it instead of burning retries. Tunnel errors and proxy-side 502s also point to the proxy. Our article on fixing proxy errors 407, 502, 504 and tunnel failures walks through each one.
Tuning settings so retries don't make things worse
A retry system can turn a small block into a big one. If a site starts sending 429s and you hit it again at once from ten fresh IPs, you've told it exactly what you are.
These starting values keep retries calm:
|
Setting |
Scrapy default |
Suggested start |
|---|---|---|
|
RETRY_TIMES |
2 |
3–5 |
|
DOWNLOAD_TIMEOUT |
180 s |
20–30 s |
|
CONCURRENT_REQUESTS_PER_DOMAIN |
1 in new projects (8 otherwise) |
1–4 |
|
DOWNLOAD_DELAY |
1 s in new projects (0 otherwise) |
1–2 s |
|
AUTOTHROTTLE_ENABLED |
False |
True |
A higher RETRY_TIMES makes sense, because each retry uses a different IP. The 180-second timeout is far too generous. One stalled proxy can hold a request slot for three minutes. If timeouts pile up, measure your proxy latency before you blame the target.
AutoThrottle is worth turning on. Error responses can raise its delay but never lower it, so the spider slows itself down as soon as blocks start.
Monitoring proxy health and retry costs
Once the spider runs, the numbers tell you which proxies are pulling their weight. Scrapy's stats collector already tracks retries, because get_retry_request() writes them:
-
retry/count: total retries
-
retry/max_reached: requests that ran out of retries
-
retry/reason_count/<reason>: retries grouped by cause
The middleware adds proxy/<host>/ok and proxy/<host>/failed for each proxy, with the password stripped so it stays out of your logs. Drop any proxy that fails far more often than the rest. The Scrapy stats docs show how to read these values from code.
Watch the retry count closely if you pay per GB. Every retry downloads the page again. If one in four requests needs a second try, you pay for that traffic too. Our guide to cutting proxy costs in web scraping shows where else to trim the bill.
The cheapest retry is the one you never need. Test new proxies for speed, leaks and errors before they go into PROXY_LIST, and the bench will stay mostly empty.