How to benchmark proxy cost per successful request for scraping
Your proxy plan says $4 per GB. Your scraper pays more than that. Every CAPTCHA page, empty response and retry uses traffic you're billed for, and none of it gives you data. Proxy cost per successful request is the number that includes all of that waste. This guide shows how to calculate it and how to run a benchmark you can repeat every month.
Why price per GB doesn't show your real cost
Pay-per-GB proxies charge for traffic, whether or not the request returns anything useful. A block page costs money. A timeout that sends half a page costs money. So does a retry that fails again.
The biggest hidden loss is the soft block. The target returns HTTP 200, but the body is a CAPTCHA, a login wall or an empty template. Your logs count it as a success. Your dataset doesn't.
AIMultiple's proxy benchmark shows how large this gap can be. Vendors advertise success rates of 95–99%, but the benchmark measured much lower rates because it rejects CAPTCHAs and login walls served with HTTP 200. On protected sites, both residential and datacenter proxies returned real pages only 55–75% of the time.
Flat-rate plans hide the same waste in a different place. Failed requests don't add to the bill, but they use up threads, time and concurrency limits. If you're choosing between plan types, see our comparison of unlimited residential proxies vs pay-per-GB.
The formula for proxy cost per successful request
Start by defining "successful" strictly. A request counts only when all of these are true:
-
The final status code is in the 2xx range after redirects
-
The body contains a marker you expect, such as a price element or a product title
-
There's no CAPTCHA, block page or login wall in the response
Then the math is simple:
Cost per successful request = total proxy spend for the test ÷ number of validated results
For pay-per-GB plans, total spend is the GB billed multiplied by your price per GB. For per-IP or unlimited plans, use the share of the monthly fee that this job used.
Here's an example with two pools. Each attempt transfers 200 KB, and the goal is 1,000 valid pages.
|
Pool A |
Pool B |
|
|---|---|---|
|
List price |
$4/GB |
$5/GB |
|
Success rate |
55% |
95% |
|
Requests needed for 1,000 valid pages |
1,818 |
1,053 |
|
Data billed |
0.36 GB |
0.21 GB |
|
Cost per 1,000 successful requests |
$1.45 |
$1.05 |
Pool B costs 25% more per GB and is about 28% cheaper per result. This is why proxy cost per successful request is a better way to compare providers than list price.
How to run a fair benchmark
A benchmark only means something when every provider gets the same conditions. Keep these fixed:
-
The same URLs. Use a few hundred real pages from your actual targets, not a test endpoint that never blocks.
-
The same settings. Use identical headers, timeouts, concurrency and retry rules for every pool.
-
The same time window. Run providers in parallel, or alternate between them. Block rates change by hour and day.
-
The same validation. Apply the content check from the previous section to every response.
Before the real run, check basics like speed, location and DNS leaks with our guide on how to test proxies before you use them.
For each request, log the URL, status code, bytes transferred, latency, attempt number and whether it passed validation. Your retry logic needs to be consistent and visible in the logs. Our Scrapy proxy rotation and retry system shows one way to set that up.
Keep proxy-side errors separate from target blocks. A 407 or 502 points to an authentication or gateway problem, not to the site defending itself. If those errors show up, fix the proxy errors and run the test again.
Finally, take spend from the provider's usage dashboard instead of your own byte counter. The figure you're billed for is the one that goes into the formula.
import pandas as pd
log = pd.read_csv("benchmark_log.csv") # one row per attempt
valid = log[log["passed_validation"]].groupby("provider").size()
billed = {"pool_a": 1.45, "pool_b": 1.05} # USD from each dashboard
for provider, spend in billed.items():
print(provider, round(spend / valid[provider] * 1000, 2), "USD per 1k")
Hidden costs that change the result
Success rate isn't the only thing that moves the number. Three other factors matter about as much.
Bandwidth per page. According to HTTP Archive's 2025 Web Almanac, the median mobile home page weighs about 2.56 MB, while the median HTML document is only 33 KB. If a headless browser loads every image, font and script through your proxy, you can pay for roughly 75 times more data than the HTML you need. Blocking unneeded resources is often the fastest saving available. Our guide to cutting proxy costs in web scraping covers the main methods.
Retries. Every retry adds traffic and time. A pool that succeeds on the third attempt can look fine in a success-rate chart while costing three times as much per result. Track attempts per valid result, not just final outcomes.
Latency. Slow proxies keep browsers, workers and servers busy for longer. That compute cost doesn't show up on your proxy invoice, but it's part of the real cost of each result. If response times are high, start with measuring and fixing proxy latency.
Using the number to make proxy decisions
Once you know your proxy cost per successful request for each target, you can route traffic based on that number instead of habit.
Many teams send everything through residential proxies by default. A better approach is to start each target on cheaper datacenter IPs and move only the targets that keep failing to residential. AIMultiple's benchmark recommends the same approach. Easy targets stay cheap, and hard ones get the IPs they need.
Set a budget per 1,000 valid results for every target, and alert when a job goes over it. A sudden jump usually means the site has changed its anti-bot setup, its page structure or both.
Run the benchmark again every month, and whenever a target's success rate drops. The results only hold for the URLs, settings and dates you tested, so an old benchmark tells you little about this month's costs.