How Many Proxies Do You Need for Web Scraping?

The honest answer is that it depends on one number you can measure in ten minutes: how fast a single IP can hit your target before it gets blocked.

The formula

proxies needed = (requests per minute you want) / (safe requests per minute per IP)

That is it. The difficulty is the denominator, and no article can tell you it — it is specific to your target.

Finding the safe rate

Take one proxy and one target. Start slow and increase until it breaks.

for i in $(seq 1 60); do
  code=$(curl -s -x "$PROXY" -o /dev/null -w "%{http_code}" "$TARGET")
  echo "$i $code"
  sleep 1
done

That is 60 requests per minute from one IP. Watch for the first 429 or 403. Halve whatever rate broke it and use that as your safe rate — you want headroom, because limits often tighten under load.

Typical ranges, purely as a starting point:

• Small sites, no protection: 30–60 req/min per IP
• Mid-size e-commerce: 5–15 req/min per IP
• Large protected sites: 1–5 req/min per IP
• Aggressive anti-bot: under 1 req/min, and IP count is not your main problem

Worked example

You want 100,000 pages in 24 hours.

100,000 / (24 × 60) ≈ 70 requests per minute

If testing shows 10 req/min is safe on that target:

70 / 10 = 7 proxies

Add headroom for failures and retries — call it 10. That is a very different purchase from the “you need hundreds of proxies” advice you will find elsewhere.

Why more proxies is often the wrong fix

When a scraper starts getting blocked, the reflex is to buy more IPs. That works only if the block is genuinely rate-based. It usually is not.

If you are blocked instantly on a fresh IP, the problem is the IP type, not the count. Datacenter ranges are published and widely blocked. One residential or ISP proxy will outperform fifty datacenter IPs on a protected target — see residential vs datacenter.

If you are blocked after a consistent number of requests, it is rate-based and more IPs genuinely helps.

If you are blocked regardless of IP, you are being fingerprinted — TLS signature, header order, missing Accept-Language, headless browser tells. More proxies will not help at all. Fix the client first.

Diagnose before you buy. The test above tells you which case you are in.

Residential is billed differently

Datacenter and ISP proxies are priced per IP per month, so “how many” is the question. Residential is billed per gigabyte, so the question becomes how much traffic you will move:

GB needed = pages × average page size

A typical HTML page is 50–200 KB if you block images and other assets, which you should. 100,000 pages at 100 KB is about 10 GB. Blocking images can cut bandwidth by 80% or more, and it is the single biggest lever on residential cost.

Concurrency is not the same as pool size

Ten proxies does not mean ten simultaneous requests. If your safe rate is 10 req/min per IP, ten proxies gives you 100 req/min spread over a minute — not 100 at once. Firing all ten simultaneously every six seconds is burstier than a human and gets noticed.

Pace requests per proxy rather than blasting the pool.

A sensible starting point

For most projects: start with 5–10 proxies, measure your actual block rate, and scale from there. You will learn more from a week of real data than from any estimate, and you will usually find you need fewer IPs and better client behaviour than you expected.

See pricing for per-proxy and per-GB costs, or our web scraping guide for the client-side work that reduces how many you need.