Why Proxies Get Banned (and How to Avoid It)

When a scraper stops working, the reflex is to blame the proxy and buy more IPs. Often the IP was never the problem. Sites check four things, roughly in this order, and only the first is about your proxy.

1. The IP’s reputation

Every IP belongs to a range, and ranges have histories.

Datacenter ranges are published. Anyone can download the address blocks belonging to AWS, Hetzner or DigitalOcean, and many sites block them wholesale — not because you did anything, but because almost no genuine customer browses from a server rack.

Residential and ISP ranges belong to consumer connections. They carry the trust of the households and businesses they were assigned to.

If you are blocked instantly, on the very first request from a fresh IP, this is almost certainly your problem. More IPs of the same type will not help. See residential vs datacenter.

2. Request rate

Blocked after a consistent number of requests rather than immediately? That is rate limiting, and it is the one case where more proxies genuinely helps.

Watch for the pattern: fine for 200 requests, then 429 on every one. Halve your rate per IP and spread across more of them — how many proxies you need covers the arithmetic.

Also: honour Retry-After. Ignoring it turns a temporary limit into a permanent ban on many sites.

3. Your client fingerprint

This is where most people are actually losing, and no proxy fixes it.

Headers. A request with no User-Agent, or with python-requests/2.31, is self-identifying. Real browsers send Accept, Accept-Language, Accept-Encoding and more, in a consistent order. Sending a Chrome User-Agent with none of Chrome’s other headers is more suspicious than sending nothing.

TLS fingerprint (JA3). Your HTTPS handshake — cipher suites, extensions, the order of them — differs between Python, Go, curl and Chrome. Sites read this. You can claim to be Chrome in your User-Agent, but if your TLS handshake says Python, the mismatch itself is the signal. Libraries like curl_cffi exist specifically to impersonate real browser handshakes.

Headless browser tells. Default Puppeteer and Playwright set navigator.webdriver = true and carry other detectable properties. Stealth plugins address the common ones — see Puppeteer and Playwright.

The diagnostic: if you are blocked on every IP you try, including good residential ones, the problem is your client, not your proxies.

4. Behaviour

Even with a trusted IP and a convincing fingerprint, you can look wrong.

• Perfectly regular timing. A request exactly every 2.000 seconds is not human. Randomise intervals.
• Impossible navigation. Landing directly on deep URLs with no referrer, in an order no person would follow.
• No supporting requests. Real browsers fetch CSS, JS, images and fonts. Fetching only HTML is a clear automation signature — though blocking images saves substantial bandwidth on residential plans, so this is a genuine trade-off.
• Rotating mid-session. Changing IP while logged in looks like a hijacked session, because that is what it resembles. See rotating vs sticky sessions.

Diagnosing quickly

Symptom	Cause	Fix
Blocked on first request, fresh IP	IP reputation	Change proxy type
Blocked after N requests, consistently	Rate limit	Slow down, add IPs
Blocked on every IP including residential	Fingerprint	Fix headers and TLS
Works, then fails after login	Rotation	Use sticky sessions
Works from browser, fails from script	Fingerprint	Match real browser headers

Protecting the IPs you have

A burned residential IP goes back into a pool and someone else gets it. Being careless costs other people, and eventually costs you when you receive an IP that someone else burned.

Keep request rates conservative, back off on the first 429 rather than the tenth, and stop hammering a target that has clearly blocked you — repeated attempts from a blocked address extend the block.

The honest summary

Buy the right proxy type for the target, then fix your client. Those two, in that order, solve the large majority of blocking. Proxy count is the third lever, not the first — and if you are blocked instantly on good residential IPs, buying more of them is money spent on the wrong problem.

If you are unsure which type suits your target, get in touch — it is usually a short conversation, and cheaper than guessing.