The Anti-Blocking Checklist: 12 Things to Fix Before Buying More Proxies

Before you spend anything on more IPs, work down this list. It is ordered by impact per unit of effort, and most of it is free.

Free fixes, in order

1. Send a real User-Agent. python-requests/2.31 identifies you instantly. Copy a current Chrome string — and update it occasionally, because a two-year-old Chrome version is its own signal.

2. Send the other headers too. Real browsers send Accept, Accept-Language, Accept-Encoding, Connection. A Chrome User-Agent arriving with none of Chrome’s companion headers is more suspicious than no User-Agent at all.

3. Reuse connections. Use a session or client object rather than a fresh connection per request. Faster, and TCP behaviour is part of your fingerprint.

4. Randomise timing. A request exactly every 2.000 seconds is not a person. Jitter between 1 and 5 seconds costs almost nothing.

5. Honour Retry-After. When a 429 includes it, wait that long. Ignoring it converts a temporary throttle into a lasting ban on many sites.

6. Back off exponentially. 1s, 2s, 4s, 8s. Immediate retries against a struggling target are how you get escalated.

7. Retry on a different proxy. Retrying the same failing IP is the most common bug in scraping code — track which you have tried per URL.

8. Set a timeout on everything. A dead proxy with no timeout hangs your run indefinitely.

Cheap fixes

9. Match your locale to your IP. A German proxy with Accept-Language: en-US and an America/New_York timezone is a contradiction. Keep IP country, language and timezone consistent — mismatches are actively worse than defaults.

10. Block images and fonts. On per-GB residential plans this cuts bandwidth 70–80%. Code in Puppeteer and Playwright.

11. Fix your TLS fingerprint. Your HTTPS handshake reveals whether you are Python, Go or Chrome, regardless of what your User-Agent claims. curl_cffi in Python impersonates real browser handshakes. This is the one most people never check, and on well-protected targets it is often the whole problem.

12. Keep sessions on one IP. Rotating mid-session looks like session hijacking. See rotating vs sticky sessions.

Then, and only then, consider proxies

Having done the above, diagnose properly before buying:

• Blocked instantly on a fresh IP → wrong proxy type. Move from datacenter to residential or ISP. More datacenter IPs will not help.
• Blocked after a consistent count → genuine rate limiting. More IPs helps. Size it with this calculation.
• Blocked on every IP including good residential → still a fingerprint problem. Go back to items 1, 2, 9 and 11.

How to test a change

Change one thing at a time and measure. Run a fixed sample — say 200 URLs — record the block rate, change one item, run it again.

Changing five things at once and seeing improvement teaches you nothing about which mattered, and you will carry four pieces of cargo-cult configuration into every future project.

The part people skip

Check whether you are allowed to. Read the site’s terms and its robots.txt. Scraping public data is broadly lawful in many jurisdictions, but personal data brings GDPR into scope, and terms of service can create contractual obligations. Our acceptable use policy sets out what is permitted on our network.

Rate-limiting yourself is also just good manners. A target that stays up is one you can keep scraping.

Summary

Eight of the twelve items cost nothing but attention, and together they resolve most blocking. Buy proxies to solve a rate problem or an IP-reputation problem — both of which you can now identify — rather than as a general remedy.

If you have worked through this and are still stuck, tell us the target and we will say which proxy type actually suits it.