How to Use Proxies with Python Requests (Properly)

The basic case is two lines. Everything that makes it survive contact with a real target — rotation, retries, timeouts, sessions — is where people get it wrong.

The basics

import requests

proxies = {
    "http":  "http://USER:PASS@HOST:PORT",
    "https": "http://USER:PASS@HOST:PORT",
}

r = requests.get("https://api.ipify.org", proxies=proxies, timeout=10)
print(r.text)

Both keys point at the same http:// proxy. That looks wrong but is correct: the https key means “use this proxy for HTTPS requests”, not “the proxy speaks HTTPS”. Requests issues a CONNECT and tunnels through it.

Always set a timeout. Without one, requests waits indefinitely. A dead proxy will hang your script forever rather than failing.

Special characters in passwords

If your password contains @, : or /, the URL parses wrong:

from urllib.parse import quote

user = quote("myuser", safe="")
pw   = quote("p@ss:word/1", safe="")
proxy = f"http://{user}:{pw}@HOST:PORT"

This is a common source of 407 errors that look like wrong credentials.

Use a Session

Creating a new connection for every request is slow and makes you look like a bot. A session reuses TCP connections and keeps cookies:

import requests

s = requests.Session()
s.proxies = {"http": PROXY, "https": PROXY}
s.headers.update({
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 "
                  "(KHTML, like Gecko) Chrome/126.0 Safari/537.36",
    "Accept-Language": "en-GB,en;q=0.9",
})

for url in urls:
    r = s.get(url, timeout=15)

A request with no User-Agent and no Accept-Language is a bot signature regardless of how good your proxy is.

Rotating proxies

import itertools, requests

PROXIES = [
    "http://user:pass@ip1:port",
    "http://user:pass@ip2:port",
    "http://user:pass@ip3:port",
]
pool = itertools.cycle(PROXIES)

def fetch(url):
    proxy = next(pool)
    return requests.get(url, proxies={"http": proxy, "https": proxy}, timeout=15)

Round-robin is fine for spreading load. It does not handle failure — a dead proxy stays in rotation forever.

Retries that skip bad proxies

import random, requests

def fetch(url, pool, attempts=3):
    tried = set()
    for _ in range(attempts):
        candidates = [p for p in pool if p not in tried]
        if not candidates:
            break
        proxy = random.choice(candidates)
        tried.add(proxy)
        try:
            r = requests.get(url, proxies={"http": proxy, "https": proxy}, timeout=15)
            if r.status_code == 429:
                continue            # rate-limited: try a different IP
            r.raise_for_status()
            return r
        except requests.RequestException:
            continue
    raise RuntimeError(f"all {attempts} attempts failed for {url}")

Note tried — retrying on the same failing proxy is the most common bug in scraping code.

SOCKS5, and the DNS leak

Requests needs an extra package:

pip install "requests[socks]"

proxies = {
    "http":  "socks5h://USER:PASS@HOST:PORT",
    "https": "socks5h://USER:PASS@HOST:PORT",
}

Use socks5h, not socks5. The h makes the proxy resolve hostnames. Without it your machine resolves them locally, so your ISP sees every domain you visit even though the traffic is proxied. Everything still works, which is what makes it dangerous. See SOCKS5 vs HTTP.

httpx, for async

import httpx, asyncio

async def main():
    async with httpx.AsyncClient(proxy="http://USER:PASS@HOST:PORT", timeout=15) as c:
        r = await c.get("https://api.ipify.org")
        print(r.text)

asyncio.run(main())

Note it is proxy= (singular) in current httpx — the older proxies= argument was removed, which trips up a lot of copied code.

Do not disable certificate verification

You will find advice to set verify=False when a proxy throws TLS errors. Don’t. It silences the warning and leaves you open to anyone on the path reading your traffic — including whoever operates the proxy.

A TLS error through a proxy usually means the proxy is intercepting rather than tunnelling. That is worth investigating, not suppressing.

Checking it works

r = requests.get("https://api.ipify.org", proxies=proxies, timeout=10)
assert r.text.strip() != requests.get("https://api.ipify.org", timeout=10).text.strip()

If both return the same address, your requests are not going through the proxy at all — usually a typo in the dict keys.

For choosing a proxy type, see residential vs datacenter. For sizing a pool, how many proxies you need.