Skip to content

How to Use Proxies with Python Requests (Properly)

Working examples for requests, sessions, rotation, retries and httpx, including the SOCKS5 DNS-leak detail and why verify=False is a mistake.

Topic
Guides
Published
Reading time
3 minutes
Written by
froxproxy Team

The basic case is two lines. Everything that makes it survive contact with a real target (rotation, retries, timeouts, sessions) is where people get it wrong.

The basics

python
import requests

proxies = {
    "http":  "http://USER:PASS@HOST:PORT",
    "https": "http://USER:PASS@HOST:PORT",
}

r = requests.get("https://api.ipify.org", proxies=proxies, timeout=10)
print(r.text)

Both keys point at the same http:// proxy. That looks wrong but is correct: the https key means “use this proxy for HTTPS requests”, not “the proxy speaks HTTPS”. Requests issues a CONNECT and tunnels through it.

Always set a timeout. Without one, requests waits indefinitely. A dead proxy will hang your script forever rather than failing.

Special characters in passwords

If your password contains @, : or /, the URL parses wrong:

python
from urllib.parse import quote

user = quote("myuser", safe="")
pw   = quote("p@ss:word/1", safe="")
proxy = f"http://{user}:{pw}@HOST:PORT"

This is a common source of 407 errors that look like wrong credentials.

Use a Session

Creating a new connection for every request is slow and makes you look like a bot. A session reuses TCP connections and keeps cookies:

python
import requests

s = requests.Session()
s.proxies = {"http": PROXY, "https": PROXY}
s.headers.update({
    "User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 "
                  "(KHTML, like Gecko) Chrome/126.0 Safari/537.36",
    "Accept-Language": "en-GB,en;q=0.9",
})

for url in urls:
    r = s.get(url, timeout=15)

A request with no User-Agent and no Accept-Language is a bot signature regardless of how good your proxy is.

Rotating proxies

python
import itertools, requests

PROXIES = [
    "http://user:pass@ip1:port",
    "http://user:pass@ip2:port",
    "http://user:pass@ip3:port",
]
pool = itertools.cycle(PROXIES)

def fetch(url):
    proxy = next(pool)
    return requests.get(url, proxies={"http": proxy, "https": proxy}, timeout=15)

Round-robin is fine for spreading load. It does not handle failure: a dead proxy stays in rotation forever.

Retries that skip bad proxies

python
import random, requests

def fetch(url, pool, attempts=3):
    tried = set()
    for _ in range(attempts):
        candidates = [p for p in pool if p not in tried]
        if not candidates:
            break
        proxy = random.choice(candidates)
        tried.add(proxy)
        try:
            r = requests.get(url, proxies={"http": proxy, "https": proxy}, timeout=15)
            if r.status_code == 429:
                continue            # rate-limited: try a different IP
            r.raise_for_status()
            return r
        except requests.RequestException:
            continue
    raise RuntimeError(f"all {attempts} attempts failed for {url}")

Note tried: retrying on the same failing proxy is the most common bug in scraping code.

SOCKS5, and the DNS leak

Requests needs an extra package:

bash
pip install "requests[socks]"
python
proxies = {
    "http":  "socks5h://USER:PASS@HOST:PORT",
    "https": "socks5h://USER:PASS@HOST:PORT",
}

Use socks5h, not socks5. The h makes the proxy resolve hostnames. Without it your machine resolves them locally, so your ISP sees every domain you visit even though the traffic is proxied. Everything still works, which is what makes it dangerous. See SOCKS5 vs HTTP.

httpx, for async

python
import httpx, asyncio

async def main():
    async with httpx.AsyncClient(proxy="http://USER:PASS@HOST:PORT", timeout=15) as c:
        r = await c.get("https://api.ipify.org")
        print(r.text)

asyncio.run(main())

Note it is proxy= (singular) in current httpx. The older proxies= argument was removed, which trips up a lot of copied code.

Do not disable certificate verification

You will find advice to set verify=False when a proxy throws TLS errors. Don’t. It silences the warning and leaves you open to anyone on the path reading your traffic, including whoever operates the proxy.

A TLS error through a proxy usually means the proxy is intercepting rather than tunnelling. That is worth investigating, not suppressing.

Checking it works

python
r = requests.get("https://api.ipify.org", proxies=proxies, timeout=10)
assert r.text.strip() != requests.get("https://api.ipify.org", timeout=10).text.strip()

If both return the same address, your requests are not going through the proxy at all. It is usually a typo in the dict keys.

For choosing a proxy type, see residential vs datacenter. For sizing a pool, how many proxies you need.

Send your first request in the next five minutes.

Create an account, pick a plan, and get your proxies set up automatically after payment. No lock-in, and nothing renews unless you switch it on.

Pay with cards or crypto

  • Visa
  • Mastercard
  • Bitcoin
  • Ethereum
  • USDT

Opens chat from Tawk.to (US). It stores cookies in your browser. Cookie Policy