Most proxy code has one error path — catch everything, rotate, retry — and that single path is why pools burn out and targets escalate. A failed proxied request is one of five different events: the proxy itself was unreachable; the proxy refused your credentials; the target refused this IP; the target rate-limited you; or the target answered 200 with a challenge page instead of content. Rotating helps with exactly one of those. Retrying immediately helps with none. This guide classifies the failures, then builds a compact Python retry layer that responds to each correctly, with exponential backoff, rotation only where it applies, and a circuit breaker so a target that has said “stop” is actually left alone. The snippets use requests; the logic ports directly to httpx and aiohttp (parameter names in Proxies in Python).
The five failures
| # | What happened | How it shows up in Python | Right response |
|---|---|---|---|
| 1 | Proxy unreachable — wrong host/port, provider outage, network | ProxyError wrapping a connection error; ConnectTimeout before any HTTP | Retry with backoff; after N failures mark the proxy endpoint down and alert. Rotation within the same gateway will not help if the gateway is down. |
| 2 | Proxy rejected you — bad credentials, IP not whitelisted, plan exhausted | ProxyError with 407; sometimes provider-specific 4xx from the gateway | Do not retry. Fix configuration; retrying a 407 in a loop can get your account throttled by the provider. |
| 3 | Target refused this IP — reputation, ASN, geo block | 403, sometimes 401/451; occasionally a connection reset | Rotate to a new exit (new session ID) and a fresh cookie jar; retry once or twice; if it persists across IPs, the problem is fingerprint or type, not the IP — stop and re-evaluate (why). |
| 4 | Target rate-limited you | 429, or 503 with Retry-After | Sleep Retry-After (or backoff) on this exit; then continue on it. Rotating instead of waiting spreads the block. See rate-limit signals. |
| 5 | Challenge served as 200 — CAPTCHA/interstitial page | Status 200 but body is not the expected content | Treat as failure type 3. Detect it by content (marker text, missing expected element, body too small), never by status alone. |
| — | Transient network / target 5xx | ReadTimeout, ConnectionError mid-stream, 500/502/504 | Retry with backoff on the same exit; these are usually not about you. |
The table is the whole article; everything below is making a program obey it.
Classify first
import requests
from enum import Enum
class Fail(Enum):
NONE = 0; PROXY_DOWN = 1; PROXY_AUTH = 2; IP_BLOCKED = 3; RATE_LIMITED = 4; CHALLENGE = 5; TRANSIENT = 6
CHALLENGE_MARKERS = (b"captcha", b"cf-chl", b"challenge-platform", b"datadome", b"px-captcha")
def classify(resp: requests.Response | None, exc: Exception | None, min_bytes: int = 2000) -> Fail:
if exc is not None:
if isinstance(exc, requests.exceptions.ProxyError):
return Fail.PROXY_AUTH if "407" in str(exc) else Fail.PROXY_DOWN
if isinstance(exc, (requests.exceptions.ConnectTimeout,)):
return Fail.PROXY_DOWN # never reached the proxy (or it never answered CONNECT)
if isinstance(exc, (requests.exceptions.ReadTimeout, requests.exceptions.ConnectionError)):
return Fail.TRANSIENT
return Fail.TRANSIENT
s = resp.status_code
if s == 429 or (s == 503 and "Retry-After" in resp.headers):
return Fail.RATE_LIMITED
if s in (401, 403, 451):
return Fail.IP_BLOCKED
if s >= 500:
return Fail.TRANSIENT
if s == 200:
body = resp.content[:20000].lower()
if len(resp.content) < min_bytes or any(m in body for m in CHALLENGE_MARKERS):
return Fail.CHALLENGE
return Fail.NONE
return Fail.TRANSIENT
Tune CHALLENGE_MARKERS and min_bytes per target: the reliable check is “does the page contain the element I came for?” — a product title, a JSON key — and its absence is the challenge signal.
Backoff, done properly
Exponential backoff with jitter: doubling delays spread out retries; jitter stops a hundred workers from retrying in lockstep. Honour Retry-After when it exists.
import random, time, email.utils
def backoff_delay(attempt: int, base: float = 1.0, cap: float = 60.0) -> float:
return random.uniform(0, min(cap, base * (2 ** attempt))) # "full jitter"
def retry_after_seconds(resp: requests.Response | None) -> float | None:
if resp is None or "Retry-After" not in resp.headers:
return None
v = resp.headers["Retry-After"].strip()
if v.isdigit():
return float(v)
dt = email.utils.parsedate_to_datetime(v) # HTTP-date form
return max(0.0, dt.timestamp() - time.time()) if dt else None
A circuit breaker per (target, exit)
When a target has refused an exit several times in a row, stop offering that exit to that target for a while instead of discovering the block on every request. Same idea for a proxy endpoint that is down.
import time
from collections import defaultdict
class Breaker:
def __init__(self, threshold=3, cooldown=300):
self.fail = defaultdict(int); self.open_until = {}
self.threshold, self.cooldown = threshold, cooldown
def allow(self, key) -> bool:
return time.time() >= self.open_until.get(key, 0)
def ok(self, key):
self.fail[key] = 0
def bad(self, key):
self.fail[key] += 1
if self.fail[key] >= self.threshold:
self.open_until[key] = time.time() + self.cooldown
self.fail[key] = 0
Putting it together
import secrets
GATE, PORT, PASS = "gate.provider.com", 7777, "PASSWORD"
breaker = Breaker()
def proxies_for(session_id: str) -> dict:
url = f"http://customer-session-{session_id}:{PASS}@{GATE}:{PORT}"
return {"http": url, "https": url}
def fetch(url: str, host: str, max_attempts: int = 5, timeout=(10, 30)) -> requests.Response | None:
sid = secrets.token_hex(4)
sess = requests.Session()
for attempt in range(max_attempts):
key = (host, sid)
if not breaker.allow(key) or not breaker.allow(("gateway", GATE)):
sid = secrets.token_hex(4); sess = requests.Session(); continue # pick a different exit
resp = exc = None
try:
resp = sess.get(url, proxies=proxies_for(sid), timeout=timeout)
except requests.RequestException as e:
exc = e
kind = classify(resp, exc)
if kind is Fail.NONE:
breaker.ok(key); breaker.ok(("gateway", GATE)); return resp
if kind is Fail.PROXY_AUTH:
raise RuntimeError("proxy rejected credentials/whitelist — fix config, not retry")
if kind is Fail.PROXY_DOWN:
breaker.bad(("gateway", GATE)); time.sleep(backoff_delay(attempt, base=2)); continue
if kind is Fail.RATE_LIMITED:
time.sleep(retry_after_seconds(resp) or backoff_delay(attempt, base=5)); continue # same exit
if kind in (Fail.IP_BLOCKED, Fail.CHALLENGE):
breaker.bad(key)
sid = secrets.token_hex(4); sess = requests.Session() # new exit + fresh cookies
time.sleep(backoff_delay(attempt)); continue
# TRANSIENT
time.sleep(backoff_delay(attempt)); continue
return None
What this does that the naive loop does not: it never retries a 407; it waits on rate limits instead of hopping; it rotates only on IP-level refusal and challenges, and pairs every rotation with a fresh cookie jar (same cookies on a new IP is the same visitor); it stops offering an exit that keeps failing on a host; and it backs off with jitter everywhere. Wire Fail counts into your metrics — a rising IP_BLOCKED rate across different exits means the fingerprint or the IP type is wrong, and no retry policy will fix that (see TLS fingerprinting and choosing the IP type).
Using a library instead
tenacity gives you decorators for retry/backoff/stop conditions and is a good base — but keep the classification and the “rotate vs wait vs stop” decision in your own code; generic retry libraries treat all exceptions alike, which is precisely the mistake. urllib3's Retry object (mountable on a requests HTTPAdapter) handles connection-level retries and Retry-After for 429/503 well, and is worth using for the TRANSIENT and RATE_LIMITED classes; it cannot rotate your proxy or detect a challenge body, so it complements rather than replaces the layer above.
Async note
In aiohttp/httpx.AsyncClient the structure is identical with await asyncio.sleep(); the one addition is a per-host semaphore so that backing off on one host does not stall the others, and so concurrency per host stays polite. A global semaphore alone lets a hundred coroutines pile onto one target the moment it is the only one not sleeping.
Frequently asked questions
Should I rotate the proxy on every error?
No. Rotate on IP-level refusal (403-class) and challenge pages, always with a fresh cookie jar. Wait — do not rotate — on 429/Retry-After. Fix configuration on 407. Retry the same exit on transient errors and 5xx.
How many times should I retry?
Three to five attempts with exponential backoff and jitter is the usual envelope; beyond that you are either facing a persistent block (rotation limit reached) or an outage (breaker should be open). Cap the total wall time per request.
How do I detect a CAPTCHA page in code?
By content, not status: check for the element you expect (a title, a JSON key), for known challenge markers in the body, and for an implausibly small response. A 200 without your content is a failure.
What is a circuit breaker for proxies?
A per-(target, exit) counter that, after several consecutive failures, stops using that exit for that target for a cooldown period — so you do not rediscover the block on every request and do not keep hammering a host that refused you.
Why not just use tenacity or urllib3 Retry?
Use them for transient and rate-limit retries; but they cannot tell a 407 from a 403 from a challenge, cannot rotate your session ID, and cannot reset cookies. Keep that decision logic in your code.
What does a rising block rate across many IPs mean?
That the IPs are not the problem: the target is identifying you by TLS/HTTP fingerprint, headers or behaviour, or the IP type is wrong for that site. Stop rotating and fix the layer that is actually failing.