robots.txt is a text file in which a site asks crawlers to stay out of certain paths; rate limits are how a site enforces how fast anyone may ask; and the difference between a scraper that runs for months and one that burns through a proxy pool in an afternoon is mostly whether it respects both. This is not primarily an ethics point, though it is that too — it is an engineering one. Every IP you have carries a finite reputation budget with each target, and impolite scraping spends it faster than rotation can replenish it. This guide covers what the signals actually mean and the practices that make proxies last.

robots.txt: what it is and is not

The Robots Exclusion Protocol (formalised as RFC 9309 in 2022) lets a site publish, at /robots.txt, which user agents may fetch which paths:

User-agent: *
Disallow: /admin/
Disallow: /search
Crawl-delay: 5

User-agent: Googlebot
Allow: /

Sitemap: https://example.com/sitemap.xml

Three facts about it that matter for proxy users:

  • It is advisory. Nothing technical stops a client from fetching a disallowed path. Compliance is voluntary — which is exactly why sites treat non-compliance as a strong bot signal: a visitor that ignores robots.txt has announced that it is an automated client that does not follow conventions.
  • It is not a legal instrument by itself, but it is evidence. In disputes over scraping, a site's robots.txt is routinely cited as notice that automated access was not permitted; ignoring it weakens the “I had no reason to think this was unwelcome” position. See the legal map.
  • It tells you where the traps are. Paths listed under Disallow for * are frequently the ones a site instruments most heavily — search endpoints, faceted navigation, infinite calendars, admin areas. Fetching them is both the most detectable and the least useful thing a scraper can do.

Rules of thumb: fetch /robots.txt once per host per day and cache it; obey Disallow for your user agent and for *; honour Crawl-delay when present (it is non-standard but widely used — the number is seconds between requests from one client); read the Sitemap lines, which are the site handing you a clean, complete URL list so you do not have to crawl for one.

Rate limits: how sites say “slower”

Rate limits are enforced, and they come with signals if you listen:

SignalMeaningCorrect response
429 Too Many RequestsYou exceeded a limit (usually per IP, sometimes per session)Stop; wait at least the Retry-After value if given; back off exponentially if not
Retry-After: 120 (header)Do not retry for 120 seconds (or until the given date)Respect it exactly; retrying earlier extends the penalty on many systems
503 with Retry-AfterOverloaded or deliberately throttlingSame as above
403 after a run of 200sSoft block: reputation exhausted for this IP or sessionRotate IP and session; slow down before resuming
CAPTCHA / challenge page with status 200The anti-bot layer wants proof (see how it scores)Treat as a block, not a success; do not hammer through it
Response times climbing steadilyYou are being deprioritised, or you are the loadReduce concurrency
X-RateLimit-Remaining / -Reset headersExplicit budget, common on APIsPace to stay under; sleep until reset when it hits zero

The costly mistake is treating every non-200 as “rotate and retry immediately.” That converts one IP's temporary limit into a burned IP, then burns the next, and teaches the target's model that your fingerprint hops addresses when refused — a pattern anti-bot vendors specifically detect. Rotation is for spreading normal load, not for evading a limit you just tripped; see rotation strategy.

Practices that keep IPs alive

  1. Find the site's per-IP tolerance and stay under half of it. Measure it once (the method is in choosing a rotation interval) rather than guessing.
  2. Add jitter. Uniform intervals are the most machine-like signal there is. Randomise delays around a mean, and vary the order in which you visit URLs.
  3. Use the sitemap. If the site publishes one, it is a full inventory of what they want indexed; crawling it is both more polite and more complete than spidering links, and it avoids the trap paths.
  4. Cache and use conditional requests. Send If-Modified-Since / If-None-Match where the site supports them; a 304 costs a fraction of a full page — in their bandwidth, your proxy GB and your IP reputation.
  5. Do not fetch what you do not need. HTML without assets where possible; block images/fonts/media in browser automation. This alone can cut per-GB proxy cost by an order of magnitude (why) and reduces the load footprint the site sees.
  6. Prefer official APIs, feeds and data exports when they exist. Many sites offer RSS, JSON endpoints or bulk downloads precisely so that people stop scraping HTML.
  7. Identify yourself honestly when the target tolerates it. A descriptive User-Agent with a contact address is the norm for research and monitoring crawlers, and some sites whitelist known-good bots. Where you must present as a browser instead, do it consistently across all layers (why half-measures fail).
  8. Back off exponentially and cap concurrency per host, not just globally. Ten hosts at two connections each is polite; one host at twenty is not.
  9. Stop when told to stop. A Retry-After, an explicit block page, or a message from the site operator each change what you should do next; the last one changes the legal posture as well.

Why politeness is cheaper

Every one of the practices above reduces one of three costs: proxy spend (fewer bytes, fewer retries), IP burn (each address lasts longer, so you need fewer of them and can use cheaper types), and detection risk (a client that behaves like a considerate crawler scores better on the behavioural layer than one that behaves like a flood). Aggressive scraping is a strategy that spends money to look more suspicious. The exception people cite — a one-off pull where nothing matters afterwards — is real but rarer than it seems, because targets share intelligence and pools are shared with your future self.

A note on the ethics of load

Beyond IP economics: the site on the other end is someone's infrastructure bill. A scraper that fetches at a rate the site can absorb is indistinguishable from traffic; one that degrades the service for real users is, at minimum, rude, and at scale it edges into the territory the law calls trespass or interference. Crawl-delay and rate limits are the site's way of saying what it can absorb. Reading them is the whole of good manners here.

Frequently asked questions

Is robots.txt legally binding?

Not on its own; it is a convention. But it serves as notice of the site's wishes and is cited in disputes, and ignoring it is a strong automation signal to anti-bot systems. Treat it as binding in practice.

What does Crawl-delay mean?

A non-standard but common directive: the number of seconds a single crawler should wait between requests. Honour it per client (per IP/session), not per whole operation.

What should I do on a 429?

Stop hitting that host from that IP; wait the Retry-After value if given, otherwise back off exponentially; only then resume, slower. Do not immediately rotate and retry — that spreads the block.

Does rotating proxies let me ignore rate limits?

It lets each IP stay under a per-IP limit while your total throughput is higher. It does not make aggressive patterns invisible: fingerprint- and behaviour-based detection links the sessions, and burned IPs are still burned.

Should I identify my scraper in the User-Agent?

Where the site tolerates crawlers, yes — it is the honest default and sometimes gets you whitelisted. Where you must appear as a browser, be consistent across TLS, headers and behaviour rather than mixing an honest UA with browser impersonation.

How fast can I scrape a site?

As fast as it tolerates from one IP times the number of IPs you have — measured, not assumed — minus a safety margin, and never faster than its Crawl-delay or the pace at which response times stay flat.