The Data Collection Handbook · Part II. Getting the Data

Chapter 4. Network Identity: Proxies and IPs

David Martin Riveros · Published August 1, 2026

A site decides how much to trust you before it reads a single thing you asked for. The request has not been parsed, the path does not matter yet, no cookie has been checked. The first and cheapest signal the server has is the address the connection came from, and that address already carries a story: who owns it, what kind of network it lives on, and whether traffic from its neighborhood has behaved. Chapter 3 chose the door. This one is about the address you knock from, because to the site the same request from two different addresses is two different events.

The last chapter ended here. A collector honoring Hacker News's published pace was refused anyway, for too many requests arriving from one address too quickly; the door and the software had nothing to do with it. Fix that and you have fixed the most common reason good collectors fail, which is usually an address that ran out of welcome rather than any clever detection.

Your address is a reputation

Every address on the internet belongs to a block that is registered to someone, and that registration is public. Anyone, a site included, can look up an address and learn in milliseconds whether it belongs to a data center full of servers, a home internet provider, or a mobile phone carrier. Those three answers carry very different trust, and the reason is who else is standing near you.

A proxy, for the record: an intermediary that forwards your request so the site sees the proxy's address instead of yours. It is how a collector controls the address it appears to come from, which is the one thing about itself the site judges first.

Start at the bottom. A data center address is a server's address, and a server does not shop for espresso machines. Sites know the big cloud providers' address ranges the way you know your neighbors' cars, so traffic from them is visibly automated before it does anything suspicious. Some of those ranges are already on public blocklists from past abuse, which makes them worse than useless: refused on sight. Clean data center addresses, ones with no bad history, still announce themselves as data center addresses. They are cheap and fast and openly machines, which is fine for sites that do not care and fatal for sites that do.

Climb to residential and the calculation inverts. A residential address is a home internet connection, the same kind your own router has, rented through a provider that routes your request out through a real household's line. To the site's address lookup it reads as a household's connection, because it is one, borrowed by the second. That does not make your traffic invisible: address-intelligence services flag known proxy pools, some residential inventory is really provider address space hosted in data centers, and the request pattern is still a machine's. What you are paying for is the collateral, because blocking that address risks blocking a real customer. These are priced by the gigabyte, because bandwidth through someone's home is the scarce thing.

At the top sits mobile. A mobile address belongs to a phone carrier, and carriers do something that makes their addresses radioactive to block: they share one address among many subscribers at once, tens to a few hundred, a design called carrier grade NAT. Block that address and you have blocked real customers along with it. Sites still challenge and rate-limit mobile traffic constantly; what they hesitate to do is hard-block the address, which is why mobile is the most trusted and the most expensive rung, kept for the sources that fight hardest.

The ladder ranks collateral damage, not quality, and reaching for the top rung by reflex is how collection budgets die. A site loses little by blocking a data center and cannot afford to block a carrier, and everything you pay climbing the ladder is buying that asymmetry. Which rung a job needs is set by how hard its sources look, not by how much you want to be safe. Most catalog and pricing work runs fine on clean data center addresses. You move up only when a specific source starts refusing them, and Chapter 3's audit is where you find out whether it does.

And move up per source, never per platform. The expensive mistake I keep meeting is a portfolio where one hard source forced residential, and then every easy source in the fleet rode the same pool, paying the hard source's rate for traffic a data center address would have carried. Route each source at the lowest rung it accepts and the bill takes the shape of Chapter 3's waterfall: the pricey addresses touch only the traffic that needs them. One routing table, checked once a quarter, keeps the proxy line tracking difficulty instead of habit.

Rotation and stickiness

Once you have addresses, you decide how to spend them, and there are two opposite ways. Rotation means using a different address for each request, spreading your traffic so no single address carries enough of it to look unusual. Stickiness means pinning one address to one line of work and keeping it there. The mistake that breaks more collectors than any detection system is using the first when the job needs the second.

The deciding question is whether the work has memory. Many collection tasks do not: fetch this product, fetch that one, and the two requests share nothing, so it does not matter that they came from different addresses. Rotation is free coverage there, and you should use it. But the moment a task spans several requests that the site expects to come from one visitor, a search that sets a session, a result you open, the next page of listings, memory appears, and rotation turns from an asset into a tell.

Picture the session that fails. Your collector lands on a site, which sets a session cookie, a small token the site issues to recognize the same visitor across requests. The collector searches, opens a result, asks for the next page. Run that with rotation on and each of those four requests leaves from a different address. From the site's chair, one visitor was just handed a session token in California and used it well under a second later from Frankfurt and again from Singapore. No human moves like that. Most sites do not tie a session to one address, because real phones hop between Wi-Fi and cellular all day, but continent-scale travel inside a second is the impossible journey that gets a session challenged or reset, and the collector, which was being polite in every other respect, looks exactly like the thing detection is built to catch. Stickiness fixes it by keeping all four steps on one address, so the session's travel history is the boring straight line a real person produces.

So the rule has two halves that people collapse into one and get wrong. Rotate between independent units of work, and stay sticky within any unit that carries session state. Rotation spreads the traffic and breaks the session; stickiness holds the session and concentrates the traffic. Sites that hand out session state have already chosen for you: stay coherent, or the session dies. Providers sell stickiness as a timed lease on an address, hold this one for ten minutes, and matching that lease to how long your session actually runs is a real part of the design, not a checkbox.

Mismatch that lease and you pay for the work twice. A lease that expires mid session hands the site an impossible visitor, the session dies, and the collector starts over: new lease, new session, the same pages fetched again, every byte of the retry billed at the same per gigabyte rate as the first attempt. On a residential pool that failure can be invisible in the error log, because every individual request succeeded. So before buying sticky sessions I time the real session, landing to last page, and buy the lease one comfortable step longer. It is the cheapest number in the proxy contract to get right.

When the country is part of the data

One more thing an address decides, and it is easy to miss because it does not look like a blocking problem. The country an address sits in can change the data itself. A price shown to a shopper in Germany is not the price shown in Brazil; currency, availability, tax, and the assortment on the shelf all shift with where the visitor appears to be. If your collector's addresses live in the wrong country, the site answers honestly and you collect the wrong truth. Nothing errors. The numbers are just not the ones your customer needs, which is the most expensive kind of wrong because nobody notices until someone downstream acts on it. When the deliverable is prices for a market, the address has to stand in that market, and that requirement sizes and shapes the pool before cost does.

Sizing the pool without guessing

Teams either do arithmetic here or pull a number from the air, and guessing is expensive in both directions. Too few addresses and the collector spends its day refused. Too many and you are renting reputation you never use. The good news is that a pool size falls out of three constraints you can each estimate, and the answer is the largest of them with a safety margin, never their sum.

The first constraint is concurrency. If your collector runs 30 requests at the same instant, those 30 need enough distinct addresses that no single one is carrying a suspicious pile of simultaneous connections. The second is the rate limit: each address tolerates only so many requests in an hour before the site starts refusing it, so your hourly volume divided by that per address budget sets a floor on how many addresses you need in rotation. The third is cooldown. When an address does get refused, it is not out for a moment, it is out for a ban, often an hour, and while it sits in the penalty box the remaining addresses have to carry the whole load. A fast rotation through addresses that keep getting burned needs a deep enough bench to cover everyone in timeout.

Take a concrete job: 100,000 requests a day, 30 in parallel, each address good for about 80 requests an hour, a one hour ban when one is burned. Concurrency asks for 42 addresses, the 30 parallel lines with 1.4 times headroom so no address carries two at once. The rate limit asks for 52, and the cooldown, with a one hour ban against a per hour budget, lands on the same 52, because those two floors only separate when a ban outlasts the rate window. So the binding constraint is 52 and a 1.5 buffer lifts the working answer to 78. That buffer covers three facts: the inputs are estimates, real addresses do not all tolerate the same load, and the day you undersize is the day traffic spikes. What the buffer is really buying is what the experiment tests, rather than taking it on trust.

There is a budget rule hiding in this formula. When a collector starts getting refused, the requests that land on my desk are usually purchases: a higher rung, a bigger pool, a new vendor. I make the first response arithmetic instead. Re-derive the pool size from the three constraints, re-probe the pace the source actually enforces, and only then price an upgrade. Most refusals I have debugged were an address short or a pace too hot, both fixable in an afternoon for nothing. A rung upgrade approved on a bad diagnosis becomes permanent, because nobody ever walks spending back down the ladder.

What undersizing actually costs

I have watched people treat pool size as a dial where a bit less money means a bit less coverage, shave the pool by a fifth to save on the proxy bill, and expect to lose a fifth of the data. The brief for this chapter carried that instinct as its hypothesis, written down before the run: the formula and a simulation would agree on the pool size, and below the right size, success would collapse fast rather than slope down gently.

The experiment is a simulation of one collection day. Requests arrive at random moments through the day, never more than the set concurrency at once. Each address carries its own tolerance, drawn a little above or below the average so the model is not pretending every address is identical, and the collector does the ordinary thing: it rotates evenly, stops using an address the instant a request on it is refused, and takes it back when the ban lifts. It never knows any address's true budget in advance. It learns, the way a real collector learns, by being refused. Then the simulation sweeps pool size from far too small to comfortably large, runs several seeded days at each size so the number is not a fluke, and records what share of the 100,000 requests got through. It is fully offline and seeded, so python run.py reproduces every figure here exactly.

The finding is the shape of the curve. At a pool of 40, the collector served 65 percent of its requests. At 56, four fifths. At 66, still under 88. Then between 68 and 70 it jumped from 98 percent to a clean 100 and stayed there for every larger pool. It does not ramp. It has a cliff, and the top of the cliff is a knee you can point at. The simulation's knee landed at 70 addresses. The formula, from the other direction and knowing none of the simulation's internals, said 78. Those two numbers agreeing, one from closed-form arithmetic and one from a seeded day of simulated refusals, is the chapter's main result: the formula lands just past the knee, which is exactly where you want a safety buffer to put you, on the flat ground above the cliff rather than on its edge.

The next numbers kill the dial idea. Shave the pool by a hair, from the knee of 70 down to 66, four addresses out of seventy, and you do not lose four addresses' worth of throughput. The collector falls from a clean 100 percent to under 88, twelve thousand of the day's hundred thousand requests gone for a six percent trim, more than double the loss a straight line would have predicted. Keep cutting and it keeps hurting: under 80 percent by the time the pool is down to 56. The mechanism is a feedback loop, and once you see it you stop trusting linear intuition near a rate limit forever. Too few addresses means each one works harder, working harder trips more bans, every banned address dumps its share onto the survivors, and the survivors, now busier, trip their own bans faster. Right below the knee the system drops off a step. That is why the buffer exists, and why the safe move is to sit above the knee where a bad day costs you nothing, not to trim toward it to save a few dollars and meet that step on the worst possible afternoon.

Carry these numbers off with two limits attached. The knee sits where it does for this job's exact inputs; change the per address tolerance or the ban length and it slides, which is why the formula takes them as inputs rather than baking in a constant. Part of what the 1.5 buffer is covering is the spread between the weakest address and the average one, since the knee is set by the addresses that tolerate least; a pool with a wider spread needs a wider buffer, and the parameters file is where you test your own. And a simulation is a clean-room argument. It proves the shape of the failure, the cliff and the feedback loop, not the precise percentage your real source will show, because a real detector is messier than a per hour counter. What transfers is the lesson rather than the decimals: near a rate limit, undersizing is a trapdoor you reach by chasing a discount, and the arithmetic that finds the knee is cheaper than the afternoon you would spend underneath it.

The addresses are handled. But an address is only the first thing a site reads about you, and a clean address running a collector that in every other respect screams robot will still get caught.