The Data Collection Handbook · Part II. Getting the Data

Chapter 4. Network Identity: Proxies and IPs

David Martin Riveros · Published August 1, 2026

A site decides how much to trust you before it reads a single thing you asked for. The request has not been parsed, the path does not matter yet, no cookie has been checked. All the server has so far is the address the connection came from, and that address already carries a story: who owns it, what kind of network it lives on, and whether traffic from its neighborhood has behaved. Chapter 3 was about which door you knock on. This chapter is about where you are standing when you knock, because the same request from two different addresses is, to the site, two different events.

The last chapter ended on exactly this. A collector honoring Hacker News's published pace was refused anyway, and the refusal was not about the door or the software. It was about too many requests arriving from one address too quickly. Fix that and you have fixed the most common reason good collectors fail: not clever detection, just an address that ran out of welcome.

Your address is a reputation

Every address on the internet belongs to a block that is registered to someone, and that registration is public. Anyone, a site included, can look up an address and learn in milliseconds whether it belongs to a data center full of servers, a home internet provider, or a mobile phone carrier. Those three answers carry very different trust, and the reason is not technical snobbery. It is about who else is standing near you.

A proxy, since the whole chapter turns on the word: an intermediary that forwards your request so the site sees the proxy's address instead of yours. It is how a collector controls the address it appears to come from, which is the one thing about itself the site judges first.

Start at the bottom. A data center address is a server's address, and a server does not shop for espresso machines. Sites know the big cloud providers' address ranges the way you know your neighbors' cars, so traffic from them is visibly automated before it does anything suspicious. Some of those ranges are already on public blocklists from past abuse, which makes them worse than useless: refused on sight. Clean data center addresses, ones with no bad history, still announce themselves as data center addresses. They are cheap and fast and honest about being machines, which is fine for sites that do not care and fatal for sites that do.

Climb to residential and the calculation inverts. A residential address is a home internet connection, the same kind your own router has, rented through a provider that routes your request out through a real household's line. To the site it is indistinguishable from a family streaming video, because it is a family's connection, borrowed by the second. Blocking it risks blocking a real customer, and that risk is exactly what you are paying for. These are priced by the gigabyte, because bandwidth through someone's home is the scarce thing.

At the top sits mobile. A mobile address belongs to a phone carrier, and carriers do something that makes their addresses radioactive to block: they share one address among thousands of subscribers at once, a design called carrier grade NAT. Block that address and you have blocked a slice of an entire city's phones. Almost no site is willing to do that, which is why mobile is the most trusted and the most expensive rung, kept for the sources that fight hardest.

The ladder is not a quality ranking, and reaching for the top rung by reflex is how collection budgets die. It is a map of collateral damage. A site loses nothing by blocking a data center and cannot afford to block a carrier, and everything you pay climbing the ladder is buying that asymmetry. Which rung a job needs is set by how hard its sources look, not by how much you want to be safe. Most catalog and pricing work runs fine on clean data center addresses. You move up only when a specific source starts refusing them, and Chapter 3's audit is where you find out whether it does.

And move up per source, never per platform. The expensive mistake I keep meeting is a portfolio where one hard source forced residential, and then every easy source in the fleet quietly rode the same pool, paying the hard source's rate for traffic a data center address would have carried. Route each source at the lowest rung it accepts and the bill takes the shape of Chapter 3's waterfall: the pricey addresses touch only the traffic that needs them. One routing table, checked once a quarter, is the difference between a proxy line that tracks difficulty and one that tracks habit.

Rotation and stickiness

Once you have addresses, you decide how to spend them, and there are two ways that are genuine opposites. Rotation means using a different address for each request, spreading your traffic so no single address carries enough of it to look unusual. Stickiness means pinning one address to one line of work and keeping it there. The mistake that breaks more collectors than any detection system is using the first when the job needs the second.

The deciding question is whether the work has memory. Many collection tasks do not: fetch this product, fetch that one, and the two requests share nothing, so it does not matter that they came from different addresses. Rotation is free coverage there, and you should use it. But the moment a task spans several requests that the site expects to come from one visitor, a search that sets a session, a result you open, the next page of listings, memory appears, and rotation turns from an asset into a tell.

Picture the session that fails. Your collector lands on a site, which sets a session cookie, a small token the site issues to recognize the same visitor across requests. The collector searches, opens a result, asks for the next page. Run that with rotation on and each of those four requests leaves from a different address. From the site's chair, one visitor was just handed a session token in California and used it thirty milliseconds later from Frankfurt and again from Singapore. No human moves like that. The session gets challenged or quietly reset, and the collector, which was being polite in every other respect, looks exactly like the thing detection is built to catch. Stickiness fixes it by keeping all four steps on one address, so the session's travel history is the boring straight line a real person produces.

So the rule has two halves that people collapse into one and get wrong. Rotate between independent units of work, and stay sticky within any unit that carries session state. Rotation buys anonymity by spending coherence. Stickiness buys coherence by spending anonymity. Sites that hand out session state have already made the choice for you: coherence wins, or the session dies. Providers sell stickiness as a timed lease on an address, hold this one for ten minutes, and matching that lease to how long your session actually runs is a real part of the design, not a checkbox.

Mismatch that lease and you pay for the work twice. A lease that expires mid session hands the site an impossible visitor, the session dies, and the collector starts over: new lease, new session, the same pages fetched again, every byte of the retry billed at the same per gigabyte rate as the first attempt. On a residential pool that failure can be invisible in the error log, because every individual request succeeded. So before buying sticky sessions I time the real session, landing to last page, and buy the lease one comfortable step longer. It is the cheapest number in the whole proxy contract to get right.

When the country is part of the data

One more thing an address decides, and it is easy to miss because it does not look like a blocking problem. The country an address sits in can change the data itself. A price shown to a shopper in Germany is not the price shown in Brazil; currency, availability, tax, and the assortment on the shelf all shift with where the visitor appears to be. If your collector's addresses live in the wrong country, the site answers honestly and you collect the wrong truth. Nothing errors. The numbers are just quietly not the ones your customer needs, which is the most expensive kind of wrong because nobody notices until someone downstream acts on it. When the deliverable is prices for a market, the address has to stand in that market, and that requirement sizes and shapes the pool before cost does.

Sizing the pool without guessing

Here is where teams either do arithmetic or pull a number from the air, and the air is expensive in both directions. Too few addresses and the collector spends its day refused. Too many and you are renting reputation you never use. The good news is that a pool size falls out of three constraints you can each estimate, and the honest answer is the largest of them with a safety margin, not their sum.

The first constraint is concurrency. If your collector runs 30 requests at the same instant, those 30 need enough distinct addresses that no single one is carrying a suspicious pile of simultaneous connections. The second is the rate limit: each address tolerates only so many requests in an hour before the site starts refusing it, so your hourly volume divided by that per address budget sets a floor on how many addresses you need in rotation. The third is cooldown. When an address does get refused, it is not out for a moment, it is out for a ban, often an hour, and while it sits in the penalty box the remaining addresses have to carry the whole load. A fast rotation through addresses that keep getting burned needs a deep enough bench to cover everyone in timeout.

Take a concrete job: 100,000 requests a day, 30 in parallel, each address good for about 80 requests an hour, a one hour ban when one is burned. Concurrency asks for 42 addresses, the rate limit and the cooldown each ask for 52, so the binding constraint is 52 and a 1.5 buffer lifts the working answer to 78. That buffer is not padding for its own sake. It is the honest admission that the three inputs are estimates, that real addresses do not all tolerate the same load, and that the day you undersize is the day traffic spikes. The chapter's experiment is about what that buffer is really buying, so it gets tested rather than trusted.

There is a budget rule hiding in this formula. When a collector starts getting refused, the requests that land on my desk are usually purchases: a higher rung, a bigger pool, a new vendor. I make the first response arithmetic instead. Re-derive the pool size from the three constraints, re-probe the pace the source actually enforces, and only then price an upgrade. Most refusals I have debugged were an address short or a pace too hot, both fixable in an afternoon for nothing. A rung upgrade approved on a bad diagnosis becomes permanent, because nobody ever walks spending back down the ladder.

What undersizing actually costs

I have watched people treat pool size as a dial where a bit less money means a bit less coverage, shave the pool by a fifth to save on the proxy bill, and expect to lose a fifth of the data. The brief for this chapter carried that instinct as its hypothesis, written down before the run: the formula and a simulation would agree on the pool size, and below the right size, success would collapse fast rather than slope down gently.

The experiment is a simulation of one collection day. Requests arrive at the set concurrency and spread across the day. Each address carries its own tolerance, drawn a little above or below the average so the model is not pretending every address is identical, and the collector does the ordinary thing: it rotates evenly, stops using an address the instant a request on it is refused, and takes it back when the ban lifts. It never knows any address's true budget in advance. It learns, the way a real collector learns, by being refused. Then the simulation sweeps pool size from far too small to comfortably large, runs several seeded days at each size so the number is not a fluke, and records what share of the 100,000 requests got through. It is fully offline and seeded, so python run.py reproduces every figure here exactly.

The shape of the answer is the finding, and it is not a slope. At a pool of 40, the collector served 65 percent of its requests. At 56, four fifths. At 66, still under 88. Then between 68 and 70 it jumped from 98 percent to a clean 100 and stayed there for every larger pool. The curve does not ramp; it has a cliff, and the top of the cliff is a knee you can point at. The simulation's knee landed at 70 addresses. The formula, from the other direction and knowing none of the simulation's internals, said 78. Those two numbers agreeing, one from closed-form arithmetic and one from a seeded day of simulated refusals, is the chapter's main result: the formula lands just past the knee, which is exactly where you want a safety buffer to put you, on the flat ground above the cliff rather than on its edge.

Now the part that kills the dial thinking. Shave the pool by a hair, from the knee of 70 down to 66, four addresses out of seventy, and you do not lose four addresses' worth of throughput. The collector falls from a clean 100 percent to under 88, twelve thousand of the day's hundred thousand requests gone for a six percent trim, more than double the loss a straight line would have predicted. Keep cutting and it keeps hurting: under 80 percent by the time the pool is down to 56. The mechanism is a feedback loop, and once you see it you stop trusting linear intuition near a rate limit forever. Too few addresses means each one works harder, working harder trips more bans, every banned address dumps its share onto the survivors, and the survivors, now busier, trip their own bans faster. Right below the knee the system does not degrade gently, it drops off a step. That is why the buffer exists, and why the safe move is to sit above the knee where a bad day costs you nothing, not to trim toward it to save a few dollars and meet that step on the worst possible afternoon.

Two honest limits before you carry these numbers off. The knee sits where it does for this job's exact inputs; change the per address tolerance or the ban length and it slides, which is the whole reason the formula takes them as inputs rather than baking in a constant. And a simulation is a clean-room argument. It proves the shape of the failure, the cliff and the feedback loop, not the precise percentage your real source will show, because a real detector is messier than a per hour counter. What transfers is the lesson, not the decimals: near a rate limit, undersizing is not a discount, it is a trapdoor, and the arithmetic that finds the knee is cheaper than the afternoon you would spend underneath it.

The addresses are handled. But an address is only the first thing a site reads about you, and a clean address running a collector that in every other respect screams robot will still get caught. What the rest of that first impression is made of, and how detection reads it, is Chapter 5.