The Data Collection Handbook · Part II. Getting the Data
Chapter 4. Network Identity: Proxies and IPs
In progress: this is the chapter plan. The full chapter, its charts, and its runnable experiment publish on this page.
What this chapter answers
Before a website reads a single byte of your request, it has already judged the address it came from. This chapter explains how the internet's address space is classified, which proxy class fits which job, and how to size a proxy pool from three constraints instead of buying on instinct.
What you will learn
- Why sites treat IP addresses as identity: every address publicly reveals whether it belongs to a datacenter, a home internet provider, or a mobile carrier, and sites trust those three groups very differently.
- The three proxy classes and their honest trade: cost ladder up, trust ladder up.
- Rotation versus sticky sessions, and when stickiness is mandatory.
- Geo-targeting: when the country of the exit address changes the data itself, as with prices and availability.
- The pool-sizing formula: concurrency, per-address rate limits, and cooldowns, take the maximum, add a buffer.
In this chapter
- Your IP is your reputation. How addresses are classified before your request is read.
- The three proxy classes. Datacenter, residential, mobile: what each buys you.
- Rotation vs stickiness. Why rotating mid-session breaks session-bound sites.
- Geo-targeting. When location changes the data, not just the access.
- Sizing the pool. The three-constraint formula, worked by hand, then simulated.
- Vendor hygiene. Multiple providers, credentials in a secrets manager, continuous quality measurement.
The experiment
The question: Given a target concurrency, a per-address rate limit, and a cooldown for overused addresses, what is the smallest pool that keeps success above 99 percent, and how sharp is the cliff below it?
A simulation that rolls the dice thousands of times: requests arrive at a set concurrency, each address tolerates a set rate, and exceeding it sidelines the address for a cooldown period. The code sweeps pool sizes, plots success against size, and compares the simulated answer with the closed-form formula. The data and code ship with the chapter, sized to run offline on a laptop with one command.
My hypothesis is that the formula and the simulation agree, and that below the knee of the curve success collapses fast: undersizing a pool by twenty percent does not cost twenty percent of results, it costs most of them.