The Data Collection Handbook · Part II. Getting the Data

Chapter 5. Looking Human: Fingerprints and Anti-Bot Systems

David Martin Riveros · Published August 2, 2026

The first time a site blocked a collector I had spent a week perfecting, I took it personally. I had matched every header a real browser sends, in the right order, down to the ones nobody thinks about. The site refused me anyway, instantly, on the first request. That is the day I learned the thing this chapter is about: a site starts judging you before you say anything, from the shape of the handshake your software makes to open the connection. The headers I had labored over were the wrong layer. The tell was underneath them.

Chapter 4 was about where you connect from. This chapter is about what your connection looks like once it opens, and why a request can be refused for how it was made rather than for anything it asked. A short honesty note first, the same one from Chapter 2: this is about collecting public data at fair load. It explains how detection works at the level of principle, so you can send traffic that is what it claims to be. It is not a recipe for defeating any particular company's product, and the durable strategy here turns out to be coherence, not evasion, which is a happier thing to have to teach.

Why sites do this at all

Put yourself on the other side for a moment, because the whole field makes more sense from there. You run a site. Your servers are sized for the humans you expect. One morning a collector that did not size itself points a thousand requests a minute at you, your pages slow down for real customers, and your on-call engineer's phone goes off. You cannot tell, from any single request, whether it is a shopper or a script. So you start looking for the difference, and detection is born, not out of malice toward collection but out of the ordinary need to keep a service standing.

That framing decides the entire posture of this chapter. The goal is not to defeat the site's defenses, which is a fight that escalates forever and that you lose the moment your traffic degrades their service. The goal is to be traffic they have no reason to stop: coherent, believable, and light. Detection is looking for things that do not add up, and the professional move is to not add up to a problem.

Three gates, not one

What people call anti-bot is really three inspections stacked on top of each other, and they read completely different things. A site may lean on any one of them, and most collectors that fail were caught at the one they were not thinking about.

The first gate is the network handshake, and it is the one that surprises people, so it gets the experiment later in this chapter. Before any web request is sent, your software and the server perform a short negotiation to set up the encrypted connection, called a TLS handshake, TLS being the protocol behind the padlock in a browser. That negotiation has a shape: which encryption options are offered, in what order, with which settings. Different software makes that negotiation differently, and the difference is stable enough to fingerprint. Your HTTP library has an accent, and it speaks before you have said a word.

The second gate is the browser environment. If the site serves a page with JavaScript, small programs that run inside the browser, that code can look around the room it is running in: the screen size, the fonts it can detect, the exact quirks of how this engine draws and calculates. A real browser answers those questions one way. A script pretending to be a browser, or a stripped-down automated browser, answers them wrong or cannot answer at all, and the gaps are the tell.

The third gate is behavior, and it needs more than one request to see anything. It watches the rhythm and the route: how long between requests, in what order pages get visited, whether the timing has the small irregularity of a hand or the metronome regularity of a loop. One request has no rhythm to read. A thousand show it plainly.

The experiment: what the socket already knows

I wanted to see the first gate with my own eyes rather than describe it from memory, so this chapter's experiment is a mirror held up to three clients. There are public services that echo back exactly what they saw of your connection, the handshake shape and all, and I sent the same simple request to one of them from three different clients: a plain Python HTTP library, an impersonating library built to mimic a real browser's handshake, and a real Chrome browser driven by automation. Same request, same destination. The only variable is who is knocking. Everything below comes from that capture, taken on the last day of July 2026, and it ships with the chapter so python run.py reproduces it.

The plain Python client gave itself away at every level at once, which is the point. It spoke an older version of the web's transport protocol, HTTP/1.1, while both the impersonator and the real browser spoke a newer one, HTTP/2. Its handshake offered a different set of encryption options, in a different arrangement, producing a network fingerprint with no resemblance to a browser's. And it announced itself, in plain text, as python-requests. A site does not need clever detection for this. Two of those three tells, the protocol version and the shape of the handshake, are set before the request's headers are read at all. The user-agent is the third, read the moment a site parses those headers. My week of perfect headers would have been read, if it was read at all, only after the connection had already been judged by the client that opened it.

Then the impersonating client, and here the data did something better than prove the obvious. The library exists to wear a real browser's handshake, and it wore it well. It spoke HTTP/2, presented the same list of encryption options as the real Chrome, and, tested across three connections, it varied part of its handshake each time exactly the way modern Chrome does. That last point matters more than it looks. Today's Chrome deliberately shuffles a piece of its handshake on every connection, so the older style of fingerprint, called JA3, the one that hashes the handshake's exact byte order, changes from one Chrome connection to the next. I had half-expected the impersonator to betray itself by being too consistent, by faking a single frozen fingerprint while real Chrome shimmered. It did not. It shuffled too. The naive tell I was ready to write about was not there, and the honest version of this chapter had to follow the data instead.

But the data did leave one seam, and finding it is the real lesson. The newer, sturdier fingerprint, JA4, is built to see through that shuffling: it sorts the shuffled parts before hashing, so the per-connection reshuffle stops changing it the way it changes JA3. Read the three clients that way and a gap appears that the shuffle was hiding. The impersonator matched real Chrome's encryption options exactly, the two share an identical cipher fingerprint, but the part of the fingerprint that summarizes the handshake's extensions still differed from genuine Chrome's. And the real browser did something the impersonator did not: across three connections its sorted fingerprint itself shifted slightly, flickering between two neighboring values as Chrome included one more or one fewer element in its handshake, while the impersonator held to a single fixed value. Genuine software varied here in a way the copy did not. The copy was excellent. It was not identical, and at this depth the first gate is not checking for a byte-perfect match to one browser. It asks whether the handshake belongs to the family it claims, and a residue like this says it does not.

I want to be careful about what this proves, because the temptation is to over-read it. It does not prove the impersonator gets caught in practice; in my experience most sites do not inspect at this depth, and against most of the web a good impersonating client sails through the first gate. What it proves is narrower and more useful: the discriminating information is present at the socket, before headers, before cookies, before behavior. The plain client is separable from a browser in the opening handshake, decisively. The impersonator is separable too, if the inspector looks closely enough, by a residue it did not know to hide. The first gate is not a formality you satisfy with a user-agent string. It is a real inspection with real signal in it, and whatever story you intend to tell has to be true down there or it is not true at all.

Coherence is the actual principle

Here is the mistake that follows people through this field for years. They treat detection as a checklist of tricks to satisfy: fix the user-agent, add the headers, match the fingerprint, and imagine that stacking enough correct details eventually adds up to invisible. It does not, because detection cares less about whether each detail is correct than about whether the details agree. The contradiction between them is its loudest signal.

A request tells a story on every layer at once. The handshake says what software you are. The user-agent says what software you claim to be. The address says what kind of connection you are on. The country says where you are. The timing says whether a hand or a loop is driving. Detection lines those up and looks for the pair that disagrees. A browser user-agent riding a Python handshake is the classic disagreement: the request announces itself as Chrome and then shakes hands like a script, and the contradiction is louder than either signal alone would have been. The handshake alone already gave the client away. Wearing a Chrome user-agent on top did not hide that. It added a claim the socket beneath refused to back, and two signals that cannot both be true are easier to catch than one that is merely unusual.

This is why coherence beats a bag of tricks, and it is genuinely good news for someone collecting legitimately. You are not trying to assemble a perfect fake from parts. You are trying to be one consistent thing all the way down. If you present as a browser, be a browser at the socket too, which is what the impersonating libraries and real automated browsers from the experiment are for. If you present as residential traffic from Germany, connect from a residential address in Germany, the Chapter 4 decision reaching up into this one. If you present as a person reading pages, do not read them on a metronome. The layers stop contradicting each other because they are all telling the same true story, and a true story is much less work to maintain than a web of matched lies.

None of this means you have to look like a person, or like a browser at all. Coherence is only the layers agreeing. A client that is openly a script, never dressed up as Chrome, kept light and asking for public data plainly, tells a true story too, and usually the easiest one to keep. Standing out as a script is not the same as being blocked. Save a real browser for pages that genuinely need one, since that path costs far more to run and keep working.

Anti-patterns that fingerprint you instantly

A few specific contradictions show up so often they are worth naming, because each one is a single detail that undoes everything else.

The impossible value is the first. A real site's own interface only ever sends certain values, because a human clicking it can only produce certain values. When a collector sends a request asking for nine million results per page because it wanted everything in one shot, it has sent a number no human interface would ever generate, and that single field marks it more reliably than any handshake. The fix is discipline: send values the real interface sends, derived from what the site actually offered, never a round sentinel you invented for convenience.

The metronome is the second. Nothing that is really a person reading pages produces perfectly even gaps. A human's timing is lumpy, distracted, irregular. A collector that fetches every 2.0 seconds exactly is not blending into human traffic, it is signing every request with a ruler. Vary the timing, and vary it because you actually should be slower, not just to look random.

The crowd of one is the third, and it is the Chapter 4 lesson wearing this chapter's clothes. One address presenting a hundred different browser identities in an hour is a contradiction in space the way the metronome is a contradiction in time: no single home connection is a hundred different people at once. Identities and addresses have to move together, or the mismatch between them becomes the signal.

When you are blocked anyway

Even coherent, respectful collection gets blocked sometimes, and how you treat that block decides whether your data is trustworthy. The one rule that matters: a block is not data. When a request comes back challenged, throttled, or refused, you have learned nothing about the thing you were collecting, only that this attempt did not land. A block does not always look like one. Sometimes it comes back as an ordinary success that carries a challenge page or an empty shell where the data should be, and that is still not data. The unforgivable error, the one that quietly poisons datasets, is to record the block as a result, to let a challenge page become a price of zero or a product marked not found. Chapter 8 is about catching exactly that. For now, hold the line that a block is an ambiguous non-answer to be retried or escalated under the back-off discipline Chapter 6 builds, never a fact to be written down.

That is the whole posture of professional collection in one sentence. Be coherent so you are rarely blocked, respectful so a block is rare and recoverable, and honest so that when a block does come, it never gets mistaken for the truth you were sent to collect. The mechanics of staying up when things go wrong, timeouts, retries, backing off, are the next chapter.