The Data Collection Handbook · Part I. Foundations

Chapter 2. Legality, Ethics, and Fair Use

David Martin Riveros · Published July 29, 2026

Every founder in this industry hears the same question, usually within five minutes of explaining what the company does. Is this even legal? I have answered it on sales calls and at least once over a family dinner. The honest answer is conditional. It hangs on what you collect, how you collect it, and how much of it you take. This chapter walks those lines in the order a lawyer would draw them, then adds the part lawyers rarely cover: how to be a decent guest on infrastructure that someone else pays for.

One thing before we start. I am not a lawyer, and this chapter is not legal advice; it is how a practitioner works these lines after years of building collection systems that paying customers depend on. When a real dispute or a real contract is on the table, hire counsel. What I can give you is the map counsel will sketch in your first meeting, and the operating habits that keep the second meeting short. A note on this chapter's title while we are here: fair use in these pages means using the open web fairly, load and manners. Copyright's formal fair use doctrine is a usage-rights question, and usage rights live on the buyer's side of the table, in Chapter 14.

The map has three zones. Public pages, which anyone can see without an account. Gated content, which sits behind a login or a paywall. And personal data, which cuts across both, because a human being can be identifiable on either side of the wall. Most legal questions in this field reduce to knowing which zone you are standing in.

The login wall is the brightest line

Public means visible to anyone with a browser and no account. A product page, a price, a news article outside the paywall, a company's careers listing. Gated means the site asked for something before showing the content: an account or a payment. That wall, not the tool you fetch with, is the brightest legal line in this field.

The reason lives in a United States law from 1986 called the Computer Fraud and Abuse Act, the CFAA, written to punish breaking into computers. For two decades nobody knew for certain whether fetching a public web page that the site did not want you to fetch counted as "access without authorization" under that law. Being wrong meant a federal claim, potentially a criminal one, hanging over an ordinary HTTP request.

hiQ Labs v. LinkedIn is the case that tested it, and you should know the whole story, because half of it gets quoted everywhere and the other half gets forgotten.

hiQ was a small analytics company that scraped public LinkedIn profiles and sold employers predictions about which of their employees might quit. In 2017 LinkedIn sent hiQ a cease-and-desist letter demanding it stop, and began blocking its traffic. hiQ sued, and a federal district court in California ordered LinkedIn to stand down while the case ran. In 2019 the Ninth Circuit Court of Appeals upheld that order and said the words this industry has been quoting ever since: the CFAA's "without authorization" concept likely does not reach data that is public in the first place. The Supreme Court sent the case back in 2021 after Van Buren v. United States, its own decision reading the CFAA as a law about gates, up or down. In 2022 the Ninth Circuit reaffirmed. The CFAA guards gates. Where no gate is up, there is nothing to crash through.

That is the famous half. Here is the ending. Back in the district court, the case turned in November 2022 on a much older kind of claim: contract. The judge found on partial summary judgment, a ruling without a full trial, that hiQ had breached LinkedIn's User Agreement. hiQ held LinkedIn accounts, which meant it had accepted that agreement, which bans scraping and fake profiles, and hiQ had done both; what remained for trial were hiQ's fallback defenses and the damages. It never got there. In December 2022 the parties settled. hiQ accepted a judgment of $500,000 and a permanent injunction barring it from ever touching LinkedIn again. By then the company was, in every practical sense, already gone; years of litigation had drained its funding and its customers long before the final order.

So when someone tells you that hiQ established that scraping public data is legal, they are wrong twice. The Ninth Circuit held something narrower: collecting public data is likely not federal hacking. And the company that won that point still lost on contract, and did not survive to enjoy its own precedent.

Three working rules fall out of that story, and they carry different weight. The heaviest: never collect through accounts. An account is a signed contract. Courts have been reluctant to enforce terms buried behind a footer link against people who never saw them, the Ninth Circuit said as much in Nguyen v. Barnes & Noble in 2014, but nobody has that defense after clicking "I agree," and hiQ's case collapsed on exactly this point. Second, treat a cease-and-desist letter as a legal event rather than an inconvenience. The CFAA piece of that is narrower than folklore has it: in Facebook v. Power Ventures in 2016 the Ninth Circuit did find a company liable for pressing on after Facebook explicitly revoked its access, but the data there sat behind Facebook's logins, and the hiQ court later distinguished public pages on exactly that ground. What a cease-and-desist changes, on any data, is everything else: it revokes whatever welcome you could once assume, it strengthens the sender's contract and trespass claims from that day forward, and it marks the moment decisions move from your roadmap to your counsel. Third, and least discussed: being right can be ruinously expensive. hiQ won its argument and lost its company. The cheapest legal strategy in this business is to collect in a way nobody feels moved to sue over, which is what the rest of this chapter is about.

If you buy data rather than collect it, the hiQ ending is also your vetting script, because you just watched what legal cost does to a vendor's product before any verdict arrives. A vendor's legal exposure is your supply risk. Ask before you sign: do any accounts touch the collection path for my sources, has any of my sources sent a cease-and-desist, and what happens to my deliveries if one arrives mid-contract? A vendor that has planned for a dispute answers those without checking with anyone.

Personal data is different

Everything above was about property and contracts. Personal data answers to a third body of law, and that law follows the person, not the page.

A price is not personal, and neither is a product title. The moment a record identifies a human being, a name, an email address, a face, a profile handle, you have left the world of property arguments, and the fact that the data was public stops helping you. The strictest regime is the GDPR, the European Union's General Data Protection Regulation, applied since 2018. It covers any information relating to an identifiable person, and it reaches across borders: wherever your servers sit, monitoring people in Europe or offering them services brings you inside it, and collecting profiles at scale is monitoring. Publishing something does not waive its protection. Under the GDPR you need a documented lawful basis to collect personal data at all. People can demand to see what you hold and, under its Article 17, demand deletion. Article 14 goes further than most engineers expect: when you collect someone's data from a source other than the person, you are generally obliged to tell them. Fines run to 4% of worldwide revenue or 20 million euros, whichever is higher.

Clearview AI found out what enforcement looks like. It built a face-search product on billions of images scraped from public websites. In September 2024 the Dutch data protection authority fined it 30.5 million euros and ordered the collection stopped, one of several European regulators to reach the same conclusion about the same database. Every one of those images was public, and to the regulators that did not matter.

California's CCPA, the California Consumer Privacy Act, in force since 2020, takes a different line: it carves "publicly available information" out of its definition of personal information, so data lawfully published in government records, or made public by the person themselves, sits outside much of the law. Do not read that as a green light. The carve-out is narrower than it sounds, other US states have passed their own variants, and the moment your dataset includes people in Europe you are back under the GDPR.

My translation of all this into practice is short. When the job allows it, collect the field and leave the person out. If the deliverable is prices and catalog structure, strip names, handles, avatars, and emails at parse time; the field you never store is the field you never have to defend or delete. When the job genuinely is about people, recruiting intelligence, say, or review authorship, then price the compliance work into the engagement: a lawful basis written down and a deletion path that actually works, reviewed by someone who has seen the GDPR up close. I have watched teams file that under paperwork for later. Later is when the access request arrives.

What the top of the web actually writes in robots.txt

There is a file that answers half the ethics questions in this chapter before you ask them, and most people have never read one. robots.txt is a plain text file at a fixed address, example.com/robots.txt, where a site declares which automated visitors, called crawlers, may fetch which paths, and how fast. Martijn Koster proposed the convention in 1994; it stayed informal custom for twenty-eight years until the IETF wrote it down as RFC 9309 in 2022. For a working collector the RFC's main value is a citable standard: the name to give when a customer or their counsel asks which rules your crawler follows. Nothing in it can stop a request. It is a preference, stated in public, in a format built for machines to read, and United States courts have not treated it as a binding contract. I respect it anyway, on every collector, and not out of fear: it is the one place where a site tells you, in advance and unambiguously, what kind of guest it wants. Ignoring it buys you little and costs you the ability to say, later and truthfully, that you followed every preference the site published.

For years I repeated the industry folklore about these files, that hardly any site blocks everything and most just fence off a few paths, without ever counting. This chapter's experiment counts.

The question: what do the most visited domains on the web actually declare in robots.txt? In particular, how many block every crawler by default, and which crawlers get singled out by name? The method fits in three sentences. Take the top 500 domains from Tranco, a public research ranking of popular domains (list ZJL5G, so anyone can pull the identical sample). Fetch one file per domain, politely: a user agent that names us truthfully (the user agent is the label a program sends identifying itself), a ten second timeout, at most one retry over https with a single fallback to plain http, and every failure recorded as a result rather than retried into submission. The snapshot, dated 2026-07-29, ships with the chapter, so python run.py re-derives the census offline, and the remaining details in this section read straight out of the shipped manifest and snapshot files; the entire run moved 8.7 MB, less than one news homepage.

146 of the 500 domains gave no usable answer, and that needs saying before any percentage does, because the census never counts an unreachable site as evidence of anything. Most of that bucket is infrastructure rather than websites: akamai.net, cloudfront.net, root-servers.net, names that serve software rather than people and mostly came back as gateway errors, plus 24 domains that never answered at all. Tranco builds its ranking from several public popularity signals, browser visits among them but also raw DNS lookup volume, so its top 500 is full of such names, and a dead end there is the expected answer. Among the 354 domains that answered like websites, 235 publish a valid robots.txt, 66.4%. Across the full sample of 500, that is 47.0%. Another 83 answered but had no file to give, mostly plain 404s (44 of them) and 403s (32), and 36 more returned something at the robots address that was not a robots.txt at all.

Walls turn out to be the exception. Only 25 of the 500 domains block all crawlers by default, 5.0% of the sample and 10.6% of the valid files, and 20 of those 25 immediately name specific crawlers and let them back in, so even most walls come with a guest list. Meanwhile 162 of the 500 use the file to point crawlers at a sitemap, which is nothing more than a machine-readable directory of the site's pages; that is 68.9% of all valid files, which means the single most common thing the top of the web does with robots.txt is invite orderly crawling. A smaller group, 31 of 500, sets a crawl-delay, asking crawlers to pause between fetches, with a median of 4 seconds and a range of 0 to 20: github.com asks for 1 second, wikipedia.org for 5, europa.eu for 10.

Before the run, this chapter's brief committed to a hypothesis: blanket blocking would be rare, most sites would express preferences rather than walls, and the recent wave of AI-crawler blocking might be far larger than the folklore admits. The data supports all three, and adds a twist I did not predict: the blockers are the biggest consumer platforms on earth. The 25 default-deny files belong to facebook.com, instagram.com, x.com, netflix.com, reddit.com, wsj.com, reuters.com, trustpilot.com, and their peers. The open web's front row is mostly open. Its most valuable rooms are the ones with the guest list.

The AI wave is real and measurable. 81 of the 500 domains, one valid file in three (34.5%), now name at least one of the seven AI crawlers the census tracks. Naming is usually a prelude to blocking: across those seven crawlers the snapshot holds 362 naming events, and 241 of them are full blocks, 66.6%. GPTBot, OpenAI's crawler, is the most named, by 60 sites, of which 37 block it fully. Bytespider, ByteDance's crawler, is the most blocked outright, by 39 of the 49 sites that name it; Meta-ExternalAgent runs at the highest rate of all, blocked by 30 of its 37 namers. CCBot, the Common Crawl crawler whose archive fed a generation of AI models, is blocked by 38 of 50. The sites doing the blocking include nytimes.com, amazon.com, cnn.com, ebay.com, and yahoo.com. When a top site bothers to write an AI crawler's name into robots.txt, it blocks it two times out of three.

One operational habit falls straight out of these numbers. The AI rows of these files are changing fast, so a collector that read robots.txt once at build time is working from a stale answer; re-read it on a schedule, the way you would re-check a price.

If you sell data that could train a model, or buy data that will, this wave changes what provenance is worth: the record of where each row came from and what the source was asking of crawlers at the time. The census above shows why: with a third of valid files now naming AI crawlers, the same page can welcome collection one quarter and refuse it the next. Keep the robots.txt you read, dated, alongside each batch it governed. It is a dated record of the preferences the site published on the day of collection, and it is the document a serious training-data buyer will ask for. A vendor who cannot produce it is selling uncertainty at full price.

Fair load, or how not to become someone's incident

Fair load is how you behave where the law is silent, and the law is silent about most of what a collector does all day. Every request your collector sends consumes compute and bandwidth the site provisioned for its human visitors, plus, past a certain point, the attention of whoever is on call. Push hard enough and the law stops being silent: a federal court in California treated crawler load on eBay's servers as trespass to chattels, interference with someone else's property, back in 2000, in eBay v. Bidder's Edge, and the hiQ consent judgment two decades later still carried the same tort. Load was what sites sued over long before anyone trained a model. But the deeper argument is commercial rather than legal. A collector that degrades its source gets noticed, then blocked, and a blocked collector delivers nothing.

In practice it comes down to four habits. Choose a request-rate ceiling on purpose and write it into the collector's configuration, instead of inheriting whatever your thread pool defaults to; where the site declared a crawl-delay, that number wins, and where it declared nothing, silence is not an invitation. Spread the volume: the same daily total can arrive as a spike that looks like an attack or as a background hum nobody notices. Picture 7,200 pages collected in a day. Sent between 9:00 and 9:40, that is 3 requests per second sustained for 40 minutes, exactly the shape a monitoring dashboard flags. Spread across the day, it is one request every 12 seconds, 300 pages per hour, a rounding error in a popular site's traffic. Schedule against the site's quiet hours instead of your own business hours. And never fetch what you already have: cache raw responses and reuse them (Chapter 7 makes storing them the default anyway), and when the site answers 429, the status code meaning "too many requests," or starts erroring, slow down automatically instead of pressing. Chapter 6 turns that back-off discipline into code.

Here is the one-line test I give engineers: if the site's operations team could see your traffic labeled with your company's name, would anything about it embarrass you? If yes, fix it before someone else finds it.

A checklist you can hand to a customer

I keep this on one page because its real job is to be shown to other people: a customer's procurement team, or a prospect who opens the call with the legality question from the top of this chapter. Buying data, it reverses: send these eleven lines to every vendor you are comparing and ask for written confirmation of each, and where a vendor hesitates, line 2 on accounts or line 4 on personal data, is where your risk lives. It also works internally. Before a new collection goes live, every line gets a yes from a person willing to put their name next to it.

1. Everything we plan to fetch is visible without an account or a payment. No exceptions, no borrowed sessions.
2. We accepted no terms of service to reach the data, and no account of ours touches the collection path.
3. We have received no cease-and-desist letter and no block aimed at us specifically from this source. If either arrives, collection pauses and counsel decides what happens next, not the roadmap.
4. No field in the output identifies a person. If the job truly is about people, the lawful basis is written down, unneeded personal fields are dropped at parse time, and a deletion request has a tested path.
5. We read the site's robots.txt before the first fetch, our target paths respect it, and any declared crawl-delay is honored.
6. Our request rate has a deliberate ceiling, written in configuration, with the day's volume spread thin instead of bunched into spikes.
7. Nothing gets fetched twice; a raw response, once collected, is kept and reused.
8. On "too many requests" and server errors, the collector slows itself down without waiting for a human.
9. Our traffic identifies itself honestly and does not impersonate a human visitor to get around a site's defenses.
10. The customer's rights to use the data are covered by contract, because collecting a record lawfully does not by itself grant the right to resell it or train a model on it. That second question belongs to the buyer's side of the table, and Chapter 14 handles it there.
11. We could explain this collection, in plain language, to the site being collected, and we would be comfortable doing it.

Item 11 is the whole chapter in one line. If it gets a yes you can defend, the other ten are usually already true.

With the lines drawn, the next problem is mechanical. Any source worth collecting has several doors into it: the official API, the page itself, the site's own hidden data feeds, and a few others less obvious. Choosing the right one is Chapter 3.