The Data Collection Handbook · Part I. Foundations
Chapter 2. Legality, Ethics, and Fair Use
In progress: this is the chapter plan. The full chapter, its charts, and its runnable experiment publish on this page.
What this chapter answers
Is this even legal? is the first question every founder in this industry gets asked. The honest answer is: it depends on what you collect, how, and how much. This chapter draws the lines so you can defend your practices to a lawyer, a customer, or the site being collected.
What you will learn
- The brightest legal line in the field: public data versus data behind a login, and what the hiQ v. LinkedIn saga actually established, including how it ended.
- What personal data changes: the practical meaning of GDPR and CCPA for a collector.
- What robots.txt can and cannot do, and why respecting crawl preferences is good business, not just good manners.
- The fair-load principle: rate limits you impose on yourself so you never degrade the service you collect from.
- The difference between collection legality (may I fetch this?) and usage rights (may I store, resell, or train models on it?), which is a contract question covered in Chapter 14.
In this chapter
- The question every founder gets asked. Why the answer is always conditional.
- Public vs gated. The login wall; terms of service in plain words; the full hiQ v. LinkedIn arc.
- Personal data is different. Minimization and the right to be forgotten, applied to collectors.
- robots.txt and being a good citizen. What the file is, what it is not, and why I respect it.
- Fair load. Spreading volume, off-peak scheduling, and never fetching the same thing twice.
- A one-page ethics checklist. Printable, usable in sales conversations.
The experiment
The question: What do the world's most-visited websites actually say in their robots.txt files: how many block everything, how many set crawl delays, and which bots do they single out?
A census over a dated snapshot of roughly 500 robots.txt files from a public top-sites list, shipped with the chapter so the result is reproducible offline. The code parses every file and aggregates what sites allow, restrict, and rate-limit, with fetch failures tracked separately so they never pollute the numbers. The data and code ship with the chapter, sized to run offline on a laptop with one command.
My hypothesis is that blanket blocking of all crawlers is rare and most sites express preferences instead. The recent wave of AI-crawler blocks may prove me wrong, and if it does, the chapter will say so: the census reports what the files say, not what I expected them to say.