Essays on AI and data
I am also writing The Data Collection Handbook, a free book on data collection best practices with a replicable experiment in every chapter. The first chapters are out, and the book is a free download as PDF or EPUB that grows as chapters publish.
Congress Wants Bots to Say Their Name. It Picked the One Name Nobody Can Check.
Last week 300 publishers asked Congress for the Stealth Bot Prohibition Act. The bill defines a bot's identity as its user-agent string: a line of text no site can check, and the one form of identity the web's biggest verifier does not count. Congress ran the same play on email in 2003, and the header stayed unverifiable until the market signed it. Bots got the signature first.
Cloudflare's Pay Per Use Pays for More. The Buyer Holds the Meter.
Cloudflare's Pay Per Use entered beta today: AI companies pay publishers when they use a page, not when they fetch it. Pay Per Crawl counted at the door, in a unit the publisher could see in its own logs. Pay Per Use pays on a count only the buyer can see: the buyer defines the use, sets the price and reports the number, and Cloudflare checks only that the URL belongs to an enrolled publisher. The model that promises publishers more is the one they cannot audit.
Bots Passed Humans in May. Ask Whose Counter.
Cloudflare's founders date the moment bots passed humans to May 2026. Thales says it happened in 2025, Fastly says humans still led in January, DataDome counts bots at 26.5 percent, and TollBit finds one AI bot visit per 31 human visits. There is no share of the web that is bots, only counters, each measuring its own customers in its own unit, and each selling the wall it counts through.
The Bot Is the User Only While It Runs on the User's Machine
Amazon's amended complaint alleges Comet for iOS moved Perplexity's shopping agent from the user's phone to Perplexity's cloud, nine days after the injunction. The Ninth Circuit's 'the bot is the user' ruling rested on where the agent ran. Which computer runs the agent now decides whose access it is, and Amazon's answer to every unnamed agent, Meta's Muse included, is: unauthorized.
The IETF Adopted the Bot Passport. The Market Issued It Fifteen Months Ago
The IETF working group adopted Web Bot Auth, the protocol that lets a crawler sign its requests, on September 1, fifteen months after Cloudflare, OpenAI, Google, Shopify, Visa and Mastercard put it in production. What the standard fixes is the format of the name on the invoice; on September 15 staying anonymous got its first default bill.
Security's Hottest New Market Is Bot Management in a Mirror
Three rounds put $270 million into agent security in five days; Fortinet bought a fourth company two weeks later. The category looks new. It is bot management with the arrow reversed: the same five functions sold to the visitor instead of the site, consolidating into the same kind of buyer at twice the speed, with F5 already on both sides of the glass.
The Web's Biggest Bot Blocker Just Built a Browser for Bots
Cloudflare, which nearly one in four websites pays to keep bots out, shipped Kitesurf, a browser built for bots: 3 to 7 times cheaper per page read than Chromium, free in beta. Beside the agent wallets, the posted prices and the September 15 default block, the summer reads as one product, an exchange with the same operator on both sides.
Banking Already Fought the AI Agent War. The Ending Was a Contract
Twenty years before agentic browsers, aggregators like Mint and Plaid logged into banks with millions of borrowed passwords. Banks throttled, blocked and sued, and what ended the war was a contract: tokens, then fees. The screen-scraping arc mapped onto the agent fight, and the deal to watch for next.
The Web's Attack Surface Just Switched Sides
The same week a court ruled the bot is the user, Black Hat research showed every AI browser tested obeying instructions hidden in the pages it reads. A year of prompt-injection disclosures, why layered defenses failed, and why the fix is architecture, not smarter models.
Your Analytics Copilot Survived Opus 4.8. The Claude 5 Upgrade Is Where It Breaks
Anthropic shipped three Claude 5 models in seven weeks, and the dangerous changes never throw an error. Why a model upgrade under your analytics is a data migration: inventory the pinned IDs, baseline golden questions, remap tiers, and delete the calibration that now works against you.
The Ninth Circuit Just Said the Bot Is the User
The Ninth Circuit overturned Amazon's injunction against Perplexity's Comet shopping agent: it was the users, not Perplexity, doing the accessing. Why that inverts the 2016 Power Ventures rule, and why the agent fight now moves from courtrooms to identity rails.
Bots Are Getting Passports, and the Passport Is a Rate Card
Cryptographic bot identity went from a Cloudflare proposal to Visa, Mastercard, Google and OpenAI in about a year, while the IETF group chartered to standardize it has yet to adopt a draft. The passport's real function is pricing: named traffic can be billed, anonymous traffic can only be blocked.
Only 4 in 10 AI Licensing Deals Still Include Training Rights
Rob Kelly's ledger of public AI content licensing deals crossed 94 in July, and only about 4 in 10 still include training rights. Live-access deals went 2, 11, 18, then 34 projected this year. Why the money moved from archives to live feeds, from an operator who sells the feed.
The First Public Map of What AI Actually Crawled
The EU AI Act makes every general-purpose AI provider publish its top crawled domains. A year in, small labs filed straight-A summaries, the frontier labs cluster at C, and four flagship models filed nothing. Enforcement starts August 2.
Does llms.txt Work? 137,000 Domains Say Not Yet
Ahrefs read a month of server logs across 137,210 domains: 97% of llms.txt files got zero requests, and the AI search bots the file was written for sent just 1.1% of the traffic that did arrive. What that means before you ship one, from a site owner who just did.
Is Robots.txt Legally Binding? Two Courts Answered in the Same Week
A US court called robots.txt a keep-off-the-grass sign. A German court made machine-readable opt-outs the only ones that count. What the split means for AI training data.
Prime Day, Measured
Of 1.19 million European prices tracked through Prime Day 2026, 74.6% never moved and only 13.8% got a real cut. What the shelf data shows, and why price history has to be built before the question is asked.
What a Web Page Costs
Cloudflare's pay-per-crawl posted the first official price for machine access to the web. An operator's math on what a page really costs, and what the premium buys.
How AI Agents Collect and Reconcile E-commerce Data at Web Scale
Variants, pricing conditions, silent breakage: what actually determines e-commerce data quality at web scale, and how AI agents change its economics.
Why Travel Data Is the Hardest Data on the Web
Travel prices are quoted, not published. Why that makes travel the hardest web data to collect, and what AI-driven collection changes about doing it honestly.
What Makes Web Data Investable
What institutional buyers actually pay for in web data: provenance, point-in-time history, coverage honesty, and the diligence questions that sort the market.
New essays by email, on AI for data collection and analysis.