Is Web Scraping Legal? US Rules and EU Limits for AI

👁 114 reads

Web scraping is a technique, and its legal treatment depends on how access is obtained and what happens to the data afterward. This guide explains key US cases and a narrower set of EU rules as of October 2026; it is general information rather than a legal opinion on a particular project.

Describe the data flow before asking whether it is legal

A project that reads its own documentation differs from one that collects public profiles, republishes articles, or builds a commercial training dataset. The label “AI scraping” does not resolve those differences. Start with the source, access method, fields collected, intended use, recipients, and retention period.

Separate collection from later activities. Obtaining a page, keeping a copy, building a retrieval index, training a model, and publishing excerpts are different acts. A legal basis for one stage does not automatically answer every question at the next stage. Technical success is not evidence of legal permission.

For a US project, identify whether the source is public without authentication or requires an account, subscription, or other access control. Record the relevant terms and any permission from the source owner. Also identify personal information and protected content. The exact facts matter more than whether the application uses a browser agent or a conventional scraper.

The sections below explain specific authorities. They are not an exhaustive survey of state laws, every later scraping case, or all rules for training AI models. A commercial project with uncertain access or reuse rights should take its actual workflow and source terms to qualified counsel.

CFAA and Van Buren: access is not the same as purpose

The Computer Fraud and Abuse Act, 18 U.S.C. Section 1030, addresses forms of unauthorized computer access. Its provisions distinguish access without authorization from exceeding authorized access. Other statutory elements matter too, so a slogan about public data is not a complete CFAA analysis.

In Van Buren v. United States, decided June 3, 2021, the Supreme Court considered a police officer who used a database he was entitled to access for an improper purpose. The Court held that the statute’s exceeding-authorized-access language concerns obtaining information from areas of a computer that are off limits to the user. Misusing otherwise permitted access was not enough under that provision.

The distinction limits an interpretation that would turn every violation of a computer-use policy into the same federal access offense. It does not authorize entering restricted areas, using another person’s credentials, or bypassing access controls. Nor does it erase contractual or other legal consequences associated with the conduct.

For an agent workflow, document the access boundary accurately. Public pages, signed-in pages, and restricted resources should not be treated as a single category. A model’s ability to navigate a page does not expand the operator’s legal authorization. Review the specific access path rather than relying on the fact that a browser can technically display the result.

hiQ v. LinkedIn: read the procedural limit

The Ninth Circuit’s April 18, 2022 opinion in hiQ Labs v. LinkedIn addressed a preliminary injunction involving publicly accessible member profiles. On remand after Van Buren, the court again affirmed the preliminary injunction. Its reasoning distinguished information available to the general public from systems requiring authorization.

The procedural posture is important. The court considered whether hiQ had raised serious questions about the CFAA issue within the preliminary-injunction framework. The opinion should not be presented as a final, nationwide declaration that every scraping business model is lawful. It concerned particular facts and a specific legal question.

The opinion also recognized that other claims could matter. A conclusion about the CFAA does not resolve contract, copyright, privacy, or other applicable claims. The article’s discussion of this 2022 appellate ruling is limited to that decision; it is not an assertion about the complete later history of the parties’ dispute.

For a builder, the useful distinction is public availability versus an authorization gate, with the surrounding facts preserved. Do not turn that distinction into advice to ignore restrictions or continue automatically after a dispute arises. An access change or cease-and-desist demand is a material fact for legal review.

Terms, copyright, and robots.txt answer different questions

Website terms can create contractual questions separate from the CFAA. Whether terms bind a particular user depends on formation, notice, assent, governing law, and the relevant facts. A prohibition in terms is not automatically a federal hacking offense, but neither is it automatically irrelevant.

Copyright concerns the protected material and the acts performed with it. The US Copyright Office explains fair use through a case-specific four-factor analysis, including purpose, nature, amount, and market effect. Describing a system as research, retrieval, or AI does not establish the outcome of that analysis.

For example, a tool that extracts a fact with a citation is not the same workflow as one that republishes an entire article archive. The amount copied, purpose, and downstream presentation differ. A citation helps identify a source but is not itself a license. Assess copying and publication separately from the initial ability to access the page.

RFC 9309 defines the Robots Exclusion Protocol and expressly distinguishes its rules from access authorization. A robots.txt allowance is not a copyright license. A disallow rule is not a complete legal judgment either. Operationally, reading and respecting crawler instructions is useful, but the protocol cannot replace review of access rights and permitted use.

EU rules: personal data and text-and-data mining

When GDPR applies, publicly visible information can still be personal data. Articles 5 and 6 require principles such as purpose limitation and data minimization, plus an appropriate lawful basis. Legitimate interests is not an automatic permission for every collection project; the relevant interests and effects need assessment.

A dataset of business information can also contain identifiable people, such as profile owners or individual contact details. Decide which fields are necessary before collecting them. Restrict retention and access according to the actual purpose. The fact that an extraction vendor can return a field does not establish that your project should keep it.

Article 4 of the DSM Directive provides a text-and-data-mining exception for lawfully accessible works and other subject matter, subject to conditions. Rights holders can expressly reserve relevant rights in an appropriate manner, including machine-readable means for content made publicly available online. Retained copies are tied to the needs of the mining activity.

This is not a blanket right to publish the source material or ignore privacy duties. National implementation and the details of the activity matter. Article 3 contains a distinct framework for qualifying research organizations and cultural heritage institutions, so do not assume that every commercial AI project receives the same treatment.

For a US operator handling EU-related data, jurisdiction and territorial scope require their own assessment. The presence of a European website or a US server is not enough to settle the entire analysis. Keep the EU discussion attached to the actual people, processing, and rights involved.

Turn the analysis into concrete project controls

An engineering specification can make legal review more useful by recording the data flow precisely. List the domains, access methods, fields, and intended outputs. Identify which source terms or licenses apply and who approved any negotiated permission. Keep dates because a source’s conditions can change.

• Prefer an official licensed API or export when it fits the intended use.
• Limit collection to the sources and fields needed for the stated purpose.
• Separate raw-source retention from derived summaries and output publication.
• Define how access objections, deletion requests, and source changes are handled.
• Reassess the workflow when it adds authenticated access or new data categories.

These controls do not guarantee legality. They make the relevant facts visible and help the implementation stay within the assessed scope. A tool change can alter where data is processed, while an agent configuration change can expand the number of pages collected. Both deserve review when they change the facts of the project.

The web data guide covers technical architecture, and the browser agents comparison explains why interactive actions need explicit boundaries. Buying a capable scraper is an infrastructure decision, not a legal clearance for every use of its output.

FAQ

Is scraping public websites always legal in the US?

No universal rule follows from public visibility. CFAA access questions are only part of the assessment. Terms, copyright, privacy, and other applicable law can still matter.

Does hiQ let me scrape authenticated accounts?

The cited opinion concerns publicly accessible profiles and a preliminary injunction. It should not be generalized into permission to enter authenticated or restricted areas.

Does robots.txt grant permission to reuse content?

No. RFC 9309 describes crawler instructions and says they are not access authorization. The file is not a general copyright or data-processing license.

Does EU Article 4 permit all AI training?

No blanket conclusion is justified. Lawful access, rights reservations, the activity’s nature, national implementation, and other applicable obligations must be assessed. The exception is not unrestricted republication permission.

Kinetyra