Web scraping occupies genuinely contested legal ground, and the honest summary is that it depends — on jurisdiction, on what is being collected, on how it is collected, and on what is done with it afterwards.

What we can offer is the operational framework we apply before taking on a data collection project. It will not replace advice from a qualified lawyer in your jurisdiction, and you should get that advice for anything commercially significant. But it will tell you which projects are straightforward, which need legal review, and which we decline.

This article is practical guidance, not legal advice.

Four questions that decide it

1. Is the data personal?

This is the first question, always, because it changes the entire analysis.

Data protection regimes such as the GDPR apply to personal data regardless of whether it was publicly accessible. “It was on a public web page” is not a lawful basis for processing. If you are collecting names, contact details, profiles or anything that identifies a person, you need a lawful basis, and you generally carry obligations around transparency, retention and the rights of the people concerned — including, potentially, telling them you hold their data.

Scraping personal data at scale for commercial use has attracted regulatory enforcement in multiple jurisdictions. This is the single most common way an otherwise sensible project becomes a serious problem.

Our position: personal data collection needs a documented lawful basis and legal sign-off before any code is written. Product listings, prices, specifications and public company information are a far more comfortable category, and that is where most legitimate commercial scraping sits.

2. What do the terms of service say?

Many sites prohibit automated access in their terms. Enforceability varies — courts in different jurisdictions have reached different conclusions, and there is a meaningful distinction between terms you actively accepted by creating an account and terms merely published in a footer.

The practical implication is clearer than the legal one. Scraping from behind a login, where you accepted terms prohibiting it, is a materially different risk from accessing public pages. Authentication means you agreed to something. Circumventing technical access controls raises the stakes further and, in some jurisdictions, moves the question towards computer misuse legislation.

We do not build systems that require creating accounts to breach the terms attached to them, or that defeat access controls.

3. Is the content itself protected?

Facts are generally not protected by copyright. A price, a specification, a stock level is a fact. But creative content — reviews, articles, descriptions, photographs — usually is, and some jurisdictions additionally protect databases as compilations even where individual facts are not.

Collecting facts for analysis is the comfortable case. Republishing someone’s original content is not. If your intended output reproduces substantial protected content rather than deriving insight from facts, that needs legal review.

4. What is the impact on the source?

This is where ethics and risk align most neatly. A scraper that makes a few thousand well-paced requests over a day is a rounding error in most sites’ traffic. A scraper that hammers a server with concurrent requests degrades service for real users and looks, from the other end, indistinguishable from an attack.

Aggressive scraping is what triggers blocks, complaints and legal letters. Restrained scraping usually goes unremarked. Being considerate is genuinely the most effective risk mitigation available.

What responsible collection looks like

Respect robots.txt

It is a convention rather than a law, and it is the clearest available statement of what a site operator wants automated clients to do. Ignoring it is a deliberate choice to disregard an explicit request, and it looks exactly like that if the matter is ever examined.

Read it, honour its disallow rules, and honour any crawl-delay it specifies.

Identify yourself

Send a descriptive user agent naming your organisation, with a URL or email address the site operator can use to contact you.

This feels counter-intuitive — surely anonymity is safer? In practice the opposite holds. An identified crawler behaving reasonably is something an operator can contact if there is a problem. An anonymous crawler disguising itself as a browser is something they block, and it removes any argument that you were acting in good faith.

Rate limit conservatively

Sequential requests with a genuine delay between them, not parallel bursts. Slower than you think you need. If a full collection run takes overnight instead of ten minutes, that is almost always an acceptable trade for not degrading someone else’s service.

Implement exponential backoff on errors, and stop entirely on repeated failures rather than retrying into a struggling server. Where possible, schedule runs outside the source’s peak hours.

Cache and collect incrementally

Do not re-download what has not changed. Use conditional requests, track what you have already collected, and fetch only deltas. This reduces load on the source, reduces your own cost, and shortens run times.

Re-scraping an entire catalogue daily when a small fraction changes is the most common inefficiency we see, and it is the behaviour most likely to get you blocked.

Prefer an API, and ask

Before building a scraper, check whether an official API, data feed or bulk download exists. It will be more stable, faster, cheaper to maintain, and unambiguously permitted.

Where there is no API, consider simply asking. Organisations sometimes provide data directly, or grant permission, when approached — an outcome that removes the risk entirely and is rarely attempted.

Store and retain deliberately

Collect only the fields you actually need, define a retention period and enforce it, secure the store appropriately, and document what you hold and why. Data minimisation reduces both regulatory exposure and the consequences of a breach.

Things we will not build

For clarity, since clients ask:

• Systems that defeat authentication or technical access controls.
• Bulk collection of personal data without a documented lawful basis.
• Scrapers deliberately disguised to evade detection while breaching stated terms.
• Collection at rates that would degrade the source’s service.
• Systems whose purpose is republishing someone else’s protected content.

This is not only an ethical position. Systems like these have a habit of becoming their client’s problem, and we would rather explain the constraint up front than deliver something that creates liability.

The engineering reality

Two expectations worth setting for anyone commissioning this work.

First, scrapers break. Sites change their markup without notice and your parser stops working, often silently — which is worse, because you keep collecting nothing while believing you are collecting data. Any serious pipeline needs validation that alerts on unexpected structure, volumes or value ranges. Budget for ongoing maintenance; a scraper is not a one-off build.

Second, collection is the easy part. The effort is in cleaning, deduplicating, normalising and validating what you collected, and in making it genuinely usable. Projects that budget for the scraper and not for the pipeline deliver a large pile of inconsistent data and no insight.

How we work on data projects

Our web scraping and data extraction engagements start with the four questions above, and we will tell you plainly if a project needs legal review before we build it. The systems we deliver identify themselves, respect robots.txt, rate limit conservatively and validate their own output — with the pipeline built by our software engineering team so the result is usable rather than merely collected.

Get in touch to discuss what data you need and whether it can be collected responsibly.