Published by ANAWAZ
Responsible web data collection begins with permission, purpose and the information you intend to use. Public visibility does not settle privacy, contractual or intellectual-property questions. Plan the legal review and the engineering controls together, before a crawler starts collecting.
A practical decision guide
| Situation | Starting point | What to verify |
|---|---|---|
| Personal information | Privacy and legal review | Document the applicable basis, purpose, retention and rights process. |
| Protected or restricted content | Permission and usage review | Check terms, licences and intended reuse. |
| Permitted public collection | Conservative crawler design | Identify the client, cache responses and stop on repeated errors. |
This guide explains planning and engineering considerations. It is not legal advice; obtain jurisdiction-specific advice for contracts, privacy and permitted data use.
Four questions that decide it
1. Is the data personal?
This is the first question, always, because it changes the entire analysis.
Data protection regimes such as the GDPR apply to personal data regardless of whether it was publicly accessible. “It was on a public web page” is not a lawful basis for processing. If you are collecting names, contact details, profiles or anything that identifies a person, you need a lawful basis, and you generally carry obligations around transparency, retention and the rights of the people concerned — including, potentially, telling them you hold their data.
Scraping personal data at scale for commercial use has attracted regulatory enforcement in multiple jurisdictions. This is the single common way an otherwise sensible project becomes a serious problem.
Personal data collection needs a documented lawful basis and legal sign-off before any code is written. Product listings, prices, specifications and public company information are a far more comfortable category, and that is where most legitimate commercial scraping sits.
2. What do the terms of service say?
Many sites prohibit automated access in their terms. Enforceability varies — courts in different jurisdictions have reached different conclusions, and there is a meaningful distinction between terms you actively accepted by creating an account and terms merely published in a footer.
The practical implication is clearer than the legal one. Scraping from behind a login, where you accepted terms prohibiting it, is a materially different risk from accessing public pages. Check the terms accepted when access was granted and the permitted uses of that access. Circumventing technical access controls raises the stakes further and, in some jurisdictions, moves the question towards computer misuse legislation.
We do not build systems that require creating accounts to breach the terms attached to them, or that defeat access controls.
3. Is the content itself protected?
Facts are generally not protected by copyright. A price, a specification, a stock level is a fact. But creative content — reviews, articles, descriptions, photographs — usually is, and some jurisdictions additionally protect databases as compilations even where individual facts are not.
Collecting facts for analysis is the comfortable case. Republishing someone’s original content is not. If your intended output reproduces substantial protected content rather than deriving insight from facts, that needs legal review.
4. What is the impact on the source?
This is where ethics and risk align most neatly. No request rate is safe for every source. Start with the operator’s permitted limits and monitor the effect of collection. A scraper that hammers a server with concurrent requests degrades service for real users and looks, from the other end, indistinguishable from an attack.
Aggressive scraping is what triggers blocks, complaints and legal letters. A low request rate does not itself establish permission. Being considerate is one useful operational control.
What responsible collection looks like
Respect robots.txt without treating it as permission
RFC 9309 defines robots.txt crawl rules and explicitly separates them from access authorisation. Honour applicable rules, but still check permission, terms and lawful reuse. Crawl-delay is an extension rather than a directive defined by that standard; where an operator specifies pacing guidance, incorporate it into your collection plan.
Identify yourself
Send a descriptive user agent naming your organisation, with a URL or email address the site operator can use to contact you.
This feels counter-intuitive — surely anonymity is safer? In practice the opposite holds. An identified crawler behaving reasonably is something an operator can contact if there is a problem. An anonymous crawler disguising itself as a browser is something they block, and it removes any argument that you were acting in good faith.
Rate limit conservatively
Sequential requests with a genuine delay between them, not parallel bursts. Slower than you think you need. If a full collection run takes overnight instead of ten minutes, that is almost an acceptable trade for not degrading someone else’s service.
Implement exponential backoff on errors, and stop entirely on repeated failures rather than retrying into a struggling server. Where possible, schedule runs outside the source’s peak hours.
Cache and collect incrementally
Do not re-download what has not changed. Use conditional requests, track what you have already collected, and fetch only deltas. This reduces load on the source, reduces your own cost, and shortens run times.
Re-scraping an entire catalogue daily when a small fraction changes is the common inefficiency teams should watch for, and it is the behaviour most likely to get you blocked.
Prefer an API, and ask
Before building a scraper, check whether an official API, data feed or bulk download exists. Check the API or feed’s licence and terms; authorised structured access can also simplify maintenance.
Where there is no API, consider simply asking. Organisations sometimes provide data directly, or grant permission, when approached — an outcome that clarifies permitted collection and reuse and is rarely attempted.
Store and retain deliberately
Collect only the fields you need, define a retention period and enforce it, secure the store appropriately, and document what you hold and why. Data minimisation reduces both regulatory exposure and the consequences of a breach.
Things we will not build
For clarity, since clients ask:
- Systems that defeat authentication or technical access controls.
- Bulk collection of personal data without a documented lawful basis.
- Scrapers deliberately disguised to evade detection while breaching stated terms.
- Collection at rates that would degrade the source’s service.
- Systems whose purpose is republishing someone else’s protected content.
This is not only an ethical position. Systems like these have a habit of becoming their client’s problem, and we would rather explain the constraint up front than deliver something that creates liability.
The engineering reality
Two expectations worth setting for anyone commissioning this work.
First, scrapers break. Sites change their markup without notice and your parser stops working, often silently — which is worse, because you keep collecting nothing while believing you are collecting data. Any serious pipeline needs validation that alerts on unexpected structure, volumes or value ranges. Budget for ongoing maintenance; a scraper is not a one-off build.
Second, collection is the easy part. The effort is in cleaning, deduplicating, normalising and validating what you collected, and in making it usable. Projects that budget for the scraper and not for the pipeline deliver a large pile of inconsistent data and no insight.
Worked example
For a supplier-price comparison, first ask whether a catalogue feed or API is available. If permitted web collection is necessary, record the source and collection time, request only required product fields, cache unchanged pages and define a stop rule. If the project expands into personal profiles or republishing photographs, reopen the permission and legal review.
This is an illustrative planning scenario, not a client case study.
Turn the decision into a plan
Write down the outcome you need, the evidence that would show success, the unresolved risks and the person responsible for the next decision. Use a bounded first step to test the difficult assumptions before committing to a larger programme.
ANAWAZ can help you scope the engineering work through our Data collection and engineering. Tell us about your current systems and the result you want to achieve.