Data collection projects have a characteristic failure mode. The scraper works, the warehouse fills up, a dashboard gets built — and eighteen months later nobody can point to a decision that was made differently because of any of it.

The data is not wrong. It is just not connected to anything. The gap sits between having data and acting on it, and it is a design problem rather than a technical one.

Here are the five stages between raw collection and a decision someone actually takes, and where projects tend to stall.

Stage 0: Start from the decision, not the data

Before the pipeline, the question that determines whether any of it matters: which recurring decision will this change, who makes it, and what would make them decide differently?

“We want visibility into competitor pricing” is not a decision. “Our category managers review pricing every Monday and currently adjust based on incomplete manual checks; we want them to see the full competitive position for their category before that meeting” is. It names the person, the cadence, and the moment the data must arrive.

Everything downstream should be designed backwards from that moment. Data that arrives after the decision point has zero value regardless of quality — a daily feed serving a decision made hourly is useless, and an hourly feed serving a monthly decision is expensive over-engineering.

If you cannot name the decision, the honest recommendation is to stop and find it first. Building the pipeline will not reveal it.

Stage 1: Collection you can trust

Covered in detail in our guide to legal and ethical web scraping, but one point bears repeating here because it is what breaks pipelines: collection fails silently.

A site changes its markup, your parser returns empty values, and the pipeline continues to run, dutifully loading nothing. Dashboards keep rendering. Nobody notices until someone asks why a competitor has apparently not changed prices in three weeks.

The fix is validation at the collection boundary, alerting on volume anomalies, unexpected null rates, structural changes, and values outside plausible ranges. Treat a sudden drop in collected records as an incident, not a data point.

Stage 2: Cleaning and normalisation

This is where most of the effort actually goes, and where most estimates are wrong.

Real-world data arrives inconsistent in every dimension. The same product is named differently across sources. Prices come with and without tax, in different currencies, sometimes as ranges. Dates arrive in several formats. Units vary. Categories do not align. Duplicates appear with subtle differences.

Two principles keep this manageable:

• Keep the raw data immutable. Store exactly what you collected, unmodified, and treat cleaning as a separate derived layer. When you discover your cleaning logic was wrong — and you will — you can reprocess from source rather than having destroyed the original.
• Make cleaning rules explicit and versioned. Cleaning logic buried in ad-hoc scripts becomes unauditable within months. When someone asks why a figure changed, you need to be able to answer.

Entity resolution — deciding that these two records describe the same real-world thing — is usually the hardest single problem, and the one most likely to be underestimated. Matching your product catalogue against competitor listings with different names, packaging and identifiers is genuinely difficult. Plan for a rules-and-review approach: automated matching with confidence scores, and a human queue for the uncertain cases. Aiming for full automation from day one generally produces confidently wrong matches.

Stage 3: Enrichment and context

A number alone rarely supports a decision. A competitor price of a given amount means nothing without your price, your margin at that price, your stock position and the recent trend.

Enrichment joins collected data to what you already know — your own catalogue, costs, sales history, inventory — and to derived context such as change over time, percentile position, and rate of movement.

This stage is where external data becomes your data, and it is frequently skipped. A dashboard showing competitor prices requires the viewer to hold all the context in their head; one showing where you sit relative to the market, with margin implications attached, supports a decision directly.

Stage 4: Detection, not display

Here is the stage that gets skipped almost every time, and it is the one that determines whether the project succeeds.

The default deliverable is a dashboard. Dashboards are passive: they require someone to remember to look, at the right time, and to spot what changed among everything that did not. In practice, dashboard usage decays sharply after launch. A dashboard nobody opens is indistinguishable from no dashboard.

What actually changes behaviour is detection — the system deciding that something has happened which warrants attention, and pushing it to the person who can act.

Not “here is all the pricing data”, but “these eleven products moved below your price this morning; here are the margin implications of matching”. Delivered where that person already works — email, the team channel, the tool they use — at the moment before their decision, not in a portal they must remember to visit.

Getting detection right takes iteration. Thresholds are wrong initially; too sensitive and people learn to ignore the alerts, too conservative and they miss what matters. Expect to tune this with the people receiving the alerts over the first weeks, and treat “I ignored that one” as valuable feedback rather than user error.

Stage 5: The feedback loop

The final stage, and the one that separates a pipeline that stays valuable from one that decays: capture what happened after the alert.

Did the category manager act on it? Was the recommendation right? Which alerts consistently get ignored, and why?

Without this, you cannot tell whether the system is working. With it, you have a measurable basis for improvement, evidence of value when the budget is questioned, and early warning when the pipeline breaks or the underlying assumptions stop holding.

It need not be elaborate — a one-click useful/not-useful on each alert, reviewed monthly, tells you most of what you need.

Sizing it honestly

A rough guide to where effort lands on a typical project: collection is a modest share, cleaning and entity resolution is the largest single block by some distance, enrichment and detection together are substantial, and presentation is smaller than anyone expects.

Budgets are frequently allocated in the opposite proportion — most of it to collection and dashboards, almost none to the cleaning and detection work that determines whether the output is trustworthy or actionable. That misallocation is the direct cause of the eighteen-months-and-no-decisions outcome.

A short checklist

1. Which specific recurring decision does this change, and who makes it?
2. When does that decision happen, and does data arrive before it?
3. How would we know within an hour if collection silently broke?
4. Is the raw data preserved so we can reprocess when cleaning logic changes?
5. How do we resolve records to real-world entities, and who reviews uncertain matches?
6. Is the output pushed to where people work, or does it wait to be visited?
7. How do we find out whether anyone acted on it?

Building one properly

Our data extraction engagements are scoped backwards from the decision they are meant to support, with the pipeline, validation and alerting built by our software engineering team. Where matching or classification is the hard part, our AI team works on the entity resolution alongside deterministic rules.

Tell us which decision you want to improve and we will design the pipeline around it.