Published by ANAWAZ
A data pipeline is valuable when a trustworthy signal reaches someone who can act on it. Before choosing a scraper, warehouse or dashboard, name the decision, its owner and the time window in which the information is useful. Then design collection, validation and delivery around that outcome.
A practical decision guide
| Situation | Starting point | What to verify |
|---|---|---|
| Raw collection | Source, timestamp and validation | Can you distinguish a missing record from a real change? |
| Clean and enriched data | Stable entities and rules | Can you explain the transformation? |
| Operational signal | Owner and action path | Who acts, by when, and how is the outcome recorded? |
Stage 0: Start from the decision, not the data
Before the pipeline, the question that determines whether any of it matters: which recurring decision will this change, who makes it, and what would make them decide differently?
“We want visibility into competitor pricing” is not a decision. “Our category managers review pricing every Monday and currently adjust based on incomplete manual checks; we want them to see the full competitive position for their category before that meeting” is. It names the person, the cadence, and the moment the data must arrive.
Everything downstream should be designed backwards from that moment. Data that arrives after the decision point has zero value regardless of quality — a daily feed serving a decision made hourly is useless, and an hourly feed serving a monthly decision is expensive over-engineering.
If you cannot name the decision, the honest recommendation is to stop and find it first. Building the pipeline will not reveal it.
Stage 1: Collection you can trust
Covered in detail in our guide to legal and ethical web scraping, but one point bears repeating here because it is what breaks pipelines: collection fails silently.
A site changes its markup, your parser returns empty values, and the pipeline continues to run, dutifully loading nothing. Dashboards keep rendering. Nobody notices until someone asks why a competitor has apparently not changed prices in three weeks.
The fix is validation at the collection boundary, alerting on volume anomalies, unexpected null rates, structural changes, and values outside plausible ranges. Treat a sudden drop in collected records as an incident, not a data point.
Stage 2: Cleaning and normalisation
This is where most of the effort goes, and where most estimates are wrong.
Real-world data arrives inconsistent in every dimension. The same product is named differently across sources. Prices come with and without tax, in different currencies, sometimes as ranges. Dates arrive in several formats. Units vary. Categories do not align. Duplicates appear with subtle differences.
Two principles keep this manageable:
- Keep the raw data immutable. Store exactly what you collected, unmodified, and treat cleaning as a separate derived layer. When you discover your cleaning logic was wrong — and you will — you can reprocess from source rather than having destroyed the original.
- Make cleaning rules explicit and versioned. Cleaning logic buried in ad-hoc scripts becomes unauditable within months. When someone asks why a figure changed, you need to be able to answer.
Entity resolution — deciding that these two records describe the same real-world thing — is usually the hardest single problem, and the one most likely to be underestimated. Matching your product catalogue against competitor listings with different names, packaging and identifiers is difficult. Plan for a rules-and-review approach: automated matching with confidence scores, and a human queue for the uncertain cases. Aiming for full automation from day one generally produces confidently wrong matches.
Stage 3: Enrichment and context
A number alone rarely supports a decision. A competitor price of a given amount means nothing without your price, your margin at that price, your stock position and the recent trend.
Enrichment joins collected data to what you already know — your own catalogue, costs, sales history, inventory — and to derived context such as change over time, percentile position, and rate of movement.
This stage is where external data becomes your data, and it is frequently skipped. A dashboard showing competitor prices requires the viewer to hold all the context in their head; one showing where you sit relative to the market, with margin implications attached, supports a decision directly.
Stage 4: Detection, not display
Here is the stage that gets skipped almost every time, and it is the one that determines whether the project succeeds.
The default deliverable is a dashboard. Dashboards are passive: they require someone to remember to look, at the right time, and to spot what changed among everything that did not. In practice, dashboard usage decays sharply after launch. A dashboard nobody opens is indistinguishable from no dashboard.
What changes behaviour is detection — the system deciding that something has happened which warrants attention, and pushing it to the person who can act.
Not “here is all the pricing data”, but “these eleven products moved below your price this morning; here are the margin implications of matching”. Delivered where that person already works — email, the team channel, the tool they use — at the moment before their decision, not in a portal they must remember to visit.
Getting detection right takes iteration. Thresholds are wrong initially; too sensitive and people learn to ignore the alerts, too conservative and they miss what matters. Expect to tune this with the people receiving the alerts over the first weeks, and treat “I ignored that one” as valuable feedback rather than user error.
Stage 5: The feedback loop
The final stage, and the one that separates a pipeline that stays valuable from one that decays: capture what happened after the alert.
Did the category manager act on it? Was the recommendation right? Which alerts consistently get ignored, and why?
Without this, you cannot tell whether the system is working. With it, you have a measurable basis for improvement, evidence of value when the budget is questioned, and early warning when the pipeline breaks or the underlying assumptions stop holding.
It need not be elaborate — a one-click useful/not-useful on each alert, reviewed monthly, tells you most of what you need.
Sizing it honestly
A rough guide to where effort lands on a typical project: collection is a modest share, cleaning and entity resolution is the largest single block by some distance, enrichment and detection together are substantial, and presentation is smaller than anyone expects.
Budgets are frequently allocated in the opposite proportion — most of it to collection and dashboards, almost none to the cleaning and detection work that determines whether the output is trustworthy or actionable. That misallocation is the direct cause of the eighteen-months-and-no-decisions outcome.
A short checklist
- Which specific recurring decision does this change, and who makes it?
- When does that decision happen, and does data arrive before it?
- How would we know within an hour if collection silently broke?
- Is the raw data preserved so we can reprocess when cleaning logic changes?
- How do we resolve records to real-world entities, and who reviews uncertain matches?
- Is the output pushed to where people work, or does it wait to be visited?
- How do we find out whether anyone acted on it?
Worked example
A procurement team does not need another chart of supplier prices. It may need a daily list of unusual price changes, checked against product identity and recent order history, with a buyer assigned to each exception. The dashboard can support the review, but the review queue and feedback loop are what connect the data to a decision.
This is an illustrative planning scenario, not a client case study.
Turn the decision into a plan
Write down the outcome you need, the evidence that would show success, the unresolved risks and the person responsible for the next decision. Use a bounded first step to test the difficult assumptions before committing to a larger programme.
ANAWAZ can help you scope the engineering work through our Data collection and engineering. Tell us about your current systems and the result you want to achieve.