A pipeline must be tested end to end
Data quality is not established by checking only the target table. Each hop can introduce loss, duplication, transformation defects, timing problems, and operational failures.
AskAQA six-checkpoint model
1. Requirements & Testability
↓
2. Source / Ingestion Readiness
↓
3. Transformation Validation
↓
4. Curated Data Quality
↓
5. Business Consumption
↓
6. Release & Operational Readiness
1. Requirements & testability
- Source and target definitions
- Business rules
- Transformation logic
- Data quality thresholds
- Incremental/full-load rules
- Failure and recovery expectations
2. Source and ingestion readiness
- File/table arrived
- Schema correct
- Row count expected
- Encoding valid
- Nulls controlled
- Duplicates assessed
- Audit metadata captured
- Failed records visible
3. Transformation validation
Validate joins, filters, calculations, mappings, type conversions, keys, deduplication, SCD logic, business rules, and exception handling.
4. Curated data quality
Apply agreed quality rules to the resulting dataset: accuracy, completeness, consistency, validity, uniqueness, timeliness, and integrity.
5. Business consumption
Validate that downstream semantic models, reports, APIs, extracts, machine-learning features, or business users interpret the data correctly.
6. Release and operational readiness
- Scheduling
- Monitoring
- Alerting
- Retry
- Recovery
- Backfill
- Ownership
- Data-quality reporting
Test both full and incremental processing
Many pipeline defects appear only during incremental loads: updates, deletes, late-arriving data, changed keys, replay, duplicate files, or watermark logic.