Day 1: Diagnosing What Is Wrong with Data

Profiling Data Structures

• Validation of source format consistency
• Verify current field data formats
• Integrity of table relationships
• Identify anomalous observations early

Handling Missing Values

• Measures of missing data extent
• Identify reasons for absent values
• Criteria for suitable treatment methods
• Record of final treatment impact

Detecting Duplicate Records

• Define correct duplication keys
• Measure actual duplicate volume
• Authoritative records to retain
• Record of duplicate removal rules

Day 2: Standardizing Data for Consistent Analysis

Standardizing Formats and Definitions

• Standardize date formats across sources
• Align different measurement units
• Normalize primary category names
• Document approved standard definitions

Building Validation Rules

• Source value accuracy requirements
• Create data quality checks
• Specify treatment of invalid values
• Record of approved validation rules

Converting Data Types

• Identify correct field types
• Handle possible conversion errors
• Check for precision loss
• Validate results after conversion

Day 3: Delivering a Trusted Analytical Dataset

Documenting Data Lineage

• Define clear decision authority
• Transformations along the flow
• Documented sources of final fields
• Update lineage after system changes

Applying Quality Controls

• Verify source value accuracy
• Apply accuracy and completeness checks
• Monitoring of recurring quality failures
• Manage appropriate corrective actions

Approving the Final Dataset

• Run final quality checks
• Record of formally accepted exceptions
• Obtain approval from owner
• Freeze the approved analytical version

• Test Profiling Data Structures through an operational scenario
• Applied case linking Standardizing Formats and Definitions to decision quality
• Use a diagnostic exercise to uncover gaps in Documenting Data Lineage
• Complete a mini-project producing a tool for Approving the Final Dataset