Tutorial

The quickstart example uses a simple dataset and is primarily intended to show a user how to run the ingest pipeline and view basic outputs. This tutorial uses a more complex study dataset and steps through each ingest stage in more depth.

In this tutorial we will cover (not necessarily in this order):

1. How to configure complex extractions with multiple messy source data files

2. How to write a transform function for the transform stage

3. How to run the ingest pipeline from end to end

4. How to test your ingest package each step of the way

5. How to debug steps 1 through 3 by:

   • Running stages individually

   • Viewing stage outputs

   • Viewing ingest logs

6. Basic usage and configuration of ingest app-level settings

Contents

• Kids First Ingest App
  • App Settings
• New Ingest Package Setup
  • Ingest Package Components
• Extract Stage
  • Example Extract Configuration
  • Preprocessing Before the Operations List
  • Files That Need Custom Readers
  • Predefined Mapping Operations
  • Important Strategic Details
• Testing Your Ingest Package
  • The test CLI Command
  • Data Validation
  • Run Subsets of the Pipeline
• Common Challenges when Extracting
  • Values Embedded in Other Values
  • Multiple Columns of The Same Kind of Thing
  • Multiple Meltable Groups
  • Sharing Operations Across Multiple Extracts
  • Data That Only Makes Sense In Relation To Another File
• Sourcing Data
  • Source Data stored by the Kids First Data Tracker
  • Source data stored in a PostgreSQL Database
• Transform Stage
  • Guided Transform Module
  • Test Your Package
• Target Service Plugins
  • Making a Target Service Plugin
  • Using a Target Service Plugin
• Understanding the Load Stage
  • What the Load Stage Expects
  • Using The Load Stage On Its Own