Jonathan Franklin
Problem
Pricesearcher’s feed ingestion, product building, indexing and warehouse jobs depended on scheduled workflows. That orchestration layer needed to move from Airflow 1 to Airflow 2 while continuing to support existing jobs. Monitoring also needed to reflect changing feed schedules: a weekly feed should not trigger the same freshness warnings as a daily one.
Architecture
Airflow scheduled the data workflows, including validation tasks around the pipelines. SNS and CloudWatch events reached Slack through Lambda functions. Supporting services included query expansion, merchant and brand enrichment, Spark warehouse uploads and Terraform-managed infrastructure.
What I built
- Upgraded DAGs, configuration and infrastructure to Airflow 2.1.3, resolved dependency conflicts and worked through the metadata migration and later move to managed Airflow.
- Added scheduled data-validation tasks and connected pipeline notifications to Slack.
- Maintained the query-expansion Lambda, merchant and brand enrichment tasks, and Spark uploads of price-history data to S3.
- Updated vulnerable dependencies, worked on idle-instance shutdown tooling and maintained web application firewall rules.
Engineering decisions
- Freshness-check grace periods followed each feed’s schedule instead of applying a daily threshold to weekly feeds.
- Operational failures were routed from SNS and CloudWatch into the team’s existing Slack channels.
Technologies
Python, Apache Airflow, AWS (Lambda, Batch, S3, SNS, CloudWatch), Docker, Jenkins, Terraform, Slack.
Results
The orchestration layer moved to Airflow 2 and then managed infrastructure. Freshness checks accounted for the revised feed cadence, and pipeline notifications reached the team’s Slack channels. My work combined migration with ongoing maintenance of the services around the data pipelines.