https://jonnie.github.io/tags/security/Recent content in Security on Jonathan FranklinHugoenTue, 17 May 2022 00:00:00 +0000https://jonnie.github.io/work/data-orchestration/Tue, 17 May 2022 00:00:00 +0000https://jonnie.github.io/work/data-orchestration/<h2 id="problem">Problem</h2> <p>Pricesearcher&rsquo;s feed ingestion, product building, indexing and warehouse jobs depended on scheduled workflows. That orchestration layer needed to move from Airflow 1 to Airflow 2 while continuing to support existing jobs. Monitoring also needed to reflect changing feed schedules: a weekly feed should not trigger the same freshness warnings as a daily one.</p> <h2 id="architecture">Architecture</h2> <p>Airflow scheduled the data workflows, including validation tasks around the pipelines. SNS and CloudWatch events reached Slack through Lambda functions. Supporting services included query expansion, merchant and brand enrichment, Spark warehouse uploads and Terraform-managed infrastructure.</p>