https://jonnie.github.io/tags/airflow/Recent content in Airflow on Jonathan FranklinHugoenFri, 12 May 2023 00:00:00 +0000https://jonnie.github.io/work/ads-revenue-platform/Fri, 12 May 2023 00:00:00 +0000https://jonnie.github.io/work/ads-revenue-platform/<h2 id="problem">Problem</h2> <p>Pricesearcher bought traffic through Google, Bing and Facebook advertising and earned revenue through affiliate and traffic partners. Comparing advertising costs with that income required joining records from separate systems with different identifiers, timezones and reporting periods. Campaign- and keyword-level reporting helped inform bid and budget decisions.</p> <h2 id="architecture">Architecture</h2> <p>Airflow and Lambda jobs imported campaign, ad-group and keyword data alongside revenue and conversion data from affiliate partners. A Python reporting API joined that data for an internal Next.js/Tailwind dashboard. The surrounding platform also used S3-hosted landing-page variants and SNS-to-Slack notifications.</p>https://jonnie.github.io/work/data-orchestration/Tue, 17 May 2022 00:00:00 +0000https://jonnie.github.io/work/data-orchestration/<h2 id="problem">Problem</h2> <p>Pricesearcher&rsquo;s feed ingestion, product building, indexing and warehouse jobs depended on scheduled workflows. That orchestration layer needed to move from Airflow 1 to Airflow 2 while continuing to support existing jobs. Monitoring also needed to reflect changing feed schedules: a weekly feed should not trigger the same freshness warnings as a daily one.</p> <h2 id="architecture">Architecture</h2> <p>Airflow scheduled the data workflows, including validation tasks around the pipelines. SNS and CloudWatch events reached Slack through Lambda functions. Supporting services included query expansion, merchant and brand enrichment, Spark warehouse uploads and Terraform-managed infrastructure.</p>