Docker on Jonathan Franklin

Data Orchestration & Platform Operations

Tue, 17 May 2022 00:00:00 +0000

Problem

Pricesearcher’s feed ingestion, product building, indexing and warehouse jobs depended on scheduled workflows. That orchestration layer needed to move from Airflow 1 to Airflow 2 while continuing to support existing jobs. Monitoring also needed to reflect changing feed schedules: a weekly feed should not trigger the same freshness warnings as a daily one.

Architecture

Airflow scheduled the data workflows, including validation tasks around the pipelines. SNS and CloudWatch events reached Slack through Lambda functions. Supporting services included query expansion, merchant and brand enrichment, Spark warehouse uploads and Terraform-managed infrastructure.

Elasticsearch Product Search Index

Sat, 11 Sep 2021 00:00:00 +0000

Problem

Pricesearcher’s search service used separate Elasticsearch product indexes for different countries. Index builds needed to carry product identity and cross-merchant links into searchable records, while accommodating market-specific merchant rules and schedules. The team also used Google Sheets to maintain smaller datasets that needed to be searchable.

Architecture

A Makefile-driven pipeline generated Avro source data through Athena, downloaded it from S3 and used a PyPy builder to populate Elasticsearch. It supported the site’s nitor index and a separate nested index type. Separate configurable builders published selected Google Sheet tabs as Elasticsearch indexes.

Multi-Source Product Search API

Fri, 16 Jul 2021 00:00:00 +0000

Problem

Pricesearcher’s website, browser extension and external integrations needed product data from both the company’s Elasticsearch index and third-party providers. Combining those sources meant dealing with different response formats, provider latency and client ranking requirements. The API also needed clearer boundaries between the interfaces used by different clients.

Architecture

The Python/Flask API combined the internal Elasticsearch index with Amazon, Yahoo and Kelkoo. Third-party requests ran asynchronously and provider failures were handled separately. Redis with RediSearch supported filtering, sorting and facets over cached results. Web, browser-extension, external-client and core APIs ran as separate deployments with their own OpenAPI specifications.

Product Web Crawler

Tue, 08 Jun 2021 00:00:00 +0000

Problem

Pricesearcher’s price comparison service used merchant websites as a source of product data. The crawl pipeline read sitemaps, found product pages and extracted structured data, with separate deployments for environments and AWS regions. Maintaining it meant handling differences between merchant sites and balancing crawl frequency against the cost of repeated requests.

Architecture

The Python crawler ran on AWS and tracked crawl state and cycles in DynamoDB. Sitemaps stored in S3 fed product-page crawling, while a management API behind Cognito and API Gateway managed merchants and ingestion sources. A companion crawler collected website metadata for the data platform.

Merchant Image Proxy

Fri, 19 Feb 2021 00:00:00 +0000

Problem

Pricesearcher displayed product images hosted by merchants, with differing URL formats and image-serving behaviour. The site needed a common endpoint for image requests, resizing and caching.

Architecture

The service was the company’s fork of willnorris/imageproxy, exposed as a signed company-domain endpoint and deployed on AWS Lambda behind API Gateway. Terraform managed separate staging and production environments.

What I built

Maintained and customised the existing Go fork.
Added options to pass through the original image or return no image when processing failed, instead of always using the default placeholder.
Fixed decoding and encoding behaviour that corrupted proxied URLs containing special characters.
Aligned the Terraform version with the deployments’ remote state.

Engineering decisions

The implementation extended the upstream proxy rather than replacing its existing resizing and caching behaviour.
Fallback behaviour was configurable between the original image, no image and the upstream default placeholder.

Technologies

Go, AWS Lambda, API Gateway, Terraform, Docker, Jenkins.

Product Catalog Pipeline: Feeds to Products

Thu, 10 Dec 2020 00:00:00 +0000

Problem

Pricesearcher needed to turn merchant feeds into a catalogue suitable for price comparison. Downloaded listings had to be normalised, assigned to the correct market and currency, and combined with reference data before indexing. Expansion into additional countries also required changes to feed processing, catalogue seeding and scheduling.

Architecture

The ingestion stage processed merchant feeds into Parquet. A Spark product builder joined them with warehouse price history, categories, translated brands, linked product IDs and image-quality data, then supplied the search index pipeline. Country-specific processing and schedules ran through the same flow.