Jonathan Franklin
Problem
Pricesearcher needed to turn merchant feeds into a catalogue suitable for price comparison. Downloaded listings had to be normalised, assigned to the correct market and currency, and combined with reference data before indexing. Expansion into additional countries also required changes to feed processing, catalogue seeding and scheduling.
Architecture
The ingestion stage processed merchant feeds into Parquet. A Spark product builder joined them with warehouse price history, categories, translated brands, linked product IDs and image-quality data, then supplied the search index pipeline. Country-specific processing and schedules ran through the same flow.
What I built
- Added download retries, country-to-currency mappings and country-specific schedules.
- Extended feed processing and product building for Greece, Hungary, Romania and Slovakia.
- Reworked country handling and fixed a bug that wrote Hungarian products to Greek tables.
- Added out-of-stock exclusions for specific merchants and maintained dependencies across the pipeline.
Engineering decisions
- The additional markets extended the existing ingestion and enrichment pipeline rather than creating a separate system for each country.
- Catalogue seeding reused the established GB catalogue with adjusted product IDs and destination tables.
- The GB feed cycle ran less frequently than other markets, while retries handled transient download failures.
Technologies
Python, Apache Spark, Parquet, MySQL, AWS (Batch, S3, Athena), Docker, Jenkins, Terraform.
Results
The pipeline supported the additional countries within the existing ingestion and enrichment flow. Download retries addressed transient feed failures, while the country-routing fix kept Hungarian output in its intended tables. The enriched catalogue supplied the search index pipeline.