https://jonnie.github.io/tags/mysql/Recent content in Mysql on Jonathan FranklinHugoenThu, 10 Dec 2020 00:00:00 +0000https://jonnie.github.io/work/product-catalog-pipeline/Thu, 10 Dec 2020 00:00:00 +0000https://jonnie.github.io/work/product-catalog-pipeline/<h2 id="problem">Problem</h2> <p>Pricesearcher needed to turn merchant feeds into a catalogue suitable for price comparison. Downloaded listings had to be normalised, assigned to the correct market and currency, and combined with reference data before indexing. Expansion into additional countries also required changes to feed processing, catalogue seeding and scheduling.</p> <h2 id="architecture">Architecture</h2> <p>The ingestion stage processed merchant feeds into Parquet. A Spark product builder joined them with warehouse price history, categories, translated brands, linked product IDs and image-quality data, then supplied the <a href="https://jonnie.github.io/work/search-index-pipeline/">search index pipeline</a>. Country-specific processing and schedules ran through the same flow.</p>