https://jonnie.github.io/tags/jenkins/Recent content in Jenkins on Jonathan FranklinHugoenFri, 16 Jul 2021 00:00:00 +0000https://jonnie.github.io/work/product-search-api/Fri, 16 Jul 2021 00:00:00 +0000https://jonnie.github.io/work/product-search-api/<h2 id="problem">Problem</h2> <p>Pricesearcher&rsquo;s website, browser extension and external integrations needed product data from both the company&rsquo;s Elasticsearch index and third-party providers. Combining those sources meant dealing with different response formats, provider latency and client ranking requirements. The API also needed clearer boundaries between the interfaces used by different clients.</p> <h2 id="architecture">Architecture</h2> <p>The Python/Flask API combined the internal Elasticsearch index with Amazon, Yahoo and Kelkoo. Third-party requests ran asynchronously and provider failures were handled separately. Redis with RediSearch supported filtering, sorting and facets over cached results. Web, browser-extension, external-client and core APIs ran as separate deployments with their own OpenAPI specifications.</p>https://jonnie.github.io/work/product-catalog-pipeline/Thu, 10 Dec 2020 00:00:00 +0000https://jonnie.github.io/work/product-catalog-pipeline/<h2 id="problem">Problem</h2> <p>Pricesearcher needed to turn merchant feeds into a catalogue suitable for price comparison. Downloaded listings had to be normalised, assigned to the correct market and currency, and combined with reference data before indexing. Expansion into additional countries also required changes to feed processing, catalogue seeding and scheduling.</p> <h2 id="architecture">Architecture</h2> <p>The ingestion stage processed merchant feeds into Parquet. A Spark product builder joined them with warehouse price history, categories, translated brands, linked product IDs and image-quality data, then supplied the <a href="https://jonnie.github.io/work/search-index-pipeline/">search index pipeline</a>. Country-specific processing and schedules ran through the same flow.</p>