Elasticsearch on Jonathan Franklin

Elasticsearch Product Search Index

Sat, 11 Sep 2021 00:00:00 +0000

Problem

Pricesearcher’s search service used separate Elasticsearch product indexes for different countries. Index builds needed to carry product identity and cross-merchant links into searchable records, while accommodating market-specific merchant rules and schedules. The team also used Google Sheets to maintain smaller datasets that needed to be searchable.

Architecture

A Makefile-driven pipeline generated Avro source data through Athena, downloaded it from S3 and used a PyPy builder to populate Elasticsearch. It supported the site’s nitor index and a separate nested index type. Separate configurable builders published selected Google Sheet tabs as Elasticsearch indexes.

Multi-Source Product Search API

Fri, 16 Jul 2021 00:00:00 +0000

Problem

Pricesearcher’s website, browser extension and external integrations needed product data from both the company’s Elasticsearch index and third-party providers. Combining those sources meant dealing with different response formats, provider latency and client ranking requirements. The API also needed clearer boundaries between the interfaces used by different clients.

Architecture

The Python/Flask API combined the internal Elasticsearch index with Amazon, Yahoo and Kelkoo. Third-party requests ran asynchronously and provider failures were handled separately. Redis with RediSearch supported filtering, sorting and facets over cached results. Web, browser-extension, external-client and core APIs ran as separate deployments with their own OpenAPI specifications.

Affiliate Offers & Vouchers Pipeline

Tue, 19 Jan 2021 00:00:00 +0000

Problem

Pricesearcher’s deals content came from affiliate networks including Commission Junction and Webgains. Their authenticated feeds used different formats and conventions for promotions, dates and links. The data needed a common shape and a connection to the merchant catalogue before it could support the site’s deals search.

Architecture

Authenticated and paginated feeds were downloaded to S3, then SQS messages triggered network-specific processing. Normalised promotions were written to an Elasticsearch index for the site’s offers and vouchers search. A separate synchronisation path maintained Kelkoo pass-through URLs between Redis and Google Sheets.