Jonathan Franklin
Problem
Pricesearcher’s search service used separate Elasticsearch product indexes for different countries. Index builds needed to carry product identity and cross-merchant links into searchable records, while accommodating market-specific merchant rules and schedules. The team also used Google Sheets to maintain smaller datasets that needed to be searchable.
Architecture
A Makefile-driven pipeline generated Avro source data through Athena, downloaded it from S3 and used a PyPy builder to populate Elasticsearch. It supported the site’s nitor index and a separate nested index type. Separate configurable builders published selected Google Sheet tabs as Elasticsearch indexes.
What I built
- Added indexing support, merchant rules and build schedules for Romania, Slovakia, Greece and Hungary.
- Built a linked-product mapping table and joined its counts into index records, including products without links.
- Added deep-link tokens, MPNs, SKUs and source-table identifiers for product resolution.
- Worked on the Google Sheets index builders and their mapper service, including document mapping, validation and reindexing.
Engineering decisions
- Country-specific inclusion rules and schedules remained within the shared indexing pipeline.
- Cluster instance sizing, JVM heap and the GB rebuild schedule were adjusted to limit resource use, explicitly trading rebuild frequency against data freshness.
- Sheet-driven indexes provided a smaller publishing route without creating a dedicated feed pipeline for each dataset.
Technologies
Python (PyPy 3.6 builder), Elasticsearch, AWS Athena, S3, Avro, Google Sheets, Docker (ECR), Terraform, Jenkins.
Results
The additional countries could use the indexing pipeline, and search records carried more information for linking merchant offers and resolving product identities. The sheet-driven builders provided a separate route for publishing smaller datasets without a dedicated feed pipeline for each one.