https://jonnie.github.io/tags/avro/Recent content in Avro on Jonathan FranklinHugoenSat, 11 Sep 2021 00:00:00 +0000https://jonnie.github.io/work/search-index-pipeline/Sat, 11 Sep 2021 00:00:00 +0000https://jonnie.github.io/work/search-index-pipeline/<h2 id="problem">Problem</h2> <p>Pricesearcher&rsquo;s search service used separate Elasticsearch product indexes for different countries. Index builds needed to carry product identity and cross-merchant links into searchable records, while accommodating market-specific merchant rules and schedules. The team also used Google Sheets to maintain smaller datasets that needed to be searchable.</p> <h2 id="architecture">Architecture</h2> <p>A Makefile-driven pipeline generated Avro source data through Athena, downloaded it from S3 and used a PyPy builder to populate Elasticsearch. It supported the site&rsquo;s <code>nitor</code> index and a separate <code>nested</code> index type. Separate configurable builders published selected Google Sheet tabs as Elasticsearch indexes.</p>