Jonathan Franklin
Problem
Pricesearcher’s deals content came from affiliate networks including Commission Junction and Webgains. Their authenticated feeds used different formats and conventions for promotions, dates and links. The data needed a common shape and a connection to the merchant catalogue before it could support the site’s deals search.
Architecture
Authenticated and paginated feeds were downloaded to S3, then SQS messages triggered network-specific processing. Normalised promotions were written to an Elasticsearch index for the site’s offers and vouchers search. A separate synchronisation path maintained Kelkoo pass-through URLs between Redis and Google Sheets.
What I built
- Added authenticated and paginated downloads, gzip content handling and network-specific configuration.
- Worked on processors for network feeds, including Commission Junction offer JSON processed with pandas.
- Worked on the Elasticsearch promotions index.
- Maintained the Kelkoo link-cache synchronisation.
Engineering decisions
- Network-specific configuration and processors isolated differences in authentication and feed format from the common pipeline stages.
- Normalisation handled missing fields, standardised promotion types and datetimes, derived merchant keys from deeplinks and mapped destination URLs to affiliate links.
- S3 storage and SQS messages separated downloading from subsequent processing.
Technologies
Python, pandas, AWS (S3, SQS, Lambda, IAM), Redis, Elasticsearch, Google Sheets.
Results
The pipeline gave the integrated networks a shared route from downloaded feeds to searchable promotions. Network-specific configuration and processors kept differences in authentication and feed format separate from the common processing and indexing stages.