Docker

Data Orchestration & Platform Operations

May 17, 2022 · Pricesearcher

Airflow migration and operational maintenance for scheduled data pipelines, including freshness checks, Slack alerts and AWS infrastructure.

pythonairflowawsdockersecurity

Elasticsearch Product Search Index

Sep 11, 2021 · Pricesearcher

Country-specific Elasticsearch indexing, product-link enrichment and Google Sheets publishing for Pricesearcher's search services.

elasticsearchpythonawsathenaavrodockerterraform

Multi-Source Product Search API

Jul 16, 2021 · Pricesearcher

Python API work combining Pricesearcher's product index with Amazon, Yahoo and Kelkoo offers, with client-specific interfaces and Redis-backed search features.

pythonflaskelasticsearchredisawsdockerterraformjenkinsopenapi

Product Web Crawler

Jun 8, 2021 · Pricesearcher

Python crawler and API work for merchant product data: S3-managed sitemaps, compressed sitemap support and crawl scheduling on AWS.

pythonawslambdas3dynamodbdockerterraform

Merchant Image Proxy

Feb 19, 2021 · Pricesearcher

Customisation and AWS deployment of the Go-based willnorris/imageproxy project for merchant product images.

goawslambdaapi-gatewayterraformdocker

Product Catalog Pipeline: Feeds to Products

Dec 10, 2020 · Pricesearcher

Merchant feed ingestion and Spark enrichment, with download retries, country-specific processing and product data prepared for search.

pythonsparkparquetmysqlawsdockerjenkins