Jonathan Franklin
Problem
Pricesearcher’s price comparison service used merchant websites as a source of product data. The crawl pipeline read sitemaps, found product pages and extracted structured data, with separate deployments for environments and AWS regions. Maintaining it meant handling differences between merchant sites and balancing crawl frequency against the cost of repeated requests.
Architecture
The Python crawler ran on AWS and tracked crawl state and cycles in DynamoDB. Sitemaps stored in S3 fed product-page crawling, while a management API behind Cognito and API Gateway managed merchants and ingestion sources. A companion crawler collected website metadata for the data platform.
What I built
- Moved sitemap sources to S3 and added support for gzipped sitemaps.
- Adjusted selected crawl schedules.
- Worked on merchant and ingestion-source API endpoints, authentication, SQLAlchemy models and deployment-time database migrations.
- Worked on the companion website-metadata crawler.
Engineering decisions
- S3-hosted sitemap sources could be maintained without redeploying the crawler.
- Less frequently changing pages were crawled less often, trading the interval between price updates against repeated-request cost.
- Separate deployments supported different environments and AWS regions.
Technologies
Python, AWS Lambda, S3, DynamoDB, Cognito and API Gateway, SQLAlchemy, Docker, Terraform, Jenkins.
Results
The changes made sitemap sources easier to maintain and extended the crawler’s handling of compressed input. Alongside the scheduling work, the management API provided a way to manage merchants and ingestion sources around the crawl pipeline.