https://jonnie.github.io/tags/dynamodb/Recent content in Dynamodb on Jonathan FranklinHugoenTue, 08 Jun 2021 00:00:00 +0000https://jonnie.github.io/work/product-web-crawler/Tue, 08 Jun 2021 00:00:00 +0000https://jonnie.github.io/work/product-web-crawler/<h2 id="problem">Problem</h2> <p>Pricesearcher&rsquo;s price comparison service used merchant websites as a source of product data. The crawl pipeline read sitemaps, found product pages and extracted structured data, with separate deployments for environments and AWS regions. Maintaining it meant handling differences between merchant sites and balancing crawl frequency against the cost of repeated requests.</p> <h2 id="architecture">Architecture</h2> <p>The Python crawler ran on AWS and tracked crawl state and cycles in DynamoDB. Sitemaps stored in S3 fed product-page crawling, while a management API behind Cognito and API Gateway managed merchants and ingestion sources. A companion crawler collected website metadata for the data platform.</p>