S3 on Jonathan Franklin

MT5 OHLCV Bars Pipeline

Fri, 04 Sep 2026 00:00:00 +0000

Problem

Trading research needed reusable histories of one-minute open, high, low, close and volume bars. MetaTrader 5 held the source data, but its Windows terminal loaded symbol history on demand. Repeated manual exports made it awkward to retrieve consistent date ranges across symbols.

Architecture

An hourly Monday-to-Friday job on a Windows EC2 instance retrieves M1 bars through the MT5 Python API. Bars are stored as compressed daily symbol/date partitions on S3, with monthly aggregates. A CLI reads selected partitions into a local cache and imports historical CSV exports into the same layout. Lambda scheduling, CloudWatch monitoring, Pulumi and PowerShell manage the runtime around the terminal.

Product Web Crawler

Tue, 08 Jun 2021 00:00:00 +0000

Problem

Pricesearcher’s price comparison service used merchant websites as a source of product data. The crawl pipeline read sitemaps, found product pages and extracted structured data, with separate deployments for environments and AWS regions. Maintaining it meant handling differences between merchant sites and balancing crawl frequency against the cost of repeated requests.

Architecture

The Python crawler ran on AWS and tracked crawl state and cycles in DynamoDB. Sitemaps stored in S3 fed product-page crawling, while a management API behind Cognito and API Gateway managed merchants and ingestion sources. A companion crawler collected website metadata for the data platform.

Affiliate Offers & Vouchers Pipeline

Tue, 19 Jan 2021 00:00:00 +0000

Problem

Pricesearcher’s deals content came from affiliate networks including Commission Junction and Webgains. Their authenticated feeds used different formats and conventions for promotions, dates and links. The data needed a common shape and a connection to the merchant catalogue before it could support the site’s deals search.

Architecture

Authenticated and paginated feeds were downloaded to S3, then SQS messages triggered network-specific processing. Normalised promotions were written to an Elasticsearch index for the site’s offers and vouchers search. A separate synchronisation path maintained Kelkoo pass-through URLs between Redis and Google Sheets.