Jonathan Franklin
Problem
Trading research needed reusable histories of one-minute open, high, low, close and volume bars. MetaTrader 5 held the source data, but its Windows terminal loaded symbol history on demand. Repeated manual exports made it awkward to retrieve consistent date ranges across symbols.
Architecture
An hourly Monday-to-Friday job on a Windows EC2 instance retrieves M1 bars through the MT5 Python API. Bars are stored as compressed daily symbol/date partitions on S3, with monthly aggregates. A CLI reads selected partitions into a local cache and imports historical CSV exports into the same layout. Lambda scheduling, CloudWatch monitoring, Pulumi and PowerShell manage the runtime around the terminal.
What I built
- Built scheduled extraction with configurable symbol filters and a tick-preload fallback when terminal history was not loaded.
- Built partitioned Parquet storage and the read-merge-deduplicate-write import process.
- Built the CLI for symbol discovery, date-range cache updates and historical M1 CSV backfills.
- Added logging, heartbeat monitoring, bank-holiday handling and timestamp/timezone fixes.
Engineering decisions
- Partitioned files on S3 matched the main symbol-and-date access pattern without adding a database and query service.
- Repeated imports are idempotent, and local cache updates skip the in-progress day rather than cache a partition still being populated.
- Scheduled startup and shutdown limit idle EC2 compute use.
Technologies
Python, MetaTrader 5, pandas and Parquet, AWS (S3, EC2, Lambda, CloudWatch), Pulumi, uv, PowerShell.
Results
The pipeline provides a shared bar history for the trading platform and its research tools. Routine retrieval is a filtered S3 read or a local cache update, while CSV imports support deeper historical backfills.
The data remains tied to the history available from the terminal and broker, with routine extraction and manual backfills feeding the same storage layout.