https://jonnie.github.io/tags/parquet/Recent content in Parquet on Jonathan FranklinHugoenFri, 04 Sep 2026 00:00:00 +0000https://jonnie.github.io/work/mt5-ohlcv-pipeline/Fri, 04 Sep 2026 00:00:00 +0000https://jonnie.github.io/work/mt5-ohlcv-pipeline/<h2 id="problem">Problem</h2> <p>Trading research needed reusable histories of one-minute open, high, low, close and volume bars. MetaTrader 5 held the source data, but its Windows terminal loaded symbol history on demand. Repeated manual exports made it awkward to retrieve consistent date ranges across symbols.</p> <h2 id="architecture">Architecture</h2> <p>An hourly Monday-to-Friday job on a Windows EC2 instance retrieves M1 bars through the MT5 Python API. Bars are stored as compressed daily symbol/date partitions on S3, with monthly aggregates. A CLI reads selected partitions into a local cache and imports historical CSV exports into the same layout. Lambda scheduling, CloudWatch monitoring, Pulumi and PowerShell manage the runtime around the terminal.</p>https://jonnie.github.io/work/product-catalog-pipeline/Thu, 10 Dec 2020 00:00:00 +0000https://jonnie.github.io/work/product-catalog-pipeline/<h2 id="problem">Problem</h2> <p>Pricesearcher needed to turn merchant feeds into a catalogue suitable for price comparison. Downloaded listings had to be normalised, assigned to the correct market and currency, and combined with reference data before indexing. Expansion into additional countries also required changes to feed processing, catalogue seeding and scheduling.</p> <h2 id="architecture">Architecture</h2> <p>The ingestion stage processed merchant feeds into Parquet. A Spark product builder joined them with warehouse price history, categories, translated brands, linked product IDs and image-quality data, then supplied the <a href="https://jonnie.github.io/work/search-index-pipeline/">search index pipeline</a>. Country-specific processing and schedules ran through the same flow.</p>