https://jonnie.github.io/tags/lambda/Recent content in Lambda on Jonathan FranklinHugoenFri, 04 Sep 2026 00:00:00 +0000https://jonnie.github.io/work/mt5-ohlcv-pipeline/Fri, 04 Sep 2026 00:00:00 +0000https://jonnie.github.io/work/mt5-ohlcv-pipeline/<h2 id="problem">Problem</h2> <p>Trading research needed reusable histories of one-minute open, high, low, close and volume bars. MetaTrader 5 held the source data, but its Windows terminal loaded symbol history on demand. Repeated manual exports made it awkward to retrieve consistent date ranges across symbols.</p> <h2 id="architecture">Architecture</h2> <p>An hourly Monday-to-Friday job on a Windows EC2 instance retrieves M1 bars through the MT5 Python API. Bars are stored as compressed daily symbol/date partitions on S3, with monthly aggregates. A CLI reads selected partitions into a local cache and imports historical CSV exports into the same layout. Lambda scheduling, CloudWatch monitoring, Pulumi and PowerShell manage the runtime around the terminal.</p>https://jonnie.github.io/work/product-web-crawler/Tue, 08 Jun 2021 00:00:00 +0000https://jonnie.github.io/work/product-web-crawler/<h2 id="problem">Problem</h2> <p>Pricesearcher&rsquo;s price comparison service used merchant websites as a source of product data. The crawl pipeline read sitemaps, found product pages and extracted structured data, with separate deployments for environments and AWS regions. Maintaining it meant handling differences between merchant sites and balancing crawl frequency against the cost of repeated requests.</p> <h2 id="architecture">Architecture</h2> <p>The Python crawler ran on AWS and tracked crawl state and cycles in DynamoDB. Sitemaps stored in S3 fed product-page crawling, while a management API behind Cognito and API Gateway managed merchants and ingestion sources. A companion crawler collected website metadata for the data platform.</p>https://jonnie.github.io/work/merchant-image-proxy/Fri, 19 Feb 2021 00:00:00 +0000https://jonnie.github.io/work/merchant-image-proxy/<h2 id="problem">Problem</h2> <p>Pricesearcher displayed product images hosted by merchants, with differing URL formats and image-serving behaviour. The site needed a common endpoint for image requests, resizing and caching.</p> <h2 id="architecture">Architecture</h2> <p>The service was the company&rsquo;s fork of <a href="https://github.com/willnorris/imageproxy">willnorris/imageproxy</a>, exposed as a signed company-domain endpoint and deployed on AWS Lambda behind API Gateway. Terraform managed separate staging and production environments.</p> <h2 id="what-i-built">What I built</h2> <ul> <li>Maintained and customised the existing Go fork.</li> <li>Added options to pass through the original image or return no image when processing failed, instead of always using the default placeholder.</li> <li>Fixed decoding and encoding behaviour that corrupted proxied URLs containing special characters.</li> <li>Aligned the Terraform version with the deployments&rsquo; remote state.</li> </ul> <h2 id="engineering-decisions">Engineering decisions</h2> <ul> <li>The implementation extended the upstream proxy rather than replacing its existing resizing and caching behaviour.</li> <li>Fallback behaviour was configurable between the original image, no image and the upstream default placeholder.</li> </ul> <h2 id="technologies">Technologies</h2> <p>Go, AWS Lambda, API Gateway, Terraform, Docker, Jenkins.</p>