Darren Reid

Latest Posts

Cloud Lock-in Reinforcement

Recently I’ve been looking at the large Cloud providers like AWS, Azure and GCP and how their value proposition has changed over the last decade. While On-Demand compute that these companies can provide are still great for small experimentation or low volume messaging/orchestration, the value they provide to any kind of larger compute needs has been getting worse. I’m putting together a more detailed post on these value changes overtime, where as this post is about how these proprietary systems, and the skills they require, are not only further locking in companies to their platforms, but also contributing to locking-in developers creating a feedback loop that I don’t believe leads to anywhere good.

Read more →

Postgres Performance Monitoring with PgHero

In a previous role, I needed to optimize a Postgres database to store 2-3 billion total rows using an r4.xlarge instance on AWS RDS. This was storing weather data parameters for the world where parts were being updated every 5-15 minutes as new data came in, and old data needed to be removed. The total data stored was only for 2-3 weeks, so the whole database would be recycled with new data in that time frame.

Read more →

Redis Optimizations

I thought I’d write up some of my experiences at Solcast building a global solar forecasting system, and some of the challenges I had storing and updating millions of forecasts every 5-15 minutes. Problem We needed somewhere to store weather and solar forecast data being generated every time a new satellite image was pulled from various geo-stationary satellites like Himawari-8/9, GOES-17, and others. Each time a new satellite image was pulled in, a set of geo-spatial positions within that region was used to generate a forecast for each point.

Read more →