Recently I’ve been looking at the large Cloud providers like AWS, Azure and GCP and how their value proposition has changed over the last decade. While On-Demand compute that these companies can provide are still great for small experimentation or low volume messaging/orchestration, the value they provide to any kind of larger compute needs has been getting worse. I’m putting together a more detailed post on these value changes overtime, where as this post is about how these proprietary systems, and the skills they require, are not only further locking in companies to their platforms, but also contributing to locking-in developers creating a feedback loop that I don’t believe leads to anywhere good.

This post is mostly a rant, so time to click off if that bothers you.

Companies trade of simpler hiring for growing hosting costs

The trade being made is very intentional from a company perspective. Job ads have always asked for hyper specific skill sets when trying to fill a position, to reduce the onboarding cost or training of new employees. This enables a more mobile workforce, with the promise of workers being replaceable cogs in the larger company machinery.

I’ve seen many positions requiring even cloud specific requirements of “Azure experience is a must”, despite that the vast majority of the offerings from AWS, Azure, and GCP are not only very similar products, but likely use the same shared open source technology under-pinnings. A good example of this would be video transcoding services. FFMpeg is such a useful and complex tool in itself, that wrapping the functionality in some easier to understand web services that only exposes functionality for the 90% most common use cases enables these platforms to sell remote compute very effectively.

The wrapping and management of these software packages, including that of more common tooling that developers still interact with like Redis, Postgres, and others, definitely has value in of itself. Cloud providers are taking away the hassle of managing physical servers for a premium. At the time of writing, the major cloud providers have been pushed hard into every software adjacent company over the last 15 years. And this combined with the percentage of developers in the workforce with 15 years or less being ~70%, means that only 30% of developers likely worked in an environment when self hosting or on-premise was the default or used at all. And this number is dropping through attrition as time marches on.

Technology always changes

This change of technology over time to a higher and higher use of Cloud offerings has actually benefited me personally as I’ve led multiple cloud-first projects, and remember far more AWS APIs than I’d care to. However, through that experience, I’ve noticed that the extractive nature of building on these platforms is getting worse over time. The sales tactics like giving large amount of cloud credits to startups, is just the tip of the iceberg when it comes to practices the large Cloud providers use to keep their customers locked-in to their platforms.

When I was building Solcast, a global weather forecasting system, on AWS, we took a $100k USD credit which had a 1 year expiration. When using those credits, I was conscious to stick with more fundamental services that offered good value. We used EC2 Spot instances almost exclusively, RDS over Aurora, Elasticache as basic Redis hosting, just to name a few.

AWS S3 was a common API, so theoretically any S3 related code should be also migratable without too much pain. However, at the beginning, before the credits, it was just myself building the tech stack, so I needed to make choices that went against avoiding lock-in to simplify my own work and achieve the company goals. AWS ECS was one of these choices. I still believe this was the correct choice at the time as it did simplify orchestration of containers, and projects like Kubernetes was still in earlier days in 2016 where is seemed like too much of a risky bet, and probably overkill.

However, there are a huge range of different incentives all aligning to keep developers using these offerings not just in their current project, but through out their whole careers, and large companies are more than happy to make this trade off if it means more interchangeable and cheaper labor. The Cloud providers play into this through their Certification programs, generous free-tiers, pricing tiers between products, huge choice in services filling every use case and gap along the way, “Solution Architects” being rebranded sales positions, and this is no where near an exhaustive list.

My hope is that the value offering will deteriorate for many compute/memory/storage intensive usage cases soon to the point where many startups will realize early on that their profitability can be much more sustainable by looking at other providers or even co-location options, despite the need for additional skills.

And I believe this is made easier today than say 10 years ago thanks to ever improving open source options. Tooling projects like Kamal by the Basecamp team which enables not only an simplified open source tool for deploying containers, but itself is built on tooling that is foundational in the same way FFMpeg is foundational to the vast majority of media encoding services. These are technologies that have been around for a long time, and are battle tested more than most. Technologies like SSH, Linux, Docker, and Containers. These are by no means shallow topics, but even having a very limited experience or understanding of how to use these technologies, I believe can by hugely beneficial in many aspects of modern software development.

You can only learn so much

When I started in my career, I was using ASP.NET in the days of .NET 2.0 (~2008), and I was building Web Forms, and dealing with things like the AJAX UpdatePanel’s shudder to perform simple callbacks to web services. All this knowledge is throw away for multiple reasons. It’s virtually not used anymore outside of large legacy projects, and mostly because it was a bad proprietary abstraction for something that existed in browsers natively. Not only that, it promoted bad development patterns and was extremely difficult to reason and interrogate how it actually worked.

This experience helped me to realize what you choose to learn is important for your career (and sanity) over the long term. Making these trade offs with intentionality is completely understandable, but you can’t make a choice about things you don’t know about. “The nearest tool is the one you’ll use”, and the people selling you those tools know how true that is. And when you are already hosted in the cloud, the nearest tools are the ones by the same cloud provider.

Right-to-repair, “Homelab” and Open source

One analogy to the end-game of the large Cloud providers would be what has happened to physical products over the last 30 years. The ability for an end-user to repair or maintain their own tools, devices, kitchenware etc has reached virtually zero. Replaceable batteries on phones use to be the default. Tearing down a device was straight forward because engineers made their devices that way, so they were serviceable. Cars were far more serviceable than they are now, I’ve changed water-pumps, belts, spark plugs, and carburetor on a car build in 1980, but can’t even approach swapping spark plugs on my 2010 car due how many parts I would have to pull apart just to reach them.

These changes over time have not occurred by accident. Large egress bills from the big three Cloud providers has always been price gouging, now they are coming for everything else. Large companies want workers to be interchangeable, as that will drive down labor costs, simplify onboarding, and outsource training.

Developers are not powerless in these matters. Open Source tooling is better than it has ever been, and compute power of old servers and workstations can still be extremely impressive. I “homelab” (generally related to self hosting, and tinkering around with old enterprise IT hardware like servers) with a dual Xeon Dell workstation from 2016 that costs me about $13 USD/month in power, and about $600 USD initial purchase price, minus the GPUs I put in it later on. It has 28 cores/56 threads, 128GB of RAM and is the equivalent of the AWS m4.10xlarge (very similar hardware), or comparing to a newer generation, the m7i.4xlarge or m7i.8xlarge depending on the benchmark. This specific setup is not even the best older workstation and server platforms for your money. And this kind of second hand hardware is plentiful and easy to buy for under $500 USD each, which is less than a month of Cloud costs for their compute equivalent.

Yes, I know Cloud takes care of a lot more than just offering compute, but the price disparity is getting more and more obvious every year, and the most common argument I hear to not co-locate or self host solutions is related to cost of labor and difficulty finding those developers with the appropriate skills.

So what?

As I stated at the start, this is mostly a rant based on my frustration at where it seems Cloud hosting and the seemingly never ending increase in its usage are going as time goes on. Playing this out, I can’t see how this won’t end with larger and larger percentages of professional developers will miss out on getting experience with the skills and tools required to reverse course, leading to higher and higher prices.

I think the only actionable advice or observations I can share is probably for developers to try to focus on foundational tooling. That is, things that underpin most other software, especially when it comes to hosting, to help avoid all the things that are working against you to get good value for money when it comes to building hosted solutions. This is based on the following opinions/observations:

• Almost everything is hosted on Linux
• Linux Containers (namespaces and cgroups) are more accessible than ever, and extremely useful
• Docker tooling is easy to use, and fairly stable/self contained
• SSH access will remain ubiquitous, flexible and secure
• Bash scripting is available in nearly every environment
• SQL interfaces are everywhere when it comes to data access

These opinions and observations above are obviously bias like everything else I’ve written above, but I think are broadly true. Enough so, that if I were to start my developer journey again, these observations would be far more useful to me than worrying about which programming language to focus on.