I’ve been developing software for over 30 years. These days, as well as developing Windows software, I also build and maintain a number of cloud-based applications. Most of the time everything runs smoothly and I can get on with what I enjoy doing — writing code.

Last week was a little different.

Monday Morning

At around 7am on Monday, one of the applications I maintain stopped working. I checked the monitoring and discovered that the database was unavailable. Google Cloud SQL had initiated a failover.

Oh, here we go.

The database runs in a high-availability configuration, with a standby instance ready to take over if the primary fails. I’ve paid for that extra redundancy precisely because I don’t want an application going offline every time Google has a problem with a server.

Except it wasn’t failing over.

The Google Cloud console showed the failover operation in progress, but nothing seemed to be happening. I checked the logs, looked at the database metrics and tried to work out whether something in the application had triggered it.

Meanwhile, the application was down and there wasn’t much I could do.

After about an hour, the operation was still stuck. Google’s documentation suggested that failovers should normally complete in a matter of minutes, so I was beginning to suspect something had gone rather badly wrong.

It had.

The database finally came back online after 99 minutes.

I then spent much of the day trying to establish what had happened. Google eventually explained that the host had experienced a storage I/O stall, which triggered the failover. But the storage couldn’t immediately be detached from the original host because of a safety mechanism designed to prevent data corruption. The failover operation eventually timed out.

So much for high availability.

While digging through the logs, I also noticed that one of our database queries was doing rather more work than it needed to. It was scanning hundreds of thousands of rows when it should only have needed a few hundred. I added a composite index, which made a considerable difference, and took the opportunity to review the database monitoring and add some better alerts.

I also increased the storage allocation, partly because I’d been investigating whether disk performance might have contributed to the problem.

A productive morning, then. Just not in the way I’d intended.

Later, I discovered that Google had suffered a power outage affecting infrastructure in our region. There had been widespread disruption to cloud services across the UK, with news reports about major websites and applications going offline.

I’d been so busy investigating my own database that I hadn’t noticed.

At least that explained why Google had decided to move things around in the first place. It also made me a little more confident that the failure wasn’t caused by anything we’d done.

Still, 99 minutes is a long time to be offline when you’re paying for high availability.

Tuesday: PHP

The following day, a critical vulnerability in PHP was reported.

I’ve been using PHP for years, and while much of my newer development uses other technologies, I still have several applications and services that depend on it. Some run in Docker containers using PHP 8.3. Others are rather older.

So I needed to find out whether we were affected.

The initial reports were alarming. A vulnerability potentially allowing remote code execution, urgent recommendations to patch, warnings about exposed servers and plenty of discussion about taking applications offline until fixes were available.

I started looking into the actual conditions required to exploit it.

There’s a big difference between a vulnerability existing in a particular version of PHP and an attacker being able to exploit it in a real application. In this case, the attack required specific conditions, and I wanted to understand which of our applications, if any, met them.

That took some digging.

I reviewed the configurations, checked how requests were being handled and applied some temporary restrictions where they made sense. For services that didn’t need multipart form submissions, for example, I could disable that functionality without affecting anything useful.

I also adjusted some Cloud Armor rules to reduce exposure while waiting for an updated runtime.

Then the patched PHP 8.3 image became available. I rebuilt the Docker images, deployed the updated services and removed the temporary restrictions that were no longer needed.

Job done.

Well, nearly.

There was still the matter of an old server running PHP 7.4.

Yes, I know.

That server had been around for years and was well overdue for replacement. We’d already moved some of the applications away from it, but there were still a few things running there that made migration awkward.

I’d been putting it off.

I applied some additional restrictions to the old server, including disabling file uploads and functionality that wasn’t required. That reduced the immediate exposure, but it was becoming increasingly difficult to justify keeping an unsupported PHP version running.

Two days later, we’d built a replacement server running PHP 8.3 and spent the day migrating everything across.

There were configuration changes, compatibility issues and all the usual little surprises you discover when moving software that’s been happily running in the same environment for years.

But we got there.

The old server is now retired, which is something I’d wanted to achieve for quite some time. It just took a security scare to move it to the top of the list.

Another day gone.

And Then Railway

In the middle of all this, a friend contacted me about an application he hosts on Railway.

He’d discovered that an API key had been exposed and was trying to work out how it had happened. We discussed what might have been compromised and what he needed to do about it.

At the time, there wasn’t much information available.

The following morning, Railway disclosed a security incident.

Interesting timing.

I don’t know whether the incident was directly responsible for his exposed credentials, but it certainly gave him something else to investigate. He had to replace the affected credentials and make sure nothing had been misused.

I wasn’t involved in running that application, but it was another example of a developer having to drop everything because of something that might have happened elsewhere in the infrastructure.

By this point, I’d had enough of infrastructure problems for one week.

I Remember When Software Was Simpler

I started developing Macro Scheduler in 1997.

Back then, I wrote Windows software. I’d write the code, compile it, build an installer and upload it to my website. Customers downloaded the software, installed it on their PCs and got on with using it.

There were problems, of course. Windows compatibility issues, antivirus false positives, installation problems and the occasional bug that somehow escaped testing.

But once the software was installed and working, it generally stayed that way.

I didn’t have to worry about whether a database in a Google data centre was available. There were no cloud services to monitor, no containers to rebuild every time a vulnerability was discovered in a dependency, and no third-party platform that could suddenly stop an application working.

If something broke, there was a reasonable chance I’d caused it myself.

Which at least meant I could fix it.

I’m not pretending everything was wonderful in 1997. The tools were more limited, distributing software was harder and there were plenty of things we can do today that would have been impractical or prohibitively expensive back then.

I rather like modern cloud development, actually.

I can build an application, deploy it to a managed platform, connect it to a database and make it available to customers around the world without buying a single server. If I need more capacity, I can usually increase it with a few clicks.

That’s pretty remarkable.

But it comes at a cost, and I don’t just mean the monthly bills.

Instead of maintaining our own hardware, we’re relying on somebody else to maintain it. Our applications depend on their networks, their storage, their security procedures and whatever internal systems they’ve built to keep everything running.

Most of the time, that’s fine.

Until it isn’t.

And when something goes wrong, you can find yourself trying to diagnose a problem in a system you can’t actually see, let alone fix.

It’s Still Your Problem

The thing about running software for customers is that it doesn’t really matter whose fault an outage is.

If an application I’ve developed stops working, the people using it aren’t particularly interested in whether Google has had a power failure or whether I’ve introduced a bug.

They just want it fixed.

Fair enough.

But when Google’s failover got stuck, I couldn’t do anything to make it complete. I could look at the logs, check the status, contact support and try to work out what was happening, but the actual problem was somewhere inside Google’s infrastructure.

I had no control over it.

That’s frustrating when you’re used to being able to investigate a problem, find the cause and fix it. With desktop software, I can usually reproduce a bug, step through the code and work out what’s wrong.

With a cloud infrastructure failure, sometimes all you can do is wait for somebody else.

And hope they know what they’re doing.

Of course, there are ways to make applications more resilient. We can add monitoring, keep backups, minimise dependencies, build in redundancy and make sure our software is kept up to date.

I do all those things.

But none of them guarantees that everything will keep working. Monday’s outage was a fairly good demonstration of that. The database had a standby instance in another zone, and the mechanism designed to switch between them still managed to leave us offline for 99 minutes.

I could have designed an entirely separate database infrastructure with additional redundancy, but then I’d have had more infrastructure to maintain, more things to monitor and another collection of things that could go wrong.

And presumably another bill from Google.

There’s a balance to be struck.

Can I Get Back to Writing Software Now?

Looking back, I did actually achieve quite a lot last week.

The database is running again, and one of its more demanding queries is now considerably more efficient. I’ve improved the monitoring, updated the PHP runtimes and finally replaced a server that should probably have been retired years ago.

None of that was wasted effort.

But very little of it was what I’d intended to spend the week doing.

I had features to develop, bugs to fix and other work planned. Instead, I spent most of my time investigating infrastructure failures, reading security advisories, updating Docker images and migrating an old server.

That’s part of the job, I suppose.

When I started writing software, most of my time was spent writing software. These days, an increasing amount seems to be spent making sure all the other bits and pieces that the software depends on are still working.

I wouldn’t want to give up the capabilities that modern cloud platforms provide. They’ve made it possible to build some fantastic things with relatively little infrastructure of our own.

But I do occasionally miss the days when I could compile an EXE, upload it to a website and consider the job done.

Although, thinking about it, someone would usually email to say their antivirus had deleted it.

So perhaps not much has changed after all.

Anyway, I’ve got some coding to catch up on.