SpinDance

The Essential Role of Software Maintenance in IoT success

No one gets excited about “maintenance.” We like buying and building new things: driving a new car off the lot, or closing on a new home. These are events we look forward to. But paying to maintain these investments? That’s a different story. No one particularly enjoys paying for oil changes or tire rotations. And we certainly don’t like replacing a broken water heater.

But while we dislike paying for maintenance, we understand timely upkeep protects our investments, and keeps us safe. Skipping or delaying maintenance might save us some money and hassle in the short run. But ultimately we’re just costing ourselves more in the future. 

One of my cousins learned this the hard way. She purchased a used red convertible on her 18th birthday. It was a beautiful car. She was extremely proud of it, and loved driving down the road with the top down. But due to some miscommunication with her father, oil changes were neglected. Eventually the engine seized up, turning that exciting car into a very expensive paper weight. All because some relatively inexpensive maintenance was missed.

Like physical investments such as cars and homes, software systems require maintenance as well. They may not need oil changes or new tires, but applications need regular updates to stay in working order. 

This is particularly true of IoT products, which are composed of many types of software, such as embedded software and cloud software. They often include web software for commercial IoT applications, and mobile apps for consumer products.

Some aspects of software maintenance are obvious. Aging user interfaces and slowing performances are two examples. We’ve all used old, out of date user interfaces, or systems that slow down with age. When these issues arise, it’s clear some work needs to be done.

But many, and perhaps most, maintenance needs hide beneath the surface. My cousin couldn’t see the oil in her car, but that didn’t make it any less critical. The same is true with software.

Unfortunately, the software industry typically does a poor job explaining the types of maintenance a software system might need. And rarely does a custom software system come with a list of regularly scheduled maintenance. This leaves the buyer holding the bag when things go wrong. This is not good for the buyer, the builder, or the end users. 

As an IoT software consultancy, SpinDance is working to change this. We’re introducing some new maintenance-centric tools for our customers in 2023. In preparation for rolling that out, we’ve been thinking a lot about the long-term health of the IoT software we build. Every software system is a bit different, so it’s impossible to have a single, definitive list of maintenance tasks. But there are some good rules of thumbs.

The Need for Software Maintenance

It might not be apparent why we need software maintenance at all. We intuitively understand why physical things need maintenance. Wear and tear is obvious when it comes to cars or buildings. But how can digital products degrade? 

Software itself doesn’t wear out per se. But the larger software ecosystem is constantly changing, evolving and improving. Hardware, operating systems, programming languages, and 3rd-party libraries are continuously changing. Unfortunately, you can’t ignore these changes. 

The usage of software changes as well. Users demand modifications. Software interface styles evolve. Systems that initially supported 100s or 1,000s of users now need to support tens of thousands. Regular maintenance can address all these issues, and more. 

Finally, as systems evolve, they can gain so-called “technical debt”, which is a natural part of the software development process. Carrying a reasonable amount of technical debt is ok, but eventually you need to pay it down to keep things manageable. 

Expended Life Expectancy For Software

The need for maintenance is primarily a function of time. The longer a system needs to stick around, the more maintenance it’s going to require. If you are going to throw something away after a single use, there is no need for a maintenance plan. On the other hand, something that is meant to last years or decades will definitely require maintenance. 

For long-living software, such as IoT software, you’ll want to put together a maintenance plan. A schedule for regular updates, a plan for contingencies, for example. And you’ll want to budget for these things. 

The plan should cover the duration of the software system’s life expectancy. A good starting point, therefore, is to ask “how long do I need to keep this software running?” From there you can begin to put an appropriate plan together. 

A Comprehensive Software Maintenance Program

What should be included in the plan?  Broadly speaking, there are four types of maintenance:

1. Regular, planned maintenance. 
2. Corrective maintenance. 
3. Emergency maintenance.
4. Improvements.

Here’s a deeper look at each type. Then we’ll wrap up by talking about budgets and timelines.

Regular Maintenance

Regular maintenance are tasks that can be anticipated for and scheduled in advance. These are the oil changes, tire rotations or fluid flushes in a car: you know you need them, and they happen on a regular basis.

Examples of software maintenance that can be scheduled include:

1. Updates to third-party libraries. Most software is built with third-party libraries. These could be open source or for-pay. These libraries are updated to improve performance and security, fix bugs, or add new features. These updates should be applied on a regular basis. The actual schedule depends on many factors, but bi-annual or annual updates are a good starting point.
2. Updates to operating systems. Operating systems like iOS, Android, Linux and Windows get updated at least once a year. Like updates to libraries, these changes include fixes and improvements. Again, planning on annual updates is a good idea.
3. Updates to programming languages. Some programming languages, like C, change very infrequently. Others, like Python, receive major updates once a year. Over time, the maintainers of these quicker-languages “deprecate” older versions. These versions no longer receive critical updates, like security fixes. Build a plan that takes the languages you’re using into consideration.
4. Adjustments due to usage changes. Software systems grow and shrink over time. As a rule of thumb, a good software architecture should be designed for 10x expected usage. For example, if a system is expected to have 100,000 users, the architecture should support 1,000,000 users. But while the architecture can support a higher level, the implementation and/or configuration may not. Launching with 10,000 users, but paying the costs of 1,000,000, is wasteful. Therefore, over time you’ll have maintenance tasks to scale up the infrastructure for your system. Conversely, near the end of its life you might scale it down.
5. Reducing technical debt. “Technical Debt” is a general term for the cruft that builds up over time in a codebase. Technical debt isn’t bugs. Rather, it is composed of things like redundant code, unneeded code, inefficient algorithms, or any of a dozen other examples of cruft. Even the best software teams generate technical debt, and having a little debt isn’t bad. But eventually it leads to maintainability and stability problems. Paying it down on a regular basis is a good idea. 

Corrective Maintenance

Corrective Maintenance is the second type of software maintenance you should plan on. This is for fixing something that is broken, but is not (yet) an emergency.  A broken tail light, a cracked windshield, or a damaged muffler are all examples of issues that require corrective action, but are not urgent. 

Here are some examples of corrective maintenance in the software world:

1. Non-critical bug fixes. All software systems have bugs. Some bugs are emergencies, but most are not. In general, it’s always cheaper to batch bug fixes together. This reduces costs in development, QA and releases. Some types of software allow quick roll out of bug fixes: cloud-based software are a good example of this. Other types of software, like embedded firmware, require much longer cycle times. Either way, set aside some money for bug fixes. For custom software, you’ll need more for the first year for a new system, and after that it will taper off. Quarterly updates are a good place to start.
2. Low-level security issues. As noted above, many software systems rely on third-party tools like libraries, operating systems and programming languages. All these systems could (and often do) experience security issues. Some of these issues are emergencies, and we’ll discuss those below. Like bugs, these issues can range from minor to severe. And like bugs, lower-level security issues can be batched up.

Emergency Maintenance 

The third type of maintenance is emergency maintenance. Emergencies can range in scope and impact. A blown tire is an emergency, but is not at the level of a hurricane. But regardless of scope, emergencies need to be dealt with as soon as possible.

Examples of software emergencies include:

1. Critical bugs. While most bugs are non-critical, some are. Critical bugs include losing critical data, or causing the system to stop functioning entirely. These issues need to be resolved quickly.
2. Zero-day security exploits. In the land of computer security, the worst sort of issue is the so-called “zero-day” exploit. This is an issue where security researchers had no prior knowledge, and a fix needs to be discovered and applied immediately. 

In the case of emergency maintenance, the issue is not just cost, but timing. It can take time to triage the issue, determine the courses of action, and apply the fix. Doing this under pressure can be stressful. Having a solid emergency plan in place ahead of time can be immensely helpful.

Here are a few tips for building an emergency plan: 

1. Have contracts in place ahead of time. It’s no fun to negotiate on top of dealing with an emergency.
2. Understand the partners’ Service Level Agreements. A Service Level Agreement, also known as an SLA, is a document that defines the level of service you can expect from a provider. For example, Amazon Web Services’ SLA states an expected system uptime of 99.95%. In plain English, this means “the system could be down for 22 minutes a month before we agree to compensate you for downtime.” Labor-centric SLAs focus on response times, after-hours support hours, and special rates. Make sure your stakeholders understand the SLAs of your partners.
3. Put together an internal and external communication plan. Your stakeholders will want regular updates, as will your end-users. Make sure you have a clear plan to communicate inside your organization, and externally. Work with your PR and legal team to make sure they’re in the loop.
4. Build-in monitoring tools. Diagnosing the source of a software problem can be tricky. Make sure you’ve built monitoring and observability tools for your systems, to aid with triage and recovery efforts.
5. Practice. Like all emergency plans, it’s a good idea to practice. This helps work out any hiccups, and can work to take the stress out of the moment. Amazon Web Services calls this a “game day” and has many solid recommendations for how to execute a practice session. 

Improvements 

The final category of maintenance are improvements. Real-world examples of improvements include fresh paint, new carpet, or new rims on a car. Improvements aren’t strictly necessary, but often create a more productive and appealing environment for users. Therefore, it’s a good idea to set aside a budget for improvements.

The scope of improvements can vary greatly. In the real world, for example, upgrading to a 4K TV is a type of improvement. So is the addition of a sunroom. Both are improvements, but the difference in scope, cost and impact are huge. In the same way, you’ll likely want to budget for smaller vs. large improvements. 

Examples of software improvements include:

1. User Interface improvements. Software aesthetics change over time. For example, skeuomorphism was popular when the iPhone launched, but Apple moved away from it in iOS 7. Operating System updates, like Windows and iOS, often enable new aesthetics, so keep an eye out for these. Out-of-date visuals can quickly turn off new users.
2. User Experience improvements. While User Interface changes typically deal with aesthetics, user experience improvements deal with the function. For example, say your product uses a series of screens to collect information from a user. User Interface improvements may improve the fonts, whitespace and colors used in the forms. User Experience improvements would look at improving the order of the fields, the wording.
3. Performance improvements. Nobody likes slow software. Monitoring and improving performance can help keep your users engaged and happy. Some performance issues can be categorized as bugs, but many are not. Work with your partners to set performance standards.
4. New features. Finally, there are new features that improve the software product. Typically there are small improvements that can be covered under a maintenance program. These should be budgeted separately from large improvements, which can often be categorized as capital improvements.

Software Maintenance Budgets & Timelines

To wrap up, let’s look at software maintenance budgets and timelines. 

First, it’s important to understand that most software maintenance can not be capitalized, but instead must be treated as an expense. There are many articles on the internet to help guide you. “Capitalization of software development costs” from accountingtools.com is a good starting point. But this is a nuanced topic, and you’ll want to talk to your accountants to get firm advice. The categorization of software improvements in particular is something you’ll want to review.

Second, you’ll want to build a multi-year budget. Some years will require more maintenance than others. Building a long-range budget will help paint the “big picture” with your stakeholders, and help them visualize where and how money needs to be allocated. 

Finally, work with your partners to set reasonable timelines. Like car and home maintenance, most service providers book up weeks or months in advance. Make sure you’re on their schedule. An annual review with your key partners is a good place to start. 

No one likes maintenance. But like a car or house, a bit of prevention goes a long way. With some planning, you’ll be able to enjoy your shiny new software for years to come.

Written by Brian Tol, CTO of SpinDance.  He can be contacted at brian.tol@spindance.com.