The Power is On, Why Doesn’t it Work?

Yes, this is just another one of my totally biased perspectives on company efficiency, especially where related to its IT policies, procedures and systems. So again feel free to switch to another post or even better, check out one of my books 🙂

So back to the title of this post, which was a question from a very untechnical factory manager a few years back. He was a smart manager though and I guess to him the question was obvious, apart from emails of course everything should work!

So let us start with some simple questions which I am sure if you are a manager or company owner you will be able to answer immediately.

• Where is your companies data processed, and stored?
• If your internet connection broke, could your business still run?
• What dependencies do your IT systems have on other systems on the Internet?
• Is the person you phone when IT problems happen the person who will actually resolve the problem, or just a middle-man who will (hopefully) deal with it?
• Do you have manual or replacement systems in place for extended issues (ransomware, serious network outages, etc)
• Have you tested any of these?

So how do you even begin asking the right questions especially when it comes to IT? Most just give in and hire an IT Manager, a CTO, or a contractor like me to deal with it, after all, that’s what directors do don’t they, delegate! Well, if that is your solution without understanding the issues, you have likely just taken the first step starting your company on the road to complexity, unreliability, fragmentation and dependency.

So, step back from the technical issues for one minute and look at your core business and ask are the solutions being proposed for the benefit of the business or for the benefit of someone else, someone with perhaps another agenda. This is not just IT by the way, just go try and buy a used car or a new laptop, or even a smart TV when you don’t know what you want, now multiply the consequences of that decision by a factor of a thousand!

To the other question; How long, from unplugging your companies Internet cable could your business run for? How long could you receive stock, could you raise production orders, process orders, schedule staff, find stock in your inventory, dispatch stock to clients and raise and process payroll and invoices? Also, do any of your machines phone home or depend upon on-line services to operate?

With all this in mind, does your company have manual systems in place for when (not if) a key system fails? Very importantly, do you have a system in place to recover and go back to this key system when it is back online and working?

Obviously, you don’t want these key systems to fail. Your company has invested a lot of money to ensure they are robust, but just how robust are they really? Here’s a scenario to think on while everything is working fine and dandy, what is your companies most important IT system and how many single points of failure does it have?

Before 2010, this would have been an easy question to answer, you probably had a database server in the office or factory, running some sort of database which your staff and production systems would access and use directly. Redundancy would have been a UPS to keep the power on during blackouts, RAID disks so a single hard disk could fail, and backups would have been to tape or disk in a different fire zone. If you were good, you may even have a redundant or mirrored server ready to take over. This server probably also processed all your files, prints and emails and to run this you had maybe a couple of IT people or contractors, who kept the hardware and software working, managed your backups and who you could call on for problems. You also likely had specialist company’s or persons like me who wrote your processing systems and owned the responsibility for maintaining them.

So as layers went it was all quite simple and understandable, all the software and data were local, and problems came down to people you knew, and the single points of failure were easy to identify. Today however this may not be the case. Your company may be dependent upon software you may not know is even there in the forms of frameworks, API’s and libraries (which may in turn be dependant upon other software). There has also been a tendency in recent times for software companies to have their software phone home every now and then for such things as over the air software updates, license and checking, bug and usage checking etc. If your software relies on cloud systems, it is also likely running on API’s all of which need to be kept in-sync and compatible for the whole not to fail, these additional layers may also not be under your control, get depreciated, add new functionality you may not be aware of and most importantly support will be dependent on a supplier you may not have an agreement with or even a contact number for.

Finally, there is the question of how much will the total cost of ownership be for these systems over their lifetimes? You may expect that the brand new system you just commissioned and paid for will work forever, however, 6-8 years is the normal and may be as little as five years as all these various dependencies go end of life or get upgraded. Good Open-Source choices can extend this quite considerably. Having bog-standard Linux or BSD back-end server, an open-source database and a production system with no dependencies will likely age seamlessly with minor security updates. A badly written system on the other hand may have multiple dependencies and may only have a life of a few years without constant updates to both the front and back end. You need to ask the questions, and you need to be able to trust that the answers you receive are correct and not driven by a technical or other agenda.

All of this assumes your business does not change, if it does change then you will be constantly updating the software to support newer practices, while potentially retaining support for the old practices or at least supporting historical views of the old data. With new features and changes, the original software code base gets bigger and has to work around existing code so as not to break it, this is called technical debt. Your brand-new system usually starts with no technical debt and increases over the life of the software, technical debt is important as it affects the support cost and increase the length of time of subsequent development changes. Well written software can mitigate technical debt, but it is always there from the first update despite what the experts say. As a business buying in to such a system, you want to know it will go at least ten years and then give you a graceful fade out period on your terms to replace it.

So, as you sit at the helm of your well-polished ship, surrounded by a bubble of people who may agree with every word you say, ask yourself how many holes are in the hull, how many rats are infesting it and how well is it being maintained, but most of all ask yourself is it heading for an iceberg.