Failure is not an option

Time was of the essence when a failing critical communications system needing fixing and upgrading.

As Neil Young advises us, “Rust never sleeps”. Such is the plight of many technology systems around the world. As soon as you take your new, shiny, state-of-the-art widget out of the box, its replacement is anywhere from a thought in a bright mind to a prototype on the test bench. It could even be available at the same supplier that you just purchased your new widget from!

Like rust, technology never sleeps. New widgets carry the promise of better capability, speed, reliability and redundancy. This enables the tech-savvy of our world to provide us with capabilities that enable us to be more productive and therefore drive better business outcomes.

However, as time progresses and systems continue to perform, upgrades are not on the agenda. And when ‘If it ain’t broke, don’t fix it’ applies, the ‘rust’ begins. And that rust can be undetectable until the structure gives way and you fall through the floor.

A Unicom client was recently presented with this unfortunate experience. A server installed and configured 10 years ago had started to become unreliable, degrading the client’s Emergency Response Pager Messaging System. This system is used to coordinate maintenance and shift roster activities for a liquefied natural gas plant. The system also is a key component in the emergency evacuation process. So, due to the critical nature of this communications platform, it is easy to see that not having it reliably available was a major problem.

Our client’s urgent initial request was for us to provide for the resurrection of their existing system. The problem was that ongoing software development for the pager messaging system ceased at Windows 7 — the software would not run on a machine configured with anything higher than that operating system. Added to this, the software was tied to the machine on which it was running via a software key — the company that owned this software was a distant memory and therefore unable to provide solutions. In short, the client was in a pickle!

Being an agent for this software back in the day, our engineering staff was able to provide a Windows 7 licence to our client, who used it to configure a new machine. We then advised them on the installation process for a backup of their software configuration, including the software key, which we had made during the initial installation 10 years ago. Hey presto, the client was back in business. But it was now acutely aware of the risk it was faced with. So the client’s next request was for us to develop and install a replacement pager messaging software system.

We had been working with the MobiCall product for some time and we viewed this as the solution. We just needed to develop a pager messaging platform within it that met the client’s criteria of dual redundancy and failover between multiple transmitters, with alarm notification and hardware and software watchdogs and so on… delivered and installed on-site immediately. There’s nothing like a critical system delivery challenge to help you lose sleep!

So original plans were dusted off and the lens of possibility was overlaid. As with many reviews of legacy communications systems that we see, it quickly became very apparent that we could offer a superior solution than the existing single level of service by integrating a number of communications platforms into the proposed solution. This was of great interest to our client. It is also an area that Unicom has developed a great deal of expertise in — the integration of systems into one platform where command, control and interoperability become available.

After a number of urgent conversations and system modelling, we presented our client with our proposal. It comprise a staged approach as follows:

1. Replace the existing system to return to them reliable and supportable operations.
2. Integrate phones, email and SMS messaging into the paging messaging system.
3. Upgrade the hardware to include current technology offerings.
4. Deploy a system-wide support agreement to manage and maintain serviceability of the entire solution well into the future.

Their simple response was, “When can we have stage one?” This was quickly backed up by an order and, in that instant, we were at it! The next few days involved developing a hardware mirror system in the Unicom tech lab. This was used to test the evolution of the software pager messaging solution, which was developed on the MobiCall software messaging platform. In conjunction with this frenzy of activity, project management plans, server and client hardware criteria, factory acceptance test and site acceptance tests documents were drafted and submitted to our client for review and approval.

Once we had a working solution, we were off to the site for the installation. Did I mention that Unicom is located in Melbourne and that our client is in the Middle East? This, along with the requirement to have a remotely supportable solution, dictated the methodology of the solution.

We designed it to include a virtual machine which would run the new MobiCall pager messaging software. This way, Unicom could acquire real-time snapshots of the working system and diagnose any issues on our mirror image hardware. Once resolved, we would upload the new configuration to the client’s site and from there they could run it and confirm. If any issues still remained, they could roll back to the previous known state and therefore suffer zero downtime and have a constant set of known-state backup configurations. This is just what you want if you intend to be serious about system support for a site on the other side of the planet.

The system proved to be a great success, and we are already working on adding features such as alarm emailing, sending pager messages to phones as text and as speech, and enabling pager messages to be sent via SIP using IP phones.

As with many of these deployments, once the client sees the bigger picture available through the full scope of an integrated messaging and workflow solution, they then start to see how they can coordinate their multiple disparate emergency communications systems and control the escalation of response to events through a single platform.

The customer now has a reliable, mission-critical emergency response alert system with dual redundancy and automatic failover, accessible from any customer web browser or telephone and supported 24/7 from Australia.

Image courtesy Bilfinger SE under CC