IT Recovery Amid Catastrophic Flooding
Businesses heavily rely on their IT infrastructure to ensure seamless operation, day in and day out. However, when Disaster strikes, what do you do? A robust Disaster Recovery plan ensures that you know who to contact and what key considerations you should have when it comes to restoring your business operations, infrastructure, and data. Disasters can come in all shapes and sizes, these could be Cyber Attacks, Hardware Failures, or in this case, Natural Disasters.
The Background:
The weekend of October 21st & 22nd 2023 saw severe flooding in many areas of the UK after days of continued heavy rain. Whilst many of our customers were fortunate to have escaped the flooding, one of our customers who operates in the Warehousing and Logistics industry was severely impacted. The ground floor of our clients' premises was subject to a significant amount of flooding which affected the majority of their IT Infrastructure. The server, thankfully located on the second floor, was spared. However, most user computers were rendered inoperable. The gravity of the situation was magnified given the 24/7 operational nature of the Warehousing and Logistics Industry.
The Response:
In the early hours of Saturday (21st) morning we were informed of the situation via our Out of Hours emergency line. We acted quickly to remotely assist and preserve what equipment we could, putting processes in place to help the business continue to operate in a timely manner. Our team quickly diverted the main office phone lines to designated mobile numbers to allow the business to maintain communication channels with their clients/customers. The onsite server was remotely powered down to prevent further damage caused by potential power issues, an on-site electrician was guided to isolate cabling on the ground floor away from Power Over Ethernet (POE) switches, which if not done could have caused significant damage to the business’ network infrastructure. Support continued throughout the day as the building continued to experience power brownouts, with many key personnel remotely lending assistance wherever possible in order to protect/recover as much of the hardware as possible. By early Saturday evening power was isolated from the ground floor and restored to the second floor, to prevent further electrical issues. Our team, through remote support and guidance, worked with onsite personnel to power the server back online, boot up the virtual machine host, and run diagnostics to ensure that no data had been compromised.
As waters began to recede on Sunday (22nd), we worked with the business to ensure an extensive inventory was taken, this allowed us to further ascertain the extent of damage to the IT equipment, provide us with details to immediately engage suppliers for quotations and expedited delivery to allow their business to continue operations as quickly as possible. This inventory of equipment also allowed us to ascertain damage to their phone system and put further diverts in place for key personnel. Whilst the office power was isolated, further power issues affected warehouse operations, remote diagnostics and a further onsite visit by our engineers, identified that their Cisco Wireless LAN controller had reverted to a previous configuration after it was forced to perform an unsafe shutdown. This blocked Wireless Access Points from reconnecting, rending the building without WiFi services and preventing the warehouse from being able to scan or move pallets, effectively rendering the warehouse offline. A swift reconfiguration of the Wireless LAN Controller restored the operations within hours.
As Monday Morning (23rd) rolled around, out-of-hours engineers liaised with staff internally to set up dedicated points of contact and begin sourcing same-day replacement hardware. Two of our seasoned and experienced onsite engineers were dispatched to the client's site to support them as the working week ramped up, configuring and deploying temporary equipment in their makeshift offices for emergency use whilst they awaited delivery of new IT hardware which began to arrive by early afternoon. Due to extensive damage to the physical network, temporary data cabling was laid to essential hardware, alongside temporary Wi-Fi setups, and critical applications were configured to work with the existing server.
Due to extensive damage to existing finance team hardware devices located on the ground floor, a collaborative effort between our Engineers and Sage enabled us to restore and reconfigure the business’ Payroll and Accounts services, ensuring that they could process weekly payroll and invoices without interruption.
This support continued throughout the week (ending Friday, October 27th), Account Managers ensured twice-daily catch-ups to assist with any issues, and our Service Desk team operated on a heightened level of support with quicker SLAs, ensuring a rapid resolution and dedicated escalation processes to any emerging issues, underscoring our commitment ensuring normal business operations. Our projects team has also liaised with the client to plan new permanent cabling and WiFi solutions once their ground floor offices are again fit for use.
Outcome:
Our quick actions, working with the business, meant that they were able to quickly resume critical operations, preserve essential data, keep minimal operational downtime, and ensure dedicated enhanced support. This has continued as we support the business to return to its expected full operational capacity. This dedication to our customers in their times of need is just a part of our business ethos. Our proactive stance ensures stronger client relationships and is instrumental in bolstering future preparedness against unforeseen disasters throughout our entire client base. Allowing us to adapt Disaster Recovery Plans for all customers based on past experiences.