Increasing dependence upon IT systems to deliver healthcare services to 140,000 women, men, children and young people means availability and uptime are mission-critical at Birmingham Women’s and Children’s NHS Foundation Trust. The installation of EcoStruxure IT Expert has helped the Trust take a major step forward in the assured delivery of services.
Birmingham Women’s and Children’s NHS Foundation Trust was founded in 2017 as a merger between two existing hospitals and is a specialist provider of healthcare services to women, children and families living in the English Midlands.
The Trust’s operations are supported by an extensive IT infrastructure whose functions were combined and continue to be integrated on the campuses of both hospitals. Following the merger, most IT equipment is centralised in what IT Infrastructure Manager Jas Purewal describes as a “shiny new data centre” at the larger of the two sites, formerly the Birmingham Children’s Hospital.
Another older data centre is located at the site of the former Birmingham Women’s Hospital and there are also various hub rooms – distributed IT facilities – throughout both campuses which variously house servers running applications needed to support healthcare, switchgear and networking equipment. Some hub rooms also host physical servers running SQL-based applications, but there is an ongoing programme to virtualise the majority of the Trust’s applications on NetApp servers housed in the main data centres.
The necessity for always-on power for IT and patient health
Continuous uptime is the top priority for the Trust’s IT department. Any impairment to the IT systems running patient-management or clinical applications inevitably impacts the delivery of treatment to patients. To ensure continuous operation in the event of a computer systems failure, the Trust maintains a detailed disaster-recovery (DR) plan which now sees the data centres in each of the constituent hospitals acting as failover resources to one another; in the event of a systems outage in one location, the servers in the other take up the load immediately.
Of more fundamental concern is the mains electricity supply itself. To guard against any service disruption, the Trust operates nearly 100 uninterruptible power supply (UPS) systems that provide battery back-up power to essential systems in the event of a utility power interruption or blackout. The UPS systems are designed to provide temporary ride-through power either until mains power is restored, or in extremis until secondary back-up power generation can be brought online. Many of the UPSs are installed outside the main data centres, in remote edge locations, on both campuses.
A runtime challenge for the Hospital Trust
Maintenance of UPS systems is a vitally important routine. Because UPS batteries have a finite operating lifetime, they must be checked and replaced at regular intervals to ensure that they are fit for purpose should they ever be required to respond to the loss of mains power. Traditionally, an inspection of the UPS battery systems has been a manual process conducted annually. The disadvantage of this method was that any degradation of the batteries that occurred between scheduled maintenance operations was not visible. This presented the risk that in the event of an outage or blackout, the UPS systems would be unable to power the IT infrastructure until the mains power was restored.
Just such a problem occurred during a routine generator test: A UPS that was specified to provide more than 20 minutes runtime to the load failed after only a few seconds, causing the main data centre to crash. According to Jas Purewal, ICT Infrastructure Manager at Birmingham Women’s and Children’s NHS Foundation Trust:
“If our systems shut down gracefully, they will come back up gracefully. But in this case, because the power was cut abruptly, it took us several hours to recover all our systems. The most important ones were returned to full operational status within an hour, however, others took three to four hours to recover. We also had to replace some disks over the next few days, so it was a very busy time with all hands on deck!”
Whilst the loss of IT services presented no risk to the health of people in the hospital’s care, the incident caused a headache for the IT department, with the realisation that an unplanned mains outage might adversely affect the hospitals’ ability to deliver critical IT services. As a result, the issue was discussed at the board level and a decision was taken to improve the resilience of the Trust’s IT systems to withstand any similar instance in the future.
Automating inspections; using data to make decisions about availability and uptime
The Trust decided that it needed a better system for monitoring key infrastructure such as UPS systems, and it engaged the support of Advanced Power Technology (APT), an Elite Partner to Schneider Electric and specialist provider of energy-efficient critical power and cooling systems, particularly for use in IT installations. APT had previous experience at the site, having installed Schneider UPS systems for the Trust.
APT took a hard look at the actual requirement and took the novel approach of recommending the installation of EcoStruxure IT Expert, Schneider Electric’s next-generation data centre infrastructure management (DCIM) software to remotely monitor and manage the UPS and battery systems. This cloud-based application allows connected hardware assets in data centres as well as distributed IT and edge locations to be monitored continuously from a central console. Status updates and alarms can be routed to any remote access device, such as a notebook, tablet computer, or smart phone.
“The main thing we like about EcoStruxure is its dashboard facility,” says Purewal. “You can very quickly see if you have a critical problem that needs immediate attention. It also has very useful features like power forecasting which allow us to estimate how long a battery is likely to last, even if it’s not in critical condition so that we can plan for timely replacement.”
Another useful function is the ability to assess the effects of increasing the load on a UPS before additional equipment is installed. “This is where EcoStruxure is very good,” says Purewal. “We can assess the impact of additional load and determine whether we need to upgrade the UPS at the same time. Using data from the application, we can decide whether to scale up or scale down the UPSs to right-size them for the IT and optimise the system for efficient operations.“
Evaluating EcoStruxure IT Expert
EcoStruxure IT’s capability extends beyond accurate monitoring of UPS systems and batteries as it can also be used to monitor equipment throughout the physical and IT layers, as well as environmental conditions such as temperature and humidity. The easy-to-use dashboard feature means that information on such matters can be easily absorbed by management who are not necessarily IT specialists. This is particularly useful in the case of the Trust where responsibility for the environmental management of the smaller IT hub rooms is shared between Purewal’s IT department and general estate management.
“The data centres are my team’s responsibility, but a lot of the networking gear is in small rooms – remote IT installations where the environment can be dusty and the ventilation poor. Overheating in these crowded facilities could easily become a problem,” he says. “With the dashboard and graphical functions available in EcoStruxure IT Expert, I can make the case to management in other departments, or higher up the chain, that some small investment in air conditioning, or perhaps moving the equipment to a more suitable room, could reduce operating costs in the long run, as well as improving availability and reducing the total cost of ownership.”
The value of Schneider Electric and partners
APT advised the Trust on the deployment of EcoStruxure IT Expert and provide ongoing support and consultancy regarding its flexibility and potential to deliver further benefits in the future. “We’ve got this great tool now from a leading vendor and that gives us great peace of mind as try to get the most benefit out of it,” says Purewal. “APT’s people have extensive knowledge that we can tap into, and they have been very supportive when we have any questions.”
John Thompson, managing director of APT said: “The purpose of annual inspections was to provide assurance to the customer that the UPS systems were sufficient to meet the needs of emergency operations. Unfortunately, when called upon in the moment of need, Birmingham Women and Children’s Hospital found this approach to be lacking. The good news is that despite the outage, there was never any threat caused to human health at the hospital. The installation of EcoStruxure IT Expert means the team now has continuous, 24x7x365 assurance of the UPS’s operational status as well as the runtime available to support the IT and critical loads.”
Moving forward, Birmingham Women’s and Children’s NHS Foundation Trust will continue to expand the amount and variety of cloud-based services from its data centre. The use of EcoStruxure IT Expert is being extended to monitor environmental conditions in the IT rooms including temperature and humidity. This will enable the IT department, in co-operation with estates management, to provide detailed information to senior management to drive investments that will not only guarantee maximum uptime but also help the Trust to manage its hardware assets in a more sustainable and efficient way.