ICA Engineering’s most recent webinar, “Business Continuity: Lessons Learned in 2021,” included topic discussions with a panel of experts in the field:
- Joseph Stevenson, Controls and Lead Engineer, President, CEO, and Founder of ICA Engineering
- Philip Rosen, COO and Director of Business Development at ICA Engineering
- Mathieu Sanders, Business Community Leader, CEO Founder at Decision Point Solutions (consultant agency on business continuity and disaster recovery)
- Troy Manning, Chemical Engineer and Process Automation Specialist at Turtle & Hughes (Rockwell automation distributor in NY and NJ)
With a discussion focused on identifying the causes and impact of disruption to operations, panelists highlighted several key business continuity lessons learned in 2021.
1. Know Your Cost of Downtime and Take Measures to Protect Infrastructure
Business owners face a variety of situations that pose a threat to business continuity, from aging systems to cybercrime. Cybersecurity is a frequent and growing challenge as perpetrators continuously find new ways to attack regardless of the measures a business takes to protect its infrastructure.
In addition to cyberattacks, many other scenarios can lead to a disruption in operational and manufacturing processes. These incidents include acts of violence, terrorism, disgruntled employees, weather events, pandemics, equipment failures, etc., that can impact businesses for days, months, and even longer.
Companies risk significant consequences in the absence of a well-developed plan designed to deal with these issues. Engineers and production managers must consider the true cost of downtime, and weigh that against the investment required to take preventative action to thwart or mitigate costly infrastructure damage.
2. Be Aware of Critical Equipment and Have an Advanced Plan in Place
Ransomware attacks, national disasters, and many other catastrophes can cause equipment to malfunction or completely break down. When the piece of equipment is critical to the entire process, companies risk a detrimental loss in production from a few hours to several months or longer.
In the case of ransomware, cybercriminals often take a multistep approach that can span years. The 2015 Sandworm attack on a Ukrainian power facility caused a three-hour power outage for the Kyiv region. This ransomware attacked controllers, uninterruptible power supply (UPS), serial ethernet communication devices, internal communications servers, and the local customer service call center. Fortunately, the hackers were unsuccessful at getting to the Siemens protective relays, which could have caused many months of power outages.
While most people focus security efforts on the type of malware used, it’s also important to consider the steps taken by criminal actors well before malware is deployed. The example of the Sandworm ransomware attack illustrates why it’s important for organizational leaders to implement preventative measures throughout the production or manufacturing process. In particular, security methods should be focused on areas that could be at risk during crucial steps of an attack. These areas include technology and pieces of equipment that are critical to your infrastructure, such as controllers, communication servers, and third-party devices.
3. Conduct a Business Impact Analysis and Exercise the New Plan
Business continuity issues primarily impact a specific company or local businesses that rely on that company’s products and services. However, third-party risks exist, so the impact can be more far-reaching than expected.
In the past, companies were adequately protected by preventative measures like fire suppression plans, earthquake plans, flood plans, etc. Today’s technology-driven manufacturing landscape requires adding the more sophisticated approach of all-hazards planning, which involves considering all the pieces of a business that interconnect and understanding what would happen if one of those pieces became unavailable. Depending on the potential risk, the plan can be for a single department or multiple departments.
When putting together all-hazards planning, set aside time to practice the plan. For example, you could provide an exercise to answer, “What would happen if you told your staff not to come in tomorrow, but to write down everything they’d need at home?” This type of approach helps leaders anticipate and avoid business continuity issues that could result from an unexpected event.
Even workable plans can have oversights that lead to a lapse in preparedness, however. That’s why it’s important to couple all-hazards planning with a comprehensive Business Impact Analysis so that you and your managers will be able to pinpoint the areas where potential risks have the greatest possibility of costly downtime and prioritize their plans accordingly.
4. Life Cycle Management Is a Must, and Here’s Why
Asset management and security solutions play a vital role when it comes to business continuity. Companies can have hundreds to thousands of corporate and industrial automation assets onsite locally or in remote locations. These assets may include physical, programmed, and configured components such as controllers, human-machine interfaces (HMIs), drives, switches, etc.
Businesses face several challenges and risks with their assets, including lost configuration and device failures, production environment change, and unauthorized and undocumented access. An asset management solution centralizes all company’s industrial automation assets. This approach provides greater control over such things as an archive and disaster discovery, operator track and trace, network security, lifecycle management, and extensibility of third-party devices.
As part of asset management, a system lifecycle evaluation is imperative. In this way, a company’s manufacturing process is cataloged so that every asset is known and inventoried, including make, model, firmware version, etc. Also, the system is backed up and documented with representations diagramming how it works along with notes on which updates have been made. This comprehensive cataloging will not only benefit you in the event of a failure or attack but help you gain a better understanding of possible points of failure so that you can take preventative action. For example, a ransomware attack on a manufacturing plant rendered its PCs and servers inoperable for months, although the automation systems weren’t affected. Had they performed an image backup of their system, the downtime could have been minimized to days. Since then, the business has implemented asset management solutions that consider its system lifecycle.
Lifecycle management is particularly important for companies that use legacy systems, as there are added considerations surrounding if and how to obtain replacement parts and how to integrate older systems with newer technologies. Whether your organization relies on legacy systems or not, it can be daunting and risky for a business to manage assets without outside help. An asset management solution can save time and money by improving efficiency, increasing security, and reducing the risk of downtime. It can even support third-party devices, so you’re assured that all bases are covered.
Final Thoughts
As threats that have the potential to cripple operations and shut down businesses continue to grow in variety and impact, it’s crucial to leverage strategies that support business continuity. From preventing and mitigating cyberattacks to preparing against the possibility of a natural disaster, the key is to take stock of your assets and create a plan to protect and preserve them. While you can do much of this planning yourself, it pays to work with experienced professionals. We offer control system life cycle and industrial automation solutions for many fields, including food and beverage, waste/wastewater, pharmaceutical, biotechnology, packaging, materials handling, and power. If you need business continuity planning assistance, consult with the experts at ICA Engineering.