On Feb. 8 the news broke that a hacker compromised a water treatment facility system in Florida via remote access software TeamsViewer and used their access to increase the Sodium Hydroxide in the water to dangerous, poisonous levels.
Thankfully the situation was avoided when an employee noticed actions being executed on the computer and followed what changes were being made and quickly reversed them.
This was a real-world incident where digital hacking could've had a direct, life-threating impact.
We heard from cybersecurity experts that are adept at ICS/critical infrastructure security on what this attack means for the industry and what lessons can be learned from it.
Lynsey Wolf, Senior Counter-Insider Threat Analyst, Security and Business Intelligence at DTEX Systems:
“Critical infrastructure is a continuously running, vital asset. Thankfully, in the attack against the water treatment facility in Oldsmar, Florida, the plant operator witnessed the malicious actor remotely access the company’s system in real-time, so was able to immediately remediate their actions. The question providers of critical infrastructure need to be asking themselves is, if this were to happen within their organization during the hours that the plant operator was not monitoring their computer, would they have caught this attack before any real, life-threatening damage was done?
The moment an outsider breaches an organization they become an “insider” threat. It’s crucial that organizations monitor employee behavior to gather workforce cyber intelligence, so that when anomalous behavior occurs that strays from the norm of what an individual would typically do day-to-day, IT teams can be alerted and immediately respond. This is has grown to be increasingly difficult to manage and identify with the shift to remote work – as this case with the water treatment facility in FL has shown – so it’s crucial now more than ever for organizations to respond with solutions that can protect their organization, their employees and their customers.”
Gil Kirkpatrick, Chief Architect at Semperis:
"There continues to be a mindset in the public and even in the infosec community that the effects of cybercrime are primarily digital: individuals lose access to their files, companies aren't able to operate efficiently, or in the worst case (as in the Ukraine) neighborhoods lose power for a while. Even though the Maersk NotPetya attack had visible physical universe effects (shipping terminals were unable to open gates to allow trucks in or out), these effects were economically damaging, but certainly not life-threatening. This attack on the Oldsmar City water treatment system has to be a wake-up call for the industry and for the public. The risk is real. Cyber criminals can kill people, they can kill them remotely, they can kill them indiscriminately and at scale, and they can do so with a high likelihood of impunity. We as an industry have to wake up to this risk and design, implement, and operate our systems with a cyber-first mindset before the unthinkable happens. Lives depend on it."
Gary Kinghorn, Marketing Director at Tempered Networks:
"Yesterday's hack of the Oldsmar, Florida water treatment plant again highlights the importance of maintaining critical infrastructure with a virtual air-gap (being off the network) from remote access. These systems should not be reachable by unauthorized attackers because of the sophistication of modern penetration tools and the complexity of these systems to make them completely free of vulnerabilities. Traditional firewalls and other remote access or VPN solutions are proving inadequate against these threats. We need to block any unauthorized access from ever reaching these critical and life maintaining systems while still allowing authorized, fully identified users remote access through secure tunnels using military-grade encryption. We have many such water districts using our solution for just these types of scenarios."
Eddie Habibi, Founder at PAS, now part of Hexagon:
"The news that a hacker infiltrated a water treatment facility in Florida and changed a configuration setting to increase the volume of a dangerous chemical (lye) has rightly been greeted with concern by the media and cybersecurity community. The cyber threat to critical infrastructure has been increasing steadily as hackers, whether nation-state actors, criminal enterprises, or lone individuals better understand how to exploit operational technology (OT) in addition to IT systems. While much of the coverage of the cyber risk to critical infrastructure to date has focused on the age of many industrial control systems and the fact that they were not designed and deployed with security in mind, in this case, the attack vector appears to have been the increased level of remote access enabled by the Florida county.
In the rush to support remote operations during the global pandemic, there are very likely many organizations who have increased remote access to industrial engineering workstations and operator consoles. Fortunately, in this case, there was a vigilant operator who noticed the 111x increase in the chemical (from 100ppm to 11,100ppm) and was able to take quick corrective action to return the configuration setting to its prior level. While industrial espionage remains a significant threat (not all cyber attacks are focused on disruption), the worst fears of many in the OT cybersecurity community were realized in this episode; namely, changing a configuration setting to harm the community served by the facility. It is a poignant reminder that the best foundation for effective OT cybersecurity is a detailed and broad asset inventory that includes relationships and dependencies among OT systems and a baseline of configuration settings. With this in place, risk assessment is far more informed, enabling organizations to more effectively assign and limit remote access at both the system and account levels. Indeed, the combination of an up-to-date asset inventory and risk-based remote access management policies is more critical now than ever before, as it enables both reduced risk as well as faster recovery in the event of an unauthorized change."
Hitesh Sheth, President and CEO at Vectra, a San Jose, Calif.-based provider of technology which applies AI to detect and hunt for cyber attackers:
“We rarely see retaliation in these cases for good reason. It’s extremely challenging to confirm the culprits. An attack can seem to bear the signature of some known adversary, but be packed with digital red herrings to obscure its origins. Placing blame usually means placing a bet. Our time is generally more wisely spent upgrading cyber defenses.”
Dirk Schrader, Global Vice President at New Net Technologies (NNT), a Naples, Florida-based provider of cybersecurity and compliance software:
“Remote sessions tools, such as TeamViewer or Remote Desktop Protocol (RDP), should never be accessible from the outside. In this case, it seems that this was the case, likely combined with weak or easy to guess passwords.
If these tools are in place an organization should have all precautionary measures in place to verify the settings, keep them in accordance with NIST or CIS controls, monitor the access and control any change happening to the device with this tools installed. Unfortunately, this is not always the case and attackers seem to have an easy play to get access to critical systems. It is easy to find about 250 systems using these tools connected the public internet, and within two minutes, to have access to an unprotected system belonging to a water utility provider in Florida. Previous research, including the Solarium report, have documented that Critical Infrastructures are vulnerable, and sometimes it is not hard at all to get access to one provider. That status is the same across all critical sectors including healthcare.
Whether there are any access logs available in this incident is an open issue. However, the original statement seems to indicate that there are none and identification and attribution will be difficult.”
Alec Alvarado, Threat Intelligence Team Lead at Digital Shadows, a San Francisco-based provider of digital risk protection solutions:
“The attack on the water treatment facility in Oldsmar is a chilling example of how cyber attacks can have more than just financial impacts. Systems belonging to our critical infrastructure are some of the most difficult to maintain. Every day, countless vulnerabilities are found, some of which are so critical that they need to be patched immediately. Enforcing a strong patch management strategy is challenging but is even more challenging in facilities that can't afford lengthy downtimes. Although we aren't sure how the threat actors got access to the Oldsmar water facility systems, it isn't farfetched to believe this attack could happen to other facilities.
Regarding attribution, little has been released, but there are some things you can conclude based on reporting. The activity doesn't seem financially motivated, which would suggest either a nation-state actor or hacktivist conducted the attack. Hacktivism usually involves a quick claim for an attack; this is done to draw attention to their movement. Hence why defacement or DDoS is so popular in hacktivist attacks. The covert nature of this attack points more toward a possible nation-state actor. The biggest thing that should come out of this attack is to remind us how much of our every day, even our drinking water, can be changed with a few clicks.”
Setu Kulkarni, Vice President, Strategy at WhiteHat Security, a San Jose, Calif.-based provider of application security:
“Connected industrial control systems will continue to profoundly impact society in ways we haven’t thought about. However, in this case, while the negative impact could have been profound – the act was not something we wouldn’t have thought about. Connected industrial control systems now have given adversaries access to our backyards and to our water supplies. What is worse is that with such remote access, the relative anonymity and the potential safe-harbor – the adversaries do not have any deterrent to launch such malicious and potentially profound attacks.“
Yaniv Bar-Dayan, CEO and co-founder at Vulcan Cyber, a vulnerability remediation orchestration provider:
“We’ve always taken steps to secure the services and resources that are important to us. In banking we used to use vaults, armored money trucks and guards to protect our money. Now we use technology like encryption and user access controls to protect our money.
Likewise, we now rely on technology for an always-on, clean water supply. Everything is digital these days, creating an extremely complex, massive attack vector that needs to be secured and managed correctly. In this case, it looks like the water treatment plant computer systems were not configured correctly allowing system-level access to unauthorized users. Poor cyber hygiene leads to insecure systems. It isn’t easy but IT security teams must diligently monitor and secure our technology to protect our life-sustaining services in an increasingly digital world.”
Kevin Dunne, President at Greenlight, a Flemington, New Jersey-based provider of integrated risk management solutions:
"Public infrastructure is one of the most vulnerable attack vectors to remote access threats. Because these systems are mission critical, it is difficult to coordinate system down time to facilitate updates to security vulnerabilities or to modernize architecture. Many of these systems were built decades ago and are still secured through traditional methods like network design and role-based access control. Many of these systems aren't adapted to today's remote access environment, so they are prime targets for hackers, who can easily gain access to the network with compromised credentials and exploit these legacy systems."
Heather Paunet, Senior Vice President at Untangle, a San Jose, Calif.-based provider of comprehensive network security for SMBs:
“As we begin 2021, governments, as well as every other type of business, continue to have their employees work remotely.
As IT departments reacted quickly in 2020 to enable all their employees to work from home, ensuring a secure work-from-home environment took a bit longer to get right. As employees transitioned to remote work, they put their work devices onto their home networks, which would not have all the safeguards in place as their in-office network had.
This can create opportunities for bad actors to hack into networks and potentially cause dangerous situations. In the case with Oldsmar's water treatment plant, it was found that someone had access to their computer system remotely. With remote access being much more common due to fewer on-premises workers, this may not have been noticed as quickly as it should have been.
When thinking about remote access, business of all sizes, and all industries should consider:
Use of VPN technologies: provide a secure tunnel, and credentials that are given to employees to access internal resources and keep critical systems protected.
Proper onboarding and offboarding: as employees join and leave a company, it is important to ensure that access is only given if needed, and revoked immediately as employees leave.
Segregation of network access: ensure that employees are only given access to the systems that they need. Putting different systems on different networks that are only accessible by the groups of employees that need them is important to ensure that if a breach does happen, less systems can be compromised.
Dedicated work devices: during times such as the rapid shift to working from home in 2020, where many employees ended up accessing systems remotely, providing a dedicated device to employees rather than allowing employees to access the corporate network from their own devices will give IT departments the most control of their infrastructure.
Continual employee training: teaching employees how to recognize phishing emails, is just as important as putting in place protective systems. As security adversaries find new ways to infiltrate networks, keeping employees trained and up-to-date will only strengthen your network security.
While cybersecurity vendors continually come up with new solutions to guard against data breaches, there are cybersecurity adversaries that are working just as hard to break down those solutions and find new ways to get ahead of those vendors.
That’s why it’s important to stay a step ahead of hackers by keeping up on the latest technologies and providing multiple security layers of protection. If a bad actor does get through the strongest barriers, having multiple security layers provides protection to help isolate the threat and minimize the impact.”
Chris Risley, CEO at Bastille:
“The water treatment system hack is troublesome because this underscores how vulnerable cities are to critical infrastructure intrusion.
There’s widespread recognition of the need to eliminate potential intrusions and attacks, but limited adoption and enforcement of security policies to combat bad actors. Bastille’s groundbreaking research and discoveries of MouseJack, KeySniffer, and KeyJack validates the company’s thesis that the IoT is already being rolled out to individuals and enterprises with wireless protocols that have not been through sufficient security vetting. As a result, Bastille expects millions of devices to be vulnerable to currently undiscovered attacks. “The water supply poisoning attempt was scary, but not surprising. They may eventually catch the culprit because the attack came in over the Internet and Internet addresses can sometimes be traced back to the perpetrator. However, the key to the attack was that the attacker took control of the target’s mouse and adjusted the amount of lye being added to the water. Five years ago, Bastille discovered MouseJack and demonstrated that you could take over a target’s mouse without ever using a wired connection. Merely reaching out to a wireless mouse dongle via radio waves can give an attacker all the control they need to carry off this kind of dangerous attack on critical infrastructure, added Risley.”
###