Water System Hack – The System Is Broken
Hackers ‘hit’ US water treatment systems
Homeland Security investigates possible terrorism in Springfield
Water system may be cyber attack victim
Has stuxnet come to our critical infrastructure shores? Is it duqu? Could it be something even worse?
“DHS and the FBI are gathering facts surrounding the report of a water pump failure in Springfield Illinois. At this time there is no credible corroborated data that indicates a risk to critical infrastructure entities or a threat to public safety,” DHS spokesman Peter Boogaard explains.
“I dislike, immensely, how the DHS tend to downplay how absolutely FUCKED the state of national infrastructure is” responds someone named “prOf” in a pastebin post that includes, according to pr0f, images of another water system that was hacked.
“I’m not going to expose the details of the box,” prOf promises. “No damage was done to any of the machinery; I don’t really like mindless vandalism. It’s stupid and silly. On the other hand, so is connecting interfaces to your SCADA machinery to the Internet. I wouldn’t even call this a hack, either, just to say. This required almost no skill and could be reproduced by a two year old with a basic knowledge of Simatic.”
Nick Catrantzos, who has written for Homeland Security Watch in the past, is an adjunct professor of Homeland Security and Emergency Management. More relevant to today’s post, Nick is the former security director for a regional water utility. Here are his thoughts on the most recent cyber event.
Spotting the Incidental Cyber Saboteur
You need not be evil to be wrong, and the true Achilles’ Heel of recent news about cyber attacks to water infrastructure in the Chicago area (details at http://www.cnn.com/2011/11/18/us/cyber-attack-investigation/index.html?iref=allsearch) is not foreign hackers of SCADA, the supervisory control and data acquisition system that makes it possible to turn a valve by remote control. Hackers have been a known external threat since the personal computer became widespread. Thus, makers of computer- and network-dependent tools like SCADA systems have to offer some protections against hackers just to make their systems marketable.
Why is no one therefore consulting other than self-avowed cyber security experts who are now issuing dire warnings about offshore SCADA hackers who may or may not be Russians? (The may-not possibility arises when these experts point out that clever hackers have the ability to misrepresent the origin of their attacks.). The same hand-wringing experts – or their fellow travelers – belong to the camp that opens the door to this vulnerability in the first place. They are not evil, just wrong.
Remote Access as Double-Edged Sword
Consider: Even the technologically challenged security professional sees the vulnerability to enabling remote access to critical systems, like water infrastructure. How do purveyors of such systems see remote access when marketing to fellow cyber aficionados? It is a selling feature, of course. Why, with remote access, the technician fielding a panic troubleshooting call at midnight can diagnose and solve the problem in pajamas instead of in the field. And the field, when it comes to water infrastructure, often turns out to be at distant sites over bad roads, poor lighting, and unattractive traveling conditions. Solving the problem from home is a win-win for all concerned, since it saves down time, isn’t it? Not if this debate includes security professionals charged with looking at the bigger picture of enterprise-wide vulnerabilities.
What makes it possible for these infrastructure attacks to abuse SCADA? Remote web access adopted in the name of expediency. What is the Achilles’ Heel? Naïve or myopic cyber professionals whose over attention to expediency permits convenient remote access for their technical support colleagues with insufficient attention to the exposure that this condition creates.
Discovering What Some Won’t Admit
How to zero in on the problem? The way not to do it is to rely exclusively on pronouncements of SCADA vendors and their like-minded counterparts in the organization who bought into web-based remote access in the first place. There is a good chance at least some of these people overlooked sharing details of remote access vulnerabilities in discussing the system before upper management and traditional security practitioners.
No, the short path to excellence in uncovering self-introduced remote access exposures is to check logs of trouble calls against field records of physical access to work sites. The more serious cyber professionals know to avoid web-based SCADA access from any home and, instead limit access to SCADA terminals that reside behind the secured perimeter of the institution’s work facilities. Maybe a SCADA technician fielding a trouble call won’t have to drive three hours to diagnose the problem at a remote field site, but he may still have to drive 20 minutes to get to a locked and alarmed office that houses a protected SCADA terminal. At least this is the ideal and advertised state of affairs. But even 20 minutes may, in time, seem too much of an imposition, so the SCADA tech quietly arranges to beta test remote access from — you guessed it — the convenience of his or her own residence. Unofficially, without a lot of fanfare. So much so, that even the boss may not realize this is happening, hence the futility of relying on the cyber function to verify its own status regarding this vulnerability. There is another way to check.
Uncovering the Rest of the Story
If expediency has come to trump security, an examination of audit trails will soon show that technician troubleshooting calls at midnight aren’t matching up to midnight access to facilities housing SCADA terminals. Maybe operators in the field are too immersed in the problem to ask or even care how a SCADA tech is responding to a trouble call. They just want help. Maybe the tech is shrewd enough to avoid volunteering details, reasoning that speed of problem resolution is more important than revealing that this is being done from home via means subject to compromise and exposure to hackers.
However, audit trails won’t lie. Whether it is via manual logs, automated access records, video surveillance archives, or a guard’s register used for having all employees sign in after normal business hours, the discrepancy will surface under scrutiny. The on-call tech who was supposed to go to an employer site to troubleshoot the problem on a protected SCADA terminal will have shown no record of having entered any employer business site at midnight. So how did he or she handle the problem? Remotely. From home. In pajamas. Expediently. And, in the process, exposing the system to exploitable vulnerability.
Caution on Experts Offering Homilies about Cyber Attack
The so-called expert who was quick to criticize government officials on this latest cyber attack claimed he was doing so out of concern that the Department of Homeland Security was deficient in sharing information with other water agencies that could be targeted. If he were truly as conversant with water security as he claimed, he would know that it is not DHS but EPA that exercises the role of lead federal agency for protection of the water infrastructure. He would also know that EPA supports Water ISAC, the Information Sharing and Analysis Center for the water sector, and that the Association of Metropolitan Water Agencies manages that function, which takes the lead in sharing this kind of threat information within the water community, while DHS and local fusion centers do their share of distributing such information as well.
Showing no sign of recognizing these particulars, how could this self-styled expert really know what information on this SCADA threat is or is not circulating within the affected community of interest? A skeptic might conclude that such considerations take a back seat, however, when dire warnings can generate free publicity.
IT vs. Ops
Some over zealous IT departments in utilities that use SCADA see SCADA as a means of supplying bandwidth on which to commingle business applications as well, thereby increasing likely needs for remote access by more employees and raising susceptibility to compromise at the same time.
If employees in Operations at water utilities don’t over concern themselves with security deficiencies in SCADA, it tends to be because they have their hands full avoiding one or two catastrophes a year when SCADA techs unthinkingly shut down the system for maintenance or cause some other disruption without telling Ops in advance. The techs forget that flow changes can result in catastrophic treatment or distribution problems that affect water quality. This often occurs after business hours or on weekends, when the techs operate on the assumption that it is the best time to tinker without users noticing or balking — true enough for the average business network, but not for 24/7 attention to water treatment and distribution.
One sign that too many debacles have been surfacing serially is when Ops wrests the SCADA function away from IT. This does wonders for reducing those kinds of snafu.