[SCADASEC] IBM is offering 'SCADA security best practices'...
Jake Brodsky
ab3a at comcast.net
Tue Feb 12 21:14:10 CST 2008
Matt, some of this is a matter of getting control systems engineers to
wrap their minds around such technologies, and to decide for themselves
whether they really do represent an improvement.
You need to understand plant mentality. It's a fiercely independent
look at things. They don't trust automation in general unless they can
see, touch, and understand what they're getting in to. Senior operators
don't like having voodoo happen on the plant. It's their lives and
their staff's lives at stake, so they tend to be very protective. You
really have to work hard to earn their trust. Most control engineers
are very cautions not to betray that trust. IT folk will have to earn
their stripes too.
For example, you mentioned Control Logix PLC gear. Redundancy can be
had. However, if you really want something to work well, you'll go for
diversity. More than one processor can slide in to that backplane. The
programs and firmware need not be the same. In fact, you're better off
if they aren't. That way you have a fighting chance that a bug such as
a memory leak in one might not be in another. They can communicate with
each other, and if one fails to send a heartbeat to another, then backup
programs can take over.
Another point is that redundancy doesn't always produce better
availability. I have personal experience that it often makes things
worse. Most of the hoopla over safety PLC gear has very little to do
with the processor and a whole lot to do with the I/O. I don't know if
you've seen the 1oo2 and 2oo3 I/O failure calculations, but it is a much
more crucial part of a PLC design.
In my experience, the PLC has extremely high reliability. What fails is
the power supplies, the networks, and the I/O. Often this is caused by
poor grounding practice. Sometimes it's simply age. We have seen
firmware bugs, but it is rare for them to affect a process. On the
hardware side of things, electrolytic capacitors in power supplies often
fail. Many of the earlier MTBF numbers I read in advertisements were
based upon semiconductor failure rates, and overlooked such mundane
things.
As for managed switches, we have them, and we're installing more. Don't
think for a minute that we're blind to what can be done. However, we
also have concerns about integrating stock SNMP tools from the IT world
in to the SCADA or DCS systems. Many aren't well suited to present the
information in a form the operators will appreciate.
Most SNMP monitoring packages presume a level of expertise that a
typical operator may not have. You have to remember that these displays
are for network monitoring, whereas the operator is really interested in
how well the sludge furnace is working. It's a peripheral issue and our
midnight shift operator may not understand the significance of an alarm
indicating that port 3 of switch 2 in the blower building was just
reconfigured from VLAN 44 to VLAN 23 at 2 AM.
The problem is as I've stated before: While I know there are many very
well trained and experienced IT networking departments out there, we
often see just the dregs in a typical utility. Their opinion of IT is
biased because the IT *they* know is not particularly stellar.
I often point out to people that if they want to know how well a company
is doing, they need look no further than the IT department of that
company. A well run company has a busy, productive IT department and
satisfied staff all around them. A poorly run company isn't likely to
have a good IT department because without knowledgeable executive
management, the various divisions won't know what to ask for and IT is
in no position to tell them what they need.
These days utilities are in terrible shape from neglect. The employees
aren't stupid. They know things aren't ideal. However some old timers
really feel a sense of duty to make their community better.
Nevertheless, they're not getting the support from management or IT that
they need.
Much of the hate and discontent is because we know that many utilities
are ill equipped to do what you rightfully point out. The few that are
capable, are aware of the things you suggest, and are at least testing
these notions right now.
Sadly, the poor opinions of the IT profession will linger for some time.
We can't overcome it overnight. The best you or anyone else here can
do is to listen closely, take small steps, prove yourselves, and then
sell as much as the operators will let you do.
Jake Brodsky
(In Comic Book Life I'm the SCADA Rabbi)
Matthew Franz wrote:
> Lief,
>
> I imagine there will be deafening silence from the comic strip writers
> (to run with Jake's metaphor) regarding the use of fancy IT tools like
> load balancers that provide the sort of flexibility you describe.
> Whether in the control center or the substations I was doing
> assessments (no refineries, Walt so you can rest easy, not a big fan
> of East Texas or LA, too smelly) you were lucky to have an unmanaged
> switch and I never saw redundant routers or swtiches which will be the
> mainstay of networks designed for high availability.
>
> If you have a properly designed network you can safely make the sort
> of changes you describe while maintaining an appropriate level of 9's.
>
> This is probably more a function of the age of the networks than the
> desire to build this into the network infrastructure -- but I'm sure
> it has to do expertise, as well. To use anything more complex than an
> unmanaged switch would invite the control of the evil IT LAN/WAN guys
> into your business.
>
> But, probably the majority of the hardware from a controller remote IO
> gear in production does have the HA capabiliies that are now common is
> most network gear. In the mid-slzed PLCs (Contrologix/Quantum class,
> maybe I'm mangling the model names from memory) I looked at I don't
> recall the sort of active/standby sort of configuration (there may
> have been some redudancy in the same backplane) across difference
> chassis.
>
> How many of the field firewalls have any sort of HA features (of
> course they can't really because most of them are Linux) so they fail
> open right?
>
> I'm sure there are counterexamples of Safety PLCs or others. Lets hope
> the next generation of designs that are coming out of scada/automation
> vendors are taking advantage of 21st century HA kit. I did see some of
> this in the Cisco designs that were proposed with Rockwell.. And it is
> only through partnership with leading IT vendors that have the
> appropriate feature sets so that end users can build resiliency into
> the infrastructure.
>
> Of course the other alternative is to just stick your head in the sand
> deny there is any overlap in technology or methodology.
>
> - mdf
>
>> Now for me, most of my servers are in a "pool" behind a load balancer.
>> I can take out 3 servers in one data center and the other servers will
>> start to stress take out the fourth and it becomes more noticeable. Now
>> I am not a SCADA engineer so I can only guess at this; With SCADA
>> devices you do not have say a pool of 50 devices in one location behind
>> a load balancer so the removal of several devices does not really cause
>> a problem. So with that in mind, more due diligence it taken in
>> patching/replacing a defective device. Hopefully, giving you the end
>> result of updated devices and NO down time. I know that I would be
>> little upset if my gas service, or my electric service were to go out as
>> a result of a system update. ;)
>>
>> For my set up just a few years ago, I started patching by taking 8
>> servers out of service from a pool of 40, and found that we could not do
>> that. It was determined that I could take 3 out without a problem, but
>> for safety and group comfort the team settled on 1server at a time.
>> Then when we added servers in a second data center we followed the same
>> plan but made it one server per data center.
>>
>> Now, I think that on a *high level* the process of patch implementation
>> and device security is for the most part going to be "the same".
>> However, the road or methods used in that process are going to be vastly
>> different. Of course since the devices at question serve vastly
>> different needs the mindsets are going to be different.
>> Now, high level (I will not include all possible areas)
>> - PROBLEM FOUND
>> - Determine possible impact/need
>> - are are at risk?
>> - can we mitigate risk by deactivating "unused features"
>> - can we be shutdown?
>> - can we be impaired but still function
>> - determine the relative importance based on impact/need
>> - fix asap
>> - fix in 30
>> - fix in 60
>> - If there is a test environment TEST (this is where it might get
>> vastly different.)
>> - Implement.
>>
>> Now it is all about the fragility of the infrastructure that is being
>> updated. That will indeed determine the road taken to fix the problem.
>>
>> just my 1.50 inflation is a pain!
>> --
>> Leif Ericksen
>>
>>
>>
>>
>> On Tue, 2008-02-12 at 12:28 -0600, wboyes at putman.net wrote:
>>> Leif,
>>>
>>> Most of the "public facing" server examples you quote are not germane to
>>> this discussion. I did not intend to talk about Yahoo, or Postini. I was
>>> speaking specifically about my own area of expertise, that of the
>>> manufacturing companies and enterprise IT in those industries. If Exxon
>>> has to take down its public-facing website because it gets hacked, not
>>> many people would care outside of Exxon, and it probably would rate less
>>> than a story on the evening news.
>>>
>>> Yes, many of the rules are the same for both plant IT and SCADA systems
>>> and the standard enterprise IT security establishment. THERE IS NOBODY
>>> ARGUING ABOUT THAT.
>>> Where we differ is that I believe, quite strongly, that there are
>>> significant differences in both implementation and philosophy between the
>>> two.
>>>
>>> Walt
>>>
>>> ----------------------------------------------
>>> Walt Boyes
>>> Editor in Chief
>>> CONTROL magazine
>>> ControlGlobal.com
>>> 555 W. Pierce Road, Ste. 301
>>> Itasca, IL 60143
>>> 630.467.1301 x 368
>>> wboyes at putman.net
>>> Read my blog, Sound Off, at www.controlglobal.com
>>>
>>>
>>>
>>> Leif Ericksen <lericksen at sbcglobal.net>
>>> Sent by: scadasec-bounces at news.infracritical.com
>>> 02/10/2008 06:45 PM
>>> Please respond to
>>> scadasec at news.infracritical.com; Please respond to
>>> lericksen at sbcglobal.net
>>>
>>>
>>> To
>>> scadasec at news.infracritical.com
>>> cc
>>>
>>> Subject
>>> Re: [SCADASEC] IBM is offering 'SCADA security best practices'...
>>>
>>>
>>>
>>>
>>>
>>>
>>> Please see in-line comments.
>>>
>>> On Sun, 2008-02-10 at 16:42 -0600, wboyes at putman.net wrote:
>>>> If what you are doing is SCADA security, instead of IT Enterprise
>>>> security, I would like to offer two observations.
>>>>
>>>> The first is that SCADA security has a somewhat different purpose than
>>>> enterprise security. Both are certainly aimed at protecting the assets
>>>> of the corporation or utility from attacks, whether interior or
>>>> exterior. But enterprise security seeks to protect the servers
>>>> primarily, and reacts to attacks by cutting off what external or edge
>>> Yes SCADA security and enterprise security are can be very different
>>> beasts. However, I would disagree that enterprise security primarily
>>> reacts to attacks by cutting off devices that the security team thinks
>>> needs to be cut off.
>>>
>>> (Keep this in mind as you read this, there is a difference between
>>> perceived attacks, actual attacks in progress, and attacks with a
>>> penetration.)
>>>
>>> Now if there is an active attack going on they (the Security) may cut
>>> off access to prevent further damage but then again depending on the
>>> attack they may simply block access to the IPS that are creating the
>>> attack. This block could mean an entire Class C, Class B, or even Class
>>> A network, how about even several networks? For instance if a
>>> companies Internet facing mail servers are under active attack they
>>> will/may not simply shutoff all access to the Internet, rather they may
>>> just block several networks or the offending IPs. Granted there
>>> might/will be fallout of "GOOD" IPs if you start blocking entire
>>> networks. But that will be relatively insignificant, if you keep the
>>> servers up and running.
>>>
>>>> devices the security people think they need to. SCADA security,
>>>> however, seeks to protect operating systems _while they are continuing
>>>> in operation_ and cannot shut down edge devices (the definition of
>>>> these is somewhat different, too, between SCADA or plant security and
>>> No real difference than my sample above. But we could even change this
>>> to the company's public facing web servers. If they are under active
>>> attack, the company may only shutdown the offending IPs or networks.
>>> The Servers will still do their jobs, but the offenders will be blocked
>>> until they alter tactics and move to a network that is not blocked.
>>>
>>>> IT enterprise security) unless nothing else will do. When we
>>>> structure a defense position, then, we must look at this critical
>>>> difference. You can shut down most, if not all, enterprise functions
>>>> for significant periods of time with little harm. You cannot shut down
>>>> a plant or a SCADA node without critical repercussions.
>>> I would disagree, a company's Internet facing web servers get a lot of
>>> notice. How about a company that provides Internet E-mail services to
>>> hundred to thousands or more people (GoodMail, Postini SBCGLOBAL/AT&T,
>>> Verizon). Those shutdowns would get noticed fairly fast. Even
>>> internally, if a company were to shutdown services it could get noticed
>>> quickly. For instance if a company shuts down its Internet gateway many
>>> people in the company will notice that.
>>>
>>> How the shutdown would impact the company really depends on how long and
>>> how wide spread the outage is. Will it be local, national or worldwide?
>>> In this modern age it seems that every bit of news eventually makes it
>>> world wide. You have got to love the Internet!
>>>
>>>> The second is that it is vital to IT Enterprise security (and quite
>>>> rightly so) that every guarded entity, be it a server, a switch, or a
>>>> computer, have the same level of software revision, and firmware
>>>> revision. On the plant floor, or in a SCADA implementation, it not
>>>> only is not necessary to do this, it can very seriously be not only
>>>> counterproductive but also actively harmful to do this.
>>>>
>>>> At the ARC Forum last week, Boeing IT experts, Craig Dupler and Steve
>>>> Venema, explained in detail how Boeing realized the truth of these two
>>>> observations, and what they have been doing for the past five years to
>>>> differentiate and distinguish plant and enterprise security systems. I
>>>> hope to have them present this in article form in _Control_ in the
>>>> coming months.
>>>>
>>>> Any security expert who has not carefully internalized these
>>>> significant differences between enterprise IT security requirements
>>>> and plant and SCADA security requirements can actually be an active
>>>> danger to the plant or SCADA implementation-- as dangerous as an
>>>> uncontrolled attacker.
>>>>
>>>> Walt
>>>>
>>> Bottom line we would have to say general internal corporate operations
>>> might have servers that can get shutdown unnoticed. But if there is any
>>> public facing part it is likely they will try to keep the servers up and
>>> running unless they know the there has been a penetration. Now if there
>>> has been penetration you will have one ugly mess on your hands. Not
>>> only will you have an ugly mess, but you will have many upset folks, and
>>> some red faces are possible in the administrator community. These red
>>> faces are likely caused by administrators *failing to do their jobs* by
>>> keeping the servers patched, or even minimally "secured".
>>>
>>> Now, we all have to remember the rules of security for computers or
>>> other networked devices.
>>>
>>> - If you must have a secure system DO NOT purchase a system.
>>> - If you must purchase a system, DO NOT take it out of the box.
>>> - If you must take it out of the box do not power it up.
>>> - If you must power it up DO NOT connect it to a network (or activate
>>> wireless networking)
>>> - If it must be connected and on the network, *YOU MUST* follow all the
>>> basic and advanced precautions to ensure that the system(s) stay safe
>>> and "secure".
>>>
>>>
>>>
>>>> ----------------------------------------------
>>>> Walt Boyes
>>>> Editor in Chief
>>>> CONTROL magazine
>>>> ControlGlobal.com
>>>> 555 W. Pierce Road, Ste. 301
>>>> Itasca, IL 60143
>>>> 630.467.1301 x 368
>>>> wboyes at putman.net
>>>> Read my blog, Sound Off, at www.controlglobal.com
>>>>
>>> There may be some differences in all between SCASA and general
>>> computer/network security. However, it seems to me that some of the
>>> basic principles can be applied to both. Although, I am sure that there
>>> are some 'special' cases that have to be dealt with in order to 'secure'
>>> the environment.
>>>
>>> --
>>> Leif Ericksen
>>>
>>>
>>> _______________________________________________
>>> To unsubscribe from this mailing list, please visit:
>>> http://news.infracritical.com/mailman/listinfo/scadasec
>>>
>>> To review our privacy statement, please visit:
>>> http://www.infracritical.com/privacy.html
>>>
>>> scadasec at news.infracritical.com
>>> http://news.infracritical.com/mailman/listinfo/scadasec
>>>
>>> _______________________________________________
>>> To unsubscribe from this mailing list, please visit:
>>> http://news.infracritical.com/mailman/listinfo/scadasec
>>>
>>> To review our privacy statement, please visit:
>>> http://www.infracritical.com/privacy.html
>>>
>>> scadasec at news.infracritical.com
>>> http://news.infracritical.com/mailman/listinfo/scadasec
>>
>> _______________________________________________
>> To unsubscribe from this mailing list, please visit:
>> http://news.infracritical.com/mailman/listinfo/scadasec
>>
>> To review our privacy statement, please visit:
>> http://www.infracritical.com/privacy.html
>>
>> scadasec at news.infracritical.com
>> http://news.infracritical.com/mailman/listinfo/scadasec
>>
>
>
>
More information about the scadasec
mailing list