Advice to a New SCADA Engineer

January 24, 2017 by

Jim Acord

Target Audience

As I have come in contact with those new to industrial control systems – whether they be supervisor control and data acquisition (SCADA) systems, building automation, process automation, or what not – I have come to the conclusion that whether the individual is trade school educated or college educated, they are not prepared for the world of automation.

So, I have decided to pass on some of my thirty-plus years of experience to a different generation. I cannot say another generation, as I have seen some "seasoned" individuals move into the automation arena from other areas of expertise. This document is some wisdom that can be applied in virtually any process network. It should not be considered a comprehensive list nor should it be considered all-inclusive. My goal is to assist one in NOT making a career-limiting action that will require one to update one resume or CV.

Learn ICS/SCADA Security

Explore realistic critical infrastructure scenarios and build your security skills with hands-on labs, on-demand courses and live boot camps.

Start Learning

Introduction

One's first days on a new job are a golden opportunity. One is not expected to know too much about the business, but one is expected to have some knowledge. Whether a SCADA engineer, a SCADA Analyst, an Automation Engineer,

an Automation Technician, or some other title, this document will apply to one.

This golden opportunity allows one to ask questions without being considered stupid or ignorant; play this card: "I'm the new guy here, and I don't understand this. Would one explain it to me, please?" That's a tough question for a more senior individual not to answer. One must ask questions but not too many questions in one session. By bringing donuts and coffee to this informal training session assist the situations. However, under no circumstances should one bring Starbucks coffee, as the people that do the real work will resent this gift.

How long does this golden opportunity exist? Well, that is dependent upon one's personality and the personality of those with whom one is involved. If one presents oneself as humble and with a thirst for understanding, coworkers will be more willing to educate one on the specifics of the given situation.

Know One's (Local) Process

One must gather an understanding of all of the processes that occur for the business to exist, with regards to automation. Therefore, one must learn what the local process is. This is the process that interacts with field sensors. It may be miles away from one's location or nearby. It may be simple or complex. The communication may be leased line, satellite, cellular, radio, microwave, or another communication media.

Take A Tour of the Local Process

Make a site visit. Walk around the entire process. Take pictures. Touch the process (keeping safety in mind). Have a local operator or site manager explain what is supposed to be occurring at this location. Ask if that is what is occurring. Ask what some of the problem situations are and what the solutions were/are. Ask if this location is similar to other locations or if it is unique.

Visit Process Control Center (Local)

Some locations may have a local control center or control room. Some locations may not. If present, ask for a tour of this facility. Take a look at the operator's display screens. Listen to see if there are any alarms occurring. Observe how the operators operate the process; this is what they get paid to do. Ask questions.

Become Familiar with the Process Hardware

What controller is used at this location? Is more than one controller used here? Are they the same brand and model? What field devices are present: valves, solenoids, switches, transmitters, transducers, positioners, relays, et cetera? Is this standardized equipment or is it unique to this site? Ask how program loads get updated. How often are updates performed? Ask what happens if the controller develops a case of amnesia.

Expert Draw One's Process

As the saying goes, "a picture is worth a thousand words." Ask a local operator or site manager to draw out the process for one. If the drawing is on a whiteboard, take a picture of it. If it is on paper, take a picture of it. Ask questions to make sure that one understand the process at this location. The odds are that this documentation is not completely accurate. However, it can serve as a basis for updated documentation.

One Draw One's Process

At a later date, one need to draw this process formally. I have used Microsoft Visio to great success. By documenting this process in a drawing, it:

• Reinforces one's understanding of the process
• Allows one to standardize one's drawings.
• Provides formal documentation for the process.

Get (Local) Process Programs

With all controllers, the manufacturer provides software to interface from a computer to the controller. It is typical that this program is used to diagnose problems with the controller and to monitor the software load running within the controller.

Obtain copies of all logs that the controller provides as well as the program load currently running in the device. This allows one to:

• See if there is a standard that has been used for the local process controller's program.
• Verify logic on local process controllers.
• Backup local process controller's programs.
• Provide a centralized location for the controllers' program loads.
• Become familiar with the information that is available from the device's logs.

Get The Big Picture

To get the big picture, one must take a tour of other sites. This is not to say that each and every site must be visited unless there is a small number and they are all close. A representative sample must be visited.

Know One's (Global) Process

After being exposed to the field operations, one should be getting an idea of the global process; the one that unifies all of the field operations and produces the finished goods for the organization.

Ask to See Any Documentation of One's Process

I'm not talking about the manufacturer's manuals. I'm speaking of the customized documents that describe what is unique about this global process. This may be drawings, documents, or handwritten scribblings on unlined paper.

The odds are that this documentation is not current. However, it can serve as a basis for updated documentation. The individual will, most likely, not be trying to mislead one; they do not have a complete understanding of the entire process.

Expert Draw One's Process

Just like before, have a local operator or coworker to draw out the process for one. If the drawing is on a whiteboard, take a picture of it. If it is on paper, take a picture of it. Ask questions to make sure that one understand the process at this location.

One Draw One's Network

As before, at a later date, one need to draw this process formally. By one documenting this process in a drawing, it:

• Reinforces one's understanding of the overall process.
• Document the interactions between field elements and centralized elements.
• Documents interaction between the centralized elements.
• Allows one to standardize on drawings.
• Provides formal documentation for the process.

Know One's Network

Correctly identify the functionality of each server:

• Alarm Management Server
• Alert Management Server
• Application Server
• Decision Support Server
• Domain Controller
• Engineering Server
• Historian
• Log Management Server
• Patch Management Server
• Real-time Server
• Remote Connectivity Server
• Security Application Server
• Update Server
• Web Server

Are the primary servers in matched pairs, i.e. a Hot/Standby server? What about secondary servers? Are any servers virtualized? If so, how? Is there any software that manages this? How can one tell which one is hot and which one is standby?

How do the various servers interact with each other? It might be extremely helpful to draw this interaction out. Think about a process flow diagram.

Where are the physical locations of all of these servers? What security is in place to physically access these servers? If these servers are in data centers, are they attached to a KVM? How is the KLVM accessible? Does the KVM function correctly?

Can any of these servers reach the internet? Can the Internet reach any of these servers?

What routers, switches, gateways, and firewalls are on the network? How manages these devices?

Verify one's network

Trust, but verify. Make sure that the documentation is correct.

After one has taken the effort to obtain all of this documentation, keep it current.

Incidentals

Other questions:

• Who does the displays?
• Is there any background programs or scripts running when a display is presented?
• What standards does this system adhere to?
• What regulatory, if any, bodies have oversight?

Donuts

This is a very important point: bring donuts. Whether they are donuts, doughnuts, crullers, tortas fritas, beignets, bismark, kleinuhringir, spurgos, buñuelo, pączki and lokma, these fried bits of dough are the gateway to the senior worker's knowledge.

Learn ICS/SCADA Security

Explore realistic critical infrastructure scenarios and build your security skills with hands-on labs, on-demand courses and live boot camps.

Start Learning

Conclusion

This bit of advice should ease the transition into a new position, a new location, or a new company.