Very interesting Dan's presentation was Very impressive Mr.Daniel Nagala give presentation at the
NATIONAL TRANSPORTATION SAFETY BOARD
PIPELINE SAFETY HEARING
Leak Detection and Response
Conference Center and Board Room
National Transportation Safety Board
Washington, D.C.
The next panel is the Independent Systems Design Panel. On this panel is Mr. Daniel Nagala from UTSI International Corporation, and Mr. Al Senftleber from Senftleber and Associates.
Panelists, I would like to remind you that you have 15 minutes to give your presentation. The yellow light in front of you will go on when you have two minutes remaining. The red light indicates that your time is up.
Mr. Chairman, staff is ready to hear from the panelists.
CHAIRMAN HALL: Well, I'd like to welcome both of you gentlemen, and, Dan, if you would please proceed. You took the Number 1 seat there. So, --unless there is a different order the two of you all have agreed to. That's fine.
MR. NAGALA: We're okay. Is this working? Yes.
Panel: Independent System Design
MR. NAGALA: First of all, I'd like to thank Chairman Hall and the other Members and staff of the Board for allowing me to address you in this forum today.
My comments are directed to the overall state of the industry from a SCADA automation perspective and include comments pertaining to vendor capabilities, operating company expectations, and common practices for system maintenance and on-going support.
The comments I'm about to make have been compiled from my own observations and experience within the pipeline automation industry and are not targeted toward any particular automation supplier, pipeline operator, or service organization.
I will begin by offering a short summary of the evolution of SCADA systems over the past 25 years, followed by a discussion of the current state of the industry from my perspective, and, finally, I will offer some suggestions for how we might improve our approach to the design, implementation and support of such systems.
While some of my comments may appear to be critical of the industry in some aspects, they are notintended to be so and are only presented here to form a basis for the suggestions I will present at the conclusion.
In order to fully appreciate where we are today and how we got here, I would like to take everyone several years back in time. During the 10-year period between 1975 and 1985, many of the foundations for pipeline SCADA systems, as we know them today, were formed.
However, during these years, the approach to implementation of such systems was substantially different than that of today's common practice.
In the latter half of the 1970s, the typical approach to a pipeline SCADA project was to build a custom system to a rigid set of requirements, generally produced as a joint effort between the operating companies, engineering and operations departments, and in many cases, it was common to have selected a software or system supplier prior to the specification effort, making the process a combination of addressing specific operational and performance requirements in parallel with detailed software design.
The outcome of such efforts generally resulted in very detailed written descriptions of the software and hardware systems to be provided, includingdefinition of all principal data structures and software modules, sometimes to the point of detailed program flow diagrams.
While the merits of such a process can be debated, the end result was a solution specifically engineered to achieve a given set of requirements for a given target operation.
These projects, while time consuming and expensive, were generally completed successfully and provided expectation levels of functionality and performance.
In the late 1970s, the emphasis was typically on performance, reliability and operational stability, and the technology choices required to achieve the desired characteristics were typically left up to the operating companies, engineering personnel, and the system supplier.
While most companies in the SCADA business prior to 1980 had their preferred technology platforms, because of the custom nature of their business, it was typical to find most of them very receptive to the use of alternate technologies wherever it was deemed to be technically appropriate.
As the early 1980s arrived, system suppliers and operating companies began to grow weary of the costand continued maintenance and support complications associated with custom systems and began to look for solutions based on more product-oriented approaches.
Between 1982 and 1985, several companies initiated R&D projects specifically related to the design and development of SCADA products that could be configured and applied to solve multiple sets of requirements without the necessity of substantial custom development on every project.
These products were typically designed and developed for a particular hardware and operating system platform. However, most of those systems still remaining in the market today have also been forwarded to other platforms.
The era of SCADA products had arrived and was in full force after about 1985 with several vendors actively competing for turnkey projects with a standard product offering.
Moving into the 1990 time frame and coinciding with wide acceptance of personal computers and local area networks within operating companies, we began to see a notable shift in the expectations for SCADA within the business enterprise. The result being apparent to new architectures, many of which incorporated tools common to traditional businessapplications but never previously used within a SCADA environment.
Notably, the incorporation of relational database management systems or at least some level of connectivity to such systems into SCADA architectures was almost universal among principal suppliers in the market by 1993.
The driving force behind these evolving architectures was to provide better and more timely ways of moving information pertaining to the physical operations of the pipeline to management and to related departments.
Pipeline SCADA systems were no longer simply operational tools implemented and supported completely by a dedicated engineering group but an integral part of the daily business operations of the enterprise.
While all of the SCADA architecture and requirements changes were taking place in the early '90s, operating companies were also undergoing radical changes from the organizations of the '70s and '80s.
Almost every company experienced a significant loss of expertise related to their SCADA systems as a result of corporate down-sizing, reorganizations, mergers and acquisitions and so on, leaving the company with more complicated systems thanever before but with less internal expertise to keep those systems operating at peak performance.
Wide acceptance of and an almost exponentially-expanding dependence on internet-related technologies since about 1997 have further served to change the appearance and expectations of SCADA systems in the eyes of many industry professionals and promise to add yet another dimension of complication to these systems while the availability of sufficient SCADA support personnel and their relative depth still remains behind that which was common several years earlier.
This brings us to where we are today. Generally speaking, most of the systems that are in use today work pretty well, and while there are always exceptions to any given set of observations, it is apparent that these systems have substantially more demands being placed upon them than ever before, but for the most part, they have kept pace with the challenge.
Still, most of these systems continue to rely largely upon the legacy of the early 1980s R&D for their base functionality, having had little enhancement to the core functionality or its underlying software architecture since its initial implementation.
It is apparent that vendor companies have had and continue to have a difficult time meeting customer expectations while attempting to remain profitable and competitive.
In the typical project today, vendors, operators and consultants all tend to place much less emphasis upon the traditional performance, reliability and core functionality components of a system, and tend to focus more on the business and enterprise integration of such systems.
In fact, many industry professionals perceive the traditional core functionality of the early years as a commodity in today's market and take for granted that it will always be there. But this view may be incorrect and may be producing an unintentional shift of focus away from these items, so as to address other more pressing project requirements.
SCADA systems are significantly more complicated than ever before. Most of this can be attributed to the fact that the underlying system architectures rely on substantially more third party hardware and software components than their predecessors of the '70s and '80s.
This has also changed the image of the typical SCADA supplier to encompass a significantamount of complex system integration tasks for both hardware and software components.
Because of this, the skill sets found within vendor companies have also changed from that of pure programming and system engineering disciplines to include specialties in network architectures and protocols, multiple operating systems, relational database administration and management, graphic display standards and related tools, just to name a few.
The task of seamlessly putting all of these components together into a single continuously-operating environment, configuring them to satisfy a given set of customer requirements, and conducting complete and thorough system functionality testing, all under a budget set forth by a tight competitive bidding process, is nearly an impossible task.
Combine this with the fact that the pipeline SCADA market is very small in comparison to other automation markets, and therefore cannot offer sufficient financial resources to fund the necessary R&D required to keep pace with emerging industry expectations and advances in relevant technologies.
The results of all of this are systems that require functionality that is not always consistent with the core competencies of suppliers, complexsoftware development or system enhancement tasks being performed under the scope of specific projects, systems that go to the field before they are ready, and systems that generally require a significant amount of on-site debug and testing prior to being approved for on-line operation. Sometimes the on-site work alone can continue for up to a year or even more.
All of these things contribute to the potential for having overburdened and unreliable systems under certain conditions of operation.
A related example of this might be an instance where a given system is required to send critical information on a periodic or exception basis to some other system or another department or to even another company.
We have seen many implementations where the preferred architecture has moved away from the former dedicated communication path or protected network transport method in favor of techniques, such as the Internet's FTP, SMTP or other related means.
While these techniques can work reasonably well most of the time, they were never designed to operate in an environment where guaranteed performance and availability was required, and when a problem occurs, many times, it will require human interventionto correct.
Usually while such a problem is present, certain system functionality is completely unavailable to individuals and/or related applications. This is an example of an implementation to address an evolving functional requirement without a complete engineering analysis to determine the appropriate solution for the desired outcome with the requisite level of reliability.
Today, it is common to find the SCADA system falling under control of corporate information technology departments. While some of these organizations have simply absorbed the previously-existing engineering departments responsible for the companies' SCADA systems, others have inherited such responsibility because the former department or its key personnel no longer exist within the company or are unavailable for some other reason.
It is this latter case that is of concern to me. In my consulting practice, I have come across groups responsible for such systems who have little or no experience in dealing with critical real-time systems, not to mention the absence of a firm understanding of fully-redundant, no-single-point-of-failure architectures, and the importance of isolatednetworks for such systems.
This is not to say that such people are not receptive to these ideas, but the underlying concepts and their critical importance to SCADA are not part of the background typically afforded to the traditional IT professional.
Although most any SCADA system can be very reliable when configured and maintained properly, that same SCADA system can be very fragile and unreliable if operated in an environment contrary to that envisioned by the original system designers.
For this reason, it is of critical importance that such system be specified, implemented, managed and maintained by personnel who possess the appropriate skill sets and background system knowledge.
I've seen many of my colleagues from the real-time modeling and simulation disciplines here today, and it's apparent that these are areas very well represented. Therefore, I will withhold any specific comments pertaining to leak detection systems and their application, except for one particular item that relates directly to the SCADA system.
Most software-based leak detection systems operate in conjunction with the SCADA system and are completely dependent upon information from thesesystems in order to perform their primary function.
However, in many cases, among those specifying performance, there's a naive lack of under-standing of the relationship between the data acquisition and processing portion of the SCADA system and the leak detection algorithm.
Many times, we see leak detection performance requirements that exceed the limits of the SCADA system and its corresponding instrumentation. In some cases, such performance requirements are being heavily influenced by various regulatory and permitting agencies.
The point that I'm trying to make here is that no matter how much work is put into producing a highly-sensitive leak detection algorithm, the quality of that algorithm can only be as good as the data that it's given to work with. The SCADA system must be considered in any such sensitivity mandate.
Overall, I don't believe that any of the difficulties present in the SCADA arena today cannot be overcome by exercising prudent system engineering, conducting thorough system testing and implementing well-defined, on-going maintenance and training programs.
As I hope I have indicated, our majordifficulties are stemming from being too aggressive in some areas at the expense of others, namely core system functionality, run-time performance, robustness and reliability, while struggling to keep pace with rapidly-increasing demands on these systems.
All of this is taking place with a workforce that has a much different set of objectives to satisfy and very different expertise than their predecessors of the 1970s and '80s.
Please keep in mind that I'm certainly not suggesting that we fall back to the '70s and stay there, but I do believe that there are some lessons that were learned during that period that have either been lost or forgotten, and we need to recover those as we move forward into the next generation.
Therefore, as I near conclusion of my comments, I would like to offer some suggestions for change for your consideration.
There are already regulations mandating operator training and qualification, and I believe that this training and qualification should extend to the SCADA engineering, support and maintenance activities.
In particular, Subparts N and G of 49 CFR Parts 192 and 195, respectively, pertain to operator and technician training. While these regulations donot -- are not directly related to SCADA and/or leak detection systems, implications are that such training programs must include specific knowledge and interpretation skills related to these systems, along with proven capabilities to adhere to company-specified recognition and response protocols.
Full-scale training simulation, covering SCADA, leak detection and related telecommunication system operation, should be a required component for all new control center projects.
All personnel with system maintenance or operator access to the system should be required to be trained accordingly on such systems and should be subject to periodic follow-up testing.
Support personnel might also be subject to other detailed off-line training specifically related to the SCADA system and the maintenance thereof. This type of program would go a long way toward resolving the experience and technical depth issues related to personnel turnover I noted earlier and would most certainly result in better-running and more reliable control centers.
New agency regulations, although some might be helpful, can only go so far. For example, mandates that attempt to define acceptable levels of CPU idletime, RAM availability, disk and network utilization, would soon become outdated, and in order to be broad enough would most likely be so vague that the range of interpretation could end up being almost like having no targets whatsoever.
Instead of mandating such items, operating companies might be required to certify that a new SCADA system or the result of a substantial upgrade are compliant with a predefined and deterministic set of criteria, and OPS is already working on a control center audit checklist that identifies various aspects of the SCADA system and its operating characteristics.
One approach might be to extend this to the site testing and commissioning phase for any given project. Rather than involving -- having regulators involved in such processes, however, the responsibility should be placed on the operating company to have a qualified SCADA system engineer witness and approve the system's operation on its own merits before it is put into on-line operation.
The term "qualified" in this sense is in reference to the operator training and qualification rule I mentioned earlier.
Particular emphasis for such testing should be placed on the performance reliability, security andcore functionality of the SCADA system, and records pertaining to such testing should remain on file for reference and comparison use during subsequent follow-up audits.
The SCADA market worldwide is very small and as such does not have sufficient revenue-generating potential to fund substantial amounts of vendor-sponsored R&D. Still, it is apparent that we need to be able to take advantage of new technologies, and perhaps more industry-sponsored or even federally-funded R&D programs could be established.
Such programs might be tasked to evaluate and recommend how a certain technology could be applied in a typical SCADA environment while still maintaining the performance reliability and security aspects of the SCADA system.
Thank you.
