Saturday, December 29, 2012

Retirement and Opportunity - A Personal Note Going into 2013

Friends and family gathered last night to wish my wife and I the best during a Nebraska Public Power District retirement party.   My retirement, effective October 31st, wraps up a twenty-one year career with a large, vertically integrated public power utility company.   Also invited and present were some of my prior Behlen Mfg. Co.  colleagues, where my career first focused on developing and supporting engineering and manufacturing solutions across a wide range of platforms and technologies.

Career and Technology Perspective.     Those of us that have been in the information technology and security fields for several decades can easily look back at our own experiences and appreciate incredible advancements.     When I started with Behlen Mfg.,  the systems were distinct and independent:  mainframe for business  (IBM 3400 series), mini for engineering design and schematics production (Synercom Technology's flavor of DEC's PDP 11/70), and a sprinkling of dedicated, often proprietary end user systems that ranged from graphics stations, word processing stations, to dumb terminals (aka tubes).    Behlen offered a great opportunity to work a wide range of challenges from programming engineering and manufacturing focused solutions (generating bill of materials, etc. based on parametric inputs on the mainframe) to eventually include "downscaling" some of mini-computer building steel frame design/iteration programs to engineering PCs.  This allowed the engineering team to further enhance, optimize frame building designs by speeding up an iterative process, permitting more than one design to be analyzed at a time, without huge additional spend.    I also had the neat infrastructure challenge to directly support Mainframe VM and mini-computers.. planning, performing key upgrades (OS, DASD storage, core to digital memory with salvage parts, etc)    Behlen was also where I helped bring on the PC revolution with computer aided design systems (CAD) including some useful CNC (computer numerical control for manufacturing automation) and more broadly used office productivity software, establishing networking (3COM, Banyan Vines), while coding up some very useful Turbo Pascal applications.

After five years with Behlen, joining NPPD offered additional opportunities to bring on server and PC local area networking "LAN" advancements, and seeing a very large commitment to mainframe based computing continue scaling up before being rapidly phased out of the organization with a Y2K focused large ERP (Enterprise Resource Planning) implementation on mini computers.      Networking during this time frame eventually transitioned from distinct architectures and implementations to the now ubiquitous TCP/IP protocol.  The Internet opened up with the first killer app being email, followed by continued world wide web and search engines advancements to help access rapidly improving capabilities while also making the Internet broadly more accessible and useful.

Over the years we have seen the rising flood of information technology increasingly encompass everything we know and care about:  smaller, faster, decreasing cost and increasingly connected.  Computing power that used to take a building with dedicated staff from the early commercial days now fits in the palm of our hands, a thousand times faster; representing over a billion-fold price/performance improvement.  All this change articulates an exponentially paced advancement that is continuing and further accelerating according to some ...more.

Increasing connectivity, capability, and dependence on information technology dynamically and dramatically ramps up real world risk considerations.    Today, a solid grasp of the security issues, including compliance, must be factored into technology strategy and decisions for organizational success.
Cyber Security Focus.   Since 2002, my focus at NPPD centered on cyber security in corporate and increasingly operational settings, e.g., fossil, nuclear.   While this work with colleagues was rewarding, an opportunity emerged after reaching retirement eligibility mid-2012 to join the ES-ISAC (Electricity Sector Information Sharing and Analysis Center), supported by NERC (North American Electric Reliability Corporation).  I have accepted the challenge, directly supporting the ES-ISAC at NERC in Washington DC.

The focus on mandatory standards and compliance enforcement dominates much of what electric utility entities think of NERC since the Energy Policy Act of 2005 and ERO (Electric Reliability Organization) designation by FERC (Federal Energy Regulatory Commission).  The challenge for the ES-ISAC is to continue building capabilities and trust with the industry, federal partners, and regulatory bodies while also striving to be increasingly forward leaning in anticipating and appropriately addressing key security challenges using automation and more traditional methods, such as NERC Alerts.  The key industry security focus areas for the ES-ISAC looking forward into 2013 include building out operational capabilities under development and further bolstering core programs  (e.g., assessments, exercises) and outreach (e.g. webinars, workshops).

Federal bodies remain acutely interested and inquisitive about what the electric power industry is doing to address security concerns even as related standards advance and compliance enforced footprints scope rapidly expand across the industry with FERC and NRC (Nuclear Regulatory Commission) driven oversight, auditing, and inspection.

I expect cyber security to continue being a challenging and rapidly evolving critical infrastructure arena.  This is an exciting time to be engaged with critical infrastructure protection!

Monday, August 6, 2012

Rules of the Game Still Apply (Terrorism)
1989 G. Gordon Liddy Article Continues To Resonate

As an avid Omni magazine reader years ago, one particular "after-the-fact" fictional article from the Jan 1989 issue captivated my attention- penned by former Nixon Administration convicted confidant G. Gordon Liddy.

The fictional memo characterizes critical infrastructure concerns and analysis from postulated events that continues to remain in many ways applicable and a challenge today:

    -  Since 9/11, air terminal facilities security upgrades provides substantial mitigation from the threat of liquid metal embrittlement (LME) agents.
    -  Potential electric grid physical attacks on high voltage transformers across a wide area would be quite debilitating and difficult to recover from even with progress on spare transformer programs.  Other significant types of physical grid damage blackout risks include larger area electromagnetic pulse attacks  (commission findings) and geomagnetic storm events being address with NERC's Geomagnetic Disturbance Task Force - GMDTF.
  5. COMPUTER DATABASE ERASURE OF WALL STREET, SIX FEDERAL RESERVE BANKS, TWO IRS SERVICE CENTERS, SEVERAL OF LARGEST COMMERCIAL BANKS, AND NUMEROUS CORPORATIONS PRODUCES FISCAL CHAOS - since 9/11, financial organizations have bolstered offsite backup facilities and continuity planning that would help at least in part mitigate the impact today.
The memo goes on to provide insights and recommendations:
  • ... the "prayer" of public officials has always been that a disaster will be either so immense as to be perceived as an "act of God" and thus engage the loyalty and team spirit of both the government and a patient populace or so small that it will go away by itself. The dread of officials is the one in between, affecting more than one choke point, the one with which government cannot cope. It is dreaded because it damages the faith of the people in their government and the way of life.
  • .. current situation is a nightmare. The people know this was not an act of God. What has happened is so immense as to be almost incomprehensible to them. The people expect their government to do something about it; to fix the problem and punish those responsible. And the American people are not patient.
  • ..  delay in the use of force, and hesitation to accept responsibility for its employment when the situation clearly demands it, will always be interpreted as a weakness. Such indecision will encourage further disorder, and will eventually, necessitate measures more severe than first instance."
    --The United States Marine Corps Small Wars Manual (1940), page 27, paragraph (d)

Cybersecurity continues  gaining an increasingly important role bolstering critical infrastructure security with a rising flood of IT risks, including those associated with Smart Grid.  The potential for serious  impacts from physical or blended attacks also demands ongoing attention.

Monday, February 20, 2012

NERC CIP V5 Drafting: Showstoppers and Tune-Ups Addressed
-Honeywell-Matrikon's Team Shares Latest as Key Next Draft Forms Up

Updated 2/24/2012
NERC CIP V5 continues to form up since the first ballot failed to pass even as much of the industry incrementally focuses more on CIP V4. The NERC approved V4 adds up to a rather straightforward application of CIP V3, plus prescriptive bright-line criteria to determine facilities in scope (instead of owner developed risk based assessment methodology permitted prior). NERC CIP V5 is a whole new ball game.

Lastest on NERC CIP V5 - Proposed Changes

Honeywell-Matrikon's [in]Security Culture Blog and webcasts continue offering solid insights for organizations focusing on where CIPs are heading, addressing related compliance challenges. From the Jan 30th posting SDT Meeting updates – Or, an informal open letter, Tom Alrich provides his opinions regarding the direction on a set of key V5 draft issues:
  1. Inventory for Low-Impact Assets
    - First draft would require an inventory of all assets for compliance. This is in conflict with the SDT's intent and should be resolved in the next draft.

  2. Asset Identification
    - First draft has a fatal flaw of requiring review of all assets to identify BES reliability Operation Services supported. Next draft should return back to the approach of starting with facility identification before going deeper to supporting assets- a much more feasible and reasonable methodology.

  3. DPs and LSEs
    - To only be included if they have one or more systems meeting the bright-line criteria.

  4. TO Control Centers
    - Transmission operators (TOPs) already on the hook, no need to also burden transmission owners.

  5. Blackstart Plants
    - First draft's direction of raising all to Medium Impact would be counterproductive to reliability. Generators have a choice on whether or not to participate in regional blackstart plans and the cost of CIP compliance significantly exceeds typical financial benefit. Many anticipate large withdrawal of blackstart units nationwide from plans, some say this is already happening. A compromise proposed would assign blackstart to Low impact if no external routable or dialup connectivity is used.

  6. Power Plant Thresholds - 1500 MW
    - Right now very few plants meet this threshold for cyber assets given multiple industrial control systems, not a single cyber system, typically supports production. Many see it likely FERC will decide to lower this threshold given increasing concerns about having sufficient bulk electric assets addressed.

    2/23 Update - A simple thought experiment:
    "How much of the Bulk Electric System would remain available if all related facilities not in the scope of NERC CIPs or Nuclear (NEI 08-09)
    were taken out of service?

    Honeywell-Matrikon's latest post by Tom Alrich explains more -
    Version 5: About those Large Plants…. (2/23): ".. The main question is whether FERC will be pleased with 7.2 percent of non-blackstart generating units being part of a plant that will be a Medium Impact facility under CIP Version 5 or a Critical Asset under Version 4."


Thursday, January 26, 2012

Project Basecamp 2012 a Hit... Are We Really Ripe for More Attacks Like Stuxnet?
-Researcher Ralph Langner says "Yes" at NATO Keynote.

Project Basecamp A Hit- But Will It work?

Researchers participating in Project Basecamp clearly demonstrated just how extremely fragile and vulnerable many Industrial Control Systems (ICSs) remain to targeted cyber attacks during DigitalBond's S4 conference this month. Amazingly, a number of persistent vulnerabilities include poorly devised "features" in addition to a bucket load of underlying software flaws. Tools released include point and click easy Metasploit modules. All of this effort to extensively demonstrate persistent ICS security problems is ultimately intended to wake up C-level executives to help amp up pressure on the vendors for secure replacements ("a Firesheep moment"). Regardless, don't expect much soon as many experts agree we've seen ten years pass with few ICS vendor security improvements. DigitalBond's site continues dishing up excellent interviews (podcasts), videos, and blog entries worth paying attention to for those interested in ICS security.

What about Stuxnet - More to come or really just a one time event?

Here’s one of the most insightful, solid presentations available explaining how Ralph Langer & team pulled apart Stuxnet, what they found, and broader implications. While the Stuxnet windows “dropper” was top tier malware in many ways, including multiple zero-days, the real rocket science was approx. 15,000 lines of crafted industrial control system (ICS) malware   “digital warhead” payload developed by seasoned engineers (Langner’s opinion- not just “hackers”) targeting specific nuclear enrichment ICS assets.

Mr. Langner makes a solid case that this was a highly successful attack (like a missile) which invites an escalation for more to come. The code and modular approach itself is reusable in many ways. He’s also written a book "Robust Control System Networks: How to Achieve Reliable Control After Stuxnet" that ICS engineers, others can benefit from focusing on designing ICS systems with robust security baked in ..more.
Today (1/26) Safari Books Online has followed through on their promise to make Langner's book available to members at my request in 2011- oh yeah!

Thursday, January 12, 2012

Welcome 2012: Leaping Into The Future With A Singularity Primer
-"On track" per Ray Kurzweil as he answers the latest critics.

The future is something I've always enjoyed focused, insightful perspective around and seems like a good topic to get my blogging mojo back in gear for 2012.

As I've touched on in a decade-plus forward look 2010 posting, Ray Kurzweil’s “Singularity is Near: When Humans Transcend Biology” 2005 book (672p) provides a science derived, profound view of how exponentially accelerating IT is driving ever increasing broader advancements. A very well executed, cited work in my opinion, with anticipated continuing advancements resulting in very dramatic changes affecting humanity over the next several decades (2020s genetics, 2030s nanotech followed by an intelligence take off, already in progress – i.e. technological singularity). One does not have to agree with all the points and conclusions raised in order to appreciate and gain much. This work was also released in 2011 as an Audible audiobook (unabridged, 25 hours).

Singularity Primer 2012:
  • Seminar Podcast: “Kurzweil's Law”- Ray Kurzweil (106m) - The Long Now Foundation - audio download is free

  • Video/Article:Kurzweil: 3 Supplements To Let You Live Until The Singularity (1m) May 2011
    - Coenzyme Q10
    - Phosphatidylcholine (
    derived from lecithin)
    - Vitamin D (perhaps the most critical of the three)

  • Movie: Transcendent Man (2009)- Netflix Instant Play Inventor and futurist Ray Kurzweil is the subject of this documentary that follows him on a world speaking tour in which he expounds on his ideas about the merging of man and machine, which he predicts will occur in the not-so-distant future. The visionary who invented the first text-to-speech synthesizer and much more raises eyebrows here with his wildly optimistic views of a technology-enhanced future. I give it B- rating.. but worth seeing once for most.
The Singularity is something that may utimately be an overwelming primary factor in shaping our future- very interesting indeed!

Saturday, November 19, 2011

False Alarm? Russia Cyber Attack on Water System SCADA Reported
-Cybersecurity back in limelight, asserting more intrusion(s)

11/23/2011 Update - A False Alarm?
*** ANSWER: Yes ***
For the initial Nov14th report per DHS- with more "pr0f" (proof) hackery being demonstrated and investigated !

As the week of Nov 14th closed, a reportedly "confirmed" water system intrusion discovered after equipment damage prompted a sensitive fusion center advisory, quickly followed by more public coverage:

- Issue discovered Nov 8th after pump burned up due to power cycling.
- Believed credentials used stemmed from supplier/vendor breach (e.g. perhaps via phishing)
- May have been compromised for months with ongoing "instability glitches" dismissed
- Involved access from Russian Internet addresses.

A Nov 10th Illinois fusion center report serving as initial notice regarding this matter was obtained by Joe Weiss, crusader for critical infrastructure security, who then broke the story providing some particulars to major media. A statement released by DHS spokesman Peter Boogaard downplayed the matter “At this time there is no credible corroborated data that indicates a risk to critical infrastructure entities or a threat to public safety.”

11/23/2011 Update
Illinois intelligence fusion center reported Tuesday 11/22 that earlier reports of a water utility hacked cannot be substantiated, according to a DHS announcement. Joe Weiss's quote to - “This smells to high holy heaven, because when you look at the Illinois report, nowhere was the word preliminary ever used,” Weiss said, noting that the fusion center — which is composed of Illinois state police, as well as representatives from the FBI and DHS — distributed the report to other critical infrastructure facilities in that state. “It was just laying out facts. How do the facts all of a sudden all fall apart?”

Following the initial DHS statement, a PGP signed posting by "pr0f" asserted evidence of gaining unauthorized access a second water treatment facility SCADA with five screen shots and statement, excerpt: "I dislike, immensely, how the DHS tend to downplay how absolutely F*****D the state of national infrastructure is....I've also seen various people doubt the possibility an attack like this could be done. So, y'know. The city of South Houston has a really insecure system. Wanna see? I know ya do... "

11/23/2011 Update
Sophos's Chester Wisniewski was contacted by the hacker "pr0f" regarding the South Houston, Texas intrusion. The hacker gained access through several methods (VNC variant, web portal) claiming he still has access. He also commented "Don't worry, I use my powers for good and such." And also pointed out, ".. I am under no illusions about my level of skill. These are the least secure systems. .. I was furious at the lack of proper government response. The response they gave was nothing more than 'Nothing happened. Probably.' When clearly something did happen."

What should utilities do?

Mr. Weiss provided some constructive broader recommendations in his post "Water System Hack - The System is Broken" Here are some specific suggestions for near term critical infrastructure cyber risk mitigation, especially for industrial control system (ICS) settings where cyber security may be lapsing, not addressed in a robust manner:
  1. Identify all ICS systems and their organizational management owners.

  2. Audit key baseline IT security controls, identify any serious remote and local access issues
    - e.g. protected perimeter, all accounts have defined need, management approval/review, access activity logging for review, antivirus where feasible, patching.

  3. Consider how to assert stronger positive owner access control, especially for remote access
    -e.g. remote access normally disabled when not needed, logging all access events, multifactor token required/kept in house for vendor call in, protected jump box use instead of opening full throat network paths, segmentation when multiple vendor solutions are involved.
    Note: A good place to start is closely studying NERC's July 2011 "Guidance for Secure Interactive Remote Access"

  4. Implemented initial improvement options based on risk informed priority.
    - proceed based on management engaged approval/direction, document and implement, monitor and report progress.

  5. Pursue ongoing, broader ICS security improvements
    - after getting basic IT-centric hardening measures in place, tools such as DHS's CSET (Cyber Security Evaluation Tool) - free for critical infrastructure organizations are available to build better understanding of ICS security susceptibilities and consequences, measure risk, and identify, prioritize further security improvements.
Any such attack damaging a water utility's pump is more akin to amateur antics than part of any organized nation state effort in my opinion. Regardless, even if this turns out to be a false alarm for causing of equipment damage, many related "what ifs" will be asked by media and others. We can expect various hats of hackers (white, grey, black) interest will also increase (SHODAN anyone?). Industrial control systems, including SCADA, are widely used to support a number of critical infrastructure functions. Secured communication paths and protected remote access must be ensured. Organizations that have blindly entrusted their vendor to adequately address cybersecurity in an increasing risk environment need to do more. People, process, technology requirements addressing security in such settings must be understood, documented, supported (with enforcement), and continue to be further developed.


- Cyber Intrusion Blamed for Hardware Failure at Water Utility- KrebsonSecurity 11/18/2011
- H(ackers)2O: Attack on City Water Station Destroys Pump- 11/18/2011
- Second Water Utility Reportedly hit by hack attack - The Register 11/18/2011
-proof of concept Intrusion
- Hacker targets South Houston Sewer System - The Houston Chronicle 11/19/2011

- What You Should Know About SHODAN and SCADA - DigitalBond 11/2/2010

Tuesday, September 20, 2011

EU BlackHat 2011: Cyberwar Overhyped, Escalating Cyber Conflict The Issue
- EU Keynote counters Ex-CIA Official's Warning

While imminent Cyberwar concerns have ramped up as of late, e.g., BlackHat 2011: Cyberwar is Coming- Ex-CIA Official Warns Black Hat 2011 Attendees, an insightful EU Black Hat 2011 - Keynote (video 1:15) with Bruce Schneier offers constructive and useful perspective:

“It’s not that that we’re fighting cyberwar, we’re increasingly seeing war-like tactics used in broader cyber conflicts. Non-nations can now deploy war-like tactics... a bunch of criminals getting tanks.. now what do you do?" - Bruce Schneier EU BlackHat 2011

Schneier points out that cyber war clearly is not happening now. Rhetoric surrounding cyberwar is exaggerated and harmful in its influence over policy. The debate language lacks good definitions - Don’t know when it starts, what it looks like, who is doing it, or when it’s over. Using the term “war” implies we’re helpless, we need to duck and cover, the government should handle it. Many measures merited in war time pose greater risk in peace time. Advantage is on the attackers side in cyber space with technology pushing capabilities out- so easy, kids can do it.

Further cyberwar high-level analysis commentary addresses topics such as preparing the battlefield, conducting attacks, etc. All advanced nations will need to have some cyber offensive capability as it's part of the war fighting theater now. It's also understood that the most advanced nations have extensive capabilities, e.g., placing logic bombs into enemy systems, potentially before broader conflicts starts. Reoccurring examples of precursor cyber-attacks being followed by more traditional military conflicts. US continues dragging feet on pursing international rules and treaties involving cyber conflicts given a perceived advantage. This stance really feeds the cyber arms race problem where every side assumes the worse. Related offensive decisions also need to be made at higher levels of government- Stuxnet types of attacks are reasonable to view as an act of war.

Critical Infrastructure concerns include widely believed examples of non-US criminal extortions, blackouts from hacking, e.g. Brazil. History is rich with market failure examples where common defense not adequately addressed by private industry. Private industry can only go so far and why we need government, with regulations only part of answer. The US is clearly more vulnerable than other nations; with risk is increasing, it's important to further address.

- 60 minutes exposé - Cyber War: Sabotaging the System 6/13/2010 (video 18:02)
- “Next war might start with blackout, not a bang.” “Art of the Possible”

Tuesday, September 6, 2011

BlackHat and Defcon 2011: Top 10 Scariest Hacks
- Network World's take on a handful meriting the most concern

Las Vegas hosted Black Hat USA 2011 and Defcon 2011 conferences dished up a number of interesting hacking demonstrations applicable for critical infrastructure organizations. The wide ranging top ten identified by Network World (full slide show) included SCADA issues (Siemens, of course) and even a pretty significant ERP system issue (SAP).

  1. Siemens S7 hack (top one!). Very scary considering just how dependent real world facilities are to systems with related security problems, issues go well beyond being specific to Siemens solutions!
  2. VoIP botnet control. Clever data ex-filtration, command and control methods using VoIP channel, touch tones phones.
  3. Powerline device takeover. Demonstrating a device that can tap into home power lines, monitor and control home alarm/security cameras, e.g., enable intruders to jam security gear then break in.
  4. Hacker drone. Off-the-shelf electronics used to create WASP (wireless aerial surveillance platform) executing flight plans while doing its work (crack codes, pick up cellphone calls, etc).
  5. Car hijack via phone networks. Using text messages over phone links to hack a Subaru Outback car alarm, unlock doors, starting vehicle. Similar to devices used in some critical infrastructure settings, raising concerns about knocking out power grids and water supplies.
  6. Hack faces to find Social Security numbers. Acquiring a person's Social Security number using nothing more than social networking photo, face recognition software, and a deducing algorithm.. interesting!
  7. Remotely shut down insulin pumps. Exposing a very difficult to resolve wireless security problem- could be fatal in wrong circumstances. The diabetic security researcher focused on issues with his own wireless pump.. "devices weren't designed with security in mind"
  8. Embedded Web server menace. Embedded web servers in photocopiers, printers may them easier to administer and be compromised, potentially pilfering produced documents. Easy fingerprinting and attack approaches demonstrated.
  9. Spreading false router tables. Demonstrated OSPF (open shortest path first) routing protocol having weaknesses permitting attackers to install false table entries on uncompromised routers, potentially affecting data streams (sending info to remote attacker) or just crippling networks.
  10. SAP flaw- Authentication. Showed how SAP system can be broken into, gaining administrative privileges. The researcher determined that half the systems examined were vulnerable to this issue. Easy to locate target systems with Google search. SAP is working towards releasing a related security update.
- Insulin pump attack prompts call for federal probe‎ - Register 8/19/2011- Committee urges investigation into security standards for wireless medical devices.
- Black Hat 2011 USA Archive video, audio, slides added since Aug 2011 conference
- DEF CON 19 Archive - site stood up 9/5 w/slides, etc from Aug 2011 conference