04 - CambiOS_ICSSecurityTesting_VulnerabilityAssessment (2).pptx

ICS Security Testing and Verification ICS Vulnerability Assessment  Meta Data © 2024 CambiOS Academy LLC All Rights 08/29/2024 2 Reserved. Overview Who is this course for?...

ICS Security Testing and Verification ICS Vulnerability Assessment  Meta Data © 2024 CambiOS Academy LLC All Rights 08/29/2024 2 Reserved. Overview Who is this course for? What will be covered in this course? Anyone with a responsibility to assure the 1. Introduction to testing effectiveness of the ICS cyber program or 2. Reality check: Threats and incidents parts of it 3. An overview of testing methodologies ICS Security Team / CISO / Internal audit 4. Vulnerability assessment 5. Penetration testing What are prerequisites? 6. Red Team exercises Familiarity with industrial control systems and 7. Wireless Audit operational technologies 8. Product / application testing (Course: Introduction to ICS) 9. Tools & Technologies for testing Basic understanding of IT and Networking (Course: IT Fundamentals) 10. Planning and setting-up a test 11. Documenting and delivering testing results Basic understanding of IT Security (Course: IT Security Fundamentals) 12. Follow-up (how to act on the results) Key questions to be addressed by this course Why should you be testing? Who should be doing the testing? What are reasons for you to consider doing Is testing really the best thing to do? Now? testing? What should you be testing? What is the What are you trying to achieve? scope? How does your maturity impact your testing? What methodologies are available? What are prerequisites to be successful? How do you identify the scope of your test? Who is the audience / recipient of the results? What testing methodology should be used How do you “prepare” your audience? when? What are risks of doing testing? What are the typical activities performed? What can / should be the outcomes of a test? When / how often should you be testing? Who is the audience and how do you prepare How big is the actual risk? for them? How do you deal with the results? How do you deal with the results? What is the best timing? When should you What are legal implications / considerations? NOT test? Who should be aware and involved? 20 questions Step 1: Write 20 questions you aim to answer for the module 1. What is a vulnerability assessment? 1. What are the typical outcomes? 2. What is the goal? Step 2: Assess your 20 questions. Write down 3 questions (per 20) that are specific and would make for good test questions. Micro Unit Welcome Topics Covered Inlay Video video Proven ICS Vulnerability Assessment Methodology 3 stages Onsite Data Collection Offsite Data Analysis Developing the Report Deliverable 6-layer approach Physical Security Network Infrastructure ICS DMZ Control Room Assets (Servers/Hosts) ICS Communications and Protocols Field Devices Tips and tricks from 10 years of experience Learning Objectives Inlay Video video After having completed this module, you will … ✔ understand a proven ICS Vulnerability Assessment Methodology. ✔ know the key stages of the methodology. ✔ be able to follow a logical 6-layer approach. ✔ apply the methodology. Using the ISA S99 Security Model Inlay Video video External Pen Testing Internal Pen Testing ICS Vulnerabilit y Assessment Micro Unit Planning a Vulnerability Assessment ICS Vulnerability Assessment Work Plan Inlay Video video Dividing the project up into Three Stages helps with scoping and planning the work Tasks can be easily grouped and assigned in the following categories: 1. Onsite Data Collection 2. Offsite Data Analysis 3. Developing the Report Deliverable  ICS Vulnerability Assessment The 3 Stages Inlay Video video Onsite Work Offsite Work Review with Client Facility Tour Walk facility with local contact looking for physical security risks Document findings with Collect all documentation, drawings, recommendations and send Interviews and configuration files to validate current architecture and make 1st draft report to client Meet with ICS engineers and network admins for system discovery recommended changes to system architecture Documentation Collect network device config files, network Prioritize and Review draft report with the drawings, and other system documentation rank security client and receive client risks suggestions and Passive Analyze router, switch and firewall modifications. Capture network traffic on all core routers / configuration files, network switches captures, host scripts, application review notes, and any scanning log Host Systems files for system vulnerabilities. Take suggestions into Server scripts to extract configurations. Review ICS application security. Create list of security findings. considerations and publish Final Report. Review Final Active Report with client for project Conduct active vulnerability scans of offline test, signoff. backup, or simulation systems Passive onsite Active onsite Offsite analysis Offsite documentation  ICS Vulnerability Assessment Sample Timeline Inlay Video video Redraw potentially ICS Vulnerability Assessment Team Resources Inlay Video video Onsite Data Collection Project Manager with Interview Skills Network Engineer with Traffic Capture Capabilities ICS Engineer with Host/Application experience Offsite Data Analysis Vulnerability Researchers who can parse PCAP files and compare Host/Application findings with CERT data Developing the Report Deliverable Technical Writer to pull all pertinent data together and frame the recommendations ICS Vulnerability Assessment Planning for the Site Visit Inlay Video video Conference call with local team ahead of time coordinate best time for the trip onsite ensure the right local resources will be present and available for interviews and assistance with collecting the network and control room data discuss any operational risks during that time (such as power plants during the heat of the summer) discuss any safety issues and proper personal protective equipment (PPE) that should be worn while onsite provide an overview of what to expect from the team send documentation request ahead of the trip so local resources can begin pulling together data and documents for the team communication encryption for emailed documents and data collected onsite Micro Unit Six Layer ICS Vulnerability Assessment Approach  Six Layer Assessment Approach Inlay Video video Onsite Work Offsite Work Review with Client Facility Tour Walk facility with local contact looking for physical security risks Document findings with Collect all documentation, drawings, recommendations and send Interviews and configuration files to validate current architecture and make 1st draft report to client Meet with ICS engineers and network admins for system discovery recommended changes to system architecture Documentation Collect network device config files, network Prioritize and Review draft report with the drawings, and other system documentation rank security client and receive client risks suggestions and Passive Analyze router, switch and firewall modifications. Capture network traffic on all core routers / configuration files, network switches captures, host scripts, application review notes, and any scanning log Host Systems files for system vulnerabilities. Take suggestions into Server scripts to extract configurations. Review ICS application security. Create list of security findings. considerations and publish Final Report. Review Final Active Report with client for project Conduct active vulnerability scans of offline test, signoff. backup, or simulation systems Passive onsite Active onsite Offsite analysis Offsite documentation Six Layer Assessment Approach Inlay Video video 1. Physical Security (Fencing, Surveillance, Guards, Gates, Locks) 2. Network Infrastructure (Switches, Routers, Firewalls, 3rd Party Connections, Modems) 3. ICS DMZ (Data Historians, Data Logging, Web Servers) 4. Mission Critical Servers / Workstations (Assessing for Host and Application Vulnerabilities Operating System Security, Application Security) 5. Communications to Field Devices (Clear text transfers, traffic injection, hijacks) 6. Field Devices (PLCs, RTUs, IEDs, Plant Equipment) ICS Vulnerability Assessment Inlay Video video Micro Unit Layer 1 – Physical Security Layer 1 – Physical Security Inlay Video video Layer 1 – Physical Security Inlay Video video Entire team takes facility tour, and all look for potential physical security gaps in areas of access control and monitoring. What steps do we take to go from the outside of the facility to the control room? Refer to any required specific compliance requirements (CFATS, NERC CIP, and others all have physical security requirements). Micro Unit Layer 2 – Network Infrastructure Layer 2 – Network Infrastructure Inlay Video video Layer 2 – Network Infrastructure Collecting network data Inlay Video video Meet with local network and ICS administrators to draw out network (think about the ISA 99 security levels, since report should include re-design recommendations) PASSIVE - Capture network traffic at all core ICS switches and routers both on the inside and outside of the firewall (using TCPDUMP) PASSIVE - Collect running configurations from all network devices ACTIVE - Scan all network segments outside of the ICS LAN that are low-risk to Operations (with NMAP, NESSUS, or similar scanners) ACTIVE - Scan secondary ICS networks, including development or testing systems, as long as the scan can not interrupt primary real-time networks (with NMAP, NESSUS, or similar scanners) Layer 2 – Network Infrastructure Where to Capture Traffic? (PASSIVE) Inlay Video video Outbound traffic to 3rd Parties and Corporate IT ICS DMZ network Control Room HMI switch Field side communications Layer 2 – Network Infrastructure Capturing Network Traffic – Practical tips Inlay Video video Linux/Mac works best, but TCPDUMP can be compiled for Windows Wireshark has been known to drop packets on larger networks. Command line has better kung foo :] Set NIC interface without an IP address so all data packets exposed to the NIC interface will be captured tcpdump -i eth0 -nn -vvv -tttt -XX -C 20 -s 0 -w filename Above syntax chops up PCAP files into 20MB files, which are easier to parse > try this out now using your own laptops Layer 2 – Network Infrastructure Where to Perform Vulnerability Scans? (ACTIVE) Inlay Video video IT networks outside of ICS firewall ICS DMZ network Secondary, Development, or Testing ICS systems NEVER SCAN LIVE PRIMARY ICS NETWORKS Layer 2 – Network Infrastructure ARP-, Port-, and Vulnerability-Scanners Inlay Video video ARP (Address Resolution Vulnerability Scanners can dive deeper into the target device(s) Scanners) on the network Perform software calls to the ARP tables on the bring back the IP address and open ports network to request the MAC address, IP address, and can determine the version of the services running on host names for each device the network knows about. the device CAIN is an example quickly compare the version of the services present on the device with its own database of known vulnerable services Port Scanners NESSUS is an example can perform ARP scans and PING scans can retrieve IP address and host names of devices can probe for the presence of particular TCP and UDP ports and retrieve their status (open, closed, filtered, etc..). can not determine if a device is vulnerable can provide what ports and services are running on the device NMAP is an example Layer 2 – Network Infrastructure Use Active Tools with CAUTION Inlay Video video ICS Vulnerability Assessment Rule: Know Thy Tools Know what your scanning tools actually do, watch them in wireshark Don’t just point-click-shoot… scanners include: NMAP Nessus OpenVAS Bindview RMS (Control Compliance Suite – Symantec) GFI LANguard N.S.S NetIQ Vulnerability Manager List needs to be verified / Retina updated by Jonathan and SNSI Marc Layer 2 – Network Infrastructure Net Flow Interviews Inlay Video video Network Engineer on the VA Team should be able to understand the current ICS and Process Control networks, and how they connect to Corporate IT and other networks. Should be able to draw the current state and recommended changes to the architecture to bring into compliance with applicable standards. Next two slides show sample diagrams for current and recommended changes. Sample Remote SCADA Site Current Network Architecture Inlay Video video Sample Remote SCADA Site Proposed Network Design (based on ISA 99) Inlay Video video Micro Unit Layer 3 and 4 – Testing ICS DMZ and Control Room Host/Application Assessment Layer 3/4 – ICS DMZ and Mission Critical Servers / Workstations Inlay Video video Layer 3/4 – ICS DMZ and Mission Critical Servers / Workstations Inlay Video video Obtain permission to capture current state of: operating system patches installed applications users/groups/permissions local security policy network configuration (all NICs) network connections services/active ports NOTE: Consider automation of this with scripts tuned for the operating system Micro Unit Layer 5 / 6 - Field Devices Layer 3/4 – ICS DMZ and Mission Critical Servers / Workstations Inlay Video video Passively Capture live ICS traffic to analyze offline for clear text transfer Layer 5 / 6 - Field Devices Scanning on a testbench or lab environment Inlay Video video NEVER ACTIVELY SCAN LIVE ICS NETWORKS Perform scans only on a testbench or lab environment Scan the controllers with NMAP (intense all TCP and UDP ports) to fully understand all ports and services exposed on the devices Many have embedded web servers, ftp servers, email servers, and backdoor ports for support (telnet) Knowing all ports and services available on each device is a required part of NERC CIP vulnerability assessments Scan the controllers with NESSUS to determine: How the device behaves when under an active scan If the device is running any vulnerable services Layer 5 / 6 - Field Devices Simulate Adverse Network Conditions Inlay Video video Determine what type of network traffic may have a negative influence on the field controllers Help create recommendations for compensating controls for attacks with high risk / impact on operations Typically test for the ability of the controller to continue to update the HMI screens while under both DoS and malformed payload packets Goal is to log the results that the attack had on the system, and the ability of the system to recover automatically once attack is over Layer 5 / 6 - Field Devices Denial of Service Attacks (DoS) Inlay Video video ping -f -s 60 (target IP Address) ping -f -s 600 (target IP Address) ping -f -s 6000 (target IP Address) ping -f -s 60000 (target IP Address) Layer 5 / 6 - Field Devices Malicious Payload Attacks Inlay Video video isic -s rand -d 192.168.0.1 hping -b -i u1 192.168.0.1 hping -x -d 1000000 –m 16 -i u1 192.168.0.1 hping -Y -i u1 192.168.0.1 hping -X -i u1 -d 6000000 -m 16 192.168.0.1 hping -y -d 65000000 -i u1 192.168.0.1 Micro Unit Offsite Data Analysis Overview  Offsite Data Analysis Inlay Video video Onsite Work Offsite Work Review with Client Facility Tour Walk facility with local contact looking for physical security risks Document findings with Collect all documentation, drawings, recommendations and send Interviews and configuration files to validate current architecture and make 1st draft report to client Meet with ICS engineers and network admins for system discovery recommended changes to system architecture Documentation Collect network device config files, network Prioritize and Review draft report with the drawings, and other system documentation rank security client and receive client risks suggestions and Passive Analyze router, switch and firewall modifications. Capture network traffic on all core routers / configuration files, network switches captures, host scripts, application review notes, and any scanning log Host Systems files for system vulnerabilities. Take suggestions into Server scripts to extract configurations. Review ICS application security. Create list of security findings. considerations and publish Final Report. Review Final Active Report with client for project Conduct active vulnerability scans of offline test, signoff. backup, or simulation systems Passive onsite Active onsite Offsite analysis Offsite documentation Offsite Data Analysis Workload Broken Down by Team Resource Inlay Video video Network Engineer (layers 2 and 5) Analyze network device configuration files Analyze network traffic captures (PCAP Analysis) Analyze and research vulnerabilities detected from active scans ICS Engineer (layers 3,4, and 6) Review output from all Host DUMP scripts for operating system vulnerabilities Analyze the information captured from onsite ICS Application Testing work Report on the results from the Field Controller scans and attack tests Project Manager (layer 1 and help with others as needed) Analyze and report on physical security findings Summarize notes from all onsite interviews Assist with Data Analysis as needed by the Network and ICS engineers Micro Unit Offsite Data Analysis – Network Engineer Offsite Data Analysis Network Engineer > Device Configuration Analysis Inlay Video video Noteworthy Network Device Configuration Checks Review all running config files from switches, routers, and firewalls to ensure Rule of Least Privilege has been used in the configuration. Check to see if network devices are at the latest firewall release available. Check for the ability to configure these devices over the network, if SSH (secure shell) is required when configuring devices, and in critical cases, configuration should only be allowed over serial (console cable) connection. Check for proper SNMP configuration. Check for proper logging for forensic and compliance purposes. Check for consistent time source for all device. If physical access is not restricted to the ports on the switches, then consider recommending port security (sticky MAC addresses). If man-in-the-middle threats exist, consider ARPwatch recommendation. Offsite Data Analysis Network Engineer > Traffic and Protocol Analysis Inlay Video video Noteworthy PCAP Traffic Dump Checks Run.CAP files through several network analysis tools NetworkMiner HoneySNAP Free Packet Analyzer Should be using an eye for detecting: Flow / network connections in and out of ICS Clear text protocols (TELNET, FTP, HTTP, most ICS Protocols) Passwords and ICS Traffic sent in the clear Known malicious protocols Excessive broadcast storms indicative of mis-matched switchport settings Excessive net flows to or from a particular source or destination address Offsite Data Analysis Network Engineer > Traffic and Protocol Analysis Inlay Video video Traffic Analysis report should indicate a protocol breakdown Tcpdstat, ipsumdump, and Netdude can generate similar protocol breakdown analysis Offsite Data Analysis Network Engineer > Traffic and Protocol Analysis Inlay Video video Offsite Data Analysis Network Engineer > Network Traffic Analysis Inlay Video video Conducting the protocol analysis allows to compare the traffic on the wire at the time of the assessment with the applications and services that should be expected. Vulnerable clear-text protocols such as TELNET, and FTP should be further researched. Any known malware protocols such also be further researched. Sorting by the hosts with the most bytes SENT or RECEIVED discloses the devices that are the most “chatty” on the network. Does those devices make sense, or has the device been compromised by a rootkit? Drawing out the communications matrix “by IP address” allows us to see what devices were communicating with other devices at the time of the assessment. Offsite Data Analysis Network Engineer > Vulnerability Scan Analysis Inlay Video video Review the results from all active Vulnerability Scans, and group similar vulnerabilities Compare vulnerability findings at the host level with those detected by the ICS engineer in the host DUMP scripts Remove any false positives from the list of vulnerabilities at both the network and host/application layer Offsite Data Analysis Network Engineer > Summarizing the security findings Inlay Video video Create overall list of security findings based on the analysis of: Network device configurations Traffic dumps (PCAP analysis) Vulnerability Scans If the same vulnerability is found on multiple devices or hosts, then group this under one vulnerability, bust site the “Systems Impacted” For each security finding, research and document a recommended remediation strategy Rank the findings, and generate individual security finding tables for each vulnerability to feed into the Draft Report Offsite Data Analysis Network Engineer > Summarizing the security findings Inlay Video video Security Finding Table (sample) Offsite Data Analysis Network Engineer > Summarizing the security findings Inlay Video video The Network Engineer should also provide the Project Manager with two network diagrams developed through interviews with the client while onsite Current State Network Diagram New Proposed Network Redesign Diagram These two diagrams should be in the Rough Draft report, and any recommended changes to the network design should be backed up with content as to why the new design will mitigate existing and future security vulnerabilities This should include how remote users access the ICS and Process Control systems, along with how data flows from the ICS system out to the Corporate IT network and 3rd parties that require real-time or historical data Offsite Data Analysis Network Engineer > Summarizing the security findings Inlay Video video Micro Unit Offsite Data Analysis – ICS Engineer Offsite Data Analysis ICS Engineer > Host DUMP script Analysis Inlay Video video Noteworthy things to look for in operating system DUMP logs Missing operating system security patches Missing application patches that impact security Excessive installed applications, or existence of applications that may not be job essential (iTunes, Peer-to-Peer file sharing, IM, weird video Codecs, games, etc…) Users/groups/permissions – especially excessive use of Administrator accounts Stale or vendor default accounts Insufficient or weak password change policy Weak or insecure AD / Windows Domain configuration Weak Local / Group Security Policy Network configuration (all NICs) Excessive network connections or connections to unknown devices Excessive or non-essential available services/active ports Connectivity to other systems Offsite Data Analysis ICS Engineer > Field Controller Testing Analysis Inlay Video video Noteworthy things from Field Controller Testing NMAP or Vulnerability Scans results Note the active services and available ports on the controller Note any security vulnerabilities exposed by the scanner Note if simply scanning the controller impairs the operation of the controller DoS Attacks Note how the field controller stands up to DoS (Denial of Service) attacks as they increase from a small byte size to larger byte size Document how the controller recovers when the attack is over Malformed Attacks Note how the field controller stands up to malformed packet attacks as various payloads are altered Document how the controller recovers when the attack is over Summarize vulnerabilities discovered Micro Unit Offsite Data Analysis – Project Manager Offsite Data Analysis Project Manager > Building Draft Report Inlay Video video Analyze physical security findings, rank them, and create security finding tables for each area that needs improvement Obtain finding tables from the Network and SCADA engineering team members, and sort all security findings from all system layers from highest security severity to lowest Compile all findings, interview results, diagrams, and security finding tables into a Rough Draft report Write the Executive Summary for the Rough Draft and send it to the Operational and IT teams for the facility being assessed Micro Unit Review with Client  Review with Client Inlay Video video Onsite Work Offsite Work Review with Client Facility Tour Walk facility with local contact looking for physical security risks Document findings with Collect all documentation, drawings, recommendations and send Interviews and configuration files to validate current architecture and make 1st draft report to client Meet with ICS engineers and network admins for system discovery recommended changes to system architecture Documentation Collect network device config files, network Prioritize and Review draft report with the drawings, and other system documentation rank security client and receive client risks suggestions and Passive Analyze router, switch and firewall modifications. Capture network traffic on all core routers / configuration files, network switches captures, host scripts, application review notes, and any scanning log Host Systems files for system vulnerabilities. Take suggestions into Server scripts to extract configurations. Review ICS application security. Create list of security findings. considerations and publish Final Report. Review Final Active Report with client for project Conduct active vulnerability scans of offline test, signoff. backup, or simulation systems Passive onsite Active onsite Offsite analysis Offsite documentation Review with Client Sample report Inlay Video video Need to add a Screenshot of and link to a sample report  Link to Module 11 Micro Unit Summary Summary Inlay Video video Passed along a time-proven methodology for safely conducting Vulnerability Assessments on ICS and Process Control Systems. Explained three stages: Onsite Data Collection Offsite Data Analysis Developing the Report Deliverable Covered how to safely collect the data onsite from the (6) layers of a typical system: 1. Physical, 2. Network Infrastructure, 3. SCADA DMZ, 4. Control Room Hosts and Applications, 5. SCADA Protocols and Communications, and 6. Field Controllers. Provided typical project timelines, team resources and roles, tasks to be performed at each stage in the project, tips and techniques for onsite and offsite work, and a sample report deliverable