LEC 6.pptx

CYB236 Chapter 5 Network Anomaly Detection Systems References 1. Gupta, N., Jindal, V., & Bedi, P. (2023). A Survey on Intrusion Detection and Prevention Systems. SN Computer Science, 4(5), 439. 2. Momand, A., Jan, S. U., & Ramzan, N. (2023). A Systematic and Comprehensive Survey of Recent Advances...

CYB236 Chapter 5 Network Anomaly Detection Systems References 1. Gupta, N., Jindal, V., & Bedi, P. (2023). A Survey on Intrusion Detection and Prevention Systems. SN Computer Science, 4(5), 439. 2. Momand, A., Jan, S. U., & Ramzan, N. (2023). A Systematic and Comprehensive Survey of Recent Advances in Intrusion Detection Systems Using Machine Learning: Deep Learning, Datasets, and Attack Taxonomy. Journal of Sensors, 2023. 3. Bijone, M. (2016). A survey on secure network: intrusion detection & prevention approaches. American Journal of Information Systems, 4(3), 69-88. lecture 5 Network Anomaly Detection Systems 2 NADS algorithms 02 Lecture Objectives 04 Taxonomy of Anomaly Detection IDS Exchange Systems Format 01 Network Anomaly Detection Systems NADS 03 3 Theoretical Foundation of Detection Network Anomaly Detection Systems  Network Anomaly Detection Systems (NADS) are crucial for identifying and predicting abnormal network behavior, utilizing personal profiles and algorithm generation to maintain network security.  Personal profiles in NADS represent normal network behavior, allowing for anomaly detection. These profiles are created by analyzing and modeling network activities, reducing multidimensional vectors into fewer dimensions for easier identification.  NADS algorithm generation involves developing techniques to detect network anomalies using genetic algorithms, fuzzy logic, and machine learning to design effective methods. DR.ASMAA HATEM RASHID 4 Network Anomaly Detection Systems: Algorithms 1) Genetic Algorithm and Fuzzy Logic: The study integrates genetic algorithm and fuzzy logic techniques to create an expert system capable of predicting and detecting network anomalies, enhancing its ability to handle complex network environments. 2) Machine learning: It techniques are utilized in network anomaly detection, enabling the creation of non-parametric and adaptive algorithms that can learn from network characteristics and adapt over time. 3) Self-Attention Mechanism: Research on deep-learning-based traffic anomaly detection models incorporates self-attention mechanisms for improved performance, focusing on identifying key network traffic data features to enhance anomaly detection. DR.ASMAA HATEM RASHID 5 Theoretical Foundation of Detection  Misuse detection and anomaly detection rely on statistical models of normal and intrusion classes. Two approaches are manual definition and machine learning. Manual definition is time-consuming and expensive, while machine learning techniques allow for less human intervention in building and maintaining anomaly detection systems (ADS) for nextgeneration IDSs due to growing complexity and attack types. DR.ASMAA HATEM RASHID 6 Taxonomy of Anomaly Detection Systems  Machine learning techniques for intrusion detection use a training data set with attributes and labels, which determine the applicability of anomaly detection techniques. Labels are typically binary. Accurate labeling is expensive, so operating modes are defined based on the availability of labels for anomaly detection techniques. er vis e Su p v is S sup emierv is e d em rc ing fo i n ar n Re t le en d pe r u s Un ed By categorizing the techniques, it becomes easier to compare and evaluate them based on their effectiveness and suitability for different applications. DR.ASMAA HATEM RASHID 7 Taxonomy of Anomaly Detection Systems: Supervised anomaly detection  Supervised anomaly detection is a method that uses a labeled dataset to construct a predictive model for future data points.  It uses algorithms like supervised neural networks, support vector machine learning, and k-nearest neighbors classifier to learn normal behavior characteristics.  However, this approach is rarely used due to data unavailability and unbalanced classes. DR.ASMAA HATEM RASHID 8 Taxonomy of Anomaly Detection Systems: Unsupervised Anomaly Detection  Unsupervised anomaly detection is a method that uses an unlabeled dataset to identify anomalies based on deviations from normal behavior.  The most commonly used algorithms for this purpose are clustering algorithms, density-based algorithms, and distancebased algorithms.  It is commonly used in fraud detection, network intrusion detection, and system health monitoring, as the model is trained without prior knowledge of the labels. DR.ASMAA HATEM RASHID 9 Taxonomy of Anomaly Detection Systems: Unsupervised Anomaly Detection  Anomaly detection methods are classified by their reporting methods, typically using scores, binary labels, or labels to report anomalies. 1) Scores: Anomaly detectors assign a numeric score to test instances, allowing analysts to rank malicious activities, set thresholds, and select significant ones. Bayesian networks like Naive Bayes provide calculated probabilities for administrators. 2) Binary Labels: Anomaly detection techniques like Decision Trees use binary labels to classify test instances as either anomalous or normal, a special case of labeling techniques. 3) Labels: Anomaly detection techniques assign labels to test instances, typically normal traffic. However, there are numerous labels for anomalies, such as normal, DoS, Probe, U2R, and R2L. Non-scoring learners like Decision Tress can apply these labels, provided enough samples are available. Anomaly detection systems can also be classified based on machine learning techniques. DR.ASMAA HATEM RASHID 10 Taxonomy of Anomaly Detection Systems: Semi-supervised Anomaly Detection  Semi-supervised anomaly detection is a method that uses both labeled and unlabeled data to create a predictive model for future data points.  It uses algorithms like semi-supervised neural networks, self- training, and co-training to identify anomalies in limited labeled data, making it useful in computer vision, natural language processing, and speech recognition. DR.ASMAA HATEM RASHID 11 Taxonomy of Anomaly Detection Systems: Reinforcement learning  Reinforcement learning is a type of machine learning training method that can be used for anomaly detection.  In reinforcement learning, an agent is situated in an environment with clear parameters defining beneficial activity and non-beneficial activity and an overarching endgame to reach.  The agent's objective is to learn a policy that guides its decision-making process to achieve the highest possible reward over time.  Reinforcement learning models have been applied to anomaly detection in various fields, including algorithmic trading and asset integrity management.  Reinforcement learning can be useful for anomaly detection in situations where the environment is complex and dynamic, and the agent needs to learn through trial-and-error interactions with its environment DR.ASMAA HATEM RASHID 12 Network Anomaly Detection Systems: Statistical methodologies  Statistical methodologies are extensively employed in Network Anomaly Detection Systems (NADS) to identify and predict abnormal behavior in computer networks, employing various techniques for anomaly detection. 1) Z-score: Z-score is a statistical measure used for anomaly detection, indicating the distance of a data point from the mean, measured in standard deviations. 2) Modified Z-score: Modified Z-score is an extension of the Z-score method that is useful for detecting outliers in non-Gaussian distributions. 3) Density-based algorithms are statistical methods used for anomaly detection, determining outliers based on a data point's deviation from a specific density threshold. 4) Parametric and non-parametric techniques are used in statistical anomaly detection models, with parametric techniques assuming data distribution, and non-parametric techniques making no assumptions. 5) Semi-supervised statistical approach: The proposed semi-supervised statistical approach for anomaly detection involves creating a probabilistic model of network normal behavior and detecting deviations from this model.13 DR.ASMAA HATEM RASHID Network Anomaly Detection Systems: Correlation Methods  Correlation methods are commonly used in Network Anomaly Detection Systems (NADS) to identify and predict abnormal behavior in computer networks. Here are some correlation-based techniques used for anomaly detection in NADS: 1) Correlation-based feature selection is a method used for anomaly detection by calculating the correlation between each feature and the target variable, selecting the most relevant features. 2) Correlation-based anomaly detection is a method that analyzes the correlation between different network variables, identifying anomalies that may not be immediately apparent when analyzing individual variables. 3) Correlation-based classification classifies network traffic as normal or anomalous by analyzing the correlation between different network features, using a deep neural network classifier. 4) Data correlation method its proposed data correlation method uses regression relations to identify anomalies by analyzing the specific properties of normal profiles and detecting changes in these properties. Correlation methods in NADS aid in identifying anomalies by analyzing the correlation between network variables, enhancing the accuracy of anomaly detection in complex network environments. DR.ASMAA HATEM RASHID 14 Network Anomaly Detection Systems: Correlation Methods  Logic methodologies are used in Network Anomaly Detection Systems (NADS) to identify and predict abnormal behavior in computer networks. Here are some logic-based techniques used for anomaly detection in NADS: DR.ASMAA HATEM RASHID Fuzzy logic Expert system s Rulebased system s Artifici al neural networ ks 15 Network Anomaly Detection Systems: Correlation Methods 1) Fuzzy logic is a method used to determine if malicious activity is occurring on a network by assigning membership degrees to network events based on their similarity to predefined rules. 2) Expert systems, computer programs that mimic human decision-making abilities, are utilized in NADS to identify and predict network anomalies using predefined rules and knowledge. 3) Rule-based systems are a logic-based method used in NADS to detect anomalies by defining rules describing normal network behavior and detecting deviations from these rules. 4) Artificial neural networks are machine learning algorithms utilized for anomaly detection in NADS, capable of learning from network characteristics and adapting to changes over time. DR.ASMAA HATEM RASHID 16 Network Anomaly Detection Systems: Fuzzy Logic  Fuzzy Logic is a logic-based methodology used in Network Anomaly Detection Systems (NADS) to identify and predict abnormal behavior in computer networks. Here are some key points about Fuzzy Logic in NADS: DR.ASMAA HATEM RASHID 1. Fuzzy Logic is a network assessment technique that assigns membership degrees to network events based on their similarity to predefined rules. 2. Fuzzy Logic uses the Exponentially Weighted Moving Average (EWMA) method to calculate thresholds for anomalies in a time interval. 3. Fuzzy Logic, when combined with genetic algorithms, can accurately create a network profile using recent data in an environment with uncertainties and imprecisions. 4. Fuzzy Logic can be used for unsupervised anomaly detection, allowing the detection of unexpected activity of users or network equipment 17 Network Anomaly Detection Systems: Artificial Intelligence  Artificial Intelligence (AI) is a technology that is increasingly being used in Network Anomaly Detection Systems (NADS) to identify and predict abnormal behavior in computer networks. Here are some ways in which AI is used in NADS: 1) Machine learning, a subset of AI, is utilized in NADS to analyze network traffic and identify anomalies through the training of algorithms and make predictions or decisions. 2) Artificial neural networks (ANNs) are machine learning algorithms used for unsupervised anomaly detection in NADS, trained on large unlabeled datasets to identify intricate patterns. 3) Fuzzy logic is a logic-based methodology used in NADS to determine if malicious activity is occurring on a network by assigning membership degrees based on predefined rules. 4) Automation, AI-driven anomaly detection algorithms can automate dataset analysis, fine-tune normal behavior parameters, and identify pattern breaches, reducing human analyst workload and improving anomaly detection accuracy. 5) Real-time analysis, AI solutions can interpret data activity in real-time, allowing for immediate detection of anomalies and quick response times DR.ASMAA HATEM RASHID 18 Network Anomaly Detection Systems: Filtering Algorithms  Filtering algorithms are commonly used in Network Anomaly Detection Systems (NADS) to identify and predict abnormal behavior in computer networks. Here are some filtering-based techniques used for anomaly detection in NADS: 1. Filter-ary-Sketch, is a filtering algorithm used in NADS for detecting traffic anomalies, identifying malicious buckets when anomalies are detected over the recorded traffic. 2. Kohonen Maps and Variable-order Markov Chains, its proposes a Linux anomaly detection and network filtering system using Kohonen Maps and Variable-order Markov Chains, clustering network traffic and modeling traffic patterns and detect anomalies. 3. A hybrid system, for anomaly detection in social networks uses multiple-level dataset filtering and machine learning algorithms, combining three algorithms for effective anomaly detection. 4. Combining Filtering and Statistical Methods, its proposes a large-scale network anomaly detection approach that combines filtering algorithms to reduce data volume, and statistical methods to detect anomalies within the filtered data. DR.ASMAA HATEM RASHID 19 Network Anomaly Detection Systems: Neural Networks  Neural networks are a type of artificial intelligence (AI) that are increasingly being used in Network Anomaly Detection Systems (NADS) to identify and predict abnormal behavior in computer networks. Here are some ways in which neural networks are used in NADS: 1) Deep neural networks (DNNs) are neural networks used for anomaly detection in NADS, learning complex patterns from rare traffic anomalies, and aiding in the detection and classification of network attacks. 2) Long-short term memory (LSTM) networks are a powerful neural network used for anomaly detection in time-series data, outperforming advanced algorithms and feed-forward neural networks. 3) Unsupervised learning is a technique that uses neural networks to detect anomalies in large unlabeled datasets, allowing them to identify patterns independently. 4) Feature extraction, Neural networks can be used for feature extraction in NADS. Feature extraction involves identifying the most relevant features in a dataset for anomaly detection DR.ASMAA HATEM RASHID 20 THANKS! Best Regards!