lec 3.pptx

Log Analysis Chapter 2 Log Analysis 2 Lecture Objectiv es 02 log analysis Applications 04 06 log analysis Technique s log analysis Use Cases 01 What is log analysis? 03 How to make log analysis? 05 log analysis Tools 3 1-What is Log Analysis  Log analysis is a branch of data analysis that involves...

Log Analysis Chapter 2 Log Analysis 2 Lecture Objectiv es 02 log analysis Applications 04 06 log analysis Technique s log analysis Use Cases 01 What is log analysis? 03 How to make log analysis? 05 log analysis Tools 3 1-What is Log Analysis  Log analysis is a branch of data analysis that involves the performance of IT infrastructure  through the reviewing and interpretation of logs (log events, audit trail records) generated by network, operating systems, applications, servers, and other hardware and software components.  For Security information and event management SIEM log analysis of companies include handling security issues , troubleshooting app performance anomalies or compliance with regulations to mitigate risks by identifying anomalies, vulnerabilities. 4 1-Examples of logs might include: 1-Sign-in and sign-out requests on a website 2-Transactions made on a currency exchange 3- Calls made to an informational API 4-Various other industry-specific actions 5 2-Why to use Log Analysis : Purpose and Benefits 1-Reduce Problem Diagnosis and Resolution Time Log file analysis allows to detect issues before or as they happen and avoid time waste, unnecessary delays, and additional costs and reduces MTTR (mean time to repair). 2-Reduce Customer Churn By analyzing log files, root cause can be detected of performance and stability issues faster, thus improve users’ experience and reduce customer churn Ex: search for HTTP errors and detect when users doesn’t receive the information they searched for or if their requests are taking too long to load 6 2-Why to use Log Analysis : Purpose and Benefits 3-Improve Resource Usage & Production Infrastructure Costs  Log analysis allows to accurately understand current resource utilization and future resource requirements. ( network bandwidth - CPU cycles - storage capacity )  It also allows to track resource usage and detect where system is struggling so that extra capacity can be added.  It allows to see underutilized or dead assets so infrastructure ca be optimized to improve productivity 7 3-How Does Log Analysis Work? The four main steps to analyze loggin process to the system are:  1-Collect – set up a log collector to gather all the logs across your infrastructure.  2-Centralize and index – ship the logs to a centralized logging platform. When collected to the central location, logs are also normalized to a common format to avoid confusion and ensure uniformity.  Along with indexing, it makes data readily available and searchable for efficient log analysis. 8 3-How Does Log Analysis Work? 3-Search and analyze – user can search for logs matching various patterns, and structures. Ex: Analyzing logs with a specific exception or severity level via reports and dashboards make information available for everyone including people outside the IT department and it’s easier to spot trends or anomalies by looking at graphs or other visual representations of data. 4-Monitor and alert – set up alerts to notify in real-time when certain conditions are not met.  This helps detect what happened, when, where, why, and how it impacted performance, to build appropriate models to avoid risks. Alerts may also have triggers, such 9 as calling a webhook to restart a service. 3-How Does Log Analysis Work? 10 4-Log Analysis Applications The three main applications of log analysis are:  1-Monitoring Logs can be used to monitor the usage of a product or service, often for security reasons. Ex: consider a private web service that allows partners of a company to make requests to their internal databases. By analyzing the logs which are created when making these requests, the company itself can identify malicious usage patterns.  2- Auditing Logs may also be used for auditing purposes, as is especially common in the financial industry. 11 4-Log Analysis Applications Cont.. Ex: Consider a regulated currency exchange that allows users to trade between currencies. If a regulator suspects that the exchange has been incorrectly processing trades, it may request access to the exchange’s logs, so as to see the transaction history.  3- Debugging If a programmer observes that a product or service is malfunctioning, he can refer to the appropriate logs to find reasons why that may be the case. 12 5-Log Analysis Techniques or  1-Normalization Processes Normalization is the process of cleaning logs so that they adhere to the same standards or formats.  Ex: if logs from various sources contain varying datetime formats, they should be normalized before proceeding.  2-Pattern recognition Pattern recognition is the process of identifying patterns in logs, so that individual log entries can be handled appropriately.  Ex: consider platform. the logs collected by an ecommerce  Log entries that refer to users signing in should be 13 separated from log entries that refer to users signing out. 5-Log Analysis Techniques or Processes  3- Classification and tagging  Classification and tagging is another process that involves categorizing individual log entries.  In this case, log entries should be further classified. EX: for example, based on keywords that may be present in the entries themselves.  4-Correlation analysis  It is the process of finding log entries that are correlated.  This refers to identifying which entries pertain to a specific event, or (pertaining to separate events) are correlated.  As in other types of data analysis, identifying correlations 14 is an essential step in drawing meaningful conclusions 5-Log Analysis Techniques or Processes  5-Artificial ignorance  Artificial ignorance is the process of “ignoring” entries which are not useful for analysis.  In web-based applications, artificial ignorance may be used to identify which logs relate to intended usage patterns.  With the help of artificial ignorance, it is possible to significantly reduce the number of logs which must be analyzed, which can speed up automatic analysis processes or even make manual analysis a possibility. 15 6-Most Common Log Analysis Use Cases Here are the most common use cases for log analysis: 1-Respond Better to Data Breaches and Other Cyber Security Incidents  logs provide a fountain of information about attackers, such as IP addresses, client/server requests, HTTP status codes, and more.  Many companies still relying only on basic firewalls or other security software to protect their data against DNS attacks. However, without log analysis, security risks can’t be understand and respond accordingly.  Logs act as a red flag. With security log analysis, suspicious activities can be tracked down and set up 16 thresholds, rules, and parameters to protect system from 6-Most Common Log Analysis Use Cases  logs use artificial intelligence and machine learning to spot patterns and behaviors that would have otherwise flown under the radar.  Furthermore, logs are extremely useful in cyber forensics. In case of an investigation, forensic log analysis can provide the time and place of every event that happened in the network or system. 2-Better System Troubleshooting  One of the most obvious use cases for log analysis is probably in troubleshooting servers, networks, or systems, from application crashes to configuration issues and hardware failure.  Fast troubleshooting helps avoid downtime and to detect 17 and solve critical system errors improving operational 6-Most Common Log Analysis Use Cases 3- Ensure Compliance with Security Policies, Regulations & Audits  Most organizations are subject to government-set standards and industry requirements they need to adhere to guarantee safety and functionality.  The require to analyze log data on a daily basis. Doing so, not only helps to defend against insider and outsider threats but also to demonstrate a willingness to comply with ISO, General Data Protection Regulation (GDPR),  In short, considering the ever-growing complexity of systems and software solutions, log analysis is the only way to ensure policies are followed, and regulations are 18 met. 6-Most Common Log Analysis Use Cases 4-Understand Online User Behavior  It enables to understand the app or webapp’s visitors’ behavior.  It shows not only how many visitors but also allows to re-trace their exact journey and on what pages they spent the most time, what were they doing on your website, why are there changes in the number of visitors, and so on.  With trends and patterns in plain view, it’s easy to spot opportunities like when is the best time to send a newsletter, when to release a new version or launch a product, when to close down your site for maintenance or 19 tests, and much more. 7-Log Analysis Tools  Many log analysis tools used to import, normalize, and process data in terms of paid solutions, and open-source solutions are available. It depend on factors like budget and technology stack, as many of these solutions offer similar functionality. Paid Log Analysis tools 1 2 3 Splunk: free and paid platform aides in all areas of data analysis, including log analysis. Retrace: This popular SaaS (Software as a Service ) solution takes your logs and finds ways to improve app performance. Sumologic: This dedicated log management tool is purpose-built for cloud applications 20 7-Log Analysis Tools Cont.. open-source solutions tools 1 2 3 Graylog: This open-core solution is — once again — a dedicated log management tool. GoAccess: This free offering helps to both visualize and analyze logs. Logz.io: This free offering targets those with cloudbased products. 21