stats notes.pdf

STATISTICS FOR DATA SCIENCE What is Data Science? Data Science is the science of analysing raw data using statistics and machine learning techniques with the purpose of drawing insights from the data. Data Science is used in many industries to allow them to make better business decisions, a...

STATISTICS FOR DATA SCIENCE What is Data Science? Data Science is the science of analysing raw data using statistics and machine learning techniques with the purpose of drawing insights from the data. Data Science is used in many industries to allow them to make better business decisions, and in the sciences to test models or theories. This requires a process of inspecting, cleaning, transforming, modelling, analyzing, and interpreting raw data. Important Disciplines Under Data Science The Constant Evolution Of Data Science Industry Statistician Data Mining Specialist Predictive Analytics Data Scientist Specialist  Data science mainly needed for: Better decision making (Whether A or B?) Predictive Analysis (What will happen next?) Pattern Discovery (Is there any hidden information in the data?) So data science is about Asking right questions and exploring the data Modelling the data using various algorithms Finally, communicating and visualising results Examples: 1) Self-driving cars 2) Airlines 3) Logistic companies like FedEx  The Various Data Science Disciplines Not everyone in the field of Data Science is a Data Scientist! Data Engineer Big Data Engineers Data Analyst Data Engineers are software The set of engineers handle A Data Analyst is someone who engineers who handle the Data Warehousing process, by processes and does statistical design, building, integration of running the analysis on data to discover data from various data sources Extract-Transform-Load (ETL) possible patterns, trends and and also manage them. procedure on data. They are also also appropriately communicate known as Big Data Engineers. the insights gotten for proper Big Data is data that contains understanding. greater variety arriving in Data Analysts are sometimes increasing volumes and with called “Junior Data Scientists” ever-higher velocity ~ Gartner or “Data Scientists in Training”  Machine Learning Engineer Data Visualization Engineer Data Scientist A Machine Learning (ML) Engineer This is someone that tells visually A Data Scientist is an analytical data is a software engineer that specializes stunning stories with data, create expert who has the technical skills to solve in making data products work in dynamic data visualizations to help complex problems and the curiosity to explore what problems needs to be solved. production. businesses/customers make meaningful decisions in an Data Scientists apply Statistics, Machine They are involved in software Learning, and analytical approaches to architecture and design; they interactive format. solve critical business problems. understand and carryout practices They basically collaborate with Data A Data Scientist is also known as a like A/B testing (A/B testing is a user Analysts and Data Scientists to make mathematician, a statistician, a experience research methodology visualizations which effectively computer programmer and an analyst communicates the insights gotten equipped with a diverse and from data to the business. wide-ranging skill set, balancing knowledge in different computer programming languages with advanced experience in data mining and visualization.  Difference between Analysis and Analytics Analysis Analytics Analytics generally refers to the future instead of Consider you have a huge data set containing data explaining past events. of various types. It explores potential future ones. Instead of tackling the entire dataset and running Analytics is essentially the application of logical and the risk of becoming overwhelmed, you separated computational reasoning to the component parts into easier to digest chunks and study them obtained in an analysis and in doing this you are individually and examine how they relate to other looking for patterns in exploring what you can do with parts and that's analysis. them in the future. One important thing to remember however is that Here analytics branches off into two areas. you perform analyses on things that have Qualitative analytics already happened in the past such as using an This is using your intuition and experience in analysis to explain how a story ended the way it conjunction with the analysis to plan your next did or how there was a decrease in the cells last business move summer. Quantitative analytics All this means that we do analyses to explain how This is applying formulas and algorithms to numbers and or why something happened you have gathered from your analysis. Business Analytics, Data Analytics, Data Science: An Introduction ❑ Qualitative analytics. - This includes working with tools that help predict future behaviour. - Therefore must be placed on the right. - In essence what we have now is qualitative Analytics which belongs to the area of business analytics. ❑ Sales Forecasting though is a future oriented activity so we can move it to the right of the black line but not too much. It must still be long on the sphere of business. So it must be in the area where business analytics and data intersect. ❑ Data Science - The most sparkly of them all is data science. - Data science is a discipline reliant on data availability while business analytics does not completely rely on data. - However, data science incorporates part of data analytics mostly the part that uses complex mathematical statistical and programming tools. - Consequently, this green rectangle representing data science on our diagram will not overlap with data analytics completely but it will reach a point beyond the area of business analytics. - An example of a discipline that belongs to the field of data science and is considered data analytics but not business analytics is the oil and gas industry and the optimization of drilling operations (It aims to optimize weight on bit, bit rotation for obtaining maximum drilling rate as well as minimizing drilling cost). - This is a perfect fit for this sub area data science can be used to improve the accuracy of predictions based on data extracted from various activities typical for drilling efficiency. - Something that involves data analytics but neither data science nor business analytics can be digital signal processing. - Digital signal is used to represent data in the form of discrete values which is an example of numeric data. - Therefore data analytics can be applied to digital signal in order to produce a higher quality signal and that’s what digital signal processing is all about. - The business intelligence or BI is the process of analysing and reporting historical business data after reports and dashboards have been prepared. - They can be used to make an informed strategic and tactical business decisions by end users such as the general manager. - Business intelligence aims to explain past events using business data. - It must go on the left of the timeline as it deals only with past events and it must sit within the data science rectangle as a subfield business intelligence fits comfortably within data science because it is the preliminary step of predictive analytics. - First you must analyse past data and extract useful insights using these inferences will allow you to create appropriate models that could predict the future of your business accurately. - As with reporting and creating dashboards these are precisely what business intelligence is all about. So we will neatly place these two into the orange rectangle. ❑ Machine Learning - The ability of machines to predict outcomes without being explicitly programmed to do so is regarded as machine learning. - Expanding on this is about creating and implementing algorithms that let machines receive data and use this data to make predictions analyse patterns and give recommendations on their own. - Machine learning cannot be implemented without data. Hence it should stay within Data analytics completely. - By definition it is about simulating human knowledge and decision making with computers. - We as humans have only managed to reach AI through machine learning the discipline we just talked about and as the data scientists we are interested in how tools from machine learning can help us improve the accuracy of our estimations. - AI is beyond our expertise - Artificial intelligence is intelligence demonstrated by machines, as opposed to the natural intelligence displayed by humans or animals. - The client retention(process of engaging existing customers to continue buying products or services from your business) and acquisition(process of gaining new customers) are two typical business activities where machine learning is involved. It helps develop models that predict what a client's next purchase would be. - For example since we could say data analytics and data science are applied in client retention and acquisition as well we can leave this term right over here. - ML can be applied to fraud prevention as another example we can feed a machine learning algorithm with prior fraudulent activity data. It will find patterns which the human brain is incapable of seeing. - Having a model which can detect such transactions or operations in real time it has helped the financial system prevent a huge amount of fraudulent activity. - When talking AI and ML usually speech and image recognition are among the most popular examples as they are already being implemented in products like Siri, Cortana, Google's assistant and more impressively self-driving cars. - Finally an example that is considered artificial intelligence but not machine learning is symbolic reasoning. - It is based on the high level human readable representations of problems in logic.  A Breakdown of Data Science  The step by step comparison between the terms and buzzwords related to each discipline - DATA - Big data is a term reserved for extremely large data and it is not just - Data is defined as information stored in a humongous in terms of volume. This data could be in various formats. digital format which can then be used as a It can be structured, Semi structure or unstructured. Big data is just base for performing analyses and decision that big. making. - You will also often see it characterized by the letter V as in big data. - As you can see there are two types of data. They may include the vision you have about big data, the value Big - Traditional data and big data: Dealing with Data carries, the visualization tools you use or the variability and the data is the first step when solving business consistency of big data and so on. problems or researching. - However, the following are probably the most important criteria. You - Traditional data is the data in the form of must remember volume as we already said. Big Data needs a tables containing numeric or text values data whopping amount of memory space typically distributed between that is structured and stored in databases minicomputers. Its size is measured in terabytes, Peta bytes, and even which can be managed from one computer. exabytes variety. - Here we are not talking just about numbers and text. Big data often implies dealing with images, audio files, mobile data and others. - Velocity when working with big data. One's goal is to make extracting patterns from it as quickly as possible. The progress that has been done in this area is remarkable outputs from huge data sets can be retrieved in real time. - Data science is a broad subject. It's - Business Intelligence is the discipline includes technology driven tools an interdisciplinary field that involved in the process of analysing, understanding, and reporting combines statistical, mathematical, available past data. This will result in having reports or dashboards and will programming, problem solving and help you on your way to making an informed strategic and tactical business data management tools. decisions. - We have divided data science into - You can extract insights and ideas about your business that will help to grow three segments business intelligence, and give you an edge of your competitors giving you added stability. traditional methods and machine - Business intelligence means understanding- learning. how your sales grew and why did competitors lose market share, Was there an increase in the price of your products or did you sell a mix of more expensive products? How did your profitability margins behave in the same time frame of a previous year? Were there client accounts that were more profitable? - This is what BI is all about understanding past business performance in order to improve future performance. - Once your BI-reports and dashboards are completed and presented it's time to apply one of two types of data science. - Traditional methods according to our - The last column we will be discussing with framework are a set of methods that are machine learning in contrast to traditional derived mainly from statistics and are methods. adapted for business. - The responsibility is left for the machine through - There is no denying that these conventional mathematics. data science tools are applicable today. They - A significant amount of computer power in are perfect for forecasting future applying AI the machine is given the ability to performance with great accuracy. predict outcomes from data without being - Regression analysis, cluster analysis and explicitly programmed to smell is all about factor analysis all of which are prime creating algorithms that let machines receive examples of traditional methods. data perform calculations and apply statistical analysis in order to make predictions with unprecedented accuracy. The Benefits of Each Discipline There are two types of data. Traditional and big data. Data driven decisions require well organized and relevant raw data stored in a digital format which can be processed and transformed into meaningful and useful information. It is the material on which you base your analysis. Without data, a decision maker wouldn't be able to test their decisions and ensure they have taken the right course of action. The data you have describes what happened in the past. It is the job of the business intelligence analyst to study the numbers and explain where and why some things went well and others not so well. Having the business context in mind the business intelligence analyst will present the data in the form of reports and dashboards. What else is needed once the patterns have been interpreted. You can forecast potential future outcomes. The application of any term related to the columns traditional methods or machine learning can be said to belong to the field of predictive analytics. There is a difference between the two. Traditional methods relate to traditional data. They were designed prior to the existence of big data where the technology simply wasn't as advanced as it is today. They involve applying statistical approaches to create predictive models. If you want to dig deeper however or tackle huge amounts of big data utilizing unconventional methods or AI then you can predict behaviour in unprecedented ways using machine learning techniques and tools. Both techniques are useful for different purposes. Traditional methods are better suited for traditional data while machine learning will have better results when it comes to tackling big data.  Techniques for Working with Traditional Data Data The gathering of raw data is Example would be the use of surveys asking people to rate how much they Collection referred to as data collection like or dislike a product or experience on a scale of 1 to 10 Preprocessing is a group of operations that will For example Customer has entered age as 942 or name basically convert your raw data into a format that is Preprocessing as UK then off course this entries are invalid which more understandable and hence useful for further you need to correct before further processing. processing. One technique is class One such category is numerical. For The other label is categorical. Here labelling. This involves example if you are storing the number you are dealing with information Class labelling the data point to of goods sold daily t. These are that cannot have mathematical Labelling the correct data type or numbers which can be manipulated manipulations. For example a arranging data by category. such as the average number of goods person's profession or place of sold per day or month. birth This can come in various forms. Say you are provided with a data set Data Cleansing The goal of data cleansing is to containing the US states and a quarter of the names are misspelled in this deal with inconsistent data. situation. Certain techniques must be performed to correct these mistakes Data cleansing and dealing with missing values are Missing Missing values are another thing you'll problems that must be solved before you can process the Values have to deal with. data further Case Shuffling Database Balancing. Specific Visualisa E R Diagram tion Relational Scema  Real Life Examples of Traditional Data Consider basic Customer data as example the difference between a numerical and categorical variable. The first column shows the id of the different customers. These numbers however cannot be manipulated. Calculating an average ID is not something that would give you any sort of useful information. This means that even though they are numbers they hold no numerical value and therefore representing categorical data. Now focus on the last column. This shows how many times that customers filed a complaint. These numbers are easily manipulated. Adding them all together to give a total number of complaints is useful information. Therefore they are numerical data. Another example we can look at is daily historical stock price data There's a column containing the dates of the observations which is considered categorical data and a column containing the stock prices which is numerical data.  Techniques for Working with Big Data Some as traditional data preprocessing can also be implemented on big data is essential to help organize the data before doing analyses or making predictions as is grouping the data into classes or categories. While working with Big Data things can get a little more complex. As you have much more variety beyond the simple distinction of numerical and categorical data. Examples of big data can be text data, digital image data, digital video data, digital audio data and more. Consequently, with a larger amount of data types comes a wider range of data cleansing methods. There are techniques that verify that a digital image observation is ready for processing and specific approaches exists that can ensure the audio quality of your file is adequate to proceed. So what about dealing with missing values. This step is a crucial one as big data has big missing values which is a big problem to exemplify. Text data mining represents the process of deriving valuable unstructured data from a text. Consider you may have a database which has stored information from academic papers about marketing expenditure. It may contain information from academic papers, blog, articles, online platforms, private Excel files and more. This means you will need to extract marketing expenditure information from many sources. This technique can find the information you need without much of a problem. Data masking If you want to maintain a credible business or governmental activity you must preserve confidential information. However when personal information is shared online it doesn't mean that it can't be touched or used for analysis. Instead you must apply some data masking techniques so you can analyse the information without compromising private details like data shuffling. Masking can be quite complex. It conceals the original data with random and false data allowing you to conduct analysis and keep all confidential information in a secure place. An example of applying data masking to big data is through what we called confidentiality preserving data mining techniques.  Real Life Examples of Big Data - Facebook keeps track of its users names, personal data, photos, videos, recorded messages and so on. This means that their data has a lot of variety in with over 2 billion users worldwide. The volume of data stored on their servers is tremendous. - Facebook requires real time reporting of the aggregated anonymised voice of its users and it applies many analytical tools for its mobile applications. - This means the company is investing in boosting its real time data processing powers increasing the velocity of its data set. - Lets take financial trading data for example what happens when we record the stock price every five seconds or every single second. - We get a data set that is incredibly voluminous requiring significantly more memory disk space in various techniques to extract meaningful information from it. Data like this would also be considered big data.  Business Intelligence (BI) Techniques Let's assume your data has been pre-processed and is ready for analysis. It is beautifully organized. This means you are ready to enter the realm of business intelligence. The job of a business intelligence analyst requires her to understand the essence of a business and strengthen that business through the power of data. So here we have techniques to measure business performance. However no A measure is the A metric refers to a value KPI Measure Metric Quantification Collecting observations In a real business where the mathematical accumulation of that derives from the number of observations is From above diagram, manipulations can be you can observe applied to these observations to show measures you obtain and significantly larger you can derive some information. aims at gauging business thousands of metrics where we variables such as observations. What we cant keep track of all possible sales volume (marked must do is quantify that For example, if you performance or progress to metrics we can extract from a as blue colour dots) information. total the revenues of compare. data set. Quantification is the all three months to or new customers process of representing If a measure is related to What you need to do is choose who have enrolled in obtain the value of something like simple the metrics that are tightly observations as numbers. your web site $350 that would be a descriptive statistics of past aligned with your business Consider your revenues measure of the objectives. These metrics are (marked as red colour from new customers for performance a metric has a called K.P.I’s, Key Performance dots). January, February and revenue of the first business meaning attached. Indicators. March were 100, 120 and quarter of that year. Key because they are related to Each monthly E.g. If you estimate the $130 respectively while Similarly add together your main business goals. revenue or each the corresponding average quarterly revenue Performance because they show customer is the number of new per new customer which number of new how successfully you have considered a single customers for the same customers for the equals 350 divided by 50 performed within a specified time observation. three months are 10, 15 same period and you that is $7. This is a metric. frame and indicators because and 25. have another their values or metrics that measure indicate something related to your business performance.  Real Life Examples of Business Intelligence - BI can be used for price optimization, hotels use price optimization very effectively by raising the price of a room at periods when many people want to visit the hotel. And by reducing it to attract visitors when demand is low they can greatly increase their profits in order to competently apply such a strategy. - They must extract the relevant information in real time and compare it with historical BI allows you to adjust your strategy to pass data. As soon as it is available. - Another application of business intelligence is enhancing inventory management over and undersupply can cause problems in a business. - However, implementing effective inventory management means supplying enough stock to meet demand with the minimal amount of waste and cost to do this well. - You can perform an in-depth analysis of past sales transactions for the purpose of identifying seasonality patterns and the times of the year with the highest cell’s. - Additionally, you could track your inventory to identify the months of which you have over or understocked a detailed analysis can even pinpoint the day or time of day were the need for a good is highest if done right business intelligence will help to efficiently manage your shipment logistics and in turn reduce costs and increase profit. - So once prepared the BMI reports and dashboards are prepared and the executives have extracted insights about the business what do you do with the information you use it to predict some future values as accurately as possible. That's why at this stage you stop dealing with analysis and start applying analytics more precisely predictive analytics. - We separate predictive analytics into two branches traditional methods which comprises classical statistical methods for forecasting and machine learning. Techniques for Working with Traditional Methods In this dataset we have house prices in dollars while the other house sizes 1) Regression measured in square feet. In business statistics, a Every row on the data table is an observation and each can be plotted on this regression is a model graph as a dot the house size is measured along the horizontal line in its price. used for quantifying On the vertical line the further to the right an observation is the larger the house size and further up the higher the price so once we've plotted all 20 causal relationships observations from our dataset our graph will appear like this. among the different The thing is there is a straight line(red) called a regression line that goes variables included in through these dots while being as close as it can be to all of them simultaneously. your analysis. Now imagine we drew another line(green). If you observe altogether dots are closer to the first red line than the second green one. This means that it more accurately represents the distribution of the observations. So in this case if y signifies the house price then B represents a coefficient A) Linear Regression which we multiply by x the house size. y = Bx So the equation professionals work with is Y equals B times X and they use the graph as visual support.  B) Nonlinear Regression – Logistic Regression - A logistic regression is a common example of a nonlinear model. - In this case unlike the house prices example the values on the vertical line won't be arbitrary integers. - They'll be ones or zeros only such a model's useful during a decision-making process. - Companies apply logistic regression algorithms to filter job candidates during their screening process. - If the algorithm estimates the probability that a prospective candidate will perform well and the company is above 50 percent it would predict one or a successful application. - Otherwise it will predict zero therefore the nonlinear nature of the logistic regression is nicely summarized by its Graph very different from the linear regression.  2) Cluster Analysis - Imagine they are derived from research on German house prices. Hence they are dispersed differently. - When the data is divided into a few groups called clusters you can apply cluster analysis. - This is another technique that will take into account that certain observations exhibit similar house sizes and prices. - For instance this cluster of observations denotes small houses but with a high price. This could be typical for houses in the city centre. - The second cluster could represent houses that are far from the city because they are quite big but cost less. - Finally, the last cluster concerns houses that are probably not in the city centre but are still in nice neighbourhoods. They are big and cost a lot. - Noticing that your data can be clustered is important so you can improve your further analysis. In our example clustering allowed us to conclude that location is a significant factor when pricing a house.  3) Factor Analysis What about a more complicated study where you consider explanatory variables apart from house size. You might have quantified the location, number of rooms, years of construction and so on which can all affect house price then when thinking about the mathematical expression corresponding to the regression model you won't just have one explanatory variable X you will have mini x1, x2, x3 and so on. Note that an explanatory variable can also be called a regress or an independent variable or a predictor variable. Imagine analysing a survey that consists of 100 questions performing any analysis on 100 different variables is tough. This means you are variables starting from x1 and going all the way up to x100. The good thing is that often different questions are measuring the same issue and this is where factor analysis comes in Assume your survey contained this question on a scale from 1 to 5. How much do you agree with the following statements. Survey: 1.I like animals 2. I care about animals. 3. I am against animal cruelty. People are likely to respond consistently to these three questions. That is however marks five to the first question does the same for the second and third questions as well. In other words, if you strongly agree with one of these three statements you won't disagree with the other two right. With factor analysis we can combine all the questions into general attitude towards animals. So instead of three variables we now have one in a similar manner. You can reduce the dimensionality of the problem from 100 variables to 10 which can be used for a regression that will deliver a more accurate prediction. To sum up we can say that clustering is about grouping observations together and factor analysis is about grouping explanatory variables together.  4) Time Series Analysis You will use this technique especially if you are working in economics or finance in these fields you will have to follow the development of certain values over time such as stock prices or sales volume you can associate time series with plotting values against time. Time will always be on the horizontal line as time is independent of any other variable therefore such a graph can end up depicting a few lines that illustrate the behaviour of your stocks over time. So when you study the visualization you can spot which stock performed well and which did not. We must admit there is a vast variety of methods that professionals can choose from.  Real Life Examples of Traditional Methods - Imagine you are the head of the user experience department of a web site selling goods on a global scale which we often abbreviate as UX. - So what is your goal as head of UX to maximize user satisfaction right. - Assume you have already designed and implemented a survey that measures the attitude of your customers towards the latest global product. - You have launched the graph where you plot your observations will likely appear in the following way when the data is concentrated in such a way. - You should cluster the observations. - Remember So when you perform cluster analysis you will find that each cluster represents a different continent. - This group may refer to the responses gathered from Asia or Europe or South America or from North America. - Once you realize there are four distinct groups it makes sense to run four separate tests. Obviously, the difference between clusters is too great for us to make a general conclusion. - Asians may enjoy using your web site one way while Europeans in another. Thus, it would be sensible to adjust your strategy for each of these groups individually. - Another noteworthy example we can give you is forecasting sales volume every business and financial company does this. - So, which traditional statistical technique that we discussed would fit the picture here. Time series analysis it is say this was your data until a certain date. What will happen next? How should you expect the cells to be for the year ahead? Will their volume increase or decrease? - Several types of mathematical and statistical models allow you to run multiple simulations which could provide you with future scenarios based on these scenarios. You can make better predictions and implement adequate strategies awesome job people. You are already acquainted with many of the data science essential terms but not all.  Machine Learning (ML) Techniques - The core of machine learning is creating an algorithm which a computer then uses to find a model that fits the data as best as possible and makes very accurate predictions based on that and how is that different from conventional methods. We provided with algorithms which give the machine directions on how to learn on its own. - A machine learning algorithm is like a trial-and-error process. Each consecutive trial is at least as good as the previous one. - Technically speaking there are four ingredients data, model, objective function and optimization algorithm. - Example. Imagine a robot holding a bow. We want to find the best way to use that bow to fire accurately. In other words the usage of the bow is our model, the best way to learn archery is to train right. We train by taking different arrows and trying to hit the target. So, the quiver of arrows will be or data or more precisely the data that the robot will use for training. - They are all arrows but they have their subtleties. There are straight ones, crooked ones, light ones, heavy ones. So we can safely say the arrows represent different data values. - We said the robot will be firing at a target. In machine learning or at least in the most common type supervised learning, we know what we are aiming for and we call it a target. - The objective function will calculate how far from the target the robot shots were on average. - Here comes the fourth ingredient the optimization algorithm. - It steps on the findings of the objective function and consists of the mechanics that will improve the robot's archery skills somehow. It's posture the way it holds the bow how strong it pulls the bowstring etc. Then the robot will take the exact same data or arrows and fire them once again with its adjusted posture. - This time the shots will be on average closer to the centre of the target. Normally the improvement will be almost unnoticeable. This entire process could have been hundreds or thousands of times until the robot finds the optimal way to fire this set of arrows and hit the centre every single time. - Nevertheless, it is important to remember that while training you won't provide the robot with a set of rules that is you won't have programmed a set of instructions like place the arrow in the middle of the bow pull the bow string and so on. - Instead, you will have given the machine a final goal to place the arrow in the centre of the target. - So you don't care if it places the arrow in the middle or in the bottom of the bow as long as it hits the target. - Another important thing is that it won't learn to shoot well right away but after a hundred thousand tries it may have learned how to be the best archer out there. - Now there can be infinite possibilities to trial, when will the robots stop training first. - The robot will learn certain things on the way and will take them into consideration for the next shots at fires for instance if it learns that it must look towards the target it will stop firing in the opposite direction. - That is the purpose of the optimization algorithm. Second it cannot fire arrows forever. - However, hitting the centre nine out of 10 times may be good enough. So, we can choose to stop it after it reaches a certain level of accuracy or fires a certain number of arrows. - So, let us follow the four ingredients at the end of the training. Our robot or model is already trained on this data. With this set of arrows most shots hit the centre so the air or the objective function is quite low or minimized as we like to say the posture the technique and all other factors cannot be improved. - So, the optimization algorithm has done its best to improve the shooting ability of the machine. - We own a robot that is an amazing Archer. So, what can you do? Give it a different bag of arrows. If they had seen most types of arrows while training it will do great with the new ones. - However, if we give it half an arrow or a longer arrow than it has seen it will not know what to do with it. - In all ordinary cases though we would expect the robot to hit the centre or at least get close. - The benefit of using machine learning is that the robot can learn to fire more effectively than a human. - It might even discover that we've been holding bows in a wrong way for centuries to conclude we must say that machine learning is not about robots.  1) Supervised Machine learning This name derives from the fact that training an algorithm resembles a teacher supervising her students. Types of Machine Learning In Supervised machine learning, it is important to mention you have been dealing with label data. In other words, you can assess the accuracy of each shot. Consider previous example, where there isn't a single target different arrows have their own targets. Let's check what the robot sees when shooting the ground, a target at a short distance a target at a further distance a target hanging on a tree far behind it a house to the side and the sky. So, having labelled data means the associating or labelling a target to a type of Arrow. You know that with a small arrow the robot is supposed to hit the closest target with a medium arrow it can reach the target located further away while with a larger arrow the target that's hanging on the tree. Finally, a crooked arrow is expected to hit the ground not reaching any target during the training process. The robot will be shooting arrows at the respective targets as well as it can. After training is finished. Ideally the robot will be able to fire the small arrow at the centre of the closest target the middle arrow at the centre of the one further away and so on. To summarize label data means we know the target prior to the shot, and we can associate that shot with the target this way. We're sure where the arrow should hit. This allows us to measure the inaccuracy of the shot through the objective function and improve the way the robot shoots through the optimization algorithm. So, what we supervise is the training itself. If a shot is far off from its target, we correct the posture. Otherwise, we don't get. 2) Unsupervised Machine learning In practice though it might happen that you won't have the time or the resources to associate the arrows with targets before giving them to the robot. In that case you could apply the other major type of M-L unsupervised learning here you will just give your robot a bag of arrows with unknown physical properties unlabelled data. This means neither you nor the robot will have separated the arrows into groups. Then you'd ask the machine to simply fire in a direction without providing it with targets. Therefore, in this case you won't be looking for a model that helps you shoot better rather you'll be looking for one which divides the arrows in a certain way. The robot will see just the ground the tree the House and the sky. Remember there are no targets. So, after firing thousands of shots during the training process we will end up having different types of arrows stuck in different areas. For instance, you may identify all the broken arrows by noticing they have fallen on the ground nearby the others you may realise are divided into small medium and large arrows. There may be anomalies like crossbow bolts in your bag that after being shot may have accumulated in a pile over here. You wouldn't want to use them with a simple bow would you. At the end of the training the robot will have fired so many times that it could discover answers that may surprise you. The machine may have managed to split the arrows not into four but into five sized categories due to discovering the crossbow bolt. Or it may have identified that some arrows are going to break soon by placing them in the Broken Arrow pile. It is worth mentioning that supervised learning can deal with such problems too and it does very often. However, if you have one million arrows you don't really have the time to assign targets to all of them do you. To save time and resources you should apply unsupervised learning. 3) Rainforcement learning The third major type of machine learning is called reinforcement learning. This time we introduce a reward system. Every time the robot fires an arrow better than before it will receive an award say a chocolate it will receive nothing if it fires worse. So instead of minimizing an error we are maximizing a reward or in other words maximizing the objective function. If you put yourselves in the shoes of the machine, you'll be reasoning in the following way. I fire an arrow and receive a reward. I'll try to figure out what I did correctly. So, I get more chocolate with the next shot or I fire an arrow and don't receive a reward. There must be something I need to improve. For me to get some chocolate on my next shot positive reinforcement.. In addition, don't forget the robot Archer was an abstract depiction of what a machine learning model can do. In reality there are robots, but the model will be a highly complex mathematical formula the arrows will be a data set and the goals will be various and quantifiable Here are the most notable approaches you will encounter when talking about machine learning support vector machines neural networks deep learning random forced models and Bazy and networks are all types of supervised learning. There are neural networks that can be applied to an unsupervised type of machine learning, but K means is the most common unsupervised approach. By the way you may have noticed we have placed deep learning in both categories. This is a relatively new revolutionary computational approach which is acclaimed as the State-of-the-art email today. Describing it briefly we can say it is fundamentally different from the other approaches. However, it has a broad practical scope of application in all M-L areas because of the extremely high accuracy of its models. Note that deep learning is still divided and supervised, unsupervised and reinforcement, so it solves the same problems but in a conceptually different way.  Real Life Examples of Machine Learning (ML) The financial sector and banks have ginormous data sets of credit card transactions. Unfortunately, banks are facing issues with fraud daily. They are tasked with preventing fraudsters from acquiring customer data and in order to keep customers funds safe they use machine learning algorithms. They take past data and because they can tell the computer which transactions in their history were legitimate and which were found to be fraudulent, they can label the data as such. So through supervised learning they train models that detect fraudulent activity when these models detect even the slightest probability of theft. They flagged the transactions and prevent the fraud in real time. Although no one in the sector has reached a perfect solution. Another example of using supervise machine learning with label data can be found in client retention. A focus of any business be it a global supermarket chain or an online clothing shop is to retain its customers. But the larger a business grows the harder it is to keep track of customer trends. A local corner shop owner will recognize and get to know their most loyal customers. They will offer them exclusive discounts to thank them for their custom. And by doing so keep them returning on a larger scale. Companies can use machine learning and past label data to automate the practice. And with this they can know which customers may purchase goods from them. This means the store can offer discounts and a personal touch in an efficient way minimizing marketing costs and maximizing profits. Popular Data Science Tools