Artificial Intelligence Fundamentals PDF

Natural Language Processing Basics. What Is Natural Language Processing?. Natural language processing (NLP), is a field of artificial intelligence (AI) that combines computer science and linguistics to give computers the ability to understand, interpret, and generate human language in a way that’s m...

Natural Language Processing Basics. What Is Natural Language Processing?. Natural language processing (NLP), is a field of artificial intelligence (AI) that combines computer science and linguistics to give computers the ability to understand, interpret, and generate human language in a way that’s meaningful and useful to humans. NLP helps computers perform useful tasks like understanding the meaning of sentences, recognizing important details in text, translating languages, answering questions, summarizing text, and generating responses that resemble human responses. NLP is already so commonplace in our everyday lives that we usually don’t even think about it when we interact with it or when it does something for us. For example, maybe your email or document creation app automatically suggests a word or phrase you could use next. You may ask a virtual assistant, like Siri, to remind you to water your plants on Tuesdays. Or you might ask Alexa to tell you details about the last big earthquake in Chile for your daughter’s science project. The chatbots you engage with when you contact a company’s customer service use NLP, and so does the translation app you use to help you order a meal in a different country. Spam detection, your online news preferences, and so much more rely on NLP. A Very Brief History of NLP.. It’s worth mentioning that NLP is not new. In fact, its roots wind back to the 1950s when researchers began using computers to understand and generate human language. One of the first notable contributions to NLP was the Turing Test. Developed by Alan Turing, this test measures a machine’s ability to answer any question in a way that’s indistinguishable from a human. Shortly after that, the first machine translation systems were developed. These were sentence- and phrase-based language translation experiments that didn’t progress very far because they relied on very specific patterns of language, like predefined phrases or sentences. By the 1960s, researchers were experimenting with rule-based systems that allowed users to ask the computer to complete tasks or have conversations. The 1970s and 80s saw more sophisticated knowledge-based approaches using linguistic rules, rule-based reasoning, and domain knowledge for tasks like executing commands and diagnosing medical conditions. Statistical approaches (i.e., learning from data) to NLP were popular in the 1990s and early 2000s, leading to advances in speech recognition, machine translation, and machine algorithms. During this period, the introduction of the World Wide Web in 1993 made vast amounts of text-based data readily available for NLP research. Since about 2009, neural networks and deep learning have dominated NLP research and development. NLP areas of translation and natural language generation, including the recently introduced ChatGPT, have vastly improved and continue to evolve rapidly. Human Language Is “Natural” Language.. What is natural language anyway? Natural language refers to the way humans communicate with each other using words and sentences. It’s the language we use in conversations, when we read, write, or listen. Natural language is the way we convey information, express ideas, ask questions, tell stories, and engage with each other. While NLP models are being developed for many different human languages, this module focuses on NLP in the English language. If you completed the Artificial Intelligence Fundamentals badge, you learned about unstructured data and structured data. These are important terms in NLP, too. Natural language–the way we actually speak–is unstructured data, meaning that while we humans can usually derive meaning from it, it doesn’t provide a computer with the right kind of detail to make sense of it. The following paragraph about an adoptable shelter dog is an example of unstructured data. Tala is a 5-year-old spayed, 65-pound female husky who loves to play in the park and take long hikes. She is very gentle with young children and is great with cats. This blue-eyed sweetheart has a long gray and white coat that will need regular brushing. You can schedule a time to meet Tala by calling the Troutdale shelter. For a computer to understand what we mean, this information needs to be well-defined and organized, similar to what you might find in a spreadsheet or a database. This is called structured data. The information included in structured data and how the data is formatted is ultimately determined by algorithms used by the desired end application. For example, data for a translation app is structured differently than data for a chatbot. Here’s how the data in the paragraph above might look as structured data for an app that can help match dogs with potential adopters. Name: Tala. Age: 5. Spayed or Neutered: Spayed. Sex: Female. And so on. Natural Language Understanding and Natural Language Generation.. Today’s NLP matured with its two subfields, natural language understanding (NLU) and natural language generation (NLG). Data processed from unstructured to structured is called natural language understanding (NLU). NLU uses many techniques to interpret written or spoken language to understand the meaning and context behind it. You learn about these techniques in the next unit. Data processed the reverse way–from structured to unstructured–is called natural language generation (NLG). NLG is what enables computers to generate human-like language. NLG involves the development of algorithms and models that convert structured data or information into meaningful, contextually appropriate, natural-like text or speech. It also includes the generation of code in a programming language, such as generating a Python function for sorting strings. In the past, NLU and NLG tasks made use of explicit linguistic structured representations like parse trees. While NLU and NLG are still critical to NLP today, most of the apps, tools, and virtual assistants we communicate with have evolved to use deep learning or neural networks to perform tasks from end-to-end. For instance, a neural machine translation system may translate a sentence from, say, Chinese, directly into English without explicitly creating any kind of intermediate structure. Neural networks recognize patterns, words, and phrases to make language processing exponentially faster and more contextually accurate. In the next unit, you learn more about our natural language methods and techniques that enable computers to make sense of what we say and respond accordingly. Learn About Natural Language Parsing.. Basic Elements of Natural Language.. Understanding and processing natural language is a fundamental challenge for computers. That's because it involves not only recognizing individual words, but also comprehending their relationships, their context, and their meaning. Our natural language, in text and speech, is characterized by endless complexity, nuances, ambiguity, and mistakes. In our everyday communication, we encounter words with several meanings; words that sound the same but are spelled differently and have different meanings; misplaced modifiers; misspellings; and mispronunciations. We also encounter people who speak fast, mumble, or who take forever to get to the point; and people who use speech patterns in accents or dialects that are different from ours. Take this sentence for example: “We saw six bison on vacation in Yellowstone National Park.” You might giggle a little as you imagine six bison in hats and sunglasses posing for selfies in front of Old Faithful. But, most likely, you understand what actually happened–that is, that someone who was on vacation in Yellowstone National Park saw six bison. Or this: “They swam out to the buoy.” If you heard someone speak this sentence without any context, you may think the people involved swam out to a male child, when in fact, they swam out to a marker in the water. The pronunciation of “boy” and “buoy” is slightly different, but the enunciation is not always made clear. While humans are able to flex and adapt to language fairly easily, training a computer to consider these kinds of nuances is quite difficult. Elements of natural language in English include: Vocabulary. The words we use. Grammar. The rules governing sentence structure. Syntax. How words are combined to form sentences according to grammar. Semantics. The meaning of words, phrases, and sentences. Pragmatics. The context and intent behind cultural or geographic language use. Discourse and dialogue. Units larger than a single phrase or sentence, including documents and conversations. Phonetics and phonology. The sounds we make when we communicate. Morphology. How parts of words can be combined or uncombined to make new words. Parsing Natural Language.. Teaching a computer to read and derive meaning from words is a bit like teaching a child to read–they both learn to recognize words, their sounds, meaning, and pronunciation. But when a child learns to read, they usually have the advantage of context from a story; visual cues from illustrations; and relationships to things they already know, like trees or animals. They also often get assistance and encouragement from experienced readers, who help explain what they’re learning. These cues help new readers identify and attach meaning to words and phrases that they can generalize to other things they read in the future. We know that computers are a different kind of smart, so while a computer needs to understand the elements of natural language described above, the approach needs to be much more scientific. NLP uses algorithms and methods like large language models (LLMs), statistical models, machine learning, deep learning, and rule-based systems to process and analyze text. These techniques, called parsing, involve breaking down text or speech into smaller parts to classify them for NLP. Parsing includes syntactic parsing, where elements of natural language are analyzed to identify the underlying grammatical structure, and semantic parsing which derives meaning. As mentioned in the last unit, natural language is parsed in different ways to match intended outcomes. For example, natural language that’s parsed for a translation app uses different algorithms or models and is parsed differently than natural language intended for a virtual assistant like Alexa. Syntactic parsing may include:  1. Segmentation.: Larger texts are divided into smaller, meaningful chunks. Segmentation usually occurs at the end of sentences at punctuation marks to help organize text for further analysis. 2. Tokenization.: Sentences are split into individual words, called tokens. In the English language, tokenization is a fairly straightforward task because words are usually broken up by spaces. In languages like Thai or Chinese, tokenization is much more complicated and relies heavily on an understanding of vocabulary and morphology to accurately tokenize language. 3. Stemming.: Words are reduced to their root form, or stem. For example breaking, breaks, or unbreakable are all reduced to break. Stemming helps to reduce the variations of word forms, but, depending on context, it may not lead to the most accurate stem. Look at these two examples that use stemming: “I’m going outside to rake leaves.” Stem = leave “He always leaves the key in the lock.” Stem = leave 4. Lemmatization.: Similar to stemming, lemmatization reduces words to their root, but takes the part of speech into account to arrive at a much more valid root word, or lemma. Here are the same two examples using lemmatization: “I’m going outside to rake leaves.” Lemma = leaf “He always leaves the key in the lock.” Lemma = leave 5. Part of speech tagging.: Assigns grammatical labels or tags to each word based on its part of speech, such as a noun, adjective, verb, and so on. Part of speech tagging is an important function in NLP because it helps computers understand the syntax of a sentence. 6. Named entity recognition (NER.): Uses algorithms to identify and classify named entities–like people, dates, places, organizations, and so on–in text to help with tasks like answering questions and information extraction. Semantic Analysis.. Parsing natural language using some or all of the steps we just described does a pretty good job of capturing the meaning of text or speech. But it lacks soft skill nuances that make human language, well, human. Semantic parsing involves analyzing the grammatical format of sentences and relationships between words and phrases to find the meaning representation. Extracting how people feel, why they are engaging, and details about circumstances surrounding an interaction all play a crucial role in accurately deciphering text or speech and forming an appropriate response. Here are several common analysis techniques that are used in NLP. Each of these techniques can be powered by a number of different algorithms to get the desired level of understanding depending on the specific task and the complexity of the analysis. Sentiment analysis.: Involves determining whether a piece of text (such as a sentence, a social media post, a review, or a tweet) expresses a positive, negative, or neutral sentiment. A sentiment is a feeling or an attitude toward something. For example, sentiment analysis can determine if this customer review of a service is positive or negative: "I had to wait a very long time for my haircut.” Sentiment helps identify and classify emotions or opinions in text to help businesses understand how people feel about their products, services, or experiences. Intent analysis.: Intent helps us understand what someone wants or means based on what they say or write. It’s like deciphering the purpose or intention behind their words. For example, if someone types, “I can’t log in to my account,” into a customer support chatbot, intent analysis would recognize that the person’s intent is to get help to access their account. The chatbot might reply with details about resetting a password or other means the user can try to access their account. Virtual assistants, customer support systems, or chatbots often use intent analysis to understand user requests and provide appropriate responses or actions. Context (discourse) analysis.: Natural language relies heavily on context. The interpretation of a statement might change based on the situation, the details provided, and any shared understanding that exists between the people communicating. Context analysis involves understanding this surrounding information to make sense of a piece of text. For example, if someone says, “They had a ball,” context analysis can determine if they are talking about a fancy dance party, a piece of sports equipment, or a whole lot of fun. It does this by considering the previous conversation or the topic being discussed. Context analysis helps NLP systems interpret words more accurately by taking into account the broader context, the relationships between words, and other relevant information. These three analysis techniques–sentiment analysis, intent analysis, and context analysis–play important roles in extracting valuable insights from text and speech data. They create a more sophisticated and accurate understanding and engagement with textual content in various applications of NLP. Summary In this module, you’ve learned about NLP at a very high level, and as it relates to the English language. To-date, the majority of NLP study is conducted using English, but you can also find a great deal of research done in Spanish, French, Farsi, Urdu, Chinese, and Arabic. NLP is a very rapidly evolving field of AI. And advancements in NLP are quickly leading to more sophisticated language understanding, cross-language capabilities, and integration with other AI fields. Data Fundamentals for AI.. The Importance of Data.. Data is a collection of facts, figures, and statistics that provide insight into various aspects of the world. Today, data is an essential component of modern life and the economy. With the rise of technology, businesses can collect, store, and analyze vast amounts of data to gain insights into their operations and customers. Data can be considered the most valuable asset in the modern world and is collected in many forms. Data provides valuable insights and information that can help individuals and organizations make better decisions. It’s now generated at an unprecedented rate. And businesses and governments are using data to gain insights into consumer behavior, market trends, and other important factors. Here are industry-specific examples of how data impacts our world. Industry Outcome Business and By analyzing data, businesses can identify new opportunities and develop Finance. new products and services that meet their customers’ needs. Healthcare and By analyzing data, researchers can identify patterns and correlations that Medicine. can lead to breakthrough discoveries. Data plays a crucial role in developing new treatments and cures for diseases. Other. Data can be used in almost any industry to improve operations and drive business success. Data-Driven Decision-Making.. Data-driven decision-making is a process of making decisions based on data analysis rather than intuition or personal experience. In modern organizations, data-driven decision-making is increasingly important due to the large amounts of data available. Data-driven decision-making can provide more accurate and reliable insights into business operations, customer behavior, and market trends. Traditional decision-making, on the other hand, relies on intuition, personal experience, and other subjective factors. While traditional decision-making can be effective in some situations, it can lead to biased decisions and missed opportunities. To implement data-driven decision-making, organizations must collect, store, and analyze data effectively. This requires the use of various tools and techniques, such as data visualization, statistical analysis, and machine learning (ML). Here are key benefits and outcomes that sum up the role of data-driven decision-making in modern organizations. Key Benefit Outcome Provides insights. By analyzing data, organizations can identify patterns and correlations that may not be apparent through other means, leading to more-informed decisions. Improves Data-driven decision-making can lead to improved performance by performance. identifying areas where organizations can reduce costs, improve efficiency, and optimize operations. Increases By using data to gain insights into customer behavior and market trends, competitiveness. organizations can develop products and services that better serve their customers’ needs. The Significance of AI.. Artificial intelligence (AI) is a technology that enables machines to learn and perform tasks that would normally require human intelligence. AI has become increasingly important in today’s world due to its ability to automate various tasks, improve efficiency, and reduce costs. AI is used in various industries, including healthcare, finance, transportation, and manufacturing to improve operations and provide better services to customers. Data is essential in enabling AI to learn and perform tasks that would normally require human intelligence, and to provide insights that improve operations and services across industries. In healthcare, AI requires large datasets of medical images and patient data to analyze and identify health risks. In finance, AI analyzes large amounts of financial data to make investment decisions and detect fraudulent activity. In manufacturing, AI uses sensor and production data to monitor equipment performance, identify maintenance issues, and optimize production processes. Here are some of the key applications of AI across industries. Healthcare.: In the healthcare industry, AI is used for medical imaging, drug discovery, and patient monitoring. AI-powered medical imaging can help physicians detect diseases and injuries more accurately, while AI-powered drug discovery can help researchers develop new drugs more quickly. AI can also be used to monitor patients’ conditions in real time, enabling healthcare providers to deliver personalized care more effectively. Finance.: In the finance industry, AI is used for fraud detection, credit scoring, and investment management. AI-powered fraud detection can help banks and other financial institutions identify fraudulent transactions more quickly and accurately, while AI-powered credit scoring can provide more accurate assessments of creditworthiness. AI can also be used to manage investments, enabling financial advisors to make more informed decisions. Manufacturing.: In the manufacturing industry, AI is used for quality control, predictive maintenance, and supply chain optimization. AI-powered quality control can help manufacturers identify defects and improve product quality, while AI-powered predictive maintenance can reduce downtime and improve efficiency. AI can also be used to optimize supply chains, enabling manufacturers to deliver products more efficiently. AI has become an essential technology in various industries due to its ability to automate tasks, improve efficiency, and reduce costs. By using AI, businesses can improve their operations and provide better services to customers, ultimately leading to increased competitiveness and better outcomes for all. In this unit, you learned about the importance of data and its role in data-driven decision-making. You also discovered the basics of AI and its various applications across different industries. In the next unit, you dive deeper into data concepts, including data types, data cleaning, and data sources. Understand Data and Its Significance.. Data Classification and Types. With data being an essential component of industries today, it’s important to understand the different types of data, data sources and collection methods, and the importance of data in AI. Data Classification.. Data can be classified into three main categories: structured, unstructured, and semi-structured. Structured data. Structured data is organized and formatted in a specific way, such as in tables or spreadsheets. It has a well-defined format and is easily searchable and analyzable. Examples of structured data include spreadsheets, databases, data lakes, and warehouses. Unstructured data. Unstructured data, on the other hand, is not formatted in a specific way and can include text documents, images, audios, and videos. Unstructured data is more difficult to analyze, but it can provide valuable insights into customer behavior and market trends. Examples of unstructured data include social media posts, customer reviews, and email messages. Semi-structured data. Semi-structured data is a combination of structured and unstructured data. It has some defined structure, but it may also contain unstructured elements. Examples of semi-structured data include XML (Extensible Markup Language) or JSON (JavaScript Object Notation) files. Data Format.. Data can also be classified by its format. Tabular data. Tabular data is structured data that is organized in rows and columns, such as in a spreadsheet. Text data. Text data includes unstructured data in the form of text documents, such as emails or reports. Image data. Image data can include visual information in the form of a brand logo, charts, and infographics. Geospatial data. Geospatial data refers to geographic coordinates and the shape of country maps, representing essential information about the Earth’s surface. Time-series data. Time-series data refers to data that can contain information over a period of time, for example, daily stock prices over the past year.. Types of Data.. Another way to classify data is by its type, which can be quantitative or qualitative. Quantitative data. Quantitative data is numerical and can be measured and analyzed statistically. Examples of quantitative data include sales figures, customer counts based on geographical location, and website traffic. Qualitative data. Qualitative data, on the other hand, is non-numerical and includes text, images, and videos. In many cases, qualitative data can be more difficult to analyze, but it can provide valuable insights into customer preferences and opinions. Examples of qualitative data include customer reviews, social media posts, and survey responses. Both quantitative and qualitative data are important in the field of data analytics across a wide range of industries. For more detail on this topic, check out the Variables and Field Types Trailhead module. Understanding different data types and classifications is important for effective data analysis. By categorizing data into structured, unstructured, and semi-structured categories, and differentiating between quantitative and qualitative data, organizations can more effectively choose the right analysis approach for gaining insights from it. Exploring different formats, such as tabular, text, and images, makes data analysis and interpretation more effective. Data Collection Methods.. Identifying data sources is an important step in data analysis. Data can be obtained from various sources, including internal, external, and public datasets. Internal data sources include data that is generated within an organization, such as sales data and customer data. External data sources include data that is obtained from outside the organization, such as market research and social media data. Public datasets are freely available datasets that can be used for analysis and research. Data collection, labeling, and cleaning are important steps in data analysis. Data collection. Data collection is the process of gathering data from various sources. Data labeling. Data labeling is assigning tags or labels to data to make it more easily searchable and analyzable. This can include assigning categories to data, such as age groups or product categories. Data cleaning. Data cleaning is the process of removing or correcting errors and inconsistencies in the data to improve its quality and accuracy. Data cleaning can include removing duplicate data, correcting spelling errors, and filling in missing data. Various techniques can be used for collecting data, such as surveys, interviews, observation, and web scraping. Surveys collect data from a group of people using a set of questions. They can be conducted online or in-person, and are often used to collect data on customer preferences and opinions. Interviews collect data from individuals through one-on-one conversations. They can provide more detailed data than surveys, but they can also be time-consuming. Observation collects data by watching and listening to people or events. This can provide valuable data on customer behavior and product interactions. Web scraping collects data from websites using software tools. It can be used to collect data on competitors, market trends, and customer reviews. Exploratory data analysis (EDA) is usually the first step in any data project. The goal of EDA is to learn about general patterns in data and understand the insights and key characteristics about it. The Importance of Data in AI.. Data is an essential component of AI, and the quality and validity of data are critical to the success of AI applications. Considerations for data quality and validity include ensuring that the data is accurate, complete, and representative of the population being studied. Bad data can have a significant impact on decision-making and AI, leading to inaccurate or biased results. Data quality is important from the beginning of an AI project. Here are a few areas of consideration that highlight the importance of data and data quality in AI. Training and performance.: The quality of the data used for training AI models directly impacts their performance. High-quality data ensures that the model learns accurate and representative patterns, leading to more reliable predictions and better decision-making. Accuracy and bias.: Data quality is vital in mitigating bias within AI systems. Biased or inaccurate data can lead to biased outcomes, reinforcing existing inequalities or perpetuating unfair practices. By ensuring data quality, organizations can strive for fairness and minimize discriminatory outcomes. Generalization and robustness.: AI models should be able to handle new and unfamiliar data effectively, and consistently perform well in different situations. High-quality data ensures that the model learns relevant and diverse patterns, enabling it to make accurate predictions and handle new situations effectively. Trust and transparency.: Data quality is closely tied to the trustworthiness and transparency of AI systems. Stakeholders must have confidence in the data used and the processes involved. Transparent data practices, along with data quality assurance, help build trust and foster accountability. Data governance and compliance.: Proper data quality measures are essential for maintaining data governance and compliance with regulatory requirements. Organizations must ensure that the data used in AI systems adheres to privacy, security, and legal standards. To achieve high data quality in AI, a robust data lifecycle is needed with focus on data diversity, representativeness, and addressing potential biases. There are various stages in the data lifecycle, and data quality is important in all of the stages. The data lifecycle includes collection, storage, processing, analysis, sharing, retention and disposal. You get more detail on the data lifecycle in the next unit. In this unit, you learned about different types of data, data sources and collection methods, and the importance of data in AI. Next, get the basics on machine learning and how it’s different from traditional programming. And learn about AI techniques and their applications in the real world. Discover AI Techniques and Applications.. Artificial Intelligence Technologies.. Artificial Intelligence is the expansive field of making machines learn and think like humans. And there are many technologies that encompass AI. Machine learning. Machine learning uses various mathematical algorithms to get insights from data and make predictions. Deep learning. Deep learning uses a specific type of algorithm called a neural network to find associations between a set of inputs and outputs. Deep learning becomes more efficient as the amount of data increases. Natural language processing. Natural language processing is a technology that enables machines to take human language as an input and perform actions accordingly. Computer vision. Computer vision is technology that enables machines to interpret visual information. Robotics. Robotics is a technology that enables machines to perform physical tasks. Machine learning (ML) can be classified into several types based on the learning approach and the nature of the problem being solved. Supervised learning. Supervised learning: In this machine learning approach, a model learns from labeled data, making predictions based on patterns it finds. The model can then make predictions or classify new, unseen data based on the patterns it has learned during training. Unsupervised learning. Unsupervised learning: Here, the model learns from unlabeled data, finding patterns and relationships without predefined outputs. The model learns to identify similarities, group similar data points, or find underlying hidden patterns in the dataset. Reinforcement learning. Reinforcement learning: This type of learning involves an agent learning through trial and error, taking actions to maximize rewards received from an environment. Reinforcement learning is often used in scenarios where an optimal decision-making strategy needs to be learned through trial and error, such as in robotics, game playing, and autonomous systems. The agent explores different actions and learns from the consequences of its actions to optimize its decision-making process. AutoML and No Code AI tools like OneNine AI and Salesforce Einstein have been introduced in recent years to automate the process of building an entire machine learning pipeline, with minimal human intervention. The Role of Machine Learning.. Machine learning is a subset of artificial intelligence that uses statistical algorithms to enable computers to learn from data, without being explicitly programmed. It uses algorithms to build models that can make predictions or decisions based on inputs. Machine Learning vs Programming.. In traditional programming, the programmer must have a clear understanding of the problem and the solution they’re trying to achieve. In machine learning, the algorithm learns from the data and generates its own rules or models to solve the problem. Importance of Data in Machine Learning.. Data is the fuel driving machine learning. The quality and quantity of data used in training a machine learning model can have a significant impact on its accuracy and effectiveness. It’s essential to ensure that the data used is relevant, accurate, complete, and unbiased. Data Quality and the Limitations of Machine Learning.. To ensure data quality, it’s necessary to clean and preprocess the data, removing any noise (unwanted or meaningless information), missing values, or outliers. While machine learning is a powerful tool for solving a wide range of problems, there are also limitations to its effectiveness including overfitting, underfitting, and bias. Overfitting. Overfitting occurs when the model is too complex and fits the training data too closely, resulting in poor generalization. Underfitting. Underfitting occurs when the model is too simple and does not capture the underlying patterns in the data. Bias. Bias occurs when the model is trained on data that is not representative of the real-world population. Machine learning is limited by the quality and quantity of data used, lack of transparency in complex models, difficulty in generalizing to new situations, challenges with handling missing data, and potential for biased predictions. While machine learning is a powerful tool, it’s important to be aware of these limitations and to consider them when designing and using machine learning models. Predictive vs Generative AI.. Predictive AI is the use of machine learning algorithms to make predictions or decisions based on data inputs. This can be used in a wide range of applications, including fraud detection, medical diagnosis, and customer churn prediction. Distinct Approaches, Different Purposes.. Predictive AI. Predictive AI is a type of machine learning that trains a model to make predictions or decisions based on data. The model is given a set of input data and it learns to recognize patterns in the data that allow it to make accurate predictions for new inputs. Predictive AI is widely used in applications such as image recognition, speech recognition, and natural language processing. Generative AI. Generative AI, on the other hand, creates new content, such as images, videos, or text, based on a given input. Rather than making predictions based on existing data, generative AI creates new data that is similar to the input data. This can be used in a wide range of applications, including art, music, and creative writing. One common example of generative AI is the use of neural networks to generate new images based on a given set of inputs. While predictive and generative AI are different approaches to artificial intelligence, they’re not mutually exclusive. In fact, many AI applications use both predictive and generative techniques to achieve their goals. For example, a chatbot might use predictive AI to understand a user’s input, and generative AI to generate a response that is similar to human speech. Overall, the choice of predictive or generative AI depends on the specific application and project goals. Now you know a thing or two about predictive AI and generative AI and their differences. For your reference, here’s a quick rundown of what each can do. Predictive AI Generative AI Can make accurate predictions based on labeled Can generate new and creative content. data. Can be used to solve a wide range of problems, Can be used in a wide range of creative including fraud detection, medical diagnosis, and applications, such as art, music, and customer churn prediction. writing. Limited by the quality and quantity of labeled data Can generate biased or inappropriate available. content based on the input data. May struggle with making predictions outside of the May struggle with understanding labeled data it was trained on. context or generating coherent content. May require significant computational resources to May not be suitable for all applications, train and deploy. such as those that require accuracy and precision. Limitations of Generative AI.. Generative AI creates new content, such as images, videos, or text, based on a given input. ChatGPT, for example, is a generative AI model that can generate human-like responses to text inputs. It works by training on large amounts of text data and learning to predict the next word in a sequence based on the previous words. While ChatGPT can generate human-like responses, it also has limitations—it may generate biased or inappropriate responses based on the data it was trained on. This is a common problem with machine learning models, as they can reflect the biases and limitations of the training data. For example, if the training data contains a lot of negative or offensive language, ChatGPT may generate responses that are similarly negative or offensive. ChatGPT may also struggle with understanding user input context or generating coherent responses. ChatGPT is only as good as the data it’s trained on. If the training data is incomplete, biased, or otherwise flawed, the model may not be able to generate accurate or useful responses. This can be a significant limitation in applications where accuracy and relevance are important. Similar to other machine learning models, data plays a critical role, so if the data it’s trained on is bad, then ChatGPT will not be very useful. The ChatGPT example demonstrates the critical role that data plays in using AI effectively. Data Lifecycle for AI.. The data lifecycle refers to the stages that data goes through, from its initial collection to its eventual deletion. The data life cycle for AI consists of a series of steps, including data collection, preprocessing, training, evaluation, and deployment. It’s important to ensure that the data used is relevant, accurate, complete, and unbiased, and that the models generated are effective and ethical. The data lifecycle for AI is an ongoing process, as models need to be continuously updated and refined based on new data and feedback. It’s an iterative process that requires careful attention to detail and a commitment to ethical and effective AI. Developers and users of ML models should ensure that their models are effective, accurate, and ethical, and make a positive impact in the world. The data lifecycle is crucial to ensuring that data is collected, stored, and used responsibly and ethically. These are the stages of the data lifecycle. 1. Data collection.: In this stage, data is collected from various sources, such as sensors, surveys, and online sources. 2. Data storage.: Once data is collected, it must be stored securely. 3. Data processing.: In this stage, data is processed to extract insights and patterns. This may include using machine learning algorithms or other data analysis techniques.  4. Data use.: Once the data has been processed, it can be used for its intended purpose, such as making decisions or informing policy. 5. Data sharing.: At times, it may be necessary to share data with other organizations or individuals. 6. Data retention.: Data retention refers to the length of time that data is kept. 7. Data disposal.: Once data is no longer needed, it needs to be disposed of securely. This may involve securely deleting digital data or destroying physical media. While AI and ML have the potential to revolutionize many industries and solve complex problems, it’s important to be aware of their limitations and ethical considerations. Continue to the next unit, to learn about the importance of data ethics and privacy. Know Data Ethics, Privacy, and Practical Implementation.. Ethics, Data, and AI.. Data collection and analysis are critical components of AI and machine learning, but they can also raise ethical concerns. As data becomes increasingly valuable and accessible, it’s important to consider the ethical implications of how it’s collected, analyzed, and used. Some examples of ethical issues in data collection and analysis include: Privacy violations.: Collecting and analyzing personal information without consent, or using personal information for purposes other than those for which it was collected. Data breaches.: Unauthorized access to or release of sensitive data, which can result in financial or reputational harm to individuals or organizations. Bias.: The presence of systematic errors or inaccuracies in data, algorithms, or decision-making processes that can cause unfair or discriminatory outcomes. Ensure Data Privacy, Consent, and Confidentiality.. To address these ethical issues, it’s important to ensure that data is collected, analyzed, and used in a responsible and ethical way. This requires strategies for ensuring data privacy, consent, and confidentiality. These strategies can help promote data privacy and confidentiality: Encryption.: Protecting sensitive data by encrypting it so that it can only be accessed by authorized users. Anonymization.: Removing personally identifiable information from data so that it can’t be linked back to specific individuals. Access controls.: Limiting access to sensitive data to authorized users, and ensuring that data is only used for its intended purpose. Address Biases and Fairness in Data-Driven Decision-Making.. One of the key challenges in data-driven decision-making is the presence of bias, which can lead to unfair or discriminatory outcomes. Bias can be introduced at any stage of the data lifecycle, from data collection to algorithmic decision-making. Addressing bias and promoting fairness requires a range of strategies, including: Diversifying data sources.: One of the key ways to address bias is to ensure that data is collected from a diverse range of sources. This can help to ensure that the data is representative of the target population and that any biases that may be present in one source are balanced out by other sources. Improving data quality.: Another key strategy for addressing bias is to improve data quality. This includes ensuring that the data is accurate, complete, and representative of the target population. It may also include identifying and correcting any errors or biases that may be present in the data. Conducting bias audits.: Regularly reviewing data and algorithms to identify and address any biases that may be present is also an important strategy for addressing bias. This may include analyzing the data to identify any patterns or trends that may be indicative of bias and taking corrective action to address them. Incorporating fairness metrics.: Another important strategy for promoting fairness is to incorporate fairness metrics into the design of algorithms and decision-making processes. This may include measuring the impact of certain decisions on different groups of people and taking steps to ensure that the decisions are fair and unbiased. Promoting transparency.: Promoting transparency is another key strategy for addressing bias and promoting fairness. This may include making data and algorithms available to the public and providing explanations for how decisions are made. It may also include soliciting feedback from stakeholders and incorporating their input into decision-making processes. Adopting these strategies helps organizations ensure their data-driven decision-making processes are fair and unbiased. To ensure that AI and machine learning are developed and deployed in a responsible and ethical manner, it’s important to have ethical frameworks and guidelines in place. So let’s deep dive into top regulatory frameworks related to data and AI. Legal and Regulatory Frameworks for Data and AI.. Data protection laws and regulations are an important component of ensuring that data is collected, analyzed, and used responsibly and ethically. Here are four important data protection laws and regulations. The California Consumer Privacy Act (CCPA).: A set of regulations that apply to companies that do business in California and collect the personal data of California residents. The Health Insurance Portability and Accountability Act (HIPAA).: A set of regulations that apply to healthcare organizations and govern the use and disclosure of protected health information in the United States. The General Data Protection Regulation (GDPR).: A set of regulations that apply to all companies that process the personal data of European Union citizens. European Union Artificial Intelligence Act (EU AI Act).: Comprehensive AI regulations banning systems with unacceptable risk and giving specific legal requirements for high-risk applications. Government agencies are responsible for enforcing these laws and regulations. They investigate complaints and data breaches, conduct audits and inspections, impose fines and penalties for noncompliance, and provide guidance and advice to organizations on how to protect data and comply with data protection laws and regulations. Best Practices for Data Lifecycle Management.. Effective data lifecycle management requires a range of best practices to ensure that data is collected, stored, and used in a responsible and ethical way. Some best practices for data lifecycle management include: Implementing data governance policies and procedures to ensure that data is collected and used in a responsible and ethical manner. Conducting regular audits and assessments to identify any weaknesses or vulnerabilities in the data lifecycle. Ensuring that the data is accurate, complete, and representative of the target population. Ensuring that data is stored securely, and that access is granted only to authorized users. Ensuring that data is used only for its intended purpose and is shared only in a responsible and ethical manner. Placing appropriate safeguards to protect the data. Ensuring that data retention policies are in place and that data is securely deleted once it’s no longer needed. By following these best practices, organizations can ensure that they are responsibly and ethically managing data, and that they’re protecting the privacy and confidentiality of individuals and organizations. AI relies on vast amounts of data to learn and make predictions. Understanding the importance of data is critical for developing effective AI models that can drive innovation and success. By understanding the fundamental concepts, individuals and organizations can effectively leverage data and AI to drive innovation and success while ensuring ethical and responsible use.