Data Engineering and Analysis PDF

Data Engineering and Analytics YARMOUK UNIVERSITY FACULTY OF INFORMATION TECHNOLOGY AND COMPUTE...

Data Engineering and Analytics YARMOUK UNIVERSITY FACULTY OF INFORMATION TECHNOLOGY AND COMPUTER SCIENCES DA 330: Data Engineering and Analysis Topic 1: Introduction to Data Engineering Dr. Rafat Hammad 1 Acknowledgements: Most of these slides have been prepared based on various online tutorials and presentations, with respect to their authors and adopted for our course. Additional slides have been added from the mentioned references in the syllabus 1 TOPIC 1 : OUTLINE ❑ What is Data Engineering? ❑ Data Engineering Learning Path ❑ What is Data? ❑ Data Lifecycle Management ❑ Data Repositories Dr. Rafat Hammad 2 2 Dr. Rafat Hammad Data Engineering and Analytics WHAT IS DATA ENGINEERING? Data Engineering: the process of designing, building, and maintaining systems that collect, store, and process data. Data engineering is a critical part of data science, as it ensures that data is collected, stored, and processed in a way that is efficient, reliable, and scalable. Without data engineering, data science would not be possible. Data Engineer: develops and maintains data architecture and pipelines. Essentially, they build the programs that generate data and aim to do so in a way that ensures the output is meaningful for operations and analysis. Dr. Rafat Hammad 3 3 RESPONSIBILITIES OF DATA ENGINEER?  1) Data collection: This involves designing and executing systems to collect and extract data from different sources. These sources could be social media, transactional databases, sensor data from IoT devices, maps, texts, documents, images, stock prices, etc.  2) Data storage: Using data warehouses or data lakes to store large volumes of data and ensuring data is organized for easy accessibility.  3) Data processing: Creating distributed processing systems to clean, aggregate, and transform data, ensuring it’s ready for analysis.  4) Data integration: Developing data pipelines that integrate data from various sources to create a comprehensive view.  5) Data quality and governance: Ensuring that data is of high-quality, reliable and adheres/complies with regulatory standards.  6) Data provisioning: Ensuring the processed data is available to end users and applications. Dr. Rafat Hammad 4 4 Dr. Rafat Hammad Data Engineering and Analytics WHAT IS A DATA ANALYST? Data Analyst: brings together data sources in a way that makes it possible to drive consolidated insights. They do the work of building systems that can model data in a clean, clear way repeatedly so that everyone can use those systems to answer questions on an ongoing basis. Responsibilities:  Descriptive statistics to summarize data.  Exploratory data analysis to understand patterns and relationships.  Creating visualizations to communicate findings.  Often involves using tools like Excel, SQL, or statistical software Dr. Rafat Hammad 5 5 WHAT IS A DATA SCIENTIST? Data Scientist: studies large data sets using advanced statistical analysis and machine learning algorithms. In doing so, they identify patterns in data to drive critical business insights, and then typically use those patterns to develop machine learning solutions for more efficient and accurate insights at scale. Critically, they combine this statistics experience with software engineering experience. Responsibilities:  Developing machine learning models for predictions and classifications.  Analyzing complex data sets to identify patterns and trends.  Extracting meaningful insights to inform business decisions.  Often involves coding in languages like Python or R. Dr. Rafat Hammad 6 6 Dr. Rafat Hammad Data Engineering and Analytics DATA ANALYST VS DATA SCIENTIST VS DATA ENGINEER Data Engineers are primarily focused on building and maintaining the systems that data scientists and data analysts use to collect, store, and analyze data. Data Analyst: Analyze data to summarize the past in visual form. Data Scientist: Analyze data to identify patterns and trends to predict future outcomes. Dr. Rafat Hammad 7 7 THE IMPORTANCE OF SOFTWARE ENGINEERING 1) Reduces complexity: Big software is always complicated and challenging to progress. Software engineering has a great solution to reduce the complication of any project. Software engineering divides big problems into various small issues. And then start solving each small issue one by one. All these small problems are solved independently to each other. 2) To minimize software cost: Software needs a lot of hardwork and software engineers are highly paid experts. A lot of manpower is required to develop software with a large number of codes. But in software engineering, programmers project everything and decrease all those things that are not needed. In turn, the cost for software productions becomes less as compared to any software that does not use software engineering method. Dr. Rafat Hammad 8 8 Dr. Rafat Hammad Data Engineering and Analytics THE IMPORTANCE OF SOFTWARE ENGINEERING (CONT.) 3) To decrease time: Anything that is not made according to the project always wastes time. And if you are making great software, then you may need to run many codes to get the definitive running code. This is a very time-consuming procedure, and if it is not well handled, then this can take a lot of time. So if you are making your software according to the software engineering method, then it will decrease a lot of time. 4) Reliable software: Software should be secure, means if you have delivered the software, then it should work for at least its given time or subscription. And if any bugs come in the software, the company is responsible for solving all these bugs. Because in software engineering, testing and maintenance are given, so there is no worry of its reliability. Dr. Rafat Hammad 9 9 THE IMPORTANCE OF SOFTWARE ENGINEERING (CONT.) 5) Handling big projects: Big projects are not done in a couple of days, and they need lots of patience, planning, and management. And to invest six and seven months of any company, it requires heaps of planning, direction, testing, and maintenance. No one can say that he has given four months of a company to the task, and the project is still in its first stage. Because the company has provided many resources to the plan and it should be completed. So to handle a big project without any problem, the company has to go for a software engineering method. 5) Effectiveness: Effectiveness comes if anything has made according to the standards. Software standards are the big target of companies to make it more effective. So Software becomes more effective in the act with the help of software engineering. Dr. Rafat Hammad 10 10 Dr. Rafat Hammad Data Engineering and Analytics TOPIC 1 : OUTLINE ❑ What is Data Engineering? ❑ Data Engineering Learning Path ❑ What is Data? ❑ Data Lifecycle Management ❑ Data Repositories Dr. Rafat Hammad 11 11 DATA ENGINEERING LEARNING PATH The following are the core topics a data engineering should master: 1) Programming- Programming is a fundamental skill for data engineers as most of the tasks performed by data engineers rely on writing programming scripts. Learning python is highly encouraged because it is widely used in many industries today. Python is easy to understand for beginners and has many modules and libraries that can be used for various tasks, including data wrangling, data analysis, machine learning, and deep learning. Lastly, python is commonly used for developing scripts. Dr. Rafat Hammad 12 12 Dr. Rafat Hammad Data Engineering and Analytics DATA ENGINEERING LEARNING PATH (CONT.) 2) Scripting and Automation- In data engineering, scripting and automation refers to the process of automating the creation and maintenance of data pipelines. This can include tasks such as provisioning resources, configuring settings, and deploying code. It can also include more complex tasks such as monitoring data flows and managing data quality. Here learners should focus on the basics of scripting languages such as Ruby or Python and how to use automation tools such as Puppet. Additionally, one should learn how to integrate automation into the data engineering workflow. Lastly, how to troubleshot and debug automation scripts. Dr. Rafat Hammad 13 13 DATA ENGINEERING LEARNING PATH (CONT.) 3) Relational Databases and SQL- Relational databases and SQL are the fundamental technologies for storing and querying data. Data engineering need to understand the following concepts:  The basics of relational databases, including how to structure data in tables and how to query data using the SQL language.  The basics of SQL, including how to select data, how to insert and update data, and how to use SQL functions and operators.  How to design efficient and effective database schema, including how to normalize data and how to choose appropriate data types.  How to optimize SQL queries for performance, including how to use indexes and how to write efficient SQL code. Dr. Rafat Hammad 14 14 Dr. Rafat Hammad Data Engineering and Analytics DATA ENGINEERING LEARNING PATH (CONT.) 4) NoSQL Databases and Map Reduce: There is a lot to learn in NoSQL databases and Map Reduce in data engineering. However, here are some key things to focus on:  How NoSQL databases work and their key features.  How to design data models for NoSQL databases.  How to query NoSQL databases using Map Reduce.  How to optimize Map Reduce jobs for performance.  How to troubleshoot and debug Map Reduce jobs. Dr. Rafat Hammad 15 15 DATA ENGINEERING LEARNING PATH (CONT.) 5) Data Analysis- There are a few key things to learn in data analysis when working in data engineering.  Firstly, it is important to understand the basics of statistical analysis and how to use various tools to effectively analyze data.  Secondly, it is also beneficial to learn how to effectively visualize data so that it can be easily interpreted.  Finally, it is also important to be familiar with the different types of data that can be collected and stored to effectively engineer data solutions. Dr. Rafat Hammad 16 16 Dr. Rafat Hammad Data Engineering and Analytics DATA ENGINEERING LEARNING PATH (CONT.) 6) Data Processing Techniques- There are a few key things to learn in Data Processing Techniques for data engineering:  Batch Processing: This is a process where data is processed in batches, typically on a schedule. This can be used to process large amounts of data efficiently.  Building Data Pipelines: This involves creating a system to efficiently move data from one place to another. This is often done using ETL (Extract, Transform, Load) tools.  Debugging: This is a process of finding and fixing errors in data processing systems. This can be done using tools like Hadoop or Spark. Dr. Rafat Hammad 17 17 DATA ENGINEERING LEARNING PATH (CONT.) 7) Big Data- The most important thing is to learn how to effectively use the tools available to manage and process large data sets. The most popular tools for this purpose include Hadoop, HDFS, MapReduce, Spark, Hive, and Pig. 8) Workflows- There are a few key concepts that are important to learn to create efficient and effective data engineering workflows. These include understanding how to extract, transform, and load data (ETL), as well as how to create and use data pipelines. Additionally, it is important to have a solid understanding of data warehousing and how to optimize data storage and retrieval. Dr. Rafat Hammad 18 18 Dr. Rafat Hammad Data Engineering and Analytics DATA ENGINEERING LEARNING PATH (CONT.) 9) Infrastructure- In infrastructure, data engineering refers to the process of designing, building, and maintaining data infrastructure. This includes the data warehouse, data lakes, data marts, and data pipelines that are necessary to support data-driven applications and analytics. Data engineers are responsible for ensuring that data is accessible, reliable, and scalable. They work with data architects to design and build data infrastructure, and with data scientists to optimize and tune it for performance. Dr. Rafat Hammad 19 19 DATA ENGINEERING LEARNING PATH (CONT.) 10) Cloud Computing-Cloud computing is a way to use technology to make it easier for businesses to work with large amounts of data. It allows businesses to store data in the cloud, which is a network of computers that can be accessed from anywhere in the world. Some key things to keep in mind include understanding how to use cloud-based data storage and processing services, as well as how to manage and monitor cloud-based data systems. Additionally, it is important to be familiar with the different types of cloud computing architectures and how they can be used to support data engineering workloads. Dr. Rafat Hammad 20 20 Dr. Rafat Hammad Data Engineering and Analytics TOPIC 2 : OUTLINE ❑ What is Data Engineering? ❑ Data Engineering Learning Path ❑ What is Data? ❑ Data Lifecycle Management ❑ Data Repositories Dr. Rafat Hammad 21 21 WHAT IS DATA? Data is defined as individual facts, such as numbers, words, measurements, observations or just descriptions of things. For example, data might include individual prices, weights, addresses, ages, names, temperatures, dates, or distances. There are two main types of data:  Quantitative data is provided in numerical form, like the weight, volume, or cost of an item.  Qualitative data is descriptive, but non-numerical, like the name, gender, or eye color of a person. Dr. Rafat Hammad 22 22 Dr. Rafat Hammad Data Engineering and Analytics CHARACTERISTICS OF DATA The following are six key characteristics of data which discussed below: 1) Accuracy 2) Validity 3) Reliability 4) Timeliness 5) Relevance 6) Completeness Dr. Rafat Hammad 23 23 CHARACTERISTICS OF DATA (CONT.) 1) Accuracy: Data should be sufficiently accurate for the intended use and should be captured only once, although it may have multiple uses. Data should be captured at the point of activity. 2) Validity: Data should be recorded and used in compliance with relevant requirements, including the correct application of any rules or definitions. This will ensure consistency between periods and with similar organizations, measuring what is intended to be measured. 3) Reliability: Data should reflect stable and consistent data collection processes across collection points and over time. Progress toward performance targets should reflect real changes rather than variations in data collection approaches or methods. Source data is clearly identified and readily available from manual, automated, or other systems and records. Dr. Rafat Hammad 24 24 Dr. Rafat Hammad Data Engineering and Analytics CHARACTERISTICS OF DATA (CONT.) 4) Timeliness: Data should be captured as quickly as possible after the event or activity and must be available for the intended use within a reasonable time. Data must be available quickly and frequently enough to support information needs and to influence service or management decisions. 5) Relevance: Data captured should be relevant to the purposes for which it is to be used. This will require a periodic review of requirements to reflect changing needs. 6) Completeness: Data requirements should be clearly specified based on the information needs of the organization and data collection processes matched to these requirements. Dr. Rafat Hammad 25 25 TYPES OF DIGITAL DATA Digital data is the electronic representation of information in a format or language that machines can read and understand. In more technical terms, Digital data is a binary format of information that's converted into a machine-readable digital format. The power of digital data is that any analog inputs, from very simple text documents to genome sequencing results, can be represented with the binary system. Types of Digital Data:  Structured  Unstructured  Semi-Structured Data Dr. Rafat Hammad 26 26 Dr. Rafat Hammad Data Engineering and Analytics TYPES OF DIGITAL DATA (CONT.) 1) Structured Data: Any data that are accessible and are stored or processed in the form of fixed-format is termed structured data. The employee table in the Database is an example of structured data. Banking transaction data is an example of structured data. The attributes present in structured data must be related to each other in some form. These data are stored in a relational database. 2) Unstructured Data: Irregular and ambiguous data, having no predefined data model and no pre-defined structure, are referred to as unstructured data. These data can be a combination of text, numbers, audio, video, images, messages, social media posts and many more. Twitter, Instagram, Facebook, and Google all are made up of unstructured data. Dr. Rafat Hammad 27 27 TYPES OF DIGITAL DATA (CONT.) 3) Semi-structured Data: These kinds of data falls between structured and unstructured data. It is a combination of partly structured data and partly unstructured data. For example, XML, and JSON are all semi- structured data. Dr. Rafat Hammad 28 28 Dr. Rafat Hammad Data Engineering and Analytics TYPES OF DIGITAL DATA (CONT.) XML (Extensible Markup Language) XML is a markup language with predefined rules for encoding data. It’s both human-readable and machine- readable, making it ideal for sharing data between systems. Dr. Rafat Hammad 29 29 TYPES OF DIGITAL DATA (CONT.) JSON (JavaScript Object Notation) JSON is a text-based format designed for transmitting data over the web. Its simplicity, compatibility with various programming languages, and ease of use make it a popular choice for data sharing. Dr. Rafat Hammad 30 30 Dr. Rafat Hammad Data Engineering and Analytics TYPES OF DIGITAL DATA (CONT.) Structured Unstructured Semi-structured Well organised data Not organised at all Partially organised It is less flexible and It is flexible and It is more flexible and difficult to scale. It is scalable. It is schema simpler to scale than schema dependent. independent. structured data but lesser than unstructured data. It is based on It is based on It is based on XML/ RDF relational database. character and binary data. Dr. Rafat Hammad 31 31 TYPES OF DIGITAL DATA (CONT.) Structured Unstructured Semi-structured Versioning over Versioning is like as a Versioning over tuples is tuples, row, tables whole data. possible. Easy analysis Difficult analysis Difficult analysis compared to structured data but easier when compared to unstructured data. Examples: Financial Examples: Media Examples: Tweets data, bar codes logs, videos, audios organised by hashtags; folder organised by topics Dr. Rafat Hammad 32 32 Dr. Rafat Hammad Data Engineering and Analytics TOPIC 2 : OUTLINE ❑ What is Data Engineering? ❑ Data Engineering Learning Path ❑ What is Data? ❑ Data Lifecycle Management ❑ Data Repositories Dr. Rafat Hammad 33 33 DATA LIFECYCLE MANAGEMENT (DLM) A data lifecycle refers to the stages that data goes through from its creation or acquisition, through its usage and maintenance, to its eventual disposal. The stages of the data lifecycle may vary depending on the organization, but they generally include the following: Dr. Rafat Hammad 34 34 Dr. Rafat Hammad Data Engineering and Analytics DATA LIFECYCLE MANAGEMENT (CONT.) Data Creation This is the first stage of data lifecycle. It refers to any input or source for generating data, including data acquisition, data capture, and data entry by applications, artificial intelligence (AI), machine learning (ML), and sensors. That said, not all data that is generated is collected and utilized. Your team should be able to identify what information should be captured, the best way of capturing the data, and what’s irrelevant or unnecessary to the project at hand. Dr. Rafat Hammad 35 35 DATA LIFECYCLE MANAGEMENT (CONT.) Data Storage When an organization generates large volumes of data from multiple sources, it is common for them to use a data warehouse to store the data and prepare it for use. The data stored in the data warehouse is cleaned and analyzed such that it can be used to make informed decisions. You should ensure that you store the data in a stable environment and properly maintain it to ensure its integrity, protection, and security. Dr. Rafat Hammad 36 36 Dr. Rafat Hammad Data Engineering and Analytics DATA LIFECYCLE MANAGEMENT (CONT.) Data Usage What value do you accrue from your data? How are you leveraging data analytics results? In this stage, you need to align value with action. How is data shared and used within your organization? You need to establish rules that define the management of the transfer and publication of data and who can access sensitive data Dr. Rafat Hammad 37 37 DATA LIFECYCLE MANAGEMENT (CONT.) Data Archival There are some data sets that can’t be destroyed immediately because they still have value from a compliance or historical perspective, and so they should be archived. The archived data is typically not active and is kept for long-term retention purposes. Most organizations leverage data warehousing capabilities for archived data that are rarely used for decision-making. They also use technology to retrieve such data if needed. Dr. Rafat Hammad 38 38 Dr. Rafat Hammad Data Engineering and Analytics DATA LIFECYCLE MANAGEMENT (CONT.) Data Destruction Keeping too much data increases the data management cost, thereby impacting the total cost of ownership and ROI (Return On Investment) of an organization’s products or services. While it’s mandatory to delete data at some point, you should also ensure that you free yourself by deleting active or archived data that doesn’t benefit your organization in any way. Dr. Rafat Hammad 39 39 DATA SOURCES: WHERE DATA RESIDES Data engineering starts with extracting data from various sources. Common data sources include: 1) Relational Databases Relational databases store structured data related to business activities, transactions, human resources, and more. Examples include SQL Server, Oracle, and MySQL, which are widely used for data analysis and projections. Dr. Rafat Hammad 40 40 Dr. Rafat Hammad Data Engineering and Analytics DATA SOURCES: WHERE DATA RESIDES (CONT.) 2) Flat Files and XML Datasets Flat files, such as CSVs, serve as public and private datasets, containing information like demographic data, financial records, or weather data. XML files offer flexibility for complex data structures, such as surveys or bank statements. 3) APIs and Web Services APIs (Application Programming Interfaces) and web services enable data retrieval through network requests. They play a vital role in applications like sentiment analysis, stock market analysis, and data validation. Dr. Rafat Hammad 41 41 DATA SOURCES: WHERE DATA RESIDES (CONT.) 4) Web Scraping Web scraping involves extracting data from unstructured sources on the internet. It can gather information ranging from text and contact details to images and product listings. Tools like Beautiful Soup and Scrapy facilitate this process. 5) Data Streams and Feeds Data streams aggregate real-time data from various sources, such as IoT devices, sensors, and social media. They are often geotagged and timestamped, supporting applications like stock market analysis and real- time event monitoring. Dr. Rafat Hammad 42 42 Dr. Rafat Hammad Data Engineering and Analytics LANGUAGES FOR DATA PROFESSIONALS Professionals in the data engineering field use various languages to accomplish their tasks. These languages fall into three main categories: 1) Query Languages Query languages, like SQL (Structured Query Language), allow users to access and manipulate data from relational databases. SQL is renowned for its simplicity and effectiveness in querying data Dr. Rafat Hammad 43 43 LANGUAGES FOR DATA PROFESSIONALS (CONT.) 2) Programming Languages Programming languages, including Python, R, and Java, enable the development and control of data engineering applications. Python stands out for its readability and extensive library support. 3) Shell Scripting Shell scripting languages like Unix and Linux are ideal for automating repetitive and time-consuming tasks. They are commonly used for file manipulation, system administration, and routine backups. Dr. Rafat Hammad 44 44 Dr. Rafat Hammad Data Engineering and Analytics TOPIC 2 : OUTLINE ❑ What is Data Engineering? ❑ Data Engineering Learning Path ❑ What is Data? ❑ Data Repositories Dr. Rafat Hammad 45 45 WHAT IS A DATA REPOSITORY? A data repository, also known as a data library or data archive, is a large database infrastructure that segments data sets for analysis, reporting, and distribution. These infrastructures function by gathering, storing, and managing data sets that form an open or restricted system. Anyone can access the original data sets or versions available within the repository and revise them in an open repository. In a restricted repository, a user requires specific qualifications to access, revise, or manipulate the data sets in the repository. Dr. Rafat Hammad 46 46 Dr. Rafat Hammad Data Engineering and Analytics TYPES OF DATA REPOSITORIES Here are some types of repositories you can use in your data storage, collection, and analysis:  1) Relational databases  2) Data warehouses  3) Data marts  4) Data lakes  5) Operational data stores  6) Data cubes  7) Metadata repositories Dr. Rafat Hammad 47 47 1) RELATIONAL DATABASES The relational database adopts the relational data representation model, which uses a straightforward and intuitive system to represent data sets in tables. This system stores data sets by their relation to one another and establishes a database that records data in rows with unique identifications known as the key. For example, you can use this system in your role in a sales department to access, create, store, and update a database of customer information. This database can include the customer's name, shipping information, and phone number in individual columns, while the system assigns a unique key to the row. Dr. Rafat Hammad 48 48 Dr. Rafat Hammad Data Engineering and Analytics 2) DATA WAREHOUSES This is a large data infrastructure or repository that incorporates data from multiple sources. This type of repository can give you access to data from multiple departments. It's best for conducting analysis and reports because it gives a consolidated view of logical and physical data from various systems. For example, this can help you find a product catalogue and the orders related to the product from different systems when making your report. Dr. Rafat Hammad 49 49 3) DATA MARTS This is a smaller data infrastructure that limits its scope to a specific department. Data marts serve as a quicker and more economical repository as it gathers information and data sets effectively within a single area of focus. This reduces the scope or extent of your analysis to a specific department, like finance, marketing, sales, or human relations. It's a useful way to make effective decisions within your department as a manager without losing the time larger data infrastructures require. Dr. Rafat Hammad 50 50 Dr. Rafat Hammad Data Engineering and Analytics 4) DATA LAKES This is a versatile data infrastructure that you can apply to various uses. You can use data lakes to store data sets you find difficult to categorize, including unstructured, semi-structured, and structured data set of any scale. This data infrastructure requires little maintenance and is relatively easy to set up because of its loose structure. The loose infrastructure enables you to apply this repository to tasks like machine learning, reporting, visualization, and analytics. Dr. Rafat Hammad 51 51 5) OPERATIONAL DATA STORES Operational data stores adopt the form of a central database to provide operational reports through an overview of timely data sets from multiple transactional systems. This form allows you to conduct insightful reports by combining data from various sources into a single source while maintaining their original format. This system provides you with the latest information available, integrates data sets from operational sources, and applies business intelligence tools to aid your decision-making process Dr. Rafat Hammad 52 52 Dr. Rafat Hammad Data Engineering and Analytics 6) DATA CUBES Data cubes are multidimensional data structures that store data in tables. This data structure provides you with a thorough view of data sets by portraying their interaction. These interactions involve the time sequence of data through multiple standpoints, including the particular attributes of data sets from daily, monthly, quarterly, or annual standpoints. For instance, you can use this data infrastructure to analyze the information relating to a customer from all perspectives. This gives you an in-depth understanding of your subject and helps you to identify trends. Dr. Rafat Hammad 53 53 7) METADATA REPOSITORIES This system deals with information or data sets relating to structures that contain the actual data. It also includes the data models and infrastructures that store and share data sets. These systems outline the methods you use to collect, categorize, and source data. You can use this system to understand the administrative structures within a group, such as your team or department, and update data structures to reflect changes in those groups. Dr. Rafat Hammad 54 54 Dr. Rafat Hammad Data Engineering and Analytics TIPS FOR USING DATA REPOSITORIES Consider applying these best practices when choosing and using a repository: 1) Use extract transformation load (ETL) tools: Using these tools when migrating data to the repository helps you maintain data quality during the transfer. This tool also allows you to integrate data from multiple sources into a consistent form on your repository. 2) Determine access: When introducing a repository to your team members, it's important to qualify the access every member has for specific aspects of the system. This helps preserve the sensitive nature of the data sets while providing every member with access to the information and data sets they require for their reports and analysis. Dr. Rafat Hammad 55 55 TIPS FOR USING DATA REPOSITORIES (CONT.) 3) Maintain flexibility: Flexibility is essential to repositories as it enables them to grow and develop along with your needs. Inflexible repositories can put your data sets at risk by failing to recognize changes to data sets or losing earlier versions of data after updates. 4) Versatility (‫)التنوع‬: Consider employing versatile repositories to incorporate multiple formats. This versatility also helps you accommodate multiple tools and features to secure your data and make user interactions more efficient. Dr. Rafat Hammad 56 56 Dr. Rafat Hammad Data Engineering and Analytics TIPS FOR USING DATA REPOSITORIES (CONT.) 5) Limit scope initially: It's important to test the applicability of a repository to your needs by initially limiting its scope. This limit helps you test the system's efficiency before improving the system's complexity by increasing the number of data subjects. 6) Automate functions: The ability to automate the functions of your repository is another important reason to ensure it has multiple features to accommodate your needs. By automating the functions of your repositories, you can improve your productivity while reducing errors. Dr. Rafat Hammad 57 57 THE END Dr. Rafat Hammad 58 58 Dr. Rafat Hammad