Introduction to Big Data PDF

Introduction to Big Data  What’s Big Data? No single definition; here is from Wikipedia: Big data is the term for a collection of data sets so large and complex that it becomes difficult to process using on- hand database management tools or traditional data processing applicati...

Introduction to Big Data  What’s Big Data? No single definition; here is from Wikipedia: Big data is the term for a collection of data sets so large and complex that it becomes difficult to process using on- hand database management tools or traditional data processing applications. The challenges include capture, curation, storage, search, sharing, transfer, analysis, and visualization. The trend to larger data sets is due to the additional information derivable from analysis of a single large set of related data, as compared to separate smaller sets with the same total amount of data, allowing correlations to be found to "spot business trends, determine quality of research, prevent diseases, link legal citations, combat crime, and determine real-time roadway traffic conditions.” 2 Big Data: 3V’s 3  Volume (Scale) Data Volume – 44x increase from 2009 2020 – From 0.8 zettabytes to 35zb Data volume is increasing exponentially Exponential increase in collected/generated data 4  4.6 30 billion RFID billion tags today 12+ TBs (1.3B in 2005) camera of tweet data phones every day world wide 100s of millions of GPS data every of enable ? TBs day d devices sold annually 25+ TBs of log data 2+ every day billion people on the 76 million smart Web by meters in 2009… end 200M by 2014 2011 Maximilien Brice, © CERN CERN’s Large Hydron Collider (LHC) generates 15 PB  The Earthscope The Earthscope is the world's largest science project. Designed to track North America's geological evolution, this observatory records data over 3.8 million square miles, amassing 67 terabytes of data. It analyzes seismic slips in the San Andreas fault, sure, but also the plume of magma underneath Yellowstone and much, much more. (http://www.msnbc.msn.com/id/44 363598/ns/technology_and_scienc e-future_of_technology/ #.TmetOdQ--uI)  Variety (Complexity) Relational Data (Tables/Transaction/Legacy Data) Text Data (Web) Semi-structured Data (XML) Graph Data – Social Network, Semantic Web (RDF), … Streaming Data – You can only scan the data once A single application can be generating/collecting many types of data Big Public Data (online, weather, finance, etc) To extract knowledge all these types of data need to linked together 8 A Single View to the Customer Banki Social ng Media Financ e Our Know Customer Gami n ng Histor y Entertai Purcha n se  Velocity (Speed) Data is begin generated fast and need to be processed fast Online Data Analytics Late decisions  missing opportunities Examples – E-Promotions: Based on your current location, your purchase history, what you like  send promotions right now for store next to you – Healthcare monitoring: sensors monitoring your activities and body  any abnormal measurements require immediate reaction 10  Real-time/Fast Data Mobile devices (tracking all objects all the time Social media and networksScientific instruments (all of us are generating data)(collecting all sorts of data) Sensor technology and networks (measuring all kinds of data) The progress and innovation is no longer hindered by the ability to collect data But, by the ability to manage, analyze, summarize, visualize, and discover knowledge from the collected data in a timely manner and in a scalable fashion 11  Real-Time Analytics/Decision Requirement Product Recommendations Learning why Customers Influence that are Relevant Behavior Switch to competitors & Compelling and their offers; in time to Counter Friend Invitations Improving the Customer to join a Marketing Game or Activity Effectiveness of a that expands Promotion while it business is still in Play Preventing Fraud as it is Occurring & preventing more proactively Some Make it 4V’s 13  Harnessing Big Data OLTP: Online Transaction Processing (DBMSs) OLAP: Online Analytical Processing (Data Warehousing) RTAP: Real-Time Analytics Processing (Big Data Architecture & technology) 14  The Model Has Changed… The Model of Generating/Consuming Data has Changed d Model: Few companies are generating data, all others are consuming data New Model: all of us are generating data, and all of us are consuming data 15 What’s driving Big Data - Optimizations and predictive analytics - Complex statistical analysis - All types of data, and many sources - Very large datasets - More of a real-time - Ad-hoc querying and reporting - Data mining techniques - Structured data, typical sources - Small to mid-size datasets 16  THE EVOLUTION OF Interactive BUSINESS INTELLIGENCE Business Intelligence & Speed In-memory Big Data: Scale RDBMS Real Time & BI Reporting Single View OLAP & QliqView, Tableau, Dataware house HANA Big Data: Graph Databases Business Objects, SAS, Batch Processing Informatica, Cognos Scale & Speed other SQL Reporting Tools Distributed Data Store Hadoop/Spark; HBase/Cassandra 1990’s 2000’s 2010’s  Big Data Analytics Big data is more real-time in nature than traditional DW applications Traditional DW architectures (e.g. Exadata, Teradata) are not well-suited for big data apps Shared nothing, massively parallel processing, scale out architectures are well-suited for big data apps 18 Big Data Technology 20  Cloud Computing IT resources provided as a service – Compute, storage, databases, queues Clouds leverage economies of scale of commodity hardware – Cheap storage, high bandwidth networks & multicore processors – Geographically distributed data centers Offerings from Microsoft, Amazon, Google, … wikipedia:Cloud Computing  Benefits Cost & management – Economies of scale, “out-sourced” resource management Reduced Time to deployment – Ease of assembly, works “out of the box” Scaling – On demand provisioning, co-locate data and compute Reliability – Massive, redundant, shared resources Sustainability – Hardware not owned  Types of Cloud Computing Public Cloud: Computing infrastructure is hosted at the vendor’s premises. Private Cloud: Computing architecture is dedicated to the customer and is not shared with other organisations. Hybrid Cloud: Organisations host some critical, secure applications in private clouds. The not so critical applications are hosted in the public cloud – Cloud bursting: the organisation uses its own infrastructure for normal usage, but cloud is used for peak loads. Community Cloud  Classification of Cloud Computing based on Service Provided Infrastructure as a service (IaaS) – Offering hardware related services using the principles of cloud computing. These could include storage services (database or disk storage) or virtual servers. – Amazon EC2, Amazon S3, Rackspace Cloud Servers and Flexiscale. Platform as a Service (PaaS) – Offering a development platform on the cloud. – Google’s Application Engine, Microsofts Azure, Salesforce.com’s force.com. Software as a service (SaaS) – Including a complete software offering on the cloud. Users can access a software application hosted by the cloud vendor on pay-per- use basis. This is a well-established sector. – Salesforce.coms’ offering in the online Customer Relationship Management (CRM) space, Googles gmail and Microsofts hotmail, Google docs. Infrastructure as a Service (IaaS)  More Refined Categorization Storage-as-a-service Database-as-a-service Information-as-a-service Process-as-a-service Application-as-a-service Platform-as-a-service Integration-as-a-service Security-as-a-service Management/ Governance-as-a-service Testing-as-a-service Infrastructure-as-a-service InfoWorld Cloud Computing Deep Dive  Key Ingredients in Cloud Computing Service-Oriented Architecture (SOA) Utility Computing (on demand) Virtualization (P2P Network) SAAS (Software As A Service) PAAS (Platform AS A Service) IAAS (Infrastructure AS A Servie) Web Services in Cloud  Enabling Technology: Virtualization App App App App App App OS OS OS Operating System Hypervisor Hardware Hardware Traditional Stack Virtualized Stack  Everything as a Service Utility computing = Infrastructure as a Service (IaaS) – Why buy machines when you can rent cycles? – Examples: Amazon’s EC2, Rackspace Platform as a Service (PaaS) – Give me nice API and take care of the maintenance, upgrades, … – Example: Google App Engine Software as a Service (SaaS) – Just run it for me! – Example: Gmail, Salesforce  Cloud versus cloud Amazon Elastic Compute Cloud Google App Engine Microsoft Azure GoGrid AppNexus  The Obligatory Timeline Slide (Mike Culver @ AWS) COBOL, Amazon.com Edsel ARPANET Internet Web Web as a Web Services, Darkness Awareness Platform Resources Eliminated 9 2 9 19 6 8 9 6 99 7 0 1 0 4 0 6 1 95 19 9 1 1 20 20 20 Dot-Com Web 2.0 Web Scale Bubble Computing  AWS Elastic Compute Cloud – EC2 (IaaS) Simple Storage Service – S3 (IaaS) Elastic Block Storage – EBS (IaaS) SimpleDB (SDB) (PaaS) Simple Queue Service – SQS (PaaS) CloudFront (S3 based Content Delivery Network – PaaS) Consistent AWS Web Services API What does Azure platform offer to developers? Your Applications Service Databas … Workflow Analytics Identity Contacts Bus e Access Reportin … … Devices … Control g Compute Storage Manage …  Google’s AppEngine vs Amazon’s Python EC2 BigTable Other API’s VMs Flat File Storage AppEngine: EC2/S3: Higher-level functionality Lower-level functionality (e.g., automatic scaling) More flexible More restrictive Coarser billing model (e.g., respond to URL only) Proprietary lock-in June 3, 2008 Google AppEngine vs. Amazo Slide 35 n EC2/S3