Introduction to Grid and Cloud Computing (University of Szeged) PDF

UNIVERSITAS SCIENTIARUM SZEGEDIENSIS UNIVERSITY OF SZEGED Department of Software Engineering Introduction to Grid and Cloud Computing Parallel Programming 2022/2023 I. Semester...

UNIVERSITAS SCIENTIARUM SZEGEDIENSIS UNIVERSITY OF SZEGED Department of Software Engineering Introduction to Grid and Cloud Computing Parallel Programming 2022/2023 I. Semester IoTCloud Research Group [email protected]  UNIVERSITY OF SZEGED Definition of Grids of Software Engineering A grid system consists of computers, storages, special instruments and services that UNIVERSITAS SCIENTIARUM SZEGEDIENSIS can be inter- and disconnected Department dynamically These resources are heterogeneous Grid Physically distributed and Internet connected through the Internet  To be accessed on- demand 2  UNIVERSITY OF SZEGED Why to use grids? of Software Engineering  Complex propblems can be solved by interoperating its resources to: UNIVERSITAS SCIENTIARUM SZEGEDIENSIS Reduce computation time Access big databases and special Department instruments Co-work with other researchers Internet 3  UNIVERSITY OF SZEGED When to use grids? of Software Engineering Once a scientific application runs on a single PC for: UNIVERSITAS SCIENTIARUM SZEGEDIENSIS days weeks Department months or even years. Such complex computational applications come from: climate modelling, gene technology, molecular modelling, other such as economic modelling. 4  UNIVERSITY OF SZEGED Why is it beneficial to use grids? of Software Engineering Former solution was to use expensive supercomputers UNIVERSITAS SCIENTIARUM SZEGEDIENSIS Grid systems have the advantage of: Department Unifying and exploiting the free capacities of a set of computers without additional investments Virtually enlarging the infrastructure of the involved institutes on-demand 5  UNIVERSITY OF SZEGED Grid application areas of Software Engineering High performance computing (HPC) To provide higher capacity than a UNIVERSITAS SCIENTIARUM SZEGEDIENSIS supercomputer or local cluster Department Requirement: parallel programming High throughput computing (HTC) To exploit free capacities of interconnected computers to execute a higher number of jobs within a time interval Collaborative work To solve problems with a group of experts 6  Example: UNIVERSITY OF SZEGED of Software Engineering Large Hadron Collider, CERN, Genf ATLAS CMS UNIVERSITAS SCIENTIARUM SZEGEDIENSIS Department ~10-15 PetaBytes /year ~108 events/year ~103 batch and interactive users LHCb 7  General Grid model UNIVERSITY OF SZEGED of Software Engineering Share free capacity UNIVERSITAS SCIENTIARUM SZEGEDIENSIS Inst1 Inst4 Department Internet Inst2 Inst3 Ask for additional capacity 8  UNIVERSITY OF SZEGED of Software Engineering Production Grid model Example: HunGrid and EGEE Sharing free capacity 24 hours/day (statically) UNIVERSITAS SCIENTIARUM SZEGEDIENSIS Institute 1 Institiute 2 Department Donor and User Donor and User Internet User1 User N On-demand access 9  The largest production Grid: EGEE UNIVERSITY OF SZEGED of Software Engineering Country participating in EGEE UNIVERSITAS SCIENTIARUM SZEGEDIENSIS Department Scale > 260 sites in 55 countries ~ 150 000 cores > 28 PB disk storage (+ 14 PB tape) > 330 000 jobs per day > 200 Virtual Organisations > 14 000 users  Desktop Grid model UNIVERSITY OF SZEGED of Software Engineering Offering free capacities on UNIVERSITAS SCIENTIARUM SZEGEDIENSIS Corporation demand Corporation / university Department / university server PC Application Internet Corporation Corporation / university / university server server Task distribution 11  UNIVERSITAS SCIENTIARUM SZEGEDIENSIS UNIVERSITY OF SZEGED Department of Software Engineering parameter study construct Internet Parallelism in production grids with the 12  UNIVERSITAS SCIENTIARUM SZEGEDIENSIS UNIVERSITY OF SZEGED Department of Software Engineering workflows Internet Two levels of parallelization with grid 13  EGEE Applications UNIVERSITY OF SZEGED of Software Engineering  >270 VOs from several scientific domains Astronomy & Astrophysics UNIVERSITAS SCIENTIARUM SZEGEDIENSIS Civil Protection Computational Chemistry Department Comp. Fluid Dynamics Computer Science/Tools Condensed Matter Physics Earth Sciences Fusion High Energy Physics Life Sciences  Further applications under evaluation Applications have moved from testing to routine and daily usage 14 ~80-95% efficiency Providing a Production Grid Infrastructure for Collaborative Science 14  UNIVERSITY OF SZEGED EGI platforms of Software Engineering Virtual Research Communities UNIVERSITAS SCIENTIARUM SZEGEDIENSIS EGI Collaboration platform Department Community Platform EGI Cloud Communit Infrastructure VM y Platform EGI Core Platform VM VM (gLite Grid) Infrastructure Platform Grid services EGI resources (cores & storage) 15  UNIVERSITY OF SZEGED Cloud infrastructure platform of Software Engineering Scientific portals (SaaS environments) UNIVERSITAS SCIENTIARUM SZEGEDIENSIS Community Scientifi Scientifi Scientifi specific, grid Department c portal c portal c portal enabled services Custom Community specific, grid enabled services Grid services middleware Grid middleware Hypervisor OS OS OS OS OS OS OS OS Hardware Hardware 16  UNIVERSITY OF SZEGED Cloud definitions of Software Engineering When a Cloud is made available in a pay- as-you-go manner to the public, we call it a UNIVERSITAS SCIENTIARUM SZEGEDIENSIS Public Cloud; Department The service being sold is Utility Computing. Current examples of public Utility Computing include: AmazonWeb Services, Google AppEngine, Microsoft Azure. 17  UNIVERSITY OF SZEGED Definitions of Software Engineering Definition by Buyya et. al.: UNIVERSITAS SCIENTIARUM SZEGEDIENSIS „A Cloud is a type of parallel and distributed system consisting of a collection of inter- Department connected and virtualized computers that are dynamically provisioned and presented as one or more unified computing resource(s) based on service-level agreements established through negotiation between the service provider and consumers.” 18  European Commission definition UNIVERSITY OF SZEGED of Software Engineering UNIVERSITAS SCIENTIARUM SZEGEDIENSIS Department A 'cloud' is an elastic execution environment of resources involving multiple stakeholders and providing a metered service at multiple granularities for a specified level of quality (of service). 19  UNIVERSITY OF SZEGED Characteristics of Software Engineering Virtual. UNIVERSITAS SCIENTIARUM SZEGEDIENSIS software, databases, Web servers, Department operating systems, storage and networking as virtual servers. On demand. add and subtract processors, memory, network bandwidth, storage. 20  UNIVERSITY OF SZEGED of Software Engineering Grid vs Clouds Cloud Computing Grid Computing UNIVERSITAS SCIENTIARUM SZEGEDIENSIS Platform Commodity node/network HW Custom node/network HW Department Environment Virtualized: Exact execution Library-based and environment can be created customized to HW, hard and cloned in the cloud, to ensure consistent arbitrary apps supported libraries across HW domains Resource HW resources can be Whole machine unit of allocation fractionally allocated, allocation maximizing utilization Quality of Service Only CPU-based QoS Strong CPU and I/O guarantee (some variation) performance guarantees Capacity “Infinite” resources Finite allocation of available resources  UNIVERSITAS SCIENTIARUM SZEGEDIENSIS UNIVERSITY OF SZEGED Department of Software Engineering Grids vs Clouds 22  UNIVERSITY OF SZEGED Cloud delivery models of Software Engineering UNIVERSITAS SCIENTIARUM SZEGEDIENSIS Software as a Service Department Platform as a Service Infrastructure as a Service *Michael Hogan, Fang Liu, Annie Sokol, Jin Tong, NIST Cloud Computing Standards Roadmap – Version 1.0, Special Publication 500-291, NIST Cloud Computing Standards Roadmap Working Group, July 5, 2011. 23  UNIVERSITY OF SZEGED of Software Engineering Cloud infrastructure deployment models Private Cloud Public Cloud UNIVERSITAS SCIENTIARUM SZEGEDIENSIS SP SP Department Community Cloud IP Hybrid Cloud IP SP SP IP1 IP1 IP1 IP2 IP3 24  IaaS utilization steps UNIVERSITY OF SZEGED of Software Engineering 1. Upload Virtual 4. Access Appliance Repository UNIVERSITAS SCIENTIARUM SZEGEDIENSIS Instantiation Support Libs VA VA Department + Service VA OS Environment 2. Delivery VA 3. Deployment VM VMM VA Host VMM VMM VMM VMM Host Host VMM VMM Host VMM Host VMM Host Host Host Host Infrastructure as a Service Cloud 25  UNIVERSITY OF SZEGED of Software Engineering Additional reading and references  Ian Foster and Carl Kesselman (Eds.). The Grid: Blueprint for a New Computing UNIVERSITAS SCIENTIARUM SZEGEDIENSIS Infrastructure. Morgan Kaufmann Publishers Inc., San Francisco, CA, USA. 1998.  EGI: http://www.egi.eu/, 2012.  EOSC: https://ec.europa.eu/research/openscience/index.cfm?pg=open-science- Department cloud, 2020.  R. Buyya, C. S. Yeo, S. Venugopal, J. Broberg, and I. Brandic: „Cloud computing and emerging it platforms: Vision, hype, and reality for delivering computing as the 5th utility”. Future Generation Computer Systems, vol. 25, no. 6, pp. 599-616, June 2009.  K. Jeffery and B. Neidecker-Lutz: „The Future of Cloud Computing, Opportunities for European Cloud Computing beyond 2010”. Expert Group Report, January 2010.  Michael Hogan, Fang Liu, Annie Sokol, Jin Tong, NIST Cloud Computing Standards Roadmap – Version 1.0,  Special Publication 500-291, NIST Cloud Computing Standards Roadmap Working Group, July 5, 2011. 26