02_devops.pdf

SOEN 6441 Lecture 2 Software Development Process Software Process and DevOps...

SOEN 6441 Lecture 2 Software Development Process Software Process and DevOps DevOps Why DevOps? What is DevOps? DevOps Practices SOEN 6441 DevOps Consequences Notes and Further Reading SOEN 6441 Department of Computer Science and Software Engineering Concordia University 2.1 Outline SOEN 6441 Software Development Process DevOps 1 Software Development Process Why DevOps? What is DevOps? DevOps Practices DevOps Consequences Notes and Further Reading 2 DevOps Why DevOps? What is DevOps? DevOps Practices DevOps Consequences 3 Notes and Further Reading 2.2  SOEN 6441 Main software engineering activities Software Development Process DevOps Why DevOps? What is DevOps? Requirements/ DevOps Practices DevOps Consequences Specification Design Notes and Further Reading Deployment Development/ Validation/Testing Implementation/Coding Integration Evolution/Maintenance 2.3  Requirement/Specification SOEN 6441 Software Development Process DevOps Software Elements Analysis Why DevOps? What is DevOps? – Extracting the requirements. Customers typically DevOps Practices DevOps Consequences know what they want, but not what software should Notes and Further Reading do. Demonstrating live code may help reduce the risk that the requirements are incorrect. Scope Analysis – An analysis of the scope of the development should be determined and clearly stated. Certain functionality may be out of scope of the development project as a function of cost or as a result of unclear requirements 2.4  SOEN 6441 Requirement/Specification Software Development Process DevOps Why DevOps? What is DevOps? Requirements DevOps Practices DevOps Consequences elicitation and Notes and Further analysis Reading Requirements specification Requirements validation User and System Requirements system document description requirements 2.5  SOEN 6441 Requirement/Specification Software Development Process DevOps Specification is the task of precisely describing Why DevOps? What is DevOps? DevOps Practices the software to be written. DevOps Consequences Notes and Further Reading Most successful specifications are written to understand and fine-tune applications that were already well-developed, although safety-critical software systems are often carefully specified prior to application development. Specifications are most important for external interfaces that must remain stable. 2.6  SOEN 6441 Design Software Development Process DevOps Why DevOps? What is DevOps? DevOps Practices DevOps Consequences Notes and Further Architectural Component Reading design design Persistence Interface design design 2.7  SOEN 6441 Validation/Testing Software Development Process DevOps Why DevOps? What is DevOps? DevOps Practices DevOps Consequences Notes and Further Reading Component System Customer testing testing testing 2.8  SOEN 6441 Validation/Testing Software Development Process DevOps Why DevOps? Component What is DevOps? DevOps Practices testing DevOps Consequences Notes and Further Reading Done by the person who writes the code Often considered as part of coding System testing Feature testing and performance testing Different levels of system testing Regression testing 2.9  SOEN 6441 Validation/Testing Software Development Process DevOps Why DevOps? Customer What is DevOps? DevOps Practices testing DevOps Consequences Notes and Further Reading Acceptance testing Field testing 2.10  SOEN 6441 Evolution/Maintenance Software Development Process DevOps Why DevOps? What is DevOps? DevOps Practices DevOps Consequences Propose Notes and Further Define System Modify system Reading requirements assessment system changes Existing system New system 2.11  SOEN 6441 Evolution/Maintenance Software Development Process DevOps Maintaining and enhancing software to Why DevOps? What is DevOps? DevOps Practices cope with newly discovered problems or DevOps Consequences Notes and Further new requirements can take far more time Reading than the initial development of the software. A small part of that is fixing bugs. Most maintenance is extending systems to do new things, which in many ways can be considered new work. 2.12  SOEN 6441 Other activities Software Development Process Coding DevOps Why DevOps? What is DevOps? – Reducing a design to code may be the most obvious DevOps Practices DevOps Consequences part of the software engineering job, but it is not Notes and Further Reading necessarily the largest portion. Deployment – After the code is appropriately tested and approved, it is moved into production environment i.e. is made available for business use. 2.13  SOEN 6441 Other activities Software Development Process Documentation DevOps Why DevOps? What is DevOps? – An important (and often overlooked) task is DevOps Practices DevOps Consequences documenting the internal design of software for the Notes and Further Reading purpose of future maintenance and enhancement. Documentation is most important for external interfaces. Software Training and Support – A large percentage of software projects fail because the developers fail to realize that it doesn't matter how much time and planning a development team puts into creating software if nobody in the client organization ends up using it. 2.14  SOEN 6441 Activities are connected to produce a software engineering process Software Development Process DevOps Why DevOps? What is DevOps? DevOps Practices DevOps Consequences Notes and Further Reading 2.15  SOEN 6441 Building vs. growing Software Development Process DevOps Why DevOps? What is DevOps? DevOps Practices Building software DevOps Consequences Notes and Further Reading The “building” metaphor: planning; specification as blueprint; components; assembly; scaffolding; etc. But the idea that planning preceded construction remained. Growing software rather than build it. Start with a very simple system that runs but has minimal functionality and then add to it and let it grow. 2.16 Process 0: The code-and-fix model SOEN 6441 Software Development Process DevOps Why DevOps? The basic model used in the earliest days What is DevOps? DevOps Practices DevOps Consequences of software development contained two Notes and Further Reading steps: 1. Write some code. 2. Fix the problems in the code. 2.17 Process 0: The code-and-fix model SOEN 6441 Software Development Process After a number of fixes, the code became so poorly DevOps Why DevOps? What is DevOps? structured that subsequent fixes were very expensive. DevOps Practices DevOps Consequences This underscored the need for a design phase prior to Notes and Further Reading coding. Frequently, even well-designed software was such a poor match to users’ needs that it was either rejected outright or expensively redeveloped. This made the need for a requirements phase prior to design evident. Code was expensive to fix because of poor preparation for testing and modification. This made it clear that explicit recognition of these phases, as well as test and evolution planning and preparation tasks in the early phases, were needed. 2.18  SOEN 6441 Process 1: Waterfall Software Development Process DevOps Why DevOps? What is DevOps? DevOps Practices DevOps Consequences Notes and Further Reading 2.19  SOEN 6441 Process 1: Waterfall Software Development Process DevOps Why DevOps? What is DevOps? Requirements DevOps Practices DevOps Consequences definition Notes and Further Reading System and software design Implementation and unit testing Integration and system testing Operations and maintenance 2.20  SOEN 6441 Pros of the Waterfall process Software Development Process DevOps Why DevOps? What is DevOps? DevOps Practices DevOps Consequences Notes and Further Reading Rigid and formal process, fits well for: -Embedded systems -Safety-critical systems 2.21  SOEN 6441 Cons of the Waterfall process Software Development Process DevOps Why DevOps? What is DevOps? DevOps Practices DevOps Consequences Notes and Further Reading Activities are isolated: -Late-changing requirements require a lot of rework! 2.22  SOEN 6441 Process 1.5: Prototype model Software Development Process DevOps Why DevOps? Requirements What is DevOps? DevOps Practices Gathering DevOps Consequences Notes and Further Build Prototype Reading Customer Customer Quick Design Evaluation of Design Prototype acceptance Refine Requirements Implement Test Maintainance 2.23  SOEN 6441 Process 2: Spiral model Software Development Process DevOps http://csse.usc.edu/TECHRPTS/1 Why DevOps? 988/usccse88-500/usccse88- What is DevOps? DevOps Practices 500.pdf DevOps Consequences Notes and Further Reading 2.24  Process 3: Incremental SOEN 6441 Development Software Development Process DevOps Why DevOps? What is DevOps? DevOps Practices DevOps Consequences Initial Notes and Further Specification version Reading Initial description Implementation Iteration versions Validation Final version 2.25  SOEN 6441 Process 3.5 Test-Driven development Software Development Process DevOps Why DevOps? What is DevOps? DevOps Practices DevOps Consequences Notes and Further Reading 2.26  Manifesto for Agile software SOEN 6441 development Software Development Process DevOps Why DevOps? What is DevOps? DevOps Practices DevOps Consequences Notes and Further Reading Individuals and interactions over processes and tools Working software over comprehensive documentation Customer collaboration over contract negotiation Responding to change over following a plan 2.27 Outline SOEN 6441 Software Development Process DevOps 1 Software Development Process Why DevOps? What is DevOps? DevOps Practices DevOps Consequences Notes and Further Reading 2 DevOps Why DevOps? What is DevOps? DevOps Practices DevOps Consequences 3 Notes and Further Reading 2.28  SOEN 6441 Why DevOps? Software Development Process DevOps Why DevOps? What is DevOps? DevOps Practices Developers and operators do not pursue DevOps Consequences Notes and Further Reading the same goals. 2.29  SOEN 6441 Software Development Software Development Process DevOps Why DevOps? What is DevOps? DevOps Practices DevOps Consequences Notes and Further Reading Design and specification Coding Testing Release engineering Evolution 3 2.30  SOEN 6441 Software Operation Software Development Process DevOps Why DevOps? What is DevOps? DevOps Practices DevOps Consequences Notes and Further Reading Monitoring Troubleshooting Capacity planning Anomaly detection Q&A Configuration Tuning 4 2.31 There is a gap between software SOEN 6441 developers and operators Software Development Process DevOps Why DevOps? What is DevOps? Does my system DevOps Practices DevOps Consequences perform well in Notes and Further Reading the field? Developers What does this error message mean? How do I resolve it? Operators 5 2.32  SOEN 6441 What problem is DevOps trying to solve? Software Development Process DevOps Why DevOps? What is DevOps? DevOps Practices DevOps Consequences Poor communication between Dev and Ops Notes and Further Reading Opposing goals Devs want to push new features Ops want to keep the system available Leads to slow release schedule Different cultures Limited capacity of operations staff Developers have limited insight into operations 2.33  SOEN 6441 Why companies care? Software Development Process DevOps Why DevOps? What is DevOps? DevOps Practices DevOps Consequences Notes and Further IBM Reading “IBM has gone from spending about 58% of its development resources on innovation to about 80%” http://devops.com/blogs/ibms-devops-journey/?utm_content=12855120 Paddy Power (Ireland): “The cycle time from a user story's conception to production has decreased from several months to 2 to 5 days. “Previously, approximately 30% of the workforce was fixing bugs. Now, bugs are so rare that the teams no longer need a bug-tracking system.” 2.34  SOEN 6441 DevOps motivation Software Development Process DevOps Organizations want to reduce time to market for Why DevOps? What is DevOps? new features, without sacrificing quality DevOps Practices DevOps Consequences Requires culture change & business-IT Notes and Further Reading alignment DevOps practices will influence... the way you organize teams the way you build systems even the structure of the systems that you build Unlikely that Devs are able to “throw their final version over the fence” and let operations worry about running it 2.35  SOEN 6441 What is DevOps? Software Development Process DevOps Why DevOps? DevOps is a set of practices intended to What is DevOps? reduce the time between committing a DevOps Practices DevOps Consequences change to a system and the change being Notes and Further Reading placed into normal production, while ensuring high quality. DevOps is the practice of operations and development engineers participating together in the entire service lifecycle, from design through the development process to production support. DevOps is also characterized by operations staff making use many of the same techniques as developers for their systems work. 2.36  SOEN 6441 What does that mean? Software Development Process DevOps Why DevOps? What is DevOps? DevOps Practices DevOps Consequences Quality of the code must be high Notes and Further Reading Testing & test-driven development Quality of the build & delivery mechanism must be high Automation & more testing A must when deploying to production 25x per day (etsy.com) Time is split: From commit until deployment to production From deployment until acceptance into normal production Means testing in production Achieving that starts before committing 2.37  SOEN 6441 DevOps Practices Software Development Process DevOps Why DevOps? What is DevOps? DevOps Practices DevOps Consequences Notes and Further Reading Treat Ops as first-class citizens throughout the lifecycle – e.g., in requirements elicitation Many decisions can make operating a system harder or easier Logging and monitoring to suit Ops Make Dev more responsible for relevant incident handling Shorten the time between finding and repairing errors 2.38  SOEN 6441 DevOps Practices Software Development Process DevOps Why DevOps? What is DevOps? DevOps Practices Use continuous deployment, automate everything DevOps Consequences Notes and Further Commits trigger automatic build, testing, deployment Reading Enforce deployment process is used by all No ad-hoc deployments Ensures changes are traceable Develop infrastructure code with the same set of practices as application code “Infrastructure as Code” : using IaaS APIs, etc., to automate creation of environments Misconfiguration can derail your application Ops scripts are traditionally more ad-hoc 2.39  SOEN 6441 DevOps Consequences Software Development Process DevOps Why DevOps? What is DevOps? Architecturally significant requirement: DevOps Practices DevOps Consequences Speed up deployment through minimizing Notes and Further Reading synchronous coordination among development teams. Synchronous coordination, like a meeting, adds time since it requires Ensuring that all parties are available Ensuring that all parties have the background to make the coordination productive Following up to decisions made during the meeting 2.40  SOEN 6441 DevOps Consequences Software Development Process DevOps Keep teams relatively small Why DevOps? What is DevOps? DevOps Practices Amazon’s “two pizza rule”: no team should be larger than DevOps Consequences can be fed with two pizzas Notes and Further Reading Advantages: make decisions quickly, less coordination overhead, more coherent units Team size becomes a major driver of the overall architecture: Small teams develop small services -> Microservices Coordination overhead is minimized by channeling most interaction through service interfaces: Team X provides service A, which is used by teams Y and Z If changes are needed, they are communicated, implemented, and added to the interface. 2.41  SOEN 6441 Continuous Deployment Pipeline Software Development Process DevOps Why DevOps? What is DevOps? DevOps Practices DevOps Consequences Notes and Further Reading Developer wants to commit code Pre-commit tests are executed locally. If successful: Code is committed Committed code is compiled, Unit tests are run. If successful: Code is built & packaged Result can be a machine image or template (assuming virtualization). If successful: Integration tests are run. If successful: Acceptance / performance tests are run. If successful: The new software/service is deployed to production 2.42 Outline SOEN 6441 Software Development Process DevOps 1 Software Development Process Why DevOps? What is DevOps? DevOps Practices DevOps Consequences Notes and Further Reading 2 DevOps Why DevOps? What is DevOps? DevOps Practices DevOps Consequences 3 Notes and Further Reading 2.43 Reading Material SOEN 6441 Software Development Process DevOps Why DevOps? What is DevOps? DevOps Practices DevOps Consequences Notes and Further Reading Supplemental [Som16, Chapters 2, 3] (Software Processes) 2.44 References SOEN 6441 Software Development Process DevOps Why DevOps? What is DevOps? DevOps Practices DevOps Consequences Notes and Further [Som16] Ian Sommerville. Reading Software Engineering. Pearson, 10th edition, 2016. http://software-engineering-book.com. 2.45