DevOps Software Engineering PDF

SWEN_CP901 - LY_KDVOP_C: DevOps Software Engineering Recorded class in lieu of class. Software Engineering R.G. Lennon Recorded class in lieu of class. Contents Review what we know at this point See where we want to go Software Engineering Software engineering is the...

SWEN_CP901 - LY_KDVOP_C: DevOps Software Engineering Recorded class in lieu of class. Software Engineering R.G. Lennon Recorded class in lieu of class. Contents Review what we know at this point See where we want to go Software Engineering Software engineering is the application of engineering approaches to the development of software which includes the design, development, deployment and maintenance of software. Computer science is the study of the processes that interact with data through the form of programs. CS focuses on the broad concepts or theory and uses algorithms or technology to come up with new ideas. SE focuses on solving specific (business) problems using the practical or implementation of a solution.  Unlimited Programming Programming is a tool for all business areas. Data analytics for wafer failures in manufacturing Medical Health Welding maps, traceability of welding and materials used Computer pattern sewing machines Marine Archaeology https://www.forbes.com/sites/quora/2017/11/01/how-will-an-increasing-supply-of-software-engineers-affect-demand-for-their-services/#1d5c39972086 Programming Languages Scripting or light weight programming languages Templates for accessible programming Languages suitable to the specific business need https://www.statista.com/statistics/793628/worldwide-developer-survey-most-used-languages/ Evolution of lifespan Projects have a finite lifespan Start at a specific date to solve a specific problem or create a specific feature Projects end with the timeline or budget allowed for the task Products evolve over time Products may involve a number of projects as the product evolves over time Products focus on high level strategies Reduce, Reuse, Recycle Modularity – simplify code and focus on a specific function for the code Cohesion –high cohesion to ensure elements in a module are specific to the module Coupling – loose/low coupling to focus on a specific function and aid portability Unit tests – create low level tests which are also reusable  Later in the course we will see how describing required functionality as user stories maps nicely under the INVEST properties. User stories should be: Independent Negotiable Valuable Estimable Small Testable  Other Synergies In this course we look at modern programming paradigms, methodologies and frameworks. We look at how modern techniques can help improve the Software Development Life Cycle (SDLC) https://www.agilealliance.org/agile101/subway-map-to-agile-practices/  The Traditional Software Development Life Cycle © R.G. Lennon Software projects often fail 1 in 6 IT projects have an average cost overrun of 200% and a schedule overrun of 70% 75% of IT executives believe their projects are “doomed from the start” Understanding software requirements is a perceived reason for a number of project failures. http://blog.capterra.com/surprising- project-management-statistics/ Requirements When an new product is required a list of requirements are created. Requirements Specification is the document which describes the requirements. Requirements are often gathered by a business analyst. Prototyping to elicit requirements is popular as it aids with User Acceptance Testing. Dynamic tests are used by Extreme Programmers to elect requirements. It is considered best to combine multiple techniques to elicit requirements.  Poor requirements elicitation isn’t the only reason for project failure. https://www.betabre akers.com/software- survival-in-2024- understanding-2023- project-failure- statistics-and-the- role-of-quality- assurance/ http://stevedempsen.blogspot.i e/2013/08/agile-software- requirements-comic.html Why Software Projects Fail “Among the most common factors: Unrealistic or unarticulated project goals Inaccurate estimates of needed resources Badly defined system requirements Poor reporting of the project's status Unmanaged risks Poor communication among customers, developers, and users Use of immature technology Inability to handle the project's complexity Sloppy development practices Poor project management http://spectrum.ieee.org/computing/software/why-software- Stakeholder politics fails https://www.betabreakers.com/software-survival-in-2024- Commercial pressures” understanding-2023-project-failure-statistics-and-the-role-of- quality-assurance/ SDLC The SDLC is the overall process of developing, implementing and retiring information systems through a multistep process from initiation, analysis, design, implementation and maintenance to disposal – NIST The SDLC Methodology provides a framework of principles, practices and procedures to guide the systems development process. There are many different methodologies which are broadly categorised as SDLC. Improving the SDLC is the single largest aid to improving project success. Traditional SDLC Phases Planning Legacy 5 Phases System Investigation Requirements System Analysis Design Design Environments Implementation Implementation Testing Testing Maintenance Release Training & Transition Operations and Maintenance Evaluation Destruction Planning Elicitation of Requirements for the new system Investigation of how the system currently runs This step refers to why and how a system should be built The two steps in planning are: Determination of the projects Business Value to the company during the feasibility stage Creation of the project plan including a list of required resources Approaches: Top-Down: long term info systems strategic plan Bottom-Up: Process mangers request change Systems Analysis Determines in more detail the basic Who, What, Why, When, Where and How of the system This requires Review of the current system in existence where applicable Detailed requirements elicitation System Modelling The requirements should then be refined and prioritized Models may be created including: process model, business logic model, data model Gathering requirements can be difficult.  Analysis should consider the PIECES framework: Performance analysis Information and data analysis Economic analysis Control and security analysis Efficiency analysis Service analysis A sample checklist for identifying problems using the PIECES Framework can be found http://www.cs.toronto.edu/~sme/CSC340F/readings/PIECES.html Note however that it does not mention security! Design There are many aspects to design including: Architectural Design: hardware, software and other infrastructure, process models, functional model, framework model Interface Design: wireframes - Balsamiq, forms, reports or places of interaction with the system Data Design: methods of data storage, logical data structure, data relationships and data access Program Design: coding elements of the system to support the user requirements Implementation Implementation of the system This phase often overlaps with the testing phase particularly in new development methodologies Strategy for implementation of tasks There are 4 types of conversion from an old system to a new system: Parallel: old and new are run in parallel while the new system is tested Pilot: a small group run the new system until it is concluded to be fully working Phased: the move to the new system happens in phases until completion Plunge/Big Bang: the entire group are moved to the new system at the same time Testing Depending on the methodology testing may occur once or may occur repeatedly throughout the development lifecycle. Test Plan include plan for access to test infrastructure Test Data Test automation Exploratory tests Tests for functional and non-functional (QoS) requirements https://msdn.microsoft.com /en- us/library/jj159342.aspx Release This may include installation on customer equipment Version Management Training & Transitioning between products or versions Customer support Operations and Maintenance Corrective maintenance Adaptive maintenance Evaluation Methodology A methodology is a formal approach consisting of a series of steps to be followed in the SDLC Methodologies specify how a project is broken down via time, tasks and outputs. The methodology should enable the stakeholders to be detailed along with management procedures and tools The methodology should provide an area for constraints to be specified. Traditional Methodologies Waterfall Sequential development through numerous phases. Documented deliverables are produced at phase termination Specialist teams are assigned specific tasks Requirements are identified before the project begins The time between system proposal and final release is significant Validation (V-Model) Also sequential development through phases but in a V shape. It is essentially the waterfall model with testing phases for each development phase. This works best with short projects where requirements are well understood in advance  BRS: Business Requirements Specification SRS: Software Requirements Specification TDD: Test Driven Development SCD: System Context Diagram http://www.softwaretestinggenius.com/software- development-models-at-a-glance  Prototyping Analysis, design and implementation phases are run concurrently All phases are repeated producing up-to-date prototypes until the system is complete Evolutionary prototyping is where the prototype is refined and improved to produce the final system Throw-away prototyping is where small prototypes are created to aid requirements elicitation Extreme Programming (XP) An agile method where new versions of software may be produced daily Increments are release to customers in short sprints The focus is on continuous testing and validation by the end user Other Agile methods are discussed in a separate handout. Martin Fowler on Methodologies Read the following: http://www.martinfowler.com/articles/newMethodology.html Summary The SDLC defines the software development life cycle There are a number of phases which have changed with the changing lifespan of software There are many methodologies which can be used to improve upon traditional approaches to the SDLC DevOps and the SDLC ©R.G.Lennon SDLC In traditional software development there was one release of software at the end of the lifecycle. Modern techniques for software development involves multiple releases of software over a period of time. Software evolves over time and lives longer. Agile Manifesto Individuals and Interactions over Processes and Tools Working Software over Comprehensive Documentation Customer Collaboration over Contract Negotiation Responding to Change over following a Plan This manifesto aims to make software development more interesting by focusing on non-repetitive tasks. What is DevOps? DevOps in its simplest form is the enhanced communication of Development and Operations during the production of software. Gartner’s Definition “DevOps represents a change in IT culture, focusing on rapid IT service delivery through the adoption of agile, lean practices in the context of a system-oriented approach. DevOps emphasizes people (and culture), and seeks to improve collaboration between operations and development teams. DevOps implementations utilize technology — especially automation tools that can leverage an increasingly programmable and dynamic infrastructure from a life cycle perspective.” Gartner’s Definition “DevOps represents a change in IT culture, focusing on rapid IT service delivery through the adoption of agile, lean practices in the context of a system-oriented approach. DevOps emphasizes people (and culture), and seeks to improve collaboration between operations and development teams. DevOps implementations utilize technology — especially automation tools that can leverage an increasingly programmable and dynamic infrastructure from a life cycle perspective.” Amazon’s Definition (w.r.t AWS) “DevOps is the combination of cultural philosophies, practices, and tools that increases an organization’s ability to deliver applications and services at high velocity: evolving and improving products at a faster pace than organizations using traditional software development and infrastructure management processes” Amazon’s Definition (w.r.t AWS) “DevOps is the combination of cultural philosophies, practices, and tools that increases an organization’s ability to deliver applications and services at high velocity: evolving and improving products at a faster pace than organizations using traditional software development and infrastructure management processes” Puppet “DevOps is a set of principles and practices — both technical and cultural — that can help organizations deploy better software, faster. While it is not a fixed methodology, automation and a collaborative culture are the foundation for accepted DevOps practices, which include: Measuring the things that have an impact on your organization’s goals Making those measurements visible to everyone Using a shared set of tools Including all teams in the software delivery process from the earliest planning stages.” Puppet “DevOps is a set of principles and practices — both technical and cultural — that can help organizations deploy better software, faster. While it is not a fixed methodology, automation and a collaborative culture are the foundation for accepted DevOps practices, which include: Measuring the things that have an impact on your organization’s goals Making those measurements visible to everyone Using a shared set of tools Including all teams in the software delivery process from the earliest planning stages.” Culture, Automation, Measurement and Sharing (CAMS) Change of culture referred to as ‘Left-Shift’ – consider quality earlier Implementation of automation to speed up the process thereby ensuring more frequent software releases. Stakeholder feedback on the quality of the product at frequent iterations during development.. Frequent Releases. Continuous release means that we have higher visibility of the product along with enabling the customer to indicate where changes are needed early in the cycle. Three Ways Systems Thinking and Flow. The entire system must be considered. Fixing a single blockage is not helpful if there are blockages elsewhere. Amplify Feedback Loops. Having short feedback loops where information is disseminated to all relevant stake holders means that change can be acted upon efficiently. Continuous experimentation & Learning. Processes must encourage and reward experimentation as this may lead to ‘out of the box’ thinking. This will encourage cultural change. Additional Reading. Trends 2021 “In a Covid world, organizations are doing DevOps in some shape or form, either by choice or by necessity. However, is most cases, DevOps is done in silos and by a separate team, which is quite an anti- pattern in itself.” Madhwesh Kulkarni, SVP and Global Head, Technology Transformation Group, Zensar Technologies “DevOps teams will start embracing smarter tools and technologies that involve AI and ML across the entire lifecycle, across code creation, code reviews, tests, observability and big data analysis.” Eran Kinsbruner, Chief Evangelist, Perfecto(link is external) at Perforce Software https://www.devopsdigest.com/2021-devops-predictions-1 DevOps Trends 2024 There are many new trends and companies vary on what they say they are. https://codesuite.org/blogs/ top-devops-trends- predictions-in-2024/ DevOps and AIOps ©R.G.Lennon AIOps The integration of Artificial Intelligence into DevOps concepts is a strong concept for this year. Where quality management uses metrics to measure quality, AIOps analyses the information further to show patterns. Image (c) Devops.com https://devops.com/devops-and-aiops-better-together/ Observability Logging creates a data set of tasks that have happened, events that have occurred and so on. Monitoring allows teams to be alerted when an issue occurs so that they can respond to the issue in a timely manner. Observability is where systems are instrumented so that further investigation into why the issue occurred can take place. Observability also enables prediction of future issues. AIOps enables the correlation of data and alerts so that further contextual information can be reviewed.  AIOps enables notifications and predictive information to be communicated to stakeholders across silos. It enables data drive actions to occur across complex systems where information might not have been previously visible. AIOps aids scalability of visibility across systems. AIOps relies heaviliy on Machine Learning. Further Reading Read both of these articles to get a sense of how DataOps evolved into AIOps to provide added value to DevOps. https://www.bmc.com/blogs/what-is-aiops/ https://www.capgemini.com/2019/08/aiops-a-myth-or-a-reality/  What did you know before? What do you know now? What do you want to know more about?