CS5351 Software Engineering 2024-2025 Semester A Software Process Model PDF

CS5351 Software Engineering 2024-2025 Semester A Software Process Model Dr W.K. Chan Department of Computer Science Email: [email protected] Website: http://www.cs.cityu.edu.hk/~wkchan 1 Motivation  A group of strong and committed programmer...

CS5351 Software Engineering 2024-2025 Semester A Software Process Model Dr W.K. Chan Department of Computer Science Email: [email protected] Website: http://www.cs.cityu.edu.hk/~wkchan 1 Motivation  A group of strong and committed programmers (say G = {Programmer A, Programmer B}) can hack on a code project to deliver an application to their users. A knows the code P and its responsibility extremely well B knows the code Q and its responsibility extremely well A can deliver the code P, and B can deliver the code Q A meets B’s needs on P; B meets A’s needs on Q A and B can check with users U to get and verify the exact user requirements whenever necessary  A perfect case. 2 Motivation (in Reality)  A group of average, sometimes emotional, programmers (say G = {Programmer A, Programmer B}) can hack on a code project to deliver an application to their users. A knows the code P and its responsibility gradually B knows the code Q and its responsibility gradually A can deliver a buggy P with incomplete functionality, and B can deliver a buggy Q with incomplete functionality A may not know B’s exact needs on P; B may not know A’s exact needs on Q. As B only knows Q gradually, B’s requirement on P changes over time. A and B can check with users U to get and verify the exact user requirements sometimes, but an old U resigns, and a replacement U comes! Some Us are inexperience in their business domain. 3 Motivation (For Larger Project In Reality) [1/3] Suppose in Month 1, the group G = {Programmer A, Programmer B}  Month 2, G = {Programmer B}, A resigned.  Month 3, G = {Programmer A1, Programmer B}, A1 replaces A.  Month 4, G = {Programmer A1}, B resigned.  Month 5, G = {Programmer A1, Programmer B1}, B1 replaces B.  Month 6, G = {Programmer B1}, A1 resigned.  Month 7, G = {Programmer A2, Programmer B1}, A2 replaces A1. Analysis  At Month 7, A2 gets the needs on code P via AB A1 B1A2 To what extent can A2 get what to be done, and what options have been tried on P? Motivation (For Larger Project In Reality) [2/3]  A2 wishes to know The requirement R that A collected from U (months ago!) The latest changes in R before A2 takes up the position The set of functions {f1, …, fn} of P that has been identified and designed but not been implemented (in full) with respect to the latest requirements The feedback (from B or U) on the implemented functions {g1,…,gm} of P with respect to the latest requirements Any known and critical issues in P but not completely fixed  B2 wishes to minimize his/her workload to pass on the information of P to A2 A1 also wishes to minimize the workload to pass info of Q to B2 5 Motivation (For Larger Project In Reality) [3/3]  U does not want A, A1, A2 and B, B1 to affect U’s daily jobs. Entertaining these A/B is a side business only. Can you expect an estate agent as U to meet with A-A2 and B-B1 instead of meeting with his/her clients (thus, no commissions are received, or a lower salary is the result)?  Everyone wishes to get quick references and move forwards  When a project is large, no one knows all the details of the whole project. Waiting for clear instructions and clear task specifications before doing the task is unrealistic. 6 Needs  The software project scenario in the previous few slides needs a strategy to control the order of project tasks and the information flow to reduce production overheads, improve information passing efficiency, give tractability from user expectation to delivery, predict and track task/project completion and outstanding issues, improve code quality, track what has been agreed not to design or implement, enable newcomers to be productive as soon as possible, and …  Otherwise….[next slide] 7 Bugs and Delays are Ubiquitous: 8 Bugs and Delays are Ubiquitous: 9 Bugs and Delays are Ubiquitous: Airport Chaos and System Upgrade  Hong Kong Airport https://www.scmp.com 10 MTR 11 Facts (quite some years ago): Software Development Project cancelled before completed on completion. 23% 28% time and within cost 89% budget. 49% more than But, only 42% of the their original original requirements budget can be implemented. 12  Problems Nothing can be useful or reused How to identify a from such cancelled projects? subset of Facts: Software Development Project requirements, cancelled before completed on which are completion. time and consistent among within them, and track cost 89% budget. their development more than But, only 42% of the their original original requirements budget can be implemented. How to streamline the Need to prioritize the 5 requirements for guide activities better to reduce the development effort cost? More Recent Findings  Standish Group’s Annual CHAOS 2020 report 66% of projects end in partial/total failure. 31% canceled; 53% with performance degrade Large project: success rate 10%; Small project: success rate 90%  McKinsey in 2020 17% of large projects go badly  BCG (2020) 70% of digital transformation efforts fall short of meeting targets. 14 Recent Situation Project Failure in 2018 https://www.cio.com  “The PMI research also identifies a number of factors common to the Champion organizations, says Langley. Besides a focus on benefits realization, champion organizations are more likely to focus on project management talent, have at least one PMO [project management office] within their organization, drive executive sponsorship and use agile approaches to project management, the research revealed.” 15  https://hbr.org/2023/10/its-time-to-end-the-battle-between-waterfall-and-agile https://www.engprax.com/post/268-higher-failure-rates-for-agile-software-projects-study-finds More Recent Findings  Harvard Business Review (2023) A hybrid of waterfall and agile methods is better  Impact Engineering (2024) 65% of projects adopting Agile requirement engineering practices fail to deliver on time. But if a more structural and “waterfall”- style requirement engineering practice is in place, the failure rate for on-time delivery is 10%. 16 Software Engineering Process A software engineering process model is a set of (1) activities, (2) techniques, (3) deliverables and (4) tools so that software engineers and project managers follow (5) best practices to use the items (1)(4) to manage software systems 17 SE Process  Activities  Collecting user requirements  Designing software  Coding/Implementation  Testing Let’s recall what  Deploying the software at they are with user sites examples on the  Maintain software next few slides.  Configuration Management  Project Management 18 SE Process  Activities Exemplified Software Requirements User Requirements Non-functional Requirements 19 SE Process  Activities Exemplified Software Architecture 20  SE Process  Activities Exemplified Software Designs Static view: Dynamic view: Component diagram Class diagram sequence diagram Database scheme User interface Design State transition diagram 21 SE Process  Activities Exemplified Software Design Patterns observer pattern visitor pattern 22 SE Process  Activities Exemplified Coding with Qulaity better coding 23 SE Process  Activities Exemplified Testing Unit testing 24 SE Process  Activities Exemplified Software Deployment Deployment diagram: Map software component to run on hardware device (or hardware virtualization) Write scripts or use a tool to publish it 25 SE Process  Activities Exemplified Software Maintenance 26  SE Process  Activities Exemplified Configuration Mgt Process Code version control 27 SE Process  Activities Exemplified Project Management Courtesy https://www.projectmanager.com/gantt-chart PMBok Gantt Chart Agile-based task board 28 SE Process  techniques Example  Collecting user  Use cases/user stories/use case requirements diagram/meeting  Designing software  UML/ patterns/ principles/ tactics  Coding/Implementation  Java / C++ /framework/platform  Testing  Unit test framework/debugger / …  Deploying the software at  Standalone software/Software user sites plug-in/app/web services/…  Maintain software  Bug reporting/software repository  Configuration  Version control/change Management management software  Project Management  Work breakdown structure/ task scheduling algorithm 29  SE Process  deliverables Example  Use cases / user stories / use case  Requirement document diagram  UML/ patterns/ principles/…  Software design document  Java / C++/…  Code listing and test script  Unit testing / debugger /…  Quality assurance (QA) report  Software plug-in/applet architecture  Software at user site  Bug reporting/software repository  Bug report and s/w release  Version control / change  Code change / patch / management software change history report  Work breakdown structure/ task  Project schedule and status scheduling algorithm tracking 30  SE Process  tools Example  Use cases / user stories / use case  UML tool / Requirement diagram modeling tool / Word  UML/ patterns/ principles/…  UML tool / SonarQube  Java / C++/…  Visual Studio/Eclipse  Unit testing / debugger /…  JUnit/ debugger /fuzzer …  Software plug-in/app/web services  Installer/plug-in framework  Bug reporting/software repository  Bug reporting system  Version control / change  Apache subversion / git management software  MS Project/Scrum tool  Work breakdown structure/ task scheduling algorithm 31 SE Process  Best Practices  Different ordering of activities defines different types of software processes  E.g., [see the details in the next few slides] Waterfall model 1960s V-shape waterfall model Incremental software development model Prototyping 1980s Spiral model Unified Process (UP) 2000s Agile Methods 2010s Hybrid of Agile Methods with Waterfall 2024 32 Many Kinds of Agile Methods All 10 are agile methods. 33 1-Minute Q&A  Question 1: On the last slide, which agile methods include the project management activities? Hint: If an agile method requires fewer specific tools, activities, process steps, and intermediate products, then the agile method is more lightweight.  Question 2: Are there any agile methods, both lightweight and including project management activities? Name them 34 Extreme Programming vs Scrum  XP has no PM.  Scum has PM (but a less tedious process than XP) and other practices/activities/work products 35 Brief Look on Process Models 36  Waterfall Model  Completely produce the full set of deliverables of each activity before starting the next activity. Requirements Design Coding / Implementation + Simple workflow Testing  Need test against design, coding details, Deployment requirements at the same Maintenance time. 37 Waterfall Method  Sequential Complete one phase nicely before the start of the next phase.  Detailed documentation Facilitate everyone comprehensively understanding each phase.  Predictability in cost and timeline through upfront and high-quality planning  Rigidity and Incompatibility with change Once confirmed, no change  Late discovery of issues  Unsuitability for undefined projects 38  V-Shape Waterfall Model Testing User validation target Acceptance Requirements Test Design validation target System Testing validation target Unit Testing / Coding / Integration Implementation Testing Software Build Deployment Maintenance + Stagewise Validation Goal 39 Problems of “Waterfall Model” and “V-Shape Waterfall Model” 1. False sense of clear-cut phases of activities. E.g., Can we completely work out the actual requirement before we design our software? It is unrealistic for real-world problems. It is also very costly to correct the problems made in upstream activities. [see the next slide] 2. Nothing is done until they are all done E.g., If we end the project at the design phase, then there is no any code delivered to the project clients, but our clients want to have something executable to help their own business! 40 Lesson Learnt: Cost of Fixing Bugs at Earlier Stage is Cheaper Costly An artifact that can be validated should be validated as Cheap early as possible? 41  Implication on following the Waterfall Model We would rather have a gracefully degraded subsystem a partial system, a system of lower quality, instead of no system at the end! Facts: Software Development Project An artifact of an early release cancelled before completed on subsystem that completion. time and can be validated within should be budget. cost 89% validated as more than But, only 42% of the early as their original original requirements possible? How? budget can be implemented. 7 42  Software Process Improvement Current Trends (1)  Some stages (activities) can be done in parallel  Example: Run technical development and user training in parallel Development and QA within each parallel track. Each parallel track fits within a self-contained subsystem. Quicker for each parallel track to move forward, thereby earlier to rectify bugs  lower cost 43  Software Process Improvement Current Trends (2)  Backward Iterations Example: Requirement Design Feedback Notice the direction of the arrows. Improve earlier stages whenever necessary Avoid unnecessary rework or design bugs by working out the more 44 accurate requirements V-Shape Waterfall Model Testing User Acceptance Requirements Test System Testing Design Unit Testing / Coding / Integration Implementation Testing Software Build Deployment Backward transition from Maintenance a later phase to an earlier phase. 45  Prototyping = Code before design and Prototyping requirement engineering. Incrementally obtain and validate the user/system requirements before a proper software design stage. Oral software prototype released to users A prototype is a Requirements preliminary version of Coding the final product: buggy with features Testing partially implemented or only with mockup Documentation screens so that users (Design & Requirement) can point out their Deployment requirements more Backward transitions easily than expressing them more abstractly. exist but are not shown Maintenance 46 here for clarity. Incremental Development  Rather than using a parallel track to work on a smaller subsystem, another way is to deliver a smaller set of requirements and gradually enlarge the requirement set.  Divide the set of requirements into subsets. Implementing one subset to construct a program version v1 then Implementing another subset on top of v1 to construct version v2 then Implementing yet another subset on top of v2 to construct version v3 then … … 47  Incremental Development Requirement subset 1 We may optionally deploy a version to Design 1 user here! Coding / Implementation 1 Testing 1 Requirement subset 2 Design 2 Maintenance Coding / Implementation Parallel tracks with each 2 Deployment stage of each subsystem Testing 2 development is possible …. 48 Spiral Model  Contains characteristics of the waterfall, prototype, and incremental development, but is more systematic.  Basic idea: plan, do prototyping, and revise the previous plans until we know the item X well through a series of iterations. 49 Step 1) Determine objectives, Step 2) Evaluate alternatives, alternatives, and constraints identify risks, and resolve risks Prototype2 Initial plan Prototype1 4 3 2 1 Detailed De sign Requirements, n Devel o pm e esig Code nt Pla D n Un it Tes t Step 4) Plan next level Sy ste Ac m Implem ning ce Te pta st Plan cen entatio Step 3) Develop, Te st Spiral Software Life Cycle validate, and verify n next-level  Software Process Improvement Current Trends Model-Based Domain Model  Sale 1 1..* Sales LineItem... date... Software... quantity the domain objects, attributes, and domain objects associations that undergo state changes Use-Case Model Engineering Process Sale 1. Customer : Cashier make NewSale() : System Operation: makeNewSale Post-conditions: Process arrives... -... 2. Cashier system system Interact makes new enterItem events operations sale. (id, quantity) 3. Cashier Operation: enterItem (various enters item identifier. endSale() Post-conditions: 4.... - A SalesLineItem instance sli was created makePayment -... (amount) modeling Use Cases System Sequence Diagrams Contracts in addition to the use Developer cases, requirements that some ideas and inspiration for the post- must be satisfied by the skills as the conditions derive from the use cases design of the software use-case Design Model realization techniques) : Register : ProductCatalog makeNewSale() create() enterItem : Sale (itemID, quantity) spec := getSpecification( itemID ) addLineItem( spec, quantity )... endSale()...... Construct Interact 51 (Rational) Unified Process weeks Every type of activity runs in parallel (vertically) with other types of activities in the same iteration.52  All diagrams are rough sketches to ease communications. Ideas of Test-Driven Development Oral write automated test case Requirements Test Script Development The minimal functionality of the code that can pass all test cases Coding + applying coding principles (or Testing code refactoring) to keep the codebase better organized. Deployment Maintenance No obvious maintenance phase. Each iteration must be short in duration in terms of time. 53 Agile Methods  Iterative development Deliver works in small increments to users  Customer collaboration evaluate and feedback on the values of requirements/increments  Response to changes Continuously evaluate the requirements, plan, and deliver increments with user feedback. Make changes rapidly  Inadequate documentation  Unpredictable delivery time and costs  Dependence on client involvement 54 Example Agile Method: XP  Extreme programming (XP) has five phases: Planning Meet with users, define user stories, and estimate story points. Plan for release through N iterations (for user stories delivery) Design Use Simple and consistent design sketch Coding Apply XP practice (see the next two slides) Testing Conduct automated unit tests and acceptance tests per user story Feedback PM and users determine the values of the user stories delivered by the 55 implementation Extreme Programming XP [an Agile Method]  Emphases: Generate feedbacks Embrace changes keep customers engaged short iteration Fix bugs early XP practices 56  http://www.extremeprogramming.org/rules.html Extreme Programming XP  Simple Rules 57 XP guidelines  Planning: programmers estimate efforts needed for implementing user stories and customer decides the scope and timing of release  Small release: monthly, or daily for small fixes  Metaphor: A shared story guides all developments by describing how the system works  Simple design: use simplest possible solution  Testing: use tests are implemented before the code. Customer write the functional tests.  Refactoring: do refactoring frequently  Pair programming: two people write code at one computer.  Collective ownership: anyone can change any part of the code at anytime  Continuous integration: integrate the code to the project codebase as soon as it is ready.  On-side customer: customers are available full-time  Coding standards: apply them.  Open worksplace: a large room with small cubicles preferred  40-hour week: No Overtime in two consecutive weeks  Just rules: Team has its own but changeable rules for all to follow 58 Popular PM elements in Agile Methods  Estimate the story point for each user story Allocate extra time for research on a user story with high uncertainty  Pick a set of user stories into the next iteration in XP (equivalent to the next sprint in Scrum)  Within a cycle (an XP iteration or a Scrum sprint), conduct (daily) standing meetings A super short meeting (15 mins in total) [3 mins] Every team member reports (1) what was done yesterday, (2) what will be done today, and (3) what blockers are encountered. For this item, other teammates may help resolve it.  Measure project velocity  At the end of each cycle, conduct a review (a demo to users about the 59 deliverable achieved in the cycle) and a self-reflection on the cycle User story  A user story is in the following format: As a , I want an , so that I can achieve a. role is what we can identify from Requirements Engineering action should be tangible (actionable) goal should be measurable and quantifiable E.g., (for AIMS): As a CityU Student, I want to download a testimonial for my student status by submitting a request in AIMS so that I can apply for my student visa extension through the HKSAR Immigration Department.  A user story usually has more information such as use cases, diagram sketch, user notes, data, reports from existing systems to clarify the context and scope of the user story. 60 Relative Story Point Estimation  It is difficult to estimate whether bar A is long and how long bar A is.  However, it would be easier to say that A is longer than B. A B  A story point is a value. E.g., Fibonacci sequence: 1, 2, 3, 5, 8, 13, 21, 34, 55, 89, 144, … E.g., T-shirt size: XXS, XS, S, M, L, XL, XXL  Pick one or more user stories as anchors. The whole team agreed on a story point for each such user story to indicate the number of expected efforts to implement the user stories, taking into account uncertainty and implementation complexity. E.g., if B is agreed to as 3 story points, and the team has the impression that A is bigger than B but not very much, A can be assigned 5 story points. 61 Project Velocity  A user story is implemented in a cycle if it passes the cycle review (by users and product owner)  An agile project is organized as a series of cycles (XP iterations or Scrum sprints).  Each cycle implements a set of user stories with assigned story points.  If a user story is too large to fit into a cycle, the user story should be broken down into multiple user stories.  The total story points delivered by the cycle are plotted. Project velocity 15 If the outstanding number of Story points 12 11 12 Max: 15 per cycle 10 Min: 10 per cycle story points is 100, then the Avg: 12 per cycle project needs ceiling(100/12) cycle = 9 more cycles to complete 62 1 2 3 4 5 Burndown Chart Problem! The project cannot make progress gradually Story points actual progress Cycle Story Points cycle 0 160 1 150 Oops! New user stories added to 2 138 the project or original estimated 3 127 story points are underestimated 4 112 5 100 Story points cycle Project velocity is the slope over a consecutive series of cycles. 63 Scheduled versus Actual  When creating a release plan or a plan for a sprint, the project manager needs a schedule.  Consider our course project (must be delivered by the end of Week 13). Suppose the project starts at Week 2, each cycle being 1 week. The last cycle is to wrap out the project and write a report, prepare a poster or videos, etc. At the end of Week 7 (5 cycles completed), the project is delayed. PM needs to take actions after each cycle: e.g., trim off some user stories and request the team to deliver more story points per cycle 64 Scrum http://en.wikipedia.org/wiki/Scrum_(development)  Is a kind of agile development process  Arguably the most widely used development process nowadays  driven by daily and periodic one-hour meetings Go through the usage scenarios/stories to identify a slice of the highest priority backlog tasks to be completed by the next iteration (Sprint) agreed upon by the customer Together with customers, discuss the goal of the current Sprint, prioritize functions to be completed, and divide into detailed tasks Conduct periodic short planning and review meetings for tasks completed and not completed in time and revise the set of backlog tasks accordingly  Customers may change their mind at any time Accept that the requirements cannot be fully understood or defined, Focus on maximizing the team’s ability to deliver quickly and respond to the selected backlog tasks. Watch this video: https://tw.voicetube.com/videos/49552 (ignore the ads) 65  Task board Scrum  A product backlog = user stories and development tasks due to the completion of some user stories (e.g., bugs found in a later cycle)  Priority the product backlog  Suppose a cycle = two weeks (so, our project has six sprints at most)  A high-priority subset of the product backlog is treated as the current sprint backlog The size of the sprint backlog is determined by the total number of story points that the team can be delivered within a cycle based on the history  Through the standing meeting, each team member picks an item from the sprint backlog. The target is to deliver the user story in the cycle. (Visualize as a task board) The tasks needed to deliver the story when it is started are all placed into the “to-do” column of the task board. When a task is being worked on, it is moved to the “in-progress” column. When the implementation of the task is completed, it is moved to the “to verify” column (which is usually merged with the-progress column)  At the end of a sprint, conduct a sprint review Product owners/users verify whether the user story is delivered. If ok, then move the task to the “Done” column.  Also, conduct a sprint retrospective (reflect on what has been done right to make progress and what to be improved to reduce or avoid blockers to improve productivity, etc) 66 Scrum Tools and Meeting https://www.mountaingoatsoftware.com/agile/scrum/scrum-tools  product backlog is a list of features desired for a final product, the bugs to be removed, technical work to set up and maintain development environment and user site, and knowledge (e.g., learn to use a new framework) to acquire by the project.  release burndown chart tracks progress on a project. The chart itself is updated after each sprint. Teams can measure progress in any unit they choose.  sprint backlog is a list of tasks to complete during a sprint. It is updated once a day.  task board is a sheet that every member of the team can use and add to over the course of a sprint, and is a visual representation of every task and what phase of completion it’s in. Usually, task boards include columns for stories, to-dos, work in process, things needing verification and items finished. Some teams also include burndown charts, notes and tests. Hang the board on a wall or digitalize it.  Meeting: The product owner shows up at the sprint planning meeting with the prioritized agile product backlog and describes the top items to the team. The team then determines which items they can complete during the coming sprint. The team then moves items from the product backlog to the sprint backlog. In doing so, they67 expand each Scrum product backlog item into one or more sprint backlog tasks  Scrum Tools https://www.mountaingoatsoftware.com/agile/scrum/scrum-tools release burndown chart https://www.mountaingoatsoftware.com/ agile/scrum/scrum-tools/product-backlog/ example Hours to go to complete sprint backlog the task task board hours 102 86 69 33 days 68 Sum: 102 86 69 33 Scrum Roles  Scrum Master is the team's coach and helps Scrum practitioners achieve their highest level of performance. This role does not provide day-to-day direction to the team and does not assign tasks to individuals. In many projects, this role is assumed by the project manager.  Product owner is responsible for prioritizing the backlog during Scrum development.  Scrum development team as a whole. 69 Comparison among Agile Methods Extreme Programming Scrum 70 Hybrid Method  E.g., “The video game developer used Waterfall to plan and develop assets like characters and initial coding to ensure that the base game would be robust and well- constructed. They switched to Agile when it came to gameplay mechanics, debugging, and post-launch updates. By applying a hybrid approach, Ubisoft successfully launched Assassin’s Creed Valhalla to commercial acclaim in November 2020.” Source: https://hbr.org/2023/10/its-time-to-end-the-battle-between-waterfall-and-agile 71 Hybrid Method  Agile PM + Waterfall  Flexibility and structure Iterative to deliver increments rapidly Adhere to (requirements) documentation  Phased and iterative Phased approach for well-defined components Iterative approach for uncertain ones  Customer involvement and predictability getting feedback early and often  Applicable when (1) Diverse stakeholder needs, (2) Varied project phases, (3) Uncertain requirements, and (4) high-risk projects with complex project structures72 Home Reading  Read the following section of this reference: https://arxiv.org/ftp/arxiv/papers/1709/1709.08439.pdf Section 3.2 Scrum  Read the differences between Scrum Master and Product Owner in Scrum https://www.yodiz.com/blog/scrum-master-vs-product-owner-diff erences-in-skills-duties-and-responsibilities-agile-methodology/  Read the examples mentioned in the Hybrid Method https://hbr.org/2023/10/its-time-to-end-the-battle-between- waterfall-and-agile 73 References 1. João M. Fernandes and Sónia M. Sousa. 2010. PlayScrum - A Card Game to Learn the Scrum Agile Method. In Proceedings of the 2010 Second International Conference on Games and Virtual Worlds for Serious Applications (VS-GAMES '10). IEEE Computer Society, Washington, DC, USA, 52-59. DOI: https://doi.org/10.1109/VS-GAMES.2010.24 Include a concise summary of Scrum 2. Videos on YouTube https://www.youtube.com/results?search_query=scrum 74