Agile Software Development Lecture (1) PDF

Agile Software Development Lecture (1) Dania Mohamed Ahmed  Introduction  Rapid development and delivery is now often the most an important requirement for software systems  Businesses operate in a fast–changing requirement, and producing a set of st...

Agile Software Development Lecture (1) Dania Mohamed Ahmed  Introduction  Rapid development and delivery is now often the most an important requirement for software systems  Businesses operate in a fast–changing requirement, and producing a set of stable software requirements is practically impossible.  Software has to evolve quickly to reflect changing business needs.  For software types requiring thorough analysis, such as safety-critical systems, a plan-driven approach remains appropriate. However, for business systems needing rapid adaptability, agile development methods are more suitable. Agile methods focus on producing useful software quickly through iterative development. Introduction Cont..  Key characteristics of agile approaches include:  Interleaved Processes: Specification, design, and implementation occur simultaneously with minimal detailed documentation. The user requirements are outlined rather than fully detailed.  Incremental Development: The software is developed in increments, with ongoing input and feedback from end-users and stakeholders to adapt to changing needs.  Extensive Tool Support: Tools for automated testing, configuration management, integration, and user interface production aid the rapid development process. Agile methods  Agile methods are a form of incremental development where software is released in small, frequent updates, typically every two or three weeks. This approach emphasizes close customer involvement to obtain rapid feedback and adapt to changing requirements. Agile development minimizes formal documentation, favoring informal communication over structured meetings and written documents.  In agile methodologies, design and implementation are central, with other activities like requirements elicitation and testing integrated into these core processes. This contrasts with plan-driven approaches, which follow distinct stages with specific outputs used to plan subsequent activities. Agile methods Cont..  The key differences between plan-driven and agile approaches to system specification:  Plan-Driven Approaches: Iteration occurs within individual stages (e.g., requirements, design, implementation) with formal documents guiding transitions between these stages. For example, evolving requirements lead to a formal requirements specification document, which then directs the design and implementation phases.  Agile Approaches: Iteration happens across activities, with requirements and design being developed concurrently rather than sequentially. While agile methods emphasize flexibility and may prioritize coding, they are not exclusively code-focused. Agile processes can also include design documentation and planned activities, with some iterations dedicated to creating system models and documentation as needed. Agile Methods Cont..  In the 1980s and early 1990s, software development focused on extensive planning, formal quality assurance, and rigorous processes, especially for large- scale projects such as those in aerospace and government. This plan-driven approach was well-suited for complex, long-term projects involving large, dispersed teams, where detailed planning and documentation were crucial for coordination.  However, when applied to smaller or medium-sized business systems, this heavyweight approach led to excessive overhead, with more time spent on planning and documentation than on actual development and testing. This inefficiency highlighted the need for more flexible methods, leading to the emergence of agile methods in the late 1990s. Plan-driven and agile development Agile manifesto  We are uncovering better ways of developing software by doing it and helping others do it. Through this work we have come to value: Individuals and interactions over processes and tools Working software over comprehensive documentation Customer collaboration over contract negotiation Responding to change over following a plan  That is, while there is value in the items on the right, we value the items on the left more.  Agile manifesto  Agile methods prioritize delivering working software quickly and adapting to changing requirements, minimizing process bureaucracy and documentation. They focus on practical, incremental development rather than extensive upfront planning. The agile manifesto emphasizes: 1. Valuing individuals and interactions over processes and tools 2. Prioritizing working software over comprehensive documentation 3. Fostering customer collaboration over strict contract negotiation 4. Adapting to change rather than rigidly following a plan  In essence, while acknowledging the importance of certain formal aspects, agile methods favor a more flexible and responsive approach to software development.  The principles of agile methods Principle Description Customer involvement Customers should be closely involved throughout the development process. Their role is provide and prioritize new system requirements and to evaluate the iterations of the system Embrace change Expect the system requirements to change, and so design the system to accommodate these changes. Incremental delivery The software is developed in increments, with the customer specifying the requirements to be included in each increment. Maintain simplicity Focus on simplicity in both the software being developed and in the development process. Wherever possible, actively work to eliminate complexity from the system. People, not process The skills of the development team should be recognized and exploited. Team members should be left to develop their own ways of working without prescriptive processes.  Agile method applicability  Agile methods advocate for incremental software development and delivery, with various processes underpinned by a common set of principles from the agile manifesto. These methods have proven effective in two main areas: 1. Product Development: For software companies creating small to medium- sized products for sale, agile approaches are now standard practice for developing software products and apps. 2. Custom System Development: In organizational contexts where customers are actively involved and there are minimal external constraints, agile methods are well-suited for developing custom systems.  Overall, agile practices are widely used for both commercial product development and internal custom software projects due to their flexibility and collaborative nature.  Problems with agile methods 1. Customer Engagement: Keeping customers consistently engaged and interested throughout the development process can be challenging. Continuous involvement is crucial, but not all customers may have the time or commitment to stay actively involved. 2. Team Suitability: Agile methods often require intense involvement and collaboration. Team members who are not accustomed to or comfortable with this level of engagement may struggle with the demands of agile practices. 3. Prioritization of Changes: When multiple stakeholders are involved, prioritizing changes can be complex. Differing priorities and needs can make it difficult to align on what changes should be implemented first. 4. Maintaining Simplicity: Agile methods emphasize simplicity, but achieving and maintaining simplicity requires extra effort. Teams must continuously refine their processes and designs to avoid unnecessary complexity. 5. Contractual Issues: Agile approaches can face challenges with contracts, particularly when dealing with iterative development. Traditional contracts often assume a fixed scope, but agile's flexible nature can complicate contract management and scope definition. Technical, human, and organizational issues  Most projects incorporate both plan-driven and agile elements. The choice between these approaches depends on several factors: 1. Detail Requirements: If a detailed specification and design are crucial before starting implementation, a plan-driven approach is likely needed. 2. Feedback and Delivery: If delivering software incrementally and receiving rapid customer feedback is feasible, agile methods are suitable. 3. System Size: Agile works best for small, co-located teams. For larger systems needing extensive teams, a plan-driven approach may be more suitable. 4. System Type: Complex systems with critical requirements (e.g., real-time systems) may need a plan-driven approach due to extensive analysis needs. Technical, human, and organizational issues 5. System Lifetime: Long-lifetime systems might require more detailed design documentation for future support. 6. Technology: Agile methods rely on tools for managing evolving designs. 7. Team Organization: Distributed or outsourced teams may need detailed design documents to coordinate effectively. 8. Cultural Issues: Traditional engineering cultures often favor plan-based methods. 9. Team Skills: Agile methods might demand higher skill levels compared to plan-based approaches. 10. Regulation: Systems subject to external regulation (e.g., safety-critical systems) typically require detailed documentation.  Extreme programming  Perhaps the best-known and most widely used agile method.  Extreme Programming (XP) is one of the most well-known and widely adopted agile methodologies. It was introduced by Kent Beck in the late 1990s and is designed to enhance customer satisfaction, flexibility, and rapid iteration.  Iterative Development: XP takes an 'extreme' approach to iterative development:  Frequent Builds: New versions of the software may be built several times per day, allowing for rapid adjustments and improvements.  Regular Deliveries: Increments are delivered to customers every two weeks, providing frequent opportunities for feedback and alignment with user needs.  Rigorous Testing: All tests must be run for every build, and a build is only accepted if it passes all tests, ensuring high quality and reliability.  XP and agile principles  Extreme Programming (XP) aligns closely with agile principles, emphasizing the following aspects: 1. Incremental Development: XP supports incremental development by delivering small, frequent releases of the system. This approach ensures that new features and improvements are regularly provided, allowing for ongoing user feedback and adaptation. 2. Customer Involvement: XP requires full-time customer engagement with the development team. This close collaboration ensures that the team has a clear understanding of customer needs and can make adjustments quickly based on their feedback. 3. People and Team Dynamics: XP promotes practices that enhance team dynamics and productivity: a. Pair Programming: Two developers work together at one workstation, which fosters collaboration, improves code quality, and facilitates knowledge sharing.  XP and agile principles b. Collective Ownership: All team members share responsibility for the codebase, allowing anyone to make changes and encouraging a sense of shared ownership. c. Avoiding Long Hours: XP advocates for sustainable work practices, emphasizing the importance of maintaining a manageable workload to prevent burnout and maintain productivity. 4. Support for Change: Regular system releases in XP facilitate adaptability. By delivering incremental updates frequently, XP supports the ability to respond to changing requirements and evolving customer needs. 5. Maintaining Simplicity: XP emphasizes the importance of simplicity in design through constant refactoring. This practice involves continuously improving and simplifying the codebase, ensuring that the software remains clean, efficient, and easy to maintain. The extreme programming release cycle  Extreme programming practices Principle or practice Description Collective ownership The pairs of developers work on all areas of the system, so that no islands of expertise develop and all the developers take responsibility for all of the code. Anyone can change anything. Continuous integration As soon as the work on a task is complete, it is integrated into the whole system. After any such integration, all the unit tests in the system must pass. Incremental planning Requirements are recorded on “story cards,” and the stories to be included in a release are determined by the time available and their relative priority. The developers break these stories into development “tasks.” On-site customer A representative of the end-user of the system (the Customer) should be available full time for the use of the XP team. In an extreme programming process, the customer is a member of the development team and is responsible for bringing system requirements to the team for implementation.  Extreme programming practices Cont.. Principle or practice Description Pair programming Developers work in pairs, checking each other's work and providing the support to always do a good job. Refactoring All developers are expected to refactor the code continuously as soon as potential code improvements are found. This keeps the code simple and maintainable. Simple design Enough design is carried out to meet the current requirements and no more. Small releases The minimal useful set of functionality that provides business value is developed first. Releases of the system are frequent and incrementally add functionality to the first release. Sustainable pace Large amounts of overtime are not considered acceptable, as the net effect is often to reduce code quality and medium-term productivity Test first development An automated unit test framework is used to write tests for a new piece of functionality before that functionality itself is implemented.  Requirements scenarios  In Extreme Programming (XP), requirements are managed collaboratively and flexibly through several key practices: 1. Customer Involvement: A customer or user is an essential part of the XP team and makes decisions about requirements, ensuring alignment with user needs. 2. User Stories: Requirements are expressed as user stories, brief descriptions of user interactions with the system, created in collaboration between the development team and the customer. 3. Story Cards and Task Breakdown: User stories are written on cards and broken down into specific implementation tasks. These tasks guide scheduling and cost estimation. 4. Prioritization and Scheduling: The customer prioritizes user stories for the next release, and the development team uses task estimates to plan and set deadlines.  This method allows for continuous refinement and prioritization of requirements, aligning with agile principles of flexibility and responsiveness. “prescribing medication” story Examples of task cards for prescribing medication Refactoring  Traditional software engineering stresses designing for change to handle future modifications, but extreme programming (XP) dismisses this idea. XP argues that preparing for anticipated changes often proves unnecessary, as actual changes may differ. Instead, XP advocates for constant refactoring—continuously improving code structure and readability.  Refactoring involves making incremental improvements, like reorganizing class hierarchies and consolidating code, to prevent software structure degradation. However, maintaining refactoring can be difficult due to development pressures and the occasional need for more significant architectural changes.  Testing in XP  In incremental development, unlike plan-driven development, there is no fixed system specification for an external team to create tests. This often leads to a less formal testing process. Extreme Programming (XP) addresses this by integrating automated testing into the core of development. In XP, testing is automated and crucial, and development progresses only when all tests pass successfully (Testing is central to XP and XP has developed an approach where the program is tested after every change has been made).  XP testing features:  Test-first development.  Incremental test development from scenarios.  User involvement in test development and validation.  Automated test harnesses are used to run all component tests each time that a new release is built.  Test-first development  Writing tests before code clarifies the requirements to be implemented.  Tests are written as programs rather than data so that they can be executed automatically. The test includes a check that it has executed correctly.  Usually relies on a testing framework such as Junit (facilitating the writing and execution of tests.).  All previous and new tests are run automatically when new functionality is added, thus checking that the new functionality has not introduced errors.  Customer involvement in testing is vital for ensuring the system meets their needs. They help create acceptance tests and write tests with the development team to align new code with their requirements. However, due to time constraints and their belief that providing initial requirements is sufficient, customers may not be able to fully engage in the testing process. Test case description for dose checking Test automation  Test automation involves writing tests as executable components before implementing a task. These tests are designed to be independent, simulate input submission, and verify that outputs meet specifications. An automated test framework, such as JUnit, facilitates the creation and execution of these tests.  With test automation, a consistent set of tests is always available for quick execution. This ensures that whenever new functionality is added, tests can be run immediately to identify and address any issues introduced by the new code. XP testing difficulties  XP testing difficulties include:  Programmers' Preferences: Developers often prefer coding over testing and may take shortcuts, resulting in incomplete tests that may not cover all possible exceptions.  Complex Tests: Writing incremental tests can be challenging, especially for complex features like user interfaces where testing 'display logic' and inter- screen workflows is difficult.  Test Coverage: Assessing the completeness of test coverage can be hard. Even with numerous tests, it's challenging to ensure that the test set covers all aspects of the system effectively. Pair programming  Pair Programming in Extreme Programming (XP) involves two programmers working together at a single workstation. This approach fosters shared ownership of code, enhances knowledge distribution across the team, and provides an informal review process, as each line of code is reviewed by both programmers.  Key aspects include:  Dynamic Pairs: Team members are paired up dynamically to ensure everyone collaborates throughout the development process.  Encourages Refactoring: Collective code review supports ongoing refactoring, benefiting the whole team.  Productivity: Studies show that productivity with pair programming is comparable to that of two programmers working independently, and in some cases, pairs may work more efficiently than individuals.  Knowledge Sharing: Pair programming helps mitigate project risks by ensuring that knowledge is shared and not reliant on individual team members.  Overall, pair programming is an effective practice for improving code quality and team collaboration while managing knowledge and risks. Advantages of pair programming 1. Collective Ownership: The team shares responsibility for the code, rather than individual programmers being held accountable for issues, promoting a collaborative approach to problem-solving. 2. Informal Review: Each line of code is reviewed by both programmers, providing an ongoing informal review process that enhances code quality. 3. Support for Refactoring: Pair programming facilitates refactoring, as improvements made by one pair benefit the entire team, encouraging broader support for ongoing code enhancements. Scrum  The Scrum agile method is a general agile method but its focus is on managing iterative development rather than specific agile practices. 1. Initial Phase:  Purpose: Establish general project objectives and design the software architecture.  Activities: Outline planning and create a foundational design for the software. 2. Sprint Cycles:  Purpose: Develop and deliver increments of the system in iterative cycles.  Activities: Each Sprint involves planning, execution, review, and reflection to incrementally build and refine the product. 3. Project Closure Phase:  Purpose: Finalize the project and complete the necessary documentation.  Activities: Wrap up the project by preparing system help frames, and user manuals, and assessing lessons learned to improve future projects. Scrum Cont..  Scrum Framework Overview: 1. Key Roles:  Product Owner: Responsible for defining the features and prioritizing the backlog. The Product Owner ensures that the team delivers value to the business and stakeholders.  Scrum Master: Acts as a facilitator and coach, helping the team adhere to Scrum practices, removing impediments, and ensuring effective communication.  Development Team: A cross-functional group of professionals who work together to deliver potentially shippable product increments. They are self- organizing and responsible for managing their own work. Scrum Cont.. 2. Artifacts:  Product Backlog: A prioritized list of features, enhancements, and fixes that need to be addressed. It is managed by the Product Owner and evolves based on changing requirements and feedback.  Sprint Backlog: A subset of the Product Backlog selected for completion during a Sprint. It includes tasks that the Development Team commits to delivering within the Sprint.  Increment: The sum of all completed Product Backlog items during a Sprint, plus the increments of all previous Sprints. It must be in a usable condition and meet the Definition of Done. Scrum Cont.. 3. Events:  Sprint: A time-boxed iteration, usually lasting 2 to 4 weeks, during which a specific set of Product Backlog items are completed. At the end of each Sprint, a potentially shippable product increment is delivered.  Sprint Planning: A meeting held at the beginning of each Sprint where the team plans what will be delivered and how the work will be accomplished.  Daily Scrum: A short, daily meeting (usually 15 minutes) where the team synchronizes, discusses progress, and identifies any impediments. It helps maintain alignment.  Sprint Review: A meeting held at the end of each Sprint to review the completed work and gather feedback from stakeholders. It helps assess progress and make necessary adjustments.  Sprint Retrospective: A meeting held after the Sprint Review where the team reflects on the Sprint process, identifies what went well, what didn’t, and how to improve in future Sprints. The Sprint cycle  The Sprint cycle  The Sprint cycle in Scrum involves the following stages: 1. Duration: Sprints are fixed-length iterations, typically lasting 2 to 4 weeks, and result in a release of the system similar to the XP development process. 2. Planning: The process starts with the Product Backlog, which lists all tasks and features to be completed. The project team collaborates with the customer to select the features for the Sprint. 3. Development: Once features are selected, the team organizes and works on the development in isolation from the customer and organization. Communication is managed through the Scrum Master, who shields the team from external distractions. 4. Review: At the end of the Sprint, the completed work is reviewed and presented to stakeholders. This concludes the Sprint, and the cycle begins anew with the next Sprint. Teamwork in Scrum  In Scrum, teamwork is organized as follows:  Scrum Master: Acts as a facilitator by arranging daily meetings, tracking the backlog, recording decisions, measuring progress, and communicating with customers and management outside the team.  Daily Meetings: The entire team participates in brief daily meetings where members share updates on their progress, discuss any issues encountered, and outline plans for the next day. This ensures that all team members are informed about the project's status and allows for quick adjustments if problems arise. Scrum benefits  Manageable Chunks: The product is divided into smaller, manageable pieces, making it easier to understand and develop.  Adaptability to Change: Unstable or changing requirements do not impede progress, as the iterative nature of Scrum accommodates adjustments throughout the project.  Improved Communication: The whole team has visibility into all aspects of the project, which enhances communication and collaboration.  Customer Engagement: Customers see regular, on-time deliveries of product increments and receive feedback on how the product is evolving, ensuring their needs are addressed.  Trust and Culture: Scrum fosters trust between customers and developers, creating a positive environment where everyone is aligned towards the project's success. Scaling agile methods  Scaling Agile methods involves adapting these approaches to handle larger, more complex projects that involve multiple teams and possibly different locations.  Successful for Small Projects: Agile methods have proven effective for small to medium-sized projects with a single, co-located team due to enhanced communication and collaboration.  Challenges of Scaling: When scaling Agile, adjustments are needed to manage larger, longer projects with multiple development teams. This includes coordinating work across teams and potentially dealing with distributed team members.  Communication Considerations: The success of Agile in smaller projects often hinges on close communication, which can be more challenging with larger, geographically dispersed teams.  Adaptation for Larger Projects: Scaling Agile requires modifications to processes and practices to ensure effective collaboration, coordination, and communication across multiple teams and locations.  Large systems development  Large systems development involves several complexities: 1. Multiple Teams and Locations: Large systems typically comprise separate, interacting systems developed by different teams, often spread across various locations and time zones. 2. Brownfield Systems: These systems often interact with existing systems, making many requirements related to these interactions rigid and less amenable to flexibility and incremental development. 3. System Configuration: Much of the work in integrating multiple systems focuses on configuration rather than new code development. Additionally, development is often constrained by external regulations and rules. 4. Extended Development Time: Large systems have lengthy procurement and development phases, which can lead to challenges in maintaining a consistent team due to personnel changes over time. 5. Diverse Stakeholders: Managing a diverse group of stakeholders is challenging, and involving all of them actively in the development process is impracticals.  Scaling out and scaling up  Scaling out and scaling up refer to different approaches for applying Agile methods in larger contexts:  Scaling Up: Focuses on using Agile methods to develop large software systems that require more than a small team. It involves adapting Agile practices to manage the complexities of bigger projects.  Scaling Out: Involves implementing Agile methods across a large organization with extensive software development experience. This process is about integrating Agile practices organization-wide.  Essential Agile Fundamentals to Maintain:  Flexible Planning: Adapting plans as needed to respond to changes.  Frequent Releases: Delivering system updates regularly.  Continuous Integration: Regularly merging code changes to detect issues early.  Test-Driven Development: Writing tests before code to ensure functionality.  Good Team Communications: Ensuring effective collaboration and information sharing.  Scaling up to large systems  Up-Front Design and Documentation: For large systems, focusing solely on coding is insufficient. More comprehensive design and system documentation are required upfront.  Cross-Team Communication: Effective communication mechanisms are crucial. This includes regular phone and video conferences and frequent short electronic meetings for progress updates among teams.  System Builds and Releases: Continuous integration of the entire system with every code change is often impractical. Instead, maintaining frequent system builds and regular releases is essential for managing large systems effectively. Scaling Out to Large Companies  Reluctance from Project Managers: Project managers unfamiliar with Agile methods may be hesitant to adopt this new approach due to perceived risks.  Compatibility with Standards: Large organizations often have established quality procedures and standards that may conflict with Agile methods, which can be bureaucratic and rigid.  Skill Variability: Agile methods are most effective with highly skilled team members, but large organizations often have a diverse range of skill levels among their employees.  Cultural Resistance: There may be resistance to Agile methods in organizations with a long history of traditional systems engineering processes.