Advanced Software Engineering PDF

Advanced Software Engineering Software Products Project-Based SE: Customers have a problem that generates requirements that are implemented by software developed by developers. customer defines a list of functionalities that they want, the developers generate the requirements fr...

Advanced Software Engineering Software Products Project-Based SE: Customers have a problem that generates requirements that are implemented by software developed by developers. customer defines a list of functionalities that they want, the developers generate the requirements from them custom adaptation required, since the functionalities needed may change Product-Based SE: Developers realize opportunities that inspires product features that are implemented by software. developer has the entire process in his hands, but needs to take the customer's needs in consideration getting product ASAP is critical => need Agile methods to rapidly develop the software Software Execution Models Stand-alone: everything installed user-side Hybrid: part of functionalities are on server-side SaaS: everything on the server Product Vision 3 Fundamental questions: Who are the targeted customers? FOR (customer) WHO NEEDS (opportunity) What is the product to be developed? THE (product) THAT (reason to buy) Why should customers buy the product? UNLIKE (competitive alternative) OUR PRODUCT (differentiation statemnt) Information sources for developing prod vision: Domain Experience: understand the software support needed by own work area experience  Product Experience: users of existing software may suggest simpler or more usable ways to provide comparable functionalities Customer Experience: understanding the customers' needs Prototyping: develop a prototype to experiment and play around with variations to find errors and missed needed things Product Manager must ensure: monitoring constraints important for the customer meet needs of the company and customers regularly communicate with customers (UX++) Product Manager Technical interactions: Product Roadmap: to set goals, milestones, success criteria User Stories and scenario dev: to identify product features and refine product vision Product Backlog: to-do list to complete project development, PM decides the priorities Acceptance Testing: PM should be involved in developing tests of the features that reflect how customers use the product Customer Testing: taking a release of a product to potential customers groups and getting feedback from them. UI Design: monitor simplicity of usability or not Agile Project Management Framework that splits project dev into several dynamic phases. Purpose = shipping products quickly minimizing the overhead (anything that doesnt contribute to fast delivery) Agile Manifesto Individuals and interactions over processes and tools; Working software over comprehensive documentation; Customer collaboration over contract negotiation; Responding to change over following a plan. Based on Incremental Development and Delivery Software = Set of Features 1. plan a product features list 2. choose 1 feature for next increment 3. refine feature description 4. implementation and automated tests 5. integrate into system and run general tests 6. deliver system increment, if all features are complete, deliver system release (12) Principles 1. early and continuous delivery of sw 2. welcome changing requirements, even late 3. deliver working sw frequently -> primary measure of progress 4. business ppl & devs must work together 5. simplicity 6. self organizing teams 7. team reflects how to become more effective, adjusting its behaviour Extreme Programming (XP) Development technique based on: Incremental planning + user stories => requirements are written as user stories, with a customer representative. Small Releases => minimal set of functionalities is developed first, then small and frequent releases. Test-driven development => write test before code to clarify what it should do. Continuous Integration => as soon as a task is completed, it is integrated in the system and autotested. Refactoring => improving structure, readability, efficiency and security. Scrum Lightweight framework for project organization with roles that act as an interface between dev team and organization. Terminology Dev Team: 4-8 developers Product Owner: identifies product features and attributes, reviews the work and helps testing ScrumMaster: Team coach that guides the team on the Scum method Product Backlog: To-do list of items (PBI) => bug fixes, features and improvements Sprint: 1-4 weeks during which the product increment is developed Scrum: daily team meeting to assess progress and discuss the work of the day 3 Pillars 1. Transparency - everything must be visible by anyone 2. Inspection - artifacts and progress must be inspected frequently 3. Adaptation - if something deviates from the limits, it must be adjusted 5 Values Commitment - Focus - Opennes - Respect - Courage PBI States Ready for: Consideration: tentative features that will be considered, but may not be included in final product Refinement: agreed important item that will be implemented, but needs more understanding Implementation: has enough detail (even dependencies with other items) to be implemented Activities Refinement: PBIs are analyzed, can lead to creation of more items Estimation: team estimates the time needed for each PBI Creation: adding more items to backlog  Prioritization: reordering PBI to take into account new circumstances Features, Scenarios & Stories Definitions Features In product-based SE, requirements arent set by the customers and can change. Developers identify product features, which are fragments of a functionality, through interviews/surveys. Personas Representation of a user to understand their background, skills and experience and to design features useful for them. 1-2, max 5 personas = key product feature Personal information + Job + Education + Interests in the product (smiling photo xd) "to step into the users' shoes" Scenarios Narrative describing the user's interaction with the product's features. may be incomplete and lack detail 3-4 scenarios for each persona, covering the persona's main responsibilities written from the user's perspective User Stories Fine-grained description of what the user wants to do with the product. As a i want to so that helps dividing work into units that are associated with priorities, to sort them a well done story should that 1-2 workdays to be developed Feature Identification Properties that a feature should have: 1. Independence: no dependency with other features (even their order of activation)  2. Coherence: linked to only a single action and no side-effects 3. Relevance: should not offer obscure functionalities that are rarely required Knowledge Sources User: user stories & scenarios Technology: taking advantage of new technologies Product: experience in existing products / research on them Domain: work area expertise (finance, event booking...) Tradeoff Factors Simplicity Functionality Control Automation Familiarity Novelty Feature Creep: features grows exponentially with users growt and demands (MC Office) Software Architecture Definitions Component element of a project implementing a Set of Features collection of service that may be used by other components Module Set of Components, providing a Service Service Coherent unit of functionality Design issues Non functional product characteristics => security or performance Product Lifetime => possibility for the product to evolve during its lifetime Software Reuse => using other products/opensource to save time  Number of users => up/down scaling possibilities Software Compatibility => mainainance of compatibility with other sw (ex: database) Non functional Quality Attributes Important for final product, not important for prototypes. Optimizing one may affect others (trade-off). Responsiveness: wait time Reliability: behaves as expected Availabilty: delivers service whenever requested Security: protects itself and users data Maintainability: updates/adding features without costs Resilience: continues delivering service after failures or attacks Controlling Complexity As the number of components increase, the relationships between them increase at a faster rate need to decompose until a certain point => Simplicity is Essential decomposition must be done when choosing the technologies for the system Design Guidelines: Separation of Concerns, Implement Once, Stable Interfaces. Distribution Architecture 1. Client-Server: clients access shared databases to do logic operation on that data 2. Model-View-Controller: Client = Browser - Controller - View Server = Model model decoupled from its presentation each view registers with the model, if model changes, views needs refreshing Technologies choices Affect and constrain the overall system architecture and are difficult/expensive to change during dev. 1. Databases Relational (MySQL): good for transaction management (ACID - Atomicity, Consistency, Isolation, and Durability) and simple data structures NoSQL: best for not well structured data and most common operation is reading (data analysis apps) 2. Delivery Platform Web-based / Desktop App Mobile intermittent connectivity, less processing power, needs power management, reduced screen size** Cloud based Software Main idea: access to virtualized resources (computing power, storage,...) on demand Advantages Scalability - Elasticity - Resilience - Cost - Startup Time - Server choice - Distributed development Scaling out = adding new virtual servers Scaling up = increasing the computational power of a server Virtualization and Containers 1. Virtual machines: guest OS with an Hypervisor that manages it on a phisical server 2. Container: exploit OS kernel to allow multiple isolated user-spaces instances lighter, less resources used, faster to start (seconds) share the same OS Docker Container management system that allows to develop and run portable applications on an isolated environment Docker Image: read only template to create containers (stored in a private/public Docker Registry) Docker Hub: platform used to share/obtain images Dockerfile: defines an image as a series of setup command Docker Volume: to ensure data persistance (containers are volatile) Docker Compose: to run application depending of multiple containers (microservices) Everything as a service Order in terms of how much of the app the vendor manages: Saas > Faas > Paas > IaaS Order in terms of the highest number of provided services: IaaS > PaaS > FaaS > SaaS 1. Infrastructure aaS Providing virtualized Server, Storage or Networking. The provider is responsible for the hardware, the client is responsible for every other aspect. Examples: EC2, S3 2. Platform aaS Providing VMs, OSs, SDK. The provider manages the infrastructure, the os and the enabling software; the client is responsible for installing and managing the application. Examples: Heroku, Azure, GAE 3. Software aaS Providing on-demand software. Everything is on the hands of the provider, the clients pay a subscription to access the product. Example: salesforce Benefits of SaaS Periodic Cash flow from client subscriptions + Payment flexibility (also Freemium) Updates are received by all clients simultaneously => no management of different versions Can deploy new versions as soon as tests are passed Users data Collection is easy From customers POV  PROs: access from several devices, no upfront cost, immediate updates CONs: privacy regulations, security concerns, network constraints, vendor lock-in Multi tenant Systems Single database schema shared by all system’s users. Items in database tagged with user identifier to provide isolation. PROs resource utilization: can optimize processes with control over resources updates: only one DB to be patched UI configurability by using user Profiles CONs inflexibility: fixed DB schema for all users (no adaptation) => 2 solutions: extra empty fields, extension table ID. security: DP is a single point of failure => 2 solutions: multilevel acces control, encryption of data complexity: no scalability capabilities Multi-instance Systems Each customer has it own system, adapted to his needs (including DB type and security controls). 1. VM-based software instance/database running on the client's VM suitable for 24/7 data access requests and frequent data sharing 2. Container-based isolated version of software/database running in a set of containers suitable for microservices architecture (indipendency), with little data sharing PROs flexibility / scalability: adapted to customer's needs security / resilience: data leakages only affect single customer CONs cost: more expensive due to cloud VMs cost update management: updates affect many instances, higher complexity Microservices Architecture Microservice = small-scale stateless service that have a single responsibility low coupling (interdependencies between services) high cohesion (less inter-service communication) can replicate only most used microservices can be developed in 2 weeks (8-10 ppl) Service Interaction 1. Synchronous: A calls B and waits for the response => easier to understand / debug 2. Asynchronous: A pushed request in a queue, B answers in a different queue => more efficient Service Communication 1. Direct: messages directly sent to service address 2. Indirect: messages sent to a message broker, that forwards messages to recipients (RabbitMQ) Data Distribution and Sharing each microservice should manage its own data data sharing should be as little as possible most sharing should be read-only, only a few services responsible for data updates include mechanism to update replicated data tolerance to some degree of data inconsistency (eventual consistency) CAP Theorem It is impossible for a web service to provide at the same time: Consistency = each service returns the right response to each request  Availability = each request eventually receives a response (A ➞ ¬ C) Partition-tolerance = tolerate delayed/lost messages during service Saga Pattern Implement each business transaction that spans multiple services as a Saga. Saga = sequence of local transactions (big transaction => many small transactions). If one small transaction fails, execute a series of compensating transactions. Saga Coordination 1. Choreography: each local transaction publishes and event that triggers the next one 2. Orchestration: an orchestrator handles each local transaction execution Compensating Failed Transations 1. Backward model: undo changes made by previously executed local transactions 2. Forware model: "retry later" policy Failure Management Something will go wrong, unavoidably Types of Service failures: 1. Internal: detected by the service and can be reported to requestor. 2. External: affects availability of the service. 3. Performance: heavy loads making the service unresponsive / slow. Splitting the monolith Monolithic apps are ok only if updates arent frequent. Problems: if app has a long lifetime, updates are inevitable low cohesion, with code kept together even if unrelated small change may require redeployment of the entire monolith Solution: find the separation points between monolith's services => the "seams"  create pakages representing each service context, moving the code into them Splitting Databases 1. understand which parts of the code read / write from the DB (tool: Hybernate) 2. detect database constraints (foreign-keys, tool: Schemaspy) 3. break foreign key relationships, 3 ways: expose data via API to main table, secondary table API calls the first shared static data: single table for all services, acting like a config file shared mutable data: new service handline the shared table, accessible via API Smells and refactoring Architectural Smell = commonly used architectural decision that negatively impacts system lifecycle qualities (can lead to issues such as poor maintainability, scalability, or extensibility) 1. design principles = blueprints for architectural design 2. architectural smells = alerts that the design diverges from the principles (can be many on a single principle) 3. refactorings = to steer the code back on track aligning it with design principles (can be many on a single smell) Examples { < Principle > => < refactoring > } 1. Independent Deployability pack each service in one container 2. Horizontal Scalability add service discovery add message router (load balancer) add message broker (message queue) 2. No API gateway (smell on the edge of the architecture) => add it 3. Isolation of Failures add message broker (failure tolerance)  add circuit breaker (easier to implement) use timeouts (not always implementable) add bulkhead (logical / phisical partition of microservices) 4. Decentralization split database (eventual consistency) add data manager (in between DB and microservices) merge services Single layer Teams => split teams by service