ISTQB Mobile Application Testing Foundation Level PDF

Mobile Application Testing Foundation Level 1. Mobile World - Business and Technology Drivers - 175 mins Keywords risk analysis, risk mitigation, risk-based testing, test strategy Learning Objectives for Business and Technology Drivers 1.1 Mobile Analytics Data MAT-1.1.1 (K2) Describe how...

Mobile Application Testing Foundation Level 1. Mobile World - Business and Technology Drivers - 175 mins Keywords risk analysis, risk mitigation, risk-based testing, test strategy Learning Objectives for Business and Technology Drivers 1.1 Mobile Analytics Data MAT-1.1.1 (K2) Describe how available mobile analytics data can be used as input for the test strategy and the test plan. HO-1.1.1 (H3) Based on data collected from one or more analytics data sources (geographical location, platform, operating system version and device type distribution), select the device types to be tested and their corresponding prioritization. Note: HO-1.1.1 and HO-1.7.1 (below) may be combined. 1.2 Business Models for Mobile App MAT-1.2.1 (K2) Distinguish between various business models for mobile applications. 1.3 Mobile Device Types MAT-1.3.1 (K1) Recall different types of mobile devices. 1.4 Types of Mobile Applications MAT-1.4.1 (K2) Distinguish between different types of mobile applications. 1.5 Mobile Application Architecture MAT-1.5.1 (K2) Distinguish between general architecture types of mobile applications. 1.6 Test Strategy for Mobile Apps MAT-1.6.1 (K3) Apply characteristics and specifics of the mobile market in preparing a test strategy. 1.7 Challenges of Mobile Application Testing MAT-1.7.1 (K2) Give examples of the challenges associated with testing mobile applications. HO-1.7.1 (H1) Gather market data such as device or operating system market share for a selected region. Gather data for screen sizes and density. Create a list of five devices and calculate the expected market coverage for this list. Note: HO-1.1.1 (see earlier) and HO-1.7.1 may be combined. 1.8 Risks in Mobile Application Testing MAT-1.8.1 (K2) Describe how risks specific to mobile applications may be mitigated. 1.1 Mobile Analytics Data There are many stakeholders in the mobile world including manufacturers, platform providers, operating system (OS) providers, market data providers, tool providers and, of course, application developers and testers. In order to contribute effectively to test planning discussions and test analysis, a mobile application tester should be aware of and familiar with the following factors: The business implications of the distribution of platforms Page 1 of 8  Mobile Application Testing Foundation Level Application downloads per platform The quantity and distribution of OS versions The market distribution of various device types, including variations based on geographical location Differing screen sizes and resolutions The various input methods Camera types There are several sources of information for the above, both free and commercial-based. These include StatCounter GlobalStats [URL1], the OS vendors themselves and other third-party sources. The mobile analytics data is used to select a device portfolio for test execution that is appropriate for the target market. Tests are run over this portfolio to test the app on a device in accordance with the importance of the device. The data related to devices and their special features, if any, may also be used to design tests specific to a device type. For example, a device with heart beat sensor may need special test cases. 1.2 Business Models for Mobile Apps There are several models which can be used to monetize the work done in creating mobile applications. These include but are not limited to: Freemium, advertisement-based, transaction-based, fee-based, and enterprise applications. In addition, in-app purchases can be applied to some of these models. There are certain advantages and disadvantages for each of these approaches and the tester should keep the business model in mind whilst testing the mobile application. In a Freemium model the applications are generally free but users have to pay if the need additional features. The applications need to provide sufficient features to be attractive to the users, whilst at the same time providing advanced features for which a large number of users would be willing to pay. Advertisement-based applications display advertisements on the screen as users interact with the applications. This strategy for revenue generation is more effective if the applications are used for relatively long periods of time. The user interface designers must take care when displaying the advertisements. They must be prominent enough without hiding essential parts of the application and they must ensure that users are not distracted and dislike using the application. Transaction-based applications charge the users either per transaction, a flat fee or a percentage of the transaction value or similar. This type of business model is suitable for a limited number of applications only and is usually applied for business and financial apps such as mobile wallets. Fee-based applications require the users to pay for downloading and installing the application. Deciding on a fee-based business model should be well-considered since large numbers of free or freemium options exist for most application types. The probability of users buying such an app increases if it provides outstanding features or usability, or when competing applications are not available. Free and enterprise applications do not charge their users. Enterprise applications are developed for internal use within the organization and provide an interface to the services provided. There are many such apps available from organizations such as banks or e-commerce companies. These apps are not generally focused on monetizing the app itself but allow revenue to be generated by directing users to the services provided by the organizations. 1.3 Mobile Device Types There is a variety of mobile devices available that support different types of applications. Page 2 of 8  Mobile Application Testing Foundation Level Typical devices include: Basic phones Feature phones Smartphones Tablets Companion devices - including wearables and some IoT (Internet of Things) devices. When testing it should be kept in mind that each type of device has specific features for particular needs. Basic phones are used for telephone and SMS only and provide very few built-in apps and games. The installation of apps or browsing is not possible. Feature phones provide limited support for apps and app installation. They provide internet access via a built-in browser and may have some additional hardware such as cameras. Smartphones provide phones with several sensors. The operating system supports features such as application installation, multimedia support and browsing. Tablets are similar to smartphones but are physically larger. They are typically used when a larger display is needed or desired and they may also support longer battery life. Companion devices and some IoT appliances are computer-powered devices commonly used together with a smartphone or tablet to extend the available functionality or to give access to the data on the phone or tablet in a more convenient way. Wearables are devices that can be worn by consumers. These can act as a companion to existing devices or function independently. Watches and fitness bands are examples of popular wearables. 1.4 Types of Mobile Applications There are three main types of mobile application: Native Browser-based Hybrid Each type of application has certain advantages and disadvantages, requiring a business decision to be made before starting the application development. Native applications are developed using platform specific software development kits (SDKs), development tools and platform specific sensors and features. They are downloaded, installed and updated from supplier stores. These apps may need testing on all supported devices. Native applications generally provide better performance, can fully utilize platform features and comply to the expectations for the platform they are developed for. The development cost is typically higher and additional challenges may apply such as the use of multiple platforms and the installation and testing on a large number of devices. Browser-based applications are accessed through a mobile browser. Since these use the typical web- development technologies and browsers, multiple platform support is easy, and the development cost is usually lower. There are four main ways in which mobile web applications are created: Mobile specific versions of websites and applications (these are also known as m(dot) sites). Usually this means that when a mobile browser addresses the application, a mobile version of Page 3 of 8  Mobile Application Testing Foundation Level the application is provided. For example, facebook.com redirects to m.facebook.com when accessed from a mobile device. Responsive web apps ensure that the design adjusts to the form factor and screen size, usually expressed as view ports. Adaptive web apps adjust the design according to some predefined sizes. There are different designs for these sizes and the features available to the user are often adjustable. Progressive web apps allow shortcuts of specific web pages to be created on the mobile home screen. They appear like native apps and sometimes even can work offline. Mobile web apps are created using common web technologies, which generally makes them easier to develop and manage compared to native and hybrid apps. They may however not be as feature-rich as native or hybrid apps and may have limited access to the platform’s native Application Programming Interfaces (APIs). The access to mobile sensors is also limited. Installability testing on devices is not needed, but browser compatibility testing is required. Hybrid applications are a combination of native app and web app. They use a native app wrapper which contains a web view to run a web application inside of a native app. These apps are downloaded from supplier stores and can access all of the device features. They are relatively easy to develop, update and maintain without updating the app installed on the device. The skills required for developing these apps are almost the same as for web development. Possible weak points for these apps include performance issues due to the use of a wrapper and possible divergences from the expected look and feel because of platform-specific aspects. Native and hybrid apps are installed physically on a device and are therefore always available to the user, even when the device has no internet connection. In comparison, browser-based applications require internet access. Some applications are pre-installed on the mobile device and others can be installed via various distribution channels, such as Apple App Store, Google Play Store, enterprise app stores (available only inside the enterprise network) and third-party app markets. Testing of each of these application types may require a different approach. The parameters to consider include: Different types of devices to be supported Sensor and device features to be used Availability under various network conditions Installability, compatibility, performance efficiency, and usability 1.5 Mobile Application Architecture There are multiple solutions to designing a mobile application. Some of the considerations in choosing a particular architecture or design decision include: Target audience Type of application Support of various mobile and non-mobile platforms Connectivity needs Data storage needs Connections to other devices including IoT appliances Page 4 of 8  Mobile Application Testing Foundation Level Architectural decisions include: Client-side architecture such as thin or fat client Server-side architecture such as single or multi-tier Connection type such as Wi-Fi, cellular data, Near Field Communication (NFC), Bluetooth Data synchronization methods such as store-and-forward, push and pull, synchronous and asynchronous communications Thin client apps do not contain application code which is customized to the device and make minimal use of mobile operating system features. These apps typically use the web browser as the front-end and JavaScript as the language for implementing client-side logic. Thick/fat client applications may have multiple layers of application code and may use mobile operating system features. These are typically Native or Hybrid applications. The server-side architectures include the following possibilities: Single-tier architectures are monolithic and have all servers on the same machine. They are less scalable and harder to secure. Multi-tier architectures spread server-side components across various units. Two-tier architectures involve separate web and database servers, whereas three-tier architectures also include an application server. Multi-tier architectures allow separation of responsibilities, provide database specialization and provide better flexibility, scalability and security. However, they may be significantly more expensive to develop, manage and host compared to single-tier architectures. There are various connection methods. A mobile device might be connected to the server via connection types such as Wi-Fi or via cellular data connections such as 2G, 3G, 4G, and 5G. Mobile applications typically operate in one of the following three modes: Never-connected apps work offline and don’t need to be connected. A simple calculator is an example of such an app. Always-connected apps require a permanent network connection during operation. All mobile web applications fall into this category, although some can operate in a limited way when partially connected. Partially-connected apps require a connection for tasks such as data transfer but can operate for long periods of time without connection. The synchronization of data between the client and the server can be conducted in the following modes: Continuous mode is where the data gets transferred as soon as it is submitted. Store-and-forward mode is where the data may be stored locally before being transferred, especially when no connectivity is available. The data transfer can be performed in the following two approaches: Synchronous data transfer is performed when the calling function waits for the called function to complete before returning. Asynchronous data transfer is performed when the called server function returns immediately, processes the data in the background and calls back the calling client function once it completes the task. This give users more control. However, implementing Page 5 of 8  Mobile Application Testing Foundation Level the handshake mechanism increases complexity concerning the availability of the client or the network when the server initiates the callback. 1.6 Test Strategy for Mobile Apps Creating a test strategy for mobile devices requires the tester to take into account all the parameters listed so far in this chapter. In addition, the risks discussed in this section and the challenges described in section 1.7 must also be considered. Typical risks are, for example: Without knowing the device proliferation data in a particular geographic location, one cannot choose the devices on which the app needs to be tested in a sustainable fashion. Without knowing the type of business model, one cannot test whether the application behavior is a good fit for that business model. Creating a test strategy for mobile application testing additionally needs to consider the following specific risks and challenges: The variety of mobile devices with device-specific defects on some of them. The availability of devices in-house or via the use of external test labs. The introduction of new technologies, devices and/or platforms during the application life cycle. The installation and upgrade of the app itself via various channels, including preserving app data and preferences. Platform issues which might impact the application. Network coverage and its impact on the app in a global context. The ability to test using the networks of various service providers. The use of mobile emulators, simulators and/or real devices for specific test levels and types of test. These challenges are described in greater detail in the section 1.7. The test strategy takes risks and challenges into account. For example: The test strategy may specify the use of mobile emulators/simulators in the early stages of development, followed by real devices in later stages. There are certain types of tests that can be performed on the mobile emulators/simulators but not all types of tests. More on this is described in section 4.3. The test strategy may consider the challenge posed by a large number of different devices by adopting one of the following approaches: o Single platform approach: Reduce scope to a single type of device, one OS version, one carrier and one network type. o Multi-platform approach: Reduce scope to a representative selection of devices and OS used by a majority of customers in the target market, based on mobile traffic or other analytical data. Page 6 of 8  Mobile Application Testing Foundation Level o Maximum coverage approach: Cover all OS versions, devices, manufacturers, carriers and network types. This is basically exhaustive testing, which is usually not economically viable, especially when considering the multitude of devices and OS versions on the market. The test strategy may consider the challenge posed by the non-availability of devices, networks or real-life conditions by using external resources, such as: o Remote device access services. This is a way to access devices over the web which are not otherwise owned. o Crowd testing services. This is as a way to access a huge group of volunteers and their devices. o Personal networks such as friends and colleagues. This makes use of one’s own private social network. o Bug hunting. This is gamified testing event using volunteers from the company or from the general public. In addition to the test levels described in [ISTQB_CTFL_2018] the test strategy also considers the common types of testing used for mobile applications (see section 3.1) and any additional levels of testing required (see section 3.2). 1.7 Challenges of Mobile Application Testing In the mobile world many additional challenges exist that are uncommon or uncritical in desktop or server software. Testers must be aware of these challenges and how they might impact the success of the application. Typical challenges in the mobile world include: Multiple platforms and device fragmentation: Multiple OS types and versions, screen sizes and quality of display. Hardware differences in various devices: Various types of sensors and difficulty in simulating test conditions for constrained CPU and RAM resources. Variety of software development tools required by the platforms. Difference of user interface designs and user experience (UX) expectations from the platforms. Multiple network types and providers. Resource-starved devices. Various distribution channels for apps. Diverse users and user groups. Various app types with various connection methods. High feedback visibility resulting from bugs that have a high impact on users which may easily result in them publishing feedback on online market places. Market place publishing which requires additional approval cycles for publishing by market place owners such as Google Play or Apple App Store. Page 7 of 8  Mobile Application Testing Foundation Level Unavailability of newly launched devices requiring the use of mobile emulators/simulators The impact of these challenges includes: Large numbers of combinations to be tested. Large numbers of devices required for testing, which drives up the cost. The need for backward compatibility to run the application on older versions of the platform. New features being released in every version of underlying operating system. Guidelines to be considered for the various platforms. Resource-starved CPUs as well as limited amount of memory and storage space. Varying bandwidth and jitter of various networks. Changes in the available upload and download speeds based on data plans. The following two examples illustrate typical challenges and their potential impact: Different devices have different types of sensors and tests need to account for these. Every new sensor added to the hardware may require additional backward compatibility testing. Some of the network challenges can be dealt with appropriately, even under varying network conditions, by using appropriate caching and prefetching strategies. However, this comes at a cost; a large number of open connections can impact the server-side performance as most apps keep the user logged-in on the server. 1.8 Risks in Mobile Application Testing The challenges mentioned in section 1.7 can appear in isolation or in combination with others. This may result in additional risks for a mobile application. A tester must be able to contribute to the product risk analysis. Common risk analysis and mitigation methods, as discussed in [ISTQB_CTFL_2018], chapter 5.5, can also be applied in the mobile context. In addition, the following mobile-specific risks and mitigation strategies exist: Risk Possible mitigation Market fragmentation Choose an appropriate selection of devices for test execution, e.g., testing the most commonly used devices. Cost of supporting multiple platforms Perform analysis to understand most used platforms, thus avoiding testing of those no longer in scope. Introduction of new technologies, Use pre-production versions of those technologies. platforms and devices Lack of availability of devices for test Apply remote device access services or crowd testing execution services. Risks from the expected usage Applying appropriate testing approaches such as field patterns of mobile applications used testing while on the go Page 8 of 8