Over the last several years, the rise of the smartphone in the wireless market has led to substantially increased capabilities in mobile devices. The emergence of an applications ecosystem for these platforms has enabled an almost limitless number of ways for consumers and enterprise users to interact with people, systems and businesses. Supporting much more than telephony, today’s smartphones function as a digital appendage, playing the role of a watch, camera, map, book, wallet, laptop, newspaper, shopping cart, game console, personal music player, and even radio and television.
The range of hardware components such as chipsets, radios, displays and batteries, along with operating systems that are used in designing mobile devices, leads to a huge variety of implementations. More importantly, it also results in a wide range of device performance levels across use cases and applications that are unique to each mobile device. Not surprisingly, this contrast in performance among devices translates to differences in the mobile experience from the vantage point of the end-user.
Recent focus has been on the user experience that results from the continued rollout of mobile broadband services, enabled by technologies such as LTE, that greatly increase the demand for IP bandwidth and resource-intensive applications such as streaming video and online gaming. Furthermore, all-data LTE networks use IP Core Network Multimedia Subsystems (IMS) that will, for the first time, shift carriers’ voice service offerings to the data realm. In the United States and Korea, the first commercial deployments of this technology, known as Voice-over-LTE (VoLTE), rolled out in 2012. The challenge of ensuring an all-around enjoyable mobile experience in the face of these changes is immense.
Overcoming these challenges requires development and execution of a comprehensive and robust series of test scenarios prior to the launch of the device. This means implementing test methodologies that offer repeatability of results, scalability in design, and flexibility across device platforms, operating systems that include Android, Windows Phone and iOS among others, as well as across wireless network technologies. In addition, the measurement systems and tools employed must offer the capabilities and feature sets that objectively measure, provide in-depth analysis and reporting, and automate the data collection process. Fortunately, solutions to meet these difficult measurement challenges do exist and will be discussed later in this article.
Why Is the Mobile Experience So Important?
A poor end-user experience on a mobile device can be devastating to a model, a product family, a brand and at times to the network operator that offers the device in its portfolio. For example, a YouTube video stream that is pixelated, choppy or interrupted midstream more than a couple of times can rapidly spiral from an annoying end-user experience to a highly frustrating one. Even in the presence of ample bandwidth, a mobile device’s inability to process and render video frames, or to correctly synchronize audio with video, can still leave the end user dissatisfied.
Similarly, the performance of a device in modulating and presenting uplink and downlink speech under conditions such as RF channel degradation with changes in signal strength, and acoustic degradation in the presence of background noise, can greatly impact end-user experience. Voice quality is becoming even more critical as operators around the world prepare to deploy VoLTE. The application that is still used most frequently on smartphones is the one that has been with us from the very beginning: making voice calls.
The successful launch of carrier-grade voice service with VoLTE is a critical element in operators’ business plans going forward. End users will not accept a step backward in voice quality when services shift to VoLTE, yet the challenge of ensuring voice quality is high on an all-data network already burdened with a multitude of bandwidth hungry services. Even now, as operators are in various stages of transition to converged voice and data networks — including dual-radio devices or Circuit-Switched Fall Back (CSFB) — there is an urgent need to monitor the quality of experience for their distinct implementations of simultaneous voice and data.
Other components that make up the smartphone, especially the battery, also play a major role in quality of the end-user experience. Although recent mobile devices have been equipped with higher capacity batteries to help alleviate the annoyance of short battery lives, the introduction of additional processing power, high definition displays, and even flash-photography-capable devices have increased the power demands on smartphones, offsetting the gains made from the increased power storage and keeping overall battery life stagnant. In fact, poor battery performance has emerged as a principle cause of subscriber dissatisfaction with LTE-equipped mobile phones.
The Keys to Measuring the Mobile Experience
A major challenge in gauging the quality of experience on a given device is identification of performance indicators that are objective, quantifiable, and a predictive measure of end-user experience. This challenge is amplified by the explosive growth in use cases supported by smartphones, which expands the range of performance areas beyond just speech quality and data speeds. Furthermore, the challenge only becomes larger and more expensive to address when considering other dimensions, such as the multiple technologies utilized by access networks, design limitations introduced by a mobile form factor and the complexities of mobility testing in the field.
Fortunately, it is now possible to accurately and objectively measure mobile experience. Some of the most important requirements are: operating system and network agnostic, access technology agnostic, end-user measurements, centralized test administration, diagnostics, benchmarking and applicability to testing in both live networks and the lab.
Operating System and Network Agnostic
In other words, measurements need to work on any device, on any network, and in any place where wireless service is offered. This requires compatibility with all major mobile operating systems and a global infrastructure that will allow service testing anywhere in the world.
Access Technology Agnostic
Mobile experience issues are not limited to the latest LTE wireless access technology, and many issues can occur when mobile phones fall back to legacy 2G and 3G technologies. Measurement systems must therefore be agnostic of the radio access technology.
To be truly indicative of the mobile experience, measurements must be made the same way users interact with the services on their mobile phone. Examples of this include acoustically coupled audio quality measurements, camera-captured video quality measurements, and application-level data measurements (e.g. browser-based web surfing).
Centralized Test Administration
With measurements often collected in many places by multiple teams over multiple days, it is critical to have a central test administration online site that provides a single interface for test configuration, reporting and analysis of test results.
Getting objective and reliable mobile experience measurements is an important first step, but it is also critical to provide diagnostic capability that aid in diagnosing and debugging any issues found.
The usefulness of quantifying the experience of a single device is greatly amplified when the measurements can be benchmarked against other devices or networks. For this reason, mobile experience measurement tools should include a database with data analysis tools for comparing metrics versus devices, networks, or time.
Applicability to Testing in Both Live Networks and the Lab
Since mobile subscribers experience services over the network, measurement tools must be capable of testing in the field on live networks. Steps must be taken to ensure the integrity of measurements made in the constantly-varying real world, and when testing demands a perfectly controlled environment for precise measurement, the tools should also work in the lab. Lab testing that allows for reliable recreation of the environment enhances the ability to troubleshoot problems when they occur.
Spirent Solutions for the Mobile Experience
Spirent Communications assists and enables device manufacturers and mobile operators to assess the quality of devices prior to launch and to ensure consistent improvement of the mobile experience. Spirent’s Fit4Launch measurement system portfolio of solutions includes Datum software and mobile client for data and multi-service performance measurement, Chromatic video measurement system, and Nomad for speech quality and call performance. Spirent’s Lab2Live portfolio of lab test solutions enables repeatable recreation of the real-world environment and uses the same Fit4Launch measurement systems for correlation of metrics between live and lab. The solutions offer objective measurement, correlation and analysis to effectively predict the quality of mobile experience. Additionally, Spirent’s comprehensive Fit4Launch and Insights suite of services deliver turnkey and cost-effective Testing-as-a-Service (TaaS) offerings that are used to evaluate the performance of a device relative to its peers or client-established benchmarks.
Improving the Mobile Experience
With expectations for smartphone services already sky-high, the introduction of LTE and VoLTE is driving an urgent need for more realistic and objective quantification of the mobile experience. While many tools promise to measure relevant metrics, only Spirent has a history of delivering solutions that work on any device, on any network, at any place in the world, and which provide centralized test administration for results analysis, benchmarking, diagnostics, and test management. Spirent’s subject matter expertise in both live network and lab-based mobile experience solutions puts it in a unique position to help the entire wireless ecosystem improve the service experience offered on the mobile devices and wireless networks of today and tomorrow.
About the Author
Joining Spirent in 2011, Ardeshir is responsible for the development of new and existing services within Spirent’s business lines. He brings with him 16 years of telecom industry experience in network management, professional service delivery and Voice over IP solutions. Prior to joining Spirent, Ardeshir was Vice President of Operations at VoIP Logic and and Founder of Sundial Network Services. He holds a BS in Telecommunications Management from New York Institute of Technology and MBA in International Business and Finance from CUNY-Baruch College.
Over the last several years, the rise of the smartphone in the wireless market has led to substantially increased capabilities in mobile devices. The emergence of an applications ecosystem for these platforms has enabled an almost limitless number of ways for consumers and enterprise users to interact with people, systems and businesses. Supporting much more than telephony, today’s smartphones function as a digital appendage...