Rob is the CTO at Applause, helping leading brands quickly release high-quality digital experiences.
My last article examined the complexities of testing and digital quality, ending with a few tips on how organizations can start comprehensive testing. We highlighted a few organizations that are doing digital quality “right” because they understand that releasing software is never going to be a “set it and forget it” process.
Testing, revision, quality assurance (QA), repeat—this all needs to be seamless. But how does an organization even know if it’s doing this right? What quality metrics should you be tracking? Are there best practices? And more importantly, given how continued economic and political uncertainty worldwide affects budgets, how can businesses demonstrate testing’s value to stakeholders in an environment where there is pressure to consistently do more with less?
The problem with doing something well behind the scenes is that it’s very hard to quantify. It’s very easy to explain the value of paying for car or health insurance when you have a car accident or get sick. It’s harder if you’re a consistently safe driver and are always healthy.
Testing is a bit like insurance for digital quality. Doing it “right” means that your users might not notice you did anything at all. You identified and resolved issues before software hit the market, and the product does what it’s supposed to do. Do it “wrong,” and there are bugs, delays, angry boards, executives and customers, as well as a brand reputation for crappy quality.
Digital Quality Defined
Quality is the foundation of an organization’s brand, and many invest a lot of time and money into making sure that bugs and defects don’t make their way into production. That said, digital quality needs to be an intersectional discipline with the customer experience front and center—not just an exercise in bug hunting.
QA teams may look to test things like functionality, localization and user experience (UX), while product and engineering teams want to release software faster, but all of these connect. The companies doing it “right” look at digital quality holistically and use testing to provide product, engineering and QA teams with a connected and cohesive picture of how customers are using the product or application.
While each group may define quality differently, there are a number of common metrics that leading organizations track, including:
• Defect metrics look at ways that software behaves in unexpected or unintended ways. These can include defect density, open/close rates, quantity, functional bugs as well as different priority-level defects that can be categorized by severity.
• User experience and satisfaction metrics help organizations track trends and preferences and make changes. These include customer satisfaction and sentiment scores, Net Promoter Scores and usability testing results, as well as app store ratings and reviews on third-party sites.
• User behavior can also give organizations additional insight into quality. For example, if a company is rolling out a new app and the number of customer support tickets skyrockets, there’s probably a problem. On the flip side, higher adoption rates and increased purchase activity show success.
• Coverage metrics look at the overall quantity of testing performed including percentage of code coverage, devices and integration and can help organizations identify where they can incorporate more testing into their overall software development process. This should include unit tests and test case count by the area and/or features most important to the business.
In addition to the organization-specific metrics, there are hard scores that organizations can track to ensure that all of their applications comply with accessibility and readability guidelines like Web Content Accessibility Guidelines (WCAG) 2.2, as well as web compatibility tests that score cross-platform performance and behavior. All of these testing metrics combine to give an overview of quality, and once incorporated into an organization’s development process, need to be continuously tweaked.
There are always going to be “escapes,” or bugs that made it through both the development and QA process into the field. These are some of the biggest measures of quality. How many issues come back from users? How many are detected in production, and are you tracking those trends over time? When this new customer and stakeholder feedback is input, the depth and quality of the software application can only get better.
Putting A Number On Quality And Testing
If an organization tracks all of these metrics and invests in testing digital quality, is there ROI? There is evidence that testing can influence other “business value” metrics like revenue, customer retention rates and time to market.
According to one analyst firm, IDC, organizations using testing solutions can realize benefits worth an annual average of $3.79 million per organization, or $54,900 per application release, and save 681 hours of staff time per application release.
Defects have a real cost, and businesses testing them under resource pressure demonstrated the following annual benefits to IDC:
1. Testing staff efficiencies ($1,620,400).
2. Risk mitigation benefits and reduced cost of bugs ($1,421,800).
3. Business productivity benefits and higher net revenue ($330,700).
4. Development productivity gains ($237,800).
5. Testing lab cost savings and efficiencies ($182,300).
And the cost of doing nothing is not insignificant! Organizations like The Consortium for Information & Software Quality have estimated that the cost of poor software quality in the United States alone has grown to at least $2.41 trillion.
Test To Meet Today’s Demands
The pressure to deliver quality software products faster than competitors, while increasing market share and optimizing resources, is nonstop for today’s organizations. Testing is a critical way to do this and key to digital quality success. It should be viewed as a worthwhile investment for organizations looking to insure their brands’ reputations.
Forbes Technology Council is an invitation-only community for world-class CIOs, CTOs and technology executives. Do I qualify?
