Programming in the Large

Most new cars come with a 3 year / 30,000 mile warranty. Most new software programs come with a legal disclaimer disavowing any responsibility for the product. Clearly the software industry has room for improvement when it comes to building quality products.

Automobiles weren't always the shining example of reliability and quality which they are today. When automobiles first started appearing on the roads in the early twentieth century, drivers thought nothing of stopping every few hundred miles to fix a flat tire. As the novelty of driving wore off and competition intensified, automobile manufactures had to improve the quality of their products in order to maintain interest and stay ahead of the competition.

Some of the same forces are at work today in the software industry. The first software systems were amazing in part because of their originality. Users were delighted, or at least content, to receive anything that generally worked. During the 1970's and 1980's software was like a dancing bear-- it's not important how well the bear can dance, it's just amazing that the bear can dance at all.

Today the bear is dancing faster (almost desperate for attention) but users expect even more. Software is becoming just another product or service: vital like electricity and clean water, but expected and sometimes taken for granted. Viewed in this light, the expectations are that software should have a level of quality comparable to other everyday products. Some would argue that it's unfair to have quality expectations of software on par with those for other products because software is more complex than other products. In particular, every software system is a unique intellectual product designed and developed rather than a copy of an existing product manufactured according to a well-defined process. Nevertheless, users increasingly view software as just another product or service and consequently expect software to have a level of quality that is on par with other everyday products and services. Expectations become reality and so quality is taking an out-sized role in product acceptance.

The demand for software quality can be a source of opportunity as well as challenge. The only thing worse than being "just another product" is being a commodity product. Commodity products compete on price alone and competition in a commodity market is often a race to the bottom. Software providers looking to distinguish their offerings can use quality as a differentiator. Just as some patrons choose a restaurant based on quality, quality can also be the deciding factor in a software purchase decision.

Quality can also be the source of pricing power. Whether it's handbags or golf clubs, the provider with the best reputation for quality usually has the power to raise prices when others in the same market segment can't. In a mature market products compete on quality and only the highest quality products can justify a price premium. Not all software is sold commercially but all software has value and its development is justified by the value that it delivers to its customers. Commercial vendors can use quality to justify a price premium. Internal software development groups can use quality to justify their continued existence.

Beyond competition, software quality is vital for some applications. The world is becoming increasingly dependent on software and it's taking on a greater role in life- and mission-critical systems. For these types of systems the costs and risks of poor quality are making quality the number one concern over and above other priorities such as schedule and budget.

Introduction

A broad range of practices are needed to achieve high levels of software quality. Although there are no strict dependencies on what order these practices are adopted, organizations striving for higher levels of software quality often adopt them in the order depicted in figure 1.

Figure 1. Route to Improved Quality

Testing is the most common approach to ensuring quality. Testing is defined as the operation of software under simulated conditions for the purpose of finding defects. No reasonable organization would skip testing, but on the other hand, many haven't evolved beyond a test-only approach to software quality. Testing alone is a narrow and somewhat limited method of avoiding defects because:

1. Testing is performed too late in the product cycle to be an efficient method of avoiding defects. With most software development methodologies, testable code isn't produced until late in the development cycle. Even when doing short iterations, a lot of work is performed before the first line of code is written. Testing is an inefficient method of detecting defects because the cost of finding and fixing a defect increases dramatically with the length of time the defect exists in the system. Starting the search for defects late in the software development life cycle increases the expense of finding and removing defects that are introduced earlier in the software development life cycle.
2. Testing addresses only a narrow, although important, dimension of quality. Testing has the potential to improve the reliability of the end product but no amount of testing will improve its maintainability, extensibility, etc. These other dimensions of software quality have to be engineered into the product from the beginning.
3. Only the quality of code is improved through testing. A software product or solution is more than just code. It also includes requirements, designs, user manuals and other documentation. The supporting documentation of a software system has a significant influence on the overall quality of the system but can't be verified through testing.

Because of the limitations of testing most organizations also adopt some form of product reviews to complement their testing efforts. Reviews vary in type from informal peer reviews, such as asking a colleague to proofread a document, to formal and rigorous inspections with their well-scripted roles and responsibilities. The combination of testing and reviews is particularly effective because reviews complement testing and compensate for many of the limitations of testing. Work products other than code can be reviewed early in the product cycle which allows errors to be found early, closer to their point of origin. The types of errors found during reviews also tend to complement those found through testing. Testing verifies that a product works under certain simulated conditions. Reviews verify that a product has the qualities it needs to work under changing or unanticipated conditions. In terms of product requirements, testing is better at verifying functional requirements and reviews are better at verifying non-functional quality requirements like maintainability, reliability, reusability, etc.

Beyond testing and reviews, when quality is crucial or when there are outside stakeholders that need additional assurance of software quality, organizations will institute some form of quality assurance. The term quality assurance is sometimes used as a synonym for testing or as a reference to an independent test group. The correct use of the term refers to the function which assures management and other project stakeholders that software products have their planned levels of quality [IEEE Std. 610-12-1990]. The quality assurance group (or individual if it is a small project):

1. Works with development engineers to define appropriate standards and procedures of software development.
2. Monitors their work through reviews and audits to verify that these standards and procedures are being followed.
3. Provides feedback to upper management and other stakeholders on progress towards quality goals.

It's more difficult to generalize software quality practices beyond quality assurance because there are fewer organizations working at this level from which to generalize. However, software process improvement models [CMMI] and approaches in other disciplines [PMI and Juran] suggest that implementation of a comprehensive system of quality management is the next step on the road to higher software quality. Quality management is an umbrella term that encompasses several quality-related processes including quality planning, quality control, quality assurance, and verification and validation. Each of these processes can be performed individually but performed together they have a synergistic effect. Quality management includes all activities undertaken to establish quality objectives and then manage and control the execution of a project toward the achievement of these objectives. Put another way, quality management is the application of engineering and management techniques toward the achievement of project quality goals. Organizations routinely plan and manage schedule and budget performance. Quality management is about planning and managing quality goals with equal or near equal precision.

The final quality-related practice discussed in this chapter is defect prevention. If poor quality is a disease, testing and inspecting is just treating the symptoms, whereas defect prevention is a cure for the underlying problem. Defect prevention is about identifying the root causes of defects and taking specific actions to remove the causes and prevent the defects from occurring again in the future. Defect prevention has the potential to improve product quality and development productivity. Fewer defects means less time spent finding and fixing errors.

Given the number and variety of quality-related topics it's impractical to cover them all in one chapter. This chapter explores the key concepts and principles of software quality and describes in more detail quality-related practices beyond testing and inspections. This includes .

The functions described here (quality control, quality assurance, quality management and defect prevention) rely on and interact with other quality-related functions that are described in more detail elsewhere. Specifically,

• Verification and Validation. Verification is ensuring that a product meets its technical specifications and validation is ensuring that it meets the user's needs and expectations. The two main approaches to verification and validation are testing and inspections. Software testing and inspections are described in more detail in chapter 19.
•  Measurement and Estimation. It's well known that you can't control what you can't measure so in order to control quality it is important to be able to measure the quality of processes and products. The details of defining and using quality metrics are discussed in chapter 23.
• Software Process Assessment and Improvement. One way to improve the quality of products is to improve the quality of the processes used to produce these products. A well-defined process has a certain quality capability and average performance. The capability of a process is the range of results that can be expected from the process and average performance is a measure of the effectiveness of the process. Without changing the process there is a certain limit or ceiling on the amount of quality that can be expected from products of that process. Process improvement is about improving the quality capability and average performance of a process. The focus of the present chapter is on understanding the quality-related components of the software development process. The focus of chapter 26 is on improving these and other components of the software development process.

Concepts and Principles

What is Quality?

Quality is an elusive concept. Most people can recognize quality and give examples of quality, but have trouble supplying a clear, unambiguous definition of the term. In everyday discourse this usually isn't a problem because language is very tolerant of ambiguity. However, before the quality goals of a project can be specified, managed, and assured, we need a good working definition of the term quality.

This first section explores different perspectives on quality in general. These perspectives provide a foundation for the definition of software quality in particular presented in the next section.

The following are the most common perspectives on quality [Garvin, Crosby]:

1. Transcendent
2. User-Based
3. Conformance to Requirements*
4. Product-Based
5. Value-Based

Figure 2 shows a conceptual depiction of each one of these perspectives on quality.

______________________
*Garvin refers to conformance to requirements as the manufacturing-based view because it is the view preferred by those responsible for the production phase of a manufacturing process. The term conformance to requirements is used here because it is more descriptive of the software development environment where products are developed rather than manufactured.
______________________

Figure 2. Perspectives on quality.

Transcendent. The transcendent view of quality equates quality with "innate excellence". According to this perspective, quality can't be described with words, it can only be understood through experience. The transcendent view of quality relies on perception and experience but isn't completely subjective. For example, wine tasting experts generally agree on the relative merits of different vintages of wine which suggests that the activity isn't completely subjective. Robert Pirsig takes on the topic of quality in Zen and the Art of Motorcycle Maintenance and concludes, "Quality is not objective... It doesn't reside in the material world....Quality is not subjective...It doesn't reside merely in the mind....Quality is neither a part of mind, nor is it a part of matter. It is a third entity which is independent of the two." [Pirsig]

The transcendent view of quality is a poor basis from which to develop an operational definition of quality. The most obvious problem is measurability. There is no precise way of quantifying innate excellence. The transcendent view of quality is better suited to products that appeal to the senses. Software is practical and utilitarian and therefore not well suited to the transcendent view of quality.

User-Based. The user-based view of quality is certainly more down-to-earth than the transcendent view. According to the user-based view of quality, quality is the satisfaction of user wants or needs. If product A satisfies more user wants and needs than product B, then product A has higher quality than product B.

There are two potential problems with this definition of quality though. The first problem is that of designing a product that simultaneously meets the needs of a diverse group of users. For example, the popular image editing program PhotoShop is used by average consumers, professional photographers, and high-end animation studios. It would be difficult to design a product that simultaneously meets the needs of this diverse group of customers. What might be satisfying to one customer might be completely unacceptable to another.

Figure 3. Quality is only one component of user satisfaction

The second problem with this perspective is more fundamental. User satisfaction is based on an aggregate of product factors--only one of which is quality. (See figure 3.) Just because a user is satisfied with a product, you can't conclude it is exclusively because of high quality. A user might be satisfied with a low quality product if the other factors that influence satisfaction are high enough to compensate for the dissatisfaction attributable to low quality. To say that user satisfaction is a good measure of product quality is to ignore the other factors. For example, there is a robust market for classic Jaguar automobiles from the early 90's. This suggests that these cars are providing user satisfaction. However, Jaguars of this vintage are notorious for electrical and mechanical problems. These cars might be meeting the overall needs of their buyers, but even their most enthusiastic admirers would freely admit that they are not high quality. More likely, it is a combination of product features including their visceral beauty and style that make them appealing in spite of their reputation for poor quality. (In fact, for some owners it's possible that the sense of adventure that comes from driving a low quality unpredictable automobile adds to their appeal.)

There is certainly value in having a definition of quality that defines quality relative to user wants and needs. However, before this perspective can serve as a basis for quality management, there has to be some way of measuring the extent to which product quality is contributing to user satisfaction independent of other product features that might also be influencing overall user satisfaction.

Conformance to requirements. Quality as conformance to requirements is a popular operational definition of quality. Philip Crosby, respected author and evangelist of quality, is probably the best-known advocate of this interpretation [Crosby, Quality is Free, p. 17]. It is also the definition used in the well-known ISO 9000 quality standard.

When quality is defined as conformance to requirements, there is no ambiguity about what quality is. A quality product is one that conforms to specified requirements and design. According to this definition of quality, a meal at McDonalds is high quality as long as it conforms to the specifications of the franchise. By the same token a meal at the Four Seasons could be considered of lesser quality if, for example, the salad fork isn't chilled.

If this definition of quality bothers you it's probably because you are equating quality with luxury or goodness [Crosby]. Viewed in this way, a Lexus has more quality than a Hyundai. A Rolex watch has more quality than a Timex watch. This is a common interpretation of quality but it's incompatible with the interpretation of quality as conformance to requirements. Quality as conformance to requirements is absolute. Quality can't be both conformance to requirements and luxury or goodness. The source of the confusion is in the use of the same term for two different concepts. A manufacture may plan a product with a certain level of quality (luxury or goodness). In order to deliver on this plan, the manufacture will need to maintain high standards of quality (conformance to requirements) during production. Quality as conformance to requirements is a good operational definition of quality because manufactures of all types of products can specify quality goals and control progress towards the accomplishment of these goals.

Another potentially confusing aspect of defining quality as "conformance to requirements" is that it seems to disregard the needs of the user. Quality as conformance to requirements doesn't discount the importance of meeting the needs of the user, it just assumes that these needs will be completely expressed in the requirements. This puts the onus on the requirements process to capture the true requirements with accuracy and precision.

The principle advantage of defining quality as conformance to requirements is that it simplifies the production or implementation phase of the product life cycle. It doesn't make it any easier to identify the right set of product features that will lead to customer satisfaction, but it does make it easier to manage and control the production or implementation phase of the product life cycle.

When quality is defined as satisfaction of user needs the whole life cycle is working toward subjective hard-to-measure goals which makes the whole process hard to control. When quality is defined as conformance to requirements, the requirements phase isn't any easier to control, but the design and production phase are working toward an objective easier-to-measure goal--conformance to written requirements. The advantage of defining quality as conformance to requirements is that it makes it possible to define quality goals in specific terms and then measure and control progress towards these goals.

Conformance to requirements is the definition of quality preferred by manufactures because it provides an objective easy-to-measure goal for quality during the production phase of the product life cycle.

Product-Based. With the product-based approach to quality, differences in quality are measured by differences in the quantity of some ingredient or attribute of the product. For example, one way to measure the quality of fabric is by thread count. Fabrics with a higher thread count are usually considered more desirable.

In practice, the quality of a product is often a function of multiple attributes, some more important than others. For example, three attributes that can be used to measure the quality of a digital camera are resolution, light sensitivity and frame rate. These aren't the only attributes by which the quality of a digital camera can be measured and how much each attribute contributes to overall product quality will be different for different buyers.

Camera Quality  =  W_Resolution * Resolution   +   W_Sensitivity * Light Sensitivity   +   W_{Storage Capacity} * Storage Capacity

The challenge when defining product quality using this definition is determining the attributes and their weights. If the choice of attributes and weights are subjective, there may be no choice of attributes and weights that is optimal for all customers.

The section on Software Quality Models in this chapter describes various models for defining software quality in terms of product attributes.

Value-Based. The value-based approach to quality considers value to be an aspect of quality. Adding features to a product and controlling its defects, beyond a certain point, will likely increase its cost. Higher cost means less value. Tying value to quality means that as the cost of a product goes up, its perceived quality goes down. According to the value-based approach to quality, a quality product is one that has the features the customer desires, limited defects (if any), and is offered at a price that is acceptable to the customer. This approach to quality explains why, when shopping for flooring, linoleum starts to look better than tile after seeing the price of each.

In practice, this view of quality confuses product quality with desire for a product. A customer may marvel at the quality of a pair of shoes, but upon seeing the price may loose interest in purchasing them. The quality of the shoes haven't changed but the customer's desire for owning a pair diminishes because they don't represent good value for the customer.

The relationship between software quality and cost is explored in the section titled Cost of Quality.

What is Software Quality?

The perspectives on quality presented above are purposefully general--they apply to any product in any environment. Our interests are more specific. Our goal is to develop a working definition of software quality that can serve as a basis for managing software quality during a project.

The motivation for defining software quality is that it is one of the 4 constraints or variables of a project (along with time, cost and features) that must be planned and managed. You can't plan and manage the quality goals of a project unless there is universal agreement among all stakeholders about what quality is*.

____________________________
*Note, we aren't yet talking about specifying the quality goals of a project. Right now we are only interested in defining software quality. Later we will build on this definition and explore techniques for specifying and managing the quality goals or requirements for a project.
____________________________

It's interesting to note that of the 4 constraints on a project, quality is the only one that is in danger of being misunderstood and therefore needs to be defined. There's no need to define time, everyone knows what a day, week or month is. There's no need to define cost because money is universally understood. There's no need to define what a feature is because, while stakeholders might argue over the utility of a feature, no one would argue whether or not something is a feature. However, as the multiple perspectives on quality presented above demonstrate, there is no universal agreement on the meaning of the term quality. If the definition of software quality isn't well-defined and understood by all stakeholders, disagreements and misunderstandings are likely to follow.

Another more subtle problem is that if quality isn't well-defined it's likely to be the first of the 4 project constraints to be compromised--irrespective of its true importance to overall project success. Project success is delivering the features and quality the customer desires at a cost and schedule that is acceptable. Cost and schedule must be balanced with features and quality. Once a project is started, if there isn't enough time and money to deliver the features and quality promised, something has to give. It should be the least important constraint, but in practice it's hard to resist compromising the most convenient constraint. Cost, time and features are well-defined and easily recognizable. If one of these is compromised, the effects will be immediately obvious. On the other hand, if quality isn't well-defined it can be compromised in favor of other more visible project constraints without any immediate consequences. Compromise schedule, budget or features and you have to publicly acknowledge a failure to meet the original project goals. Compromise on quality and it's someone else's problem for another day. The bottom line is: when quality is well-defined, tradeoffs among project constraints are more likely to be based on true long-term priorities rather than short-term conveniences.

Figure 4. Project success is finding the right balance among cost, time, features and quality

Quality is important but it's not everything. Quality is only one component of project success. Project success is delivering desired features at an acceptable level of quality according to the agreed upon schedule and budget. Anyone who declares an unwillingness to compromise on quality probably means well, but isn't considering the bigger picture, which is project success. Users may be willing to accept a little less quality if it means faster delivery, less cost, or more features. Engineers have an especially hard time compromising on quality because:

1. They are closely involved with product creation and so understandably have pride of authorship.
2. Engineers are directly affected by poor quality because they have to maintain the software into the future.
3. Conversely, engineers are less affected by compromises to the other project constraints.

When it comes to setting project goals, quality is just one component of project success that must be balanced with other considerations such as cost and schedule.

Having made the case for why it is important to develop a working definition of software quality at the beginning of a project, we now turn our attention toward deriving such a definition. The IEEE provides two definitions relevant to the topic: one for quality in general and another for software quality in particular.  The IEEE standard glossary of software engineering terminology defines quality as:

Quality - (1) The degree to which a system, component, or process meets specified requirements. (2) The degree to which a system, component, or process meets customer or user needs or expectations. [IEEE Std. 610-12-1990]

This definition offers what are probably the two most common interpretations of the word quality, (1) conformance to requirements, and (2) measure of user satisfaction. As long as the requirements completely and accurately reflect user needs and expectations, these two definitions are equivalent. When user needs aren't fully reflected in the written requirements, however, there is the potential for disagreements and misunderstandings. For example, at the end of a project the development staff may declare the result to be of high quality because it conforms to the specified requirements. Looking at the same result, the customer may disagree and declare the result to be of low quality because it doesn't meet their needs. According to this definition of quality, both are correct.

The definition for quality above is also broader than most definitions because it explicitly includes software processes as well as the products of these processes. Process quality is important because the only practical way of producing a quality product is with a quality process.

The second definition from the IEEE is from the IEEE standard for quality metrics. It defines software quality as:

Software Quality - the degree to which software possesses a desired combination of attributes. [IEEE Std 1061-1998] (Italics added for emphasis.) 

Despite its brevity, this definition incorporates both the user-based and product-based views of quality. Using the desired in the definition suggests a user-based view of quality where quality is relative to the desires and priorities of the users and project stakeholders. Using the word attributes in the definition implies that software quality is best measured by the degree to which certain product attributes are present. The quality attributes of software are the familiar "ilities" of software including usability, reliability, maintainability, etc. Software quality attributes are discussed in more detail in the next two sections.

Building on the aforementioned definitions from the IEEE, software quality is defined here as:

Software Quality - degree to which the software, (1) conforms to specified requirements, (2) meets the needs and expectations of customers, users and stakeholders in general (3) is designed and developed according to sound engineering practices and standards.

The three parts to this definition aren't three alternative definitions, but rather the three aspects of software quality. Each contributes to the overall quality of the product. Software quality is the degree to which all three of these aspects are met.

Figure 5. Software Quality

Figure 5 illustrates each part of this definition of software quality in the context of the products and process of software development. The written requirements should accurately and completely represent the needs and expectations of the user. The written requirements should include non-functional or quality requirements as well as functional requirements. Non-functional quality requirements may be expressed as a desired combination of attributes (i.e. usability, maintainability, etc.) as suggested in the IEEE definition for software quality.

Because the definition above takes into account the needs and expectations of all stakeholders, it doesn't provide an absolute definition of software quality. What is high quality to one stakeholder may be low quality to another. For example, consider the conceptual diagram in figure 6. From the perspective of marketing, product₁ has high quality. From the perspective of development, the same product is perceived to have low quality because the product's attributes don't reflect the priorities of developers. (This assumes developers are stakeholders with legitimate authority to influence the product.) Product₂ is just the opposite. It is perceived as low quality by marketing and high quality by development.

Disagreements over what is or isn't quality can be avoided by having clear and specific written requirements. The more the written requirements reflect the true needs and expectations of stakeholders, the less ambiguity there is about what is or isn't high quality.

Figure 6. Product quality is relative to stakeholder priorities

Long-term software quality (i.e. the ability to modify and extend the system over time) is influenced by the development process used to produce the software. There are no guarantees that following certain methodologies will lead to a quality product, but there are generally recognized standards and procedures for professionally developed software. Following these standards and procedures helps to ensure the long-term quality of the resulting software.

Quality software is software that conforms to implicit requirements and implicit expectations of professionally developed software. Just as it's impossible to fully specify product requirements, it's impossible to fully specify engineering standards of performance. For example, it's unnecessary to state that all words in the system dialog should be spelled correctly. Certain professional standards of quality are just assumed. To avoid misunderstandings, it is important that customers and developers have a shared understanding of what are reasonable expectations for professionally developed software.

Hierarchical Nature of Software Quality

From the user's perspective, several factors collectively influence the perceived quality of a software system. Among them, the frequency and severity of defects certainly ranks high. No one is going to be pleased with a system that has so many defects that it impedes the ability to get work done. Beyond defects, other factors that affect users' perceptions of quality include: usability, performance, robustness, etc.

When planning for software quality it's important to keep in mind the hierarchical nature of these factors [Humphrey 2005, p 136]. A relevant analogy is Abraham Maslow's theory of human motivation. Maslow's popular theory of motivation posits that humans are driven by a hierarchy of needs. Certain lower level needs including safety and security have to be met before higher level needs such as those for loving and belonging can be addressed. Once a need is satisfied, it no longer motivates and the next higher level need takes its place.

Figure 7. The primary factors that influence users'
perception of software quality change as quality improves.

There is an analogous hierarchy of factors that drive users' perceptions of software quality. Ease of use doesn't mean much if the frequency of defects prevents the software from being used at all. Lower level quality factors have to be addressed before higher level factors become relevant. The general order in which quality factors should be addressed is [Universal Principles of Design, p 107]:

• Correctness. Quality starts with understanding the real problem and delivering a correct solution. If users ask for a mortgage calculator and they are given a bond trading system, the system wont be used and any other quality factors don't matter.

  System requirements are a mix of functional and non-functional requirements. The quality of a software system is measured by the degree to which functional as well as non-functional requirements are met. Correctness is the degree to which stated and implied functional requirements are met. The word implied makes it clear that quality isn't strictly conformance to written requirements. If a desired feature isn't implemented because it's not included in the requirements, it's a deficiency of the requirements process that also diminishes the overall quality of the product.

• Reliability and Availability. Beyond correctness, users look for reliability and availability. A mortgage calculator isn't effective unless it is available when needed and able to operate reliably for the duration it is needed. The frequency and severity of defects determines the reliability and availability of a software system. For most software systems, zero defects is an impossible goal. If there are any defects, they should interfere with the normal operation of the system. This usually means no severe defects and a limited number of minor ones.

• Usability. Usability refers to the ease of use or user friendliness of an application. Usability is the measure of effort required to learn and use the application. A mortgage calculator is easy to use if users understand the meaning of each field and can easily calculate the mortgage for different scenarios.

• Proficiency. Proficiency addresses the needs of power users. It is a measure of how well the software enables work to be done quickly and efficiently. A mortgage calculator that responds quickly (performance) and has short-cuts for frequently used operations is proficient.

• Enjoyment. Products at this level achieve an emotional appeal that goes beyond mere functionality. The best examples of companies with products at this level are: Apple, BMW and Hermes. Imagining an enjoyable mortgage calculator requires a stretch of the imagination, but an enjoyable mortgage calculator is one that you enjoy using because it makes you feel good. The feeling might come from a coherent design style, superior craftsmanship and/or attention to detail.

In addition to being hierarchical, another subtle distinction among software quality factors is that some lead to satisfaction while others lead to dissatisfaction. Furthermore, these two sets of factors are mostly disjoint. Lower level quality factors such as correctness, availability and reliability have to be present in order to avoid negative feelings (dissatisfaction), where as high level factors such as usability, proficiency and enjoyment are needed to create a positive impression (satisfaction). Users expect correctness and reliability, they are delighted when they get proficiency and enjoyment.

 

Figure 8. Quality factors causing satisfaction are not the same as those causing dissatisfaction

This dynamic is much like Herzberg's theory on employee motivation. Herzberg discovered that the factors of work that cause dissatisfaction are different from those that cause satisfaction. For example, if there isn't enough light in a room, people will be dissatisfied. However, once there is adequate lighting, you can't create more satisfaction by adding more light. In the case of product quality, controlling the severity and number of defects is necessary to avoid a negative perception of the product, but controlling defects alone isn't enough to create a positive impression. The system must also possess the quality characteristics that are important to end users.

Quality is more than just the absence of defects. You might have heard that "you can't test quality into a product." Simply removing defects isn't enough to ensure quality. Removing defects only reveals the inherent quality of the product. If all the defects are removed but what is left doesn't have the quality characteristics that are important to the customer, it won't be considered a quality product. As an analogy, having a lush garden isn't simply a matter of removing all the weeds. Weeding only reveals the plants underneath. If there isn't a healthy selection of plants amongst the weeds, just removing the weeds isn't going to yield an attractive garden.  

Software Quality Models

Models are common tools in software development. There are software life-cycle models (spiral, waterfall, etc.), process improvement models (CMMI, ISO 9000, etc.), and cost estimation models (COCOMO, SLIM, etc.). Models are used to manage complexity. They provide an abstraction of a system or phenomenon from a particular perspective for a particular purpose. This section introduces another type of software development model--the software quality model. Software quality models provide a framework for expressing and measuring software quality as a collection of desirable attributes (the product-based perspective on quality mentioned above). Just as software life-cycle models help manage the software development process by suggesting activities and their timing, software quality models help manage the specification and measurement of quality in software systems by specifying quality characteristics, their relationships, and associated metrics.

This section presents three well-know quality models:

1. McCall's quality model
2. Boehm's quality model
3. The ISO/IEC 9126 Product Quality Standard

These models can be used as-is or as a starting point for creating project-specific or organization-specific software quality models.

All three models have a common hierarchical structure which is illustrated in figure 8. High level quality factors represent external quality characteristics of interest to users, customers and managers. The models relate these high-level quality factors to lower-level subfactors that represent attributes of the software or production process. These subfactors are more meaningful to developers and technical staff. In order to quantify the presence of each factor in a system, metrics are defined for all subfactors. It may also be possible to define direct metrics for some higher level quality factors.

Figure 9. Framework for quality models

McCall's Quality Model

McCall's quality model is shown in figure 10. It groups software quality characteristics according to the main phases in the software life cycle. The high-level categories of quality characteristics are:

1. Product operation - quality characteristics related to using the product.
2. Product revision - quality characteristics related to maintaining the product in its current environment.
3. Product transition - quality characteristics related to porting the product to a new environment.

Figure 10. McCall's Quality Model

Like most software quality models the high level quality factors in McCall's model generally represent external emergent system properties. Emergent system properties are system characteristics not attributable to specific elements of the system but rather those that emerge from the structure and interaction of individual system elements. Low level quality factors in the model refer to more tangible internal properties of the system.

McCall's model links high-level quality characteristics of interest to end users to low-level system properties that developers understand. More generally, it provides a vocabulary of quality factors that users and managers can use when setting the quality goals of a project. For example, end users may decide that usability is an important quality requirement. When quality requirements are stated in terms of high-level quality factors, it's not always clear to technical staff what specific product properties are needed in order to accomplish the desired system qualities. McCall's quality model breaks down high-level quality factors into their constituent components or quality criteria. These low level quality criteria are more familiar to technical staff and represent product properties that can be built directly into work products.

Besides suggesting quality characteristics and their relationships, McCall's model--and the others that will be examined--also suggest metrics for measuring the degree to which quality characteristics are present. McCall's model breaks down high-level quality factors that are difficult to measure into low-level detailed quality criteria which are more tangible and measurable. For each quality criteria, McCall suggests one or more metrics for measuring the degree to which the system possesses or exhibits the associated quality criteria. For example, storage efficiency may have the associated metrics: (1) total disk space required, and (2) average internal memory required while the program is running. Every quality criteria has at least one metric, but metrics may also be associated with some high level quality factors. For example, mean time to failure (MTTF) might be used to measure reliability.

One of the key features of McCall's model is that it provides a quantitative measure of quality. Metric values are expressed in quantitative terms and the relationship between quality criteria and quality factors is formally expressed with a linear equation of the form:

QF_a = c₁*m₁ + c₂*m₂ + ... + c_n*m_n

Where:

QF_a is the degree to which quality factor a is present.

c_i is a constant which denotes the relative importance of the measurement m_i to quality factor a. This constant may be determined by using regression analysis on historical data from similar projects.

m_i is a metric value associated with quality factor QF_a.

As an example, the quality criteria "self-descriptiveness" which is associated with quality factor maintainability might have the the metric "number of modules with standard headers". If 9 out of 10 modules have a standard header, and historical data suggests that this metric has a 30% influence on the quality factor maintainability, the formula becomes:

QF_{maintainability} = ... + 30% * 9/10

Note, that the the number of modules with standard headers is divided by the total number of modules. Metric values should be normalized to remove any bias related to the size of the product.

The formal relationship between metrics and quality factors is captured in the equation's coefficients (c_i in the equation above). McCall showed that for a limited domain significant correlation could be established between predicted quality and actual quality. However, its not clear that a quality model calibrated for one domain is generalizable to other domains. Like other software models calibrated from empirical data, the accuracy of McCall's model depends heavily on tuning and using the model on projects that are similar in type.

One of the ways McCall's software quality model is distinguished from the others that are presented here is that it defines software broadly to include not only code but also requirements and design. A few of the quality characteristics in McCall's model are limited to code (i.e. efficiency and instrumentation), but most are general enough to apply to code and other pre-code artifacts such as requirements and design. For example, traceability and completeness are two desirable characteristics of requirements and design documents.

Boehm's Quality Model

Boehm's quality model is shown in figure 11. It has the same basic categories of quality characteristics found in McCall's model. In Boehm's model, overall system quality (called general utility in the model) is the aggregation of portability, usability (called "as is utility" in the model), and maintainability.

Figure 11. Boehm's Quality Model

Like McCall's software quality model, Boehm's quality model is hierarchical with metrics defined for lower-level quality attributes. Boehm's model is similar to McCall's but does have a few notable differences. One obvious difference is its factoring of quality characteristics. As an example, in McCall's model maintainability and testability are at the same level which means in McCall's model a product can be maintainable without being testable and vise versa. In Boehm's model, testability is a subfactor of maintainability, which means an increase in testability implies an increase in maintainability. By itself this isn't necessary an important distinction but it does point out the difficulty of having a definitive quality model.

Another way Boehm's model is different from McCall's model is that Boehm's model applies only to code. Boehm's quality model includes 151 metrics which provide measurements of 12 primitive quality characteristics. Each metric is expressed in the form of a question. Stated in this way the metrics form a checklist of good programming practice. Table 1 shows a partial checklist for the quality characteristic self-descriptiveness. Self-descriptiveness is the extent to which a program or module contains enough information for the reader to understand the function of the module and its dependencies. Self-descriptiveness is a measure of a product's testability and understandability.

Table 1. Partial checklist for the quality self-descriptiveness in Boehm's quality model

By expressing quality metrics in the form of checklists, the metrics used for quality measurement can also be used for code reviews and general process documentation and improvement. The metrics developed in Boehm's original work are for Fortran programs, making them mostly outdated by today's standards. However, the quality characteristics and their factorings as shown in figure 11 above are just as applicable today as when they were proposed in 1978, demonstrating that the principles of quality are timeless.

The ISO/IEC 9126 Product Quality Standard

ISO/IEC 9126 is an international standard for defining and measuring software product quality. It comes from the International Organization for Standards (ISO) which is also the publisher of the popular ISO 9000 series of process quality standards (ISO 9000, ISO 9001, ISO 9000-3) ISO/IEC 9126 is a software product quality standard designed to complement the well-known process quality standards offered by the ISO.

ISO 9126 isn't remarkably different from the other software quality models discussed here, but it is distinguished in some important ways. First, it is an internationally recognized standard and a safe choice for any organization looking for a standard way to define and measure product quality. In the 80's there was a popular saying that nobody got fired for buying an IBM computer. Today, it can be said that no one gets fired for choosing to follow ISO standards. A universal standard provides a common language and framework for specifying and measuring software product quality. Having a common definition of software quality makes it easier to compare the quality of one product with another. Just as some organizations rely on internationally recognized ISO process quality standards to ensure a certain level of process quality, ISO 9126 can be used to ensure a certain level of product quality. Second, the ISO has shown a willingness to revise the standard and keep it up-to-date. Quality characteristics are generally timeless, but the metrics associated with quality characteristics depend on current technology and product environments. Since being introduced in 1999, the ISO 9126 standard has gone through 4 significant revisions or updates.

The original version of ISO 9126 introduced in 1991 was rather sparse. The original title of the standard, "Information technology -- Software product evaluation -- Quality characteristics and guidelines for their use", included almost as much information as the actual standard. As the title suggests the original standard offered quality characteristics and a process for applying them. The standard was essentially 6 high level quality characteristics and their definition. The standard claimed that the 6 quality characteristics were sufficient to represent any aspect of software quality. The model was limited because it didn't offer a standard factoring or taxonomy of quality characteristics or associated metrics. Sub-characteristics and metrics were suggested in an annex to the standard but they weren't part of the official standard. Overall the standard identified the elements of a software quality model for specifying and measuring software product quality, but fell short of offering a complete and comprehensive model.

Figure 12 shows the evolution of the ISO 9126 standard since its introduction in 1991. The original standard is now represented in two separate sets of standards. The ISO 9126-x series represents the expanded quality model, and the ISO 14598-x series details a process model for evaluating quality.

The new standard has evolved significantly from its roots. The two main changes are (1) suggested quality sub-characteristics that were informative in the 1991 standard are now normative, and (2) an expended set of suggested metrics have been added for measuring quality characteristics from multiple perspectives.

Figure 12. Evolution of ISO/IEC 9126

ISO 9126-1 contains the original quality model and its extensions. The quality sub-characteristics that were specified in an annex in the original standard have been modified slightly and are now part of the official standard. Figure 13 shows the full taxonomy of quality characteristics in the ISO 9126-1 standard. It also illustrates the connection to the other two related standards for external and internal metrics which are defined in ISO 9126-2 and ISO 9126-3, respectively.

Figure 13. ISO 9126 Quality Model

The quality model in ISO 9126-1 is the foundation for the other three parts of the ISO 9126 standard which define metrics and techniques for measuring quality from three different perspectives: internal quality, external quality, and quality in use.  Figure 14 shows the relationship between these three perspectives.

Figure 14. Quality Perspectives

ISO 9126-2 defines external metrics for measuring sub-characteristics in the base quality model. These are measures of quality from the perspective of the external behavior of the product at its interface. The difference between external quality metrics and quality in use metrics is that external quality metrics measure the external behavior of the product while it is running in a simulated environment. Quality in use metrics measure effects on actual end users while the product is running in its production environment. The difference is analogous to the difference between system testing a product with fabricated data and acceptance testing with real user data in the presence of actual users.

ISO 9126-3 defines internal metrics for measuring sub-characteristics in the base quality model. Some quality characteristics can only be measured with internal product metrics. For example, the maintainability of a solution isn't visible from a product's external interface. Internal quality metrics are typically measurements of static structural properties of the implementation or intermediate forms of the implementation.

ISO 9126-4 defines quality in use metrics that measure the results of using the system in a specific environment. Quality in use is a measure of quality from the user's perspective while using the product in its environment. Quality in use measures the ease and extent to which users can accomplish their goals. As figure 15 shows, quality in use metrics are based on the smaller set of user-oriented quality characteristics: effectiveness, productivity, safety, and satisfaction.

Figure 15. ISO 9126-4 Quality Model

The same basic quality characteristic from the ISO 9126-1 model can be measured in three separate ways depending on the perspective taken. For example, consider the quality characteristic efficiency. Internal efficiency can be measured by analyzing the asymptotic complexity of the algorithms used in the implementation. In terms of efficiency, an O(n) algorithm is more efficient or has higher internal quality than an O(n²) algorithm. External efficiency can be measured by simulation. If a product performs well in a simulated environment it is of high quality. From the quality in use perspective, program efficiency indirectly influences productivity and satisfaction. The product has sufficient quality if performance doesn't negatively impact the productivity or satisfaction of end users.

The ISO 9126 standard defines sub-characteristics and associated metrics but allows for application-specific quality characteristics and metrics. It would be difficult to mandate a specific closed set of quality characteristics or metrics because they are heavily dependent on implementation technology and the application domain. The ISO 9126 standard also doesn't define an absolute measure of quality because, again, the relative importance of each characteristic is application-dependent. For example, reliability is more important for mission critical applications; efficiency for time-critical applications; and usability for interactive applications.

Recent updates make the ISO 9126 standard one of the most current. At the very least it offers another taxonomy of quality characteristics and metrics to consider. The standard will be of special interest to organizations that are already using ISO process standards

Cost of Quality

Everyone is for higher quality, but the reality is that expenditures on quality improvement initiatives are economically justified only when the expected savings from higher quality is greater than the initial upfront investment it takes to achieve higher quality. The cost of quality is both a metric and economic model for understanding the cost-benefit trade-offs of pursuing higher levels of quality. It can be used at the start of a project to justify an investment in quality improvement or at the conclusion of a project to measure the outcomes of such expenditures.

Quality is one of the 4 variables of a project that has to be balanced against the other three which are cost, schedule and features. The following conceptual formula expresses the general relationship between these four variables:

Cost * Schedule = Features * Quality

The formula suggests that improving quality requires some combination of higher cost, longer schedule, or fewer features. Holding schedule and features constant, quality correlates with cost:

Quality = a_constant * Cost

In practice the true relationship between quality and cost is not so simple. In some cases it's possible to increase quality without increasing costs. To understand why, consider that cost represents not only the money spent on quality improvement (mainly defect prevention and early reviews), but also the money not spent on late stage testing and rework when defects are avoided through additional quality improvement efforts. In other words:

Cost-of-Quality = (cost of defect prevention and early reviews)  +  (cost of late stage testing and rework)

Increasing the first category of costs (prevention) tends to reduce the second category (rework). When the amount saved by avoiding defects rises faster than the amount invested in order to avoid these defects, quality is free*.

________________________
*Philip Crosby was so convinced of this dynamic that he titled one of his books Quality is Free, lest anyone would miss the main point which was that the major costs of quality are the costs associated with poor quality. Crosby maintained that, as the voluntary costs of defect prevention are increased, the involuntary costs of poor quality (mainly rework) decrease by an even greater amount thus making quality free [Crosby].
________________________

How much does quality cost? Is quality free? Is there a point of diminishing returns, beyond which an investment in quality is no longer cost-justified? The cost of quality model helps to answer these questions by modeling the relationship between quality and costs*.

_____________________
*When discussing the cost of quality, quality is defined as freedom from defects. It explicitly doesn't mean luxury or features. More luxury and features almost always means higher cost.
_____________________

Formally the cost of quality is the sum of conformance costs and non-conformance costs. Conformance costs further breakdown into prevention and appraisal costs. Conformance costs are the costs associated with attaining quality. These are the voluntary investment costs for things like defect prevention and early reviews. Non-conformance costs are the costs of dealing with defects. This includes things like debugging and rework resulting from exposed defects. These are involuntary costs and a source of potential savings if defects can be reduced. Non-conformance costs may be internal failure costs (cost of errors found before the product is released), and external failure costs (cost of errors found once the product has been released). Figure 16 shows the taxonomic breakdown of these costs and figure 17 shows when these costs generally occur during the software life cycle.

Figure 16. Taxonomy of Software Quality Costs

Figure 17. Relative Timing of Cost of Quality Expenses

The following is a closer look at these 4 categories of costs. They are also summarized in table 2 below.

Prevention costs -  These are the costs associated with preventing defects from ever occurring. If you can prevent errors from occurring you can avoid the expense of finding and fixing them (not to mention the wasted effort spent making the error in the first place). Prevention costs include the costs of training, early reviews, quality planning, root cause analysis, baseline configuration management, and process improvement initiatives.

Appraisal costs - These are the costs associated with analyzing and testing the product to make sure it conforms to requirements. The most common appraisal costs are the costs for testing, inspections and quality control.

Internal failure costs - These are the costs associated with fixing defects found prior to product release. This includes the cost of rework, retesting, and the administrative overhead of pre-release defect management. It may also include the cost of updating documentation, additional reviews and inspections.

External failure costs - These are the costs associated with fixing defects found after the product has been released. The process of diagnosing and fixing an external failure is much more complicated and involved than fixing an internal failure so costs are higher. There are the regular expenses that go along with internal failures, plus additional expenses that come with field defects. Less tangible external failure costs are the costs associated with possible damage to reputation and the loss of future sales. External failure costs may also include the unpredictable costs of litigation.

Table 2. Software Quality Cost Categories

The cost of quality is the sum of all conformance and non-conformance costs. The cost of quality is a real expense that shows up indirectly in the budget of every project or directly when the cost of quality is being tracked. It is also an expense that can be controlled by the project manager. In order to control the cost of quality the project manager must understand the complex relationship between conformance costs and non-conformance costs.

There are three notable aspects of the relationship between conformance and non-conformance costs. First, the two categories of costs tend to be inversely related. Spending more on conformance should reduce non-conformance costs. Spending less on conformance will likely increase non-conformance costs. Second, conformance costs are voluntary while non-conformance costs are involuntary. The project manager decides how much to spend on conformance, and this in turn determines how much will be spent involuntarily on non-conformance. Third, conformance costs represent an investment that must be justified by even larger anticipated savings in non-conformance costs. It is an investment that usually has a point of diminishing returns beyond which spending more on conformance won't reduce failure costs enough to justify the initial expense and risk.

The project manager is responsible for determining the optimal amount to spend on conformance activities such that the overall cost of quality (conformance and non-conformance costs) is minimal. Projects vary in size and budget so the amount won't be an absolute number but rather some portion of the total budget. The optimal cost of quality is usually expressed as the cost per good unit of product produced or as a percentage of total development costs.

The problem of finding the optimal cost of quality is made more difficult by the fact that the optimal balance between conformance and non-conformance costs depends on the characteristics of the project and the maturity of the development process being used. There isn't one precise answer, but there are models for understanding in general the forces that affect the optimal balance in different environments. Figure 18 shows two such models. Figure 18.a shows the traditional cost of quality model and figure 18.b shows the contemporary one. The models depict the general relationship between costs and quality for different types of projects.

Figure 18.a. Traditional Cost of Quality Model

Figure 18.b. Contemporary Cost of Quality Model

Both models show the costs of quality in terms of conformance and non-conformance as quality increases (the x axis). In both models the total cost of quality is the sum of conformance and non-conformance costs. Also in both models, non-conformance costs drop to 0 as quality reaches 100 % or zero defects. The models differ in their assumptions about the amount of investment needed in conformance to achieve perfect quality. In the traditional model, a sharp increase in conformance cost is needed to achieve perfect quality. The contemporary model shows a more modest increase in conformance costs as quality rises, thus making it possible to have both perfect quality and minimal cost.

"Quality is free but perfection you have to pay for."

Which model is correct? They are both correct, for the project environments they depict. The conventional cost of quality model in figure 18.a is appropriate for project environments where the cost of failure is low and the effort required to achieve zero defects increases rapidly as product quality approaches 100%. For example, this model accurately depicts the cost of removing all of the errors from a large daily newspaper. The cost of an error is minimal and the cost of removing every single error is prohibitively expensive. It could take as much time to find the last single error as it took to find all of the other errors combined.  The economic impact of a reader finding an error isn't worth the extraordinary effort it would take to find all the errors. To paraphrase Crosby, the traditional cost of quality model seems to be saying that quality is free but perfection you have to pay for.

The contemporary cost of quality model in figure 18.b is more appropriate in environments where:

• The cost of failure is high which justifies increased spending on conformance.
• There are more effective development and appraisal methods (usually automated) which reduces the cost of conformance. More effective development methods reduces the total number of errors and more effective appraisal methods reduces the cost of finding and removing the errors that are introduced. In some cases complete testing may be both feasible and economical.

As an example, consider the development process for microprocessors. The cost of putting a defective microprocessors into production is very high thus justifying increased spending on conformance. Modern production processes limit the number and types of errors introduced during development, and automated testing tools in most cases make it economical to perform complete testing during appraisal. This reduces the cost of conformance.

So, which model is most appropriate when the environment is software development? Again, it depends on the specific project, but for the average software development project the traditional cost of quality model is probably more appropriate. Not all of the trends that are making the contemporary cost of quality model more applicable in manufacturing apply when the product is software.

The reason for this rather pessimistic assessment is that first, software development is a human-intensive activity. Humans are fallible, so it follows that software will always have defects as long as humans are in the loop. Second, all but the most trivial software systems are too complex to completely test. There will always be some inputs that haven't been tested. The difficulty of preventing all defects and the inability to test for all inputs, suggests that for most software products the goal of zero defects is either impossible or prohibitively expensive. This implies that for most software projects there is a point of diminishing returns beyond which improvements in quality are not cost-justified.

The above conclusion is mainly a theoretical result--and not a very original one at that. Most software practitioners are well aware that for most software systems complete testing is impractical. What would be more useful is some way of predicting the point of diminishing returns for a software system. Looking at figure 18.a it appears that the rising cost of conformance (principally appraisal costs) is what causes the total cost of quality curve to reverse course and head upward. However, a rapid rise in conformance costs doesn't by itself determine the low point in the cost of quality curve. The cost of quality is conformance costs plus non-conformance costs. If the cost of a failure is high it can justify high, even exponentially increasing, conformance costs. For example, figure 19 shows the cost of achieving ultra high reliability with the primary flight control software on the Space Shuttle. As you might expect, developing near perfect code for the Space Shuttle is an order of magnitude more expensive than developing what might be considered very reliable software (1 defect per KSLOC). In spite of the high conformance costs necessary to achieve near-perfect code, the total cost of quality is still minimal because the cost of failure is even greater. The high cost of a defect in the Space Shuttle software in terms of human life, vehicle expense, and national pride can justify large expenditures on conformance. In general, a rapid rise in appraisal costs alone isn't enough to trigger a point of diminishing returns. Applications that have very high reliability requirements can justify very high, even exponentially increasing, conformance costs.

Figure 19. Cost of ultra-high reliability in space shuttle software

[Image is from: Herb Krasner, Using the Cost of Quality Approach for Software, Crosstalk, Nov. 1998. Original data is from: Krasner, H., "A Case History of the NASA Space Shuttle Onboard Systems Project," SEMATECH Technology Transfer Report 94092551A-TR, Oct. 31, 1994]

As the cost of failure rises, increased spending on conformance is justified and the point of diminishing returns in figure 18.a shifts to the right. In other words, a high cost of failure can justify large investments in quality improvement. When calculating the cost of failure the non-obvious costs are often overlooked. These include:

1. The intangible costs of failure, including loss of good will and future sales.
2. The user's cost of failure. Factoring in the user's cost of failure may justify even higher investments in conformance.

The point of diminishing returns is also shifted right by improvements in the standard development process. With an improved development process fewer defects will be injected and conformance activities will be more efficient and effective. This makes higher quality more economical.

In summary, the cost of quality is both a model and metric for planning and monitoring process improvement initiatives. Some practical applications of cost of quality principles are [Plunkett and Dale 1990, Managing Quality]:

1. Cost of quality estimates can be used to justify spending on quality improvement initiatives. Tracking conformance and non-conformance costs over a period of time provides a strong argument for investing in defect prevention and early reviews.
2. Candidate process improvement initiatives can be compared and prioritized on the basis of their return on investment (ROI) as measured by cost of quality estimates. The initiatives with the highest ROI can then be scheduled for implementation.
3. In a showdown between budget, schedule and product quality, quality usually loses because it's harder to quantify. Cost of quality analysis provides quantitative data on the value of quality which can be used to make informed decisions regarding tradeoffs between budget, schedule and quality.
4. Cost of quality estimates can be used to budget spending on quality improvement programs. The budget for quality improvement programs can be based on expected returns rather than a fixed percentage of development costs.
5. The cost of quality can be used to assess the performance of process improvement initiatives.
6. The cost of quality provides a single measure for comparing the success of diverse projects.

Software Quality Control

The meaning of the term quality control in software development is still evolving. The IEEE offers these candidate definitions:

Quality Control - ... (1) A set of activities designed to evaluate the quality of developed or manufactured products. ... (2) The process of verifying one's own work or that of a co-worker. (3) Synonym for quality assurance. [IEEE Std. 610-12-1990]

In everyday discourse the term is typically used as a general reference to any quality-related activity, as in "Our support costs are killing us. We need better quality control."

Outside of the software industry the term has a more precise meaning. In the manufacturing and service industries, quality control refers to a managerial process for maintaining the quality of a result [PMBOK 2000] [Juran] [Gryna]. Once an operation achieves a certain level of performance, quality controls are put into place to maintain that level of performance. For example, consider a customer service call center at a consumer products company. After careful product design and training, the average length of a customer service call might be 3 minutes. If this is acceptable, a quality control process is put into place to maintain this performance. If a 3 minute customer wait is unacceptable, some combination of improved product design and/or employee training is needed to reduce the the average length of a service call. Once the wait time reaches an acceptable level, quality controls are put into place to maintain the capability of the operation.

Quality control is a particular application of the standard process control feedback loop. The basic steps of process control are:

1.  Establish plans and standards of performance
2.  Perform work according to plan
3.  Measure performance relative to plan
4.  If actual performance deviates significantly from plan, take corrective action to bring performance back into line with plan

Figure 20. Standard Process Control Feedback Loop

You might recall another application of process control is the control function in the project management process. At the project level, the project manager exercises control over the project by reviewing status reports and taking corrective action as needed in order to keep the project moving forward as planned. Quality control is process control performed on a smaller scale. Quality control is performed by workers as part of their day-to-day routine. They take measurements of results and immediately use these data to make adjustments as needed in order to maintain satisfactory results.

The definition for quality control that will be used here is one that is consistent with the use of the term in the manufacturing and service industries. Here quality control is defined as:

Quality Control - a technique for maintaining planned levels of quality in products and processes. Specifically, the act of measuring performance of a control subject, comparing performance to plans, and taking corrective action when actual performance deviates significantly from plans.

This definition is consistent with the emerging uses of quality control in the practice of software engineering [CMMI] [SPC articles].

Quality control involves quality planning, and when current performance is unsatisfactory, quality improvement. The relationship between these three essential ingredients to quality management are best illustrated by the Juran Trilogy diagram. (See figure 21.)

Figure 21. The Juran Trilogy diagram. [Juran 1999]

Quality management starts with quality planning. During the planning phase quality goals are set and the means for accomplishing these goals are planned. Quality control activities start during the production or development phase of operation. The diagram shows the changing values of some control subject over time. For example, the control subject might be the number of errors found in different modules during system test. A defined process has a certain inherent capability that characterizes the quality that can be expected from the process. One way to depict this capability is by the amount of variation inherent in the process. The more variation there is, the less capable the process is*. The original zone of control in figure 21 shows a wide variation in the results of the process. This suggests a less capable process. For example, you would expect a poorly defined development process to produce modules of unpredictable quality. The purpose of quality control is to maintain the status quo or the inherent capability of the process. The out-of-range value in the figure indicates a potential problem in the process that has a specific cause which should be investigated. For example, analysis might reveal that fewer errors were found in a particular module because the test cases for that module were mistakenly excluded from those used during the system test. With a quality control process in place, an error such as this can be found and corrected quickly before it has a chance to impact the quality of the product.

___________________________
*In some cases high variability might seem benign as long as the average performance is acceptable. However, the more variation in a process the less predictable it is, and predictability in a process is highly valued.
___________________________

The center part of figure 21 illustrates a change in the process that occurs as a result of some quality improvement initiative. Quality improvement is a change in the process that improves the inherent capability of the process. The figure shows a reduction in variation (zone of quality control is less) as well as performance (average absolute measure of the control subject is less). Note, that after the quality improvement initiative there is a new zone of quality control.

Quality control and quality improvement are both techniques for minimizing error and improving quality. They are distinguished by the types of problems they eliminate. Quality control is a process for controlling variation by detecting and correcting sporadic quality problems that have special or assignable causes. Quality improvement is a process for correcting chronic causes of quality problems. Chronic quality problems have no single identifiable assignable cause. They are the cumulative effect of many small essentially random causes of poor performance. The only way to fix chronic quality problems is by changing the process. (Statistical techniques for distinguishing between sporadic and chronic causes of quality problems are described in more detail in chapter 20.)

The Quality Control Process

Quality control occurs during the production or development stage of a software project. This section presents a 6-step process for performing quality control. A running example is used to illustrate each step.  Figure 22 shows a flowchart for the process. 

Figure 22. Quality Control Process [Juran 4.5 1999]

Step 0. The quality control process executes within the larger quality management function. The planning step of the quality control process can begin once project quality goals have been set and activities for accomplishing these goals have been identified and planned. These activities may come from the software development process being followed. For example, reviews, testing and quality assurance are all quality management activities that might be planned for a project.

Example - For the running example assume that the following quality goals have been identified: The defect density during system test should be 3 or fewer defects per thousand lines of code (KLOC). The finished product should have 10% portability. This quality goal is stated using the metric for portability introduced above. Ten percent portability implies that the cost of moving the system to a new platform should be 10% or less of the cost of developing the system in the new environment without the benefit of reusing work from a preexisting system. Furthermore, assume that a standard software development process is being followed which includes planned activities for inspections, quality assurance, and testing (unit, integration, system).

Step 1. The first step of the quality control process is to identify control subjects--product or process features that need to be controlled in order to achieve the quality goals of the project.

When selecting control subjects, preference should be given to leading indicators. Leading indicators expose quality problems early when corrective action is more effective and economical. One of the tenets of modern quality management is that it's better to build quality into a product than to test errors out. Measuring and responding to leading quality indicators is one way to build quality into a product.

Example - Given the project quality goals specified above, a reasonable set of control subjects are: Quality of code at code reviews Time spent reviewing code Portability of code The first control subject, the quality of code at code reviews, should be a good leading indicator of quality in the final product. If an unusually large number of errors are found during reviews, that's an early warning that the overall project quality goal of 3 or fewer defects in the final product may be in jeopardy. Finding this out early gives the project manager time to react and take corrective action such as rewriting error prone modules or scheduling additional training for programmers. When selecting control subjects, some thought should be given to the interpretation and analysis of their values. For example, the first control subject, the quality of code at code reviews, seems like a good predictor of final product quality. However, finding few or no defects during a review doesn't necessarily imply high code quality. It could mean the code review was hastily performed. Without additional information it's impossible to tell. The purpose of the second control subject, time spent reviewing code, is to provide more information with which to interpret the values of the first control subject, the quality of code at code reviews.

Step 2. Once control subjects have been selected, the next step is to establish metrics or the means of measuring these control subjects. Sometimes a control subject will have more than one candidate measure. For example, product size can be measured in lines of code (LOC) or function points. The metric selected should (1) correlate with the quality characteristic being measured, (2) be trusted by project stakeholders, and (3) require minimal effort to collect. (See chapter x for more information on measuring quality.)

Example - Reasonable metrics for the three quality characteristics identified in step 1 are: The quality of code at code reviews will be measured by tracking defect density. For this example, defect density will be defined as the number of major defects found during a code review per KLOC (thousand lines of code) reviewed. Time spent reviewing code during a code review will be measured by review rate. Review rate will be defined as the duration of the actual review meeting, irrespective of the number of participants, expressed in lines of code (LOC) per hour. For example, if a team reviews 400 LOC in 2 hours, the time spent reviewing will be 200 LOC/hour. The same metric for measuring the portability of the overall system will be used to measure the portability of the code as it is written. Portability of code will be estimated by expert opinion. As code is reviewed the reviewers will be asked to estimate the portability of each module reviewed. The average of the reviewers' estimates will be used to estimate the portability of the reviewed code. Notice that the means of measuring a control subject has two parts: (1) a unit of measure, and (2) a sensor or way of making the measurement. For example, the unit of measure for code quality is defects per thousand lines of code, and the sensor is a code review.

Step 3. The third step of the quality control process is to identify specific performance goals or target values for the control subjects. The target values for control subjects are set with project quality goals in mind. Target values should be sufficient such that meeting them will provide reasonable assurance that the overall quality goals for the project will be met.

Project quality goals are set at the beginning of the project based on business needs and a consideration for the capability of the development process. The project quality goals will dictate the control subject quality goals. When project quality goals are translated into control subject quality goals, the feasibility of the project quality goals starts to become evident. What seemed like realistic project quality goals at the beginning of the project may reveal  themselves to be overly optimistic. The capability of the development process and experience on past projects suggest what are reasonable quality goals for control subjects. If project quality goals require control subject quality goals that are unreasonable for the existing development process, changes in the development process will be needed. Possible changes include adding more reviews, practicing defect prevention, reusing existing components, etc.

Example - Reasonable target values for the quality metrics identified in step 2 are: The target value for the number of defects found during a code review is 5-25 major defects per KLOC. The target value for review rate is 150 LOC/Hour. The target for portability of code reviewed is 10%. The assumption will be that these target values are in line with the historic performance of the development process being used. No changes to the development process will be required.

Steps 4-6. The first three steps of the quality control process are preparatory. The remaining steps form the feedback loop which is performed at regular intervals during the execution of the project. Quality control is an ongoing activity. Values for control subjects are measured over time to detect deviations from plans and to identify trends. In steps 4-6, results are measured and compared with plans. If actual results deviate significantly from plans, corrective action is taken to bring performance back into line with plans.

Example - Figure 23 shows a graphical display of defect detection rates recorded during code reviews over a period of time. The data are being displayed in the form of a run chart. (Run charts are described in more detail in chapter 20.) The standard process being used predicts that between 5 and 25 errors will be found per thousand lines of code (KLOC) reviewed. These values are represented as specification limits on the run chart in figure 23. The recorded data shows two instances when the number of defects found fall outside of the expected range. These deviations signal potential problems in the product and/or process. The high number of errors found in the 3rd review could be an early warning of poor code quality or just thorough reviewing. The low number of errors found during the 6th review could be an sign of perfunctory or hastily performed reviews or simply high code quality. Values outside the range of the specification limits should be investigated. If a problem is found, corrective action should be taken to bring performance back in line with plans. Figure 23. Run Chart for Defects Found During Code Reviews Review rate is another control value being tracked. Knowing the review rate for each review will help interpret defect detection rates. For example, if the review rate for the 6th review is low this suggests that the review was inadequate and the low defect rate isn't necessarily the result of high quality code.

Limits of Quality Control in Software Development

Having just discussed the union of software development and quality control, it's time for a reality check. Quality control is a regular practice in manufacturing, but software development is not manufacturing (see sidebar). The mass production of identical physical items is fundamentally different from the design and development of software systems. Software development is primarily an intellectual design activity where every product is unique. This makes it difficult to achieve the degree of precision and control during software development that is commonplace in manufacturing.

Software Development is not Manufacturing Manufacturing is sometimes used as a metaphor for software development. The metaphor is useful only where the similarities and differences between the two disciplines are well understood. Software development is not manufacturing, so practices common in manufacturing don't necessarily apply to software development. What follows is a brief comparative look at the manufacturing and software development product cycles. The comparison will reveal why software development is less predictable and controllable than manufacturing and why practices such as quality control, which are common in manufacturing, don't automatically apply to the products and processes of software development. The manufacturing product cycle, illustrated in figure 24.a, consists of three distinct phases: (1) a product is designed according to customer specifications, (2) an efficient production process is established for creating the product according to the design, and (3) the production process is transferred to an operations phase where the process executes continuously turning out essentially identical items. The production process is specific to the product and is worked out in great detail in advance. It is highly predictable, repeatable and controllable. Figure 24. Manufacturing Compared to Software Development The software development product cycle has rough analogs to the manufacturing product cycle. (1) a product is designed according to customer specifications, (2) the development phase, including design and implementation, is planned and executed, (3) the source code is compiled and linked into a finished product. Creating another instance of the product is as simple as duplicating a CD-ROM or copying a file. The obvious difference between software development and manufacturing is that software development is purely design and development. It doesn't have a production phase*. Or rather, the production phase is completely automatic and trivial from the developer's perspective. This is the source of the problem with the software development as manufacturing metaphor. The comparison is often made between software development and the production phase of manufacturing, when a more apt comparison would be between software development and the product development phase of manufacturing. It's important to make the distinction so as not to expect the same predictability and control during software development that's routine during the production phase of manufacturing. For example, quality control works well during the production phase of manufacturing because the production process is well-defined and repeatable. Quality control doesn't work so well during software development because design and development is not a deterministic process. ___________________________ *Some might argue that the analog to the production phase in manufacturing is the coding stage of software development. Coding might be more routine than design but it is not as predictable and repeatable as the production phase in manufacturing. It's a poor analogy principally because the design stage in manufacturing is not the same as the design stage in software development. The design for a manufactured product is fully specified. The design for a software product is a high-level abstraction that is validated and refined during the implementation and testing phases. In software development, a significant amount of design and innovation occurs after the design phase proper. In terms of the manufacturing analogy, "the source code is the design" and not the product of software development [Reeves, What is software design? 1992, reprinted in Robert Martin 2003]. The product of software development is the output from the compiler. ___________________________

Controlling the quality of manufactured products is based on two principles. First, most products are aggregates of more primitive components. The quality of a product is determined by the quality of its constituent components. Second, the quality of a component is inversely proportional to the amount of variation in the attributes of that component. Low variation means high quality and vise versa. For example, ball bearings are one of the constituent components of a motor. The quality of a motor is determined in part by the quality of the ball bearings used in its construction. High quality ball bearings are ones with measurements that vary little from their specification. (See figure 25)

Figure 25. Quality control with ball bearings

It's possible to control the quality of manufactured products because (1) the variation in components of a manufactured product can be controlled with precision, and (2) there is a direct link between controlling variation in these components and the quality of the products built from these components.

Quality control is much less certain when the product is software. The main problems are:

1. It's difficult to specify quality metrics along with target values for the constructive elements of software. For example, figure 26 shows a conceptual control chart for design documents. What should the control metric be? What should the target value and allowable tolerances be? Every software system is unique so it is hard to define universal measurements. When there are measurements the range of permissible values is often too wide to be of use. Further complicating the problem is the dynamic environment in which software is developed. People, tools, and technology are continuously changing. Control metrics that are valid today may not be suitable in the future.
2. The instruments of measurement are often imprecise and subjective. The physical attributes of manufactured products can be measured with arbitrary precision, but there is no precise and objective way of measuring characteristics of software such as maintainability and robustness.
3. Options for corrective actions or control are more limited. For example, if the quality control process does detect a module with excessive errors, what realistically can be done to prevent this from happening again in the future? Developers don't come with alignment adjustment screws.
4. Even if is is possible to control variation in the constructive elements of software, the relationship between controlling variation in these elements and the quality of the resulting software system is not well defined. For example, several modules in a system may have a high cyclomatic complexity rating but that doesn't necessarily imply that the overall system will be complex. The reverse is also true. You can have a complex system composed of relatively simple individual modules.

Figure 26. Quality control with design documents

During manufacturing it's possible to control the quality of a product by controlling the quality of its constituent components. The process of software development is less predictable and controllable. There is less control over the evolving elements of a software system and no guarantee that controlling the quality of these elements will ensure the quality of the resulting system.

Most of the discussion up until this point has been about quality control at the product level--controlling the quality of the evolving products of software development. Quality control may also be used at the project level for controlling the execution of the project [Kan]. For example, specific goals for the number of defects injected and detected at each phase can be set and tracked. Goals can be set based on past experience with similar types of projects and the expected capability of the process being used. Actual results provide feedback on the state of the project. When actual results are significantly different from plan, corrective action can be taken to bring results back in line with plan. This is quality control at the project level. Variation in the execution of the project is tracked and controlled to ensure the project is performed as expected.

Statistical Methods of Quality Control

Quality control is often performed with the aid of statistical methods. Statistical methods of quality control include statistical process control (SPC), statistical sampling, and design of experiments [Montgomery].  This section shows the connection between the process aspects of quality control described here, and the statistical techniques of quality control discussed elsewhere [tbd: reference].

The example used here to introduce the idea of statistical quality control is a non-technical one. Consider the use of quality control during the production phase of a manufacturing process that packages breakfast cereal. One of the quality goals is to be able to control the amount of cereal that goes into each box. There is a certain amount of variation in any process so the amount of cereal will vary between boxes. Any less than the advertised weight and the customer will be dissatisfied. Any more than the advertised weight and profits are reduced. The quality control process as described in this section can be used to keep the production process performing to plan. For this example assume the following quality control parameters:

• Control subject: amount of cereal placed in each box
• Target value: 16.3 (This will allow for some variation without going under the advertised weight.)
• Acceptable tolerance: 16.0 - 16.6

During operation samples are selected and measured for conformance to the stated quality control goals. Figure 27 shows the results in the form of a run chart:

Figure 27. Performance of process is well within specification limits

The results look good. The run chart in figure 27 shows that the weight of selected samples is well within the specification limits. Because the recorded values for all measurements fall within the specification limits, there is no need to intervene with corrective action. This is the most that can be expected from the process of quality control as described in this chapter.

The quality control process described in this chapter keeps performance on track but it doesn't maximize the capability of the process. Statistical methods go beyond what can be expected from the process of quality control alone. Statistical methods quantify the data in ways that reveal new information which can be used to maximize the capability of the process. For example, here are three specific ways statistical methods of quality control can enhance the results of the example above:

1. The performance of the cereal packing process is well within the range of the specification limits. In fact most values fall within a narrow range around the target value. This suggests that there is an opportunity to lower the target value of the process and still meet the original quality goals of the process. Statistical methods of quality control can be used to determine how much the target value can be lowered and still maintain a reasonable level of confidence that all boxes will always have at least 16.0 ounces of product. Remember, lowering the average volume of each box will raise profits.
2. In the example above control limits are based on specification limits. Specification limits are based on expectations for the process. They reflect external requirements. When a process is performing within specification limits (external requirements) there will be fewer false negatives then when quality control limits are set according to the capability of the process. For example, consider point A in figure 27. It is within the specification limits so it doesn't trigger corrective action. However, in the context of the other values, this errant value appears to be a symptom of a problem in the process that has an assignable cause and is not just part of the random noise of the process. The role of quality control is to distinguish between deviations with assignable causes that can be investigated and corrected as part of the quality control process, and those that are part of the random noise of the process which can only be fixed by changes to the process (i.e. process improvement). Statistical methods of quality control can be used to identify more effective control limits--those based on the capability of the process rather than external requirements. The result is more accurate identification of deviations due to assignable causes which can be investigated and corrected as part of the quality control process.
3. Statistical methods of quality control can also be used to determine the average performance of a process. For instance, in the example above the average performance of the process (fill rate per box of cereal) is a measure of the cost of quality. It is the additional expense incurred because of the inability to fill each box with exactly 16.0 ounces of product. Knowing the average performance of the process would allow us to calculate the cost of quality for this process.

Statistical process control is a collection of tools for performing statistical quality control. One of the primary tools of statistical process control is the control chart. Figure 28 shows what a control chart for the example above might look like*. A control chart has control limits rather than specification limits. The control limits are calculated based on a stable set of measurements from the process. They define the capability of the process. Values outside of the control limits are statistically likely to be the result of some specific identifiable cause that can be corrected as part of the quality control process. Variations within the control limits are likely to be due to the random noise in the process and are better addressed by fundamental changes to the process.

______________________________
*The graph shown is more of a pseudo or conceptual control chart. The data values on a real control chart are the average values for a small number of samples and not the values of individual samples that appear to be shown in the control chart in this example.
______________________________

Figure 28. Conceptual Control Chart

17.4 Software Quality Assurance

Software quality assurance means different things to different people. Some people use the term as a synonym for testing or as a synonym for the group responsible for testing, as in "the product must pass through quality assurance before we can ship it." Quality assurance is not the same as testing though. Testing is about detecting defects. Quality assurance is about preventing them and assuring others that they have been prevented. Quality assurance may use the results of testing but it is primarily an oversight function. The goal of testing is to identify defects. The goal of quality assurance is to assure stakeholders, especially those not directly involved with day-to-day operations, that the product has achieved a certain level of quality.

Quality assurance is a common practice in manufacturing and the service industry. Consider the following examples of quality assurance in everyday activities:

• When you purchase a new item of cloths you are likely to find a slip of paper tucked into the pocket assuring you that the garment has been inspected.
• While refueling your automobile you are likely to see a sticker on the side of the pump assuring you that the meter has recently been checked for accuracy.
• When you eat at a restaurant you can take comfort in knowing that the city where the restaurant is located publishes and enforces strict guidelines for food preparation.

These are all examples of quality assurance in action. In all cases there is a person or organization responsible for ensuring that goods or services are being delivered according to approved standards and procedures. In all cases the person doing the inspection has no direct control over the quality of the product or service. The inspector's job is only to verify that the product or service being delivered meets established standards.

Software quality assurance is the practice of quality assurance applied to the products and processes of software development. The IEEE Standard 12207 defines software quality assurance as,

 “... a process for providing adequate assurance that the software products and processes in the product life cycle conform to their specific requirements and adhere to their established plans.” [IEEE 12207]

Expanding on each element of this definition provides a deeper understanding of the term quality assurance:

1. "... providing adequate assurance ..." - A key result or outcome of performing SQA is that external stakeholders have increased confidence in the quality of the product.
2. "... software products and processes ..." - SQA is not only concerned with the quality of products but also the processes used to produce these products. SQA assures that products adhere to their specified requirements and quality standards, and that development processes are planned and performed according to established standards and procedures. Product quality is directly affected by process quality. It's not sufficient to focus exclusively on product quality alone. Some long-term quality results are difficult or impossible to achieve by measuring and controlling product quality alone. For example, the best way to assess the quality of a new concrete floor is to observe the process used to cure the concrete. Just inspecting the result doesn't provide an accurate indication of long-term wear or quality.
3. "... conform to their specified requirements ..."  - SQA judges the quality of products and processes by comparing them to established specifications. There are standards of performance for work products at every stage of the software life cycle. For example, code should meet coding standards and conform to the specified design.
4. "... adhere to their established plans." - A key component of quality assurance is ensuring that there is are adequate plans and that these plans are followed.

Figure 29 outlines the key activities of SQA roughly in the order they would occur during the software life cycle.

Figure 29. Responsibilities of SQA during the software lifecycle

At the beginning of a project SQA works with the project staff to identify and plan appropriate standards and procedures of software development. These standards and procedures typically come from the organization's standard process. The organization's standard process may be owned and maintained by a separate organizational entity such as the software engineering process group (SEPG). If so, the SEPG usually has responsibility for defining and maintaining the standards and procedures while SQA ensures that they are planned and followed.

During a project SQA monitors development processes for conformance to procedures and evaluates products for conformance to standards. Process monitoring and product evaluation are performed through reviews and audits.

The results of process monitoring and product evaluations are reported to management and other stakeholders. When there are deviations from plan, SQA may recommend actions for bringing performance back into line with plan. This reporting function provides another source of information on project status and progress, especially with respect to the quality goals of the project. Management uses this information to make decisions about the project such as when to transition from one phase to the next.

The SQA function plays an important role in overall project management. Recall that the four variables of a project that must be managed are: schedule, cost, features and quality. Compared to the other three variables of a project, quality is difficult to evaluate from outside of the project. The SQA function makes the quality of the evolving product visible outside of the project. (See figure 30.) It externalizes the quality of the evolving product, putting quality on equal footing with the other 3 variables of a project.

Figure 30. Quality Assurance makes project quality visible

Standards and Procedures of SQA

Standards and procedures provide structure and consistency to the process of software development. Following standards and procedures make software development more predictable, repeatable and manageable. As a point of definition:

• Standards represent the criteria for judging the quality of products
• Procedures represent the criteria for judging the performance of processes

SQA is responsible for enforcing an organization's standards and procedures for software development.

Most standards apply to either code or documentation--the two main products of software development. Coding standards include naming conventions, format conventions, constructs that are encouraged/discouraged, etc. Document standards typically specify format and desired content for written documents such as the requirements document, architecture document, verification and validation plans, test case specification, etc.

A process is a sequence of steps for transforming inputs into outputs. Procedures are the specific steps to be followed when performing a process. All processes should have defined procedures. Most software development processes include procedures for:

• Requirements management (primarily change control)
• Configuration management
• Problem reporting and corrective action
• Inspections and reviews

Organizational Independence

The organizational placement of the SQA function is key to its effectiveness. SQA is an oversight function, and to be effective in this role SQA should be organizationally separate from the staff they are monitoring. This gives SQA the freedom to provide objective unbiased assessments of process and product quality. SQA may report preliminary results of evaluations to the project manager, but the allegiance and formal reporting relationship of those performing the function should be with an organizational entity above the project manager. (See figure 31.)

Figure 31. The SQA Function Requires Organizational Independence

Rational for SQA

SQA is primarily an oversight or audit function. Many people are wary of audits and may even resent having their work checked. The monitoring may be perceived as a sign of mistrust or lack of confidence.

Resistance to SQA can be overcome by educating staff on the role and importance of SQA. When workers understand the role and value of SQA, they are usually more willing to have their work audited. It's easier to accept auditing as part of software development when you consider that auditing is an essential and accepted practice in other areas of life. For example,

• When rock climbing it's common practice to check your partner's buckle routine and tie-in knot before starting a climb. When performance is life- or mission-critical we accept, even welcome, having our work checked. Humans are not perfect. Over time, human error is inevitable and systems that can't tolerate error should be designed with redundancies to verify the performance of human activity.
• Most publicly held companies in the US must register with the Securities and Exchange Commission (SEC) and file regular financial statements. The SEC requires many of these statements to be audited by an independent public accountant. The company's management prepares the reports but an independent auditor certifies their accuracy. The purpose of the audit is to provide additional assurance to outside stakeholders (primarily shareholders) that the information provided by the company is complete and accurate. It is the structural relationship between management and shareholders that creates the need for an independent audit. The company is owned by its shareholders but ran by management. Management controls the company and is the best source for information on internal operations and finances. Shareholders rely on management for the information they receive and subsequently use to make decisions about where and how much to invest. This division of responsibility makes for strong capital markets but relies on the independent audit to make the relationship work to the benefit of all involved.

In software development, product quality is difficult to accurately measure from outside the project. External project stakeholders need reliable information about the quality of the evolving product in order to make decisions that affect the project. The development staff are the best source of information on product quality, but the motivations and priorities of developers are sometimes in conflict with those of external stakeholders. SQA helps balance the structural relationship between developers and stakeholders for the long-term benefit of all.

Oversight plays an important role when performance is critical or when independent assessment is needed to balance certain structural relationships. For life- or mission-critical software systems SQA can provide additional assurance that the system produced will meet its intended requirements. The quality of the final product is more certain when it is verified by the activities of SQA.

SQA Planning

SQA activities must be planned. Planning for SQA may be included in the main project plan or specified in a separate Software Quality Assurance Plan (SQAP). This section describes the key elements of a SQAP. The elements presented here are derived from and are a subset of those found in the IEEE standard for software quality assurance plans [IEEE 730-2002]. The characteristics of the system and project dictate the level of rigor and formality needed in the SQAP. Only the essential elements of a SQAP are presented here. The elements presented here may not be sufficient for safety- or mission-critical systems*. These systems may require the full details of the IEEE standard.

The essential elements of a SQAP are:

1. Purpose
2. Standards and procedures
3. Reviews and audits
4. Management
5. Problem reporting and corrective action

-------------------------------
* Another reason to consider following the full details of the IEEE standard is to avoid being found legally negligent. One of the criteria for establishing legal liability when work results in a catastrophic failure is whether or not the work was performed according to standards established by a representative consensus of industry professionals. It is an open question whether or not software development practices are mature enough to be standardized [IEEE Computer, 27(9) 1994, Evaluation of Software Engineering Standards], but to the extent that they are, the collection of IEEE standards arguably represent the closest work we have to standards established by a representative consensus of industry professionals.
-------------------------------

Purpose

This introductory section of the SQAP describes the motivation and justification for performing the SQA function. What is to be achieved? What are the characteristics of the system or project that justify the additional assurance that SQA provides? Is the system safety critical? Does the project have inherent risks?

This section may also include specific quality objectives. Quality objectives may be process objectives, defect objectives (expected number of defects injected and detected during each phase of the software life cycle), or desired quality characteristics of the system (usability, maintainability, robustness, etc). Some example quality objectives are:

• There will be fewer than 5 defects found per thousand lines of code (KLOC).
• The system will be maintainable. After each code review, participants will be asked to rate the maintainability of the code reviewed in comparison to the other code they have reviewed in the past year.
• There will be automated unit tests for every module.
• All code will be inspected before integration.

Notice that each quality objective is specified in a way that is measurable and verifiable. The priority of each objective should also be clearly stated in case there are conflicts between objects or resource constraints that make it impossible to accomplish all objectives.

Standards and Procedures

This section of the SQAP should list the standards and procedures that will be used during the course of the project. These are the standards and procedures that SQA will make certain are adequate, planned and followed. In most cases this section will contain references to standards and procedures defined elsewhere. Any planned deviations from or tailoring of external standards and procedures should also be documented here.

Standards and procedures are the basis for assessing compliance. They should be well-defined and documented because technical staff have the right to know the criteria for success.

Reviews and Audits

Reviews and audits are the primary tools used by SQA during a project to evaluate product quality and process performance. An audit is a formal type of review performed by independent reviewers. This section should specify what is to be evaluated (which products and processes), the type of evaluation (review or audit), the timing or frequency of the evaluation, and the specific acceptance criteria. Criteria may be specified directly or referenced if available in other external documents or standards. The following table gives an example of what data are expected in this section.

Table 3. Sample SQA Assessment Schedule

Management

The software quality assurance function is like a mini project with all the planning and management needs of a full project. At a minimum the SQAP should specify an organization structure, schedule and budget.

The organizational structure defines the organizational elements responsible for SQA. The SQA function may be centralized or decentralized. The recommended approach is to have an independent organizational element responsible for SQA. The organizational structure should also define the relationship between SQA and the other organizational elements, especially those responsible for the work that will be reviewed. Associated with each organizational element are roles with their assigned authority and responsibility. Because SQA is an oversight function, it's important to clearly define SQA's authority to monitor and evaluate the processes and products of development.

Part of defining roles is defining the expectations of each role. SQA is responsible for evaluating and reporting on the quality of products, but what responsibility do they have for the actual quality of the products they monitor? If a product is found to be of poor quality and they accurately detect and report this, do they bear any responsibility for the defects in the product? The theory of authority and responsibility suggests no. The theory holds that because SQA doesn't have any authority or control over the quality of the product, they shouldn't be held accountable for any deficiencies in the product. If, on the other hand, SQA becomes an active participant in reviews and takes some responsibility for finding defects then they may be held partially accountable for the quality of the resulting products.

The activities of SQA need to be broken down into tasks which are scheduled, budgeted, and coordinated with tasks and milestones in the main project plan. (See chapter 4 for guidelines on defining, scheduling and budgeting tasks.)

Problem Reporting and Corrective Action

This section should describe the procedures for reporting, tracking and resolving problems or issues identified by SQA. Attempting to first resolve problems locally engenders trust between SQA and those performing the work monitored. Problems that can't be resolved locally are usually escalated to senior management for resolution.

17.5 Software Quality Management

The four main constraints on a project are: time, cost, features and quality. Over optimizing one is a wasted opportunity with another. Under performing on any leads to customer dissatisfaction. Key to project success is setting agreeable targets for each constraint at the beginning of a project and then managing the execution of the project to deliver on these targets. Chapter 4 describes how to manage time, cost, and features. This section describes how to manage project quality with the same clarity and precision.

Managing quality goes beyond the familiar procedural approach to quality control [Jalote 2000/2004]. With the procedural approach to quality control there are defined procedures for verification and validation. Test cases are written to verify specific requirements, and reviews are planned for selected work products. The assumption is that if tests and reviews are performed according to their defined procedures, the resulting product will have adequate quality. No attempt is made to assess the effectiveness of testing and reviews or verify that a certain level of quality has been achieved with the product.

The quality management approach to quality control builds on the procedural approach adding more control and predictability. Quality management starts with a quantitative quality goal for the project. From this goal, intermediate quality sub goals are defined for various stages of the project. These intermediate quality sub goals are quality milestones or control points for managing project quality. They motivate action and provide a baseline for measuring progress at various stages of development. With the procedural approach to quality control, performance is measured by the number of test cases executed and reviews performed. With the quality management approach to quality control, performance is measured by the degree to which quantitative quality goals are satisfied. The idea is to make product quality as predictable and controllable as the project's schedule and budget.

The basic principles of quality management are much like those that underlie time or resource management: you set goals, make plans, track progress, and take corrective action when actual progress deviates significantly from plans [Humphrey 1990]. The key to effective quality management is having explicit numeric quality goals that can be measured objectively. What gets measured gets done. If a project has accurate measures for schedule and budget but not quality, the project is likely to meet its schedule and budget goals at the expense of its quality goals.

Setting the Project Quality Goal

The quality management process is driven by the project quality goal which is decided during the planning phase of the project at the same time schedule and budget goals are being determined. A de facto metric for expressing quality goals is defect count [Fenton 1997]. The project quality goal is typically expressed as the maximum number of defects that can be tolerated during late stage testing or during the first year of operation. Results from late stage testing are available early but results during the first year of operation provide a more accurate account of the project's true quality.

Defect count is just one metric for specifying and tracking system quality. Other potential metrics are cost of quality, defect density and test case coverage. Managing project quality by defect count is common because the data are relatively easy to obtain, it's available early in the development process, and it correlates reasonably well to quality as experienced by the customer.

Planning to make any errors might seem like defeatism or even condoning mediocrity, but it's unrealistic to expect no errors. As long as software engineering remains a human activity it will be an error-prone activity. With current development technology, perfect software is impossible or prohibitively expensive for most projects. The project quality goal may be ambitious to motivate action, but it must also be realistic since it is the basis for planning and control actions.

The feasibility of a particular quality goal can be assessed by comparing it to quality performance on past projects and the capability of the existing development process. If the desired quality goal is less than what can be expected from the existing development process, appropriate process improvement actions must be planned in order to close the gap. Keeping quantitative data on completed projects is the best way to know what level of quality can be expected from a development process and what effect process improvement actions are likely to have.

The project quality goal is based on customer needs but the customer may not be able to fully quantify their quality requirements. If the customer has experience with an existing system from the same development team, the quality goal for the new system can be expressed relative to the known quality of the existing system. For example, a customer might request a new system with quality that is equal to or better than that of the existing system. Assuming defect data has been kept on the existing system, developers can translate a request like this into a quantitative quality goal, such as: "The new system shall have at most 30 moderate defects and no sever defects during the first year of operation."

The following example illustrates the main considerations when setting the quality goal for a project.

Example. In this hypothetical example a customer has requested a new system, ACME-2. This request comes a little more than a year after a similar system, ACME-1, was delivered to the same customer. The customer is satisfied with the quality of ACME-1 and has requested that the quality of the new system be as good or better than that of the existing system. Table 4 shows a summary of the defect data collected for ACME-1. According to the data, ACME-1 had 93 defects reported against it during its first year of operation. If the perceived quality of the new system ACME-2 is to be as good or better than that of ACME-1, the new system should have no more than 93 defects reported against it during its first year of operation.

Table 4. Historic defect data for ACME-1

The data in table 4 not only shows the absolute number of defects found in ACME-1, but also defect density which is a measure of the quality of the development process used to create the system. Defect density is the number of defects found per unit of code during a certain time period. The defect density for ACME-1 after 1 year of use is 17 defects per KLOC, with 11% of these defects being found in production. If the same development process is used for ACME-2, similar results can be expected. Table 5 shows the estimated defect profile of ACME-2 assuming no changes in the development process. Notice that the defect density is predicted to remain the same at 17 defects per KLOC, but since ACME-2 is estimated to be 10% larger than ACME-1, the number of production errors is also expected to increase by 10%.

Table 5. Estimated quality profile of ACME-2 based on experience with ACME-1

It should be pointed out that the performance of the development process during the ACME-1 project is just one data point and may not be representative of the true quality potential of the development process. Performance on a single project can be corroborated by comparing it with data on the capability of the development process. Process capability is the range of results expected from a process. For the purposes of this example assume that the data in table 6 represents the quality capability of the development process used to create ACME-1. The data represents the aggregate performance of the development process experienced over several similar projects. Comparing  project performance in table 4 with process capability in table 6 shows that  the performance of the development process during the ACME-1 project is in line with the capability of the development process. This makes the quality profile of ACME-1 a reasonable baseline from which to estimate the quality performance of future projects.

Table 6. Process Capability

Because the new system is larger than the previous system, the development process will need to be improved just to maintain the same level of quality from the customer's perspective. This illustrates why it is important to track both the quality of the development process as well as product quality from the customer's perspective. Most organizations strive for continuous process improvement. This means improving the quality of their development process as well as release-to-release product quality. If the system size grows between releases, it's important to focus on release-to-release quality. If the system size should decrease between releases, the development team will want to focus on the quality of the development process (i.e. defect density) in order to meet internal quality improvement goals.

The quality goal for the ACME-2 project is 93 or fewer defects reported during the first year of operation. Assuming no changes in the development process the new system is estimated to have 103 defects during the first year of operation. The next section on planning for project quality explores process improvement options for meeting the desired project quality goal.

Planning for Project Quality

Preparation and planning are the keys to success in any complex endeavor. The quality plan specifies the course of action for accomplishing the project quality goal.

Planning for project quality is much like planning for other project goals. For example, most projects start with a desired target completion date and then work backwards defining intermediate tasks or milestones throughout the project for tracking and controlling progress toward the goal of competing the project on time. Planning for project quality follows a similar course of action. Once the quality goal is set, intermediate quality sub goals are defined for tracking and controlling progress towards the desired project quality goal. Intermediate quality goals motivate action and provide milestones for measuring progress toward the project quality goal.

The main elements of a project quality plan are:

1. The project quality goal including background and justification.
2. Description of quality measures. How will product quality be measured? If quality is measured by defect count, what is considered a defect? The specific metric used is less important than having consistent conventions for measuring quality so that comparisons can be made between projects.
3. Basic quality control actions such as testing and reviews. What are the main quality control actions planned?
4. Planned enhancements to the standard process. If the project quality goal is more than what can be expected from the existing development process, process improvement actions must be planned.
5. Intermediate quality sub goals. These are the intermediate quality milestones planned at appropriate stages throughout the project that will guide the execution of the project. Quality sub goals are typically specified in terms of expected defect injection and removal rates for each stage of the project.

The quality plan may be a separate document or integrated with the main project plan or quality assurance plan.

The example in the previous section introduced the ACME-2 project and derived the following quality goal for it: 93 or fewer defects during the first year of production. The example that follows develops the main elements of a quality plan for accomplishing this project quality goal.

Example.  Recall from the previous example that, based on the estimated size of the new project and expected defect density of the development process, process improvement initiatives need to be planned in order to meet the desired project quality goal.

Planning for process improvement starts with a thorough understanding of the existing process. For this example, performance during the ACME-1 project is being used as a baseline for understanding the existing development process. Table 7 shows the quality profile of the ACME-1 project. It shows the defect injection and removal rates for various stages from requirements through the first year of operation. Residual defects are latent defects that exist in the product at the start of a stage but have not yet been identified. The number of residual defects at the start of a stage can of course only be calculated after a certain duration of time has passed. When a defect is found it becomes a residual defect in all phases where it was present but not yet detected. Another metric in table 7, removal efficiency, is especially revealing. Removal efficiency is the percentage of existing defects that are removed by a removal activity. For example, a removal activity that is 33% efficient means that 2/3rds of the defects that exist in the product at the start of the removal activity are not detected by the activity. The formula for calculating removal efficiency is:

Removal efficiency = defects removed / (residual defects + injected defects)

Note that removal efficiency is a dynamic number--it goes down over time as latent defects are discovered. The number of defects detected during a stage is fixed at the conclusion of the stage but residual and injected defects go up as latent defects are detected and traced back to their point of origin. Consequently, when comparing removal efficiency (and other dynamic metrics) between projects, it's important that the measures be for the same time period. For example, if application A has been in production for a year it is unfair to compare its removal efficiency to application B that is just entering production. Application A has been exposed to more rigorous testing that has probably had the effect of lowering its removal efficiency.

Also notice that table 7 breaks out data separately for testing and review activities. Research shows that inspections are the most efficient method of defect removal [Jones 97]. It is nearly impossible to have high development efficiency without also having high inspection efficiency. Knowing the efficiencies of different defect removal activities provides insights as to where process improvement efforts should be focused. 

Table 7. Defect Data by stage for ACME-1

Once the existing development process is well-understood you can begin to look for ways to improve it. Opportunities for process improvement can be found by comparing the performance of the existing development process with organization and industry norms. Areas of the existing process that compare unfavorably are good candidates for process improvement initiatives.

Table 8 shows data on the capability of the existing development process. This data represents the aggregate performance of the development process calculated over several projects. Comparing process performance (table 7) with process capability (table 8), reveals that inspection efficiency during the ACME-1 project is slightly below the organization average--62% vs. 65%. Looking closer at removal efficiency by stage in both table 7 and table 8, it appears that low removal efficiency during code inspections accounts for a significant portion of the difference (especially when you consider the number of defects injected during the coding phase). Still more analysis is needed in order to determine if poor efficiency is due to the number of inspections performed or the effectiveness of individual inspections. For the purposes of this example assume that the problem was simply a matter of inspections not being performed. Assume that the 54% removal efficiency during the coding phase of ACME-1 was achieved by inspecting only 20% of the code base. Based on this analysis one tactic for improving the performance of the development process during the ACME-2 project is to increase the number of code inspections performed.

Table 8. Process Capability by Stage

Now that we have identified an opportunity for process improvement we can begin to construct the quality plan for the ACME-2 project. The quality goal for the ACME-2 project is 93 or fewer defects during the first year of production. The estimated size of ACME-2 is 55 KLOC. Assuming that defect density remains the same at 17 defects per KLOC, development efficiency will need to increase from 89% to 90% in order to accomplish the desired quality goal of 93 or fewer defects during the first year of production. Without any other changes in the development process, this can be accomplished if removal efficiency during the coding phase is increased from 50% to 56%. Some analysis is required to determine what portion or percentage of the ACME-2 code base needs to be inspected in order to reach this goal. Table 9 shows the estimated quality profile for the ACME-2 project assuming the development process is changed to include more code inspections. The data in table 9 also provides quality sub goals for each phase of development for tracking and controlling progress towards the project quality goal. For example, according to the data the development team should expect to find 58 defects during the requirements phase. If significantly more or fewer defects are found this is an indication that the team is not on track to accomplish the quality goal for the project.

Table 9. Quality goals for ACME-2 assuming improved inspection efficiency

Another option for improving the existing development process is to take action to prevent defects from occurring. This will lower the defect density and, assuming removal efficiency doesn't go down, improve overall product quality.

Data on defect distribution in tables 7 and 8 show that about half of all defects are introduced during the coding phase. This makes the coding phase an attractive target for defect prevention actions. Table 9 shows that without taking any action to reduce the number of defects injected during the ACME-2 project, 477 defects are expected to be injected during the coding phase. If this can be reduced by 20% the quality goal for the ACME-2 project can be accomplished through defect prevention alone.  Table 10 shows the estimated quality profile for the ACME-2 project assuming a 20% reduction in the number of defects injected during the coding phase. Just making the estimate doesn't make it so though. The quality plan must also justify the estimate and include detailed plans for achieving it. The defect prevention process is described in more detail in the next section.

Table 10. Quality goals for ACME-2 assuming improved defect prevention

The examples in this section showed defect detection and defect prevention activities being applied separately. In practice, the most effective approach to quality improvement is implementing some combination of both.

Tracking and Controlling Project Quality

Tracking and control are universal management functions just as important for quality management as for schedule, budget and scope management. Tracking and control are used collectively to ensure that a project, once started, makes steady progress towards established goals. The general concept of tracking and control is discussed on chapter 4. In the context of quality management, tracking and controlling consist of:

• Planning - Quantitative quality goals are established at the beginning of a project. Goals are set in terms of product quality and process performance. For example, the expected number of defects found during a review activity is a measure of product quality. The number of inspections performed is a measure of process performance.
• Tracking - At certain points during the project, product quality and process performance are reviewed against plans. Actual results should meet target values or fall within ranges set for target values. When there are discrepancies additional analysis may be necessary in order to explain deviations.
• Controlling - When performance deviates significantly from plans, corrective action must be taken to mitigate effects of potential or actual problems. The alternative to corrective action is to make changes to commitments and plans. This, however, should only be done in response to changes in the goals and objectives of the project. When plans or commitments do change, these changes must be communicated to all stakeholders.

Tracking and control at the project level is similar in concept to quality control at the product level. The difference is one of scope. Tracking and control at the product level drives tracking and control at the project level.

17.6 Defect Prevention

The benefits of finding errors early are well known. The cost of finding and fixing a defect goes up dramatically the longer it remains in the product. The ultimate in early defect detection though is defect prevention, that is preventing errors from ever occurring in the first place. Defect prevention is a systematic process for avoiding errors by identifying the underlying causes of errors that have occurred in the past and taking specific actions to eliminate these causes in order to prevent the errors from reoccurring again in the future.

Defect prevention and detect detection are complementary approaches to achieving better quality products. Defect detection looks for defects in work products and defect prevention looks for ways to improve the development process so that errors are avoided. Errors that can't be prevented must be detected and removed. (See figure 32.)

Figure 32. Errors that can't be prevented must be detected and removed

Defect prevention improves product quality and development productivity through incremental improvements to the underlying development process. This makes defect prevention primarily a method for improving the development process with the expected benefits being improved product quality and development productivity.

This section on defect prevention is divided into two parts. The first half explores the benefits of defect prevention in terms of process improvement, quality improvement and productivity improvement. The second half describes the activities of the defect prevention process in more detail.

Process Improvement

Defect prevention is an effective method of process improvement because process changes that come from defect prevention activities are systematic and driven by actual defect data. In most development environments there is no shortage of ideas for improving the development process. However, most ideas are based on personal perceptions and experiences. There is no assurance that these individual ideas for process improvement are more broadly applicable. Process improvement changes initiated by the defect prevention process are based on a statistical analysis of all problems so they are more likely to address the most urgent deficiencies in the development process.

Quality Improvement

In order for defect prevention to improve the quality of products it must prevent errors that otherwise would not have been found during the testing and review process*. Defect prevention can improve the quality of delivered software because testing tends to remove a certain percentage of errors. If defect prevention causes the product to enter the test phase with fewer inherent errors there will be fewer residual errors in the product after testing. Figure 33 illustrates this concept. The stroke pattern in both images depicts the fact that for any non-trivial system it's impractical and sometimes impossible to test all paths or operating conditions. Figure 33.b illustrates a reduction in defect density (fewer dots) that can be expected from using defect prevention. Because the characteristics of an error that make it preventable are independent of the characteristics that make it detectable, the defects prevented are just as likely to be among those detected as those not detected. The net result is that defect prevention reduces the overall number of residual errors that escape testing because some of the errors prevented will be errors that would not have been detected during inspection and testing.

_______________
*Notice the argument here is strictly for improving the quality of products. If defect prevention avoids an error that would have been found with testing, it doesn't improve the quality of the product but it still can improve overall productivity because it avoids the wasted effort of finding and fixing the defect and making any collateral changes.
---------------------------

(a) High Initial Defect Density

(b) Low Initial Defect Density

Figure 33. Fewer inherent errors before testing means fewer residual errors after testing.

The assertion that defect prevention tends to remove a certain percentage of all errors can be empirically tested by comparing the reduction in the number of errors found during development to the reduction in the number of errors found in production. If defect prevention does in fact remove a certain percentage of errors, then it should reduce both development and production errors by equal percentages.

The quantitative data in the literature seem to support the idea that defect prevention removes a certain percentage of all errors. Reporting on the results of a defect prevention program at IBM, Mays notes a 54% decrease in defects during development and preliminary estimates of a 60% decrease in field defects [Mays 1990]. Humphrey reports on a different project that experienced a 50% reduction in defects found during development and an even greater 78% reduction in field defects. Humphrey goes on to explain why defect prevention might eliminate a greater percentage of post-release defects than pre-release defects. He surmises that when fewer errors are injected during development, inspections (and possibly testing) are likely to be more efficient. The defect removal efficiency of inspections is likely to go up when there are fewer errors in the product being inspected. As an example, consider a small document with 70 or more errors in it. At the very least the large number of errors creates a distraction that makes it harder to do a thorough review. Reviewers are likely to give up in frustration before finding all of the errors that are easily detectable. If instead, the document starts with just 10-15 errors, reviewers have a better chance of finding most or all of the errors.

Productivity Improvement

Productivity is the output (amount and quality) per unit of labor or effort:

Figure 34. Productivity

Defect prevention improves productivity because it reduces the amount of effort it takes to deliver a certain output*. Preventing defects tends to increase productivity primarily because it takes less effort to avoid a mistake than it does to make a mistake, detect it, and then correct it. In other words, when the added burden of rework is taken into account, it usually takes less overall effort to do things right the first time.

___________
As mentioned in the previous section, defect prevention also improves product quality. Because productivity is both the amount and quality of output per unit of effort, defect prevention can also increase productivity even when effort isn't reduced. Increasing product quality for the same amount of effort also increases productivity. It's much harder to measure the effects of improved quality on productivity though.
--------------------

As section 17.2.5 explains, the cost of quality curve is U shaped for most applications. This implies that for most applications there is a point of diminishing returns beyond which additional investments in defect prevention aren't paid back with lower failure costs. This implies that it is possible to overspend on quality improvement.  One of the characteristics of the DP process that mitigates this concern is that defect prevention is a controlled process. It's not an all or nothing approach to defect prevention. The actions with the highest predicted return on investment can be implemented first. Additional corrective actions can be implemented as justified based on results achieved.

Another way that preventing defects improves productivity is that, when combined with testing, defect prevention makes it possible to achieve high levels of product quality more efficiently than with testing alone. Recall from section 17.2.5 that with testing alone, as the percentage of defects detected approaches 100%, it becomes exponentially more expensive to find additional errors. (See figure 35.)

Figure 35. Effort required to remove additional defects rises rapidly as the percentage of defects detected approaches 100%

The last few defects in a product are expensive to remove because they are difficult to detect. They might be difficult to detect but they aren't necessarily difficult to prevent. For example, consider an application that is close to its point of diminishing returns when testing reaches 90% efficiency. Finding the last 10% of defects with testing alone is likely to be very expensive. However, another option for improving the quality of the product is defect prevention. Preventing just 40% of the errors that would have been injected into the product is likely to remove 40% of the remaining residual errors. It would be much more expensive to find these additional errors with testing alone.

For most applications there is a point of diminishing returns for both defect detection and defect prevention. The best value for your quality dollar is when defect prevention is combined with defect detection and both processes are allowed to work in their "linear zone". For a given amount of effort the overall level of quality achieved is greater than what either method could accomplish with the same amount of effort working alone. Figure 36 shows a conceptual example. The combined effort for defect prevention and defect detection is less than the effort required to get the same level of removal efficiency with either defect prevention or defect detection alone.

Figure 36. It's more efficient to remove errors with prevention and detection then with either one alone.

The Defect Prevention Process

The defect prevention process is a systematic process for improving the quality of delivered software through incremental improvements to the underlying development process [Jones].  Figure 37 illustrates the main steps in the defect prevention process which are:

1. Work is performed and verified according to a defined development process. Data on any defects are gathered and recorded for analysis in the next step.

2. Causal analysis is performed on defects to determine their underlying causes.

3. Corrective actions are proposed for eliminating the underlying causes and preventing similar types of defects in the future.

4. Corrective actions are prioritized and the most promising ones are selected for implementation. New ideas for process improvement are first tried locally on a single project. Those that prove valuable at the local level are made a permanent part of the standard development process.

Figure 37. The Defect Prevention Process

Defect prevention is a planned and systematic form of continuous process improvement. Causal analysis of defects drives changes in the underlying development process. The changes are implemented and the cycle starts again on the changed process.

Defect prevention is a supporting process. A prerequisite to performing defect prevention is the existence of a well-defined software development process. Most of the ideas for preventing defects are process improvement ideas. A well-defined process provides the structure and framework needed for implementing and institutionalizing process changes. Figure 38 shows how the defect prevention process integrates with the software development process. The solid lines represent elements of a standard software development process and the dashed lines represent the additional elements of defect prevention.

Figure 38. Defect prevention relies on the framework of an existing development process

Planning for Defect Prevention

Defect prevention is a simple process but it still requires some up-front planning. Like any other planning exercise you must decide who does what when.

The major events during the defect prevention process are:

1. The Stage Kickoff Meeting - At the beginning of each task or stage of development a stage kickoff meeting is held. The purpose of the kickoff meeting is to increase awareness of quality issues specific to the current stage of development.
2. The Causal Analysis Meeting - After a certain amount of development work has been performed a causal analysis meeting is held to analyze the defects found. The purpose of the meeting is to identify the underlying causes of these defects and suggest specific actions for preventing similar types of defects in the future.
3. The Action Team Meeting - Periodically an action team meeting is held to prioritize and implement the suggested actions.

One of the first decisions in planning for defect prevention is deciding when and how often to hold stage kickoff, causal analysis and action team meetings. A kickoff meeting is normally held at the beginning of every task or stage of development. The timing of the causal analysis and action team meetings depends on the needs of the project and the life cycle model being followed. One option is to hold causal analysis meetings after every stage of development and action team meetings at regular intervals [Mays 1990 IBM Systems Journal]. (See figure 39.) This arrangement is probably best suited to projects following the waterfall life cycle model where a substantial amount of work is performed during each phase. When developing software in short frequent iterations, distinct stages may be difficult to discern and the amount of work performed during a stage of development may be insufficient to justify a causal analysis meeting.

Figure 39. Causal analysis being performed after each phase

A more appropriate timing for defect prevention activities during a project following an iterative life cycle model is to schedule causal analysis and action team meetings after each iteration. (See figure 47.) The amount of work performed during an iteration is usually sufficient to justify a causal analysis meeting. In general, causal analysis should be delayed until there are enough significant defects to make the activity worthwhile--typically 15-20 errors. On the other hand, the sooner causal analysis is performed and corrective actions are implemented, the greater the quality and productivity will be because corrective actions will have had longer to affect the current project.

Figure 40. Causal analysis being performed after each Iteration

A third option for scheduling defect prevention activities is by percentage of work complete. At Infosys the general guideline is to perform causal analysis after 20% of the system has been coded, reviewed and unit tested, and again after 50% of the system has been coded, reviewed and unit tested [Jalote, 2002]. (See figure 48.) Twenty percent into a project is late enough for substantial problems to emerge, but early enough that corrective actions will have a chance to contribute to the quality and productivity of the current project. Performing another round of defect prevention at the 50% mark provides the chance to verify that corrective actions are having their intended effect as well as plan additional corrective actions for the second half of the project.

Figure 41. Scheduling causal analysis based on percent complete

Another convenient time for performing causal analysis is a few months after the product has been released. Errors that make it all the way to the customer are the most important to prevent because they are the most expensive to fix and the most damaging to the reputation of the product. When an error reaches the customer it indicates a problem in the testing process as well as the development process.

The two main groups of individuals that share responsibility for the defect prevention process are the causal analysis team and the action team. The causal analysis team is comprised of the technical staff responsible for development. Programmers shouldn't test their own code but they should perform causal analysis on their own errors. The person who makes an error probably understands the error and contributing factors better than anyone else. As May observes, "direct causal analysis by the developer making the error results in a more accurate determination of the cause of the defect and more relevant preventive actions." [May] The action team is responsible for ensuring that suggested actions are implemented. The action team is staffed with individuals that have the authority and resources to effect changes in the development process.

Both the causal analysis and action teams should be trained in defect prevention. In addition, the causal analysis team should be trained in statistical techniques such as Pareto charts and cause-effect diagrams. (Statistical tools for quality control are discussed in chapter 20.)

Stage Kickoff Meeting

A stage kickoff meeting is held at the beginning of every task or stage of development. The individuals performing the work meet to review the upcoming activities including feedback from earlier rounds of defect prevention. The meeting is used to increase awareness of quality issues right before the work begins. Many errors are caused by oversight or lack of awareness. When developers are reminded of the most common errors made during a task right before they perform the task, they are less likely to repeat the errors.

The stage kickoff meeting is the primary mechanism for introducing feedback from causal analysis into the development process. Recommended changes from causal analysis are typically first introduced during the stage kickoff meeting of a single project. Those that prove helpful locally are later migrated to the standard development process and made available to all projects.

The main agenda items at a stage kickoff meeting are [Humphry, Jones, Mays]:

1. Review the inputs to the current stage for completeness and understandability.
2. Specify and clarify the expected outputs of the current stage. Examples and document templates are especially helpful in clarifying expected outputs. Performance criteria and quality assurance guidelines may also be reviewed.
3. Review the list of common errors specific to the current stage. Many errors are caused by oversight or lack of awareness. A common solution for preventing errors caused by oversight is to add a reminder to a list of common errors which is reviewed at the start of the stage of development where the error may occur.
4. Review guidelines from the standard development process that are relevant to the current stage. Guidelines are expressed in the form of standards, procedures, checklists, etc. Methods and tools for the current phase may also be reviewed.
5. Discuss and set internal team quality goals. Most projects are already working towards release quality goals, typically specified in a quality plan or project plan.  Internal quality goals are goals over and above any existing quality goals. The motivation for having more aggressive internal quality goals is that release quality goals are set based on the expected capability of the standard development process. Internal quality goals are more ambitious and reflect the improved capabilities expected from defect prevention activities. Because they are based on speculative process improvements that haven't been proven yet, internal quality goals are never published. Internal project quality goals motivate and garner commitment from those performing the work. Quality goals are typically stated in terms of the estimated number of defects injected and detected during a phase of development.

Collecting Data on Defects

To aid later causal analysis, data on defects are gathered as defects are detected and corrected. It's important to collect as much information as possible up front while the details of the problems are fresh in the minds of those dealing with them.

Most organization getting started with defect prevention will already have a system in place for tracking and reporting defects. In most cases this system can be made to work for defect prevention with the addition of a few extra fields. The fields important for defect prevention, that might not be included in a general defect tracking database are:

1. Cause category - The cause category is a first-level classification of the defect. Example cause categories are: process, people, technology, and training [infosys]; oversight, education, communication failure, and transcription error [Jones 84, Mays 90]. Specifying a cause category helps focus the efforts of the causal analysis team. Having a fixed number of predefined cause categories also facilitates statistical analysis of defects by type.
2. Specific cause - This is a free-form field for capturing the complete and specific cause for the defect. Preliminary speculation on the cause of a defect may be added at the time it is detected or corrected, but the final determination of the cause isn't made until causal analysis when all of the information on the defect is available for consideration.
3. Recommended preventive actions - This field documents the specific actions recommended for preventing similar types of defects from occurring again in the future.
4. Assigned actions - This field is for tracking the status of action items. Data on preventive actions may be stored directly in this field, or more likely, specified in a separate database of preventive actions. When actions are tracked in a separate database this field becomes a "foreign key" or index that connects records in the defect database with records in the preventive actions database.

More information on defect data tracking and reporting can be found in chapter 16.

Causal Analysis

Causal analysis meetings are held periodically during a project, usually at the end of a development phase after the work products of that phase have been verified and all defects found in the phase have been corrected and documented. A causal analysis meeting is similar to a formal inspection and has some of the same sensitivities found at formal inspections. The participants are those directly responsible for the work just completed. These are the individuals responsible for the errors which are being analyzed. As with inspections, the focus should be on the errors and how to improve the process rather than the faults of the individual. To encourage open and honest discussion of defects, managers generally don't attend.

During a causal analysis meeting participants analyze recent defects, determine their root causes, and suggest ways of avoiding them in the future.

The focus of causal analysis is on error prevention, but for errors detected in a stage later than the stage in which they were introduced, some consideration may also be given to suggestions for finding the error earlier, closer to the phase it was introduced.

The product or output of causal analysis is a prioritized list of specific suggestions or actions for preventing the defects that were analyzed. These preventive actions are forwarded to the action team for implementation.

To clarify the concepts, here is simple example that illustrates the key activities of causal analysis.

Preliminaries. For this example, assume that 22 errors were found during the implementation phase of one iteration. The errors have been corrected and data on each error has been recorded in a database.

Review Defects. The first order of business during causal analysis is to decide which errors to review in more detail. It may not be feasible or worthwhile to perform detailed causal analysis on all errors. Priority should be given to errors with high failure costs and those more likely to occur again in the future. A Pareto chart showing the distribution of errors among a set of common causes helps to identify the categories with the most errors. Figure 49 shows what a Pareto chart might look like for the 22 hypothetical errors assumed for this example. The chart suggests that errors caused by oversight and insufficient technical education are more prevalent and therefore deserve the most attention.

Common Cause Categories

Figure 42. Pareto Diagram

Root Cause Analysis. The best preventive actions come from thoughtful consideration of specific causes. Just knowing the common cause category of a defect is not enough to suggest useful preventive actions. Table 11. shows a portion of the defect data record for a specific error from the oversight category. The specific error is exceeding the number of database connections that can be open at one time. During root cause analysis additional information is usually drawn out. Did the programmer really forget to close database connections as specified in the defect record, or was the programmer unaware that database connects must be closed. If it was the latter this suggests a different common cause category--insufficient technical education--which would dictate a different response. For the rest of this example, assume the root cause was simply forgetting to close a connection after use.

A useful tool for analyzing defects with multiple apparent causes is a cause-effect diagram (also called fishbone or Ishikawa diagram). It is used to identify and organize the possible causes of a defect. (Cause-effect diagrams are described in more detail in chapter 20.)

Table 11. A portion of a defect data record

Preventive Actions. Once the root causes of defects are understood, attention can shift to looking for ways of preventing them from occurring again in the future. There are a couple of ways to avoid the problem of programmers forgetting to close database connections. First, a reminder to close database connections can be added to the common errors list for the implementation phase. Updating the common errors list for a particular development phase is one of the most common changes made to prevent defects. Second, and potentially more costly, the interface onto the database can be redesigned--made more abstract--so that programmers don't have to be concerned with opening and closing database connections.

Output of the causal analysis phase are specific suggestions for preventing defects. When there is more than one response to a particular defect, the suggested actions should be prioritized. These recommended actions are usually documented in the defect record for the error they are meant to address. Table 12 shows the updates to the suggested actions field of the record for the defect being analyzed in this example.

Table 12. Suggested Action Items

The Action Team

The causal analysis team generates ideas for preventing defects and the action team implements them. The action team meets regularly to review, prioritize and implement suggested actions from causal analysis. The action team is staffed with individuals with the authority to authorize work in the areas most likely affected by action items such as the development process, tools and education.

The main responsibilities of the action team are:

1. Review and prioritize action items. Typically there are more ideas for preventing defects than there is time and resources available to implement them. The action team considers each suggestion and selects the most promising ones for implementation. Factors considered in the decision process are the consequences of not addressing the cause, the cost of implementing the suggestion, and the expected impact or effectiveness of the proposed solution.

2. Assign responsibility for implementing each action item. Typically action items are divided among the members of the action team for implementation. If there is someone outside the team better qualified to implement an action item, the item is assigned to this individual and a member from the action team takes responsibility for tracking the item to completion.

3. Track and report status of action items. During each meeting the action team not only considers new action items but also monitors the progress of open items. It is the team's responsibility to ensure that all items selected for implementation are completed. There are no dependencies on defect prevention action items which makes it easy to put off their completion. The interest the action team shows in open items motivates progress.

4. Provide feedback. Developers performing causal analysis are encouraged and energized to come up with new ideas for defect prevention when they see their suggestions being taken seriously. The final step in the implementation of an action item should be to notify the individual submitter and organization of its completion. When a suggestion is not selected for implementation the submitter should be notified of the rational for the decision.

Summary

tbd