SOFTWARE TESTING

Yogesh Singh

4843/24, 2nd Floor, Ansari Road, Daryaganj, Delhi - 110002, India Cambridge University Press is part of the University of Cambridge. It furthers the University’s mission by disseminating knowledge in the pursuit of education, learning and research at the highest international levels of excellence. www.cambridge.org Information on this title: www.cambridge.org/9781107012967 © Yogesh Singh 2012 This publication is in copyright. Subject to statutory exception and to the provisions of relevant collective licensing agreements, no reproduction of any part may take place without the written permission of Cambridge University Press. First published 2012 First Paperback edition 2012 Reprint 2013, 2014, 2016, 2017 Printed in India A catalogue record for this publication is available from the British Library Library of Congress Cataloguing in Publication data Singh, Yogesh, 1966Software testing / Yogesh Singh. p. cm. Includes bibliographical references. Summary: “Discusses the concept of Software Testing with real-life case studies and solved examples”-- Provided by publisher. ISBN 978-1-107-01296-7 (hardback) 1. Computer software--Testing. 2. Computer software--Quality control. I. Title QA76.76.T48S56 2011 4--dc22 2011012053’005.1 ISBN 978-1-107-65278-1 Paperback Cambridge University Press has no responsibility for the persistence or accuracy of URLs for external or third-party internet websites referred to in this publication, and does not guarantee that any content on such websites is, or will remain, accurate or appropriate.

Contents

List of Figures List of Tables Preface Acknowledgements

1. Introduction 1.1

1.2

1.3

Some Software Failures 1.1.1 The Explosion of the Ariane 5 Rocket 1.1.2 The Y2K Problem 1.1.3 The USA Star-Wars Program 1.1.4 Failure of London Ambulance System 1.1.5 USS Yorktown Incident 1.1.6 Accounting Software Failures 1.1.7 Experience of Windows XP Testing Process 1.2.1 What is Software Testing? 1.2.2 Why Should We Test? 1.2.3 Who Should We Do the Testing? 1.2.4 What Should We Test? Some Terminologies 1.3.1 Program and Software 1.3.2 Verification and Validation 1.3.3 Fault, Error, Bug and Failure 1.3.4 Test, Test Case and Test Suite 1.3.5 Deliverables and Milestones 1.3.6 Alpha, Beta and Acceptance Testing 1.3.7 Quality and Reliability

xi xv xxi xxiii

1 1 1 2 3 3 3 3 4 4 5 14 15 16 19 19 20 21 21 22 22 23

iv

Contents

1.3.8 Testing, Quality Assurance and Quality Control 1.3.9 Static and Dynamic Testing 1.3.10 Testing and Debugging 1.4 Limitations of Testing 1.4.1 Errors in the Software Requirement and Specification Document 1.4.2 Logical Bugs 1.4.3 Difficult to Measure the Progress of Testing 1.5 The V Shaped Software Life Cycle Model 1.5.1 Graphical Representation 1.5.2 Relationship of Development and Testing Parts Multiple Choice Questions Exercises Further Reading

2. Functional Testing 2.1

Boundary Value Analysis 2.1.1 Robustness Testing 2.1.2 Worst-Case Testing 2.1.3 Robust Worst-Case Testing 2.1.4 Applicability 2.2 Equivalence Class Testing 2.2.1 Creation of Equivalence Classes 2.2.2 Applicability 2.3 Decision Table Based Testing 2.3.1 Parts of the Decision Table 2.3.2 Limited Entry and Extended Entry Decision Tables 2.3.3 ‘Do Not Care’ Conditions and Rule Count 2.3.4 Impossible Conditions 2.3.5 Applicability 2.4 Cause-Effect Graphing Technique 2.4.1 Identification of Causes and Effects 2.4.2 Design of Cause-Effect Graph 2.4.3 Use of Constraints in Cause-Effect Graph 2.4.4 Design of Limited Entry Decision Table 2.4.5 Writing of Test Cases 2.4.6 Applicability Multiple Choice Questions Exercises Further Reading

3. Essentials of Graph Theory 3.1

What is a Graph? 3.1.1 Degree of a Node 3.1.2 Regular Graph

23 23 24 24 24 24 26 26 27 27 28 34 35

37 38 43 44 46 48 63 63 65 81 81 82 82 83 83 96 97 97 97 99 99 99 102 105 108

110 110 112 113

Contents v

3.2

Matrix Representation of Graphs 3.2.1 Incidence Matrix 3.2.2 Adjacency Matrix 3.3 Paths and Independent Paths 3.3.1 Cycles 3.3.2 Connectedness of a Graph 3.4 Generation of a Graph from Program 3.4.1 Program Graphs 3.4.2 DD Path Graphs 3.5 Identification of Independent Paths 3.5.1 Cyclomatic Complexity 3.5.2 Graph Matrices Multiple Choice Questions Exercises Further Reading

4. Structural Testing 4.1

Control Flow Testing 4.1.1 Statement Coverage 4.1.2 Branch Coverage 4.1.3 Condition Coverage 4.1.4 Path Coverage 4.2 Data Flow Testing 4.2.1 Define/Reference Anomalies 4.2.2 Definitions 4.2.3 Identification of du and dc Paths 4.2.4 Testing Strategies Using du-Paths 4.2.5 Generation of Test Cases 4.3 Slice Based Testing 4.3.1 Guidelines for Slicing 4.3.2 Creation of Program Slices 4.3.3 Generation of Test Cases 4.4 Mutation Testing 4.4.1 Mutation and Mutants 4.4.2 Mutation Operators 4.4.3 Mutation Score Multiple Choice Questions Exercises Further Reading

5.

113 114 114 116 117 117 123 124 127 144 144 150 159 161 163

165 165 166 167 167 167 173 174 174 175 175 176 197 197 198 202 212 212 216 216 223 226 228

0 5.1

Verification Methods 5.1.1 Peer Reviews

230 231

vi

Contents

5.1.2 Walkthroughs 5.1.3 Inspections 5.1.4 Applications 5.2 Software Requirements Specification (SRS) Document Verification 5.2.1 Nature of the SRS Document 5.2.2 Characteristics and Organization of the SRS Document 5.2.3 SRS Document Checklist 5.3 Software Design Description (SDD) Document Verification 5.3.1 Organization of the SDD Document 5.3.2 The SDD Document Checklist 5.4 Source Code Reviews 5.4.1 Issues Related to Source Code Reviews 5.4.2 Checklist of Source Code Reviews 5.5 User Documentation Verification 5.5.1 Review Process Issues 5.5.2 User Documentation Checklist 5.6 Software Project Audit 5.6.1 Relevance Scale 5.6.2 Theory and Practice Scale 5.6.3 Project Audit and Review Checklist 5.7 Case Study Multiple Choice Questions Exercises Further Reading

6. Creating Test Cases from Requirements and Use Cases 6.1

6.2

6.3

6.4

Use Case Diagram and Use Cases 6.1.1 Identification of Actors 6.1.2 Identification of Use Cases 6.1.3 Drawing of Use Case Diagram 6.1.4 Writing of Use Case Description Generation of Test Cases from Use Cases 6.2.1 Generation of Scenario Diagrams 6.2.2 Creation of Use Case Scenario Matrix 6.2.3 Identification of Variables in a Use Case 6.2.4 Identification of Different Input States of a Variable 6.2.5 Design of Test Case Matrix 6.2.6 Assigning Actual Values to Variables Guidelines for generating validity checks 6.3.1 Data Type 6.3.2 Data Range 6.3.3 Special Data Conditions 6.3.4 Mandatory Data Inputs 6.3.5 Domain Specific Checks Strategies for Data Validity 6.4.1 Accept Only Known Valid Data

231 231 232 233 233 233 235 238 239 239 241 241 242 243 244 244 245 246 246 246 257 279 282 283

285 285 286 287 288 290 292 293 294 295 296 296 296 316 316 316 316 316 316 317 317

Contents vii

6.4.2 Reject Known Bad Data 6.4.3 Sanitize All Data 6.5 Database Testing Multiple Choice Questions Exercises Further Reading

7.

Selection, Minimization and Prioritization of Test Cases for Regression Testing 7.1

What is Regression Testing? 7.1.1 Regression Testing Process 7.1.2 Selection of Test Cases 7.2 Regression Test Cases Selection 7.2.1 Select All Test Cases 7.2.2 Select Test Cases Randomly 7.2.3 Select Modification Traversing Test Cases 7.3 Reducing the Number of Test Cases 7.3.1 Minimization of Test Cases 7.3.2 Prioritization of Test Cases 7.4 Risk Analysis 7.4.1 What is Risk? 7.4.2 Risk Matrix 7.5 Code Coverage Prioritization Technique 7.5.1 Test Cases Selection Criteria 7.5.2 Modification Algorithm 7.5.3 Deletion Algorithm Multiple Choice Questions Exercises Further Reading

8. Software Testing Activities 8.1

8.2

8.3

Levels of Testing 8.1.1 Unit Testing 8.1.2 Integration Testing 8.1.3 System Testing 8.1.4 Acceptance Testing Debugging 8.2.1 Why Debugging is so Difficult? 8.2.2 Debugging Process 8.2.3 Debugging Approaches 8.2.4 Debugging Tools Software Testing Tools 8.3.1 Static Software Testing Tools

322 322 326 331 333 334

335 335 336 337 339 339 339 339 340 340 341 342 342 343 346 347 347 352 363 364 365

368 368 369 370 373 373 374 374 375 377 378 379 379

viii

Contents

8.3.2 Dynamic Software Testing Tools 8.3.3 Process Management Tools 8.4 Software Test Plan Multiple Choice Questions Exercises Further Reading

9. Object Oriented Testing 9.1

What is Object Orientation? 9.1.1 Classes and Objects 9.1.2 Inheritance 9.1.3 Messages, Methods, Responsibility, Abstraction 9.1.4 Polymorphism 9.1.5 Encapsulation 9.2 What is Object Oriented Testing? 9.2.1 What is a Unit? 9.2.2 Levels of Testing 9.3 Path Testing 9.3.1 Activity Diagram 9.3.2 Calculation of Cyclomatic Complexity 9.3.3 Generation of Test Cases 9.4 State Based Testing 9.4.1 What is a State Machine? 9.4.2 State Chart Diagram 9.4.3 State Transition Tables 9.4.4 Generation of Test Cases 9.5 Class Testing 9.5.1 How Should We Test a Class? 9.5.2 Issues Related to Class Testing 9.5.3 Generating Test Cases Multiple Choice Questions Exercises Further Reading

381 382 382 383 386 387

389 389 390 391 393 394 394 395 395 395 396 396 400 401 404 404 406 407 408 411 412 412 412 415 417 418

10. Metrics and Models in Software Testing

420

10.1 Software Metrics 10.1.1 Measure, Measurement and Metrics 10.1.2 Applications 10.2 Categories of Metrics 10.2.1 Product Metrics for Testing 10.2.2 Process Metrics for Testing 10.3 Object Oriented Metrics Used in Testing

420 420 421 422 422 423 423

Contents ix

10.3.1 Coupling Metrics 10.3.2 Cohesion Metrics 10.3.3 Inheritance Metrics 10.3.4 Size Metrics 10.4 What Should We Measure During Testing? 10.4.1 Time 10.4.2 Quality of Source Code 10.4.3 Source Code Coverage 10.4.4 Test Case Defect Density 10.4.5 Review Efficiency 10.5 Software Quality Attributes Prediction Models 10.5.1 Reliability Models 10.5.2 An Example of Fault Prediction Model in Practice 10.5.3 Maintenance Effort Prediction Model Multiple Choice Questions Exercises Further Reading

11. Testing Web Applications 11.1 What is Web Testing? 11.1.1 Web Application versus Client Server Application 11.1.2 Key Areas in Testing Web Applications 11.2 Functional Testing 11.3 User Interface Testing 11.3.1 Navigation Testing 11.3.2 Form Based Testing 11.3.3 User Interface Testing Checklist 11.4 Usability Testing 11.4.1 What is Usability and Usability Testing? 11.4.2 Identification of Participants 11.4.3 Development of Usability Testing Questionnaire 11.4.4 Setting up Environment for Conducting Test 11.4.5 Conducting the Test 11.4.6 Analyze the Results and Observations 11.5 Configuration and Compatibility Testing 11.5.1 Browser Testing 11.5.2 Guidelines and Checklist for Configuration and Compatibility Testing 11.6 Security Testing 11.7 Performance Testing 11.7.1 Load Testing 11.7.2 Stress Testing 11.7.3 Performance Testing Checklist 11.8 Database Testing 11.9 Post-Deployment Testing

424 424 425 426 427 427 428 429 429 429 430 430 437 442 446 449 451

453 453 453 455 456 458 458 459 461 463 463 464 465 468 469 469 469 470 470 471 476 476 479 479 480 482

x

Contents

11.10 Web Metrics Multiple Choice Questions Exercises Further Reading

12. Automated Test Data Generation

485 486 490 492

494

12.1 What is Automated Test Data Generation? 12.1.1 Test Adequacy Criteria 12.1.2 Static and Dynamic Test Data Generation 12.2 Approaches to Test Data Generation 12.2.1 Random Testing 12.2.2 Symbolic Execution 12.2.3 Dynamic Test Data Generation 12.3 Test Data Generation using Genetic Algorithm 12.3.1 Initial Population 12.3.2 Crossover and Mutation 12.3.3 Fitness Function 12.3.4 Selection 12.3.5 Algorithm for Generating Test Data 12.4 Test Data Generation Tools Multiple Choice Questions Exercises Further Reading

494 495 495 496 496 496 501 502 503 503 504 505 505 511 512 513 514

Appendix I Appendix II Appendix III References Answers to Multiple Choice Questions Index

517 541 594 612 617 621

List of Figures

1.1 1.2 1.3 1.4 1.5 1.6 1.7 1.8 1.9 1.10 2.1 2.2 2.3 2.4 2.5 2.6 2.7 2.8 2.9 2.10 2.11 2.12 2.13 2.14 2.15 2.16 3.1 3.2 3.3 3.4 3.5 3.6 3.7 3.8

Program ‘Minimum’ to find the smallest integer out of a set of integers Modified program ‘Minimum’ to find the smallest integer out of a set of integers Final program ‘Minimum’ to find the smallest integer out of a set of integers Phase wise cost of fixing an error Control flow graph of a 10 to 20 statement program [MYER04] Components of the software Documentation manuals Operating system manuals A typical example V shaped software development life cycle model Functional (Black Box) testing Five values for input ‘x’ of ‘Square’ program Selected values for input values x and y Valid input domain for the program ‘Addition’ Graphical representation of inputs Graphical representation of inputs Graphical representation of inputs Graphical representation of inputs Graphical representation of inputs Equivalence classes of input domain Steps for the generation of test cases Basic notations used in cause-effect graph Constraint symbols for any cause-effect graph Example of cause-effect graph with exclusive (constraint) and requires constraint Cause-effect graph of rental car problem Cause-effect graph of triangle classification problem Undirected and directed graphs Graphs with loop and parallel edges Types of graphs Regular graphs Cyclic graph Disconnected graphs Strongly connected graph Weakly connected graph

5 11 13 15 18 19 19 20 25 27 37 38 39 39 40 44 45 46 64 65 96 97 98 99 100 101 111 111 112 113 117 118 118 119

xii

List of Figures

3.9 3.10 3.11 3.12 3.13 3.14 3.15 3.16 3.17 3.18 3.19 3.20 3.21 3.22 3.23 3.24 3.25 3.26 3.27 3.28 3.29 3.30 3.31 3.32 3.33 3.34 3.35 4.1 4.2 4.3 4.4 4.5 4.6 4.7 4.8 4.9 6.1 6.2 6.3 6.4 6.5

6.6

Basic constructs of a program graph Program ‘Square’ and its program graph Program to find the largest among three numbers Program graph to find the largest number amongst three numbers as given in Figure 3.11 DD path graph and mapping table of program graph in Figure 3.10 DD path graph of the program to find the largest among three numbers Source code of determination of division of a student problem Program graph DD path graph of program to find division of a student Source code for classification of triangle problem Program graph of classification of triangle problem DD path graph of the program to classify a triangle Source code for determination of day of the week Program graph for determination of day of the week DD path graph for determination of day of the week Program graph Program graph with 3 connected components Program graphs and graph matrices Connection matrix for program graph shown in Figure 3.26(b) Various paths of program graph given in Figure 3.26(a) Program graph DD path graph for determination of division problem Graph matrix for determination of division problem Connection matrix for determination of division problem DD path graph for classification of triangle problem Graph matrix for classification of triangle problem Connection matrix for classification of triangle problem Source code with program graph Portion of a program Two slices for variable ‘c’ Example program Some slices of program in Figure 3.11 Slices of program for determination of division of a student Mutant1 (M1) of program to find the largest among three numbers Mutant2 (M2) of program to find the largest among three numbers Mutant3 (M3) of program to find the largest among three numbers Components of use case diagram Use case diagram of the URS Basic and alternative flows with pre-and post-conditions Basic and alternative flows for login use case (a) Login (b) Change password Basic and alternative flows for ‘maintain school’, ‘programme’, ‘scheme’, ‘paper’, or ‘student details’ use cases (a) Add details (b) Edit details (c) Delete details (d) View details Login form 6.7 Change password form

124 125 126 126 127 128 130 130 132 134 134 136 139 140 143 145 146 151 152 153 154 155 156 156 157 158 158 166 198 199 199 202 207 214 214 215 288 289 293 294

301 318 320

List of Figures

6.8 6.9 7.1 7.2 7.3 7.4 7.5 7.6 7.7 8.1 8.2 8.3 8.4 9.1 9.2 9.3 9.4 9.5 9.6 9.7 9.8 9.9 9.10 9.11 9.12 9.13 9.14 9.15 9.16 9.17 10.1 10.2 10.3 10.4 10.5 11.1 11.2 11.3 11.4 11.5 11.6 11.7 11.8 11.9

Maintain school details form Maintain program details form Steps of regression testing process Program for printing value of z Threshold by quadrant Alternate threshold by quadrant Threshold by diagonal quadrant Threshold based on high ‘Impact of Problem’ value Threshold based on high ‘probability of occurrence of problem’ value Levels of testing Unit under test with stubs and driver Coupling amongst units Integration approaches Class and its objects Class courier The categories around fast track courier A class hierarchy An example of an activity diagram Program to determine division of a student Activity diagram of function validate() Activity diagram of function calculate() Program to determine largest among three numbers Activity diagram for function validate() Activity diagram for function maximum() A typical state machine diagram Typical life cycle of a process State chart diagram for class stack State chart diagram of withdrawal from ATM Specification for the class stack Class ATM withdrawal as a function of Relationship between and μ Relationship between and ROC curve for (a) Model I (b) Model II (c) Model III (d) Model IV using LR method Comparison between actual and predicted values for maintenance effort Two-tier architecture of client-server application Three-tier architecture of web based applications Web application process Homepage of online shopping web application Sample online registration form Usability testing steps Participant characteristics for online shopping website Usability testing questionnaire Sample questions for online shopping website

xiii

322 324 337 338 343 344 345 345 346 369 370 371 372 390 391 392 393 397 399 400 400 403 403 404 405 406 407 410 413 414 431 431 432 442 446 454 454 455 456 460 464 464 468 468

xiv

List of Figures

11.10 11.11 11.12 11.13 12.1 12.2 12.3 12.4 12.5 II-1 II-2 III-1 III-2 III-3 III-4 III-5 III-6 III-7 III-8 III-9 III-10

Security threats Working of a firewall Post deployment testing questionnaire Typical post deployment testing procedure of a web application Program for determination of nature of roots of a quadratic equation Program graph of program given in Figure 12.1 A typical program Flow chart of various steps of genetic algorithm Program to divide two numbers Basic and alternative flows for maintain faculty details use case (a) Add a faculty (b) Edit a faculty (c) Delete a faculty (d) View a faculty Basic and alternative flows for maintain registration details use case (a) Add student registration details (b) Edit student registration details Validity checks for scheme form Test case with actual data values for the scheme form Validity checks for paper form Test case with actual data values for paper form Validity checks for student form Test case with actual data values for the student form Validity checks for faculty form Test case with actual data values for the faculty form Validity checks for maintain registration details Test case with actual data values for the student registration form

472 473 484 484 497 498 501 506 507 577 587 595 597 598 600 601 603 604 605 607 611

List of Tables

1.1 1.2 1.3 1.4 1.5 1.6 1.7 1.8 1.9 2.1 2.2 2.3 2.4 2.5 2.6 2.7 2.8 2.9 2.10 2.11 2.12 2.13 2.14 2.15 2.16 2.17 2.18 2.19 2.20 2.21 2.22 2.23 2.24 2.25

Inputs and outputs of the program ‘Minimum’ Some critical/typical situations of the program ‘Minimum’ Possible reasons of failures for all nine cases Reasons for observed output Results of the modified program ‘Minimum’ Persons and their roles during development and testing Test case template Test cases for function of Figure 1.9 Typical test cases where outputs are different Test cases for the ‘Square’ program Test cases for the program ‘Addition’ Boundary value test cases to find the largest among three numbers Boundary value test cases for the program determining the division of a student Boundary value test cases for triangle classification program Boundary value test cases for the program determining the day of the week Robustness test cases for two input values x and y Worst test cases for the program ‘Addition’ Robust worst test cases for the program ‘Addition’ Robust test cases for the program to find the largest among three numbers Worst case test cases for the program to find the largest among three numbers Robust test cases for the program determining the division of a student Worst case test cases for the program for determining the division of a student Robust test cases for the triangle classification program Worst case test cases for the triangle classification program Robust test cases for program for determining the day of the week Worst case test cases for the program determining day of the week Test cases for program ‘Square’ based on input domain Test cases for the program ‘Addition’ Test cases for program ‘Square’ based on output domain Test cases for program ‘Addition’ based on output domain Output domain test cases to find the largest among three numbers Input domain test case Output domain test cases Input domain test cases

6 7 8 9 12 16 21 25 26 38 40 41 41 42 43 44 45 47 48 48 51 52 55 55 59 59 63 64 65 65 66 67 68 70