General Principles of Software Validation; Final Guidance for Industry and FDA Staff

• Date: October 24, 2007
• Source: www.fda.gov

Validation of Automated Process Equipment & Quality System Software

SECTION 6. VALIDATION OF AUTOMATED
PROCESS EQUIPMENT AND QUALITY SYSTEM
SOFTWARE 

The Quality System regulation requires that "when computers or automated data processing systems are used as part of production or the quality system, the [device] manufacturer shall validate computer software for its intended use according to an established protocol." (See 21 CFR §820.70(i)). This has been a regulatory requirement of FDA's medical device Good Manufacturing Practice (GMP) regulations since 1978.

In addition to the above validation requirement, computer systems that implement part of a device manufacturer's production processes or quality system (or that are used to create and maintain records required by any other FDA regulation) are subject to the Electronic Records; Electronic Signatures regulation. (See 21 CFR Part 11.) This regulation establishes additional security, data integrity, and validation requirements when records are created or maintained electronically. These additional Part 11 requirements should be carefully considered and included in system requirements and software requirements for any automated record `keeping systems. System validation and software validation should demonstrate that all Part 11 requirements have been met.

Computers and automated equipment are used extensively throughout all aspects of medical device design, laboratory testing and analysis, product inspection and acceptance, production and process control, environmental controls, packaging, labeling, traceability, document control, complaint management, and many other aspects of the quality system. Increasingly, automated plant floor operations can involve extensive use of embedded systems in:

• programmable logic controllers;
• digital function controllers;
• statistical process control;
• supervisory control and data acquisition;
• robotics;
• human-machine interfaces;
• input/output devices; and
• computer operating systems.

Software tools are frequently used to design, build, and test the software that goes into an automated medical device. Many other commercial software applications, such as word processors, spreadsheets, databases, and flowcharting software are used to implement the quality system. All of these applications are subject to the requirement for software validation, but the validation approach used for each application can vary widely.

Whether production or quality system software is developed in-house by the device manufacturer, developed by a contractor, or purchased off-the-shelf, it should be developed using the basic principles outlined elsewhere in this guidance. The device manufacturer has latitude and flexibility in defining how validation of that software will be accomplished, but validation should be a key consideration in deciding how and by whom the software will be developed or from whom it will be purchased. The software developer defines a life cycle model. Validation is typically supported by:

• verifications of the outputs from each stage of that software development life cycle; and
• checking for proper operation of the finished software in the device manufacturer's intended use environment.

6.1. HOW MUCH VALIDATION EVIDENCE IS NEEDED?

The level of validation effort should be commensurate with the risk posed by the automated operation. In addition to risk other factors, such as the complexity of the process software and the degree to which the device manufacturer is dependent upon that automated process to produce a safe and effective device, determine the nature and extent of testing needed as part of the validation effort. Documented requirements and risk analysis of the automated process help to define the scope of the evidence needed to show that the software is validated for its intended use. For example, an automated milling machine may require very little testing if the device manufacturer can show that the output of the operation is subsequently fully verified against the specification before release. On the other hand, extensive testing may be needed for:

• a plant-wide electronic record and electronic signature system;
• an automated controller for a sterilization cycle; or
• automated test equipment used for inspection and acceptance of finished circuit boards in a life-sustaining / life-supporting device.

Numerous commercial software applications may be used as part of the quality system (e.g., a spreadsheet or statistical package used for quality system calculations, a graphics package used for trend analysis, or a commercial database used for recording device history records or for complaint management). The extent of validation evidence needed for such software depends on the device manufacturer's documented intended use of that software. For example, a device manufacturer who chooses not to use all the vendor-supplied capabilities of the software only needs to validate those functions that will be used and for which the device manufacturer is dependent upon the software results as part of production or the quality system. However, high risk applications should not be running in the same operating environment with non-validated software functions, even if those software functions are not used. Risk mitigation techniques such as memory partitioning or other approaches to resource protection may need to be considered when high risk applications and lower risk applications are to be used in the same operating environment. When software is upgraded or any changes are made to the software, the device manufacturer should consider how those changes may impact the "used portions" of the software and must reconfirm the validation of those portions of the software that are used. (See 21 CFR §820.70(i).)

6.2. DEFINED USER REQUIREMENTS

A very important key to software validation is a documented user requirements specification that defines:

• the "intended use" of the software or automated equipment; and
• the extent to which the device manufacturer is dependent upon that software or equipment for production of a quality medical device.

The device manufacturer (user) needs to define the expected operating environment including any required hardware and software configurations, software versions, utilities, etc. The user also needs to:

• document requirements for system performance, quality, error handling, startup, shutdown, security, etc.;
• identify any safety related functions or features, such as sensors, alarms, interlocks, logical processing steps, or command sequences; and
• define objective criteria for determining acceptable performance.

The validation must be conducted in accordance with a documented protocol, and the validation results must also be documented. (See 21 CFR §820.70(i).) Test cases should be documented that will exercise the system to challenge its performance against the pre-determined criteria, especially for its most critical parameters. Test cases should address error and alarm conditions, startup, shutdown, all applicable user functions and operator controls, potential operator errors, maximum and minimum ranges of allowed values, and stress conditions applicable to the intended use of the equipment. The test cases should be executed and the results should be recorded and evaluated to determine whether the results support a conclusion that the software is validated for its intended use.

A device manufacturer may conduct a validation using their own personnel or may depend on a third party such as the equipment/software vendor or a consultant. In any case, the device manufacturer retains the ultimate responsibility for ensuring that the production and quality system software:

• is validated according to a written procedure for the particular intended use; and
• will perform as intended in the chosen application.

The device manufacturer should have documentation including:

• defined user requirements;
• validation protocol used;
• acceptance criteria;
• test cases and results; and
• a validation summary

that objectively confirms that the software is validated for its intended use.

6.3. VALIDATION OF OFF-THE-SHELF SOFTWARE AND AUTOMATED EQUIPMENT

Most of the automated equipment and systems used by device manufacturers are supplied by third-party vendors and are purchased off-the-shelf (OTS). The device manufacturer is responsible for ensuring that the product development methodologies used by the OTS software developer are appropriate and sufficient for the device manufacturer's intended use of that OTS software. For OTS software and equipment, the device manufacturer may or may not have access to the vendor's software validation documentation. If the vendor can provide information about their system requirements, software requirements, validation process, and the results of their validation, the medical device manufacturer can use that information as a beginning point for their required validation documentation. The vendor's life cycle documentation, such as testing protocols and results, source code, design specification, and requirements specification, can be useful in establishing that the software has been validated. However, such documentation is frequently not available from commercial equipment vendors, or the vendor may refuse to share their proprietary information.

Where possible and depending upon the device risk involved, the device manufacturer should consider auditing the vendor's design and development methodologies used in the construction of the OTS software and should assess the development and validation documentation generated for the OTS software. Such audits can be conducted by the device manufacturer or by a qualified third party. The audit should demonstrate that the vendor's procedures for and results of the verification and validation activities performed the OTS software are appropriate and sufficient for the safety and effectiveness requirements of the medical device to be produced using that software.

Some vendors who are not accustomed to operating in a regulated environment may not have a documented life cycle process that can support the device manufacturer's validation requirement. Other vendors may not permit an audit. Where necessary validation information is not available from the vendor, the device manufacturer will need to perform sufficient system level "black box" testing to establish that the software meets their "user needs and intended uses." For many applications black box testing alone is not sufficient. Depending upon the risk of the device produced, the role of the OTS software in the process, the ability to audit the vendor, and the sufficiency of vendor-supplied information, the use of OTS software or equipment may or may not be appropriate, especially if there are suitable alternatives available. The device manufacturer should also consider the implications (if any) for continued maintenance and support of the OTS software should the vendor terminate their support.

For some off-the-shelf software development tools, such as software compilers, linkers, editors, and operating systems, exhaustive black-box testing by the device manufacturer may be impractical. Without such testing - a key element of the validation effort - it may not be possible to validate these software tools. However, their proper operation may be satisfactorily inferred by other means. For example, compilers are frequently certified by independent third-party testing, and commercial software products may have "bug lists", system requirements and other operational information available from the vendor that can be compared to the device manufacturer's intended use to help focus the "black-box" testing effort. Off-the-shelf operating systems need not be validated as a separate program. However, system-level validation testing of the application software should address all the operating system services used, including maximum loading conditions, file operations, handling of system error conditions, and memory constraints that may be applicable to the intended use of the application program.

For more detailed information, see the production and process software references in Appendix A.

APPENDIX A - REFERENCES

Food and Drug Administration References

Design Control Guidance for Medical Device Manufacturers, Center for Devices and Radiological Health, Food and Drug Administration, March 1997.

Do It by Design, An Introduction to Human Factors in Medical Devices, Center for Devices and Radiological Health, Food and Drug Administration, March 1997.

Electronic Records; Electronic Signatures Final Rule, 62 Federal Register 13430 (March 20, 1997).

Glossary of Computerized System and Software Development Terminology, Division of Field Investigations, Office of Regional Operations, Office of Regulatory Affairs, Food and Drug Administration, August 1995.

Guidance for the Content of Pre-market Submissions for Software Contained in Medical Devices, Office of Device Evaluation, Center for Devices and Radiological Health, Food and Drug Administration, May 1998.

Guidance for Industry, FDA Reviewers and Compliance on Off-the-Shelf Software Use in Medical Devices, Office of Device Evaluation, Center for Devices and Radiological Health, Food and Drug Administration, September 1999.

Guideline on General Principles of Process Validation, Center for Drugs and Biologics, & Center For Devices and Radiological Health, Food and Drug Administration, May 1987.

Medical Devices; Current Good Manufacturing Practice (CGMP) Final Rule; Quality System Regulation, 61 Federal Register 52602 (October 7, 1996).

Reviewer Guidance for a Pre-Market Notification Submission for Blood Establishment Computer Software, Center for Biologics Evaluation and Research, Food and Drug Administration, January 1997

Student Manual 1, Course INV545, Computer System Validation, Division of Human Resource Development, Office of Regulatory Affairs, Food and Drug Administration, 1997.

Technical Report, Software Development Activities, Division of Field Investigations, Office of Regional Operations, Office of Regulatory Affairs, Food and Drug Administration, July 1987.

Other Government References

W. Richards Adrion, Martha A. Branstad, John C. Cherniavsky. NBS Special Publication 500-75, Validation, Verification, and Testing of Computer Software, Center for Programming Science and Technology, Institute for Computer Sciences and Technology, National Bureau of Standards, U.S. Department of Commerce, February 1981.

Martha A. Branstad, John C Cherniavsky, W. Richards Adrion, NBS Special Publication 500-56, Validation, Verification, and Testing for the Individual Programmer, Center for Programming Science and Technology, Institute for Computer Sciences and Technology, National Bureau of Standards, U.S. Department of Commerce, February 1980.

J.L. Bryant, N.P. Wilburn, Handbook of Software Quality Assurance Techniques Applicable to the Nuclear Industry, NUREG/CR-4640, U.S. Nuclear Regulatory Commission, 1987.

H. Hecht, et.al., Verification and Validation Guidelines for High Integrity Systems. NUREG/CR-6293. Prepared for U.S. Nuclear Regulatory Commission, 1995.

H. Hecht, et.al., Review Guidelines on Software Languages for Use in Nuclear Power Plant Safety Systems, Final Report. NUREG/CR-6463. Prepared for U.S. Nuclear Regulatory Commission, 1996.

J.D. Lawrence, W.L. Persons, Survey of Industry Methods for Producing Highly Reliable Software, NUREG/CR-6278, U.S. Nuclear Regulatory Commission, 1994.

J.D. Lawrence, G.G. Preckshot, Design Factors for Safety-Critical Software, NUREG/CR-6294, U.S. Nuclear Regulatory Commission, 1994.

Patricia B. Powell, Editor. NBS Special Publication 500-98, Planning for Software Validation, Verification, and Testing, Center for Programming Science and Technology, Institute for Computer Sciences and Technology, National Bureau of Standards, U.S. Department of Commerce, November 1982.

Patricia B. Powell, Editor. NBS Special Publication 500-93, Software Validation, Verification, and Testing Technique and Tool Reference Guide, Center for Programming Science and Technology, Institute for Computer Sciences and Technology, National Bureau of Standards, U.S. Department of Commerce, September 1982.

Delores R. Wallace, Roger U. Fujii, NIST Special Publication 500-165, Software Verification and Validation: Its Role in Computer Assurance and Its Relationship with Software Project Management Standards, National Computer Systems Laboratory, National Institute of Standards and Technology, U.S. Department of Commerce, September 1995.

Delores R. Wallace, Laura M. Ippolito, D. Richard Kuhn, NIST Special Publication 500-204, High Integrity Software, Standards and Guidelines, Computer Systems Laboratory, National Institute of Standards and Technology, U.S. Department of Commerce, September 1992.

Delores R. Wallace, et.al. NIST Special Publication 500-234, Reference Information for the Software Verification and Validation Process. Computer Systems Laboratory, National Institute of Standards and Technology, U.S. Department of Commerce, March 1996.

Delores R. Wallace, Editor. NIST Special Publication 500-235, Structured Testing: A Testing Methodology Using the Cyclomatic Complexity Metric. Computer Systems Laboratory, National Institute of Standards and Technology, U.S. Department of Commerce, August 1996.

International and National Consensus Standards

ANSI / ANS-10.4-1987, Guidelines for the Verification and Validation of Scientific and Engineering Computer Programs for the Nuclear Industry, American National Standards Institute, 1987.

ANSI / ASQC Standard D1160-1995, Formal Design Reviews, American Society for Quality Control, 1995.

ANSI / UL 1998:1998, Standard for Safety for Software in Programmable Components, Underwriters Laboratories, Inc., 1998.

AS 3563.1-1991, Software Quality Management System, Part 1: Requirements. Published by Standards Australia [Standards Association of Australia], 1 The Crescent, Homebush, NSW 2140.

AS 3563.2-1991, Software Quality Management System, Part 2: Implementation Guide. Published by Standards Australia [Standards Association of Australia], 1 The Crescent, Homebush, NSW 2140.

IEC 60601-1-4:1996, Medical electrical equipment, Part 1: General requirements for safety, 4. Collateral Standard: Programmable electrical medical systems. International Electrotechnical Commission, 1996.

IEC 61506:1997, Industrial process measurement and control - Documentation of application software. International Electrotechnical Commission, 1997.

IEC 61508:1998, Functional safety of electrical/electronic/programmable electronic safety-related systems. International Electrotechnical Commission, 1998.

IEEE Std 1012-1986, Software Verification and Validation Plans, Institute for Electrical and Electronics Engineers, 1986.

IEEE Standards Collection, Software Engineering, Institute of Electrical and Electronics Engineers, Inc., 1994. ISBN 1-55937-442-X.

ISO 8402:1994, Quality management and quality assurance - Vocabulary. International Organization for Standardization, 1994.

ISO 9000-3:1997, Quality management and quality assurance standards - Part 3: Guidelines for the application of ISO 9001:1994 to the development, supply, installation and maintenance of computer software. International Organization for Standardization, 1997.

ISO 9001:1994, Quality systems - Model for quality assurance in design, development, production, installation, and servicing. International Organization for Standardization, 1994.

ISO 13485:1996, Quality systems - Medical devices - Particular requirements for the application of ISO 9001. International Organization for Standardization, 1996.

ISO/IEC 12119:1994, Information technology - Software packages - Quality requirements and testing, Joint Technical Committee ISO/IEC JTC 1, International Organization for Standardization and International Electrotechnical Commission, 1994.

ISO/IEC 12207:1995, Information technology - Software life cycle processes, Joint Technical Committee ISO/IEC JTC 1, Subcommittee SC 7, International Organization for Standardization and International Electrotechnical Commission, 1995.

ISO/IEC 14598:1999, Information technology - Software product evaluation, Joint Technical Committee ISO/IEC JTC 1, Subcommittee SC 7, International Organization for Standardization and International Electrotechnical Commission, 1999.

ISO 14971-1:1998, Medical Devices - Risk Management - Part 1: Application of Risk Analysis. International Organization for Standardization, 1998.

Software Considerations in Airborne Systems and Equipment Certification. Special Committee 167 of RTCA. RTCA Inc., Washington, D.C. Tel: 202-833-9339. Document No. RTCA/DO-178B, December 1992.

Production Process Software References

The Application of the Principles of GLP to Computerized Systems, Environmental Monograph #116, Organization for Economic Cooperation and Development (OECD), 1995.

George J. Grigonis, Jr., Edward J. Subak, Jr., and Michael Wyrick, "Validation Key Practices for Computer Systems Used in Regulated Operations," Pharmaceutical Technology, June 1997.

Guide to Inspection of Computerized Systems in Drug Processing, Reference Materials and Training Aids for Investigators, Division of Drug Quality Compliance, Associate Director for Compliance, Office of Drugs, National Center for Drugs and Biologics, & Division of Field Investigations, Associate Director for Field Support, Executive Director of Regional Operations, Food and Drug Administration, February 1983.

Daniel P. Olivier, "Validating Process Software", FDA Investigator Course: Medical Device Process Validation, Food and Drug Administration.

GAMP Guide For Validation of Automated Systems in Pharmaceutical Manufacture,Version V3.0, Good Automated Manufacturing Practice (GAMP) Forum, March 1998:   Volume 1, Part 1: User Guide     Part 2: Supplier Guide   Volume 2: Best Practice for User and Suppliers.

Technical Report No. 18, Validation of Computer-Related Systems. PDA Committee on Validation of Computer-Related Systems. PDA Journal of Pharmaceutical Science and Technology, Volume 49, Number 1, January-February 1995 Supplement.

Validation Compliance Annual 1995, International Validation Forum, Inc.

General Software Quality References

Boris Beizer, Black Box Testing, Techniques for Functional Testing of Software and Systems, John Wiley & Sons, 1995. ISBN 0-471-12094-4.

Boris Beizer, Software System Testing and Quality Assurance, International Thomson Computer Press, 1996. ISBN 1-85032-821-8.

Boris Beizer, Software Testing Techniques, Second Edition, Van Nostrand Reinhold, 1990. ISBN 0-442-20672-0.

Richard Bender, Writing Testable Requirements, Version 1.0, Bender & Associates, Inc., Larkspur, CA 94777, 1996.

Frederick P. Brooks, Jr., The Mythical Man-Month, Essays on Software Engineering, Addison-Wesley Longman, Anniversary Edition, 1995. ISBN 0-201-83595-9.

Silvana Castano, et.al., Database Security, ACM Press, Addison-Wesley Publishing Company, 1995. ISBN 0-201-59375-0.

Computerized Data Systems for Nonclinical Safety Assessment, Current Concepts and Quality Assurance, Drug Information Association, Maple Glen, PA, September 1988.

M. S. Deutsch, Software Verification and Validation, Realistic Project Approaches, Prentice Hall, 1982.

Robert H. Dunn and Richard S. Ullman, TQM for Computer Software, Second Edition, McGraw-Hill, Inc., 1994. ISBN 0-07-018314-7.

Elfriede Dustin, Jeff Rashka, and John Paul, Automated Software Testing - Introduction, Management and Performance, Addison Wesley Longman, Inc., 1999. ISBN 0-201-43287-0.

Robert G. Ebenau and Susan H. Strauss, Software Inspection Process, McGraw-Hill, 1994. ISBN 0-07-062166-7.

Richard E. Fairley, Software Engineering Concepts, McGraw-Hill Publishing Company, 1985. ISBN 0-07-019902-7.

Michael A. Friedman and Jeffrey M. Voas, Software Assessment - Reliability, Safety, Testability, Wiley-Interscience, John Wiley & Sons Inc., 1995. ISBN 0-471-01009-X.

Tom Gilb, Dorothy Graham, Software Inspection, Addison-Wesley Publishing Company, 1993. ISBN 0-201-63181-4.

Robert B. Grady, Practical Software Metrics for Project Management and Process Improvement, PTR Prentice-Hall Inc., 1992. ISBN 0-13-720384-5.

Les Hatton, Safer C: Developing Software for High-integrity and Safety-critical Systems, McGraw-Hill Book Company, 1994. ISBN 0-07-707640-0.

Janis V. Halvorsen, A Software Requirements Specification Document Model for the Medical Device Industry, Proceedings IEEE SOUTHEASTCON '93, Banking on Technology, April 4th -7th, 1993, Charlotte, North Carolina.

Debra S. Herrmann, Software Safety and Reliability: Techniques, Approaches and Standards of Key Industrial Sectors, IEEE Computer Society, 1999. ISBN 0-7695-0299-7.

Bill Hetzel, The Complete Guide to Software Testing, Second Edition, A Wiley-QED Publication, John Wiley & Sons, Inc., 1988. ISBN 0-471-56567-9.

Watts S. Humphrey, A Discipline for Software Engineering. Addison-Wesley Longman, 1995. ISBN 0-201-54610-8.

Watts S. Humphrey, Managing the Software Process, Addison-Wesley Publishing Company, 1989. ISBN 0-201-18095-2.

Capers Jones, Software Quality, Analysis and Guidelines for Success, International Thomson Computer Press, 1997. ISBN 1-85032-867-6.

J.M. Juran, Frank M. Gryna, Quality Planning and Analysis, Third Edition, , McGraw-Hill, 1993. ISBN 0-07-033183-9.

Stephen H. Kan, Metrics and Models in Software Quality Engineering, Addison-Wesley Publishing Company, 1995. ISBN 0-201-63339-6.

Cem Kaner, Jack Falk, Hung Quoc Nguyen, Testing Computer Software, Second Edition, Vsn Nostrand Reinhold, 1993. ISBN 0-442-01361-2.

Craig Kaplan, Ralph Clark, Victor Tang, Secrets of Software Quality, 40 Innovations from IBM, McGraw-Hill, 1995. ISBN 0-07-911795-3.

Edward Kit, Software Testing in the Real World, Addison-Wesley Longman, 1995. ISBN 0-201-87756-2.

Alan Kusinitz, "Software Validation", Current Issues in Medical Device Quality Systems, Association for the Advancement of Medical Instrumentation, 1997. ISBN 1-57020-075-0.

Nancy G. Leveson, Safeware, System Safety and Computers, Addison-Wesley Publishing Company, 1995. ISBN 0-201-11972-2.

Michael R. Lyu, Editor, Handbook of Software Reliability Engineering, IEEE Computer Society Press, McGraw-Hill, 1996. ISBN 0-07-039400-8.

Steven R. Mallory, Software Development and Quality Assurance for the Healthcare Manufacturing Industries, Interpharm Press,Inc., 1994. ISBN 0-935184-58-9.

Brian Marick, The Craft of Software Testing, Prentice Hall PTR, 1995. ISBN 0-13-177411-5.

Steve McConnell, Rapid Development, Microsoft Press, 1996. ISBN 1-55615-900-5.

Glenford J. Myers, The Art of Software Testing, John Wiley & Sons, 1979. ISBN 0-471-04328-1.

Peter G. Neumann, Computer Related Risks, ACM Press/Addison-Wesley Publishing Co., 1995. ISBN 0-201-55805-X.

Daniel Olivier, Conducting Software Audits, Auditing Software for Conformance to FDA Requirements, Computer Application Specialists, San Diego, CA, 1994.

William Perry, Effective Methods for Software Testing, John Wiley & Sons, Inc. 1995. ISBN 0-471-06097-6.

William E. Perry, Randall W. Rice, Surviving the Top Ten Challenges of Software Testing, Dorset House Publishing, 1997. ISBN 0-932633-38-2.

Roger S. Pressman, Software Engineering, A Practitioner's Approach, Third Edition, McGraw-Hill Inc., 1992. ISBN 0-07-050814-3.

Roger S. Pressman, A Manager's Guide to Software Engineering, McGraw-Hill Inc., 1993 ISBN 0-07-050820-8.

A. P. Sage, J. D. Palmer, Software Systems Engineering, John Wiley & Sons, 1990.

Joc Sanders, Eugene Curran, Software Quality, Addison-Wesley Publishing Co., 1994. ISBN 0-201-63198-9.

Ken Shumate, Marilyn Keller, Software Specification and Design, A Disciplined Approach for Real-Time Systems, John Wiley & Sons, 1992. ISBN 0-471-53296-7.

Dennis D. Smith, Designing Maintainable Software, Springer-Verlag, 1999. ISBN 0-387-98783-5.

Ian Sommerville, Software Engineering, Third Edition, Addison Wesley Publishing Co., 1989. ISBN 0-201-17568-1.

Karl E. Wiegers, Creating a Software Engineering Culture, Dorset House Publishing, 1996. ISBN 0-932633-33-1.

Karl E. Wiegers, Software Inspection, Improving Quality with Software Inspections, Software Development, April 1995, pages 55-64.

Karl E. Wiegers, Software Requirements, Microsoft Press, 1999. ISBN 0-7356-0631-5.