What is Safety Testing?

March 11, 2020

Testing technology and software systems is crucial for successful product development. Inappropriate or late tests, underestimated testing efforts, or wrong test technology choices have often led projects to crisis as well as failure. This crisis is usually due to negligence of the imbalance between constructive software engineering and analytic quality assurance. The most appropriate way of handling such a situation is to perform safety testing during the process of software or program development. By doing so, the engineer can avoid future failures and frustration, as well as keep a track of errors and hazards in the system.

What is Safety Testing in Software Systems?

Safety testing in software systems aims at optimizing system safety in the design, development, use, and maintenance of software systems and their integration with safety-critical hardware systems in a production environment. It verifies and validates all the testable Software Safety Significant Requirements, in accordance with their assigned LOR, through Software Developmental and System Integration Testing, which is executed to verify the implementation of mitigations, uncovers safety anomalies, and identify and/or verify the resolution of hazards. Safety Testing ensures that the functioning software does not generate hazards and that the monitoring systems must perform flawlessly. Example: Back-up computer should start automatically when primary computer fails.

The Importance of Safety Testing:

Testing products and their process of development is a necessary requirement. It allows developer the opportunity to check for faulty features or identify undetected errors or bugs. Similarly, Safety Testing in software systems too is immensely useful for various reasons. It provides a proper analysis of system and its design, properties, data and other features that may or may not cause a safety crisis or failure of the software. Mentioned here are some of the most common failure mechanisms that should be evaluated during the safety analysis process:

Software is failing or unable to perform the required function, i.e., either the function is never executed or there is no answer produced.
Software is executing the wrong function or a function not required. For example, getting wrong answers, issuing the wrong control instructions, or doing the right action but under inappropriate conditions.
Failure of software to recognize a safety-critical function and to initiate the appropriate fault tolerant response.
There is a timing or sequence problem in the software and it is failing to ensure that two things happen at the same or different times as well as in a particular order.
There are specification errors which is the major cause of system or software failures or errors. These include improperly stated, omitted, misunderstood or incorrect specifications and requirements.
Design and coding errors: Usually introduced by the programmer, this error can arise from specification errors and are direct result of poor structured programming techniques. These errors can consist of incomplete interfaces, timing errors, incorrect interfaces, incorrect algorithms, logic errors, lack of self-tests, overload faults, endless loops, and syntax errors.
Hardware or computer induced errors though are not as common as others, but they do exist. Possibilities include random power supply transients, hardware failure modes that are not identified and/or corrected by the software to revert the system to a safe state among others.
Documentation Errors: Poor or incomplete documentation can be a major cause of error through miscommunication, which could further lead to software errors mentioned above.
Debugging or Software Change Induced Errors: These errors can be traced back to programming and coding, poor structured techniques, poor documentation as well as poor specification requirements. Software change induced errors assist in validating the necessity for software configuration.

Software Safety Planning and Management:

The single most important step that precedes all other phases of safety program is software system safety planning and management. Safety planning should be used to impose provisions for accommodating safety well before each of the software life cycle phases, like requirements, design, coding, and testing starts in the cycle.

By performing a detailed planning before the execution of test cycles, a programmer can ensure that critical program interfaces and support are identified and formal lines of communication are established between disciplines and among engineering functions. Also, there should be planned provisions that can assure best results. To assist a programmer in executing these aspects, here is a check list of things that needs to be taken care of while planning a software system safety testing.

Software safety organization is properly chartered and a safety team is commissioned at the beginning of the tests or programming.
Acceptable levels of software risk are defined consistently with risks defined for the entire system.
Interfaces between software and the rest of the system’s functions are clearly delineated and understood.
Software application concepts are examined to identify hazards/risks within safety critical software functions.
Requirements and specifications are examined for hazards (e.g. identification of hazardous commands, processing limits, sequence of events, timing constraints, failure tolerance, etc.)
Proper incorporation of design and implementation into the software safety requirements.
Test plans and procedures can achieve the intent of the software safety verification requirements.
Satisfactory results of software safety verification efforts.

Guidance for Safety-critical Software Testing:-

Testing safety-critical software demonstrates that the software complies with the requirements stated by the programmer and shows the errors which could result in a hazard. This type of testing ensure that the monitoring system performs flawlessly and there are no errors left undetected. However, to achieve this it is necessary to follow the below mentioned guidelines while creating software test cases:

The tests cases should show compliance with all software requirements, whether they are of high or low level.
The correct functionality under normal and off normal conditions much be verified before the creation of test cases.
Test cases created by the programmer should ensure that the software requirements are tested by Software Requirements Coverage Analysis.
Structural Coverage Analysis should be used to ensure the software structure are exercised.

Safety Assessment Report:-

The Safety Assessment Report contains the details of the safety analysis performed on a system, like the test conducted, the analysis performed and their results as well as the compliance assessment. The main purpose of this report is to provide management with an overall assessment of the risk associated with the system, including the software executing within the system context of an operational environment. Following are some of the information that a safety assessment report provides:

The safety criteria and the methodology used to classify and rank software significant hazard or error, including any assumptions made from which the criteria and methodologies were derived.
It should contain the results of the analysis and tests performed.
Should include the errors and hazards that pose a risk and its assessment.
To reduce the safety risk the list of significant hazards and the specific safety recommendations or precautions should be provided.
It should also include a discussion of the engineering decisions made that affect the residual risk at a system level.

Conclusion:-

Safety Testing in Software Systems ensures that contributing factors and resultant hazards associated with the system are identified and eliminated. In complex systems where there are many interactions involved, the safety–critical functionality too is identified and thoroughly analysed. Furthermore, it validates that the system and its design, data, development, use and maintenance is of exceptional quality; no hazards, errors or bug are left undetected and the results match the requirements of the tester. Hence, Safety Testing provides the testers and programmers a golden opportunity to test their products for any risk before they are delivered and also ensures cost effective development.