Testing Oracle

July 1, 2019

---

Introduction :

Have you ever wondered what does the term "Oracle" imply? Well, Oracle has an interesting fact associated with it. If we go back to history and divert our attention to Greek mythology, we come to know that in those times, a person who could foresee or tell prophecies , were known as "Oracle". But how is this terminology even remotely associated with technology? Let us understand . It is believed that in Greek mythology, the prophecies were often misunderstood by people and the outcome of their actions was based on their own interpretation of the situation. Henceforth, the Greek mythology is used as a metaphor to the idea behind drawing a conclusion based on certain assumption. That is why the term "Oracle" is used in the context of "testing".

So technically we can say that a software testing oracle is a tool that helps to decide whether the program passed our test or not.

What is Oracle?

The word Oracle basically deals with the process of generating results, by providing inputs to the system under test and evaluating the outcome with the expected output. There are various approaches to Oracle or one can say types of Oracle:

• True Oracle- it helps to produce independent result of all expected outputs. It is however a little expensive to implement and time-consuming also. But no errors can go undetected. Use - Algorithm Validation.
• Heuristic Oracle- this approach does verify the values as well as checks for consistency between the remaining values. Less expensive, faster and easier to create and implement than true Oracle. Use- Algorithm Verification.
• Sampling Oracle- It selects a specific collection of inputs or results as sample. It is capable of selecting easily computed or recognized results. Use - Boundary Testing.
• Consistent Oracle- It performs a regression test, by comparing current results with the previous ones. It is the fastest method and can verify large amounts of data. Use- Regression Test.
• No Oracle- It does not check for the accuracy of the results produced. It just focuses on getting the output.

There have been many researches and analyses on the process of test oracle. We can confine our focus to few major concepts to develop a good understanding of the subject. For instance, there is a term used in the context of test oracle, random testing, it is an efficient strategy to produce results and requires test oracles to assure adequate evaluation of test results.

Features of Test Oracle :

• Completeness of information from oracle
• Accuracy of information from oracle
• Independence of oracle from the system under test - the oracle must not be dependant on the algorithms, libraries, or the operating system used.
• Pace of Predictions
• Time taken to execute oracle
• The extent to which the results could be used
• Oracle's consistency with respect to the changes made in the system under test

Oracle as Black-Box Testing:

Oracle's working can be broadly classified as-

• A 'generator' to provide the results that are expected of each test case.
• A 'comparator' to verify/compare expected and actual result.
• An 'evaluator' to ensure whether the verified results are close enough to the pass results.

Few demerits of test oracle can be, oracles only address a small subset of inputs and outputs. The tester may need to set values for a variable. We may generate a set of test inputs based on the previous observation of a custom model, previous release of a program etc.

Conclusion :

What makes testing in Oracle in an automated framework is that , one is free to write their own code. We can create automated tests to check the output of the program, so that we are able to analyse the deviations in the expected and actual output. Oracle can be thought of as a way to analyse and then devise a strategy to proceed to the next level. So far, based on our understanding about oracle, we have come to know that the test results vary from time to time. It may be the result of the type of framework chosen for conducting the tests or our approach to the testing process. Therefore we can assume that an efficient test oracle is the one that performs effectively irrespective of the platforms, compilers, code, algorithm being used. Thus an Oracle is a 'way' that establishes a connection between the process chosen for testing and the System Under Test.