Software Testing

Software Testing#

Generally, software testing is “the process of executing a program with the intent of finding errors” (Myers, 1979) or, in other terms, “the process of […] evaluating a system […] to verify that it satisfies specific requirements.” (IEEE) There are decades worth of research and discussions on how to test software and on how to formalize testing.

Methods for software testing can, for example, be categorized by their scope. Then, categories range from unit testing, dealing with small portions of code such as a function or a method, to integration testing, dealing with the interaction of multiple units, to larger scopes validating the interplay of modules or even different computer systems.

Software testing can also be categorized by whether we have a test oracle, i.e., a means of telling what the correct output of a software is.

Here, we’ll first focus on unit testing with a test oracle and learn techniques to specify requirements for the software under test. Later, we’ll also see how to still write meaningful tests if we don not have a test oracle.

If we write tests before or while we implement a software, we call this process test driven development.

Unit testing with a test oracle#

Unit testing takes (relatively) small portions of code (such as a function or a method) and validates it with one or multiple tests. For the case where we have a test oracle, i.e., if there are inputs to our software for which we know the correct or expected outputs, we can describe a unit test as follows: For given inputs, called the test data, make sure that the test result matches the expected result.

Simple example: Adding three numbers#

Suppose we have a function meant to add three numbers:

def sum_three_numbers(a, b, c):
    return a + b + c

We could now make sure (or assert) that the sum of the numbers 1, -1, and 2 evaluates to 2:

assert 2 == sum_three_numbers(1, -1, 2)

Here, sum_three_numbers() is our software under test, 1, -1, 2 are our test data, the result of sum_three_numbers(1, -1, 2) is our test result, and 2 is our expected result.

If the result of the expression after assert does not evaluate to True, it will raise an AsssertionError.

Example of a failing test: Multiplying numbers#

Suppose we have a function meant to multiply two numbers. And suppose this function has a bug:

def multiply(a, b):
    """This function is wrong."""
    return a / b

Let’s start with testing a special case:

assert 0 == multiply(0, 1)

And add another test:

assert 21 == multiply(3, 7)

---------------------------------------------------------------------------
AssertionError                            Traceback (most recent call last)
Cell In[5], line 1
----> 1 assert 21 == multiply(3, 7)

AssertionError: 

Now we get an assertion error, because the result of multiply_wrong(3, 7) is not 21. Let’s fix the function and run the tests again:

def multiply(a, b):
    """This implementation should be correct."""
    return a * b

assert 0 == multiply(0, 1)
assert 21 == multiply(3, 7)

Now the function returns the expected result for all our tests.

Unit testing without a test oracle#

Let’s again consider the sum_three_numbers() function from above as our software under test, but now suppose we don’t have an oracle, i.e. we don’ know (or we don’t use the knowledge about) the result for given test data. But we can still use our knowledge of basic rules of algebra and validate that our software satisfies those.

For example, we know that \(a + b + c = b + c + a\). So let’s validate that our function sum_three_numbers() satisfies this relation:

assert sum_three_numbers(1, 2, 3) == sum_three_numbers(2, 3, 1)
assert sum_three_numbers(-11, 99, 137.5) == sum_three_numbers(99, 137.5, -11)

Or we could check that \(-(a + b + c) = - a - b - b\):

assert (- sum_three_numbers(7, 11, -6)) == sum_three_numbers(-7, -11, 6)

We could also check that \(x\cdot (a + b + c) = x\cdot a + x\cdot b + x\cdot c\):

a, b, c = 2, 3, 17
x = 77

assert x * sum_three_numbers(a, b, c) == sum_three_numbers(x * a, x * b, x * c)

Note that in this section, we never validated the value of the outputs agains known true results, but only validated that the changes of the output for a given change to the test data is correct. (This technique is also called metamorphic testing.)

Automation: Test cases#

There are whole software packages (see, e.g., https://docs.pytest.org) for automating and tracking software tests and there are platforms for even further automating the test procedure (see, e.g., https://www.jenkins.io/).

The essence of software testing, however, can be implemented without these helpers by simply writig code which is complementary to a part of code which we intend to test. The units of this complemetary code are called test cases. If our software under test is a function, then we write a second function which reflects one or multiple test cases which we want to apply to our original function.

Example: We want to find the family name of a person in a string which can have different formats (e.g. “John Doe”, or “Curie, Marie”, or even “LF Richardson”). It really easy to come up with the expected results: “Doe”, “Curie”, “Richardson”. So let’s create a function meant to extract the familiy name and another function validating the first for these three test cases:

def find_family_name(name_string):
    return name_string.split(" ")[1]


def test_find_family_name():
    assert find_family_name("John Doe") == "Doe"
    assert find_family_name("Curie, Marie") == "Curie"
    assert find_family_name("Lewis Fry Richardson") == "Richardson"


test_find_family_name()

---------------------------------------------------------------------------
AssertionError                            Traceback (most recent call last)
Cell In[11], line 11
      7     assert find_family_name("Curie, Marie") == "Curie"
      8     assert find_family_name("Lewis Fry Richardson") == "Richardson"
---> 11 test_find_family_name()

Cell In[11], line 7, in test_find_family_name()
      5 def test_find_family_name():
      6     assert find_family_name("John Doe") == "Doe"
----> 7     assert find_family_name("Curie, Marie") == "Curie"
      8     assert find_family_name("Lewis Fry Richardson") == "Richardson"

AssertionError: 

So our test revealed that our software under test did not work for the input "Curie, Marie".

Test-driven development#

With our tests cases for the function find_family_name() defined, we can now go ahead and improve the function, re-evaluating the test as often as we like. Once all test cases succeed, we can be confident, that our software under test satisfies the requirements specified in our test cases.

This does not necessarily generalize to all conceivable inputs of the function, but it lends confidence at least for those cases or classes of cases we ran our tests for.

The process of defining test cases and then implementing the code to satisfy the test cases is called test driven development. So let’s fix our name-finding function by first defining tests and then implementing the solutions:

Simple names: Given Name and Family Name#

def find_family_name(name_string):
    return name_string.split(" ")[1]

def test_find_family_name_simple_twoparts():
    assert find_family_name("John Doe") == "Doe"
    assert find_family_name("Marie Tharp") == "Tharp"


def test_find_family_name_reversed():
    assert find_family_name("Curie, Marie") == "Curie"
    assert find_family_name("Meitner, Lise") == "Meitner"


def test_find_family_name_simple_multipart():
    assert find_family_name("Lewis Fry Richardson") == "Richardson"

test_find_family_name_simple_twoparts()
test_find_family_name_reversed()
test_find_family_name_simple_multipart()

---------------------------------------------------------------------------
AssertionError                            Traceback (most recent call last)
Cell In[14], line 2
      1 test_find_family_name_simple_twoparts()
----> 2 test_find_family_name_reversed()
      3 test_find_family_name_simple_multipart()

Cell In[13], line 7, in test_find_family_name_reversed()
      6 def test_find_family_name_reversed():
----> 7     assert find_family_name("Curie, Marie") == "Curie"
      8     assert find_family_name("Meitner, Lise") == "Meitner"

AssertionError: 

So we find that the reversed order where the given name is attached to the family name with a comma does not work. Let’s add this:

def find_family_name(name_string):
    if "," in name_string:
        return name_string.split(",")[0]
    else:
        return name_string.split(" ")[1]

test_find_family_name_simple_twoparts()
test_find_family_name_reversed()
test_find_family_name_simple_multipart()

---------------------------------------------------------------------------
AssertionError                            Traceback (most recent call last)
Cell In[16], line 3
      1 test_find_family_name_simple_twoparts()
      2 test_find_family_name_reversed()
----> 3 test_find_family_name_simple_multipart()

Cell In[13], line 12, in test_find_family_name_simple_multipart()
     11 def test_find_family_name_simple_multipart():
---> 12     assert find_family_name("Lewis Fry Richardson") == "Richardson"

AssertionError: 

So we fixed this one. Let’s tackle names with multiple given names as well:

def find_family_name(name_string):
    if "," in name_string:
        return name_string.split(",")[0]
    elif len(name_string.split(" ")) == 2:
        return name_string.split(" ")[1]
    else:
        return name_string.split(" ")[-1]

And run our tests again:

test_find_family_name_simple_twoparts()
test_find_family_name_reversed()
test_find_family_name_simple_multipart()

And find them all to succeed.