Subversion Repositories c0or1k

                                TESTING HOWTO

                               By Bahadir Balban

HOW TO ADD TESTS

Tests in this project are meant to test basic functionality of the microkernel
but cover test cases comprehensively. The special trait with these tests is
that each is meant to test only one aspect of the microkernel, rather than
trying to test multiple aspects at once.

A benefit of having this approach is that when a test fails, it is easier to
spot the problem as opposed to a failure that is caused by a complex
interaction of multiple aspects of the microkernel.

The software must be also tested for more complex interactions, but these
tests are not the place to create them. When a new test is added, it must cover
a valid test case but the test itself must be as shallow as possible.


EXAMPLE

A thread creation test, while using other aspects of the microkernel such as
the l4_exchange_registers system call, should only test thread creation aspect
of the microkernel. A separate l4_exchange_registers test would need to be
created who then uses the thread creation aspect of the microkernel, but only
testing the parameters to l4_exchange_registers system call, covering it
exhaustively. In essence, each test must be shallow in itself, and have a single
focus.


Here's a step-by-step approach to testing:

1.) Create one test that tests one thing well.
    This is about the above paragraphs. Test one aspect, so we know that that
    aspect is functional. When there are lots of changes to code and many
    invariables, this will make sure that these tests are one solid invariable
    in the equation. They should be as deterministic and simple as possible
    so that other variants can be debugged using the tests as anchor points.

    Each test must be significantly simple. E.g. If I have 2 test cases that
    require slightly differing input to a test function, I would not use a
    loop of i = 0..2 where parameters change when i == 0 and i == 1. That
    would be difficult to understand, and complicated. Instead, each of those
    inputs should be 2 test cases. This way, I focus %100 on the debugged code,
    I don't put a tiny fraction of my focus trying to understand the test.

2.) Number (1) is unexpectedly hard to achieve. Especially a person who is
    accustomed to design complex systems that work will have a hard time in
    applying an approach that is completely the opposite. Make sure to stop
    thinking smart and get used to thinking not moderately dumb but very dumb.

3.) Make sure to escalate errors properly.
    When porting to a new architecture or where there are lots of variables,
    your tests will fail. It is an extra burden to find out how. Error
    escalation provides a quick and easy means to provide error details.

4.) Make sure to print only when necessary.
    Tests should print out values only when a tested aspect goes wrong.
    a) It should describe what the test was trying to do, i.e. describe what
       was expected to happen. This way it is quicker to see expected results.
    b) It should describe what has happend, contrary to expected. This should
       tell what went wrong.
    c) It should tell no more. Printout clutter is a huge negative factor in
       making tests effective.

4.) Complicating the tests:
    It is often necessary that simple tests must be the building blocks for
    more complicated tests so that complicated interactions in the system
    can be tested.

    Complicating the test cases should be done with a systematic approach:

    a) Tests are like building blocks. Each test that tests one aspect of the
       software should be designed as a rock solid way of testing that one
       aspect. See (1) above for the first step. Design extremely simple tests.

    b) A design should be carefully made such that multiple aspects as
       described in (a) provide a highly deterministic and reproducable
       sequence of events when combined together. While this significantly
       narrows down the tested domain of functionality, it equally much ensures
       that the test case is covered in full control, thus should anything
       go wrong and test fails, it is much easier to find out what went wrong.

       E.g. a test that randomly creates threads would be almost useless since
       any failure of the test would be irreproducable and very hard to use
       to recover any bug.

    c) Test cases should slowly and surely grow into a larger domain of
       functionality by rigorously and systematically following steps (a) and
       (b). If these steps are not rigorously followed, the tests would be
       diluted in quality, and the testing structure would quickly become a
       mess.

     Some examples:
     ~~~~~~~~~~~~~~
     1) Test 1: Extended ipc on already mapped pages.
     2) Test 2: Extended ipc on page faulting pages.
     3) Test 3: Extended ipc on page faulting pages, but non contiguous
        physical pages.
     4) Test 4: Extended ipc on page faulting pages, but non contiguous
        physical pages, and between threads in different address spaces.
     5) Test 5: Extended ipc on page faulting pages, but non contiguous
        physical pages, and between threads in different address spaces,
        but using the same virtual addresses on both threads.

      You see, a single ipc test is systematically growing into a more complex
      test, but at each step, only one possibly variable aspect is added and
      tested. The other test aspects have already been validated by earlier,
      simpler combinations of test cases.