Subversion Repositories openrisc

Some Thoughts on Automated Network Testing for eCos
***************************************************

Hugo Tyson, Red Hat, Cambridge UK, 2000-07-28


Requirements
============

This thinking is dominated by the need for automated continuous testing
of the StrongARM EBSA-285 boards, which have two ethernet interfaces.
We also have some needs for ongoing eCos network testing.

 o TCP testing: move a large amount of data, checking its correctness.
   (with several streams running in parallel at once)

 o UDP testing: similar but using UDP.

 o TFTP testing: an external server, from a choice of LINUX, NT, SunOS, and
   another EBSA board, get from the target some files, of sizes 0, 1, 512,
   513, 1048576 bytes.  (with several streams running in parallel at once)

 o TFTP testing: put to the target some files, ....

 o TFTP testing: the target tftp client code does the same, getting and
   putting, to an external server.

   [ All that TFTP test makes explicit testing of UDP unneccessary; UDP
   testing would need sequence numbers and so on, so we may as well use
   TFTP as the implementation of that]

 o FTP test: we have a trivial "connect" test; continue to use it.

 o Performance testing: TCP_ECHO, TCP_SOURCE, TCP_SINK programs work in
   concert to measure throughput of a partially loaded target board.
   Source and Sink apps run on an external host.

 o Flood pings: the target floods the hosts on its two external interfaces
   whilst they flood it.  This is left going for a long time, and store
   leaks or crashes are checked for.

Orthogonally to these "feature tests" are requirements to run these tests
with and without these features in combinations:

 o The "realtime test harness" operating - it checks interrupt latencies
   and so on.  This is written and works.

 o Booting statically, via bootp, via DHCP statically/leased on the two
   interfaces in combination.

 o Simulated failure of the network, of the kinds "drop 1 in N packets",
   "drop all for 0 < random() < 30 Seconds" and the like.  Corrupted
   packets being sent out by the target, also!

Needs
---------------

We have some other requirements:

 o Support testing of other net-enabled targets!

 o Run tests at a reasonable rate, so do NOT require a reboot of, say, a
   LINUX host every test run to reconfigure the network environment.

 o Feasible: do NOT require anything too complex in terms of controlling
   the network environment.

 o Do not use too many machines.  The farm is full already.


Other Goals
-----------

These are some ideas that are useful but not strictly necessary:

 o Re-use/work-with the existing test infrastructure

 o Provide the sort of results information that the existing test
   infrastructure does.

 o Work with standard testing *host* computers of various kinds.

 o Support conveniently debugging these test examples at developers' desks
   - not just in the farm.


Details
=======

Because of the flood pinging and malformed packet requirements, the target
boards need to be on an isolated network.

The target board's two interfaces need to be on distinct networks for the
stack to behave properly.



Strategy
========

I believe we can implement everything we need for the host computers to do
using a daemon or server, (cf. the serial test filter) which sits on the
host computer waiting to be told what test we are about to run, and takes
appropriate action.

Note that this does work even for situations where the target is passive,
eg. being a TFTP server.  The target simply "does" TFTP serving for a set
period of time - or perhaps until a cookie file exists in its test file
system - and then performs a set of consistency checks (including on the
state of the test FS), thus creating a PASS/FAIL test result.  It can also
periodically run those checks anyway, and choose FAIL at any time.

But who tells the host daemon what to do?  The target does, of course.
That way, the host is stateless, it simply runs that daemon doing what it's
bid, and does NOT ever have to report test results.  This has enormous
advantages, because it means we gather test results from what the target
said, and no other source, thus minimizing changes to the farm software.
It also means that to add a new test, we can asynchronously add the feature
to the test daemons of that is required, then add a new testcase in the
usual manner, with all the usual (compile-time) testing of its
applicability as usual.

Network Topology
----------------

The idea is that we can initially have a setup like this:

    house network <---> [net-testfarm-machine] serial -------+
                                                             |
                                                           serial
    house network <----> eth0 [LINUX BOX] eth1 <---> eth0 [ EBSA ]
                              [ dhcpd   ] eth2 <---> eth1 [      ]
                              [ tftpd   ]
                              [ ftpd    ]
                              [ testd   ]

for developing the system.  Testd is our new daemon that runs tcp_sink, or
tcp_source, or a floodping, or does tftp to the target (rather than vice
versa) as and when the target instructs it.  The target can report test
results to the net-testfarm-machine as usual, but with bigger timeouts &c
configured in the test farm.

This system can then be generalized to

        test-server1            test-server2            test-serverN
           eth0                   eth0                     eth0
            |                      |                        |
            |                      |                        |
           eth0                    |                        |
        target1                 target4                 targetM
           target2                target5                 targetM+1
              target3               target6                 targetM+2
           eth1
            |
            |
        test-server11
                
where target1,2,3 have 2 ethernet interfaces and the others have only one.

And further, provided the testd protocol supports targets choosing one
server from many which offer service, which would be a good thing:

        test-server1            test-server2            test-serverN
           eth0                   eth0                     eth0
         [LINUX]                 [Solaris]                [NT4.0]
            |                      |                        |
            +----------------------+------------------------+
            |                      |                        |
           eth0                    |                        |
        target1                 target4                 targetM
           target2                target5                 targetM+1
              target3               target6                 targetM+2
           eth1
            |
            +-----------------------+
            |                       |
         [LINUX]                 [NT4.0]
        test-server11          test-server12

which would IMHO be a good thing IN ADDITION to a completely partitioned
set of test network as above.  The partitioned set of test networks is
required also because we need to test all of:

 Target asks for BOOTP vs. DHCP  -X-  Server does only BOOTP vs. DHCP

in combinations on the different interfaces.  Simply setting up servers
that way statically is best, rather than trying to script controls for
servers that have them offering bootp one minute and DHCP the next.


Test Farm
---------

Orthogonal to the network topology, the network test farm is connected to
all these targets in the usual manner by serial lines.  That way the tests
can run and busy/cripple that local network without affecting the house
network *and* without affecting the debug connection, and with the
advantage that tests net traffic can interfere with each other, providing a
diverse network environment for testing, rather than a quiet net.

For testing with GDB connection over the network, which is desirable, I
suggest either keeping those machines separate, or having the farm's
connection into the test-net be via second interfaces fitted to the server
machines.

Otherwise, it's a standard test farm, which knows to choose only from a
special set of perms, and which has waaay longer timeouts.


Test Cases
----------

In this way, tests of the form "make an external machine do tftp to the
target board" are implemented by means of a standard eCos test case, which
requests that action from a server, then waits for it to be told AOK or
just for a certain time, and reports as such to the test farm as usual.

These special tests are only compiled in certain perms.

Those same special perms select between the various initialization options
required also: DHCP or BOOTP or static initialization, and so on, in the
usual manner.


Implementation
--------------

Just a quick note on this: DHCP has a lot of the properties we want for the
test protocol.  We should take a copy of that and use different port
numbers, re-use a lot of the code, since server code is also available.

Or something simpler; none of this seems especially challenging.


That's all for now.
.ends