Subversion Repositories openrisc

Host Software for Automated Network Testing for eCos

====================================================

Hugo Tyson, Red Hat, Cambridge UK, 2000-10-20

What Host Software?

-------------------

This note refers to the package CYGPKG_NET_AUTOTEST which lives in

packages/net/autotest in the eCos repository.

It contains, amongst other things, some software to run a "network testing

server" which eCos network test cases (from the same package) use to test

the network stack.

The reason such a server is needed is simple: testing the eCos network

stack involves an external, standard, computer _doing_things_to_ the target

board as well as having the target board's test program do things itself.

For example, one test case (in the main network package, net/tcpip) to test

TFTP will get a file from a server using TFTP and put the data to another

file on the server using TFTP.  This tests the ability of the eCos TCPIP

stack to act as a TFTP client.  But it does not test the eCos TCPIP stack's

TFTP server at all.  To do that an external agent must act as client -

using the standard tftp(1) app on linux, for example - to put and get data

to and from an eCos testcase running on the target, offering a TFTP server.

Quick Start Guide

-----------------

Set up your LINUX machine to serve DHCP to the target you want to test;

ensure that any 10.0.0.x addresses used are given a class C (/24) netmask.

Provide a genuine gateway to the internet via some route; the routing tests

expect to ping www.cygnus.com!  Also let it have the normal networking apps

(tftp, ftp, ping) and servers (tftpd, ftpd) available, plus the SNMP

package from UCD: snmpwalk and snmpstatus and standard MIBs.

Make sure it is on on the same network(s) as the target you want to test.

Set up aliases for loopback interface on the servers: on server for eth0 on

the target, add an alias for 10.0.3.1/24.  For eth1, 10.0.4.1/24.

With a checked-out eCos source tree, change directory into either

packages/net/autotest/current/host or a copy of it, and type make.

Follow the printed instruction to change the simple executable "./_suping"

to su root execution.  You will need to become root temporarily to do this.

Make sure . is on your path and type "awaitorder | obey.sh" to run the

server; it prints info about messages it receives and sends.

The test server is now running.  You can now run test built from the

CYGPKG_NET_AUTOTEST package (packages/net/autotest) on your target and they

will talk to this host test server.

Categories of Tests

-------------------

 o Host passive

        The host is server, the eCos test case is the proactive client.

        Such tests use the host to talk to, where the eCos testcase is the

        client, and the host is the server.  The network testing server is

        not involved in these tests, but other servers which must be set up

        there are used.

 o Host proactive

        The host is client to the eCos server.

        The target board running eCos is the server, the host must run

        scripts to do thing to it and report success or otherwise over the

        net.  The eCos test case typically starts a server, issues order

        about how to test it, then sleeps for a time until the test is

        deemed complete.

Building the Network Testing Server

-----------------------------------

Cd into either packages/net/autotest/current/host or a copy of it, and type

make.  The standard makefile just compiles ./awaitorder.c into ./awaitorder

the executable that waits for a request from a testcase and prints it to

its stdout.  It also builds some other utilities that are used by the test

scripts, for example ./tmpfile from ./tmpfile.c - this just creates a file

of random data for passing back and forth.  It also builds _suping: make

sure this is suid-root to be able to flood-ping the target.

Running the Network Testing Server

----------------------------------

Make sure "." is on your path, early enough, then

        awaitorder | obey.sh

This is all you need to do to run the testing server.

"./awaitorder" just waits for a connection on TCP port 9980 and prints what

it receives.  For example, to test it:

                % telnet masala 9980

 you type --->  Eggs, bacon, beans, mushrooms, toast and coffee

                Acknowledged

                Connection closed by foreign host.

                %

and awaitorder will print as follows:

                 % awaitorder

                awaitorder: connection from 172.16.19.13:3739

                ORDERS: Eggs, bacon, beans, mushrooms, toast and coffee

and continue waiting for the next order.

"./obey.sh" does rather more.  It assumes that its stdin is connected to a

source of orders, and it expects that lines of input will look like this:

        ORDERS: TFTP_SERV_GET 10.16.19.171 300

or more generally

        "ORDERS:" [XFAIL]   

The set of valid orders (TFTP_SERV_GET, TFTP_SERV_PUT ...) will expand with

time as we add more tests.

Having checked that an order looks generally valid, and set up some

environment, it then switches on the order to execute a test script to do

what the testcase that sent the order wants.  The extra params are given to

the test script.  The test script is invoked in background, so that several

sessions can be running at once to hit the target board hard, or of course

one network testing server can be servicing several target boards running

different testcases all at once.  After the test script is launched,

obey.sh just loops back to reading an input line, to get the next order.

"XFAIL" in parameter 1 means the target is configured to simulate network

unreliability, or otherwise be uncooperative.  It is passed both to the

individual testing scripts and to the system for returning status messages

to the test target.

This structure is designed to allow separate development and testing of the

individual parts of the network testing server and also of new testcases;

the eCos testcase and its test script can be developed on your desk without

running the network test server - you only need to knit them all together

at the end by adding a simple clause to obey.sh to invoke the test script

automatically.

Test Script Environment

-----------------------

There is some IO redirection nastiness in obey.sh to separate errors from

good status from status to return to the eCos testcase and so on.

awaitorder takes care to issue complaints to stderr, whilst obviously its

stdout is piped in to obey.sh

obey.sh takes care to issue problem reports to stderr, and mere commentary

to stdout; neither is paricularly verbose right now, so there's no need to

so anything special.

Individual test scripts are invoked as follows.  Stdout is directed to a

logfile named uniquely and for the test in question, according the the

variable $LG.  Stderr is piped to an acknowledgement agent, by default

"./sendack.sh", which is invoked with the target address as its argument.

This returns status messages from the host testing script to the eCos

testcase via TCP port 9990.

Let's look at the two example switch cases for the TFTP tests.  There are

two to mostly put versus mostly get files with the eCos TFTP server.

    # Now the main swicth that runs the test script: this is where

    # new testcases are added:

    case $TEST in

    TFTP_SERV_GET)

    tftpget.sh $TARGET $WF/tftpget.$unique ${PARAMS[@]} \

            2>&1 >$LG/tftpget.$unique | $SENDACK $TARGET & ;;

    TFTP_SERV_PUT)

    tftpput.sh $TARGET $WF/tftpput.$unique ${PARAMS[@]} \

            2>&1 >$LG/tftpput.$unique | $SENDACK $TARGET & ;;

OK, this is horrid.  But breaking up one example:

this first part is the invokation of the script:

    tftpput.sh $TARGET $WF/tftpput.$unique ${PARAMS[@]}

its arguments are

    tftpput.sh    

 o  is $TARGET, from the original order.

 o obey.sh has invented  with a unique id, in the workfiles

   directory, $WF.  It will typically be "/tmp/auto/wf/tftpput.1138".  In

   this example, the test script will make files called

   "/tmp/auto/wf/tftpput.1138.src" and "/tmp/auto/wf/tftpput.1138.tmp" to

   play with; one is created by the tmpfile program, the other read back

   from the TFTP server.

 o  is part of the extra params supplied by the eCos testcase,

   which in this case will serve TFTP for about 5 minutes, so it tells the

   test script to diddle it for about 5 minutes too.

 o  is another param supplied by the eCos testcase, which tells

   the script to create a testfile of a particular size to exercise edge

   conditions wrt the blocksize.

The second part of the example is the IO redirection and backgrounding:

            2>&1 >$LG/tftpput.$unique | $SENDACK $XFAIL $TARGET & ;;

This sets the script's fd2 to a copy of its stdout, then sets its stdout to

go to the logfile, typically "/tmp/auto/log/tftpput.1138", then pipes the

original stdout which now occupies fd2 into $SENDACK, the acknowledgement

agent, which is itself invoked with the target's IP address and optionally

XFAIL to tell it not to be so upset if it fails to return the result.  The

whole thing is run in background by the "&" and the ";;" is the end of this

case within the switch statement.  Phew!

More briefly, the test script's stdout goes into the logfile and the test

script's stderr goes into $SENDACK (./sendack.sh) and thence to the

testcase.

Typical Output

--------------

This is the output from awaitorder | obey.sh from running just one eCos

testcase, which requests 3 simultaneous TFTP sessions for 5 minutes:

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

awaitorder: connection from 10.16.19.171:34554

test TFTP_SERV_PUT; target 10.16.19.171; params 300 512

awaitorder: connection from 10.16.19.171:46052

test TFTP_SERV_PUT; target 10.16.19.171; params 300 513

awaitorder: connection from 172.16.19.171:25388

test TFTP_SERV_PUT; target 172.16.19.171; params 300 512

10.16.19.171: result INFO:<10.16.19.171 is up>

172.16.19.171: result INFO:<172.16.19.171 is up>

10.16.19.171: result INFO:<10.16.19.171 is up>

10.16.19.171: result PASS:

10.16.19.171: result PASS:

172.16.19.171: result PASS:

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

and the eCos test case prints, amongst other things, on receipt of the

results via sendack.sh

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

INFO:<10.16.19.171 is up> (10.16.19.171) (from 10.16.19.13:4764)

INFO:<172.16.19.171 is up> (172.16.19.171) (from 172.16.19.13:4765)

INFO:<10.16.19.171 is up> (10.16.19.171) (from 10.16.19.13:4766)

....

PASS: (10.16.19.171) (from 10.16.19.13:4775)

PASS: (10.16.19.171) (from 10.16.19.13:4776)

PASS: (172.16.19.171) (from 172.16.19.13:4777)

....

PASS:

EXIT:

%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%%

This output format will change over time as more features may be added, but

this example should convey the general idea.

Individual Host Testing Scripts

-------------------------------

The params and invocation of these is entirely a matter of agreement

between the eCos testcase and the test script.  A new case arm in the

obey.sh switch statement must be written for any new testcase, so the

script's invocation is not prescribed in any way.

For the sake of easier separate development of testcases, it would be

better to follow the pattern of the other testcases, also for consistency

and comprehensibility.  Here are some guidelines and requirements, but

first some assumptions:

 o One test script "does things to" one target board, and more specifically

   one target IP address.  If you want several things done to your target

   board at once, invoke several scripts at once - all on the same IP

   address if you want, that's fine.

 o The test script may need to be told that the target network stack is

   configured to simulate network unreliability, so the test script should

   not expect perfect communications.  Eg. weaker protocols will in fact

   fail - the point of the test is verify that the target does not hang or

   crash or assert-fail and so on; host side network failures within the

   script are unimportant.  Otherwise, the test script should expect

   correct operation in the operations it performs.

 o The network test server can serve testing to many eCos target boards at

   once.  Therefore any particular test script can be invoked several times

   at once to deal with different target boards, if they happen at random

   to run the same testcase at the same time as each other.  Therefore a

   test script should only use temporary files which are uniquely named per

   invocation.

These lead to requirements and guidelines:

 o Take the flag XFAIL optionally as the first argument, if it is XFAIL

   shift to get the other arguments.

 o Take the target IP address as the next argument; it's an obvious place

   for it and you'll certainly need it anyhow.

 o For workfiles, have obey.sh supply a parameter to name such files

   uniquely.  Use a full pathname.  Include the test type also, for example

   "/tmp/auto/wf/tftpput.1138" identifies the file as belonging to the

   tftpput testcase.  Make required workfiles be second and subsequent

   arguments.

 o Further arguments which are arbitrary and passed from the testcase

   should be last, and passed all together.  It's not convenient in the

   shell to split out words 2 through the end from an array if you already

   quoted word one elsewhere, so instead put all the extras together as in

   ${PARAMS[@]} in the tftpput.sh invocation discussed above.

Other Environment

-----------------

 o DHCP

Set up DHCP to support the target board - it's the easiest way.

The tests know which server to talk to by looking in the server ID field of

the bootp information record; if DHCP or BOOTP is used, this will be the

server that served it.  It is also possible to set up the stack with static

initialization of that field - if you do this, make sure it points to a

machine that is running the network test server.

Setting up the server to serve DHCP to the target boards only is easy;

keeping other DHCP servers, who offer dynamic addresses to any new net

presence, from offering service to them requires a little trick.  A dynamic

server should be set up to supply a static address to the target board's

MAC address, with the additional key: "deny booting;".  This does nothing

malicious, but it lets the server recognize the target board and ignore it.

This ensures that the network test server machine "wins" the race to set up

the target, and so ensures that the test talks to the network test server

rather than trying to talk to a global DHCP server.

 o /tmp clearance

The ongoing testing fills /tmp with log files and work files.  The logs

will likely not be very large, but the work files could be a few Mb per

test run.

A cron job to remove everything older than an hour, or some such might be

useful; or if the machine can stand it, just wipe /tmp/auto every night.

 o Ping!

The tests will expect to ping (and flood ping) the server; it should be

allowed to do this back.  An suid executable (CARE!) is needed to enable

this.  "./_suping" is built by the host makefile: make it be suid root.

It is a trivial app that execs ping after setting the effective UID to

root.  This is necessary because ping is an suid-root executable anyway, to

be able to get at raw sockets; it checks the effective UID to decide

whether to permit flood pinging; we need this for the flood tests floodping

and floodpingmux.

 o Network aliases in the loopback interface

The routing tests require a network setup like this:

  10.0.3.1      A.B.C.D   A.B.C.E        N.M.O.P   N.M.O.Q      10.0.4.1

  -------- Box1 ----------------- target ----------------- Box2 --------

Optionally with 10.0.3 and 10.0.4 being real networks, with other stations

on them; addresses 10.0.x.99 are also pinged.  Alternatively, 10.0.3 and

10.0.4 can be aliases for the loopback device in Box1 and Box2 (which may

be the same box really).  A typical setup might really look like:

  HOST: eth0 172.16.19.13 ------ 172.16.19.171 eth0 on TARGET

  HOST: eth1 10.16.19.13 -------- 10.16.19.171 eth1 on TARGET

  HOST: lo:0 10.0.3.1

  HOST: lo:1 10.0.4.1

with all netmasks being /24 (255.255.255.0).

This means that the target can only ping addresses 10.0.3.x and 10.0.4.x if

a route has been set up via 172.16.19.13 or 10.16.19.13 (respectively, for

the case of having two distinct hosts).  The routing tests verify that

behaviour.

 o TFTP server and its files

No test of the eCos TFTP client yet, so no details here.

 o FTP server and its files

No test for the eCos FTP client yet, so no details here.

.fin