Powerful testing with DejaGNU

Although I complain about DejaGNU from time to time, it still deserves this post about running tests with multiple variations, which allows you to run your tests with a combination of different options.

make check RUNTESTFLAGS='--target_board=qemu-system-arm\{-marm,-mthumb\}\{-march=armv7-a,-march=armv4t\} call-ar-st.exp'

The test case is compiled and executed four times with different compilation flags, {-marm, -mthumb} x {-march=armv7-a,-march=armv4t}, and you can see the following from console:

 === gdb tests ===

Schedule of variations:
 qemu-system-arm/-marm/-march=armv7-a
 qemu-system-arm/-marm/-march=armv4t
 qemu-system-arm/-mthumb/-march=armv7-a
 qemu-system-arm/-mthumb/-march=armv4t

The test summary is appended to gdb.sum and aggregated like this:

 === gdb Summary for qemu-system-arm/-mthumb/-march=armv4t ===

# of expected passes 45
# of unexpected failures 1

 === gdb Summary ===

# of expected passes 180
# of unexpected failures 4

Note that the test is compiled and run four times in a serialized manner. GDB extends it to allow each variations running in parallel.

make -j2 -k check//qemu-system-arm/{-marm,-mthumb} TESTS='gdb.base/call-ar-st.exp'

GDB test harness starts two instances of running in parallel, for -marm and -mthumb, in directories testsuite.qemu-system-arm.-marm and testsuite.qemu-system-arm.-mthumb respectively. The compiled executables are not overwritten because they are located in different directories.

Last but not least, you need this line in the board file (qemu-system-arm.exp in my case):

process_multilib_options ""