In general, seeing a "PASS" from the sender(s) and receiver(s) for each execution is a good indication that all was successful. Reeceivers will fail if the simple checksum calculated for the payload and trace data doesn't match. Senders will fail if a returned message doesn't have its matching tag (meaning it was returned to the wrong sender). Both will error on a timeout either no route information, or receiver did not receive the expected number of messages. Receivers send an 'ack' for message type 5, so for some tests the number of ack messages sent will not be the same as the number of messages received. Senders loop through message types 0-9 inclusive, unless otherwise directed on the command line (e.g. the rts test sends nothing but message type 5 messages so that all messages are ack'd). Receivers will generate a final histogram of message types received. For example mtype histogram: 0 0 0 0 0 100000 0 0 0 0 0 is generated for the rts test -- all messages are type 5 and thus all other message type bins should be 0. By default, senders send 10 messages at a rate of about 1/sec. Receivers give up after 20 seconds, so even though the rate and number of messages sent can be adjusted from the command line, if the combination is such that the total number of messages sent requires more than 20 seconds to send the tests will fail. Specific examples The output is chopped to the last few lines. Return to sender test with 20 senders sending 5K messages each: ksh run_rts_test.ksh -s 20 -d 180 -n 5000 [PASS] sent=5000 rcvd=4999 rts-ok=4999 failures=0 retries=4 mtype histogram: 0 0 0 0 0 100000 0 0 0 0 0 [PASS] 100000 messages; good=100000 acked=99983 bad=0 bad-trace=0 bad-sub_id=0 [PASS] sent=5000 rcvd=5000 rts-ok=5000 failures=0 retries=4 [PASS] sent=5000 rcvd=4998 rts-ok=4998 failures=0 retries=2 [PASS] sent=5000 rcvd=4998 rts-ok=4998 failures=0 retries=2 [PASS] sent=5000 rcvd=5000 rts-ok=5000 failures=0 retries=4 [PASS] sent=5000 rcvd=4998 rts-ok=4998 failures=0 retries=2 [PASS] sent=5000 rcvd=5000 rts-ok=5000 failures=0 retries=4 [PASS] sent=5000 rcvd=5000 rts-ok=5000 failures=0 retries=2 [PASS] sent=5000 rcvd=5000 rts-ok=5000 failures=0 retries=4 [PASS] sent=5000 rcvd=4999 rts-ok=4999 failures=0 retries=2 [PASS] sent=5000 rcvd=4999 rts-ok=4999 failures=0 retries=4 [PASS] sent=5000 rcvd=4999 rts-ok=4999 failures=0 retries=5 [PASS] sent=5000 rcvd=4999 rts-ok=4999 failures=0 retries=1 [PASS] sent=5000 rcvd=5000 rts-ok=5000 failures=0 retries=4 [PASS] sent=5000 rcvd=4997 rts-ok=4997 failures=0 retries=2 [PASS] sent=5000 rcvd=4999 rts-ok=4999 failures=0 retries=2 [PASS] sent=5000 rcvd=5000 rts-ok=5000 failures=0 retries=2 [PASS] sent=5000 rcvd=5000 rts-ok=5000 failures=0 retries=3 [PASS] sent=5000 rcvd=5000 rts-ok=5000 failures=0 retries=1 [PASS] sent=5000 rcvd=4998 rts-ok=4998 failures=0 retries=2 [PASS] sender rc=0 receiver rc=0 Important notes + The receiver will only retry acks for a finite number of tries before giving up, thus the total acs sent may still be less than messages received. As a cross validation, the total acks sent by the receiver should match the recvd count sum over all senders. + The recvd and rts-ok counts for each sender should match. If they don't the receiver should mark the overall state as a failure as this indicates that a return to sender message was returned to the wrong place. Multiple Receiver test Test run with 10 receivers and sender sending 10K messages. The histograms and status messages were reorganised for easier reading here. ksh run_multi_test.ksh -r 10 -d 180 -n 10000 mtype histogram: 1000 1000 1000 1000 1000 1000 1000 1000 1000 1000 0 mtype histogram: 1000 1000 1000 1000 1000 1000 1000 1000 1000 1000 0 mtype histogram: 1000 1000 1000 1000 1000 1000 1000 1000 1000 1000 0 mtype histogram: 1000 1000 1000 1000 1000 1000 1000 1000 1000 1000 0 mtype histogram: 1000 1000 1000 1000 1000 1000 1000 1000 1000 1000 0 mtype histogram: 1000 1000 1000 1000 1000 1000 1000 1000 1000 1000 0 mtype histogram: 1000 1000 1000 1000 1000 1000 1000 1000 1000 1000 0 mtype histogram: 1000 1000 1000 1000 1000 1000 1000 1000 1000 1000 0 mtype histogram: 1000 1000 1000 1000 1000 1000 1000 1000 1000 1000 0 mtype histogram: 1000 1000 1000 1000 1000 1000 1000 1000 1000 1000 0 [PASS] 10000 messages; good=10000 acked=1000 bad=0 bad-trace=0 bad-sub_id=0 [PASS] sent=10000 rcvd=10000 rts-ok=10000 failures=0 retries=0 [PASS] 10000 messages; good=10000 acked=1000 bad=0 bad-trace=0 bad-sub_id=0 [PASS] 10000 messages; good=10000 acked=1000 bad=0 bad-trace=0 bad-sub_id=0 [PASS] 10000 messages; good=10000 acked=1000 bad=0 bad-trace=0 bad-sub_id=0 [PASS] 10000 messages; good=10000 acked=1000 bad=0 bad-trace=0 bad-sub_id=0 [PASS] 10000 messages; good=10000 acked=1000 bad=0 bad-trace=0 bad-sub_id=0 [PASS] 10000 messages; good=10000 acked=1000 bad=0 bad-trace=0 bad-sub_id=0 [PASS] 10000 messages; good=10000 acked=1000 bad=0 bad-trace=0 bad-sub_id=0 [PASS] 10000 messages; good=10000 acked=1000 bad=0 bad-trace=0 bad-sub_id=0 [PASS] 10000 messages; good=10000 acked=1000 bad=0 bad-trace=0 bad-sub_id=0 [PASS] sender rc=0 receiver rc=0 Important notes: + histograms should show messages for all types, except type 10 which are never sent. + sender should receive only 1/10th of the number of messages sent back as acks; modulo receiver giving up on an ack retry, so as before the sum of ack counts should match the sender's received count. + sender should fail if the received count does not match the rts-ok count indicating that a return to sender was sent to the wrong spot (very unlikely here as there is only one sender). Retries The retries counter for a sender is the number of times that a retry send loop had to be entered in order to successfully send a message. The sender will never give up on a send attempt, but retrying will affect latency of that message. A count of less than 10/10000 messages is good, but it also depends on the rate that the sender is attempting. The higher the rate, the more likely the need to retry, and thus the higher this counter will be.