Gearman testing

So basically I like Gearman and would like to use it.  I can think of a few things I could throw it's way.  But I've discovered that with the current build that I lose data along the way.  I usually still get a request on the worker (but not always), but it will have lost the arguments along the way.  Test files: gearman_test.zip

Make and install Gearman 1.10 and JSON::XS from CPAN then follow these simple steps:

ServerA:
sudo su
tar -xzvf gearman_test.tar.gz
cd gearman_test
gearmand
perl worker.pl [--json] [--storable]

ServerB:
sudo su
tar -xzvf gearman_test.tar.gz
cd gearman_test
perl bench.pl --server="ip:port" [--json] [--storable] [--loops=x]


bench.pl will default the port if you don't provide one.  Obviously make sure that our json/storable options match between worker and benchmark tool.  If you provide
neither then it will skip serialization and just use a plain scalar.  Default for loops is 100000.

If I do a localhost test I normally don't see problems.  It only happens when going across external interfaces which implies maybe a latecy issue or something.  None
of the many other network related tools we have running show any issues with our network.  And the amount of faults will vary, but you seem to have to run it continuously for a minute or more to see it. Example:

[jstephens@host (12:16:03) ~/gearman_test]$ perl bench.pl --json --loops=10 --ip=xx.xx.xx.xx
10 loops of other code took: 0 wallclock secs ( 0.00 usr +  0.00 sys =  0.00 CPU)
With 10 requests and  bad requests
[jstephens@host (12:16:11) ~/gearman_test]$ perl bench.pl --json --loops=10000 --ip=xx.xx.xx.xx
10000 loops of other code took:21 wallclock secs ( 4.67 usr +  1.19 sys =  5.86 CPU) @ 1706.48/s (n=10000)
With 10000 requests and  bad requests
[jstephens@host (12:16:49) ~/gearman_test]$ perl bench.pl --json --loops=100000 --ip=xx.xx.xx.xx
100000 loops of other code took:211 wallclock secs (46.41 usr + 14.81 sys = 61.22 CPU) @ 1633.45/s (n=100000)
With 99207 requests and 793 bad requests

[jstephens@host (12:20:26) ~/gearman_test]$ ping xx.xx.xx.xx
PING xx.xx.xx.xx (xx.xx.xx.xx) 56(84) bytes of data.
64 bytes from xx.xx.xx.xx: icmp_seq=1 ttl=61 time=0.178 ms
64 bytes from xx.xx.xx.xx: icmp_seq=2 ttl=61 time=0.199 ms
64 bytes from xx.xx.xx.xx: icmp_seq=3 ttl=61 time=0.159 ms
64 bytes from xx.xx.xx.xx: icmp_seq=4 ttl=61 time=0.167 ms
64 bytes from xx.xx.xx.xx: icmp_seq=5 ttl=61 time=0.176 ms
64 bytes from xx.xx.xx.xx: icmp_seq=6 ttl=61 time=0.196 ms
64 bytes from xx.xx.xx.xx: icmp_seq=7 ttl=61 time=0.153 ms
64 bytes from xx.xx.xx.xx: icmp_seq=8 ttl=61 time=0.170 ms
64 bytes from xx.xx.xx.xx: icmp_seq=9 ttl=61 time=0.183 ms
64 bytes from xx.xx.xx.xx: icmp_seq=10 ttl=61 time=0.177 ms
64 bytes from xx.xx.xx.xx: icmp_seq=11 ttl=61 time=0.189 ms
64 bytes from xx.xx.xx.xx: icmp_seq=12 ttl=61 time=0.159 ms
64 bytes from xx.xx.xx.xx: icmp_seq=13 ttl=61 time=0.175 ms
64 bytes from xx.xx.xx.xx: icmp_seq=14 ttl=61 time=0.185 ms
64 bytes from xx.xx.xx.xx: icmp_seq=15 ttl=61 time=0.188 ms
64 bytes from xx.xx.xx.xx: icmp_seq=16 ttl=61 time=0.208 ms
64 bytes from xx.xx.xx.xx: icmp_seq=17 ttl=61 time=0.172 ms
64 bytes from xx.xx.xx.xx: icmp_seq=18 ttl=61 time=0.179 ms
64 bytes from xx.xx.xx.xx: icmp_seq=19 ttl=61 time=0.369 ms
64 bytes from xx.xx.xx.xx: icmp_seq=20 ttl=61 time=0.202 ms
64 bytes from xx.xx.xx.xx: icmp_seq=21 ttl=61 time=0.163 ms

--- xx.xx.xx.xx ping statistics ---
21 packets transmitted, 21 received, 0% packet loss, time 20000ms
rtt min/avg/max/mdev = 0.153/0.187/0.369/0.046 ms