Re: [ERROR] NAKACK.handleXmitReq()

On Saturday 16 September 2006 09:23, Bela Ban wrote:
> Ok, 600 billion 256b messages without problems (see below). Average 
> message rate of 82K messages/sec

(Bela, I send this message again to the mailing list because it was too big and 
the moderator rejected it )


My test failed with out of memory here you have the logs:


Host: oks
-----------------

Configuration is: 

----------------------- TEST -----------------------
Date: Fri Sep 15 10:39:12 MSD 2006
Run by: comfw

log_interval:   1000000
msg_size:       256
config: /home/comfw/public/perf_tests/config/config.txt
transport:      org.jgroups.tests.perf.transports.JGroupsTransport
num_senders:    3
sender: true
start_port:     7800
props:  /home/comfw/public/perf_tests/hosts/oks/protocol.xml
dump_transport_stats:   false
num_msgs:       2000000000
processing_delay:       0
num_members:    3
jmx:    false
JGroups version: 2.4CR2
-------------------------------------------------------
GMS: address is 10.50.3.38:7800
-------------------------------------------------------
-- waiting for 3 members to join
-- 10.50.3.38:7800 joined
-- 10.50.3.37:7800 joined
-- 10.50.3.40:7800 joined
-- members: 3
-- sending 2000000000 256b messages
-- received 1000000 messages
-- received 2000000 messages
-- received 3000000 messages

<...snip...>

++ sent 501000000
-- received 1568000000 messages
-- received 1569000000 messages
460 [ERROR] TP.receive(): - failed handling data from 10.50.3.37:7800
java.lang.OutOfMemoryError: Java heap space
-- received 1570000000 messages

<...snip...>

++ sent 725000000
-- received 2275000000 messages
-- received 2276000000 messages
-- received 2277000000 messages
18818302 [ERROR] TP.receive(): - failed handling data from 10.50.3.37:7800
java.lang.OutOfMemoryError: Java heap space
++ sent 726000000
-- received 2278000000 messages

<...snip...>

++ sent 1524000000
-- received 4893000000 messages
88971904 [ERROR] BasicConnectionTable$Connection._send(): - exception is java.la
ng.OutOfMemoryError: Java heap space
-- received 4894000000 messages
-- received 4895000000 messages
++ sent 1525000000

<..snip...>

-- received 5906000000 messages
-- received 5907000000 messages
121494302 [ERROR] ConnectionTableNIO$WriteHandler.run(): - Thread (nioWriteHandl
erThread) caught Throwable
java.lang.OutOfMemoryError: Java heap space
++ sent 1916000000
-- received 5908000000 messages
[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor676]
[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor672]
[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor671]
[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor674]
[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor675]
[Unloading class sun.reflect.GeneratedSerializationConstructorAccessor673]
-- received 5909000000 messages
-- received 5910000000 messages
++ sent 1917000000
-- received 5911000000 messages
Exception in thread "nioReadHandlerThread" java.lang.OutOfMemoryError: Java 
heap space
121848991 [WARN] ConnectionTableNIO$ReadHandler.run(): - Read operation on 
socket failed
java.io.IOException: Peer closed socket
        at 
org.jgroups.blocks.ConnectionTableNIO$ReadHandler.readHeader(ConnectionTableNIO.java:746)
        at 
org.jgroups.blocks.ConnectionTableNIO$ReadHandler.readOnce(ConnectionTableNIO.java:699)
        at 
org.jgroups.blocks.ConnectionTableNIO$ReadHandler.run(ConnectionTableNIO.java:638)
        at java.lang.Thread.run(Thread.java:595)
121849379 [WARN] ConnectionTableNIO$ReadHandler.run(): - Read operation on 
socket failed
java.io.IOException: Peer closed socket
        at 
org.jgroups.blocks.ConnectionTableNIO$ReadHandler.readHeader(ConnectionTableNIO.java:746)
        at 
org.jgroups.blocks.ConnectionTableNIO$ReadHandler.readOnce(ConnectionTableNIO.java:699)
        at 
org.jgroups.blocks.ConnectionTableNIO$ReadHandler.run(ConnectionTableNIO.java:638)
        at java.lang.Thread.run(Thread.java:595)
121864212 [ERROR] BasicConnectionTable$Connection._send(): - exception is 
java.lang.reflect.InvocationTargetException
121888660 [WARN] ConnectionTableNIO$ReadHandler.run(): - Read operation on 
socket failed
java.io.IOException: Peer closed socket
        at 
org.jgroups.blocks.ConnectionTableNIO$ReadHandler.readHeader(ConnectionTableNIO.java:746)
        at 
org.jgroups.blocks.ConnectionTableNIO$ReadHandler.readOnce(ConnectionTableNIO.java:699)
        at 
org.jgroups.blocks.ConnectionTableNIO$ReadHandler.run(ConnectionTableNIO.java:638)
        at java.lang.Thread.run(Thread.java:595)
121888691 [WARN] ConnectionTableNIO$ReadHandler.run(): - Read operation on 
socket failed
java.io.IOException: Peer closed socket
        at 
org.jgroups.blocks.ConnectionTableNIO$ReadHandler.readHeader(ConnectionTableNIO.java:746)
        at 
org.jgroups.blocks.ConnectionTableNIO$ReadHandler.readOnce(ConnectionTableNIO.java:699)
        at 
org.jgroups.blocks.ConnectionTableNIO$ReadHandler.run(ConnectionTableNIO.java:638)
        at java.lang.Thread.run(Thread.java:595)
121944702 [ERROR] MPING.sendMcastDiscoveryRequest(): - failed sending discovery 
request
java.io.InterruptedIOException: operation interrupted
        at java.net.PlainDatagramSocketImpl.send(Native Method)
        at java.net.DatagramSocket.send(DatagramSocket.java:612)
        at org.jgroups.protocols.MPING.sendMcastDiscoveryRequest(MPING.java:255)
        at org.jgroups.protocols.PING.sendGetMembersRequest(PING.java:211)
        at org.jgroups.protocols.PingSender.run(PingSender.java:59)
        at java.lang.Thread.run(Thread.java:595)
121944702 [WARN] ConnectionTableNIO.run(): - 10.50.3.37:7800 is already there, 
will terminate connection
121944704 [WARN] ConnectionTableNIO.run(): - 10.50.3.40:7800 is already there, 
will terminate connection
121975845 [ERROR] BasicConnectionTable$Connection._send(): - exception is 
java.lang.reflect.InvocationTargetException
122014520 [WARN] ConnectionTableNIO.run(): - 10.50.3.37:7800 is already there, 
will terminate connection
122014521 [WARN] ConnectionTableNIO.run(): - 10.50.3.40:7800 is already there, 
will terminate connection
122014529 [WARN] ConnectionTableNIO.run(): - 10.50.3.37:7800 is already there, 
will terminate connection
122014554 [WARN] ConnectionTableNIO.run(): - 10.50.3.40:7800 is already there, 
will terminate connection
122074766 [WARN] ConnectionTableNIO.run(): - 10.50.3.37:7800 is already there, 
will terminate connection
122074793 [WARN] ConnectionTableNIO.run(): - 10.50.3.40:7800 is already there, 
will terminate connection
122075777 [WARN] ConnectionTableNIO.run(): - 10.50.3.37:7800 is already there, 
will terminate connection
122075804 [WARN] ConnectionTableNIO.run(): - 10.50.3.40:7800 is already there, 
will terminate connection
122115129 [WARN] ConnectionTableNIO.run(): - 10.50.3.37:7800 is already there, 
will terminate connection
122115135 [WARN] ConnectionTableNIO.run(): - 10.50.3.40:7800 is already there, 
will terminate connection
122116129 [WARN] ConnectionTableNIO.run(): - 10.50.3.37:7800 is already there, 
will terminate connection
122116154 [WARN] ConnectionTableNIO.run(): - 10.50.3.40:7800 is already there, 
will terminate connection
122144942 [WARN] ConnectionTableNIO.run(): - 10.50.3.37:7800 is already there, 
will terminate connection
122144963 [WARN] ConnectionTableNIO.run(): - 10.50.3.40:7800 is already there, 
will terminate connection
122145937 [WARN] ConnectionTableNIO.run(): - 10.50.3.37:7800 is already there, 
will terminate connection
122145972 [WARN] ConnectionTableNIO.run(): - 10.50.3.40:7800 is already there, 
will terminate connection
122198490 [WARN] ConnectionTableNIO.run(): - 10.50.3.40:7800 is already there, 
will terminate connection
122198497 [WARN] ConnectionTableNIO.run(): - 10.50.3.37:7800 is already there, 
will terminate connection
122199497 [WARN] ConnectionTableNIO.run(): - 10.50.3.37:7800 is already there, 
will terminate connection
122199500 [WARN] ConnectionTableNIO.run(): - 10.50.3.40:7800 is already there, 
will terminate connection
122222440 [WARN] ConnectionTableNIO.run(): - 10.50.3.37:7800 is already there, 
will terminate connection
122223450 [WARN] ConnectionTableNIO.run(): - 10.50.3.37:7800 is already there, 
will terminate connection
154713629 [WARN] ConnectionTableNIO$ReadHandler.run(): - Read operation on 
socket failed
java.io.IOException: Peer closed socket
        at 
org.jgroups.blocks.ConnectionTableNIO$ReadHandler.readHeader(ConnectionTableNIO.java:746)
        at 
org.jgroups.blocks.ConnectionTableNIO$ReadHandler.readOnce(ConnectionTableNIO.java:699)
        at 
org.jgroups.blocks.ConnectionTableNIO$ReadHandler.run(ConnectionTableNIO.java:638)
        at java.lang.Thread.run(Thread.java:595)
154719135 [WARN] ConnectionTableNIO.run(): - 10.50.3.37:7800 is already there, 
will terminate connection
154720144 [WARN] ConnectionTableNIO.run(): - 10.50.3.37:7800 is already there, 
will terminate connection


Host: vezer (coordinator)
--------------------------------

Configuration is: 

----------------------- TEST -----------------------
Date: Fri Sep 15 10:37:51 MSD 2006
Run by: comfw

log_interval:   1000000
msg_size:       256
config: /home/comfw/public/perf_tests/config/config.txt
transport:      org.jgroups.tests.perf.transports.JGroupsTransport
num_senders:    3
sender: true
start_port:     7800
props:  /home/comfw/public/perf_tests/hosts/vezer/protocol.xml
dump_transport_stats:   false
num_msgs:       2000000000
processing_delay:       0
num_members:    3
jmx:    false
JGroups version: 2.4CR2


-------------------------------------------------------
GMS: address is 10.50.3.37:7800
-------------------------------------------------------
-- waiting for 3 members to join
-- 10.50.3.37:7800 joined
-- 10.50.3.38:7800 joined
-- 10.50.3.40:7800 joined
-- members: 3
-- sending 2000000000 256b messages

<...snip...>

++ sent 1993000000
-- received 5910000000 messages
-- received 5911000000 messages
++ sent 1994000000
0 [WARN] FD.up(): - I was suspected by 10.50.3.38:7800; ignoring the SUSPECT 
message and sending back a HEARTBEAT_ACK
39831 [WARN] ConnectionTableNIO.run(): - 10.50.3.38:7800 is already there, will 
terminate connection
41381 [WARN] ConnectionTableNIO.run(): - 10.50.3.38:7800 is already there, will 
terminate connection
68583 [WARN] ConnectionTableNIO.run(): - 10.50.3.38:7800 is already there, will 
terminate connection
-- received 5912000000 messages
97397 [WARN] ConnectionTableNIO$ReadHandler.run(): - Read operation on socket 
failed
java.io.IOException: Peer closed socket
        at 
org.jgroups.blocks.ConnectionTableNIO$ReadHandler.readHeader(ConnectionTableNIO.java:746)
        at 
org.jgroups.blocks.ConnectionTableNIO$ReadHandler.readOnce(ConnectionTableNIO.java:699)
        at 
org.jgroups.blocks.ConnectionTableNIO$ReadHandler.run(ConnectionTableNIO.java:638)
        at java.lang.Thread.run(Thread.java:595)
++ sent 1995000000
-- received 5913000000 messages
++ sent 1996000000
-- received 5914000000 messages
167207 [WARN] ConnectionTableNIO$ReadHandler.run(): - Read operation on socket 
failed
java.io.IOException: Peer closed socket
        at 
org.jgroups.blocks.ConnectionTableNIO$ReadHandler.readHeader(ConnectionTableNIO.java:746)
        at 
org.jgroups.blocks.ConnectionTableNIO$ReadHandler.readOnce(ConnectionTableNIO.java:699)
        at 
org.jgroups.blocks.ConnectionTableNIO$ReadHandler.run(ConnectionTableNIO.java:638)
        at java.lang.Thread.run(Thread.java:595)
167216 [WARN] ConnectionTableNIO$ReadHandler.run(): - Read operation on socket 
failed
java.io.IOException: Peer closed socket
        at 
org.jgroups.blocks.ConnectionTableNIO$ReadHandler.readHeader(ConnectionTableNIO.java:746)
        at 
org.jgroups.blocks.ConnectionTableNIO$ReadHandler.readOnce(ConnectionTableNIO.java:699)
        at 
org.jgroups.blocks.ConnectionTableNIO$ReadHandler.run(ConnectionTableNIO.java:638)
        at java.lang.Thread.run(Thread.java:595)
++ sent 1997000000
-- received 5915000000 messages
++ sent 1998000000
227454 [WARN] ConnectionTableNIO$ReadHandler.run(): - Read operation on socket 
failed
java.io.IOException: Peer closed socket
        at 
org.jgroups.blocks.ConnectionTableNIO$ReadHandler.readHeader(ConnectionTableNIO.java:746)
        at 
org.jgroups.blocks.ConnectionTableNIO$ReadHandler.readOnce(ConnectionTableNIO.java:699)
        at 
org.jgroups.blocks.ConnectionTableNIO$ReadHandler.run(ConnectionTableNIO.java:638)
        at java.lang.Thread.run(Thread.java:595)
228465 [WARN] ConnectionTableNIO$ReadHandler.run(): - Read operation on socket 
failed
java.io.IOException: Peer closed socket
        at 
org.jgroups.blocks.ConnectionTableNIO$ReadHandler.readHeader(ConnectionTableNIO.java:746)
        at 
org.jgroups.blocks.ConnectionTableNIO$ReadHandler.readOnce(ConnectionTableNIO.java:699)
        at 
org.jgroups.blocks.ConnectionTableNIO$ReadHandler.run(ConnectionTableNIO.java:638)
        at java.lang.Thread.run(Thread.java:595)
-- received 5916000000 messages
++ sent 1999000000
267815 [WARN] ConnectionTableNIO$ReadHandler.run(): - Read operation on socket 
failed
java.io.IOException: Peer closed socket
        at 
org.jgroups.blocks.ConnectionTableNIO$ReadHandler.readHeader(ConnectionTableNIO.java:746)
        at 
org.jgroups.blocks.ConnectionTableNIO$ReadHandler.readOnce(ConnectionTableNIO.java:699)
        at 
org.jgroups.blocks.ConnectionTableNIO$ReadHandler.run(ConnectionTableNIO.java:638)
        at java.lang.Thread.run(Thread.java:595)
268815 [WARN] ConnectionTableNIO$ReadHandler.run(): - Read operation on socket 
failed
java.io.IOException: Peer closed socket
        at 
org.jgroups.blocks.ConnectionTableNIO$ReadHandler.readHeader(ConnectionTableNIO.java:746)
        at 
org.jgroups.blocks.ConnectionTableNIO$ReadHandler.readOnce(ConnectionTableNIO.java:699)
        at 
org.jgroups.blocks.ConnectionTableNIO$ReadHandler.run(ConnectionTableNIO.java:638)
        at java.lang.Thread.run(Thread.java:595)
java.lang.NullPointerException
        at 
org.jgroups.blocks.BasicConnectionTable$Connection.getSockAddress(BasicConnectionTable.java:267)
        at 
org.jgroups.blocks.BasicConnectionTable$Connection.access$100(BasicConnectionTable.java:246)
        at 
org.jgroups.blocks.BasicConnectionTable$Connection$Sender.start(BasicConnectionTable.java:591)
        at 
org.jgroups.blocks.BasicConnectionTable$Connection.send(BasicConnectionTable.java:345)
        at 
org.jgroups.blocks.BasicConnectionTable.send(BasicConnectionTable.java:215)
        at org.jgroups.protocols.TCP_NIO.send(TCP_NIO.java:50)
        at org.jgroups.protocols.BasicTCP.sendToSingleMember(BasicTCP.java:61)
        at org.jgroups.protocols.TP.doSend(TP.java:950)
        at org.jgroups.protocols.TP.access$2100(TP.java:45)
        at org.jgroups.protocols.TP$Bundler.bundleAndSend(TP.java:1662)
        at org.jgroups.protocols.TP$Bundler.send(TP.java:1585)
        at org.jgroups.protocols.TP$Bundler.access$100(TP.java:1568)
        at org.jgroups.protocols.TP.send(TP.java:925)
        at org.jgroups.protocols.TP.down(TP.java:725)
        at org.jgroups.stack.Protocol.receiveDownEvent(Protocol.java:517)
        at org.jgroups.stack.Protocol.passDown(Protocol.java:551)
        at org.jgroups.protocols.Discovery.down(Discovery.java:333)
        at org.jgroups.stack.Protocol.receiveDownEvent(Protocol.java:517)
        at org.jgroups.stack.Protocol.passDown(Protocol.java:551)
        at org.jgroups.protocols.MERGE2.down(MERGE2.java:208)
        at org.jgroups.stack.Protocol.receiveDownEvent(Protocol.java:517)
        at org.jgroups.stack.Protocol.passDown(Protocol.java:551)
        at org.jgroups.protocols.FD_SOCK.down(FD_SOCK.java:407)
        at org.jgroups.stack.Protocol.receiveDownEvent(Protocol.java:517)
        at org.jgroups.stack.Protocol.passDown(Protocol.java:551)
        at org.jgroups.protocols.FD.down(FD.java:313)
        at org.jgroups.stack.Protocol.receiveDownEvent(Protocol.java:517)
        at org.jgroups.stack.Protocol.passDown(Protocol.java:551)
        at org.jgroups.stack.Protocol.down(Protocol.java:577)
        at org.jgroups.stack.Protocol.receiveDownEvent(Protocol.java:517)
        at org.jgroups.stack.Protocol.passDown(Protocol.java:551)
        at org.jgroups.protocols.pbcast.NAKACK.send(NAKACK.java:621)
        at org.jgroups.protocols.pbcast.NAKACK.down(NAKACK.java:409)
        at org.jgroups.stack.Protocol.receiveDownEvent(Protocol.java:517)
        at org.jgroups.stack.Protocol.passDown(Protocol.java:551)
        at org.jgroups.protocols.pbcast.STABLE.down(STABLE.java:283)
        at org.jgroups.stack.Protocol.receiveDownEvent(Protocol.java:517)
        at org.jgroups.stack.Protocol.passDown(Protocol.java:551)
        at org.jgroups.protocols.pbcast.GMS.down(GMS.java:809)
        at org.jgroups.stack.Protocol.receiveDownEvent(Protocol.java:517)
        at org.jgroups.stack.Protocol.passDown(Protocol.java:551)
        at org.jgroups.protocols.FC.handleDownMessage(FC.java:383)
        at org.jgroups.protocols.FC.down(FC.java:300)
        at org.jgroups.stack.Protocol.receiveDownEvent(Protocol.java:517)
        at org.jgroups.protocols.FC.receiveDownEvent(FC.java:294)
        at org.jgroups.stack.Protocol.passDown(Protocol.java:551)
        at org.jgroups.protocols.FRAG2.down(FRAG2.java:167)
        at org.jgroups.stack.Protocol.receiveDownEvent(Protocol.java:517)
        at org.jgroups.stack.Protocol.passDown(Protocol.java:551)
        at 
org.jgroups.protocols.pbcast.STREAMING_STATE_TRANSFER.down(STREAMING_STATE_TRANSFER.java:376)
        at org.jgroups.stack.Protocol.receiveDownEvent(Protocol.java:517)
        at org.jgroups.stack.Protocol.passDown(Protocol.java:551)
        at org.jgroups.protocols.pbcast.FLUSH.down(FLUSH.java:179)
        at org.jgroups.stack.Protocol.receiveDownEvent(Protocol.java:517)
        at org.jgroups.stack.ProtocolStack.down(ProtocolStack.java:385)
        at org.jgroups.JChannel.down(JChannel.java:1175)
        at org.jgroups.JChannel.send(JChannel.java:584)
        at 
org.jgroups.tests.perf.transports.JGroupsTransport.send(JGroupsTransport.java:83)
        at org.jgroups.tests.perf.Test.sendMessages(Test.java:401)
        at org.jgroups.tests.perf.Test.main(Test.java:681)
-- received 5917000000 messages
297628 [WARN] ConnectionTableNIO$ReadHandler.run(): - Read operation on socket 
failed
java.io.IOException: Peer closed socket
        at 
org.jgroups.blocks.ConnectionTableNIO$ReadHandler.readHeader(ConnectionTableNIO.java:746)
        at 
org.jgroups.blocks.ConnectionTableNIO$ReadHandler.readOnce(ConnectionTableNIO.java:699)
        at 
org.jgroups.blocks.ConnectionTableNIO$ReadHandler.run(ConnectionTableNIO.java:638)
        at java.lang.Thread.run(Thread.java:595)
298623 [WARN] ConnectionTableNIO$ReadHandler.run(): - Read operation on socket 
failed
java.io.IOException: Peer closed socket
        at 
org.jgroups.blocks.ConnectionTableNIO$ReadHandler.readHeader(ConnectionTableNIO.java:746)
        at 
org.jgroups.blocks.ConnectionTableNIO$ReadHandler.readOnce(ConnectionTableNIO.java:699)
        at 
org.jgroups.blocks.ConnectionTableNIO$ReadHandler.run(ConnectionTableNIO.java:638)
        at java.lang.Thread.run(Thread.java:595)
++ sent 2000000000
351184 [WARN] ConnectionTableNIO$ReadHandler.run(): - Read operation on socket 
failed
java.io.IOException: Peer closed socket
        at 
org.jgroups.blocks.ConnectionTableNIO$ReadHandler.readHeader(ConnectionTableNIO.java:746)
        at 
org.jgroups.blocks.ConnectionTableNIO$ReadHandler.readOnce(ConnectionTableNIO.java:699)
        at 
org.jgroups.blocks.ConnectionTableNIO$ReadHandler.run(ConnectionTableNIO.java:638)
        at java.lang.Thread.run(Thread.java:595)
351185 [ERROR] BasicConnectionTable$Connection._send(): - exception is 
java.lang.reflect.InvocationTargetException
352184 [WARN] ConnectionTableNIO$ReadHandler.run(): - Read operation on socket 
failed
java.io.IOException: Peer closed socket
        at 
org.jgroups.blocks.ConnectionTableNIO$ReadHandler.readHeader(ConnectionTableNIO.java:746)
        at 
org.jgroups.blocks.ConnectionTableNIO$ReadHandler.readOnce(ConnectionTableNIO.java:699)
        at 
org.jgroups.blocks.ConnectionTableNIO$ReadHandler.run(ConnectionTableNIO.java:638)
        at java.lang.Thread.run(Thread.java:595)
375125 [WARN] ConnectionTableNIO$ReadHandler.run(): - Read operation on socket 
failed
java.io.IOException: Peer closed socket
        at 
org.jgroups.blocks.ConnectionTableNIO$ReadHandler.readHeader(ConnectionTableNIO.java:746)
        at 
org.jgroups.blocks.ConnectionTableNIO$ReadHandler.readOnce(ConnectionTableNIO.java:699)
        at 
org.jgroups.blocks.ConnectionTableNIO$ReadHandler.run(ConnectionTableNIO.java:638)
        at java.lang.Thread.run(Thread.java:595)
376135 [WARN] ConnectionTableNIO$ReadHandler.run(): - Read operation on socket 
failed
java.io.IOException: Peer closed socket
        at 
org.jgroups.blocks.ConnectionTableNIO$ReadHandler.readHeader(ConnectionTableNIO.java:746)
        at 
org.jgroups.blocks.ConnectionTableNIO$ReadHandler.readOnce(ConnectionTableNIO.java:699)
        at 
org.jgroups.blocks.ConnectionTableNIO$ReadHandler.run(ConnectionTableNIO.java:638)
        at java.lang.Thread.run(Thread.java:595)
462060 [WARN] CoordGmsImpl$MergeTask.run(): - merge responses from subgroup 
coordinators <= 1 ([sender=10.50.3.37:7800, view=[10.50.3.37:7800|3] 
[10.50.3.37:7800, 10.50.3.40
:7800], digest=10.50.3.40:7800: [2000873580 : 2000873581], 10.50.3.37:7800: 
[2000866427 : 2000866428]]). Cancelling merge
560428 [ERROR] CoordGmsImpl.sendMergeCancelledMessage(): - coords or merge_id 
== null
560428 [ERROR] CoordGmsImpl.sendMergeCancelledMessage(): - coords or merge_id 
== null
604571 [ERROR] CoordGmsImpl.sendMergeCancelledMessage(): - coords or merge_id 
== null
604571 [ERROR] CoordGmsImpl.sendMergeCancelledMessage(): - coords or merge_id 
== null
675726 [WARN] CoordGmsImpl$MergeTask.run(): - merge responses from subgroup 
coordinators <= 1 ([]). Cancelling merge
675727 [ERROR] CoordGmsImpl.sendMergeCancelledMessage(): - coords or merge_id 
== null
675727 [ERROR] CoordGmsImpl.sendMergeCancelledMessage(): - coords or merge_id 
== null
705729 [ERROR] CoordGmsImpl.sendMergeCancelledMessage(): - coords or merge_id 
== null
705729 [ERROR] CoordGmsImpl.sendMergeCancelledMessage(): - coords or merge_id 
== null
792835 [WARN] CoordGmsImpl$MergeTask.run(): - merge responses from subgroup 
coordinators <= 1 ([sender=10.50.3.37:7800, view=[10.50.3.37:7800|3] 
[10.50.3.37:7800, 10.50.3.40
:7800], digest=10.50.3.40:7800: [2000873585 : 2000873586], 10.50.3.37:7800: 
[2000866432 : 2000866433]]). Cancelling merge
823658 [WARN] CoordGmsImpl$MergeTask.run(): - merge responses from subgroup 
coordinators <= 1 ([]). Cancelling merge
823659 [ERROR] CoordGmsImpl.sendMergeCancelledMessage(): - coords or merge_id 
== null
823659 [ERROR] CoordGmsImpl.sendMergeCancelledMessage(): - coords or merge_id 
== null
859981 [WARN] CoordGmsImpl$MergeTask.run(): - merge responses from subgroup 
coordinators <= 1 ([]). Cancelling merge
859982 [ERROR] CoordGmsImpl.sendMergeCancelledMessage(): - coords or merge_id 
== null
859982 [ERROR] CoordGmsImpl.sendMergeCancelledMessage(): - coords or merge_id 
== null
932777 [WARN] CoordGmsImpl$MergeTask.run(): - merge responses from subgroup 
coordinators <= 1 ([]). Cancelling merge
932778 [ERROR] CoordGmsImpl.sendMergeCancelledMessage(): - coords or merge_id 
== null
932779 [ERROR] CoordGmsImpl.sendMergeCancelledMessage(): - coords or merge_id 
== null
1030117 [WARN] CoordGmsImpl$MergeTask.run(): - merge responses from subgroup 
coordinators <= 1 ([]). Cancelling merge
1030117 [ERROR] CoordGmsImpl.sendMergeCancelledMessage(): - coords or merge_id 
== null
1030118 [ERROR] CoordGmsImpl.sendMergeCancelledMessage(): - coords or merge_id 
== null
1059809 [WARN] CoordGmsImpl$MergeTask.run(): - merge responses from subgroup 
coordinators <= 1 ([]). Cancelling merge
1059810 [ERROR] CoordGmsImpl.sendMergeCancelledMessage(): - coords or merge_id 
== null
1059811 [ERROR] CoordGmsImpl.sendMergeCancelledMessage(): - coords or merge_id 
== null
1097613 [WARN] CoordGmsImpl$MergeTask.run(): - merge responses from subgroup 
coordinators <= 1 ([]). Cancelling merge
1097614 [ERROR] CoordGmsImpl.sendMergeCancelledMessage(): - coords or merge_id 
== null
1097614 [ERROR] CoordGmsImpl.sendMergeCancelledMessage(): - coords or merge_id 
== null
1190893 [ERROR] CoordGmsImpl.sendMergeCancelledMessage(): - coords or merge_id 
== null
1190893 [ERROR] CoordGmsImpl.sendMergeCancelledMessage(): - coords or merge_id 
== null
1216565 [WARN] CoordGmsImpl$MergeTask.run(): - merge responses from subgroup 
coordinators <= 1 ([]). Cancelling merge
1216565 [ERROR] CoordGmsImpl.sendMergeCancelledMessage(): - coords or merge_id 
== null
1216566 [ERROR] CoordGmsImpl.sendMergeCancelledMessage(): - coords or merge_id 
== null

<...snip...>

32420889 [WARN] CoordGmsImpl$MergeTask.run(): - merge responses from subgroup 
coordinators <= 1 ([]). Cancelling merge
32420889 [ERROR] CoordGmsImpl.sendMergeCancelledMessage(): - coords or merge_id 
== null
32420889 [ERROR] CoordGmsImpl.sendMergeCancelledMessage(): - coords or merge_id 
== null
32496669 [WARN] CoordGmsImpl$MergeTask.run(): - merge responses from subgroup 
coordinators <= 1 ([sender=10.50.3.37:7800, view=[10.50.3.37:7800|3] 
[10.50.3.37:7800, 10.50.3.
40:7800], digest=10.50.3.40:7800: [2000874046 : 2000874047], 10.50.3.37:7800: 
[2000866933 : 2000866934]]). Cancelling merge
32590291 [WARN] CoordGmsImpl$MergeTask.run(): - merge responses from subgroup 
coordinators <= 1 ([]). Cancelling merge
32590291 [ERROR] CoordGmsImpl.sendMergeCancelledMessage(): - coords or merge_id 
== null
32590292 [ERROR] CoordGmsImpl.sendMergeCancelledMessage(): - coords or merge_id 
== null
32669710 [WARN] CoordGmsImpl$MergeTask.run(): - merge responses from subgroup 
coordinators <= 1 ([sender=10.50.3.37:7800, view=[10.50.3.37:7800|3] 
[10.50.3.37:7800, 10.50.3.
40:7800], digest=10.50.3.40:7800: [2000874048 : 2000874049], 10.50.3.37:7800: 
[2000866937 : 2000866938]]). Cancelling merge
32726946 [WARN] CoordGmsImpl$MergeTask.run(): - merge responses from subgroup 
coordinators <= 1 ([sender=10.50.3.37:7800, view=[10.50.3.37:7800|3] 
[10.50.3.37:7800, 10.50.3.
40:7800], digest=10.50.3.40:7800: [2000874048 : 2000874049], 10.50.3.37:7800: 
[2000866937 : 2000866938]]). Cancelling merge
32763481 [WARN] CoordGmsImpl$MergeTask.run(): - merge responses from subgroup 
coordinators <= 1 ([sender=10.50.3.37:7800, view=[10.50.3.37:7800|3] 
[10.50.3.37:7800, 10.50.3.
40:7800], digest=10.50.3.40:7800: [2000874050 : 2000874051], 10.50.3.37:7800: 
[2000866938 : 2000866939]]). Cancelling merge
32832678 [WARN] CoordGmsImpl$MergeTask.run(): - merge responses from subgroup 
coordinators <= 1 ([]). Cancelling merge
32832678 [ERROR] CoordGmsImpl.sendMergeCancelledMessage(): - coords or merge_id 
== null
32832679 [ERROR] CoordGmsImpl.sendMergeCancelledMessage(): - coords or merge_id 
== null
32866307 [WARN] ConnectionTableNIO$ReadHandler.run(): - Read operation on 
socket failed
java.io.IOException: Peer closed socket
        at 
org.jgroups.blocks.ConnectionTableNIO$ReadHandler.readHeader(ConnectionTableNIO.java:746)
        at 
org.jgroups.blocks.ConnectionTableNIO$ReadHandler.readOnce(ConnectionTableNIO.java:699)
        at 
org.jgroups.blocks.ConnectionTableNIO$ReadHandler.run(ConnectionTableNIO.java:638)
        at java.lang.Thread.run(Thread.java:595)
32871815 [WARN] ConnectionTableNIO$ReadHandler.run(): - Read operation on 
socket failed
java.io.IOException: Peer closed socket
        at 
org.jgroups.blocks.ConnectionTableNIO$ReadHandler.readHeader(ConnectionTableNIO.java:746)
        at 
org.jgroups.blocks.ConnectionTableNIO$ReadHandler.readOnce(ConnectionTableNIO.java:699)
        at 
org.jgroups.blocks.ConnectionTableNIO$ReadHandler.run(ConnectionTableNIO.java:638)
        at java.lang.Thread.run(Thread.java:595)
32872824 [WARN] ConnectionTableNIO$ReadHandler.run(): - Read operation on 
socket failed
java.io.IOException: Peer closed socket
        at 
org.jgroups.blocks.ConnectionTableNIO$ReadHandler.readHeader(ConnectionTableNIO.java:746)
        at 
org.jgroups.blocks.ConnectionTableNIO$ReadHandler.readOnce(ConnectionTableNIO.java:699)
        at 
org.jgroups.blocks.ConnectionTableNIO$ReadHandler.run(ConnectionTableNIO.java:638)
        at java.lang.Thread.run(Thread.java:595)
32881908 [WARN] ConnectionTableNIO$ReadHandler.run(): - Read operation on 
socket failed
java.io.IOException: Peer closed socket
        at 
org.jgroups.blocks.ConnectionTableNIO$ReadHandler.readHeader(ConnectionTableNIO.java:746)
        at 
org.jgroups.blocks.ConnectionTableNIO$ReadHandler.readOnce(ConnectionTableNIO.java:699)
        at 
org.jgroups.blocks.ConnectionTableNIO$ReadHandler.run(ConnectionTableNIO.java:638)
        at java.lang.Thread.run(Thread.java:595)
32881909 [WARN] ConnectionTableNIO$ReadHandler.run(): - Read operation on 
socket failed
java.io.IOException: Peer closed socket
        at 
org.jgroups.blocks.ConnectionTableNIO$ReadHandler.readHeader(ConnectionTableNIO.java:746)
        at 
org.jgroups.blocks.ConnectionTableNIO$ReadHandler.readOnce(ConnectionTableNIO.java:699)
        at 
org.jgroups.blocks.ConnectionTableNIO$ReadHandler.run(ConnectionTableNIO.java:638)
        at java.lang.Thread.run(Thread.java:595)
32893054 [WARN] CoordGmsImpl$MergeTask.run(): - merge responses from subgroup 
coordinators <= 1 ([]). Cancelling merge
32893055 [ERROR] CoordGmsImpl.sendMergeCancelledMessage(): - coords or merge_id 
== null
32893055 [ERROR] CoordGmsImpl.sendMergeCancelledMessage(): - coords or merge_id 
== null

Host: oster
-------------------

Configuration is: 

----------------------- TEST -----------------------
Date: Fri Sep 15 10:39:39 MSD 2006
Run by: comfw

log_interval:   1000000
msg_size:       256
config: /home/comfw/public/perf_tests/config/config.txt
transport:      org.jgroups.tests.perf.transports.JGroupsTransport
num_senders:    3
sender: true
start_port:     7800
props:  /home/comfw/public/perf_tests/hosts/oster/protocol.xml
dump_transport_stats:   false
num_msgs:       2000000000
processing_delay:       0
num_members:    3
jmx:    false
JGroups version: 2.4CR2


-------------------------------------------------------
GMS: address is 10.50.3.40:7800
-------------------------------------------------------
-- waiting for 3 members to join
-- 10.50.3.37:7800 joined
-- 10.50.3.40:7800 joined
-- 10.50.3.38:7800 joined
-- members: 3
-- sending 2000000000 256b messages

<...snip...>

++ sent 1999000000
-- received 5324000000 messages
-- received 5325000000 messages
++ sent 2000000000

<...snip...>

-- received 5908000000 messages
-- received 5909000000 messages
-- received 5910000000 messages
-- received 5911000000 messages
0 [WARN] FD.up(): - I was suspected by 10.50.3.38:7800; ignoring the SUSPECT 
message and sending back a HEARTBEAT_ACK
11279 [WARN] FD.up(): - I was suspected by 10.50.3.38:7800; ignoring the 
SUSPECT message and sending back a HEARTBEAT_ACK
22229 [WARN] FD.up(): - I was suspected by 10.50.3.38:7800; ignoring the 
SUSPECT message and sending back a HEARTBEAT_ACK
32409 [WARN] FD.up(): - I was suspected by 10.50.3.38:7800; ignoring the 
SUSPECT message and sending back a HEARTBEAT_ACK
42749 [WARN] FD.up(): - I was suspected by 10.50.3.38:7800; ignoring the 
SUSPECT message and sending back a HEARTBEAT_ACK
52759 [WARN] FD.up(): - I was suspected by 10.50.3.38:7800; ignoring the 
SUSPECT message and sending back a HEARTBEAT_ACK
65069 [WARN] FD.up(): - I was suspected by 10.50.3.38:7800; ignoring the 
SUSPECT message and sending back a HEARTBEAT_ACK
65399 [WARN] ConnectionTableNIO.run(): - 10.50.3.38:7800 is already there, will 
terminate connection
73429 [WARN] ConnectionTableNIO.run(): - 10.50.3.38:7800 is already there, will 
terminate connection
-- received 5912000000 messages
160725 [WARN] ConnectionTableNIO$ReadHandler.run(): - Read operation on socket 
failed
java.io.IOException: Peer closed socket
        at 
org.jgroups.blocks.ConnectionTableNIO$ReadHandler.readHeader(ConnectionTableNIO.java:746)
        at 
org.jgroups.blocks.ConnectionTableNIO$ReadHandler.readOnce(ConnectionTableNIO.java:699)
        at 
org.jgroups.blocks.ConnectionTableNIO$ReadHandler.run(ConnectionTableNIO.java:638)
        at java.lang.Thread.run(Thread.java:595)
-- received 5913000000 messages
-- received 5914000000 messages
230539 [WARN] ConnectionTableNIO$ReadHandler.run(): - Read operation on socket 
failed
java.io.IOException: Peer closed socket
        at 
org.jgroups.blocks.ConnectionTableNIO$ReadHandler.readHeader(ConnectionTableNIO.java:746)
        at 
org.jgroups.blocks.ConnectionTableNIO$ReadHandler.readOnce(ConnectionTableNIO.java:699)
        at 
org.jgroups.blocks.ConnectionTableNIO$ReadHandler.run(ConnectionTableNIO.java:638)
        at java.lang.Thread.run(Thread.java:595)
230571 [WARN] ConnectionTableNIO$ReadHandler.run(): - Read operation on socket 
failed
java.io.IOException: Peer closed socket
        at 
org.jgroups.blocks.ConnectionTableNIO$ReadHandler.readHeader(ConnectionTableNIO.java:746)
        at 
org.jgroups.blocks.ConnectionTableNIO$ReadHandler.readOnce(ConnectionTableNIO.java:699)
        at 
org.jgroups.blocks.ConnectionTableNIO$ReadHandler.run(ConnectionTableNIO.java:638)
        at java.lang.Thread.run(Thread.java:595)
-- received 5915000000 messages
290811 [WARN] ConnectionTableNIO$ReadHandler.run(): - Read operation on socket 
failed
java.io.IOException: Peer closed socket
        at 
org.jgroups.blocks.ConnectionTableNIO$ReadHandler.readHeader(ConnectionTableNIO.java:746)
        at 
org.jgroups.blocks.ConnectionTableNIO$ReadHandler.readOnce(ConnectionTableNIO.java:699)
        at 
org.jgroups.blocks.ConnectionTableNIO$ReadHandler.run(ConnectionTableNIO.java:638)
        at java.lang.Thread.run(Thread.java:595)
291821 [WARN] ConnectionTableNIO$ReadHandler.run(): - Read operation on socket 
failed
java.io.IOException: Peer closed socket
        at 
org.jgroups.blocks.ConnectionTableNIO$ReadHandler.readHeader(ConnectionTableNIO.java:746)
        at 
org.jgroups.blocks.ConnectionTableNIO$ReadHandler.readOnce(ConnectionTableNIO.java:699)
        at 
org.jgroups.blocks.ConnectionTableNIO$ReadHandler.run(ConnectionTableNIO.java:638)
        at java.lang.Thread.run(Thread.java:595)
-- received 5916000000 messages
331152 [WARN] ConnectionTableNIO$ReadHandler.run(): - Read operation on socket 
failed
java.io.IOException: Peer closed socket
        at 
org.jgroups.blocks.ConnectionTableNIO$ReadHandler.readHeader(ConnectionTableNIO.java:746)
        at 
org.jgroups.blocks.ConnectionTableNIO$ReadHandler.readOnce(ConnectionTableNIO.java:699)
        at 
org.jgroups.blocks.ConnectionTableNIO$ReadHandler.run(ConnectionTableNIO.java:638)
        at java.lang.Thread.run(Thread.java:595)
332171 [WARN] ConnectionTableNIO$ReadHandler.run(): - Read operation on socket 
failed
java.io.IOException: Peer closed socket
        at 
org.jgroups.blocks.ConnectionTableNIO$ReadHandler.readHeader(ConnectionTableNIO.java:746)
        at 
org.jgroups.blocks.ConnectionTableNIO$ReadHandler.readOnce(ConnectionTableNIO.java:699)
        at 
org.jgroups.blocks.ConnectionTableNIO$ReadHandler.run(ConnectionTableNIO.java:638)
        at java.lang.Thread.run(Thread.java:595)
-- received 5917000000 messages
360981 [WARN] ConnectionTableNIO$ReadHandler.run(): - Read operation on socket 
failed
java.io.IOException: Peer closed socket
        at 
org.jgroups.blocks.ConnectionTableNIO$ReadHandler.readHeader(ConnectionTableNIO.java:746)
        at 
org.jgroups.blocks.ConnectionTableNIO$ReadHandler.readOnce(ConnectionTableNIO.java:699)
        at 
org.jgroups.blocks.ConnectionTableNIO$ReadHandler.run(ConnectionTableNIO.java:638)
        at java.lang.Thread.run(Thread.java:595)
361991 [WARN] ConnectionTableNIO$ReadHandler.run(): - Read operation on socket 
failed
java.io.IOException: Peer closed socket
        at 
org.jgroups.blocks.ConnectionTableNIO$ReadHandler.readHeader(ConnectionTableNIO.java:746)
        at 
org.jgroups.blocks.ConnectionTableNIO$ReadHandler.readOnce(ConnectionTableNIO.java:699)
        at 
org.jgroups.blocks.ConnectionTableNIO$ReadHandler.run(ConnectionTableNIO.java:638)
        at java.lang.Thread.run(Thread.java:595)
414511 [WARN] ConnectionTableNIO$ReadHandler.run(): - Read operation on socket 
failed
java.io.IOException: Peer closed socket
        at 
org.jgroups.blocks.ConnectionTableNIO$ReadHandler.readHeader(ConnectionTableNIO.java:746)
        at 
org.jgroups.blocks.ConnectionTableNIO$ReadHandler.readOnce(ConnectionTableNIO.java:699)
        at 
org.jgroups.blocks.ConnectionTableNIO$ReadHandler.run(ConnectionTableNIO.java:638)
        at java.lang.Thread.run(Thread.java:595)
415521 [WARN] ConnectionTableNIO$ReadHandler.run(): - Read operation on socket 
failed
java.io.IOException: Peer closed socket
        at 
org.jgroups.blocks.ConnectionTableNIO$ReadHandler.readHeader(ConnectionTableNIO.java:746)
        at 
org.jgroups.blocks.ConnectionTableNIO$ReadHandler.readOnce(ConnectionTableNIO.java:699)
        at 
org.jgroups.blocks.ConnectionTableNIO$ReadHandler.run(ConnectionTableNIO.java:638)
        at java.lang.Thread.run(Thread.java:595)


thanks iand regards
        gianluca

-- 
Gianluca Puggelli
skype:pugg1138

-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys -- and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV