Re: [ERROR] NAKACK.handleXmitReq()

It is almost impossible to see from those snippets what caused your 
OOMs. Did you run *exactly* the same configuration I emailed you ? You 
probably need to connect to one of the instances with a profiler, to see 
what's causing the leak.
Can you post the config you used, plus the VM startup options again ?

Gianluca Puggelli wrote:
> On Saturday 16 September 2006 09:23, Bela Ban wrote:
>   
>> Ok, 600 billion 256b messages without problems (see below). Average 
>> message rate of 82K messages/sec
>>     
>
> (Bela, I send this message again to the mailing list because it was too big 
> and 
> the moderator rejected it )
>
>
> My test failed with out of memory here you have the logs:
>
>
> Host: oks
> -----------------
>
> Configuration is: 
>
> ----------------------- TEST -----------------------
> Date: Fri Sep 15 10:39:12 MSD 2006
> Run by: comfw
>
> log_interval:   1000000
> msg_size:       256
> config: /home/comfw/public/perf_tests/config/config.txt
> transport:      org.jgroups.tests.perf.transports.JGroupsTransport
> num_senders:    3
> sender: true
> start_port:     7800
> props:  /home/comfw/public/perf_tests/hosts/oks/protocol.xml
> dump_transport_stats:   false
> num_msgs:       2000000000
> processing_delay:       0
> num_members:    3
> jmx:    false
> JGroups version: 2.4CR2
> -------------------------------------------------------
> GMS: address is 10.50.3.38:7800
> -------------------------------------------------------
> -- waiting for 3 members to join
> -- 10.50.3.38:7800 joined
> -- 10.50.3.37:7800 joined
> -- 10.50.3.40:7800 joined
> -- members: 3
> -- sending 2000000000 256b messages
> -- received 1000000 messages
> -- received 2000000 messages
> -- received 3000000 messages
>
> <...snip...>
>
> ++ sent 501000000
> -- received 1568000000 messages
> -- received 1569000000 messages
> 460 [ERROR] TP.receive(): - failed handling data from 10.50.3.37:7800
> java.lang.OutOfMemoryError: Java heap space
> -- received 1570000000 messages
>
> <...snip...>
>
> ++ sent 725000000
> -- received 2275000000 messages
> -- received 2276000000 messages
> -- received 2277000000 messages
> 18818302 [ERROR] TP.receive(): - failed handling data from 10.50.3.37:7800
> java.lang.OutOfMemoryError: Java heap space
> ++ sent 726000000
> -- received 2278000000 messages
>
> <...snip...>
>
> ++ sent 1524000000
> -- received 4893000000 messages
> 88971904 [ERROR] BasicConnectionTable$Connection._send(): - exception is 
> java.la
> ng.OutOfMemoryError: Java heap space
> -- received 4894000000 messages
> -- received 4895000000 messages
> ++ sent 1525000000
>
> <..snip...>
>
> -- received 5906000000 messages
> -- received 5907000000 messages
> 121494302 [ERROR] ConnectionTableNIO$WriteHandler.run(): - Thread 
> (nioWriteHandl
> erThread) caught Throwable
> java.lang.OutOfMemoryError: Java heap space
> ++ sent 1916000000
> -- received 5908000000 messages
> [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor676]
> [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor672]
> [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor671]
> [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor674]
> [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor675]
> [Unloading class sun.reflect.GeneratedSerializationConstructorAccessor673]
> -- received 5909000000 messages
> -- received 5910000000 messages
> ++ sent 1917000000
> -- received 5911000000 messages
> Exception in thread "nioReadHandlerThread" java.lang.OutOfMemoryError: Java 
> heap space
> 121848991 [WARN] ConnectionTableNIO$ReadHandler.run(): - Read operation on 
> socket failed
> java.io.IOException: Peer closed socket
>         at 
> org.jgroups.blocks.ConnectionTableNIO$ReadHandler.readHeader(ConnectionTableNIO.java:746)
>         at 
> org.jgroups.blocks.ConnectionTableNIO$ReadHandler.readOnce(ConnectionTableNIO.java:699)
>         at 
> org.jgroups.blocks.ConnectionTableNIO$ReadHandler.run(ConnectionTableNIO.java:638)
>         at java.lang.Thread.run(Thread.java:595)
> 121849379 [WARN] ConnectionTableNIO$ReadHandler.run(): - Read operation on 
> socket failed
> java.io.IOException: Peer closed socket
>         at 
> org.jgroups.blocks.ConnectionTableNIO$ReadHandler.readHeader(ConnectionTableNIO.java:746)
>         at 
> org.jgroups.blocks.ConnectionTableNIO$ReadHandler.readOnce(ConnectionTableNIO.java:699)
>         at 
> org.jgroups.blocks.ConnectionTableNIO$ReadHandler.run(ConnectionTableNIO.java:638)
>         at java.lang.Thread.run(Thread.java:595)
> 121864212 [ERROR] BasicConnectionTable$Connection._send(): - exception is 
> java.lang.reflect.InvocationTargetException
> 121888660 [WARN] ConnectionTableNIO$ReadHandler.run(): - Read operation on 
> socket failed
> java.io.IOException: Peer closed socket
>         at 
> org.jgroups.blocks.ConnectionTableNIO$ReadHandler.readHeader(ConnectionTableNIO.java:746)
>         at 
> org.jgroups.blocks.ConnectionTableNIO$ReadHandler.readOnce(ConnectionTableNIO.java:699)
>         at 
> org.jgroups.blocks.ConnectionTableNIO$ReadHandler.run(ConnectionTableNIO.java:638)
>         at java.lang.Thread.run(Thread.java:595)
> 121888691 [WARN] ConnectionTableNIO$ReadHandler.run(): - Read operation on 
> socket failed
> java.io.IOException: Peer closed socket
>         at 
> org.jgroups.blocks.ConnectionTableNIO$ReadHandler.readHeader(ConnectionTableNIO.java:746)
>         at 
> org.jgroups.blocks.ConnectionTableNIO$ReadHandler.readOnce(ConnectionTableNIO.java:699)
>         at 
> org.jgroups.blocks.ConnectionTableNIO$ReadHandler.run(ConnectionTableNIO.java:638)
>         at java.lang.Thread.run(Thread.java:595)
> 121944702 [ERROR] MPING.sendMcastDiscoveryRequest(): - failed sending 
> discovery request
> java.io.InterruptedIOException: operation interrupted
>         at java.net.PlainDatagramSocketImpl.send(Native Method)
>         at java.net.DatagramSocket.send(DatagramSocket.java:612)
>         at 
> org.jgroups.protocols.MPING.sendMcastDiscoveryRequest(MPING.java:255)
>         at org.jgroups.protocols.PING.sendGetMembersRequest(PING.java:211)
>         at org.jgroups.protocols.PingSender.run(PingSender.java:59)
>         at java.lang.Thread.run(Thread.java:595)
> 121944702 [WARN] ConnectionTableNIO.run(): - 10.50.3.37:7800 is already 
> there, will terminate connection
> 121944704 [WARN] ConnectionTableNIO.run(): - 10.50.3.40:7800 is already 
> there, will terminate connection
> 121975845 [ERROR] BasicConnectionTable$Connection._send(): - exception is 
> java.lang.reflect.InvocationTargetException
> 122014520 [WARN] ConnectionTableNIO.run(): - 10.50.3.37:7800 is already 
> there, will terminate connection
> 122014521 [WARN] ConnectionTableNIO.run(): - 10.50.3.40:7800 is already 
> there, will terminate connection
> 122014529 [WARN] ConnectionTableNIO.run(): - 10.50.3.37:7800 is already 
> there, will terminate connection
> 122014554 [WARN] ConnectionTableNIO.run(): - 10.50.3.40:7800 is already 
> there, will terminate connection
> 122074766 [WARN] ConnectionTableNIO.run(): - 10.50.3.37:7800 is already 
> there, will terminate connection
> 122074793 [WARN] ConnectionTableNIO.run(): - 10.50.3.40:7800 is already 
> there, will terminate connection
> 122075777 [WARN] ConnectionTableNIO.run(): - 10.50.3.37:7800 is already 
> there, will terminate connection
> 122075804 [WARN] ConnectionTableNIO.run(): - 10.50.3.40:7800 is already 
> there, will terminate connection
> 122115129 [WARN] ConnectionTableNIO.run(): - 10.50.3.37:7800 is already 
> there, will terminate connection
> 122115135 [WARN] ConnectionTableNIO.run(): - 10.50.3.40:7800 is already 
> there, will terminate connection
> 122116129 [WARN] ConnectionTableNIO.run(): - 10.50.3.37:7800 is already 
> there, will terminate connection
> 122116154 [WARN] ConnectionTableNIO.run(): - 10.50.3.40:7800 is already 
> there, will terminate connection
> 122144942 [WARN] ConnectionTableNIO.run(): - 10.50.3.37:7800 is already 
> there, will terminate connection
> 122144963 [WARN] ConnectionTableNIO.run(): - 10.50.3.40:7800 is already 
> there, will terminate connection
> 122145937 [WARN] ConnectionTableNIO.run(): - 10.50.3.37:7800 is already 
> there, will terminate connection
> 122145972 [WARN] ConnectionTableNIO.run(): - 10.50.3.40:7800 is already 
> there, will terminate connection
> 122198490 [WARN] ConnectionTableNIO.run(): - 10.50.3.40:7800 is already 
> there, will terminate connection
> 122198497 [WARN] ConnectionTableNIO.run(): - 10.50.3.37:7800 is already 
> there, will terminate connection
> 122199497 [WARN] ConnectionTableNIO.run(): - 10.50.3.37:7800 is already 
> there, will terminate connection
> 122199500 [WARN] ConnectionTableNIO.run(): - 10.50.3.40:7800 is already 
> there, will terminate connection
> 122222440 [WARN] ConnectionTableNIO.run(): - 10.50.3.37:7800 is already 
> there, will terminate connection
> 122223450 [WARN] ConnectionTableNIO.run(): - 10.50.3.37:7800 is already 
> there, will terminate connection
> 154713629 [WARN] ConnectionTableNIO$ReadHandler.run(): - Read operation on 
> socket failed
> java.io.IOException: Peer closed socket
>         at 
> org.jgroups.blocks.ConnectionTableNIO$ReadHandler.readHeader(ConnectionTableNIO.java:746)
>         at 
> org.jgroups.blocks.ConnectionTableNIO$ReadHandler.readOnce(ConnectionTableNIO.java:699)
>         at 
> org.jgroups.blocks.ConnectionTableNIO$ReadHandler.run(ConnectionTableNIO.java:638)
>         at java.lang.Thread.run(Thread.java:595)
> 154719135 [WARN] ConnectionTableNIO.run(): - 10.50.3.37:7800 is already 
> there, will terminate connection
> 154720144 [WARN] ConnectionTableNIO.run(): - 10.50.3.37:7800 is already 
> there, will terminate connection
>
>
> Host: vezer (coordinator)
> --------------------------------
>
> Configuration is: 
>
> ----------------------- TEST -----------------------
> Date: Fri Sep 15 10:37:51 MSD 2006
> Run by: comfw
>
> log_interval:   1000000
> msg_size:       256
> config: /home/comfw/public/perf_tests/config/config.txt
> transport:      org.jgroups.tests.perf.transports.JGroupsTransport
> num_senders:    3
> sender: true
> start_port:     7800
> props:  /home/comfw/public/perf_tests/hosts/vezer/protocol.xml
> dump_transport_stats:   false
> num_msgs:       2000000000
> processing_delay:       0
> num_members:    3
> jmx:    false
> JGroups version: 2.4CR2
>
>
> -------------------------------------------------------
> GMS: address is 10.50.3.37:7800
> -------------------------------------------------------
> -- waiting for 3 members to join
> -- 10.50.3.37:7800 joined
> -- 10.50.3.38:7800 joined
> -- 10.50.3.40:7800 joined
> -- members: 3
> -- sending 2000000000 256b messages
>
> <...snip...>
>
> ++ sent 1993000000
> -- received 5910000000 messages
> -- received 5911000000 messages
> ++ sent 1994000000
> 0 [WARN] FD.up(): - I was suspected by 10.50.3.38:7800; ignoring the SUSPECT 
> message and sending back a HEARTBEAT_ACK
> 39831 [WARN] ConnectionTableNIO.run(): - 10.50.3.38:7800 is already there, 
> will terminate connection
> 41381 [WARN] ConnectionTableNIO.run(): - 10.50.3.38:7800 is already there, 
> will terminate connection
> 68583 [WARN] ConnectionTableNIO.run(): - 10.50.3.38:7800 is already there, 
> will terminate connection
> -- received 5912000000 messages
> 97397 [WARN] ConnectionTableNIO$ReadHandler.run(): - Read operation on socket 
> failed
> java.io.IOException: Peer closed socket
>         at 
> org.jgroups.blocks.ConnectionTableNIO$ReadHandler.readHeader(ConnectionTableNIO.java:746)
>         at 
> org.jgroups.blocks.ConnectionTableNIO$ReadHandler.readOnce(ConnectionTableNIO.java:699)
>         at 
> org.jgroups.blocks.ConnectionTableNIO$ReadHandler.run(ConnectionTableNIO.java:638)
>         at java.lang.Thread.run(Thread.java:595)
> ++ sent 1995000000
> -- received 5913000000 messages
> ++ sent 1996000000
> -- received 5914000000 messages
> 167207 [WARN] ConnectionTableNIO$ReadHandler.run(): - Read operation on 
> socket failed
> java.io.IOException: Peer closed socket
>         at 
> org.jgroups.blocks.ConnectionTableNIO$ReadHandler.readHeader(ConnectionTableNIO.java:746)
>         at 
> org.jgroups.blocks.ConnectionTableNIO$ReadHandler.readOnce(ConnectionTableNIO.java:699)
>         at 
> org.jgroups.blocks.ConnectionTableNIO$ReadHandler.run(ConnectionTableNIO.java:638)
>         at java.lang.Thread.run(Thread.java:595)
> 167216 [WARN] ConnectionTableNIO$ReadHandler.run(): - Read operation on 
> socket failed
> java.io.IOException: Peer closed socket
>         at 
> org.jgroups.blocks.ConnectionTableNIO$ReadHandler.readHeader(ConnectionTableNIO.java:746)
>         at 
> org.jgroups.blocks.ConnectionTableNIO$ReadHandler.readOnce(ConnectionTableNIO.java:699)
>         at 
> org.jgroups.blocks.ConnectionTableNIO$ReadHandler.run(ConnectionTableNIO.java:638)
>         at java.lang.Thread.run(Thread.java:595)
> ++ sent 1997000000
> -- received 5915000000 messages
> ++ sent 1998000000
> 227454 [WARN] ConnectionTableNIO$ReadHandler.run(): - Read operation on 
> socket failed
> java.io.IOException: Peer closed socket
>         at 
> org.jgroups.blocks.ConnectionTableNIO$ReadHandler.readHeader(ConnectionTableNIO.java:746)
>         at 
> org.jgroups.blocks.ConnectionTableNIO$ReadHandler.readOnce(ConnectionTableNIO.java:699)
>         at 
> org.jgroups.blocks.ConnectionTableNIO$ReadHandler.run(ConnectionTableNIO.java:638)
>         at java.lang.Thread.run(Thread.java:595)
> 228465 [WARN] ConnectionTableNIO$ReadHandler.run(): - Read operation on 
> socket failed
> java.io.IOException: Peer closed socket
>         at 
> org.jgroups.blocks.ConnectionTableNIO$ReadHandler.readHeader(ConnectionTableNIO.java:746)
>         at 
> org.jgroups.blocks.ConnectionTableNIO$ReadHandler.readOnce(ConnectionTableNIO.java:699)
>         at 
> org.jgroups.blocks.ConnectionTableNIO$ReadHandler.run(ConnectionTableNIO.java:638)
>         at java.lang.Thread.run(Thread.java:595)
> -- received 5916000000 messages
> ++ sent 1999000000
> 267815 [WARN] ConnectionTableNIO$ReadHandler.run(): - Read operation on 
> socket failed
> java.io.IOException: Peer closed socket
>         at 
> org.jgroups.blocks.ConnectionTableNIO$ReadHandler.readHeader(ConnectionTableNIO.java:746)
>         at 
> org.jgroups.blocks.ConnectionTableNIO$ReadHandler.readOnce(ConnectionTableNIO.java:699)
>         at 
> org.jgroups.blocks.ConnectionTableNIO$ReadHandler.run(ConnectionTableNIO.java:638)
>         at java.lang.Thread.run(Thread.java:595)
> 268815 [WARN] ConnectionTableNIO$ReadHandler.run(): - Read operation on 
> socket failed
> java.io.IOException: Peer closed socket
>         at 
> org.jgroups.blocks.ConnectionTableNIO$ReadHandler.readHeader(ConnectionTableNIO.java:746)
>         at 
> org.jgroups.blocks.ConnectionTableNIO$ReadHandler.readOnce(ConnectionTableNIO.java:699)
>         at 
> org.jgroups.blocks.ConnectionTableNIO$ReadHandler.run(ConnectionTableNIO.java:638)
>         at java.lang.Thread.run(Thread.java:595)
> java.lang.NullPointerException
>         at 
> org.jgroups.blocks.BasicConnectionTable$Connection.getSockAddress(BasicConnectionTable.java:267)
>         at 
> org.jgroups.blocks.BasicConnectionTable$Connection.access$100(BasicConnectionTable.java:246)
>         at 
> org.jgroups.blocks.BasicConnectionTable$Connection$Sender.start(BasicConnectionTable.java:591)
>         at 
> org.jgroups.blocks.BasicConnectionTable$Connection.send(BasicConnectionTable.java:345)
>         at 
> org.jgroups.blocks.BasicConnectionTable.send(BasicConnectionTable.java:215)
>         at org.jgroups.protocols.TCP_NIO.send(TCP_NIO.java:50)
>         at org.jgroups.protocols.BasicTCP.sendToSingleMember(BasicTCP.java:61)
>         at org.jgroups.protocols.TP.doSend(TP.java:950)
>         at org.jgroups.protocols.TP.access$2100(TP.java:45)
>         at org.jgroups.protocols.TP$Bundler.bundleAndSend(TP.java:1662)
>         at org.jgroups.protocols.TP$Bundler.send(TP.java:1585)
>         at org.jgroups.protocols.TP$Bundler.access$100(TP.java:1568)
>         at org.jgroups.protocols.TP.send(TP.java:925)
>         at org.jgroups.protocols.TP.down(TP.java:725)
>         at org.jgroups.stack.Protocol.receiveDownEvent(Protocol.java:517)
>         at org.jgroups.stack.Protocol.passDown(Protocol.java:551)
>         at org.jgroups.protocols.Discovery.down(Discovery.java:333)
>         at org.jgroups.stack.Protocol.receiveDownEvent(Protocol.java:517)
>         at org.jgroups.stack.Protocol.passDown(Protocol.java:551)
>         at org.jgroups.protocols.MERGE2.down(MERGE2.java:208)
>         at org.jgroups.stack.Protocol.receiveDownEvent(Protocol.java:517)
>         at org.jgroups.stack.Protocol.passDown(Protocol.java:551)
>         at org.jgroups.protocols.FD_SOCK.down(FD_SOCK.java:407)
>         at org.jgroups.stack.Protocol.receiveDownEvent(Protocol.java:517)
>         at org.jgroups.stack.Protocol.passDown(Protocol.java:551)
>         at org.jgroups.protocols.FD.down(FD.java:313)
>         at org.jgroups.stack.Protocol.receiveDownEvent(Protocol.java:517)
>         at org.jgroups.stack.Protocol.passDown(Protocol.java:551)
>         at org.jgroups.stack.Protocol.down(Protocol.java:577)
>         at org.jgroups.stack.Protocol.receiveDownEvent(Protocol.java:517)
>         at org.jgroups.stack.Protocol.passDown(Protocol.java:551)
>         at org.jgroups.protocols.pbcast.NAKACK.send(NAKACK.java:621)
>         at org.jgroups.protocols.pbcast.NAKACK.down(NAKACK.java:409)
>         at org.jgroups.stack.Protocol.receiveDownEvent(Protocol.java:517)
>         at org.jgroups.stack.Protocol.passDown(Protocol.java:551)
>         at org.jgroups.protocols.pbcast.STABLE.down(STABLE.java:283)
>         at org.jgroups.stack.Protocol.receiveDownEvent(Protocol.java:517)
>         at org.jgroups.stack.Protocol.passDown(Protocol.java:551)
>         at org.jgroups.protocols.pbcast.GMS.down(GMS.java:809)
>         at org.jgroups.stack.Protocol.receiveDownEvent(Protocol.java:517)
>         at org.jgroups.stack.Protocol.passDown(Protocol.java:551)
>         at org.jgroups.protocols.FC.handleDownMessage(FC.java:383)
>         at org.jgroups.protocols.FC.down(FC.java:300)
>         at org.jgroups.stack.Protocol.receiveDownEvent(Protocol.java:517)
>         at org.jgroups.protocols.FC.receiveDownEvent(FC.java:294)
>         at org.jgroups.stack.Protocol.passDown(Protocol.java:551)
>         at org.jgroups.protocols.FRAG2.down(FRAG2.java:167)
>         at org.jgroups.stack.Protocol.receiveDownEvent(Protocol.java:517)
>         at org.jgroups.stack.Protocol.passDown(Protocol.java:551)
>         at 
> org.jgroups.protocols.pbcast.STREAMING_STATE_TRANSFER.down(STREAMING_STATE_TRANSFER.java:376)
>         at org.jgroups.stack.Protocol.receiveDownEvent(Protocol.java:517)
>         at org.jgroups.stack.Protocol.passDown(Protocol.java:551)
>         at org.jgroups.protocols.pbcast.FLUSH.down(FLUSH.java:179)
>         at org.jgroups.stack.Protocol.receiveDownEvent(Protocol.java:517)
>         at org.jgroups.stack.ProtocolStack.down(ProtocolStack.java:385)
>         at org.jgroups.JChannel.down(JChannel.java:1175)
>         at org.jgroups.JChannel.send(JChannel.java:584)
>         at 
> org.jgroups.tests.perf.transports.JGroupsTransport.send(JGroupsTransport.java:83)
>         at org.jgroups.tests.perf.Test.sendMessages(Test.java:401)
>         at org.jgroups.tests.perf.Test.main(Test.java:681)
> -- received 5917000000 messages
> 297628 [WARN] ConnectionTableNIO$ReadHandler.run(): - Read operation on 
> socket failed
> java.io.IOException: Peer closed socket
>         at 
> org.jgroups.blocks.ConnectionTableNIO$ReadHandler.readHeader(ConnectionTableNIO.java:746)
>         at 
> org.jgroups.blocks.ConnectionTableNIO$ReadHandler.readOnce(ConnectionTableNIO.java:699)
>         at 
> org.jgroups.blocks.ConnectionTableNIO$ReadHandler.run(ConnectionTableNIO.java:638)
>         at java.lang.Thread.run(Thread.java:595)
> 298623 [WARN] ConnectionTableNIO$ReadHandler.run(): - Read operation on 
> socket failed
> java.io.IOException: Peer closed socket
>         at 
> org.jgroups.blocks.ConnectionTableNIO$ReadHandler.readHeader(ConnectionTableNIO.java:746)
>         at 
> org.jgroups.blocks.ConnectionTableNIO$ReadHandler.readOnce(ConnectionTableNIO.java:699)
>         at 
> org.jgroups.blocks.ConnectionTableNIO$ReadHandler.run(ConnectionTableNIO.java:638)
>         at java.lang.Thread.run(Thread.java:595)
> ++ sent 2000000000
> 351184 [WARN] ConnectionTableNIO$ReadHandler.run(): - Read operation on 
> socket failed
> java.io.IOException: Peer closed socket
>         at 
> org.jgroups.blocks.ConnectionTableNIO$ReadHandler.readHeader(ConnectionTableNIO.java:746)
>         at 
> org.jgroups.blocks.ConnectionTableNIO$ReadHandler.readOnce(ConnectionTableNIO.java:699)
>         at 
> org.jgroups.blocks.ConnectionTableNIO$ReadHandler.run(ConnectionTableNIO.java:638)
>         at java.lang.Thread.run(Thread.java:595)
> 351185 [ERROR] BasicConnectionTable$Connection._send(): - exception is 
> java.lang.reflect.InvocationTargetException
> 352184 [WARN] ConnectionTableNIO$ReadHandler.run(): - Read operation on 
> socket failed
> java.io.IOException: Peer closed socket
>         at 
> org.jgroups.blocks.ConnectionTableNIO$ReadHandler.readHeader(ConnectionTableNIO.java:746)
>         at 
> org.jgroups.blocks.ConnectionTableNIO$ReadHandler.readOnce(ConnectionTableNIO.java:699)
>         at 
> org.jgroups.blocks.ConnectionTableNIO$ReadHandler.run(ConnectionTableNIO.java:638)
>         at java.lang.Thread.run(Thread.java:595)
> 375125 [WARN] ConnectionTableNIO$ReadHandler.run(): - Read operation on 
> socket failed
> java.io.IOException: Peer closed socket
>         at 
> org.jgroups.blocks.ConnectionTableNIO$ReadHandler.readHeader(ConnectionTableNIO.java:746)
>         at 
> org.jgroups.blocks.ConnectionTableNIO$ReadHandler.readOnce(ConnectionTableNIO.java:699)
>         at 
> org.jgroups.blocks.ConnectionTableNIO$ReadHandler.run(ConnectionTableNIO.java:638)
>         at java.lang.Thread.run(Thread.java:595)
> 376135 [WARN] ConnectionTableNIO$ReadHandler.run(): - Read operation on 
> socket failed
> java.io.IOException: Peer closed socket
>         at 
> org.jgroups.blocks.ConnectionTableNIO$ReadHandler.readHeader(ConnectionTableNIO.java:746)
>         at 
> org.jgroups.blocks.ConnectionTableNIO$ReadHandler.readOnce(ConnectionTableNIO.java:699)
>         at 
> org.jgroups.blocks.ConnectionTableNIO$ReadHandler.run(ConnectionTableNIO.java:638)
>         at java.lang.Thread.run(Thread.java:595)
> 462060 [WARN] CoordGmsImpl$MergeTask.run(): - merge responses from subgroup 
> coordinators <= 1 ([sender=10.50.3.37:7800, view=[10.50.3.37:7800|3] 
> [10.50.3.37:7800, 10.50.3.40
> :7800], digest=10.50.3.40:7800: [2000873580 : 2000873581], 10.50.3.37:7800: 
> [2000866427 : 2000866428]]). Cancelling merge
> 560428 [ERROR] CoordGmsImpl.sendMergeCancelledMessage(): - coords or merge_id 
> == null
> 560428 [ERROR] CoordGmsImpl.sendMergeCancelledMessage(): - coords or merge_id 
> == null
> 604571 [ERROR] CoordGmsImpl.sendMergeCancelledMessage(): - coords or merge_id 
> == null
> 604571 [ERROR] CoordGmsImpl.sendMergeCancelledMessage(): - coords or merge_id 
> == null
> 675726 [WARN] CoordGmsImpl$MergeTask.run(): - merge responses from subgroup 
> coordinators <= 1 ([]). Cancelling merge
> 675727 [ERROR] CoordGmsImpl.sendMergeCancelledMessage(): - coords or merge_id 
> == null
> 675727 [ERROR] CoordGmsImpl.sendMergeCancelledMessage(): - coords or merge_id 
> == null
> 705729 [ERROR] CoordGmsImpl.sendMergeCancelledMessage(): - coords or merge_id 
> == null
> 705729 [ERROR] CoordGmsImpl.sendMergeCancelledMessage(): - coords or merge_id 
> == null
> 792835 [WARN] CoordGmsImpl$MergeTask.run(): - merge responses from subgroup 
> coordinators <= 1 ([sender=10.50.3.37:7800, view=[10.50.3.37:7800|3] 
> [10.50.3.37:7800, 10.50.3.40
> :7800], digest=10.50.3.40:7800: [2000873585 : 2000873586], 10.50.3.37:7800: 
> [2000866432 : 2000866433]]). Cancelling merge
> 823658 [WARN] CoordGmsImpl$MergeTask.run(): - merge responses from subgroup 
> coordinators <= 1 ([]). Cancelling merge
> 823659 [ERROR] CoordGmsImpl.sendMergeCancelledMessage(): - coords or merge_id 
> == null
> 823659 [ERROR] CoordGmsImpl.sendMergeCancelledMessage(): - coords or merge_id 
> == null
> 859981 [WARN] CoordGmsImpl$MergeTask.run(): - merge responses from subgroup 
> coordinators <= 1 ([]). Cancelling merge
> 859982 [ERROR] CoordGmsImpl.sendMergeCancelledMessage(): - coords or merge_id 
> == null
> 859982 [ERROR] CoordGmsImpl.sendMergeCancelledMessage(): - coords or merge_id 
> == null
> 932777 [WARN] CoordGmsImpl$MergeTask.run(): - merge responses from subgroup 
> coordinators <= 1 ([]). Cancelling merge
> 932778 [ERROR] CoordGmsImpl.sendMergeCancelledMessage(): - coords or merge_id 
> == null
> 932779 [ERROR] CoordGmsImpl.sendMergeCancelledMessage(): - coords or merge_id 
> == null
> 1030117 [WARN] CoordGmsImpl$MergeTask.run(): - merge responses from subgroup 
> coordinators <= 1 ([]). Cancelling merge
> 1030117 [ERROR] CoordGmsImpl.sendMergeCancelledMessage(): - coords or 
> merge_id == null
> 1030118 [ERROR] CoordGmsImpl.sendMergeCancelledMessage(): - coords or 
> merge_id == null
> 1059809 [WARN] CoordGmsImpl$MergeTask.run(): - merge responses from subgroup 
> coordinators <= 1 ([]). Cancelling merge
> 1059810 [ERROR] CoordGmsImpl.sendMergeCancelledMessage(): - coords or 
> merge_id == null
> 1059811 [ERROR] CoordGmsImpl.sendMergeCancelledMessage(): - coords or 
> merge_id == null
> 1097613 [WARN] CoordGmsImpl$MergeTask.run(): - merge responses from subgroup 
> coordinators <= 1 ([]). Cancelling merge
> 1097614 [ERROR] CoordGmsImpl.sendMergeCancelledMessage(): - coords or 
> merge_id == null
> 1097614 [ERROR] CoordGmsImpl.sendMergeCancelledMessage(): - coords or 
> merge_id == null
> 1190893 [ERROR] CoordGmsImpl.sendMergeCancelledMessage(): - coords or 
> merge_id == null
> 1190893 [ERROR] CoordGmsImpl.sendMergeCancelledMessage(): - coords or 
> merge_id == null
> 1216565 [WARN] CoordGmsImpl$MergeTask.run(): - merge responses from subgroup 
> coordinators <= 1 ([]). Cancelling merge
> 1216565 [ERROR] CoordGmsImpl.sendMergeCancelledMessage(): - coords or 
> merge_id == null
> 1216566 [ERROR] CoordGmsImpl.sendMergeCancelledMessage(): - coords or 
> merge_id == null
>
> <...snip...>
>
> 32420889 [WARN] CoordGmsImpl$MergeTask.run(): - merge responses from subgroup 
> coordinators <= 1 ([]). Cancelling merge
> 32420889 [ERROR] CoordGmsImpl.sendMergeCancelledMessage(): - coords or 
> merge_id == null
> 32420889 [ERROR] CoordGmsImpl.sendMergeCancelledMessage(): - coords or 
> merge_id == null
> 32496669 [WARN] CoordGmsImpl$MergeTask.run(): - merge responses from subgroup 
> coordinators <= 1 ([sender=10.50.3.37:7800, view=[10.50.3.37:7800|3] 
> [10.50.3.37:7800, 10.50.3.
> 40:7800], digest=10.50.3.40:7800: [2000874046 : 2000874047], 10.50.3.37:7800: 
> [2000866933 : 2000866934]]). Cancelling merge
> 32590291 [WARN] CoordGmsImpl$MergeTask.run(): - merge responses from subgroup 
> coordinators <= 1 ([]). Cancelling merge
> 32590291 [ERROR] CoordGmsImpl.sendMergeCancelledMessage(): - coords or 
> merge_id == null
> 32590292 [ERROR] CoordGmsImpl.sendMergeCancelledMessage(): - coords or 
> merge_id == null
> 32669710 [WARN] CoordGmsImpl$MergeTask.run(): - merge responses from subgroup 
> coordinators <= 1 ([sender=10.50.3.37:7800, view=[10.50.3.37:7800|3] 
> [10.50.3.37:7800, 10.50.3.
> 40:7800], digest=10.50.3.40:7800: [2000874048 : 2000874049], 10.50.3.37:7800: 
> [2000866937 : 2000866938]]). Cancelling merge
> 32726946 [WARN] CoordGmsImpl$MergeTask.run(): - merge responses from subgroup 
> coordinators <= 1 ([sender=10.50.3.37:7800, view=[10.50.3.37:7800|3] 
> [10.50.3.37:7800, 10.50.3.
> 40:7800], digest=10.50.3.40:7800: [2000874048 : 2000874049], 10.50.3.37:7800: 
> [2000866937 : 2000866938]]). Cancelling merge
> 32763481 [WARN] CoordGmsImpl$MergeTask.run(): - merge responses from subgroup 
> coordinators <= 1 ([sender=10.50.3.37:7800, view=[10.50.3.37:7800|3] 
> [10.50.3.37:7800, 10.50.3.
> 40:7800], digest=10.50.3.40:7800: [2000874050 : 2000874051], 10.50.3.37:7800: 
> [2000866938 : 2000866939]]). Cancelling merge
> 32832678 [WARN] CoordGmsImpl$MergeTask.run(): - merge responses from subgroup 
> coordinators <= 1 ([]). Cancelling merge
> 32832678 [ERROR] CoordGmsImpl.sendMergeCancelledMessage(): - coords or 
> merge_id == null
> 32832679 [ERROR] CoordGmsImpl.sendMergeCancelledMessage(): - coords or 
> merge_id == null
> 32866307 [WARN] ConnectionTableNIO$ReadHandler.run(): - Read operation on 
> socket failed
> java.io.IOException: Peer closed socket
>         at 
> org.jgroups.blocks.ConnectionTableNIO$ReadHandler.readHeader(ConnectionTableNIO.java:746)
>         at 
> org.jgroups.blocks.ConnectionTableNIO$ReadHandler.readOnce(ConnectionTableNIO.java:699)
>         at 
> org.jgroups.blocks.ConnectionTableNIO$ReadHandler.run(ConnectionTableNIO.java:638)
>         at java.lang.Thread.run(Thread.java:595)
> 32871815 [WARN] ConnectionTableNIO$ReadHandler.run(): - Read operation on 
> socket failed
> java.io.IOException: Peer closed socket
>         at 
> org.jgroups.blocks.ConnectionTableNIO$ReadHandler.readHeader(ConnectionTableNIO.java:746)
>         at 
> org.jgroups.blocks.ConnectionTableNIO$ReadHandler.readOnce(ConnectionTableNIO.java:699)
>         at 
> org.jgroups.blocks.ConnectionTableNIO$ReadHandler.run(ConnectionTableNIO.java:638)
>         at java.lang.Thread.run(Thread.java:595)
> 32872824 [WARN] ConnectionTableNIO$ReadHandler.run(): - Read operation on 
> socket failed
> java.io.IOException: Peer closed socket
>         at 
> org.jgroups.blocks.ConnectionTableNIO$ReadHandler.readHeader(ConnectionTableNIO.java:746)
>         at 
> org.jgroups.blocks.ConnectionTableNIO$ReadHandler.readOnce(ConnectionTableNIO.java:699)
>         at 
> org.jgroups.blocks.ConnectionTableNIO$ReadHandler.run(ConnectionTableNIO.java:638)
>         at java.lang.Thread.run(Thread.java:595)
> 32881908 [WARN] ConnectionTableNIO$ReadHandler.run(): - Read operation on 
> socket failed
> java.io.IOException: Peer closed socket
>         at 
> org.jgroups.blocks.ConnectionTableNIO$ReadHandler.readHeader(ConnectionTableNIO.java:746)
>         at 
> org.jgroups.blocks.ConnectionTableNIO$ReadHandler.readOnce(ConnectionTableNIO.java:699)
>         at 
> org.jgroups.blocks.ConnectionTableNIO$ReadHandler.run(ConnectionTableNIO.java:638)
>         at java.lang.Thread.run(Thread.java:595)
> 32881909 [WARN] ConnectionTableNIO$ReadHandler.run(): - Read operation on 
> socket failed
> java.io.IOException: Peer closed socket
>         at 
> org.jgroups.blocks.ConnectionTableNIO$ReadHandler.readHeader(ConnectionTableNIO.java:746)
>         at 
> org.jgroups.blocks.ConnectionTableNIO$ReadHandler.readOnce(ConnectionTableNIO.java:699)
>         at 
> org.jgroups.blocks.ConnectionTableNIO$ReadHandler.run(ConnectionTableNIO.java:638)
>         at java.lang.Thread.run(Thread.java:595)
> 32893054 [WARN] CoordGmsImpl$MergeTask.run(): - merge responses from subgroup 
> coordinators <= 1 ([]). Cancelling merge
> 32893055 [ERROR] CoordGmsImpl.sendMergeCancelledMessage(): - coords or 
> merge_id == null
> 32893055 [ERROR] CoordGmsImpl.sendMergeCancelledMessage(): - coords or 
> merge_id == null
>
> Host: oster
> -------------------
>
> Configuration is: 
>
> ----------------------- TEST -----------------------
> Date: Fri Sep 15 10:39:39 MSD 2006
> Run by: comfw
>
> log_interval:   1000000
> msg_size:       256
> config: /home/comfw/public/perf_tests/config/config.txt
> transport:      org.jgroups.tests.perf.transports.JGroupsTransport
> num_senders:    3
> sender: true
> start_port:     7800
> props:  /home/comfw/public/perf_tests/hosts/oster/protocol.xml
> dump_transport_stats:   false
> num_msgs:       2000000000
> processing_delay:       0
> num_members:    3
> jmx:    false
> JGroups version: 2.4CR2
>
>
> -------------------------------------------------------
> GMS: address is 10.50.3.40:7800
> -------------------------------------------------------
> -- waiting for 3 members to join
> -- 10.50.3.37:7800 joined
> -- 10.50.3.40:7800 joined
> -- 10.50.3.38:7800 joined
> -- members: 3
> -- sending 2000000000 256b messages
>
> <...snip...>
>
> ++ sent 1999000000
> -- received 5324000000 messages
> -- received 5325000000 messages
> ++ sent 2000000000
>
> <...snip...>
>
> -- received 5908000000 messages
> -- received 5909000000 messages
> -- received 5910000000 messages
> -- received 5911000000 messages
> 0 [WARN] FD.up(): - I was suspected by 10.50.3.38:7800; ignoring the SUSPECT 
> message and sending back a HEARTBEAT_ACK
> 11279 [WARN] FD.up(): - I was suspected by 10.50.3.38:7800; ignoring the 
> SUSPECT message and sending back a HEARTBEAT_ACK
> 22229 [WARN] FD.up(): - I was suspected by 10.50.3.38:7800; ignoring the 
> SUSPECT message and sending back a HEARTBEAT_ACK
> 32409 [WARN] FD.up(): - I was suspected by 10.50.3.38:7800; ignoring the 
> SUSPECT message and sending back a HEARTBEAT_ACK
> 42749 [WARN] FD.up(): - I was suspected by 10.50.3.38:7800; ignoring the 
> SUSPECT message and sending back a HEARTBEAT_ACK
> 52759 [WARN] FD.up(): - I was suspected by 10.50.3.38:7800; ignoring the 
> SUSPECT message and sending back a HEARTBEAT_ACK
> 65069 [WARN] FD.up(): - I was suspected by 10.50.3.38:7800; ignoring the 
> SUSPECT message and sending back a HEARTBEAT_ACK
> 65399 [WARN] ConnectionTableNIO.run(): - 10.50.3.38:7800 is already there, 
> will terminate connection
> 73429 [WARN] ConnectionTableNIO.run(): - 10.50.3.38:7800 is already there, 
> will terminate connection
> -- received 5912000000 messages
> 160725 [WARN] ConnectionTableNIO$ReadHandler.run(): - Read operation on 
> socket failed
> java.io.IOException: Peer closed socket
>         at 
> org.jgroups.blocks.ConnectionTableNIO$ReadHandler.readHeader(ConnectionTableNIO.java:746)
>         at 
> org.jgroups.blocks.ConnectionTableNIO$ReadHandler.readOnce(ConnectionTableNIO.java:699)
>         at 
> org.jgroups.blocks.ConnectionTableNIO$ReadHandler.run(ConnectionTableNIO.java:638)
>         at java.lang.Thread.run(Thread.java:595)
> -- received 5913000000 messages
> -- received 5914000000 messages
> 230539 [WARN] ConnectionTableNIO$ReadHandler.run(): - Read operation on 
> socket failed
> java.io.IOException: Peer closed socket
>         at 
> org.jgroups.blocks.ConnectionTableNIO$ReadHandler.readHeader(ConnectionTableNIO.java:746)
>         at 
> org.jgroups.blocks.ConnectionTableNIO$ReadHandler.readOnce(ConnectionTableNIO.java:699)
>         at 
> org.jgroups.blocks.ConnectionTableNIO$ReadHandler.run(ConnectionTableNIO.java:638)
>         at java.lang.Thread.run(Thread.java:595)
> 230571 [WARN] ConnectionTableNIO$ReadHandler.run(): - Read operation on 
> socket failed
> java.io.IOException: Peer closed socket
>         at 
> org.jgroups.blocks.ConnectionTableNIO$ReadHandler.readHeader(ConnectionTableNIO.java:746)
>         at 
> org.jgroups.blocks.ConnectionTableNIO$ReadHandler.readOnce(ConnectionTableNIO.java:699)
>         at 
> org.jgroups.blocks.ConnectionTableNIO$ReadHandler.run(ConnectionTableNIO.java:638)
>         at java.lang.Thread.run(Thread.java:595)
> -- received 5915000000 messages
> 290811 [WARN] ConnectionTableNIO$ReadHandler.run(): - Read operation on 
> socket failed
> java.io.IOException: Peer closed socket
>         at 
> org.jgroups.blocks.ConnectionTableNIO$ReadHandler.readHeader(ConnectionTableNIO.java:746)
>         at 
> org.jgroups.blocks.ConnectionTableNIO$ReadHandler.readOnce(ConnectionTableNIO.java:699)
>         at 
> org.jgroups.blocks.ConnectionTableNIO$ReadHandler.run(ConnectionTableNIO.java:638)
>         at java.lang.Thread.run(Thread.java:595)
> 291821 [WARN] ConnectionTableNIO$ReadHandler.run(): - Read operation on 
> socket failed
> java.io.IOException: Peer closed socket
>         at 
> org.jgroups.blocks.ConnectionTableNIO$ReadHandler.readHeader(ConnectionTableNIO.java:746)
>         at 
> org.jgroups.blocks.ConnectionTableNIO$ReadHandler.readOnce(ConnectionTableNIO.java:699)
>         at 
> org.jgroups.blocks.ConnectionTableNIO$ReadHandler.run(ConnectionTableNIO.java:638)
>         at java.lang.Thread.run(Thread.java:595)
> -- received 5916000000 messages
> 331152 [WARN] ConnectionTableNIO$ReadHandler.run(): - Read operation on 
> socket failed
> java.io.IOException: Peer closed socket
>         at 
> org.jgroups.blocks.ConnectionTableNIO$ReadHandler.readHeader(ConnectionTableNIO.java:746)
>         at 
> org.jgroups.blocks.ConnectionTableNIO$ReadHandler.readOnce(ConnectionTableNIO.java:699)
>         at 
> org.jgroups.blocks.ConnectionTableNIO$ReadHandler.run(ConnectionTableNIO.java:638)
>         at java.lang.Thread.run(Thread.java:595)
> 332171 [WARN] ConnectionTableNIO$ReadHandler.run(): - Read operation on 
> socket failed
> java.io.IOException: Peer closed socket
>         at 
> org.jgroups.blocks.ConnectionTableNIO$ReadHandler.readHeader(ConnectionTableNIO.java:746)
>         at 
> org.jgroups.blocks.ConnectionTableNIO$ReadHandler.readOnce(ConnectionTableNIO.java:699)
>         at 
> org.jgroups.blocks.ConnectionTableNIO$ReadHandler.run(ConnectionTableNIO.java:638)
>         at java.lang.Thread.run(Thread.java:595)
> -- received 5917000000 messages
> 360981 [WARN] ConnectionTableNIO$ReadHandler.run(): - Read operation on 
> socket failed
> java.io.IOException: Peer closed socket
>         at 
> org.jgroups.blocks.ConnectionTableNIO$ReadHandler.readHeader(ConnectionTableNIO.java:746)
>         at 
> org.jgroups.blocks.ConnectionTableNIO$ReadHandler.readOnce(ConnectionTableNIO.java:699)
>         at 
> org.jgroups.blocks.ConnectionTableNIO$ReadHandler.run(ConnectionTableNIO.java:638)
>         at java.lang.Thread.run(Thread.java:595)
> 361991 [WARN] ConnectionTableNIO$ReadHandler.run(): - Read operation on 
> socket failed
> java.io.IOException: Peer closed socket
>         at 
> org.jgroups.blocks.ConnectionTableNIO$ReadHandler.readHeader(ConnectionTableNIO.java:746)
>         at 
> org.jgroups.blocks.ConnectionTableNIO$ReadHandler.readOnce(ConnectionTableNIO.java:699)
>         at 
> org.jgroups.blocks.ConnectionTableNIO$ReadHandler.run(ConnectionTableNIO.java:638)
>         at java.lang.Thread.run(Thread.java:595)
> 414511 [WARN] ConnectionTableNIO$ReadHandler.run(): - Read operation on 
> socket failed
> java.io.IOException: Peer closed socket
>         at 
> org.jgroups.blocks.ConnectionTableNIO$ReadHandler.readHeader(ConnectionTableNIO.java:746)
>         at 
> org.jgroups.blocks.ConnectionTableNIO$ReadHandler.readOnce(ConnectionTableNIO.java:699)
>         at 
> org.jgroups.blocks.ConnectionTableNIO$ReadHandler.run(ConnectionTableNIO.java:638)
>         at java.lang.Thread.run(Thread.java:595)
> 415521 [WARN] ConnectionTableNIO$ReadHandler.run(): - Read operation on 
> socket failed
> java.io.IOException: Peer closed socket
>         at 
> org.jgroups.blocks.ConnectionTableNIO$ReadHandler.readHeader(ConnectionTableNIO.java:746)
>         at 
> org.jgroups.blocks.ConnectionTableNIO$ReadHandler.readOnce(ConnectionTableNIO.java:699)
>         at 
> org.jgroups.blocks.ConnectionTableNIO$ReadHandler.run(ConnectionTableNIO.java:638)
>         at java.lang.Thread.run(Thread.java:595)
>
>
> thanks iand regards
>       gianluca
>
>   

-- 
Bela Ban
Lead JGroups / Manager JBoss Clustering Group
JBoss - a division of Red Hat


-------------------------------------------------------------------------
Take Surveys. Earn Cash. Influence the Future of IT
Join SourceForge.net's Techsay panel and you'll get the chance to share your
opinions on IT & business topics through brief surveys -- and earn cash
http://www.techsay.com/default.php?page=join.php&p=sourceforge&CID=DEVDEV