osdir.com


[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Kahadb Activemq holds the lock but does start the broker service or connectors.


Hi,

We had shared file locker with kaha db based persistence adapter in our 3
node master-slave-slave architecture. 
Master activemq was restarted couple of time and after a cycle or so it was
seen that even though the activemq  java process is up and it holds the lock
it doesn't start  the broker service and open the transports. 

There are no errors in the activemq logs on master nodes. 

The other two slave nodes continue to be slaves since the master node
doesn't release the lock. 

The query command on activemq doesn't come out and hangs:

[root@vsd-sudhar-set1-node1 bin]# ./activemq query
INFO: Loading '/opt/vsd/amq//bin/env'
INFO: Using java '/usr/java/latest/bin/java'
Java Runtime: Oracle Corporation 1.8.0_172 /usr/java/jre1.8.0_172-amd64
  Heap sizes: current=1005056k  free=994570k  max=1005056k
    JVM args: -Xms1G -Xmx1G
-Djava.util.logging.config.file=logging.properties
-Djava.security.auth.login.config=/opt/vsd/amq/conf/login.config
-Dorg.apache.activemq.SERIALIZABLE_PACKAGES=* -Dvsd.rest.http.retries=5
-Dvsd.rest.http.retry.interval=400
-Dvsd.host.addr=vsd-sudhar-set1-node1.mv.nuagenetworks.net,vsd-sudhar-set1-node2.mv.nuagenetworks.net,vsd-sudhar-set1-node3.mv.nuagenetworks.net
-Dactivemq.classpath=/opt/vsd/amq/conf:/opt/vsd/amq//../lib/:
-Dactivemq.home=/opt/vsd/amq -Dactivemq.base=/opt/vsd/amq
-Dactivemq.conf=/opt/vsd/amq/conf -Dactivemq.data=/opt/vsd/amq/data
Extensions classpath:
 
[/opt/vsd/amq/lib,/opt/vsd/amq/lib/camel,/opt/vsd/amq/lib/optional,/opt/vsd/amq/lib/web,/opt/vsd/amq/lib/extra]
ACTIVEMQ_HOME: /opt/vsd/amq
ACTIVEMQ_BASE: /opt/vsd/amq
ACTIVEMQ_CONF: /opt/vsd/amq/conf
ACTIVEMQ_DATA: /opt/vsd/amq/data
Connecting to JMX URL: service:jmx:rmi:///jndi/rmi://localhost:1099/jmxrmi

But on slave nodes same command is successful. 

Process status:
root      5638     1  0 Jun25 ?        00:00:27 /usr/java/latest/bin/java
-Xms1G -Xmx1G -Djava.util.logging.config.file=logging.properties
-Djava.security.auth.login.config=/opt/vsd/amq/conf/login.config
-Dorg.apache.activemq.SERIALIZABLE_PACKAGES=* -Dvsd.rest.http.retries=5
-Dvsd.rest.http.retry.interval=400
-Dvsd.host.addr=vsd-sudhar-set1-node1.mv.nuagenetworks.net,vsd-sudhar-set1-node2.mv.nuagenetworks.net,vsd-sudhar-set1-node3.mv.nuagenetworks.net
-Dcom.sun.management.jmxremote -Djava.awt.headless=true
-Djava.io.tmpdir=/opt/vsd/amq/tmp -Xdebug -Xnoagent -Djava.compiler=NONE
-Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=5008
-Dactivemq.classpath=/opt/vsd/amq/conf:/opt/vsd/amq//../lib/:
-Dactivemq.home=/opt/vsd/amq -Dactivemq.base=/opt/vsd/amq
-Dactivemq.conf=/opt/vsd/amq/conf -Dactivemq.data=/opt/vsd/amq/data -jar
/opt/vsd/amq/bin/activemq.jar start

Lock held by same process:
[root@vsd-sudhar-set1-node1 bin]# lsof /opt/vsd/amq/data/database/lock 
COMMAND  PID USER   FD   TYPE DEVICE SIZE/OFF       NODE NAME
java    5638 root  133u   REG   0,44        8 2152124800
/opt/vsd/amq/data/database/lock

I have attached the thread dump and activemq log of the master node. 

Can someone help why the broker is not comming up?

threadDump1.txt
<http://activemq.2283324.n4.nabble.com/file/t379043/threadDump1.txt>  
activemq.log
<http://activemq.2283324.n4.nabble.com/file/t379043/activemq.log>  




--
Sent from: http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html