osdir.com


[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: Status of Demo Server on Apache VM Infrastructure


Hi guys,

I am assuming that the suggested form of development for Fineract CN is to
write unit tests to test the code which you just wrote locally and then
push the code to a remote cluster or a massive VM and wait 40+ minutes to
run the API / integration tests there. A lot of the confusion regarding the
stacktraces in identity, rhythm, timeouts and auto-shutdown might or might
not have anything to do with the resource crunch in my local. (i7, 8GB, SSD)

Maybe a FAQ section in "How to build Fineract CN" to help devs with local
setup would go a long way to bring in contributions to the core modules.

1. Change demo server orchestration logic from running Identity,
Organization, Customer, Accounting, Portfolio, Deposit, Teller, Reporting,
Cheque, Payroll, Group and Notification services to running only Identity,
Organization, Customer and working set of your choice (Portfolio and
Accounting OR Deposit and Accounting OR Group and Portfolio)

2. Leave out Rhythm from orchestration logic for now. A JIRA ticket
tracking ability of Rhythm to run with low resource footprint will probably
help.


@Myrle, can you elaborate on FINCN-25?  Is the motivation of this ticket to
reduce resource footprint only or also to reduce the startup time from 40
minutes to a reasonable couple of minutes? Do you have any debugging tips
with respect to the slow startup caused by services is due to registering
with eureka or provisioning or sequential startup?

Maybe we should also follow up in Cassandra mailing list regarding
configuring Cassandra to run within 8GB of RAM and include the same config
as part of the FAQ.


Regards,
Avik.
ᐧ

On Wed, Sep 19, 2018 at 2:23 AM Courage Angeh <courageangeh@xxxxxxxxx>
wrote:

> Thanks for the update. Ed.
>
> If you need an extra hand please let me know Aleks.
>
> On Tue, Sep 18, 2018, 4:27 PM Ed Cable <edcable@xxxxxxxxx> wrote:
>
> > Hi all,
> >
> > Just to bring this thread back up to top of everyone's inbox. Aleks
> > estimates he has about a day left to get the demo server up and running
> is
> > hoping to tackle it by tomorrow. Here are the two approaches he suggested
> > now that we have the second VM allocated:
> >
> > - continue with the JUnit based startup sequence of the demo module; the
> > one thing that I would need to change is to split up the startup sequence
> > and recompile
> > - create a set of Docker Compose files to do the startup on the 2 VM
> > instances (maybe I could borrow some stuff from Courage)
> >
> > I would prefer the latter, because you have more fine grained control and
> > it's easier move service instances around... and you don't have to
> > recompile anything.
> >
> > Ed
> >
> > On Tue, Jul 17, 2018 at 4:09 AM Myrle Krantz <myrle@xxxxxxxxxx> wrote:
> >
> > > Hey Aleks,
> > >
> > > I don’t see anything in that log that would explain the behavior you’re
> > > observing. Markus did most of the work on deposits. Markus? Do you have
> > any
> > > ideas?
> > >
> > > Best Regards,
> > > Myrle
> > >
> > >
> > >
> > > On Fri 13. Jul 2018 at 07:06 Aleksandar Vidakovic <
> > > cheetah@xxxxxxxxxxxxxxxxx>
> > > wrote:
> > >
> > > > @Myrle: with your proposed changes things look a bit better. I've
> > > disabled
> > > > Rhythm as you suggested and get past all microservice startups with a
> > lot
> > > > less exceptions (one is coming repeatedly from Netty native
> transport,
> > > > because epoll is not available).
> > > >
> > > > The one exception (actually it happened on 2 occasions) that is still
> > > > there:
> > > >
> > > > [snip]
> > > >
> > > > 13:33:45.080 [main] INFO
> > > o.a.f.c.d.s.DepositAccountManagementApplication -
> > > > Started DepositAccountManagementApplication in 55.187 seconds (JVM
> > > running
> > > > for 62.772)
> > > > 13:34:45.730 [AsyncResolver-bootstrap-executor-0] INFO
> > > > c.n.d.s.r.aws.ConfigClusterResolver - Resolving eureka endpoints via
> > > > configuration
> > > > Resolving artifact
> > > > org.apache.fineract.cn.teller:service-boot:jar:0.1.0-BUILD-SNAPSHOT
> > > > Resolving metadata
> > > >
> > > >
> > >
> >
> org.apache.fineract.cn.teller:service-boot:0.1.0-BUILD-SNAPSHOT/maven-metadata.xml
> > > > from /root/.m2/repository (enhanced)
> > > > Resolved metadata
> > > >
> > > >
> > >
> >
> org.apache.fineract.cn.teller:service-boot:0.1.0-BUILD-SNAPSHOT/maven-metadata.xml
> > > > from /root/.m2/repository (enhanced)
> > > > Resolved artifact
> > > > org.apache.fineract.cn.teller:service-boot:jar:0.1.0-BUILD-SNAPSHOT
> > from
> > > > /root/.m2/repository (enhanced)
> > > > 13:34:56.800 [DiscoveryClient-0] ERROR
> > c.n.discovery.TimedSupervisorTask
> > > -
> > > > task supervisor timed out
> > > > java.util.concurrent.TimeoutException: null
> > > > at java.util.concurrent.FutureTask.get(FutureTask.java:205)
> > > > at
> > > >
> > >
> >
> com.netflix.discovery.TimedSupervisorTask.run(TimedSupervisorTask.java:64)
> > > > at
> > > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> > > > at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> > > > at
> > > >
> > > >
> > >
> >
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
> > > > at
> > > >
> > > >
> > >
> >
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
> > > > at
> > > >
> > > >
> > >
> >
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> > > > at
> > > >
> > > >
> > >
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> > > > at java.lang.Thread.run(Thread.java:748)
> > > > 13:34:56.801 [DiscoveryClient-1] ERROR
> > c.n.discovery.TimedSupervisorTask
> > > -
> > > > task supervisor timed out
> > > > java.util.concurrent.TimeoutException: null
> > > > at java.util.concurrent.FutureTask.get(FutureTask.java:205)
> > > > at
> > > >
> > >
> >
> com.netflix.discovery.TimedSupervisorTask.run(TimedSupervisorTask.java:64)
> > > > at
> > > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> > > > at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> > > > at
> > > >
> > > >
> > >
> >
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
> > > > at
> > > >
> > > >
> > >
> >
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
> > > > at
> > > >
> > > >
> > >
> >
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> > > > at
> > > >
> > > >
> > >
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> > > > at java.lang.Thread.run(Thread.java:748)
> > > > 13:34:56.916 [DiscoveryClient-2] ERROR
> > c.n.discovery.TimedSupervisorTask
> > > -
> > > > task supervisor timed out
> > > > java.util.concurrent.TimeoutException: null
> > > > at java.util.concurrent.FutureTask.get(FutureTask.java:205)
> > > > at
> > > >
> > >
> >
> com.netflix.discovery.TimedSupervisorTask.run(TimedSupervisorTask.java:64)
> > > > at
> > > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> > > > at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> > > > at
> > > >
> > > >
> > >
> >
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
> > > > at
> > > >
> > > >
> > >
> >
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
> > > > at
> > > >
> > > >
> > >
> >
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> > > > at
> > > >
> > > >
> > >
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> > > > at java.lang.Thread.run(Thread.java:748)
> > > > 13:35:08.586 [main] INFO  o.s.c.a.AnnotationConfigApplicationContext
> -
> > > > Refreshing
> > > >
> > > >
> > >
> >
> org.springframework.context.annotation.AnnotationConfigApplicationContext@7dc5e7b4
> > > > :
> > > > startup date [Fri Jul 13 13:35:08 UTC 2018]; root of context
> hierarchy
> > > > 13:35:10.182 [main] INFO
> > o.s.b.f.a.AutowiredAnnotationBeanPostProcessor
> > > -
> > > > JSR-330 'javax.inject.Inject' annotation found and supported for
> > > autowiring
> > > > 13:35:10.371 [main] INFO
> > > > o.s.c.s.PostProcessorRegistrationDelegate$BeanPostProcessorChecker -
> > Bean
> > > > 'configurationPropertiesRebinderAutoConfiguration' of type [class
> > > >
> > > >
> > >
> >
> org.springframework.cloud.autoconfigure.ConfigurationPropertiesRebinderAutoConfiguration$$EnhancerBySpringCGLIB$$ffaf96f9]
> > > > is not eligible for getting processed by all BeanPostProcessors (for
> > > > example: not eligible for auto-proxying)
> > > > 13:35:10.435 [background-preinit] INFO
> > > o.h.validator.internal.util.Version
> > > > - HV000001: Hibernate Validator 5.3.0.Final
> > > >
> > > > [/snip]
> > > >
> > > > It looks like the whole startup procedure goes through and I can see
> > some
> > > > kind of "summary" that lists all REST endpoints. But it stays in that
> > > state
> > > > for only a minute or so and then a cascade of shutdowns are triggered
> > > > again. The only thing that stays alive is Eureka which complains
> about
> > > the
> > > > inaccessible services.
> > > >
> > > > Any idea what else we could do other than just ordering an additional
> > > > machine?
> > > >
> > > > Cheers,
> > > >
> > > > Aleks
> > > >
> > > > On Mon, Jun 18, 2018 at 5:42 PM Myrle Krantz <myrle@xxxxxxxxxx>
> wrote:
> > > >
> > > > > Hey Aleks,
> > > > >
> > > > > It looks like that exception is coming from Rhythm.  I suspect it's
> > not
> > > > the
> > > > > cause of your difficulties, since you say your other services are
> > going
> > > > > down too.  You can run most of the Fineract server without Rhythm.
> > You
> > > > > could edit the demo-server script to leave it out for the purposes
> of
> > > > > testing.  The consequence will be a lack of interest calculations
> for
> > > > > disbursed loans, but for finding out what the real problem is, it'd
> > be
> > > > good
> > > > > to get this one out of the way.
> > > > >
> > > > > Best Regards,
> > > > > Myrle
> > > > >
> > > > > (FYI: I've removed the non-list subscribers from the to.  Most of
> > them
> > > > are
> > > > > already subscribed, and those who aren't, don't want to be.)
> > > > >
> > > > > On Wed, Jun 13, 2018 at 1:40 AM Aleksandar Vidakovic <
> > > > > cheetah@xxxxxxxxxxxxxxxxx> wrote:
> > > > >
> > > > > > I managed to further improve the startup procedure (aka less
> > > exceptions
> > > > > > than before)... there are still some timeouts, but not as many as
> > > > before.
> > > > > >
> > > > > >  The demo server still dies after this exception:
> > > > > >
> > > > > > 23:27:41.896 [AsyncResolver-bootstrap-executor-0] INFO
> > > > > > c.n.d.s.r.aws.ConfigClusterResolver - Resolving eureka endpoints
> > via
> > > > > > configuration
> > > > > > 23:27:44.564 [AsyncResolver-bootstrap-executor-0] INFO
> > > > > > c.n.d.s.r.aws.ConfigClusterResolver - Resolving eureka endpoints
> > via
> > > > > > configuration
> > > > > > 23:28:43.669 [AsyncResolver-bootstrap-executor-0] INFO
> > > > > > c.n.d.s.r.aws.ConfigClusterResolver - Resolving eureka endpoints
> > via
> > > > > > configuration
> > > > > > 23:28:45.990 [pool-9-thread-1] ERROR
> > > > > o.s.s.s.TaskUtils$LoggingErrorHandler
> > > > > > - Unexpected error occurred in scheduled task.
> > > > > > org.springframework.transaction.CannotCreateTransactionException:
> > > Could
> > > > > not
> > > > > > open JPA EntityManager for transaction; nested exception is
> > > > > > javax.persistence.PersistenceException:
> > > > > > java.sql.SQLNonTransientConnectionException: Could not read
> > > resultset:
> > > > > > unexpected end of stream, read 0 bytes from 4
> > > > > > Query is : set autocommit=0
> > > > > > at
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.springframework.orm.jpa.JpaTransactionManager.doBegin(JpaTransactionManager.java:431)
> > > > > > at
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.springframework.transaction.support.AbstractPlatformTransactionManager.getTransaction(AbstractPlatformTransactionManager.java:373)
> > > > > > at
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.springframework.transaction.interceptor.TransactionAspectSupport.createTransactionIfNecessary(TransactionAspectSupport.java:430)
> > > > > > at
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.springframework.transaction.interceptor.TransactionAspectSupport.invokeWithinTransaction(TransactionAspectSupport.java:276)
> > > > > > at
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.springframework.transaction.interceptor.TransactionInterceptor.invoke(TransactionInterceptor.java:96)
> > > > > > at
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.springframework.aop.framework.ReflectiveMethodInvocation.proceed(ReflectiveMethodInvocation.java:179)
> > > > > > at
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.springframework.aop.framework.CglibAopProxy$DynamicAdvisedInterceptor.intercept(CglibAopProxy.java:655)
> > > > > > at
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.fineract.cn.rhythm.service.internal.service.Drummer$$EnhancerBySpringCGLIB$$70dfc8e3.checkForBeatsNeeded(<generated>)
> > > > > > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> > > > > > at
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> > > > > > at
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> > > > > > at java.lang.reflect.Method.invoke(Method.java:498)
> > > > > > at
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.springframework.scheduling.support.ScheduledMethodRunnable.run(ScheduledMethodRunnable.java:65)
> > > > > > at
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.springframework.scheduling.support.DelegatingErrorHandlingRunnable.run(DelegatingErrorHandlingRunnable.java:54)
> > > > > > at
> > > > >
> > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> > > > > > at
> java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
> > > > > > at
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
> > > > > > at
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
> > > > > > at
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> > > > > > at
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> > > > > > at java.lang.Thread.run(Thread.java:748)
> > > > > > Caused by: javax.persistence.PersistenceException:
> > > > > > java.sql.SQLNonTransientConnectionException: Could not read
> > > resultset:
> > > > > > unexpected end of stream, read 0 bytes from 4
> > > > > > Query is : set autocommit=0
> > > > > > at
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.hibernate.jpa.spi.AbstractEntityManagerImpl.convert(AbstractEntityManagerImpl.java:1692)
> > > > > > at
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.hibernate.jpa.spi.AbstractEntityManagerImpl.convert(AbstractEntityManagerImpl.java:1602)
> > > > > > at
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.hibernate.jpa.spi.AbstractEntityManagerImpl.throwPersistenceException(AbstractEntityManagerImpl.java:1700)
> > > > > > at
> > > > > >
> > > >
> > org.hibernate.jpa.internal.TransactionImpl.begin(TransactionImpl.java:48)
> > > > > > at
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.springframework.orm.jpa.vendor.HibernateJpaDialect.beginTransaction(HibernateJpaDialect.java:189)
> > > > > > at
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.springframework.orm.jpa.JpaTransactionManager.doBegin(JpaTransactionManager.java:380)
> > > > > > ... 20 common frames omitted
> > > > > > Caused by: java.sql.SQLNonTransientConnectionException: Could not
> > > read
> > > > > > resultset: unexpected end of stream, read 0 bytes from 4
> > > > > > Query is : set autocommit=0
> > > > > > at
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.mariadb.jdbc.internal.util.ExceptionMapper.get(ExceptionMapper.java:123)
> > > > > > at
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.mariadb.jdbc.internal.util.ExceptionMapper.throwException(ExceptionMapper.java:69)
> > > > > > at
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.mariadb.jdbc.MariaDbStatement.executeQueryEpilog(MariaDbStatement.java:242)
> > > > > > at
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.mariadb.jdbc.MariaDbStatement.executeInternal(MariaDbStatement.java:270)
> > > > > > at
> > > > > >
> > > > >
> > > >
> > >
> >
> org.mariadb.jdbc.MariaDbStatement.executeUpdate(MariaDbStatement.java:399)
> > > > > > at
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.mariadb.jdbc.MariaDbConnection.setAutoCommit(MariaDbConnection.java:584)
> > > > > > at
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> com.jolbox.bonecp.ConnectionHandle.setAutoCommit(ConnectionHandle.java:1292)
> > > > > > at
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.hibernate.resource.jdbc.internal.AbstractLogicalConnectionImplementor.begin(AbstractLogicalConnectionImplementor.java:67)
> > > > > > at
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.hibernate.resource.jdbc.internal.LogicalConnectionManagedImpl.begin(LogicalConnectionManagedImpl.java:238)
> > > > > > at
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.hibernate.resource.transaction.backend.jdbc.internal.JdbcResourceLocalTransactionCoordinatorImpl$TransactionDriverControlImpl.begin(JdbcResourceLocalTransactionCoordinatorImpl.java:214)
> > > > > > at
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.hibernate.engine.transaction.internal.TransactionImpl.begin(TransactionImpl.java:52)
> > > > > > at
> > > > > >
> > > > >
> > > >
> > >
> >
> org.hibernate.internal.SessionImpl.beginTransaction(SessionImpl.java:1512)
> > > > > > at
> > > > > >
> > > >
> > org.hibernate.jpa.internal.TransactionImpl.begin(TransactionImpl.java:45)
> > > > > > ... 22 common frames omitted
> > > > > > Caused by: org.mariadb.jdbc.internal.util.dao.QueryException:
> Could
> > > not
> > > > > > read resultset: unexpected end of stream, read 0 bytes from 4
> > > > > > Query is : set autocommit=0
> > > > > > at
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.mariadb.jdbc.internal.protocol.AbstractQueryProtocol.getResult(AbstractQueryProtocol.java:909)
> > > > > > at
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.mariadb.jdbc.internal.protocol.AbstractQueryProtocol.executeQuery(AbstractQueryProtocol.java:604)
> > > > > > at
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.mariadb.jdbc.MariaDbStatement.executeInternal(MariaDbStatement.java:261)
> > > > > > ... 31 common frames omitted
> > > > > > Caused by: java.io.EOFException: unexpected end of stream, read 0
> > > bytes
> > > > > > from 4
> > > > > > at
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.mariadb.jdbc.internal.packet.read.ReadPacketFetcher.getReusableBuffer(ReadPacketFetcher.java:168)
> > > > > > at
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.mariadb.jdbc.internal.protocol.AbstractQueryProtocol.getResult(AbstractQueryProtocol.java:900)
> > > > > > ... 33 common frames omitted
> > > > > > 23:29:05.500 [qtp6422064-20] INFO
> > > o.a.f.c.l.c.ServiceExceptionFilter -
> > > > > > Responding with a service error ServiceError{code=409,
> > > > > message='Application
> > > > > > identity-v1 already exists!'}
> > > > > > 23:29:21.282 [Thread-15] INFO
> > > > > > o.s.b.c.e.AnnotationConfigEmbeddedWebApplicationContext - Closing
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.springframework.boot.context.embedded.AnnotationConfigEmbeddedWebApplicationContext@11438d26
> > > > > > :
> > > > > > startup date [Tue Jun 12 23:02:02 UTC 2018]; parent:
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.springframework.context.annotation.AnnotationConfigApplicationContext@7dc5e7b4
> > > > > > 23:29:23.717 [Thread-15] INFO  o.s.c.s.DefaultLifecycleProcessor
> -
> > > > > Stopping
> > > > > > beans in phase 2147483647
> > > > > > 23:29:24.267 [Thread-15] INFO  o.s.c.s.DefaultLifecycleProcessor
> -
> > > > > Stopping
> > > > > > beans in phase 0
> > > > > > 23:29:24.557 [Thread-15] INFO
> o.s.b.a.e.jmx.EndpointMBeanExporter
> > -
> > > > > > Unregistering JMX-exposed beans on shutdown
> > > > > > 23:29:24.579 [Thread-15] INFO
> o.s.b.a.e.jmx.EndpointMBeanExporter
> > -
> > > > > > Unregistering JMX-exposed beans
> > > > > >
> > > > > > The only thing I've left: start the databases (aka Cassandra) on
> a
> > > > > separate
> > > > > > machine (e. g. on Digital Ocean) and see how this is working
> then.
> > > > > >
> > > > > > ... FYI.
> > > > > >
> > > > > > Cheers,
> > > > > >
> > > > > > Aleks
> > > > > >
> > > > > > On Tue, Jun 12, 2018 at 11:16 PM Aleksandar Vidakovic <
> > > > > > cheetah@xxxxxxxxxxxxxxxxx> wrote:
> > > > > >
> > > > > > > @Rajan: it's not running, because the demo server dies... See
> > logs
> > > > from
> > > > > > > previous message...
> > > > > > >
> > > > > > > We just have to figure that one out and we should be good to
> go.
> > > > > > >
> > > > > > >
> > > > > > > On Tue, Jun 12, 2018, 9:11 PM Rajan Maurya <
> > > rajanmaurya154@xxxxxxxxx
> > > > >
> > > > > > > wrote:
> > > > > > >
> > > > > > >> Thanks, Alex,
> > > > > > >>
> > > > > > >> I can see web app is running but I can't log in with the
> > > credentials
> > > > > > that
> > > > > > >> you shared earlier
> > > > > > >>
> > > > > > >> Tenant : playground
> > > > > > >> Username : operator
> > > > > > >> Password : init1@l
> > > > > > >>
> > > > > > >> Thanks
> > > > > > >>
> > > > > > >> On Wed, Jun 13, 2018 at 12:32 AM Aleksandar Vidakovic <
> > > > > > >> cheetah@xxxxxxxxxxxxxxxxx> wrote:
> > > > > > >>
> > > > > > >> > Hi all,
> > > > > > >> >
> > > > > > >> > ... sweeeet! I got a considerable step further.
> > > > > > >> >
> > > > > > >> > @Markus: thanks for the conversation... gave me an idea how
> to
> > > fix
> > > > > > (most
> > > > > > >> > of) it!
> > > > > > >> >
> > > > > > >> > The problem was - as suspected - the memory (or lack of).
> The
> > VM
> > > > > > >> instance
> > > > > > >> > at Apache has net 30G available. 16G are needed for
> Cassandra
> > > and
> > > > in
> > > > > > my
> > > > > > >> > previous tests I tried to run the demo server with up to
> 16G.
> > > The
> > > > > > >> problem
> > > > > > >> > is that the demo server needs considerably more memory than
> > 16G.
> > > > > > >> According
> > > > > > >> > to my observation it's around 23-24G. Even if you limit the
> > demo
> > > > > > server
> > > > > > >> > memory allocation with "-Xmx" to 12G (for example) it would
> > just
> > > > > > >> continue
> > > > > > >> > to consume more memory.
> > > > > > >> >
> > > > > > >> > My solution: add a swap file. Not ideal in terms of
> > performance,
> > > > but
> > > > > > it
> > > > > > >> > gets the whole thing going... at least mostly (please read
> > on).
> > > > > > >> >
> > > > > > >> > Current status:
> > > > > > >> >
> > > > > > >> >    - Cassandra (and all other Docker containers) are still
> > > > running;
> > > > > > >> >    Cassandra died in my previous attempts at the latest when
> > the
> > > > > > Teller
> > > > > > >> >    application started, but with the swap file it's still
> > > running
> > > > > > >> >    - all Spring Boot apps are starting now
> > > > > > >> >    - I get once in a while some spurious timeouts (I think
> > when
> > > > > either
> > > > > > >> the
> > > > > > >> >    Eureka server is contacted or the config server)
> > > > > > >> >    - there are also sometimes exceptions concerning MySQL
> > > > > connections,
> > > > > > >> but
> > > > > > >> >    doesn't seem to matter overall
> > > > > > >> >    - I could issue an authentication request with Postman to
> > > > > > >> >    http://fineract-vm.apache.org:2021/identity/v1; just to
> > test
> > > > if
> > > > > > >> > anything
> > > > > > >> >    is responding
> > > > > > >> >    - PROBLEM: but the demo server (the Spring Boot apps)
> > > > eventually
> > > > > > dies
> > > > > > >> >    after a couple of minutes; I don't think that the memory
> is
> > > the
> > > > > > >> problem
> > > > > > >> > at
> > > > > > >> >    this point; I think it's more of a timeout problem
> > concerning
> > > > the
> > > > > > >> >    communication with Eureka and/or the config server (maybe
> > > > because
> > > > > > the
> > > > > > >> >    system is a bit slow)
> > > > > > >> >
> > > > > > >> > ... and here the memory footprint ("free -h") when
> everything
> > is
> > > > > > >> running,
> > > > > > >> > just FYI:
> > > > > > >> >
> > > > > > >> >               total        used        free      shared
> > > > buff/cache
> > > > > > >> >  available
> > > > > > >> > Mem:            31G         31G        226M        8.4M
> > > > 169M
> > > > > > >> >  47M
> > > > > > >> > Swap:           15G        6.8G        9.2G
> > > > > > >> >
> > > > > > >> > ... after the Payroll application is started (approx.) I see
> > > these
> > > > > > >> > exceptions appearing in the logs (a whole series of them):
> > > > > > >> >
> > > > > > >> > 18:19:36.465 [AsyncResolver-bootstrap-executor-0] INFO
> > > > > > >> > c.n.d.s.r.aws.ConfigClusterResolver - Resolving eureka
> > endpoints
> > > > via
> > > > > > >> > configuration
> > > > > > >> > Resolving artifact
> > > > > > >> >
> > > org.apache.fineract.cn.group:service-boot:jar:0.1.0-BUILD-SNAPSHOT
> > > > > > >> > Resolving metadata
> > > > > > >> >
> > > > > > >> >
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.fineract.cn.group:service-boot:0.1.0-BUILD-SNAPSHOT/maven-metadata.xml
> > > > > > >> > from /root/.m2/repository (enhanced)
> > > > > > >> > Resolved metadata
> > > > > > >> >
> > > > > > >> >
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.fineract.cn.group:service-boot:0.1.0-BUILD-SNAPSHOT/maven-metadata.xml
> > > > > > >> > from /root/.m2/repository (enhanced)
> > > > > > >> > Resolved artifact
> > > > > > >> >
> > > org.apache.fineract.cn.group:service-boot:jar:0.1.0-BUILD-SNAPSHOT
> > > > > > from
> > > > > > >> > /root/.m2/repository (enhanced)
> > > > > > >> > 18:19:44.689 [DiscoveryClient-0] ERROR
> > > > > > >> c.n.discovery.TimedSupervisorTask -
> > > > > > >> > task supervisor timed out
> > > > > > >> > java.util.concurrent.TimeoutException: null
> > > > > > >> > at java.util.concurrent.FutureTask.get(FutureTask.java:205)
> > > > > > >> > at
> > > > > > >> >
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
> com.netflix.discovery.TimedSupervisorTask.run(TimedSupervisorTask.java:64)
> > > > > > >> > at
> > > > > > >>
> > > > >
> > java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
> > > > > > >> > at java.util.concurrent.FutureTask.run(FutureTask.java:266)
> > > > > > >> > at
> > > > > > >> >
> > > > > > >> >
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$201(ScheduledThreadPoolExecutor.java:180)
> > > > > > >> > at
> > > > > > >> >
> > > > > > >> >
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
> java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
> > > > > > >> > at
> > > > > > >> >
> > > > > > >> >
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
> > > > > > >> > at
> > > > > > >> >
> > > > > > >> >
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
> > > > > > >> > at java.lang.Thread.run(Thread.java:748)
> > > > > > >> >
> > > > > > >> > ... but it still seems to have no problem starting the next
> > app
> > > > > > (Group)
> > > > > > >> and
> > > > > > >> > things look good for a while...
> > > > > > >> >
> > > > > > >> > ... until suddenly it seems that app instances are beginning
> > to
> > > > > > >> shutdown...
> > > > > > >> > and I'm not sure why this happens:
> > > > > > >> >
> > > > > > >> > 18:27:19.603 [AsyncResolver-bootstrap-executor-0] INFO
> > > > > > >> > c.n.d.s.r.aws.ConfigClusterResolver - Resolving eureka
> > endpoints
> > > > via
> > > > > > >> > configuration
> > > > > > >> > 18:27:39.974 [qtp1631119258-22] INFO
> > > > > > >> c.datastax.driver.core.ClockFactory -
> > > > > > >> > Using native clock to generate timestamps.
> > > > > > >> > 18:27:39.975 [qtp1631119258-22] WARN
> > > > > > >> c.datastax.driver.core.CodecRegistry
> > > > > > >> > - Ignoring codec LocalDateTimeCodec [timestamp <->
> > > > > > >> java.time.LocalDateTime]
> > > > > > >> > because it collides with previously registered codec
> > > > > > LocalDateTimeCodec
> > > > > > >> > [timestamp <-> java.time.LocalDateTime]
> > > > > > >> > 18:27:41.309 [qtp1631119258-22] INFO
> > > > > > c.d.d.c.p.DCAwareRoundRobinPolicy
> > > > > > >> -
> > > > > > >> > Using data-center name 'datacenter1' for
> > DCAwareRoundRobinPolicy
> > > > (if
> > > > > > >> this
> > > > > > >> > is incorrect, please provide the correct datacenter name
> with
> > > > > > >> > DCAwareRoundRobinPolicy constructor)
> > > > > > >> > 18:27:41.319 [qtp1631119258-22] INFO
> > > > > > com.datastax.driver.core.Cluster -
> > > > > > >> > New Cassandra host /127.0.0.1:9042 added
> > > > > > >> > 18:27:41.661 [AsyncResolver-bootstrap-executor-0] INFO
> > > > > > >> > c.n.d.s.r.aws.ConfigClusterResolver - Resolving eureka
> > endpoints
> > > > via
> > > > > > >> > configuration
> > > > > > >> > 18:27:51.110 [Thread-15] INFO
> > > > > > >> > o.s.b.c.e.AnnotationConfigEmbeddedWebApplicationContext -
> > > Closing
> > > > > > >> >
> > > > > > >> >
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
> org.springframework.boot.context.embedded.AnnotationConfigEmbeddedWebApplicationContext@11438d26
> > > > > > >> > :
> > > > > > >> > startup date [Tue Jun 12 17:55:35 UTC 2018]; parent:
> > > > > > >> >
> > > > > > >> >
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
> org.springframework.context.annotation.AnnotationConfigApplicationContext@7dc5e7b4
> > > > > > >> > 18:27:51.341 [AsyncResolver-bootstrap-executor-0] INFO
> > > > > > >> > c.n.d.s.r.aws.ConfigClusterResolver - Resolving eureka
> > endpoints
> > > > via
> > > > > > >> > configuration
> > > > > > >> > 18:27:53.373 [Thread-15] INFO
> > > o.s.c.s.DefaultLifecycleProcessor -
> > > > > > >> Stopping
> > > > > > >> > beans in phase 2147483647
> > > > > > >> > 18:27:54.331 [Thread-15] INFO
> > > o.s.c.s.DefaultLifecycleProcessor -
> > > > > > >> Stopping
> > > > > > >> > beans in phase 0
> > > > > > >> > 18:27:54.736 [Thread-15] INFO
> > > > o.s.b.a.e.jmx.EndpointMBeanExporter -
> > > > > > >> > Unregistering JMX-exposed beans on shutdown
> > > > > > >> > 18:27:54.740 [Thread-15] INFO
> > > > o.s.b.a.e.jmx.EndpointMBeanExporter -
> > > > > > >> > Unregistering JMX-exposed beans
> > > > > > >> > 18:27:54.798 [Thread-15] INFO
> > > o.s.j.e.a.AnnotationMBeanExporter -
> > > > > > >> > Unregistering JMX-exposed beans on shutdown
> > > > > > >> > 18:27:54.800 [Thread-15] INFO
> > > o.s.j.e.a.AnnotationMBeanExporter -
> > > > > > >> > Unregistering JMX-exposed beans
> > > > > > >> > 18:27:54.937 [Thread-15] INFO
> > > > > > >> > o.s.o.j.LocalContainerEntityManagerFactoryBean - Closing JPA
> > > > > > >> > EntityManagerFactory for persistence unit 'metaPU'
> > > > > > >> > 18:27:58.376 [async-processor-1] INFO
> > > > > > >> o.f.c.internal.util.VersionPrinter -
> > > > > > >> > Flyway 3.2.1 by Boxfuse
> > > > > > >> > 18:27:58.677 [async-processor-1] INFO
> > > > > > >> o.f.c.i.dbsupport.DbSupportFactory -
> > > > > > >> > Database: jdbc:mysql://localhost:3306/seshat (MySQL 10.3)
> > > > > > >> > 18:28:00.620 [async-processor-1] INFO
> > > > > > >> o.f.core.internal.command.DbValidate
> > > > > > >> > - Validated 2 migrations (execution time 00:01.169s)
> > > > > > >> > 18:28:00.901 [Thread-15] INFO
> > > o.e.jetty.server.AbstractConnector
> > > > -
> > > > > > >> Stopped
> > > > > > >> > ServerConnector@e162a35{HTTP/1.1,[http/1.1]}{0.0.0.0:2020}
> > > > > > >> > 18:28:01.268 [Thread-15] INFO
> > > > o.e.j.server.handler.ContextHandler -
> > > > > > >> > Stopped o.s.b.c.e.j.JettyEmbeddedWebAppContext@10850d17
> > > > > > >> >
> > > > > > >> >
> > > > > > >>
> > > > > >
> > > > >
> > > >
> > >
> >
> {/provisioner/v1,file:///tmp/jetty-docbase.5486166363042759626.2020/,UNAVAILABLE}
> > > > > > >> >
> > > > > > >> > ... the list is actually a lot longer... in the rest of the
> > log
> > > > you
> > > > > > can
> > > > > > >> see
> > > > > > >> > that one by one all the apps are shutting down.
> > > > > > >> >
> > > > > > >> > The last (error) entries I see are these:
> > > > > > >> >
> > > > > > >> > E18:29:31.278 [DefaultMessageListenerContainer-3] ERROR
> > > > > > >> > o.s.j.l.DefaultMessageListenerContainer - Could not refresh
> > JMS
> > > > > > >> Connection
> > > > > > >> > for destination 'cheques-v1' - retrying using
> > > > > > >> FixedBackOff{interval=5000,
> > > > > > >> > currentAttempts=0, maxAttempts=unlimited}. Cause: Error
> while
> > > > > > >> attempting to
> > > > > > >> > retrieve a connection from the pool; nested exception is
> > > > > > >> > javax.jms.JMSException: Could not connect to broker URL:
> > > > > > >> > tcp://localhost:61616. Reason: java.net.ConnectException:
> > > > Connection
> > > > > > >> > refused (Connection refused)
> > > > > > >> >
> > > > > > >> > ... I guess an embedded ActiveMQ was also shutdown in the
> > > process
> > > > of
> > > > > > the
> > > > > > >> > previous errors.
> > > > > > >> >
> > > > > > >> > So far from here... any input on the above highly
> appreciated.
> > > > > > Overall I
> > > > > > >> > think we are close.
> > > > > > >> >
> > > > > > >> > Cheers,
> > > > > > >> >
> > > > > > >> > Aleks
> > > > > > >> >
> > > > > > >> >
> > > > > > >> > On Tue, Jun 12, 2018 at 6:16 PM Markus Geiss <
> mage@xxxxxxxxxx
> > >
> > > > > wrote:
> > > > > > >> >
> > > > > > >> > > One thing we maybe can do, given we are part of the Apache
> > > > Family,
> > > > > > is
> > > > > > >> to
> > > > > > >> > > ask the Cassandra community for some suggestions.
> > > > > > >> > >
> > > > > > >> > > Cheers
> > > > > > >> > >
> > > > > > >> > > Markus
> > > > > > >> > >
> > > > > > >> > > On Tue, Jun 12, 2018 at 6:04 PM Markus Geiss <
> > mage@xxxxxxxxxx
> > > >
> > > > > > wrote:
> > > > > > >> > >
> > > > > > >> > > > Hey all,
> > > > > > >> > > >
> > > > > > >> > > > we are not running Cassandra in a container, we have
> > > dedicated
> > > > > VMs
> > > > > > >> to
> > > > > > >> > run
> > > > > > >> > > > our cluster.
> > > > > > >> > > >
> > > > > > >> > > > Given this we are not the best persons to help here,
> > sorry.
> > > (;
> > > > > > >> > > >
> > > > > > >> > > > Cheers
> > > > > > >> > > >
> > > > > > >> > > > Markus
> > > > > > >> > > >
> > > > > > >> > > > On Tue, Jun 12, 2018 at 10:35 AM Ed Cable <
> > > edcable@xxxxxxxxx>
> > > > > > >> wrote:
> > > > > > >> > > >
> > > > > > >> > > >> Adding Patric directly to this thread too so he can
> give
> > > his
> > > > > > input
> > > > > > >> > > >> regarding to the challenges with Cassandra.
> > > > > > >> > > >>
> > > > > > >> > > >> Ed
> > > > > > >> > > >> On Tue, Jun 12, 2018 at 12:53 AM Aleksandar Vidakovic <
> > > > > > >> > > >> cheetah@xxxxxxxxxxxxxxxxx> wrote:
> > > > > > >> > > >>
> > > > > > >> > > >>> Hi Victor,
> > > > > > >> > > >>>
> > > > > > >> > > >>> .... this is the relevant part of the
> docker-compose.yml
> > > > file:
> > > > > > >> > > >>>
> > > > > > >> > > >>>   cassandra0:
> > > > > > >> > > >>>     image: cassandra:3.11.1
> > > > > > >> > > >>>     container_name: cassandra0
> > > > > > >> > > >>>     ports:
> > > > > > >> > > >>>       - 9042:9042
> > > > > > >> > > >>>       - 9160:9160
> > > > > > >> > > >>>       - 7199:7199
> > > > > > >> > > >>>       - 8778:8778
> > > > > > >> > > >>>     volumes:
> > > > > > >> > > >>>       - ./cassandra:/etc/cassandra
> > > > > > >> > > >>>     environment:
> > > > > > >> > > >>>       - CASSANDRA_START_RPC=true
> > > > > > >> > > >>>       - CASSANDRA_SEEDS=cassandra0
> > > > > > >> > > >>>       - CASSANDRA_CLUSTER_NAME=fineract_cluster
> > > > > > >> > > >>>     ulimits:
> > > > > > >> > > >>>       memlock: -1
> > > > > > >> > > >>>       nproc: 32768
> > > > > > >> > > >>>       nofile: 100000
> > > > > > >> > > >>>
> > > > > > >> > > >>> ... and just to be complete... here's the Docker
> service
> > > > > > >> > configuration
> > > > > > >> > > >>> ("/lib/systemd/system/docker.service"):
> > > > > > >> > > >>>
> > > > > > >> > > >>> [Unit]
> > > > > > >> > > >>> Description=Docker Application Container Engine
> > > > > > >> > > >>> Documentation=https://docs.docker.com
> > > > > > >> > > >>> After=network-online.target docker.socket
> > > firewalld.service
> > > > > > >> > > >>> Wants=network-online.target
> > > > > > >> > > >>> Requires=docker.socket
> > > > > > >> > > >>>
> > > > > > >> > > >>> [Service]
> > > > > > >> > > >>> Type=notify
> > > > > > >> > > >>> # the default is not to use systemd for cgroups
> because
> > > the
> > > > > > >> delegate
> > > > > > >> > > >>> issues
> > > > > > >> > > >>> still
> > > > > > >> > > >>> # exists and systemd currently does not support the
> > cgroup
> > > > > > feature
> > > > > > >> > set
> > > > > > >> > > >>> required
> > > > > > >> > > >>> # for containers run by docker
> > > > > > >> > > >>> ExecStart=/usr/bin/dockerd -H fd://
> > > > > > >> > > >>> ExecReload=/bin/kill -s HUP $MAINPID
> > > > > > >> > > >>> LimitNOFILE=1048576
> > > > > > >> > > >>> # Having non-zero Limit*s causes performance problems
> > due
> > > to
> > > > > > >> > accounting
> > > > > > >> > > >>> overhead
> > > > > > >> > > >>> # in the kernel. We recommend using cgroups to do
> > > > > > container-local
> > > > > > >> > > >>> accounting.
> > > > > > >> > > >>> LimitNPROC=infinity
> > > > > > >> > > >>> LimitCORE=infinity
> > > > > > >> > > >>> LimitMEMLOCK=infinity
> > > > > > >> > > >>> # Uncomment TasksMax if your systemd version supports
> > it.
> > > > > > >> > > >>> # Only systemd 226 and above support this version.
> > > > > > >> > > >>> TasksMax=infinity
> > > > > > >> > > >>> TimeoutStartSec=0
> > > > > > >> > > >>> # set delegate yes so that systemd does not reset the
> > > > cgroups
> > > > > of
> > > > > > >> > docker
> > > > > > >> > > >>> containers
> > > > > > >> > > >>> Delegate=yes
> > > > > > >> > > >>> # kill only the docker process, not all processes in
> the
> > > > > cgroup
> > > > > > >> > > >>> KillMode=process
> > > > > > >> > > >>> # restart the docker process if it exits prematurely
> > > > > > >> > > >>> Restart=on-failure
> > > > > > >> > > >>> StartLimitBurst=3
> > > > > > >> > > >>> StartLimitInterval=60s
> > > > > > >> > > >>>
> > > > > > >> > > >>> [Install]
> > > > > > >> > > >>> WantedBy=multi-user.target
> > > > > > >> > > >>>
> > > > > > >> > > >>> ... the one tweak I did there is to set LimitMEMLOCK
> to
> > > > > > >> infinity...
> > > > > > >> > do
> > > > > > >> > > >>> you
> > > > > > >> > > >>> think the problems are file handle related? Should I
> set
> > > > > > >> LimitNOFILE
> > > > > > >> > > also
> > > > > > >> > > >>> to infinity?
> > > > > > >> > > >>>
> > > > > > >> > > >>> Appreciate the help.
> > > > > > >> > > >>>
> > > > > > >> > > >>> Cheers,
> > > > > > >> > > >>>
> > > > > > >> > > >>> Aleks
> > > > > > >> > > >>>
> > > > > > >> > > >>> On Tue, Jun 12, 2018 at 9:09 AM Victor Romero <
> > > > > > >> > > >>> victor.romero@xxxxxxxxxxxxxxx>
> > > > > > >> > > >>> wrote:
> > > > > > >> > > >>>
> > > > > > >> > > >>> > Hi Aleks,
> > > > > > >> > > >>> >
> > > > > > >> > > >>> > Can you share the ulimits flags that the cassandra's
> > > > > container
> > > > > > >> is
> > > > > > >> > > using
> > > > > > >> > > >>> > while it is running?
> > > > > > >> > > >>> >
> > > > > > >> > > >>> > The values are being set in the composer file? Or in
> > the
> > > > > > >> upstart or
> > > > > > >> > > >>> > systemd docker's deamon config files?
> > > > > > >> > > >>> >
> > > > > > >> > > >>> >
> > > > > > >> > > >>> >
> > > > > > >> > > >>> > ⁣Enviado desde TypeApp ​
> > > > > > >> > > >>> >
> > > > > > >> > > >>> > En jun. 11, 2018 10:02 AM, en 10:02 AM, Aleksandar
> > > > > Vidakovic <
> > > > > > >> > > >>> > cheetah@xxxxxxxxxxxxxxxxx> escribió:
> > > > > > >> > > >>> > >... and I should have attached the Cassandra log
> > > dump...
> > > > so
> > > > > > >> here
> > > > > > >> > it
> > > > > > >> > > >>> > >is...
> > > > > > >> > > >>> > >
> > > > > > >> > > >>> > >On Mon, Jun 11, 2018 at 4:49 PM Aleksandar
> Vidakovic
> > <
> > > > > > >> > > >>> > >cheetah@xxxxxxxxxxxxxxxxx> wrote:
> > > > > > >> > > >>> > >
> > > > > > >> > > >>> > >> Hi all,
> > > > > > >> > > >>> > >>
> > > > > > >> > > >>> > >> ... as you might have noticed I did a couple of
> > > > restarts
> > > > > > >> > today...
> > > > > > >> > > >>> the
> > > > > > >> > > >>> > >> problem I am facing now: I can't get Cassandra to
> > run
> > > > in
> > > > > a
> > > > > > >> > stable
> > > > > > >> > > >>> > >way.
> > > > > > >> > > >>> > >>
> > > > > > >> > > >>> > >> Things look quite OK for a while when running the
> > > demo
> > > > > > >> server,
> > > > > > >> > but
> > > > > > >> > > >>> > >then
> > > > > > >> > > >>> > >> suddenly Cassandra dies (sometimes it starts
> > > > misbehaving
> > > > > > with
> > > > > > >> > the
> > > > > > >> > > >>> > >deposit
> > > > > > >> > > >>> > >> microservice startup, sometimes with portfolio or
> > > > > teller).
> > > > > > I
> > > > > > >> > tried
> > > > > > >> > > >>> to
> > > > > > >> > > >>> > >> increase the memory (4G, 8G and 16G) and set some
> > > > Docker
> > > > > > >> limits
> > > > > > >> > to
> > > > > > >> > > >>> > >> "infinity" (especially LimitMEMLOCK).
> > > > > > >> > > >>> > >>
> > > > > > >> > > >>> > >> I've attached Cassandra's log dump... maybe
> someone
> > > can
> > > > > > help
> > > > > > >> out
> > > > > > >> > > >>> > >here? Is
> > > > > > >> > > >>> > >> it even possible to run Fineract CN on 32GB of
> > > memory?
> > > > > > >> > > >>> > >>
> > > > > > >> > > >>> > >> Other than that the setup would be ready to go...
> > we
> > > > just
> > > > > > >> need
> > > > > > >> > to
> > > > > > >> > > >>> get
> > > > > > >> > > >>> > >the
> > > > > > >> > > >>> > >> database running more reliably.
> > > > > > >> > > >>> > >>
> > > > > > >> > > >>> > >> Cheers,
> > > > > > >> > > >>> > >>
> > > > > > >> > > >>> > >> Aleks
> > > > > > >> > > >>> > >>
> > > > > > >> > > >>> > >> On Mon, Jun 11, 2018 at 2:38 PM Aleksandar
> > Vidakovic
> > > <
> > > > > > >> > > >>> > >> cheetah@xxxxxxxxxxxxxxxxx> wrote:
> > > > > > >> > > >>> > >>
> > > > > > >> > > >>> > >>> @Rajan: I had to restart it again (needed to add
> > > some
> > > > > > >> > additional
> > > > > > >> > > >>> > >reverse
> > > > > > >> > > >>> > >>> proxy configuration for the web UI to work)...
> and
> > > > there
> > > > > > >> were
> > > > > > >> > > more
> > > > > > >> > > >>> > >>> exceptions that I hope will be fixed now.
> > > > > > >> > > >>> > >>>
> > > > > > >> > > >>> > >>> Just FYI
> > > > > > >> > > >>> > >>>
> > > > > > >> > > >>> > >>> On Mon, Jun 11, 2018 at 1:47 PM Aleksandar
> > > Vidakovic <
> > > > > > >> > > >>> > >>> cheetah@xxxxxxxxxxxxxxxxx> wrote:
> > > > > > >> > > >>> > >>>
> > > > > > >> > > >>> > >>>> No prob.... Let me know how it goes... Can get
> > back
> > > > to
> > > > > > >> testing
> > > > > > >> > > >>> only
> > > > > > >> > > >>> > >>>> later tonight.
> > > > > > >> > > >>> > >>>>
> > > > > > >> > > >>> > >>>> Cheers
> > > > > > >> > > >>> > >>>>
> > > > > > >> > > >>> > >>>>
> > > > > > >> > > >>> > >>>> On Mon, Jun 11, 2018, 1:45 PM Rajan Maurya
> > > > > > >> > > >>> > ><rajanmaurya154@xxxxxxxxx>
> > > > > > >> > > >>> > >>>> wrote:
> > > > > > >> > > >>> > >>>>
> > > > > > >> > > >>> > >>>>> Sorry missed the 30 min, I will test after 30
> > min.
> > > > > > >> > > >>> > >>>>>
> > > > > > >> > > >>> > >>>>> Big thanks for this 🙂
> > > > > > >> > > >>> > >>>>>
> > > > > > >> > > >>> > >>>>>
> > > > > > >> > > >>> > >>>>>
> > > > > > >> > > >>> > >>>>>
> > > > > > >> > > >>> > >>>>> On Mon, Jun 11, 2018 at 5:12 PM Aleksandar
> > > > Vidakovic <
> > > > > > >> > > >>> > >>>>> cheetah@xxxxxxxxxxxxxxxxx> wrote:
> > > > > > >> > > >>> > >>>>>
> > > > > > >> > > >>> > >>>>> > @Rajan: can't see the image you posted.
> > > > > > >> > > >>> > >>>>> >
> > > > > > >> > > >>> > >>>>> > ... and as I said: the services are still
> > > > > starting...
> > > > > > >> and
> > > > > > >> > the
> > > > > > >> > > >>> > >best
> > > > > > >> > > >>> > >>>>> estimate
> > > > > > >> > > >>> > >>>>> > I have right now (as already mentioned):
> 30min
> > > > > > >> > > >>> > >>>>> >
> > > > > > >> > > >>> > >>>>> > On Mon, Jun 11, 2018 at 1:40 PM Rajan
> Maurya <
> > > > > > >> > > >>> > >>>>> rajanmaurya154@xxxxxxxxx>
> > > > > > >> > > >>> > >>>>> > wrote:
> > > > > > >> > > >>> > >>>>> >
> > > > > > >> > > >>> > >>>>> > > [image: image.png]
> > > > > > >> > > >>> > >>>>> > >
> > > > > > >> > > >>> > >>>>> > >
> > > > > > >> > > >>> > >>>>> > >
> > > > > > >> > > >>> > >>>>> > >
> > > > > > >> > > >>> > >>>>> > >
> > > > > > >> > > >>> > >>>>> > > On Mon, Jun 11, 2018 at 5:08 PM Rajan
> > Maurya <
> > > > > > >> > > >>> > >>>>> rajanmaurya154@xxxxxxxxx>
> > > > > > >> > > >>> > >>>>> > > wrote:
> > > > > > >> > > >>> > >>>>> > >
> > > > > > >> > > >>> > >>>>> > >> [image: image.png]
> > > > > > >> > > >>> > >>>>> > >> I am getting this.
> > > > > > >> > > >>> > >>>>> > >>
> > > > > > >> > > >>> > >>>>> > >>
> > > > > > >> > > >>> > >>>>> > >>
> > > > > > >> > > >>> > >>>>> > >>
> > > > > > >> > > >>> > >>>>> > >>
> > > > > > >> > > >>> > >>>>> > >> On Mon, Jun 11, 2018 at 5:05 PM
> Aleksandar
> > > > > > Vidakovic
> > > > > > >> <
> > > > > > >> > > >>> > >>>>> > >> cheetah@xxxxxxxxxxxxxxxxx> wrote:
> > > > > > >> > > >>> > >>>>> > >>
> > > > > > >> > > >>> > >>>>> > >>> Hi all,
> > > > > > >> > > >>> > >>>>> > >>>
> > > > > > >> > > >>> > >>>>> > >>> so... the demo server is (almost)
> ready...
> > > it
> > > > > took
> > > > > > >> me a
> > > > > > >> > > >>> > >moment
> > > > > > >> > > >>> > >>>>> and a
> > > > > > >> > > >>> > >>>>> > >>> couple
> > > > > > >> > > >>> > >>>>> > >>> of restarts to figure out some boot
> > > > failures...
> > > > > > the
> > > > > > >> > > >>> services
> > > > > > >> > > >>> > >are
> > > > > > >> > > >>> > >>>>> quite
> > > > > > >> > > >>> > >>>>> > >>> resource hungry and default settings
> won't
> > > do
> > > > > it.
> > > > > > >> > > >>> > >>>>> > >>>
> > > > > > >> > > >>> > >>>>> > >>> The services are still starting and this
> > > will
> > > > > > take a
> > > > > > >> > > while
> > > > > > >> > > >>> > >(my
> > > > > > >> > > >>> > >>>>> best
> > > > > > >> > > >>> > >>>>> > guess
> > > > > > >> > > >>> > >>>>> > >>> right now is around 30min or so).
> > > > > > >> > > >>> > >>>>> > >>>
> > > > > > >> > > >>> > >>>>> > >>> You can access the web UI at:
> > > > > > >> > > >>> > >http://fineract-vm.apache.org/login
> > > > > > >> > > >>> > >>>>> > >>>
> > > > > > >> > > >>> > >>>>> > >>> Credentials:
> > > > > > >> > > >>> > >>>>> > >>>
> > > > > > >> > > >>> > >>>>> > >>> Tenant : playground
> > > > > > >> > > >>> > >>>>> > >>> Username : operator
> > > > > > >> > > >>> > >>>>> > >>> Password : init1@l
> > > > > > >> > > >>> > >>>>> > >>>
> > > > > > >> > > >>> > >>>>> > >>> Note (to self): the safest way to
> compile
> > > this
> > > > > app
> > > > > > >> is
> > > > > > >> > > with
> > > > > > >> > > >>> > >NodeJS
> > > > > > >> > > >>> > >>>>> > 8.11.1
> > > > > > >> > > >>> > >>>>> > >>> (I tried with 10.3.0 before, won't
> work).
> > > > > > >> > > >>> > >>>>> > >>>
> > > > > > >> > > >>> > >>>>> > >>> ... and the webservices at:
> > > > > > >> > > >>> > >>>>> > >>>
> > > > > > >> > > >>> > >>>>> > >>> Identity Service:
> > > > > > >> > > >>> > >http://fineract-vm.apache.org:2021/identity/v1
> > > > > > >> > > >>> > >>>>> > >>> Office Service:
> > > > > > >> > > >>> http://fineract-vm.apache.org:2023/office/v1
> > > > > > >> > > >>> > >>>>> > >>> Customer Service:
> > > > > > >> > > >>> > >http://fineract-vm.apache.org:2024/customer/v1
> > > > > > >> > > >>> > >>>>> > >>> Accounting Service:
> > > > > > >> > > >>> > >>>>>
> > http://fineract-vm.apache.org:2025/accounting/v1
> > > > > > >> > > >>> > >>>>> > >>> Portfolio Service:
> > > > > > >> > > >>> > >>>>>
> http://fineract-vm.apache.org:2026/portfolio/v1
> > > > > > >> > > >>> > >>>>> > >>> Deposit Service:
> > > > > > >> > > >>> > >http://fineract-vm.apache.org:2027/deposit/v1
> > > > > > >> > > >>> > >>>>> > >>> Teller Service:
> > > > > > >> > > >>> http://fineract-vm.apache.org:2028/teller/v1
> > > > > > >> > > >>> > >>>>> > >>> Reporting Service:
> > > > > > >> > > >>> > >>>>>
> http://fineract-vm.apache.org:2029/reporting/v1
> > > > > > >> > > >>> > >>>>> > >>> Cheque Service:
> > > > > > >> > > >>> > >http://fineract-vm.apache.org:2030/cheques/v1
> > > > > > >> > > >>> > >>>>> > >>> Payroll Service:
> > > > > > >> > > >>> > >http://fineract-vm.apache.org:2031/payroll/v1
> > > > > > >> > > >>> > >>>>> > >>>
> > > > > > >> > > >>> > >>>>> > >>> Note: restarting the services takes
> quite
> > a
> > > > > > >> while... if
> > > > > > >> > > you
> > > > > > >> > > >>> > >>>>> encounter
> > > > > > >> > > >>> > >>>>> > >>> connection problems retry a couple of
> > > minutes
> > > > > > later
> > > > > > >> to
> > > > > > >> > > >>> > >ensure I
> > > > > > >> > > >>> > >>>>> am not
> > > > > > >> > > >>> > >>>>> > >>> currently working on something.
> > > > > > >> > > >>> > >>>>> > >>>
> > > > > > >> > > >>> > >>>>> > >>> For the moment I'll restart the backend
> > > > services
> > > > > > >> once
> > > > > > >> > per
> > > > > > >> > > >>> > >day or
> > > > > > >> > > >>> > >>>>> so to
> > > > > > >> > > >>> > >>>>> > >>> reset the data; I guess we have to
> figure
> > > out
> > > > > how
> > > > > > we
> > > > > > >> > want
> > > > > > >> > > >>> to
> > > > > > >> > > >>> > >>>>> handle
> > > > > > >> > > >>> > >>>>> > this
> > > > > > >> > > >>> > >>>>> > >>> (also concerning passwords etc.).
> > > > > > >> > > >>> > >>>>> > >>>
> > > > > > >> > > >>> > >>>>> > >>> If you have any suggestions where to put
> > > this
> > > > > demo
> > > > > > >> > server
> > > > > > >> > > >>> > >>>>> configuration
> > > > > > >> > > >>> > >>>>> > >>> then let me know here.
> > > > > > >> > > >>> > >>>>> > >>>
> > > > > > >> > > >>> > >>>>> > >>> Let me know if you encounter any
> problems
> > (I
> > > > > have
> > > > > > >> not
> > > > > > >> > > >>> > >extensively
> > > > > > >> > > >>> > >>>>> > tested
> > > > > > >> > > >>> > >>>>> > >>> it
> > > > > > >> > > >>> > >>>>> > >>> yet).
> > > > > > >> > > >>> > >>>>> > >>>
> > > > > > >> > > >>> > >>>>> > >>> Cheers,
> > > > > > >> > > >>> > >>>>> > >>>
> > > > > > >> > > >>> > >>>>> > >>> Aleks
> > > > > > >> > > >>> > >>>>> > >>>
> > > > > > >> > > >>> > >>>>> > >>>
> > > > > > >> > > >>> > >>>>> > >>> On Mon, Jun 11, 2018 at 12:00 PM
> > Aleksandar
> > > > > > >> Vidakovic <
> > > > > > >> > > >>> > >>>>> > >>> cheetah@xxxxxxxxxxxxxxxxx> wrote:
> > > > > > >> > > >>> > >>>>> > >>>
> > > > > > >> > > >>> > >>>>> > >>> > Progress! All the modules seem to be
> > > > starting
> > > > > > fine
> > > > > > >> > > now...
> > > > > > >> > > >>> > >just
> > > > > > >> > > >>> > >>>>> have
> > > > > > >> > > >>> > >>>>> > to
> > > > > > >> > > >>> > >>>>> > >>> > work on a couple of improvements on
> the
> > > > config
> > > > > > >> files.
> > > > > > >> > > >>> > >>>>> > >>> >
> > > > > > >> > > >>> > >>>>> > >>> > Currently having another stab to get
> the
> > > > > webapp
> > > > > > >> > > running.
> > > > > > >> > > >>> > >>>>> > >>> >
> > > > > > >> > > >>> > >>>>> > >>> > BTW: it should be fairly easy to adapt
> > the
> > > > > > >> simplistic
> > > > > > >> > > >>> > >Docker
> > > > > > >> > > >>> > >>>>> Compose
> > > > > > >> > > >>> > >>>>> > >>> file
> > > > > > >> > > >>> > >>>>> > >>> > for a CI pipeline.
> > > > > > >> > > >>> > >>>>> > >>> >
> > > > > > >> > > >>> > >>>>> > >>> > Will keep you posted.
> > > > > > >> > > >>> > >>>>> > >>> >
> > > > > > >> > > >>> > >>>>> > >>> > Cheers,
> > > > > > >> > > >>> > >>>>> > >>> >
> > > > > > >> > > >>> > >>>>> > >>> > Aleks
> > > > > > >> > > >>> > >>>>> > >>> >
> > > > > > >> > > >>> > >>>>> > >>> > On Mon, Jun 11, 2018 at 9:41 AM
> > Aleksandar
> > > > > > >> Vidakovic
> > > > > > >> > <
> > > > > > >> > > >>> > >>>>> > >>> > cheetah@xxxxxxxxxxxxxxxxx> wrote:
> > > > > > >> > > >>> > >>>>> > >>> >
> > > > > > >> > > >>> > >>>>> > >>> >> I am still working on it... Will ping
> > you
> > > > > here
> > > > > > >> when
> > > > > > >> > > it's
> > > > > > >> > > >>> > >ready
> > > > > > >> > > >>> > >>>>> (hope
> > > > > > >> > > >>> > >>>>> > >>> >> soon).
> > > > > > >> > > >>> > >>>>> > >>> >>
> > > > > > >> > > >>> > >>>>> > >>> >> On Mon, Jun 11, 2018, 9:40 AM Rajan
> > > Maurya
> > > > <
> > > > > > >> > > >>> > >>>>> > rajanmaurya154@xxxxxxxxx>
> > > > > > >> > > >>> > >>>>> > >>> >> wrote:
> > > > > > >> > > >>> > >>>>> > >>> >>
> > > > > > >> > > >>> > >>>>> > >>> >>> Hi Alex,
> > > > > > >> > > >>> > >>>>> > >>> >>>
> > > > > > >> > > >>> > >>>>> > >>> >>> Is the demo server up, If yes then
> > > what's
> > > > > the
> > > > > > >> base
> > > > > > >> > > url
> > > > > > >> > > >>> > >so I
> > > > > > >> > > >>> > >>>>> can
> > > > > > >> > > >>> > >>>>> > >>> access.
> > > > > > >> > > >>> > >>>>> > >>> >>>
> > > > > > >> > > >>> > >>>>> > >>> >>>
> > > > > > >> > > >>> > >>>>> > >>> >>>
> > > > > > >> > > >>> > >>>>> > >>> >>>
> > > > > > >> > > >>> > >>>>> > >>> >>> On Fri, Jun 8, 2018 at 2:03 PM Rajan
> > > > Maurya
> > > > > <
> > > > > > >> > > >>> > >>>>> > >>> rajanmaurya154@xxxxxxxxx>
> > > > > > >> > > >>> > >>>>> > >>> >>> wrote:
> > > > > > >> > > >>> > >>>>> > >>> >>>
> > > > > > >> > > >>> > >>>>> > >>> >>> > Hi Alex,
> > > > > > >> > > >>> > >>>>> > >>> >>> >
> > > > > > >> > > >>> > >>>>> > >>> >>> > This is really great news, I will
> > wait
> > > > > until
> > > > > > >> > then.
> > > > > > >> > > >>> > >>>>> > >>> >>> >
> > > > > > >> > > >>> > >>>>> > >>> >>> > Thanks
> > > > > > >> > > >>> > >>>>> > >>> >>> > Rajan Maurya
> > > > > > >> > > >>> > >>>>> > >>> >>> >
> > > > > > >> > > >>> > >>>>> > >>> >>> >
> > > > > > >> > > >>> > >>>>> > >>> >>> > On Fri, Jun 8, 2018 at 11:36 AM
> > > > Aleksandar
> > > > > > >> > > Vidakovic
> > > > > > >> > > >>> <
> > > > > > >> > > >>> > >>>>> > >>> >>> > cheetah@xxxxxxxxxxxxxxxxx> wrote:
> > > > > > >> > > >>> > >>>>> > >>> >>> >
> > > > > > >> > > >>> > >>>>> > >>> >>> >> Hi Rajan,
> > > > > > >> > > >>> > >>>>> > >>> >>> >>
> > > > > > >> > > >>> > >>>>> > >>> >>> >> My plan is to get this up and
> > running
> > > > > until
> > > > > > >> > > Monday.
> > > > > > >> > > >>> > >>>>> > >>> >>> >>
> > > > > > >> > > >>> > >>>>> > >>> >>> >> Cheers,
> > > > > > >> > > >>> > >>>>> > >>> >>> >>
> > > > > > >> > > >>> > >>>>> > >>> >>> >> Aleks
> > > > > > >> > > >>> > >>>>> > >>> >>> >>
> > > > > > >> > > >>> > >>>>> > >>> >>> >> On Fri, Jun 8, 2018, 7:23 AM
> Rajan
> > > > > Maurya <
> > > > > > >> > > >>> > >>>>> > >>> rajanmaurya154@xxxxxxxxx>
> > > > > > >> > > >>> > >>>>> > >>> >>> >> wrote:
> > > > > > >> > > >>> > >>>>> > >>> >>> >>
> > > > > > >> > > >>> > >>>>> > >>> >>> >> > Hi Markus, Mark, Myrle, Alex,
> Ed
> > > > > > >> > > >>> > >>>>> > >>> >>> >> >
> > > > > > >> > > >>> > >>>>> > >>> >>> >> > I am mentoring
> fineract-cn-mobile
> > > > GSoC
> > > > > > >> > project.
> > > > > > >> > > I
> > > > > > >> > > >>> > >am
> > > > > > >> > > >>> > >>>>> waiting
> > > > > > >> > > >>> > >>>>> > for
>