Hope this is the right place to ask, but I have a 5 Raspberry Pi 3 cluster running Flink on which I’m hitting memory issues. Each Pi has 1Gb and is running with a 256Gb USB drive, however I don’t seem to be able to configure Java to use more than 256M of heap. The memory problem, I’m reasonably sure, lies in in the key store backed by the filestore (state.backend: filesystem). Either it is not caching to disk or has insufficient RAM to function properly. How can I set the configuration to increase the heap size?
The task manger pane shows the following for each of the five PIs
akka.tcp://flink@wright:35869/user/taskmanager 456052018-04-28, 10:35:25224923 MB224 MB108 MB
My flink configuration is
# The RPC port where the JobManager is reachable.
# The heap size for the JobManager JVM, I have to comment these out
# otherwise Flink refuses to run on the Pi
# jobmanager.heap.mb: 1024
# The heap size for the TaskManager JVM
# taskmanager.heap.mb: 1024
# The number of task slots that each TaskManager offers. Each slot runs one parallel pipeline.
# Specify whether TaskManager memory should be allocated when starting up (true) or when
# memory is required in the memory manager (false)
# Important Note: For pure streaming setups, we highly recommend to set this value to `false`
# as the default state backends currently do not use the managed memory.
# The parallelism used for programs that did not specify and other parallelism.