osdir.com


[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Flink not rolling log files


I am using Flink in EMR with following configuration. 

 {
      "Classification": "flink-log4j",
      "Properties": {
            "log4j.logger.no":"DEBUG", 
            "log4j.appender.file":"org.apache.log4j.rolling.RollingFileAppender",
            "log4j.appender.file.RollingPolicy.FileNamePattern":"logs/log.%d{yyyyMMdd-HHmm}.log",
            "log4j.appender.file.RollingPolicy":"org.apache.log4j.rolling.TimeBasedRollingPolicy",
            "log4j.appender.file.append":"false",
            "log4j.appender.file.layout":"org.apache.log4j.PatternLayout",
            "log4j.appender.file.layout.ConversionPattern":"%d{yyyy-MM-dd HH:mm:ss,SSS} %-5p %-60c %x - %m%n"

      }
    }

FYI this configuration get written into flink's log4j.properties.As you can see even after this setting taskmanager and jobmanager log files are not getting rolled

[hadoop@ip-XXXXXX ~]$ sudo ls -lh  /mnt/var/log/hadoop-yarn/containers/application_DDDDDDDDDDD_0002/container_DDDDDDDDDDDD_0002_01_000002
total 7.0G
-rw-r--r-- 1 yarn yarn 770K Aug 17 14:02 taskmanager.err
-rw-r--r-- 1 yarn yarn 6.0G Aug 17 14:02 taskmanager.log
-rw-r--r-- 1 yarn yarn 526K Aug 17 13:54 taskmanager.out

Can somebody help me to give pointer about how to roll these log files? Note that these files are also being copied into s3.