osdir.com


[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

S3 logs not working with CeleryExecutor


Hello Team,
I am trying to setup S3 logging with CeleryExecutor (apache-airflow 1.10).
Files are not written to S3. I have configured in airflow.cfg like below,

remote_logging = True

remote_log_conn_id = s3_connection_mine

remote_base_log_folder = s3://mybucket/airflow/logs/


when i tried with LocalExecutor, it works perfectly fine but not with
CeleryExecutor. Also CeleryExecutor is working fine without S3 logs. But i
want the logs to be in S3. I can see the celery logging is defaulted to log
= LoggingMixin().log inside default_celery.py. Can you please help me to
configure S3 logging with CeleryExecutor.


Thanks,

Bhavani