[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: How to consolidate log files?

AFAIK, airflow doesn't provide log in this way. Multiple tasks would run in
different processes and potentially in parallel, thus writing to the same
file at run time would produce log file with mix log lines from different
tasks. Also I believe airflow now does not seperate stdour and stderr, they
all go to same place. Not sure if there's a good point in code to
consolidate the logs from different tasks. Maybe you can have a separate
script/service to do the log consolidate job since the log structure and
format are known.

Kevin Y

On Mon, Apr 30, 2018 at 12:16 PM, mad2271@xxxxxxxxxxxx <mad2271@xxxxxxxxxxxx
> wrote:

> Hi Mailing List,
> Is there a way to consolidate airflow task logs by stdout and stderr?
> Currently, the structure is something like /logs/taskname/task_run_date/log1.txt
> which is the log for a particular taskname at a particular run date.  What
> I would like is two large log files for all tasks, something like
> /logs/errors.txt and /logs/outputs.txt  Which would contain all the stderr
> and stdout messages for all runs of all tasks regardless of run date. I
> essentially want two very large log files.  For example, if I have task A
> and task B, instead of having two directories and then subdirectories for A
> and B, I would just like two files one with errors from A and B and one for
> outputs from A and B. Does airflow provide this information?
> Thanks!