The need for LocalTaskJob
Hi Max, Dan et al,
Currently, when a scheduled task runs this happens in three steps:
3. Raw task instance
It uses (by default) 5 (!) different processes:
2. Bash + Airflow
3. Bash + Airflow
I think we can merge worker and LocalTaskJob as the latter seems exist only to track a particular task. This can be done within the worker without side effects. Next to thatI think we can limit the amount of (airflow) processes to 2 if we remove the bash dependency. I don’t see any reason to depend on bash.
Can you guys shed some light on what the thoughts were around those choices? Am I missing anything on why they should exist?
Verstuurd vanaf mijn iPad