OSDir


[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

feature suggestion


Hello,

I wanted to submit a feature suggestion. I have a scenario where I'd like
to create a subdag where the number of tasks in it are dependent on a
previous task. Ideally, this parameter could  be passed in via xcom. For
example:

Task A does something that generates 10 files. Task B is a subdag that at
runtime, creates 10 tasks within the sub dag. Each task will process one of
the 10 files. That way the processing of the 10 files can run in parallel.
Task C will then start when all tasks in the subdag are complete.

My particular scenario is to process the results of a BigQuery export to
GCS. The export creates n number of files and you don't know that number
until the export is done running.

Thanks,
Andrew
-- 
Andrew Harmon
(202) 615-6433