[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: feature suggestion

The new lineage functionality (while in very early states) can do this for you.


> On 20 Jun 2018, at 19:46, Andrew Harmon <andrewhharmon@xxxxxxxxx> wrote:
> Hello,
> I wanted to submit a feature suggestion. I have a scenario where I'd like
> to create a subdag where the number of tasks in it are dependent on a
> previous task. Ideally, this parameter could  be passed in via xcom. For
> example:
> Task A does something that generates 10 files. Task B is a subdag that at
> runtime, creates 10 tasks within the sub dag. Each task will process one of
> the 10 files. That way the processing of the 10 files can run in parallel.
> Task C will then start when all tasks in the subdag are complete.
> My particular scenario is to process the results of a BigQuery export to
> GCS. The export creates n number of files and you don't know that number
> until the export is done running.
> Thanks,
> Andrew
> -- 
> Andrew Harmon
> (202) 615-6433