OSDir


[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: feature suggestion


ok, just took a look at the latest docs. Not sure I fully understand it,
but i guess you can pass a value as an outlet into the subdag? And the
subdag can use that to create n task?  i'm a bit weary of using it if it's
not in a stable release. Is there a time frame on 1.10 release?

Thanks,
Andrew

On Wed, Jun 20, 2018 at 3:00 PM Bolke de Bruin <bdbruin@xxxxxxxxx> wrote:

> The new lineage functionality (while in very early states) can do this for
> you.
>
> B.
>
> > On 20 Jun 2018, at 19:46, Andrew Harmon <andrewhharmon@xxxxxxxxx> wrote:
> >
> > Hello,
> >
> > I wanted to submit a feature suggestion. I have a scenario where I'd like
> > to create a subdag where the number of tasks in it are dependent on a
> > previous task. Ideally, this parameter could  be passed in via xcom. For
> > example:
> >
> > Task A does something that generates 10 files. Task B is a subdag that at
> > runtime, creates 10 tasks within the sub dag. Each task will process one
> of
> > the 10 files. That way the processing of the 10 files can run in
> parallel.
> > Task C will then start when all tasks in the subdag are complete.
> >
> > My particular scenario is to process the results of a BigQuery export to
> > GCS. The export creates n number of files and you don't know that number
> > until the export is done running.
> >
> > Thanks,
> > Andrew
> > --
> > Andrew Harmon
> > (202) 615-6433
>
> --
Andrew Harmon
(202) 615-6433