OSDir

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: Assigning multiple pools to a task


Yes,  after looking at the code, it looks like implementing a many to many
between pools and tasks requires significant redesign.

My workaround is to implement it externally by borrowing required resources
from an external pool using a sensor, and returning them back once the task
is done using another sensor.  They are basically entry/exit sensors for a
task

Thx

On Sat, May 12, 2018, 2:28 PM Dennis O'Brien <dennis@xxxxxxxxxxxxxxxx>
wrote:

> A task is assigned to a pool by the task specifying the name of the pool.
> The docs suggest that the pool argument is a string, not a list of strings.
> https://airflow.apache.org/code.html#baseoperator
>
> And looking at the code it does seem like this relationship of one task
> assigned to zero or one pools is strict.
>
> It's an interesting idea of trying to associated a task with multiple
> pools.  I think things would get complicated fast to avoid deadlock
> situations.  Do you know of any other work flow systems that support
> one-to-many relationship?
>
> Is it possible to bundle the resources into a single resource?  I'm
> guessing this is not an option, and that tasks use different combinations
> of these resources.
>
> cheers,
> Dennis
>
>
> On Fri, May 11, 2018 at 1:04 PM Neelesh <neeleshs@xxxxxxxxx> wrote:
>
> > Hello,
> >
> > Is it possible to assign multiple pools to a task?
> >
> > I'm wrapping a bash operator around some heavy-weight custom data
> > processing script that cannot be modified. This script cannot be
> modified,
> > and it uses multiple resources. Im trying to use pools as proxies for
> these
> > resources.
> >
> > Thanks
> >
>