[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: Looking for collaborators

Hello Everyone,

We recently worked on an open source Machine Learning Work Bench, (not
released to public yet) where a "github for ML models" is envisioned. It is
fundamentally based on the idea that an ML Application a DAG and is
specified via an YAML file and under the hood it is parsed and mapped to an
airflow DAG.

We wanted this tool to be an open source alternative to the commercial
offerings such as a Amazon SageMaker, Google Cloud ML etc.. AirFlow is seen
as one of the alternatives to the orchestrators. Others being Dask for

I see that the compiling/parsing yaml suggested in this threads fits into
what we already did, and there is huge scope to take that idea forward.
More about the principles and the scope is discussed in my blog here
If there is interest, we can push the project to the public and collaborate.

Btw, I initiated a baby experimental project to see the application of
KanBans as a way capture the nodes in a DAG here. Technically it is
possible to spec an ML App as a Kanban and parse it into an airflow DAG.


That specification needs to support CRUD operations an ML App.

yet, where we used AirFlow as one of the DAG orchestrators.

On Tue, Oct 2, 2018 at 12:10 PM Sai Phanindhra <phani8996@xxxxxxxxx> wrote:

> Hi Siddarth,
>      Your idea sounds interesting and challenging. I would like to
> contribute to it.
> On Tue, 2 Oct 2018 at 06:08, siddharth gupta <sid88in@xxxxxxxxx> wrote:
> > Hello Folks!
> >
> > I am working on an Airflow Plugin to auto generate DAG code from the UI
> and
> > YAML (already have initial design on how it can work etc). I am looking
> for
> > folks interested in collaborating. Please reach out to me if you are
> > interested :)
> >
> > You can reach out to me at sid88in@xxxxxxxxx
> >
> > Thanks
> > Sid
> >
> --
> Sai Phanindhra,
> Ph: +91 9043258999