[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: Airflow 1.10 Migration Duration

I checked with our Ops guy and he mentioned that when he upgraded from
1.8.x to 1.9.x, it took a few seconds. We had 3M rows in the task_instance
table and run MySQL 5.7.


On Tue, Sep 25, 2018 at 4:54 PM Matt Davis <jiffyclub@xxxxxxxxx> wrote:

> Hi folks,
> Here at Clover we're excitedly migrating to Airflow 1.10 (thanks for
> everyone's hard work on that!). We're finding that it's taking about 2
> hours to apply all the migrations to go from Airflow 1.8 to 1.10, largely
> driven by the 10 million rows in our task_instance table. That got us
> wondering what kind of maintenance people do on their Airflow metadata
> databases. Do folks mostly put up with long migrations and generally longer
> queries, or are y'all doing periodic cleanups of your metadata DB to keep
> it fairly light?
> Thanks,
> Matt Davis