[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [DISCUSS] Unsustainable situation with ptests

+1 There are many problems with the test infrastructure and in my opinion
it has not become number one bottleneck for the project. I was looking at
the infrastructure yesterday and I think the current infrastructure (even
its own set of problems) is still under-utilized. I am planning to increase
the number of threads to process the parallel test batches to start with.
It needs a restart on the server side. I can do it now, it folks are okay
with it. Else I can do it over weekend when the queue is small.

I listed the improvements which I thought would be useful under
https://issues.apache.org/jira/browse/HIVE-19425 but frankly speaking I am
not able to devote as much time as I would like to on it. I would
appreciate if folks who have some more time if they can help out.

I think to start with https://issues.apache.org/jira/browse/HIVE-19429 will
help a lot. We need to pack more test runs in parallel and containers
provide good isolation.

For the short term green runs, I think we should @Ignore the tests which
are known to be failing since many runs. They are anyways not being
addressed as such. If people think they are important to be run we should
fix them and only then re-enable them.

Also, I feel we need light-weight test run which we can run locally before
submitting it for the full-suite. That way minor issues with the patch can
be handled locally. May be create a profile which runs a subset of
important tests which are consistent. We can apply some label that
pre-checkin-local tests are runs successful and only then we submit for the

More thoughts are welcome. Thanks for starting this conversation.

On Fri, May 11, 2018 at 1:27 PM, Jesus Camacho Rodriguez <
jcamacho@xxxxxxxxxx> wrote:

> I believe we have reached a state (maybe we did reach it a while ago) that
> is not sustainable anymore, as there are so many tests failing / timing out
> that it is not possible to verify whether a patch is breaking some critical
> parts of the system or not. It also seems to me that due to the timeouts
> (maybe due to infra, maybe not), ptest runs are taking even longer than
> usual, which in turn creates even longer queue of patches.
> There is an ongoing effort to improve ptests usability (
> https://issues.apache.org/jira/browse/HIVE-19425), but apart from that,
> we need to make an effort to stabilize existing tests and bring that
> failure count to zero.
> Hence, I am suggesting *we stop committing any patch before we get a green
> run*. If someone thinks this proposal is too radical, please come up with
> an alternative, because I do not think it is OK to have the ptest runs in
> their current state. Other projects of certain size (e.g., Hadoop, Spark)
> are always green, we should be able to do the same.
> Finally, once we get to zero failures, I suggest we are less tolerant with
> committing without getting a clean ptests run. If there is a failure, we
> need to fix it or revert the patch that caused it, then we continue
> developing.
> Please, let’s all work together as a community to fix this issue, that is
> the only way to get to zero quickly.
> Thanks,
> Jesús
> PS. I assume the flaky tests will come into the discussion. Let´s see
> first how many of those we have, then we can work to find a fix.