OSDir


[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: Rename Nexmark jobs to Perf instead of PostCommit


@Mikhail I must disagree. I have set Nexmark tests as PostCommit mainly because they were used to discover regressions on all the beam model as soon as possible: when the new regression commit comes into master. The graphs (see https://beam.apache.org/documentation/sdks/java/nexmark/) help a lot in that way. As Reuven said, performance testing is more a second aim even if it is part of regression.

Etienne

Le mardi 14 août 2018 à 15:40 -0700, Mikhail Gryzykhin a écrit :
@Reuven Lax Thank you for clarifying. It makes sense to keep them as post-commits then.

@Ahmet Altay I don't see them being flaky atm. So we are good here. 

I would prefer to separate perf tests mostly because of requirements, such as runtime, environment, trigger restrictions. Right now we have a separate setup with no GHPRB phrase trigger for Nexmark tests.

My understanding was that due to the "performance" nature of Nexmark tests, that we can not treat them as post-commit tests and have to create special setup for them. 

--Mikhail

Have feedback


On Mon, Aug 13, 2018 at 10:37 AM Ahmet Altay <altay@xxxxxxxxxx> wrote:
I replied before seeing Reuven's comment. In that case it makes sense to keep them as post commit tests.

Are these test flaky because they fail some performance metrics? If that is the case it makes sense to separate them from functional tests. 

On Mon, Aug 13, 2018 at 10:30 AM, Reuven Lax <relax@xxxxxxxxxx> wrote:
Nexmark was primarily written as a set of functionality tests, not performance tests. In fact some of the Nexmark queries are written in knowingly inefficient ways, in order to better exercise more features of Beam. We also use Nexmark as performance tests (mostly out of convenience, because we already have those tests), however I believe that is a secondary use.

Reuven

On Mon, Aug 13, 2018 at 9:36 AM Mikhail Gryzykhin <migryz@xxxxxxxxxx> wrote:
Hi everyone,

As I can understand, a lot of tests in Nexmark set are performance tests. I suggest to rename(or split) the set to performance tests. 

Performance tests are much less reliable compared to post-commit tests and should have different requirements. Additionally, they are much more flaky.

Splitting out performance tests to separate set will allow us to treat failures with lower priority and add more tolerance for flakes compared to what we have decided for post-commit tests.

This will also be more organic to use different builder from PostcommitJobBuilder, since we will want different requirements for perf tests.

I do not believe we have a problem with this in current state, but I expect this to become an issue in the future as amount of perf tests grows.

Regards,
--Mikhail

Have feedback