osdir.com

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: Parallelizing test runs


Hey Rafael, looks like we need more 'INSTANCE_TEMPLATES' quota [1]. Can you take a look? I've filed [BEAM-4722]: https://issues.apache.org/jira/browse/BEAM-4722 

[1] https://github.com/apache/beam/pull/5861#issuecomment-401963630 

On Mon, Jul 2, 2018 at 11:33 AM Rafael Fernandez <rfernand@xxxxxxxxxx> wrote:
OK, Scott just sent https://github.com/apache/beam/pull/5860 . Quotas should not be a problem, if they are, please file a JIRA under gcp-quota.

Cheers,
r

On Mon, Jul 2, 2018 at 10:06 AM Kenneth Knowles <klk@xxxxxxxxxx> wrote:
One thing that is nice when you do this is to be able to share your results. Though if all you are sharing is "they passed" then I guess we don't have to insist on evidence.

Kenn

On Mon, Jul 2, 2018 at 9:25 AM Scott Wegner <scott@xxxxxxxxxx> wrote:
A few thoughts:

* The Jenkins job getting backed up is beam_PostCommit_Java_ValidatesRunner_Dataflow_Gradle_PR [1]. Since Mikhail refactored Jenkins jobs, this only runs when explicitly requested via "Run Dataflow ValidatesRunner", and only has 8 total runs. So this job is idle more often than backlogged.

* It's difficult to reason about our exact quota needs because Dataflow jobs get launched from various Jenkins jobs that have different parallelism configurations. If we have budget, we could enable concurrent execution of this job and increase our quota enough to give some breathing room. If we do this, I recommend limiting the max concurrency via throttleConcurrentBuilds [2] to some reasonable limit.

* This test suite is meant to be an exhaustive post-commit validation of Dataflow runner, and tests a lot of different aspects of a runner. It would be more efficient to run locally only the tests affected by your change. Note that this requires having access to a GCP project with billing, but most Dataflow developers probably have access to this already. The command for this is:

./gradlew :beam-runners-google-cloud-dataflow-java:validatesRunner -PdataflowProject=myGcpProject -PdataflowTempRoot=gs://myGcsTempRoot --tests "org.apache.beam.MyTestClass"


On Mon, Jul 2, 2018 at 8:33 AM Lukasz Cwik <lcwik@xxxxxxxxxx> wrote:
The validates runner test parallelism is controlled here and is currently set to be "unlimited":
https://github.com/apache/beam/blob/fbfe6ceaea9d99cb1c8964087aafaa2bc2297a03/runners/google-cloud-dataflow-java/build.gradle#L115

Each test fork is run on a different gradle worker, so the number of parallel test runs is limited to the max number of workers configured which is controlled here:
It is currently configured to 3 * number of CPU cores.

We are already running up to 48 Dataflow jobs in parallel.


On Sat, Jun 30, 2018 at 9:51 AM Rafael Fernandez <rfernand@xxxxxxxxxx> wrote:
- How many resources to ValidatesRunner tests use?
- Where are those settings?

On Sat, Jun 30, 2018 at 9:50 AM Reuven Lax <relax@xxxxxxxxxx> wrote:
The specific issue only affects Dataflow ValidatesRunner tests. We currently allow only one of these to run at a time, to control usage of Dataflow and of GCE quota. Other types of tests do not suffer from this issue.

I would like to see if it's possible to increase Dataflow quota so we can run more of these in parallel. It took me 8 hours end to end to run these tests (about 6 hours for the run to be scheduled). If there was a failure, I would have had to repeat the whole process. In the worst case, this process could have taken me days. While this is not as pressing as some other issues (as most people don't need to run the Dataflow tests on every PR), fixing it would make such changes much easier to manage.

Reuven

On Sat, Jun 30, 2018 at 9:32 AM Rafael Fernandez <rfernand@xxxxxxxxxx> wrote:
+Reuven Lax told me yesterday that he was waiting for some test to be scheduled and run, and it took 6 hours or so. I would like to help reduce these wait times by increasing parallelism. I need help understanding the continuous minimum of what we use. It seems the following is true:

  • There seems to always be 16 jenkins machines on (16 CPUs each)
  • There seems to be three GKE machines always on (1 CPU each)
  • Most (if not all) unit tests run on 1 machine, and seem to run one-at-a-time <-- I think we can safely parallelize this to 20.
With current quotas, if we parallelize to 20 concurrent unit tests, we still have room for 80 other concurrent dataflow jobs to execute, with 75% of CPU capacity. 

Thoughts? Additional data?

Thanks,
r