[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Proposed changes to CircleCI testing workflow

I'd like to give you a quick update on the work that has been done
lately on running tests using CircleCI. Please let me know if you have
any objections or don't think this is going into the right direction, or
have any other feedback!

We've been using CircleCI for a while now and results are used on
constant basis for new patches. Not only by committers, but also by
casual contributors to run unit tests. Looks like people find the
service valuable and we should keep using it. Therefor I'd like to make
some improvements that will make it easier to add new tests and to
continue making CircleCI an option for all contributors, both on paid
and free plans.

The general idea of the changes implemented in #14806, is to consolidate
the existing config to make it more modular and have smaller jobs that
can be scheduled ad-hoc by the developer, instead of running a few big
jobs on every commit. Reorganizing and breaking up the existing config
was done using the new 2.1 config features. Starting jobs on request,
instead of automatically, is done using the manual approval feature,
i.e. you now have to click on that job in the workflow page in order to
start it. I'd like to see us having smaller, more specialized groups of
tests that we can run more selectively during development, while still
being able to run bigger tests before committing, or firing up all of
them during testing and releasing. Other example of smaller jobs would
be testing coverage (#14788) or cqlsh tests (#14298). But also
individual jobs for different ant targets, like burn, stress or benchmarks.

We'd now also be able to run tests using different docker images and
different JDKs. I've already updated the used image to also include Java
11 and added unit and dtest jobs to the config for that. It's now really
easy to run tests on Java 11, although these won't pass yet. It seems to
be important to me to have this kind of flexibility, given the
increasingly diverse ecosystem of Java distributions. We can also add
jobs for packaging and doing smoke tests by installing and starting
packages on different docker images (Redhat, Debian, Ubuntu,..) at a
later point.

As for the paid vs free plans issue, I'd also like us to discuss how we
can make tests faster and less resource intensive in general. As a
desired consequence, we'd be able to move away from multi-node dtests,
to something that can be run using the free plan. I'm looking forward to
see if #14821 can get us into that direction. Ideally we can add these
tests into a job that can be completed on the free plan and encourage
contributors to add new tests there, instead of having to write a dtest,
which they won't be able to run on CircleCI without a paid plan.

Whats changing for you as a CircleCI user?
* All tests, except unit tests, will need to be started manually and
will not run on every commit (this can be further discussed and changed
anytime if needed)
* Updating the config.yml file now requires using the CircleCI cli tool
and should not be done directly (see #14806 for technical details)
* High resource settings can be enabled using a script/patch, either run
manually or as commit hook (again see ticket for details)
* Both free and paid plan users now have more tests to run

As already mentioned, please let me know if you have any thoughts on
this, or if you think this is going into the wrong direction.


To unsubscribe, e-mail: dev-unsubscribe@xxxxxxxxxxxxxxxxxxxx
For additional commands, e-mail: dev-help@xxxxxxxxxxxxxxxxxxxx