[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [DISCUSS] Gradle for the build ?

Hi all!
Coming from the Go SDK side, I'm in favour of splitting the repo after the portability APIs have stablized. The reason being that it would be all to easy for changes not be propagated to the SDKs, which would cause painful drift.

As for the Go Gradle experience, it agree it leaves much to be desired, since it's not cleaning up after itself in it's attempt to ensure reproduceable go builds. BEAM-5379 has been filled as a possible solution to that, with the suggestion to move to the Go Modules supported by Go 1.11.

Resolving this is on the short list of contributions I want to make this quarter.

From a pure Go SDK contributor standpoint, I generally use the standard go tools (mostly the go tool) and its existing IDE integrations rather than attempt to make Gradle work for me, outside of building the core components. I can't speak for everyone of course but it's my view on it.

On Thu, Oct 11, 2018, 10:33 AM Udi Meiri <ehudm@xxxxxxxxxx> wrote:
I agree with the points made that our Gradle configuration is too complicated (learning Groovy and BeamModulePlugin come to mind).

But I mainly write for Python SDK, and I've done experiments in the past with Bazel builds, which was a much better fit for Python as an incremental and parallel build and test system.

On Thu, Oct 11, 2018 at 4:56 AM Alexey Romanenko <aromanenko.dev@xxxxxxxxx> wrote:
I agree with all previous points related to Gradle-related problems. Among all others, I’d highlight IDE integration issue since it can have real negative impact on attracting new contributors and, finally, community grow. The more obstacles there are to start code contribution, the more chances that it will distract newcomers.

In this way I see two points to do:
1) Continue working on IDE integration improvement. I hope there are no principal barriers for that and it’s only a question of applied forces, otherwise, it can be a very serious issue.
2) Improve our “Get started” documentation for contributors to make the process of first-time contribution as smooth and simple as possible.
On 11 Oct 2018, at 00:22, Tim Robertson <timrobertson100@xxxxxxxxx> wrote:

Thank you JB for starting this discussion.

Others comment on many of these points far better than I can, but my experience is similar to JB.

1. IDEA integration (and laptop slowing like crazy) being the biggest contributor to my feeling of being unproductive
2. Not knowing the correct way to modify the build scripts which I put down to my own limitations

It seems we also need to help build Gradle expertise in our community, so that those that are motivated are empowered to contribute.

Nicely phrased. +1

On Wed, Oct 10, 2018 at 7:15 PM Scott Wegner <scott@xxxxxxxxxx> wrote:
> Perhaps we should go through and prioritize (and add missing items to) BEAM-4045

+1. It's hard to know where to start when there's such a laundry list of tasks. If you're having build issues, will you make sure it is represented in BEAM-4045, and "Vote" for the issues that you believe are the highest priority?

I agree that the Gradle build is far from perfect (my top gripes are IDE integration and parallel/incremental build support). I believe that we're capable of making our build great, and continuing our investment in Gradle would be a shorter path than changing course again. Remember that our Maven build also had it's share of issues, which is why we as a community voted to replace it [1][2].

It seems we also need to help build Gradle expertise in our community, so that those that are motivated are empowered to contribute. Does anybody have a good "Getting Started with Gradle" guide they recommend? Perhaps we could also link to it from the website/wiki.

[1] https://lists.apache.org/thread.html/225dddcfc78f39bbb296a0d2bbef1caf37e17677c7e5573f0b6fe253@%3Cdev.beam.apache.org%3E
[2] https://lists.apache.org/thread.html/bd399ecb17cd211be7c6089b562c09ba9116649c9eabe3b609606a3b@%3Cdev.beam.apache.org%3E

On Wed, Oct 10, 2018 at 2:40 AM Robert Bradshaw <robertwb@xxxxxxxxxx> wrote:
Some rough stats (because I was curious): The gradle files have been edited by ~79 unique contributors over 696 distinct commits, whereas the maven ones were edited (over a longer time period) by ~130 unique contributors over 1389 commits [1]. This doesn't capture how much effort was put into these edits, but neither is restricted to a small set of experts. 

Regarding "friendly for other languages" I don't think either is necessarily easy to learn, but my impression is that the maven learning curve shallower for those already firmly embedded in the Java ecosystem (perhaps due to leveraging existing familiarity, and perhaps some due to the implicit java-centric conventions that maven assumed about your project), whereas with gradle at least I could keep pulling on the string to unwind things to the bottom. The "I just want to build/test X without editing/viewing the build files" seemed more natural with Gradle (e.g. I can easily list all tasks).

That being said, I don't think everyone needs to understand the full build system. It's important that there be a critical mass that do (we have that for both, and if we can simplify to improve this that'd be great), it's easy enough to do basic changes (e.g. add a dependency, again I don't think the barrier is sufficiently different for either), and works well out of the box for someone who just wants to look up a command on the website and edit code (the CLI is an improvement with Gradle, but it's clear that (java) IDE support is a significant regression). 

Personally, I don't know much about IDE configuration (admittedly the larger issue), but one action item I can take on is trying to eliminate the need to do a "git clean" after building certain targets (assuming I can reproduce this).

Perhaps we should go through and prioritize (and add missing items to) BEAM-4045 https://issues.apache.org/jira/issues/?jql=parent%20%3D%20BEAM-4045%20ORDER%20BY%20priority%20DESC ? There's always a long tail with this kind of thing, and looking at the whole list can be daunting, but putting it in the correct order and knocking off the top N items could possibly go a long way. 

- Robert

[1] The commands I ran were (with and without the uniq)

$ find . -name 'build.gradle' | xargs git log | grep Author: | grep -o '[^< ]*@' | sort | uniq | wc
$ find . -name 'pom.xml' | xargs git log | grep Author: | grep -o '[^< ]*@' | sort | uniq | wc

On Wed, Oct 10, 2018 at 10:31 AM Etienne Chauchot <echauchot@xxxxxxxxxx> wrote:
Hi all,
I must admit that I agree on the status especially regarding 2 points:
1. new contributors obstacles: gradle learning curve might be too long for spare-time contributors, also complex scripted build takes time to understand comparing to self-descriptive one.
2. IDE integration kind of slows down development.

Now, regarding how we improve the situation, I think we need to discuss and identify tasks and tackle them all together even if they are not sexy tasks as Ismaël mentioned.


Le mardi 09 octobre 2018 à 10:04 +0200, Jean-Baptiste Onofré a écrit :
Hi guys,

I know that's a hot topic, but I have to bring this discussion on the table.

Some months ago, we discussed about migrating our build from Maven to
Gradle. One of the key expected improvement was the time to build.
We proposed to do a PoC to evaluate the impacts and improvements, but
this PoC was actually directly a migrate on master.

Now, I would like to bring facts here:

1. Build time
On my machine, the build time is roughly 1h15. It's pretty long, and
regarding what the build is doing, I don't see huge improvement provided
by Gradle.
2. Build reliability
Even worse, most of the time, we need to use --no-parallel and
--no-daemon to have a reliable build (it's basically recommended for
release). It has an impact on build time, and we loose part of Gradle
3. Release and repositories
Even if couple of releases has been performed with Gradle, it's not
obvious to see improvements around artifacts handling. I got my
repository polluted twice (that's part of the trick Gradle is doing to
speed up the build dealing around the repository).
4. IDE integration
We already had some comments on the mailing lists about the IDE
integration. Clearly, the situation is not good on that front too. The
integration on IDE (especially IntelliJ) is not good enough right now.

We are working hard to grow up the community, and from a contributor
perspective, our build system is not good today IMHO.
As a contributor, I resumed my work on some PRs, and I'm spending so
much time of the build, largely more than working on the PRs code itself.

So, obviously, the situation is not perfect, at least from a contributor

The purpose of this thread is not again to have a bunch of replied
ending nowhere. I would like to be more "pushy" and let's try to be
concrete. So basically, we only have two options:

1. Improve the build, working hard on Gradle front. Not sure if it makes
such sense from a contributor perspective, as Maven is really well known
from most of contributors (and easier to start with IMHO).
2. Back on Maven. That's clearly my preferred approach. IDE integration
is better, Maven is well known from the contributors as already said.
The effort is not so huge. We tried to use Gradle, we don't have the
expected results now, that's not a problem, it's part of a project lifetime.

Thoughts ?