[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: Java > 8 support

When we were supporting Java 7, there were Java 8 based modules in Maven. We can follow a similar pattern where Java11 (or any other version) have their own projects and as the base Java version moves up, we can merge those modules back in to simplify the set of modules being developed.

On Wed, Oct 17, 2018 at 8:14 AM Ismaël Mejía <iemejia@xxxxxxxxx> wrote:
I want to give more context on the previous work towards Java 11 support. The issue was not necessarily that gradle was not ok to support Java 11, but this is also a good point Lukasz.

Months ago, we built a maven profile that basically set up Java to be strictly compatible with the ‘minimal’ java.base profile and used Java version (9/10) at the time. We found and fixed some missing dependencies (because Java 11 removed some of the Java EE stuff from the base profile), not sure if these changes were migrated to the gradle build.

We also dealt with a good chunk of updates at the time of plugins/bytebuddy and other deps to guarantee that the execution was done strictly with the latest version of Java and its generated bytecode (the origin of the docker build images was to guarantee this isolation). At the time I run the tests of most IOs and most extensions with the existing (java 8 compiled) direct runner in JRE 10 with success but of course this was not the ultimate goal but a good way to evaluate the state of the code base (I remember some GCP tests were broken and the ApiSurfaceTests too).

One important thing to remember is that the goal on this work was ‘limited’ to guarantee that a user that had ONLY Java 11 installed could build and generate Beam artifacts and run Beam programs in the direct runner. The limited scope was in part because of some issues of a full migration to Java 11:

- Support for Java modules, this is a too much radical change and probably far from the scope of Beam at the moment.
- Support for other runners, given that most execution systems do not support yet Java 11 this could imply a lot of extra work. In the meantime, this could be easier now due to the portability work, anyway, still not sure that Hadoop/Spark/Flink will be supporting Java 11 anytime soon.

Let’s not forget that we still are and probably will be for a long time backwards compatible with Java 8 so we need to ensure that Java 11 only things don’t get into the code base.

If you Lukasz or someone else is up to take this task, I think the immediate thing to do is to get back to the same state we were before the move to gradle, build the equivalent of the maven profile for the gradle build and tackle code generation and classpath issues, then try to fix missing parts and push this into the CI too (similar to the recent python 3 work) to guarantee that new changes don’t break Java 11 compatibility. I sadly lack of bandwidth to lead this task but I will gladly help with reviews, or feedback if someone else is up to the task.

Not sure about more recent details, but probably worth evaluating at some moment is if we there is some interest on supporting multi-release jars with Java 11 classes or the dream scenario of having a AOT build of the direct runner.

On Wed, Oct 17, 2018 at 12:06 PM Łukasz Gajowy <lukasz.gajowy@xxxxxxxxx> wrote:
FYI: If I'm correct, migrating to Java 11 may require Migrating to Gradle 5, which to me is ok, but there are some issues on the way (such as [1] or maybe more that I'm not aware of). 

I did some research and it seems that Groovy 2.4.15 (the default version of Groovy for Gradle 4.10.2) is unable to work with Java 11 due to Nest based access control problem [2]. The very same error happened when I tried to compile "core" module with java11. This issue is solved in Groovy 2.5.3 [3]I was unable to run Gradle 4.10.2 with Groovy 2.5.3 (NoClassDefError) and found out that the support for Groovy 2.5.3 will be added in Gradle 5.0RC1 (milestone) [4]. 

Other than that I had some issues with spottless and findbugs (I turned them off for the sake of my experiment). 


czw., 11 paź 2018 o 13:10 Łukasz Gajowy <lukasz.gajowy@xxxxxxxxx> napisał(a):
FWIW, the issue regarding java 11 support to gradle might be helpful here [1]. Quoting: "there are only some minor corner cases that don't work. Overall Java 11 support is pretty solid already" but some users reported problems in comments with Checkstyle plugin and MacOS + Gradle4.10.2 (this might be important for us). 


[1] https://github.com/gradle/gradle/issues/5120

czw., 11 paź 2018 o 02:06 Pablo Estrada <pabloem@xxxxxxxxxx> napisał(a):
Hello all,
If I understand you correctly Ismael, a good amount of 'beam-sdks-java-core' tests are already passing with Java 11, so the amount of work necessary on the core module should be relatively small. Is this correct? Are there improvements that may be missing in terms of modularization?

There is also the work necessary to build/run tests with Gradle....

I am also curious... how much work do you estimate is necessary to support Java 11 with some of the existing sources? I understand that we have many, many sources, but perhaps some of the more popular ones (e.g. TextIO)?


On Wed, Oct 10, 2018 at 12:59 AM Arif Kasim <arifkasim@xxxxxxxxxx> wrote:
Thanks for the clarification Ismaël.

  •  Arif Kasim
  •  Strategic Cloud Engineer
  •  Google, Inc.
  •  arifkasim@xxxxxxxxxx  

On Wed, Oct 10, 2018 at 9:41 AM Ismaël Mejía <iemejia@xxxxxxxxx> wrote:
Just wanted to clarify, there is already a JIRA for ongoing work on
Java 11 support.

I led the initial work on supporting what at the time was Java 9/10,
so far the biggest blockers were around the ApiSurface tests (not at
all compatible with these versions) but at the time we were at 5 tests
from getting sdks/core passing. Notice also that the scope of this
JIRA evolved to support only the LTS version (Java 11), and
specifically to support only sdks/core + direct runner. Supporting all
IOs or runners really is more a question of the dependencies working
nicely with Java 11 so this will probably take long time. Also the
idea so far does NOT include supporting the Java module system at all.

I stopped working on this during the move to gradle because it was too
hard to tackle both Java evolving and all the ongoing changes in the
build system. If somebody in the community wants to contribute in this
area it will be greatly appreciated, notice that all the work we did
on the build system for this needs to be implemented now in gradle
On Sat, Oct 6, 2018 at 5:55 PM Romain Manni-Bucau <rmannibucau@xxxxxxxxx> wrote:
> @Reuven: bytebuddy by itself no but the way beam tries to inject the proxy class is. There are other strategies you can use in bytebuddy which work.
> Romain Manni-Bucau
> @rmannibucau |  Blog | Old Blog | Github | LinkedIn | Book
> Le sam. 6 oct. 2018 à 17:51, Reuven Lax <relax@xxxxxxxxxx> a écrit :
>> Romain, do you have any more details on the ByteBuddy incompatibility? Is ByteBuddy incompatible with the Java 11 JRE, or just with new language features?
>> On Fri, Oct 5, 2018 at 10:20 AM Romain Manni-Bucau <rmannibucau@xxxxxxxxx> wrote:
>>> Hi Arif,
>>> AFAIK bytebuddy code is not java 11 friendly otherwise it runs (but it means your pipeline is very very simple since it does not have a dofn ;)) if your engine supports it. Also note that the modules not being named you can have to use some weird import names or even unstable ones if you want to use modules (but there is no real reason to do that yet in java).
>>> Romain Manni-Bucau
>>> @rmannibucau |  Blog | Old Blog | Github | LinkedIn | Book
>>> Le ven. 5 oct. 2018 à 19:10, Arif Kasim <arifkasim@xxxxxxxxxx> a écrit :
>>>> Hello,
>>>> What's the status of java version > 8 support for beam? Thanks.
>>>> -Arif.