OSDir


[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: Vendoring / Shading Protobuf and gRPC


That's really cool !

I'm thinking about the Spark runner where grpc deps are not easy to deal
with.

Thanks !
Regards
JB

On 10/07/2018 23:09, Lukasz Cwik wrote:
> With the merge of PR #5594[1], we started shading all gRPC / Protobuf
> dependencies within all the modules that depended on the model/*
> dependencies by vendoring them. The vendored versions are built and
> packaged into the model jars (they should be separated out once I figure
> out how to generate proto code using a shaded import path). Note that
> this cleaned up several issues where we were incorrectly built shaded
> jars without repackaging in some locations or the shading process was
> corrupting the contents of some of the jars.
> 
> Note that the majority of the code base (especially related to
> portability) should be using imports under the
> org.apache.beam.vendor.protobuf.v3 or org.apache.beam.vendor.grpc.v1
> paths. I have yet to figure out a clean way to get Intellij to recognize
> these vendored paths. My only solution so far has been to manually add
> one of the built model jars to the compile classpath of the module being
> worked on in Intellij as described here[2]. I would greatly appreciate
> some ideas on how to improve this integration because from a few
> attempts configuring the intellij gradle pluglin scope sections didn't
> produce the result that I was expecting.
> 
> I also added a simple test
> task /validateShadedJarDoesntLeakNonOrgApacheBeamClasses/ that validates
> the shaded jar doesn't contain classes without repackaging which we
> should apply to any module that performs shading to ensure that classes
> are relocated and we don't accidentally expose stuff. I filed
> BEAM-4753[3] to this end.
> 
> Note, these are the following modules that still depend on protobuf that
> are shaded away and could move to use a vendored variant of protobuf:
> * sdks/java/core
> * sdks/java/extensions/sql
> 
> These modules expose protobuf because it is part of the API surface:
> * sdks/java/extensions/protobuf
> * sdks/java/io/google-cloud-platform (I believe that gRPC could be
> shaded here but preferrably the IO module would do it so we wouldn't
> have this maintenance burden.)
> 
> 1: https://github.com/apache/beam/pull/5594
> 2: https://stackoverflow.com/questions/1051640/correct-way-to-add-external-jars-lib-jar-to-an-intellij-idea-project
> 3: https://issues.apache.org/jira/browse/BEAM-4753

-- 
Jean-Baptiste Onofré
jbonofre@xxxxxxxxxx
http://blog.nanthrax.net
Talend - http://www.talend.com