OSDir


[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [VOTE] Release 1.6.0, release candidate #4


I did not use the tools/list_deps.py script as I wasn't aware that it existed.

Even if I were I wouldn't have used it and in fact would advocate for removing it. It manually parses and constructs dependency information which is utterly unnecessary as maven already provides this functionality, with the added bonus of also accounting for dependencyManagement and transitive dependencies which we obviously have to take into account.

I used this one-liner instead:
|mvn dependency:list | ||grep| |":.*:.*:.*"| || ||grep| |-||v| |-e ||"Finished at"| |-e ||"Some problems"| || ||cut| |-d] -f2- | ||sed| |'s/:[a-z]*$//g'| || ||sort| |-u

|which I have documented here: https://cwiki.apache.org/confluence/display/FLINK/Dependencies

On 08.08.2018 15:06, Aljoscha Krettek wrote:
+1

- verified checksum and gpg files
- verified LICENSE and NOTICE: NOTICE didn't change from 1.5, LICENSE had one unnecessary part removed

Side comment: I'm not sure whether the "Verify that the LICENSE and NOTICE file is correct for the binary and source releases" part is valid anymore because we only have one LICENSE and NOTICE file. also "The LICENSE and NOTICE files in flink-dist/src/main/flink-bin refer to the binary distribution and mention all of Flink's Maven dependencies as well" can be dropped because we don't have them anymore.

I came to the same conclusion on dependencies. I used tools/list_deps.py and diff'ed the output for 1.5 and 1.6, that's probably what Chesnay also did ... :-)

On 8. Aug 2018, at 14:43, Chesnay Schepler <chesnay@xxxxxxxxxx> wrote:

+1

- verified source release contains no binaries
- verified correct versions in source release
- verified compilation, tests and E2E-tests pass (on travis)
- verified checksum and gpg files

New dependencies (excluding dependencies where we simply depend on a different version now):
    Apache licensed:
        io.confluent:common-utils:jar:3.3.1
        io.confluent:kafka-schema-registry-client:jar:3.3.1
        io.prometheus:simpleclient_pushgateway:jar:0.3.0
        various Apache Nifi dependencies
        various Apache Parquet dependencies
        various ElasticSearch dependencies
    CDDL:
        javax.ws.rs:javax.ws.rs-api:jar:2.1
    Bouncycastle (MIT-like):
        org.bouncycastle:bcpkix-jdk15on:jar:1.59
        org.bouncycastle:bcprov-jdk15on:jar:1.59
    MIT:
        org.projectlombok:lombok:jar:1.16.20

On 08.08.2018 13:28, Till Rohrmann wrote:
Thanks for reporting these problems Chesnay. The usage string in
`standalone-job.sh` is out dated and should be updated. The same applies to
the typo.

When calling `standalone-job.sh start --job-classname foobar.Job` please
make sure that the user code jar is contained in the classpath (e.g.
putting the jar in the lib directory). Documenting this behaviour is part
of the pending issue FLINK-10001.

We should fix all of these issues. They are, however, no release blockers.

Cheers,
Till

On Wed, Aug 8, 2018 at 11:31 AM Chesnay Schepler <chesnay@xxxxxxxxxx> wrote:

I found some issues with the standalone-job.sh script.

I ran "./bin/standalone-job.sh start" as described by the usage string.

     2018-08-08 09:22:34,385 ERROR
     org.apache.flink.runtime.entrypoint.ClusterEntrypoint         -
     Could not parse command line arguments [--configDir,
     /home/zento/svn/flink-1.6.0/flink-1.6.0/conf].
     org.apache.flink.runtime.entrypoint.FlinkParseException: Failed to
     parse the command line arguments.
              at

org.apache.flink.runtime.entrypoint.parser.CommandLineParser.parse(CommandLineParser.java:52)
              at

org.apache.flink.container.entrypoint.StandaloneJobClusterEntryPoint.main(StandaloneJobClusterEntryPoint.java:143)
     Caused by: org.apache.commons.cli.MissingOptionException: Missing
     required option: j
              at

org.apache.commons.cli.DefaultParser.checkRequiredOptions(DefaultParser.java:199)
              at
     org.apache.commons.cli.DefaultParser.parse(DefaultParser.java:130)
              at
     org.apache.commons.cli.DefaultParser.parse(DefaultParser.java:81)
              at

org.apache.flink.runtime.entrypoint.parser.CommandLineParser.parse(CommandLineParser.java:50)
             ... 1 more

The script should fail earlier if no jar is provided, with a better
error message.
It is also undocumented, and the usage instructions don't appear correct.

Passing a jar with the -j option leads to a ClassNotFoundException:
"./bin/standalone-job.sh start -j examples/streaming/WordCount.jar"

     2018-08-08 09:26:30,562 ERROR
     org.apache.flink.runtime.entrypoint.ClusterEntrypoint         -
     Cluster initialization failed.
     java.lang.reflect.UndeclaredThrowableException
              at

org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1854)
              at

org.apache.flink.runtime.security.HadoopSecurityContext.runSecured(HadoopSecurityContext.java:41)
              at

org.apache.flink.runtime.entrypoint.ClusterEntrypoint.startCluster(ClusterEntrypoint.java:189)
              at

org.apache.flink.container.entrypoint.StandaloneJobClusterEntryPoint.main(StandaloneJobClusterEntryPoint.java:158)
     Caused by: org.apache.flink.util.FlinkException: Could not load the
     provied entrypoint class.
              at

org.apache.flink.container.entrypoint.StandaloneJobClusterEntryPoint.createPackagedProgram(StandaloneJobClusterEntryPoint.java:92)
              at

org.apache.flink.container.entrypoint.StandaloneJobClusterEntryPoint.retrieveJobGraph(StandaloneJobClusterEntryPoint.java:75)
              at

org.apache.flink.runtime.entrypoint.JobClusterEntrypoint.createDispatcher(JobClusterEntrypoint.java:107)
              at

org.apache.flink.runtime.entrypoint.ClusterEntrypoint.startClusterComponents(ClusterEntrypoint.java:353)
              at

org.apache.flink.runtime.entrypoint.ClusterEntrypoint.runCluster(ClusterEntrypoint.java:232)
              at

org.apache.flink.runtime.entrypoint.ClusterEntrypoint.lambda$startCluster$0(ClusterEntrypoint.java:190)
              at java.security.AccessController.doPrivileged(Native Method)
              at javax.security.auth.Subject.doAs(Subject.java:422)
              at

org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1836)
              ... 3 more
     Caused by: java.lang.ClassNotFoundException:
     examples/streaming/WordCount.jar
              at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
              at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
              at
     sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:338)
              at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
              at

org.apache.flink.container.entrypoint.StandaloneJobClusterEntryPoint.createPackagedProgram(StandaloneJobClusterEntryPoint.java:89)
              ... 11 more

So this seems to not work at all, but maybe I'm using it wrong?

(There's also typo in "Could not load the provied entrypoint class")

On 08.08.2018 10:33, Piotr Nowojski wrote:
+1 from my side

I’ve spent some time playing around with various examples (batching,
streaming and SQL) on EMR 6 nodes cluster with yarn deployment, with
different configuration options (number of task
managers/memory/Flip6/credit base flow control/metrics) and everything
looks now fine (after fixing
https://issues.apache.org/jira/browse/FLINK-9969 <
https://issues.apache.org/jira/browse/FLINK-9969> ).
Piotrek

On 7 Aug 2018, at 17:17, Till Rohrmann <trohrmann@xxxxxxxxxx> wrote:

Hi everyone,
Please review and vote on the release candidate #4 for the version
1.6.0,
as follows:
[ ] +1, Approve the release
[ ] -1, Do not approve the release (please provide specific comments)


The complete staging area is available for your review, which includes:
* JIRA release notes [1],
* the official Apache source release and binary convenience releases to
be
deployed to dist.apache.org [2], which are signed with the key with
fingerprint 1F302569A96CFFD5 [3],
* all artifacts to be deployed to the Maven Central Repository [4],
* source code tag "release-1.6.0-rc4" [5],
* website pull request listing the new release and adding announcement
blog
post [6].

Please use this document for coordinating testing efforts: [7]

The vote will be shortened since we only adde a minor fix on top of the
RC
3. It will close on Wednesday 6:30pm CET. It is adopted by majority
approval, with at least 3 PMC affirmative votes.

Thanks,
Your friendly Release Manager

[1]

https://issues.apache.org/jira/secure/ReleaseNote.jspa?projectId=12315522&version=12342760
[2] https://dist.apache.org/repos/dist/dev/flink/flink-1.6.0/
[3] https://dist.apache.org/repos/dist/release/flink/KEYS
[4]
https://repository.apache.org/content/repositories/orgapacheflink-1178
[5] https://github.com/apache/flink/tree/release-1.6.0-rc4
[6] https://github.com/apache/flink-web/pull/117
[7]

https://docs.google.com/document/d/1upBFZQ7tbaSkYvDiLqfUFXKg8Xxs-lVheEfb66e4jpo/edit?usp=sharing
Pro-tip: you can create a settings.xml file with these contents:

<settings>
<activeProfiles>
   <activeProfile>flink-1.6.0</activeProfile>
</activeProfiles>
<profiles>
   <profile>
     <id>flink-1.6.0</id>
     <repositories>
       <repository>
         <id>flink-1.6.0</id>
         <url>

https://repository.apache.org/content/repositories/orgapacheflink-1178/
         </url>
       </repository>
       <repository>
         <id>archetype</id>
         <url>

https://repository.apache.org/content/repositories/orgapacheflink-1178/
         </url>
       </repository>
     </repositories>
   </profile>
</profiles>
</settings>

And reference that in you maven commands via --settings
path/to/settings.xml. This is useful for creating a quickstart based on
the
staged release and for building against the staged jars.