osdir.com

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Job jumbo jar can't be extracted on Windows


When I try to "flink run" a job jar that includes dependent jar's - it fails on Windows with the following Exception:

org.apache.flink.client.program.ProgramInvocationException: Unknown I/O error while extracting contained jar files.
        at org.apache.flink.client.program.PackagedProgram.extractContainedLibraries(PackagedProgram.java:752)
        at org.apache.flink.client.program.PackagedProgram.<init>(PackagedProgram.java:194)
        at org.apache.flink.client.cli.CliFrontend.buildProgram(CliFrontend.java:833)
        at org.apache.flink.client.cli.CliFrontend.run(CliFrontend.java:201)
        at org.apache.flink.client.cli.CliFrontend.parseParameters(CliFrontend.java:1020)
        at org.apache.flink.client.cli.CliFrontend.lambda$main$9(CliFrontend.java:1096)
        at org.apache.flink.runtime.security.NoOpSecurityContext.runSecured(NoOpSecurityContext.java:30)
        at org.apache.flink.client.cli.CliFrontend.main(CliFrontend.java:1096)
Caused by: org.apache.flink.client.program.ProgramInvocationException: An I/O error occurred while creating temporary file to extract nested library 'lib/flink-connector-kafka-0.10_2.11-1.4.1.jar'.
        at org.apache.flink.client.program.PackagedProgram.extractContainedLibraries(PackagedProgram.java:708)
        ... 7 more
Caused by: java.io.IOException: Unable to create temporary file, C:\Users\XXX\AppData\Local\Temp\1863208689_4625824260650653022lib\flink-connector-kafka-0.10_2.11-1.4.1.jar
        at java.io.File$TempDirectory.generateFile(Unknown Source)
        at java.io.File.createTempFile(Unknown Source)
        at java.io.File.createTempFile(Unknown Source)
        at org.apache.flink.client.program.PackagedProgram.extractContainedLibraries(PackagedProgram.java:702)
        ... 7 more

I think we have a problem in PackagedProgram.java - it fails to extract contained jar's on Windows, because on Windows File.separatorChar is '\', but JarEntry.getName() returns '/' as file separator.
Line 699 of https://github.com/apache/flink/blob/master/flink-clients/src/main/java/org/apache/flink/client/program/PackagedProgram.java is no-op on Windows.

I think we need to replace line 699 with
            name = name.replace('/', '_');
as I don't see a way to have JarEntry use platform-specific file separator, and ZIP spec dictates the use of forward-slashes anyway.
Line 680 already uses hard-coded '/'.


Alex Nekrassov