osdir.com


[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

RE: 'mvn verify' fails at flink-hadoop-fs


I now see that the test that fails for me already has this check. But I don't think it is done correctly.

https://github.com/apache/flink/blob/master/flink-filesystems/flink-hadoop-fs/src/test/java/org/apache/flink/runtime/fs/hdfs/HdfsBehaviorTest.java has verifyOS() but a failed assumption in that method doesn't stop us from executing createHDFS() where there is no check, and thus the test fails.

I think we need to remove verifyOS(), and move the check directly to createHDFS() instead.


Similar problem may exist in:
flink-connectors\flink-connector-filesystem\src\test\java\org\apache\flink\streaming\connectors\fs\bucketing\BucketingSinkMigrationTest.java
flink-fs-tests\src\test\java\org\apache\flink\hdfstests\ContinuousFileProcessingMigrationTest.java
flink-fs-tests\src\test\java\org\apache\flink\hdfstests\HDFSTest.java

-----Original Message-----
From: Chesnay Schepler [mailto:chesnay@xxxxxxxxxx] 
Sent: Tuesday, July 10, 2018 3:10 PM
To: dev@xxxxxxxxxxxxxxxx; NEKRASSOV, ALEXEI <an4828@xxxxxxx>
Subject: Re: 'mvn verify' fails at flink-hadoop-fs

That flat-out disables all tests in the module, even those that could run on Windows.

We commonly add an OS check to respective tests that skip the tests, with an "Assume.assumeTrue(os!=windows)" statement in a "@BeforeClass" 
method.

On 10.07.2018 21:00, NEKRASSOV, ALEXEI wrote:
> I added lines below to flink-hadoop-fs/pom.xml, and that allowed me to turn off the tests that were failing for me.
> Do we want to add this change to master?
> If so, do I need to document this new switch somewhere?
>
> (
> the build then hang for me at flink-runtime, but that's a different 
> issue Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 
> 25.969 sec - in 
> org.apache.flink.runtime.taskmanager.TaskManagerRegistrationTest
> )
>
> 	<build>
> 		<plugins>
> 			<plugin>
> 				<groupId>org.apache.maven.plugins</groupId>
> 				<artifactId>maven-surefire-plugin</artifactId>
> 				<configuration>
> 					<skipTests>${skipHdfsTests}</skipTests>
> 				</configuration>
> 			</plugin>
> 		</plugins>
> 	</build>
>
> -----Original Message-----
> From: Chesnay Schepler [mailto:chesnay@xxxxxxxxxx]
> Sent: Tuesday, July 10, 2018 10:36 AM
> To: dev@xxxxxxxxxxxxxxxx; NEKRASSOV, ALEXEI <an4828@xxxxxxx>
> Subject: Re: 'mvn verify' fails at flink-hadoop-fs
>
> There's currently no workaround except going in and manually disabling them.
>
> On 10.07.2018 16:32, Chesnay Schepler wrote:
>> Generally, any test that uses HDFS will fail on Windows. We've 
>> disabled most of them, but some slip through from time to time.
>>
>> Note that we do not provide any guarantees for all tests passing on 
>> Windows.
>>
>> On 10.07.2018 16:28, NEKRASSOV, ALEXEI wrote:
>>> I'm running 'mvn clean verify' on Windows with no Hadoop libraries 
>>> installed, and the build fails (see below).
>>> What's the solution? Is there a switch to skip Hadoop-related tests?
>>> Or I need to install Hadoop libraries?
>>>
>>> Thanks,
>>> Alex
>>>
>>>
>>> Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 
>>> 1.726 sec <<< FAILURE! - in 
>>> org.apache.flink.runtime.fs.hdfs.HdfsBehaviorTest
>>> org.apache.flink.runtime.fs.hdfs.HdfsBehaviorTest  Time elapsed:
>>> 1.726 sec  <<< ERROR!
>>> java.lang.UnsatisfiedLinkError:
>>> org.apache.hadoop.io.nativeio.NativeIO$Windows.access0(Ljava/lang/String;I)Z
>>>           at
>>> org.apache.hadoop.io.nativeio.NativeIO$Windows.access0(Native Method)
>>>           at
>>> org.apache.hadoop.io.nativeio.NativeIO$Windows.access(NativeIO.java:570)
>>>           at org.apache.hadoop.fs.FileUtil.canWrite(FileUtil.java:996)
>>>           at
>>> org.apache.hadoop.hdfs.server.common.Storage$StorageDirectory.analyzeStorage(Storage.java:484)
>>>           at
>>> org.apache.hadoop.hdfs.server.namenode.FSImage.recoverStorageDirs(FSImage.java:293)
>>>           at
>>> org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:202)
>>>           at
>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:891)
>>>           at
>>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:638)
>>>           at
>>> org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:503)
>>>           at
>>> org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:559)
>>>           at
>>> org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:724)
>>>           at
>>> org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:708)
>>>           at
>>> org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1358)
>>>           at
>>> org.apache.hadoop.hdfs.MiniDFSCluster.createNameNode(MiniDFSCluster.java:996)
>>>           at
>>> org.apache.hadoop.hdfs.MiniDFSCluster.createNameNodesAndSetConf(MiniDFSCluster.java:867)
>>>           at
>>> org.apache.hadoop.hdfs.MiniDFSCluster.initMiniDFSCluster(MiniDFSCluster.java:702)
>>>           at
>>> org.apache.hadoop.hdfs.MiniDFSCluster.<init>(MiniDFSCluster.java:374)
>>>           at
>>> org.apache.hadoop.hdfs.MiniDFSCluster$Builder.build(MiniDFSCluster.java:355)
>>>           at
>>> org.apache.flink.runtime.fs.hdfs.HdfsBehaviorTest.createHDFS(HdfsBeh
>>> a
>>> viorTest.java:65)
>>>
>>> Running org.apache.flink.runtime.fs.hdfs.HdfsKindTest
>>> Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 
>>> 0.081 sec - in org.apache.flink.runtime.fs.hdfs.HdfsKindTest
>>> Running
>>> org.apache.flink.runtime.fs.hdfs.LimitedConnectionsConfigurationTest
>>> Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 
>>> 0.017 sec - in 
>>> org.apache.flink.runtime.fs.hdfs.LimitedConnectionsConfigurationTest
>>>
>>> Results :
>>>
>>> Tests in error:
>>>     HdfsBehaviorTest.createHDFS:65 ▒ UnsatisfiedLink 
>>> org.apache.hadoop.io.nativeio...
>>>
>>> Tests run: 24, Failures: 0, Errors: 1, Skipped: 1
>>>
>>> [INFO]
>>> --------------------------------------------------------------------
>>> -
>>> ---
>>> [INFO] Reactor Summary:
>>> [INFO]
>>> [INFO] force-shading ...................................... SUCCESS 
>>> [
>>> 2.335 s] [INFO] flink ..............................................
>>> SUCCESS [
>>> 29.794 s]
>>> [INFO] flink-annotations .................................. SUCCESS 
>>> [
>>> 2.198 s] [INFO] flink-shaded-hadoop ................................
>>> SUCCESS [  0.226 s] [INFO] flink-shaded-hadoop2 
>>> ............................... SUCCESS [
>>> 11.015 s]
>>> [INFO] flink-shaded-hadoop2-uber .......................... SUCCESS 
>>> [
>>> 16.343 s]
>>> [INFO] flink-shaded-yarn-tests ............................ SUCCESS 
>>> [
>>> 13.653 s]
>>> [INFO] flink-shaded-curator ............................... SUCCESS 
>>> [
>>> 1.386 s] [INFO] flink-test-utils-parent ............................
>>> SUCCESS [  0.191 s] [INFO] flink-test-utils-junit 
>>> ............................. SUCCESS [  3.318 s] [INFO] 
>>> flink-metrics ...................................... SUCCESS [  
>>> 0.212 s] [INFO] flink-metrics-core .................................
>>> SUCCESS [  3.502 s] [INFO] flink-core 
>>> ......................................... SUCCESS
>>> [01:30 min]
>>> [INFO] flink-java ......................................... SUCCESS
>>> [01:31 min]
>>> [INFO] flink-queryable-state .............................. SUCCESS 
>>> [
>>> 0.186 s] [INFO] flink-queryable-state-client-java ..................
>>> SUCCESS [  4.099 s] [INFO] flink-filesystems 
>>> .................................. SUCCESS [  0.198 s] [INFO] 
>>> flink-hadoop-fs .................................... FAILURE [  
>>> 8.672 s]
>>>
>>>
>>>
>>>
>>
>