osdir.com


[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

RE: 'mvn verify' fails at flink-hadoop-fs


I added lines below to flink-hadoop-fs/pom.xml, and that allowed me to turn off the tests that were failing for me. 
Do we want to add this change to master?
If so, do I need to document this new switch somewhere?

(
the build then hang for me at flink-runtime, but that's a different issue
Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 25.969 sec - in org.apache.flink.runtime.taskmanager.TaskManagerRegistrationTest
)

	<build>
		<plugins>
			<plugin>
				<groupId>org.apache.maven.plugins</groupId>
				<artifactId>maven-surefire-plugin</artifactId>
				<configuration>
					<skipTests>${skipHdfsTests}</skipTests>
				</configuration>
			</plugin>
		</plugins>
	</build>

-----Original Message-----
From: Chesnay Schepler [mailto:chesnay@xxxxxxxxxx] 
Sent: Tuesday, July 10, 2018 10:36 AM
To: dev@xxxxxxxxxxxxxxxx; NEKRASSOV, ALEXEI <an4828@xxxxxxx>
Subject: Re: 'mvn verify' fails at flink-hadoop-fs

There's currently no workaround except going in and manually disabling them.

On 10.07.2018 16:32, Chesnay Schepler wrote:
> Generally, any test that uses HDFS will fail on Windows. We've 
> disabled most of them, but some slip through from time to time.
>
> Note that we do not provide any guarantees for all tests passing on 
> Windows.
>
> On 10.07.2018 16:28, NEKRASSOV, ALEXEI wrote:
>> I'm running 'mvn clean verify' on Windows with no Hadoop libraries 
>> installed, and the build fails (see below).
>> What's the solution? Is there a switch to skip Hadoop-related tests?
>> Or I need to install Hadoop libraries?
>>
>> Thanks,
>> Alex
>>
>>
>> Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 1.726 
>> sec <<< FAILURE! - in 
>> org.apache.flink.runtime.fs.hdfs.HdfsBehaviorTest
>> org.apache.flink.runtime.fs.hdfs.HdfsBehaviorTest  Time elapsed: 
>> 1.726 sec  <<< ERROR!
>> java.lang.UnsatisfiedLinkError: 
>> org.apache.hadoop.io.nativeio.NativeIO$Windows.access0(Ljava/lang/String;I)Z
>>          at
>> org.apache.hadoop.io.nativeio.NativeIO$Windows.access0(Native Method)
>>          at
>> org.apache.hadoop.io.nativeio.NativeIO$Windows.access(NativeIO.java:570)
>>          at org.apache.hadoop.fs.FileUtil.canWrite(FileUtil.java:996)
>>          at
>> org.apache.hadoop.hdfs.server.common.Storage$StorageDirectory.analyzeStorage(Storage.java:484)
>>          at
>> org.apache.hadoop.hdfs.server.namenode.FSImage.recoverStorageDirs(FSImage.java:293)
>>          at
>> org.apache.hadoop.hdfs.server.namenode.FSImage.recoverTransitionRead(FSImage.java:202)
>>          at
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFSImage(FSNamesystem.java:891)
>>          at
>> org.apache.hadoop.hdfs.server.namenode.FSNamesystem.loadFromDisk(FSNamesystem.java:638)
>>          at
>> org.apache.hadoop.hdfs.server.namenode.NameNode.loadNamesystem(NameNode.java:503)
>>          at
>> org.apache.hadoop.hdfs.server.namenode.NameNode.initialize(NameNode.java:559)
>>          at
>> org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:724)
>>          at
>> org.apache.hadoop.hdfs.server.namenode.NameNode.<init>(NameNode.java:708)
>>          at
>> org.apache.hadoop.hdfs.server.namenode.NameNode.createNameNode(NameNode.java:1358)
>>          at
>> org.apache.hadoop.hdfs.MiniDFSCluster.createNameNode(MiniDFSCluster.java:996)
>>          at
>> org.apache.hadoop.hdfs.MiniDFSCluster.createNameNodesAndSetConf(MiniDFSCluster.java:867)
>>          at
>> org.apache.hadoop.hdfs.MiniDFSCluster.initMiniDFSCluster(MiniDFSCluster.java:702)
>>          at
>> org.apache.hadoop.hdfs.MiniDFSCluster.<init>(MiniDFSCluster.java:374)
>>          at
>> org.apache.hadoop.hdfs.MiniDFSCluster$Builder.build(MiniDFSCluster.java:355)
>>          at
>> org.apache.flink.runtime.fs.hdfs.HdfsBehaviorTest.createHDFS(HdfsBeha
>> viorTest.java:65)
>>
>> Running org.apache.flink.runtime.fs.hdfs.HdfsKindTest
>> Tests run: 6, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.081 
>> sec - in org.apache.flink.runtime.fs.hdfs.HdfsKindTest
>> Running
>> org.apache.flink.runtime.fs.hdfs.LimitedConnectionsConfigurationTest
>> Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.017 
>> sec - in 
>> org.apache.flink.runtime.fs.hdfs.LimitedConnectionsConfigurationTest
>>
>> Results :
>>
>> Tests in error:
>>    HdfsBehaviorTest.createHDFS:65 ▒ UnsatisfiedLink 
>> org.apache.hadoop.io.nativeio...
>>
>> Tests run: 24, Failures: 0, Errors: 1, Skipped: 1
>>
>> [INFO]
>> ---------------------------------------------------------------------
>> ---
>> [INFO] Reactor Summary:
>> [INFO]
>> [INFO] force-shading ...................................... SUCCESS [  
>> 2.335 s] [INFO] flink .............................................. 
>> SUCCESS [
>> 29.794 s]
>> [INFO] flink-annotations .................................. SUCCESS [  
>> 2.198 s] [INFO] flink-shaded-hadoop ................................ 
>> SUCCESS [  0.226 s] [INFO] flink-shaded-hadoop2 
>> ............................... SUCCESS [
>> 11.015 s]
>> [INFO] flink-shaded-hadoop2-uber .......................... SUCCESS [
>> 16.343 s]
>> [INFO] flink-shaded-yarn-tests ............................ SUCCESS [
>> 13.653 s]
>> [INFO] flink-shaded-curator ............................... SUCCESS [  
>> 1.386 s] [INFO] flink-test-utils-parent ............................ 
>> SUCCESS [  0.191 s] [INFO] flink-test-utils-junit 
>> ............................. SUCCESS [  3.318 s] [INFO] 
>> flink-metrics ...................................... SUCCESS [  0.212 
>> s] [INFO] flink-metrics-core ................................. 
>> SUCCESS [  3.502 s] [INFO] flink-core 
>> ......................................... SUCCESS
>> [01:30 min]
>> [INFO] flink-java ......................................... SUCCESS
>> [01:31 min]
>> [INFO] flink-queryable-state .............................. SUCCESS [  
>> 0.186 s] [INFO] flink-queryable-state-client-java .................. 
>> SUCCESS [  4.099 s] [INFO] flink-filesystems 
>> .................................. SUCCESS [  0.198 s] [INFO] 
>> flink-hadoop-fs .................................... FAILURE [  8.672 
>> s]
>>
>>
>>
>>
>
>