osdir.com

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [HELP] Bisect


I think I find it at
https://github.com/apache/flink/commit/4e616a8362744c15a71e0e57ad68fbb52266c837

Thanks,
tison

陈梓立 <wander4096@xxxxxxxxx> 于2018年8月2日周四 下午3:14写道:

> Hi devs,
>
> At tag release-1.5.1, I find LOG contains noise like below.
>
> ```
> 13:58:40,752 INFO
> org.apache.flink.runtime.dispatcher.StandaloneDispatcher      - Job
> a18a72a9c335b5cc964f43075418dedc reached globally terminal state FINISHED.
> 13:58:40,753 INFO  org.apache.flink.runtime.minicluster.MiniCluster
>       - Shutting down Flink Mini Cluster
> 13:58:40,753 INFO
> org.apache.flink.runtime.dispatcher.DispatcherRestEndpoint    - Shutting
> down rest endpoint.
> 13:58:40,754 INFO  org.apache.flink.runtime.jobmaster.JobMaster
>       - Stopping the JobMaster for job Flink Java Job at Thu Aug 02
> 13:58:29 CST 2018(a18a72a9c335b5cc964f43075418dedc).
> 13:58:40,754 INFO  org.apache.flink.runtime.taskexecutor.TaskExecutor
>       - Stopping TaskExecutor akka://flink/user/taskmanager_0.
> 13:58:40,755 INFO
> org.apache.flink.runtime.state.TaskExecutorLocalStateStoresManager  -
> Shutting down TaskExecutorLocalStateStoresManager.
> 13:58:40,755 INFO  org.apache.flink.runtime.jobmaster.JobMaster
>       - Close ResourceManager connection 58178cf2968f728837df335f4a8a63a5:
> JobManager is shutting down..
> 13:58:40,755 INFO
> org.apache.flink.runtime.dispatcher.StandaloneDispatcher      - Stopping
> dispatcher akka://flink/user/dispatcher65cb755a-d441-48b5-8942-52465b77bae2.
> 13:58:40,755 INFO
> org.apache.flink.runtime.dispatcher.StandaloneDispatcher      - Stopping
> all currently running jobs of dispatcher
> akka://flink/user/dispatcher65cb755a-d441-48b5-8942-52465b77bae2.
> 13:58:40,756 INFO  org.apache.flink.runtime.jobmaster.slotpool.SlotPool
>       - Suspending SlotPool.
> 13:58:40,756 INFO  org.apache.flink.runtime.jobmaster.slotpool.SlotPool
>       - Stopping SlotPool.
> 13:58:40,758 INFO
> org.apache.flink.runtime.resourcemanager.StandaloneResourceManager  -
> Disconnect job manager 93c37c8311c692f1a5b0c7caaa074a3a@akka://flink/user/jobmanager_1
> for job a18a72a9c335b5cc964f43075418dedc from the resource manager.
> 13:58:40,758 INFO
> org.apache.flink.runtime.resourcemanager.slotmanager.SlotManager  - Closing
> the SlotManager.
> 13:58:40,758 INFO
> org.apache.flink.runtime.resourcemanager.slotmanager.SlotManager  -
> Suspending the SlotManager.
> 13:58:40,759 INFO
> org.apache.flink.runtime.resourcemanager.slotmanager.SlotManager  -
> Unregister TaskManager 8c28415e2960d2f7a49a12cf329e2444 from the
> SlotManager.
> 13:58:40,759 INFO  org.apache.flink.runtime.taskexecutor.TaskExecutor
>       - JobManager for job a18a72a9c335b5cc964f43075418dedc with leader id
> 93c37c8311c692f1a5b0c7caaa074a3a lost leadership.
> 13:58:40,759 INFO
> org.apache.flink.runtime.resourcemanager.StandaloneResourceManager  - Slot
> request with allocation id a3f91d383ba2e647c07894a0974af2fb for job
> a18a72a9c335b5cc964f43075418dedc failed.
> org.apache.flink.util.FlinkException: The assigned slot
> 2cd8d254-283d-472c-8ae2-6499e4ba04bf_7 was removed.
> at
> org.apache.flink.runtime.resourcemanager.slotmanager.SlotManager.removeSlot(SlotManager.java:895)
> ...
> 13:58:40,761 INFO
> org.apache.flink.runtime.rest.handler.legacy.backpressure.StackTraceSampleCoordinator
> - Shutting down stack trace sample coordinator.
> 13:58:40,763 INFO
> org.apache.flink.runtime.resourcemanager.StandaloneResourceManager  - Slot
> request with allocation id 25f5cd03e1bd5654c54c20143fd05a7e for job
> a18a72a9c335b5cc964f43075418dedc failed.
> org.apache.flink.util.FlinkException: The assigned slot
> 2cd8d254-283d-472c-8ae2-6499e4ba04bf_6 was removed.
> at
> org.apache.flink.runtime.resourcemanager.slotmanager.SlotManager.removeSlot(SlotManager.java:895)
> ...
> 13:58:40,766 INFO
> org.apache.flink.runtime.dispatcher.StandaloneDispatcher      - Stopped
> dispatcher akka://flink/user/dispatcher65cb755a-d441-48b5-8942-52465b77bae2.
> 13:58:40,766 INFO
> org.apache.flink.runtime.resourcemanager.StandaloneResourceManager  - Slot
> request with allocation id 32047761d6066628b196ac13b7b2993e for job
> a18a72a9c335b5cc964f43075418dedc failed.
> org.apache.flink.util.FlinkException: The assigned slot
> 2cd8d254-283d-472c-8ae2-6499e4ba04bf_5 was removed.
> at
> org.apache.flink.runtime.resourcemanager.slotmanager.SlotManager.removeSlot(SlotManager.java:895)
> ...
>
> (repeat likely)
> 13:58:40,768 INFO
> org.apache.flink.runtime.resourcemanager.StandaloneResourceManager  - Slot
> request with allocation id befc46831f1f0ec5ef54cd660d12aa88 for job
> a18a72a9c335b5cc964f43075418dedc failed.
> org.apache.flink.util.FlinkException: The assigned slot
> 2cd8d254-283d-472c-8ae2-6499e4ba04bf_1 was removed.
> at
> org.apache.flink.runtime.resourcemanager.slotmanager.SlotManager.removeSlot(SlotManager.java:895)
> ...
> 13:58:40,784 INFO
> org.apache.flink.runtime.dispatcher.DispatcherRestEndpoint    - Removing
> cache directory
> /var/folders/ff/92vml6qd147dncsvvw28qlrc0000gn/T/flink-web-ui
> 13:58:40,785 INFO  org.apache.flink.runtime.io.disk.iomanager.IOManager
>       - I/O manager removed spill file directory
> /var/folders/ff/92vml6qd147dncsvvw28qlrc0000gn/T/flink-io-22be945e-daeb-49e9-ab3b-22521e9d2280
> 13:58:40,785 INFO  org.apache.flink.runtime.io.network.NetworkEnvironment
>       - Shutting down the network environment and its components.
> 13:58:40,796 INFO  org.apache.flink.runtime.taskexecutor.JobLeaderService
>       - Stop job leader service.
> 13:58:40,796 INFO  org.apache.flink.runtime.taskexecutor.TaskExecutor
>       - Stopped TaskExecutor akka://flink/user/taskmanager_0.
> 13:58:40,804 INFO  org.apache.flink.runtime.rpc.akka.AkkaRpcService
>       - Stopping Akka RPC service.
> 13:58:40,834 INFO  org.apache.flink.runtime.blob.PermanentBlobCache
>       - Shutting down BLOB cache
> 13:58:40,834 INFO  org.apache.flink.runtime.blob.TransientBlobCache
>       - Shutting down BLOB cache
> 13:58:40,836 INFO  org.apache.flink.runtime.blob.BlobServer
>       - Stopped BLOB server at 0.0.0.0:50685
> 13:58:40,836 INFO  org.apache.flink.runtime.rpc.akka.AkkaRpcService
>       - Stopped Akka RPC service.
> ```
>
> Note that the job is FINISHED and RM, TM, JM and so on shutdown correctly.
>
> To reproduce it, run
> `flink-examples/flink-examples-batch/src/main/java/org/apache/flink/examples/java/ml/LinearRegression.java`
>
> The code snip relevant is here
> <https://github.com/apache/flink/blob/628b71dfabe72a99eb6d54994fc870abed1f0268/flink-runtime/src/main/java/org/apache/flink/runtime/resourcemanager/slotmanager/SlotManager.java#L783>
> .
>
> But it could not be reproduced in master, which infers there is a commit
> suppress it. So I come to here ask for help to accurately bisect it.
>
> Thanks,
> tison
>