Did you know that tracking YARN task status in DolphinScheduler can be a challenge? In the past, the initial approach was based on applicationId, but this method has its limitations. For instance, when using Flink Stream Application mode, the client Shell exits immediately, causing DolphinScheduler to mark the task as successful, even if the YARN task is still running.
To address this issue, developers refactored the process, implementing two core interfaces: submitApplication and trackApplicationStatus. This allows for more accurate tracking of YARN tasks in DolphinScheduler.
In addition, there are three states to consider when tracking YARN task status: FINISHED, FAILED, and KILLED. However, within the FINISHED state, there is also a FinalStatus, which requires further evaluation to determine the actual outcome.
Source: https://dev.to/chen_debra_3060b21d12b1b0/how-to-track-the-yarn-task-status-in-dolphinscheduler-36jc
isphere_devs / Programmers - InfoSphere
npub16k…8ds4r
2024-10-29 07:03:43
Author Public Key
npub16klxfzuzsxckxdxtfjfwnpympay6yujycq602h5kgnuzxr6wdfgsp8ds4rPublished at
2024-10-29 07:03:43Event JSON
{
"id": "cfb97060b49e007d10b15608f0649e73fa34832747f53ce731a488525228cd47",
"pubkey": "d5be648b8281b16334cb4c92e9849b0f49a27244c034f55e9644f8230f4e6a51",
"created_at": 1730185423,
"kind": 1,
"tags": [],
"content": "Did you know that tracking YARN task status in DolphinScheduler can be a challenge? In the past, the initial approach was based on applicationId, but this method has its limitations. For instance, when using Flink Stream Application mode, the client Shell exits immediately, causing DolphinScheduler to mark the task as successful, even if the YARN task is still running.\n\nTo address this issue, developers refactored the process, implementing two core interfaces: submitApplication and trackApplicationStatus. This allows for more accurate tracking of YARN tasks in DolphinScheduler.\n\nIn addition, there are three states to consider when tracking YARN task status: FINISHED, FAILED, and KILLED. However, within the FINISHED state, there is also a FinalStatus, which requires further evaluation to determine the actual outcome.\n\nSource: https://dev.to/chen_debra_3060b21d12b1b0/how-to-track-the-yarn-task-status-in-dolphinscheduler-36jc",
"sig": "0124b021498f5798136b6c5562baa8200424d32dc300fb029d1ce60dc1c364d0994a0ef1fd4cc47e8ec9a5885dc097306e768cb514f48a7cf5b549378c38720b"
}