[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: Need help regarding memory leak issue

trying to understand the situation you are having.  

By saying 'kills the appllication', is that a leak in the application itself, or the workers being the root cause?  Also are you running ML models inside Python SDK DoFn's?  Then I suppose it is running some predictions rather than model training? 

On Thu, Nov 15, 2018 at 1:08 PM Rakesh Kumar <rakeshkumar@xxxxxxxx> wrote:
I am using Beam Python SDK to run my app in production. The app is running machine learning models. I am noticing some memory leak which eventually kills the application. I am not sure the source of memory leak. Currently, I am using object graph to dump the memory stats. I hope I will get some useful information out of this. I have also looked into Guppy library and they are almost the same.

Do you guys have any recommendation for debugging this issue? Do we have any tooling in the SDK that can help to debug it? 
Please feel free to share your experience if you have debugged similar issues in past.

Thank you,

Ruoyun  Huang