OSDir


[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: Go SDK: Teardown() not being called with dataflow runner


Hi Eduardo,
These differences are described by the link I sent (https://github.com/apache/beam/blob/master/sdks/java/core/src/main/java/org/apache/beam/sdk/transforms/DoFn.java#L465-L666) - it documents what kind of things it's best to do in each method. Please let me know if something is still unclear.

On Tue, Jul 3, 2018 at 1:35 PM eduardo.morales@xxxxxxxxx <eduardo.morales@xxxxxxxxx> wrote:
FinishBundle() does the job.

Should I keep using Setup()? What is the difference between Setup() and StartBundle()?

Thanks again.

On 2018/07/03 20:10:21, Henning Rohde <herohde@xxxxxxxxxx> wrote:
> Teardown has very loose guarantees on when it's called and you essentially
> can't rely on it. Currently, for Go on non-direct runners, we hang on to
> the bundle descriptors forever and never destroy them (and in turn never
> call Teardown). Even if we didn't, failures/restarts could cause Teardown
> to not be called.
>
> If something _must_ happen, FinishBundle is the right method.
>
> Thanks,
>  Henning
>
> On Tue, Jul 3, 2018 at 10:37 AM eduardo.morales@xxxxxxxxx <
> eduardo.morales@xxxxxxxxx> wrote:
>
> > Essentially I have the following code:
> >
> > type Writer struct {
> >   Pool WriterPool
> > }
> >
> > func (w *Writer) Setup() {
> >  w.Pool = Init()
> > }
> >
> > func (w* Writer) ProcessElement(ctx, elem Elem) {
> >   w.Pool.Add(elem)
> > }
> >
> > func (w* Writer) Teardown() {
> >   w.Pool.Write()
> >   w.Pool.Close()
> > }
> >
> > beam.ParDo0(scope, &Writer{}, elemCollection)
> >
> > The above code runs fine with the direct runner but not with dataflow.
> >
> >  I added log lines to the above methods, and the ones in Teardown() never
> > appear in the logs.
> > If I change my code as follows:
> >
> > func (w* Writer) ProcessElement(ctx, elem Elem) {
> >   w.Pool.Add(elem)
> >   w.Pool.Write()
> > }
> >
> > Then I see the data being written, but I lose the ability to pool, plus I
> > am leaking connections.
> >
> > Is this a known issue, or I am going something wrong?
> >
> > Thanks again for the help.
> >
>