osdir.com

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: Go SDK: How are re-starts handled?


I am sorry. I am not expressing myself correctly. Let me do it though code:

if err := beamx.Run(ctx, pipe); err != nil {
  // How do I know 'err' is the result of a pipeline already running, as opposed to some
  // other problem that may need  special attention.
}

On 2018/06/22 23:10:13, Lukasz Cwik <lcwik@xxxxxxxxxx> wrote: 
> The job name is a user chosen value[1]. If you don't specify something, a
> job name is generated for you automatically[2].
> 
> 1:
> https://github.com/apache/beam/blob/c1927cd339c57125e29a651e614fb5105abf6d33/sdks/go/pkg/beam/options/jobopts/options.go#L38
> 2:
> https://github.com/apache/beam/blob/c1927cd339c57125e29a651e614fb5105abf6d33/sdks/go/pkg/beam/options/jobopts/options.go#L71
> 
> On Fri, Jun 22, 2018 at 3:28 PM eduardo.morales@xxxxxxxxx <
> eduardo.morales@xxxxxxxxx> wrote:
> 
> >
> >
> > On 2018/06/22 21:35:29, Lukasz Cwik <lcwik@xxxxxxxxxx> wrote:
> > > There can only be one pipeline in Dataflow with the same job name so if
> > you
> > > attempt to submit another job with the same job name you'll get back an
> > > identifier for the currently executing pipeline.
> >
> > But beam.Run() only returns an error. How do I get the job name back?
> > My guess is that I have to use a different API (
> > https://godoc.org/google.golang.org/api/dataflow/v1b3). Is that the
> > correct way to detect job name collisions?
> >
> > Thanks again.
> >
> > > On Fri, Jun 22, 2018 at 2:27 PM eduardo.morales@xxxxxxxxx <
> > > eduardo.morales@xxxxxxxxx> wrote:
> > >
> > > > If I have a k8s process launching dataflow pipelines, what happens when
> > > > the process is restarted? Can Apache Beam detect a running pipeline and
> > > > join accordingly? or will the pipeline be duplicated?
> > > >
> > > > Thanks in advance.
> > > >
> > >
> >
>