osdir.com


[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: Duplicate key unique constraint error


Also what's your schedule interval? I'm just trying to confirm that this
isn't a "run every minute, or anytime someone blinks" kind of DAG.

Max

On Tue, Oct 30, 2018 at 5:48 AM Brian Greene <
brian@xxxxxxxxxxxxxxxxxxxxxxxxx> wrote:

> How do you trigger it externally?
>
> We have several custom operators that trigger other jobs and we had to be
> really careful or we’d get duplicate keys for the dag run and it would fail
> to kick off.
>
> One scheduler, but we saw it repeatedly and have it noted as a thing to
> watch out for.
>
> Brian
>
> Sent from a device with less than stellar autocorrect
>
> > On Oct 29, 2018, at 2:03 PM, Abhishek Sinha <abhishek@xxxxxxxxxxxx>
> wrote:
> >
> > Attaching the scheduler crash logs as well.
> >
> > https://pastebin.com/B2WEJKRB
> >
> >
> >
> >
> > Regards,
> >
> > Abhishek Sinha | m: +919035191078 | e: abhishek@xxxxxxxxxxxx
> >
> >
> > On Tue, Oct 30, 2018 at 12:19 AM Abhishek Sinha <abhishek@xxxxxxxxxxxx>
> > wrote:
> >
> >> Max,
> >>
> >> We always trigger the DAG externally. I am not sure if there is still
> any
> >> backfill involved.
> >>
> >> Is there a way where I can find out in logs, if more than one instance
> of
> >> scheduler is running?
> >>
> >>
> >> On 29 October 2018 at 10:43:19 PM, Maxime Beauchemin (
> >> maximebeauchemin@xxxxxxxxx) wrote:
> >>
> >> The stacktrace seems to be pointing in that direction. Id check that
> >> first. It seems like it **could** be a race condition with a backfill as
> >> well, unclear.
> >>
> >> It's still a bug though, and the scheduler should make sure to handle
> this
> >> and not raise/crash.
> >>
> >> On Mon, Oct 29, 2018, 10:05 AM Abhishek Sinha <abhishek@xxxxxxxxxxxx>
> >> wrote:
> >>
> >>> Max,
> >>>
> >>> I do not think there was more than one instance of scheduler running.
> >>> Since the scheduler crashed and it has been restarted, I cannot
> confirm it
> >>> now. Is there any log that can provide this information?
> >>>
> >>> Could there be a different cause apart from multiple scheduler
> instances
> >>> running?
> >>>
> >>>
> >>> On 29 October 2018 at 9:30:56 PM, Maxime Beauchemin (
> >>> maximebeauchemin@xxxxxxxxx) wrote:
> >>>
> >>> Abhishek, are you running more than one scheduler instance at once?
> >>>
> >>> Max
> >>>
> >>> On Mon, Oct 29, 2018 at 8:17 AM Abhishek Sinha <abhishek@xxxxxxxxxxxx>
> >>> wrote:
> >>>
> >>>> The issue is happening more frequently now. Can someone please look
> into
> >>>> this?
> >>>>
> >>>>
> >>>>
> >>>>
> >>>> On 24 September 2018 at 12:42:49 PM, Abhishek Sinha (
> >>> abhishek@xxxxxxxxxxxx
> >>>> )
> >>>> wrote:
> >>>>
> >>>> Can someone please help in looking into this issue? It is critical
> since
> >>>> this has come up in one of our production environment. Also, this
> issue
> >>> has
> >>>> appeared only once till now.
> >>>>
> >>>>
> >>>>
> >>>>
> >>>> Regards,
> >>>>
> >>>> Abhishek
> >>>>
> >>>> On 20-Sep-2018, at 10:18 PM, Abhishek Sinha <abhishek@xxxxxxxxxxxx>
> >>> wrote:
> >>>>
> >>>> Any update on this?
> >>>>
> >>>>
> >>>>
> >>>>
> >>>> Regards,
> >>>>
> >>>> Abhishek
> >>>>
> >>>> On 18-Sep-2018, at 12:48 AM, Abhishek Sinha <abhishek@xxxxxxxxxxxx>
> >>> wrote:
> >>>>
> >>>> Pastebin: https://pastebin.com/K6BMTb5K
> >>>>
> >>>>
> >>>>
> >>>>
> >>>> Regards,
> >>>>
> >>>> Abhishek
> >>>>
> >>>> On 18-Sep-2018, at 12:31 AM, Stefan Seelmann <mail@xxxxxxxxxxxxxxxxxx
> >
> >>>> wrote:
> >>>>
> >>>> On 9/17/18 8:19 PM, Abhishek Sinha wrote:
> >>>>
> >>>> Any update on this?
> >>>>
> >>>> Please find the scheduler error log attached.
> >>>>
> >>>> Can you share the full python stack trace?
> >>>>
> >>>>
> >>>> Seems the mailing list doesn't allow attachments. Either post the
> >>>> stacktrace inline, or post it somewhere at pastebin or so.
> >>>>
> >>>
> >>>
>