osdir.com

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: not able to launch dataflow job - permission issues with service account


As the message says, enable the api from cloud console of gcloud cli utility. (Refer dataflow docs)

If that’s enabled, ensure the robot account has the permissions it needs - dataflow api, gcs and anything else you are trying to access. 

On Thu, Nov 22, 2018 at 04:33 Unais T <tpunais@xxxxxxxxx> wrote:

I am trying to run a simple data flow job in google cloud - its running perfectly in local - but when I tried to launch it I'm getting the following error - I tried debug a lot

can someone help on this

INFO:root:Created job with id: [2018-11-22_02_57_07-12079060901530487381]
INFO:root:To access the Dataflow monitoring console, please navigate to https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2018-11-22_02_57_07-12079060901530487381?project=kkfas-main-account-221808
INFO:root:Job 2018-11-22_02_57_07-12079060901530487381 is in state JOB_STATE_PENDING
INFO:root:2018-11-22T10:57:08.052Z: JOB_MESSAGE_DETAILED: Autoscaling is enabled for job 2018-11-22_02_57_07-12079060901530487381. The number of workers will be between 1 and 1000.
INFO:root:2018-11-22T10:57:08.072Z: JOB_MESSAGE_DETAILED: Autoscaling was automatically enabled for job 2018-11-22_02_57_07-12079060901530487381.
INFO:root:2018-11-22T10:57:40.405Z: JOB_MESSAGE_ERROR: Workflow failed. Causes: There was a problem refreshing your credentials. Please check:
1. Dataflow API is enabled for your project.
2. There is a robot service account for your project:
service-[project number]@dataflow-service-producer-prod.iam.gserviceaccount.com should have access to your project. If this account does not appear in the permissions tab for your project, contact Dataflow support.
INFO:root:Job 2018-11-22_02_57_07-12079060901530487381 is in state JOB_STATE_FAILED
Traceback (most recent call last):
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/runpy.py", line 162, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/runpy.py", line 72, in _run_code
    exec code in run_globals
  File "/Users/u/Projects//digital/test.py", line 49, in <module>
    run()
  File "/Users/u/Projects//dataflow//digital/test.py", line 44, in run
    return p.run().wait_until_finish()
  File "/Users/u/VirtualEnv/dataflow/lib/python2.7/site-packages/apache_beam/runners/dataflow/dataflow_runner.py", line 1122, in wait_until_finish
    (self.state, getattr(self._runner, 'last_error_msg', None)), self)
apache_beam.runners.dataflow.dataflow_runner.DataflowRuntimeException: Dataflow pipeline failed. State: FAILED, Error:
Workflow failed. Causes: There was a problem refreshing your credentials. Please check:
1. Dataflow API is enabled for your project.
2. There is a robot service account for your project:
service-[project number]@dataflow-service-producer-prod.iam.gserviceaccount.com should have access to your project. If this account does not appear in the permissions tab for your project, contact Dataflow support.