osdir.com

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: not able to launch dataflow job - permission issues with service account


My problem I generated a service account - it has all permission and I downloaded the credentials.json for that - But when I'm trying to launch the job - I think it still taking the default account.

Is some configuration I have to do take the service account I created?

On Thu, Nov 22, 2018 at 10:08 PM Ankur Chauhan <ankur@xxxxxxxxxxxx> wrote:
As the message says, enable the api from cloud console of gcloud cli utility. (Refer dataflow docs)

If that’s enabled, ensure the robot account has the permissions it needs - dataflow api, gcs and anything else you are trying to access. 

On Thu, Nov 22, 2018 at 04:33 Unais T <tpunais@xxxxxxxxx> wrote:

I am trying to run a simple data flow job in google cloud - its running perfectly in local - but when I tried to launch it I'm getting the following error - I tried debug a lot

can someone help on this

INFO:root:Created job with id: [2018-11-22_02_57_07-12079060901530487381]
INFO:root:To access the Dataflow monitoring console, please navigate to https://console.cloud.google.com/dataflow/jobsDetail/locations/us-central1/jobs/2018-11-22_02_57_07-12079060901530487381?project=kkfas-main-account-221808
INFO:root:Job 2018-11-22_02_57_07-12079060901530487381 is in state JOB_STATE_PENDING
INFO:root:2018-11-22T10:57:08.052Z: JOB_MESSAGE_DETAILED: Autoscaling is enabled for job 2018-11-22_02_57_07-12079060901530487381. The number of workers will be between 1 and 1000.
INFO:root:2018-11-22T10:57:08.072Z: JOB_MESSAGE_DETAILED: Autoscaling was automatically enabled for job 2018-11-22_02_57_07-12079060901530487381.
INFO:root:2018-11-22T10:57:40.405Z: JOB_MESSAGE_ERROR: Workflow failed. Causes: There was a problem refreshing your credentials. Please check:
1. Dataflow API is enabled for your project.
2. There is a robot service account for your project:
service-[project number]@dataflow-service-producer-prod.iam.gserviceaccount.com should have access to your project. If this account does not appear in the permissions tab for your project, contact Dataflow support.
INFO:root:Job 2018-11-22_02_57_07-12079060901530487381 is in state JOB_STATE_FAILED
Traceback (most recent call last):
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/runpy.py", line 162, in _run_module_as_main
    "__main__", fname, loader, pkg_name)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/runpy.py", line 72, in _run_code
    exec code in run_globals
  File "/Users/u/Projects//digital/test.py", line 49, in <module>
    run()
  File "/Users/u/Projects//dataflow//digital/test.py", line 44, in run
    return p.run().wait_until_finish()
  File "/Users/u/VirtualEnv/dataflow/lib/python2.7/site-packages/apache_beam/runners/dataflow/dataflow_runner.py", line 1122, in wait_until_finish
    (self.state, getattr(self._runner, 'last_error_msg', None)), self)
apache_beam.runners.dataflow.dataflow_runner.DataflowRuntimeException: Dataflow pipeline failed. State: FAILED, Error:
Workflow failed. Causes: There was a problem refreshing your credentials. Please check:
1. Dataflow API is enabled for your project.
2. There is a robot service account for your project:
service-[project number]@dataflow-service-producer-prod.iam.gserviceaccount.com should have access to your project. If this account does not appear in the permissions tab for your project, contact Dataflow support.