osdir.com


[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: Running SpannerWriteIT on dataflow


You want to run this task[1] (either on your machine or by opening a Github PR and using a trigger phrase). Tracing back from that task, you'll find that the root ":javaPostCommit"[2]  task is responsible for running that task and a bunch of others and that the Java SDK Post Commit Tests[3] triggered via the phrase "Run Java PostCommit" is responsible for launching the ":javaPostCommit" task.

You may want to update the include/exclude filter defined in the task to reduce the number of ITs running and or modify the root build.gradle file to only list the single task underneath the javaPostCommit. Note that running the ITs via the trigger phrase without these modifications will run a bunch of tests.

If executing on your machine, you'll want override the values set for the GCP project and GCS location used during the tests by specifying the command line flags -PdataflowProject=<myGCPproject> and -PdataflowTempRoot=<gs://my/bucket>


On Wed, Nov 7, 2018 at 7:29 AM Wout Scheepers <Wout.Scheepers@xxxxxxxxxxxxxxxxxxx> wrote:

Hey all,

 

I’m still running into a bug when streaming into spanner, which I describe in the comments of https://issues.apache.org/jira/browse/BEAM-4796.

I think the cause is a missing equals method on SpannerSchema, for which I get a warning in the worker logs when running on Dataflow.

 

To reproduce this, I would like to run the SpannerWriteIT integration test on dataflow. Could anyone point me into the right direction on how to do this?

 

Thanks in advance

- Wout