A quick follow-up on using current PortableRunner.
I followed the exact three steps as Ankur and Maximilian shared in
https://beam.apache.org/roadmap/portability/#python-on-flink ; ; The
wordcount example keeps hanging after 10 minutes. I also tried
specifying explicit input/output args, either using gcs folder or local
file system, but none of them works.
Spent some time looking into it but conclusion yet. At this point
though, I guess it does not matter much any more, given we already have
the plan of merging PortableRunner into using java reference runner
Still appreciated if someone can try out the python-on-flink
in case it is just due to my local machine setup. Thanks!
On Thu, Nov 8, 2018 at 5:04 PM Ruoyun Huang <ruoyun@xxxxxxxxxx
I am working on migrating existing PortableRunner to using java ULR
(Link to Notes
If this issue is non-trivial to solve, I would vote for removing
this default behavior as part of the consolidation.
On Thu, Nov 8, 2018 at 2:58 AM Maximilian Michels <mxm@xxxxxxxxxx
In the long run, we should get rid of the Docker-inside-Docker
which was only intended for testing anyways. It would be cleaner to
start the SDK harness container alongside with JobServer container.
Short term, I think it should be easy to either fix the
the mounted "docker" executable or use a Docker image for the
which comes with Docker pre-installed.
Thanks for reporting this Ruoyun!
On 08.11.18 00:10, Ruoyun Huang wrote:
> Thanks Ankur and Maximilian.
> Just for reference in case other people encountering the same
> message, the "permission denied" error in my original email
> due to dockerinsidedocker issue that Ankur mentioned.
> Didn't make the link when you said it, had to discover that
in a hard
> way (I thought it is due to my docker installation messed up).
> On Tue, Nov 6, 2018 at 1:53 AM Maximilian Michels
> <mailto:mxm@xxxxxxxxxx <mailto:mxm@xxxxxxxxxx>>> wrote:
> Please follow
> On 06.11.18 01:14, Ankur Goenka wrote:
> > Hi,
> > The Portable Runner requires a job server uri to work
> > default job server docker image is broken because of
> > docker issue.
> > Please refer to
> how to
> > run a wordcount using Portable Flink Runner.
> > Thanks,
> > Ankur
> > On Mon, Nov 5, 2018 at 3:41 PM Ruoyun Huang
> <mailto:ruoyun@xxxxxxxxxx <mailto:ruoyun@xxxxxxxxxx>>
> > <mailto:ruoyun@xxxxxxxxxx <mailto:ruoyun@xxxxxxxxxx>
<mailto:ruoyun@xxxxxxxxxx <mailto:ruoyun@xxxxxxxxxx>>>> wrote:
> > Hi, Folks,
> > I want to try out Python PortableRunner, by
> > command:
> > *sdk/python: python -m apache_beam.examples.wordcount
> > --output=/tmp/test_output --runner PortableRunner*
> > It complains with following error message:
> > Caused by: java.lang.Exception: The user defined
> > caused an exception: java.io.IOException: Cannot
> > "docker": error=13, Permission denied
> > at
> > at
> > at
> > ... 1 more
> > Caused by:
> > java.io.IOException: Cannot run program "docker":
> > Permission denied
> > at
> > ... 7 more
> > My py2 environment is properly configured, because
> > works. Also I tested my docker installation by
> > hello-world ', no issue.
> > Thanks.
> > --
> > ================
> > Ruoyun Huang
> Ruoyun Huang