osdir.com


[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: --archives flag missing from SparkSubmitHook


Looking at this again, I think one could just use set
`spark.yarn.dist.archives` in the conf passed to the job. If that works,
please disregard :)

On Mon, Sep 17, 2018 at 1:09 PM Ben Laird <br.laird@xxxxxxxxx> wrote:

> The current SparkSubmitHook doesn't appear to support the --archives flag.
> From the Spark docs:
>
> "spark.yarn.dist.archives (none): Comma separated list of archives to be
> extracted into the working directory of each executor."
> https://spark.apache.org/docs/latest/running-on-yarn.html
>
> This is necessary for deploying zipped virtualenvs or other packages
> across the cluster. For now, I'll have to maintain my own copy of this Hook
> but will contribute this back if others aren't planning on it.
>
> Curious if there is context on why this isn't included?
>
> Ben
>