Re: --archives flag missing from SparkSubmitHook
Looking at this again, I think one could just use set
`spark.yarn.dist.archives` in the conf passed to the job. If that works,
please disregard :)
On Mon, Sep 17, 2018 at 1:09 PM Ben Laird <br.laird@xxxxxxxxx> wrote:
> The current SparkSubmitHook doesn't appear to support the --archives flag.
> From the Spark docs:
> "spark.yarn.dist.archives (none): Comma separated list of archives to be
> extracted into the working directory of each executor."
> This is necessary for deploying zipped virtualenvs or other packages
> across the cluster. For now, I'll have to maintain my own copy of this Hook
> but will contribute this back if others aren't planning on it.
> Curious if there is context on why this isn't included?