OSDir

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Spark submit operator doesn't support properties-file


Hi all,
I see that spark submit operator [0] doesn't support properties-file[1] parameter. 
I need resolve properties from file for my job and now I'm reading properties line-by-line from file and build conf property to spark submit. It is not user friendly for me.
Do community have objections to  implement this property in spark submit hook? 
I would like create issue on Jira and Implement it, if I can.

[0] https://github.com/apache/incubator-airflow/blob/master/airflow/contrib/hooks/spark_submit_hook.py#L202) 
[1] https://jaceklaskowski.gitbooks.io/mastering-apache-spark/content/spark-submit.html#properties-file   

Best Regards,
Anton