OSDir


[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: KubernetesPodOperator: Invalid arguments were passed to BaseOperator


Hi Taylor,

Thanks, I was thinking about something similar like you're suggesting. But
I'm not confident if the sys.exit() won't kill the whole Airflow process.
For example, if you do an airflow initdb, also the examples will be
initialised, and if you don't have kubernetes installed, it will hit the
sys.exit().

Cheers, Fokko

2018-05-30 20:08 GMT+02:00 Taylor Edmiston <tedmiston@xxxxxxxxx>:

> I used requests instead of kube as an example, but what do you think about
> doing something like this?  I'm happy to put this into a PR if it would
> solve the pain point today.
>
> import logging
>
> try:
>     import requests
> except ModuleNotFoundError:
>     import sys
>     logging.warning('kube not installed - skipping kube examples')
>     sys.exit()
>
> resp = requests.get('http://example.com')
> print(resp)
> ...
>
> Taylor
>
> *Taylor Edmiston*
> Blog <https://blog.tedmiston.com/> | CV
> <https://stackoverflow.com/cv/taylor> | LinkedIn
> <https://www.linkedin.com/in/tedmiston/> | AngelList
> <https://angel.co/taylor> | Stack Overflow
> <https://stackoverflow.com/users/149428/taylor-edmiston>
>
>
> On Wed, May 30, 2018 at 4:40 AM, Driesprong, Fokko <fokko@xxxxxxxxxxxxxx>
> wrote:
>
> > Hi Craig,
> >
> > This is something that needs to be fixed. I agree with you this is very
> > dirty. In your installation you're not installing the kubernetes stuff,
> so
> > the KubernetesPodOperator is ignored. We need to figure out how to have
> > example dags that are not compatible with the vanilla installation, or we
> > need to remove the kubernetes example for now, and move it to the
> > documentation.
> >
> > Cheers, Fokko
> >
> > 2018-05-30 2:11 GMT+02:00 Craig Rodrigues <crodr001@xxxxxxxxx>:
> >
> > > I tested master branch by putting the following in my requirements.txt:
> > >
> > > git+https://github.com/rodrigc/incubator-airflow@
> > > master#egg=apache-airflow[celery,crypto,emr,hive,hdfs,
> > > ldap,mysql,postgres,redis,slack,s3]
> > >
> > > and did a pip install -r requirements.txt
> > >
> > > When I started the airflow webserver, I saw deprecation warnings.  I
> > > put some additional debugging in models.py to through an exception so
> > that
> > > I could see the
> > > full stacktrace:
> > >
> > > [2018-05-29 14:00:34,419] {models.py:307} ERROR - Failed to import:
> > > /Users/c-craigr/airflow2/lib/python2.7/site-packages/
> > > airflow/example_dags/example_kubernetes_operator.py
> > > Traceback (most recent call last):
> > >   File "/Users/c-craigr/airflow2/lib/python2.7/site-packages/
> > airflow/models.py",
> > > line 304, in process_file
> > >     m = imp.load_source(mod_name, filepath)
> > >   File "/Users/c-craigr/airflow2/lib/python2.7/site-packages/
> > > airflow/example_dags/example_kubernetes_operator.py", line 53, in
> > <module>
> > >     dag=dag)
> > >   File "/Users/c-craigr/airflow2/lib/python2.7/site-packages/
> > airflow/utils/decorators.py",
> > > line 98, in wrapper
> > >     result = func(*args, **kwargs)
> > >   File "/Users/c-craigr/airflow2/lib/python2.7/site-packages/
> > airflow/models.py",
> > > line 2308, in __init__
> > >     raise Exception("Invalid use of args or kwargs")
> > > Exception: Invalid use of args or kwargs
> > >
> > >
> > > If looks like example_kubernetes_operator.py, this code is the source
> of
> > > the exception:
> > >
> > > k = KubernetesPodOperator(
> > >     namespace='default',
> > >     image="ubuntu:16.04",
> > >     cmds=["bash", "-cx"],
> > >     arguments=["echo", "10"],
> > >     labels={"foo": "bar"},
> > >     name="airflow-test-pod",
> > >     in_cluster=False,
> > >     task_id="task",
> > >     get_logs=True,
> > >     dag=dag)
> > >
> > >
> > > Without my extra debugging, the deprecation warning looks like this:
> > >
> > > [2018-05-29 14:06:27,567] {example_kubernetes_operator.py:30} WARNING
> -
> > > Could not import KubernetesPodOperator
> > > /Users/c-craigr/airflow2/lib/python2.7/site-packages/
> > airflow/models.py:2315:
> > > PendingDeprecationWarning: Invalid arguments were passed to
> BaseOperator.
> > > Support for passing such arguments will be dropped in Airflow 2.0.
> > Invalid
> > > arguments were:
> > > *args: ()
> > > **kwargs: {'name': 'airflow-test-pod', 'image': 'ubuntu:16.04',
> 'labels':
> > > {'foo': 'bar'}, 'namespace': 'default', 'cmds': ['bash', '-cx'],
> > > 'arguments': ['echo', '10'], 'in_cluster': False, 'get_logs': True}
> > >   category=PendingDeprecationWarning
> > >
> > >
> > >
> > > What is the correct fix for this?  It looks like a lot of operators
> pass
> > > in arguments which are not
> > > processed by BaseOperator, and thus trip over this deprecation warning.
> > >
> > > --
> > > Craig
> > >
> >
>