OSDir


[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [VOTE] Airflow 1.10.0rc1


+1 (binding)

* Sigs look good
* Artifact has incubating in name
* LICENSE/NOTICE/DISCLAIMER look good
   - nit: DISCLAIMER is not word wrapped to 80 chars as I've seen in
other projects.
* Nit: Last release Sebb called out "Copyright 2016 and onwards" in
NOTICE as being imprecise.  This language remains.
* Spot check on license headers looks ok.
* Nit: Last release there was a request for instructions on how to
build the code.  This isn't included.

-Jakob

On 11 July 2018 at 19:50, Sid Anand <sanand@xxxxxxxxxx> wrote:
> FYI!
> I just installed the release candidate. The first thing I noticed is a
> missing tool tip for the Null State in the Recent Tasks column on the
> landing page. Since the null globe is new to this UI, users will likely
> hover over it to inquire what it means... and will be left wanting. Of
> course, they could click on the globe, which will take them to
> http://localhost:8080/admin/taskinstance/?flt1_dag_id_equals=example_bash_operator&flt2_state_equals=null,
> which will always show an empty list, leaving them a bit more confused.
>
> -s
>
> On Wed, Jul 11, 2018 at 3:13 PM Carl Johan Gustavsson
> <carl.johan@xxxxxxxxxxx.invalid> wrote:
>
>> Hi Bolke,
>>
>> (Switching email to avoid moderation on my emails.)
>>
>> The normal Airflow test suite does not fail as it uses a LC_ALL set to
>> utf-8.
>>
>> I think it is a proper test though, it is a minimal reproducible version of
>> the code that fails. And the only difference in behaviour is at 3.7 which
>> we don’t support anyway so I’m fairly sure it is broken for all supported
>> Python 3 versions.
>>
>> I now tried running the tests in docker using 3.5 with the LC_ALL/LANG
>> unset and I see the same failure.
>>
>> I don’t think this is a big thing though and we could release it without
>> the fix I made. I think most people run it with a sane LC_ALL, but
>> apparently we didn’t.
>> Here’s the log for the test:
>>
>> > docker run -t -i -v `pwd`:/airflow/ python:3.5 bash
>> root@b99b297df111:/# locale
>> LANG=C.UTF-8
>> LANGUAGE=
>> LC_CTYPE="C.UTF-8"
>> LC_NUMERIC="C.UTF-8"
>> LC_TIME="C.UTF-8"
>> LC_COLLATE="C.UTF-8"
>> LC_MONETARY="C.UTF-8"
>> LC_MESSAGES="C.UTF-8"
>> LC_PAPER="C.UTF-8"
>> LC_NAME="C.UTF-8"
>> LC_ADDRESS="C.UTF-8"
>> LC_TELEPHONE="C.UTF-8"
>> LC_MEASUREMENT="C.UTF-8"
>> LC_IDENTIFICATION="C.UTF-8"
>> LC_ALL=
>> > unset LANG
>> root@b99b297df111:/# locale
>> LANG=
>> LANGUAGE=
>> LC_CTYPE="POSIX"
>> LC_NUMERIC="POSIX"
>> LC_TIME="POSIX"
>> LC_COLLATE="POSIX"
>> LC_MONETARY="POSIX"
>> LC_MESSAGES="POSIX"
>> LC_PAPER="POSIX"
>> LC_NAME="POSIX"
>> LC_ADDRESS="POSIX"
>> LC_TELEPHONE="POSIX"
>> LC_MEASUREMENT="POSIX"
>> LC_IDENTIFICATION="POSIX"
>> LC_ALL=
>> root@b99b297df111:/# pip install -e .[devel]
>> root@b99b297df111:/airflow# ./run_unit_tests.sh
>> + export AIRFLOW_HOME=/root/airflow
>> + AIRFLOW_HOME=/root/airflow
>> + export AIRFLOW__CORE__UNIT_TEST_MODE=True
>> + AIRFLOW__CORE__UNIT_TEST_MODE=True
>> + export AIRFLOW__TESTSECTION__TESTKEY=testvalue
>> + AIRFLOW__TESTSECTION__TESTKEY=testvalue
>> + export AIRFLOW_USE_NEW_IMPORTS=1
>> + AIRFLOW_USE_NEW_IMPORTS=1
>> +++ dirname ./run_unit_tests.sh
>> ++ cd .
>> ++ pwd
>> + DIR=/airflow
>> + export PYTHONPATH=:/airflow/tests/test_utils
>> + PYTHONPATH=:/airflow/tests/test_utils
>> + nose_args=
>> + which airflow
>> + echo 'Initializing the DB'
>> Initializing the DB
>> + airflow resetdb
>> + yes
>> Traceback (most recent call last):
>>   File "/usr/local/bin/airflow", line 6, in <module>
>>     exec(compile(open(__file__).read(), __file__, 'exec'))
>>   File "/airflow/airflow/bin/airflow", line 21, in <module>
>>     from airflow import configuration
>>   File "/airflow/airflow/__init__.py", line 35, in <module>
>>     from airflow import configuration as conf
>>   File "/airflow/airflow/configuration.py", line 106, in <module>
>>     DEFAULT_CONFIG = f.read()
>>   File "/usr/local/lib/python3.5/encodings/ascii.py", line 26, in decode
>>     return codecs.ascii_decode(input, self.errors)[0]
>> UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 21082:
>> ordinal not in range(128)
>> + '[' '' ']'
>> + '[' -z '' ']'
>> + nose_args='--with-coverage     --cover-erase     --cover-html
>> --cover-package=airflow     --cover-html-dir=airflow/www/static/coverage
>>   --with-ignore-docstrings     --rednose     --with-timer     -s     -v
>> --logging-level=DEBUG '
>> + echo 'Starting the unit tests with the following nose arguments:
>> --with-coverage' --cover-erase --cover-html --cover-package=airflow
>> --cover-html-dir=airflow/www/static/coverage --with-ignore-docstrings
>> --rednose --with-timer -s -v --logging-level=DEBUG
>> Starting the unit tests with the following nose arguments: --with-coverage
>> --cover-erase --cover-html --cover-package=airflow
>> --cover-html-dir=airflow/www/static/coverage --with-ignore-docstrings
>> --rednose --with-timer -s -v --logging-level=DEBUG
>> + nosetests --with-coverage --cover-erase --cover-html
>> --cover-package=airflow --cover-html-dir=airflow/www/static/coverage
>> --with-ignore-docstrings --rednose --with-timer -s -v --logging-level=DEBUG
>> nose.plugins.cover: ERROR: Coverage not available: unable to import
>> coverage module
>> Failure: UnicodeDecodeError ('ascii' codec can't decode byte 0xe2 in
>> position 21082: ordinal not in range(128)) ... ERROR
>> Failure: UnicodeDecodeError ('ascii' codec can't decode byte 0xe2 in
>> position 21082: ordinal not in range(128)) ... ERROR
>> ======================================================================
>> 1) ERROR: Failure: UnicodeDecodeError ('ascii' codec can't decode byte 0xe2
>> in position 21082: ordinal not in range(128))
>> ----------------------------------------------------------------------
>>    Traceback (most recent call last):
>>     /usr/local/lib/python3.5/site-packages/nose/failure.py line 39 in
>> runTest
>>       raise self.exc_val.with_traceback(self.tb)
>>     /usr/local/lib/python3.5/site-packages/nose/loader.py line 418 in
>> loadTestsFromName
>>       addr.filename, addr.module)
>>     /usr/local/lib/python3.5/site-packages/nose/importer.py line 47 in
>> importFromPath
>>       return self.importFromDir(dir_path, fqname)
>>     /usr/local/lib/python3.5/site-packages/nose/importer.py line 94 in
>> importFromDir
>>       mod = load_module(part_fqname, fh, filename, desc)
>>     /usr/local/lib/python3.5/imp.py line 245 in load_module
>>       return load_package(name, filename)
>>     /usr/local/lib/python3.5/imp.py line 217 in load_package
>>       return _load(spec)
>>     <frozen importlib._bootstrap> line 693 in _load
>>
>>     <frozen importlib._bootstrap> line 673 in _load_unlocked
>>
>>     <frozen importlib._bootstrap_external> line 697 in exec_module
>>
>>     <frozen importlib._bootstrap> line 222 in _call_with_frames_removed
>>
>>     airflow/__init__.py line 35 in <module>
>>       from airflow import configuration as conf
>>     airflow/configuration.py line 106 in <module>
>>       DEFAULT_CONFIG = f.read()
>>     /usr/local/lib/python3.5/encodings/ascii.py line 26 in decode
>>       return codecs.ascii_decode(input, self.errors)[0]
>>    UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position
>> 21082: ordinal not in range(128)
>> ======================================================================
>> 2) ERROR: Failure: UnicodeDecodeError ('ascii' codec can't decode byte 0xe2
>> in position 21082: ordinal not in range(128))
>> ----------------------------------------------------------------------
>>    Traceback (most recent call last):
>>     /usr/local/lib/python3.5/site-packages/nose/failure.py line 39 in
>> runTest
>>       raise self.exc_val.with_traceback(self.tb)
>>     /usr/local/lib/python3.5/site-packages/nose/loader.py line 418 in
>> loadTestsFromName
>>       addr.filename, addr.module)
>>     /usr/local/lib/python3.5/site-packages/nose/importer.py line 47 in
>> importFromPath
>>       return self.importFromDir(dir_path, fqname)
>>     /usr/local/lib/python3.5/site-packages/nose/importer.py line 94 in
>> importFromDir
>>       mod = load_module(part_fqname, fh, filename, desc)
>>     /usr/local/lib/python3.5/imp.py line 245 in load_module
>>       return load_package(name, filename)
>>     /usr/local/lib/python3.5/imp.py line 217 in load_package
>>       return _load(spec)
>>     <frozen importlib._bootstrap> line 693 in _load
>>
>>     <frozen importlib._bootstrap> line 673 in _load_unlocked
>>
>>     <frozen importlib._bootstrap_external> line 697 in exec_module
>>
>>     <frozen importlib._bootstrap> line 222 in _call_with_frames_removed
>>
>>     tests/__init__.py line 25 in <module>
>>       from .configuration import *
>>     tests/configuration.py line 28 in <module>
>>       from airflow import configuration
>>     airflow/__init__.py line 35 in <module>
>>       from airflow import configuration as conf
>>     airflow/configuration.py line 106 in <module>
>>       DEFAULT_CONFIG = f.read()
>>     /usr/local/lib/python3.5/encodings/ascii.py line 26 in decode
>>       return codecs.ascii_decode(input, self.errors)[0]
>>    UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position
>> 21082: ordinal not in range(128)
>>
>> [error] 100.00% nose.failure.Failure.runTest: 0.0004s
>>
>> -----------------------------------------------------------------------------
>> 2 tests run in 0.060 seconds.
>> 2 errors (0 tests passed)
>>
>>
>> --
>> Carl Johan Gustavsson
>>
>> On 11 July 2018 at 23:24:01, Bolke de Bruin (bdbruin@xxxxxxxxx) wrote:
>>
>> Hi Carl,
>>
>> That is not a real check, ie. Does Airflow have the same issue clean
>> install on 3.5? Travis’ tests run on 3.5.
>>
>> B.
>>
>> Verstuurd vanaf mijn iPad
>>
>> > Op 10 jul. 2018 om 15:10 heeft Carl Johan Gustavsson <
>> carl.j.gustavsson@xxxxxxxxx> het volgende geschreven:
>> >
>> > Hi Bolke,
>> >
>> >
>> > I did a quick test on 3.5.5, 3.4.0 and 3.7.0 on OS X now, all but 3.7
>> breaks
>> >
>> > Quick repro:
>> >
>> > ➜ ~ pyenv local 3.5.5
>> > ➜ ~ locale
>> > LANG=
>> > LC_COLLATE="C"
>> > LC_CTYPE="C"
>> > LC_MESSAGES="C"
>> > LC_MONETARY="C"
>> > LC_NUMERIC="C"
>> > LC_TIME="C"
>> > LC_ALL=
>> > ➜ ~ cat testweird.txt
>> > ’
>> > ➜ ~ python
>> > Python 3.5.5 (default, Jul 7 2018, 17:00:56)
>> > [GCC 4.2.1 Compatible Apple LLVM 9.1.0 (clang-902.0.39.1)] on darwin
>> > Type "help", "copyright", "credits" or "license" for more information.
>> > >>> open('testweird.txt').read()
>> > Traceback (most recent call last):
>> > File "<stdin>", line 1, in <module>
>> > File "/Users/cjg/.pyenv/versions/3.5.5/lib/python3.5/encodings/ascii.py",
>> line 26, in decode
>> > return codecs.ascii_decode(input, self.errors)[0]
>> > UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 0:
>> ordinal not in range(128)
>> > >>>
>> >
>> >
>> > Maybe not a blocking change but it is a breaking change from 1.9 I guess.
>> >
>> >
>> > / Carl Johan
>> >
>> >
>> >> On 10 July 2018 at 14:48:47, Bolke de Bruin (bdbruin@xxxxxxxxx) wrote:
>> >>
>> >> Hi Carl,
>> >>
>> >> Did you this on python 3.5 as well? 3.6 is not an officially supported
>> (yet). As a workaround is available I won’t consider this blocking btw.
>> >>
>> >> Bolke
>> >>
>> >> Verstuurd vanaf mijn iPad
>> >>
>> >> Op 10 jul. 2018 om 11:53 heeft Carl Johan Gustavsson <
>> carl.j.gustavsson@xxxxxxxxx> het volgende geschreven:
>> >>
>> >>> Hi,
>> >>>
>> >>> First of all, thank you for all the work with the release management.
>> >>>
>> >>> I ran in to a weird issue testing the RC1, running under Python 3.6.0 /
>> Ubuntu 14.04.5, upgrading from a master build from February.
>> >>>
>> >>> Jul 10 08:50:33 hostname supervisord: airflow-webserver-01 Traceback
>> (most recent call last):
>> >>> Jul 10 08:50:33 hostname supervisord: airflow-webserver-01 File
>> "/opt/virtualenv/tictail/pipeline/bin/airflow", line 21, in <module>
>> >>> Jul 10 08:50:33 hostname supervisord: airflow-webserver-01 from airflow
>> import configuration
>> >>> Jul 10 08:50:33 hostname supervisord: airflow-webserver-01 File
>>
>> "/opt/virtualenv/tictail/pipeline/lib/python3.6/site-packages/airflow/__init__.py",
>> line 35, in <module>
>> >>> Jul 10 08:50:33 hostname supervisord: airflow-webserver-01 from airflow
>> import configuration as conf
>> >>> Jul 10 08:50:33 hostname supervisord: airflow-webserver-01 File
>>
>> "/opt/virtualenv/tictail/pipeline/lib/python3.6/site-packages/airflow/configuration.py",
>> line 106, in <module>
>> >>> Jul 10 08:50:33 hostname supervisord: airflow-webserver-01
>> DEFAULT_CONFIG = f.read()
>> >>> Jul 10 08:50:33 hostname supervisord: airflow-webserver-01 File
>> "/opt/virtualenv/tictail/pipeline/lib/python3.6/encodings/ascii.py", line
>> 26, in decode
>> >>> Jul 10 08:50:33 hostname supervisord: airflow-webserver-01 return
>> codecs.ascii_decode(input, self.errors)[0]
>> >>> Jul 10 08:50:33 hostname supervisord: airflow-webserver-01
>> UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 20770:
>> ordinal not in range(128)
>> >>>
>> >>> Removing the ’ on
>>
>> https://github.com/apache/incubator-airflow/blob/master/airflow/config_templates/default_airflow.cfg#L613
>> solved the issue for me, and digging a bit deeper it seems Airflow now
>> requires setting LC_ALL=en_US.UTF-8 in the environment or similar to force
>> Python to read the file as utf-8 and not ascii. (I think this was changed
>> in to default to utf-8 in Python 3.7).
>> >>>
>> >>> I see 3 solutions for this
>> >>> 1. Document that Airflow need to run with LC_ALL=en_US.UTF-8 or
>> similar.
>> >>> 2. Change the default config file to not contain non-ascii characters.
>> >>> 3. Always read the file as unicode regardless of the LC_ALL
>> environment, by the encoding='utf-8’ parameter to open().
>> >>>
>> >>> I think 3 is the best solution, and I can prepare a PR for that if
>> necessary .
>> >>>
>> >>> I guess this counts as -1 (non-binding)
>> >>>
>> >>> All the best
>> >>>
>> >>> Carl Johan
>> >>>
>> >>>
>> >>>
>> >>>> On 8 July 2018 at 22:02:33, Bolke de Bruin (bdbruin@xxxxxxxxx) wrote:
>> >>>>
>> >>>> Hey all,
>> >>>>
>> >>>> I have cut Airflow 1.10.0 RC1. This email is calling a vote on the
>> release,
>> >>>> which will last for 72 hours. Consider this my (binding) +1.
>> >>>>
>> >>>> Airflow 1.10.0 RC 1 is available at:
>> >>>>
>> >>>> https://dist.apache.org/repos/dist/dev/incubator/airflow/1.10.0rc1/ <
>> https://dist.apache.org/repos/dist/dev/incubator/airflow/1.10.0rc1/>
>> >>>>
>> >>>> apache-airflow-1.10.0rc1+incubating-source.tar.gz is a source release
>> that
>> >>>> comes with INSTALL instructions.
>> >>>> apache-airflow-1.10.0rc1+incubating-bin.tar.gz is the binary Python
>> "sdist"
>> >>>> release.
>> >>>>
>> >>>> Public keys are available at:
>> >>>>
>> >>>> https://dist.apache.org/repos/dist/release/incubator/airflow/ <
>> https://dist.apache.org/repos/dist/release/incubator/airflow/>
>> >>>>
>> >>>> The amount of JIRAs fixed is over 700. Please have a look at the
>> changelog.
>> >>>>
>> >>>> Please note that the version number excludes the `rcX` string as well
>> >>>> as the "+incubating" string, so it's now simply 1.10.0. This will
>> allow us
>> >>>> to rename the artifact without modifying the artifact checksums when
>> we
>> >>>> actually release.
>> >>>>
>> >>>>
>> >>>> Cheers,
>> >>>> Bolke
>>