OSDir


[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

Re: [VOTE] Airflow 1.10.0rc1


That said I can't reproduce it myself (I will take discussion to the Jira ticket)

-ash

> On 12 Jul 2018, at 13:36, Ash Berlin-Taylor <ash_airflowlist@xxxxxxxxxxxxxx> wrote:
> 
> Possible blocking regression here - .airflowignore doesn't seem to be respected anymore
> 
> https://issues.apache.org/jira/browse/AIRFLOW-2729 <https://issues.apache.org/jira/browse/AIRFLOW-2729>
> 
> -ash
>> On 12 Jul 2018, at 08:45, Bolke de Bruin <bdbruin@xxxxxxxxx> wrote:
>> 
>> Hi Jakob,
>> 
>> Thanks. We do include an INSTALL document that explains how to well install airflow and is quite a standard location for install instructions. Or did the install instruction needed to be included in the Vote?
>> 
>> B.
>> 
>> 
>> 
>> Verstuurd vanaf mijn iPad
>> 
>>> Op 12 jul. 2018 om 07:54 heeft Jakob Homan <jghoman@xxxxxxxxx> het volgende geschreven:
>>> 
>>> +1 (binding)
>>> 
>>> * Sigs look good
>>> * Artifact has incubating in name
>>> * LICENSE/NOTICE/DISCLAIMER look good
>>> - nit: DISCLAIMER is not word wrapped to 80 chars as I've seen in
>>> other projects.
>>> * Nit: Last release Sebb called out "Copyright 2016 and onwards" in
>>> NOTICE as being imprecise.  This language remains.
>>> * Spot check on license headers looks ok.
>>> * Nit: Last release there was a request for instructions on how to
>>> build the code.  This isn't included.
>>> 
>>> -Jakob
>>> 
>>>> On 11 July 2018 at 19:50, Sid Anand <sanand@xxxxxxxxxx> wrote:
>>>> FYI!
>>>> I just installed the release candidate. The first thing I noticed is a
>>>> missing tool tip for the Null State in the Recent Tasks column on the
>>>> landing page. Since the null globe is new to this UI, users will likely
>>>> hover over it to inquire what it means... and will be left wanting. Of
>>>> course, they could click on the globe, which will take them to
>>>> http://localhost:8080/admin/taskinstance/?flt1_dag_id_equals=example_bash_operator&flt2_state_equals=null,
>>>> which will always show an empty list, leaving them a bit more confused.
>>>> 
>>>> -s
>>>> 
>>>> On Wed, Jul 11, 2018 at 3:13 PM Carl Johan Gustavsson
>>>> <carl.johan@xxxxxxxxxxx.invalid> wrote:
>>>> 
>>>>> Hi Bolke,
>>>>> 
>>>>> (Switching email to avoid moderation on my emails.)
>>>>> 
>>>>> The normal Airflow test suite does not fail as it uses a LC_ALL set to
>>>>> utf-8.
>>>>> 
>>>>> I think it is a proper test though, it is a minimal reproducible version of
>>>>> the code that fails. And the only difference in behaviour is at 3.7 which
>>>>> we don’t support anyway so I’m fairly sure it is broken for all supported
>>>>> Python 3 versions.
>>>>> 
>>>>> I now tried running the tests in docker using 3.5 with the LC_ALL/LANG
>>>>> unset and I see the same failure.
>>>>> 
>>>>> I don’t think this is a big thing though and we could release it without
>>>>> the fix I made. I think most people run it with a sane LC_ALL, but
>>>>> apparently we didn’t.
>>>>> Here’s the log for the test:
>>>>> 
>>>>>> docker run -t -i -v `pwd`:/airflow/ python:3.5 bash
>>>>> root@b99b297df111:/# locale
>>>>> LANG=C.UTF-8
>>>>> LANGUAGE=
>>>>> LC_CTYPE="C.UTF-8"
>>>>> LC_NUMERIC="C.UTF-8"
>>>>> LC_TIME="C.UTF-8"
>>>>> LC_COLLATE="C.UTF-8"
>>>>> LC_MONETARY="C.UTF-8"
>>>>> LC_MESSAGES="C.UTF-8"
>>>>> LC_PAPER="C.UTF-8"
>>>>> LC_NAME="C.UTF-8"
>>>>> LC_ADDRESS="C.UTF-8"
>>>>> LC_TELEPHONE="C.UTF-8"
>>>>> LC_MEASUREMENT="C.UTF-8"
>>>>> LC_IDENTIFICATION="C.UTF-8"
>>>>> LC_ALL=
>>>>>> unset LANG
>>>>> root@b99b297df111:/# locale
>>>>> LANG=
>>>>> LANGUAGE=
>>>>> LC_CTYPE="POSIX"
>>>>> LC_NUMERIC="POSIX"
>>>>> LC_TIME="POSIX"
>>>>> LC_COLLATE="POSIX"
>>>>> LC_MONETARY="POSIX"
>>>>> LC_MESSAGES="POSIX"
>>>>> LC_PAPER="POSIX"
>>>>> LC_NAME="POSIX"
>>>>> LC_ADDRESS="POSIX"
>>>>> LC_TELEPHONE="POSIX"
>>>>> LC_MEASUREMENT="POSIX"
>>>>> LC_IDENTIFICATION="POSIX"
>>>>> LC_ALL=
>>>>> root@b99b297df111:/# pip install -e .[devel]
>>>>> root@b99b297df111:/airflow# ./run_unit_tests.sh
>>>>> + export AIRFLOW_HOME=/root/airflow
>>>>> + AIRFLOW_HOME=/root/airflow
>>>>> + export AIRFLOW__CORE__UNIT_TEST_MODE=True
>>>>> + AIRFLOW__CORE__UNIT_TEST_MODE=True
>>>>> + export AIRFLOW__TESTSECTION__TESTKEY=testvalue
>>>>> + AIRFLOW__TESTSECTION__TESTKEY=testvalue
>>>>> + export AIRFLOW_USE_NEW_IMPORTS=1
>>>>> + AIRFLOW_USE_NEW_IMPORTS=1
>>>>> +++ dirname ./run_unit_tests.sh
>>>>> ++ cd .
>>>>> ++ pwd
>>>>> + DIR=/airflow
>>>>> + export PYTHONPATH=:/airflow/tests/test_utils
>>>>> + PYTHONPATH=:/airflow/tests/test_utils
>>>>> + nose_args=
>>>>> + which airflow
>>>>> + echo 'Initializing the DB'
>>>>> Initializing the DB
>>>>> + airflow resetdb
>>>>> + yes
>>>>> Traceback (most recent call last):
>>>>> File "/usr/local/bin/airflow", line 6, in <module>
>>>>>  exec(compile(open(__file__).read(), __file__, 'exec'))
>>>>> File "/airflow/airflow/bin/airflow", line 21, in <module>
>>>>>  from airflow import configuration
>>>>> File "/airflow/airflow/__init__.py", line 35, in <module>
>>>>>  from airflow import configuration as conf
>>>>> File "/airflow/airflow/configuration.py", line 106, in <module>
>>>>>  DEFAULT_CONFIG = f.read()
>>>>> File "/usr/local/lib/python3.5/encodings/ascii.py", line 26, in decode
>>>>>  return codecs.ascii_decode(input, self.errors)[0]
>>>>> UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 21082:
>>>>> ordinal not in range(128)
>>>>> + '[' '' ']'
>>>>> + '[' -z '' ']'
>>>>> + nose_args='--with-coverage     --cover-erase     --cover-html
>>>>> --cover-package=airflow     --cover-html-dir=airflow/www/static/coverage
>>>>> --with-ignore-docstrings     --rednose     --with-timer     -s     -v
>>>>> --logging-level=DEBUG '
>>>>> + echo 'Starting the unit tests with the following nose arguments:
>>>>> --with-coverage' --cover-erase --cover-html --cover-package=airflow
>>>>> --cover-html-dir=airflow/www/static/coverage --with-ignore-docstrings
>>>>> --rednose --with-timer -s -v --logging-level=DEBUG
>>>>> Starting the unit tests with the following nose arguments: --with-coverage
>>>>> --cover-erase --cover-html --cover-package=airflow
>>>>> --cover-html-dir=airflow/www/static/coverage --with-ignore-docstrings
>>>>> --rednose --with-timer -s -v --logging-level=DEBUG
>>>>> + nosetests --with-coverage --cover-erase --cover-html
>>>>> --cover-package=airflow --cover-html-dir=airflow/www/static/coverage
>>>>> --with-ignore-docstrings --rednose --with-timer -s -v --logging-level=DEBUG
>>>>> nose.plugins.cover: ERROR: Coverage not available: unable to import
>>>>> coverage module
>>>>> Failure: UnicodeDecodeError ('ascii' codec can't decode byte 0xe2 in
>>>>> position 21082: ordinal not in range(128)) ... ERROR
>>>>> Failure: UnicodeDecodeError ('ascii' codec can't decode byte 0xe2 in
>>>>> position 21082: ordinal not in range(128)) ... ERROR
>>>>> ======================================================================
>>>>> 1) ERROR: Failure: UnicodeDecodeError ('ascii' codec can't decode byte 0xe2
>>>>> in position 21082: ordinal not in range(128))
>>>>> ----------------------------------------------------------------------
>>>>> Traceback (most recent call last):
>>>>>  /usr/local/lib/python3.5/site-packages/nose/failure.py line 39 in
>>>>> runTest
>>>>>    raise self.exc_val.with_traceback(self.tb)
>>>>>  /usr/local/lib/python3.5/site-packages/nose/loader.py line 418 in
>>>>> loadTestsFromName
>>>>>    addr.filename, addr.module)
>>>>>  /usr/local/lib/python3.5/site-packages/nose/importer.py line 47 in
>>>>> importFromPath
>>>>>    return self.importFromDir(dir_path, fqname)
>>>>>  /usr/local/lib/python3.5/site-packages/nose/importer.py line 94 in
>>>>> importFromDir
>>>>>    mod = load_module(part_fqname, fh, filename, desc)
>>>>>  /usr/local/lib/python3.5/imp.py line 245 in load_module
>>>>>    return load_package(name, filename)
>>>>>  /usr/local/lib/python3.5/imp.py line 217 in load_package
>>>>>    return _load(spec)
>>>>>  <frozen importlib._bootstrap> line 693 in _load
>>>>> 
>>>>>  <frozen importlib._bootstrap> line 673 in _load_unlocked
>>>>> 
>>>>>  <frozen importlib._bootstrap_external> line 697 in exec_module
>>>>> 
>>>>>  <frozen importlib._bootstrap> line 222 in _call_with_frames_removed
>>>>> 
>>>>>  airflow/__init__.py line 35 in <module>
>>>>>    from airflow import configuration as conf
>>>>>  airflow/configuration.py line 106 in <module>
>>>>>    DEFAULT_CONFIG = f.read()
>>>>>  /usr/local/lib/python3.5/encodings/ascii.py line 26 in decode
>>>>>    return codecs.ascii_decode(input, self.errors)[0]
>>>>> UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position
>>>>> 21082: ordinal not in range(128)
>>>>> ======================================================================
>>>>> 2) ERROR: Failure: UnicodeDecodeError ('ascii' codec can't decode byte 0xe2
>>>>> in position 21082: ordinal not in range(128))
>>>>> ----------------------------------------------------------------------
>>>>> Traceback (most recent call last):
>>>>>  /usr/local/lib/python3.5/site-packages/nose/failure.py line 39 in
>>>>> runTest
>>>>>    raise self.exc_val.with_traceback(self.tb)
>>>>>  /usr/local/lib/python3.5/site-packages/nose/loader.py line 418 in
>>>>> loadTestsFromName
>>>>>    addr.filename, addr.module)
>>>>>  /usr/local/lib/python3.5/site-packages/nose/importer.py line 47 in
>>>>> importFromPath
>>>>>    return self.importFromDir(dir_path, fqname)
>>>>>  /usr/local/lib/python3.5/site-packages/nose/importer.py line 94 in
>>>>> importFromDir
>>>>>    mod = load_module(part_fqname, fh, filename, desc)
>>>>>  /usr/local/lib/python3.5/imp.py line 245 in load_module
>>>>>    return load_package(name, filename)
>>>>>  /usr/local/lib/python3.5/imp.py line 217 in load_package
>>>>>    return _load(spec)
>>>>>  <frozen importlib._bootstrap> line 693 in _load
>>>>> 
>>>>>  <frozen importlib._bootstrap> line 673 in _load_unlocked
>>>>> 
>>>>>  <frozen importlib._bootstrap_external> line 697 in exec_module
>>>>> 
>>>>>  <frozen importlib._bootstrap> line 222 in _call_with_frames_removed
>>>>> 
>>>>>  tests/__init__.py line 25 in <module>
>>>>>    from .configuration import *
>>>>>  tests/configuration.py line 28 in <module>
>>>>>    from airflow import configuration
>>>>>  airflow/__init__.py line 35 in <module>
>>>>>    from airflow import configuration as conf
>>>>>  airflow/configuration.py line 106 in <module>
>>>>>    DEFAULT_CONFIG = f.read()
>>>>>  /usr/local/lib/python3.5/encodings/ascii.py line 26 in decode
>>>>>    return codecs.ascii_decode(input, self.errors)[0]
>>>>> UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position
>>>>> 21082: ordinal not in range(128)
>>>>> 
>>>>> [error] 100.00% nose.failure.Failure.runTest: 0.0004s
>>>>> 
>>>>> -----------------------------------------------------------------------------
>>>>> 2 tests run in 0.060 seconds.
>>>>> 2 errors (0 tests passed)
>>>>> 
>>>>> 
>>>>> --
>>>>> Carl Johan Gustavsson
>>>>> 
>>>>> On 11 July 2018 at 23:24:01, Bolke de Bruin (bdbruin@xxxxxxxxx) wrote:
>>>>> 
>>>>> Hi Carl,
>>>>> 
>>>>> That is not a real check, ie. Does Airflow have the same issue clean
>>>>> install on 3.5? Travis’ tests run on 3.5.
>>>>> 
>>>>> B.
>>>>> 
>>>>> Verstuurd vanaf mijn iPad
>>>>> 
>>>>>> Op 10 jul. 2018 om 15:10 heeft Carl Johan Gustavsson <
>>>>> carl.j.gustavsson@xxxxxxxxx> het volgende geschreven:
>>>>>> 
>>>>>> Hi Bolke,
>>>>>> 
>>>>>> 
>>>>>> I did a quick test on 3.5.5, 3.4.0 and 3.7.0 on OS X now, all but 3.7
>>>>> breaks
>>>>>> 
>>>>>> Quick repro:
>>>>>> 
>>>>>> ➜ ~ pyenv local 3.5.5
>>>>>> ➜ ~ locale
>>>>>> LANG=
>>>>>> LC_COLLATE="C"
>>>>>> LC_CTYPE="C"
>>>>>> LC_MESSAGES="C"
>>>>>> LC_MONETARY="C"
>>>>>> LC_NUMERIC="C"
>>>>>> LC_TIME="C"
>>>>>> LC_ALL=
>>>>>> ➜ ~ cat testweird.txt
>>>>>> ’
>>>>>> ➜ ~ python
>>>>>> Python 3.5.5 (default, Jul 7 2018, 17:00:56)
>>>>>> [GCC 4.2.1 Compatible Apple LLVM 9.1.0 (clang-902.0.39.1)] on darwin
>>>>>> Type "help", "copyright", "credits" or "license" for more information.
>>>>>>>>> open('testweird.txt').read()
>>>>>> Traceback (most recent call last):
>>>>>> File "<stdin>", line 1, in <module>
>>>>>> File "/Users/cjg/.pyenv/versions/3.5.5/lib/python3.5/encodings/ascii.py",
>>>>> line 26, in decode
>>>>>> return codecs.ascii_decode(input, self.errors)[0]
>>>>>> UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 0:
>>>>> ordinal not in range(128)
>>>>>>>>> 
>>>>>> 
>>>>>> 
>>>>>> Maybe not a blocking change but it is a breaking change from 1.9 I guess.
>>>>>> 
>>>>>> 
>>>>>> / Carl Johan
>>>>>> 
>>>>>> 
>>>>>>> On 10 July 2018 at 14:48:47, Bolke de Bruin (bdbruin@xxxxxxxxx) wrote:
>>>>>>> 
>>>>>>> Hi Carl,
>>>>>>> 
>>>>>>> Did you this on python 3.5 as well? 3.6 is not an officially supported
>>>>> (yet). As a workaround is available I won’t consider this blocking btw.
>>>>>>> 
>>>>>>> Bolke
>>>>>>> 
>>>>>>> Verstuurd vanaf mijn iPad
>>>>>>> 
>>>>>>> Op 10 jul. 2018 om 11:53 heeft Carl Johan Gustavsson <
>>>>> carl.j.gustavsson@xxxxxxxxx> het volgende geschreven:
>>>>>>> 
>>>>>>>> Hi,
>>>>>>>> 
>>>>>>>> First of all, thank you for all the work with the release management.
>>>>>>>> 
>>>>>>>> I ran in to a weird issue testing the RC1, running under Python 3.6.0 /
>>>>> Ubuntu 14.04.5, upgrading from a master build from February.
>>>>>>>> 
>>>>>>>> Jul 10 08:50:33 hostname supervisord: airflow-webserver-01 Traceback
>>>>> (most recent call last):
>>>>>>>> Jul 10 08:50:33 hostname supervisord: airflow-webserver-01 File
>>>>> "/opt/virtualenv/tictail/pipeline/bin/airflow", line 21, in <module>
>>>>>>>> Jul 10 08:50:33 hostname supervisord: airflow-webserver-01 from airflow
>>>>> import configuration
>>>>>>>> Jul 10 08:50:33 hostname supervisord: airflow-webserver-01 File
>>>>> 
>>>>> "/opt/virtualenv/tictail/pipeline/lib/python3.6/site-packages/airflow/__init__.py",
>>>>> line 35, in <module>
>>>>>>>> Jul 10 08:50:33 hostname supervisord: airflow-webserver-01 from airflow
>>>>> import configuration as conf
>>>>>>>> Jul 10 08:50:33 hostname supervisord: airflow-webserver-01 File
>>>>> 
>>>>> "/opt/virtualenv/tictail/pipeline/lib/python3.6/site-packages/airflow/configuration.py",
>>>>> line 106, in <module>
>>>>>>>> Jul 10 08:50:33 hostname supervisord: airflow-webserver-01
>>>>> DEFAULT_CONFIG = f.read()
>>>>>>>> Jul 10 08:50:33 hostname supervisord: airflow-webserver-01 File
>>>>> "/opt/virtualenv/tictail/pipeline/lib/python3.6/encodings/ascii.py", line
>>>>> 26, in decode
>>>>>>>> Jul 10 08:50:33 hostname supervisord: airflow-webserver-01 return
>>>>> codecs.ascii_decode(input, self.errors)[0]
>>>>>>>> Jul 10 08:50:33 hostname supervisord: airflow-webserver-01
>>>>> UnicodeDecodeError: 'ascii' codec can't decode byte 0xe2 in position 20770:
>>>>> ordinal not in range(128)
>>>>>>>> 
>>>>>>>> Removing the ’ on
>>>>> 
>>>>> https://github.com/apache/incubator-airflow/blob/master/airflow/config_templates/default_airflow.cfg#L613
>>>>> solved the issue for me, and digging a bit deeper it seems Airflow now
>>>>> requires setting LC_ALL=en_US.UTF-8 in the environment or similar to force
>>>>> Python to read the file as utf-8 and not ascii. (I think this was changed
>>>>> in to default to utf-8 in Python 3.7).
>>>>>>>> 
>>>>>>>> I see 3 solutions for this
>>>>>>>> 1. Document that Airflow need to run with LC_ALL=en_US.UTF-8 or
>>>>> similar.
>>>>>>>> 2. Change the default config file to not contain non-ascii characters.
>>>>>>>> 3. Always read the file as unicode regardless of the LC_ALL
>>>>> environment, by the encoding='utf-8’ parameter to open().
>>>>>>>> 
>>>>>>>> I think 3 is the best solution, and I can prepare a PR for that if
>>>>> necessary .
>>>>>>>> 
>>>>>>>> I guess this counts as -1 (non-binding)
>>>>>>>> 
>>>>>>>> All the best
>>>>>>>> 
>>>>>>>> Carl Johan
>>>>>>>> 
>>>>>>>> 
>>>>>>>> 
>>>>>>>>> On 8 July 2018 at 22:02:33, Bolke de Bruin (bdbruin@xxxxxxxxx) wrote:
>>>>>>>>> 
>>>>>>>>> Hey all,
>>>>>>>>> 
>>>>>>>>> I have cut Airflow 1.10.0 RC1. This email is calling a vote on the
>>>>> release,
>>>>>>>>> which will last for 72 hours. Consider this my (binding) +1.
>>>>>>>>> 
>>>>>>>>> Airflow 1.10.0 RC 1 is available at:
>>>>>>>>> 
>>>>>>>>> https://dist.apache.org/repos/dist/dev/incubator/airflow/1.10.0rc1/ <
>>>>> https://dist.apache.org/repos/dist/dev/incubator/airflow/1.10.0rc1/>
>>>>>>>>> 
>>>>>>>>> apache-airflow-1.10.0rc1+incubating-source.tar.gz is a source release
>>>>> that
>>>>>>>>> comes with INSTALL instructions.
>>>>>>>>> apache-airflow-1.10.0rc1+incubating-bin.tar.gz is the binary Python
>>>>> "sdist"
>>>>>>>>> release.
>>>>>>>>> 
>>>>>>>>> Public keys are available at:
>>>>>>>>> 
>>>>>>>>> https://dist.apache.org/repos/dist/release/incubator/airflow/ <
>>>>> https://dist.apache.org/repos/dist/release/incubator/airflow/>
>>>>>>>>> 
>>>>>>>>> The amount of JIRAs fixed is over 700. Please have a look at the
>>>>> changelog.
>>>>>>>>> 
>>>>>>>>> Please note that the version number excludes the `rcX` string as well
>>>>>>>>> as the "+incubating" string, so it's now simply 1.10.0. This will
>>>>> allow us
>>>>>>>>> to rename the artifact without modifying the artifact checksums when
>>>>> we
>>>>>>>>> actually release.
>>>>>>>>> 
>>>>>>>>> 
>>>>>>>>> Cheers,
>>>>>>>>> Bolke
>>>>> 
>