[Python-Dev] Tests failing on Windows with TESTFN
On 28/07/2018 17:27, Jeremy Kloth wrote:
> On Sat, Jul 28, 2018 at 8:41 AM Tim Golden <mail at timgolden.me.uk> wrote:
>> 1) Why are these errors occurring? ie are we dodging a root cause issue
> The root cause is how Windows handles file deletions. When a file is
> removed, it is not immediately removed from the directory, instead, it
> is simply marked for deletion. Only once all open handles to the file
> are closed, is it removed from the directory. The most common cause
> of additional open handles to a file is a antivirus scanner.
Thanks, Jeremy. I'm already aware of that. (If you look back at
https://bugs.python.org/issue7443 you'll find me explaining the same to
MvL some years ago). [In case the tone isn't clear, there's absolutely
no sarcasm implied here. I just wanted to make it clear that I'm at
least partly au fait with the situation :)].
Although things have moved on since that discussion and
test.support.unlink has grown some extra legs, all it's done really is
to push the bump along the carpet for a bit. I've got a newly-installed
Win10 machine with the typical MS Antivirus & TortoiseGitCache vying for
locks with every other process. I've just done a full test run:
python -mtest -j0 -v > test.log
and I've got a mixture of Permission (winerror 13) & Access errors
(winerror 5). The former are usually attempting to open a TESTFN file;
the latter are attempting to unlink one. Most of the Error 5 are
os.unlink, but at least one is from test.support.unlink.
I understand the unable-to-recreate (virus checkers with share-delete
handles) but I'm not so clear on what's blocking the unlink. IOW I think
we have more than one issue.
Here's the thing. The TESTFN approach creates a directory per process
test_python_<pid> and some variant of @test_<pid>_tmp inside that
directory. But the same filename in the same directory is used for every
test in that process. Now, leaving aside the particular mechanism by
which Windows processes might be holding locks which prevent removal or
re-addition, that already seems like an odd choice.
I think that was my starting point: rather than develop increasingly
involved and still somewhat brittle mechanisms, why not do what you'd
naturally do with a new test and use tempfile? I was expecting someone
to come forward to highlight some particular reason why the TESTFN
approach is superior, but apart from a reference to the possibly cost of
creating a new temporary file per test, no-one really has.
> If you are not seeing the RuntimeWarnings, then something else is amiss.
I think I've seen one RuntimeWarning in the many, many times I've been
running through tests today.