OSDir

[Date Prev][Date Next][Thread Prev][Thread Next][Date Index][Thread Index]

[no subject]


Legitimate Use Cases for Raising StopIteration in a Generator
------------------------------------------------------------

In a producer/consumer generator chain, the input generator signals
it is done by raising StopIteration and the output generator signals
that it is done by raising StopIteration (this isn't in dispute).

That use case is currently implemented something like this:

    def middleware_generator(source_generator):
        it = source_generator()
        input_value = next(it)
        output_value = do_something_interesting(input_value)
        yield output_value

Termination of the input stream will then terminate middleware stream.
You can see several real world examples of this code pattern in
Fredrik Lundh's pure python verion of ElementTree
(prepare_descendant, prepare_predicate, and iterfind).

Under the proposal, the new logic would be to:
1) catch input stream StopIteration
2) return from the generator
3) which in turn raises StopIteration yet again.

This doesn't make the code better in any way.  The new code
is wordy, slow, and unnecessarily convoluted:

    def middleware_generator(source_generator):
        it = source_generator()
        try:
            input_value = next(it)
        except StopIteration:
            return             # This causes StopIteration to be reraised
        output_value = do_something_interesting(input_value)
        yield output_value

I don't look forward to teaching people to have to write code like this
and to have to remember yet another special case rule for Python.


Is next() Surprising?
---------------------

The PEP author uses the word "surprise" many times in describing the
motivation for the PEP.  In the context of comparing generator expressions
to list comprehenions, I can see where someone might be surprised that
though similar in appearance, their implementations are quite different
and that some of those differences might not expected.

However, I believe this is where the "surprise" ends.

The behavior of next(it) is to return a value or raise StopIteration.
That is fundamental to what is does (what else could it do?).

This is as basic as list indexing returning a value or raising an IndexError,
or as dict lookups returning a value or raising a KeyError.

If someone in this thread says that they were suprised that next() could
raise StopIteration, I don't buy it.

Being able to consume a value from an iterator stream is a fundamental
skill and not hard to learn (when I teach iterators and generators, the
operation of next() has never been a stumbling block).

The Python iterator protocol isn't rocket science.  Like almost every
other language, it is based on the Iterator Design Pattern found in
the Gang of Four Design patterns book.

For the discussion of the PEP to go forward, it is necessary to stop
"acting surprised" about everything related to iterators and
generators.  If there is any surprise, it is confined to uncommon cases
that make visible the differences between generator expressions and
list comprehensions (IIRC, that is how this thread started).

Also, it pays to remember that we have 13 years worth of real world
experience with generators and iterators in Python.  By and large,
it has been one of our biggest success stories.   Please keep that
in mind when experiencing the temptation to change it.

TL;DR

Sorry for the long post, but I've been thinking about this deeply for
a couple days and wanted to get all the thoughts out to others for
consideration,


Raymond
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://mail.python.org/pipermail/python-dev/attachments/20141122/7c3d87f6/attachment.html>