Iterables and Iterators

Two different things

numbers = [1, 2, 3]

print(hasattr(numbers, "__iter__"))      # True  - it is iterable
print(hasattr(numbers, "__next__"))      # False - it is not an iterator

iterator = iter(numbers)
print(hasattr(iterator, "__next__"))     # True

print(next(iterator))     # 1
print(next(iterator))     # 2
print(next(iterator))     # 3
# print(next(iterator))   # StopIteration

What a for loop really does

for value in [10, 20, 30]:
    print(value)

# Exactly equivalent
iterator = iter([10, 20, 30])
while True:
    try:
        value = next(iterator)
    except StopIteration:
        break
    print(value)

Every for loop calls iter() once, then next() repeatedly, and stops when StopIteration is raised. Nothing else is involved. Understanding this explains a whole family of otherwise puzzling behaviours.

An iterator is used up

numbers = [1, 2, 3]

print(list(numbers))     # [1, 2, 3]
print(list(numbers))     # [1, 2, 3] - a list can be walked again

iterator = iter(numbers)
print(list(iterator))    # [1, 2, 3]
print(list(iterator))    # []  <- already exhausted

squares = (n * n for n in range(5))     # a generator, so an iterator

print(sum(squares))      # 30
print(sum(squares))      # 0  - nothing left
print(list(squares))     # []

This is the most common surprise with iterators. If you need to walk the values twice, either keep the underlying iterable, or materialise the iterator into a list once with values = list(iterator).

with open("notes.txt", encoding="utf-8") as handle:
    print(sum(1 for _ in handle))        # counts the lines
    print(sum(1 for _ in handle))        # 0 - the file is at the end
    handle.seek(0)
    print(sum(1 for _ in handle))        # counts again

next with a default

iterator = iter([1, 2])

print(next(iterator, "done"))     # 1
print(next(iterator, "done"))     # 2
print(next(iterator, "done"))     # done - no exception


people = [{"name": "Meera"}, {"name": "Arun"}]
match = next((p for p in people if p["name"] == "Zara"), None)
print(match)                       # None

next(generator, default) is the idiomatic "find the first match, or nothing". It stops at the first hit instead of filtering the whole collection.

Writing an iterator class

class Countdown:
    """Counts down from a starting number to one."""

    def __init__(self, start):
        self.current = start

    def __iter__(self):
        return self                 # the object is its own iterator

    def __next__(self):
        if self.current <= 0:
            raise StopIteration
        value = self.current
        self.current -= 1
        return value


for n in Countdown(4):
    print(n, end=" ")               # 4 3 2 1
print()

c = Countdown(3)
print(list(c))        # [3, 2, 1]
print(list(c))        # []  - self.current is now 0, so it is exhausted

Separating the iterable from the iterator

class CountdownIterator:
    def __init__(self, start):
        self.current = start

    def __iter__(self):
        return self

    def __next__(self):
        if self.current <= 0:
            raise StopIteration
        value = self.current
        self.current -= 1
        return value


class Countdown:
    """Reusable: each iteration gets a fresh iterator."""

    def __init__(self, start):
        self.start = start

    def __iter__(self):
        return CountdownIterator(self.start)


c = Countdown(3)
print(list(c))        # [3, 2, 1]
print(list(c))        # [3, 2, 1] - reusable

print(list(zip(c, c)))    # two independent walks over the same object

This is how list works: the list is the iterable, and iter(list) returns a separate list_iterator each time. That is why you can nest loops over the same list.

A practical iterator

class Paginated:
    """Walks a sequence in fixed size pages."""

    def __init__(self, items, size):
        if size < 1:
            raise ValueError("page size must be at least 1")
        self.items = items
        self.size = size

    def __iter__(self):
        for start in range(0, len(self.items), self.size):
            yield self.items[start:start + self.size]

    def __len__(self):
        return -(-len(self.items) // self.size)      # ceiling division


pages = Paginated(list(range(1, 11)), 3)

print(len(pages), "pages")
for number, page in enumerate(pages, start=1):
    print(f"page {number}: {page}")

Note that __iter__ uses yield. A method containing yield returns a generator, which is already an iterator - so there is no __next__ to write. The next note covers generators fully, and in practice they replace almost every hand written iterator class.

iter() with two arguments

import random

# Call the function repeatedly until it returns the sentinel value
rolls = iter(lambda: random.randint(1, 6), 6)
print(list(rolls))              # every roll before the first 6

with open("data.bin", "rb") as handle:
    for chunk in iter(lambda: handle.read(4096), b""):
        print(len(chunk))       # read until an empty result

The two argument form calls the first argument repeatedly and stops when it returns the second. It replaces a while True loop with a break.

Infinite iterators

class Cycle:
    """Repeats a sequence forever."""

    def __init__(self, items):
        self.items = list(items)
        self.index = 0

    def __iter__(self):
        return self

    def __next__(self):
        if not self.items:
            raise StopIteration
        value = self.items[self.index]
        self.index = (self.index + 1) % len(self.items)
        return value


import itertools

colours = Cycle(["red", "green", "blue"])
print(list(itertools.islice(colours, 7)))
# [' red', 'green', 'blue', 'red', 'green', 'blue', 'red']

An iterator never has to end. That is impossible for a list and perfectly normal for an iterator, and it is why lazy evaluation matters.

Checking whether something is iterable

from collections.abc import Iterable, Iterator

print(isinstance([1, 2], Iterable))          # True
print(isinstance([1, 2], Iterator))          # False
print(isinstance(iter([1, 2]), Iterator))    # True
print(isinstance("abc", Iterable))           # True
print(isinstance(42, Iterable))              # False


def flatten(value):
    """Flatten nested iterables, leaving strings alone."""
    if isinstance(value, Iterable) and not isinstance(value, (str, bytes)):
        for item in value:
            yield from flatten(item)
    else:
        yield value


print(list(flatten([1, [2, [3, "ab"]], 4])))     # [1, 2, 3, 'ab', 4]

The exclusion of str matters: a string is iterable, so without that check it would be split into characters and then recursed on forever.

Common mistakes

• Iterating an iterator twice and getting nothing the second time.
• Calling len() on an iterator; there is no length to report.
• Indexing an iterator; it does not support subscripting.
• Forgetting to raise StopIteration, producing an infinite loop.
• Returning self from __iter__ when the object should be reusable.
• Treating a string as a container of words when iterating; it yields characters.

Best practices

• Write __iter__ with yield rather than a separate iterator class.
• Return a fresh iterator from __iter__ when the object should be walkable more than once.
• Use next(it, default) instead of catching StopIteration.
• Materialise into a list when you genuinely need several passes.
• Use collections.abc.Iterable for checks rather than testing for __iter__ by hand.

Practice

1. Write the equivalent while loop for a for loop, using iter and next explicitly.
2. Show that a generator is exhausted after one pass, and give two ways to work around it.
3. Write an iterator class producing the first n even numbers.
4. Write a reusable iterable and prove it can be walked twice, including in nested loops.
5. Use the two argument form of iter to read a file in fixed size chunks.

Conclusion

An iterable hands out iterators; an iterator hands out values and remembers its place. A for loop is iter plus repeated next until StopIteration. Everything lazy in Python - generators, files, range, itertools - is built on that protocol.