Advanced Python · Lesson 34 of 39

Iterators and Generators

5 min read Updated Nov 7, 2026 Saurabh Samir

On this page

What a for loop really does
Generator functions: yield
Why generators save memory
Infinite sequences
Generator pipelines
yield from
Practice
Summary

Every for loop you have written relies on a simple protocol called iteration. Understanding it unlocks generators: functions that produce values one at a time, on demand. Generators let you process a 10 GB log file or an endless stream of sensor data using almost no memory.

In this lesson

Iterables vs iterators, iter() and next()
Writing generator functions with yield
Generator expressions and lazy pipelines
Why generators save memory

What a for loop really does

An iterable is anything you can loop over (list, string, dict, file…). Calling iter() on it gives an iterator – an object that hands out one value per next() call and raises StopIteration when it runs out.

colors = ["red", "green"]
it = iter(colors)
print(next(it))
print(next(it))
try:
    next(it)
except StopIteration:
    print("StopIteration – nothing left")

Output

red
green
StopIteration – nothing left

for color in colors: … behind the scenes

it = iter(colors)get an iterator
color = next(it)ask for the next value
Run the loop bodythen go back to step 2
StopIterationthe loop ends quietly

Iterators are single-use: once exhausted, they stay empty.

squares = map(lambda n: n * n, [1, 2, 3])   # map returns an iterator
print(list(squares))
print(list(squares))     # already used up

Output

[1, 4, 9]
[]

Generator functions: yield

Writing an iterator class by hand is verbose. A generator function does the same job: any function that contains yield returns a generator. Each yield hands out a value and pauses the function, keeping all its local variables, until the next value is requested.

def countdown(n):
    print("starting")
    while n > 0:
        yield n
        n -= 1
    print("done")

gen = countdown(3)
print(type(gen).__name__)
print(next(gen))
print(next(gen))
for value in gen:        # continues where it paused
    print(value)

Output

generator
starting
3
2
1
done

Notice that "starting" appeared only when the first value was requested – calling countdown(3) itself runs nothing.

return vs yield

returnsends one value and the function is finished
yieldsends one value and the function pauses, ready to resume

Why generators save memory

import sys

as_list = [n * n for n in range(1_000_000)]
as_gen = (n * n for n in range(1_000_000))      # generator expression

print(sys.getsizeof(as_list) > 1_000_000)   # over a megabyte
print(sys.getsizeof(as_gen) < 500)
print(sum(as_gen))

Output

True
True
333332833333500000

The list stores a million numbers (megabytes of memory). The generator stores only its current state (a couple of hundred bytes) and computes each value when sum() asks for it.

Infinite sequences

Because values are produced on demand, a generator can be endless – you just stop asking:

from itertools import islice

def fibonacci():
    a, b = 0, 1
    while True:
        yield a
        a, b = b, a + b

print(list(islice(fibonacci(), 12)))

def order_ids(prefix="ORD"):
    n = 1000
    while True:
        n += 1
        yield f"{prefix}-{n}"

ids = order_ids()
print(next(ids), next(ids), next(ids))

Output

[0, 1, 1, 2, 3, 5, 8, 13, 21, 34, 55, 89]
ORD-1001 ORD-1002 ORD-1003

Generator pipelines

Chain small generators like stages on an assembly line. Each line of data flows through every stage before the next line is even read – so this works the same for a 10-line file and a 10-million-line file:

log_lines = [
    "200 GET /home 12ms",
    "500 POST /pay 340ms",
    "200 GET /blog 25ms",
    "404 GET /old 3ms",
    "500 GET /api 512ms",
]

def parse(lines):
    for line in lines:
        status, method, path, ms = line.split()
        yield {"status": int(status), "path": path, "ms": int(ms[:-2])}

def errors(records):
    return (r for r in records if r["status"] >= 500)

slow_errors = [r["path"] for r in errors(parse(log_lines)) if r["ms"] > 300]
print(slow_errors)

Output

['/pay', '/api']

In a real script, log_lines would simply be an open file – files are iterators that read one line at a time.

yield from

yield from hands out every value of another iterable – handy for recursive generators:

def walk(tree):
    for key, children in tree.items():
        yield key
        yield from walk(children)

folders = {"src": {"app": {}, "tests": {}}, "docs": {}}
print(list(walk(folders)))

Output

['src', 'app', 'tests', 'docs']

Common mistake: reusing a generator

A generator can be looped over only once. If you need the values twice, either call the generator function again or convert it with list() (if the data fits in memory).

Practice

Try it yourself

Write a generator evens(limit) that yields even numbers up to limit.

Show solution

def evens(limit):
    for n in range(0, limit + 1, 2):
        yield n

print(list(evens(10)))

Output

[0, 2, 4, 6, 8, 10]

Write chunks(items, size) that yields lists of size items – useful for sending data in batches.

Show solution

def chunks(items, size):
    for start in range(0, len(items), size):
        yield items[start:start + size]

print(list(chunks(list(range(1, 8)), 3)))

Output

[[1, 2, 3], [4, 5, 6], [7]]

Quick check

What turns a normal function into a generator function?

Using yield anywhere in its body.

What does next() raise when an iterator is exhausted?

StopIteration.

What is the difference between [x for x in data] and (x for x in data)?

The first builds a full list in memory; the second is a lazy generator that produces values one at a time.

Summary

for loops call iter() once and next() until StopIteration.
yield pauses a function and resumes it on the next request.
Generators are lazy and single-use; they handle huge or infinite data in tiny memory.

Finished this lesson?

Mark it complete to track your progress, then carry on with the next lesson.

Next lesson