Iterators, Generators and Comprehensions

Generators are how Python processes data that does not fit in memory — log files, database results, API pages — one item at a time. Interviewers ask about them to test your understanding of laziness, memory and the iterator protocol behind every for loop.

1. How does a for loop work under the hood?

It calls iter(obj) to get an iterator, then repeatedly calls next() until StopIteration is raised. Any object implementing __iter__ (returning an iterator) is iterable; an iterator implements __next__ and remembers its position.

nums = [10, 20]
it = iter(nums)
print(next(it))   # 10
print(next(it))   # 20
# next(it) -> StopIteration: the for loop catches this and stops

2. Write a custom iterator class.

Implement __iter__ returning the iterator and __next__ returning the next value or raising StopIteration. In practice a generator function is shorter — show both if asked.

class Countdown:
    def __init__(self, start):
        self.current = start
    def __iter__(self):
        return self
    def __next__(self):
        if self.current <= 0:
            raise StopIteration
        self.current -= 1
        return self.current + 1

print(list(Countdown(3)))   # [3, 2, 1]

def countdown(start):        # same thing as a generator
    while start > 0:
        yield start
        start -= 1

3. What makes generators memory-efficient?

A generator produces one value at a time and keeps only its local state between yields, so memory stays constant no matter how many values it yields. A list comprehension builds every element up front.

import sys

squares_list = [n * n for n in range(1_000_000)]
squares_gen = (n * n for n in range(1_000_000))
print(sys.getsizeof(squares_list))   # about 8 MB
print(sys.getsizeof(squares_gen))    # about 200 bytes
print(sum(squares_gen))              # computed lazily

4. Build a lazy data pipeline with generators.

Chain small generators, each doing one job. Nothing runs until the final consumer pulls values, and memory stays flat even for huge files — the Unix pipe idea in Python.

def read_lines(path):
    with open(path, encoding="utf-8") as f:
        yield from f

def errors(lines):
    return (line for line in lines if " ERROR " in line)

def services(lines):
    return (line.split()[2] for line in lines)

from collections import Counter
top = Counter(services(errors(read_lines("app.log")))).most_common(3)

5. What does yield from do?

It delegates to another iterable, yielding each of its values — shorter than a nested for loop, and it also passes values and exceptions through properly. Handy for recursive generators such as walking a tree.

def flatten(items):
    for item in items:
        if isinstance(item, list):
            yield from flatten(item)
        else:
            yield item

print(list(flatten([1, [2, [3, 4]], 5])))   # [1, 2, 3, 4, 5]

6. Which itertools functions are most useful in interviews?

chain (join iterables), islice (slice any iterator), groupby (group consecutive items), combinations and permutations, product (nested loops), accumulate (running totals), count and cycle. Python 3.12 added batched for chunking.

from itertools import combinations, accumulate, islice, batched

print(list(combinations("ABC", 2)))      # [('A','B'), ('A','C'), ('B','C')]
print(list(accumulate([5, 10, 15])))      # [5, 15, 30]
print(list(islice(range(100), 3)))        # [0, 1, 2]
print(list(batched(range(7), 3)))         # [(0,1,2), (3,4,5), (6,)]

7. When should you use a comprehension and when a loop?

Use a comprehension to build a collection from another with a simple transform and filter — it is concise and fast. Use a regular loop when the logic needs several statements, side effects (printing, saving) or exception handling. Nested comprehensions beyond two levels hurt readability.

# good
active_emails = [u.email for u in users if u.active]

# too clever — write a loop instead
result = [x * y for x in range(5) if x % 2 for y in range(x) if y > 1]

8. Why can a generator be consumed only once?

A generator is an iterator: once it has yielded all values it is exhausted, and iterating again yields nothing. Recreate it, store results in a list if you need several passes, or use itertools.tee to split it.

gen = (x for x in range(3))
print(list(gen))   # [0, 1, 2]
print(list(gen))   # [] — already exhausted