How do generators and yield reduce memory usage?

Short answer

A generator produces values on demand, one at a time, instead of building a full collection, so memory stays roughly constant no matter how large the data is.

Explanation

When execution reaches yield, the function pauses and keeps its local state, then resumes from the same point when the next item is requested. This is ideal for huge files or staged data pipelines. The trade-offs: no random access by index, no way to know the length up front, and errors surface during consumption rather than when the generator is created. yield from delegates generation to another generator.

Code example

def read_errors(path):
    with open(path, encoding="utf-8") as f:
        for line in f:              # Line by line
            if "ERROR" in line:
                yield line.rstrip()

for entry in read_errors("app.log"):
    print(entry)

Common mistakes

• Calling list() on a generator over a huge file, which cancels the benefit and consumes memory.
• Expecting the error to appear when the generator is created when it actually appears on the first next().

Follow-up questions

• When do generators improve memory use and when do they not improve time?
• What is the difference between an iterable and an iterator?

Sources

• Python documentation — Generators (Functional Programming HOWTO)