When do generators improve memory use and when do they not improve time?

Short answer

Generators lower peak memory use because they produce items on demand, but they do not necessarily speed up processing, and may be slower if you need to pass over the data several times.

Explanation

A generator is ideal for streaming processing: reading a huge file, filtering it, and aggregating it without loading it entirely. But if you will iterate over the results more than once, or need len, indexed access, or sorting, building a list once is faster and clearer. Each step in a generator has a small overhead for resuming the function. Measure memory and time together before judging.

Code example

import sys

squares_list = [n * n for n in range(1_000_000)]
squares_gen = (n * n for n in range(1_000_000))

print(sys.getsizeof(squares_list))  # ~8 MB For indicators only
print(sys.getsizeof(squares_gen))   # ~200 bytes

total = sum(squares_gen)  # Single consumption only

Common mistakes

• Converting a generator to a list and then iterating over it only once.
• Using a generator and then iterating over it twice, so the second pass is empty.

Follow-up questions

• How do generators and yield reduce memory usage?

Sources

• Python documentation — Generator expressions