What is the concurrent.futures module for?

Short answer

It offers a unified interface to submit work to a ThreadPoolExecutor or ProcessPoolExecutor through submit and map, and to follow the results as Futures with as_completed.

Explanation

Use the executor inside with to guarantee shutdown and waiting for the tasks. An exception inside a task does not surface until you call future.result(), so read every result or check exception(). map returns results in input order, whereas as_completed returns them in completion order. ProcessPoolExecutor requires the function and arguments to be picklable, so do not pass lambdas or closures.

Code example

from concurrent.futures import ThreadPoolExecutor, as_completed

with ThreadPoolExecutor(max_workers=8) as pool:
    futures = {pool.submit(fetch, url): url for url in urls}
    for future in as_completed(futures):
        url = futures[future]
        try:
            print(url, len(future.result()))
        except Exception as err:
            print(f"{url} failed: {err}")

Common mistakes

• Submitting tasks without reading result(), so exceptions are silently lost.
• Passing a lambda to a ProcessPoolExecutor.

Follow-up questions

• How do you run a blocking function inside asyncio?

Sources

• Python documentation — concurrent.futures