Python: Multiprocessing under Linux and Windows

Ran into an interesting issue with the Python multiprocessing library under Windows, where the code that I had written would hang. It was the equivalent of this:

from multiprocessing import Pool

...

with Pool() as pool:
    results = pool.starmap(foo, arglist)

This was in part because of how Windows was forking / pulling back the results.

The work-around I used was the following:

if os.name == 'nt':
    from multiprocessing.pool import ThreadPool as Pool
else:
    from multiprocessing import Pool

...

with Pool() as pool:
    results = pool.starmap(foo, arglist)

Which did the trick. This is because ThreadPool uses Python's native threading (which is Global Interpreter Locked, yes) to handle multi-processing instead of the OS-level forking / spawning.

Posting this here in case I run into this again.


links

social