If you are reading this post, most likely you’re still working with python 2.7 series for some reason. In such case, one of the small annoyances you risk running into is nearly complete lack of information what’s happening when the multiprocessing module is used and an uncaught exception is thrown in the child process. In such situation, the standard traceback you get won’t be useful since it will be limited to sources of the multiprocessing module itself (and not of your code or other modules used).

One way to fix that is to print the traceback yourself:

from multiprocessing import Pool
import traceback

def exception_test(x):
    try:
        raise Exception()
    except Exception, ex:
        print traceback.format_exc()
        raise ex

if __name__ == "__main__":
    pool = Pool(processes=2)
    pool.map(exception_test, range(4))

This way you get a complete trace, including the line numbers.

Notes:

  • This is a TLDR version of this post on handling exceptions when multiprocessing module is used. Read if you want to get a decorator version of the above or use the joblib package as an alternative solution.
  • This is one of the many (small) reasons to move to python 3. In the latest 3.6 python version (earlier versions from 3.x series are also likely to have this), uncaught exception in child process gives you also a proper trace. See this post for some more reasons to move to python 3.6.

Flask and bower - download jQuery, others automatically Look ma, I made a browser game!

Leave a Reply

Your email address will not be published.