Pickle performance bottlenecks when using multiprocessing

Some while ago I have written a parameter scan (regularization in logistic regression to be specific) that was taking a bit to long to execute. Since the machine on which it was executed was essentially other-user-free (and had some 20 cores laying unused 🙂 ) I decided to go multiprocessing. Usually I pick multiprocessing.Pool.map on such ocassions where boilerplate… Continue reading

Being defensive with pickle in evolving environment

Pickle is an in-house python object persistence solution. Although very useful, care must be taken when using it with class definitions that may change, i.e. are under active development. Consider the following example Both printouts will show you var1 and var2 instance variables and no var3, despite the fact that class logic changed in meantime.… Continue reading