I've shifted my batch processing routine.
I had been putting items to process in a json file, reading them all into memory, processing them all, and writing the results as json into another file.
But too often — like just now — I'd wait a long while until it was almost done, and then there'd be an error, meaning it wouldn't write anything to the file, losing all the work I had waited for.
So now I find myself putting everything into a temporary mongodb collection, querying the collection for a non-processed item, processing that item, and writing it back into the collection.
mongodb is so great.