batch processing

I've shifted my batch processing routine.

I had been putting items to process in a json file, reading them all into memory, processing them all, and writing the results as json into another file.

But too often — like just now — I'd wait a long while until it was almost done, and then there'd be an error, meaning it wouldn't write anything to the file, losing all the work I had waited for.

So now I find myself putting everything into a temporary mongodb collection, querying the collection for a non-processed item, processing that item, and writing it back into the collection.

mongodb is so great.