rejection reports..

same ol' same ol', settup up another report, where I first need to do some data munging..

I'm currently storing rejections inside a "history" attribute of each record. I decided to do this for atomicity purposes — that is, when an item is rejected, I want to reset it's state so that another person can redo it, while simultaneously making a note that the rejection happened.

But having a separate "history" table would be easier for running queries.. I feel the laughing of my SQL friends.. "everything should be a separate table"..

hm.. I would consider thinking about the history as "roughly right", but since there are payments involved, I think it should be exact. Of course, as it stands, payments are only for accepted items anyway.. though I do want people to have an exact accounting of why they weren't payed for certain things..

I feel like this is yet another argument in favor of paying by the hour — things are just so much easier to keep track of that way.

hm.. I'm currently building a "rejects" table based on the history in each record. This is done every day when payments are processed. I think I'll stick with this for now.. though I think I'll flatten the table, because I currently have one entry for each user with an array of their rejects..

oh.. I see why I have it that way.. it's a silly reason which can replaced by deriving a unique key for each rejection record..

hm.. and I'm currently building up some huge list of rejections into memory, which might fail at some point.. this seems like a job for server-side javascript (and by server-side, I mean running on the mongoDB instance itself using eval).. of course, I want to use the "nolock" option, but that option isn't available in db.eval, which is all mongojs provides.. does mongojs really not provide db.runCommand?..

AHA! It doesn't have db.runCommand, but does have db.eval, which I didn't realize existed. great.

db.eval('function (x) {return x;}', [3], {nolock:true}, function (err, x) { console.log(x) })

Ok, so let's rewrite the payment processor do run an eval that updates the rejects table the way I want..

oh, it's so beautiful to run javascript on the mongodb server:


db.eval("" + function () {
    var recs = db.records.find()
    for (var i = 0; i < recs.length(); i++) {
        var rec = recs[i]
        if (rec.history) {
            for (var ii = 0; ii < rec.history.length; ii++) {
                var h = rec.history[ii]
                if (h.reviewedBy && !h.reviewAccept) {
                    h._id = h.answeredBy + " " + h.answeredAt
                    h.batch = rec.batch
                    h.question = rec.question
                    h.category = rec.category
                    db.rejects.insert(h)
                }
            }
        }
    }
}, { nolock : true }, p.set)

..too bad I can't use u.js, but at least it cuts out all kinds of transmission times.. and I can write the code with syntax highlighting, just prepending "" + to convert it to a string..

javascript/mongodb +1

ok, where was I.. oh, right, I want to drop the rejects table and rebuild it.. before I drop it, I'll make a backup.. ok.. backup downloaded.. rejects table dropped.. ok, running payments script.. man this takes a while, I hope it finishes before midnight when it is scheduled to run again.. I suppose if it gets close, I can cancel the cron job (I'm not confident that it would work fine to run this script concurrently with itself..).. oh, speak of the devil, it finished.. good, the rejects table seems to have been created ok..

so now that we have a nice rejects table, let's create the api call download it.. I'll just model it after the csv query..

done.. pushing.. people informed.