work notes

I'm creating another system similar to the one before.. let's start by forking it.. hm.. I can't seem to fork my own repository. I can't believe github doesn't have that feature.. oh well, I'll do it the old-fashioned way.. create nar-nar repo.. copy stuff over from february-fire..

now it looks like we can just follow along in the readme file to it up on heroku..

I guess I can't "heroku ps:scale web=2" until I push something.

let's run this thing locally.. ok, it shows the february-fire stuff.

I think a good place to start with this new project is to load the database with some examples of the sort of stuff that will be there.


{ "username" : "fakeuser", "name" : "Fake User", "img" : null }
{ "username" : "testuser", "name" : "Test User", "img" : null }
{ "username" : "bbernard", "name" : "Barney Bernard", "img" : null }

great.. ok, now for this task, people are going to be grabbing batches of tasks, rather than just one.. so let's write the grabBatch rpc..

first let's add a "availableToGrabAt : 0" to each fake user.. done..

on to writing grabBatch..

oh, I first need to add a "random" field to each record, so I can grab a random batch.. wait.. I don't need them to be random. Randomness was good when allowing people to preview items and decide what to grab, but in this case, they're just gonna get stuck with whatever batch we give them (and they can grab a new batch if they don't like it).

so still writing grabBatch.. let's see.. we want a query like this

find all records where the availableToGrabAt time is less than now, sorted by _id descending (which will show the most recently added tasks first — since I've convinced now that for moderation tasks, it's best to do most-recent-first, since those are the highest priority items, since stuff that hasn't been looked at yet has some evidence in favor of it not being so bad.. since it hasn't triggered any alerts elsewhere in the system).

find all records where the availableToGrabAt time is less than now, sorted by _id descending,
goes to
db.records.find({ availableToGrabAt : { $lt : _.time() }).sort({_id : -1}).limit(16)

and actually we want to update, marking all these things as grabbed..

..hm.. not sure how to write the update.. because of the limit 16.. how do I do that? Does the update query support a limit?.. look like no. A person there suggested that doing a find and then an update is faster than a bunch of findAndModify's.. though, it seems like I would still need multiple updates..

oh well, I'm going to do multiple findAndModify's. It's easier to make sure the code is correct, and I'll worry about it only if it's too slow..

so, in the best of conditions, it's taking about 30 milliseconds to do 16 findAndModify's.. that strikes me as pretty slow.. I think I'm going to try the find and update.. ok, that's about 4 milliseconds. Worth it I suppose.. of course it has a higher danger of returning no items, even when there are items available, since it could find some free items, but then those items could be taken before it can update them.. I suppose if it returns nothing, we can run the original findAndModify code.. of course, then that code will be pretty dead, e.g., not used very often.. another strategy is to try again to find some items.. and keep trying so long as the find returns stuff, but the update fails.. or maybe try doing that 10 times.. ok, done..

now I should add a couple indexes.. done

ok, now the rpc is working. next I want to display the items.. but before I do that, I want to rethink the interface.. I had problems before with updating to new versions of the interface, and giving people messages about what's going on.. let's think for a moment about an elegant solution for that..