csv gist

here's a simple csv parser and outputter — requires underscore

this and that

I feel like I'm scrambling a bit, getting things done on various projects that people want before I leave for San Antonio, where I'll still be working, but empirically not much work gets done at conferences..

I'm about to leave for my flight.. pack.. what do I need..

- computer
- power cable
- cell phone charger
- extra key for rental car, since I'll be returning it
- pajamas
- clothes
- more clothes
- something else that I don't remember.. probably..

ready

let's see, my flight leaves at 1:40pm, bords at 1:10pm, I need 30min to get through security, so 12:40pm, 10min to get from rental car to main terminal and get ticket, so 12:30pm, and 30min to get to rental car place, so 12:00pm leave here.. it's 10:50pm now, so I have 1 hour to kill..

game of thrones?

yes

report

I'm supposed to generate a report for february-fire or rejected answers. It should be a table, with 3 "columns": "name of answerer", "the question", and "name of the editor that rejected it, and their reason"

authorName
question
answer
url
editorName
rejectionReason

clone a script file

hack, hack

done.. I hope to find a general purpose reporting solution.. maybe some way where people can download a complete dump of the database formatted as csv..

idea notes

these idea came up today.. just jotting them down:

- if you do a favor for someone, they feel obligated to reciprocate. I have probably been manipulated by this.

- "can you please do BIG-X for me? or, it would also help if you could do small-x for me.." — may have more chance of succeeding than just saying "can you please do small-x for me?"

- a friend said they feel "not themselves" when not thinking in words. I feel the opposite — I feel "not myself" when talking to people.

- two things that I think are feelings, but I didn't think of as feelings before, are a sense of "reality", e.g., interpretations of the world that only exist inside my head, but seem "real", and also the sense of "understanding".

falling asleep

I had a meeting at 1pm, but I couldn't manage to fall asleep before then, until about an hour before, at which point I did fall asleep, sleeping through my 1pm meeting, and not waking up until 7:15pm, 15 minutes after a 7pm thing I was supposed to do as well..

despite this blunder, I think things are in good shape. nar nar launched today, and seems to be running smoothly.

I'll be leaving for a conference tomorrow, so I have a 1pm meeting with an airplane — hopefully I'll fair better with that..

I feel so close to having things together.. I need to start hiring people. The whole point of these projects.. well.. one of the hopes with these projects was to try using humanscripting on them, and hopefully the urgency will settle soon so I can start to add features to them using humanscripting.

!(a <= b)

several times recently I've found myself typing something like:

!(x <= 42)

but why not

x > 42

you might ask. The reason is that x might be null, and comparisons with null are always false — null is not greater than anything, and it is not less than anything — ...

hm..

you know, I wrote that, and then I decided to do a quick check, and..

null < 0 is false
null > 0 is false
null == 0 is false
null >= 0 is true!

so.. um..

update: Kyle (see comment) is right — JavaScript is coercing null to 0 when using <, >, <= or >=, so "null < 1" is true as well

hm.. well.. I started this post thinking I was doing something clever, but it turns out I'm doing something wrong.. what I was claiming about null does appear to be true of undefined though, e.g., "undefined < 1" is false

NODE_ENV=production

to make sure express is operating as expressly as possible in production, I set NODE_ENV=production

BUT, this seems to turn errors into "Internal Server Error", rather than providing a stack trace. I would rather have users see the stack trace, so that they can tell me the stack trace when they get an error, rather than say "it says 'Internal Server Error'".

how do I tell express to do this, while still keeping in production mode?.. I guess I use app.use(express.errorHandler({ dumpExceptions: true, showStack: true })), which apparently also styles the errors.. not sure I want that, but it'll do

ok, let's make sure nar-nar still works..

Oddly enough, this error screen is an indicator of success, rather than failure..

AHA!

I had complained earlier about mongoHQ crumbling under 10 requests per second, but I discovered it was my fault (as usual).

I neglected to set auto_reconnect for mongo-connect (which is just used for sessions — I wasn't sure how to give it a reference to mongojs's connection, so it opens it's own connection)

When I tried again, setting this flag, with only 2 dynos, the server was handling 450 requests per second before crumbling.

With 4 dynos, it still seems like it dies at about 400-500 requests per second..

With 8 dynos, the same.. so I'm guessing the database it happy to handle about 400 requests per second.  Anyway, that's a lot more than 10.

I need to update this setting in february-fire and nar-nar..

nar-nar done, and it still works.. good sign..
february-fire done..

thought..

first, I can't seem to sleep. I'm supposed to get up by 1 tomorrow. I had the same problem yesterday, and ended up getting very little sleep..

I think it's a combination of several projects going on that involve servers, and also I'm supposed to leave Friday for a conference, so I need to make sure everything is in order to be gone for a week..

anyway..

my thought was that I wonder if mongoHQ was really crapping out at 10 requests per second.. I have this lingering fear that my dynos might be losing their connections with the database, and not recreating them. I fear this because there's a flag called auto_reconnect that defaults to false, and I'm not setting to true..

also, after I hammered the server with the Blitz add-on, I actually needed to reset the server before I could access the site (which I did by pushing a commit)..

so my thought is, maybe I've been doing it wrong. Maybe I'm suppose to set auto_reconnect to true, and somehow the Blitz thing is causes the connection to the database to get lost, and then it fails forever after that since it has no connection.

I'd like to test this.

I have the energy to test this.

But.. I'm suppose to sleep. But I can't sleep.. maybe I'll test this, and then go in early to work to talk to people about nar nar, and then leave early to sleep..

so, actually mongojs uses auto_reconnect by default, so that's good, but mongo-connect does not — which I'm using for sessions — so that's bad..

databases..

I was talking a bit today about my mongoDB instance in mongoHQ being slow.

I need to make sure it is slow.

A friend who is a big advocate of App Engine recommended using that, and when I complained about it not supporting node.js, he said "no, use App Engine as a database". That's an interesting idea.. probably not for the scale of stuff I'm doing right now.. but I wonder how many requests per second App Engine's Datastore can handle..

thought of bug while trying to sleep..

in nar-nar, I'm using "insert" to add users, which is usually fine, because they won't exist at first, and the first time they are added, I grab useful information about them, like their name and such..

however, nar-nar has a thing for granting permissions, which creates users before they log in, so that it has a place to store their permissions, which means that when they do log in, it won't store the new information about them..

so I need to use upsert..

incidentally, it is a bit scary to me that I catch bugs like this in my mind, because it means that my mind is probably doing something useful as it's all nervous, which means that it's probably good to remain nervous.. which.. is sad.. because I don't like being nervous..

ok, this time I tested it before checking it in.. good.. now upload to heroku..

nar nar getting ready to launch

nar nar is scheduled to launch today, provided that I make a few changes to it — I want to see people use it, so I'm going to make those changes now..

let's see, let's get my development environment setup for nar nar. I wonder if Sublime Text 2 has some notion of a development environment.. I remember some sort of side panel..

ahh sweet, the folder's view is pretty nice..

..I need some music..

(man, I didn't sleep all that long, but I feel pretty rested :)

..music.. Tegan and Sara! their new album is so great..

Sublime Text 2, I give you a +1 for being so awesome.

ok, ready.. what are the tasks I need to do..

- rework the interface to have checkboxes next to different fields that people can click saying "this field is bad", and change "reject" to "notify this person that stuff is bad, and they need to change it", and after some time, have a button saying "suspend account".

- make two checkboxes at the top for viewing both kinds of tasks ("notifications" and "suspensions")

first, let's write the function that displays the data the way I want..

ok, great. I'm happy with how it looks and works now.. let's wire-up the business logic..

(music.. the magnetic fields.. the first of their albums that I fell in love with..)

ok, hm.. I think the new version is all working now.. let's upload it to heroku.. ok, seems to be running fine..

now let's update the script that gets data from the spreadsheet in anticipation of a change.. a field called "portraiturl" might change to "portraiturl_100" or some such.. I'd like to find a field that begins with "portrait...", though, I'd also like the app to alert me if the spreadsheet changes in a way I don't expect.. I wonder what a good way for the app to alert me is.. I would have though that printing something to the console, and then having the Logging add-on alert me would do, but that add-on doesn't seem to be seeing my console output.. there must be a better logging tool..

well, it's uploaded.. seems to be running.. I'm going to bed. Well see if we end up getting actual use of this tool tomorrow :)

hm.. I didn't actually do the "portraiturl_100" change I was wanting to do.. I decided it would be too risky — I would mess it up, and it's hard to test — but maybe not so hard to test. I can just mark one of the spreadsheets and unread.. let's try..

first some music.. Garbage.. I recall liking their most recent album..

ok, what am I doing.. right, want to search for a key beginning with "portrait..."..

hm.. for testing.. I need to make my JavaScriptEval use the new u.js..

hm.. usually glittle.org updates right away when I check stuff in, but it hasn't updated yet.. (JavaScriptEval should probably be a separate repo any..)

oh, I checked into "master". I didn't think there was a master.. I don't think there is.. my local repo must be out of date.. ok, deleted it and cloned it again.. now it works. good.

ok, what was I going to do with JavaScriptEval? oh right, test some code..

ok, the core bit of code I changed seems to work in a vacuum, now let's test the whole script..

..let's check on when the script is going to be executed by the cron job.. I have 10 minutes I think..

great, it didn't break anything, that I can tell.. now bed.

can't sleep, guess I'll work

my mind is too.. um.. awake.. to sleep

let's see. I'm running 4 dynos right now for february-fire. I don't think I need that many, but I have no idea how many I do need. I want to test it. My idea for testing it is to create a new heroku account with 2 dynos, and run a 5000 person blitz on it, and see what happens..

so let's set that up.. first, let's create a new directory: february-ice..

next let's point my development environment at it..

next, let's get my readme instructions up..

ok, now let's run through them..

hm.. the git repo is still somehow linked to the old heroku app.. maybe if I delete it.. ok.. now how do I create a new .git directory? "git init" seemed to do it..

now to link it to heroku.. Stack Overflow says "git remote add heroku git@heroku.com:project.git"..

ok, good, it's uploading..

done.. it's all setup (pretty easy.. very satisfied with heroku right now)..

now let's get a dump of the database from february-fire..

actually I have one already.. let's upload it..

sweet, it's actually running in the new heroku app. I can login and everything..

..hopefully it's truly separated.. it has it's own database.. I'm not running the payments cron job..

ok.. now we want to stress test this sucker..

..for safety's sake, I just ran a test to make sure the new app is really using the new database.. so paranoid..

ok, let's add blitz..

ok, added, the expensive plan, so let's blitz and remove it asap..

what url do we want to blitz to?

so.. 2 dynos apparently does not serve 5000 blitz people..

let's try 4..

hm.. it seems to be failing at about 64 users, even with 4 dynos.. I wonder if the indexes have finished building in mongodb.. I wonder how to check that..

well, "explain" says it's using an index, at least it is now, I guess I can try again..

looks like I get about 10 hits/second before the app crumbles. Increasing the number of dynos, even to 6, doesn't seem to help. I'm guessing the database is crumbling. I suppose I could try again with a fancier database.. in any case, that's good information. I feel like my current 4 dynos is probably unnecessary, since the real bottleneck is the database.

huh.. I didn't realize I could change the name of my heroku apps. sweet!

game of thrones...

...is finally on amazon instant! (season 2 that is)

what next..

I want to sleep, but I want to make sure everything is out of the way.. I could either just assume it is not important because I'm tired, or I could go through pima and make sure.. let's try making sure..

ok, good. everything is in order. that is, everything has been proactively procrastinated as much as possible in pima.

let's sleep.

talking to myself

note to self: I want to try talking to myself

of course, I'm sortof talking to myself right now, but I'm not talking, I'm writing, and I feel like I'm a different person when I'm talking to other people.. I think I might learn something talking to myself. I think I might be talking to a different part of my brain, that is not quite connected in the way I would assume to the rest of my brain.. maybe.. worth a try.. I'll need to do it sometime when I'm alone..

february-fire bug 2

there was another bug report. This person said that they just see a blank screen. I had them take a screenshot, even though the page was blank. I was thinking I'd see a mistake in the url bar — in particular, I was suspicious that they had used https, since another person had made that mistake.. but I'm really glad I had them take a screenshot because I saw something else: it said in the lower-left corner "waiting for ...", which is very strange..

since I can log in myself, I figured the error might have to do with new users trying to log in, so I deleted one of my test accounts and tried to log in with that, and sure enough, a blank screen, and a "waiting for...". It seems to be continually redirecting for login..

crap..

ok, I've reproduced the problem locally..

sigh..

I had this:


db.collection('users').update({ _id : user._id }, { $set : _.omit(user, '_id') }, function () {
if (err) return done(err)
done(null, user)
}, { upsert: true })

I need this:


db.collection('users').update({ _id : user._id }, { $set : _.omit(user, '_id') }, { upsert: true }, function () {
if (err) return done(err)
done(null, user)
})

I obviously didn't test that upsert.. It always impresses me how often when I don't test something, it doesn't work. I mean, I generally test everything, because I figure I probably did it wrong, but sometimes things are hard to test, and I figure, it looks right.. it really looks right..

great, that works. Let's upload this fix..

ok, done. Let's see if the person can login now.

new project

so, there's more to do on the february-fire project.. more data.. possibly different tasks..

hm.. the data is in an xlsx file. I don't have Excel installed. I could get it again.. maybe I should.. but I could also get SkyDrive, and I think that will at least let me open an excel file.. let's try..

(more music: Blue October)

hm.. so many cloud shaped logos these days..

hm.. I successfully loaded it, but I can't scroll to the bottom very easily. I use two-fingered scrolling, but it's huge.. and when I try to move my mouse over to the little scroll thingy that pops up, it disappears right before I get to it, as if there's code in place to prevent me from grabbing it..

huh.. I fail. I can't get to the bottom of the spreadsheet. Well, I'm sure I could figure out a way, but I've decided it's not important enough..

ok, I've sent an e-mail asking questions.. I suppose that means I'm done for now on this..

february-fire bug

I got the following bug report:

It shows: "grabbed for next -9 minutes" . Before, the clock will show the time since you took the question. Now, whenever I take a question, it will show -8 minutes.

I think I know what's going on.. I think when I display the "grabbed for next X minutes", I'm checking "availableToAnswerAt", whereas I should sometimes be checking "availableToReviewAt".. let's see if I'm right..

nope. I did that correctly.. hm.. I guess the bug must be more subtle..

It does make sense for them to see a negative number of minutes, if they've had the task for more than 60 minutes without grabbing it again..

but they claim that when they take new questions, it shows -8 minutes.. hm.. is it possible that their computer's clock is off?

I seems to work for me.. and it doesn't seem like the system could do this differently depending on the user..

I think I'll need to ask them if this is still happening, and to send me a screenshot..

research interview

A friend proposed doing a retrospective on a project I did a couple years back. It is here: TurKit Retrospective: An Interview with Greg Little

I think the interview approach worked really well. I suck at writing. Well, I hate it in any case. But I don't mind talking about research. And if the interviewer knows what they're doing, they can withhold the stupid things I say, and make stuff make sense.

And I only spent like 10-20 minutes being interviewed. It is so much more efficient than writing! I'll have to ask her how long she spent writing it up though — it may not be a net efficiency gain.

deciding what to do next

what to do now.. I have a meeting at 2pm.. hm.. I want to check how much has been paid out for february-fire, to make sure it hasn't fallen off the deep-end..

hm.. I tried looking at the logs in Logentries, but I can't find it there.. I get the feeling that Logentries isn't keeping my console-output logs :(

ok, the payout so far seems about right. phew. I think I'm going to take the advice to NOT have payments in my system. My system can have an api for saying how much people have earned, but some other system should actually be making the payments.. I'll need to discuss that..

a feature for nar-nar

goal: I want to add a feature to nar-nar. Sometimes people will review something, make a suggestion to the relevant party, and they'll want to look at it again 48 hours later and make sure that the suggestion was followed through with. So I want a button for that. And I think I also want a "notes" field that will be shown when the task resurfaces, to give the person doing the resurfaced task a note about what they're meant to check.

so.. let's do it. First, let's get the development environment all pointing to nar-nar rather than february-fire — the projects are sisters, and it's easy to work in one while thinking I'm working in the other.. done

next.. let's make sure the project is in a good state — I may have been testing something in it before.. done. Git says that nothing is changed since the last commit.

ok.. now.. let's run it, so I can see where I'll want to make changes, and get a read-eval-print-loop going.. done

ok.. first, currently I have the "reject" button itself take the worker to a page to process the rejection. I think there should be a separate link to that page, since they'll also want to go to it in the case that they will press the soon-to-be "check in 48 hours" button.. done

now let's add the "check in 48 hours" button, just the button itself, it won't do anything yet..

hm.. it's been a while since I've had an html alignment issue:

the yellow "check in 48 hours" button is the same height as the other buttons, but somehow making it have two lines of text moves it up a slight amount..

the behavior is a bit weirder than I thought:

I'm tempted to just make the "check in 48 hours" button a single-line wide button. I'll do that for now.. probably it will stay that way forever.. done

ok, I need a "notes" field too..

hah! I fixed the button issue as a side-effect of trying to get the obo button and notes to line up; namely, I made everything "float:left":

..it's like deciding not to look for my lost keys, and then finding them.

ok.. let's decide what this will look like once accept or reject or "check in 48 hours" is pressed. I had an "undo" button there.. I think I want to keep that, but have it on the far left, and after it I can say "accepted, notes: blah blah"..

done.. ish..

I need to decide how this "check in 48 hours" will actually work. My thought is to make it use the availableToGrabAt thing, since then the query for finding available tasks can be the same. On the other hand, it will be a bit hacky, and I'll need to do things like not clear that when the task is done, since it won't really be done yet..

if I make a separate field, I'll need to add that field to my query for available tasks.. how bag would that be? It will need to be included in the index of course, but the index is cheap, I think.. hm.. I think.. hm.. let's look at the code a bit..

ok, I think I want a "status" field, which can be "ready", "accepted", "rejected", or "check in 48 hours".. actually, we'll make a non-existant status field mean "ready"..

ok, I kindof went into the programming zone of remembering lots of bits of stuff in lots of places and making a fairly large change in one go.. which means.. now I need to test this glob of code..

hm.. when I grab a task, I set the availableToGrabAt to 1 hour from now, but I shouldn't do that for a "check again in 48 hours" task.. unfortunately I update the 1-hour-from-now in an update.. I suppose I could limit the update to only items that don't have their status set to "check again in 48 hours".. perhaps that is best.. though, I probably do want to set their availableToGrabAt time if it's in the past.. hm.. what is a clever way to do this..

ok, after all that, it turns out I'm only ever updating availableToGrabAt to 1 hour from now for tasks for which availableToGrabAt is less than the current time. The reason I thought this was an issue is that when I refresh the page, all the tasks that I had marked as "check again later" were displayed as ready to check again now.. so why is that happening..

(need music: Melissa Ferrick.. one of my favorite artists..)

AHA! I do "availableToGrabAt = 1000 * 60 * 60 * 48" instead of "availableToGrabAt = _.time() + 1000 * 60 * 60 * 48"

ok, now when someone is looking at an item that was marked as "check again later", how will they know it? Perhaps a little message.. done.. hm.. testing that seems like a pain.. well, it's not critical. I think I'll just inspect it later and make sure it happens..

ok.. hm.. I'm tempted to move the admin stuff into the main page. It would be easy, and it would make it easier to explain where the admin interface is located.. but it is not critical.. let's see what is critical..

hm.. they want to see how many items remain to do..

hm.. how do I use "runCommand" in mongojs?.. ahh.. apparently it's like this: db.myCollection.find(...).count(callback)

that's nice. I like that better than the "runCommand" anyway, and apparently it's valid mongodb syntax too..

what else.. oh.. I want to include the name "nar nar" in the tool, since it's cute.. done.

ok, now let's make sure it's all running correctly on heroku..

(more music.. hm.. I'm not super happy with either Spotify's nor last.fm's recommendations.. they have my entire listening history.. alas.. ok, it's Nicki Minaj time.. hm.. it is no longer Nicki Minaj time. let's try Kings of Leon)

ok, it's up.. I think I'll clear some tasks out of the database (they aren't real anyway.. they've been done elsewhere, since this system isn't ready yet)..

ok, good.. now in a couple hours, some tasks should appear.. I think this is done enough for my meeting tomorrow..

work notes

so.. there's a feature I want to add to a nar-nar, but I don't want to do it just yet.. I think I'll eat, and maybe sleep first..

but I wanted to write a note: I think I'm close to the point where I'll want to start writing human scripts to add more features. These past couple projects have been up against a deadline, so I wasn't willing to wait.. but now.. now I think they're ready. But I'll need to figure out some easy way for people to run and test these projects on their own machines.. it might be worth creating a "debug script" that gets it up and running with some play data..

work here

huh, I didn't notice this, but I was calling these things "work notes", and then I started calling them "work here". Now, "work here" is a thing I often type in programs when I want to remember to go back to that place (it has the nice property of producing a syntax error in case I forget).

work here

continuing work on nar-nar.. I added permission abilities. Let's upload that to heroku, and give some users permissions there..

ok, uploaded.. now, let's add the users.. first I'll need to manually set my own permissions.. oops, I set the users, but didn't include myself, bumping my own permissions down to nothing.. I'll just reset them manually, again..

ok, good.. now let's test that the new e-mail processor is working on heroku..

oh yes, need to tell heroku about the new gmail account..

ok, it ran successfully, and processes all 2 pending e-mails.. now let's setup a cron job to do it every hour this time (rather than every 10 minutes)..

done. what else..

ok, I've conceptualized a change that I want to make, but for some reason I am having difficulty bringing myself to make it. Perhaps I am tired.

work notes

ok, I think the thing to do now is some nar-nar stuff. First, they've changed the spreadsheet where I'm getting data to include two additional columns. I'll need to display those in the UI, but first, I notice that the csv file has "|" as a delimiter instead of ",". What kind of csv is that? I wonder if that is part of the standard, or if I need to explicitly tell my csv library to use "|" as a delimiter. Let's find out.. seems that I need to tell it to use "|". How do I do that..

ok, my the csv parser I've been using is way too complicated. and I can't figure out how to change the delimiter  even though it claims to have the ability. I found a simpler csv parser, and modified it to my needs..

I think I'll rename cron.js to grab_csv_from_email.js..

ok, let's stop the cron job on heroku, so it doesn't interfere with my testing..

ok, now let's make the most recent csv in my e-mail "unread"..

hm.. I couldn't log into the gmail account that houses the csv files.. I was able to "recover" the account by answering some questions, like when I created the account, what my previous password was, and such.. I'm a little worried.. I can't imagine that someone tried to "hack" the account, and it doesn't seem like they did anything with it if they did.. hm.. hm..

ugg.. I don't understand, node-imap is giving me a "Error: Invalid credentials".. maybe I should try logging in with a different gmail account (but not my main account, I think.. )..

stranger and stranger.. that one gives me a "Error: Web login required: http://support.google.com/mail/bin/answer.py?answer=78754 (Failure)"

um.. now that one is also saying my password is wrong.. I find that hard to believe. I feel like there is something else at work here..

I feel like this might be a good time to enter information into my main gmail account to.. er.. make sure I don't lose it..

ok, turned on "2 factor authentication" for my main gmail account. we'll see how that goes..

now let's try accessing that other account again..

AHA! I see what was happening. I would type in a password, press enter, and then it would show me a captcha, but delete the password, so I'd enter the captcha, and then it would complain that I put in the wrong password. The trick is to type my password, press enter, retype my password, do the captcha, and then press sign in.

..so I think my password wasn't wrong after all, even for my other account. I was just being confused by the user interface removing my password without me noticing.

ugg.. I can access the other account I tried, but not the account I want to access. I can sign into the account I want to access using the web, so I know my password is correct..

(nice, my phone now uses an app-specific-password for gmail that I can revoke if I lose my phone)

ugg, how frustrating.. I think google has locked out non-web access to that account because I was checking it too frequently. I was checking it every 10 minutes, and I've read in a couple places that checking more than every 10 minutes is bad. I wonder if there's a way to convince gmail that I've changed, and I was not check it so often anymore, or if I need to create a second bogus e-mail account and forward stuff to that instead..

(let's put on some music: Matchbox Twenty.. I want to see them in concert. They're in Las Vegas at the end of March..)

ok, good, I can login from my new bogus gmail account from node.js. Hopefully I won't anger google's servers again with this account..

good, it's reading the new file format and such.. what next.. oh yes, the interface needs to display more information than it was displaying before. I did have things all in a grid, but with this extra information, that may be too compact.. though the extra information is only for some items.. so some items could be in a grid, but the grid wouldn't look super great if it was broken up by these larger items.. hm..

i could put them in a list..

yeah, I think a list may be easiest..

ok, I'm supposed to make the images 100x100, now I have a link to the image, but it's an S3 link, and it's signed, so I can't just modify it to give me the 100x100 version (which I happen to know exists).. hm.. I'm not sure how to get it.. I could scrape it from their profile page, but I'm not sure how to get their profile page either.. I have their username, but I don't know an api call to go from username to profile..

hm.. I may punt on that.. I think it will be easier for them to put it in the spreadsheet, since that process has access to the database..

(more music.. The Killers)..

ok, they want to restrict who has access to this tool, and they want to be able to update that list.. so I guess I can't do it with an environment variable.. hm..

I suppose I can add a "clearance" field to users. I was worried that some users would need to be cleared before they even log in, but I think that's fine.. I can create those user entries before they log in, and they'll just look like { _id : "someuser", clearance : 1 }, and when they log in, it will add more information to the record, like their name and such..

..now, where should I put this admin interface? I could make a special html page for it.. I'd rather just keep one html page, but maybe it would be easiest to keep the admin interface separate.. sure, I'll go try a separate page..

hm.. I think I'd like to do this authentication with middleware, but that means my middleware would need to sometimes not call "next", e.g., when I redirect people to a login page.. is that allowed? I guess I'll have to try it..

ugg.. I don't understand.. I have some middleware, and I'm not calling "next", yet it manages to do the next thing in the chain.. how?..

oh god, it makes a difference whether I put it before or after the passport stuff.. why would express care! my mental model of middleware must be wrong somehow..

express! why!

ok, it's not the passport stuff per se, it's after my first use of app.get. Are no middleware allowed after that point? they obviously are allowed, and they're even called.. they just don't care about "next" anymore, it seems..

I wish I could find documentation for that..

AHA! fucking hell, here it is:

Note that if you don't explicitly use the router, it is implicitly added by Express at the point you define a route (which is why your routes still worked even though you commented out app.use(app.router)).

Thanks josh3736

It appears that all the "routing" things, like app.get, app.post, app.all, are sortof grouped together and all appear as if they were added wherever the first of any of these calls is made.. and middleware after that point still gets called, but after the whole group of routing things..

I guess one way to think of this is that routing is a single middleware item, as opposed to a utility for creating a new middleware each time I call app.get or app.post (which is what I had thought before)..

(more music: Crash Test Dummies)..

ok, I added my thing to the router middleware, using app.all('*', my_access_checking_middleware). stupid. there should not be a conceptual difference between app.use() and app.all('*'), yet there is..

one good thing about express though is that it's used so much that there was a Stack Overflow question addressing this subtle use case.

anyway, what was I doing? something with access control.. right..

hm.. I've been checking for things in sets with _.has(mySet, key), but that will return true if the set has the key, but the value of the key is false. I think I want a _.setHas or _.inSet.. yeah, the second one.. (I don't think this is a problem for previous cases where I've used _.has, since I generally don't keep keys in the object with a false value.. I would generally delete them entirely).. I suppose I could redefine my personal notion of a set to not care about the value.. hm.. I think for now I'll never have false values in a set, but also use _.inSet..

hm.. another argument to think about sets as objects where every key is in the set is that it works better for iterating over the keys (I don't need to filter based on the value).

I wonder if this means I should remove _.inSet, since it implies a different notion of sets.. or make _.inSet work like _.has, proactively implying my desired notions of a set, but with a redundant function.. hm.. I think I'll keep _.inSet as it is. I should never have set values of false, but if I ever do have one, it still seems correct to say it is not in the set.

(more music: Girlyman, ooh, a new album by them!)

ok, good, I've implemented setting permissions in the most inefficient way possible.. first I go through each existing user and update their clearance based on the list of people I got, and then I go through each person in the list I got and update their clearance, and possible create a user for them..

..hehe, I was about to say "inefficient, but correct!", but it isn't even that, because I have two lists of people, admins and workers, and I represent them with clearance levels 2 and 1 respectively, and I iterate over the workers second, overriding the fact that they may have been an admin..

hm.. I want a _.setSub for this (subtracting a set from a set)..

ok, done. it's now possible to set permissions..

work notes

I made a change to the way payments are calculated, which should pay everyone a little bit more — more correctly. I'd like to test it.. let's see when the last payment cron job was executed.. looks like 15 minutes ago. Perfect. Plenty of time till the next one. Let's just try running it with "heroku run node pay_people.js".. ok, it seems to be paying people a little more. Of course, people also did more work since I last paid them, but I think it's more than that.. oh well, I'm pretty sure it's right, and I don't think it's going completely off the rails, so that's good at least..

I had a thought about updating the version without scaring people too much. First, I could update only when people try to grab a task, so I know they won't be in the state where they just submitted an answer (and may therefore be worried about whether that answer got submitted correctly).. and in the new version, for future updates, if I deal with the textbox-contents-disappearing-issue, then I can make it just refresh the page without alerting them..

so.. what to do.. I think I need to go through e-mail and todos.

postponing

had a thought: maybe it's good to push everything back.. if something still seems important a month from now, it's probably important.. it's like a more extreme version of the saying "don't make the decision now, sleep on it first".

pull request

my first pull request to be accepted by someone, yay!

self actualization

Here's a rough sketch of my interpretation of Maslow's hierarchy of needs.

Now I often worry about what my ultimate goal should be.

But if I'm in the bottom non-self-actualized part of the chart, then a useful goal seems to be getting to the top.

Relations to other thoughts:

I've heard the idea "if you can't be happy with the way things are now, then you can never be happy". I think that may just be false. I can't always be self-actualized, because sometimes my lower-level needs are in danger.

I think there may be two sorts of depression, for me, and I think I often get them confused. When I'm not self-actualized, part of my depression is simply that I'm not self-actualized, but knowing that, I think I can simply accept the fact.

work notes

was about to get ice cream, celebrating the completion of the payments thing — because I had done all I could do without getting some questions answered first — but actually I can do more. I don't know how much to pay people, but I know a lower-bound on how much to pay them, so I can pay them that, and I can raise it later if I discover it should be higher, since the system will remember how much it has paid people, and pay them the extra that they deserve..

so we decided that one person was successfully paid, so I assume payment is working. Let's plug in the conservative payment estimate, and make it pay everyone..

..need music.. Catie Curtis..

ok, let's pay people. here goes..

well, it's slowly paying people, but seems to be working..

now we also need to display to people in the UI that they've been paid..

..hm.. I got a payment error paying someone $0.75. I think maybe $1 is the minimum payment I can make (this was the first payment under $1, and there were payments for exactly $1)..

..ok, I changed the min-payment to $1, let's try running it again..

ok, it's running again.. seems to be going fine.. let's see if these are showing up in the logs..

hm.. I don't see them in the logs.. let's try looking again in a bit.. for now, the payments seemed to go fine, so let's set that up to run on a cron job, and also check in the UI that shows how much one has been paid..

ok, I've added a cron job using the heroku scheduler to run every hour and pay people.

At this point, I would say the bare-minimum has been done for payments.. people get paid what the system thinks they're owed, and they can see how much they've been paid in the interface (although they would need to refresh it to see.. I suppose the interface should refresh that information every hour too).

Hm.. I feel like I want to put in the refresh-every-hour and also change the version number, so people see the new interface, and then call it done..

oh, it updates the user every time the user submits a task anyway, so that's good enough for now.. let's just try updating the version, and see how badly that goes..

hm.. I'm afraid to update the version, because if someone is working on a question, the behavior is that it will wait for them to submit, then tell them that an updated interface is available, asking them to refresh, but also telling them that they can dismiss the dialog if they want to save their work.. implying that they should save their work, but then when they refresh, the task they were working on will no longer be available (because they'll have successfully submitted it), but they won't realize that, and will wonder how to get it back so they can paste in their saved work. alas..

we'll just leave it be for now. Things seem to be working. Let's not rock the boat.

it is a bit stressful writing code where real money is involved.

work notes

ok, I was going to do 4 things,
1. eat
2. put on music
3. deal with people having reviewed their own work, so I don't have special cases for that in the payment code
4. I'd also like to shower and get ice cream.. ice cream for after payments are done I suppose, but I'll need to buy it before midnight..

and I had decided to shower, and then buy ice cream, and get Subway..

I showered, and bought ice cream, but Subway was closed (I'm guessing they close at 9pm), so I got Safeway frozen dinner.. now let's put on some music and eat, and then we'll be ready for number 3..

ok, good.. let's do this.

I feel like I'm performing surgery on data.. let's do it on our local copy, and make sure it's right.. we'll just write temporary code directly into pay_people.js, since this is just something I need to run once, and that code already has the infrastructure to access the database..

ok, what do I want to do.. I think I want to iterate through each record, and see if there are any cases of people reviewing their own word, and print those all out, so I can see how much there is to operate on..

..oops, lucky catch.. I was going to be overwriting my _output.txt file, so I looked at what was there, and there was data of stats I had calculated for different users, and there was a user with deservedCents = null, and it occurred to me that it could be null if the user didn't do any reviewing, since my calculation is: u.answerAcceptedCount * 28 + u.reviewAcceptCount * 4, which will be null if either u.answerAcceptedCount or u.reviewAcceptCount is null.. so I think I want: (u.answerAcceptedCount || 0) * 28 + (u.reviewAcceptCount || 0) * 4..

fortunately I don't think it would have screwed things up to much.. it would have not paid people what they deserved, but it wouldn't corrupt the database, meaning that once the error was fixed, everyone would get paid correctly..

..anyway, that's not what I was working on.. I'm listing instances of people reviewing their own work..

ok, there are 84 instances. I think they may all have only one thing in their history, since they happened pretty early on.. let's see.. yup. So that makes things a bit easier. The query we can use to get them all is.. well.. something like:

find records where record.answeredBy and (record.answeredBy == record.reviewedBy)..

but I'm not sure mongodb allows comparing record fields with other fields.. let's see..

hm.. apparently "yes", but, it involves sending javascript to the server, e.g., db.T.find( { $where: function() { return this.Grade1 > this.Grade2 } } );

I wonder if mongoHQ will let me do that.. I wonder if I can do that from the mongojs node.js driver..

answer to "can I do it from mongojs" is YES, but I need to send the function as a string, e.g., db.collection('records').find( { $where: '' + function() { return this.answeredBy && (this.answeredBy == this.reviewedBy) } }, p.set). note where I have: '' +. in front of function () {...}.

answer to "can I do it on mongoHQ" is also YES. sweet! :)

ok.. so my update will be something like:


db.records.update({
$where: function() { return this.answeredBy && (this.answeredBy == this.reviewedBy) }

...

hm.. the update part would also need javascript, but I don't see a thing for running javascript there..

so I guess we'll have to do it with code..

(put on another album: Sufjan Stevens)

..hm.. all but one of the instances of a reviewer reviewing their own work is one person..

hm.. this person actually modified their answer during some of the reviews.. I do want someone else to review their work, but I suppose I want to review their updated version, so I'll need to copy it out of the history.. wait, no I don't. The history has the old version, so that's good.

ok, it seems to be working.
let's try running it on the real database.. crossing fingers..
seems to work.. good.. now let's remove the little "temp" flag I put there so I could query for the items I changed to make sure they were good..
done

let's put the code that did that somewhere for reference.. how about here:


db.collection('records').find( { $where: '' + function() { return this.answeredBy && (this.answeredBy == this.reviewedBy) } }, p.set)
_.each(p.get(), function (doc) {
db.collection('records').update({ _id : doc._id }, {
$set : {
// work here
temp : true,

availableToReviewAt : 0,
'history.0.answeredBy' : doc.answeredBy,
'history.0.answeredAt' : doc.answeredAt,
'history.0.reviewedBy' : doc.reviewedBy,
'history.0.reviewedAt' : doc.reviewedAt,
'history.0.reviewAccept' : true
},
$unset : {
reviewedAt : null,
reviewedBy : null
}
}, p.set)
console.log("done: " + doc.reviewedBy)
p.get()
})

Ok, now it is on to the payment code again..
ok, it seems to be working.. let's strip out the scaffolding, and run it on heroku, but make it artificially throw an exception after making the first payment, saying who was paid.. and let's check that that payment went through..

ok.. ready to press go.. hm.. let's throw the error before the payment is made, and make sure it works, and see who is going to be paid, so I can find them on oDesk and see how much they've already been paid for comparison purposes..

hm.. it's not as easy as I thought to guarantee that I'll be looking at the same person as the "first to get paid" on the second run.. let's look through the teams and see if anyone has been paid.. if they're all zero, that will make things easier..

ahh, how about I'll filter for a particular person to pay..

I ran: heroku run node pay_people.js

it's running...
hm.. it threw an exception, as expected, sortof, except that it's complaining: Cannot find module '/app/pay_people.js'... but it was running for a while before showing that.. so I suspect it may actually be an error trying to display the error.. let's see if anything was logged or changed in the database..

oh.

I forgot to upload my code.

let's do that..

ok, it ran.. but it ran entirely too fast. Something did not go right. Let's see what happened..

well, the console.log did show up in my local console using heroku run, so that's convenient. We'll just add back some console.logs..

(need more music.. Tilly and the Wall)

..oh, I need to remember to create a fake version of february-fire and do a full-on 5000 person blitz attack against it, to see how many dynos I need..

ahh, I see my mistake. I was returning from my _.each loop for all non-matching users, but I was returning "false", which is a special return value to break out of the _.each loop, so it was entering the loop, not finding the user it wanted, returning false, and not checking any other users..

oh yes, I need to first login to the running heroku app, so it can get my accessToken..

ok, it did it.. things to check..
did the person get paid? hm.. it doesn't look like it.. at least, I don't see it in the oDesk interface..
did I get a "payment receipt"? yes.. hm..

ugg.. this person has two contracts in two different team rooms.. I must have paid the other one, but that one already has been paid a fair amount, so I'm not sure if my payment was included.. is there a way to view payment history?.. hm.. I can't find one in the UI.. maybe there's an API call for it..

hm.. I ran this API call: https://www.odesk.com/gds/finreports/v2/buyer_teams/TEAM_REF/billings?tq=SELECT%20amount%20WHERE%20provider__reference%20=%20'PROVIDER_REF'&tqx=out:json

and it showed a bunch of data, and those bits of data added up to the amount of money that the UI says this person has been paid..

but none of those bits corresponds to my payment (they are each for an amount, and my amount isn't among them)..

hm.. I'm guessing the payment failed.. how how? the API call succeeded.. where did the money go?

AHA! It did NOT fail :) -- what I was looking at before was "billings", which shows how much money  we lost, but that's 10% more than the amount of money they gained, which there's another API call for with the word "earnings", and when I do that, I do see my payment.

so phew..

hm.. it occurrs to me that before I can run this, I need to confirm whether the 28 cents and 4 cents rewards are before or after oDesk's 10% cut. I assumed they were after when I paid this person, so I may have overpaid them..

I did send an e-mail asking this earlier.. let's see if they got back to me..

I haven't heard back. I don't think I can go further with payment before know, so I'll wait till tomorrow. It's just about done though, so I suppose ice cream is still in order..

work notes

ok, I'm still trying to pay people in the february-fire system using the oDesk API. I want to create a script that does it that I will run as a heroku cron job.

The plan for the script is to iterate through all the tasks, and construct stats for each worker saying how much money each worker deserves, and then we'll pay each worker the difference between how much they deserve, and how much we've already paid them, and then increment a counter of how much we've already paid them (and if the program has an error between paying and updating the counter, then we'll end up paying them again, but hopefully that won't happen often, and will only be for small amounts). I feel like a "perfect" solution would involve querying oDesk to see how much has been paid, but in theory, a person could have been given a bonus manually for some reason, and my system wouldn't realize that it hadn't given them that money, so it would need to do more than just query how much has been paid for the engagement.. somehow it would need a breakdown of the transactions themselves..

hm.. there are some old answers that were reviewed by the same person who wrote them, before the logic was in place to prevent that. I'll probably want to re-open those questions for review, but I also want to make sure the reviewers are still paid (since it was my fault and not their fault that they were able to review their answer, and presumably they did spend time reviewing it, even though they probably ended up deciding it was good). Since I'll be recalculating everything from ground truth in the cron job, I need to trick that calculation into still thinking that these reviewers did a reviewing task, even though there is no record of it.. perhaps I'll make a record of it in the "history" attribute, which currently just has records of rejected jobs — I'll make a special entry for an "accepted" job.

Ok, I've written all the code. Now to test it. But I don't want to hurt anything while testing it.. I think I'll copy the database from heroku to my local machine, and test it here.. how do we get a dump of the database again..

I see a lot of stuff about mongodb 2.2.. I want to make sure that's the version I'm using both locally and in MongoHQ. Let's check.. locally I have.. 2.2.3, and on MongoHQ they have.. also 2.2.3. Great.

ok, now to download the database using mongodump..
something like this.. I need to adjust it for my needs..
mongodump -h flame.mongohq.com:27053 -d YOUR_HEROKU_APP_NAME -u heroku -p HEROKU_PASSWORD -o db/backups/

ok, got it in my _dumps directory. Now to "restore" it to my local mongod..

so, it seems to be working, locally.. though I haven't actually called the payments API.. I'm not sure how to test that function..

I also included spitting out "PAYMENT ERROR!" to the logs if there is a payment error, and I think I can make the Logwhatever plugin email me if that ever happens..

..hm.. what to do to test this sucker for real.. it will involve doing "heroku run" I think.. maybe I can make it artificially fail after the first payment, and make sure that that payment happened.. that will also let me test the error in Logwhatever..

ok, before continuing payment stuff, let's do a few things:
1. eat
2. put on music
3. deal with people having reviewed their own work, so I don't have special cases for that in the payment code
4. I'd also like to shower and get ice cream.. ice cream for after payments are done I suppose, but I'll need to buy it before midnight..

ok, let's shower, and then buy ice cream, and get Subway..

work here

I need to pay some people using the oDesk api. There is the custom payment api, but it requires an engagement, and all I have is a username.. so somehow I need to find an engagement from a username..

the engagements api let's me supply a provide reference. Let's try that..

hm.. I don't actually have provider references, I have usernames.. I could have saved their references, but I didn't. Hm.. I could make everyone log in again, and this time save their provider reference..

I guess we want "status=active"..

ok, it seems to work. But there may be more than one engagement, so I'll need to figure out which one I want. I know the team rooms that people were hired into, so I guess I'll find the engagement that is in one of those team rooms.

ok, here are some steps
1. add environment variables for PAYER, which will be me for now
2. store the token for the PAYER when they login (we'll need it to make API calls on their behalf)
3. from now on, when people login, store their ref
4. write a script to gather everyone's ref, by:
    - listing all the engagements for all the teamrooms I care about,
    - and matching these with people's usernames (which is listed in the engagement as provider__id)

let's add _.omit to u.js.. done

working on 1.. using _.omit to do an upsert: I have an object "user" with all the stuff for a user, including the _id. I want to find a user with that _.id, and if I fail, I want to set all the user's properties using { $set : user }, but I can't, because _id is in user, and mongodb doesn't want me to $set the _id, so I use { $set : _.omit(user, '_id') }.. now I should care about an error (whereas I didn't with insert, since an error probably meant that the user was already there).. oh yeah, and we want to grab the ref too.. ok, let's try logging in locally.. first, let's make sure a user for me is already there locally.. yup.. now let's try logging in and see if my information gets appropriately updated.. (and get another CD playing)..

ugg, it's caching an old page.. how to clear the cache? I wish there was just a button for that (or better yet, why doesn't refreshing the page clear it? I suppose it makes sense in some cases, but I don't think I would mind if refreshing always cleared the cache for the current page, because that's usually my goal when refreshing..).. ahh, Command-Shift-R (for mac)..

oops, it didn't set the token, because I forgot to do something with the PAYER environment variable..

ugg, it's still logged in.. how to clear cookies for the current page? ahh, I can delete them from the Resources tab in Chrome JavaScript debugger panel thing.

success. I got the tokens. I guess I've done 1, 2 and 3. Now on to 4. What should this script be called? It is a temporary thing I think, since in general, I'll be getting people's refs as they login, so I'll have those on hand in order to lookup their engagements. So maybe let's call this temp.js..

hm.. I notice that I'm still using an old technique in february-fire that I updated in nar-nar. I could update february-fire, but that would mean testing it. I think I'll leave it, and just do things the new way in new projects. Retroactive fixing is cumbersome, and projects will die eventually anyway.

ok, I got my user, now let's make an oDesk API call with the token.. success..

ok, first, I need to list all my teamrooms, to get the refs for the ones I care about.. I'll also want to store these for future reference when trying to get the correct engagement for someone, since it will need to be one of these rooms..

oops, node-odesk wants just 'team/v2/teamrooms', not 'api/team/v2/teamrooms'.

hm.. I have access to 13 team rooms.

(put on another CD)..

ok, I found the teamrooms I'm interested in. Now we want to get the engagement for each team..

hm.. node-odesk forces me to manually add query parameters for GET requests. I'd rather pass them in as an object, just like for POST requests. I think I'll change this in node-odesk and send them a pull-request.. and probably they'll reject it and make me feel stupid somehow, per usual..

..I guess the first step is to make the change locally (as in in the version inside my node_modules directory), and test it, then we can copy it into my forked version of node-odesk..

seems to work, let's put it in our fork and check it in.. ok, here's my commit.. and here's the pull request.  We'll see what comes of that.

ok, we were trying to list the engagements in a team. It's giving me back 10 engagements, but there are many more. It gives some pagination information:



"total_count": "990",
"total_items": "2",
"paging": {
    "offset": "0",
    "count": "10"
}

I assume there are 990 items.. but what does total_items=2 mean? Hm.. anyway.. let's see, how do we tell it to give us the next page.. (put on another CD.. maybe I should call them albums..)

hm.. I had this error: I had the code fs.writeFileSync('./_output.txt', blah), but I forgot to require 'fs', so that was undefined, but it didn't display the error :( — hm.. it seems like node's process.on('uncaughtException', function (err) {}) doesn't catch ReferenceError's..

hm.. ok, so here's what to expect when calling api/hr/v2/engagements under different conditions:

- if there are multiple engagements to return, then there will be an array in result.engagements.engagement.

- if there is just one engagement to return, then result.engagements.engagement will be it (it will not be inside an array)

- if there are no engagements to return, then result.engagements.engagement will not exist at all (it will not be an empty array)

I'm guessing the reason for this is that the oDesk API can return xml as well, and this is the way it converts xml stuff into json.

ok, I think I'll write a utility function for grabbing lists of stuff. I've had to do this before for getting offers, and I can see a similarity..

interesting.. need to concatenate multiple arrays together, and the fastest way seems to be  [].concat.apply([], arrays), which I believe because of some guys speed test..

great, it works.. I wonder if it is worth trying to add to node-odesk.. I'd need to convert it not to use fibers.. I think I'll do it if they accept my other pull request, otherwise not..

ok, let's get all the engagements.. we've gotten the ones for one team, but there are multiple teams to get..

well, I want to store that utility function somewhere while I wait for the node-odesk people.. maybe a gist.. maybe I'll just put it here.. it's not quite gist worthy (since it currently relies on my personal utility library)

function getAll(path, params) {

var kind = path.match(/([^\/]+)s(\?|$)/)[1]

var kinds = kind + 's'

if (!params) params = {}

var accum = []

var offset = 0

var pageSize = 100

var p = _.promiseErr()

while (true) {

params.page = offset + ';' + pageSize

o.get(path, params, p.set)

var a = p.get()[kinds]

var b = a[kind]

if (b) {

if (b instanceof Array)

accum.push(b)

else

accum.push([b])

}

offset += pageSize

if (offset >= a.lister.total_count)

break

}

return [].concat.apply([], accum)

}

ok, I have all the engagements.. I guess what I want to do now is iterate through each user in the database, find the engagement for their username, and store their ref back in the database (ultimately we'll be getting their engagement from the ref, and at that time, we'll probably cache the engagement reference too, but I want to make sure that code works, so I won't do it here, even though I'll have the engagement refs.. though.. I probably won't normally have so many to do at once.. maybe.. well, it will be in at least an hour long cron job, and I think it can finish inside of an hour, so that's fine)..

hm.. I want _.find.. need to add it to u.js.. done..

ok, I updated everyone's ref.

though, there were about 7 people that I couldn't find an engagement for.. I should ask about these people.. ok, it looks like I know who 5 of them are (2 of them are my test accounts), so I just need to ask about 2.. and between those 2 people, they've only answered 1 question, so it may not be a big deal.

so now I need to write a script for paying people.

let's think about that..

strategy for now

do things that will have lasting impact...

of course, I also want to put the dishes away — how do I justify that?

18:45 hous

I went to bed at 7:00am and woke up at 1:45am.

work notes

ok, the interface now shows a name, username and image, and has an "accept" and "reject" button. Pressing accept or reject replaces those buttons with an undo button, in case the clicked the wrong thing..

now when they're done with a batch, make it so they can click "grab new batch" to get more.. done..

now, when they click "reject", we want to send them to an external link to do some stuff.. done..

ok.. I think it's time to put this on heroku. Let's upload it, and see where it's at... I suspect it will run, but there won't be anything in the database..

hm.. I'm getting a strange error: "Error: Cannot find module './_utils'". I searched for references to _utils, and there are four, all on oauth stuff..

AHA! I added "_*" to .gitignore. Did I have that in february-fire? yes.. hm.. I wonder if I had already checked in those files before adding "_*". I guess I can look in the git history.. I think so..

ok, so, I definitely want to ignore some stuff with _, but not the things inside of node_modules I guess.. how do I do that? Hm.. I guess "/_*" for now, since all my stuff with underscores is in the root directory..

(I guess that answers my question about whether I can modify stuff inside node_modules, and expect heroku to use my modified version, rather than refetching and overriding it using npm install)

good! it's running.. but there are no tasks, as expected.

I want to populate the database using the email.js script, which I'm hoping to run on a worker dyno every so often, as in a cron job.. let's see how to do that..

first step: heroku run node email.js

success. It ran on a dyno. So now I just need to schedule it.. done, using heroku's scheduler addon.

now maybe it would make more sense to call email.js --> cron.js,

poetry and art

I talked with my friend about my previous art and poetry post. I mentioned that I've had poems explained to me, so I know they can be got, and this is one of the people who has explained a poem to me. So I asked him to explain the Dickinson poem. The first thing he did was correct my spelling of Dickinson. Here's the poem again, with his interpretation after:

SUCCESS is counted sweetest
By those who ne’er succeed.
To comprehend a nectar
Requires sorest need.
Not one of all the purple host
Who took the flag to-day
Can tell the definition,
So clear, of victory,
As he, defeated, dying,
On whose forbidden ear
The distant strains of triumph
Break, agonized and clear.

His interpretation — me paraphrasing:
First stanza: People who succeed all the time don't really understand or appreciate success. "nectar" refers to success.
Second stanza: Royal nobility, when they win a battle, although they won, don't really understand what it means to win. "purple" refers to nobility.
Third stanza: The soldier on the ground dying, far from where the celebration of victory is taking place, is the guy who really understands what the victory means. "forbidden ear" refers to the fact that, from the point of view of the celebrating nobility, the soldier is not meant to hear the sounds of their celebration. The soldier is a lesser entity who's purpose is just to fight.

I think his interpretation makes sense.

I asked him how he came up with it. He said that he essentially reads over the words like "nectar", "purple" and "forbidden ear" and tries inserting different interpretations for them until he finds one that fits.

I also talked with him a bit about the Picasso painting. I'll add it here again for reference:

He said he didn't understand what the hand-looking thing near the woman's neck was. If it was a hand, did that mean the thing beneath it was a foot? I told him I didn't know what it was. It looked like a hand to me, but I wasn't sure how it was meant to be interpreted in the rest of the image — it does seem awkwardly positioned. But I said it didn't bother me. It wasn't the point of the painting, for me.

He pointed me to another artist, Kandinsky:

He said he likes this style better than cubism, and there is no worry of getting caught on anomalies like the awkwardly placed hand in Picasso's work.

I said I like this work too, and it occurred to me that I comprehend Picasso's work in a similar way to how I hear music — I love music with words, but I don't actually know what the words mean in sequence. I take the words as suggestive. Similarly, I take the visual elements in Picasso's painting as suggestive, evoking thoughts of a woman and a chair and such. And in a sense, the Kandinsky painting is like instrumental music, which I like, but tends not to be as powerful for me as music with words.

He also pointed me to the poem "if", which begins:

If you can keep your head when all about you
Are losing theirs and blaming it on you;
If you can trust yourself when all men doubt you,
But make allowance for their doubting too...

This poem does make sense to me. And I think it's more analogous to a renaissance painting — that is, a literal painting that is meant to be exactly what it looks like. I do like the message of the poem, but it is not the sort of poem I mean when I say I don't get poetry.

work notes

at least at first, inputs for this system are going to be sent as e-mail attachments, since they used to be processed manually. So I want to find e-mails, parse the attachment.. and I want to do this on heroku with node, so probably using a "worker dyno".

Step 1: figure out how to find the e-mail using node.. ok, done, using node-imap to fetch the message and mailparser to parse out the attachment. I now have a script that grabs an e-mail with a csv, uploads the items to the database, and then marks that file as read..

I also added a "time" field. I could just use the default mongodb _id, but I want to know when the item was originally generated (which is provided in the csv), rather than when the item was added to the database. I'll need to update my indexes for sorting by time too (since they currently sort by _id)..

Step 2: figure out how to run this file on a worker dyno..

..maybe before I do that, I want to make the display of tasks work.. what was I blocked on there. Oh yes. Architecting a nice way of refreshing the client on server updates.

The best I can think of now is the suggested long-poll, and updating textboxes on refreshes. And actually there won't be any textboxes in this interface, so I can punt on that for now.

So how do we do a long poll.. let's first make sure heroku supports that.. hm.. sortof. From what I gather, it supports long polling with a 10 second duration.. hm..

I'm going to punt on versioning altogether for now. Let's get the interface working.. ok, it shows the tasks, with accept and reject buttons. Let's make accept accept, and make an rpc call, and show "accepted / undo" in the interface. And the rpc call doesn't need to tell the user that it was successful  because these people are being paid by the hour, and if it wasn't successful, the task will be given to someone else anyway. Pay by the hour is so much easier to program for.. well, easier anyway..

almost there, but need to pick someone up from the airport..

work notes

I'm creating another system similar to the one before.. let's start by forking it.. hm.. I can't seem to fork my own repository. I can't believe github doesn't have that feature.. oh well, I'll do it the old-fashioned way.. create nar-nar repo.. copy stuff over from february-fire..

now it looks like we can just follow along in the readme file to it up on heroku..

I guess I can't "heroku ps:scale web=2" until I push something.

let's run this thing locally.. ok, it shows the february-fire stuff.

I think a good place to start with this new project is to load the database with some examples of the sort of stuff that will be there.


{ "username" : "fakeuser", "name" : "Fake User", "img" : null }
{ "username" : "testuser", "name" : "Test User", "img" : null }
{ "username" : "bbernard", "name" : "Barney Bernard", "img" : null }

great.. ok, now for this task, people are going to be grabbing batches of tasks, rather than just one.. so let's write the grabBatch rpc..

first let's add a "availableToGrabAt : 0" to each fake user.. done..

on to writing grabBatch..

oh, I first need to add a "random" field to each record, so I can grab a random batch.. wait.. I don't need them to be random. Randomness was good when allowing people to preview items and decide what to grab, but in this case, they're just gonna get stuck with whatever batch we give them (and they can grab a new batch if they don't like it).

so still writing grabBatch.. let's see.. we want a query like this

find all records where the availableToGrabAt time is less than now, sorted by _id descending (which will show the most recently added tasks first — since I've convinced now that for moderation tasks, it's best to do most-recent-first, since those are the highest priority items, since stuff that hasn't been looked at yet has some evidence in favor of it not being so bad.. since it hasn't triggered any alerts elsewhere in the system).

find all records where the availableToGrabAt time is less than now, sorted by _id descending,
goes to
db.records.find({ availableToGrabAt : { $lt : _.time() }).sort({_id : -1}).limit(16)

and actually we want to update, marking all these things as grabbed..

..hm.. not sure how to write the update.. because of the limit 16.. how do I do that? Does the update query support a limit?.. look like no. A person there suggested that doing a find and then an update is faster than a bunch of findAndModify's.. though, it seems like I would still need multiple updates..

oh well, I'm going to do multiple findAndModify's. It's easier to make sure the code is correct, and I'll worry about it only if it's too slow..

so, in the best of conditions, it's taking about 30 milliseconds to do 16 findAndModify's.. that strikes me as pretty slow.. I think I'm going to try the find and update.. ok, that's about 4 milliseconds. Worth it I suppose.. of course it has a higher danger of returning no items, even when there are items available, since it could find some free items, but then those items could be taken before it can update them.. I suppose if it returns nothing, we can run the original findAndModify code.. of course, then that code will be pretty dead, e.g., not used very often.. another strategy is to try again to find some items.. and keep trying so long as the find returns stuff, but the update fails.. or maybe try doing that 10 times.. ok, done..

now I should add a couple indexes.. done

ok, now the rpc is working. next I want to display the items.. but before I do that, I want to rethink the interface.. I had problems before with updating to new versions of the interface, and giving people messages about what's going on.. let's think for a moment about an elegant solution for that..

meditation

I was brushing my teeth, and I had a thought about meditation. I decided I should meditate "my own way", as opposed to trying to follow the advice of other people about how to meditate.

So I sat down in a chair by the fire, and thought. I looked inwardly. I looked at my thoughts. I tried to bring my mind to a halt, so I could see what was going on. I tried to identify thoughts. I had insights of various sorts, and then they would fade. I could put them into words if I tried, but it was hard, and I decided it may not be worth it.

I had one thought where I thought about how I care about what other people think, and that thought, that caring, was represented as a box sortof behind my eyes, as if it was "watching me", and I mentally moved it to the side, arranging things differently. And it made sense somehow, and seemed better. And then I was afraid that my mind would move the box back where it was, as if moving the box in my mind would undo what had been done, and I zoomed in on this fear and thought about it.. and I decided that the real insight was the understanding itself, not the mental moving of boxes, and that there was no need to fear having the image of moving the box back, and then those images faded.

I had another thought that I wanted to remember various insights, but I decided that I really only needed to remember one insight, and that was "meditate every day". And the other insights would come again. Sort of like trying to become good at a sport — it's not super important to remember each bit of advice about how to play, but rather it's most important to just play, and relearn those bits of advice over and over from experience until they settle in.. the things I'm writing in this blog post I didn't try to remember, these are just the things I happen to remember.

I also had another thought, which was "meditation is about introspection". It's about spending time looking at the inner world of the mind, as opposed to the external world. That is to say, meditation is about looking at how the mind thinks, rather than working through a problem. Meditation is "meta thought" — thought about thought. And the way to do it is just to think about thinking. All of these things like "clear your mind" and such seem like red herrings, as if some magic will happen from clearing your mind — and what are you meant to do after you clear your mind? I think a better approach is just to think about thinking, and do whatever is helpful for doing that.

planning and logistics

I had a thought about a theory about why planning and logistics are so hard for me..

I want my mind to be clear. I like to not be worried about stuff. It is easier to think that way.

When I have something planned, it spawns a process in my brain that keeps pinging my brain saying "hey, remember about that thing that is planned".

You'd think I could just write it down to get rid of that pining, and if I write it down, and it's long in the future, the pinging mostly goes away, but it is replaced with "hey, remember to keep checking your calendar thing, because there's stuff on it in the future."

And the more stuff I have going on, the more pinging there is, with processing saying "hey remember about this" and "hey remember about that".

Each ping is like an interrupt, in the CPU sense. It pulls my brain out of the current thought process, and puts it, if only for a short time, in this other process to remember about some thing, and there's context switching costs and such.

Stuff that's pending consumes a disproportionate amount of CPU time in my brain, compared to the actual size of the tasks involved that are pending. For instance, if there is a 30 minute meeting coming up 3 days from now, it will consume much much more than 30 minutes of CPU time effectively making sure that I remember to be at the specific time and place of the meeting.

If it was not my responsibility to remember stuff to do.. If that planning was done by someone else.. If I had someone to say "think about X until further notice"..

But then again I want to be in control. I've tried planning out a bunch of stuff to do, and I just can't convince my brain to be satisfied with all those decisions when it comes time to do them. My brain might want to think about something else.. and if someone else was telling me what to think about, I'm pretty sure I would second guess it every time.

Perhaps one solution is a time for planned stuff, and a time for nothing to be planned.. something like "Wednesday, Thursday and Friday are for scheduled stuff. If you want to meet, I can meet one of those three days. Otherwise, you can still ping me and see if I'm free, but you can't plan anything with me."