work notes

ok, I think the thing to do now is some nar-nar stuff. First, they've changed the spreadsheet where I'm getting data to include two additional columns. I'll need to display those in the UI, but first, I notice that the csv file has "|" as a delimiter instead of ",". What kind of csv is that? I wonder if that is part of the standard, or if I need to explicitly tell my csv library to use "|" as a delimiter. Let's find out.. seems that I need to tell it to use "|". How do I do that..

ok, my the csv parser I've been using is way too complicated. and I can't figure out how to change the delimiter  even though it claims to have the ability. I found a simpler csv parser, and modified it to my needs..

I think I'll rename cron.js to grab_csv_from_email.js..

ok, let's stop the cron job on heroku, so it doesn't interfere with my testing..

ok, now let's make the most recent csv in my e-mail "unread"..

hm.. I couldn't log into the gmail account that houses the csv files.. I was able to "recover" the account by answering some questions, like when I created the account, what my previous password was, and such.. I'm a little worried.. I can't imagine that someone tried to "hack" the account, and it doesn't seem like they did anything with it if they did.. hm.. hm..

ugg.. I don't understand, node-imap is giving me a "Error: Invalid credentials".. maybe I should try logging in with a different gmail account (but not my main account, I think.. )..

stranger and stranger.. that one gives me a "Error: Web login required: http://support.google.com/mail/bin/answer.py?answer=78754 (Failure)"

um.. now that one is also saying my password is wrong.. I find that hard to believe. I feel like there is something else at work here..

I feel like this might be a good time to enter information into my main gmail account to.. er.. make sure I don't lose it..

ok, turned on "2 factor authentication" for my main gmail account. we'll see how that goes..

now let's try accessing that other account again..

AHA! I see what was happening. I would type in a password, press enter, and then it would show me a captcha, but delete the password, so I'd enter the captcha, and then it would complain that I put in the wrong password. The trick is to type my password, press enter, retype my password, do the captcha, and then press sign in.

..so I think my password wasn't wrong after all, even for my other account. I was just being confused by the user interface removing my password without me noticing.

ugg.. I can access the other account I tried, but not the account I want to access. I can sign into the account I want to access using the web, so I know my password is correct..

(nice, my phone now uses an app-specific-password for gmail that I can revoke if I lose my phone)

ugg, how frustrating.. I think google has locked out non-web access to that account because I was checking it too frequently. I was checking it every 10 minutes, and I've read in a couple places that checking more than every 10 minutes is bad. I wonder if there's a way to convince gmail that I've changed, and I was not check it so often anymore, or if I need to create a second bogus e-mail account and forward stuff to that instead..

(let's put on some music: Matchbox Twenty.. I want to see them in concert. They're in Las Vegas at the end of March..)

ok, good, I can login from my new bogus gmail account from node.js. Hopefully I won't anger google's servers again with this account..

good, it's reading the new file format and such.. what next.. oh yes, the interface needs to display more information than it was displaying before. I did have things all in a grid, but with this extra information, that may be too compact.. though the extra information is only for some items.. so some items could be in a grid, but the grid wouldn't look super great if it was broken up by these larger items.. hm..

i could put them in a list..

yeah, I think a list may be easiest..

ok, I'm supposed to make the images 100x100, now I have a link to the image, but it's an S3 link, and it's signed, so I can't just modify it to give me the 100x100 version (which I happen to know exists).. hm.. I'm not sure how to get it.. I could scrape it from their profile page, but I'm not sure how to get their profile page either.. I have their username, but I don't know an api call to go from username to profile..

hm.. I may punt on that.. I think it will be easier for them to put it in the spreadsheet, since that process has access to the database..

(more music.. The Killers)..

ok, they want to restrict who has access to this tool, and they want to be able to update that list.. so I guess I can't do it with an environment variable.. hm..

I suppose I can add a "clearance" field to users. I was worried that some users would need to be cleared before they even log in, but I think that's fine.. I can create those user entries before they log in, and they'll just look like { _id : "someuser", clearance : 1 }, and when they log in, it will add more information to the record, like their name and such..

..now, where should I put this admin interface? I could make a special html page for it.. I'd rather just keep one html page, but maybe it would be easiest to keep the admin interface separate.. sure, I'll go try a separate page..

hm.. I think I'd like to do this authentication with middleware, but that means my middleware would need to sometimes not call "next", e.g., when I redirect people to a login page.. is that allowed? I guess I'll have to try it..

ugg.. I don't understand.. I have some middleware, and I'm not calling "next", yet it manages to do the next thing in the chain.. how?..

oh god, it makes a difference whether I put it before or after the passport stuff.. why would express care! my mental model of middleware must be wrong somehow..

express! why!

ok, it's not the passport stuff per se, it's after my first use of app.get. Are no middleware allowed after that point? they obviously are allowed, and they're even called.. they just don't care about "next" anymore, it seems..

I wish I could find documentation for that..

AHA! fucking hell, here it is:

Note that if you don't explicitly use the router, it is implicitly added by Express at the point you define a route (which is why your routes still worked even though you commented out app.use(app.router)).

Thanks josh3736

It appears that all the "routing" things, like app.get, app.post, app.all, are sortof grouped together and all appear as if they were added wherever the first of any of these calls is made.. and middleware after that point still gets called, but after the whole group of routing things..

I guess one way to think of this is that routing is a single middleware item, as opposed to a utility for creating a new middleware each time I call app.get or app.post (which is what I had thought before)..

(more music: Crash Test Dummies)..

ok, I added my thing to the router middleware, using app.all('*', my_access_checking_middleware). stupid. there should not be a conceptual difference between app.use() and app.all('*'), yet there is..

one good thing about express though is that it's used so much that there was a Stack Overflow question addressing this subtle use case.

anyway, what was I doing? something with access control.. right..

hm.. I've been checking for things in sets with _.has(mySet, key), but that will return true if the set has the key, but the value of the key is false. I think I want a _.setHas or _.inSet.. yeah, the second one.. (I don't think this is a problem for previous cases where I've used _.has, since I generally don't keep keys in the object with a false value.. I would generally delete them entirely).. I suppose I could redefine my personal notion of a set to not care about the value.. hm.. I think for now I'll never have false values in a set, but also use _.inSet..

hm.. another argument to think about sets as objects where every key is in the set is that it works better for iterating over the keys (I don't need to filter based on the value).

I wonder if this means I should remove _.inSet, since it implies a different notion of sets.. or make _.inSet work like _.has, proactively implying my desired notions of a set, but with a redundant function.. hm.. I think I'll keep _.inSet as it is. I should never have set values of false, but if I ever do have one, it still seems correct to say it is not in the set.

(more music: Girlyman, ooh, a new album by them!)

ok, good, I've implemented setting permissions in the most inefficient way possible.. first I go through each existing user and update their clearance based on the list of people I got, and then I go through each person in the list I got and update their clearance, and possible create a user for them..

..hehe, I was about to say "inefficient, but correct!", but it isn't even that, because I have two lists of people, admins and workers, and I represent them with clearance levels 2 and 1 respectively, and I iterate over the workers second, overriding the fact that they may have been an admin..

hm.. I want a _.setSub for this (subtracting a set from a set)..

ok, done. it's now possible to set permissions..