more data munging

ok, the remaining 8 items are done.. now I've got to compile everything into a big pile of.. er.. a csv file.

(oh, I want to sing out loud.. I wonder if my roommate is away)

ok, let's run a giant query and store everything in a file for further processing..

note: beware that...

    function hello() {
        return "hi 1"
    }

    console.log(hello())
    return

    function hello() {
        return "hi 2"
    }

...will still print out "hi 2".

this was tripping me up because I had a giant function with test code, and I would effectively comment out everything beneath my current code with a "return" statement, but it wouldn't really be commented out if I was reusing a function name..

back to what I was doing.. running some sort of query.. done

ok.. large csv file created and sent.

data surgery notes

accounting for items.. I was processing some jobs, but there were some duplicate jobs, and I didn't do a good job of dealing with those, so now I need to do a bit of data surgery...

I have two batches of jobs in a single mongodb. I tagged the second batch with a "batch" field. I want to tag the first batch also.. done

not quite all the items in the first batch are done; there's about 8 remaining. People are working on the second batch, which is large, but I want to prioritize these 8 from the first batch..

..there are actually 5 items that aren't even uploaded, so let's upload those.. done

now to prioritize them, let's do:

db.records.update({batch:"first_batch_name", availableToAnswerAt : 0}, {$set:{availableToAnswerAt : -1}}, {multi:true})

..doing availableToAnswerAt = -1 should push them to the top, since we sort by that when showing people available jobs..

we also need to do it for availableToReviewAt:


db.records.update({batch:"first_batch_name", availableToReviewAt : 0}, {$set:{availableToReviewAt : -1}}, {multi:true})

hm.. I'll probably need to run those queries again, in case someone picks up a task, and returns it, since it will set it's "availableAt" time to 0 instead of -1.. I could hack the code to do this for me, but that would require testing, and it's probably easier to just manually hand-hold this process..

hopefully these items will get done today..

oh..

and I have this same issue for a couple new batches, which I should fix preemptively..

ok, good, I think everything is in order now. probably. I thought it was in order before, but now I really thinks it's in order :)

fixing unicode mess

ok.. I have three systems running that are all failing to send unicode correctly in json responses.

first, let's patch them all, so they work going forward..

here's the change I want to make: 'application/json; charset=utf-8'

february-fire — done
nar-nar — done
third project which is really another instance of february-fire — done

next, let's try to repair the damage. There were some questions marked as "bad", which I'm guessing looked like unicode garble — of course, that means they were probably in a foreign language, so they probably are bad for our purposes, but let's make sure..

hm.. it looks like there is some unicode gibberish in the original questions — at least, when I load it as utf-8 — meaning that for at least some questions, there was a unicode mixup before the file ever came to me.. so.. that's good ;)

unicode mess

I think I may have messed up some unicode. Let's check the unicode pipeline.

First, I'll need some sample utf-8 unicode. We need a unicode character that is definitely not ascii. How about something Chinese: "你好 world". I think that means "hello world".

Ok, let's put that in the Sublime Text editor, and try to save it as utf-8.. done.

Now let's try loading it and printing it to the console — we'll see if the mac's console supports unicode.. hm.. it does. great.

Now let's try printing it out as json, just to see what it does — I think I'd prefer if it escaped it with \u, but I don't think it will.. it does not escape it.

Let's try saving it back in a file as json, and loading it again as utf-8.. success

Great, now let's try saving it in mongodb, and see what it looks like in the mongodb console.. that works too.. so far, so good..

Now let's try getting it back from mongodb in javascript.. works..

Hm.. let's see if node.js assumes that javascript files are utf-8.. we'll put "你好 world" as a literal inside a javascript file and save it as utf-8, and try running it.. works.. I guess the default is utf-8.

Now let's try sending the json to a webpage using ajax.. FAILURE: when I point the browser at the json directly, it shows "ä½ å¥½ world". I assume that if it executed the page as javascript, it would also be wrong.

..how to fix this.. somehow the browser needs to be informed that this json is utf-8 encoded. I'm guessing this will be header information.. though I'm surprised since I'm using.. oh wait, I was going to say I was using an external library to form the json http response, but I'm not, so that makes it even more likely that I'm just doing it wrong.. let's look it up..

I found my answer in this stackoverflow question: What does “Content-type: application/json; charset=utf-8” really mean?

Success!!

..and, sadness that I failed before. But mostly Success!!

Let's make sure that I can load that with ajax and put it into a div.. yup.

self-programming

when frustrated, accept situation

when angry, forgive all involved, including myself

when sad, be sad

when stuck solving problem, get more information

phone password

My five-year-old nephew borrowed my phone and said he knew what to do — I looked and saw the password screen and said "you don't know what to do in this part". But I was wrong. He knew my password. Somehow.

6:45

I woke up at 6:45 today and thought it was 6:45 pm, judging by the light, and how much I had expected to sleep.