3/31/13
3/30/13
why no applicants?
hm.. I've posted a number of humanscripts which are not getting any applicants. I wonder why. One possibility is that I haven't designated any "skills" for them, and maybe people search by skill. One reason I haven't designated any skills is that the oDesk API doesn't currently support that.. I'll try adding skills manually, and if that helps, I'll need to talk to someone about adding skill support to the API.
february-fire first humanscript
finally! I'm posting a humanscript for february-fire — this was the secret goal all along with taking on that project. Here is the humanscript in github issue form: https://github.com/dglittle/february-fire/issues/1
ok, posted it.. I put it as a $50 job, since it involves comprehending my code a little bit. we'll see if we get any takers..
ok, posted it.. I put it as a $50 job, since it involves comprehending my code a little bit. we'll see if we get any takers..
throbber
I created this issue: https://github.com/dglittle/misc/issues/1
A friend came along and did it: https://github.com/alltom/misc/blob/gh-pages/throbber.html
Here's a running version: http://dglittle.github.com/misc/throbber.html
I'm not quite sure how he found out about it.
I sent him a link to the job posting on oDesk so he could apply, and I could pay him.
Done, and done :)
Next, I want to use that throbber in nodesk.. here's the issue: https://github.com/dglittle/nodesk/issues/2
A friend came along and did it: https://github.com/alltom/misc/blob/gh-pages/throbber.html
Here's a running version: http://dglittle.github.com/misc/throbber.html
I'm not quite sure how he found out about it.
I sent him a link to the job posting on oDesk so he could apply, and I could pay him.
Done, and done :)
Next, I want to use that throbber in nodesk.. here's the issue: https://github.com/dglittle/nodesk/issues/2
a couple humanscript side projects
I had a couple of ideas that I want to humanscript. One is an advanced-chess server. The other is an add-on to the yabla language learning video player — I want it to show translations for each word directly beneath each word.
For the advanced-chess project, I created this issue: https://github.com/dglittle/advanced-chess/issues/1 , which has someone create a chess board using unicode characters. The next step will be to draw a chess board given a FEN string.
For the yabla project, I created this issue: https://github.com/dglittle/yabla-extension/issues/1 , which has someone create a starting point for a chrome extension that can interface with yabla. The next step will be to have it augment the way the transcription is shown.
I posted both of these issues on oDesk using nodesk.
For the advanced-chess project, I created this issue: https://github.com/dglittle/advanced-chess/issues/1 , which has someone create a chess board using unicode characters. The next step will be to draw a chess board given a FEN string.
For the yabla project, I created this issue: https://github.com/dglittle/yabla-extension/issues/1 , which has someone create a starting point for a chrome extension that can interface with yabla. The next step will be to have it augment the way the transcription is shown.
I posted both of these issues on oDesk using nodesk.
3/29/13
more humanscript work notes
I just created an issue here: https://github.com/dglittle/misc/issues/1
I created a super simple job — maybe too simple. It just says "github issue: create a throbber", and the task description just has a link to the github issue. We'll see how that goes.
Now to post the other one.. done. Ok, let's see how those turn out.
update: hm.. I don't give much indication up front about how big the task is — maybe I can just say in the title "about 1 hour". Ok, I edited the titles to include "(about 1 hour of work)".
I created a super simple job — maybe too simple. It just says "github issue: create a throbber", and the task description just has a link to the github issue. We'll see how that goes.
Now to post the other one.. done. Ok, let's see how those turn out.
update: hm.. I don't give much indication up front about how big the task is — maybe I can just say in the title "about 1 hour". Ok, I edited the titles to include "(about 1 hour of work)".
humanscript notes
I posted four humanscripts last night.
Notes:
For one job, I asked someone to write a simple function for creating lorem-ipsum style random text. The person I hired had submitted completed code as part of their cover letter. I should have hired them to add the functionality I needed to an existing open-source lorem-ipsum generator.
For another job, a hired the first applicant who answered my three questions correctly, and they sent me a screenshot for verification this morning — it looked correct.
Two of the humanscripts failed to get any applicants. I need to figure out why. One asks someone to create a simple throbber, and insert it into a larger project. I think I should split that one into two tasks: creating the throbber, and inserting it.
After talking to my friend, I think it will also help to make the jobs into github issues, for better accountability (something for me to put links to from here), and also to provide more context to the workers about the job.
So next steps:
- split the throbber job into two sequential jobs, and post the first one as an issue
- repost the alignment job as an issue
Notes:
For one job, I asked someone to write a simple function for creating lorem-ipsum style random text. The person I hired had submitted completed code as part of their cover letter. I should have hired them to add the functionality I needed to an existing open-source lorem-ipsum generator.
For another job, a hired the first applicant who answered my three questions correctly, and they sent me a screenshot for verification this morning — it looked correct.
Two of the humanscripts failed to get any applicants. I need to figure out why. One asks someone to create a simple throbber, and insert it into a larger project. I think I should split that one into two tasks: creating the throbber, and inserting it.
After talking to my friend, I think it will also help to make the jobs into github issues, for better accountability (something for me to put links to from here), and also to provide more context to the workers about the job.
So next steps:
- split the throbber job into two sequential jobs, and post the first one as an issue
- repost the alignment job as an issue
oatmeal
I had one of those moments where something seems good, then bad, then good, then bad, then good.
I made some oatmeal,
but then realized I was out of raisins,
but then realized I had another container of raisins,
but then realized that container didn't have enough raisins,
so I decided to put the oatmeal in the fridge for now,
but then realized I didn't have the lid I needed,
but then realized why I didn't have that lid,
because there was already perfectly good oatmeal in the fridge using the lid,
which I am now eating :)
I made some oatmeal,
but then realized I was out of raisins,
but then realized I had another container of raisins,
but then realized that container didn't have enough raisins,
so I decided to put the oatmeal in the fridge for now,
but then realized I didn't have the lid I needed,
but then realized why I didn't have that lid,
because there was already perfectly good oatmeal in the fridge using the lid,
which I am now eating :)
3/27/13
posting a human script
I want to post a humanscript to make a change to february-fire. The task I'm interested in is allowing admins to change who the editors are.
I want to clone my old script. It would be great to have a clone feature in nodesk.
After thinking for a long while, I think the right next step is to make february-fire easily runnable after forking it, including having it generate just enough test data for the person who runs it to know it's running correctly.
I think I'll need to do this bit myself.
I want to clone my old script. It would be great to have a clone feature in nodesk.
After thinking for a long while, I think the right next step is to make february-fire easily runnable after forking it, including having it generate just enough test data for the person who runs it to know it's running correctly.
I think I'll need to do this bit myself.
random
♖ : I wish random.org supported jsonp. I suppose they're afraid of getting too much traffic. It is unfortunate that true randomness is so elusive. I wonder if a scalable random server could be created if every person asking for random data also supplied some somewhat random data that could be aggregated into true random data, so the more people asking for random data, the more random data there would be
♞ : Some transcendental meditation article mentioned two interesting things: first, using a mantra, where the sound of it matters somehow; second, letting bubbles of thought enter the mind. I tried meditating for 20 minutes, using the word "om" — they say it's not good to say what your mantra is, but I disagree. I'm not sure how far I got, but not 20 minutes, because my alarm woke me up. Afterwards, I tried thinking in "bubbles", thinking complete thoughts, complete sentences that made sense, with words, and I literally imagined them in bubbles. This seemed to hold potential goodness — mental discipline.
♗ : I often wonder what to optimize for. I often wonder how to make decisions — like deciding when to get up. I think one potential solution is optimizing for myself some time in the future, like myself 10 years from now.
♞ : Some transcendental meditation article mentioned two interesting things: first, using a mantra, where the sound of it matters somehow; second, letting bubbles of thought enter the mind. I tried meditating for 20 minutes, using the word "om" — they say it's not good to say what your mantra is, but I disagree. I'm not sure how far I got, but not 20 minutes, because my alarm woke me up. Afterwards, I tried thinking in "bubbles", thinking complete thoughts, complete sentences that made sense, with words, and I literally imagined them in bubbles. This seemed to hold potential goodness — mental discipline.
♗ : I often wonder what to optimize for. I often wonder how to make decisions — like deciding when to get up. I think one potential solution is optimizing for myself some time in the future, like myself 10 years from now.
3/26/13
password security
note: The dime is irrelevant. The dime is there to convince the camera to focus on the blank screen, as opposed to the reflection of the ceiling in the screen.
..We originally used a key instead of a dime, but decided not to give away two passwords with one photo.
..We originally used a key instead of a dime, but decided not to give away two passwords with one photo.
thoughts passing my way
♞ : some thoughts passed my way today
♖ : such as?
♞ : it's good to write, even if nobody reads it
♖ : phew
♞ : yes, also, IQ is a measure of the improbability of intelligence, not the amount of intelligence. about one in one hundred people will have an IQ over 135, but maybe such people are only a little more intelligent than the average human. IQ doesn't tell us.
♖ : hm.. and with 6.7 billion people, some person definitely — by definition — has an IQ of 195, but I suppose that's not enough smarts to go from nothing and come up with something smarter than what people have come up with..
♖ : such as?
♞ : it's good to write, even if nobody reads it
♖ : phew
♞ : yes, also, IQ is a measure of the improbability of intelligence, not the amount of intelligence. about one in one hundred people will have an IQ over 135, but maybe such people are only a little more intelligent than the average human. IQ doesn't tell us.
♖ : hm.. and with 6.7 billion people, some person definitely — by definition — has an IQ of 195, but I suppose that's not enough smarts to go from nothing and come up with something smarter than what people have come up with..
♞ : yes, also, it would be cool to have a game where a computer helps you, like Advanced Chess, and it would be great if it was a web-based game, and the AI was written in JavaScript. that would make it easy for people to hack on new AIs and share them.
♖ : hm.. I wonder if there's a chess AI implemented in JavaScript.. yup. and it's better than me! it would be easy to put together an Advanced Chess web-app with that..
♞ : you know, we're chess pieces..
♖ : hm.. I wonder if there's a chess AI implemented in JavaScript.. yup. and it's better than me! it would be easy to put together an Advanced Chess web-app with that..
♞ : you know, we're chess pieces..
♖ : coincidence
3/25/13
energy
I thought I lacked energy because I wasn't happy.
But now I'm happy, and still lack energy.
I thought I lacked energy because I felt constrained.
Now I feel free, and I still lack energy.
What's up, body?
But now I'm happy, and still lack energy.
I thought I lacked energy because I felt constrained.
Now I feel free, and I still lack energy.
What's up, body?
3/24/13
externalities
I feel like externalities are an unsolved problem waiting on an elegant solution.
by externalities, I mean things like car alarms, which annoy lots of people, and yet that cost is not born by the person who buys the alarm, or bilboard advertising, where the person paying for an ugly ad only needs to pay the person who owns the land beneath the sign, rather than the area that can see it
by externalities, I mean things like car alarms, which annoy lots of people, and yet that cost is not born by the person who buys the alarm, or bilboard advertising, where the person paying for an ugly ad only needs to pay the person who owns the land beneath the sign, rather than the area that can see it
self programming patch 2
previous version:
in doing this, I've noticed that sometimes I'm happy, but still find myself not doing the things I want because I feel constrained for various reason outside my control.
I think that I may be able to reinterpret these "reasons outside my control" as within my control, since they often are within my control, I just feel pressure to act one way or another, but the choice is ultimately mine.
I think the meta idea of this self programming patch is to "take responsibility" for how I feel. That is, I decide that all the negative things I feel are my own fault, which makes them within my power to fix, since things that are not my fault are harder to fix.
- when frustrated, accept situation and convert to sadness
- when angry, forgive all involved, including myself, and convert to sadness
- when sad, be sad
- when frustrated or angry, reinterpret situation as positive and happy
in doing this, I've noticed that sometimes I'm happy, but still find myself not doing the things I want because I feel constrained for various reason outside my control.
I think that I may be able to reinterpret these "reasons outside my control" as within my control, since they often are within my control, I just feel pressure to act one way or another, but the choice is ultimately mine.
I think the meta idea of this self programming patch is to "take responsibility" for how I feel. That is, I decide that all the negative things I feel are my own fault, which makes them within my power to fix, since things that are not my fault are harder to fix.
serenity prayer
♛: i've thought about creating philosophical advertising. here's an ad idea:
♔: reminds me of that saying: "give me strength to accept the things i cannot change, the courage to change the things i can, and the wisdom to know the difference"
♛: ah yes.. i don't like that saying. i think it's better to accept everything, even the things i can change, and go from there, otherwise i'll always worry about each thing wondering if i have employed sufficient wisdom deciding whether it is one of the things i can change or not
♔: er.. to be consistent, shouldn't you have accepted the saying rather than change it?
♛: um.. i.. accept.. that it was said
♔: reminds me of that saying: "give me strength to accept the things i cannot change, the courage to change the things i can, and the wisdom to know the difference"
♛: ah yes.. i don't like that saying. i think it's better to accept everything, even the things i can change, and go from there, otherwise i'll always worry about each thing wondering if i have employed sufficient wisdom deciding whether it is one of the things i can change or not
♔: er.. to be consistent, shouldn't you have accepted the saying rather than change it?
♛: um.. i.. accept.. that it was said
3/20/13
poetry interpretation
Part One: Life II |
OUR share of night to bear, | |
Our share of morning, | |
Our blank in bliss to fill, | |
Our blank in scorning. | |
Here a star, and there a star, | 5 |
Some lose their way. | |
Here a mist, and there a mist, | |
Afterwards—day! (taken from here) |
I want to try interpreting poems. Perhaps I can become better at it. Here's my attempt for today. After reading the poem above, I have this to say (that rhymes with today, bwahaha):
I interpret the poem as being about thinking and assessing at night, and acting during the day. "Our share of night to bear" means we all get some time at night to assess, but we might view it negatively as "bearing" it, or more positively as "morning", and we can chose to interpret the previous day ("our blank") positively, filled with "bliss", or "scorn" the things that happened...
The stars and mist might refer to ideas, and the forgetting of ideas. We have some ideas, but some don't seem good "lose their way", and some we forget in the "mist".. and after we've done all this thinking and assessing, we need to try our hand at the next day.
Anyway, that has some holes, but whatever. We'll try again tomorrow. Maybe.
idea: github wars
here's the idea:
- programmers go to a github wars site and click I want to compete
- they are given an issue from github (maybe people could tag suitable issues)
- they are given a 1 hour time limit (they must commit within 1 hour of being told which issue to work on)
- another programmer is also given this issue, and 1 hour (not necessarily at the same time, so the first programmer's code would need to be hidden from them)
- after both programmers have committed, a winner is chosen by the person with the issue (and perhaps by some other programmers)
- programmers get a chess-elo-style score based on wins and loses
drawing each other
- people draw each other
- you can't see their picture of you until you submit your picture of them
- awesome
(thanks Michael Levinson for link)
3/19/13
3/16/13
3/14/13
consolidating rpcs
rpc1()
rpc2()
problem: requires two tcp connections, when we could theoretically use just one
rpc("rpc1", "rpc2")
problem: makes rpcs look like second-class citizens
rpc1() // ...push into rpc list
rpc2() // ...push into rpc list
finalize_rpcs()
problem: need to remember to call finalize_rpcs()
rpc1() // ...setTimeout(finalize_rpcs, 0)
rpc2() // ...kills previous timeout, setTimeout(finalize_rpcs, 0)
good.
rpc2()
problem: requires two tcp connections, when we could theoretically use just one
rpc("rpc1", "rpc2")
problem: makes rpcs look like second-class citizens
rpc1() // ...push into rpc list
rpc2() // ...push into rpc list
finalize_rpcs()
problem: need to remember to call finalize_rpcs()
rpc1() // ...setTimeout(finalize_rpcs, 0)
rpc2() // ...kills previous timeout, setTimeout(finalize_rpcs, 0)
good.
3/13/13
humanscripting with nodesk (part 1: getting off ground)..
ok, the time has come, finally.. let's get nodesk up and running and start humanscripting again..
step 1: hm.. I want to create an app on heroku and link it to the current nodesk repo.. I'm not sure exactly what "heroku create" does.. hm.. let's backup everything in nodesk, clear it out, and try running "heroku create"..
hm.. seems to have create an app ok.. now I want to name this one "nodesk", let's try renaming it on heroku.. actually, let's see if there's a command-line for that.. yup: heroku apps:rename newname..
ok, I got http://nodesk.herokuapp.com, sweet..
now let's put a skeleton app up there..
ok, we'll copy from nar-nar, since that's the most recent project..
ok, "hello world" version is up..
now I'm thinking I want to merge nodeutil.js and u.js.. hm.. that actually seems like something I should humanscript later..
ugg.. I tried upgrading node with brew, but apparently I didn't install it with brew, so I tried installing it with brew, but it is caught in some intermediate state where I can't run the old node or the new node.. I'm nodeless! nodeless... nodeless...
I'm back! noded. just needed to delete some files it was complaining about.. hopefully unnecessary files..
ok.. let's get login working..
hm.. I just noticed that the following code in login.js...
db.collection('users').update({ _id : user._id }, { $set : _.omit(user, '_id') }, { upsert: true }, function (err) {
if (err) return done(err)
done(null, user)
})
...was missing the err, so err was referring to the error in the enclosing function. I think I'll fix this in nodesk's version of login.js, but I don't want to endanger the running apps like nar-nar and february-fire by "fixing" this in them, since it's more likely to introduce problems than prevent them..
ugg.. login isn't working: 500 failed to obtain request token (status: 401 data: � sK��IMQ(�W(K��LI,IU�O,-�P(�L�K,)-JUH� �g�� ���,)
step 1: hm.. I want to create an app on heroku and link it to the current nodesk repo.. I'm not sure exactly what "heroku create" does.. hm.. let's backup everything in nodesk, clear it out, and try running "heroku create"..
hm.. seems to have create an app ok.. now I want to name this one "nodesk", let's try renaming it on heroku.. actually, let's see if there's a command-line for that.. yup: heroku apps:rename newname..
ok, I got http://nodesk.herokuapp.com, sweet..
now let's put a skeleton app up there..
ok, we'll copy from nar-nar, since that's the most recent project..
ok, "hello world" version is up..
now I'm thinking I want to merge nodeutil.js and u.js.. hm.. that actually seems like something I should humanscript later..
ugg.. I tried upgrading node with brew, but apparently I didn't install it with brew, so I tried installing it with brew, but it is caught in some intermediate state where I can't run the old node or the new node.. I'm nodeless! nodeless... nodeless...
I'm back! noded. just needed to delete some files it was complaining about.. hopefully unnecessary files..
ok.. let's get login working..
hm.. I just noticed that the following code in login.js...
db.collection('users').update({ _id : user._id }, { $set : _.omit(user, '_id') }, { upsert: true }, function (err) {
if (err) return done(err)
done(null, user)
})
...was missing the err, so err was referring to the error in the enclosing function. I think I'll fix this in nodesk's version of login.js, but I don't want to endanger the running apps like nar-nar and february-fire by "fixing" this in them, since it's more likely to introduce problems than prevent them..
ugg.. login isn't working: 500 failed to obtain request token (status: 401 data: � sK��IMQ(�W(K��LI,IU�O,-�P(�L�K,)-JUH� �g�� ���,)
things that are different:
- I'm using a different odesk api key and secret
- I'm potentially using a different version of passport
I guess I could try with the old keys..
..the old keys work. So I guess the keys are the issue..
..ahh, right, I set my "nodesk" odesk app to use the non-oauth authentication.. I guess I should re-create it..
ok, login working :)
now the first thing I want is the ability to post a job.. hm.. I'm tired.. let's take a break for a bit..
hopefully last urgent task before humanscripting..
my goal has been to humanscript a lot of these tasks, but they've been to urgent.. this is hopefully the last one, and it's pretty simple. I just need to increase the delay for warned accounts from 48 hours to 5 days, and add 3 days of time to all the currently warned accounts..
step 1: make warning add 5 days instead of 48 hours.. done
step 2: update existing items in warned queue by adding 3 days (might make warned items that happened very recently have 8 days, but that's probably not worth worrying about).. this seems like a job for mongodb-side javascript..
db.eval(function () {
var a = db.records.find({'status.warnUntil':{$exists:true}})
for (var i = 0; i < a.length(); i++) {
var r = a[i]
db.records.update({_id:r._id}, {$set:{'status.warnUntil':r.status.warnUntil + (1000 * 60 * 60 * 24 * 3)}})
}
}, {nolock:true})
step 1: make warning add 5 days instead of 48 hours.. done
step 2: update existing items in warned queue by adding 3 days (might make warned items that happened very recently have 8 days, but that's probably not worth worrying about).. this seems like a job for mongodb-side javascript..
db.eval(function () {
var a = db.records.find({'status.warnUntil':{$exists:true}})
for (var i = 0; i < a.length(); i++) {
var r = a[i]
db.records.update({_id:r._id}, {$set:{'status.warnUntil':r.status.warnUntil + (1000 * 60 * 60 * 24 * 3)}})
}
}, {nolock:true})
...let's test it locally first.. ok, seems to work.. ran on server.. seems to have worked.. super. let's let people know.
oh.. I missed something.. I changed the "warnUntil" value, but that feeds into the availableToGrabAt value, which I need to also change (but only if it hasn't been grabbed)..
db.eval(function () {
var a = db.records.find({'status.warnUntil':{$exists:true}})
for (var i = 0; i < a.length(); i++) {
var r = a[i]
db.records.update({_id:r._id,availableToGrabAt:(r.status.warnUntil - (1000 * 60 * 60 * 24 * 3))}, {$set:{availableToGrabAt:r.status.warnUntil}})
}
}, {nolock:true})
..that seems to have repaired it.
ok.. oh, and there's one other thing I need to do.. I added a feature to february-fire that people now want in the clone of february-fire (I'll need to humanscript some merger of these two projects.. though one of them will probably end before that happens anyway..)
phew, ok, done..
idea
try to measure different team structures by putting people in those structures and having them play a game against another team with a possibly different structure, where the measure of team-structure-quality is how often teams with that structure win the game
suspected origin: Tom Malone, Ramesh Johari and myself
suspected origin: Tom Malone, Ramesh Johari and myself
rejection reports..
same ol' same ol', settup up another report, where I first need to do some data munging..
I'm currently storing rejections inside a "history" attribute of each record. I decided to do this for atomicity purposes — that is, when an item is rejected, I want to reset it's state so that another person can redo it, while simultaneously making a note that the rejection happened.
But having a separate "history" table would be easier for running queries.. I feel the laughing of my SQL friends.. "everything should be a separate table"..
hm.. I would consider thinking about the history as "roughly right", but since there are payments involved, I think it should be exact. Of course, as it stands, payments are only for accepted items anyway.. though I do want people to have an exact accounting of why they weren't payed for certain things..
I feel like this is yet another argument in favor of paying by the hour — things are just so much easier to keep track of that way.
hm.. I'm currently building a "rejects" table based on the history in each record. This is done every day when payments are processed. I think I'll stick with this for now.. though I think I'll flatten the table, because I currently have one entry for each user with an array of their rejects..
oh.. I see why I have it that way.. it's a silly reason which can replaced by deriving a unique key for each rejection record..
hm.. and I'm currently building up some huge list of rejections into memory, which might fail at some point.. this seems like a job for server-side javascript (and by server-side, I mean running on the mongoDB instance itself using eval).. of course, I want to use the "nolock" option, but that option isn't available in db.eval, which is all mongojs provides.. does mongojs really not provide db.runCommand?..
AHA! It doesn't have db.runCommand, but does have db.eval, which I didn't realize existed. great.
db.eval('function (x) {return x;}', [3], {nolock:true}, function (err, x) { console.log(x) })
Ok, so let's rewrite the payment processor do run an eval that updates the rejects table the way I want..
oh, it's so beautiful to run javascript on the mongodb server:
db.eval("" + function () {
var recs = db.records.find()
for (var i = 0; i < recs.length(); i++) {
var rec = recs[i]
if (rec.history) {
for (var ii = 0; ii < rec.history.length; ii++) {
var h = rec.history[ii]
if (h.reviewedBy && !h.reviewAccept) {
h._id = h.answeredBy + " " + h.answeredAt
h.batch = rec.batch
h.question = rec.question
h.category = rec.category
db.rejects.insert(h)
}
}
}
}
}, { nolock : true }, p.set)
..too bad I can't use u.js, but at least it cuts out all kinds of transmission times.. and I can write the code with syntax highlighting, just prepending "" + to convert it to a string..
javascript/mongodb +1
ok, where was I.. oh, right, I want to drop the rejects table and rebuild it.. before I drop it, I'll make a backup.. ok.. backup downloaded.. rejects table dropped.. ok, running payments script.. man this takes a while, I hope it finishes before midnight when it is scheduled to run again.. I suppose if it gets close, I can cancel the cron job (I'm not confident that it would work fine to run this script concurrently with itself..).. oh, speak of the devil, it finished.. good, the rejects table seems to have been created ok..
so now that we have a nice rejects table, let's create the api call download it.. I'll just model it after the csv query..
done.. pushing.. people informed.
I'm currently storing rejections inside a "history" attribute of each record. I decided to do this for atomicity purposes — that is, when an item is rejected, I want to reset it's state so that another person can redo it, while simultaneously making a note that the rejection happened.
But having a separate "history" table would be easier for running queries.. I feel the laughing of my SQL friends.. "everything should be a separate table"..
hm.. I would consider thinking about the history as "roughly right", but since there are payments involved, I think it should be exact. Of course, as it stands, payments are only for accepted items anyway.. though I do want people to have an exact accounting of why they weren't payed for certain things..
I feel like this is yet another argument in favor of paying by the hour — things are just so much easier to keep track of that way.
hm.. I'm currently building a "rejects" table based on the history in each record. This is done every day when payments are processed. I think I'll stick with this for now.. though I think I'll flatten the table, because I currently have one entry for each user with an array of their rejects..
oh.. I see why I have it that way.. it's a silly reason which can replaced by deriving a unique key for each rejection record..
hm.. and I'm currently building up some huge list of rejections into memory, which might fail at some point.. this seems like a job for server-side javascript (and by server-side, I mean running on the mongoDB instance itself using eval).. of course, I want to use the "nolock" option, but that option isn't available in db.eval, which is all mongojs provides.. does mongojs really not provide db.runCommand?..
AHA! It doesn't have db.runCommand, but does have db.eval, which I didn't realize existed. great.
db.eval('function (x) {return x;}', [3], {nolock:true}, function (err, x) { console.log(x) })
Ok, so let's rewrite the payment processor do run an eval that updates the rejects table the way I want..
oh, it's so beautiful to run javascript on the mongodb server:
db.eval("" + function () {
var recs = db.records.find()
for (var i = 0; i < recs.length(); i++) {
var rec = recs[i]
if (rec.history) {
for (var ii = 0; ii < rec.history.length; ii++) {
var h = rec.history[ii]
if (h.reviewedBy && !h.reviewAccept) {
h._id = h.answeredBy + " " + h.answeredAt
h.batch = rec.batch
h.question = rec.question
h.category = rec.category
db.rejects.insert(h)
}
}
}
}
}, { nolock : true }, p.set)
..too bad I can't use u.js, but at least it cuts out all kinds of transmission times.. and I can write the code with syntax highlighting, just prepending "" + to convert it to a string..
javascript/mongodb +1
ok, where was I.. oh, right, I want to drop the rejects table and rebuild it.. before I drop it, I'll make a backup.. ok.. backup downloaded.. rejects table dropped.. ok, running payments script.. man this takes a while, I hope it finishes before midnight when it is scheduled to run again.. I suppose if it gets close, I can cancel the cron job (I'm not confident that it would work fine to run this script concurrently with itself..).. oh, speak of the devil, it finished.. good, the rejects table seems to have been created ok..
so now that we have a nice rejects table, let's create the api call download it.. I'll just model it after the csv query..
done.. pushing.. people informed.
3/12/13
cake and eating it
one reason I love javascript is that I feel like I can use an imperative style when I want, and a functional style when I want..
I was just thinking it would be nice if it also allowed a "fast" style, like an asm block in C, some designation like fast { ... } where the stuff inside the ... would be some restricted language with tighter assumptions, but it would map more closely to machine instructions, and could be optimized to be super fast.
I was just thinking it would be nice if it also allowed a "fast" style, like an asm block in C, some designation like fast { ... } where the stuff inside the ... would be some restricted language with tighter assumptions, but it would map more closely to machine instructions, and could be optimized to be super fast.
twice
Instead of a view count on youtube, I'd like a count of how many people viewed the video twice.
When showing top songs on spotify, I want to know which songs people listened to twice.
When I search for x on google, I want to know which links users clicked on twice.
When showing top songs on spotify, I want to know which songs people listened to twice.
When I search for x on google, I want to know which links users clicked on twice.
generating reports 2
back from dancing.
dancing is so great.
ok.. got a downloadable csv report locally.. uploaded it.. app is down.. turns out I upgraded a module, and that module has files that begin with _ which were not checked it, because I ignore stuff starting with _, so I need to only ignore such files in the root folder..
ok, it's up.. good, I successfully downloaded a csv file from heroku..
now I need a thing to show counts as well.. I did this in a similar project, so I can probably just copy that code over.. done
good.. people notified.. done
dancing is so great.
ok.. got a downloadable csv report locally.. uploaded it.. app is down.. turns out I upgraded a module, and that module has files that begin with _ which were not checked it, because I ignore stuff starting with _, so I need to only ignore such files in the root folder..
ok, it's up.. good, I successfully downloaded a csv file from heroku..
now I need a thing to show counts as well.. I did this in a similar project, so I can probably just copy that code over.. done
good.. people notified.. done
talking to myself
I've been talking to myself. Trying. It's embarrassing. I feel like when I first started trying to dance. Even though I was all alone, I was embarrassed to dance in front of myself.
By that token, I feel like there is probably some good to be gained by talking to myself in terms of confidence building.
By that token, I feel like there is probably some good to be gained by talking to myself in terms of confidence building.
3/11/13
generating reports
people want to be able to get results from february-fire. Currently they ask me, and I run database queries and generate a csv file. I want to generate that file automatically from a web address.
My plan is this blah.heroku.com/csv?batch=batchName&begin=3-10-2013&end=3-17-2013
ok.. let's get a stand-in for this that returns "hello world".. done
next, make it echo the input arguments.. done
ok.. now let's run a query to get all the stuff I want..
hm.. the query would be easier if I had a single doneAt field, whereas there are currently two ways a record could be done: reviewAt or rejectedAt.. I think I'll add a common doneAt field, and convert everything else to have it two, which should just be a couple of queries.. oh wait.. I guess it won't quite be a couple queries, since I need to set a different value for each item.. drat..
hm.. I guess I could do it with an eval.. let's try that, and see how it goes..
but first I need to update the code so it adds this, so items from now-on will have it.. done
ok, so now let's update all the other items with eval.. did it locally, and it works.. here we go.. hm.. this query doesn't delete anything, but I think I'll take a backup anyway, because it's a pretty intense query..
seemed to work.. horray.. here's the query for reference:
db.runCommand({
eval: function () {
var count = 0
var recs = db.records.find({reviewedAt:{$exists:true},
doneAt:{$exists:false}})
for (var i = 0; i < recs.length(); i++) {
var rec = recs[i]
db.records.update({_id:rec._id}, {$set:{doneAt:rec.reviewedAt}})
count++
}
var recs = db.records.find({rejectedAt:{$exists:true},
doneAt:{$exists:false}})
for (var i = 0; i < recs.length(); i++) {
var rec = recs[i]
db.records.update({_id:rec._id}, {$set:{doneAt:rec.rejectedAt}})
count++
}
return count
},
nolock: true
})
note: "length()" instead of "length"
ok.. now that we have the doneAt field, let's go back to working on the csv api call..
..ok, I got streaming working on the api call — apparently mongojs upgraded to support streaming between the time I started february-fire, and now.. what luck
..but I need to go to dance
My plan is this blah.heroku.com/csv?batch=batchName&begin=3-10-2013&end=3-17-2013
ok.. let's get a stand-in for this that returns "hello world".. done
next, make it echo the input arguments.. done
ok.. now let's run a query to get all the stuff I want..
hm.. the query would be easier if I had a single doneAt field, whereas there are currently two ways a record could be done: reviewAt or rejectedAt.. I think I'll add a common doneAt field, and convert everything else to have it two, which should just be a couple of queries.. oh wait.. I guess it won't quite be a couple queries, since I need to set a different value for each item.. drat..
hm.. I guess I could do it with an eval.. let's try that, and see how it goes..
but first I need to update the code so it adds this, so items from now-on will have it.. done
ok, so now let's update all the other items with eval.. did it locally, and it works.. here we go.. hm.. this query doesn't delete anything, but I think I'll take a backup anyway, because it's a pretty intense query..
seemed to work.. horray.. here's the query for reference:
db.runCommand({
eval: function () {
var count = 0
var recs = db.records.find({reviewedAt:{$exists:true},
doneAt:{$exists:false}})
for (var i = 0; i < recs.length(); i++) {
var rec = recs[i]
db.records.update({_id:rec._id}, {$set:{doneAt:rec.reviewedAt}})
count++
}
var recs = db.records.find({rejectedAt:{$exists:true},
doneAt:{$exists:false}})
for (var i = 0; i < recs.length(); i++) {
var rec = recs[i]
db.records.update({_id:rec._id}, {$set:{doneAt:rec.rejectedAt}})
count++
}
return count
},
nolock: true
})
note: "length()" instead of "length"
ok.. now that we have the doneAt field, let's go back to working on the csv api call..
..ok, I got streaming working on the api call — apparently mongojs upgraded to support streaming between the time I started february-fire, and now.. what luck
..but I need to go to dance
3/10/13
verbal me
some aggregated thoughts about verbal me
verbal me is relatively unemotional, and can talk about my emotional state in a pretty detached manner
verbal me is not the me I am usually — maybe — and I want to try talking to it sometime when I'm alone
I've had difficulty getting hypnotized, and I wonder if it has to do with hypnosis operating through words, and my subconscious not being verbal
verbal me is relatively unemotional, and can talk about my emotional state in a pretty detached manner
verbal me is not the me I am usually — maybe — and I want to try talking to it sometime when I'm alone
I've had difficulty getting hypnotized, and I wonder if it has to do with hypnosis operating through words, and my subconscious not being verbal
just a little more..
currently the process for getting records into nar-nar is a bit indirect — things go into a spreadsheet, which is sent to an e-mail, and nar-nar reads the spreadsheet from the e-mail (this was not my idea — this was the way it worked previously since people were processing the spreadsheet by hand).
We want a more direct approach, so I need to create an API entry point to add items..
here it is: http://blah.herokublah.com/add?uid=greglittle&profile_key=~12345&created_ts_gmt=1362956465001
seems to work..
now I need to upload that..
hm.. I also changed things like db.collection('records').update to just db.records.update, but there are many such changes, and although I used search-and-replace, I'm worried.. I think I'll revert those changes and just start doing it the new way going forward, rather than worry about testing all the old stuff..
ok, added.. now let's tell the people who care..
done
phew
We want a more direct approach, so I need to create an API entry point to add items..
here it is: http://blah.herokublah.com/add?uid=greglittle&profile_key=~12345&created_ts_gmt=1362956465001
seems to work..
now I need to upload that..
hm.. I also changed things like db.collection('records').update to just db.records.update, but there are many such changes, and although I used search-and-replace, I'm worried.. I think I'll revert those changes and just start doing it the new way going forward, rather than worry about testing all the old stuff..
ok, added.. now let's tell the people who care..
done
phew
the nar-nar saga continues...
The database rose from the grave, thanks to mongoHQ.. so now I need to do something with it..
here's where I'm at:
- I have the database of deleted items
- I have csv dumps of previous spreadsheet inputs
- I still don't have quite all the profileKeys I need
here's what I want:
- currently I'm storing a history of actions inside each records.. I want instead a separate history table, since we're more interested in the history of actions taken by a user, rather than the history of actions taken on a record — and I need a place to store actions like "grabbed a batch of records".. the main motivation for this is that I'm going to be giving someone else direct access to the database, and I want it to be more comprehensible for them (the more comprehensible it is for them, the less they need to get help from me).
step 1: start storing new actions in a new table/collection.. done
step 2: backup database.. done
step 3: remove the "history" attribute from all the records.. done
step 4: remove remove img, name, new_id, overview, title
note: I'll want to remove obo and username, since those can be reconstructed from _id now
step 5: set obo and username in the same function that fetches title and overview from oDesk.. done
step 6: remove username and obo from records..
oops, before I do that, let's make the uploader not add these fields either.. well.. let's let the updater run first..
hum-de-dum.. it's running.. waiting.. ok, it finished..
so now I need to re-remove all the stuff I don't want: img, name, new_id, overview, title, username, obo.. done
..and I now need to check-in the uploader not adding these fields.. done.. I just need to remember to check that that works the next time the uploader is executed..
step 7: merge mongoHQ's backup into my recent backup, marking the recovered items as recovered=true.. upload recent backup to "narnarRecent".. good, now upload the HQ backup into "narnarHQ".. good.. now write script to copy items from narnarHQ which don't already appear in narnarRecent and put them there, adding the "recovered" flag..
..oops, forgot to set recovered flag.. let's kill narnarRecent and restore it again.. done..
ok, the script is running..
..this should put me in a position where I have just one backup database to do stuff from,
and there are a couple things I want to do from it..
- restore the recovered items into heroku's database
- re-create the histories into the new history system..
- check the spreadsheets to see if there are items that never even made it into nar-nar because of failed inserts..
bwahahaha! the script was taking a while checking for the existence of items, and on-the-fly, I added an index, and then suddenly I get about 100-fold speedup :)
ok. I also want to put the _on_ice.txt items into narnarRecent.. done
now let's see which items from the spreadsheets are not in narnarRecent.. hm.. a lot it seems.. but they're old.. I think they're before the start of nar-nar..
the oldest item in nar-nar is 1361491264000 (Feb 21st).. let's see if there are any items after that time.. nope.. super
so now all that remains is the following:
- recover missing records from narnarRecent into heroku
- recreate history from narnarRecent into heroku
to do this, I'm just waiting on the profileKeys, which should come when the data warehouse updates.. I'm not sure how often it does that though..
I guess I don't need the profile keys for the history.. though I sortof do, since I need to recreate histories for some records which don't exist, though I suppose I can do that fine, since the history just shows what the record looked like when the user did stuff to it, and I have that information..
though I don't really have that information — I just have what the record looked like when it was first created.. hm..
I feel like the old history is impure.. hm.. what is it's purpose.. it's purpose is supposed to be checking the quality of people making assessments..
hm.. hm.. I'll ask about the history. I don't think it will actually get used, so maybe I can just provide an archival dump of what I currently have.
here's where I'm at:
- I have the database of deleted items
- I have csv dumps of previous spreadsheet inputs
- I still don't have quite all the profileKeys I need
here's what I want:
- currently I'm storing a history of actions inside each records.. I want instead a separate history table, since we're more interested in the history of actions taken by a user, rather than the history of actions taken on a record — and I need a place to store actions like "grabbed a batch of records".. the main motivation for this is that I'm going to be giving someone else direct access to the database, and I want it to be more comprehensible for them (the more comprehensible it is for them, the less they need to get help from me).
step 1: start storing new actions in a new table/collection.. done
step 2: backup database.. done
step 3: remove the "history" attribute from all the records.. done
step 4: remove remove img, name, new_id, overview, title
note: I'll want to remove obo and username, since those can be reconstructed from _id now
step 5: set obo and username in the same function that fetches title and overview from oDesk.. done
step 6: remove username and obo from records..
oops, before I do that, let's make the uploader not add these fields either.. well.. let's let the updater run first..
hum-de-dum.. it's running.. waiting.. ok, it finished..
so now I need to re-remove all the stuff I don't want: img, name, new_id, overview, title, username, obo.. done
..and I now need to check-in the uploader not adding these fields.. done.. I just need to remember to check that that works the next time the uploader is executed..
step 7: merge mongoHQ's backup into my recent backup, marking the recovered items as recovered=true.. upload recent backup to "narnarRecent".. good, now upload the HQ backup into "narnarHQ".. good.. now write script to copy items from narnarHQ which don't already appear in narnarRecent and put them there, adding the "recovered" flag..
..oops, forgot to set recovered flag.. let's kill narnarRecent and restore it again.. done..
ok, the script is running..
..this should put me in a position where I have just one backup database to do stuff from,
and there are a couple things I want to do from it..
- restore the recovered items into heroku's database
- re-create the histories into the new history system..
- check the spreadsheets to see if there are items that never even made it into nar-nar because of failed inserts..
bwahahaha! the script was taking a while checking for the existence of items, and on-the-fly, I added an index, and then suddenly I get about 100-fold speedup :)
ok. I also want to put the _on_ice.txt items into narnarRecent.. done
now let's see which items from the spreadsheets are not in narnarRecent.. hm.. a lot it seems.. but they're old.. I think they're before the start of nar-nar..
the oldest item in nar-nar is 1361491264000 (Feb 21st).. let's see if there are any items after that time.. nope.. super
so now all that remains is the following:
- recover missing records from narnarRecent into heroku
- recreate history from narnarRecent into heroku
to do this, I'm just waiting on the profileKeys, which should come when the data warehouse updates.. I'm not sure how often it does that though..
I guess I don't need the profile keys for the history.. though I sortof do, since I need to recreate histories for some records which don't exist, though I suppose I can do that fine, since the history just shows what the record looked like when the user did stuff to it, and I have that information..
though I don't really have that information — I just have what the record looked like when it was first created.. hm..
I feel like the old history is impure.. hm.. what is it's purpose.. it's purpose is supposed to be checking the quality of people making assessments..
hm.. hm.. I'll ask about the history. I don't think it will actually get used, so maybe I can just provide an archival dump of what I currently have.
+1 for mongoHQ
I deleted half my database here, and it seemed like it would cost $250 for mongoHQ to give me the most recent backup, but it was free — apparently because I'm on heroku — and they got back to me quickly :)
3/9/13
up-to-date profiles in nar-nar..
nar-nar shows profiles, and people review them, but the people need to click a link to see the newest stuff in the profile. It would be nice if the tool showed the newest version of the profile.
getting this information requires a "profile key". My profile key is "~01f894483a6d5ee6c1". Using this key, one can go to https://www.odesk.com/users/~01f894483a6d5ee6c1 to see my profile. One can also get that information in json format by going to http://www.odesk.com/api/profiles/v1/providers/~01f894483a6d5ee6c1.json.
it used to be that nar-nar wasn't receiving the profile keys, and they're not easy to get just from a user's username. But now it is receiving them..
so.. here's what I want to happen:
- I want all the records in nar-nar to have a profileKey.. new one's should get it the way they're getting it.. for old ones, I think I'll need to run some SQL queries on the main database
- when accounts are loaded for review, I want them to update to the latest and greatest information based on calls to the http://www.odesk.com/api/profiles/v1/providers API call.. assuming this is fast enough (I'll be making 10 such API calls for each batch..)
- there's also an issue where I'm ignoring errors inserting stuff into the database. I do this because that error usually means that the item was already there; however, it may mean an actual error, in which case I want to know about it.. and it may be that there were some accounts that I failed to insert into nar-nar and don't know about because they had an error..
so, where to begin.. I feel like the place to start is making sure that all new stuff is processed correctly, and then try and repair the old stuff..
this involves two things for new stuff:
- get the profileKey
- check for errors when inserting into the database
checking for errors when inserting stuff is easier when using an "upsert", which either updates the item, or inserts it. I think this would make sense if the _id's are usernames.. though currently they are not.. they're a concatenation of _id and other information which may change..
I think it would be easier to handle this if the _id's were usernames. That would just make more sense. Hm.. how to convert them.. I think I need to delete records, and insert new ones with the new _id. One issue is that the system is live, meaning people are using it, so I don't want to change _id's of records that people are processing, since they'll get errors when they submit those results..
I think we want a script that works as follows:
- find items with old _id which are not grabbed by anyone, delete and readd them (telling me if it fails)
- keep running this until all the items use the new _id
of course, I should probably start by making the insertion script use the new _id..
ok, let's have a temporary flag that says whether an item is processed.. we'll call it "new_id".
hm.. let's see when the cron job is scheduled next.. actually, I know it's every hour, and right now it's 7:24, so I should be fine.. let's see if there's a new csv ready.. yup, there's a new one (added at 7:03), so we'll need to remember to mark it as unread after processing it with my test script..
ok, script seems to work.. let's upload it to nar-nar, so all the incoming stuff is processed correctly..
ok, I put some stuff to effectively only insert items if they don't exist already.. this seems silly.. I feel like another approach would be to check for the error that happens in insert when the item already exists, and ignore just that error, but that seems less stable, since I'd probably have to check for some text in the error which might change in the future.. let's do a quick check to see if mongodb has an option to pass to insert to repress that error.. hm.. I don't see one
humpf, ok, well, I guess I'll use my hack.. but I need to test it..
hm.. my hack doesn't work.. it finds an object with the given _id, but the object doesn't meet my other criteria, so instead of updating it, it tries to insert it, which fails, because an object with the given _id already exists..
let's see what the error looks like for failing an insert.. it looks like this: "MongoError: E11000 duplicate key error index"
let's see if that "E11000" seems likely to stick around.. well, let's assume so..
ok, I think the error object has a "name" of "MongoError" and "code" of "11000".. let's verify that in a test script.. ok good, this seems to work:
if (e.name == 'MongoError' && e.code == 11000) {
...
}
..and doesn't seem too short-lived..
ok, done.. it is now adding the profileKey and using the username as the _id.. we'll need to test that on the server when the next batch comes along..
for now, let's fix the old items..
let's make a script that searches for records without "new_id" which are not currently grabbed by anyone, delete them, and re-add them with the new _id..
ugg.. despite my precautions, I managed to delete half the database.. long story short, mongojs's forEach does not wait for the first callback to finish before making the second callback, so the script was blocked on many pseudo-threads removing items, and I just thought it was hung-up waiting to process the first item — which seemed to be taking too long — so I terminated the process..
..I think I need to ask mongoHQ for a backup.. hm.. apparently this costs money.. $250. sigh..
ok, e-mail sent to mongoHQ.. although my precautions did not include backing up the database (cry), I did at least print out items before deleting them, so many items can be recovered that way (though not all, because I ran the script a bit before adding even that precaution)..
ok, the items I printed have been added back.. and the script now seems to be correctly processing the remaining items.. though it's going to take a while.. fortunately the script doesn't need to run in one go. it will pick up where it left off..
ok, let's figure out how to get the profileKeys for the old items..
ok, I have access to the database.. just need to figure out where the data that I want is located.. I'd like a visual view of all the tables.. I had some program for that before I re-installed my operating system.. I wonder what it was called..
I think it was called "navicat".. hm.. not free, but a 30 day trial. I suppose I have another 30 days since I wiped my operating system..
great, I found the table I need.. now it has many items.. I think I'd like to get a dump of the last 2 months..
hm.. the most recent one is a day ago, so I guess I need to wait a day or two before it's caught up.. or.. let's see, how recently have I been getting profileKeys in the spreadsheets..
hm.. there's like a 5 hour gap.. drat.. I'll have to wait.. or maybe I should just set the items I have for now..
ok, let's gather all the spreadsheets I have into a giant table.. it could be useful..
humpf, there is a lot of effort to repair the past.. it's so much easier to just make things work better going forward, and in this case, I feel like people would be in support of that.. maybe I shouldn't be spending time trying to repair the past at all.. alas, I don't quite have the authority to make that call..
so, here's where things are at:
- I screwed up and deleted a lot of data. Hopefully mongoHQ will get back to me about that tomorrow
- I have a script updating a bunch of records in the most inefficient way possible, so it will take a while.. probably a couple more hours
- I have almost all the data I need to set all the profileKeys, but not quite.. hopefully I'll have it tomorrow
I'm waiting on so many things.. all of which are probably not worth much, since really this system just needs to work better going forward. I just want to cross this thing off my list, and I can't :(
update:
I slept a bit.. it finished updating the records to have usernames as keys..
now it's setting as many profileKeys as it can. It was doing that in a super inefficient way too, but I found a way to make it more efficient, involving mongodb's eval.. I could have done a similar thing with the script to delete/re-add items, but it would have been more dangerous, and I wanted to make more sure that I was alerted to any mishaps with that script.. the script to add profileKeys is a less dangerous script..
done.. some items remain undone, presumably because they were added during the 5-hour gap, so I'll need to set them later when the database updates..
let's see how many of them will actually be an issue.. hm.. it's a much smaller number, but non-zero..
the whole point of trying to update all the items to include a profileKey is so I don't need to write special-case code that does one thing if I have the profileKey, and another if I don't.. so I feel like having any items that violate this policy is as bad as having many items which violate it.
so I'll have to wait.. OR, I could download all the offending items and remove them from the database (it's not as if I haven't removed lots of other items accidentally), and add them back later when I add back the rest..
..before deleting anything, we'll naturally go ahead and backup the database..
done. note to self: the deleted items are in "_on_ice.txt"
so now, since everything has a profileKey, we can add code to take advantage of that..
ok, it works.. though now I feel like all the information stored in my records is essentially useless, since I fetch it all again before displaying it using the profileKey. I think I should just be storing the profileKey, and when people mark things as bad, store what it looked like when they marked it as bad..
getting this information requires a "profile key". My profile key is "~01f894483a6d5ee6c1". Using this key, one can go to https://www.odesk.com/users/~01f894483a6d5ee6c1 to see my profile. One can also get that information in json format by going to http://www.odesk.com/api/profiles/v1/providers/~01f894483a6d5ee6c1.json.
it used to be that nar-nar wasn't receiving the profile keys, and they're not easy to get just from a user's username. But now it is receiving them..
so.. here's what I want to happen:
- I want all the records in nar-nar to have a profileKey.. new one's should get it the way they're getting it.. for old ones, I think I'll need to run some SQL queries on the main database
- when accounts are loaded for review, I want them to update to the latest and greatest information based on calls to the http://www.odesk.com/api/profiles/v1/providers API call.. assuming this is fast enough (I'll be making 10 such API calls for each batch..)
- there's also an issue where I'm ignoring errors inserting stuff into the database. I do this because that error usually means that the item was already there; however, it may mean an actual error, in which case I want to know about it.. and it may be that there were some accounts that I failed to insert into nar-nar and don't know about because they had an error..
so, where to begin.. I feel like the place to start is making sure that all new stuff is processed correctly, and then try and repair the old stuff..
this involves two things for new stuff:
- get the profileKey
- check for errors when inserting into the database
checking for errors when inserting stuff is easier when using an "upsert", which either updates the item, or inserts it. I think this would make sense if the _id's are usernames.. though currently they are not.. they're a concatenation of _id and other information which may change..
I think it would be easier to handle this if the _id's were usernames. That would just make more sense. Hm.. how to convert them.. I think I need to delete records, and insert new ones with the new _id. One issue is that the system is live, meaning people are using it, so I don't want to change _id's of records that people are processing, since they'll get errors when they submit those results..
I think we want a script that works as follows:
- find items with old _id which are not grabbed by anyone, delete and readd them (telling me if it fails)
- keep running this until all the items use the new _id
of course, I should probably start by making the insertion script use the new _id..
ok, let's have a temporary flag that says whether an item is processed.. we'll call it "new_id".
hm.. let's see when the cron job is scheduled next.. actually, I know it's every hour, and right now it's 7:24, so I should be fine.. let's see if there's a new csv ready.. yup, there's a new one (added at 7:03), so we'll need to remember to mark it as unread after processing it with my test script..
ok, script seems to work.. let's upload it to nar-nar, so all the incoming stuff is processed correctly..
ok, I put some stuff to effectively only insert items if they don't exist already.. this seems silly.. I feel like another approach would be to check for the error that happens in insert when the item already exists, and ignore just that error, but that seems less stable, since I'd probably have to check for some text in the error which might change in the future.. let's do a quick check to see if mongodb has an option to pass to insert to repress that error.. hm.. I don't see one
humpf, ok, well, I guess I'll use my hack.. but I need to test it..
hm.. my hack doesn't work.. it finds an object with the given _id, but the object doesn't meet my other criteria, so instead of updating it, it tries to insert it, which fails, because an object with the given _id already exists..
let's see what the error looks like for failing an insert.. it looks like this: "MongoError: E11000 duplicate key error index"
let's see if that "E11000" seems likely to stick around.. well, let's assume so..
ok, I think the error object has a "name" of "MongoError" and "code" of "11000".. let's verify that in a test script.. ok good, this seems to work:
if (e.name == 'MongoError' && e.code == 11000) {
...
}
..and doesn't seem too short-lived..
ok, done.. it is now adding the profileKey and using the username as the _id.. we'll need to test that on the server when the next batch comes along..
for now, let's fix the old items..
let's make a script that searches for records without "new_id" which are not currently grabbed by anyone, delete them, and re-add them with the new _id..
ugg.. despite my precautions, I managed to delete half the database.. long story short, mongojs's forEach does not wait for the first callback to finish before making the second callback, so the script was blocked on many pseudo-threads removing items, and I just thought it was hung-up waiting to process the first item — which seemed to be taking too long — so I terminated the process..
..I think I need to ask mongoHQ for a backup.. hm.. apparently this costs money.. $250. sigh..
ok, e-mail sent to mongoHQ.. although my precautions did not include backing up the database (cry), I did at least print out items before deleting them, so many items can be recovered that way (though not all, because I ran the script a bit before adding even that precaution)..
ok, the items I printed have been added back.. and the script now seems to be correctly processing the remaining items.. though it's going to take a while.. fortunately the script doesn't need to run in one go. it will pick up where it left off..
ok, let's figure out how to get the profileKeys for the old items..
ok, I have access to the database.. just need to figure out where the data that I want is located.. I'd like a visual view of all the tables.. I had some program for that before I re-installed my operating system.. I wonder what it was called..
I think it was called "navicat".. hm.. not free, but a 30 day trial. I suppose I have another 30 days since I wiped my operating system..
great, I found the table I need.. now it has many items.. I think I'd like to get a dump of the last 2 months..
hm.. the most recent one is a day ago, so I guess I need to wait a day or two before it's caught up.. or.. let's see, how recently have I been getting profileKeys in the spreadsheets..
hm.. there's like a 5 hour gap.. drat.. I'll have to wait.. or maybe I should just set the items I have for now..
ok, let's gather all the spreadsheets I have into a giant table.. it could be useful..
humpf, there is a lot of effort to repair the past.. it's so much easier to just make things work better going forward, and in this case, I feel like people would be in support of that.. maybe I shouldn't be spending time trying to repair the past at all.. alas, I don't quite have the authority to make that call..
so, here's where things are at:
- I screwed up and deleted a lot of data. Hopefully mongoHQ will get back to me about that tomorrow
- I have a script updating a bunch of records in the most inefficient way possible, so it will take a while.. probably a couple more hours
- I have almost all the data I need to set all the profileKeys, but not quite.. hopefully I'll have it tomorrow
I'm waiting on so many things.. all of which are probably not worth much, since really this system just needs to work better going forward. I just want to cross this thing off my list, and I can't :(
update:
I slept a bit.. it finished updating the records to have usernames as keys..
now it's setting as many profileKeys as it can. It was doing that in a super inefficient way too, but I found a way to make it more efficient, involving mongodb's eval.. I could have done a similar thing with the script to delete/re-add items, but it would have been more dangerous, and I wanted to make more sure that I was alerted to any mishaps with that script.. the script to add profileKeys is a less dangerous script..
done.. some items remain undone, presumably because they were added during the 5-hour gap, so I'll need to set them later when the database updates..
let's see how many of them will actually be an issue.. hm.. it's a much smaller number, but non-zero..
the whole point of trying to update all the items to include a profileKey is so I don't need to write special-case code that does one thing if I have the profileKey, and another if I don't.. so I feel like having any items that violate this policy is as bad as having many items which violate it.
so I'll have to wait.. OR, I could download all the offending items and remove them from the database (it's not as if I haven't removed lots of other items accidentally), and add them back later when I add back the rest..
..before deleting anything, we'll naturally go ahead and backup the database..
done. note to self: the deleted items are in "_on_ice.txt"
so now, since everything has a profileKey, we can add code to take advantage of that..
ok, it works.. though now I feel like all the information stored in my records is essentially useless, since I fetch it all again before displaying it using the profileKey. I think I should just be storing the profileKey, and when people mark things as bad, store what it looked like when they marked it as bad..
3/8/13
brain to brain
one rat tell another rat which button to press, with it's mind
the guy says doing this with humans will rely on the development of non-invasive technologies..
why? sign me up
guilt
I am still vulnerable to guilt. I think this is related to my phobia of offending people.
I feel like guilt may be a harmful emotion to have. I wish I could expunge it from my psyche.
But it is really bad? If I didn't feel guilty, would I go around hurting people? I feel like I wouldn't.. but..
I guess my main complaint is that guilt is a loud and obnoxious emotion. At the round-table of my mental decision-making committee, guilt is pounding the table and yelling so that others can't converse sensibly.
My guilt should feel guilty for hurting the rest of me.
I feel like guilt may be a harmful emotion to have. I wish I could expunge it from my psyche.
But it is really bad? If I didn't feel guilty, would I go around hurting people? I feel like I wouldn't.. but..
I guess my main complaint is that guilt is a loud and obnoxious emotion. At the round-table of my mental decision-making committee, guilt is pounding the table and yelling so that others can't converse sensibly.
My guilt should feel guilty for hurting the rest of me.
3/7/13
sadness and such
The far left is frustration. The grey left is sadness. The middle is I'm not sure, maybe repressed emotion. The grey right is serenity. The far right is winning a prize.
Sadness and serenity are smooth, and don't overwhelm me. They are also oddly similar, like accomplishing something, and feeling good about it, but feeling sad that the journey is over.
Serenity in spiral with sadness
Serenity and sadness as sine wave in emotional plane, with their "imaginary" counterpart outside the emotional plane — intellectual understanding and acceptance perhaps.
Sadness and serenity are smooth, and don't overwhelm me. They are also oddly similar, like accomplishing something, and feeling good about it, but feeling sad that the journey is over.
Serenity in spiral with sadness
Serenity and sadness as sine wave in emotional plane, with their "imaginary" counterpart outside the emotional plane — intellectual understanding and acceptance perhaps.
fixing pima
pima is taking a while to save, like 4 seconds... I patched this earlier by making it save less often, but now I have time to track down the problem. I suspect there is just a super large card in there, probably from a super long e-mail..
how to find it.. I used to have an "eval" feature in pima. I may need to re-add that. Hm.. I should be humanscripting this.. I've lived with the long saving till now..
hm.. before that, I have some merging to do.. my current version is not in sync with github.. done
ok, let's humanscript this:
hm.. I need to see how it will run for someone starting from scratch, with no data..
let's try cloning the repo in a new place..
ok, I added some instructions to the readme..
let's try the humanscript again:
- clone https://github.com/dglittle/PIMA
- put these instructions in humanscript/hs_748569937/hs.txt
- see README to run it
- add "eval" button next to "save" button
- pressing "eval" button should:
var div = new jquery div
var top = new jquery div
top.append("run" button)
top.append("close" button)
var input = textarea with 100% width and height
var output = textarea with 100% width and height
div.append(_.splitVert(1, top, _.splitHorz(50, input, output)))
"close" button should call _.closeDialog()
"run" should eval the code in input,
where any calls to console.log will print out to the output textarea
_.dialog(div)
- test it
run it,
press "eval"
type: console.log("hi")
press "run"
should see "hi" in output
take screenshot
put in humanscript/hs_748569937
- check in
- make pull request to original repo
Ok, I want to post this, but my current system is too cumbersome. I'm supposed to paste this into a json file, and run some command line on it.. I want an interface running at some url where I can just paste my humanscript into a textbox, press "post", and later press "hire", and later press "pay/close".. how is this different from the oDesk interface itself? Well, eventually I want the system to automatically hire people.. before it does that, the main benefit will just be a convenient way to track my jobs separately from the other people who post jobs into the same teamroom as me.. and allow me to add little efficiency gains, like setting the defaults the way I want..
it sounds like I want another heroku/node.js/mongodb app.. it will take a day to build this.. I probably can't do it until my non-meeting days, which will start Friday or Saturday..
so in the meantime, it sounds like I need to either solve my slow-saving pima issue myself, or wait..
or.. I could post it on oDesk manually.. though then I would need to track it manually..
or.. I could have my assistant post it on oDesk for me.. let's do that.. e-mail sent.
how to find it.. I used to have an "eval" feature in pima. I may need to re-add that. Hm.. I should be humanscripting this.. I've lived with the long saving till now..
hm.. before that, I have some merging to do.. my current version is not in sync with github.. done
ok, let's humanscript this:
hm.. I need to see how it will run for someone starting from scratch, with no data..
let's try cloning the repo in a new place..
ok, I added some instructions to the readme..
let's try the humanscript again:
- clone https://github.com/dglittle/PIMA
- put these instructions in humanscript/hs_748569937/hs.txt
- see README to run it
- add "eval" button next to "save" button
- pressing "eval" button should:
var div = new jquery div
var top = new jquery div
top.append("run" button)
top.append("close" button)
var input = textarea with 100% width and height
var output = textarea with 100% width and height
div.append(_.splitVert(1, top, _.splitHorz(50, input, output)))
"close" button should call _.closeDialog()
"run" should eval the code in input,
where any calls to console.log will print out to the output textarea
_.dialog(div)
- test it
run it,
press "eval"
type: console.log("hi")
press "run"
should see "hi" in output
take screenshot
put in humanscript/hs_748569937
- check in
- make pull request to original repo
Ok, I want to post this, but my current system is too cumbersome. I'm supposed to paste this into a json file, and run some command line on it.. I want an interface running at some url where I can just paste my humanscript into a textbox, press "post", and later press "hire", and later press "pay/close".. how is this different from the oDesk interface itself? Well, eventually I want the system to automatically hire people.. before it does that, the main benefit will just be a convenient way to track my jobs separately from the other people who post jobs into the same teamroom as me.. and allow me to add little efficiency gains, like setting the defaults the way I want..
it sounds like I want another heroku/node.js/mongodb app.. it will take a day to build this.. I probably can't do it until my non-meeting days, which will start Friday or Saturday..
so in the meantime, it sounds like I need to either solve my slow-saving pima issue myself, or wait..
or.. I could post it on oDesk manually.. though then I would need to track it manually..
or.. I could have my assistant post it on oDesk for me.. let's do that.. e-mail sent.
drawing ephemeral
some friends and I are continuing the creativity/introspection sessions from CrowdCamp. We each spent 10 minutes trying to draw the word "ephemeral". Here's my sketches/notes:
3/6/13
self-programming patch
I wrote before:
- when frustrated, accept situation
- when angry, forgive all involved, including myself
- when sad, be sad
I think sadness may be the feeling produced when accepting frustrating or anger-inducing things, so:
- when frustrated, accept and convert to sadness
- when angry, forgive and convert to sadness
3/5/13
notes from CrowdCamp part 2
idea nuggets from CrowdCamp and CSCW 2013:
- a lot of creativity is "riffing". see this and this
- here's an interesting way to create a story: tell someone that you have a story in mind, and have them ask you yes-or-no questions about it. You can answer randomly, and they'll effectively create a story from their questions. see this
- from a key-note, and from someone's interpretation of the key-note:
- longer one-on-one interactions with people are more effective for one's reputation than lots of small interactions, even if the longer one-on-one interactions exclude small peripheral interactions. The reason is that reputation consists of stories people tell about you, and not the interactions themselves, and longer in-depth interactions inspire better stories.
- it is valuable to have people be creative in separate groups, and exchange ideas more rarely, like every year. I don't know if the following anecdote is true, but I got the impression that Steve Jobs had many groups at Apple design an iPhone separately, without talking to each other.
- this is the first speaker I've heard make fun of the people asking questions. To one person he said: "people who start questions that way haven't been listening, but have just been focussed on the question they want to ask". To another person he said: "have less of a preamble to your question".
- the Microsoft Kinect can quickly identify all the pixels in an image that are part of a human. see this
- the river walk is nice. possible reasons: the road is replaced with a river; there are escher-esque stairs; people are laid back; it feels like a cozy canyon.
- Winter Mason suggested that oDesk include offline non-remote work as well, e.g., managing the payroll of small companies that have on-site employees.
- Anand Kalmari informed me that oDesk has healthcare options for contractors, which I didn't know about, see this.
- Anand also showed me a demo of a system for hiring writers quickly on Mobileworks, and I told him about the humanscripting idea.
- I talked with a few other people about humanscripting, including: Nikki Kittur, Walter Lasecki, and Nicolas Kokkalis. The idea that seemed to resonate most with people was thinking about specifications imperatively rather than declaratively
- I saw Nicolas present the GmailValet paper; I hope this generates interest in valet-keys for more services
- I told someone that it was exhausting to talk to lots of people at parties at the conference. They said that's what alcohol was for. This finally made sense. Unfortunately, I had to drive, so I couldn't employ that strategy.
- I saw the future or crowdwork paper. I think everyone has similar visions of where things are going, and I'm sure we'll get there. Though, I was reminded that it may be worth coming up with a compelling positive vision for the future of work as a defense against potential negative reactive regulations.
- I heard Rob Miller talk about plans for expanding a nice paired-research system he's been experimenting with. The idea is to pair people up every week so they can help each other, and presumably cross-pollinate ideas. Some people external to his research group are on the list, including me, and he pitched the idea of having groups at multiple schools on the list, where the system would usually pair people with someone at their own school, but sometimes pair them with external people. This idea seemed linked to the key-note mentioned above.
- I had multiple discussions with Walter Lasecki, who is an interesting person.
- He experiments with extreme sleep schedules, which interest me. I expressed my concern that waking up is hard, and asked him if waking up was hard for him. He said it was like sticking forks in his eyes. But he manages.
- He has 12 undergrads working for him, and he's starting to hire people on oDesk as well. I want to pick his brain about his process. It seems like he is doing something right with regard to management.
- I'd like to spend some time with his group (Jeff Bigham's group) at Rochester. I've been there before, while interning at Xerox. In fact, I was wearing a sweater at CSCW that their group gave to me :) — in fact, the logo on that sweater was drawn by Kyle Murray, who is now in Rob's group at MIT, and will be interning at oDesk this summer, extending the humanscript idea, which he created. So, lot's of social ties.
- write more nuggets here when I remember...
- oh, I re-had an idea of extracting idea-nuggets from papers, and posting them, and trying to get the community to think in terms of idea-nuggets, rather than papers — assuming this is a good idea
speeding
While in San Antonio at this conference, I was a bit paranoid about drinking too much because I had to drive 30 minutes to my sister's house every day, which turned out to be a good policy, since I got pulled over for speeding — 76 in a 65 — and I got a sobriety test to boot.
He started with some questions about where I was coming from, which I answered accurately, but with a slowness characteristic of me, and he asked if I had been drinking. Yes. How much? Almost a whole beer. Are you sure that's all? Yes. How long ago? Two hours.
Then he asked, are you an educated man? Er.. yes. What is your highest level of education? Um.. phd. His followup to this: are you coordinated? Yes.
Then he gave me a sobriety test that I could perform in the comfort and convenient of my car. It involved flipping my hand over and over and counting. I was pretty scared, but I passed.
I asked a couple people about the test the next day. One person said "oh, that's hard, when you're drunk." The other person was drunk and performed the test.. he did very well, as far as I could tell.
I've gotten pulled over a number of times before, without ever getting a ticket.. two differences this time: I had drunk something that evening; I was in Texas.
The ticket is $190.
notes from CrowdCamp part 1
I arrived a few minutes late searching for parking near the river walk in San Antonio. I found the room where everyone was meeting, only to see everyone walking out of it with long roles of paper. I was given a role of paper myself, and found myself following the crowd to an open space in the Hyatt's large atrium.
Liz Gerber led us in an ice breaking session. Activities included making random suggestions of what to do with the long role of paper. Someone suggested that everyone hit me — fortunately someone else made another suggestion before everyone was able to follow through with that. We also ended up tracing each other's heads on the paper. Ice broken. Thanks Liz ;)
Next, we spent some time trying to divide into groups to hack on some project over the weekend — the main point of CrowdCamp is to hack on some project. I proposed a group that would spend its time trying to be creative, as a group, and introspecting about that process. My thought was: we often try to figure out how to get Mechanical Turk workers to approach expert levels of creativity through some sort of magical workflow. But I'm more interested in taking creative experts and trying to achieve super-human levels of creativity, and the people at this workshop seemed like "creative experts", by some definition.
The first thing our group did was have each person spend 10 minutes on their own with a pen and paper trying to create a logo for CrowdCamp. Here's what I did:
I drew a path through the notes just now to show the order of the sketches. I started by drawing "CrowdCamp" with a tent next to it. Then I experimented with drawing a stylized bathroom person next to a tent. I liked the idea of having stylized bathroom people engaged in normal camp activities, and drew them roasting marshmallows. I then tried to make it look like a crowd of bathroom people roasting marshmallows. I thought this approach might be getting to complex for a logo, and decided to move toward making the CrowdCamp logo look like a conventional camp sign. I looked on google images for some examples of camp signs, and tried to adapt them.
After we had all spent 10 minutes sketching alone, we each shared our process with the group. All four of us had a similar process: we drew sketches of ideas, and then riffed on those ideas with new sketches, and if we got stuck, we started with a new sketch. Here's another group member's sketches for this task:
I drew the path based on what they said — they started with traditional camp imagery, just like me, including a tent and marshmallows, but then decided to abandon that path. They restarted with a focus on trying to show "working together", and trying to use symmetry. This idea eventually led to a celtic braid of cables.
As we shared our ideas, I was inspired by other people's ideas, and wanted to riff on those. There is a dotted line in my set of notes above going to an image of a crowd of people hammering down each spike for a tent. Unfortunately I don't have the other group member's sketch that inspired me to draw that, but I liked the idea of showing a crowd doing something that really only required one person, because I thought it was funny.
We tried the logo creation task again later on, trying to create a logo for UIST — which is a conference we were all familiar with — but this time we did it twice. The first time we came up with our own ideas. Then we passed our notes to the person next to us and spent 10 minutes riffing on their sketches.
Here's a sketch done by me, where the words user, interface, software and technology flow down in a braid:
Here's a sketch done by another group member where they stylized my sketch:
Here's a sequence of sketches done by another group member:
And here are a couple sketches done by me based on those ideas:
Note that we didn't talk to each other about our sketches before exchanging them, and I wasn't sure what was going on with the eye in the hand, or with the broken computer. After we discussed everything, I learned that the eye, hand and sound-waves were meant to represent different senses, since user interfaces involve senses. Also, the broken computer was meant to represent "breaking free of old interfaces". I found these explanations useful, and I may have riffed on the ideas better if we had told each other what we were thinking before exchanging papers. In any case, I'm still proud of the guy with his head in a monitor trying to program himself :)
In addition to sketching logos, we did a 10 minute session on a couple of other creative tasks as well: writing a verse of a song, and thinking of a website to build that could facilitate creative introspection.
Writing the verse was interesting for a couple of reasons. First, we didn't specify what it should be, and so some of our time was spent drilling down to what our song should be about. This was a creative process in itself. Second, the task was somewhat uncomfortable for everyone, since songs are so personal, and people were — at least I was — embarrassed.
In any case, I feel like I learned a lot about creativity, seeing other people's creative processes. And by "a lot", I really mean that I grew to appreciate the importance of riffing on ideas. I feel like a lot of creativity is essentially hill-climbing with random restarts.
We didn't get a chance to explore too much into super-human creativity, though we did try exchanging ideas after thinking alone for 10 minutes, and that did seem better than just thinking alone for 20 minutes.
Liz Gerber led us in an ice breaking session. Activities included making random suggestions of what to do with the long role of paper. Someone suggested that everyone hit me — fortunately someone else made another suggestion before everyone was able to follow through with that. We also ended up tracing each other's heads on the paper. Ice broken. Thanks Liz ;)
Next, we spent some time trying to divide into groups to hack on some project over the weekend — the main point of CrowdCamp is to hack on some project. I proposed a group that would spend its time trying to be creative, as a group, and introspecting about that process. My thought was: we often try to figure out how to get Mechanical Turk workers to approach expert levels of creativity through some sort of magical workflow. But I'm more interested in taking creative experts and trying to achieve super-human levels of creativity, and the people at this workshop seemed like "creative experts", by some definition.
The first thing our group did was have each person spend 10 minutes on their own with a pen and paper trying to create a logo for CrowdCamp. Here's what I did:
I drew a path through the notes just now to show the order of the sketches. I started by drawing "CrowdCamp" with a tent next to it. Then I experimented with drawing a stylized bathroom person next to a tent. I liked the idea of having stylized bathroom people engaged in normal camp activities, and drew them roasting marshmallows. I then tried to make it look like a crowd of bathroom people roasting marshmallows. I thought this approach might be getting to complex for a logo, and decided to move toward making the CrowdCamp logo look like a conventional camp sign. I looked on google images for some examples of camp signs, and tried to adapt them.
After we had all spent 10 minutes sketching alone, we each shared our process with the group. All four of us had a similar process: we drew sketches of ideas, and then riffed on those ideas with new sketches, and if we got stuck, we started with a new sketch. Here's another group member's sketches for this task:
I drew the path based on what they said — they started with traditional camp imagery, just like me, including a tent and marshmallows, but then decided to abandon that path. They restarted with a focus on trying to show "working together", and trying to use symmetry. This idea eventually led to a celtic braid of cables.
As we shared our ideas, I was inspired by other people's ideas, and wanted to riff on those. There is a dotted line in my set of notes above going to an image of a crowd of people hammering down each spike for a tent. Unfortunately I don't have the other group member's sketch that inspired me to draw that, but I liked the idea of showing a crowd doing something that really only required one person, because I thought it was funny.
We tried the logo creation task again later on, trying to create a logo for UIST — which is a conference we were all familiar with — but this time we did it twice. The first time we came up with our own ideas. Then we passed our notes to the person next to us and spent 10 minutes riffing on their sketches.
Here's a sketch done by me, where the words user, interface, software and technology flow down in a braid:
Here's a sketch done by another group member where they stylized my sketch:
Here's a sequence of sketches done by another group member:
And here are a couple sketches done by me based on those ideas:
Note that we didn't talk to each other about our sketches before exchanging them, and I wasn't sure what was going on with the eye in the hand, or with the broken computer. After we discussed everything, I learned that the eye, hand and sound-waves were meant to represent different senses, since user interfaces involve senses. Also, the broken computer was meant to represent "breaking free of old interfaces". I found these explanations useful, and I may have riffed on the ideas better if we had told each other what we were thinking before exchanging papers. In any case, I'm still proud of the guy with his head in a monitor trying to program himself :)
In addition to sketching logos, we did a 10 minute session on a couple of other creative tasks as well: writing a verse of a song, and thinking of a website to build that could facilitate creative introspection.
Writing the verse was interesting for a couple of reasons. First, we didn't specify what it should be, and so some of our time was spent drilling down to what our song should be about. This was a creative process in itself. Second, the task was somewhat uncomfortable for everyone, since songs are so personal, and people were — at least I was — embarrassed.
In any case, I feel like I learned a lot about creativity, seeing other people's creative processes. And by "a lot", I really mean that I grew to appreciate the importance of riffing on ideas. I feel like a lot of creativity is essentially hill-climbing with random restarts.
We didn't get a chance to explore too much into super-human creativity, though we did try exchanging ideas after thinking alone for 10 minutes, and that did seem better than just thinking alone for 20 minutes.
more unicode pathways
I forgot a couple of unicode pathways to check:
- sending data in a GET request from chrome — check
- sending data in a POST request from chrome — check
- sending data in a GET request from node.js — check
- sending data in a POST request from node.js — failed
..the failure there is not node, but me. I was not sending the proper header information in my _.wget utility. Fixed. I should probably put nodeutil.js in the myutil github repo..
dance
I feel very good after dance lessons. The lessons are one-on-one, and sometimes another of her students joins in. I think I enjoy dancing in a small group the same way I enjoy talking in a small group.
3/4/13
more data munging
ok, the remaining 8 items are done.. now I've got to compile everything into a big pile of.. er.. a csv file.
(oh, I want to sing out loud.. I wonder if my roommate is away)
ok, let's run a giant query and store everything in a file for further processing..
note: beware that...
function hello() {
return "hi 1"
}
console.log(hello())
return
function hello() {
return "hi 2"
}
...will still print out "hi 2".
this was tripping me up because I had a giant function with test code, and I would effectively comment out everything beneath my current code with a "return" statement, but it wouldn't really be commented out if I was reusing a function name..
back to what I was doing.. running some sort of query.. done
ok.. large csv file created and sent.
(oh, I want to sing out loud.. I wonder if my roommate is away)
ok, let's run a giant query and store everything in a file for further processing..
note: beware that...
function hello() {
return "hi 1"
}
console.log(hello())
return
function hello() {
return "hi 2"
}
...will still print out "hi 2".
this was tripping me up because I had a giant function with test code, and I would effectively comment out everything beneath my current code with a "return" statement, but it wouldn't really be commented out if I was reusing a function name..
back to what I was doing.. running some sort of query.. done
ok.. large csv file created and sent.
data surgery notes
accounting for items.. I was processing some jobs, but there were some duplicate jobs, and I didn't do a good job of dealing with those, so now I need to do a bit of data surgery...
I have two batches of jobs in a single mongodb. I tagged the second batch with a "batch" field. I want to tag the first batch also.. done
not quite all the items in the first batch are done; there's about 8 remaining. People are working on the second batch, which is large, but I want to prioritize these 8 from the first batch..
..there are actually 5 items that aren't even uploaded, so let's upload those.. done
now to prioritize them, let's do:
db.records.update({batch:"first_batch_name", availableToAnswerAt : 0}, {$set:{availableToAnswerAt : -1}}, {multi:true})
..doing availableToAnswerAt = -1 should push them to the top, since we sort by that when showing people available jobs..
we also need to do it for availableToReviewAt:
db.records.update({batch:"first_batch_name", availableToReviewAt : 0}, {$set:{availableToReviewAt : -1}}, {multi:true})
I have two batches of jobs in a single mongodb. I tagged the second batch with a "batch" field. I want to tag the first batch also.. done
not quite all the items in the first batch are done; there's about 8 remaining. People are working on the second batch, which is large, but I want to prioritize these 8 from the first batch..
..there are actually 5 items that aren't even uploaded, so let's upload those.. done
now to prioritize them, let's do:
db.records.update({batch:"first_batch_name", availableToAnswerAt : 0}, {$set:{availableToAnswerAt : -1}}, {multi:true})
..doing availableToAnswerAt = -1 should push them to the top, since we sort by that when showing people available jobs..
we also need to do it for availableToReviewAt:
db.records.update({batch:"first_batch_name", availableToReviewAt : 0}, {$set:{availableToReviewAt : -1}}, {multi:true})
hm.. I'll probably need to run those queries again, in case someone picks up a task, and returns it, since it will set it's "availableAt" time to 0 instead of -1.. I could hack the code to do this for me, but that would require testing, and it's probably easier to just manually hand-hold this process..
hopefully these items will get done today..
oh..
and I have this same issue for a couple new batches, which I should fix preemptively..
ok, good, I think everything is in order now. probably. I thought it was in order before, but now I really thinks it's in order :)
oh..
and I have this same issue for a couple new batches, which I should fix preemptively..
ok, good, I think everything is in order now. probably. I thought it was in order before, but now I really thinks it's in order :)
fixing unicode mess
ok.. I have three systems running that are all failing to send unicode correctly in json responses.
first, let's patch them all, so they work going forward..
here's the change I want to make: 'application/json; charset=utf-8'
february-fire — done
nar-nar — done
third project which is really another instance of february-fire — done
next, let's try to repair the damage. There were some questions marked as "bad", which I'm guessing looked like unicode garble — of course, that means they were probably in a foreign language, so they probably are bad for our purposes, but let's make sure..
hm.. it looks like there is some unicode gibberish in the original questions — at least, when I load it as utf-8 — meaning that for at least some questions, there was a unicode mixup before the file ever came to me.. so.. that's good ;)
first, let's patch them all, so they work going forward..
here's the change I want to make: 'application/json; charset=utf-8'
february-fire — done
nar-nar — done
third project which is really another instance of february-fire — done
next, let's try to repair the damage. There were some questions marked as "bad", which I'm guessing looked like unicode garble — of course, that means they were probably in a foreign language, so they probably are bad for our purposes, but let's make sure..
hm.. it looks like there is some unicode gibberish in the original questions — at least, when I load it as utf-8 — meaning that for at least some questions, there was a unicode mixup before the file ever came to me.. so.. that's good ;)
unicode mess
I think I may have messed up some unicode. Let's check the unicode pipeline.
First, I'll need some sample utf-8 unicode. We need a unicode character that is definitely not ascii. How about something Chinese: "你好 world". I think that means "hello world".
Ok, let's put that in the Sublime Text editor, and try to save it as utf-8.. done.
Now let's try loading it and printing it to the console — we'll see if the mac's console supports unicode.. hm.. it does. great.
Now let's try printing it out as json, just to see what it does — I think I'd prefer if it escaped it with \u, but I don't think it will.. it does not escape it.
Let's try saving it back in a file as json, and loading it again as utf-8.. success
Great, now let's try saving it in mongodb, and see what it looks like in the mongodb console.. that works too.. so far, so good..
Now let's try getting it back from mongodb in javascript.. works..
Hm.. let's see if node.js assumes that javascript files are utf-8.. we'll put "你好 world" as a literal inside a javascript file and save it as utf-8, and try running it.. works.. I guess the default is utf-8.
Now let's try sending the json to a webpage using ajax.. FAILURE: when I point the browser at the json directly, it shows "ä½ å¥½ world". I assume that if it executed the page as javascript, it would also be wrong.
..how to fix this.. somehow the browser needs to be informed that this json is utf-8 encoded. I'm guessing this will be header information.. though I'm surprised since I'm using.. oh wait, I was going to say I was using an external library to form the json http response, but I'm not, so that makes it even more likely that I'm just doing it wrong.. let's look it up..
I found my answer in this stackoverflow question: What does “Content-type: application/json; charset=utf-8” really mean?
Success!!
..and, sadness that I failed before. But mostly Success!!
Let's make sure that I can load that with ajax and put it into a div.. yup.
First, I'll need some sample utf-8 unicode. We need a unicode character that is definitely not ascii. How about something Chinese: "你好 world". I think that means "hello world".
Ok, let's put that in the Sublime Text editor, and try to save it as utf-8.. done.
Now let's try loading it and printing it to the console — we'll see if the mac's console supports unicode.. hm.. it does. great.
Now let's try printing it out as json, just to see what it does — I think I'd prefer if it escaped it with \u, but I don't think it will.. it does not escape it.
Let's try saving it back in a file as json, and loading it again as utf-8.. success
Great, now let's try saving it in mongodb, and see what it looks like in the mongodb console.. that works too.. so far, so good..
Now let's try getting it back from mongodb in javascript.. works..
Hm.. let's see if node.js assumes that javascript files are utf-8.. we'll put "你好 world" as a literal inside a javascript file and save it as utf-8, and try running it.. works.. I guess the default is utf-8.
Now let's try sending the json to a webpage using ajax.. FAILURE: when I point the browser at the json directly, it shows "ä½ å¥½ world". I assume that if it executed the page as javascript, it would also be wrong.
..how to fix this.. somehow the browser needs to be informed that this json is utf-8 encoded. I'm guessing this will be header information.. though I'm surprised since I'm using.. oh wait, I was going to say I was using an external library to form the json http response, but I'm not, so that makes it even more likely that I'm just doing it wrong.. let's look it up..
I found my answer in this stackoverflow question: What does “Content-type: application/json; charset=utf-8” really mean?
Success!!
..and, sadness that I failed before. But mostly Success!!
Let's make sure that I can load that with ajax and put it into a div.. yup.
self-programming
when frustrated, accept situation
when angry, forgive all involved, including myself
when sad, be sad
when stuck solving problem, get more information
phone password
My five-year-old nephew borrowed my phone and said he knew what to do — I looked and saw the password screen and said "you don't know what to do in this part". But I was wrong. He knew my password. Somehow.
6:45
I woke up at 6:45 today and thought it was 6:45 pm, judging by the light, and how much I had expected to sleep.
Subscribe to:
Posts (Atom)