November 2008 Archives

I took a new job at a software company in Palo Alto, which means an 80 mile commute every day through Bay Area combat traffic.  The first two weeks wore hard on my motorcycle - a 14 year old Ninja 500.  Last week on my ride home, the left turn signal stopped working.  Fuck.

If you thought debugging a software problem was hard, try debugging a hardware problem.  There are some salient facts about hardware problems that make them a real bitch:

1. You need to buy a lot of tools.
2. There's a real possibility that you will fuck something up beyond repair.
3. There's a real possibility that you will injure yourself.
4. If it's your primary vehicle, you need to have it up and running on Monday.
   

A blinker light stops working, which means electricity isn't flowing.  Sounds easy, but to get access to the wires, I needed to take the whole damn thing apart:

I'm working with a startup now on a text summarization project.  The requirements are fairly loose: "take all this text and make it smaller", solving the tl;dr problem (too long; didn't read).  There are a couple of critical details, namely identifying the sentiment of the text, and a few others that are excruciatingly domain-specific.

At first glance, this seems approachable with some natural language processing libraries.  Oh no.  There be dragons.  At Pressflip, I had myself into a few NLP libraries, and the only takeaway I got from all that experience was "Don't use NLP.  Ever."

Why?  NLP is yucky.  It's complicated, the field is rife with academic shitheaddery, there are some major-asspain licensing issues with a couple of software packages and best of all, it's balls slow.  Plus, if you venture down the road of natural language processing, the law of diminishing returns will pull you into a dark alley, pummel you with a tire iron, take you wallet, and then just to be a prick, steal your shoes so you need to walk home barefoot.

My point is, for 99 practical projects out of 100, you can cheat your way out of NLP.  Cook up some fancy shit with word frequencies and logarithms.  Reach back into your information retrieval notes for inspiration.  TF*IDF can take you a long way if you know how to use it.

When I was brainstorming the project I'm working on, my first thought was some hand-waving business about a part-of-speech tagger and a Markov Chain to figure out probabilities of part-of-speech transitions and all that fancy shit.  Factor in a little bit of sentiment detection from God-knows-where and that was my sketch.  Then practicality set in: how much time do you want to spend on this?  If you are considering NLP as the answer to a real problem, it's virtually certain that you're overthinking it.

That being said, NLP does have its place: making the best fucking Wikipedia search engine there ever was with technology licensed from Xerox and then selling yourself to Microsoft.

Ubuntu upgrades always, always, always fuck up the same things: network connections and graphics drivers.  Without fail, if you upgrade, your wireless connection won't work and any closed-source video card drivers you need will get ill.  8.10 Intrepid Ibex is no exception.

I figured out how to get my wireless connection back, but the NVidia drivers are still a mystery.  I don't care about free software idealism, I care that my shit works.  I'm willing to jump through minor hoops to make it work, like Ubuntu's "restricted drivers" lecture, but now that doesn't even work:



Why is this such a pain in the balls?  I press the activate button and nothing happens. AWESOME. Just because I'm used to wasting hours fixing things like this doesn't mean I enjoy it.

NVidia has 70-ish percent market share of all GPUs.  If your shit doesn't work out of the box on 70% of the graphics cards out there, who has failed?

[Update, the next day]:  I found the fix.  Navigate to System > Administration > Synaptic Package Manager.  From there, go to Settings > Repositories.  In the "Ubuntu Software" tab, check the "Proprietary device drivers" box.  Or edit /etc/apt/sources.list if you want to show your chest hair.

I'm glad to see that passive-aggressive Debian superiority is alive and well.