Ted Dziuba

Corporate Competence

2008-07-04T17:33:25Z

I really love it when people just do their jobs. I feel gifted whenever I call a company and get a customer support representative who know what they are doing and actually cares about me.

It's rare, but it happens.

Worst ISP Ever

For a while, I had Comcast's cable internet service. It was clear after two years of putting up with their horseshit that they don't care about customers at all.

Oh, wait, they set up a Twitter account.

Fantastic, but my BitTorrent shit still didn't work on their network. Their installation staff is rude and has questionable hygeine, and their customer support representatives are downright lazy.

Switch to AT&T Now

When I moved, my first order of business was to call Comcast and tell them it's over. They said my service wouldn't end until I brought back my cable modem, and of course, the place I need to bring it back to is only open during working hours.

I took off work early to get this little brick of dissatisfaction back to its rightful owner, because fuck them.

At the same time, I was waiting for AT&T to show up and install U-Verse internet service. They did, and shit was impressive.

They told me the tech would be at my house any time from noon to 2pm on a Sunday. The tech showed up at noon on the dot.
It took him about an hour to set up the service. When he left, he gave me a card with his direct cell phone number. If I had any problem in the next ten days, I called him directly and he would come fix it.
An hour after he left, the service went out. I called him, and he was back at my house within 30 minutes. It turns out there was something wrong with the line from the street to my house, and he had to get another tech out to fix it. That guy showed up, fixed the problem, and was on his way. The two of them were at my place until 8pm on a Sunday until the job was done right.

I've been using the service for almost a week now and it's great.

No BitTorrent fuckery. All my torrents work great, and I can seed.
10 megabits downstream, 1.5 megabits upstream.

Great job, AT&T, you actually care about the people paying your salaries.

Having Fun At Mahalo's Expense

2008-07-03T03:57:29Z

Holy crap this is awesome. Mahalo, the "human powered search engine", now lets the general public edit parts of its pages.

For the uninformed, Mahalo is a for-profit installation of MediaWiki founded by notable blowhard Jason Calacanis and backed by Sequoia Capital. They're aiming to be a hand-vetted search engine to compete with Google. Right.

Anyway, now you can edit their pages anonymously. Since they're a capitalist version of Wikipedia, they need to pay people to review edits, so they aren't too quick on the uptake. Here's how to do it:

Step 1: Search for something at mahalo.com

Step 2: Click the 'edit' link

Step 3: Add your verbiage. Remember, this is MediaWiki syntax, so you can link to other Mahalo pages.

Step 4: Hit 'save' and collect your winnings

A search engine that anyone can edit. What could possibly go wrong?

Practical Unique Identifiers

2008-07-02T05:16:13Z

There have been a handful of places within the Persai pipeline where I have needed unique identifiers of varying length. 64 bits here, 32 bits there. I'm not the only one to ever have to solve this problem, but I could never find a concise toolbox of information on it.

Automatic Increment or Not

MySQL has the AUTO_INCREMENT modifier for integral record keys. That's great, if you're using MySQL. In general, prefer a non-automatically increasing record identifier, unless you have a specific reason. Here's why:

You may actually have to think about thread synchronization at some point when creating records.
If these identifiers become publicly visible, they can leak information about how many records are in your database.
If you make identifiers out of other pieces of data (say URLs), then you can't get the identifier value of a given datum without a table lookup. And even then, you'll need another index on that field.

There are a few cases where automatic increment identifiers are good, though:

You are using a MySQL database and are setting up a simple structure of tables. (i.e. MySQL handles synchronization for you and it's actually harder to not use automatic increment)
The creation order of records is really important to you, but not important enough to store a timestamp field.

Making an Identifier Out Of Arbitrary Data

Easy, right? Just hash whatever data you've got. It's not reversible and spread uniformly over the identifier space. However, many times the output of a standard hashing algorithm is too big. SHA-1, for example, is 160 bits wide. Way too long for most purposes.

In this case, I truncate the output. Yes, this is mathematically valid, because any good hashing algorithm's output will be uniform over the range of the function. And by uniform, I mean really uniform. For example, if you take the first 64 bits of a 160-bit SHA-1 hash and call that your unique identifier, the probability of a collision is going to be uniform over the space of all 64-bit numbers. If it wasn't (i.e. the first 64-bits of a SHA-1 hash were distributed, say normally), then the hash function would be cryptographically insecure.

Don't try to swing your dick around and come up with your own hash function. You'll screw it up. I know I have.

GUIDs

I got an e-mail from a reader about using GUIDs for unique identifiers. This fits with the hashing scheme, but for the most part, I think GUIDs are far too large, especially if you are storing a lot of records. GUIDs are 128 bits wide, so if you have a hundred million records, that's about 1.5GB worth of identifiers. Use a 64-bit identifier, and your space is halved, without a significant increase in collision probability.

Making An Identifier Easier On The Eyes

If you need to put a unique identifier in a URL, it can't look too nerdy. For example, this URL looks like shit:

http://www.website.com/document?id=1b25a53bf21d0206

Too many numbers. So, to make it look better, Base-64 encode it. It will lengthen the code a little, but it's much easier to look at:

http://www.website.com/document?id=ZnJvc3RlZCBidXR0cw==

Eh, well it looks better to me. Personal taste, I guess.

You'll need to make sure that your Base-64 alphabet doesn't include the + and / characters: they aren't URL safe.

Sort Orderings

Don't worry about sort ordering unless you have to worry about sort ordering. Duh. The vast majority of Persai's data is stored simply as files, and for most purposes we don't have to care about the processing order. We're fortunate in that regard (well maybe not fortunate, I mean that's like saying you're fortunate that you're not fat because you exercise and eat sensibly).

Anyway, there are a couple of places in Persai where sort order matters. The ordering of recommendations, for example. There, though, we're just ordering by time, and we need to display the exact time, not just the relative times of the recommendations, so we store a date field and order data by it in the store.

This drives one of my earlier points home: if you need ordering by time, don't count on an automatic increment unique identifier to do it. It's much more robust to store a timestamp.

In fact this point goes deeper. Very rarely do you actually need records sorted by record identifier. What you need is the records sorted by some other value that happens to be reflected in the record identifier by virtue of automatic increment and the insertion order. It's always more robust to store the actual value you need to sort by.

I'm Not Going To Tell You How To Write Code

Because I don't really care. This is how I do it, though.

So Close Now I Can Taste It

2008-07-01T02:49:42Z

Matt, Kyle and I are wrapping things up for the public launch. I have spent the last week writing Nagios alerts for some really esoteric details of our system. We're sending Persai out into the world in July, if everything goes right. It has been a long time coming - the system has been in development for over a year.

That Ride Sucks, Anyway

Why is this thing taking so long to launch? We've had a lot of failures along the way, but we've finally converged on a system that I believe to be correct. The focus of the product has changed a little bit, too.

Our pitch used to be: a content filtering and delivery system that learns about what you're really interested in, and shows you news based on your feedback.

That doesn't sit too well with investors, it turns out.

Now, it's: a persistent search that learns the nuanced details of what you like.

That one picked up a few ears, and got a few pens working.

We ended up re-writing the internals of this beast three times before we had what's running today. Now it's lean and mean. Our classifiers went through a massive reduction in feature space: about six orders of magnitude. Some of our linguistics algorithms got ten times faster, and we are crawling thousands more sources.

The next couple of weeks are going to be very busy and very exciting. We will likely have a launch party, but we're not pissing away our investors' money on it. It'll be a barbecue in the back yard of the "office" with a bunch of beer. Maybe 111 Minna next time.

Datacenter Fire Causes Persai Outage

2008-06-01T18:32:45Z

A generator fire at a datacenter has caused Persai to be down since yesterday.

We'll be moving ourselves from that datacenter shortly - this is why the service is in beta.

Yeah, hosted applications are the wave of the future. Yessiree.

An Engineer's Guide To Weight Loss

2008-05-20T15:28:00Z

After a year of the Google cafeteria and another year of eating low cost Startup FounderChow, I put on a few pounds. Now, I'm starting to shed them, and I'll tell you how.

Before I get into it, I want to lay down a few prerequisites. There are a lot of diet guides out there that will bullshit you into thinking that the process is easy. This is a lie. Dieting and exercising suck. This is possibly the most miserable thing you can do to yourself. You are not going to have fun.

To that end, if you are more than 50 pounds overweight, are unmarried, have no children, and your only reason to get up in the morning is your shitty software job, the healthy lifestyle is not for you. You are better off eating yourself to the grave: you will get much more satisfaction out of life by eating cheeseburgers than you will by torturing the pounds of fat off your gut.

Simple I/O Operation

The science of weight loss is simple: eat fewer calories than your burn. You have heard this before, I trust. To follow this principle, you need to start quantifying. I use a web service called FitDay to track the calories I eat versus the calories I burn.

Start by running a 1,000 calorie per day deficit. To lose a pound of fat, you need to burn around 3,500 calories, so you'll lose two pounds in a week. Just to be clear, doing this sucks ass. However, there are a few easy ways to trim calories here and there.

Go Easy On The Drinking
When I go out on a Friday night to have a few beers, it's not hard for me to consume 800 calories worth of booze. Yes, liquor helps to numb the pain of writing XML parsers all day, but it comes at an expense. To compensate, take up smoking. I smoke more cigars now: it's a good zero-calorie alternative.

Eat One Serving
There are eight servings in a box of Barilla pasta. I used to eat half a box of pasta in a single sitting, 4 servings worth. Since you're counting your calories anyhow, you'll already be monitoring servings. You will also spend less money this way: since I started counting my calories, I've been spending 50% less per week on food. More money for cigars.

Drink More Coffee
Caffeine is an appetite suppressant. In large enough quantities, it can be used as an amphetamine. Drink up.

Ice Cream Keeps You Sane
Low-fat ice cream has around 120 calories per half cup. Fat calories keep you feeling satiated for longer. The Dreyer's brand (sold as Edy's on the east coast) doesn't suck that much.

After you have been limiting your calorie intake for two weeks, your stomach will shrink enough that it takes significantly less food to satisfy you. So that's step one: stop eating so damn much. Step two is exercise. And yes, it's awful.

See What Condition My Condition Was In

The only real benefit to exercise is being able to hold your nose over people who don't exercise. That's pretty cool if you're looking to take your misery out on your co-workers. Protip: it's better for your general well-being to be a prick to your colleagues than to your family.

You will lose more weight by dieting than by exercise if you are eating 1,000 fewer calories per day than you burn by doing nothing, so use exercise only as a supplement to your calorie loss.

If you're going to exercise, use an elliptical machine. Treadmills are terrible: they make you run. If you're like me, you have horrific flashbacks of being 10 year old, sucking wind, being the one that got nailed by the cops because your friends were all physically fit and managed to get away. Failure.

If you're going to pussyfoot around and work out for 30 minutes in your "fatburn" zone three times a week, don't even bother. You're just wasting your time. One hour per day, hard. You should be close to vomiting by the end of that hour.

Well, That's It

Easy, huh. Stop eating so damn much and get off your fat lazy ass.

Weekend Science Project

2008-05-10T23:37:36Z

I built an HD antenna today. Using these instructions and less than $15 worth of materials, I can get a few local channels in HD over the air. Check this sucker out:

Ugly as hell, but I can watch Lost in HD without paying Comcast an extra dime.

JavaFX Native Look and Feel

2008-05-10T17:14:02Z

I have been toying around with JavaFX, Sun's answer to Adobe AIR and Microsoft Silverlight. Since JavaFX is pretty much an easy way to do Swing, you can get Swing's pluggable look and feel in Java FX programs. Thank Christ, because the Swing UI components look like shit:

Versus the GTK look and feel:

This was done by adding the following snippet to my JavaFX code:


import javax.swing.UIManager;
 
 UIManager.setLookAndFeel(
            UIManager.getSystemLookAndFeelClassName());

Oh neat, native UI components.

Eclipse Crashes in Ubuntu Hardy Heron

2008-04-27T01:37:22Z

I just upgraded my workstation to Hardy and got a SIGSEGV when starting Eclipse. It appears to be a bug in the Sun JVM that ships with Hardy, and it only happens on AMD64.

If you're set on using this runtime, the fix is to disable the JIT compiler by launching Eclipse with -Xint, but that's comes with a severe performance penalty.

The fix I used was to simply downgrade the Hardy JRE (6-06-0ubuntu1) to the Gutsy version (6-03-0ubuntu2). You'll have to edit /etc/apt/sources.list to add a Gutsy repository.

I'm pretty sure this is the Sun bug: http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=6614100

And this is the Ubuntu bug: https://bugs.launchpad.net/ubuntu/+source/eclipse/+bug/174759

I'm Going To Scale My Foot Up Your Ass

2008-04-24T14:03:43Z

Engineers love to talk about scalability. It makes us feel like the bad ass, dick-swingin' motherfuckers that we wish we could be.

After we talk about scalability with our co-workers (Yeah, Rails doesn't scale!), we flex our true engineering prowess by writing a post about it on our blog. Once that post hits Reddit, son, everyone will know how hardcore you really are. Respect.

People Who Talk Big About Scalability Don't Need To Worry About It

Fact: every chest-thumping blog post I have seen written about scalability is either about architecture, Memcached, or both. Some asshole who writes shitty code starts pontificating about "scalable architecture" with data storage, web frontends, whatever-the-fuck. Dude, your app isn't having scalability problems because of the architecture. It's having scalability problems because you coded a ton of N^2 loops into it and you're too self-important to get peer reviews on your commits.

And let's not forget the tools who discover Memcached for the first time, install it on a web server, and notice how fast their app runs now. Yeah, welcome to the modern age. Hope you know what a cache expiry policy is.

If You Haven't Discussed Capacity Planning, You Can't Discuss Scalability

You don't need to worry about scalability on your Rails-over-Mysql application because nobody is going to use it. Really. Believe me. You're going to get, at most, 1,000 people on your app, and maybe 1% of them will be 7-day active. Scalability is not your problem, getting people to give a shit is.

Unless you know what you need to scale to, you can't even begin to talk about scalability. How many users do you want your system to handle? A thousand? Hundred thousand? Ten million? Here's a hint: the system you design to handle a quarter million users is going to be different from the system you design to handle ten million users.

Of course you'll point to the engineer's wet dream: linear scalability. Lulz but when we get more users we just add more machines you are so stupid ted. uncov sucks.

Yeah, great, well it doesn't exist. Oh no, go ahead and try out Amazon SimpleDB and think to yourself that it will scale linearly. Then, when you get enough users that the latency becomes a problem, blame it on "those shitty Amazon datacenters".

Choosing Technology Don't Mean Shit If You Don't Know How To Use It

The most common butthurt about scalability is this: choose a technology. If you like the technology, claim "technology X scales better!" If you don't like it, claim "technology X doesn't scale!"

Saying "Rails doesn't scale" is like saying "my car doesn't go infinitely fast". Alternatively, saying "We'll have no problems scaling because we're using Django" is like saying "I will win every race because my car is the most powerful". Maybe so, but you suck at driving, and you're up against professionals.

If you're having scalability problems and blaming it on a single technology, chances are, you're doing it wrong.

tl;dr

Shut up about scalability, no one is using your app anyway.

Don't Serialize Java Objects In Hadoop SequenceFiles

2008-04-08T15:28:29Z

Not if you can avoid it, at least.

Hadoop provides you with the Writable interface if you want to write your object to a SequenceFile. It's up to you to implement the write() and readFields() methods for your object. It's easy if your object is simple: just write each of your instance variables to a DataOutput and read them back in the same order from a DataInput.

Don't Write Your Object As A Serialized Byte Array

I got lazy when I was implementing the Writable interface with one of our classes because it had a ton of instance variables. I figured I'd just serialize it to a byte array, then write the array length and the whole array to the DataOutput. And on the read, well, just unserialize the object from the byte array. This was my write():


@Override
public void write(DataOutput out) throws IOException {
	ByteArrayOutputStream byteOutStream = new ByteArrayOutputStream();
	ObjectOutputStream objectOut = new ObjectOutputStream(byteOutStream);
		
	objectOut.writeObject(getContainedObject());
	objectOut.close();
		
	byte[] serializedObject= byteOutStream.toByteArray();
		
	out.writeInt(serializedObject.length);
	out.write(serializedModel);

}

Naw, dude. Bad idea.

I knew that I'd be paying some overhead in both space and time for this little scheme, but I didn't know how much. It was just a little bit per object, but when we started seeing MapReductions take way too much time in I/O, it was time to revisit this.

What This Cost In Space And Time

First, the Java serialization space overhead. On a toy example of this object, serialization to a byte array used 953 bytes. Properly writing out the instance variables consumed 296 bytes. In production, doing it the right way shrunk a 1,600-record SequenceFile from 1.4GB to 825MB.

Time savings were great, too. In the same toy example, it took my JVM 7.2 milliseconds to serialize the object and 1.7 milliseconds to unserialize. Doing with with stream I/O only took 76,000 nanoseconds to serialize, 58,000 nanoseconds to unserialize.

I love order-of-magnitude improvements.

Lesson learned: get off your lazy ass and do it right.

Plugged in the New York Times

2008-03-20T19:34:18Z

Me, Persai, and Uncov got a plug in the New York Times today. We've been in VentureBeat, Slate, and now NYT, but not TechCrunch. Something tells me that's not an accident.

A Magic Elixir

2008-02-28T03:47:14Z

In software, there are no silver bullets. In internal combustion engine mechanics, however, there are plenty. And I just discovered one.

It's called Sea Foam, and it will cure what ails 'ya.

My wife's first motorcycle was a Honda Rebel 250. She upgraded too late in the season and couldn't sell the starter before winter showed up. Winter time is a dead zone for the used motorcycle market in the San Francisco Bay Area, so the Rebel sat in the parking garage for 5 months. Being lazy, I didn't properly store it.

We went to fire it up yesterday to prepare it for its 15 minutes of Craigslist fame, and it wouldn't turn over. Gasoline, if left long enough, will degrade into a mucky varnish that cakes the inside of your carburetors.

I poured half a can of Sea Foam into the tank and let it sit for a few minutes. I cranked it again and it made a few pathetic putts. A few more cranks, a few more putts, but after about 5 tries, the Rebel roared to life.

A six dollar bottle of some petroleum distillate has the same end effect as a three hundred dollar carburetor job.

I am detecting much win in this sector.

Amazon EC2 Is Half As Fast As It Should Be

2008-02-23T23:30:27Z

Update: This post is made of fail. I trusted other peoples' benchmarks instead of doing my own. If you want details, go read Don MacAskill's butthurt response on the SmugMug blog.

The fun part about running a virtualized server environment on a heterogeneous hardware setup is that you can play word games with the specifications. Let's take, oh, I don't know, Amazon EC2 for example. This article is going to be all "science", but the takeaway is this: in Amazon's EC2 environment, you only get half of the CPU performance you would expect.

When EC2 launched, the specifications for the machine were "the equivalent of a 1.7GHz x86 processor". Crappy by the day's standards, but only a dime per hour. Fine.

As EC2 developed, Amazon came up with the idea of the "Compute Unit" to describe the power you get out of the instances. From the documentation:

One EC2 Compute Unit provides the equivalent CPU capacity of a 1.0-1.2 GHz 2007 Opteron or 2007 Xeon processor. This is also the equivalent to an early-2006 1.7 GHz Xeon processor referenced in our original documentation.

I won't get nitpicky about how they think that 2006 megahertz are slower than 2007 megahertz; I just want to show how nebulous the specification is.

Processes Run Half As Fast As You Think They Should

I figured this little nugget out last week. I had failed a piece of code to the point where an infinite loop would happen in an edge case. It took a while to debug this because looking at top in the EC2 instance only showed 40-50% CPU usage.

At the same time, I noticed another metric, the CPU %st, hovering around 50%. This stands for "CPU time stolen", and you'll notice that as your CPU usage rises, so does steal-time. What exactly does "stolen" mean? Time is stolen when your instance requests CPU time but the Xen virtualizer chooses to give that CPU time to something else, such as somebody else's instance.

I am not the only one to notice this, either. There are threads on the AWS forums by other users who are seeing their code run half as fast as it should, given the specifications for a Compute Unit. These posts are met by dismissive replies from Amazon employees. Great job.

How They Can Get Away With This

I think that many EC2 users are more I/O bound than CPU bound. If you make a simple Rails app backed by MySQL, chances are you're not going to consistently burn the CPU, so you won't even notice the slowdown. However, if you have some work that is CPU bound, this restriction becomes painfully obvious.

They also get to mince words with the equivalent-to metric. You don't actually get 1.7 billion clock cycles per second. This comparison is made by considering the machine as a whole: disk controller speed, memory bus speed, and evidently to a much lesser extent, CPU speed.

After doing Uncov, it's hard for me to tell the difference between a swindle and incompetence.

Good Persai Writeup on Slate

2008-02-20T23:25:39Z

A controlled experiment to test its learning ability:

http://www.slate.com/id/2184810/