Will be demonstrating Cisco’s WAAS tomorrow in NYC, so today we spent some time going through a testing protocol so we can show people different things.

After we finished we had dinner at Murphy’s in NJ. Strange place. It’s not a classy steakhouse or anything - nor does it have aspirations to be one.

The menu is, to quote Kipling, as immutable as the hills. Apparently any substitutions or deviations are swiftly and sternly stamped out, as though they signify an impending revolution that threatens all that we hold holy. Dressing on the side? Heresy! Burn!

I got the 24oz Delmonico. I was urged not to ask anything about it, lest they bring out someone to take me to the back. He also suggested generous amounts of A1.

At least it was inexpensive (about $17) and properly cooked. If you’re looking for flavor and marbling, look elsewhere. Much of it looked like solid marble, though. Had to surgically remove a good amount of gristle.

Better than the steak at Bowling Green, I have to admit.

D

Well, aside from visiting Flames and helping fix a severe customer problem. Those were rewarding. I still haven’t pooped that steak, BTW.

I was supposed to only stay for 1 day in Manhattan, fix the issue, ba da bing. I ended up staying an extra day - had no extra clothes and no time to get anything. Washed my undies on my own and used the hair dryer over a period of hours to dry them. I learned my lesson now and will always have extra stuff with me.

So I try to go back home today and guess what - Air Traffic Control computers had a major glitch (abcnews.go.com/Business/wireStory?id=3259992) that messed up the whole country’s air travel. Thousands of flights delayed and canceled. Mine was canceled, after I spent about 10 hours in the airport. Another 2 hours in the line to simply rebook the flight since they had 3 people trying to serve hordes. And all because, at least according to the report, a system failed and the failover system didn’t have the capacity to sustain the whole load.

So, while I wait in the airport to catch a stand-by flight tomorrow morning, unbathed and frankly looking a bit menacing, I decided to vent a bit. No hotels, no cars.

Maybe this is too much conjecture and if I’m wrong please enlighten me, but let’s enumerate some of the things wrong with this picture:

1. First things first: While it’s cool to fail over to a completely separate location, typically you want a robust local cluster first so you can fail over to another system in the original location.
2. If the original location is SO screwed up (meaning that a local cluster has failed, which typically means something really ominous for most places) ONLY THEN do you fail over to another facility altogether.
3. Last but not least: Whatever facility you fail over to has to have enough capacity (demostrated during tests) to sustain enough load to let operations proceed. Ideally, for critical systems, the loss of any one site should hardly be noticeable.

According to the report none of the aforementioned simple rules were followed. Someone made the decision to fail over to another facility, which promptly caved under the load. A cascade effect ensued.

I mean, seriously: One of the most important computer systems in the country does not have a well-thought-out and -tested DR implementation. Guys, those are rookie mistakes. Like some airports having 1 link to the outside world, or 2 links but with the same provider. Use some common sense!

So, I guess I’ll put that in the list together with using what’s tantamount to unskilled labor securing our airports instead of highly trained and well-paid personnel that’s been screened extremely intensely and actually takes pride in the job. Maybe some of those unskilled people are running the computers, it might be like the Clone Army in Star Wars. A mass of cheap, expendable labor that collectively has the IQ of my left nut (I’m not being overly harsh - my left nut is quite formidable). The armed forces heading the same way isn’t the most reassuring thought, either.

Yes, I’m upset!!!

D

Not amazing news but an official announcement nonetheless: Saw this (www.macnn.com/articles/07/06/06/zfs.in.leopard/) and I couldn’t resist posting. This means a few things:

1. Sun figured out how to make ZFS bootable (at least on OSX)
2. Someone figured out how to deal with ZFS and resource forks (I can’t believe they are willing to break compatibility with so much software otherwise).

Now I just need a Mac so I can run some benchmarks before and after. I have some buddies that might oblige… finally the Macs get a decent FS.

Now if only Apple could lose the silly Mach legacy, it’s a common misconception that the kernel in OSX is FreeBSD - it ain’t. Run lmbench (www.bitmover.com/lmbench/) on different platforms and compare results such as context switching, thread creation and whatnot. Then you’ll see why OSX can’t always make a decent server OS.

D

I was helping a client in the Wall Street District today with some rather obscure CIFS performance issues (Opportunistic Locks anyone? Berzerk BDCs causing issues? Multi-user Access DBs over WAN?)

Had to stay overnight (unplanned) so after putting in some solid hours I decided to get some steak, and NYC is the place to get decent steak.

Did some research and found out that Flames was walking distance from my hotel, so I went.

Got a T-Bone this time (usually go for strip or ribeye but the waiter insisted, even though they had far more expensive cuts on offer). Some creamed spinach and a small salad and I was set.

Flames is one of those fancy places where they cut your steak for you. At least they don’t feed you or, indeed, help you masticate.

Not that they would need to - the dry-aged steak had fantastic flavor and was reasonably tender (not the most tender but good). I wish it had been a tad less cooked but it was still great, and I devoured it in atavistic glory, almost beating the man-pelt on my chest in ecstasy. It’s been a while since I’ve had proper dry-aged beef.

The creamed spinach wasn’t too creamy or salty. The salad was just OK, I typically use salads for intestinal lubrication anyway and it served the purpose.

I did overhear some patrons asking for well done steaks, this is one of those places where they won’t try to talk you out of it, sadly. I think steakhouses should make you actually sign a waiver if you want to commit such culinary atrocity.

I also overheard a waiter trying to sell some $100 “Kobe” steak to some ladies, telling them how they massage the cows 4 times a day. I discreetly shook my head at them and they got the message.

Anyway - long story short, strongly recommended, and don’t dare order anything beyond medium-rare.

Now back to washing and drying my Superman underoos - I had no change of clothes and I’m writing this naked. It kinda is an appropriate image for this review though…

D

I’ve known about IBRIX for a while, but it was refreshing to talk to a decent techie that knew the product. They have improved it a lot over the past year.

For the uninitiated, IBRIX can be either

1. A network-based filesystem using the IBRIX client and protocol
2. Also accessible using NFS or CIFS
3. SAN-based parallel filesystem

The product’s claim to fame is it’s scalability and performance (realized by adding extra nodes “hot”). Their most famous client is probably Pixar, they replaced a ton of NetApp boxes with an IBRIX cluster and realized huge performance benefits and vastly reduced costs. I always liked cool filesystem technologies and this definitely falls under the realm of “cool”. Some highlights based on notes I took on my Blackberry during the session and questions I asked:

• No limits on filesystem size (they have deployed single namespace filesystems several PB in size).
• 300mb/s read, 200mb/s write on small box per node. Bigger boxes can do 1.2GB/s per node, of course your storage needs to be able to keep up.
• No limit on the number of nodes.
• Automatic rebalancing of data over time. When you add new disk you rebalance to keep things humming.
• Dedicated ibrix backup node, works with 3rd party backup SW, can have many backup servers for backup speed.
• Has snaps now (global), this was a failing of the product before since it was lacking snapshots.
• No real limit on the number of files per FS.
• Biggest file size they have tested on production is an 8TB file, no software limit.
• Nodes use FC to access storage, clients use Ethernet.
• Client on Windows or Linux, otherwise general NFS and CIFS. Client is fastest.
• Your prod servers can be the ibrix nodes but very compute-intensive. They recommend the client (IP-based, bonded). or get an 8-core box.
• There is no single lock manager - this is the coolest thing. There is global metadata and global locking, all nodes participate equally.
• How are node failures handled? All nodes interchangeable. All see same storage. Storage allocated to remaining servers if you lose a node.
  Can lose all but 1 server.
• Back-end storage size per node? Unlimited.
• Multipathing per node? Powerpath works. Can do bonded GigE up to 8 ports per.
• How are files allocated? The file inode contains the info concerning which node it needs to go to. Round-robin allocation or preferred servers per file type. Also if server over 50% full then it’s skipped.
• All volumes accessible by all nodes.
• Can stripe huge files across many nodes.

I’m stoked! I can think of so many uses for this product:

1. Data mining
2. Digital media
3. Oil and gas
4. Backups

D

I had some great customer meetings in OH this week. One meeting took me to Bowling Green, cute town.

The locals like to eat steak at Trotter’s Tavern. They only serve fist-sized and -shaped chunks of sirloin in some weird sauce that has at least some Worcestershire in it but is more tangy. No other cut choices, you get either 10 or 16 ounces and that’s it.

I asked the waitress how it was aged and got a blank stare back. I could almost read her mind: “we just defrost it in the microwave”.

Well, had it been cooked properly it might have been OK, but mine was well-done (which I hadn’t asked for). Ate it anyway, as is my idiom, but I can’t say I recommend the place. Maybe if you get the 10-ouncer and ask for medium rare it might be medium by the time you get it. It’s tough to cook a thick piece of meat properly.

At least the place is relatively inexpensive, their most expensive piece is $25 and comes with all the trimmings.

There was one weird thing though: The restroom was festooned with carvings (yes, carvings) asserting the gayness of various people.

D