Wes Maldonado: Data Junkie

The new Ruby on Rails patches by Acunote/Nimble Method are speeding up Ruby on Rails by a claimed 5X… a friend of mine tried just the ActiveRecord::Base change and reported this:

Yup. I actually just did the ActiveRecord::Base change alone and got a good boost. The code change is pretty solid too. That led me to seriously look at some other code as well. If you happen to use the actsasmodified plugin anywhere, let me know, I have a simple fix for it as well. Between those two patches, my tests run about 20% faster.

I am a certified dork because papers about boarding airplanes interest me.

• Analog Concurrency: Boarding a Plane

• Linked from the above in the comments is this paper: Novel Approaches to Airplane Boarding (PDF Link)

Fractal: A mathematically generated pattern that is reproducible at any magnification or reduction.

At a seminar called Lean-Agile Software Testing – Practices and Challenges Jean McAuliffe described lean/agile processes as a fractal. She was talking about applying agile principles at each iteration of development. Principles are hard to apply because they are nebulous and numerous. What is the formula for the agile fractal?

business value = f(Cost, Time, Functionality, Quality)

In reality the formula is a wonderfully complex optimization problem.

Business Owners Perspective

max(business value) = f( min(cost), min(time), max(Functionality), max(Quality) )

And to the agile team we rename a couple of things…

Agile Team Perspective

max(business value) = f( max( Size_story) , min(Size_story * Risk_story), max(Unit Tests), max(Acceptance Tests) )

And when it really comes down to it, you have to try to solve both formulas simultaneously for max(business value):

max(business value) = f(x)

where x {

f() { min(cost), min(time), max(Functionality), max(Quality) },

f() { max( Size_story) , min(Size_story * Risk_story), max(Unit Tests), max(Acceptance Tests) }

}

There are only many variations on the theme for agile fractals, let me know about yours.

It’s time to re-examine those long running batch jobs. Could you partition the data to allow for MapReduce? I bet you can. I know I’ve always wanted an affordable way to fire up 30 servers and run MapReduce operations against giant datasets, it’s confirmed; I’m a dork.

Tom White sent me a note this week to inform me that he had implemented a Hadoop file system on top of S3. This file system can be used as a full or partial replacement for HDFS, the Hadoop Distributed File System.

Because bandwidth between EC2 instances and data stored in S3 is not metered or billed, this is a very cost-effective way to process large amounts of data.

Hadoop Filesystem Using S3

This is a good reason to remember that no everyone has a fast connection also.

I got an email from ORCSWEB, my most excellent hosting company (check them out) that I’d used over 230 GIGs of bandwidth for the month. Oy. First, I was happy that the site is doing well, then I was disturbed. Something MUST be wrong.

Notice anything odd there? Yes, my favicon.ico used 27 GIGS of bandwidth in the month. Yikes.

Turns out my icon was 70k, as I made it a wonderful high-quality “Vista” icon in an attempt to make things pleasant and everything for folks. Of course, I didn’t noticed when it was taking up over 11% of my monthly bandwidth.

FavIcon.ico can be a bandwidth hog

Using 230G of bandwidth is impressive. 25G for favicon.ico is just amazing.

A notice to readers: I’ll be updating my FavIcon.ico to a 250K icon later today.

Most people probably will not run into a situation where boxing and unboxing are going to really matter in their applications, others do. Read this post to see how a module’s heap usage went from 103MB to 16MB.

Take the TagLib.ByteVector class. It serializes many formats of data into a collection of bytes. However, it was using the System.Collections.ArrayList class to store these bytes! As a byte type is stack-based data, and ArrayList stores heap-based data, each stack-based member of the collection must be boxed for storage and unboxed for retrieval.

Simply by replacing ArrayList with List, the excessive memory problem goes away, since the type to be stored in the collection is known at compile time, and thus boxing/unboxing is no longer necessary.

http://abock.org/2006/10/30/cracking-down-on-heap-abuse-part-1/