Obviously it’s not a Michael Bolton CD, so I chose the Foo Fighters and then was greeted by this screen:

It all looked fine until I got to Track 7, and then I started to get a little worried.  Fortunately, this CD was not a mix of Foo Fighters and Michael Bolton and all I had to do was manually rename the tracks.  CD Lookup Fail!

For those in the crowd, Digg is a social news aggregation site where articles are submitted and then the general populous “Diggs” them up or down.  The algorithms are unknown and changing, but the general idea is that enough people Digg your stuff fast enough, you end up on the front page of Digg and thousands/millions of people visit the site in a short time.

This sudden increase in traffic often causes servers to crash when people visit sites that don’t have the architecture to support the huge quantity of requests.  This is the Digg Effect.

I used to follow the technology category on Digg, but stopped after a few months, when I couldn’t stand the ridiculous (and often vulgar) comments left by Diggers when they would Digg up and down.  There seemed to be a few intelligent voices in the crowd, but for the most part, it’s just stupid people saying even stupider things.

So I’ve started calling it the Real Digg Effect.  The real effect is that the world fills up with anonymous comments made by people who didn’t think twice before typing and submitting.  This is what Seth Godin referred to in a recent post called Monkeys With Megaphones.  Seth’s point is valid, but it assumes that you’ve got the time to sift through all the trash.  Do you have the time?

It was the same calendar item each time and even though I had my settings to continue syncing on failure, that didn’t happen, so I couldn’t get my whole calendar to sync anymore. A friend at work has been using SyncMyCal quite successfully for awhile, so I finally went that route. Even though it is $15 more than the $10 gSyncIt, I figured the price was still nice to get my calendars synched without having to deal with errors all the time. I want my technology to work and get out of my way.

Not 20 minutes after making my purchase and registering SyncMyCal, I talked to another friend here at work who has been using Plaxo to sync his contacts and calendars with Google. Plaxo is Free. Buyer’s Remorse sets in.

In addition to syncing your calendars and contacts between every conceivable operating system and program, Plaxo is also integrated with a number of web applications most notably for me, Last.fm, Flickr, del.icio.us, etc.

After waiting for 4 weeks to make my purchase, why couldn’t I have waited for another 30 minutes? Buyer’s Remorse.

Problem now is that I can’t seem to remember any of the ones I looked at before, and I’m really drawing a Google Blank. Any help from the masses?

I’m excited for this version of WordPress however, since it now has native support for tags. This is something that has been in Typo for ages, but the lack of a really good/cheap Rails host (well, maybe it’s laziness and lack of willingness to try different hosts) means that I had to wait for WordPress to include it before I could use it.

Yes, there were plugins before, but I try to keep the mgmt of my site to a minimum, and stay out of the code as much as possible. I consider myself fully capable of diving into the code and the hardcore database stuff, but when you’ve got better things to do, why spend the time doing it?

Now that I no longer have homework to do, my spare time has dramatically increased.  I spend more time at work than I ever spent at school, but the time is well spent and when I’m done working, I’m done for the day.  What will I do with all my leisure time?

There are a few plans, but at the top of the list will be the creation of a new domain.  I’m going to move all my web analytics related content to a new domain (still to be decided upon).  Since all my rails work is centered around web development, that will go along with the web analytics stuff to make a new Rails/Web Development/Web Analytics/SEO Dabbling blog.

This blog won’t be going anywhere, but it’s going to become more about me, my life, and my computer dabbling hobbies, wading into various *Nix distros and griping about software I don’t like.  So once I get the new site up and running, I’ll let you all know where my intelligent sounding posts are going, and this one will be for my rants and hobbies.

I’m working at Omniture now, just got all my benefits papers filled out and submitted and looking forward to the completion of training so I can actually get to work.  This is going to be fun.

I’ll just throw out the warning up front that this is not a typical blog post. For my Information Architecture class, I’ve been instructed to condense a semester’s worth of learning into one blog post.

So my goal with this post is not to rehash all the research and work I’ve done throughout the semester, but to make a good parts version and wrap everything up. So here we go in no particular order. I wrote a random number generator to sort them for me. A real random-number generator, not a psuedo-random number generator.

Lesson #1: Ruby is not a toy. And Rails is not a buzzword.
Dr. Liddle, I can already see you shaking your head and smiling as you read this. To borrow a phrase, you’re thinking that all I’ve got is a hammer and everything looks like a nail.

While I admit that there may be a bit of truth in it, the fact is that I really don’t know that much about Ruby or Rails. But there are very smart people in the world that know just about everything that there is to know about it, and they’re working very hard to make it better.

It is not a solution for every problem. I’ve never seen a decent client-side GUI built on Ruby, but when you add Rails and throw it onto the web, it becomes a very powerful, very simple, and very stress-relieving application platform. It makes AJAX very simple. It teaches good practice. It has good separation of components. It has a very active friendly community around it. No it is not the solution to everything, but it is going to go places, and you should pay more attention to it instead of just smiling and shaking your head whenever I bring it up =)

Lesson #2: Your mileage may vary.
During this semester, I’ve spent countless hours on the internet, looking up different solutions to different problems, to put together programs and projects for technologies I’ve never even heard of before.

Even though I ran into the same problems and error messages that others had run into, their solutions did not always work for me. There are lots of folks that Google themselves onto my blog, and while I hope that my solutions work for them, that assumption requires a certain amount of naivety, and it’s not an assumption I’d make.

Sometimes an authoritative answer cannot be found, so we turn to the blogosphere. The so-called Web 2.0 movement has enabled millions of people to build themselves a spot on the internet and granted access to amazing information resources, but just keep in mind where this knowledge is coming from. Don’t be surprised when someone else’s solution does not work for you.

Lesson #3: More servers and computing power is not always the best answer.
Sometimes when you are presented with the same old problem, the same old answer is not the best one. I wrote a paper about Memcached this semester. It’s an innovative way to solve the ages-old problem of scalability.

Obviously when you get bigger, you’ve got to scale up, but the lesson taught by Memcached is that up is not the only direction to scale. Proper use of innovative technologies can enable you to scale out as well as up. And Memcached is free, so throwing more money on the fire isn’t always the best solution either.

I actually just ran across an article on Digg that provided instructions for surviving The Digg Effect. The article didn’t mention anything about buying new servers, but rather instructions for tuning your server, optimizing your DB, caching your results, and tweaking your PHP (read programming language of choice). That’s all free too.

Lesson #4: A Gentoo installation should not be undertaken in the middle of a busy semester.
This is pretty much self-explanatory.

Lesson #5: There is no perfect one-size-fits all solution.
There are a million ways to architect a system to support your application. Maybe Model Driven Architecture will fit your needs. Maybe it’s an extensible Service Oriented Architecture. Maybe it’s a simple Web Service that you publish for others to consume.

The point is that what works best for someone else, may not be your best option. Figure out what you really need, do your research, talk to people who know what they’re doing, and decide what works best for you.

Lesson #6: Don’t reinvent the wheel.
If you’re having some kind of pain with your infrastructure, chances are you’re not the first. Google is your friend. Or in an attempt to not be search-engine specific, the internet is your friend. There are a lot of smart and friendly people that can help you out.

Lesson #7: If you’re supporting a really really big, revenue-generating infrastructure, you’re going to want the right tools for the job.
This is somewhat related to the above point. We talked a lot about different architectures, applications, frameworks, and tools for solving your enterprise-level problems. Standards are developed for a reason. There are some industry standard tools for getting important tasks done. log4J and JMS are examples of accepted, widely used tools for enterprise logging and messaging. If a particular part of your IT infrastructure is not helping to generate a strategic advantage for your business, using tried-and-true solution is an excellent idea.

Lesson #8: Database optimization is not for the faint of heart.
Part of the reason it’s so difficult is because it’s different for every application and every person. It is hard, but it is not impossible.

Awhile ago I wrote about my thoughts for the best way to approach it. Benjamin Swanson, a good friend of mine, developed his own framework for performing the task. It’ll take time, but so do most other things that are worth doing.

Lesson #9: Putting in the extra time is worth the extra time.
I’ve found through personal experience that taking the time to do something right is worth the effort. Throughout my school career, I’ve built a reputation for being a go-to guy and getting things done because I’ve been willing to put in the time it takes to get things done. I find it rewarding when my peers ask me for help with something I’ve figured out.

But that is not why I do it. For me, putting in the time to learn something is its own reward.

*** Update ***

Lesson #10: Make your technology fit your business.
Perhaps this lesson is more powerful when stated in the longer form. Make your technology fit your business; don’t make your business fit your technology. Technology serves a purpose, but it is only a tool. When applied to the business world, it should solve a business need and more importantly, should make it easier for the business people to do their jobs. The MISM program at BYU emphasizes both the business and the technology, but in most cases, it’s the business that makes the money. The technology exists to help us do business better (well, obviously there are other uses for technology, but let’s keep things in context here).

Low-hanging fruit
Don’t waste time trying to tune every query in your database. It can’t be done. Find the ones that take all the time and tune them. The slow queries log is a good place to start. My understanding is that you can set the parameters for the slow queries log in your my.cnf file.

log_slow_queries=/var/lib/mysql/slow­queries.log
long_query_time=2
log_long_format

EXPLAIN
Once you’ve found the queries that are taking a long time, you can use the EXPLAIN command (or the MySQL Query Browser graphical version) to see which parts of the query are taking the longest. It gives a lot of information about the query itself, but besides the times themselves the most useful information is related to the indexes. You can see how many records each portion has to crunch through, which indexes are available, and which indexes are being used. This leads to tip #3.

Index, Index, and….oh, Index.
Proper indexing is probably your best friend when it comes to increasing query performance. This has limited benefits for extremely dynamic databases with a high proportion of INSERTs and UPDATEs because the indexes will have to be rebuilt each time the table is changed, but where SELECTs are the bulk of the queries (most databases) indexing can contribute to huge increases in speed

More Info
http://dev.mysql.com/doc/refman/5.0/en/optimization.html
http://dev.mysql.com/doc/refman/5.0/en/explain.html
http://dev.mysql.com/tech-resources/presentations/presentation-oscon2000-20000719/index.html
http://jpipes.com/presentations/mysql_perf_tuning.pdf [pdf warning]
http://20bits.com/2007/04/10/10-tips-for-optimizing-mysql-queries-that-dont-suck/
http://immike.net/blog/2007/04/09/how-not-to-optimize-a-mysql-query/