Toluu

mattheaton.com

Matt Heaton - Bluehost Blog

Hyper Speed MySQL DatabasesNovember 9 2008

The last several months we have been feverishly testing many different hardware server configurations trying to squeeze every last bit of performance from each of them. One of our main focuses have been towards improving MySQL performance and segregating MySQL users resources from one another so that a user that is abusing the system wont’ affect others on the same server.

The software side of this equation and related custom MySQL patches that make it happen will follow in a few days on this blog along with GPLed source code to help out. However, for this entry I want to focus on the hardware side of things.

How have we increased performance on the hardware side? We are putting in SSDs of course! SSDs or solid state drives have been getting a lot of attention lately - and rightly so! They are EXTREMELY fast. The problem with SSDs are that many aren’t compatible with certain raid controllers, many are slow, and many have extreme write delays. The ones we use have none of these problems. Even random writes which is typically the slowest stat for SSDs outperform any regular platter based drive we have ever tested.

To illustrated how fast our software and hardware combination is compared to just a few months ago I took a server with 500 clients on it and benchmarked it in every way I could. We then took five of these servers and combined ALL the users onto a single new server with our new platform with the SSD and MySQL caching system. With

The Linux Kernel = The Solution = The ProblemNovember 3 2008

Linux is an amazing operating system. I have written about it several times over the years. All of our company’s servers run on CentOS Linux (about a thousand) and a good chunk of our desktops and work stations run Ubuntu linux. The speed and reliability of the main linux kernel (Think engine of a car) is unmatched in my estimation. What is even more impressive about the Linux kernel is its unbelievably rapid pace of development. Herein lies our problem.

The linux kernel is developed simultaneously for so many different work loads that it is impossible to ship a Linux Distribution that is tuned for your specific workload. If I could use the car analogy one more time - Linux could just as easily be an all electric 50 HP car, or a 1,000 HP Dragster, or a Semi Truck. It literally can and is used to power your phone to the fastest super computer in the world. With that type of flexibility how do you wrangle the most power out of the linux kernel for your specific needs?

Unfortunately, right now the answer is - With a GREAT DEAL of effort. I mentioned earlier that we use CentOS. This is a free rebranded version of Redhat Enterprise Linux which is a server class linux distribution. Meaning it is geared toward customers with a heavy server workload. The problem is the linux kernel they use in CentOS is SLOW, outdated, and certainly not tuned to our workload.

In a way this is a big benefit for us. I know of no other hosting company on the planet

Chicken with your head cut off…November 2 2008

True story - In 1945 there was a chicken in Colorado that had its head cut off (Sorry, gruesome I know). What’s incredible about this particular chicken is that the axe missed its jugular vein but left enough of the brain stem attached for this chicken to live. Not only did this chicken live, but it actually seemed to thrive. In fact, in the following two years the chicken actually put on 6 lbs.

This chicken was later named Mike and actually toured the country being shown at different shows and events. The chicken made its owner up to $4,500 a month (63 years ago!!!). People by the thousands paid 25 cents to see Mike and were amazed at how Mike seemed to live a completely “normal” life.

Mike was fed through an eyedropper, but other than that he simply went about his life. Even headless, Mike would happily “peck” for food blissfully unaware that he had no head.

Now this story is amazing to me for several reasons, but the one that stood out to me most was that a chicken with literally his head taken off could function as he did before. He was a drone. He did the same activities day in and day out but never had to “think” for himself.

Sometimes as I watch people that do the same thing day after day with no ambition or action to change their circumstances or their lives that they are no better off than Mike. We all have the capability to live a full life despite our circumstances, but it requires work, dedication, and proper

Happy Halloween!October 30 2008

Sorry…. Lately I have sort of disappeared from my blog. Perhaps its because I have been delving DEEP into the underpinnings of the linux kernel ferreting out problems that lesser hosting companies would run screaming from!

Anyway, I’m almost back to normal life and to prove it I offer up the following physical evidence!

Let it never be said that I let a potential marketing opportunity slip by. All those trick or treaters need a website to sell or trade their candy on Ebay. Ok, jokes aside it is good to be back. I will try and be MUCH more active than I have been lately.

Happy Halloween!

Matt Heaton / President Bluehost.com

Collectl - A System Admins’ Dream ToolOctober 29 2008

We run Linux on all our servers at Bluehost and Hostmonster.  Our particular flavor of linux is CentOS 4 and 5.  For a variety of reasons we feel that Linux is head and shoulders better than Windows in virtually every way.  But in order to really squeeze the best performance out of a system or diagnose a performance related issue you used to have to rely on many different  tools to give you the bits and pieces on information you would need.

Enter “collectl” (Yes that is an L on the end of collect).  Collectl is indispensable to any system admin.  It replaces sar, vmstat, top, atop, iostat, and many other tools that I USED to use.   Now instead of having to rely on those various tools that did 80% of what I needed I just use collectl.

Collectl was the brain child of Mark Seger over at HP’s Scalable Computing & Infrastructure group.  He developed this as a tool to monitor huge clusters of HP servers, but saw the benefit that the “average” linux user could get from using this tool.

The best part of collectl in my opinion is that it is CONSTANTLY being updated and improved.  This means that you have access to data and reporting that is available in many of the newer kernels.  So many of the other tools out there simply don’t keep up and so get left by the wayside as the rapid development of the kernel continues.

An example of this is when I need realtime IO breakdown of processes that simply can’t be had without the data