BVLog Bryan Voss’ mental synchronization point

14May/070

Server sprawl

Just heard that we will be getting 7 more Linux boxes in as part of a new system we're implementing. We already have 9 in the rack for that particular system and it's not live yet. Our primary software vendor, McKesson, appears to be standardizing on Red Hat Linux and Oracle lately. On one hand, it's nice to see Linux taking a larger percentage of the datacenter, but on the other hand, the server sprawl is ridiculous. The particular system we're implementing has three DB servers and five frontend webservers as well as several ancillary servers. These are dual-core, dual-CPU systems with 8GB RAM. Granted, some of that is for a test system, but still, four webservers with four hyper-threaded cores each should handle several orders of magnitude more traffic than our ~200-bed facility could ever generate.

Why we can't virtualize all this is a question that I've been asking, but the answer is always, "McKesson doesn't support that platform." So what? They support RHEL on Intel. It doesn't matter if it's bare metal or virtual. As long as we manage the virtualization platform properly, performance loss should be negligible. It's ridiculous to see a rack 75% full of servers just to support one system. We could conceivably fill two racks with ESX servers and run the entire datacenter on them. As it stands, our new datacenter is around 75% full already. We're running close to 200 servers. Our UPS and air conditioning capacity are inching closer to max and we're still getting in more servers. This has got to stop at some point.

I suspect part of the problem is that we're buying our hardware and software as one cohesive system. Every new application brings along several new servers. I don't know if our software vendors even support purchasing just the software without the associated hardware.

For one of the systems I maintain, the various applications don't play well together on the same box, so the vendor splits it out into several 1U servers. These servers are sitting at 99% idle all day. That is an ideal candidate for virtualization. I suspect 90% of our systems are similar, in that they run on separate servers just to avoid compatibility issues or performance concerns. The fact that the vendors can't get their own applications to work nicely with each other is ridiculous, but I can't understand why they aren't actively pursuing virtualization as a target platform. Why not deliver your application as several virtual machine images pre-configured to work together and all burned on a single DVD? We can just plop them onto an ESX system and start them up.

Maybe other industries are already getting to this point. Maybe vendors are already looking into that sort of solution. I just need to see some results soon. Since I started in this position 2.5 years ago, we have added about 80 servers and a mass of infrastructure to try to support them. At some point, the daily care and feeding of all these servers is going to overwhelm the operations staff unless we get more people in to help out or automation of tasks gets a lot easier. Just applying OS updates is getting difficult to keep up with. We try to install updates on all internal servers at least every 90 days, but sometimes that slips when we're busy with other projects. That's one issue that virtualization doesn't help with. In fact, the easier it is to throw another VM into the mix, the less consideration is given to the maintenance associated with that server.

And while I'm on a rant, storage requirements are another problem. We can't possibly hope to recover from a disaster using tape. At our current rate, it would take around a year to recover data from tape back to disk. We're moving towards a second datacenter a few blocks away with a secondary SAN and Centera to provide some disaster recovery. Surely other healthcare facilities are experiencing the same issues. Why can't we work together to provide reciprocal redundancy? We could co-locate two racks of equipment at another facility's datacenter across the state and put a couple of racks of their equipment in our datacenter. As long as our connectivity is sufficient, we should be able to save considerable money over building a secondary datacenter ourselves. Are we so competitive with each other that we can't work together to save our patients some money? I realize that sanctity of patient data is a factor to consider, but if we can encrypt data when writing to tape, surely we can encrypt data when writing to a redundant SAN across a fiber link.

Probably a pie-in-the-sky vision. Too many good old boys smoking cigars together and slapping each other on the backs during a round of golf to expect anything to change during my career. I should probably just be glad that I'm getting more servers to support and stop whining about saving money and making things easier. Grumble, grumble...

Comments (0) Trackbacks (0)

No comments yet.


Leave a comment

No trackbacks yet.