BVLog Bryan Voss’ mental synchronization point


Welcome to the datacenter/sauna

What I did today:

Datacenter temperature graph - 2007/01/17

Whee! Our new high-dollar datacenter got a little steamy after the chiller on the roof decided to quit. Apparently nobody has bothered configuring it to alert when it goes down. I was actually in the middle of performing a scheduled maintenance from home at around 4:30 am when I noticed that things were acting funny and decided to go in and take a look.

When I opened the door to the datacenter, I was hit with a balmy 108 degree Fahrenheit wave of air. I dragged in a big fan and propped it in the door to help move some of the air out and then scrambled around paging people for a few minutes. Funny how people's pagers mysteriously stop working in the middle of the night and then work fine the next day.

Several systems had already shut down due to the heat. A couple of the guys showed up and we started manually shutting down the remaining systems until Maintenance was able to get the chiller working again and the temperature started dropping.

Nice shakedown run for our IP KVM solution. In our old datacenter, we had a KVM in each rack. It took around 20 minutes to manually shut down 120 servers with 3-4 people working together. When we moved to the new datacenter, we added Avocent IP KVMs and removed the rack KVMs. We've also added about 60 new servers. With three people working on shutting down servers, we were able to get about 20 servers down in 20 minutes. Obviously, the KVM solution needs some reassessment.

Anyway, we're back in business and things are once again running smoothly. Once the chiller came back up, it did an admirable job of cooling the place down.

Comments (0) Trackbacks (0)

No comments yet.

Leave a comment

No trackbacks yet.