BVLog Bryan Voss’ mental synchronization point


Racks and cables and wireties, oh my!

I spent almost all day at work yesterday recabling an entire rack while it was running. The whole rack is devoted to our medical imaging (PACS) system, so it's very painful to schedule downtime.

We moved the rack to our new datacenter about a month or so ago. In the process, we pulled the rackmount UPSes out and connected directly to the central redundant UPS system. The rack was originally configured with one UPS per server (!), meaning the bottom half of the rack was filled with UPSes and the top with servers. We installed vertical PDUs in the back of the rack on either side and had to swap the power cables out on each server. Since we were already at the end of our scheduled downtime, we had to frantically get everything back up and running without cleaning up the rat's nest of cables in the back.

I was pulling a test server out of the rack yesterday and got into the back to disconnect it. It was such a mess that I decided to spend the time to clean it up. Thankfully, the power and network are redundant on that rack (we're slowly working towards doing this on all racks). The rack was staged by the vendor and shipped to us pre-cabled. I suspect that a large part of the expense involved in purchasing the system went towards all the cables and wireties jammed into the back of that thing. Why use a three foot cable when you can use a 10 foot? I was practically wading in snipped wireties by the time I got all the old cabling out.

Early in the process, I was merrily pulling cables when I heard a knock on the datacenter door. The rack I was working on just happened to be near the door, otherwise I never would have heard anything over the blasting fans in the room. I opened the door and there was our PACS admin (the guy who handles the clinical side of the medical imaging system). Apparently, I had disconnected the power cable on the database server's SCSI tray. Surprisingly, the database stopped responding. We spent 30 minutes bringing it back up and calling vendor support to make sure everything was back to normal.

After that mishap, I was much more careful about pulling cables. With the PDUs we're using, the power cables fall out at the slightest wiggle. The PDUs come with a bracket that you can add to wiretie the cables down. I hadn't mounted the brackets yet due to the time constraints we had when originally moving the rack. So, I spent the time yesterday mounting the brackets and tying everything down snugly.

When I finished all the cabling, I closed the rack and cleaned up the mess on the floor. I was about to roll my cart full of old cables out the door and leave for the day when I stopped. I went back to the rack, opened the back doors and just spent a couple of minutes admiring the niceness of all the clean cabling. With a sigh of satisfaction, I closed the doors and went on my way. It's those little moments of relishing a job well done that make the daily hassles worthwhile.

Comments (0) Trackbacks (0)

No comments yet.

Leave a comment

No trackbacks yet.