The case of the disappearing eth0
There have been a couple of occasions in the past week that I have lost an ethernet interface when swapping machines around. Looking back into my murky past, I can recall a couple of other times that I probably encountered the same issue. Don't recall how I resolved the issue before, but I have a definite solution now. I figured I should note it here so I can look it up later and so others can benefit from it.
Scenario 1: I build a Debian virtual machine using VMWare Workstation on my laptop. I later move the VM to a VMWare Server box. On first boot, VMWare Server asks if I want to assign a new UUID and I select yes. It turns out that the MAC address assigned to the virtual ethernet device is affiliated with the VMWare UUID. When the UUID changes, the MAC address changes. Debian assigns eth devices based on MAC address and therefore eth0 is lost after the MAC changes. The issue shows up when I try to start networking on the VM and eth0 doesn't come up.
Scenario 2: I install Debian on a PC-class box and tinker with it a while. It breaks (something to do with heat, probably a fan failure). I move the hard drive to an identical box and it boots fine, but eth0 doesn't come up. Same as above. Since Debian assigns the eth devices based on MAC address and the new ethernet device has a different MAC address, I get no eth0.
Solution: A comment on this post pointed me down the path of enlightenment.
/etc/udev/rules.d/z25_persistent-net.rules contains the MAC address to eth device mappings. Delete the lines like below, noting the module name on the "# PCI device" line:
# PCI device xxxxxx:xxxxxx ([module])
SUBSYSTEM=="net", DRIVERS=="?*", ATTRS{address}=="xx:xx:xx:xx:xx:xx", NAME="eth0"
This removes the MAC to eth device mapping info. Now we need to restart udev to allow the change to take effect:
/etc/init.d/udev restart
Next step is to "bounce" the kernel module for the ethernet device. Use the module name from the z25_persistent-net.rules file noted above:
modprobe -r [module]
modprobe [module]
"ifconfig" should now show the eth0 interface as up and running. If not, try "ifup eth0" and check "ifconfig" again. That rascally ethernet interface can't hide for long!
Update 2009/03/11: This post details a method that does not require modifying individual VMs. Probably a better solution for template VMs or virtual appliances.
Fixing VMWare Server directory permissions on Debian hosts
I often create a VMWare virtual machine using VMWare Workstation on my laptop, then later move it to a Debian machine running VMWare Server. As a result of the move from Windows to Linux, permissions on the VM directory end up being wrong and I get a blank black screen when connecting the VMWare Server Console to the VM. I normally just fix the permissions manually when I notice the problem.
Today, I was helping my father deal with the same situation via email and started typing out all the steps to create a group and fix permissions on the VM directory when I thought, "Why not be a good little sysadmin and write a script to do this?" So, here's the resulting BASH script:
#!/bin/bash
# Fix VMWare Server permissions on Debian host
# Bryan Voss 2007/07/19
# *** Config items ***
# Group that all VMWare users will belong to
VMGROUP=vmware
# *** /Config items ***
# Find directory where VMs are stored
VMDIR=`grep vmdir /etc/vmware/config | cut --delimiter=' ' -f 3 | cut --delimiter='"' -f 2`
# Add group that VMs should belong to
addgroup --system $VMGROUP
# Fix permissions on directories under VMDIR
cd $VMDIR
chgrp -R $VMGROUP *
find . -type d -exec chmod g+rwxs \{\} \;
# Fix permissions on vmx files
find . -name *.vmx -exec chmod -R +x,g+rwx \{\} \;
# Fix permissions on all other files
chmod -R g+rw *
Make sure any users who will be connecting via VMWare Server Console are members of whatever group you set VMGROUP to ("vmware" by default).
I have saved this as /usr/local/sbin/vmware-fixperms on my VMWare Server boxes and will probably use it pretty often. Maybe others will benefit from it.
RHEL3 and LVM
So there I was, installing Red Hat Enterprise Linux AS 3 on a new box. "RHEL3," you ask? Yes, the application that will be running on that box requires not version 5, not even version 4, but version 3.
Anyway, I boot into the GUI installer. The mouse doesn't work. Apparently the KVM module on this box is confused about the type of mouse it wants to present to the OS. Ok, no problem. I just reboot into the text-mode installer and go merrily on my way. I get to the partitioning step and spend several minutes trudging through the arcane requirements document provided by the vendor. for some reason, they want two volume groups rather than one that consumes the entire disk. [shrug] Calculate how to size the partitions. Create physical volumes. Done. Ok, time to create a volume group. Wait a minute. Where's the LVM button? The GUI installer has an LVM button on the partition screen that allows you to create volume groups and logical volumes. The text-mode installer is missing the LVM button!
I hit www.redhat.com and navigate my way to the install guide for RHEL3. Try the index first. Nothing about LVM. Ok, check the table of contents. Text mode installer user interface. Nope, nothing there. Maybe something in the Disk Druid buttons section? What's this? "Note, LVM is only available in the graphical installation program." Aargh!
Reboot yet again into the graphical installer with the plan to use the keyboard to navigate through it. Wiggle the mouse just for fun. Hey, it works! I guess the KVM module saw my dilemma and decided to have mercy on me.
Linux sysadmin tip #1: LVM
There's no excuse for not using logical volumes when setting up a Linux system anymore. All the major distros support LVM. All the major rescue disks support it. The benefits of LVM are too great to ignore. You set up a volume group containing all your disks, then create logical volumes on top of that. The logical volumes can be expanded when necessary until the volume group is filled. When that happens, just add another disk and expand the volume group across it. You can then continue expanding the logical volumes as needed. Logical volumes work on top of RAID arrays and SAN LUNs, as well.
About the only thing better is Solaris' ZFS, which I've just begun experimenting with recently. But that's a topic for another post...
Do not be afraid
I am occasionally asked how I learned so much about [insert technical subject here]. My answer is always: "I just started playing with it until I figured it out." The point I try to get across is that I learned it by doing it.
A coworker asked me to sit down with him and teach him some things about Linux. We can spend all day talking, but the only way to figure out the ideas and concepts behind the Linux shell is to live in it for a while. I told him that around 10 years ago, I determined that I needed to learn Linux since it looked like it was going to be an important platform and doggone it, I didn't have the money to keep buying commercial software. I threw out Windows entirely and used Linux for everything at home. I started a sysadmin job at a small manufacturing company and proceeded to migrate everything I could to Linux. I was probably overzealous in my attempts, but I learned a huge amount as part of the process. I spent countless hours reading man pages and howtos. I signed up on the local Linux users group email list and asked questions. Eventually, I got good enough that I didn't have to post many questions anymore. Then I got good enough that I was able to post responses and help other people with their Linux problems.
I never would have gotten where I am today without banging my head against seemingly insurmountable problems, breaking all kinds of systems and rebuilding them, doggedly sticking with Linux even though it would have been trivial to solve a problem with Windows. I have since learned to back down occasionally and recognize the particular situations where Linux is the right choice and to accept the situations where it's not the right choice. But I wouldn't trade my history for anything. It's what got me to where I am today.
Here's the point in all this: do not be afraid to play with a system. Build a test box if the system is mission critical. Just the experience of building the test system is useful. Break it and rebuild it.
Learn the history of whatever project or system you're experimenting with. Feel the mindset and methods of the developers. There's probably a reason they did it that way.
Once the system is in production, it will eventually break in some unexpected and odd way. Don't be afraid to open the hood and fix it, just document what you did (an internal blog makes a great worklog).
Welcome uncertainty, it's a learning opportunity in disguise. The only way to gain certainty is to act decisively. No route is perfect. Pick what looks like the best one and run with it. If it turns out to be the wrong route, at least you learned something along the way.
But above all, do not be afraid.
Server sprawl
Just heard that we will be getting 7 more Linux boxes in as part of a new system we're implementing. We already have 9 in the rack for that particular system and it's not live yet. Our primary software vendor, McKesson, appears to be standardizing on Red Hat Linux and Oracle lately. On one hand, it's nice to see Linux taking a larger percentage of the datacenter, but on the other hand, the server sprawl is ridiculous. The particular system we're implementing has three DB servers and five frontend webservers as well as several ancillary servers. These are dual-core, dual-CPU systems with 8GB RAM. Granted, some of that is for a test system, but still, four webservers with four hyper-threaded cores each should handle several orders of magnitude more traffic than our ~200-bed facility could ever generate.
Why we can't virtualize all this is a question that I've been asking, but the answer is always, "McKesson doesn't support that platform." So what? They support RHEL on Intel. It doesn't matter if it's bare metal or virtual. As long as we manage the virtualization platform properly, performance loss should be negligible. It's ridiculous to see a rack 75% full of servers just to support one system. We could conceivably fill two racks with ESX servers and run the entire datacenter on them. As it stands, our new datacenter is around 75% full already. We're running close to 200 servers. Our UPS and air conditioning capacity are inching closer to max and we're still getting in more servers. This has got to stop at some point.
I suspect part of the problem is that we're buying our hardware and software as one cohesive system. Every new application brings along several new servers. I don't know if our software vendors even support purchasing just the software without the associated hardware.
For one of the systems I maintain, the various applications don't play well together on the same box, so the vendor splits it out into several 1U servers. These servers are sitting at 99% idle all day. That is an ideal candidate for virtualization. I suspect 90% of our systems are similar, in that they run on separate servers just to avoid compatibility issues or performance concerns. The fact that the vendors can't get their own applications to work nicely with each other is ridiculous, but I can't understand why they aren't actively pursuing virtualization as a target platform. Why not deliver your application as several virtual machine images pre-configured to work together and all burned on a single DVD? We can just plop them onto an ESX system and start them up.
Maybe other industries are already getting to this point. Maybe vendors are already looking into that sort of solution. I just need to see some results soon. Since I started in this position 2.5 years ago, we have added about 80 servers and a mass of infrastructure to try to support them. At some point, the daily care and feeding of all these servers is going to overwhelm the operations staff unless we get more people in to help out or automation of tasks gets a lot easier. Just applying OS updates is getting difficult to keep up with. We try to install updates on all internal servers at least every 90 days, but sometimes that slips when we're busy with other projects. That's one issue that virtualization doesn't help with. In fact, the easier it is to throw another VM into the mix, the less consideration is given to the maintenance associated with that server.
And while I'm on a rant, storage requirements are another problem. We can't possibly hope to recover from a disaster using tape. At our current rate, it would take around a year to recover data from tape back to disk. We're moving towards a second datacenter a few blocks away with a secondary SAN and Centera to provide some disaster recovery. Surely other healthcare facilities are experiencing the same issues. Why can't we work together to provide reciprocal redundancy? We could co-locate two racks of equipment at another facility's datacenter across the state and put a couple of racks of their equipment in our datacenter. As long as our connectivity is sufficient, we should be able to save considerable money over building a secondary datacenter ourselves. Are we so competitive with each other that we can't work together to save our patients some money? I realize that sanctity of patient data is a factor to consider, but if we can encrypt data when writing to tape, surely we can encrypt data when writing to a redundant SAN across a fiber link.
Probably a pie-in-the-sky vision. Too many good old boys smoking cigars together and slapping each other on the backs during a round of golf to expect anything to change during my career. I should probably just be glad that I'm getting more servers to support and stop whining about saving money and making things easier. Grumble, grumble...
diff ‘Linux traceroute’ ‘Windows tracert’
A coworker and I were trying to debug a remote connectivity issue between a Windows box connected via a Juniper SSL VPN to a RHEL box. We were able to do a tracert from a Windows box through the Juniper, but a traceroute under Linux would not work. We checked for iptables rules, routing, etc. to no avail.
I vaguely remembered something about a difference between Windows and Linux/UNIX traceroute. Something about one using ICMP and the other using UDP. I finally found in the traceroute man page: "-I Use ICMP ECHO instead of UDP datagrams." I slapped a -I argument in there and traceroute began to work.
My coworker said, "Well, I guess we just have to remember that Linux traceroute is buggy." Grrr. I said, "Or maybe it's Windows traceroute that's buggy and non-standard. [thoughtful pause] Although it does seem like ICMP would be the protocol to use for traceroute."
We talked about reading the RFPs for an answer, then shrugged and went on with our duties. It would be nice to be able to argue that one implementation of traceroute more closely adheres to the RFPs than another. Honestly though, outside the two of us having the discussion, there's probably nobody else in the office that even knows what an RFP is.
Handy Linux/Bash tidbits
Lest this blog remain a simple mirror of my del.icio.us entries, I should probably provide something interesting to read. Here goes...
Every time I set up a new Linux box, I install some utilities and configure a few things to create a common environment that I'm comfortable in. I'll document those things here for my own benefit, but also in the hopes that it might help someone else.
My distro of choice is Debian, so YMMV with other distros.
apt-get -u install ssh less vim screen
edit root .bashrc:
Modify or comment out the "export PS1" line and add:
PS1='${debian_chroot:+($debian_chroot)}\[\033[01;31m\]\
\u@\h\[\033[00m\]:\[\033[01;36m\]\w\[\033[00m\]\$ '
This makes the prompt red and the current directory cyan, something like: root@tetra:/var/tmp$
I do the same for my account, but make the prompt green. This makes it easy to tell whether I'm myself or root:
PS1='${debian_chroot:+($debian_chroot)}\[\033[01;32m\]\
\u@\h\[\033[00m\]:\[\033[01;36m\]\w\[\033[00m\]\$ '
This give you something like: bvoss@tetra:/var/tmp$
I also like to make sure "alias rm='rm -i'" and "alias dir='ls -AlFh --color | less -REX'" are present in there somewhere. And, yes, I use "dir" to get a directory listing. Very DOS of me, innit?