We all know it can happen, a BIOS update of some kind bricks whatever just got flashed, but it's one of those things you hope happens to other people first so you know not to go there. It happened to me recently, which got me thinking about continuous deployment from a hardware POV. Hardware being what it is, hard, you can't iterate and roll-back the way you can do software. There is no such thing as Vagrant for Embedded Systems that I've found!

The problem of, "when do I update the firmware for my server," is one that faces anyone with a physical infrastructure. There isn't really a globally accepted best-practice for this one, though the closest I can find is:

If the vendor lists the update as critical, apply it.
If you're experiencing one of the problems listed in the fixes, apply it.
If vendor tech-support tells you to apply it, apply it.
Otherwise, don't apply it.

But only apply it to a test device first to verify it actually fixes the problem. Then roll it out.

Doing so pro-actively is kind of risky, and only really useful in repurposing scenarios. Also, this 'best practice' assumes you have identical hardware to actually test with. Which a lot of us don't, and often can't due to slight differences between servers of the same model.

So. For those of us who are working on infrastructures either small enough to not be able to afford test hardware, or diverse enough that there is no such thing as a common class of machine, what are we to do?

Hope, mostly, and trust in your vendor support contracts to ship you new hardware in case you get a brick.

Or, trust in your redundancies and treat new-firmware-updates like a lost-server outage. If you get a brick, you're still within your failure tolerance and know not to go there for the rest of 'em. This is the approach we ended up taking, and it worked. We were running without our scale-test environment for a few days but production was unaffected until we could unbrick the affected machines.

In our case I suspect we had a v1.0 hardware revision, and the newest firmware was only backwards compatible for v1.0a and newer or something. I don't have proof of this, but that's what it feels like. Of course, this eventuality was not mentioned in the release-notes anywhere. Thus, testing.

Most UNIX users have heard of the nice utility used to run a command with a lower priority to make sure that it only runs when nothing more important is trying to get a hold of the CPU:

nice long_running_script.sh

That's only dealing with part of the problem though because the CPU is not all there is. A low priority command could still be interfering with other tasks by stealing valuable I/O cycles (e.g. accessing the hard drive).

Prioritizing I/O

Another Linux command, ionice, allows users to set the I/O priority to be lower than all other processes.

Here's how to make sure that a script doesn't get to do any I/O unless the resource it wants to use is idle:

sudo ionice -c3 hammer_disk.sh

The above only works as root, but the following is a pretty good approximation that works for non-root users as well:

ionice -n7 hammer_disk.sh

You may think that running a command with both nice and ionice would have absolutely no impact on other tasks running on the same machine, but there is one more aspect to consider, at least on machines with limited memory: the disk cache.

Polluting the disk cache

If you run a command (for example a program that goes through the entire file system checking various things, you will find that the kernel will start pulling more files into its cache and expunge cache entries used by other processes. This can have a very significant impact on a system as useful portions of memory are swapped out.

For example, on my laptop, the nightly debsums, rkhunter and tiger cron jobs essentially clear my disk cache of useful entries and force the system to slowly page everything back into memory as I unlock my screen saver in the morning.

Thankfully, there is now a solution for this in Debian: the nocache package.

This is what my long-running cron jobs now look like:

nocache ionice -c3 nice long_running.sh

Turning off disk syncs

Another relatively unknown tool, which I would certainly not recommend for all cron jobs but is nevertheless related to I/O, is eatmydata.

If you wrap it around a command, it will run without bothering to periodically make sure that it flushes any changes to disk. This can speed things up significantly but it should obviously not be used for anything that has important side effects or that cannot be re-run in case of failure.

After all, its name is very appropriate. It will eat your data!

At 11:32PM, Saturday May 11th, I got an email from MX Toolbox notifying me that a SBS 2008 machine that I support had gone unresponsive. It’s 600 miles away from me in another state. This was not a strange occurrence with this server.

A Cluster of Prior Failures

Five years ago a small office with a minimal budget needed a SBS implementation. I recommended an HP ML 115 G5 with four hard drives and onboard RAID provided by an NVIDIA chipset. I have regretted that decision for all five years. Here’s a post of mine concerning that chipset and the troubles I’ve had with it.

In short, I have poor insight into and control over the entire server’s health. Some examples include:

• I couldn’t update the hard drives’ firmware, which was a big deal because the serial numbers of those hard drives fell into a set of drives that have a known problem with suddenly going offline. The firmware update has to be applied through HP’s support tools, which are not supported on the ML 110/115. After much research and seeking help from HP, I was told that, in essence, I was left out to dry.
• The ML 110/115 does not support the ProLiant Support Pack nor does that model support the Insight Control Manager. Keeping drivers updated and staying abreast of the various components’ health was virtually impossible.
• There was also no HP ILO CLI interface available which made doing things like firmware updates especially difficult remotely.
• The on-board storage controller had poor support form Nvidia, and offered very slim storage management features or reporting on hard drive health.

For years I hit the management ceiling with that box which probably cost my client more of my time and theirs than had a more robust server been purchased for twice the hardware cost. And then what I had been dreading for years finally happened…

Two Months Ago

“Did you reboot the server?” That’s never a question you want to hear, especially when you did not reboot a server. I VPN’d into that office’s network and checked for the presence of the server on the network. Yes, the server was down. One power cycle later, the OS loaded just fine.

I checked the event logs and it turns out there was a massive flurry of parity errors that came out of nowhere. The server froze as a result. The controller was apparently dying. After a reboot, the data appeared fine, and there were no more parity errors coming from the Nvidia storage driver. I knew something had to be done, but being remote and working with an office that has a shoestring budget (and can often only afford used shoestrings) made the options few and unattractive.

What’s worse, as I started investigating things further, I noticed that the ILO Advanced card that was in the server was no longer showing on the network. Aaaaand the BIOS clock would reset to July 2009 after being shut down (BIOS battery dying) causing strange problems with Active Directory and other applications running on the network that relied on accurate time (read: everything). AAAaaaaaand the two mirror sets (one for the system volume and one for the email server’s databases) had split apart and could not be re-synced because the Nvidia storage management software no longer recognized that any hard drives were connected.

The options, as I saw them, were for the business to either buy a new RAID controller, BIOS battery, and perhaps ILO card (and then scramble to perform the complex surgery remotely on their own, or pay a local consultant to coordinate with me, or pay to ship me on site) or get a new server altogether (and pay a local consultant to coordinate with me, or… you get the idea). Either way, it started to look more and more like a total forklift migration was necessary.

Two Months Later

Yes, it’s been about two months and the server is still riding in the same perilous state. Split mirrors, bi-monthly freezes that require a power cycle to recover from, and a lot of hoping and praying that data is not corrupted. Welcome to the world of supporting small business IT where people re-use tea bags and don’t run heat or AC in order to save money and keep the business open.

That Saturday night, it was getting late and I was thinking about bed. I checked my email one last time for anything pressing when I saw a MX Toolbox alert. This is never good. I scanned the email, saw what host was causing the alert, and knew that I was dead in the water. I could get into that client’s network via both a SonicWall VPN and unattended TeamViewer installations that existed on most of the workstation PCs. However, it was all futile because I didn’t have hardware level access to the server as a result of the ILO’s failure. The office has a Lantronix Spider KVMoIP device that was being used to work on a workstation migration for one employee, and was therefore not hooked up to the main office server. That was two layers of out of band management that was not doing any good for the most important technology asset in the building.

All of this meant that someone would have to show up at the office to power cycle the PC. The technical debt and compound interest of failure had already mounted fairly high by that point, considering the state of the server. However, things were about to get comical.

I’ll Gladly Pay You Tomorrow for Out of Band Management Today

What happened in the next 24 hours was a morbid comedy of oversights and compounded problems that ended in a whiplash inducing facepalm.

First, I needed to email three people who would most likely be in the vicinity of that office so I could coordinate with one of them to drop by on their Sunday morning and power cycle the server. Except the server is what does email for the organization so I can’t send to their organization email addresses (this is a Microsoft SBS machine). I only know of one employee’s non-work address, and I also happen to know the gmail address of another employee’s son.

I email those two people and tell them of the situation. As it turns out, two key workers are out, traveling to a convention in Texas. That makes access to email even more vital than normal. Everyone knows the situation and there’s not much more I can do so I get to bed. It’s not until about 2PM on Sunday, Mother’s Day here in the USA, that I hear back from one worker who has just enough time to skip by the office and power cycle the server.

Myself, I’m in the midst of a Mother’s Day dinner with my own family so I had ditched my phone… just moments before the employee called me from the remote office. I missed the call and the employee left a voicemail expressing a state of confusion over which server to power cycle. The organization is small and only has two servers. One is the SBS machine and the other is a HP MicroServer that is used as a network monitoring station and catchall for various extraneous services. I had assumed that over the years everyone had each server’s role understood by sight so I simply asked him to power cycle the SBS server, expecting that it would be known which piece of hardware that was. The fellow power cycled both servers since he couldn’t get in touch with me directly.

Okay, no big deal. The MicroServer is just running CentOS and OpenNMS. They’re resilient and can handle a sudden shutdown. As I listened to that voicemail, I checked to see if I could remotely connect to the server that had been down all night. I couldn’t. Great. Time to call the office and talk to the person who was on site and see what else could be done. Except the voicemail had been left over an hour ago and the employee had naturally left shortly after power cycling the server. I called his cell phone back, but he’s didn’t pick up. I left a voicemail.

A little later that Sunday I get in touch with another employee in the area who lives closer. He’s on his way out to pick up Mother’s Day dinner for his wife and can swing by to check out the server. First, I have him power cycle it again. Maybe the first guy just clicked the power button and didn’t hold it in? I held out hope for such a simple explanation. However, after I instructed this second person on how to make sure the server had shut down and then powered up, I waited for the duration of the standard bootup but nothing was showing up. It became apparent that the server was not coming back online.

“Do you know where the Spider is?” I asked hopefully. “No, I dunno where the other guy put it.” Gah! The Spider is a well known piece of equipment in that office, and it’s very rare that it can’t be found. I was about to concede defeat for that Sunday when, after some searching, the employee found the Spider. A few minutes of scrambling around and he had the thing hooked up to the server. Except… now I couldn’t get to the Spider. The fellow had to leave to pick up dinner and I wasn’t about to ruin his family’s Mother’s Day so I told him I’d see what I could do remotely, expecting nothing to be successful.

In the process of hooking up the Lantronix Spider, the employee had pulled the network cable out of the server and put it into the Spider. Then from the spider’s cascade port (it’s essentially a one port switch) he had connected a patch cable to the server’s LAN port. That made me wonder… perhaps it was a port on the ProCurve switch that was bad? That would explain both the server and now the Lantronix Spider being inaccessible. Or maybe the port spontaneously shut down as a result of some bug. Crazier things have happened.

I browsed to the switch’s management interface. “Please enter your username and password!” Okay, no problem! “Wait… I can’t remember what the password is… NOOOOOO!” The organization uses KeePass to store important passwords and software keys. The KeePass file is on the server. The server that is down.

But wait! I have a copy of the keepass databases on my own storage. Once a month or so I copy the files to my local storage so that I have an in-sync copy just in case. Whew! I find the switch’s login credentials and begin inspecting things. I looked, hoping for some bad news concerning the switch’s health (at least that would mean the server was okay), but the switch looked perfect. Nothing was amiss.

I’ve always been told to troubleshoot network problems from the lowest layer first. I had pretty much ruled out the physical layer. Layer 2 seemed healthy. Not much that can go wrong on a small, single subnet LAN. Layer three, IP… IP addresses… I gritted my teeth. I knew what the problem was. The Lantronix Spider is set to pick up an address via DHCP. Specifically it’s a DHCP reservation on the network’s DHCP server. The server that’s down. I wanted the network layer benefits of a static IP address, however I also wanted it to be easily portable between networks. My original idea was that the Spider could be used to support PCs on other LANs, like perhaps workers that were based in their home office that didn’t come into the organization’s building very often. With the Spider getting an IP address via DHCP, I could just tell someone to take it home with them and I’d only be left with walking them through configuring port forwarding, or getting TeamViewer set up on a PC on their LAN so I could get in and access the Spider via a local web browser. Except now the Spider was barking out forlorn DHCP discover packets and not getting any response back.

I fired up Network Monitor on an office PC to be sure. Yep, there it was. A DHCP discover request broadcasting every sixty seconds or so. Okay, I can handle this. The small office has a SonicWall firewall that has DHCP services on it. I only need to enable them, check its list of leases to find what IP address it was given, and I’ll be good! I mosey my web browser on over to the firewall’s administrative page. I stare at it. It wants the password for the admin user. “Password… password… I had to change it a few weeks ago. What did I choose…”

Oh well, I’ll look in the organization’s copied password file that I keep on my local storage! Yay foresight!! I found the firewall admin password and entered it. “Password Failure. Please Retry.” What?! Then I remembered that I had changed the firewall password due to security policy about two weeks ago. However, I hadn’t copied the organization’s password file to my local storage in a month. I had the old password in my copy of the password file, but not the new one. The new one was on the server that was currently down. Backups are taken every few hours, but a restoration needs to be done on functioning hardware. Super.

So that means I did it again. I couldn’t log in to the interface because I didn’t have the long password committed to memory. For super important passwords like that, I do keep a disaster recovery hard copy around. It’s essentially a few pages spelling out the most important usernames and password for the organization. However, only two people have that physical copy of information. While I could call them up and have them read off the password to me, I wasn’t ready to do that.

Instead, I turned to the HP MicroServer running CentOS 6. I have OpenNMS installed on it and have plans to install some ticketing software and maybe smokeping or M/Monit. Now, however, it’s going to be an impromptu DHCP server. Fortunately I can remember the password for the MicroServer! A quick ‘yum install dhcpd’ later and… “Couldn’t resolve host ‘centos-distro.cavecreek.net’” WHAT DEVILRY IS THIS?! But of course; DNS for the network is performed by the SBS server… which is down. After facepalming, I changed resolv.conf to point to OpenDNS and continued my march towards a functioning DHCP server on the network. After a few minutes I have dhcpd running and it quickly hands out a lease to the Spider.

And it was then that I saw it. After logging into the Spider, I viewed the remote console and saw a Windows installation screen on the server. Suddenly, I remembered what happened. In the process of preparing for a migration away from the failing hardware, I needed to experiment with making an unattended installation file. I had a remote worker put the SBS 2008 install CD in the main server’s tray. Of course, rebooting caused the server to boot into the high boot priority CD drive. I sat in horror, thinking about my cascade of failures. Nevertheless, that wasn’t the time to flail in self loathing. I simply needed to hit “cancel” and get out of the installation welcome screen to boot from the hard drive.

Except the Spider was unable to interact with the server as a remote keyboard or mouse. I’ve used the Spider on that very server in the past, and it worked great at all stages of the boot process. In the years that I’ve worked with that office I’ve had to check BIOS settings, ILO firmware settings, and storage controller settings, all using either the Spider or the ILO itseld. But now, for some unexplained reason, the Spider was not able to input anything. I couldn’t move the mouse, I couldn’t press keys. So I sat and stared at the remote video in complete disbelief.

It was a simple matter of leaving a voicemail for someone and telling them to remove the disc from the DVD drive the next time they were in the office. The next morning the worker that I left a message for did just that, power cycled the server, and it booted up as normal. Life continued.

I was abashed.

More about my conclusions concerning the situation later. In the mean time, got a similar story to share? Let me know in the comments below or contact me and you can write a guest blog post about it.

I might not notice it at the time, but I can always tell how busy I am by how many blog posts I manage to get live. By my count, I've been doing about one every eight days so far this month (if you count this one). So I'm behind :-) So what's been going on?

LOPSA-East

But I've been doing good, fun things. For instance, on May 3rd and 4th, I went to LOPSA-East, which was yet another really great conference. There was somewhere around 150 attendees this year, and it was really nice to see everyone again from previous years.

Way back in October of 2011 (were some of you even born then?), I asked about a class on SSDs, to see if there was any interest. Well, in October of 2011, the earliest I could have done it was spring of 2012, and didn't get around to finishing the course before then, so spring of 2013 it was, and I taught the SSD class on Saturday afternoon. Only three years in the making. That's cool, right? :-D

If you were in my class, you probably have the slides from the USB key. If you weren't in my class, then you'll be happy to know that since I don't really intend to teach the class again (although if my feedback is overwhelmingly positive, I'll consider it), I opted to have it recorded, and whenever that goes live, I'll be linking to it from here and including my full slide deck, too.

Storage Field Day

At the end of April, I went to Denver to do Storage Field Day. I haven't had a chance to write about the things I saw yet, but I'm very excited to talk about what we saw with Pernix Data. If you want to see some cool ideas, watch the videos there. I'll write more as soon as I get time.

LOPSA stuff

We're still in the swing of the election season. You might have seen when I updated my earlier post that the LOPSA Live transcript had been posted. That was the first of two candidate sessions. The other is tonight at 9pm, so follow the instructions by Aaron Sachs for connecting to #LOPSA-Live on Freenode and come ask the candidates good, hard questions.

The election is coming up next month. I've posted my series of discussions on internal concerns (including membership numbers, member communications, and operational transparency. Starting tomorrow, I'm going to start posting discussions related to external concerns - we have a lot of problems with marketing and how we're seen externally...when we're seen at all. Make sure to watch for those blog entries, too.

LISA Training

I haven't posted anything about it here, but I'm working with Dan Klein to help get training ideas for LISA'13. For the past several years, I've been involved as a blogger at the LISA conference (along with Ben Cotton, Marius Ducea, Greg Riedesel, and many others. I'm planning on continuing that for as long as they'll have me, but it's also nice to be able to contribute to the program in some small way, too. This means that if there's training that you think LISA should have, but doesn't, let me know and I'll do my best to figure out how we can have it.

Actual, "I get paid to do this" work stuff

At work, we've been doing all kinds of things. I've now got a production vSphere cluster, a new Nimble storage box, I'm trying desperately to get new gear for my core switch (I'm going with a pair of Nexus 5548s and six FEX to go along), and I need to order more five or six server racks to replace some of the ones we have now.

I continue to be mystified by the way that academia works. Specifically, budgeting and deadlines. For reasons that I'm unable to fathom, in order to get things on this year's budget, I have to order hardware and have it delivered and in my space by the end of June. Not, "ordered and paid for". Ordered, delivered, and in my space. I've thought about it, and I can't come up with any kind of compelling reason for this rule. Anyone with more experience in academia than I have want to weigh in? I'm at a loss.

Personal Stuff

I've finally bit the bullet and decided to get LASIK.

I'm in a large-ish metro area now, and the technology has been continually developing for a couple of decades, and I think it's matured to the point where I'm cool with people cutting my eye open and burning part of it away using lasers. I can't be 100% about technology enhancing our lives unless I walk the walk and take advantage of it, so I'm doing it.

I went in last week for my "free consultation", which determined that I was an excellent fit for normal "LASIK" surgery. If my cornea had been too thin, I guess I could have gotten either LASEK or PRK, both of which work well but have a longer healing and recovery time. Turns out my cornea is just fine.

Also, can I just say - they have the coolest eye equipment I've ever seen there. I've worn glasses or contacts since elementary school, and I've lived in a dozen cities or so since then, so I've seen my share of optometry equipment, but man, the toys the LASIK guys have are nuts. I'm practically blind, so when they said, "take off your glasses and look in this machine, and you'll see a hot-air balloon", I thought, "please, I'll be lucky to see a blurry light". Sure enough, looking into the machine, it was blurry...for a second. Then, like a camera, it "autofocused" and just like that, they had nearly my exact prescription. Awesome!

So the whole "lasering my eyeballs" thing is happening tomorrow afternoon. I honestly can't wait. I've been thinking about it for years, and having it this close is really exciting. I'll make sure to update early next week with the results.

So there you go. That's what I've been up to. I'll try to get back to posting more regularly, and maybe even on topics that you care about! Wouldn't that be exciting? ;-)

We'll see. Thanks!

Changing my ssh port from the default port (22) has been one of my standard processes for quite some time when I build new servers or virtual machines. However, I see arguments crop up regularly about it (like this reddit thread or this other one).

Before I go any further, let’s settle the “security through obscurity” argument. (This could probably turn into its own post but I’ll be brief for now.) Security should always be applied in layers. This provides multiple levels of protection from initial attacks, like information gathering attempts or casual threats against known vulnerabilities. In addition, these layers of security should be applied within the environment so that breaking into one server after getting a pivot point in the environment should be just as difficult (if not more difficult) than the original attack that created the pivot point. If “security through obscurity” tactics make up one layer of a multi-layered solution, I’d encourage you to obscure your environment as long as it doesn’t affect your availability.

The key takeaway is:

Security through obscurity is effective if it’s one layer in a multi-layer security solution

Let’s get back to the original purpose of the post.

The biggest benefit to changing the port is to avoid being seen by casual scans. The vast majority of people hunting for any open ssh servers will look for port 22. Some will try the usual variants, like 222 and 2222, but those are few and far between. I ran an experiment with a virtual machine exposed to the internet which had sshd listening on port 22. The server stayed online for one week and then I changed the ssh port to 222. The number of attacks dropped by 98%. Even though this is solely empirical evidence, it’s clear that moving off the standard ssh port reduces your server’s profile.

If it’s more difficult to scan for your ssh server, your chances of being attacked with an ssh server exploit are reduced. A determined attacker can still find the port if they know your server’s IP address via another means (perhaps via a website you host) and they can launch attacks once they find it. Paranoid server administrators might want to check into port knocking to reduce that probability even further.

Remembering the non-standard ssh port can be annoying, but if you have a standard set of workstations that you use for access your servers, just utilize your ~/.ssh/config file to specify certain ports for certain servers. For example:

Host *.mycompany.com
  Port 4321
 
Host nonstandard.mypersonalstuff.com
  Port 2345
 
Host *.mypersonalstuff.com
  Port 5432

If you run into SELinux problems with a non-standard ssh port, there are plenty of guides on this topic.. The setroubleshoot-server package helps out with this as well.

# semanage port -a -t ssh_port_t -p tcp 4321
# semanage port -l | grep ssh
ssh_port_t                     tcp      4321,22

Here is my list of ssh lockdown practices when I build a new server:

• Update the ssh server package and ensure that automatic updates are configured
• Enable SELinux and allow a non-standard ssh port
• Add my ssh public key to the server
• Disable password logins for ssh
• Adjust my AllowUsers setting in sshd_config to only allow my user
• Disable root logins
• For servers with sensitive data, I install fail2ban

Changing your ssh server’s port from the default: Is it worth it? is a post from: Major Hayden's blog.

Thanks for following the blog via the RSS feed. Please don't copy my posts or quote portions of them without attribution.

Today my main machine was down for about 8 hours. Oops.

That meant when I got home, after a long and dull train journey, I received a bunch of mails from various hosts each saying:

• Failed to fetch slaughter policies from rsync://www.steve.org.uk/slaughter

Slaughter is my sysadmin utility which pulls policies/recipies from a central location and applies them to the local host.

Slaughter has a bunch of different transports, which are the means by which policies and files are transferred from the remote "central host" to the local machine. Since git is supported I've now switched my policies to be fetched from the master github repository.

This means:

• All my servers need git installed. Which was already the case.
• I can run one less service on my main box.
• We now have a contest: Is my box more reliable than github?

In other news I've fettled with lumail a bit this week, but I'm basically doing nothing until I've pondered my way out of the hole I've dug myself into.

Like mutt lumail has the notion of "limiting" the display of things:

• Show all maildirs.
• Show all maildirs with new mail in them.
• Show all maildirs that match a pattern.
• Show all messages in the currently selected folder(s)
  • More than one folder may be selected :)
• Shall all unread messages in the currently selected folder(s).

Unfortunately the latter has caused an annoying, and anticipated, failure case. If you open a folder and cause it to only show unread messages all looks good. Until you read a message. At which point it is no longer allowed to be displayed, so it disappears. Since you were reading a message the next one is opened instead. WHich then becomes marked as read, and no longer should be displayed, because we've said "show me new/unread-only messages please".

The net result is if you show only unread messages and make the mistake of reading one .. you quickly cycle through reading all of them, and are left with an empty display. As each message in turn is opened, read, and marked as non-new.

There are solutions, one of which I documented on the issue. But this has a bad side-effect that message navigation is suddenly complicated in ways that are annoying.

For the moment I'm mulling the problem over and I will only make trivial cleanup changes until I've got my head back in the game and a good solution that won't cause me more pain.

Normally on this blog I tend to write about more complicated tasks or fancy Linux tricks and completely overlook some of the most basic tasks that a SysAdmin needs to know. Today I have decided that I will make my blog a little more comprehensive and add some posts with some of the basics.

Along with this I will be starting a new category, called Sysadmin Basics and I will try to post an additional article each week that covers some of the more basic concepts and commands used by Linux and Unix Sysadmins.

Remove Directories with the rmdir command

The rmdir command is used to delete and remove empty directories. I bolded empty as it is important to note that rmdir will only remove a directory if there are no files within that directory. If you want to remove a directory and all files within that directory, skip down to the rm section of this article.

Remove a single empty directory

# rmdir somedir/

Remove multiple empty directories (in a single tree)

# rmdir -p somedir/a/b/c/d/e/f/whoa

While rmdir will not remove directories with files in it; rmdir will recursively remove a directory tree that has no files. In the example somedir only has directory a within it, and the a directory only has b which only has c and so on.

Remove multiple empty directories

The above command will also fail if there are multiple directories in one single directory, to handle that scenario you can list the directories individually and include the –ignore-fail-on-non-empty flag.

# rmdir --ignore-fail-on-non-empty -p somedir/a/b/c/ somedir/a2/b2/

Without the –ignore-fail-on-non-empty flag the command will still print that somedir is not empty even though it removes somedir. This is due to the fact that both command line arguments ask rmdir to remove somedir and rmdir cannot remove that directory until the last step.

Removing Files and Directories with the rm Command

While the rmdir command is solely for directories the rm command can remove both files and directories. With the right combination of flags rm will also remove entire directories, files and all.

Remove a file

# rm a-file
 rm: remove regular empty file `a-file'? y

On it’s own rm will not prompt a user before removing a file; to keep systems safe from accidental file removals some distributions of Linux will ship with an alias for rm with the default .bashrc file. This alias gives the interactive (-i) flag for rm, this tells rm to prompt the user before removing files and directories.

# alias
alias rm='rm -i'

Remove a file without being prompted

While you can simply unalias the rm alias, a simplier and generally used method to remove files without being prompted is to add the force (-f) flag to the rm command. It is advisable that you only add the force (-f) flag if you really know what you are removing.

# rm -f b-file

Remove a file without being prompted and with verbosity

If you don’t want to be prompted for each file removable but also want to keep an eye on rm in case the command starts removing unexpected files, you can simply add the verbose (-v) flag.

# rm -fv c-file
 removed `c-file'

Remove multiple files

There are many ways to remove multiple files, one method is to simply list each file you want to remove.

# rm -f a-file b-file

Removing multiple files with a wildcard

The bash command line supports wildcards and regex statements. A simplier way to remove all files that end in the word file is to simply state *file. I suggest being cautious with wildcards as it is entirely possible to remove a file without meaning to.

# rm -f *file

Remove files using a regex

Another common method of deleting files is to use regex statement, the below would remove anything that looks like files-0 through files-9 but would not remove files-a or files-list.

# rm -f files-[0-9]

Remove a directory and all of it’s contents with rm

If you want to simply remove an entire directory and all of the contents within, including both files and directories the easiest method is to add the recursive (-R) flag to rm. If you are in any way unsure of what you are doing than drop the force (-f) and replace it with verbose (-v) or interactive (-i).

# rm -Rf somedir/

You may have read the Popular Science article:

Thieves Stole $45 Million From ATMs Because The U.S. Uses Absurd 40-Year-Old Technology

Let me quote:

So why is the US so far behind? Infrastructure is a major factor; countries like Japan and the UK are much smaller, so replacing all the old point-of-sale machines and ATMs is easier.

Bullshit.

Bullshit. Bullshit. Bullshit.

The reason is that bank executives had the choice between paying a lot of money to do the right thing or a little money to consultants who would tell them what they wanted to hear. It's a big win for consultants.

WHY IS POPULAR SCIENCE BEING SO ANTI-CONSULTANT???

Everyone got what they asked for. What's so bad about that?

And besides, I'm sure the banks are insured for this kind of thing.

The real headline should be, "Insurance companies lose $45 million from signing contracts with banks that couldn't care less because they've signing contracts with insurance companies that remove the need for them to give a shit."

Amirite? (No, really, can someone from the banking industry confirm?)

Adding static routes in Linux can be troublesome, but also absolutely necessary depending on your network configuration. I call static routes troublesome because they can often be the cause of long troubleshooting sessions wondering why one server can’t connect to another.

This is especially true when dealing with teams that may not fully understand or know the remote servers IP configuration.

The Default Route

Linux, like any other OS has a routing table that determines what is the next hop for every packet.

Print the routing table contents

There are numerous commands that show the routing table but today we will use the ip command as this command will be replacing the route command in future releases.

# ip route show
 10.1.6.0/26 dev eth0 proto kernel scope link src 10.1.6.21
 10.1.7.0/24 dev eth1 proto kernel scope link src 10.1.7.41
 default via 10.1.6.1 dev eth0

As you can see in the example routing table there are numerous routes however 1 route shows as the default route. This routing table tells the system that if the IP that is being communicated to does not fall into any of the other routes than send the packets to the default route defined as 10.1.6.1. The default route basically acts as a catchall for any packet that isn’t being told what to do in the above routes.

Our Example System

In today’s article I will be referencing an example network configuration in order to show how static routes are added, why to add them and some basic troubleshooting.

Example Interface Configuration

eth0:

# cat /etc/sysconfig/network-scripts/ifcfg-eth0
DEVICE=eth0
BOOTPROTO=static
IPADDR=10.1.6.21
NETMASK=255.255.255.192
ONBOOT=yes

eth1:

# cat /etc/sysconfig/network-scripts/ifcfg-eth1
DEVICE=eth1
BOOTPROTO=static
IPADDR=10.1.7.41
NETMASK=255.255.255.0
ONBOOT=yes

Example Default Route Configuration

# cat /etc/sysconfig/network
NETWORKING=yes
HOSTNAME=testing.example.com
GATEWAY=10.1.6.1

The GATEWAY configuration in /etc/sysconfig/network tells the system that 10.1.6.1 is the default route. This configuration could also be added to /etc/sysconfig/network-scripts/ifcfg-eth0 file; However if multiple ifcfg-<interface> files have a GATEWAY this may provide unexpected results as there can only be one default route.

Example Why we need a static route

For our example network configuration we have two interfaces; eth0 (10.1.6.21) for the internet, and eth1 (10.1.7.41) for the internal network. If we were to hook up to a backup server such as 10.1.5.202 we would want the connectivity to go through eth1 the internal network, rather than eth0 which is the internet network.

Since 10.1.5.202 is not in the same subnet at eth1 (10.1.7.0/24) the routing table does not automatically route the packet through eth1 and would then hit the “catchall” default route out eth0. To force all of our packets destined for 10.1.5.202 out eth1 we will need to set up a static route.

Adding a Static Route

Adding the route to the current routing table

Adding the static route is a fairly simple task however before we start we must first know the gateway for the internal network; for our example the gateway is 10.1.7.1.

Adding a single IP

# ip route add 10.1.5.202/32 via 10.1.7.1 dev eth1

The above command adds a route that tells the system to send all packets for 10.1.5.202 and only that IP to 10.1.7.1 from device eth1.

Adding a subnet of IP’s

In order to add a whole subnet than you will need to change the CIDR on the end of the IP. In this case I want to add anything in the 10.1.5.0 – 10.1.5.255 IP range. To do that I can specify the netmask of 255.255.255.0 in CIDR format (/24) at the end of the IP itself.

If a CIDR (or netmask) is not specified the route will default to a /32 (single ip) route.

# ip route add 10.1.5.0/24 via 10.1.7.1 dev eth1

The difference between these two routes is that the second will route anything between 10.1.5.0 and 10.1.5.255 out eth1 with 1 route command. This is useful if you need to communicate with multiple servers in a network and don’t want to manage lengthy routing tables.

Adding the route even after a network restart

While the commands above added the static route they are only in the routing table until either the server or network service is restarted. In order to add the route permanently the route can be added to the route-<interface> file.

# vi /etc/sysconfig/network-scripts/route-eth1

Append:

10.1.5.0/24 via 10.1.7.1 dev eth1

If the above configuration file does not already exist than simply create it and put only the route itself in the file (# comments are ok). When the interface is restarted next the system will add any valid route in the route-eth1 file to the routing table.

I highly suggest that when possible anytime you add a route to the route-<interface> files that the interface itself is restarted to validate whether the route is actually in place correctly or not. I have been on many late night calls where a static route was not added correctly to the configuration files and was removed on the next reboot, which is also long after everyone has forgotten that a static route was required.

Troubleshooting a Static Route

Check if the route is in the routing table

Before performing any deep down troubleshooting steps the easiest and first step should be to check if the routing table actually has the route you expect it to have.

# ip route show
 10.1.5.0/24 via 10.1.7.1 dev eth1
 10.1.6.0/26 dev eth0 proto kernel scope link src 10.1.6.21
 10.1.7.0/24 dev eth1 proto kernel scope link src 10.1.7.41
 default via 10.1.6.1 dev eth0

Use tcpdump to see tcp/ip communication

The easiest way that I have found to find out whether a static route is working correctly or not is to use tcpdump to look at the network communication. In our example above we were attempting to communicate to 10.1.5.202 through device eth1.

# tcpdump -qnnvvv -i eth1 host 10.1.5.202
tcpdump: listening on eth1, link-type EN10MB (Ethernet), capture size 96 bytes
16:50:35.880941 IP (tos 0x10, ttl 64, id 59563, offset 0, flags [DF], proto: TCP (6), length: 60) 10.1.7.41.41403 > 10.1.5.202.22: tcp 0
16:50:35.881266 IP (tos 0x0, ttl 59, id 0, offset 0, flags [DF], proto: TCP (6), length: 60) 10.1.5.202.22 > 10.1.7.41.41403: tcp 0

The above tcpdump command will only listen on eth1 and output only results that to or from 10.1.5.202.

TCP connections require communication from both the source and the destination, to validate a static route you can simply initiate a tcp connection (telnet to port 22 in this case) from the server with the static route to the destination server. In the output above you can see communication from 10.1.7.41 to 10.1.5.202 from the eth1 interface, this line alone shows that the static route is working correctly.

If the static route was incorrect or missing the tcpdump output would look similar to the following.

# tcpdump -qnnvvv -i eth1 host 10.1.5.202
tcpdump: listening on eth1, link-type EN10MB (Ethernet), capture size 96 bytes
16:50:35.881266 IP (tos 0x0, ttl 59, id 0, offset 0, flags [DF], proto: TCP (6), length: 60) 10.1.5.202.22500 > 10.1.7.41.22: tcp 0

In the above, only the target server is communicating over eth1.

Lumail <http://lumail.org> received two patches today, one to build on Debian Unstable, and one to build on OpenBSD.

The documentation of the lua primitives is almost 100% complete, and the repository has now got a public list of issues which I'm slowly working on.

Even though I can't reply to messages I'm cheerfully running it on my mail box as a mail-viewer. Faster than mutt. Oddly enough. Or maybe I'm just biased.

Following the recent post on how to investigate limit related issues which gave instructions what to check if you suspect a system limit to be hit I want to share this Nagios check to cover the open file descriptor limit. Note that existing Nagios plugins like this only check the global limit, only check one application or do not output all problems. So here is my solution which does:

1. Check the global file descriptor limit
2. Uses lsof to check all processes "nofile" hard limit

It has two simple parameters -w and -c to specify a percentage threshold. An example call:

./check_nofile_limit.sh -w 70 -c 85

could result in the following output indicating two problematic processes:

WARNING memcached (PID 2398) 75% of 1024 used CRITICAL apache (PID 2392) 94% of 4096 used

Here is the check script doing this:

#!/bin/bash

# Check "nofile" limit for all running processes using lsof

MIN_COUNT=0	# default "nofile" limit is usually 1024, so no checking for 
		# processes with much less open fds needed

WARN_THRESHOLD=80	# default warning:  80% of file limit used
CRITICAL_THRESHOLD=90	# default critical: 90% of file limit used

while getopts "hw:c:" option; do
	case $option in
		w) WARN_THRESHOLD=$OPTARG;;
		c) CRITICAL_THRESHOLD=$OPTARG;;	
		h) echo "Syntax: $0 [-w <warning percentage>] [-c <critical percentage>]"; exit 1;;
	esac
done

results=$(
# Check global limit
global_max=$(cat /proc/sys/fs/file-nr 2>&1 |cut -f 3)
global_cur=$(cat /proc/sys/fs/file-nr 2>&1 |cut -f 1)
ratio=$(( $global_cur * 100 / $global_max))

if [ $ratio -ge $CRITICAL_THRESHOLD ]; then
	echo "CRITICAL global file usage $ratio% of $global_max used"
elif [ $ratio -ge $WARN_THRESHOLD ]; then
	echo "WARNING global file usage $ratio% of $global_max used"
fi

# We use the following lsof options:
#
# -n 	to avoid resolving network names
# -b	to avoid kernel locks
# -w	to avoid warnings caused by -b
# +c15	to get somewhat longer process names
#
lsof -wbn +c15 2>/dev/null | awk '{print $1,$2}' | sort | uniq -c |\
while read count name pid remainder; do
	# Never check anything above a sane minimum
	if [ $count -gt $MIN_COUNT ]; then
		# Extract the hard limit from /proc
		limit=$(cat /proc/$pid/limits 2>/dev/null| grep 'open files' | awk '{print $5}')

		# Check if we got something, if not the process must have terminated
		if [ "$limit" != "" ]; then
			ratio=$(( $count * 100 / $limit ))
			if [ $ratio -ge $CRITICAL_THRESHOLD ]; then
				echo "CRITICAL $name (PID $pid) $ratio% of $limit used"
			elif [ $ratio -ge $WARN_THRESHOLD ]; then
				echo "WARNING $name (PID $pid) $ratio% of $limit used"
			fi
		fi
	fi
done
)

if echo $results | grep CRITICAL; then
	exit 2
fi
if echo $results | grep WARNING; then
	exit 1
fi

echo "All processes are fine."

Use the script with caution! At the moment it has no protection against a hanging lsof. So the script might mess up your system if it hangs for some reason. If you have ideas how to improve it please share them in the comments!

Today I’ll be live blogging the May 2013 CentOS Dojo event held in Scottsdale Arizona. I can’t find a specific itenerary for what talks will be given when, but this isn’t a multiple track event, so we all get what they give, and it looks like some cool talks will be given. The event starts at 9:30AM Arizona time (currently the same time as Los Angeles). Refresh this page for updates!

Talk #1: Pinterest and scaling

Older code at Pinterest is Python but the newer stuff is in the Java stack. Use Ostrich for Java processes. Python is using Sentry, StatsD, distributed tracing system is an internal tool called Kafka.

Application metrics are in Varnish / MySQL / Nginx / Redis. There are some tools like Percona’s toolkit.

If Pinterest is creating codes and needs metrics, they don’t like gauges. Gauges are a “right now” view. That’s not as useful because you can miss a lot if there is a giant spike between collection periods. They use “counters”, and while you might not know what happened between two sample periods, but you absolutely know that something happened and how much happened.

The nice thing about a counter is that if there’s a collection period downtime, counters in your code are always putting up numbers.

They choose to measure requests per second to get ideas of which systems are unbalanced. Read vs write requests, and of course errors. They measure latency, but also payload size. They saw latency issues in memcache, but they noticed that payload was huge which explained the latency.

• Ganglia is used for Cluster Metrics and Host Metrics
• OpenTSDB for high cardinality and alerting data (Some metrics are really unique and ephemeral, like memcache slabs and EC2 autoscaling.
• Kafka for event data
• Graphite for Application Metrics. Simple incremental monitoring for new applications as they’re made.

They use cgroups that allow them to run collectors in isolation. Kernel processes are separated from user space processes and user code is protected from kernel code. So there’s a cgroup for monitoring that has constrained resources. That way monitoring doesn’t take down the entire system. In the worst case scenario of bad monitoring code, then the monitoring gets killed with the OOM controller. Not the core services that you’re offering.

They use Ganglia because it requires the least amount of touch. It’s set up and they’ve had very few problems with it. They use RRDcache to buffer the stats to disk and don’t use solid states. They don’t use EBS because they don’t want to. They use local ephemeral storage.

Graphite and StatsD are used. The talk on this was very vast and over my head. Sorry.

OpenTSDB. They have 60 billion points on a 10 node cluster. They keep network performance data, memcacheD ketama ring and slab stats, HBase dynamic metrics.

Their graphite front end is written in Flask. Other tools like D3 and rickshaw for UI elements. JSON is good. Data sources need to speak JSON to their front ends (or so people say).

They use R for really complex, weird data queries. If you have data of any kind, you can look at it with R in esoteric ways, it’s just a little more difficult. Pinterest has a data science team that handles it.

Talk 2: Sensu as a monitoring system

Joe Miller (www.getpantheon.com) speaking. Sensu is a monitoring tool / framework. They do PaaS with Drupal.

Sensu has been called “the monitoring router.” It’s really a framework for building a monitoring tool. It’s Ruby 2.0 based, RabbitMQ for messaging, and Redis for basic amount of state storage. JSON is everywhere. You can simply re-use Nagios plugins. It is packaged with all the gems and Ruby packages that are needed.

Leverage your existing config management system to attach checks to your infrastructure. They tend to focus on Chef. There is a Puppet project.

The GUIs for Sensu are slim. There are two options. The Sensu-dashboard (which is included in the omnibus package that has everything in it). It has no user authentication and is stateless. No roles. The other option is Sensu-admin which is a separate package that has users, roles, scheduled downtimes, etc. It’s a Rails app.

Sensu runs on Checks, Events, and Handlers. Checks create events and are fed into handlers. Handlers are passed data simply through STDIN. There are handlers available to send information to logstache, graylog, and many more. It’s all in the community. Another way to diagram how Sensu works is that there is a Sensu-Server on one end, and Sensu-Clients on the other end with rabbitMQ in the middle. If you have a client, a node that needs to be monitored, it gets subscribed to a type, like webserver. It is monitored however webservers are defined. Messages passed from client to server, and then the server can fire off actions like alerts.

You don’t have to discover new systems, you don’t have to add it to any other monitoring system, it gooks into config management. The config files specific to Sensu are all JSON.

Some examples: Because every sensu client heartbeats via rabbit to the server, if it disappears for 180 seconds it generates an event that is sent to the default handler. However, for one large company, it will run a decommission script to remove that server from configuration management tools, and other systems that keep track of the system. Obviously, that’s intended for large cloud instances with lots of ethereal, come-and-go, servers.

18 month old project. Seems very malleable and able to morph with rapidly changing environments, or rather, make a static and rigid environment more flexible with the help of config management.

Talk #3

Linux Enablement for Calxeda ARM Servers by Rob Herring. He works for Calxeda, an ARM processor licesnsee that makes server deployments. They are making a “EnergyCore” arch for processors, quad core Cortex based ARM processors, with a fifth core dedicated to the IPMI, OOB management, etc. It includes a Fabric Switch on board with 10Gbit links, and of course an IO controller is included. It’s all the functionality of a server on one SOC. No South Bridge, North Bridge, etc.

The amazing part is that they are making four-node cards, so four SOCs, that are baked into one card that is about the size of a PCI card. The system board is essentially passive to connect all the nodes together. The first generation processor is called the ECX-1000, AKA “High Bank”, and it has a 1.4GHz quad core CortexA9. The next level up is the ECX-2000 CortexA15 that has virtualization support, KVM and Xen.

There are some example systems that are on the market. There are already example systems that are using massive amounts of these cards, including systems from HP and Boston Servers.

The Linux kernel has increasing ARM support, and major distros are including support for it, but that might be the next hurdle. The hardware is useful and can be used for low power, highly distributed applications, but the distribution support might need to come up more and more. UEFI, ACPI, and LPAE need to grow on Linux for ARM. A demonstration included running OpenStack on a Calxeda box and spinning up a new VM.

Talk #4

Mike McLane – Performance tuning as a web hosting provider. Works on the Genesis Team that is in GoDaddy to performance optimize and tune the services. GoDaddy started as web nodes attached to NAS.  Then went to multiple web nodes attached to NASs with a hardware load balancer in front.

They use nodes with lots of RAM and local disk space, but relatively conservative CPUS. CentOS 6.current is used. Apache 2.2.current but moving to Apache 2.4.current. All customer content is served via NAS (NFS) over local storage. GD is not using RHEL / CentOS 3 as is rumored. It’s  6.current.

They run mod_fcgid for FastCGI. mod_fastcgi was not ideal for scalable web hosting. They do run mod_rewrite, which is to dispel a myth running on the internet concerning GoDaddy not using mod_rewrite.

The use NAS because it was comparable to DAS / local storage for user experience so the didn’t change anything.

GD decided to take a close look at performance but they didn’t have quite the amount of data that they would need to make deep, scientific analysis of the large systems. They did queries for the top fastest loading pages, top slowest, 95th percentile, fastest 250 pages, etc. all measured across all customers, but it ended up being all over the place and just a bunch of numbers. Very hard to make hard conclusions from it.

They created the CEM (Customer Experience Monitor) that was basically a vanilla WordPress install on each and every host in the environment and then they’d load it and measure the performance. It wasn’t the best judge of performance. They watched all the numbers and none of the numbers really pointed to any one thing that was causing slowness. They used the flot javascript library to plot their data.

The problem was later found to be PHP serialization that if two requests came in at the exact same time, two processes would be spun up but the requests would be processed one after the other. It turns out there was a one second sleep in mod_fcgid that would be triggered to keep the box from being thrashed under certain circumstances. mod_fscgid author was notified and he made a nice patch, but apparently it’s not in the stable branch right at the moment.

That was a good first major win for tuning the GD stack. Then they implemented graphite to gauge the performance of the machines across the board. They sometimes use Apache bench to dump the stats to a CSV and then plot it out. Good ol’ iperf for bandwidth measurement.

It’s important to test performance form cold, warm and hot starts if you have different levels of cache. Cache can skew things.

They noticed that certain Xeon processors had problems with cstates. Even if power savings mode was disabled in the BIOS, some CentOS kernels would have stability and performance issues surrounding power state options.

Talk #5: Logging Love w/ Rashid Khan Elasticsearch Developer

Worked at a newspaper, they logged direct to disk, he then created Kibana. If you store locally, you have to ssh into each server, or use a really bad for loop for ssh. Then you get smarter and do a central syslog server. But then you end up making a massive grep/sed/awk spaghetti script. Log search scripts would break as things changed.

Then he started looking at commercial log analyzers, but they were punishing and hard. Enter: Logstash and Elasticsearch. Logstash is like an event pipe that works over the network. It takes things in with various inputs, amqp, eventlog, exec, irc, tcp, redis, twitter, udp, xmpp, etc. and etc.

Log files suck. Dates in particular. So many different types of date possibilities. Logstash can take in different date inputs and normalize. Logstash can use filters like grep, geoip, grep, metrics, multiline. Then outputs can be things like circonus, cloudwatch, elasticsearch, more and more and more. Rashid uses Elastic search to store logstash information in. It’s schema free, clusters very well, JSON over HTTP, does search and aggregation.

However, he needed an interface to manage all the data gathered and outputted by Logstash and held by elastic search. enter Kibana! It’s been re-written a number of times, PHP first, then Ruby to match Logstash, except Logstash is Ruby on Java through jruby and Kibana didn’t work well that way for a number of reasons. Now it’s all simple HTML and JavaScript interfacing with the ElasticSearch RESTful API.

Talk #6

Der.Hans talking about SSH and Bash tricks.

Set your shell’s prompt to something informative. $PS1 is the variable. \u@\h is a minimal set of information, that shows username and hostname. Sometimes you need the FQDN instead though. If you have your own account just use your shell config files on the remote machines, or you can use SSH config files. However, SSH config files runs before SSH launches your shell, so config files run with system default shell, not your user account’s shell.

In the case of not being able to modify any of those files, you can call ssh with a script that set up the remote environment on a temporary basis.

Obfuscate the names in known_hosts. If you put command= in your auth keys file, whatever command you put after the equal sign is run when you open a connection to that machine.

Set PermitRootLogin=”no” – never log in as root. You can pipe straight to ssh so you can tar on one machine and pipe it to ssh. Fuse-sshfs can mount a remote system as a local system.

Parallel SSH! for i in $( cat hostnames.txt ); do ssh $i some_command; done. However, Shmux is better! Also, ClusterSSH, parallel SSH, Mussh, Massh. sssl runs on a port and listens for incoming connections, determines what is intended for HTTP and what is for SSH and then sends the traffic on to the proper daemon.

SSH can perform local and remote tunnels, etc. etc. Check the -L and -R switches. Lots of examples, but I was paying attention and not writing.

Talk #7

Mike Dorman CentOS & Xen at GoDaddy. A couple years ago Go Daddy released a product called cloud servers. It turned out to not be such a great fit for GD’s target group. GD is more focused on the SMB that needs to have a simple website and have a simple online store. It’s been EOL’d and was completely turned off last week.

The development group for that products has come back together and they’re revamping it to an internal private cloud platform that will be sued internally. The vision is kind of like Google where they have their infrastructure that runs everything concerning the company, public and private. Maybe in the future GD will get off of the physical focus of dropping in more metal, and be more cloud based.

To provision a new server, there are server build times of 10, 12, to 15 weeks because of “The Checklist” that five, six, or seven groups have to look at and approve. The idea is to have a cloud platform to make it faster for servers to spin up and develop internal products and projects.

One of the troubles with monitoring it all is that the dynamic creation and destruction of machines is hard with traditional tools. They’ve also been trying to decouple the idea of physical servers from services. Physical servers are like puppies. You name them, care about them, and nurse them back to health when they’re sick. Virtual servers are like cattle. They’re given numbers, they’re driven hard, and if they get sick you shoot them in the head.

GoDaddy went with CloudStack at the outset of their decision to go cloud. Clustering and HA functions gave them trouble. It was essentially a forked version of cloudstack with code that couldn’t be submitted back up to the core project for various reasons.

They looked at CloudStack, and they ran into the problem that is in the OpenStack community about “clouds of clouds” or “pods of pods” where you can’t really grow a cloud too big. There were some management problems with it as well where it was not operationally simple to deal with. There are also concerns about the OpenStack community looking at the big players, their aren’t a lot of good contributions back into the core. A lot of the main users of OpenStack are essentially using forked versions of the project.

So, GD developed their own internal management stack and called it Norm, after the robot from some cartoon. It’s basically CloudStack, rebuilt into something very different.

So why not use XenServer as the hypervisor? It worked okay. You do have official Microsoft support if you want to run MSFT products in XenServer. However, it looks like Linux, but it can’t be treated like Linux. It has the network layer configured through a different configuration system. Management of XenServer couldn’t be done like a regular Linux machine.

Now they use stock CentOS and open source Xen. There is more of a community around CentOS, but the XenServer community isn’t comparable in a support sense. Patch management is easier because none of the XenServer patches are able to be patched through regular management systems like SpaceWalk. They never found a way to update XenServers to scale.

So, now after trying all the major cloud stacks, they’re building from CentOS and Xen raw and real.

I’m continuing to write over at The Virtualization Practice, and it’s been fun so far. Those of you following what I’ve been doing have probably seen me take a real turn towards converged infrastructures in the last six months, both for TVP and for TechTarget. Not that I don’t think the public cloud is attractive to many, but hardware vendors are doing some real interesting things that are keeping on-site IT fairly attractive. Plus the local telco lobbies and myopic/dirty legislators seem to be keeping inexpensive bandwidth, the Achilles heel of the cloud, to a minimum in most non-urban places. Anyhow, we’ve got:

• A Look at the Dell Active System 800

wherein I’m trying to figure out if Dell’s converged anything worth using (I think so, especially if you need physical deployments or want respectable amounts of RAM on your physical nodes). I largely ignore their vStart lineup, which is all basically the Active System without as much standardization and integrated management, but hey, I don’t have infinite time here, and vStart basically just converges how you buy a bunch of gear. Not that vStart is bad, just not what I consider convergence.

I also wrote about:

• OpenStack and the Software Defined Data Center

a few weeks ago during the OpenStack Summit as they were announcing some new, badass features in Grizzly. The rate OpenStack has been moving forward is very impressive, doing major releases every 4-5 months as they sprint to catch VMware (or pass them, if you need features like distributed tiered storage, software-defined networking, or that new-fangled IPv6 crap so you can be part of the 21st century and also talk to mobile devices, or anybody in Asia… not that I’m bitter). You still need a mad scientist or two on staff to run the thing yourself, but there’s finally enough good stuff in the base product that we’re seeing some real vendor offerings with real enterprise support at price points that are going to start making people think about using OpenStack somewhere other than their lab.

I just wonder when we’re going to start seeing the inevitable fragmentation of OpenStack, including a loss of interoperability. At a certain point all open source projects endure some infighting, or someone will have a good idea they want to keep to themselves, and the open source code means a codebase fork is the functional equivalent of taking one’s ball and going home. I hope I’m wrong, but I’d bet money it’ll happen in the next year. I’ll also bet that it won’t be VMware’s fault, though they’ll somehow end up the scapegoat. I just worry that the currently-bleak third-party tool ecosystem for OpenStack will remain that way if there’s fragmentation.

Anyhow, thanks for reading. Y’all rock.

This post was written by Bob Plankers for The Lone Sysadmin - Virtualization, System Administration, and Technology. Licensed under the Creative Commons Attribution-NonCommercial-ShareAlike 3.0 Unported License and copyrighted © 2005-2013. All rights reserved.

Related posts:

1. OpenStack Isn’t Our Savior from Lock-In or Support Costs
2. Shameless Self Promotion – VCE, TVP, and 787 Edition
3. OpenStack, Lock-In, Support Costs, and Open Source Free Lunches

What is your opinion of WAN Accelerators? Please post to the comments section.

I haven't used or configured a WAN Accelerator but from the reading I've done so far it seems that the older the protocol is, the better a W.A. will help. Older protocols were designed when we were ignorant of the networking realities we have today. More modern protocols tend to do their own compression, caching, don't do stupid things that fail over high latency links, and so on. In particular: Am I right but if you mostly have home-grown protocols, you can tune them better than a W.A. can accelerate them?

Please post to the comments section.

Tonight at 9pm Eastern time, there is going to be a special town-hall style Q&A meeting with the candidates who are running for the LOPSA Board of Directors. The meeting happens on IRC, but if you don't have an IRC client, LOPSA Leadership Committee member Aaron Sachs wrote instructions on how to connect with the web client, so check those out.

You might remember that I'm running for election, as well. My candidate statement is at that link, and I've written a series of articles on my LOPSA blog expounding on the internal concerns that I feel need more concentration by the Board.

Tonight would be a great time to get a feel for how the potential Board members feel about topics relevant to our profession.

I'll be reminding people to tune in via twitter prior to the meeting, so watch for it there. I'd really appreciate your support and your questions, so please come and take part. This is your profession, so start taking part in it now.

Edit: The session has finished and you can read the transcript here:
https://lopsa.org/content/lopsa-live-may-9-2013-candidate-forum-transcript.

A massive marathon has resulted in my lumail mail client working well.

Functionally the application looks little different to the previous C-client, but it is a lot cleaner, neater, and nicer internally.

The configuration file luamail.lua gives a good flavour of the code, and the github repository has brief instructions.

Initially I decied that the navigation/index stuff was easy and the rest of the program would be hard; dealing with GPG-signatures, MIME-parts, etc.

But I'm stubborn enough to keep going.

If I can get as far as reading messages, with MIME handled properly, and replying then I can switch to using it immediately which will spur further development.

I'm really pleased with the keybinding code, and implementing the built-in REPL-like prompt was a real revelation. Worht it for that alone.

The domain name lumail.org was available. So I figured why not?

I hope to teach a "how to interview" class at an upcoming conference. Here's one of the points I'll be making.

How can one interview question help me understand what the candiate does and doesn't know Unix?

Here's the question:

What happens with I type this at a shell prompt:

telnet www.wikipedia.org 80 RETURN

Usually the candiate will explain just the command: "It opens a connection to port 80 on wikipedia". That's a good answer.

The follow-up question is, "Please give me more detail".

At this point they might explain how DNS works or how TCP network connections work. Whichever they explain, you can ask about the other one.

That is, of course, skipping the fact that the command was started: You can ask them to explain how a process is created in Unix. You can ask them to explain how command lines are parsed.

Basically you can drill down and down and down and down. Any time the candiate says, "I don't know" ask them to guess. After the guess, move to some other aspect of the system.

You can do this in 15 minutes or 15 hours depending on how detailed you ask the candidate to get.

The point is that you will learn a lot about what the candidate does and doesn't know. Don't get hung up if they don't know half the issues you bring up, just make sure the parts that are known are the parts the job opening requires knowledge. Watch the "guesses" to see if they are logical or do they require what I will politely call "magical thinking". Not knowing how DNS works is forgivable for some positions. Thinking that TCP works with hostnames instead of IP addresses is not forgivable. Thinking that "the computer just knows" is a magical thinking and is not forgivable.

I call this "Zeno's Interview Question" because you never get to the end. There is always more to discuss if you break the situation down into smaller and smaller parts to be described. Generally the conversation doesn't go linearly. You often ask "what happens before that?" When all is said and done you probably have traversed the timeline back and forth and back and forth; moving up and down layers of abstractions again and again.

You can use different command lines depending on if the position is for a sysadmin or software developer. The above example is good for a sysadmin because it involves DNS and networking. A systems programmer might be asked "cat *" instead, especially if you want to introduce that there is a file called "-f" in the current directory. You'll learn a lot about whether or not the candidate understands how command line parsing works.

Here is a list of topics this can generate:

• How USB keyboards transmit keystrokes.
• How keystrokes get processed through the kernel.
• How the unix concept of pty "raw vs. cooked" mode.
• How the shell parses lines (quotes then variables then {} then globs).
• How the shell executes commands: aliases then relative path or search $PATH.
• How commands are executed: all about fork() and exec()
• How a process starts: the Unix disk I/O system, virtual memory, paging executables into memory, bss vs data. Jumping to the first instruction.
• How argv is passed to the process.
• How a process parses flags.
• How DNS lookups work (in the simple case)
• How DNS lookups work (in the recursive case)
• To send the DNS packet we need to discuss:
• How L3 routing works
• How L2 packet passing works (Ethernet Switching, "cam" tables)
• VLANs
• Finally we have an IP address. Now we discuss...
• How a TCP connection is created.
• How sequence numbers work.
• How a TCP connection sends data.
• Nagle's Algorithm
• My fingers are getting tired.

(Note: the above list is incomplete, by definition. It probably also includes a few errors.)

We discussed this at the monthly meeting of the New Jersey Chapter of LOPSA a few years ago. It was 60 minutes before we got to the point that a packet hit the network. At the 2-hour mark we stopped the meeting because, to be honest, we were all pretty exhausted.

If you use this as an interview question I highly recommend you first tell the candidate how much time you've allocated for this question. If they know you want this to last 20 minutes, for example, it helps them give an indication of how much detail is being expected.

The best part of this question is that there's no way to "cheat". If the candidate has heard this question before it doesn't help them. The only way to "cheat" is to read all the books and other resources required to learn all of this. That's called education.

I’m a little behind on my reading, but I wanted to address Major Hayden’s blog posts about disabling Security-Enhanced Linux, or SELinux, which brings mandatory access control to Linux. Mandatory access control is a completely different permission model for UNIX-based hosts, and Mr. Hayden feels it is underutilized:

After many discussions with fellow Linux users, I’ve come to realize that most seem to disable SELinux rather than understand why it’s denying access. In an effort to turn the tide, I’ve created a new site as a public service to SELinux cowards everywhere: stopdisablingselinux.com.

It’s pretty rare for me to argue against a security technology but in my eyes SELinux isn’t a solution to very many problems. I know how SELinux works, what it does, how to configure it and troubleshoot it, and as a result I disable it everywhere. Here’s why:

1. While SELinux ships as part of most Linux distributions, few distribution maintainers have it set up correctly for all the services that ship with their OS. Perhaps web servers work fine if you use the defaults, but deviate from those, or install another service like a mail server or tftpd and there’s a real good chance it’ll malfunction. And by “malfunction” I mean “fail silently” which just leads to a bunch of time spent troubleshooting. That’s time out of my life I’m never getting back again. On top of that, patching those distributions sometimes resets the SELinux configurations, which both annoying and a symptom of Linux vendors not caring enough to do a good job. You can deal with patching problems with more testing, but that’s also more time and effort down the drain.

2. Absolutely no third-party vendors support SELinux[0]. Which is unfortunate, as that’s one of the most compelling use cases since those vendors often have gaping, unpatched security holes in their products. As a result it’s rare that you’ll have an entire environment with SELinux turned on, which means you’ve just added yet another variable to account for in your data center (has SELinux enabled, has SELinux in permissive mode, or does not have SELinux enabled), three times as many system configurations to account for, and you should do three times more testing, too.

3. SELinux protects the interface between applications and the OS. It doesn’t inherently protect the application, beyond keeping the host itself secure. That’s important, but less so now that we have clouds and virtualization. Back when we bought one expensive server, ran one OS image on that server, and ran a ton of different apps alongside each other in that single OS image it was very important to protect one application from another. Now that we have lots of smaller OS images running in isolation inside virtual machine monitors the focus is on application security. If an intruder breaks through the application and manages to do something to the OS you redeploy the application on a fresh VM. Only this time it’s with the latest OS & application patches applied, too.

For busy IT shops and busy sysadmins return on investment is always a factor. In my eyes SELinux is a tool which will never save you enough time to justify having implemented it in the first place. Very low ROI. When it is up to me I’d rather see scarce IT time and money spent on more productive security initiatives:

• Implementing regular OS patching. So few places do this well or in a timely fashion and it treats the root cause of many security problems directly. SELinux treats symptoms, not causes.
• Implementing better service design, so that patching and system outages can take place without service outages, and that services might be scalable to handle additional workload from users or denial-of-service attacks.
• Implementing configuration management tools, like Chef or Puppet. If time is money these systems print money. They open the door to better testing, rapid & repeatable deployments, elastic performance scaling, and good system documentation. I’ve also found they also lead to better, tighter security controls. For example, instead of adding large IP ranges to firewall rules “because it’s easier” you add just the specific IPs you need. With these tools it’s not a big deal to add another IP later if you need to. It’s also possible to do configuration audits with these tools, thereby finding discrepancies, potential backdoors, permission errors, etc. Heck, proper implementation of these tools should make enabling SELinux easier, but I still would take a critical look at the overall ROI of that decision versus other moves you can make.
• Implementing better application security, whether it’s code reviews, automated code testing, IDS/IPS, patching third-party apps, etc. The application is where your data lives and is often the weakest link in the security chain. Nowadays the OS is expendable, and rather than protect the OS under the application we should focus instead on hardening the application itself. You focus on intruders coming into your house via the doors and windows, right? Not through the middle of the basement floor.

In an ideal world we’d all have time to set this sort of thing up, and it would work perfectly. This isn’t an ideal world, and as such compromises are made. Don’t let someone shame you into wasting your time by calling you a SELinux coward. You’re in good company[1].

—-

[0] I have not checked all of them. It’s possible one or two do, but I would be surprised.

[1] At the very least you have company. :)

This post was written by Bob Plankers for The Lone Sysadmin - Virtualization, System Administration, and Technology. Licensed under the Creative Commons Attribution-NonCommercial-ShareAlike 3.0 Unported License and copyrighted © 2005-2013. All rights reserved.

Related posts:

1. Stop Patching your OS
2. Bad Day For People Who Actually Patch
3. Revenge of the Developer