Planet SysAdmin


Upcoming sysadmin conferences/events

Contact me to have your event listed here.


August 21, 2014

TaoSecurity

Air Force Leaders Should Read This Book

I just finished reading The Icarus Syndrome: The Role of Air Power Theory in the Evolution and Fate of the U.S. Air Force by Carl Builder. He published this book in 1994 and I wish I had read it 20 years ago as a new Air Force second lieutenant. Builder makes many interesting points in the book, but in this brief post I'd like to emphasize one of his concluding points: the importance of a mission statement.

Builder offers the following when critiquing the Air Force's mission statement, or lack thereof, around the time of his study:

[Previous] Air Force of Staff, General John P. McConnell, reportedly endorsed the now-familiar slogan

     The mission of the Air Force is to fly and fight. 

Sometime later, the next Chief, General John D. Ryan, took pains to put it more gruffly:

     The job of the Air Force is to fly and to fight, and don't you ever forget it. (p 266)

I remember hearing "Fly, Fight, Win" in the 1990s as well.

Builder correctly criticizes these mission statements on multiple grounds, none more compelling than this: how are non-flyers supposed to interpret this statement? It's simply a reminder and reinforcement of the second-class status of non-flyers in the Air Force. Furthermore, Builder more or less also notes that "fight" is often eclipsed but non-combat missions, such as airlift or humanitarian relief. Finally, Builder doesn't ask the question explicitly, but how does one define "winning"? Would wars in Iraq or Afghanistan be a "win"? That's a demoralizing way to think in my opinion.

Builder offers a wonkish, but conceptually more useful, mission statement on p 284:

The mission of the Air Force is the military control and exploitation of the aerospace continuum in support of the national interests.

The author immediately notes that one Air Force officer criticized Builder's mission statement as too "academic," but I think this particular policy wonk is on target.

Curious as to what the current Air Force mission statement says, I checked the Our Mission page and read at the top:

The mission of the United States Air Force is to fly, fight and win … in air, space and cyberspace.

Wow. That's even worse than before. Not only does it still insult non-flyers, but now the mission involves "flying" in "cyberspace."

I strongly suggest Air Force leaders read Builder's book. It's as relevant today as it was 20 years ago.

by Richard Bejtlich (noreply@blogger.com) at August 21, 2014 09:56 PM

RISKS Digest

The Lone Sysadmin

Google Chrome Missing The URL in the Address Bar

Chrome Origin Chip in Address Bar Omnibox

The latest Google Chrome update changed the omnibox/address bar so you cannot see the URL by default. I, for one, hate it. I’m not being curmudgeonly[0], I copy URLs all the time and it’s just another step for me to click on the “origin chip” (as it’s being called) to see the URL.

Here’s how to change it back:

  1. Go to chrome://flags/#origin-chip-in-omnibox (you’ll likely have to cut & paste this in)
  2. Pick “Disabled”
  3. Click the “Relaunch Now” button at the bottom.

Rejoice in having Chrome the way you like it.

Chrome Origin Chip Gone in Address Bar Omnibox

 

—-

[0] Yeah, I know, I am. GET OFF MY LAWN.


Did you like this article? Please give me a +1 back at the source: Google Chrome Missing The URL in the Address Bar

This post was written by Bob Plankers for The Lone Sysadmin - Virtualization, System Administration, and Technology. Licensed under the Creative Commons Attribution-NonCommercial-ShareAlike 3.0 Unported License and copyrighted © 2005-2013. All rights reserved.

by Bob Plankers at August 21, 2014 07:21 PM

Standalone Sysadmin

Ubuntu and SNMP

After running Ubuntu for about two years now, I have a laundry list of complaints. Whether Ubuntu is automagically starting daemons that I install, or the relative difficulty of running an internal repo, or (and I'm heartily agreeing with my coworker Nick here) that it doesn't actually include a firewall out of the box....there are very basic issues I have with running and managing Ubuntu machines.

The one that inspired this entry, though, is like, super dumb and annoying.

Suppose I'm trying to do something like snmpwalk on a switch:

$ snmpwalk -v 2c -c public myswitch.mydomain
-bash: /usr/bin/snmpwalk: No such file or directory

Of course, I need snmp. Lets install that:

~$ sudo apt-get install snmp
Reading package lists... Done
Building dependency tree
Reading state information... Done
The following extra packages will be installed:
libperl5.18 libsensors4 libsnmp-base libsnmp30
Suggested packages:
lm-sensors snmp-mibs-downloader
The following NEW packages will be installed:
libperl5.18 libsensors4 libsnmp-base libsnmp30 snmp
0 upgraded, 5 newly installed, 0 to remove and 36 not upgraded.
Need to get 1,168 kB of archives.
After this operation, 4,674 kB of additional disk space will be used.

and try it again:

$ snmpwalk -v 2c -c public myswitch.mydomain
iso.3.6.1.2.1.1.1.0 = STRING: "Cisco NX-OS(tm) n5000, Software (n5000-uk9), Version 6.0(2)N2(3), RELEASE SOFTWARE Copyright (c) 2002-2012 by Cisco Systems, Inc. Device Manager Version 6.2(1), Compiled 12/17/2013 2:00:00"
iso.3.6.1.2.1.1.2.0 = OID: iso.3.6.1.4.1.9.12.3.1.3.1084
iso.3.6.1.2.1.1.3.0 = Timeticks: (575495258) 66 days, 14:35:52.58
iso.3.6.1.2.1.1.4.0 = STRING: "me@here"
iso.3.6.1.2.1.1.5.0 = STRING: "myswitch"
iso.3.6.1.2.1.1.6.0 = STRING: "snmplocation"
iso.3.6.1.2.1.1.7.0 = INTEGER: 70
iso.3.6.1.2.1.1.8.0 = Timeticks: (4294966977) 497 days, 2:27:49.77
iso.3.6.1.2.1.1.9.1.2.1 = OID: iso.3.6.1.6.3.1
iso.3.6.1.2.1.1.9.1.2.2 = OID: iso.3.6.1.6.3.16.2.2.1
iso.3.6.1.2.1.1.9.1.2.3 = OID: iso.3.6.1.6.3.10.3.1.1
...snip...

Well, we're close, but all we have are a bunch of OIDs. I'd really like names. If you've read my introduction to SNMP, you know that it's not loading the MIBs. Weird. On RHEL/CentOS, that's kind of automatic. Maybe there's another package?

Well, that snmp-mibs-downloader that was listed as a suggested package above sounds pretty promising. Lets install that.

$ sudo apt-get install snmp-mibs-downloader
...snip lots of installing MIBS...

So basically, 300+ MIBs were just installed into /var/lib/mibs/ - this is awesome. Lets run that command again:

$ snmpwalk -v 2c -c public myswitch.mydomain
iso.3.6.1.2.1.1.1.0 = STRING: "Cisco NX-OS(tm) n5000, Software (n5000-uk9), Version 6.0(2)N2(3), RELEASE SOFTWARE Copyright (c) 2002-2012 by Cisco Systems, Inc. Device Manager Version 6.2(1), Compiled 12/17/2013 2:00:00"
iso.3.6.1.2.1.1.2.0 = OID: iso.3.6.1.4.1.9.12.3.1.3.1084
iso.3.6.1.2.1.1.3.0 = Timeticks: (575577418) 66 days, 14:49:34.18
iso.3.6.1.2.1.1.4.0 = STRING: "me@here"
iso.3.6.1.2.1.1.5.0 = STRING: "myswitch"
iso.3.6.1.2.1.1.6.0 = STRING: "snmplocation"
iso.3.6.1.2.1.1.7.0 = INTEGER: 70
iso.3.6.1.2.1.1.8.0 = Timeticks: (4294966977) 497 days, 2:27:49.77
iso.3.6.1.2.1.1.9.1.2.1 = OID: iso.3.6.1.6.3.1
iso.3.6.1.2.1.1.9.1.2.2 = OID: iso.3.6.1.6.3.16.2.2.1
iso.3.6.1.2.1.1.9.1.2.3 = OID: iso.3.6.1.6.3.10.3.1.1
...snip...

That's strange. As it turns out, though, Ubuntu has yet another trick up its sleeve to screw with you. Check out /etc/snmp/snmp.conf:

msimmons@nagios:/var/log$ cat /etc/snmp/snmp.conf
#
# As the snmp packages come without MIB files due to license reasons, loading
# of MIBs is disabled by default. If you added the MIBs you can reenable
# loaging them by commenting out the following line.
mibs :

This file's entire purpose in life is to stop you from having MIBs out of the box.

Obviously, you can comment out that line and then things work:

$ snmpwalk -v 2c -c public myswitch.mydomain
SNMPv2-MIB::sysDescr.0 = STRING: Cisco NX-OS(tm) n5000, Software (n5000-uk9), Version 6.0(2)N2(3), RELEASE SOFTWARE Copyright (c) 2002-2012 by Cisco Systems, Inc. Device Manager Version 6.2(1), Compiled 12/17/2013 2:00:00
SNMPv2-MIB::sysObjectID.0 = OID: SNMPv2-SMI::enterprises.9.12.3.1.3.1084
DISMAN-EVENT-MIB::sysUpTimeInstance = Timeticks: (575602144) 66 days, 14:53:41.44
SNMPv2-MIB::sysContact.0 = STRING: me@here
SNMPv2-MIB::sysName.0 = STRING: myswitch
SNMPv2-MIB::sysLocation.0 = STRING: snmplocation
SNMPv2-MIB::sysServices.0 = INTEGER: 70
SNMPv2-MIB::sysORLastChange.0 = Timeticks: (4294966977) 497 days, 2:27:49.77
SNMPv2-MIB::sysORID.1 = OID: SNMPv2-MIB::snmpMIB
SNMPv2-MIB::sysORID.2 = OID: SNMP-VIEW-BASED-ACM-MIB::vacmBasicGroup
SNMPv2-MIB::sysORID.3 = OID: SNMP-FRAMEWORK-MIB::snmpFrameworkMIBCompliance
SNMPv2-MIB::sysORID.4 = OID: SNMP-MPD-MIB::snmpMPDCompliance
SNMPv2-MIB::sysORID.5 = OID: SNMP-USER-BASED-SM-MIB::usmMIBCompliance
SNMPv2-MIB::sysORDescr.1 = STRING: The MIB module for SNMPv2 entities
SNMPv2-MIB::sysORDescr.2 = STRING: View-based Access Control Model for SNMP.
SNMPv2-MIB::sysORDescr.3 = STRING: The SNMP Management Architecture MIB.
SNMPv2-MIB::sysORDescr.4 = STRING: The MIB for Message Processing and Dispatching.
SNMPv2-MIB::sysORDescr.5 = STRING: The management information definitions for the SNMP User-based Security Model.
SNMPv2-MIB::sysORUpTime.1 = Timeticks: (4294966977) 497 days, 2:27:49.77
SNMPv2-MIB::sysORUpTime.2 = Timeticks: (4294966977) 497 days, 2:27:49.77
...snip...

But...if you're not actually running an snmp server, and you just want to use snmp for querying...if you get rid of that file entirely, it ALSO fixes the problem.

Anyway, just another annoying thing I've found that I thought I'd share.

by Matt Simmons at August 21, 2014 03:23 PM

Everything Sysadmin

Maybe the problem was...

they wouldn't let me wear ear plugs.

August 21, 2014 11:58 AM

Steve Kemp's Blog

Updating Debian Administration

Recently I've been getting annoyed with the Debian Administration website; too often it would be slower than it should be considering the resources behind it.

As a brief recap I have six nodes:

  • 1 x MySQL Database - The only MySQL database I personally manage these days.
  • 4 x Web Nodes.
  • 1 x Misc server.

The misc server is designed to display events. There is a node.js listener which receives UDP messages and stores them in a rotating buffer. The messages might contain things like "User bob logged in", "Slaughter ran", etc. It's a neat hack which gives a good feeling of what is going on cluster-wide.

I need to rationalize that code - but there's a very simple predecessor posted on github for the curious.

Anyway enough diversions, the database is tuned, and "small". The misc server is almost entirely irrelevent, non-public, and not explicitly advertised.

So what do the web nodes run? Well they run a lot. Potentially.

Each web node has four services configured:

  • Apache 2.x - All nodes.
  • uCarp - All nodes.
  • Pound - Master node.
  • Varnish - Master node.

Apache runs the main site, listening on *:8080.

One of the nodes will be special and will claim a virtual IP provided via ucarp. The virtual IP is actually the end-point visitors hit, meaning we have:

Master HostOther hosts

Running:

  • Apache.
  • Pound.
  • Varnish

Running:

  • Apache.

Pound is configured to listen on the virtual IP and perform SSL termination. That means that incoming requests get proxied from "vip:443 -> vip:80". Varnish listens on "vip:80" and proxies to the back-end apache instances.

The end result should be high availability. In the typical case all four servers are alive, and all is well.

If one server dies, and it is not the master, then it will simply be dropped as a valid back-end. If a single server dies and it is the master then a new one will appear, thanks to the magic of ucarp, and the remaining three will be used as expected.

I'm sure there is a pathological case when all four hosts die, and at that point the site will be down, but that's something that should be atypical.

Yes, I am prone to over-engineering. The site doesn't have any availability requirements that justify this setup, but it is good to experiment and learn things.

So, with this setup in mind, with incoming requests (on average) being divided at random onto one of four hosts, why is the damn thing so slow?

We'll come back to that in the next post.

(Good news though; I fixed it ;)

August 21, 2014 08:50 AM

Chris Siebenmann

How data flows around on the client during an Amanda backup

An Amanda backup session involves a lot of processes running on the Amanda client. If you're having a problem with slow backups it can be somewhere between useful and important to understand how everything is connected to everything else and where things go.

A disclaimer: my current information is probably incomplete and it only covers one case. Hopefully it will at least give people an idea of where to start looking and how data is likely to flow around their Amanda setup.

Let's start with an ASCII art process tree of a tar-based backup with indexing (taken from an OmniOS machine with Amanda 3.3.5):

amandad
   sendbackup (1)
      sendbackup (2)
         sh -c '....'
            tar -tf -
            sed -e s/^\.//
      tar --create --file - /some/file/system ....

(In real life the two sendbackup processes are not numbered in pstree or whatever; I'm doing it here to be clear which one I'm talking about.)

Call the 'sh -c' and everything under it the 'indexing processes' and the 'tar --create' the 'backup process'. The backup process is actually making the backup; the indexing processes are reading a copy of the backup stream in order to generate the file index that amrecover will use.

Working outwards from the backup process:

  • the backup process is reading from the disk and writing to its standard output. Its standard output goes to sendbackup (2).

  • sendbackup (2) reads its standard input, which is from the backup process, and basically acts like tee; it feeds one copy of the data to amandad and another into the indexing processes.

  • the indexing processes read standard input from sendbackup (2) and wind up writing their overall standard output to amandad (the tar and sed are in a shell pipeline).

  • sendbackup (1) sits there reading the standard error of all processes under it, which I believes it forwards to amandad if it sees any output. This process will normally be not doing anything.

  • amandad reads all of these incoming streams of data (the actual backup data, the index data, and the standard error output) and forwards them over the network to your Amanda server. If you use a system call tracer on this amandad, you'll also see it reading from and writing to a pipe pair. I believe this pipe pair is being used for signaling purposes, similar to waiting for both file activity and other notifications.

    (Specifically this is being done by GLib's g_wakeup_signal() and g_wakeup_acknowledge() functions, based on tracing library call activity. I believe they're called indirectly by other GLib functions that Amanda uses.)

Under normal circumstances, everything except sendbackup (1) will be churning away making various system calls, primarily to read and write from various file descriptors. The overall backup performance will be bounded by the slowest component in the whole pipeline of data shuffling (although the indexing processes shouldn't normally be any sort of bottleneck).

Depending on the exact Amanda auth setting you're using, streams from several simultaneous backups may flow from your amandad process to the server either as multiple TCP streams or as one multiplexed TCP stream. See amanda-auth(7) for the full details. In all cases I believe that all of this communication runs through a single amandad process, making it a potential bottleneck.

(At this point I don't know whether amandad is an actual bottleneck for us or something else weird is going on.)

(I knew some of the Amanda internal flow at the time that I wrote ReasoningBackwards, but I don't think I wrote it down anywhere apart from implicitly in that entry and it was in less detail than I do now.)

by cks at August 21, 2014 04:43 AM

August 20, 2014

Warren Guy

Global DNS Tester Update

Global DNS Tester Update

In the few days since launching my Global DNS Tester, I've made a few significant improvements. It's no longer limited to looking up A records alone; you can now compare A, AAAA, PTR, CNAME, NS, MX, and SOA records returned for any given host/IP from up to 100 public nameservers simultaneously.

I've also made a bunch of iterative improvements to the interface, and some minor performance improvements.


Read full post

August 20, 2014 06:59 AM

Chris Siebenmann

Explicit error checking and the broad exception catching problem

As I was writing yesterday's entry on a subtle over-broad try in Python, it occurred to me that one advantage of a language with explicit error checking, such as Go, is that a broad exception catching problem mostly can't happen, especially accidentally. Because you check errors explicitly after every operation, it's very hard to aggregate error checks together in the way that a Python try block can fall into.

As an example, here's more or less idiomatic Go code for the same basic operation:

for _, u := range userlist {
   fi, err := os.Stat(u.hdir)
   if err != nil || !(fi.IsDir() && fi.Mode().Perm() == 0) {
      fmt.Println(u.name)
   }
}

(Note that I haven't actually tried to run this code so it may have a Go error. It does compile, which in a statically typed language is at least a decent sign.)

This does the stat() of the home directory and then prints the user name if either there was an error or the homedir is not a mode 000 directory, corresponding to what happened in the two branches of the Python try block. When we check for an error, we're explicitly checking the result of the os.Stat() call and it alone.

Wait, I just pulled a fast one. Unlike the Python version, this code's printing of the username is not checking for errors. Sure, the fmt.Println() is not accidentally being caught up in the error check intended for the os.Stat(), but we've exchanged this for not checking the error at all, anywhere.

(And this is sufficiently idiomatic Go that the usual tools like go vet and golint won't complain about it at all. People ignore the possibility of errors from fmt.Print* functions all the time; presumably complaining about them would create too much noise for a useful checker.)

This silent ignoring of errors is not intrinsic to explicit error checking in general. What enables it here is that Go, like C, allows you to quietly ignore all return values from a function if you want instead of forcing you to explicitly assign them to dummy variables. The real return values of fmt.Println() are:

n, err := fmt.Println(u.name)

But in my original Go code there is nothing poking us in the nose about the existence of the err return value. Unless we think about it and remember that fmt.Println() can fail, it's easy to overlook that we're completely ignoring an error here.

(We can't do the same with os.Stat() because the purpose of calling it is one of the return values, which means that we have to at least explicitly ignore the err return instead of just not remembering that it's there.)

(This is related to how exceptions force you to deal with errors, of course.)

PS: I think that Go made the right pragmatic call when it allowed totally ignoring return values here. It's not completely perfect but it's better than the real alternatives, especially since there are plenty of situations where there's nothing you can do about an error anyways.

Sidebar: how you can aggregate errors in an explicit check language

Languages with explicit error checks still allow you to aggregate errors together if you want to, but now you have to do it explicitly. The most common pattern is to have a function that returns an error indicator and performs multiple different operations, each of which can fail. Eg:

func oneuser(u user) error {
   var err error
   fi, err := os.Stat(u.hdir)
   if err != nil {
      return err
   }
   if !(fi.IsDir() && fi.Mode().Perm() == 0) {
      _, err = fmt.Println(u.name)
   }
   return err
}

If we then write code that assumes that a non-nil result from oneuser() means that the os.Stat() has failed, we've done exactly the same error aggregation that we did in Python (and with more or less the same potential consequences).

by cks at August 20, 2014 05:57 AM

August 19, 2014

Racker Hacker

Audit RHEL/CentOS 6 security benchmarks with ansible

Ansible logoSecuring critical systems isn’t easy and that’s why security benchmarks exist. Many groups and communities distribute recommendations for securing servers, including NIST, the US Department of Defense (DoD), and the Center for Internet Security (CIS).

Although NIST and DoD are catching up quickly with newer OS releases, I’ve found that the CIS benchmarks are updated very regularly. CIS distributes auditing tools (with paid memberships) that require Java and they’re cumbersome to use, especially on servers where Java isn’t normally installed.

A better way to audit security benchmarks

I set out to create an Ansible playbook that would allow users to audit and (carefully!) remediate servers. The result is on GitHub. Before we go any further, I’d just like to state that I’m not affiliated with CIS in any way and this repository hasn’t been endorsed by CIS. Use it at your own risk.

Getting the playbook onto a machine is easy:

git clone https://github.com/major/cis-rhel-ansible.git

PLEASE review the README and NOTES files in the GitHub repository prior to running the playbook.

What-Did-You-Do-Chris-Farley-Gif

Seriously. I mean it. This playbook could knock production environments offline.

The tasks are split into sections (just like the CIS benchmarks themselves) and each section is split into Level 1 and 2 requirements.

Benchmark levels

Level 1 requirements provide good security improvements without a tremendous amount of intrusion into production workloads. With that said, they can still cause issues.

Level 2 requirements provide stronger security improvements but they can adversely affect production server environments. This is where you find things like SELinux, AIDE (including disabling prelinking), and some kernel tweaks for IPv6.

How to use it

I strongly recommend some dry runs with Ansible’s check mode before trying to modify a production system. Also, you can run the playbook against a freshly-installed system and then deploy your applications on top of it. Find out what breaks and disable certain benchmarks that get in the way.

The entire playbook takes less than a minute to run locally on a Rackspace Performance Cloud Server. Your results may vary over remote ssh connections, but I was seeing the playbooks complete over ssh within three to four minutes.

You can also review the variables file to find all the knobs you need to get more aggressive in your audits. If you spot something potentially destructive that needs a variable added, let me know (or submit a pull request).

It’s open source

The entire repository is licensed under Apache License 2.0, so please feel free to submit issues, pull requests, or patches.

The post Audit RHEL/CentOS 6 security benchmarks with ansible appeared first on major.io.

by Major Hayden at August 19, 2014 12:00 PM

Chris Siebenmann

An example of a subtle over-broad try in Python

Today I wrote some code to winnow a list of users to 'real' users with live home directories that looks roughly like the following:

for uname, hdir in userlist:
   try:
      st = os.stat(hdir)
      if not stat.S_ISDIR(st.st_mode) or \
         stat.S_IMODE(st.st_mode) == 0:
            continue
      # looks good:
      print uname
   except EnvironmentError:
      # accept missing homedir; might be a
      # temporarily missing NFS mount, we
      # can't tell.
      print uname

This code has a relatively subtle flaw because I've accidentally written an over-broad exception catcher here.

As suggested by the comment, when I wrote this code I intended the try block to catch the case where the os.stat failed. The flaw here is that print itself does IO (of course) and so can raise an IO exception. Since I have the print inside my try block, a print-raised IO exception will get caught by it too. You might think that this is harmless because the except will re-do the print and thus presumably immediately have the exception raised again. This contains two assumptions: that the exception will be raised again and that if it isn't, the output is in a good state (as opposed to, say, having written only partial output before an error happened). Neither are entirely sure things and anyways, we shouldn't be relying on this sort of thing when it's really easy to fix. Since both branches of the exception end up at the same print, all we have to do is move it outside the try: block entirely (the except case then becomes just 'pass').

(My view is that print failing is unusual enough that I'm willing to have the program die with a stack backtrace, partly because this is an internal tool. If that's not okay you'd need to put the print in its own try block and then do something if it failed, or have an overall try block around the entire operation to catch otherwise unexpected EnvironmentError exceptions.)

The root cause here is that I wasn't thinking of print as something that does IO that can throw exceptions. Basic printing is sufficiently magical that it feels different and more ordinary, so it's easy to forget that this is a possibility. It's especially easy to overlook because it's extremely uncommon for print to fail in most situations (although there are exceptions, especially in Python 3). You can also attribute this to a failure to minimize what's done inside try blocks to only things that absolutely have to be there, as opposed to things that are just kind of convenient for the flow of code.

As a side note, one of the things that led to this particular case is that I changed my mind about what should happen when the os.stat() failed because I realized that failure might have legitimate causes instead of being a sign of significant problems with an account that should cause it to be skipped. When I changed my mind I just did a quick change to what the except block did instead of totally revising the overall code, partly because this is a small quick program instead of a big system.

by cks at August 19, 2014 02:35 AM

RISKS Digest

August 18, 2014

[bc-log]

Removing packages and configurations with apt-get

Yesterday while re-purposing a server I was removing packages with apt-get and stumbled upon an interesting problem. After I removed the package and all of it's configurations, the subsequent installation did not re-deploy the configuration files.

After a bit of digging I found out that there are two methods for removing packages with apt-get. One of those method should be used if you want to remove binaries, and the other should be used if you want to remove both binaries and configuration files.

What I did

Since the method I originally used caused at least 10 minutes of head scratching; I thought it would be useful to share what I did and how to resolve it.

On my system the package I wanted to remove was supervisor which is pretty awesome btw. To remove the package I simply removed it with apt-get remove just like I've done many times before.

# apt-get remove supervisor
Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following packages will be REMOVED:
  supervisor
0 upgraded, 0 newly installed, 1 to remove and 0 not upgraded.
After this operation, 1,521 kB disk space will be freed.
Do you want to continue [Y/n]? y
(Reading database ... 14158 files and directories currently installed.)
Removing supervisor ...
Stopping supervisor: supervisord.
Processing triggers for ureadahead ...

No issues so far, the package was removed according to apt without any issues. However, after looking around a bit I noticed that the /etc/supervisor directory still existed. As well as the supervisord.conf file.

# ls -la /etc/supervisor
total 12
drwxr-xr-x  2 root root 4096 Aug 17 19:44 .
drwxr-xr-x 68 root root 4096 Aug 17 19:43 ..
-rw-r--r--  1 root root 1178 Jul 30  2013 supervisord.conf

Considering I was planning on re-installing supervisor and I didn't want to cause any weird configuration issues as I moved from one server role to another I did what any other reasonable Sysadmin would do. I removed the directory...

# rm -Rf /etc/supervisor

I knew the supervisor package was removed, and I assumed that the package didn't remove the config files to avoid losing custom configurations. In my case I wanted to start over from scratch, so deleting the directory sounded like a reasonable thing.

# apt-get install supervisor
Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following NEW packages will be installed:
  supervisor
0 upgraded, 1 newly installed, 0 to remove and 0 not upgraded.
Need to get 0 B/314 kB of archives.
After this operation, 1,521 kB of additional disk space will be used.
Selecting previously unselected package supervisor.
(Reading database ... 13838 files and directories currently installed.)
Unpacking supervisor (from .../supervisor_3.0b2-1_all.deb) ...
Processing triggers for ureadahead ...
Setting up supervisor (3.0b2-1) ...
Starting supervisor: Error: could not find config file /etc/supervisor/supervisord.conf
For help, use /usr/bin/supervisord -h
invoke-rc.d: initscript supervisor, action "start" failed.
dpkg: error processing supervisor (--configure):
 subprocess installed post-installation script returned error exit status 2
Errors were encountered while processing:
 supervisor
E: Sub-process /usr/bin/dpkg returned an error code (1)

However, it seems supervisor could not start after re-installing.

# ls -la /etc/supervisor
ls: cannot access /etc/supervisor: No such file or directory

There is good reason why supervisor wouldn't restart; because the /etc/supervisor/supervisord.conf file was missing. Shouldn't the package installation deploy the supervisord.conf file? Well, technically no. Not with the way I removed the supervisor package.

Why it didn't work

How remove works

If we look at apt-get's man page a little closer we can see why the configuration files are still there.

remove
  remove is identical to install except that packages are removed
  instead of installed. Note that removing a package leaves its
  configuration files on the system.

As the manpage clearly says, remove will remove the package but leaves configuration files in place. This explains why the /etc/supervisor directory was lingering after removing the package; but it doesn't explain why a subsequent installation doesn't re-deploy the configuration files.

Package States

If we use dpkg to look at the supervisor package, we will start to see the issue.

# dpkg --list supervisor
Desired=Unknown/Install/Remove/Purge/Hold
| Status=Not/Inst/Conf-files/Unpacked/halF-conf/Half-inst/trig-aWait/Trig-pend
|/ Err?=(none)/Reinst-required (Status,Err: uppercase=bad)
||/ Name                             Version               Architecture          Description
+++-================================-=====================-=====================-========================================
rc  supervisor                       3.0b2-1               all                   A system for controlling process state

With the dpkg package manager a package can have more states than just being installed or not-installed. In fact there are several package states with dpkg.

  • not-installed - The package is not installed on this system
  • config-files - Only the configuration files are deployed to this system
  • half-installed - The installation of the package has been started, but not completed
  • unpacked - The package is unpacked, but not configured
  • half-configured - The package is unpacked and configuration has started but not completed
  • triggers-awaited - The package awaits trigger processing by another package
  • triggers-pending - The package has been triggered
  • installed - The packaged is unpacked and configured OK

If you look at the first column of the dpkg --list it shows rc. The r in this column means the package is remove, which as we saw above means the configuration files are left on the system. The c in this column shows that the package is in the state of config-files. Meaning, only the configuration files are deployed on this system.

When running apt-get install the apt package manager will lookup the current state of the package, when it sees that the package is already in the config-files state it simply skips the configuration file portion of the package installation. Since I manually removed the configuration files outside of the apt or dpkg process the configuration files are gone and will not be deployed with a simple apt-get install.

How to resolve it and remove configurations properly

Purging the package from my system

At this point, I found myself with a broken installation of supervisor. Luckily, we can fix the issue by using the purge option of apt-get.

# apt-get purge supervisor
Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following packages will be REMOVED:
  supervisor*
0 upgraded, 0 newly installed, 1 to remove and 0 not upgraded.
1 not fully installed or removed.
After this operation, 1,521 kB disk space will be freed.
Do you want to continue [Y/n]? y
(Reading database ... 14158 files and directories currently installed.)
Removing supervisor ...
Stopping supervisor: supervisord.
Purging configuration files for supervisor ...
dpkg: warning: while removing supervisor, directory '/var/log/supervisor' not empty so not removed
Processing triggers for ureadahead ...

Purge vs Remove

The purge option of apt-get is similar to the remove function however with one difference. The purge option will remove both the package and configurations. After running apt-get purge we can see that the package was fully removed by running dpkg --list again.

# dpkg --list supervisor
dpkg-query: no packages found matching supervisor

Re-installation without error

Now that the package has been fully purged, and the state of it is now not-installed; we can re-install without errors.

# apt-get install supervisor
Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following NEW packages will be installed:
  supervisor
0 upgraded, 1 newly installed, 0 to remove and 0 not upgraded.
Need to get 0 B/314 kB of archives.
After this operation, 1,521 kB of additional disk space will be used.
Selecting previously unselected package supervisor.
(Reading database ... 13833 files and directories currently installed.)
Unpacking supervisor (from .../supervisor_3.0b2-1_all.deb) ...
Processing triggers for ureadahead ...
Setting up supervisor (3.0b2-1) ...
Starting supervisor: supervisord.
Processing triggers for ureadahead ...

As you can see from the output above, the supervisor package has been installed and started. If we check the /etc/supervisor directory again we can also see the necessary configuration files.

# ls -la /etc/supervisor/
total 16
drwxr-xr-x  3 root root 4096 Aug 17 19:46 .
drwxr-xr-x 68 root root 4096 Aug 17 19:46 ..
drwxr-xr-x  2 root root 4096 Jul 30  2013 conf.d
-rw-r--r--  1 root root 1178 Jul 30  2013 supervisord.conf

You should probably just use purge in most cases

After running into this issue I realized, most of the times I ran apt-get remove I really wanted the functionality of apt-get purge. While it is nice to keep configurations handy in case we need them after re-installation, using remove all the time also leaves random config files to clutter your system. Free to cause configuration issues when packages are removed then re-installed.

In the future I will most likely default to apt-get purge.


Originally Posted on BenCane.com: Go To Article

by Benjamin Cane at August 18, 2014 07:45 AM

Chris Siebenmann

The potential issue with Go's strings

As I mentioned back in Things I like about Go, one of the Go things that I really like is its strings (and slices in general). From the perspective of a Python programmer, what makes them great is that creating strings is cheap because they often don't require a copy. In Python, any time you touch a string you're copying some or all of it and this can easily have a real performance impact. Writing performant Python code requires considering this carefully. In Go, pretty much any string operation that just takes a subset of the string (eg trimming whitespace from the front and the end) is copy-free, so you can throw around string operations much more freely. This can make a straightforward algorithm both the right solution to your problem and pretty efficient.

(Not all interesting string operations are copy-free, of course. For example, converting a string to all upper case requires a copy, although Go's implementation is clever enough to avoid this if the string doesn't change, eg because it's already all in upper case.)

But this goodness necessarily comes with a potential badness, which is that those free substrings keep the entire original string alive in memory. What makes Go strings (and slices) so cheap is that they are just references to some chunk of underlying storage (the real data for the string or the underlying array for a slice); making a new string is just creating a new reference. But Go doesn't (currently) do partial garbage collection of string data or arrays, so if even one tiny bit of it is referred to somewhere the entire object must be retained. In other words, a string that's a single character is (currently) enough to keep a big string from being garbage collected.

This is not an issue that many people will run into, of course. To hit it you need to either be dealing with very big original strings or care a lot about memory usage (or both) and on top of that you have to create persistent small substrings of the non-persistent original strings (well, what you want to be non-persistent). Many usage patterns won't hit this; your original strings are not large, your subsets cover most of the original string anyways (for example if you break it up into words), or even the substrings don't live very long. In short, if you're an ordinary Go programmer you can ignore this. The people who care are handling big strings and keeping small chunks of them for a long time.

(This is the kind of thing that I notice because I once spent a lot of effort to make a Python program use as little memory as possible even thought it was parsing and storing chunks out of a big configuration file. This made me extra-conscious about things like string lifetimes, single-copy interned strings, and so on. Then I wrote a parser in Go, which made me consider all of these issues all over again and caused me to realize that the big string representing my entire input file was going to be kept in memory due to the bits of it that my parser was clipping out and keeping.)

By the way, I think that this is the right tradeoff for Go to make. Most people using strings will never run into this, while it's very useful that substrings are cheap. And this sort of cheap substrings also makes less work for the garbage collector; instead of a churn of variable length strings when code is using a lot of substrings (as happens in Python), you just have a churn of fixed-size string references.

Of course there's the obvious fix if your code starts running into this: create a function that 'minimizes' a string by turning it into a []byte and then back. This creates a minimized string at the cost of an extra copy over the theoretical ideal implementation and can be trivially done in Go today.

Sidebar: How strings.ToUpper() et al avoid unnecessary copies

All of the active transformation functions like ToUpper() and ToTitle() are implemented using strings.Map() and functions from the unicode package. Map() is smart enough to not start making a new string until the mapping function returns a different rune than the existing one. As a result, any similar direct use of Map() that your code has will get this behavior for free.

by cks at August 18, 2014 04:46 AM

August 17, 2014

Geek and Artist - Tech

The learning gap in deploying Javascript apps

I’ve recently been building a website for my wife using relatively modern tools – lots of client-side Javascript and relatively minimal CSS and HTML. A recent-ish email alerted me to the existence of a Berlin-based Famous framework meetup group, which initially made no sense to me, but after I checked out the actual framework website my interest was piqued. I’ve got next to no website building experience (at least from the front-end point of view), and what I would only describe as barely competent Javascript skills. Nevertheless it appealed more to me to learn more about this world than simply building a generic WordPress-based site and customising some stylesheets.

There are some great tools out there these days for Javascript learners, for example Codecademy. I used this myself to brush up on the weird scoping rules, function hoisting, manipulating the DOM etc. That’s enough to generally get you started with Javascript as a language and also in the context of execution in the browser which comes with additional constraints and possibilities. On top of that you have tutorials on using jQuery which are usually quite understandable and make a lot of sense if you have already learned about the DOM and manipulating content on a page. So far so good.

The Famous framework introduces a new paradigm for creating responsive, dynamic content on a webpage for both desktop and mobile devices. Fortunately on their website they also provide a bunch of tutorials which give you a pretty good overview of functionality and how to use the framework. Their starter pack containing further examples for most of the library functions also helps considerably. I took a few times to go through all of it and still find some aspects confusing but ultimately I was able to build a website that actually worked.

Great – now I need to run this stuff on a real webserver. If you have had even tangential contact with real front-end developers you have probably heard of such terms as “minification“, “Javascript obfuscation” and perhaps “RequireJS” or “CommonJS modules” and so on. I was already somewhat aware of CommonJS modules, having encountered them in front-end code for my company’s website and could see that they provide a nice solution to the Javascript scoping and code reuse problems. Fortunately, using the scaffolding provided in the Famous starter kit you get code that already has the CommonJS module pattern built-in, so building the website code followed this pattern from the start. If this hadn’t been the case, I’m not sure how I would have found some good examples for getting started with it.

The website built, I was only vaguely aware that RequireJS had to be part of the solution but left this part of the puzzle aside while I pondered how to minify the code. I could have just downloaded the famous.min.js file from their website and copied all the files to my webserver but I felt like this wasn’t how it is meant to be (but with no real way knowing the right way to do it). This lack of knowledge resulted in a frustrated mailing list post but ultimately no better solution. There was a suggestion to use RequireJS, and a lot more Googling on my part but I still didn’t make much progress. Eventually I decided I’d create a “vendor” directory, cloned the Famous repo into it and… got stuck again. I just didn’t know how to join the dots between the original Javascript code and a tidy single-file representation of the library dependencies and my own app code.

After a conversation with a coworker I was armed with links to such tools as Bower (which I played with a bit, before realising I still didn’t know what I was doing), Grunt, Gulp, Yeoman, and also found an interesting Hackernews thread that seemed to contain useful information but was still a little out of my reach. All these tools, but still I had no good idea what I needed to accomplish and how to do it. I decided in the end to just use the Yeoman Famous generator module and generate a Famous site scaffolding to see what such a thing looked like. This contained just a tiny example of Famous code but more importantly had all of the minification tooling, Javascript linting and build preparation and deployment tasks already baked in without having to build them from scratch. I copied the relevant parts over to my existing repository, fixed all the linter errors and ended up with the minified, deployable Javascript files I’d been hoping for.

I’m happy with the end result, but saddened by the apparent gap in learning possibilities for this part of the Javascript world. There are plenty of opportunities to learn basic Javascript, figure out how to make a static website slightly more dynamic and perhaps use some fairly standard convenience libraries like jQuery or Underscore but not much more than that. When it comes to building reusable modules with RequireJS or minifying, linting and building your code for deployment this feels like an area that suffered from problems felt acutely by long-time Javascript developers, that now fortunately have a variety of good solutions – and there has been a lot of rapid development in this area in the last few years. Sadly, getting learners up to speed with these problems and their solutions doesn’t seem to have kept up with the technologies themselves.

My recommendation: If you’re in a similar position to me, check out Yeoman and the generators available for it. Use that and build up your knowledge of how the individual components and libraries are used together by inspecting what scaffolding code it provides for you. I can only hope that we start to see more tutorials on Codecademy and other similar sites on Javascript deployment subjects and the problems that the tools mentioned above solve for modern websites.

by oliver at August 17, 2014 01:07 PM

Chris Siebenmann

The challenges of diagnosing slow backups

This is not the techblog entry I thought I was going to write. That entry would have been a quietly triumphal one about finding and squelching an interesting and obscure issue that was causing our backups to be unnaturally slow. Unfortunately, while the particular issue is squelched our backups of at least our new fileservers are still (probably) unnaturally slow. Certainly they seem to be slower than we want. So this entry is about the challenge of trying to figure out why your backups are slow (and what you can do about it).

The first problem is that unless the cause is quite simple, you're going to wind up needing to make detailed observations of your systems while the backups are running. In fact you'll probably have to do this repeatedly. By itself this is not a problem as such. What makes it into one is that most backups are run out of hours, often well out of hours. If you need to watch the backups and the backups start happening at 11pm, you're going to be staying up. This has various knock-on consequences, including that human beings are generally not at their sharpest at 11pm.

(General purpose low level metrics collection can give you some immediate data but there are plenty of interesting backup slowdowns that cannot be caught with them, including our recent one. And backups run during the day (whether test runs or real ones) are generally going to behave somewhat differently from nighttime backups, at least if your overall load and activity are higher in the day.)

Beyond that issue, a running backup system is generally a quite complex beast with many moving parts. There are probably multiple individual backups in progress in multiple hosts, data streaming back to backup servers, and any number of processes in communication with each other about all of this. As we've seen, the actual effects of a problem in one piece can manifest far away from that piece. In addition pieces may be interfering with each other; for example, perhaps running enough backups at once on a single machine causes them to contend for a resource (even an inobvious one, since it's pretty easy to spot saturated disks, networks, CPU, et al).

Complex systems create complex failure modes, which means that there are a lot of potential inobvious things that might be going on. That's a lot of things to winnow through for potential issues that pass the smell test, don't contradict any evidence of system behavior that you already have, and ideally that can be tested in some simple way.

(And the really pernicious problems don't have obvious causes, because if they did they would be easy to at least identify.)

What writing this tells me is that this is not unique to backup systems and that I should probably sit down to diagram out the overall backup system and its resources, then apply Brendan Gregg's USE Method to all of the pieces involved in backups in a systematic way. That would at least give me a good collection of data that I could use to rule things out.

(It's nice to form hypotheses and then test them and if you get lucky you can speed everything up nicely. But there are endless possible hypotheses and thus endless possible tests, so at some point you need to do the effort to create mass winnowings.)

by cks at August 17, 2014 04:50 AM

August 16, 2014

Chris Siebenmann

Caches should be safe by default

I've been looking at disk read caching systems recently. Setting aside my other issues, I've noticed something about basically all of them that makes me twitch as a sysadmin. I will phrase it this way:

Caches should be safe by default.

By 'safe' I mean that if your cache device dies, you should not lose data or lose your filesystem. Everything should be able to continue on, possibly after some manual adjustment. The broad purpose of most caches is to speed up reads; write accelerators are a different thing and should be labeled as such. When your new cache system is doing this for you, it should not be putting you at a greater risk of data loss because of some choice it's quietly made; if writes touch the cache at all, it should default to write-through mode. To do otherwise is just as dangerous as those databases that achieve great speed through quietly not really committing their data to disk.

There is a corollary that follows from this:

Caches should clearly document when and how they aren't safe.

After I've read a cache's documentation, I should not be in either doubt or ignorance about what will or can happen if the cache device dies on me. If I am, the documentation has failed. Especially it had better document the default configuration (or the default examples or both), because the default configuration is what a lot of people will wind up using. As a corollary to the corollary, the cache documentation should probably explain what I get for giving up safety. Faster than normal writes? It's just required by the cache's architecture? Avoiding a write slowdown that the caching layer would otherwise introduce? I'd like to know.

(If documenting this stuff makes the authors of the cache embarrassed, perhaps they should fix things.)

As a side note, I think it's fine to offer a cache that's also a write accelerator. But I think that this should be clearly documented, the risks clearly spelled out, and it should not be the default configuration. Certainly it should not be the silent default.

by cks at August 16, 2014 02:41 AM

August 15, 2014

Everything Sysadmin

Simple bucket-ized stats in awk

Someone recently asked how to take a bunch of numbers from STDIN and then break them down into distribution buckets. This is simple enough that it should be do-able in awk.

Here's a simple script that will generate 100 random numbers. Bucketize them to the nearest multiple of 10, print based on # of items in bucket:

while true ; do echo $[ 1 + $[ RANDOM % 100 ]] ; done | head -100 | awk '{ bucket = int(($1 + 5) / 10) * 10 ; arr[bucket]++} END { for (i in arr) {print i, arr[i] }}' | sort -k2n,2 -k1n,1

Many people don't know that in bash, a single quote can go over multiple lines. This makes it very easy to put a little bit of awk right in the middle of your code, eliminating the need for a second file that contains the awk code itself. Since you can put newlines anywhere, you can make it very readable:

#!/bin/bash

while true ; do
  echo $[ 1 + $[ RANDOM % 100 ]]
done | head -100 | \
  awk '
      {
        bucket = int(($1 + 5) / 10) * 10 ;
        arr[bucket]++
      }
      END {
        for (i in arr) {
          print i, arr[i]
        }
      }
' | sort -k2n,2 -k1n,1

If you want to sort by the buckets, change the sort to sort -k1n,1 -k2n,2

If you want to be a little more fancy, separate out the bucket function into a separate function. What? awk can do functions? Sure it can. You can also import values from the environment using the -v flag.

#!/bin/bash

# Bucketize stdin to nearest multiple of argv[1], or 10 if no args given.
# "nearest" means 0..4.999 -> 0, 5..14.999 -> 10, etc.

# Usage:
# while true ; do echo $[ 1 + $[ RANDOM % 100 ]]; done | head -99 | bucket.sh 8

awk -v multiple="${1:-10}" '

function bucketize(a) {
  # Round to the nearest multiple of "multiple"
  #  (nearest... i.e. may round up or down)
  return int((a + (multiple/2)) / multiple) * multiple;
}

# All lines get bucketized.
{ arr[bucketize($1)]++ }

# When done, output the array.
END {
  for (i in arr) {
    print i, arr[i]
  }
}
' | sort -k2n,2 -k1n,1

I generally use Python for scripting but for something this short, awk makes sense. Sadly using awk has become a lost art.

August 15, 2014 03:15 PM

TechRepublic Network Administrator

ARM's battle for the datacentre: The contenders

How will the first generation of enterprise-ready ARM servers stack up against traditional datacentre boxes?

by Nick Heath at August 15, 2014 02:15 PM

Steve Kemp's Blog

A tale of two products

This is a random post inspired by recent purchases. Some things we buy are practical, others are a little arbitrary.

I tend to avoid buying things for the sake of it, and have explicitly started decluttering our house over the past few years. That said sometimes things just seem sufficiently "cool" that they get bought without too much thought.

This entry is about two things.

A couple of years ago my bathroom was ripped apart and refitted. Gone was the old and nasty room, and in its place was a glorious space. There was only one downside to the new bathroom - you turn on the light and the fan comes on too.

When your wife works funny shifts at the hospital you can find that the (quiet) fan sounds very loud in the middle of the night and wakes you up..

So I figured we could buy a couple of LED lights and scatter them around the place - when it is dark the movement sensors turn on the lights.

These things are amazing. We have one sat on a shelf, one velcroed to the bottom of the sink, and one on the floor, just hidden underneath the toilet.

Due to the shiny-white walls of the room they're all you need in the dark.

By contrast my second purchase was a mistake - The Logitech Harmony 650 Universal Remote Control should be great. It clearly has the features I want - Able to power:

  • Our TV.
  • Our Sky-box.
  • OUr DVD player.

The problem is solely due to the horrific software. You program the device via an application/website which works only under Windows.

I had to resort to installing Windows in a virtual machine to make it run:

# Get the Bus/ID for the USB device
bus=$(lsusb |grep -i Harmony | awk '{print $2}' | tr -d 0)
id=$(lsusb |grep -i Harmony | awk '{print $4}' | tr -d 0:)

# pass to kvm
kvm -localtime ..  -usb -device usb-host,hostbus=$bus,hostaddr=$id ..

That allows the device to be passed through to windows, though you'll later have to jump onto the Qemu console to re-add the device as the software disconnects and reconnects it at random times, and the bus changes. Sigh.

I guess I can pretend it works, and has cut down on the number of remotes sat on our table, but .. The overwhelmingly negative setup and configuration process has really soured me on it.

There is a linux application which will take a configuration file and squirt it onto the device, when attached via a USB cable. This software, which I found during research prior to buying it, is useful but not as much as I'd expected. Why? Well the software lets you upload the config file, but to get a config file you must fully complete the setup on Windows. It is impossible to configure/use this device solely using GNU/Linux.

(Apparently there is MacOS software too, I don't use macs. *shrugs*)

In conclusion - Motion-activated LED lights, more useful than expected, but Harmony causes Discord.

August 15, 2014 12:14 PM

Chris Siebenmann

A consequence of NFS locking and unlocking not necessarily being fast

A while back I wrote Cross-system NFS locking and unlocking is not necessarily fast, about one drawback of using NFS locks to communicate between processes on different machines. This drawback is that it may take some time for process B on machine 2 to find out that process A on machine 1 has unlocked the shared coordination file. It turns out that this goes somewhat further than I realized at the time. Back then I looked at cross-system lock activity, but it turns out that you can see long NFS lock release delays even when the two processes are on the same machine.

If you have process A and process B on the same machine, both contending for access to the same file via file locking, you can easily see significant delays between active process A releasing the lock and waiting process B being notified that it now has the lock. I don't know enough about the NLM protocol to know if the client or server kernels can do anything to make the process go faster, but there are some client/server combinations where this delay does happen.

(If the client's kernel is responsible for periodically retrying pending locking operations until they succeed, it certainly could be smart enough to notice that another process on the machine just released a lock on the file and so now might be a good time for a another retry.)

This lock acquisition delay can have a pernicious spiraling effect on an overall system. Suppose, not entirely hypothetically, that what a bunch of processes on the same machine are locking is a shared log file. Normally a process spends very little time doing logging and most of their time doing other work. When they go to lock the log file to write a message, there's no contention, they get the lock, they basically immediately release the lock, and everyone goes on fine. But then you hit a lock collision, where processes A and B both want to write. A wins, writes its log message, and unlocks immediately. But the NFS unlock delay means that process B is then going to sit there for ten, twenty, or thirty seconds before it can do its quick write and release the lock in turn. Suppose during this time another process, C, also shows up to write a log message. Now C may be waiting too, and it too will have a big delay to acquire the lock (if locks are 'fair', eg FIFO, then it will have to wait both for B to get the lock and for the unlock delay after B is done). Pretty soon you have more and more processes piling up waiting to write to the log and things grind to a much slower pace.

I don't think that there's a really good solution to this for NFS, especially since an increasing number of Unixes are making all file locks be NFS aware (as we've found out the hard way before). It's hard to blame the Unixes that are doing this, and anyways the morally correct solution would be to make NLM unlocks wake up waiting people faster.

PS: this doesn't happen all of the time. Things are apparently somewhat variable based on the exact server and client versions involved and perhaps timing issues and so on. NFS makes life fun.

Sidebar: why the NFS server is involved here

Unless the client kernel wants to quietly transfer ownership of the lock being unlocked to another process instead of actually releasing it, the NFS server is the final authority on who has a NFS lock and it must be consulted about things. For all that any particular client machine knows, there is another process on another machine that very occasionally wakes up, grabs a lock on the log file, and does stuff.

Quietly transferring lock ownership is sleazy because it bypasses any other process trying to get the lock on another machine. One machine with a rotating set of processes could unfairly monopolize the lock if it did that.

by cks at August 15, 2014 05:48 AM

Warren Guy

New DNS troubleshooting utility: Global DNS Tester

Helping a friend diagnose a DNS problem earlier, I stumbled across a huge list of public nameservers (more than 3,000 at present) at public-dns.tk. Inspired, I hacked together a simple script to query a random set of them and display the results.

So, I've just published a simple web based utility (URL at end of this post) for checking the A record of a hostname from a random set of global public nameservers. It allows you to query up to 100 servers at a time, either from all available global nameservers, or filtered by country. You might find it useful for diagnosing DNS propagation delays, nameserver connectivity issues, geotargeted DNS, and more. The nameserver list is updated about once an hour. I may release the source code at a later time, if I get around to cleaning it up a bit.


Read full post

August 15, 2014 01:03 AM

August 14, 2014

Chris Siebenmann

Bind mounts with systemd and non-fstab filesystems

Under normal circumstances the way you deal with Linux bind mounts on a systemd based system is the same as always: you put them in /etc/fstab and systemd makes everything work just like normal. If you can deal with your bind mounts this way, I recommend that you do it and keep your life simple. But sometimes life is not simple.

Suppose, not entirely hypothetically, that you are dealing with base filesystems that aren't represented in /etc/fstab for one reason or another; instead they appear through other mechanisms. For example, perhaps they appear when you import a ZFS pool. You want to use these filesystems as the source of bind mounts.

The first thing that doesn't work is leaving your bind mounts in /etc/fstab. There is no way to tell systemd to not create them until something else happens (eg your zfs-mount.service systemd unit finishes or their source directory appears), so this is basically never going to do the right thing. If you get bind mounts at all they are almost certainly not going to be bound to what you want. At this point you might be tempted to think 'oh, systemd makes /etc/fstab mounts into magic <name>.mount systemd units, I can just put files in /etc/systemd/system to add some extra dependencies to those magic units'. Sadly this doesn't work; the moment you have a real <name>.mount unit file it entirely replaces the information from /etc/fstab and systemd will tell you that your <name>.mount file is invalid because it doesn't specify what to mount.

In short, you need real .mount units for your bind mounts. You also need to force the ordering, and here again we run into something that would be nice but doesn't work. If you run 'systemctl list-units -t mount', you will see that there are units for all of your additional non-fstab mounts. It's tempting to make your bind mount unit depend on an appropriate mount unit for its source filesystem, eg if you have a bind mount from /archive/something you'd have it depend on archive.mount. Unfortunately this doesn't work reliably because systemd doesn't actually know about these synthetic mount units before the mount appears. Instead you can only depend on whatever .service unit actually does the mounting, such as zfs-mount.service.

(In an extreme situation you could create a service unit that just used a script to wait for the mounts to come up. With a Type=oneshot service unit, systemd won't consider the service successful until the script exits.)

The maximally paranoid set of dependencies and guards is something like this:

[Unit]
After=zfs-mount.service
Requires=zfs-mount.service
RequiresMountsFor=/var
ConditionPathIsDirectory=/local/var/local

(This is for a bind mount from /local/var/local to /var/local.)

We can't use a RequiresMountsFor on /local/var, because as far as systemd is concerned it's on the root filesystem and so the dependency would be satisfied almost immediately. I don't think the Condition will cause systemd to wait for /local/var/local to appear, just stop the bind mount from trying to be done if ZFS mounts happened but they didn't managed to mount a /local/var for some reason (eg a broken or missing ZFS pool).

(Since my /var is actually on the root filesystem, the RequiresMountsFor is likely gilding the lily; I don't think there's any situation where this unit can even be considered before the root filesystem is mounted. But if it's a separate filesystem you definitely want this and so it's probably a good habit in general.)

I haven't tested using local-var.mount in just the Requires here but I'd expect it to fail for the same reason that it definitely doesn't work reliably in an After. This is kind of a pity, but there you go and the Condition is probably good enough.

(If you don't want to make a bunch of .mount files, one for each mount, you could make a single .service unit that has all of the necessary dependencies and runs appropriate commands to do the bind mounting (either directly or by running a script). If you do this, don't forget to have ExecStop stuff to also do the unmounts.)

Sidebar: the likely non-masochistic way to do this for ZFS on Linux

If I was less stubborn, I would have set all of my ZFS filesystems to have 'mountpoint=legacy' and then explicitly mentioned and mounted them in /etc/fstab. Assuming that it worked (ie that systemd didn't try to do the mounts before the ZFS pool came up), this would have let me keep the bind mounts in fstab too and avoided this whole mess.

by cks at August 14, 2014 03:20 AM

August 13, 2014

Racker Hacker

Start Jenkins on Fedora 20

Installing Jenkins on Fedora 20 is quite easy thanks to the available Red Hat packages, but I ran into problems when I tried to start Jenkins. Here are the installation steps I followed:

wget -O /etc/yum.repos.d/jenkins.repo http://pkg.jenkins-ci.org/redhat/jenkins.repo
rpm --import http://pkg.jenkins-ci.org/redhat/jenkins-ci.org.key
yum -y install jenkins
systemctl enable jenkins
systemctl start jenkins

Your first error will show up if Java isn’t installed. You can fix that by installing Java:

yum -y install java-1.7.0-openjdk-headless

After installing Java, Jenkins still refused to start. Nothing showed up in the command line or via journalctl -xn, so I jumped into the Jenkins log file (found at /var/log/jenkins/jenkins.log):

Aug 13, 2014 2:21:44 PM org.eclipse.jetty.util.log.JavaUtilLog info
INFO: jetty-8.y.z-SNAPSHOT
Aug 13, 2014 2:21:46 PM org.eclipse.jetty.util.log.JavaUtilLog info
INFO: NO JSP Support for , did not find org.apache.jasper.servlet.JspServlet

My Java knowledge is relatively limited, so I tossed the JSP error message into Google. A stackoverflow thread was the first result and it talked about a possible misconfiguration with Jetty. I tried their trick of using the OPTIONS environment variable, but that didn’t work.

Then I realized that there wasn’t a Jetty package installed on my server. Ouch. The installation continues:

yum -y install jetty-jsp

Jenkins could now get off the ground and I saw the familiar log messages that I’m more accustomed to seeing:

Aug 13, 2014 2:24:26 PM hudson.WebAppMain$3 run
INFO: Jenkins is fully up and running

Much of these problems could stem from the fact that Jenkins RPM’s are built to suit a wide array of system versions and the dependencies aren’t configured correctly. My hope is that the Jenkins project for Fedora 21 will alleviate some of these problems and give the user a better experience.

The post Start Jenkins on Fedora 20 appeared first on major.io.

by Major Hayden at August 13, 2014 02:39 PM

httpry 0.1.8 available for RHEL and CentOS 7

Red Hat Enterprise Linux and CentOS 7 users can now install httpry 0.1.8 in EPEL 7 Beta. The new httpry version is also available for RHEL/CentOS 6 and supported Fedora versions (19, 20, 21 branched, and rawhide).

Configuring EPEL on a RHEL/CentOS server is easy. Follow the instructions on EPEL’s site and install the epel-release RPM that matches your OS release version.

If you haven’t used httpry before, check the output on Jason Bittel’s site. It’s a handy way to watch almost any type of HTTP server and see the traffic in an easier to read (and easier to grep) format.

The post httpry 0.1.8 available for RHEL and CentOS 7 appeared first on major.io.

by Major Hayden at August 13, 2014 01:20 PM

Chris Siebenmann

How you create a systemd .mount file for bind mounts

One of the types of units that systemd supports is mount units (see 'man systemd.mount'). Normally you set up all your mounts with /etc/fstab entries and you don't have to think about them, but under some specialized circumstances you can wind up needing to create real .mount service files for some mounts.

How to specify most filesystems is pretty straightforward, but it's not quite clear how you specify Linux bind mounts. Since I was just wrestling repeatedly with this today, here is what you need to put in a systemd .mount file to get a bind mount:

[Mount]
What=/some/old/dir
Where=/the/new/dir
Type=none
Options=bind

This corresponds to the mount command 'mount --bind /some/old/dir /the/new/dir' and an /etc/fstab line of '/some/old/dir /some/new/dir none bind'. Note that the type of the mount is none, not bind as you might expect. This works because current versions of mount will accept arguments of '-t none -o bind' as meaning 'do a bind mount'.

(I don't know if you can usefully add extra options to the Options setting or if you'd need an actual script if you need to, eg, make a bind mountpoint read-only. If you can do it in /etc/fstab you can probably do it here.)

A fully functioning .mount unit will generally have other stuff as well. What I've wound up using on Fedora 20 (mostly copied from the standard tmp.mount) is:

[Unit]
DefaultDependencies=no
Conflicts=umount.target
Before=local-fs.target umount.target

[Mount]
[[ .... whatever you need ...]]

[Install]
WantedBy=local-fs.target

Add additional dependencies, documentation, and so on as you need or want them. For what it's worth, I've also had bind mount units work without the three [Unit] bits I have here.

Note that this assumes a 'local' filesystem, not a network one. If you're dealing with a network filesystem or something depending on one, you'll need to change bits of the targets (systemd documentation suggests to remote-fs.target).

by cks at August 13, 2014 04:34 AM

August 12, 2014

That grumpy BSD guy

Password Gropers Take the Spamtrap Bait

We have just seen a new level of password gropers' imbecility. Or, Peak Stupid.

Regular readers of this column know that I pay attention to my logs, as any sysadmin worth his or her salt would. We read logs, or at least skim summaries because in between the endless sequence of messages that were successfuly delivered, users who logged in without a hitch and web pages served, from time to time unexpected things turn up. Every now and then, the unexpected log entries lead to discoveries that are useful, entertaining or even a bit bizarre.

In no particular order, my favorite log discoveries over the years have been
  • the ssh bruteforcer password guessers, a discovery that in turn lead to some smart people developing smart countermeasures that will shut each one of them down, automatically.
  • the faked sender addresses on spam, leading to bounces to non-existent users. Once again, thanks to the OpenBSD developers, I and others have been feeding those obviously fake addresses back to spamdb, slowing spammers down and generating publishable blacklists.
  • and never forgetting the relatively recent slow, distributed ssh bruteforcers, also known as "The Hail Mary Cloud", who in all likelihood had expected to avoid detection by spreading their access attempts over a long time and a large amount of hosts in coordination that was only detectable by reading the logs in detail.

After the Hail Mary Cloud cycle of attempted attacks, I thought I'd seen it all. 

But of course I was wrong.  Now we have more likely than not found evidence of another phenomenon which by perverse coincidence or fate appears to combine features of all these earlier activities.

Early in July 2014, my log summaries started turning up failed logins to the pop3 service (yes, I still run one, for the same bad reasons more than a third of my readers probably do: inertia).

At our site (which by now serves mainly as my personal lab in addition to serving a few old friends and family), pop3 activity besides successful logins by the handful of users who still use the service had been quite low for quite a while. And if we for now ignore the ssh bruteforcers who somehow never seem to tire of trying to log in as root, my log summaries had been quite boring for quite a while. But then the failed pop3 logins starting this July were for the unlikely targets admin, sales and info, user names that have never actually existed on that system, so I just kept ignoring the irregular and infrequent attempts.

My log summaries seemed to indicate that whoever is in charge of superonline.net (see the log summary) should tighten up a few things, but then it's really not my problem.

But then on July 18th, this log entry presented itself:

Jul 18 17:01:14 skapet spop3d[28606]: authentication failed: no such user: malseeinvmk - host-92-45-149-176.reverse.superonline.net (92.45.149.176)

That user name malseeinvmk is weird enough that I remembered adding it as part of one of the spamtrap addresses at one point. I had to check:

$ grep malseeinvmk sortlist
malseeinvmk@bsdly.net

which means yes, I remembered correctly. That user name is part of one of the more than twenty-eight thousand addresses on my traplist page, added at some point between 2006 and now to my spamdb database as a spamtrap, and as such known to be non-deliverable. So I started paying a bit more attention, and sure enough, over the next few days the logs (preserved here) were still turning up more entries from the traplist page. We would see local parts (usernames) only, of course, but grep would find them for us.

Now, trying to log on to a system with user names that are known not to exist sounds more than a little counterproductive at the best of times. But from my perspective, the activity was in fact quite harmless and could be potentially fun to watch, so I adjusted my log rotation configuration to preserve the relevant logs for a little longer than the default seven days.

Coming back to peek at the results every now and then, I noticed fairly soon that the attempts in almost all cases were concentrated in the first two minutes past every full hour. There are a couple of hour long periods with attempts spread more or less evenly with a few minutes in between, but the general pattern is a anything from one to maybe ten attempts in the :00:00-:02:00 interval.

The pattern was not exactly the Hail Mary Cloud one, but similar enough in that the long silent intervals could very well be an attempt at hiding in between other log noise.

But that returns us to the question, Why would anybody treat a list of known to be non-existent user names as if they actually offered a small chance of access?

They could be trying to weed out bad entries. One possible explanation would be that whoever is at the sending end here is trying to weed out the bad addresses in a long list that may or may not be the wanted quality. If an attempted login gives an indication whether the user exists or not, it might be worth trying.

They could be thinking the list is all good, and they want access. Brute force password guessing is not limited to ssh. We will explore this option further in a bit.

This could be an elaborate joke. The Hail Mary Cloud got passing mention in some mainstream press, and there are people out there who might be able to pull this off just for the hell of it.

Let's put each of those hypotheses to the test.

First, when you try to log in to the service, do you get any indication whether the user you attempt to log in as exists?

Here's what it looks like when a valid user logs in:

$ telnet skapet.bsdly.net pop3
Trying 213.187.179.198...
Connected to skapet.bsdly.net.
Escape character is '^]'.
+OK Solid POP3 server ready
USER peter
+OK username accepted
PASS n0neof.y3RB1Z
+OK authentication successful

At this point, I would have been able to list or retrieve messages or even delete them. But since my regular mail client does that better than I do by hand, I close the session instead:

quit
+OK session ended
Connection closed by foreign host.

And in case you were wondering, that string is not my current password, if it ever was.

Next, let us compare with what happens when you try logging in as a user that does not exist:

$ telnet skapet.bsdly.net pop3
Trying 213.187.179.198...
Connected to skapet.bsdly.net.
Escape character is '^]'.
+OK Solid POP3 server ready
USER jallaballa
+OK username accepted
PASS Gakkazoof
-ERR authentication failed
Connection closed by foreign host.

Here, too, the user name is tentatively accepted, but the login fails anyway without disclosing whether the password was the only thing that was invalid. If weeding out bad entries from the list of more than twenty-eight thousand was the objective, they're not getting any closer using this method.

Unless somebody actually bothered to compromise several hundred machines in order to pull off a joke that would be funny to a very limited set of people, the inescapable conclusion is that we are faced with would-be password guessers who
  • go about their business slowly and in short bursts, hoping to avoid detection
  • base their activity on a list that was put together with the explicit purpose of providing no valid information

If throwing semi-random but 'likely' user names and passwords at random IP addresses in slow motion had monumental odds against succeeding, I'm somewhat at a loss to describe the utter absurdity of this phenomenon. With trying to sneak under the radar to guess the passwords of users that have never existed, I think we're at the point where the Internet's bottom-feeding password gropers have finally hit peak stupidity.

More likely than not, this is the result of robots feeding robots with little or no human intervention, also known as excessive automation. Automation in IT is generally a good thing, but but I have a feeling somebody is about to discover the limits of this particular automation's usefulness.

I hate to break it to you, kids, but your imaginary friends, listed here, never actually existed, and they're not coming back.

And if this is actually a joke that has somebody, somewhere rolling on the floor laughing, now is a good time to 'fess up. There is still the matter of a few hundred compromised hosts to answer for, which may be a good idea to clear up as soon as possible.

As usual, I'll be tracking the activities of the miscreants and will refresh these resources at semi-random intervals as long as the activity persists:

At the rate they're going at the moment, we could be seeing them hang on for quite a while. And keep in mind that the list generally expands by a few new finds every week.

Update 2014-08-19: The attempts appear to have stopped. After some 3798 access attempts by 849 hosts trying 2093 user IDs, the last attempt so far was

Aug 17 22:40:58 skapet spop3d[20058]: authentication failed: no such user: jonatas-misbruke - host-92-45-135-208.reverse.superonline.net (92.45.135.208)

I take this as an indication that these access attempts are at least to some extent monitored, and with those numbers of attempts with a total of 0 successes, any reasonably constructed algorithm would have found reason to divert resources elsewhere. We can of course hope that some of the participating hosts have been cleaned up (although nobody actually wrote me back about doing that), and of courseyou can't quite rule out the possibility that whoever runs the operations reads slashdot.

But then again, the fact that the pop3 password gropers have moved on from my site should not lead you to believe that your site is no longer a target.

Update 2014-08-21: I spoke too soon about the pop3 password gropers giving up and moving on. In fact, only a few hours after the last update (in the early morning CEST hours of, two more attempts occured

Aug 20 05:07:41 skapet spop3d[2943]: authentication failed: no such user: info - 94.102.63.160
Aug 20 05:31:58 skapet spop3d[15882]: authentication failed: no such user: info - host-92-45-150-182.reverse.superonline.net (92.45.150.182)


before another long silence set in and I decided to wait a bit before making any new announcements.

But at just after ten in the morning CEST on August 21st, the slow and distributed probing with usernames lifted from my spamtraps page resumed. At the time of this writing, the total number of attempts has reached 3822, with 856 participating hosts and attempts on 2103 distinct user names. I'll publish refreshed files at quasi-random intervals, likely once per day if not more frequently.

Book of PF News: All ten chapters and the two appendixes have been through layout and proofing, front and back matter is still a work in progress, expected arrival for physical copies is still not set. Those most eager to read the thing could try early access or preorder.

by noreply@blogger.com (Peter N. M. Hansteen) at August 12, 2014 09:35 PM

August 11, 2014

Chris Siebenmann

Copying GPT partition tables from disk to disk

There's a number of situations where you want to replicate partition tables from one disk to another disk; for example, if you are setting up mirroring or (more likely) replacing a dead disk in a mirrored setup with a new one. If you're using old fashioned MBR partitioning, the best tool for this is sfdisk and it's done as follows:

sfdisk -d /dev/OLD | sfdisk /dev/NEW

Under some situations you may need 'sfdisk -f'.

If you're using new, modern GPT partitioning, the equivalent of sfdisk is sgdisk. However it gets used somewhat differently and you need two operations:

sgdisk -R=/dev/NEW /dev/OLD
sgdisk -G /dev/NEW

For obvious reasons you really, really don't want to accidentally flip the arguments. You need sgdisk -G to update the new disk's partitions to have different GUIDs from the original disk, because GUIDs should be globally unique even if the partitioning is the same.

The easiest way to see if your disks are using GPT or MBR partitioning is probably to run 'fdisk -l /dev/DISK' and look at what the 'Disklabel type' says. If it claims GPT partitioning, you can then run 'sgdisk -p /dev/DISK' to see if sgdisk likes the full GPT setup or if it reports problems. Alternately you can use 'gdisk -l /dev/DISK' and pay careful attention to the 'Partition table scan' results, but this option is actually kind of dangerous; under some situations gdisk will stop to prompt you about what to do about 'corrupted' GPTs.

Unfortunately sgdisk lacks any fully supported way of dumping and saving a relatively generic dump of partition information; 'sgdisk -b' explicitly creates something which the documentation says should not be restored on anything except the original disk. This is a hassle if you want to create a generic GPT based partitioning setup which you will exactly replicate on a whole fleet of disks (not that we use GPT partitioning on our new iSCSI backends, partly for this reason).

(I suspect that in practice you can use 'sgdisk -b' dumps for this even if it's not officially supported, but enhh. Don't forget to run 'sgdisk -G' on everything afterwards.)

(This is the kind of entry that I write so I have this information in a place where I can easily find it again.)

by cks at August 11, 2014 05:42 PM