Planet SysAdmin

Upcoming sysadmin conferences/events

Contact me to have your event listed here.

September 17, 2014

Chris Siebenmann

In praise of Solaris's pfiles command

I'm sure that at one point I was introduced to pfiles through a description that called it the Solaris version of lsof for a single process. This is true as far as it goes and I'm certain that I used pfiles as nothing more than this for a long time, but it understates what pfiles can do for you. This is because pfiles will give you a fair amount more information than lsof will, and much of that information is useful stuff to know.

Like lsof, pfiles will generally report what a file descriptor maps to (file, device, network connection, and Solaris IPC 'doors', often with information about what process is on the other end of the door). Unlike on some systems, the pfiles information is good enough to let you track down who is on the other end of Unix domain sockets and pipes. Sockets endpoints are usually reported directly; pipe information generally takes cross-correlating with other processes to see who else has an S_IFIFO with the specific ino open.

(You would think that getting information on the destination of Unix domain sockets would be basic information, but on some systems it can take terrible hacks.)

Pfiles will also report some socket state information for sockets, like the socket flags and the send and receive buffers. Personally I don't find this deeply useful and I wish that pfiles also showed things like the TCP window and ACK state. Fortunately you can get this protocol information with 'netstat -t inet -P tcp' or 'netstat -v -t inet -P tcp' (if you want lots of details).

Going beyond this lsof-like information, pfiles will also report various fcntl() and open() flags for the file descriptor. This will give you basic information like the FD's read/write status, but it goes beyond this; for example, you can immediately see whether or not a process has its sockets open in non-blocking mode (which can be important). This is often stuff that is not reported by other tools and having it handy can save you from needing deep dives with DTrace, a debugger, or the program source code.

(I'm sensitive to several of these issues because my recent Amanda troubleshooting left me needing to chart out the flow of pipes and to know whether some sockets were nonblocking or not. I could also have done with information on TCP window sizes at the time, but I didn't find the netstat stuff until just now. That's how it goes sometimes.)

by cks at September 17, 2014 06:04 AM

September 16, 2014

Standalone Sysadmin

Nagios Config Howto Followup

One of the most widely read stories I've ever posted on this blog is my Nagios Configuration HowTo, where I explained how I set up my Nagios config at a former employer. I still think that it's a good layout to use if you're manually building Nagios configs. In my current position, we have a small manual setup and a humongous automated monitoring setup. We're moving toward a completely automated monitoring config using Puppet and Nagios, but until everything is puppetized, some of it needs to be hand-crafted, bespoke monitoring.

For people who don't have a single 'source of truth' in their infrastructure that they can draw monitoring config from, hand-crafting is still the way to go, and if you're going to do it, you might as well not drive yourself insane. For that, you need to take advantage of the layers of abstraction in Nagios and the built-in object inheritance that it offers.

Every once in a while, new content gets posted that refers back to my Config HowTo, and I get a bump in visits, which is cool. Occasionally, I'll get someone who is interested and asks questions, which is what happened in this thread on Reddit. /u/sfrazer pointed to my config as something that he references when making Nagios configs (Thanks!), and the original submitter replied:

I've read that write up a couple of times. My configuration of Nagios doesn't have an objects, this is what it looks like

(click to embiggen)

And to understand what you are saying, just by putting them in the file structure you have in your HowTo that will create an inheritance?

I wanted to help him understand how Nagios inheritance works, so I wrote a relatively long response, and I thought that it might also help other people who still need to do this kind of thing:


No, the directories are just to help remember what is what, and so you don't have a single directory with hundreds of files.

What creates the inheritance is this:

You start out with a host template:

msimmons@nagios:/usr/local/nagios/etc/objects$ cat generic-host.cfg
define host {
    name generic-host
    notifications_enabled   1
    event_handler_enabled   1
    flap_detection_enabled  1
    failure_prediction_enabled  1
    process_perf_data   1
    retain_status_information   1
    retain_nonstatus_information    1
    max_check_attempts 3
    notification_period 24x7
    contact_groups systems
    check_command check-host-alive
    register 0

So, what you can see there is that I have a host named "generic-host" with a bunch of settings, and "register 0". The reason I have this is that I don't want to have to set all of those settings for every other host I make. That's WAY too much redundancy. Those settings will almost never change (and if we do have a specific host that needs to have the setting changed, we can do it on that host).

Once we have generic host, lets make a 'generic-linux' host that we can have the linux machines use:

msimmons@monitoring:/usr/local/nagios/etc/objects/linux$ cat generic-linux.cfg 
define host { 
    name     linux-server
    use generic-host
    check_period    24x7
    check_interval  5
    retry_interval  1
    max_check_attempts  5
    check_command   check-host-alive
    notification_interval 1440
    contact_groups  systems
    hostgroups linux-servers
    register 0

define hostgroup {
    hostgroup_name linux-servers
    alias Linux Servers

Alright, so you see we have two things there. A host, named 'linux-server', and you can see that it inherits from 'generic-host'. I then set some of the settings specific to the monitoring host that I'm using (for instance, you probably don't want notification_interval 1440, because that's WAY too long for most people - a whole day would go between Nagios notifications!). The point is that I set a bunch of default host settings in 'generic-host', then did more specific things in 'linux-server' which inherited the settings from 'generic-host'. And we made it 'register 0', which means it's not a "real" host, it's a template. Also, and this is important, you'll see that we set 'hostgroups linux-servers'. This means that any host we make that inherits from 'linux-server' will automatically be added to the 'linux-servers' hostgroup.

Right below that, we create the linux-servers hostgroup. We aren't listing any machines. We're creating an empty hostgroup, because remember, everything that inherits from linux-servers will automatically become a member of this group.

Alright, you'll notice that we don't have any "real" hosts yet. We're not going to yet, either. Lets do some services first.

msimmons@monitoring:/usr/local/nagios/etc/objects$ cat check-ssh.cfg
define command{
   command_name   check_ssh
   command_line   $USER1$/check_ssh $ARG1$ $HOSTADDRESS$

This is a short file which creates a command called "check_ssh". This isn't specific to Linux or anything else. It could be used by anything that needed to verify that SSH was running. Now, lets build a service that uses it:

msimmons@monitoring:/usr/local/nagios/etc/objects/services$ cat generic-service.cfg 
define service{
        name                            generic-service  
        active_checks_enabled           1     
        passive_checks_enabled          1      
        parallelize_check               1       
        obsess_over_service             1        
        check_freshness                 0         
        notifications_enabled           1          
        event_handler_enabled           1           
        flap_detection_enabled          1            
        failure_prediction_enabled      1             
        process_perf_data               1
        retain_status_information       1 
        retain_nonstatus_information    1  
        is_volatile                     0   
        check_period                    24x7 
        max_check_attempts              3     
        normal_check_interval           10     
        retry_check_interval            2       
        contact_groups                  systems
      notification_options    w,u,c,r        
        notification_interval           1440
        notification_period             24x7       
         register                        0            

This is just a generic service template with sane settings for my environment. Again, you'll want to use something good for yours. Now, something that will inherit from generic-service:

msimmons@monitoring:/usr/local/nagios/etc/objects/linux$ cat linux-ssh.cfg
define service { 
    use generic-service
    service_description Linux SSH Enabled
    hostgroup_name linux-servers
    check_command check_ssh 

Now we have a service "Linux SSH Enabled". This uses check_ssh, and (importantly), 'hostgroup_name linux-servers' means "Every machine that is a member of the hostgroup 'linux-servers' automatically gets this service check".

Lets do the same thing with ping:

define command{
        command_name    check_ping
        command_line    $USER1$/check_ping -H $HOSTADDRESS$ -w $ARG1$ -c $ARG2$ -p 5

 define service { 
    use generic-service
    service_description Linux Ping
    hostgroup_name linux-servers
    check_command check_ping!3000.0,80%!5000.0,100

Sweet. (If you're wondering about the exclamation marks on the check_ping line in the Linux Ping service, we're sending those arguments to the command, which you can see set the warning and critical thresholds).

Now, lets add our first host:

msimmons@monitoring:/usr/local/nagios/etc/objects/linux$ cat 
define host{
       use linux-server

That's it! I set the host name, I set the IP address, and I say "use linux-server" so that it automatically gets all of the "linux-server" settings, including belonging to the linux host group, which makes sure that it automatically gets assigned all of the Linux service checks. Ta-Da!

Hopefully this can help people see the value in arranging configs like this. If you have any questions, please let me know via comments. I'll be happy to explain! Thanks!



by Matt Simmons at September 16, 2014 10:22 PM

Steve Kemp's Blog

Applications updating & phoning home

Personally I believe that any application packaged for Debian should neither phone home, attempt to download plugins over HTTP at run-time, or update itself.

On that basis I've filed #761828.

As a project we have guidelines for what constitutes a "serious" bug, which generally boil down to a package containing a security issue, causing data-loss, or being unusuable.

I'd like to propose that these kind of tracking "things" are equally bad. If consensus could be reached that would be a good thing for the freedom of our users.

(Ooops I slipped into "us", "our user", I'm just an outsider looking in. Mostly.)

September 16, 2014 07:42 PM


We Need More Than Penetration Testing

Last week I read an article titled  People too trusting when it comes to their cybersecurity, experts say by Roy Wenzl of The Wichita Eagle. The following caught my eye and prompted this post:

[Connor] Brewer is a 19-year-old sophomore at Butler Community College, a self-described loner and tech geek...

Today he’s what technologists call a white-hat hacker, hacking legally for companies that pay to find their own security holes. 

When Bill Young, Butler’s chief information security officer, went looking for a white-hat hacker, he hired Brewer, though Brewer has yet to complete his associate’s degree at Butler...

Butler’s security system comes under attack several times a week, Young said...

Brewer and others like him are hired by companies to deliberately attack a company’s security network. These companies pay bounties if the white hackers find security holes. “Pen testing,” they call it, for “penetration testing.”

Young has repeatedly assigned Brewer to hack into Butler’s computer system. “He finds security problems,” Young said. “And I patch them.”

On the face of it, this sounds like a win-win story. A young white hat hacker does something he enjoys, and his community college benefits from his expertise to defend itself.

My concern with this article is the final sentence:

Young has repeatedly assigned Brewer to hack into Butler’s computer system. “He finds security problems,” Young said. “And I patch them.”

This article does not mention whether Butler's CISO spends any time looking for intruders who have already compromised his organization. Finding security problems and patching them is only one step in the security process.

I still believe that the two best words ever uttered by Bruce Schneier were "monitor first," and I worry that organizations like those in this article are patching holes while intruders maneuver around them within the compromised network.

by Richard Bejtlich ( at September 16, 2014 12:47 PM

A Brief History of Network Security Monitoring

Last week I was pleased to deliver the keynote at the first Security Onion Conference in Augusta, GA, organized and hosted by Doug Burks. This was probably my favorite security event of the year, attended by many fans of Security Onion and the network security monitoring (NSM) community.

Doug asked me to present the history of NSM. To convey some of the milestones in the development of this operational methodology, I developed these slides (pdf). They are all images, screen captures, and the like, but I promised to post them. For example, the image at left is the first slide from a Webinar that Bamm Visscher and I delivered on 4 December 2002, where we presented the formal definition of NSM the first time. We defined network security monitoring as

the collection, analysis, and escalation of indications and warnings to detect and respond to intrusions.

You may recognize similarities with the intelligence cycle and John Boyd's Observe - Orient - Decide Act (OODA) loop. That is not an accident.

During the presentation I noted a few key years and events:

  • 1986: The Cliff Stoll intrusions scare the government, military, and universities supporting gov and mil research.
  • 1988: Lawrence Livermore National Lab funds three security projects at UC Davis by supporting the Prof Karl Levitt's computer science lab. They include AV software, a "security profile inspector," and the "network security monitor."
  • 1988-1990: Todd Heberlein and colleagues code and write about the NSM platform.
  • 1991: While instrumenting a DISA location suffering from excessive bandwidth usage, NSM discovers 80% of the clogged link is caused by intruder activity.
  • 1992: Former FBI Director, then assistant AG, Robert Mueller writes a letter to NIST warning that NSM might not be legal.
  • 1 October 1992: AFCERT founded.
  • 10 September 1993: AFIWC founded.
  • End of 1995: 26 Air Force sites instrumented by NSM.
  • End of 1996: 55 Air Force sites instrumented by NSM.
  • End of 1997: Over 100 Air Force sites instrumented by NSM.
  • 1999: Melissa worm prompts AFCERT to develop dedicated anti-malware team. This signaled a shift from detection of human adversaries interacting with victims to detection of mindless code interacting with victims.
  • 2001: Bamm Visscher deploys SPREG, the predecessor to Sguil, at our MSSP at Ball Aerospace.
  • 13 July 2001: Using SPREG, one of our analysts detects Code Red, 6 days prior to the public outbreak. I send a note to a mailing list on 15 July.
  • February 2003: Bamm Visscher recodes and releases Sguil as an open source NSM console.

As I noted in my presentation,. the purpose of the talk was to share the fact that NSM has a long history, some of which happened when many practitioners (including myself) were still in school.

This is not a complete history, either. For more information, please see my 2007 post Network Security Monitoring History and the foreword, written by Todd Heberlein, of my newest book The Practice of Network Security Monitoring.

Finally, I wanted to emphasize that NSM is not just full packet capture or logging full content data. NSM is a process, although my latest book defines seven types of NSM data. One of those data types is full content. You can read about all of them in the first first chapter of my book at the publisher Web site.

by Richard Bejtlich ( at September 16, 2014 12:11 PM

Chris Siebenmann

My collection of spam and the spread of SMTP TLS

One of the things that my sinkhole SMTP server does that's new on my workstation is that it supports TLS, unlike my old real mail server there (which dates from a very, very long time ago). This has given me the chance to see how much of my incoming spam is delivered with TLS, which in turn has sparked some thoughts about the spread of SMTP TLS.

The starting point is that a surprising amount of my incoming spam is actually delivered with TLS; right now about 30% of the successful deliveries have used TLS. This is somewhat more striking than it sounds for two reasons; first, the Go TLS code I'm relying on for TLS is incomplete (and thus not all TLS-capable sending MTAs can actually do TLS with it), and second a certain amount of the TLS connection attempts fail because the sending MTA is offering an invalid client certificate.

(I also see a fair number of rejected delivery attempts in my SMTP command log that did negotiate TLS, but the stats there are somewhat tangled and I'm not going to try to summarize them.)

While there are some persistent spammers, most of the incoming email is your typical advance fee fraud and phish spam that's send through various sorts of compromised places. Much of the TLS email I get is this boring sort of spam, somewhat to my surprise. My prejudice is that a fair amount of this spam comes from old and neglected machines, which are exactly the machines that I would expect are least likely to do TLS.

(Some amount of such spam comes from compromised accounts at places like universities, which can and do happen to even modern and well run MTAs. I'm not surprised when they use TLS.)

What this says to me is that support for initiating TLS is fairly widespread in MTAs, even relatively old MTAs, and fairly well used. This is good news (it's now clear that pervasive encryption of traffic on the Internet is a good thing, even casual opportunistic encryption). I suspect that it's happened because common MTAs have enabled client TLS by default and the reason they've been able to do that is that it basically takes no configuration and almost always works.

(It's clear that at least some client MTAs take note when STARTTLS fails and don't try it again even if the server MTA offers it to them, because I see exactly this pattern in my SMTP logs from some clients.)

PS: you might wonder if persistent spammers use TLS when delivering their spam. I haven't done a systematic measurement for various reasons but on anecdotal spot checks it appears that my collection of them basically doesn't use TLS. This is probably unsurprising since TLS does take some extra work and CPU. I suspect that spammers may start switching if TLS becomes something that spam filtering systems use as a trust signal, just as some of them have started advertising DKIM signatures.

by cks at September 16, 2014 03:26 AM

RISKS Digest

September 15, 2014

Everything Sysadmin

In NYC for Velocity?

If you are in NYC for Velocity, please check out my tutorial on Monday, "Office Hours" on Tuesday, book signing on Wednesday, or come to my book party on Wednesday night!

...or just stop me randomly in the hallway and say "hi!" I love meeting new people!

September 15, 2014 04:00 PM

Brewster Rockit explains network latency.

Sunday's "Brewster Rockit" comic strip explained bandwidth vs. latency better than I've ever seen is some text books:

When I interview anyone for a technical position I always ask them to explain the difference between bandwidth and latency. It is an important concept, especially in today's networked world. Years ago most candidates didn't know the difference. Lately most candidates I interview know the difference, but have a difficult time putting it into words. Fewer can explain it in a mathematical or scientific way.

Latency is how long information takes to get from one place to another. Bandwidth is how much data per second is sent. Suppose you are querying a database that is very far away. The time it takes the query to go to the server and for the reply to come back could be very long (say, 1 second). If you do 1,000 database queries and each time wait for the query to complete before moving on to the next one, the task will take 1,000 seconds. If you increase the bandwidth between the two points the speed improvement will be nil or next to nil because the time it takes for the information to travel outweighs the transmission time. However if you send all your queries one after the next, without waiting for the replies, then wait and catch all the replies as they arrive, the entire list of 1,000 queries might happen in a matter of seconds. However, if you can't do the second query until you have information from the first query (say, the first query looks up a person's ID number and the second query uses the ID number as part of the query) you have to find some other optimization or you will simply have to wait 1,000 seconds for the entire process to run.

One situation I give the candidate is that they are dealing with a developer or manager that can't understand why doubling the bandwidth between the NYC office and Australia won't improve performance of a particular technical issue. Whether the candidate is a software developer, operations engineer/sysadmin, or desktop support engineer, there is a good chance they will find themselves in this situation. I roleplay as the person that needs the explanation and ask dumb questions. The candidate should be able to give a reasonable explanation in a couple minutes.

Maybe next time I should just ask them if they read Brewster Rockit.

September 15, 2014 03:00 PM

Safari Books Online update

Previously Safari Books Online (the O'Reilly thing... not the Apple thing) had a rough draft of The Practice of Cloud System Administration. Now it has the final version:


September 15, 2014 03:00 PM

Chris Siebenmann

I want my signed email to work a lot like SSH does

PGP and similar technologies have been in the news lately, and as a result of this I added the Enigmail extension to my testing Thunderbird instance. Dealing with PGP through Enigmail reminded me of why I'm not fond of PGP. I'm aware that people have all sorts of good reasons and that PGP itself has decent reasons for working the way it does, but for me the real strain point is not the interface but fundamentally how PGP wants me to work. Today I want to talk just about signed email, or rather however I want to deal with signed email.

To put it simply, I want people's keys for signed email to mostly work like SSH host keys. For most people the core of using SSH is not about specifically extending trust to specific, carefully validated host keys but instead about noticing if things change. In practical use you accept a host's SSH key the first time you're offered it and then SSH will scream loudly and violently if it ever changes. This is weaker than full verification but is far easier to use, and it complicates the job of an active attacker (especially one that wants to get away with it undetected). Similarly, in casual use of signed email I'm not going to bother carefully verifying keys; I'm instead going to trust that the key I fetched the first time for the Ubuntu or Red Hat or whatever security team is in fact their key. If I suddenly start getting alerts about a key mismatch, then I'm going to worry and start digging. A similar thing applies to personal correspondents; for the most part I'm going to passively acquire their keys from keyservers or other methods and, well, that's it.

(I'd also like this to extend to things like DKIM signatures of email, because frankly it would be really great if my email client noticed that this email is not DKIM-signed when all previous email from a given address had been.)

On the other hand, I don't know how much sense it makes to even think about general MUA interfaces for casual, opportunistic signed email. There is a part of me that thinks signed email is a sexy and easy application (which is why people keep doing it) that actually doesn't have much point most of the time. Humans do terribly at checking authentication, which is why we mostly delegate that to computers, yet casual signed email in MUAs is almost entirely human checked. Quick, are you going to notice that the email announcement of a new update from your vendor's security team is not signed? Are you going to even care if the update system itself insists on signed updates downloaded from secure mirrors?

(My answers are probably not and no, respectively.)

For all that it's nice to think about the problem (and to grumble about the annoyances of PGP), a part of me thinks that opportunistic signed email is not so much the wrong problem as an uninteresting problem that protects almost nothing that will ever be attacked.

(This also ties into the problem of false positives in security. The reality is that for casual message signatures, almost all missing or failed signatures are likely to have entirely innocent explanations. Or at least I think that this is the likely explanation today; perhaps mail gets attacked more often than I think on today's Internet.)

by cks at September 15, 2014 05:43 AM

Standalone Sysadmin

Mount NFS share to multiple hosts in vSphere 5.5

One of the annoying parts of making sure that you can successfully migrate virtual resources across a vSphere datacenter is ensuring that networks and datastores are not only available everywhere, but also named identically.

I inherited a system that was pretty much manually administered, without scripts. I've built a small powershell script to make sure that vSwitches can be provisioned identically when spinning up a new vHost, and there's really no excuse for not doing it for storage, except that not all of my hosts should have all of the same NFS datastores that another host has. I could do some kind of complicated menu system or long command line options, but that's hardly better than doing it individually.

Tonight, I learned about a really nice feature of the vSphere 5.5 web interface (which I am generally not fond of) - the ability to take a specific datastore and mount it on multiple hosts. (Thanks to Genroo on #vmware on Freenode for letting me know that it was a thing).

Log into the vSphere web interface and select "vCenter"



Select Datastores


Right click on the datastore you want to mount on other hosts. Select "All vCenter Actions", then 'Mount Datastore to Additional Host"



Pick the hosts you want to mount the datastore to.


The mount attempt will show up in the task list. Make sure that the hosts you select have access to NFS mount the datastore, otherwise it will fail. (You can see the X here, my failed attempt to rename a datastore after I created it using the wrong network for the filer. I'll clean that up shortly.

Anyway, hopefully this helps someone else in the future.

by Matt Simmons at September 15, 2014 01:03 AM

September 14, 2014

Chris Siebenmann

My current hassles with Firefox, Flash, and (HTML5) video

When I've written before about my extensions, I've said that I didn't bother with any sort of Flash blocking because NoScript handled that for me. The reality turns out to be that I was sort of living a charmed life, one that has recently stopped working the way I want it and forced me into a series of attempts at workarounds.

How I want Flash and video to work is that no Flash or video content activates automatically (autoplay is evil, among other things) but that I can activate any particular piece of content if and when I want to. Ideally this activation (by default) only last while the window is open; if I discard the window and re-visit the URL again, I don't get an autoplaying video or the like. In particular I want things to work this way on YouTube, which is my single most common source of videos (and also my most common need for JavaScript).

For a long time, things worked this way with just NoScript. Then at some point recently this broke down; if I relied only on NoScript, YouTube videos either would never play or would autoplay the moment the page loaded. If I turned on Firefox's 'ask to activate' for Flash, Firefox enabled and disabled things on a site-wide basis (so the second YouTube video I'd visit would autoplay). I wound up having to add two extensions to stop this:

  • Flashblock is the classic Flash blocker. Unlike Firefox's native 'ask to activate', it acts by default on a per-item basis, so activating one YouTube video I watch doesn't auto-play all future ones I look at. To make Flashblock work well I have disabled NoScript's blocking of Flash content so that I rely entirely on Flashblock; this has had the useful side effect of allowing me to turn on Flash elements on various things besides YouTube.

    But recently YouTube added a special trick to their JavaScript arsenal; if Flash doesn't seem to work but HTML5 video does, they auto-start their HTML5 player instead. For me this includes if Flashblock is blocking their Flash, so I had to find some way to deal with that.

  • StopTube stops YouTube autoplaying HTML5 videos. With both Flashblock and StopTube active, YouTube winds up using Flash (which is blocked and enabled by StopTube). I don't consider this ideal as I'd rather use HTML5, but YouTube is what it is. As the name of this addon sort of suggests, StopTube has the drawback that it only stops HTML5 video on YouTube itself. HTML5 video elsewhere are not blocked by it, including YouTube videos embedded on other people's pages. So far those embedded videos aren't autoplaying for me, but they may in the future. That might force me to not whitelist YouTube for JavaScript (at which point I almost might as well turn JavaScript off entirely in my browser).

    (An energetic person might be able to make such an addon starting from StopTube's source code.)

Some experimentation suggests that I might get back to what I want with just NoScript if I turn on NoScript's 'Apply these restrictions to whitelisted sites too' option for embedded content it blocks. But for now I like Flashblock's interface better (and I haven't been forced into this by being unable to block autoplaying HTML5 video).

There are still unfortunate aspects to this setup. One of them is that Firefox doesn't appear to have an 'ask to activate' (or more accurately 'ask to play') option for its HTML5 video support; this forces me to keep NoScript blocking that content instead of being able to use a nicer interface for enabling it if I want to. It honestly surprises me that Firefox doesn't already do this; it's an obviously feature and is only going to be more and more asked for as more people start using auto-playing HTML5 video for ads.

(See also this question and its answers.)

by cks at September 14, 2014 05:16 AM

September 13, 2014

Chris Siebenmann

What can go wrong with polling for writability on blocking sockets

Yesterday I wrote about how our performance problem with amandad were caused by amandad doing IO multiplexing wrong by only polling for whether it could read from its input file descriptors and assuming it could always write to its network sockets. But let's ask a question: suppose that amandad was also polling for writability on those network sockets. Would it work fine?

The answer is no, not without even more code changes, because amandad's network sockets aren't set to be non-blocking. The problem here is what it really means when poll() reports that something is ready for write (or for that matter, for read). Let me put it this way:

That poll() says a file descriptor is ready for writes doesn't mean that you can write an arbitrary amount of data to it without blocking.

When I put it this way, of course it can't. Can I write a gigabyte to a network socket or a pipe without blocking? Pretty much any kernel is going to say 'hell no'. Network sockets and pipes can never instantly absorb arbitrary amounts of data; there's always a limit somewhere. What poll()'s readiness indicator more or less means is that you can now write some data without blocking. How much data is uncertain.

The importance of non-blocking sockets is due to an API decision that Unix has made. Given that you can't write an arbitrary amount of data to a socket or a pipe without blocking, Unix has decided that by default when you write 'too much' you get blocked instead of getting a short write return (where you try to write N bytes and get told you wrote less than that). In order to not get blocked if you try a too large write you must explicitly set your file descriptor to non-blocking mode; at this point you will either get a short write or just an error (if you're trying to write and there is no room at all).

(This is a sensible API decision for reasons beyond the scope of this entry. And yes, it's not symmetric with reading from sockets and pipes.)

So if amandad just polled for writability but changed nothing else in its behavior, it would almost certainly still wind up blocking on writes to network sockets as it tried to stuff too much down them. At most it would wind up blocked somewhat less often because it would at least send some data immediately every time it tried to write to the network.

(The pernicious side of this particular bug is whether it bites you in any visible way depends on how much network IO you try to do how fast. If you send to the network (or to pipes) at a sufficiently slow rate, perhaps because your source of data is slow, you won't stall visibly on writes because there's always the capacity for how much data you're sending. Only when your send rates start overwhelming the receiver will you actively block in writes.)

Sidebar: The value of serendipity (even if I was wrong)

Yesterday I mentioned that my realization about the core cause of our amandad problem was sparked by remembering an apparently unrelated thing. As it happens, it was my memory of reading Rusty Russell's POLLOUT doesn't mean write(2) won't block: Part II that started me on this whole chain. A few rusty neurons woke up and said 'wait, poll() and then long write() waits? I was reading about that...' and off I went, even if my initial idea turned out to be wrong about the real cause. Had I not been reading Rusty Russell's blog I probably would have missed noticing the anomaly and as a result wasted a bunch of time at some point trying to figure out what the core problem was.

The write() issue is clearly in the air because Ewen McNeill also pointed it out in a comment on yesterday's entry. This is a good thing; the odd write behavior deserves to be better known so that it doesn't bite people.

by cks at September 13, 2014 04:48 AM

September 12, 2014

Steve Kemp's Blog

Storing and distributing secrets.

I run a number of hosts, and they are controlled via a server automation tool I wrote called slaughter [Documentation].

The policies I use to control my hosts are public and I don't want to make them private because they server as good examples.

Because the roles are public I don't want to embed passwords in them, which means I need something to hold secrets securely. In my case secrets are things like plaintext-passwords. I want those secrets to be secure and unavailable from untrusted hosts.

The simplest solution I could think of was an IP-address based ACL and a simple webserver. A client requests something like:


That returns a JSON object, if the requesting host is permitted to read the data. Otherwise it returns a HTTP 403 error.

The layout is very simple:

|-- secrets
|   |--
|   |   `-- auth.json
|   |--
|   |   `-- example.json
|   `--
|       `-- chat.json

Each piece of data is beneath a directory/symlink which controls the read-only access. If the request comes in from the suitable IP it is granted, if not it is denied.

For example a failing case:

skx@desktop ~ $ curl
missing/permission denied

A working case :

root@chat ~ # curl
{ "steve": "haha", "bot": "notreally" }

(The JSON suffix is added automatically.)

It is hardly rocket-science, but I couldn't find anything else packaged neatly for this - only things like auth/secstore and factotum. So I'll share if it is useful.

Simple Secret Sharing, or Steve's secret storage.

September 12, 2014 08:10 PM

The Nubby Admin

My Simple Trick for Quickly Making Secure Passwords

I’m frequently required to make test accounts on various systems. I need to create passwords that are obnoxiously long and complex many times in a day. My method for creating those passwords requires OpenSSL to be installed on my workstation:

$ alias secpass="openssl rand -base64 24"
$ secpass
$ thvxAWI5RT9Skmb/9zQ041zxF+P2enKe

A base 64 password has, in some cases, caused certain poorly designed systems to choke if there are special symbols included. In those cases I use the -hex option:

$ alias hexpass="openssl rand -hex 24"
$ hexpass
$ e1289af1316f33c91346c5512465c8488e6f2b5c3ff588fc

Ta da! Instant passwords as long and obnoxious as you please.


by WesleyDavid at September 12, 2014 11:15 AM

RISKS Digest