Planet SysAdmin

Chris Siebenmann: A general point about SSH personal keys

Thu, 09 Feb 2012 05:19:29 +0000

A general point about SSH personal keys

Recently I've seen a number of articles on suggested good ways to use SSH securely and other SSH tricks (unfortunately I can't find URLs to all of them, so I'm not going to try to put any here). As it happens I have a few modest suggestions on this, but before I started I wanted to make a broad meta-point about the use of personal SSH keys, aka SSH identities.

The big thing to understand about all advice about SSH personal keys is that when you choose to use personal keys for your own logins, you are deciding to balance convenience with security. After all, if security was your primary concern you would not use personal keys at all; you would use one time passwords with two-factor authentication.

(Things are different for cron'd scripts and the like, when there is no human there to interact with the system. I'm purely talking about using SSH identities to avoid typing passwords.)

Now, everyone has different views of the amount of security that they need and the convenience that they want. People fall along a spectrum between the two poles and where you wind up is not necessarily where I do. Thus, people's security advice about personal keys is not necessarily right for you even if it's correct (in some sense). The trick is to understand your particular tradeoffs and circumstances, to figure out what irritates you and what you need, and then to pick what works for you rather than blindly following someone else's suggestions and being either frustrated or dangerously insecure (in your environment) or both.

Yes, some things will make you less secure than others but they can also be more convenient (and vice versa). Sometimes this is the right tradeoff for you and sometimes it is not (even if it's the right tradeoff for me or whoever you're reading). And yes, there are some SSH tricks that usually increase both security and convenience. These are excellent things to know when you can find them.

(Sadly, my suggestions to come are not of this nature.)

PS: as always when you consider security related issues, you want to think about not just security in the abstract but security in the concrete in your environment with your risks.

USENIX Update: Cascadia IT Conference 2012

Wed, 08 Feb 2012 17:25:28 +0000

The second Cascadia IT Conference will be held March 23-24 in Seattle, Washington. LISA’11 co-chair, Tom Limoncelli, will be one of the speakers at the event. He’ll be giving talks on Time Management for System Administrators, The Limoncelli Test, and Ganeti virtual cluster management software. Other talks at the 2012 Cascadia IT Conference include: Essential [...]

CiscoZine: Nmap for IOS? No, IOSMap

Wed, 08 Feb 2012 09:24:44 +0000

The Tcl shell can be used to run Cisco IOS CLI EXEC commands within a Tcl script. Using the Tcl shell to run CLI commands allows customers to build menus to guide novice users through tasks, to automate repetitive tasks, and to create custom output for show commands. Not everyone knows that it is possible to implement a port scanning tool like a light Nmap. Surfing the web I have found a tool named IOSMap, a Cisco port scanning tool. It is not mandatory know Tcl to use this script; the only thing you need to know is how execute a [...]

SysAdmin1138: Looking to the future person

Wed, 08 Feb 2012 08:01:58 +0000

It is becoming increasingly clear that we'll be hiring a second full-time IT-type person some time this year. What's more, they're likely to be directly reporting to me. That would turn me into a straight up Manager, something I haven't been before. This is somewhat scary! The closest I've been was two jobs ago when I was providing work direction to two other people, but the actual time-card signoffs and vacation approvals was handled by someone else.

Which means I'm giving thought on what we'll need in terms of skillset for this nebulous person. As I see it, there are five primary knowledge domains that we're interested in:

The "buying things" one is new to this chart, in the past I've always had either a purchasing department to handle things like chasing down late orders, or have had single-source contracts that take a lot of the choice out of what we can buy. When you get to the senior levels and get 'recommend' powers (if not straight up purchasing authority) this kind of thing is actually pretty key. In fact, right now I have a specific order that I need in my hands by next Thursday OR ELSE, and I'm having to apply mallet to a supplier to get it. I never had to do this kind of thing before.

Anyway.

The tricky part is, what weight do we assign to each knowledge domain? Based on my workload right now, I'd have to say "automation coding" is primary. However, come August the pain we're feeling may be somewhere else entirely. Come August, our product should have been released for several months and we'll then have a lot of operational experience, and what we may need most of all is someone to share the midnight callout duties more than anything else.

And then there is the whole, "Now that I'm a manager-type person, what work is most suited to that?" question.

For me, of those five domains the Automation Coding portion is the domain with the highest interrupt costs; it takes me a good while to get back on track after someone asks me something random. Considering it's my job to be interrupted (even though that's a bad thing for systems administrators), this suggests having Someone Else do the automation coding part is a good idea. On the other hand, for large web-scale and highly homogeneous infrastructures automation coding is the majority of the systems engineering work needed to make it all function.

Right this moment, it's looking like my #2 will have to be someone with some years under their belt, which runs against our company's proven track-record of hiring promising people right out of college. Automation coding is that curious mix of a lot of Software Engineering crossed with the domain specific knowledges of OS lore, management frameworks, and application-specific functioning. Of these, it's the OS lore that's the hardest to train for.

However...

If I've done my job right, by the time we release the automation framework should be largely completed and should be good enough to handle at least half a year's worth of scale. If we release, and the industry loves us like a loving thing, we'll be scaling out madly. At that point, having another set of hands well versed in the Network Configuration and Server Hardware parts will be more important than a systems programmer. THAT is someone who could be a fresh-from-college person, much like our summer intern IT last year.

Won't know until we get there.

Chris Siebenmann: Choosing the superblock format for Linux's software RAID

Wed, 08 Feb 2012 06:20:38 +0000

Choosing the superblock format for Linux's software RAID

Linux's software RAID implementation stores metadata about the RAID device in each physical device involved in the RAID, in what mdadm calls 'RAID superblocks' by analogy to the filesystem superblocks that describe filesystems. In modern versions of software RAID there are a number of different formats for these RAID superblocks with different tradeoffs involved in each one, and one of the decisions you need to make when you create a software RAID array is what format you want to use.

(Even if you don't actively make a decision, mdadm will pick a format for you. Sometimes it will whine irritatingly at you about the situation, which is how I discovered the whole issue.)

In my opinion, at the moment there are three sensible options to choose from: the 0.90 format and then two variants of the 'version-1' metadata format.

0.90 is the original metadata format, which is widely understood and used. For most people, the most potentially important limitation of 0.90 metadata is that component devices can't be larger than 2 TB.
The 0.90 superblock goes at the end of the underlying partition.
1.0 puts the superblock at the end of the underlying partition.
1.2 puts the superblock 4 Kb from the start of the underlying partition It's the sort of default for modern versions of mdadm.

(You can see what format your current RAID arrays are using by looking at /proc/mdstat. If an array doesn't say 'super <something>' it's using 0.90 format metadata; otherwise, it's using whatever version it says it is. Many relatively modern systems, such as Ubuntu 10.04, either don't support anything past 0.90 or default to 0.90 in system setup.)

Where the superblock goes is potentially important for RAID-1 arrays. A RAID-1 array with the superblock at the end can relatively easily have whatever filesystem it contains mounted read-only without the RAID running, because the filesystem will start at the start of the underlying raw partitions; this can be important sometimes. A RAID-1 array with the superblock at or near the start of the underlying partitions can't have the raw partitions used this way, because you have to look somewhat beyond the start of the raw partition to see the filesystem.

(Some versions of mdadm will explicitly warn you about this or even quiz you about it if you don't specify a format explicitly.)

If you want to use a modern format and are going to directly use the RAID-1 array for a filesystem, I would use 1.0 format (this is what I've done for my new / and /boot). For swap areas you might as well use 1.2 format; if you ever need to use swap without software RAID, you can just destroy the 1.2 superblocks with mkswap. For LVM physical volumes you can argue back and forth either way; right now I've chosen 1.2 format because I really don't want to think about what it would take to safely bring up an LVM physical volume without software RAID running.

(LVM physical volumes have their own metadata, which normally goes at the start of the 'raw' partition that LVM is using but which can be replicated to the end as well. See pvcreate's manpage.)

As far as I know you can't change the superblock format of an array after it has been created, at least not without destroying it and recreating it. You can sort of do this without an extra disk with sufficient work, but really you want to get it right at creation time.

PS: note that in theory you can use dmsetup to gain access to filesystems or other sorts of data that doesn't begin at the start of a raw partition, so you can get at a filesystem embedded inside the raw partition of a RAID-1 array with 1.2 format metadata. However this requires user level intervention, which means that you're going to need a rescue environment or rescue disk of some sort.

Ben's Practical Admin Blog: iLO PS Library & Script Updates

Tue, 07 Feb 2012 22:39:19 +0000

The iLO PS Library has been updated to version 1.1.2. This release has minor bugfixes and a new function for parsing RIBCL output to obtain the CSR that is created. Latest version of the script can be downloaded of the iLO PS Page. I have completely re-written the iLO SSL Signing Script in the last [...]

MDLog:/sysadmin: HowTo completely remove a file from Git history

Tue, 07 Feb 2012 19:40:06 +0000

I just started working on a new project and as you would expect one of the first things I did was to download its git repository from github. These were just some scripts and should have been very small ~5M, but the clone from gitbhub took about one hour as the full repo folder was 1.5G… (with the biggest size under .git/objects/pack) Crazy… What was in the git repository history that would cause something like this? I assumed that at some point in time the repository was much bigger (probably from some file/s that don’t exist anymore), but how could I find out what were those files? And more important howto remove them from history? Well if you came here from a google search on “how to remove a file from git history” then you probably know there are plenty of docs and howtos on how to achieve this but from my experience none of them really worked. This is why I decided to document the steps needed to identify the file from the git repo history that is using all that space and to have it removed fully and bring the repository to a manageable size.

First we need to identify the file that is causing this issue; and for this we will verify all the packed objects and look for the biggest ones:
git verify-pack -v .git/objects/pack/*.idx | sort -k 3 -n | tail -5
(and grab the revisions with the biggest files). Then find the name of the files in those revisions:
git rev-list --objects --all | grep <revision_id>

Next, remove the file from all revisions:
git filter-branch --index-filter 'git rm --cached --ignore-unmatch <filename>' rm -rf .git/refs/original/

Edit .git/packed-refs and remove/comment any external pack-refs. Without this the cleanup might not work. I my case I had refs/remotes/origin/master and some others branches.
vim .git/packed-refs

Finally repack and cleanup and remove those objects:
git reflog expire --all --expire-unreachable=0 git repack -A -d git prune

Hopefully these steps will help you completely remove those un-wanted files from your git history. Let me know if you have any problems after following these simple steps.

Note: if you want to test these steps here is how to quickly create a test repo:
# Make a small repo mkdir test cd test git init echo hi > there git add there git commit -m 'Small repo' # Add a random 10M binary file dd if=/dev/urandom of=testme.txt count=10 bs=1M git add testme.txt git commit -m 'Add big binary file' # Remove the 10M binary file git rm testme.txt git commit -m 'Remove big binary file'

TechRepublic Network Administrator: How SAS, Near Line (NL) SAS, and SATA disks compare

Tue, 07 Feb 2012 18:20:44 +0000

Scott Lowe breaks down the differences in reliability and performance between SAS, Near-Line SAS, and SATA drives.

TechRepublic IT Security: VeriSign repeatedly hacked in 2010

Tue, 07 Feb 2012 14:39:25 +0000

Verisign was repeatedly attacked in 2010 but the extent of what was stolen is unknown, and the company didn't even own up to it until late 2011.

Racker Hacker: Using OpenSSL's s_client command with web servers using Server Name Indication (SNI)

Tue, 07 Feb 2012 14:07:41 +0000

One of the handiest tools in the OpenSSL toolbox is s_client. You can quickly view lots of details about the SSL certificates installed on a particular server and diagnose problems. For example, use this command to look at Google's SSL certificates:

openssl s_client -connect encrypted.google.com:443

You'll see the chain of certificates back to the original certificate authority where Google bought its certificate at the top, a copy of their SSL certificate in plain text in the middle, and a bunch of session-related information at the bottom.

This works really well when a site has one SSL certificate installed per IP address (this used to be a hard requirement). With Server Name Indication (SNI), a web server can have multiple SSL certificates installed on the same IP address. SNI-capable browsers will specify the hostname of the server they're trying to reach during the initial handshake process. This allows the web server to determine the correct SSL certificate to use for the connection.

If you try to connect to rackerhacker.com with s_client, you'll find that you receive the default SSL certificate installed on my server and not the one for this site:

$ openssl s_client -connect rackerhacker.com:443
Certificate chain
 0 s:/C=US/ST=Texas/L=San Antonio/O=MHTX Enterprises/CN=*.mhtx.net
   i:/C=US/O=SecureTrust Corporation/CN=SecureTrust CA
 1 s:/C=US/O=SecureTrust Corporation/CN=SecureTrust CA
   i:/C=US/O=Entrust.net/OU=www.entrust.net/CPS incorp. by ref. (limits liab.)/OU=(c) 1999 Entrust.net Limited/CN=Entrust.net Secure Server Certification Authority

Add on the -servername argument and s_client will do the additional SNI negotiation step for you:

$ openssl s_client -connect rackerhacker.com:443 -servername rackerhacker.com
Certificate chain
 0 s:/OU=Domain Control Validated/OU=PositiveSSL/CN=rackerhacker.com
   i:/C=GB/ST=Greater Manchester/L=Salford/O=Comodo CA Limited/CN=PositiveSSL CA
 1 s:/C=GB/ST=Greater Manchester/L=Salford/O=Comodo CA Limited/CN=PositiveSSL CA
   i:/C=US/ST=UT/L=Salt Lake City/O=The USERTRUST Network/OU=http://www.usertrust.com/CN=UTN-USERFirst-Hardware
 2 s:/C=US/ST=UT/L=Salt Lake City/O=The USERTRUST Network/OU=http://www.usertrust.com/CN=UTN-USERFirst-Hardware
   i:/C=SE/O=AddTrust AB/OU=AddTrust External TTP Network/CN=AddTrust External CA Root
 3 s:/C=SE/O=AddTrust AB/OU=AddTrust External TTP Network/CN=AddTrust External CA Root
   i:/C=SE/O=AddTrust AB/OU=AddTrust External TTP Network/CN=AddTrust External CA Root

You may be asking yourself this question:

Why doesn't the web server just use the Host: header that my browser sends already to figure out which SSL certificate to use?

Keep in mind that the SSL negotiation must occur prior to sending the HTTP request through to the remote server. That means that the browser and the server have to do the certificate exchange earlier in the process and the browser wouldn't get the opportunity to specify which site it's trying to reach. SNI fixes that by allowing a Host: header type of exchange during the SSL negotiation process.

Using OpenSSL's s_client command with web servers using Server Name Indication (SNI) is a post from: Major Hayden's Racker Hacker blog.

Thanks for following the blog via the RSS feed. Please don't copy my posts or quote portions of them without attribution.

TechRepublic IT Security: Juvenile cyber-delinquency: Laws that are turning kids into criminals

Tue, 07 Feb 2012 14:00:50 +0000

Deb Shinder considers the problem of juvenile delinquency when it concerns kids and computers. Do we need a better strategy for dealing with juvenile cybercrime? Take the opinion poll.

Anton Chuvakin - Security Warrior: Links for 2012-02-06 [del.icio.us]

Tue, 07 Feb 2012 08:00:00 +0000

Prediction Provides Questions; Not Answers

Chris Siebenmann: The advantage of HDMI for dual displays

Tue, 07 Feb 2012 04:25:00 +0000

The advantage of HDMI for dual displays

One of the interesting things that happened during my five years of hardware hibernation is that when I woke up, even low end (aka passively cooled) graphics cards could suddenly drive two digital outputs. Back in 2006 it was common for cards to have one analog and one digital out (eg, the ATI X300 in my work machine had VGA plus DVI), but getting dual digital out required an expensive card with an often noisy fan.

(I actually went through two such cards at work, each time deciding that I couldn't see enough advantage to driving my second display digitally instead of via analog VGA to be worth putting up with the noise. Possibly I wasn't sensitized enough to VGA artifacts and issues.)

What I have to thank for this is HDMI. Now, I'm aware that there's a lot to dislike about HDMI (see eg HDCP), but from my perspective the great thing about it is that it's given even low end cards a second digital output; it seems to be common for cards to have both DVI and HDMI. Some modern displays can be directly driven over HDMI and for the others, a simple cable will go from HDMI to DVI. And so my 2011 low end, passively cooled graphics card will now drive both my displays at work digitally, one directly with DVI and one with an HDMI to DVI cable, which is something that I never managed nicely before now.

(I believe that this has resolution limits. I don't use really big LCDs, so these haven't affected me.)

One of the interesting questions for me is why this happened. Why did graphics card vendors start putting HDMI on everything, where they only rarely did dual DVI? I think that part of the reason is that HDMI uses a physically small connector. DVI uses a relatively big connector and if you look at the back of a graphics card (especially a dual-DVI graphics card), there just isn't all that much physical space there; it's hard to get two DVI connectors and anything else in. By contrast, HDMI connectors are much smaller (I can't find the exact dimensions, but some sources say a third of the size). This makes it much easier to find the physical room for a HDMI connector on a card edge and on a circuit board.

(For example, my current graphics card just fits in VGA, DVI, and HDMI connectors with basically no spare room.)

PS: I don't think it's a coincidence that DisplayPort, the theoretical next generation replacement for DVI, also has a small connector. I suspect that the graphics card layout designers had a few words with people.

(Of course pretty much everything seems to be going to small connectors, with large ones proving awkward. Consider SATA versus IDE, for example. Someone who knew more about electronics than I do could probably write a fascinating article about all of the developments that made narrow-connector interfaces feasible and preferable to the old wide connector ones.)

(2 comments.)

TechRepublic IT Security: Why are websites getting your mobile-phone number?

Mon, 06 Feb 2012 16:24:13 +0000

Are mobile-service providers leaking data into traffic destined for web servers? That's what a research analyst set out to prove.

TechRepublic Network Administrator: Virtualize everything (except in these four scenarios)

Mon, 06 Feb 2012 15:08:42 +0000

Virtualization should be the de facto standard in deploying workloads today. However, there are valid exceptions. Rick Vanover lists four of them in this blog post.

The Nubby Admin: Iomega Jaz Disks – Why Didn’t They Take Off?

Mon, 06 Feb 2012 14:51:14 +0000

I’ve cleaned many a back room computer closet and found quite a number of artifacts that belie how far technology has come in just slightly more than a decade. One such item is the Iomega Jaz drive. I found a package of Jaz disks at a client and, instead of throwing them out, I dissected them and then threw them out.

I really didn’t know what to expect while cracking the cases open, but what I encountered made me stagger. When I opened up the case, I saw what amounted to a hard drive platter sitting loose on a plastic spindle:

The precious platters were protected by the mighty power of… bendy aluminium.

The bendy aluminium was held in place and allowed the freedom to move by a paper clip. Paperclips make the world go ’round!

I took a few more picture of the dissection and posted them on my Flickr account. Take a look at the ignominy here.

So how about it? Are you as appalled as I am that such a product made it to the consumer? Spew your ire in the comments below.

Slaptijack: Cisco IPS Inline Bypass Mode

Mon, 06 Feb 2012 14:00:37 +0000

On occasion, you may need to put your Cisco IPS into bypass mode. Bypass mode allows the IPS to pass traffic without inspecting it. This may be useful if you suspect the IPS is causing a problem, if you are going to run a one-time scan that would set the IPS off, or you need to update some part of the IPS software.

Inline Bypass Modes

There are three inline bypass modes:

Auto: In this mode, the sensor determines whether or not it should be in inline bypass mode. If the monitoring process is down, the sensor automatically shifts into inline bypass mode until the monitoring process returns.
Off: Inline bypass mode is disabled. Traffic will be forwarded to and is inspected by the Analysis Engine.
On: Inline bypass mode is disabled. Traffic flows but is not inspected.

There are two methods for putting the IPS into bypass mode.

Command-Line Interface (CLI)

This is really straightforward:

sensor# configure terminal
sensor(config)# service interface
sensor(config-int)# bypass-mode on
sensor(config-int)# exit
Apply Changes:?[yes]:

IPS Device Manager (IDM)

The IDM has a dedicated Bypass Mode Pane. Simply select "On" and save your configuration.

TechRepublic IT Security: The three types of online attackers

Mon, 06 Feb 2012 14:00:34 +0000

Patrick Lambert describes the three main groups of attackers that security specialists are guarding against.

Chris Siebenmann: My view on what will kill 'traditional' system administration

Mon, 06 Feb 2012 04:56:12 +0000

My view on what will kill 'traditional' system administration

Phil Hollenback recently wrote DevOps Is Here Whether You Like It Or Not, in which he writes that traditional system administration is dying. While I sort of agree with him about the death, I don't think it's necessarily for the reasons that Phil points to.

Fundamentally, there has always been a divide between small systems and large systems. Large systems have had to automate and when that automation involved applications, it involved the developers; small systems did not have to automate, and often do not automate because the costs of automation are larger than the costs of doing everything by hand. Moving to virtualization doesn't change that (at least for my sort of system administration, which has always had very little to do with shoving actual physical hardware around); if you have only a few virtualized servers and services, you can perfectly well keep running them by hand and it will probably be easier than learning Chef, Puppet, or CFEngine and then setting up an install.

(If you're future-proofing your career you want to learn Chef or Puppet anyways, so go ahead and use them even in a small environment.)

There are two things that I think will change that, and Phil points to one of them. Heroku is not just a virtualization provider; they are what I'll call a deployment provider, where if you write your application to their API you can simply push it to them without having to configure servers directly. We've seen deployment providers before (eg Google App Engine), but what distinguishes Heroku is how unconstrained and garden variety your API choices are. You don't need to write to special APIs to build a Heroku-ready application; in many cases, if you build an application in a sensible way it's automatically Heroku-ready. This is very enticing to developers because (among other things) it avoids lockin; if Heroku sucks for you, you can easily take your application elsewhere.

(This has historically not been true of other deployment providers, which makes writing things to, say, the Google AppEngine API a very big decision that you have to commit to very early on.)

Deployment providers like Heroku remove traditional system administration entirely. There's no systems or services to configure, and the developers are deeply involved in deployment because a non-developer can't really take a random application and deploy it for the developers. If there is an operations group, it's one that worries about higher level issues such as production environment performance and how to control the movement of code from development to production.

The other thing is general work to reduce the amount of knowledge you need to set up a Chef or Puppet-based environment with certain canned configurations. Right now my impression is that we're still at the stage where someone with experience has to write the initial recipe to configure all N of your servers correctly, and you might as well call that person a sysadmin (ie, they understand Apache config files, package installation on Ubuntu, etc). However it's quite possible that this is going to change over time to the point where we'll see programs shipped with Chef or Puppet recipes to install them in standard setups. At that point you won't need any special knowledge to go from, say, writing a Django-based application to installing it on the virtualization environment of your choice. This really will be the end of developers needing conventional sysadmins in order to get stuff done.

The general issue of the amount of hardware in a small business (and virtualizing the hardware) ties into a larger question of how much hardware the business of the future is going to need or want, but that's a different entry. I will just observe that the amount of servers that you need for a given amount of functionality has been steadily shrinking for years.

Sidebar: what virtualization does change now

I think that plain virtualization does mark a sea change today in one way: it moves sysadmins away from a model of upgrading OSes to a model of recreating their customizations on top of a new version of the OS. Possibly it moves away from upgrading software versions in general to 'build new install with new software versions from scratch, then configure'.

This is partly because the common virtualization model is 'provide base OS version X image, you customize from there' and partly because most virtualization makes it easy to build new server instances. It's much easier to start a new Ubuntu 12.04 image than it is to find a spare server to use as your 12.04 version of whatever.

(Note that virtualization may not make it any easier to replace your Ubuntu 10.04 server with a new 12.04 server; there are a host of low level practical issues that you can still run into unless you already have a sophisticated management environment built up.)

I don't think that this is a huge change for system administration, partly because this is pretty how much we've been doing things here for years. We basically never upgrade servers in place; we always build new servers from scratch. Among other things, it's much cleaner and more reproduceable that way.

Chris Siebenmann: Link: Filenames.WTF

Mon, 06 Feb 2012 00:28:08 +0000

Link: Filenames.WTF

In Filenames.WTF, Daniel Rutter runs down the reasons first why paying attention to file extensions is ridiculous, and then the reasons why it's still the best solution to the problem that we have. Spoiler: it's because people have spent decades creating file formats that suck.

(Via philliph on Twitter.)

Racker Hacker: The Kerberos-hater's guide to installing Kerberos

Sun, 05 Feb 2012 21:03:52 +0000

As promised in my earlier post entitled Kerberos for haters, I've assembled the simplest possible guide to get Kerberos up an running on two CentOS 5 servers.

Also, I don't really hate Kerberos. It's a bit of an inside joke with my coworkers who are studying for some of the RHCA exams at Rackspace. The additional security provided by Kerberos is quite good but the setup involves a lot of small steps. If you miss one of the steps or if you get something done out of order, you may have to scrap the whole setup and start over unless you can make sense of the errors in the log files. A lot of my dislikes for Kerberos comes from the number of steps required in the setup process and the difficulty in tracking down issues when they crop up.

To complete this guide, you'll need the following:

two CentOS, Red Hat Enterprise Linux or Scientific Linux 5 servers or VM's
some patience

Here's how I plan to name my servers:

kdc.example.com - the Kerberos KDC server at 192.168.250.2
client.example.com - the Kerberos client at 192.168.250.3

CRITICAL STEP: Before getting started, ensure that both systems have their hostnames properly set and both systems have the hostnames and IP addresses of both systems in /etc/hosts. Your server and client must be able to know the IP and hostname of the other system as well as themselves.

First off, we will need NIS working to serve up the user information for our client. Install the NIS server components on the KDC server:

[root@kdc ~]# yum install ypserv

Set the NIS domain and set a static port for ypserv to make it easier to firewall off. Edit /etc/sysconfig/network on the KDC server:

NISDOMAINNAME=EXAMPLE.COM
YPSERV_ARGS="-p 808"

Manually set the NIS domain on the KDC server and add it to /etc/yp.conf:

[root@kdc ~]# nisdomain EXAMPLE.COM
[root@kdc ~]# echo "domain EXAMPLE.COM server kdc.example.com" >> /etc/yp.conf

Adjust /var/yp/securenets on the KDC server for additional security:

[root@kdc ~]# echo "255.0.0.0 127.0.0.0" >> /var/yp/securenets
[root@kdc ~]# echo "255.255.255.0 192.168.250.0" >> /var/yp/securenets

Start the NIS server and generate the NIS maps:

[root@kdc ~]# /etc/init.d/ypserv start; chkconfig ypserv on
[root@kdc ~]# make -C /var/yp

I usually like to prepare my iptables rules ahead of time so I ensure that it doesn't derail me later on. Paste this into the KDC's terminal:

iptables -N SERVICES
iptables -I INPUT -j SERVICES
iptables -A SERVICES -p tcp --dport 111 -j ACCEPT -m comment --comment "rpc"
iptables -A SERVICES -p udp --dport 111 -j ACCEPT -m comment --comment "rpc"
iptables -A SERVICES -p tcp --dport 808 -j ACCEPT -m comment --comment "nis"
iptables -A SERVICES -p udp --dport 808 -j ACCEPT -m comment --comment "nis"
iptables -A SERVICES -p tcp --dport 88 -j ACCEPT -m comment --comment "kerberos"
iptables -A SERVICES -p udp --dport 88 -j ACCEPT -m comment --comment "kerberos"
iptables -A SERVICES -p udp --dport 464 -j ACCEPT -m comment --comment "kerberos"
iptables -A SERVICES -p tcp --dport 749 -j ACCEPT -m comment --comment "kerberos"
/etc/init.d/iptables save

We need our time in sync for Kerberos to work properly. Install NTP on both nodes, start it, and ensure it comes up at boot time:

[root@kdc ~]# yum -y install ntp && chkconfig ntpd on && /etc/init.d/ntpd start
[root@client ~]# yum -y install ntp && chkconfig ntpd on && /etc/init.d/ntpd start

Now we're ready to set up Kerberos. Start by installing some packages on the KDC:

[root@kdc ~]# yum install krb5-server krb5-workstation

We will need to make some edits to /etc/krb5.conf on the KDC to set up our KDC realm. Ensure that the default_realm is set:

default_realm = EXAMPLE.COM

The [realms] section should look like this:

[realms]
EXAMPLE.COM = {
	kdc = 192.168.250.2:88
	admin_server = 192.168.250.2:749
}

The [domain_realm] section should look like this:

[domain_realm]
kdc.example.com = EXAMPLE.COM
client.example.com = EXAMPLE.COM

Add validate = true within the pam { } block of the [appdefaults] section:

[appdefaults]
 pam = {
   validate = true

Adjust /var/kerberos/krb5kdc/kdc.conf on the KDC:

[realms]
EXAMPLE.COM = {
	master_key_type = des-hmac-sha1
	default_principal_flags = +preauth
}

There's one last configuration file to edit on the KDC! Ensure that /var/kerberos/krb5kdc/kadm5.acl looks like this:

*/admin@EXAMPLE.COM	    *

We're now ready to make a KDC database to hold our sensitive Kerberos data. Create the database and set a good password which you can remember. This command also stashes your password on the KDC so you don't have to enter it each time you start the KDC:

kdb5_util create -r EXAMPLE.COM -s

On the KDC, create a principal for the admin user as well as user1 (which we'll create shortly). Also, export the admin details to the kadmind key tab. You'll get some extra output after each one of these commands but I've snipped it to reduce the length of the post.

[root@kdc ~]# kadmin.local
kadmin.local:  addprinc root/admin
kadmin.local:  addprinc user1
kadmin.local:  ktadd -k /var/kerberos/krb5kdc/kadm5.keytab kadmin/admin
kadmin.local:  ktadd -k /var/kerberos/krb5kdc/kadm5.keytab kadmin/changepw
kadmin.local:  exit

Let's start the Kerberos KDC and kadmin daemons:

[root@kdc ~]# /etc/init.d/krb5kdc start; /etc/init.d/kadmin start
[root@kdc ~]# chkconfig krb5kdc on; chkconfig kadmin on

Now that the administration work is done, let's create a principal for our KDC server and stick it in it's keytab:

[root@kdc ~]# kadmin.local
kadmin.local:  addprinc -randkey host/kdc.example.com
kadmin.local:  ktadd host/kdc.example.com

Transfer your /etc/krb5.conf from the KDC server to the client. Hop onto the client server, install the Kerberos client package and add some host principals:

[root@client ~]# yum install krb5-workstation
[root@client ~]# kadmin.local
kadmin.local:  addpinc --randkey host/client.example.com
kadmin.local:  ktadd host/kdc.example.com

There aren't any daemons on the client side, so the configuration is pretty much wrapped up there for Kerberos. However, we now need to tell both servers to use Kerberos for auth and your client servers needs to use NIS to get user data.

On the KDC:
- run authconfig-tui
- choose Use Kerberos from the second column
- press Next
- don't edit the configuration (authconfig got the data from /etc/krb.conf)
- press OK
On the client:
- run authconfig-tui
- choose Use NIS and Use Kerberos
- press Next
- enter your NIS domain (EXAMPLE.COM) and NIS server (kdc.example.com or 192.168.250.2)
- press Next
- don't edit the Kerberos configuration (authconfig got the data from /etc/krb.conf)
- press OK

Got NIS problems? If the NIS connection stalls on the client, ensure that you have the iptables rules present on the KDC that we added near the beginning of this guide. Also, if you forgot to add both hosts to both servers' /etc/hosts, go do that now.

Let's make our test user on the KDC. Don't add this user to the client -- we'll get the user information via NIS and authenticate via Kerberos shortly. We'll also rebuild our NIS maps after adding the user:

[root@kdc ~]# useradd user1
[root@kdc ~]# passwd user1
[root@kdc ~]# make -C /var/yp/

On the client, see if you can get the password hash for the user1 account via NIS:

[root@client ~]# ypcat -d EXAMPLE.COM -h kdc.example.com passwd | grep user1
user1:$1$sUlSTlCv$riK5El3z8N4y.mi5Fe3Q60:500:500::/home/user1:/bin/bash

You can see why NIS isn't a good way to authenticate users. Someone could easily pull the hash for any account and brute force the hash on their own server. Go back to the KDC and lock out the user account:

[root@kdc ~]# usermod -p '!!' user1

Go back to the client and try to pull the password hash now:

[root@client ~]# ypcat -d EXAMPLE.COM -h kdc.example.com passwd | grep user1
user1:!!:500:500::/home/user1:/bin/bash

On the plus side, the user's password hash is now gone. On the negative side, you've just prevented this user from logging in locally or via NIS. Don't worry, the user can log in via Kerberos now. Let's prepare a home directory on the client for the user:

[root@client ~]# mkdir /home/user1
[root@client ~]# cp -av /etc/skel/.bash* /home/user1/
[root@client ~]# chown -R user1:user1 /home/user1/

Note: In a real-world scenario, you'd probably want to export this user's home directory via NFS so they didn't get a different home directory on every server.

While you're still on the client, try to log into the client via the user. Use the password that you used when you created the user1 principal on the KDC.

[root@client ~]# ssh user1@localhost
user1@localhost's password:
[user1@client ~]$ whoami
user1

List your Kerberos tickets and you should see one for your user principal:

[user1@client ~]$ klist
Ticket cache: FILE:/tmp/krb5cc_500_fCKPnZ
Default principal: user1@EXAMPLE.COM
 
Valid starting     Expires            Service principal
02/05/12 14:18:53  02/06/12 00:18:53  krbtgt/EXAMPLE.COM@EXAMPLE.COM
	renew until 02/05/12 14:18:53

Your KDC should have a couple of lines in its /var/log/krb5kdc.log showing the authentication:

Feb 05 14:18:53 kdc.example.com krb5kdc[4694](info): AS_REQ (12 etypes {18 17 16 23 1 3 2 11 10 15 12 13}) 192.168.250.3: ISSUE: authtime 1328473133, etypes {rep=16 tkt=16 ses=16}, user1@EXAMPLE.COM for krbtgt/EXAMPLE.COM@EXAMPLE.COM
Feb 05 14:18:53 kdc.example.com krb5kdc[4694](info): TGS_REQ (7 etypes {18 17 16 23 1 3 2}) 192.168.250.3: ISSUE: authtime 1328473133, etypes {rep=16 tkt=18 ses=18}, user1@EXAMPLE.COM for host/client.example.com@EXAMPLE.COM

The first line shows that the client asked for a Authentication Server Request (AS_REQ) and the second line shows that the client then asked for a Ticket Granting Server Request (TGS_REQ). In layman's terms, the client first asked for a ticket-granting ticket (TGT) so it could authenticate to other services. When it actually tried to log in via ssh it asked for a ticket (and received it).

YOU JUST CONFIGURED KERBEROS!

From here, the sky's the limit. Another popular implementation of Kerberos is encrypted NFSv4. You can even go crazy and use Kerberos with apache.

Let me know if you have any questions about this post or if you spot any errors. With this many steps, there's bound to be a typo or two in this guide. Keep in mind that there are some obvious spots for network-level and service-level security improvements. This guide was intended to give you the basics and it doesn't cover all of the security implications involved with a Kerberos implementation.

The Kerberos-hater's guide to installing Kerberos is a post from: Major Hayden's Racker Hacker blog.

Thanks for following the blog via the RSS feed. Please don't copy my posts or quote portions of them without attribution.

Steve Kemp's Blog: Some domains just don't learn

Sun, 05 Feb 2012 13:24:44 +0000

For the past few years the anti-spam system I run has been based on a simplified version of something I previously ran commercially.

Although the code is similar in intent there were both explicit feature removals, and simplifications made.

Last month I re-implimented domain-blacklisting - because a single company keeps ignoring requests to remove me.

So LinkedIn.com if you're reading this:

I've never had an account on your servers.
I find your junk mail annoying.
I suspect I'll join your site/service when hell freezes over.

I've also implemented TLD-blacklisting which has been useful.

TLD-blacklisting in my world is not about blocking mail from foo@bar.ph (whether in the envelope sender, or the from: header), instead it is about matching the reverse DNS of the connecting client.

If I recieve a connection from 1.2.3.4 and the reverse DNS of that IP address matches, say, /\.sa$/i then I default to denying it.

My real list is longer, and handled via files:
steve@steve:~$ ls /srv/_global_/blacklisted/tld/ -C
ar  br  cn  eg  hr  in  kr  lv  mn  np  ph  ro  sg  tg  ua  ve  zw
aw  cc  cy  gm  hu  is  kz  ma  my  nu  pk  rs  sk  th  ug  vn
be  ch  cz  gr  id  it  lk  md  mz  nz  pl  ru  su  tr  uy  ws
bg  cl  ec  hk  il  ke  lt  mk  no  om  pt  sa  sy  tw  uz  za

On average I'm rejecting about 2500 messagse a day at SMTP-time, and 30 messages, or so, hit my SPAM folder after being filtered with CRM114 after being accepted for delivery. (They are largely from @hotmail and @yahoo, along with random compromised machines. The amount of times I see a single mail from a host with RDNS mysql.example.org is staggering.).

(Still looking forward to the development of Haraka, a node.js version of qpsmtpd.)

ObQuote: "Mr. Mystery Guest? Are you still there? " - Die Hard

Anton Chuvakin - Security Warrior: Links for 2012-02-04 [del.icio.us]

Sun, 05 Feb 2012 08:00:00 +0000

The Valley of Death Between IT and Security

Chris Siebenmann: What five years of PC technology changed for me

Sun, 05 Feb 2012 07:28:44 +0000

What five years of PC technology changed for me

This fall I got a new home machine, just a bit over exactly five years after I got my previous home machine. It happens that I saved the invoice for my five year old machine, so I dug it out today in order to do a comparison about what five years of progress in PC technology did and didn't change for me.

First off, the progress of five years got me much better prices. My recent home machine cost me only about 60% of what my old home machine did. By itself, this is pretty impressive. Apart from that, running down the major components:

CPU: AMD dual core versus much faster Intel quad core. The Intel CPU was cheaper but not by a substantial amount; I think the AMD was probably closer to the high end at the time. I don't know what the benchmark results are, but I got a substantial performance improvement.
RAM: This is perhaps the most striking change on a purely numerical level; in 2006 I got 2GB of RAM for more than twice as much as what 16 GB of RAM cost me in 2011. Even in 2006, 2 GB was clearly economizing (I remember debating with myself over 2 GB versus the extra money for 4 GB and deciding that 2 GB should be good enough). In 2011, 16 GB is as much as the motherboard will take with current DIMM densities.
In short, desktop RAM has become stupid cheap.
(One index of the change is that in 2006, the 2 GB of RAM cost more than the CPU and was the most expensive single component. In 2011, the 16 GB cost only a bit over half of the CPU.)
motherboard: the modern era features more SATA, less IDE, more USB, and not even one external serial port. Motherboards are unexciting. Even in 2006 the motherboard had onboard sound and gigabit Ethernet. The 2011 motherboard probably has better onboard sound, but in practice this doesn't matter to me; my sound needs are modest.
(The 2006 motherboard was a bit cheaper than the 2011 motherboard, but neither were particularly expensive or advanced ones.)
Hard drives changed only moderately at one level; in 2006 I got 320 GB drives for somewhat over twice what 2011's 500 GB drives cost me. In 2011, 500 GB drives are nowhere near state of the art; in 2006, 320 GB drives were not that far out of it.
(This was before the floods in Thailand.)
On another level, they changed a lot. The 320 GB hard drives of 2006 were my only storage. The 500 GB drives of 2011 are only for the operating system; my data lives on a pair of 1.5 TB drives (that I had upgraded to some time ago). 500 GB is way overkill for the OS, but there's no real point in using drives that are any smaller; it's not like I'd have saved any significant amount of money.
Video card: ATI X800 GT versus ATI HD 5450 with double the memory for less than a third of the price. Toms Hardware theoretically puts these two cards in almost the same performance category, although I'm not sure that's really true. In practice, what happened between 2006 and 2011 is that graphics cards shifted to the point where a basic passively cooled card was clearly more than good enough for what I was doing, even for driving dual displays digitally.
(I don't yet have dual displays at home, but I do at work and my work machine uses the same card. In fact, my work machine is now a clone of my home machine, just as it was in 2006.)
optical drives: in 2006 a DVD burner cost about four times what it did in 2011, and I thought I would listen to CDs enough to justify having a separate CD/DVD reader (rather than put wear and tear on an expensive burner).
(I was wrong; my CD listening had already dropped off a cliff in early 2006 and never recovered. I still kind of miss that sometimes.)
Power supply: in 2006 I didn't trust the power supply that came with the case to really be a good solid one that delivered enough power so I bought a separate one as well. In 2011 I couldn't find any reason to worry about it so I didn't; the power supply you get with a decent quiet case these days is going to be quite good, more than you need (for a PC like the kind I build), and efficient.

In 2006, the most expensive components were the RAM, the CPU, the two hard drives together, and then the video card. In 2011, the most expensive components were the CPU, the motherboard, and the case (more or less tying with the RAM). Another way to put it is that in 2011, the video card, the DVD burner, the hard drives, and pretty much the RAM were all what I considered trivial expenses in the overall machine.

(One comment.)

My SysAd Blog: How to Exclude a Directory for TAR

noreply@blogger.com (esofthub) — Sun, 05 Feb 2012 04:35:41 +0000

I am doing a few aesthetic upgrades, so naturally I want to backup some of my website files. However, I do not want everything, i.e. transient files such images. I tarred up the website's primary directory but excluded its images sub-directory.

Frankly speaking, I did not want to waste time and bandwidth downloading images. Here is the syntax to exclude a directory.

# tar cvfp mytarball.tar /mypath/Example.com_DIR --exclude=/mypath/Example.com_DIR/images

Tar everything in the current directory but exclude two files

# tar cvpf mytar.tar * --exclude=index.html --exclude=myimage.png

Racker Hacker: Get notifications instead of automatic updates in Scientific Linux

Sat, 04 Feb 2012 19:01:54 +0000

Scientific Linux installations have a package called yum-autoupdate by default and the package contains two files:

# rpm -ql yum-autoupdate
/etc/cron.daily/yum-autoupdate
/etc/sysconfig/yum-autoupdate

The cron job contains the entire script to run automatic updates once a day and the configuration file controls its behavior. However, you can't get the same functionality as Fedora's yum-updatesd package where you can receive notifications for updates rather than automatically updating the packages.

To get those notifications in Scientific Linux, just make two small edits to this portion of /etc/cron.daily/yum-autoupdate:

173           echo "    Starting Yum with command"
174           echo "     /usr/bin/yum -c $TEMPCONFIGFILE -e 0 -d 1 -y update"         
175   fi
176   /usr/bin/yum -c $TEMPCONFIGFILE -e 0 -d 1 -y update > $TEMPFILE 2>&1
177   if [ -s $TEMPFILE ] ; then

Adjust the update commands to look like this:

173           echo "    Starting Yum with command"
174           echo "     /usr/bin/yum -c $TEMPCONFIGFILE -e 0 -d 1 -y check-update"         
175   fi
176   /usr/bin/yum -c $TEMPCONFIGFILE -e 0 -d 1 -y check-update > $TEMPFILE 2>&1
177   if [ -s $TEMPFILE ] ; then

Since you won't be auto-updating with this script any longer, you may want to comment out the EXCLUDE= line in /etc/sysconfig/yum-autoupdate so that you'll receive notifications for all packages with updates. Also, to avoid having your changes updated with a newer yum-autoupdate package later, add the package to your list of excluded packages in /etc/yum.conf.

Get notifications instead of automatic updates in Scientific Linux is a post from: Major Hayden's Racker Hacker blog.

Thanks for following the blog via the RSS feed. Please don't copy my posts or quote portions of them without attribution.

TaoSecurity: Impressions: Network Warrior, 2nd Ed

noreply@blogger.com (Richard Bejtlich) — Sat, 04 Feb 2012 15:18:01 +0000

Five years ago I reviewed the first edition of Network Warrior by Gary A. Donahue. Thank to O'Reilly I can post my "impressions" of the second edition of this great book. Although I read almost all of it, I am unable to post another review because Amazon.com has my previous review attached to the new edition.

In brief, Network Warrior, 2nd Ed is the book to read if you are a network administrator trying to get to the next level. All of my praise from the previous review apply to the new book. The book is really that good, primarily because it combines very clear explanations with healthy doses of real-world experience. Thanks to Mr Donahue for taking the time to update his book!

TaoSecurity: Impressions: Windows Sysinternals Administrator's Reference

noreply@blogger.com (Richard Bejtlich) — Sat, 04 Feb 2012 15:01:12 +0000

Mark Russinovich and Aaron Margosis have written another awesome addition to the Microsoft Press catalog, Windows Sysinternals Administrator's Reference. Per my policy, because I did not read the whole book I am only posting "impressions" here and not a full Amazon.com review.

In brief this book will tell you more about the awesome Sysinternals tools than you might have thought possible. One topic that caught my attention was using Process Monitor to summarize network activity (p 139). This reminded me of Event Tracing for Windows and Network Tracing in Windows 7. I remain interested in this capability because it can be handy for incident responders to collect network traffic on endpoints without installing new software, relying instead on native OS capabilities.

I suggest keeping a copy of this book in your team library if you run a CIRT. Thorough knowledge of the Sysinternals tools is a great benefit to anyone trying to identify compromised Windows computers.

TaoSecurity: Impressions: The Tangled Web

noreply@blogger.com (Richard Bejtlich) — Sat, 04 Feb 2012 14:23:43 +0000

Six years ago I reviewed Michal Zalewski's first book, Silence on the Wire. Michal is a security researcher who has consistently created high-quality content for a very long time, so I was pleased to receive a review copy of his newest book The Tangled Web.

I did not read the whole book, hence I'm posting only my "impressions" here. I recommend reading this book if you want to know a lot, and I mean a lot, about how screwed up Web browsers, protocols, and related technologies truly are. Because many points of the book are tied to specific browser versions, I suspect its shelf life to degrade a little more rapidly than some other technical titles. Still, I am shocked by the amount of research and documentation Michal performed to create The Tangled Web.

As always, Michal's content is highly readable, very detailed, and well-sourced. It's a great example for other technical authors. Great work Michal!

TaoSecurity: The Toughest Question in Digital Security

noreply@blogger.com (Richard Bejtlich) — Sat, 04 Feb 2012 13:35:30 +0000

The toughest question in digital security is "who cares?"

The recent Tweet by hogfly (@4n6ir) made me ponder this question. He points to an Aviation Week story by David Fulghum, Bill Sweetman, and Amy Butler titled China's Role In JSF's Spiraling Costs. It says in part:

How much of the F-35 Joint Strike Fighter’s spiraling cost in recent years can be traced to China’s cybertheft of technology and the subsequent need to reduce the fifth-generation aircraft’s vulnerability to detection and electronic attack?

That is a central question that budget planners are asking, and their queries appear to have validity. Moreover, senior Pentagon and industry officials say other classified weapon programs are suffering from the same problem. Before the intrusions were discovered nearly three years ago, Chinese hackers actually sat in on what were supposed to have been secure, online program-progress conferences, the officials say.

The full extent of the connection is still being assessed, but there is consensus that escalating costs, reduced annual purchases and production stretch-outs are a reflection to some degree of the need for redesign of critical equipment. Examples include specialized communications and antenna arrays for stealth aircraft, as well as significant rewriting of software to protect systems vulnerable to hacking.

It is only recently that U.S. officials have started talking openly about how data losses are driving up the cost of military programs and creating operational vulnerabilities, although claims of a large impact on the Lockheed Martin JSF are drawing mixed responses from senior leaders. All the same, no one is saying there has been no impact.

While claiming ignorance of details about effects on the stealth strike aircraft program, James Clapper, director of national intelligence, says that Internet technology has “led to egregious pilfering of intellectual capital and property. The F-35 was clearly a target,” he confirms.

The point of this article is to question the impact, in business and operational terms, of the cyberwar China continues to prosecute against the West.

The toughest question in digital security is "who cares" because it is usually extremely difficult to determine the impact of an intrusion. Consider the steps required to define the business and operational impact of the theft of intellectual property (as one example -- there are many others).

The victim must learn that an intrusion occurred.
The victim must determine exactly what IP was stolen.
The victim must understand the adversary's capability and intention to exploit the stolen IP.
The victim must recognize when the adversary exploits the stolen IP by using it in an operational context.
The victim must determine what countermeasures or changes in courses of actions are possible to mitigate the adversary's exploitation of the stolen IP.
The victim must synthesize most or all of the previous points into an assessment of the business and operational cost of the IP theft.

Steps 1 and 2 are largely technical, but 3-6 are more business-focused. From what I have seen, everyone who is a victim in the ongoing cyberwar struggles to conduct "battle damage assessment" (BDA) for digital intrusions. Articles like the one I cited are examples showing how difficult it is to determine if anyone should care about China's exploitation of Western IP.

Chris Siebenmann: Understanding a subtle Twitter feature

Sat, 04 Feb 2012 03:49:10 +0000

Understanding a subtle Twitter feature

One part of getting on Twitter has been following people, which led me to discover that when you follow someone Twitter doesn't show you all of their public tweets. To summarize what I think is the rule, Twitter excludes any conversations they're having that purely involve other people you don't also follow. Their tweets in the conversation will appear in their public timeline, but not in your view of their tweets.

(This may only apply to relatively new Twitter accounts, or even only to some of them. I've seen Twitter give two different interfaces to two new accounts.)

On the one hand, when I discovered this I was infuriated. If you really did want to see everything (for example, so you could find other people to follow based on who your initial people had interesting conversations with), this made having a Twitter account worse than just perusing the Twitter pages of interesting people.

On the other hand, once I thought about it more I've come to reluctantly admire Twitter's trick with this feature. What it is, from my perspective, is a clever way to reduce the volume impact of following someone and thus make doing so less risky. Without it, following someone would immediately expose you to both their general remarks and to the full flow of whatever conversations they have. With Twitter's way, you are only initially exposed to people's general remarks; you ramp up your exposure to their conversations by following more people, and ramp it down by the reverse.

My feeling is that exposure to an overwhelming firehose of updates is the general problem of social networking. Social networks usually want you to be active and to follow lots of people. But if those people are themselves active, the more people you follow the more volume descends on you, and it's especially bad when you follow very socially active users, the ones having a lot of conversations. This creates a disincentive to follow people and pushes you to scale back. Twitter has this especially badly because it has no separate 'comment' mechanism (comments are important for reducing volume). Twitter's trick here is thus a clever way to reduce the firehose in a natural way that doesn't require user intervention and tuning; you could see it as a way of recreating something like comments in a system that doesn't naturally have them.

Once I realized this, it's certainly been working the way that Twitter probably intended. When I'm considering whether or not to follow someone I don't really look at the volume of their tweets in general; I mostly look just at the volume of their non-conversation tweets, because those are the only ones that I'm going to see. Often this makes me more willing to follow people (and thereby furthers Twitter's overall goal of getting me more engaged with their service).

(4 comments.)

USENIX Update: LISA’12: First Peek

Fri, 03 Feb 2012 18:34:05 +0000

Now that the LISA’12 Call for Papers is officially open, it’s time to start getting officially excited about our 2012 event, which will be held in San Diego from December 9-14. In this interview, I asked LISA’12 program chair Carolyn Rowland to give us a super early look at what’s in store for LISA attendees [...]

HolisticInfoSec.org: toolsmith: Splunk app - Windows Security Operation Center

noreply@blogger.com (Russ McRee) — Fri, 03 Feb 2012 16:24:00 +0000

Prerequisites

Windows 2003, 2008, 7

Splunk (Free or Enterprise)

Introduction

As a volunteer handler for the SANS Internet StormCenter, I am privileged to work with some incredibly bright, highly capableinformation security professionals. As saidindividuals create new tools or update those they maintain I have the advantageof early awareness and access. Bojan Zdrnja’s Splunk app, Windows Security OperationsCenter (referred to as WSOC hereafter) is a perfect example. By the time youread this a new version should be available on Splunkbase.

Bojan bought me up to speed on his latest effort viaemail.

The latest version of WSOC contains bug fixes (mainlyminor search tweaks) along with a couple of new dashboards:

1. Adashboard for up-to-date servers with patches

2. DirectoryServices dashboards

The Directory Services dashboards are very useful as theyshow changes to objects in AD including creations, deletions, and modifications.These views are excellent for auditors.

In the future Bojan plans to add support for otherproducts normally found in Microsoft environments, including infrastructureelements such as DNS/DHCP, IIS, SQL server, and perhaps TMG. WSOC’s primarypurpose is to cover all potential security views an auditor or informationsecurity personnel might want purview of; there’ll be no run-of-the-mill operationalmonitoring here ;-).

Bojan offered many favorite use cases. People are not alwaysaware of what's going on in their Windows environments. In almost everyimplementation he’s encountered he found automated tools/services filling logs inabundance. As an example, when the tool tries to access a resourceautomatically, it generates an AD authentication failure event and then itsuccessfully authenticates through NTLM. This causes logs to growsubstantially. The same dashboards can be used to easily spot infected machinesor brute force attacks on the network, thanks to Splunk's excellent visualizationcapabilities. WSOC includes a table that shows a distinct count of failed loginattempts per username per machine, so if a machine is brute forcing, even ifit's slow, you'll be able to see it.

Auditors are particularly fond of the user/group managementdashboards. They produce ready evidence, in one view, of which users were addedto which group. When coupled with change requests, yours becomes anorganization that is then better prepared for audits.

The dashboard showing installed services supports this welltoo as any installed service should have an accompanied change request (seefurther discussion below).

Bojan wanted to stress the missing patches dashboard asextremely valuable. This information is collected from the local Windows Updateagent on every server. Of course, in order for it to be accurate, the WindowsUpdate agent must be able to connect to WSUS or Microsoft's update server, butassuming it can, results will populate nicely showing servers that have missingpatches and those that are all up to date.

Windows SecurityOperation Center installation

You’ll need a Splunk installation to make use of WinSOC.I’ll assume you have some familiarity with Splunk and its installation. If not,ping me via russ at holisticinfosec dot org and I’ll send you copy of adetailed Splunk article I wrote for Admin magazine in June 2010. You can alsomake use of the extensive online Splunk documentation resources.

A panoply of Splunk application goodness is available onthe Splunkbase site, WSOC included. For the easiestinstallation method, from the Splunk UI, click App| Find More Apps…, then searchWindows Security Operations Center followed by clicking the Install Freebutton.

Alternatively if you’ve acquired the .tar.gz for the appyou can, again via the Slunk UI navigate to App| Manage Apps… | Install app from file and select theapp from the location you’ve downloaded it to. Installation is also possiblefrom the Splunk CLI.

Once installed WSOC will present itself from the Splunkmenu under App as WindowsSecurity Operations Center. Once you’ve navigated to the WSOC app, options willinclude:

· About

o Includestop sending servers, top source types, and contributing Domain Controllers (ifapplicable)

· Login Events

o IncludesActive Directory, NTLM, and RDP successful and failed attempts

· Directory services

o Accessand changes

· User management

o UserAccount and Group Management

· Change Control

o AdvancedActivity Monitor

o WindowsInstallations and Patch Status Overviews

o ProcessTracking

o TimeSynchronization

· Windows firewall

o Configurationchanges

o Allowedand blocked connections

o Allowedand blocked binds

· Saved Searches

o Preconfiguredqueries, too plentiful to list

· Search

o StandardSplunk search UI

You’ve got to remember to set your audit and loggingpolicies to be sure they capture the appropriate level of success and failure inorder to be properly indexed by Splunk from the security event log. Recognize the profound differences between Window Server 2003 and 2008 withspecial attention to Event IDs. WSOC is largely optimized for Windows 2008/7event types but can be tuned for older versions if you know how to manageSplunk app configurations and query parameters.

Remember too that you can configure Splunk as a lightforwarder (CLI only) on target Windows servers and send all events to a coreSplunk collector running WSOC, thus aggregating all events in one index and UI.Note the 500MB a day limitation on the free version of Splunk.

Using WindowsSecurity Operations Center

I ran WSOC through its paceson a Windows Server 2003 virtual machine image that I literally had not touchedin two years (prior snapshot: 9/11/09). With WSOC and Splunk installed Ipatched the VM and generated a number of different logon events via RDP andlocally. I also made changes to users and groups as well as updated browsers,Flash, and Java.

WSOC smartly reported on allrelated activity.

Under Change Control | Windows Installation Overview I notedall installations that wrote to the security event log (the default WSOCmonitored log source) as seen in Figure 1.

Figure 1:WSOC Windows installation details

As configured out of the box,if an event is not written to the security event log WSOC will not pick it up.As Bojan said, this app is intended as a security auditor’s tool as opposed toan operational health tool.

The default search covers thelast 7 days from query time but the chronology drop down menu offers a rangefrom 15 minutes to All time. Licensed versions of Splunk can also leveragereal time reporting.

Process Tracking is also great view to monitor oncritical servers. Unwelcome or unfamiliar processes may jump out at youparticularly if you’ve baselined normal expectations for your systems.

I am currently not runningActive Directory or a domain controller in my lab which left a lot of WSOCfunctionality testing off the table (Directory Services, etc.) but that shouldnot preclude you from doing so. Via Local Users and Groups I added an eviluser, deleted some users created during testing on the VM in 2009, and deleteda couple of non-essential groups. Evidence of the activity immediatelypresented itself via User management | UserAccount Management and GroupManagement as seen in Figure 2.

Figure 2: WSOC user account monitoring

It’s a tad unseemly for WSOCto label UI panes as Added WindowsDomain accounts and DeletedWindows Domain accounts given that the activity was local accountspecific, but you get the idea.

If you drill into View results you’ll receive all thedetail not immediately available in the preliminary app pane.

Figure 3 shows WSOC nabbingme for having created the user Ima, short for Ima Hacker. J

Figure 3: Ima Hacker bagged and tagged

I love the Saved Search feature and ran Windows – Server restarts for you asan example knowing I’d intentionally triggered one of those events.

Results are noted in Figure 4where you can see the fact that the reboot was spawned by Internet Explorer(Windows Update).

Figure 4: WSOC captures system restarts

Lastly, the Advanced Activity Monitor, under Change control, offers search capacityvia unique identifiers. In Figure 5, you’ll see all the New added services attributed to my user account.

Figure 5: WSOC shows added services

I did some customization ofthe app to capture Windows Server 2003 Windows Firewall-related events but beaware that by default the app checks events 4946, 4947, 4948, 5156, 5157, 5158,and 5159 (Windows Server 2008 Event IDs). Enable Audit MPSSVC Rule-Level PolicyChange onWindows 7 and 2008 for this to capture Window Firewall events correctly.Windows 2003 Event IDs are a different event code hierarchy that is not coveredby WSOC but is east enough to customize for if you’re still running 2003.

I imagine you can see thevalue in WSOC, particularly from an audit and awareness perspective. The nicething about Splunk apps is they can be enhanced and built upon with relativeease. Bojan and team also offer a supported, licensed version so that’s anoption for you as well.

In Conclusion

WSOC is slick, particularly for teams already making useof Splunk. Once (or if) you’re comfortable with Splunk, you’ll find that appssuch as WSOC and others make it invaluable for centralized, correlated data.

Again, if you want to read deeper dives into the power ofSplunk and apps, ping me via email if you have questions (russ atholisticinfosec dot org).

Cheers…until next month.

Acknowledgements

BojanZdrnja, project lead, INFIGO IS

Chris Siebenmann: Understanding Resident Set Size and the RSS problem on modern Unixes

Fri, 03 Feb 2012 07:11:21 +0000

Understanding Resident Set Size and the RSS problem on modern Unixes

On a modern Unix system with all sorts of memory sharing between processes, Resident Set Size is a hard thing to explain; I resorted to a very technical description in my entry on Linux memory stats. To actually understand RSS, let's back up and imagine a hypothetical old system that has no memory sharing between processes at all; each page of RAM is either free or in use by exactly one process.

(We'll ignore the RAM the operating system itself uses. In old Unixes, this was an acceptable simplification; memory was statically divided between memory used by the OS and memory used by user programs.)

In this system, processes acquire new pages of RAM by trying to access them and then either having them allocated or having them paged (back) in from disk. Meanwhile, the kernel is running around trying to free up memory, generally using some approximation of finding the least recently used page of RAM. How aggressively the operating system tries to reclaim pages depends on how much free memory it has; the less free memory, the faster the OS tries to grab pages back. In this environment, the resident set size of a process is how many pages of RAM it has. If the system is not thrashing, ie if there's enough memory to go around, a process's RSS is how much RAM it actually needs in order to work at its current pace.

(All of this is standard material from an operating system course.)

The problem of RSS on modern Unix systems is how to adopt this model to an environment where processes share significant amounts of memory with each other. In the face of a lot of sharing, what does it mean for a process to have a resident set size and how do you find the right pages to free up?

There are at least two approaches the kernel can take to reclaiming pages, which we can call the 'physical' and 'process' approaches. In the physical approach the kernel continues to scan over physical RAM to identify candidate pages to be freed up; when it finds one, it takes it away from all of the processes using it at once (this is the 'global' removal of my earlier entry). In the process approach the kernel scans each process more or less independently, finding candidate pages and removing them only from the process (a 'local' removal); only once a candidate page has been removed from all processes using it is it actually freed up.

(Scanning each 'process' is a simplification. Really the kernel scans each separate set of page tables; there are situations where multiple processes share a single set of page tables.)

The problem with the process approach is that the kernel can spend a great deal of time removing pages from processes when the pages will never actually be reclaimed for real. Imagine two processes with a shared memory area; one process uses it actively and one process only uses it slowly. The kernel can spend all the time it wants removing pages of the shared area from the less active process without ever actually getting any RAM back, because the active process is keeping all of those pages in RAM anyways.

So, why doesn't everyone use the physical approach? My understanding is that the problem with the physical approach is that it is often not necessarily a good fit for how the hardware manages virtual memory activity information. Per my earlier entry, every process mapping a shared page of RAM can have a different page table entry for it. To find out if the page of RAM has been accessed recently you may have to find and look at all of those PTEs (with locking), and do so for every page of physical RAM you look at.

My impression is that most current Unixes normally use per-process scanning, perhaps falling back on physical scanning if memory pressure gets sufficiently severe.

(I suspect and hope that virtual memory management in the face of shared pages have been studied academically, just as the older and simpler model of virtual memory has been, but I'm out of contact with OS research.)

Racker Hacker: Kerberos for haters

Fri, 03 Feb 2012 04:29:32 +0000

I'll be the first one to admit that Kerberos drives me a little insane. It's a requirement for two of the exams in Red Hat's RHCA certification track and I've been forced to learn it. It provides some pretty nice security features for large server environments. You get central single sign ons, encrypted authentication, and bidirectional validation. However, getting it configured can be a real pain due to some rather archaic commands and shells.

Here's Kerberos in a nutshell within a two-server environment: One server is a Kerberos key distribution center (KDC) and the other is a Kerberos client. The KDC has the list of users and their passwords. Consider a situation where a user tries to ssh into the Kerberos client:

sshd calls to pam to authenticate the user
pam calls to the KDC for a ticket granting ticket (TGT) to see if the user can authenticate
the KDC replies to the client with a TGT encrypted with the user's password
pam (on the client) tries to decrypt the TGT with the password that the user provided via ssh
if pam can decrypt the TGT, it knows the user is providing the right password

Now that the client has a a TGT for that user, it can ask for tickets to access other network services. What if the user who just logged in wants to access another Kerberized service in the environment?

client calls the KDC and asks for a ticket to grant access to the other service
KDC replies with two copies of the ticket:
- one copy is encrypted with the user's current TGT
- a second copy is encrypted with the password of the network service the user wants to access
the client can decrypt the ticket which was encrypted with the current TGT since it has the TGT already
client makes an authenticator by taking the decrypted ticket and encrypting it with a timestamp
client passes the authenticator and the second copy of the ticket it received from the KDC
the other network service decrypts the second copy of the ticket and verifies the password
the other network service uses the decrypted ticket to decrypt the authenticator it received from the client
if the timestamp looks good, the other network service allows the user access

Okay, that's confusing. Let's take it one step further. Enabling pre-authentication requires that clients send a request containing a timestamp encrypted with the user's password prior to asking for a TGT. Without this requirement, an attacker can ask for a TGT one time and then brute force the TGT offline. Pre-authentication forces the client to send a timestamped request encrypted with the user's password back to the KDC before they can ask for a KDC. This means the attacker is forced to try different passwords when encrypting the timestamp in the hopes that they'll get a TGT to work with eventually. One would hope that you have something configured on the KDC to set off an alarm for multiple failed pre-authentication attempts.

Oh, but we can totally kick it up another notch. What if an attacker is able to give a bad password to a client but they're also able to impersonate the KDC? They could reply to the TGT request (as the KDC) with a TGT encrypted with whichever password they choose and get access to the client system. Enabling mutual authentication stops this attack since it forces the client to ask the KDC for the client's own host principal password (this password is set when the client is configured to talk to the KDC). The attacker shouldn't have any clue what that password is and the attack will be thwarted.

By this point, you're either saying "Oh man, I don't ever want to do this." or "How do I set up Kerberos?". Stay tuned if you're in the second group. I'll have a dead simple (or as close to dead simple as one can get with Kerberos) how-to on the blog shortly.

In the meantime, here are a few links for extra Kerberos bedtime reading:

Kerberos for haters is a post from: Major Hayden's Racker Hacker blog.

Thanks for following the blog via the RSS feed. Please don't copy my posts or quote portions of them without attribution.

TechRepublic Network Administrator: Foxit Reader for PDFs in the enterprise

Thu, 02 Feb 2012 14:00:14 +0000

Derek Schauland discovered a PDF replacement program for Adobe Reader in his office. Here are his thoughts on the Foxit Enterprise Reader.

CiscoZine: How to monitor devices with Cacti

Thu, 02 Feb 2012 08:50:21 +0000

There are many ways to monitor devices: netflow, span port, switchport and so on. Today I will explain how to monitor bandwith, CPU, … of routers and switches using SNMP and Cacti. Simple Network Management Protocol (SNMP) is an “Internet-standard protocol for managing devices on IP networks. Devices that typically support SNMP include routers, switches, servers, workstations, printers, modem racks, and more.” It is used mostly in network management systems to monitor network-attached devices for conditions that warrant administrative attention. SNMP is a component of the Internet Protocol Suite as defined by the Internet Engineering Task Force (IETF). It consists [...]

Geek and Artist - Tech: WAT

Wed, 01 Feb 2012 23:32:02 +0000

I’ve been spreading this excellent talk by Gary Bernhardt around my co-workers and friends who universally love it. I’m also proud to say the “WAT” meme has (hopefully permanently) entered my team’s culture as we find it adequately sums up our feelings towards various bits of software that we either have to work with or maintain.

Half my team is away on business trips or sick leave at the moment so it was a relatively quiet day, which the rest of us spent squashing bugs. Unfortunately for us, most of our codebase is in Ruby and we have started cultivating a reasonable collection of Javascript in some of our web-oriented interfaces so you can imagine that Gary’s talk was particularly poignant.

A co-worker had submitted a review request this afternoon (yes, code reviews are awesome) and I was talking a brief look through it as I was not that familiar with this piece of code. He’d admirably refactored some quite nasty stuff but I was a bit perplexed as to some of the logic. An example might be similar to this:

def check_vm_state(vm, vmstate)
  if vmstate[vm][:host]
    return vmstate[vm][:host]
  end
  raise InvalidHypervisorError
end

OK, it wasn’t nearly as bad as this but you get the point. We’re meant to pass in some kind of hash which contains status information, pull out something relevant to the VM in question and return it; if not, throw an error. What threw me was the test for this which was clearly just some fudged parameters but I couldn’t figure out what was going on:

assert_equals some_valid_value check_vm_state(1,2)

Again, for your sanity’s sake it was a little more than this. I wondered how this could possibly work, and my co-worker did the same. “But clearly the tests are passing!” someone exclaimed. Let’s take a look:

1.8.7 :001 > 2[1]
 => 1
1.8.7 :002 > 2[1][:host]
 => 0

WAT.

Apparently the [] operator on a Fixnum will retrieve the binary digit value at that index. OK, not too unreasonable. But what about the index off the :host symbol?!? Well, easy – it is cast to its unique identifier which is an integer value – in this case 16473 but this changes every time you run Ruby. This bit index clearly can’t exist for the number 2 so the return value is 0. WAT.

Here’s another cool one. We had some awful code that was a bunch of if statements testing for equality to various things – which in most instances you would just replace with a case/switch statement and be done with it. So we tried, and failed (initially):

> foo = 'hello'
 => "hello"
> case foo
>   when 'hello'
>   puts 'hello'
>   when 'goodbye'
>   puts 'goodbye'
>   else
>     puts 'something else'
>   end
hello
 => nil

Absolutely no surprises there. Except in this case we’re more interested in the type of the object we are dealing with since our particular piece of code is making use of Ruby’s duck-typing to do some smart manipulation of various objects in similar ways. So now:

> foo.class
 => String
> case foo.class
>   when String
>   puts 'string'
>   when Fixnum
>   puts 'fixnum'
>   else
>     puts 'something else'
>   end
something else
 => nil

Er… WAT? Obviously we failed to take into account that the case statement calls the === method on the operand in each when statement and it behaves completely differently depending on whether it is used with an Object or a Module (which Class inherits from as do String, Fixnum etc). For Module it will only return true if the thing being compared to is an instance or descendent of the thing being operated on, whereas Objects just compare equality (and not identity).

I’m sure this behaviour is also overridden in other types to “make sense” under the circumstances, but unfortunately just serves to confuse by its inconsistency. I realise this represents something like a raised middle finger to dyed-in-the-wool Rubyists but it really isn’t following any principle of least surprise that I know about.

Anyway, after all of us said WAT about nine-thousand times I came to the conclusion that duck-typing is only cool if not every single basic type in the language responds to most method calls, usually in completely different and unexpected ways. If it swims like a duck, looks like a duck and quacks like a duck, it could be a duck. Or some kind of shapeshifting, organism impersonating cyborg warrior from the future intent on the destruction of our minds and all we hold dear to us.

On a slightly less inflammatory note, Ruby is actually still quite nice to use once you know all it’s little quirks. I went to my first Clojure meetup in Berlin tonight and was introduced to some pretty awesome concepts. I’m not sure I’m completely sold but it may actually be time to broaden my horizons somewhat but I guess I’m stuck with Ruby for a while yet

Anton Chuvakin - Security Warrior: Monthly Blog Round-Up – January 2012

anton@chuvakin.org (Anton Chuvakin) — Wed, 01 Feb 2012 22:37:25 +0000

Here is my next monthly "Security Warrior" blog round-up of top 5 popular posts/topics this month:

“On Free Log Management Tools” is a companion to the checklist below (updated version)
“Simple Log Review Checklist Released!” is often at the top – the checklist is still a very useful tool for many people
“Updated With Community Feedback SANS Top 7 Essential Log Reports DRAFT2”, “SANS Top 5 Essential Log Reports Update!” and their predecessor “Top5 SANS Log Reports Update DRAFT” also show up close to the top. IF YOU WANT TO VOLUNTEER TO FINISH THIS DOCUMENT- PLEASE EMAIL ME!
“On Choosing SIEM” is about the least wrong way of choosing a SIEM tool – as well as why the right way is so unpopular.
My classic PCI DSS log review series is last on my Top 5: “Complete PCI DSS Log Review Procedures.”

In addition, I’d like to draw your attention to a few posts from my Gartner blog:

Also see my past monthly and annual “Top Posts” – 2007, 2008, 2009, 2010, 2011.

Disclaimer: all this content was written before I joined Gartner on Aug 1, 2011 and is solely my personal view at the time of writing. For my current security blogging, go here.

Monthly Blog Round-Up – December 2011

HolisticInfoSec.org: 2011 Toolsmith Tool of the Year: OWASP ZAP

noreply@blogger.com (Russ McRee) — Wed, 01 Feb 2012 20:58:00 +0000

Congratulations to the OWASP ZAP team!
The Zed Attack Proxy is the 2011 Toolsmith Tool of the Year.
ZAP finished with 338 votes (36.5% of the total), slightly edging out Security Onion.
SO finished a strong second place with 328 votes (35.4%).
Volatility came in third with 152 (16.4%) and Armitage right on their heels in fourth with 148 votes (16%).

I am donating $50 to the OWASP ZAP project to honor this win.
I ask that those of you with the wherewithal and resources to do so please visit the project page and donate in any capacity you can.

Congratulations and thank you to all participants this year and I look forward to a strong 2012.

Chris Siebenmann: A ZFS pool scrub wish: suspending scrubs

Wed, 01 Feb 2012 16:38:21 +0000

A ZFS pool scrub wish: suspending scrubs

Like sensible people, we scrub our pools periodically in order to turn up latent problems. Because pool scrubs have a visible impact on responsiveness (at least in the lightly patched Solaris 10 update 8 that we're running), we only run scrubs on weekends (and only scrub one pool per fileserver). However, we've recently started running into problems where pool scrubs slow the fileservers down enough that backups have started failing.

The obvious way around this is to switch things to only doing scrubs when backups aren't running. Except there's a problem: we run backups every day, they run for a fairly long time every day, and some of our pools take up to fifteen hours to scrub. If we only scrub when backups aren't running, there just isn't a fifteen hour gap that our biggest pools need.

(It's possible that they would scrub somewhat faster if they never overlapped with backups, but that's only a vague possibility. And as the pools get more data, they'll take longer and longer to scrub.)

Which brings me to my wish: I wish you could suspend ZFS pool scrubs. Not stop them and start them again from the start, but just put one to sleep by telling the pool to remember where the scrub was but do no further scrub IO for now, then later resume the scrub from where it left off. This would allow us to do even big scrubs around the backups, and in fact we could schedule scrubs much more liberally than we do right now. For example, we might have a couple of hours in a weekday early morning after backups have finished that we could use to get some scrubbing in.

(I'd be perfectly happy if this was only an in-memory pause, so that if you rebooted your system or exported the pool you lost it and had to start from scratch. As an in-memory pause it ought to be relatively simple to implement.)

PS: I checked and this doesn't seem to be in Illumos, at least based on the current Illumos zpool manpage.

Racker Hacker: OpenStack bleeding-edge Python packages are now available

Wed, 01 Feb 2012 15:05:16 +0000

I sometimes enjoy living on the edge occasionally and that sometimes means I keep up with OpenStack changes commit by commit. If you're in the same boat as I am, you may save some time by using my repository of bleeding-edge Python packages from the OpenStack projects:

pypi.mhtx.net

Python packages are updated moments after the commit is merged into the repositories under OpenStack's github account.

Although the packages will contain the latest code available, rest assured that the code has passed an initial code review (by humans), unit tests, and varying levels of functional or integrated testing. There may still be a bug or two cropping up after that, so be aware of that as you utilize these packages.

The package versions utilize a standard format:

[package]-[version]-[git commit count]-[short commit hash]

If you need to check the git log up to that particular commit, just run git log:

git log [short commit hash]

Instructions for configuring pip or easy_install are provided within the repository.

In addition, the repository is accessible via IPv4 and IPv6.

OpenStack bleeding-edge Python packages are now available is a post from: Major Hayden's Racker Hacker blog.

Thanks for following the blog via the RSS feed. Please don't copy my posts or quote portions of them without attribution.

CiscoZine: January 2012: three Cisco vulnerabilities

Wed, 01 Feb 2012 13:19:21 +0000

The Cisco Product Security Incident Response Team (PSIRT) has published three important vulnerability advisories: Cisco IronPort Appliances Telnet Remote Code Execution Vulnerability Cisco IP Video Phone E20 Default Root Account Cisco Digital Media Manager Privilege Escalation Vulnerability Cisco IronPort Appliances Telnet Remote Code Execution Vulnerability Cisco IronPort Email Security Appliances (ESA) and Cisco IronPort Security Management Appliances (SMA) contain a vulnerability that may allow a remote, unauthenticated attacker to execute arbitrary code with elevated privileges. Vulnerable Products The following Cisco IronPort Email Security Appliances (ESA) and Cisco IronPort Security Management Appliances (SMA) are affected by this vulnerability: Cisco IronPort Email [...]

Racker Hacker: Create a local PyPi repository using only mod_rewrite

Wed, 01 Feb 2012 04:02:49 +0000

Regular users of Python's package tools like pip or easy_install are probably familiar with the PyPi repository. It's a one-stop-shop to learn more about available Python packages and get them installed on your server.

However, certain folks may find the need to host a local PyPi repository for their own packages. You may need it to store Python code which you don't plan to release publicly or you may need to add proprietary patches to upstream Python packages. Regardless of the reason to have it, a local PyPi repository is relatively easy to configure.

You'll need to start with a base directory for your PyPi repository. For this example, I chose /var/pypi. The directory structure should look something like this:

/var/pypi/simple/[package_name]/[package_tarball]

For a package like pip, you'd make a structure like this:

/var/pypi/simple/pip/pip-1.0.2.tar.gz

Once you have at least one package stored locally, it's time to configure apache. Here's a snippet from the virtual host I configured:

DocumentRoot /var/pypi/
ServerName pypi.example.com
 
Options +Indexes
 
RewriteEngine On
RewriteRule ^/robots.txt - [L]
RewriteRule ^/icons/.* - [L]
RewriteRule ^/index\..* - [L]
 
RewriteCond /var/pypi/$1 !-f
RewriteCond /var/pypi/$1 !-d
RewriteRule ^/(.*)/?$ http://pypi.python.org/$1 [R,L]

The last set of rewrite directives check to see if the request refers to an existing file or directory under your document root. If it does, your server will reply with a directory listing or with the actual file to download. If the directory or file doesn't exist, apache will send the client a redirection to the main PyPi site.

Reload your apache configuration to bring in your new changes. Let's try to download the pip tarball from our local server in the example I mentioned above:

$ curl -I http://pypi.example.com/simple/pip/
HTTP/1.1 200 OK
 
$ curl -I http://pypi.example.com/simple/pip/pip-1.0.2.tar.gz
HTTP/1.1 200 OK

I've obviously snipped a bit of the response above, but you can see that apache is responding with 200's since it has the directories and files that I was trying to retrieve via curl. Let's try to get something we don't have locally, like kombu:

$ curl -I http://pypi.example.com/simple/kombu/
HTTP/1.1 302 Found
Location: http://pypi.python.org/simple/kombu/

Our local PyPi repository doesn't have kombu so it will refer our Python tools over to the official PyPi repository to get the listing of available package versions for kombu.

Now we need to tell pip to use our local repository. Edit ~/.pip/pip.conf and add:

[global]
index-url = http://pypi.example.com/simple/

If you'd rather use easy_install, edit ~/.pydistutils.cfg and add:

[easy_install]
index_url = http://pypi.example.com/simple/

Once your tools are configured, try installing a package you have locally and try to install one that you know you won't have locally. You can add -v to pip install to watch it retrieve different URL's to get the packages it needs. If you spot any peculiar behavior or unexpected redirections, double-check your mod_rewrite rules in your apache configuration and check the spelling of your directories under your document root.

Create a local PyPi repository using only mod_rewrite is a post from: Major Hayden's Racker Hacker blog.

Thanks for following the blog via the RSS feed. Please don't copy my posts or quote portions of them without attribution.

Chris Siebenmann: The solution to the modern X font handling mystery

Wed, 01 Feb 2012 02:53:11 +0000

The solution to the modern X font handling mystery

I wrote last time about my attempts to work out just why xterm was rendering the same font differently on Ubuntu and Fedora. Thanks to comments from Adam Sampson and some additional digging, I now have an answer and some theories. As it happens, the answer illuminates yet more issues with modern X font handling.

In the modern Xft/FreeType/Fontconfig world, fonts are specified more or less as a font name and a size. With most programs that allow explicit specification of the font name you can augment the name with additional attributes, partly to modify the exact font that gets matched and partly to control how it's rendered. All of this is sort of covered in the fontconfig user documentation.

(An example could be 'DejaVu Sans Mono:style=bold:hintstyle=hintslight'. This shows both a modification of the font selection process and a rendering instruction. A similar sort of syntax can be used if you want to find, eg, all of the monospace fonts on the system.)

Fontconfig also has system-wide configuration files, found in /etc/fonts/conf.d/. In most packages that I'm familiar with, the global configuration is a default and explicit specification of things override them. However, this is not the case for fontconfig; at least for some settings, fontconfig's global settings silently override anything you specify explicitly. The only way to override these settings yourself is to have a $HOME/.fonts.conf file (and you can't unset the settings so that you can pick them on the fly, only set them to whatever personal global value you want).

You can probably guess the rest of the story. As spotted by Adam Sampson, Ubuntu's fontconfig package has a global config file that is explicitly forces hinting to be set to hintslight, while Fedora has no config file and is defaulting to hintfull. Because this is set in a global config file you can't override it on the xterm command line, which fooled me into thinking that this setting wasn't the culprit.

(You can include ':hintstyle=hint<whatever>' in a -fa argument all you want, but it is silently ignored.)

Overriding that (with a personal .fonts.conf file that forces hintfull hinting) got Ubuntu rendering to be almost the same as Fedora rendering. The remaining difference turns out to be due to the specific versions and compilation options of my version of FreeType. Interestingly, this is not just a small visual difference; at least under some circumstances the Ubuntu FreeType library renders DejaVu Sans Mono characters a pixel or so taller than my Fedora FreeType library does, meaning that an 80x50 xterm on Ubuntu is visibly taller than a Fedora 80x50 xterm. (They are both the same width.)

I don't know for sure why gnome-terminal, Firefox, and TK applications were unaffected by this, but my theory is that all of them use the Gnome preferences system. Gnome has its own preferences settings for how to render fonts and these appear to completely override fontconfig's views on the subject, so Gnome applications were using the 'right' hinting style for my tastes. I would have probably seen the same rendering of DejaVu Sans Mono in any other Gnome application that used it as the monospace font (a good example is probably gedit).

(Why this happened for some fonts and not for others presumably has to do with how the fonts were hinted, or maybe some fonts specify that they can only be hinted at some levels. I don't know if this means that the fonts that weren't affected are less hinted than DejaVu Sans Mono and so on, or just hinted differently.)

Security Monkey: Why I Love Routerpwn? Simplicity!

Tue, 31 Jan 2012 21:56:47 +0000

One HTML file. 138 web exploits. 3 unique generators. Remote exploits (some 0-day). Copied to nearly any device. Connect to any open WiFi network. Own the router. Profit. Behold, one of my favorite tools: routerpwn! Watch routerpwn's creator Pedro Jo...

Security Monkey: The F-BOMB: Backdoor Device For Under $100 That Drops From The Sky!

Tue, 31 Jan 2012 21:26:57 +0000

There is some insane stuff coming out of ShmooCon this year. Stuff that is so cool that I wish I could go back in time and use some of these devices during my investigative work. For example: The F-BOMB! It's a Falling/Ballistically Launched Object t...

TechRepublic IT Security: Infographic: Kim Dotcom and the Megaupload story

Tue, 31 Jan 2012 19:33:11 +0000

While Megaupload's attorney tries to negotiate a deal with the government to preserve users' legitimate hosted files, check out this snapshot of Kim Dotcom fun facts and figures.

TechRepublic Network Administrator: System Center 2012 licensing primer

Tue, 31 Jan 2012 18:42:38 +0000

Scott Lowe breaks down the changes in the licensing structure for Microsoft System Center 2012. Here are some of the basics.

TechRepublic IT Security: World IPv6 launch day set: Security pitfalls to look out for

Tue, 31 Jan 2012 14:00:15 +0000

Patrick Lambert warns that when the IPv6 launch date officially arrives this June 6, it will be prudent to watch out for some security gaps in the initial days. Here are some issues to think about.

TechRepublic Network Administrator: Storage vMotion virtual disk format options with vSphere 5

Tue, 31 Jan 2012 14:00:05 +0000

New disk options make Storage vMotion tasks give vSphere administrators more control over virtual machine disk formats. Rickatron shows how in this blog post.

Everything Sysadmin: Who to trust?

Tue, 31 Jan 2012 14:00:02 +0000

We are two people. The person that calmly makes plans and the person that executes them. The first person is calm and thoughtful and has the right amount of doubt to make sure a plan will work. The second person rushes to judgement and is full of hubris. "What was I thinking! I can do it more/better/differently." is what the second person says. The second person often forgets how much work went into the planning or the rationale for why things were set in a particular order.

If an outside knows of the plan, it can confuse things if the second person "optimizes" the plan leaving those other people out of the loop. The second person often thinks they're the only one that knows the plan, but often they are forgetting someone.

I've had to learn that if someone in my todo list is marked as being in a specific order, I should "trust the plan" and follow it... against the recommendation of that second person.

A friend of mine recently said her plan in the morning involved seeing her son off on a trip then getting 4 things done at home. What happened was she made a last-minute decision to drive him to the event personally, which meant a series of problems including some delays that prevented those 4 things from getting done.

Why didn't she listen to that person that, the night before, carefully constructed a good plan?

I do a lot of volunteer work and often we spend a lot of time working on a plan and later when executing the plan people will start to make changes. This brings up all the "second person" problems but at an even bigger scale. You'll often hear me saying, "Trust the process" over and over.

Once we were stuffing envelopes for a big mailing. It was a rather complicated project creating 3000 pieces to be mailed. Previously we had ended up in a situation where we ended up with 2000 properly stuffed, labeled, and stamped envelopes plus 1000 envelopes that just had stamps, and a different 1000 envelopes that were stuffed and had address labels stuck on them. We stuff the envelopes; only stuffed envelopes get labels, and only labeled envelopes get stamps. Three assembly lines, one that feeds the next. If you notice, the order also reflects the cost-of-replacement: stamps are expensive so you don't want to put them on until you know the envelope is otherwise prepared. When you run out of contents, no more stamps are consumed.

Sometimes the labeling process was the bottleneck and someone outside the planning process would "help" by labeling empty envelopes. They don't realize the potential problem they are causing, or the confusion.

Every morning I do my "5 minutes of planning". I look at my calendar then check my todo list for the day. I re-arrange my todo list, often pushing things around to be in priority order. I do this on the train so it is ready when I get to work. By the time I get working on stuff I've often forgotten the rationale for the order things are in, so I've had to train myself to "trust the process" and do the tasks in the order "the other me" proscribed.

Because if I don't do that, I end up spending the morning writing a blog post instead of working on my todo list. And that can disorient my entire day.

Standalone Sysadmin: Manhattan World Tour – 1 Night Only! Wednesday 2/1/2012!

Tue, 31 Jan 2012 12:27:46 +0000

Just a note…I’m back in NYC this week doing some contracting work. I’ve got Wednesday (tomorrow) night open for dinner, if anyone wants to hang out. I’m meeting some friends who are sysadmin-types at Brother Jimmy’s BBQ in Union Square at 7pm, so you’re welcome to come by! See you there!

Chris Siebenmann: Where is Oracle going with Solaris?

Tue, 31 Jan 2012 05:17:09 +0000

Where is Oracle going with Solaris?

(Disclaimer: rambling ahead.)

Once upon a time, back when Sun was still Sun, it was possible to kind of see what they thought the future market for Solaris was. Solaris wasn't Linux, but they could load it with attractive features (ZFS, DTrace, arguably Zones, etc) to make up for being not-Linux and then sell it for a relatively low price to hook the low end of the market. Arguably Sun skipped the bit where they upsold to more lucrative services later.

(In this view, the free Linux distributions serve as a valuable initial hook for higher end commercial Linuxes like Red Hat Enterprise. A small company is unlikely to buy RHEL right away; instead they can progressively move closer, first with Debian or Ubuntu, then with CentOS, and finally they start paying Red Hat when they get tired of the alternatives. Since very few people were going to jump from a Linux to Solaris, Solaris needed a similar entry-level hook.)

Then Oracle took over Solaris and now I don't understand how they see its future. The initial moves were straightforward: Oracle drastically raised prices and effectively drastically reduced hardware availability. Then of course they killed off other features that made Solaris attractive, like source availability. As far as I can see this took out the bottom end of the Solaris market entirely.

(It's hard to find current pricing for Solaris on non-Oracle hardware. The best I could find on Oracle's own website was $1k per core per year; it's not clear if you can get a better deal through either Dell or HP, which were at one point theoretically reselling Solaris on their own hardware. I couldn't configure a low-end 1U Dell server with Solaris, for what that's worth.)

One possible answer is that Oracle has no real plans for Solaris's future. In this view, they're treating it as a declining asset and milking it to get as much money as possible from those people who have to have Solaris. As the ranks of those people dwindle, Solaris itself will dwindle away with them. Eventually Oracle will politely sunset it and no one will really care. In this view, the relatively high prices for Solaris (and the outrageously high ones for non-Oracle hardware) are somewhat deliberately designed to discourage new customers; the last thing Oracle wants is for Solaris to actually get popular, because then Oracle would have to start spending real money on it.

Another possible answer is that Oracle thinks that Solaris has a viable future on big iron but not on low end hardware. I'm a professional skeptic about big iron in general, so I'm not well placed to evaluate how realistic this is. I think you can make a case that big iron customers are mostly insensitive to both the exact operating system (they care about the apps, which are often layered on top of a database to start with) and the licensing costs, but will value various (theoretical) Solaris virtues like resilience and inspectability with DTrace (especially if Oracle integrates DTrace support into their database products). On the other hand they do care about TCO (and there can be a lot of money involved in that TCO with big iron and Solaris licensing) and I'm not sure Oracle has a good sales pitch for Solaris against the relentless march of cheaper Linuxes.

(I'm not persuaded by the variant of this where Solaris is supposed to be the true home of Oracle's database software, because it requires customers to either like or be neutral to Solaris and its increased costs. If everyone wants to run Oracle on RHEL, it's hard to make Solaris Oracle's true home.)

All of this is mostly but not entirely academic to me, since it seems clear that we have too little money to interest Oracle. Still, I just can't stop wondering; there was a time when Solaris looked like it had a place in the general Unix future.

(You can argue that Solaris still does, in the form of Illumos and distributions using it. Especially as apparently a whole lot of the Sun technical people have left Oracle and settled at various other places that are working on Illumos; this makes Illumos the technical future of Solaris, and the technical future is the interesting one.)

PS: I would probably be better informed about the speculation on this if I actually followed Solaris news. I don't, because it seems very unlikely that anything Solaris news is going to affect us; Oracle would have to perform one of the world's most spectacular sudden reverses in order to be relevant to us again.

Chris Siebenmann: HTML is not a SGML dialect and never really has been

Mon, 30 Jan 2012 20:33:44 +0000

HTML is not a SGML dialect and never really has been

There is a persistent story that makes the rounds among the web specification world (for example, in this otherwise realistic article on XHTML) that HTML is a SGML dialect but web browsers persistently mishandle and mis-parse certain SGML features such as minimization. Although I have pandered to this belief before, it is false in practice and in reality.

HTML is really a documentation standard; the standard followed behind existing practice, not preceded it. In the very beginning, people just created browsers and a vague format that the browsers understood. This format was inspired by SGML, but it was never an SGML dialect and as such it never had various obscure SGML features. At some point, when people in the W3C were writing down the HTML standard of the time (or perhaps evolving it), they decided to 'fix' this obvious omission by writing into the new version of the HTML specification that it was a SGML dialect.

(Looking at the historical specifications via wikipedia, this appears to go as far back as HTML 2.0.)

You can guess what happened next. All of the browsers of the time promptly ignored this new bit of the standard, and pretty much every browser written since then has as well; none of them ever parsed HTML as SGML, supporting all of the little odd SGML features that that implies. HTML may be an SGML dialect as far as the W3 standards and their validator are concerned, but it is not in real life and anyone who writes HTML believing otherwise is going to have problems.

As you might expect, HTML5 very firmly puts a stake in this particular issue; the current spec draft says explicitly (emphasis mine):

For compatibility with existing content and prior specifications, this specification describes two authoring formats: one based on XML (referred to as the XHTML syntax), and one using a custom format inspired by SGML (referred to as the HTML syntax).

Perhaps someday all of the common HTML validators will be updated to understand HTML as it really is.

TechRepublic IT Security: Insidious insiders: Psychology provides clues in handling invisible threats

Mon, 30 Jan 2012 17:00:28 +0000

Dominic Vogel considers the insider threat risk in organizations and suggests that cross-departmental collaboration could help shore up data loss prevention methods.

TechRepublic IT Security: Repurposed software: Apps gone rogue

Mon, 30 Jan 2012 15:50:41 +0000

You come up with a dynamite app. Companies are thrilled with it. Next thing you know, your app is used to deny visitors access to a website. Michael Kassner considers repurposed software.

TechRepublic Network Administrator: Eight steps to restore an individual Exchange 2010 mailbox with System Center Data Protection Manager

Mon, 30 Jan 2012 14:00:11 +0000

Exchange admins often have to restore an individual mailbox rather than an entire database. John Joyner shows you eight steps to do this using PowerShell commands and Microsoft System Center DPM.

SysAdmin1138: Judicial rubber-hoses

Mon, 30 Jan 2012 08:29:53 +0000

The other day a Colorado court ordered a defendant to produce the unencrypted contents of their own laptop. This is what I called "rubber hose cryptography", and previously we've heard of efforts in the UK to compel decryption. It has now happened here, and not at the US border. Unlike the UK, this decryption demand in Colorado is not based on a law that specifically says that courts can demand this.

Wired article

The counter-argument is quite clearly the 5th amendment right guaranteeing the ability to not self-incriminate. If that decryption key only exists in your head, and disclosing it would incriminate you, then you don't have to yield the key.

This judge disagreed. I'm not a lawyer, so I can't tell what legal hairs were split to come to this decision. But the fact remains that this judgment stands. The only concession he appears to have made for the defendant is to preclude the prosecution from using the act of disclosure as a 'confession', but the data yielded by the disclosure is still admissible.

Chris Siebenmann: Dealing with Fitts' Law on widescreen displays

Mon, 30 Jan 2012 02:13:49 +0000

Dealing with Fitts' Law on widescreen displays

One of the usual sayings derived from Fitts' Law is that four of the five easiest locations to reach with the mouse are the four corners of the screen, because they require very little precision (the edges trap the mouse and guide it into the corner). Over the years I've made some modifications to my desktop environment to make better use of this principle. The most important one is how I use the top left corner; I have my taskbar equivalent arranged so that when an iconified terminal window gets output, I can just zoom my mouse to that corner and click in order to reveal the terminal window.

Zooming to a corner is a fast operation in most setups; it works fine on a single monitor, even a single widescreen monitor, and on a normal dual-monitor setup such as my work desktop. But recently (for reasons beyond the scope of this blog) my work setup got updated to dual widescreen monitors, which revealed two problems with my application of Fitts' Law in this environment.

The first problem is that the sheer number of side to side pixels in a pair of 1920x1200 LCD panels seems to be a bit too many to easily zoom a mouse across. My mouse pointer generally winds up in the middle of the right hand display; getting it to the top left corner of the left display was no longer anything like a little flick of the wrist. The second problem is that the top left corner was sufficiently physically far off to the side that it was no longer an easy casual action to glance at it to see if there was anything with new output that I needed to deiconify; I was less glancing off a bit and more peering off into the distance.

(I had my old dual displays relatively flat against each other, but I think that I probably need to move the new displays into a much more pronounced V shape.)

My current solution to this issue exploits Fitts' Law once again. The often-overlooked fifth easy to reach location is 'where the mouse is right now', or failing that 'some large area very near where the mouse is'. So I've created a new mouse button binding for my window manager; if the mouse is over the root window, hitting the left button with Shift+Control now de-iconifies the (alphabetically) first terminal window. My mouse is frequently parked over the root window and when it's not there's generally an exposed patch of the root window close to it.

(Technically the binding toggles the window's iconified state, which means that I can flip the first window back and forth from iconified to not. This is a great way to fidget.)

To deal with the 'too far to look' issue and to make things in my terminal windows taskbar easier to reach in general, I've repositioned it so that it's at the top left corner of my second (right) display; this puts it more or less in the center of my overall workspace and makes it easier to both reach and look at. I don't think this move away from a screen corner is a loss for Fitts' Law because everything except the first window already had to be targeted carefully.

Of course, now I just have to train myself out of a many years habit of reflexively looking and going to the top left of the left display. This shouldn't take too long, right?

(What I'd really like to do is duplicate my taskbar equivalent in the top left of both displays. Unfortunately this isn't possible right now with my window manager.)

PS: I experimented briefly with increasing the mouse acceleration (which would make everything effectively closer) but didn't like the effects it had on my ability to target things with the mouse in general; I kept overshooting and missing stuff. Possibly I would have acclimatized with time and I just gave up too soon.