Planet SysAdmin

Doug Maughan's CACM article & Roadmap for Cybersecurity Research

2010-01-31T15:04:03+00:00

UI fix freezes NYSE, affects 975 stocks

2010-01-31T15:04:03+00:00

False positives galore in SARs

2010-01-31T15:04:03+00:00

Problems starting a jail with ezjail

2010-01-31T15:03:41+00:00

Things change. We must evolve.

Integrating the Tape Library with an existing Bacula installation

2010-01-31T15:03:41+00:00

Adding a new tape drive isn't always simple

cssh - cluster ssh

2010-01-31T15:03:41+00:00

Sometimes you need more than one

System installation, or licenses for redistribution

2010-01-31T15:03:17+00:00

Part of my job is to prep Windows laptops for coworkers; it is my goal to provide laptops fully setup, this means among other things, I need to have Adobe Reader and Adobe Flash Player installed when they recieve the machine. Unfortunately Adobe wishes to make that a little difficult.

The license for Adobe Reader however does not permit me as IT support for an employee to distribute Adobe Reader on a machine I am providing to the employee. Adobe would want me to simply point the employee at the download site for Adobe Reader.

For me to distribute these products, I am supposed to agree to a different license agreement, one intended for distribution. This likely isn't a problem for most as most IT employees, as most never actually read license agreements. My company however has a clear policy that employees are not allowed to agree to contracts without the approval of the lawyers.

Considering that Adobe is interested in having Adobe Reader and Flash Player installed on as many machines as possible, why must they throw additional obstcles in my way?

Let's touch base, or Harassment

2010-01-31T15:03:17+00:00

I have spoken with many salespeople in the past week. In the past week, I have spoken with over a dozen salespeople (I sought pricing on business cellphone plans and a new photocopier). I have heard the phrase "I just wanted to touch base" many times. I have grown to have a great dislike of that phrase.

I do believe that every time I have heard "I just wanted to touch base" in the last seven days, it was prefaced by an interrupting phone call and followed by an annoying couple of minutes of a sales weasel not accepting my quite clear and simple statement of "Thank you for the quote, I am evaluating all of the options and will get back to you with questions, concerns, or with our decision." I wouldn't hold the phone call against them if I had not made it abundantly clear to each of these people that email is my preferred method of contact. Worse is that in most cases, it has taken more than a few such phone calls from each of them to get them to leave me alone.

So ignoring the sexual harassment angle of the phrase I am clearly starting to associate the phrase with annoyance.

Sysadmins Law 9, or Backups must be automated

2010-01-31T15:03:17+00:00

We have a reasonable system to perform backups of laptops that are regularly on our local network. For laptops that live most of their life connected to the office over the internet and through a VPN connection it is much harder. Instead of an automated backup, the user must initiate the backup themselves. Automated messages are sent out when it has been a week without backups. And I talk to them on the phone a few days after that if a backup hasn't been done.

He came into my office complaining that his laptop blue-screened on boot, and it had been crashing regularly for the few days before. Booting off other media quickly showed the disk was failing in a pretty spectacular way. When I inform the user of this, he says, "I guess I should have backed up like you told me to, eh?" Yeah, I guess so.

Which brings me to sysadmin law 9

Backups that are not automated are not done.

Thinking about syndication feeds and spoilers

2010-01-31T06:40:09+00:00

Thinking about syndication feeds and spoilers

DWiki has always had the ability to do the common blog thing of 'click here to see the rest of the entry'; when I put it in, I expected to use it for things like the detailed stats at the end of this entry. Because I am crazy that way, I built the feature so that it could apply on the main page (pages, really), in syndication feed entries, or both, depending on what options I turned on in any particular entry.

In practice, it turned out that I really don't like using cuts in syndication feed entries, for at least two reasons. First, syndication feed readers already have good ways to skip parts of entries and even whole entries (or at least they should), which makes cutting for volume mostly unnecessary. Second, partial entries are in annoying in general because they effectively force you out of your syndication feed reader and into your browser in order to read the full entry.

(In fact it turns out that I don't like cuts very much in general, so I barely use them even on the main pages.)

However, this does leave one case unhandled: spoilers. Places like the anime blogging community have come up with decent Javascript-based solutions for people who are reading your main site, but this is a complete non-starter in syndication feeds. In fact you can't even count on the old 'set the colour of the text to the background colour' trick, as modern syndication feed readers can strip styling as well.

My reluctant conclusion is that handling spoilers may well call for using a cut even in syndication feeds, with the annoyance of having to click off to read the entry being the lesser of two evils. The other approach is just to note that there will be spoilers at the start of an entry and count on people to use their feed reader's 'skip to next entry' feature.

(Spoilers are not generally relevant to WanderingThoughts, but they sometimes come up for me elsewhere.)

Two Dimensional Thinking and APT

2010-01-30T20:18:06+00:00

I expect many readers will recognize the image at left as representing part of the final space battle in Star Trek II: The Wrath of Khan. During this battle, Kirk and Spock realize Khan's tactics are limited. Khan is treating the battle like it is occuring on the open seas, not in space. Spock says:

He is intelligent, but not experienced. His pattern indicates two-dimensional thinking.

I though this quote could describe many of the advanced persistent threat critics, particularly those who claim "it's just espionage" or "there's nothing new about this." Consider this one last argument to change your mind. (Ha, like that will happen. For everyone else, this is how I arrive at my conclusions.)

I think the problem is APT critics are thinking in one or two dimensions at most, when really this issue has at least five. When you only consider one or two dimensions, of course the problem looks like nothing new. When you take a more complete look, it's new.

Offender. We know who the attacker is, and like many of you, I know this is not their first activity against foreign targets. I visited the country as an active duty Air Force intelligence officer in 1999. I got all the briefings, etc. etc. This is not the first time I've seen network activity from them. Wonderful.

Defender. We know the offender has targeted national governments and militaries, like any nation-state might. What's different about APT is the breadth of their target base. Some criticize the Mandiant report for saying:

The APT isn't just a government problem; it isn't just a defense contractor problem. The APT is everyone's problem. No target is too small, or too obscure, or too well-defended. No organization is too large, two well-known, or too vulnerable. It's not spy-versus-spy espionage. It's spy-versus-everyone.

The phrasing here may be misleading (i.e., APT is not attacking my dry cleaner) but the point is valid. Looking over the APT target list, the victims cover a broad sweep of organizations. This is certainly new.

Means. Let's talk espionage for a moment. Not everyone has the means to be a spy. You probably heard how effective the idiots who tried bugging Senator Landrieu's office were. With computer network exploitation (at the very least), those with sufficient knowledge and connectivity can operate at nearly the same level as a professional spy. You don't have to spend nearly as much time teaching tradecraft for CNE, compared to spycraft. You can often hire someone with private experience as a red teamer/pen tester and then just introduce them to your SOPs. Try hiring someone who has privately learned national-level spycraft.

Motive. Besides "offender," this is the second of the two dimensions that APT critics tend to fixate upon. Yes, bad people have tried to spy on other people for thousands of years. However, in some respects even this is new, because the offender has his hands in so many aspects of the victim's centers of power. APT doesn't only want military secrets; it wants diplomatic, AND economic, AND cultural, AND...

Opportunity. Connectivity creates opportunity in the digital realm. Again, contrast the digital world with the analog world of espionage. It takes a decent amount of work to prepare, insert, handle, and remove human spies. The digital equivalent is unfortunately still trivial in comparison.

To summarize, I think a lot of APT critics are focused on offender and motive, and ignore defender, means, and opportunity. When you expand beyond two-dimensional thinking, you'll see that APT is indeed new, without even considering technical aspects.

Example of Threat-Centric Security

2010-01-30T15:34:46+00:00

In my last post I mentioned the need to take threat-centric approaches to advanced persistent threat. No sooner than I had posted those thoughts do I read this:

Beijing 'strongly indignant' about U.S.-Taiwan arms sale

The Obama administration announced the sale Friday of $6 billion worth of Patriot anti-missile systems, helicopters, mine-sweeping ships and communications equipment to Taiwan in a long-expected move that sparked an angry protest from China.

In a strongly worded statement on Saturday, China's Defense Ministry suspended military exchanges with the United States and summoned the U.S. defense attache to lodge a "solemn protest" over the sale, the official Xinhua news agency reported.

"Considering the severe harm and odious effect of U.S. arms sales to Taiwan, the Chinese side has decided to suspend planned mutual military visits," Xinhua quoted the ministry as saying. The Foreign Ministry said China also would put sanctions on U.S. companies supplying the equipment.

It would have been interesting if the Obama administration had announced its arms sale in these terms:

"Considering the severe harm and odious effect of the advanced peristent threat, the American side has decided to sell the following arms to Taiwan."

It's time for the information security community to realize this problem is well outside our capability to really make a difference, without help from our governments.

You're Certified! Wait, Maybe Not...

2010-01-30T11:32:03+00:00

As both a holder and a co-author of the first CompTIA Security+ Certification, I was recently alarmed to find out that CompTIA had very quietly posted an update regarding the renewal policy of their "+" series certifications. "...effective January...

Mandiant M-Trends on APT

2010-01-30T09:13:05+00:00

If you want to read a concise yet informative and clue-backed report on advanced persistent threat, I recommend completing this form to receive the first Mandiant M-Trends report.

Mandiant occupies a unique position with respect to this problem because they are one of only two security service companies with substantial counter-APT consulting experience.

You may read blog posts and commentary from other security service providers who either 1) suddenly claim counter-APT expertise or 2) deride "APT" as just a marketing term, or FUD, or some other term to hide their inexperience with this problem. The fact remains that, when organizations meet in closed forums to do real work on this problem, the names and faces are fairly constant. They don't include those trying to make an APT "splash" or those pretending APT is not a real problem.

Mandiant finishes its report with the following statement:

[T]his is a war of attrition against an enemy with extensive resources. It is a long fight, one that never ends. You will never declare victory.

I can already hear the skeptics saying "It never ends, so you can keep paying Mandiant consulting fees!" or "It never ends, so you can keep upgrading security products!" You're wrong, but nothing I say will convince some of you. The fact of the matter is that until the threat is addressed at the nation-state to nation-state level, victim organizations will continue to remain victims. This is not a problem that is going to be solved by victims better defending themselves. The cost is simply too great to take a vulnerability-centric approach. We need a threat-centric approach, where those with the authority to apply pressure on the threat are allowed to do so, using a variety of instruments of national power. This is the unfortunate reality of the conflict in which we are now engaged.

I'm puzzled about DNS glue records in the modern world

2010-01-30T07:40:28+00:00

I'm puzzled about DNS glue records in the modern world

I recently wound up reading a message about upcoming behavior changes to the .com/.net/.edu nameservers (via Dawn Keenan). The short summary of that message is that as of March 1st, those nameservers will mostly stop returning information about out of zone glue records.

Up until I read the message, I had not realized that these nameservers were still carrying (and in fact returning) out of zone glue records and it was just that modern DNS systems were not paying attention to them; I had innocently thought that out of zone glue was entirely gone, due to the era of glue record hell being long over. In fact, it turns out that they were doing so in a way that let people create circular loops of nameservers, which are going to break on March 1st.

(A longer explanation of the situation is in the message; see there for the details, which make my head hurt.)

Some experimentation shows that the out of zone glue records do not have to be from in-zone glue records for their own domain. As an example, suppose you have A.com, with NS records n1.a.com and n2.a.com, and B.com, with NS records n1.a.com and n3.a.com. If you do an NS lookup for B.com at the .com nameservers, you'll get back IP addresses for both n1.a.com, which is in-zone glue for A.com and thus necessary to carry in the .com zone, and n3.a.com, which is not. This means that something in the whole nameserver and zone management process knows or is looking up n3.a.com's IP address, and is adding it to the .com DNS information.

All of this leaves me puzzled about what the rules are for out of zone glue records in the modern world, although it probably varies by DNS zone (and maybe even registrar).

(The client-side resolver rules are clearer; apparently pretty much you always ignore any out of zone A records that you get back.)

See you at MacWorld!

2010-01-30T06:10:15+00:00

No such table: django_admin_log

2010-01-29T21:44:16+00:00

If you get this error when trying to save something in the Django admin, it’s probably because you forgot to synchronize the database after adding the admin application. If you are using a pre-1.2 version of Django, you can simply:

$ ./managy.py syncdb

Django 1.2 (which is currently in alpha) adds multi-database support. If you want to synchronize a specific database, you can specify that on the command line:

$ ./managy.py syncdb --database=default

links for 2010-01-29

2010-01-29T19:00:46+00:00

HURT FEELINGS REPORT

"To assist whiners in documenting hurt feelings, and to provide leaders with a list of soldiers who require additional counseling,
NCO leadership, and extra duty.."

Home Grown / Open Source vs Appliances

2010-01-29T17:52:12+00:00

The last few days I have been having a pretty good debate with a friend about the virtues of open source vs Appliances. At times its gotten pretty heated but its all in good fun. The current debate centers around email infrastructure. There are options on the table to use an appliance, or a 3rd [...]

Teaching Java How to Commit Suicide

2010-01-29T17:39:00+00:00

At $work, we have a lot of java processes that are ran via cron and other wrappers that do some pretty critical tasks. The apps have been written so that the whole thing is wrapped in a try/catch that will call system.exit(1) should something not go right. Our wrapper scripts watch for a non-zero exit code, and alert Nagios if something went wrong.

This works great except for when a VM encounters an outOfMemory exception (OOM). The Java VM attempts to continue on, but if the main thread hits this exception, the entire VM will exit. However, the application code that exits with a status of 1 never gets called, so the application ends up dying with a status of 0. Well, Sun (Oracle now I guess) gave us a new option in Java 6 that was backported to 1.4.2_12 and up that allows us to tell Java to run a shell command when it encounters an OOM exception. By adding the option

-XX:OnOutOfMemoryError="kill -9 %p"

to our Java command line, the VM will execute a shell that calls the kill command, with an argument of the PID of the VM. The -9 option to kill will cause the VM to exit with a non-zero status, so that our wrappers will pick up the error and alert the right people.

Note: this feature was never backported to Java5 - sorry!

Planet Network Management Highlights 2010 Week 4

2010-01-29T15:50:36+00:00

Highlights from Planet Network Management + Planet Sys Admin for Week 4.

Orion Additional Polling Engine and Modules — What do I buy for modules…Nothing because it’s now free! – Solarwinds start giving away extra polling engines…
Red Hat Launches opensource.com – Tarus Balog ponders what Red Hat will do with the opensource.com web site
CACE Technologies Launches SDMS Product Family – CACE announce the release of the SDMS product family which delivers a complete, end-to-end, enterprise-wide solution for line-rate network recording, monitoring, and analysis.
WhatsUp Gold Acquires Dorian Software Inc. – Ipswitch announce the purchase of Dorian Software, Inc makers of log file analysis and consolidation products.
New Nessus Videos – Scanning With Credentials
Wireshark 1.2.6 and 1.0.11 Released
New SolarWinds Free Tool – Network Config Generator
Release 6.1 – now with 105% more customer goodness
Network Janitor – Wrapping it Up (by Tony Fortunato)
Nagios RPMs bundled with Novell’s SLES

IPv6-capable devices: Make sure they are ready

2010-01-29T14:00:34+00:00

You need a new router, but your budget is in shambles. On top of that, someone tells you IPv4 addresses are running out. Great, what’s the smart thing to do?

—————————————————————————————

Joe Klein, an IPv6 consultant has been mentoring me for years. Recently, he helped with my article about IPv4 addresses running out. During one of our conversations, he suggested I promote replacing IPv4 networking equipment with IPv4/IPv6 ready equipment as the need arose. That way, costly re-buys will not be required in a year or two. Seems simple enough, so why write about it?

Little did I know. My mistake was thinking that it’s either IPv6-ready or not. It seems there are varying degrees of readiness and interoperability amongst manufacturers, and that’s a problem.

What I’ve learned

You may be familiar with the Wi-Fi Alliance and its ability to get wireless-networking companies to focus on standardization and interoperability. Thankfully, there are groups doing the same thing with IPv6 equipment.

IPv6 Ready Logo Program

The IPv6 Forum has a service called IPv6 Ready Logo. It’s a qualification program that assures devices they test are IPv6 capable. It reminds me of the Wi-Fi Alliance. I say that because once certified, they allow qualified products to display their logo. The IPv6 Forum objectives are to:

Verify protocol implementation and validate interoperability of IPv6 products.

Provide access to free self-testing tools.

Provide IPv6 Ready Logo testing laboratories across the globe dedicated to provide testing assistance or services.

IPv6 experts I talked to, suggest only paying attention to devices given the Phase-2 approval (gold logo). That makes sense, as they are given the full treatment:

“The Phase 2 Logo expands the “core IPv6 protocols” test coverage to approximately 450 tests and adds new extended test categories. The Logo background color is Gold. The Phase 2 Logo has been available since February 16, 2005.”

This link will take you to their approved list. I wasn’t familiar with this organization, which concerned me. So, I asked Joe Klein what he thought:

“It is a good program aimed at the private sector. Actual testing in the U.S. is performed at the University of New Hampshire, a pioneer in IPv6 testing.”

That’s one resource. The Department of Defense (DoD) is committed to IPv6 and will likely be the first federal organization completely converted to IPv6. They also have a process for qualifying IPv6 equipment.

JITC/DISA

The task of certifying IPv6 products was given to the Joint Interoperability Test Command (JITC), part of the Defense Information Systems Agency (DISA). To help standardize IPv6 qualification procedures, the JITC follows what’s called the IPv6 Generic Test Plan. The PDF is 216 pages long, so I thought I’d summarize. First, the devices to be tested:

“The source requirement document, DoD IPv6 Standard Profiles for IPv6 Capable Products, identifies six product classes for IPv6 network devices: Host/Workstation, Network Appliance/Simple Server, Advanced Server, Router, Layer-3 Switch, and Information Assurance Device.”

Next, the procedure for checking compliance with IPv6 RFCs:

“Conformance testing will consist of automated test equipment that provides controlled data inputs to elicit a response from a device under test and evaluate that response in accordance with the requirements in the corresponding IPv6 Request for Comment.”

Finally interoperability between devices is tested by placing the equipment in a network that simulates the DoD network:

“Data traffic will be generated and transmitted across the network to assess the device’s capability to effectively pass IPv6 traffic and perform other IPv6-related functions in a realistic operational environment.”

After JITC qualifies a product, it is added to the Unified Capabilities Approved Products List. Fortunately, JITC makes the list available to the public. I once again asked Joe Klein what he thought about the DoD process, here is what he said:

“The DoD takes the chance of buying the wrong product very seriously. As of June 2010, all new networking products must pass testing. This is a lesson for any business/tech person. Require that the equipment you purchase is tested by one of the above programs, to mitigate the risk of a product that does not support IPv6.”

Sounds like good advice.

Final thoughts

In the process of researching this article, one thing stood out. Saying a device is IPv6-capable can have multiple meanings. So, make sure the device has been certified by an independent source.

MySQL: The total number of locks exceeds the lock table size

2010-01-29T13:12:21+00:00

This problem has cropped up for me a few times, but I’ve always forgotten to make a post about it. If you’re working with a large InnoDB table and you’re updating, inserting, or deleting a large volume of rows, you may stumble upon this error:

ERROR 1206 (HY000): The total number of locks exceeds the lock table size

InnoDB stores its lock tables in the main buffer pool. This means that the number of locks you can have at the same time is limited by the innodb_buffer_pool_size variable that was set when MySQL was started. By default, MySQL leaves this at 8MB, which is pretty useless if you’re doing anything with InnoDB on your server.

Luckily, the fix for this issue is very easy: adjust innodb_buffer_pool_size to a more reasonable value. However, that fix does require a restart of the MySQL daemon. There’s simply no way to adjust this variable on the fly (with the current stable MySQL versions as of this post’s writing).

Before you adjust the variable, make sure that your server can handle the additional memory usage. The innodb_buffer_pool_size variable is a server wide variable, not a per-thread variable, so it’s shared between all of the connections to the MySQL server (like the query cache). If you set it to something like 1GB, MySQL won’t use all of that up front. As MySQL finds more things to put in the buffer, the memory usage will gradually increase until it reaches 1GB. At that point, the oldest and least used data begins to get pruned when new data needs to be present.

So, you need a workaround without a MySQL restart?

If you’re in a pinch, and you need a workaround, break up your statements into chunks. If you need to delete a million rows, try deleting 5-10% of those rows per transaction. This may allow you to sneak under the lock table size limitations and clear out some data without restarting MySQL.

To learn more about InnoDB’s parameters, visit the MySQL documentation.

Links for 2010-01-28 [del.icio.us]

2010-01-29T08:00:00+00:00

Power Policy Configuration and Deployment in Windows

A theory about Apple's new iPad

2010-01-29T07:59:33+00:00

A theory about Apple's new iPad

Like a lot of other people, I'm not very interested in the iPad myself for all of the obvious reasons for an open source person who likes doing random things to his computers (see Tim Bray for a representative example). But I have a theory about what Apple is up to here, and it goes like this:

The iPad is a computer for people who do not like computers, not a computer for people who like computers.

The problem with making computers for people who like computers is that it is more and more a limited market with little potential for real growth. Most people who like computers already have one (or several), and these computers are generally pretty adequate ones. Without major technology improvements to obsolete existing hardware on a regular basis, you are are down to getting your sales from a moderate stream of new people, people replacing worn out machines, and whatever market share you can steal from your competitors (who are all trying to steal your market share in turn).

(In short, selling computers to people who like computers has become a mature market. Mature markets are boring and unspectacular, and companies in mature markets don't grow much.)

Selling computers to people who do not like computers is much easier; either they don't have a computer yet or they don't much like the one that they have. This is a growth market, potentially a very large one, provided that you actually have a computer that these people will like. Which is where the iPad's restrictions come into the picture.

People who do not actively like computers do not care about a lot of the computer stuff; they just want the computer to do things for them. All of the fiddling around that is necessary (even on a Mac) to get the computer to do things and to keep it doing those things is an annoyance to these people, and if you want to sell to them you need to make as much of it vanish as possible. Apple doesn't intrinsically need a closed and controlled box to do this, but it does need something that just works, all the time, and getting that is less effort with a closed box than with an open one (and it's in Apple's inclinations anyways). And Apple is very good about making the magic work.

Netbooks made vague attempts at this market, but they failed to be sufficiently appealing, ie sufficiently different from the same old computer that these people don't like. Apple is not making that mistake; it has targeted this market as the iPad's primary market, so the iPad's limitations are entirely deliberate and consciously thought out. And Apple is doing this in order to tap into another major market and that market's explosive growth potential, just as it did with the iPod and the iPhone.

(I'm not the only person thinking along these lines; see here, here, John Gruber, and here, for a random sampling pulled from Hacker News's front pages. Also, my thinking about this owes a debt to Thom Hogan's writing on dpreview about what camera companies will need to do in order to keep growing despite the DSLR market maturing and flattening.)

Sidebar: on some iPad restrictions

Lack of a physical keyboard probably isn't going to be much of a drawback for the 'don't like computers' market, because I suspect that they don't spend all that much time typing away on a personal computer, or even interacting with it (which will help avoid the well studied fatigue effects of long term touchscreen usage). Similarly, not being able to run multiple applications at once sounds awfully like a feature, not a flaw, since it avoids all sorts of confusions and annoyances and likely mimics how these people already prefer to use computers (especially ones with relatively small screens).

The appeal of a single, well designed and simple place and process to get applications that just work should hopefully be obvious. The applications may be not worth your money, but people waste money all the time; what they won't be is dangerous to the overall experience.

(Think of it like iTunes. What you get may turn out to be bad music, but the experience of doing it is pretty decent and the music will always actually play. This is a lot different from the experience of getting either digital music or software in a more open environment.)

(3 comments.)

Sysadmins aren’t (necessarily) programmers, they’re people who can program

2010-01-29T04:07:44+00:00

I remember back when I was first learning Linux, and I encountered shell scripts. I wasn’t a programmer, and I didn’t “get” it. I distinctly remember thinking, “well, THAT’S something I won’t have to learn”. Ha!

As it turns out, I was incorrect. Writing scripts is an essential skill for a system administrator. In Linux/Unix, we’re blessed to have an amazing development environment, where as administrators running on Windows had to make due with batch files until the dot net revolution came along to introduce ASP and VB script. Now, with powershell, they’ve actually got a great environment to write systems scripts in. Between that and things like Windows Server Core, I’m beginning to wonder about the Redmond camp. But I’m digressing…

Writing scripts isn’t an optional tool in an effective administrator’s tool belt. It’s absolutely vital to efficiently performing many, many tasks. Personally, I use the bash shell, because it’s the default, and it’s what I started on. You should use whatever you feel comfortable with, whether it’s a shell script or perl. Heck, I’ve been desperate enough to even do a couple of things in CLI-mode PHP, just because I’m more fluent in it than I am perl (which is a shame, and something I’m going to be working on rectifying).

My last “fun” bit of shell programming was probably a cron job that checked for a new tsunami warning and emailed me the text of the alert. Before that, I wrote an entire RSS reader in bash. With bookmark support. Yeah, I’m a sick man.

What kind of fun things have you done with scripts lately?

Google exploit update; what no Flash?

2010-01-28T23:09:59+00:00

Microsoft has published a critical security update, MS10-002, as of Thursday, 21 January. They rushed out this patch that covers seven different IE vulnerabilties, even as attackers were working on converting the Google exploit so it succeeds against later versions of IE. If you thought there was only one IE vulnerability to patch, you might be wondering why seven get patched [...]

USENIX TaPP ‘10 Program Available

2010-01-28T19:45:21+00:00

Join us February 22, 2010, in San Jose, CA, for the 2nd USENIX Workshop on the Theory and Practice of Provenance (TaPP ‘10). The TaPP workshop series builds upon a set of Workshops on Principles of Provenance organized in 2007–2009, which helped raise the profile of this area within diverse research communities, such as databases, security, [...]

LISA ‘10 Call for Participation Now Available

2010-01-28T19:35:42+00:00

The Call for Participation for the 24th Large Installation System Administration Conference (LISA ‘10) is now available. Participation opportunities include refereed papers, invited talks, and more. The annual LISA conference is the meeting place of choice for system and network administrators and engineers. The conference serves as a venue for a lively, diverse, and rich mix [...]

Convert milliarcseconds to degrees

2010-01-28T19:14:08+00:00

I was recently working with a GPS device that returned coordinates in milliarcseconds (mas). Unfortunately, not all mapping applications (Google Maps, for example) can understand milliarcseconds. Therefore, I had to convert milliarcseconds to degrees.

The key to the conversion is understanding that one degree is equivalent to 60 arcminutes (arcmin or MOA). Therefore, one degree has 3,600 arcseconds (arcsec) or 3,600,000 mas.

So, to convert millarcseconds to degrees, simply divide by 3,600,000.

What should you write your LISA paper about?

2010-01-28T18:37:18+00:00

Status: draft

The Usenix LISA 2010 "Call for Participation" is out. I encourage everyone to think about what they're doing to improve system administration, what innovation they've brought to their network, and write a paper about it.

People often ask me for a definition of "system administrator". TPOSANA/2ed has a great definition in the preface (read it and see).

But lately I've been thinking that one way to define it is in terms of lifecycle. I think of sysadmins as being the people that are responsible for technology from cradle-to-grave. Developers might create it. Executives might pay for it. Customers might request it. But we facilitate the cradle-to-grave process:

acquisition
deployment
maintenance
repair
scaling
decommission

(Hmm... am I missing a phase?)

Each of these phases can be a big deal and can be done well or badly.

Most sysadmins are mired in the operational aspects (keeping it running) and don't even realize the other phases exist. Consider the (usually Windows) desktop support person at most companies: the OS comes loaded from the vendor, is never changed, just maintained until the machine is thrown away. We (they) get a reputation for being janitors, not engineers.

The biggest innovations come from focusing on the other parts: getting customer requirements as part of acquisition; deploying things very well (especially desktops or things that have opportunity for mass-production cloning, etc.), scaling (growing from 100 machines , 10,000 machines, 1 million machines; or from hundreds to thousands to millions of users), decommissioning: securely destroying information, and so on.

You can optimize these phases individually or look for cross-phase improvements. If you have optimized all of these phases the only thing left is release engineering and scaling.

Surrounding these issues is all the soft skills that relate to "professionalism": how you deal with people, manage your time, etc.

So what are you going to write your LISA paper about? Which one of these phases have you improved? Did you write a new tool? Develop a new technique? Start an open-source project? Or maybe you have perfected a cross-phase methodology?

Alternatively, maybe you've noticed that there is a thing that sysadmins do, and you can study many groups of sysadmins doing that thing, and can draw comparisons and conclusions about what worked best.

Or maybe you're facing a problem that nobody else has faced, and it is worthwhile to publish your results. Maybe you've scaled Apache to more users than anyone else has, and learned something useful. Maybe you've broken from tradition "on a hunch" and your new disk backup software works better for a particularly common edge-case (Just kidding... I'm sick of papers about backup systems.) Heck, maybe you're just the first sysadmin to deploy a new technology (10G ethernet to the desktop?) and learned something about managing it that the engineers that invented it would have never expected.

What is your sysadmin team most proud of? What thing are you doing that others think is "futuristic" or "cutting edge"?

William Gibson famously wrote, "The future is here. It's just not evenly distributed yet." I suspect that most people reading this blog live in the future and by writing or speaking about what we're doing, we can spread it to others.

That's the goal of LISA, isn't it?

Read the "Call for Participation" here.

Evolving best-practice

2010-01-28T18:21:00+00:00

As of this morning, everyone's home-directory is now on the Microsoft cluster. The next Herculean task is to sort out the shared volume. And this, this is the point where past-practice runs smack into both best-practice, and common-practice.

You see, since we've been a NetWare shop since, uh, I don't know when, we have certain habits ingrained into our thinking. I've already commented on some of it, but that thinking will haunt us for some time to come.

The first item I've touched on already, and that's how you set permissions at the top of a share/volume. In the Land of NetWare, practically no one has any rights to the very top level of the volume. This runs contrary to both Microsoft and Posix/Unix ways of doing it, since both environments require a user to have at least read rights to that top level for anything to work at all. NetWare got around this problem by creating traverse rights based on rights granted lower down the directory structure. Therefore, giving a right 4 directories deep gave an inplicit 'read' to the top of the volume. Microsoft and Posix both don't do this weirdo 'implicit' thing.

The second item is the fact that Microsoft Windows allows you to declare a share pretty much anywhere, and NetWare was limited to the 'share' being the volume. This changed a bit when Novell introduced CIFS to NetWare, as they introduced the ability to declare a share anywhere; however, NCP networking still required root-of-volume only. At the same time, Novell also allowed the 'map root' to pretend there is a share anywhere but it isn't conceptually the same. The side-effect of being able to declare a share anywhere is that if you're not careful, Windows networks have share-proliferation to a very great extent.

In our case, past-practice has been to restrict who gets access to top-level directories, greatly limit who can create top-level directories, and generally grow more permissive/specific rights-wise the deeper you get in a directory tree. Top level is zilch, first tier of directories is probably read-only, second tier is read/write. Also, we have one (1) shared volume upon which everyone resides for ease of sharing.

Now, common-practice among Microsoft networks is something I'm not that familiar with. What I do know is that shares proliferate, and many, perhaps most, networks have the shares as the logical equivalent of what we use top-level directories for. Where we may have a structure like this, \\cluster-facshare\facshare\HumRes, Microsoft networks tend to develop structures like \\cluster-facshare\humres instead. Microsoft networks rely a lot on browsing to find resources. It is common for people to browse to \\cluster-facshare\ and look at the list of shares to get what they want. We don't do that.

One thing that really gets in the way of this model is Apple OSX. You see, the Samba version on OSX machines can't browse cluster-shares. If we had 'real' servers instead of virtual servers this sort of browse-to-the-resource trick would work. But since we have a non-trivial amount of Macs all over the place, we have to pay attention to the fact that all a Mac sees when they browse to \\cluster-facshare\ is a whole lot of nothing. We're already running into this, and we only have our user-directories migrated so far. We have to train our Mac users to enter the share as well. For this reason, we really need to stick to the top-level-directory model as much as possible, instead of the more commonly encountered MS-model of shares. Maybe a future Mac-Samba version will fix this. But 10.6 hasn't fixed it, so we're stuck for another year or two. Or maybe until Apple shoves Samba 4 into OSX.

Since we're on a fundamentally new architecture, and can't use common-practice, our sense of best-practice is still evolving. We come up with ideas. We're trying them out. Time will tell just how far up our heads are up our butts, since we can't tell from here just yet. So far we're making extensive use of advanced NTFS permissions (those permissions beyond just read, modify, full-control) in order to do what we need to do. Since this is a deviation from how the Windows industry does things, it is pretty easy for someone who is not completely familiar with how we do things to mess things up out of ignorance. We're doing it this way due to past-practice and all those Macs.

In 10 years I'm pretty sure we'll look a lot more like a classic Windows network than we do now. 10 years is long enough for even end-users to change how they think, and is long enough for industry-practice to erode our sense of specialness more into a compliant shape.

In the mean time, as the phone ringing off the hook today foretold, there is a LOT of learning, decision-making, and mind-changing to go through.

What I want in a mobile task management software

2010-01-28T16:11:12+00:00

The biggest impediment to recording a todo item is that it is inconvenient. I use that excuse to tell myself, "oh, i'll write it down later". Later never comes.

The fewer clicks to the "add a task" prompt, the less likely I can give myself that excuse.

90% of time management is mental.

That's why I recommend paper (no boot-up time), and PDAs like the original Palm that make it very fast (minimal clicks) to write down an idea.

A related excuse happens when I'm in the NYC subway. With no internet connectivity (2G, 3G, or WiFi), any great idea I have on the subway is destined to be not recorded, and often forgotten, if the app I'm using requires the network.

What would be optimal? A "record a task" button right on the phone. You would press-and-hold the button, it would wake up and say "Recording". You would then say your task and use speech-to-text technology to transcribe the idea. If the speech-to-text server isn't reachable, it should hold the audio clip until it can be reached; possibly doing the translation in the background.

The on-screen or physical keyboard should be available too, of course, but what I really want is a super smart, voice activated, task recorder.

What is Cat5e?

2010-01-28T14:37:13+00:00

Everybody knows, if you’re still dealing with Cat-5 to use Cat-5e. I mean, Cat-6 would be better, but it’s still really expensive to have installed. So we use Cat-5, except it’s ENHANCED for that extra special protection against interference and crosstalk. But what is it? What, if any, is the actual, real, honest-to-God, difference between Cat-5 and Cat-5e? I’ve read the wikipedia entry, and while informative, it could really do with a rewrite. One of the few sections that mentions Cat-5e is The Cat 5e 350mhz debacle. The title makes me believe that maybe the 350mhz denotation is how you can tell, but then it goes and says this:

“Although the performance of this new 350 MHz cable was slightly better it was an easy way to sell the consumer on future proofing their needs while charging around 15% more and leading to a higher margin on the 350 MHz cable than the standard 5e cable.”

Well, alright then, what IS a standard 5e cable?

I looked at some more resources, and found some sadly incorrect suggestions and no real answers. For instance, this wiki answers post says that you need Cat-5e to get Gb/s speeds, that 5e is rated at 350mhz, and that there’s less crosstalk (though it doesn’t say why). Of course, it also says “Unless every single component in the network is gigabit rated, then you will never have a gigabit network, because your network will always run at the speed of your slowest device.”, so you can understand my skepticism.

The second table on this page sounds pretty solid, basically displaying that there are standards for Cat5E that don’t exist for Cat5, with the assumption that the cable meeting those standards will perform better/faster than a cable without those particular requirements.

But of course, I wanted to ask, because you can’t trust everything you read on the internet. Case in point? connectworld.net, who either desperately need to update their page, or desperately need to stop taking drugs:

“The Simple Answer:
CAT-5 is rated to 100M
CAT-5e is rated to 350M
CAT-6 and CAT6e is rated to 550M or 1000M depending on your source
CAT-7 is supposedly rated to 700M or presumably 1000M

Today there is no approved CAT-6 or CAT-7. While some folks are selling products they call Level 6 or 7, there aren’t even specs for them, making CAT-5e the best available option.”

Sad, really. I blame the schools.

So now I ask you. I assume that there is an actual, real difference between Cat-5 and Cat-5e. what is it?

It appears that Cat-5 cable is defined by the TIA/EIA-568-A standard. The only time I’ve ever paid attention to that standard was when crimping my own ends, because TIA/EIA-568-A specifies the order of the wires in the RJ45 (really 8P8C) end. It also uses the green pair first, which I thought was weird, since I learned orange first.

However, when I go to the TIA/EIA-568-A wiki page, I’m redirected to the TIA/EIA-568-B page. At a table way down at the bottom are the lines:

Cat 5: Currently unrecognized by TIA/EIA. Defined up to 100 MHz, and was frequently used on 100 Mbit/s Ethernet networks. May be unsuitable for 1000BASE-T gigabit ethernet.

Cat 5e: Currently defined in TIA/EIA-568-B. Defined up to 100 MHz, and is frequently used for both 100 Mbit/s and 1000BASE-T Gigabit Ethernet networks.

Fascinating!

I still don’t know what the actual difference is, aside from orange going first in Cat-5e, apparently, but hey, more light on the subject, I guess.

Also, it appears that Able Cables had someone research a write a paper on the differences on the difference between TIA/EIA-568A and TIA/EIA-568B. Here is his conclusion:

9.0 Conclusion

Historically 568B was the specification on choice due to its early development and implemented base, but as the market and the political climate has changed over the years 568A has become the more dominate and preferred specification. This is only due to a desire by world standards organizations to provide a specification as backwardly compatible as possible. All new installations should be carried out using the 568A standard and cables only to be terminated to 568B specification on existing 568B systems.

Honestly, I’m not entirely sure what to make of that.

Can anyone here with more cable experience give me a hand?

BTW, I also posed this question on serverfault.

Always sign exactly what you are authenticating

2010-01-28T07:00:59+00:00

Always sign exactly what you are authenticating

I can't claim to know very much about cryptography programming, but I like to think that I have picked up on a few mistakes to avoid. Here's one of them: you want to sign exactly what you are authenticating, not some mangled version of it.

(Note that 'canonicalizing' things is a form of mangling them.)

Suppose, as a not entirely hypothetical example, that you are signing some sort of web request with a bunch of (URL) parameters. In order to deal with annoying software, you define a canonical form for these URL parameters, which is to sort them into alphabetical order and concatenate them all together; you sign this mangled, canonical result.

Great, except that that this canonical form has just allowed an attacker to turn 'login=fred&next=10' into 'login=fredne&xt=10' (and worse is possible if you do not sort the parameters into order but use them in the request's order). As Colin Percival puts it, good crypto signatures are designed so it is very difficult to produce collisions and when you mangle what you sign, part of what you do is that you create opportunities for attackers to produce deliberate collisions. This rarely ends well.

Another part of what you do when you mangle is that you are no longer necessarily verifying what you think you are verifying. Instead of verifying what you are actually about to act on and use, you are verifying something else, some mangled transformation of it. This is almost invariably a mistake, one that attackers will be happy to take advantage of to slip dangerous things into the gap between what you verify and what you act on.

(It is possible that your mangling will be un-exploitable. But the historical odds are against you; over and over, people who have done this sort of mangling and imprecise verification have turned out to have created exploitable vulnerabilities. And if you are writing crypto code, you should not be betting on things going your way.)

Sidebar: how I think you have to do canonicalization

Disclaimer: you should not necessarily trust what I write about how to do crypto (as opposed to how not to do crypto), because I am not an experienced crypto person.

I believe that the corollary to this is that if you absolutely have to do mangling and canonicalization, you must do it as part of generating the plaintext; you take raw input, transform it into the canonical form, sign the canonical form, and output the canonical form and the signature. On verification, you canonicalize, verify the canonicalization, and then use the canonicalization for further processing, not the raw input.

If you cannot use your canonical form as input to the rest of your processing or as your public plaintext, you need a new canonical form. Try again.

(One comment.)

Hot Scalability Links for January 28 2010

2010-01-28T05:52:51+00:00

Google's Research Areas of Interest: Building scalable, robust cluster applications. At Google we see distributed systems as a technology in its infancy, with huge gaps in the supporting research that represent some of the most important problems in the space. Here are some examples: Resource sharing, Balancing cost, performance, and reliability, Self-maintaining systems.
Amazon SimpleDB: A Simple Way to Store Complex Data by Paul Tremblett. The most effective way I have found to understand SimpleDB is to think about it in terms of something else we all use and understand -- a spreadsheet.
Rackspace Cloud Servers versus Amazon EC2: Performance Analysis. The Bitsource conducted a review of the two cloud computing platforms, Rackspace Cloud Servers and Amazon Elastic Compute Cloud (EC2), to get a general idea of overall system performance.
Private Clouds Are Not The Future by Jame Hamilton. Private clouds are better than nothing but an investment in a private cloud is an investment in a temporary fix that will only slow the path to the final destination: shared clouds.
What is the right way to measure scale? by Daniel Abadi. So which scales better? Is using the number of nodes a better proxy than size of data? Hadoop can “scale” to 3800 nodes. So far, all we know is that Greenplum can “scale” to 96 nodes. Can it handle more nodes?

REAL PCI Compliance Percentages?

2010-01-28T05:05:00+00:00

Reading "UK Security Breach Investigations Report 2010" [PDF] (source) makes for some truly blood-curdling reading.

Example:

"Prior to having suffered a cardholder data compromise, 26% of the organisations had believed themselves to be PCI DSS compliant upon submission of completed
Self Assessment Questionnaires. The investigations also revealed that none of the organisations met all requirements of the PCI DSS."

This is how it starts. It is all downhill from there:

"Indeed, in just over one quarter of the cases, none of the twelve requirements were met."

and

"7Safe has found that all the merchants who have been subject to a breach and have completed an ASV scan
have believed themselves to be secure based solely on the results of this scan"

I literally cannot read any further, since I am starting to get angry! Can somebody come and kick those merchants in the balls, please? Actually, no. Do not kick them! Stand on their balls. :-) Then have then buy our book!

Administering Users in Linux via CLI

2010-01-27T23:57:54+00:00

The other day, I mentioned that my review of LISA ‘09 was published in Linux Pro Magazine. Well, I went to the bookstore to pick up a copy. It wasn’t there, but to my surprise, I saw the special “Linux Shell Handbook” edition was out, which includes a four page article that I wrote about administering users in Linux with the command line tools! How cool! So I picked up a copy for my junior admin and another to read myself. Very neat to see my name in a printed magazine. I can’t wait to see it on the cover of a book!

iPad vs. Netbook: Netbook FTW

2010-01-27T21:38:03+00:00

I’m glad to see the iPad is announced. It looks like an interesting device, not quite a notebook, not quite an iPhone. I, however, don’t see how it’s anything beyond a portal to give Apple more money.

Please, if you see I’ve made an error here let me know in the comments. Thank you!

1. AT&T. Seriously, a “breakthrough” deal with AT&T is like being the fastest reader in remedial reading class. You’re still in remedial reading class.

2. No Flash. It’s astonishing how much stuff I watch in Flash on my laptop, and it being missing on this device is going to be a big hole. Lots of stuff is in YouTube, but not everything, and HTML5 isn’t going to solve this problem for quite a while, either.

3. You still need a desktop to dock this thing to, for anything beyond basic downloading or web browsing. Despite what some apps can do (including iWork), it really is just a standalone viewer of content.

4. The dock and keyboard setup is a kludge. It appears clunky, certainly not easily portable, and looks like it’ll fall over when you try to click on something by touching it. I’m skeptical. I think a netbook or cheap laptop will continue to smoke the iPad for anybody who needs to type anything. Heck, you can add an external USB keyboard and a mouse to a netbook.

5. E-books with DRM. There is no mention of being able to use anything but EPUB format books. I’d like to be able to read things from Project Gutenberg, for example, or anything that an independent party might like to push out. Furthermore, it looks as if EPUB doesn’t work well for technical books or books that need precision graphics placement (comic books, for example).

6. No user-replaceable batteries, though it’s not a huge deal because you can charge just about anywhere. If their battery life figures aren’t inflated it should be enough for a day’s use. Plus, with a tablet I’m anticipating third-party form-fitting add-ons that boost battery life. I worry about wear on the battery, though — after a year or so of daily charging batteries lose significant capacity.

7. It is still tied to the draconian App Store policies. Apple still controls who can put what on this device, and their policies are not consumer-friendly. Take Google Voice as an example. Maybe Apple should watch their first commercial, “1984,” and see what their message was then.

8. No multitasking. On a real computing device you can switch between apps and not lose your place. I understand the implications for battery life and whatnot, but I’d like the option to quickly switch between apps, like an SSH client and a web browser, and keep my sessions.

Looking at Apple’s list of things they think the iPad can do better:

Browsing: netbooks for the win. A netbook has Flash and can run any web browser, not just the Apple-prescribed browser and technologies.

Email: netbooks for the win. The external keyboard is a kludge, not portable, and I’m guessing they added it because typing on the screen sucks.

Photos: I’ve changed this based on new information in the comments. I didn’t realize that the iPad had a camera connector, so I’ll dub this a tie. Netbooks are more flexible and can run more software packages, but the display & interface on the iPad will likely smoke a netbook’s. What does remain to be seen is if iPhoto or something like it will be ported to Windows. If that happens it’s iPad FTW.

Video: netbooks for the win. Aside from the lack of Flash on the iPad, which disables most Internet video players, you are only able to watch video encoded with Apple-prescribed codecs.

Music: iPad. The iPod is the standard, and the iPad will draw on that heritage.

Games: iPad. The games for the iPhone and iPad are so-so, but netbooks really don’t have the ability or interface to play anything.

eBooks: libraries for the win. A paperback book doesn’t take any power, can be read in many differing conditions, isn’t made of toxic waste, isn’t locked to a carrier, doesn’t have a monthly fee (though one would argue your library has a fee in the form of taxes), can be loaned to your cousin, is available at millions of locations, has an easy-to-use interface, can be dropped on the floor or crushed in your luggage, and can be donated to or borrowed from a library or a book swap when you’re done with it.

I think, for now, I’ll stick with a netbook and a paperback.

Review of Professional Penetration Testing Posted

2010-01-27T20:28:17+00:00

Amazon.com just posted my three star review of Professional Penetration Testing by Thomas Wilhelm. From the review:

I had fairly high hopes for Professional Penetration Testing (PPT). The book looks very well organized, and it is published in the new Syngress style that is a big improvement over previous years. Unfortunately, PPT should be called "Professional Pen Testing Project Management." The vast majority of this book is about non-technical aspects of pen testing, with the remainder being the briefest overview of a few tools and techniques. You might find this book useful if you either 1) know nothing about the field or 2) are a pen testing project manager who wants to better understand how to manage projects. Those looking for technical content would clearly enjoy a book like Professional Pen Testing for Web Applications by Andres Andreu, even though that book is 3 years older and focused on Web apps.

This is my 300th Amazon.com book review. I wish I had planned the review schedule such that I reviewed a five star book for number 300.

I reported my 200th book review for Building an Internet Server With FreeBSD 6 in August 2006.

The enduring cipher: Unbreakable for nearly 100 years

2010-01-27T19:10:41+00:00

One cryptographic cipher has been mathematically proven to be unbreakable when it is used correctly, but it is only very rarely used. Chad Perrin breaks down the one-time pad cipher.

It seems to be a truism of cryptography that any cipher, no matter how strong it is considered to be in its heyday, eventually becomes a weak cipher. What people fear when they use the current favorite strong cipher is that someone will crack it — will find a shortcut that greatly reduces the time required to use brute force calculation to decrypt something without the key.

Even if a cipher is never broken, and we forever-more need the same average number of CPU clock cycles to decrypt something encrypted by a given cipher without using the key; it still takes less time every few months to achieve the same goal than it did a few months ago. This is because computers keep getting faster, allowing us to squeeze the same number of CPU clock cycles into a shorter period of time.

Ever-more complex and clever algorithms are designed to provide ever greater resistance to brute force cryptanalysis, and to replace older algorithms that have been broken or otherwise become obsolete. It is always an arms race — privacy against the attempt to penetrate that privacy.

Amongst all the wreckage of the broken and rusty ciphers that have fallen by the wayside through history, one cipher has endured for the last 93 years. It is called the one-time pad. In 1917, Gilbert Vernam developed a cipher known as the Vernam Cipher, which used teletype technology with a paper tape key to encrypt and decrypt data. The result was a symmetric cipher that was quite strong for its time.

U.S. Army Captain Joseph Mauborgne realized that by using truly random keys, where no part of the key was repeated (except perhaps at random), the Vernam cipher could be made much stronger. From the idea of using paper tape keys, a pad of paper with rows of random letters or numbers on each page as the means of recording keys was developed. Two copies of the same pad could be given to two people, and by using each character on each page only once (and destroying each page as its last character is used to encrypt or decrypt a message), they could pass encrypted messages between them without fear of an intercepted message ever being decrypted without the help of the key. Because of the technique of distributing key stream data on pads of paper, this cipher became known as the one-time pad.

Claude Shannon, known as the father of information theory, mathematically proved the unbreakability of the one-time pad cipher when it is used properly — including destroying any pages containing used key data so that it will not be used, and so that unauthorized copies of any messages cannot later be decrypted if someone gets his hands on your used pages from the pad. The same concept for key management can be employed digitally, of course, with the proviso that one must be very careful to avoid letting the inherent weaknesses of computers introduce flaws into the one-time pad system. For instance, expensive data recovery operations might be able to reconstruct “deleted” files, including used one-time pad data. There are things you can do to help obscure such data when simply deleting files is not enough, but one must be careful.

The one-time pad cipher can be extremely inconvenient at times, which is why it is not used more often. We do actually need theoretically weaker (if cleverer) ciphers, such as AES/Rijndael and Twofish because of that inconvenience:

Because the one-time pad cipher is a symmetric cipher, both parties to an encrypted communication must have the exact same key data. For certain use cases for encryption, this makes a one-time pad completely useless, because to securely exchange the key data so both parties have it, one must have a secure means of sharing data that would work just as well for sharing the eventual messages themselves. Only in cases in which you do not know what messages you will need to send, and where you will not be able to use whatever secure means was used to exchange the key data (such as physically handing it to the person) at the later time, is the one-time pad cipher useful.
A one-time pad encryption key must be as long as the message it is used to encrypt and decrypt. Thus, if you want to encrypt or decrypt a three gigabyte file, you need three gigabytes of one-time pad key data.
The same one-time pad data can not be shared securely among more than two people; for example, in cases where different messages will be sent between some recipients of the key data, which should not be readable to other recipients, using the same one-time pad amongst all of them subverts the security of the cipher. By contrast, with an asymmetric cipher you can provide the same public key to dozens of people, and they will all be able to use that same public key to encrypt messages for you without any fear the other people who have the public key will be able to read it — as long as the cipher is not broken and the state of the art of computing technology does not advance enough to reasonably brute force decrypt the messages. This is because when something is encrypted with the public key, only the associated private key can be used to decrypt it.
Reusing a key potentially breaks the security of the one-time pad cipher because it suffers a known-plaintext vulnerability. Kerckhoffs’ principle states that a cryptosystem should be secure so long as the the key remains secret, but where the encrypted and unencrypted (plaintext) versions of a given message are both known, the key can be derived in the case of the one-time pad cipher. This is not a problem if each string of key data is used only once, because if the plaintext is captured by the “enemy” you have already lost the game anyway. If a key is reused, however, one message’s plaintext can be used as part of the set of tools used to determine the key for decrypting another message. The moral of the story is: Don’t use a given one-time pad key more than once. There is a reason it is called a “one-time” pad.
When the last of your one-time pad is used up, you cannot securely send messages back and forth — encrypted using that same cipher — any longer unless you securely exchange new random key data. This kind of thing can really cramp your style when trying to communicate with someone on the other side of the world.

Other factors come into play in making use of the one-time pad cipher impractical in some circumstances, too, but these should give you a good start on seeing why other, theoretically weaker ciphers are still important.

The way the one-time pad works is deceptively simple. It involves merely comparing each of two datums, such as two letters or numbers, and using that comparison to produce a new datum. This is done for every such datum in the message you want to encrypt. The process of performing this comparison is simple and easy, one datum at a time, and (relatively) computationally cheap. A simple operator known as the XOR operator can be used to perform such a comparison. In its simplest form, the XOR operator as applied to binary numbers works something like this:

First, you need a message. Let’s use the word “short” as our message. Yes, that is a short message.
Next, you translate that message into a binary representation. Using ASCII encoding to translate the word “short”, you end up with the following string of ones and zeros:
```
0111001101101000011011110111001001110100
```
The last thing you need is a key that is exactly as long as the message. In this case, let’s use this string of ones and zeros as our key:
```
0110010101101010001110010010011101100100
```
Finally, with the XOR operator, you basically perform a simple subtraction. Where the first character in each case is a zero, you see that 0 - 0 = 0. Similarly, the second character in each is a one, and 1 - 1 = 0. Where one character is a one and the other a zero, though, you get either a 1 or a -1 result. If you take the absolute value of the result, that means that both will give you a 1 result. Thus, subtracting and taking the absolute value provides the following:
```
0111001101101000011011110111001001110100
0110010101101010001110010010011101100100
----------------------------------------
0001011000000010010101100101010100010000
```

Of course, there are no ASCII translations for some of those groups of eight binary characters in the resulting string of ones and zeros, so it is difficult to represent the data in a concise form. Using ASCII encoding is well-suited to computer use, but the traditional number-and-letter approach to implementing the one-time pad cipher is much better suited to analog, by-hand translations.

The core algorithm for the one-time pad cipher is obviously incredibly simple. It is only in designing the rest of the software that surrounds this algorithm, and finding the right use case for the cipher, that the real problems of secure cryptographic software development arise. On the other hand, if it absolutely, positively has to be securely encrypted, the one-time pad is the only cipher that is provably unbreakable when used properly — given our current understanding of mathematics.

Lessons From A Stolen iPhone

2010-01-27T18:36:46+00:00

So you're in Mexico (Cabo San Lucas to be exact) and flying back to the USA. You send a reassuring text to a friend once you land letting them know that you'll be out of customs in 30 minutes and to show up on time at the north curb. You're nearly off the plane when you realize... WHERE IS MY iPHONE? Such is the story of Mr. Sam, located

Ipswitch acquires Dorian Software Creations Inc

2010-01-27T16:05:11+00:00

Ipswitch, the people responsible for creating What’s Up Gold, have acquired Dorian Software Creations. Dorian Software are publishers of event log management software.

Dorian’s event log management solutions for Windows and Syslog environments include:

Event Archiver for automated collection, centralization and secure storage of log data;
Event Analyst for event examination, correlation and comprehensive reporting for audit and compliance;
Event Alarm for monitoring, alerting and notification on key defined events;
Event Rover for on-the-fly forensics and log data mining.

Dorian products are scheduled to be available from Ipswitch in March.

Microsoft Mediaroom Personal Server Odyssey

2010-01-27T15:09:15+00:00

Ewwww, scary isn’t it. No Its not Halloween, but you may have entered the twilight zone. Right, I never touch Microsoft products. Well in actuality sometimes I do (I just don’t brag about it). Some of the development at $work uses Microsofts Mediaroom, and I have a “Personal Server” (great name right?) that the developers use. [...]

Now Mobile Ready

2010-01-27T14:46:29+00:00

I’d like to thank Dale Mugford and Duane Storey from BraveNewCode for the nice Wordpress plugin and bundled mobile theme. If you visit my site on your mobile device you should get a slimmed down page, let me know what ya think. ©2010 cmdln.org (a sysadmin blog). All Rights Reserved..

PC-BSD 8.0-RC Released

2010-01-27T12:17:13+00:00

From the announcement:

AT&T's mad unbundling and the damage it did to Unix

2010-01-27T06:56:24+00:00

AT&T's mad unbundling and the damage it did to Unix

Once upon a time, AT&T had Unix, and Unix was big (by the standards of the time); it contained all sorts of things that made it useful. AT&T decided that it wanted to make as much money as possible from Unix, so it had a brilliant idea: it would sell a bunch of pieces of 'Unix' separately, as additional cost extras. After all, Unix systems were primarily used to run canned vendor-developed software (or at least AT&T no doubt maintained), so a lot of pieces weren't strictly necessary and thus could be split off, and you could even tell the customers that they were saving money by only buying what they needed.

This created AT&T's mad unbundling of Unix components, where they separated out previously all-included things like the C compiler, troff and associated commands (as the 'Documenter's Workbench'), and so on. Pretty soon the baseline version of (AT&T) Unix was significantly stripped down compared to its predecessors.

(AT&T also repeated this most every time that they made substantial improvements to commands, so you had 'new awk', improved versions of ksh, and so on, all of which cost more money.)

This had both immediate and long term effects. The immediate effect was to create annoyingly crippled versions of Unix, ones where you could not do things like compile your own programs or even read third-party manpages, and to divert a great deal of people's efforts to getting around these limitations. Naturally, people used to BSD Unix (which included C compilers, troff, and so on) hated these environments.

(The height of such workarounds is Henry Spencer's awf, a nroff clone written in old-style awk.)

The long term effect was to make Unix, all Unixes, less attractive and less useful at a crucial time for Unix in general. By maximizing its theoretical income, AT&T determinedly minimized (AT&T) Unix's appeal in general. The C compiler situation was an especially clever own goal, and it was saved from being a complete disaster only by the existence of the FSF's GNU C Compiler and the accompanying great effort to bootstrap GCC on as many Unixes as possible and make available the precompiled binaries. Without GCC, I really think that Unix would have done much worse in many places, and certainly important Unix augmentations like Perl would have spread significantly more slowly.

(Unix vendors could have crippled the GCC effort by not shipping the header files for the standard library; fortunately, they by and large did not. Note that this was by and large in the pre-Internet era, where many people could not just download precompiled binaries for Perl and so on from some Internet site. Even assuming that they would or could have installed lots of precompiled binaries from random third parties.)

Similar effects happened with the improved commands that AT&T developed; because they were extra cost, they spread through the Unix world only very slowly (if at all), and they almost might not have been written at all. Certainly their improvements did nothing to make Unix more attractive, because for most people they weren't part of their Unix, just as troff wasn't.

Display a CCK Filefield or Imagefield Upload Widget on Your Own Custom Form

2010-01-27T05:15:16+00:00

Took a fair amount of googling around to find the solution to this one. With the Node Gallery 3.x branch, we needed a way to quickly add an image to an existing gallery. We could have displayed the whole node form, but there's a lot of things on that form that we can just use the defaults for 99% of the time. We need just three fields filled in: Title, Caption, and the imagefield itself.

Energy Sector v China

2010-01-26T22:18:41+00:00

The aftershocks of Google v China continue to rumble as more companies are linked to the advanced persistent threat. Mark Clayton from the Christian Science Monitor wrote a story titled US oil industry hit by cyberattacks: Was China involved? I found these excerpts interesting.

At least three US oil companies were the target of a series of previously undisclosed cyberattacks that may have originated in China and that experts say highlight a new level of sophistication in the growing global war of Internet espionage.

The oil and gas industry breaches, the mere existence of which has been a closely guarded secret of oil companies and federal authorities, were focused on one of the crown jewels of the industry: valuable “bid data” detailing the quantity, value, and location of oil discoveries worldwide...

The companies – Marathon Oil, ExxonMobil, and ConocoPhillips – didn’t realize the full extent of the attacks, which occurred in 2008, until the FBI alerted them that year and in early 2009. Federal officials told the companies proprietary information had been flowing out, including to computers overseas...

“What these guys [corporate officials] don’t realize, because nobody tells them, is that a major foreign intelligence agency has taken control of major portions of their network,” says the source familiar with the attacks. “You can’t get rid of this attacker very easily. It doesn’t work like a normal virus. We’ve never seen anything this clever, this tenacious...”

Many experts say the theft of this kind of information – about, for instance, the temperature and valve settings of chemical plant processes or the source code of a software company – can give competitors an advantage, and over time could degrade America’s global economic competitiveness...

Even more basic, many corporate executives aren’t aware of how sophisticated the new espionage software has become and cling to outdated forms of electronic defense...

[B]ased on the kind of information that was being stolen, federal officials said a key target appeared to be bid data potentially valuable to “state-owned energy companies...”

China would certainly be interested in this kind of data, experts say. With the country’s economy consuming huge amounts of energy, China’s state-owned oil companies have been among the most aggressive in going after available leases around the world, particularly in Nigeria and Angola, where many US companies are also competing for tracts...

“What I’m saying to you is that it’s not just the oil and gas industry that’s vulnerable to this kind of attack: It’s any industry that the Chinese decide they want to take a look at,” says an FBI source. “It’s like they’re just going down the street picking out what they want to have.”

Expect more denials from party spokesmen in China.

Making enterprise ShortName service shorter

2010-01-26T21:33:36+00:00

Previously I wrote about the Google Apps shortname service which lets you set up a tinyurl service for your enterprise.

The article implies that the service can be used without using the FQDN. This is not true. In other words, I had said that "go.example.com/lunch" could be shortened to "go/lunch".

There is a workaround that makes it work. It is difficult to configure, but I've set up a Community Wiki on ServerFault.com that explains all the steps. As a wiki, I hope people can fill in the items I left blank, particularly specific configuration snippets for ISC BIND, Windows DHCP server, Linux DHCP clients, and so on.

The new article is here: How to set up Google ShortName service for my domain, so that the FQDN isn't needed

On Log Context

2010-01-26T19:42:35+00:00

Recently somebody asked me: what do I mean by LOG CONTEXT? And what is “log enrichment correlation?”

This picture explains it clearly:

For each element in the log message shown, you can gather some contextual information. Contextual here simply means that the information is gathered NOT from this particular log entry. For example, a log entry might contain an IP address, but its DNS name needs to be grabbed from the DNS server, which “enriches” the log entry and makes it more useful.

One of the ways SIEM and log management systems performs such enrichment is by gathering and displaying context information. Context information is the additional information required to make the limited details available within the log entry more meaningful. Context information does not come from the logs themselves [not from the entry in question – it might come from other logs], but originates in the surrounding IT environment such as other systems inside or outside the organization.

As I say above, one of the simplest example of context data is name resolution: the DNS names or Windows NetBIOS host names are added to the logs. While the log file may have already provided IP addresses, the added context of a human-readable name makes the log more meaningful. Normally, DNS names are not present in logs, but have to be obtained by queries to a DNS server. The SIEM tool might find context data in a variety sources, including:

Windows name services, DNS and NIS servers: to map IP addresses to names
Defined asset groups: internal or external status of an IP address
Asset management systems: to gather information about systems, their ownership, compliance relevant of each system or group of systems
WHOIS servers: WHOIS information for external addresses shows who owns them and where they are located
Geo-location: show the physical location of the system
Active directory and LDAP servers: to map user names to actual user identities
Attack details and vulnerability information: to gather additional details about the log data and/or the log source.

BTW, back in 2008, I did a poll on what context is the most useful for log analysis. That is what came back – it shows that some useful context is simply documentation on what the log mean (might be pulled from the internal knowledge base of a SIEM product):

Enjoy!

Product: HyperGraphDB - A Graph Database

2010-01-26T19:30:57+00:00

With the success of Neo4j as a graph database in the NoSQL revolution, it's interesting to see another graph database, HyperGraphDB, in the mix. Their quick blurb on HyperGraphDB says it is a: general purpose, extensible, portable, distributed, embeddable, open-source data storage mechanism. It is a graph database designed specifically for artificial intelligence and semantic web projects, it can also be used as an embedded object-oriented database for projects of all sizes.

From the NoSQL Archive the summary on HyperGraphDB is: API: Java (and Java Langs), Written in:Java, Query Method: Java or P2P, Replication: P2P, Concurrency: STM, Misc: Open-Source, Especially for AI and Semantic Web.

So it has some interesting features, like software transactional memory and P2P for data distribution, but I found that my first and most obvious question was not answered: what the heck is a hypergraph and why do I care? Buried in the tutorial was:

A HyperGraphDB database is a generalized graph of entities. The generalization is two-fold:

Links/edges "point to" an arbitrary number of elements instead of just two as in regular graphs

Links can be pointed to by other links as well.

OK, but I wish there was some explanation of why this is valuable. What can I do with it that I can't do with normal graphs? Given that there have been concerns over the complexity of the API this would seem a natural topic to cover. I assume it's cool, it sounds cool, but I would like to know why :-)

In any case it looks like an interesting product to take a look at. Database options are expanding fast.

Excerpts from Randy George's "Dark Side of DLP"

2010-01-26T18:59:11+00:00

Randy George wrote a good article for InformationWeek titled The Dark Side of Data Loss Prevention. I thought he made several good points that are worth repeating and expanding.

[T]here's an ugly truth that DLP vendors don't like to talk about: Managing DLP on a large scale can drag your staff under like a concrete block tied to their ankles.

This is important, and Randy explains why in the rest of the article.

Before you fire off your first scan to see just how much sensitive data is floating around the network, you'll need to create the policies that define appropriate use of corporate information.

This is a huge issue. Who is to say just what activity is "authorized" or "not authorized" (i.e., "business activity" vs "information security incident")? I have seen a wide variety of activities that scream "intrusion!" only to hear, "well, we have a business partner in East Slobovistan who can only accept data sent via netcat in the clear." Notice I also emphasized "who." It's not just enough to recognize badness; someone has to be able to classify badness, with authority.

Once your policies are in order, the next step is data discovery, because to properly protect your data, you must first know where it is.

Good luck with this one. When you solve it at scale, let me know. This is actually the one area where I think "DLP" can really be rebranded as an asset discovery system, where the asset is data. I'd love to have a DLP deployment just to find out what is where and where it goes, under normal conditions, as perceived by the DLP product. That's a start at least, and better than "I think we have a server in East Slobovistan with our data..."

Then there's the issue of accuracy... Be prepared to test the data identification capabilities you've enabled. The last thing you want is to wade through a boatload of false-positive alerts every morning because of a paranoid signature set. You also want to make sure that critical information isn't flying right past your DLP scanners because of a lax signature set.

False positives? Signature sets? What is this, dead technology? That's right. Let's say your DLP product runs passively in alert-only mode. How do you know if you can trust it? That might require access to the original data or action to evaluate how and why the DLP product came to the alert-worthy conclusion that it did.

Paradoxically, if the DLP product is in active blocking mode, your analysts have an easier time separating true problems from false problems. If active DLP blocks something important, the user is likely to complain to the help desk. At least you can figure out what the user did that upset both DLP and the denied user.

However, as with intrusion-detection systems, not all actions can be automated, and network-based DLP will generate events that must be investigated and adjudicated by humans. The more aggressively you set your protection parameters, the more time administrators will spend reviewing events to decide which communications can proceed and which should be blocked.

Ah, we see the dead technology -- IDS -- mentioned explicitly. Let's face it -- running any passive alerting technology, and making good sense of the output, requires giving the analyst enough data to make a decision. This is the core of NSM philosophy, and why NSM advocates collecting a wide variety of data to support analysis.

For earlier DLP comments, please see Data Leakage Protection Thoughts from last year.

Are TSA policies a bad joke?

2010-01-26T15:03:02+00:00

As the idea that TSA policies are a joke becomes ever more popular with the American people, one TSA screener decides to prove them right — literally.

Probably the most famous modern IT security and cryptography expert in the world, Bruce Schneier, coined the term “security theater” in his book Beyond Fear. The book refers to security procedures designed to give the impression that something is being done to enhance security without actually providing any real security benefit at all. Since then, the term has increasingly been applied to airport security measures, particularly those enacted by the United States’ Transportation Security Administration.

There is some good reason for the harsh scrutiny applied to TSA policies. Completely aside from the complaints about invasions of privacy and annoying complexity added to the process of flying, some very dramatic failures have affected the image of the TSA very negatively. The implementation and use of a “Terror Watch List” has produced many false positives, including both an eight year old boy and U.S. Senator Ted Kennedy. Recently on people’s minds is the case of Umar Farouk Abdulmutallab who attempted to blow up an airliner on Christmas day 2009 — who escaped detection by airport security, but was stopped by a civilian passenger. This case might be quite as strong an indictment of airport security procedures if not for the fact that it was not the first time it happened. In 2001, Richard Reid became famous for trying to blow up an airplane with a bomb concealed in his shoe. He too escaped detection by airport security and was subdued by passengers as well.

The simplistic procedures employed to check incoming passengers’ shoes as a result of the Richard Reid incident were widely criticized as being in the spirit of security theater. It does, however, at least appear to address the problem directly in some way at first glance. The first piece of news many of us heard about new policies for trying to prevent future threats to airline security after the Christmas day bomber was caught sounded like the punchline to a joke, on the other hand. Proposals were made to prevent passengers from leaving their seats, because Umar Farouk Abdulmutallab apparently spent twenty minutes in the airplane’s restroom as it approached its destination in Detroit just before he was tackled by another passenger — a passenger who, of course, left his seat to stop the would-be bomber.

The most recent case of airport security falling flat on its face, however, actually is a punchline. It just happens to be a particularly bad punchline that rolls many of the best known mistakes of the last few years into a single, terrible, awful joke that should never have been made. Passengers have made a point of introducing something like humor into the increasingly frustrating process of trying to get through airport security, as in the case of TSA Communication, but this time around the culprit is a TSA screener.

As reported in It was no joke at security gate, passenger Rebecca Solomon had a terrifying 20 seconds while passing through airport security:

After pulling her laptop out of her carry-on bag, sliding the items through the scanning machines, and walking through a detector, she went to collect her things.

A TSA worker was staring at her. He motioned her toward him.

Then he pulled a small, clear plastic bag from her carry-on - the sort of baggie that a pair of earrings might come in. Inside the bag was fine, white powder.

Of course, the bag was not hers, and neither was the white powder. She had never seen it before, and the TSA screener knew it:

Put yourself in her place and count out 20 seconds. Her heart pounded. She started to sweat. She panicked at having to explain something she couldn’t.

Now picture her expression as the TSA employee started to smile.

Just kidding, he said. He waved the baggie. It was his.

It really does not get much worse than this for the image of a government agency whose image was already among the worst in the country. The article summed up the event succinctly and accurately:

The last thing we expect is a joke from a Transportation Security Administration screener - particularly one this stupid.

Can it get any worse?

Product Spotlight: Barracuda Networks SSL-VPN

2010-01-26T14:00:41+00:00

Derek Schauland looks at the features of the Barracuda SSL-VPN, a browser-based VPN appliance, in this product spotlight.

—————————————————————————————

Remote access to corporate resources is becoming common place within many organizations. IT staff require access from the road, just in case they are out of the office and receive a help desk call. Road warriors require access to files and documents or maybe even their desktops while traveling to visit a customer. There are many ways to enable this type of access, some secure, some not so secure, but the SSL-VPN appliance from Barracuda Networks makes it very simple to provide browser-based VPN to a number of employees while working with your existing security appliances and firewalls.

Specifications

The Barracuda SSL-VPN comes in several flavors:

Model	Concurrent Connections
180	15
280	25
380	50
480	100
680	500

The unit plugs into your network either inside the firewall, which requires configuration changes to route SSL traffic to the new device, or in the DMZ where it will forward requests into your environment. In the box you will find a quick-start configuration guide, the SSL-VPN appliance, and a power adapter.

Supported operating systems:

Windows 7
Windows Vista
Windows XP

Hardware requirements:

An Internet connection is required to access the VPN: there are no other hardware requirements on the PC end.

Who’s it for?

The SSL-VPN is ideal for organizations that need to add remote connectivity for their users without worrying about clients being installed and maintained on their PCs or a device being configured to live in their home office.

What problem does it solve?

The SSL-VPN allows access to published Citrix applications, remote desktop sessions, network shares, and even supports a client to allow tunneling to specific machines or devices to happen behind the scenes. There are no applications to install for typical use and for tunneling; a client based on Java can be downloaded after login. The process for the user becomes very straight forward.

Security is also a concern when allowing access to your network from the outside. The SSL-VPN addresses this by integrating with Active Directory or allowing the administrator to create and manage the user database directly on the device. When a user connects to the device, they need to log on and when they disconnect; the session is terminated and does not stay open constantly.

Standout features

Antivirus and malware checking: When a file is copied to the network through the VPN, the file is checked for viruses as it is moved to the network.

Easy configuration: The plug and play nature of the device eases the workload on the IT staff.

Simple to use: Users within an organization simply point their browser to an address or host on the Internet and log in. Items they are able to access appear on a desktop style list once logged in.

Immediate replacement packages: Barracuda offers a license option for immediate replacement which brings some peace of mind in the event that the device malfunctions or needs to be replaced.

Figure A

At-a-glance information for the administrator

Figure B

User homepage after login

What’s wrong?

Some of the larger models can get a bit pricey, so consider the number of users that will connect to the device before ordering.

Competitive products

Cisco ASA Firewalls
Sonicwall SSL-VPN appliances

Bottom line for business

In today’s workforce, employees want to be productive from everywhere they might be, both in the office and out. Traditional VPNs are very useful for organizations but require specific applications on the client PC and/or devices, configured to allow a point-to-point VPN connection. With an SSL-VPN, the client is a Web browser on the PC and a network or VPN logon. The device takes care of the rest, including auditing of sessions and users connecting to the device. This ability to track the usage of the VPN can make justification very simple for IT and management.

Links for 2010-01-25 [del.icio.us]

2010-01-26T08:00:00+00:00

Mac bookmark toolbar favicons | userstyles.org

Why the modern age is great

2010-01-26T06:30:16+00:00

Why the modern age is great

In some quarters, it is popular to grumble about how much better things were back in the good old days of computers, Unix, or whatever. I don't hold with this view at all; I think that the modern age is great, and now I'll tell you one reason why.

I started working with Unix systems what is now quite a long time ago. Back in those days, you couldn't get SCCS without paying extra money to AT&T (and sometimes you couldn't get it at all, because your Unix vendor hadn't paid extra for it themselves), and you couldn't get RCS without a Unix source license, because RCS needed a customized version of diff (and there was no 'GNU diff'). Thus, if you had a source license, you generally used RCS and counted yourself lucky.

(Certainly universities pretty much didn't have the budget to buy extras from AT&T. Source control? That was a luxury, we weren't commercial developers, we hardly needed that. (AT&T's mad unbundling of the useful pieces of Unix is another rant entirely.))

Contrast that to today, where I find myself vaguely agonizing over which highly sophisticated distributed version control system to use. This shows both how far we've come and how plain nice our computing environments have become; when the big issue is just how awesome my version control system is going to be, we're doing pretty well.

(System administration has been going through this for decades. Huge swatches of boring routine work that people did even fifteen years ago are now completely gone, at least if you're not using Solaris 10.)

Communication

2010-01-25T22:20:35+00:00

“I know that you believe you understand what you think I said, but I’m not sure you realize that what you heard is not what I meant.” — Robert McCloskey

Haiti: Fact Finding Mission, Churches Helping Churches

2010-01-25T18:56:00+00:00

A coalition of churches quickly formed following the quake in Haiti, Churches Helping Churches, made up of several churches including Pastor Mark Driscoll of Mars Hill Church in Seattle. They went on site last week to assist the churches in Haiti and assess the needs.

Yesterday Pastor Mark preached a special sermon which told the entire story of his trip. If your interested in the situation on the ground in Haiti and particularly in the state of the churches there watch the sermon here: 32 Hours in Haiti

If you would like to help the churches in Haiti to continue helping the people of Haiti please consider a donation to churcheshelpingchurches.com.

Storage tiers

2010-01-25T18:38:11+00:00

Events have pushed us to give a serious look at cheaper storage solutions. What's got our attention most recently is HP's new LeftHand products. That's some nice looking kit, there. But there was an exchange there that really demonstrated how the storage market has changed in the last two years:

HP: What kind disk are you thinking of?
US: Oh, probably mid tier. 10K SAS would be good enough.
HP: Well, SAS only comes in 15K, and the next option down is 7.2K SATA. And really, the entire storage market is moving to SAS.

Note the lack of Fibre Channel drives. Those it seems are being depreciated. Two years ago the storage tier looked like this:

SATA
SAS/SCSI
FC

Now the top end has been replaced.

SATA
SAS
SSD

We don't have anything that requires SSD-levels of performance. Our VMWare stack could run quite happily on sufficient SAS drives.

Back in 2003 when we bought that EVA3000 for the new 6 node NetWare cluster, clustering required shared storage. In 2003, shared storage meant one of two things:

SCSI and SCSI disks, if using 2 nodes.
Fibre Channel and FC Disks if using more than 2 nodes.

With 6 nodes in the cluster, Fibre Channel was our only choice. So that's what we have. Here we are 6+ years later, and our I/O loads are very much mid-tier. We don't need HPC-level I/O ops. CPU on our EVA controllers rarely goes above 20%. Our I/O is significantly randomized, so SATA is no good. But we need a lot of it, so SSDs become prohibitive. Therefore SAS is what we should be using if we buy new.

Now if only we had some LTO drives to back it all up.