Planet SysAdmin

• Writing udev rules

I’m re-reading the classic The Data Warehouse Toolkit by Ralph Kimball. It’s a great book for learning about data warehousing. I figured it was worth a refresh. In my current project, I’m working with some large data sets and I’ll be using data marts to make analyzing the data easier. 

I’m at the section where he talks about CRM and adding a customer dimension. I’ve always wondered why he uses a customer dimension instead of an accumulating snapshot customer fact table.

There’s lots of interesting things about a customer that can be expressed as facts, like spending (total, yearly, quarterly, etc) and customer service requests. Adding a ‘has purchased’ dimension makes segmenting easy too. Data about the customer, like first purchase, last purchase and location, goes from being evil outriggers to full blown dimensions of their own. 

Once you have a customer dimension, you can drill across data marts to get customer-centric slices of queries. Alternatively, the customer fact table can have a view to turn it into a dimension. If you’re data sets are really big, you can create mini dimensions from the customer fact table (which he recommends anyway).

 

I suppose hard drive and memory sizes were a lot smaller and CPU’s were a lot slower in 2002.. 

I just wanted to note that the MySQL bug I mentioned in my post First Issue of BSD Magazine Release will be fixed in MySQL 5.1.25 and 6.0.6, according to the bug report. I am really impressed by the developers' speedy reaction and resolution of the problem. When the code is available I plan to test it.

Thanks to several of you for asking for my opinion of the article Carpet bombing in cyberspace: Why America needs a military botnet by Col. Charles W. Williamson III. I'd like to cite a few excerpts and comment directly.

The world has abandoned a fortress mentality in the real world, and we need to move beyond it in cyberspace. America needs a network that can project power by building an af.mil robot network (botnet) that can direct such massive amounts of traffic to target computers that they can no longer communicate and become no more useful to our adversaries than hunks of metal and plastic. America needs the ability to carpet bomb in cyberspace to create the deterrent we lack...

This is interesting. Why do we need to project force in cyberspace to deter our enemies? Cyberwar is usually cited as a means of conducting asymmetric warfare, meaning one side is much weaker than other in conventional means. Cyberwar is expected to be conducted against US assets (critical infrastructure) because the enemy lacks the capability to destroy or degrade that asset using kinetic weapons. If we can deter enemies using our existing, overwhelming kinetic force, why possess an ability to "carpet bomb in cyberspace?"

Today’s air base defense concept still uses a layered defense in depth, but it starts as far as possible from the air bases, then relies on close-in defense only as a last resort. That capability in cyberspace can exist in an af.mil botnet...

The U.S. would not, and need not, infect unwitting computers as zombies. We can build enough power over time from our own resources.

Rob Kaufman, of the Air Force Information Operations Center, suggests mounting botnet code on the Air Force’s high-speed intrusion-detection systems. Defensively, that allows a quick response by directly linking our counterattack to the system that detects an incoming attack. The systems also have enough processing speed and communication capacity to handle large amounts of traffic.

Oh, that's a great idea. Let's tie up the really only useful element of the Air Force's defense -- that which provides some degree of situational awareness -- with the task of packeting someone.

Next, in what is truly the most inventive part of this concept, Lt. Chris Tollinger of the Air Force Intelligence, Surveillance and Reconnaissance Agency envisions continually capturing the thousands of computers the Air Force would normally discard every year for technology refresh, removing the power-hungry and heat-inducing hard drives, replacing them with low-power flash drives, then installing them in any available space every Air Force base can find. Even though those computers may no longer be sufficiently powerful to work for our people, individual machines need not be cutting-edge because the network as a whole can create massive power.

I see... so the very network that is important enough to be deemed a "weapons system," thanks to the logistics, communication, and related traffic it carries, is going to be filled with tons of DDoS traffic from recycled PCs? Do you think QoS is supposed to take care of this problem?

After that, the Air Force could add botnet code to all its desktop computers attached to the Nonsecret Internet Protocol Network (NIPRNet). Once the system reaches a level of maturity, it can add other .mil computers, then .gov machines.

To generate the right amount of power for offense, all the available computers must be under the control of a single commander, even if he provides the capability for multiple theaters. While it cannot be segmented like an orange for individual theater commanders, it can certainly be placed under their tactical control.

I am sure the botnet software installed would be super secure. Can you say "biggest latent botnet" in history? Every single .mil and .gov computer under the control of a single commander -- probably a Russian or Chinese infiltrator? Just who is Col. Williamson working for, anyway?

This is a really dumb idea, at least as presented. I'm all for Taking the Fight to the Enemy, but building a botnet on operational networks, especially on operational defensive systems and even production equipment, is just wrong. If we want to remove someone from the network, it's far simpler to disable the right cable using conventional means.

Let's assume the Air Force did build a botnet, on separate, non-production computers, on dark space, ready to point towards an enemy. Where would that target be? No single, or handful, of computers DDoS'd Estonian infrastructure. (Every military planner loves to cite Estonia these days.) If someone decided to DDoS one or more US computers or routers, where would be point our botnet? Where would Estonia have pointed any botnet it owned -- Russia? Back at the computers DDoSing Estonian assets? How is this supposed to work? "Don't DDoS us or we'll DDoS you?" Mutually Assured DDoS?

There are smarter ways to conduct operations in cyberspace, and this is not one of them. Back to the drawing board, sir.

Sitting in traffic today I realized that teams of people are either like stop lights or four-way stop signs.

Stop lights are nice because everybody knows the rules, and they aren’t flexible. Everybody knows what everybody else should be doing, which is either sitting there idling, burning expensive fuel, or driving forward full-blast. Big queues build up sometimes behind a stop light, blocking other streets. When the stop lights aren’t timed perfectly (and they rarely are) you get these gobs of cars hurrying, then waiting, then hurrying again. One thing is true, though: that clueless guy talking on his cell phone doesn’t mess things up too much. Even they can figure out when to go.

Stop signs are different, especially the four-way kind. Your driving instructor told you that things move around to the right, but in practice there are tons of shortcuts, simple optimizations people make. Like if the fellow across from you is going straight you can, too, at the same time. Or being able to turn right, out of order. If you’re waiting for your turn at the game you keep moving steadily forward, always advancing. When you get to the game, though, you find that things are very fluid, and you had better know what you’re doing or you’ll mess it up for everybody. That clueless guy on his cell phone screws up the whole system, though, requiring heroism from others to get things back to normal.

Seems to me that’s exactly like most teams. Lots of teams start out as a stop sign, but eventually they get someone who is the equivalent of a cell phone idiot, not paying attention to their job, messing it up for everybody. So it takes a hero to fix things that cell phone idiot breaks, but eventually the hero can’t keep up, and management puts in a stop light. Lots of rules, lots of forced latency, and very little flexibility forcing everybody else down to cell phone idiot’s level, a lowest common denominator. The strangest thing is that “stop lights” get labeled as progress. Managers pat each other on the back for the standardization, the procedural improvements, etc. when all they really did was encourage universal mediocrity by not removing the cell phone idiot from the team. Congratulations, you crippled your team and kept substandard employees! Real progress would be if teams took down the stop signs altogether, by finding and removing delays.

It’s been years since I’ve seen a yield sign, or an intersection without a sign at all. I miss them.

ShareThis

Thanks to the patient readers who submitted questions while I've been on the road for work. I'd like to post a few questions here, along with my answers. Identities of those asking questions have been preserved unless noted otherwise, as is my policy.

How does something like Sguil relate to something like OSSIM? I find that I would love to use Sguil for analysis, but it doesn’t deal with HIDS, and I feel if I run both on the same network, I am overlapping a bit of things, as well as using a bit of resources redundantly?

I see Sguil and OSSIM as different products. Sguil is primarily (and currently) an analyst console for network security monitoring. OSSIM (from what I have seen, and from what I have heard speaking directly with developers) is more of an interface to a variety of open source tools. That sounds similar but it is somewhat different. I don't see a reason why you have to choose between the two.

I think it is important to realize that although OSSIM has the term "SIM" in the name, it's really not a SIM. Most people consider a SIM to be a system that interprets logs from a variety of sources, correlates or otherwise analyzes them, and presents more intelligence information to the analyst. OSSIM doesn't really accept that much from other log sources; it relies on output from other open source tools. I am sure I am going to hear from a bunch of satisfied OSSIM users who claim I am ignorant, but my group decided not to use OSSIM because it was less SIM than we needed and too much portal to open source applications. If you want that, it's still helpful.

In your book you stated that Sguil is really used for real-time monitoring, but what happens when you are a small company, and don’t employ 24x7 staff? Does the analyst come in the next morning and work thru alerts that come thru the previous evening?

That is one model. In another model, you set Sguil to auto-categorize all alerts, and then query for something of interest. Sguil was originally built for a 24x7 SOC environment, but you don't necessarily have to use it that way.

I have been [in a new job as an analyst at a] MSSP for 3-weeks and have formed an opinion that slightly mirrors your points about MSSP's being ticket-shops; in my opinion, MSSP, and specifically the division that I am in is like a glorified and/or specialized help/service desk. We get tickets, we fix things, we close tickets, repeat, etc. This is like a help desk except instead of dealing with say desktops and servers, we are dealing with firewalls and IDS'.

I had a conversation with a friend who helped land me the job this afternoon and one of the things that he pointed out to me was that I would have to get used to the fact that our customers (government and commercial) are not interested in situational awareness or tactical traffic analyses, or NSM in general. In fact, to my company NSM is a product by [insert vendor name here]. :)

This is funny, but true. Please don't get the impression that I am complaining, I willingly chose to work for this company and am happy to have the opportunity to learn new technologies (different firewalls, different IDS') from a different perspective and within many disparate networks. It's just that I have come to the conclusion that all Information Security is NOT Information Warfare and am not sure how to cope with this. I am a packet-head and an analyst at heart, but as I have been told, our customer's do not place the same premium on understanding their traffic that I do, nor does my company by that extension because it is not a salable service.

Wow, doesn't that question just punch you in the gut? I feel your pain. MSSPs exist to make money, and differentiation by the real issue -- detecting and ejecting intruders -- doesn't appear on the balance sheet. If anyone disagrees, re-read MSSPs: What Really Matters and read near the bottom: As Bamm Visscher asks, "Is your MSSP just a 'Security Device Management Provider'?" (SDMP?)

I have anecdotal evidence from a variety of sources that many companies are taking in-house some of the security services they previously outsourced. Some are doing so because they are getting little to no value for their MSSP dollar. Others realize that almost all of the MSSPs are just SDMPs, and the customer demands someone who has a better chance understanding their business and actually improving security. Those who retain MSSPs are usually checking PCI or other regulatory boxes or not clued in to the fact most MSSPs are terrible. A very small minority is happy with their MSSP, and I can probably name the company or two providing the service. (Please don't ask for their names.) Some customers are hoping everything ends up in the cloud anyway, so security becomes someone else's problem! (Sorry!)

To specifically address your concerns -- I would do the best you can with your situation, but if you decide you really aren't happy, I would look for alternatives. Either find a MSSP that operates how you would like it to, or find a company or agency with a good in-house operation. Now that you've seen how a ticket shop operates it's easy to identify one in the future.

Do you know if there has been any progress with FreeBSD 7.0 in coupling up Snort inline with a bridge-mode FreeBSD machine? I think that this would be a match made in heaven. The last time I did research on this, it wasn't yet possible because the kernel can't handle divert sockets.

Sorry, I have not tried this recently.

Are you handling AV issues? I wanted to know if you had tied that into your IR plan and any lessons learned you might be able to share. Right now our AV is handled by the systems team but when they get an alert "IF" they look at it they typically re-run a scan or maybe some spyware tools and call it good, no traffic monitoring, no application base lining, typically my team will come along after the fact when we see traffic that falls out of spec and question what's happened recently on the box.

I have lobbied to now pull this into my team (Network Ops and Security), increase headcount, and I have an idea on how to handle it but wanted to see if you've already dealt with it.

Great question. Ideally antivirus is integrated into an overall Security Operations Center, since AV is both a detection and containment mechanism. However, AV often seems to be run by separate groups (a dedicated AV team, or the end user desktop team, or another batch of people). I recommend integrating access to the AV console into your own processes. Either formally establish a process to involve your incident responders when notified by the AV team of a situation they realize is problematic, or offer support when you observe troublesome behavior on the AV console. Preferably the AV team escalates suspected compromises to the IRT, but you may have to be a little more aggressive if you want to compensate for lack of cooperation between the teams.

I just checked the sign-up page for TCP/IP Weapons School (TWS) at Black Hat USA 2008 on 2-3 and 4-5 August 2008, at Caesars Palace, Las Vegas, NV. Apparently (according to the color coding) there are only a few seats left in the weekday class, but more seats in the weekend class. These are my last scheduled training classes in 2008.

The cost for each two-day class is now $2400 until 1 July, $2600 until 31 July, and $2900 starting 1 August. (I don't set the prices.) Register while seats are still available -- both of my sessions in Las Vegas last year sold out.

Also, there's only one week left to register for my Network Security Operations (NSO) class at Techno Security 2008 on Saturday 31 May 2008 at the Myrtle Beach Marriott Resort at Grande Dunes, a great family vacation spot.

This is the only planned offering of NSO in 2008. I'll attend the conference after the one day class. I can accommodate 25 students and each seat costs $995 for the one day class. The great news about registering for NSO is that if you sign up for the class, you get a free ticket to the entire Techno Security 2008 conference. Early registration for Techno ended 31 March 2008, so registration for the conference alone is $1295. Take my class and you get the class plus the conference for $995! In other words, if you still want to attend Techno, take my class and it's cheaper. Sounds crazy, but it's true.

If you'd like to register for my NSO class, please check out the details here and return the registration form (.pdf) to me as quickly as you can. The deadline for registration is Friday 23 May 2008, and seats are first-come-first-serve. Thank you.

For those of you who have asked, TWS is an advanced packet analysis class, while NSO teaches how to build network security operations using primarily open source tools in your enterprise.

I just finished watching a great program on my favorite channel (The History Channel) called True Caribbean Pirates. It traces the story of piracy in the Caribbean from the 16th through the early 18th centuries. I was mostly interested in learning how the great powers of the day dealt with this problem, since I blogged about modern Pirates in the Malacca Strait and 18th and 19th century pirates off the Barbary Coast.

If many modern information security practitioners had been tasked with protecting commerce in the face of piracy, they would probably have bought ever more elaborate but largely ineffective defensive measures.

Instead, the royal navies of the area decided to hunt down pirates and hang them. Sure, the pirates continued their raids for a long time, but eventually the main players (England, France, Spain, Holland) stopped warring amongst themselves and directed their offensives against the pirates.

We're not going to see any fundamental changes in information security until those we elect to protect our rights rise to the task and go on the offensive. Private companies (especially modern ones) aren't in a position to "strike back" against threats -- that's the role for the police and militaries of the world. It's time to kill some pirates, not leave "critical infrastructure protection" to the "private sector."

For related thoughts please see last year's post Taking the Fight to the Enemy Revisited.

This will show you how to configure a Movemail account in Thunderbird. The setup works by using Fetchmail to grab the email from remote mailboxes and giving it to Procmail to put into a local file on the users system. Thunderbird will then grab (move) the mail from the file to the users home dir under the .thunderbird dir. Why do this? Well, if you have a bad connection to an IMAP server that goes up and down all the time along with other POP and IMAP accounts this pulls in all your mail from all your accounts and keeps it locally....

Some years ago I ported tcpdrop to Solaris from the FreeBSD version. I did it very quickly as a proof of concept and never got round to quite getting the error handling right or worrying about Solaris 10 privileges.

After spending the required 14 seconds looking at the privileges stuff, it became pretty clear that the required privilege for using tcpdrop was PRIV_SYS_IP_CONFIG. This cannot be asserted in a non-global zone, so if you are one of the many people who have emailed me asking if it can work in a non-global zone, the answer is “no, it can’t”. Not only that, but there’s nothing I can do about it.

Also in this release, I fixed up the error messages so that they are at least correct :)

The next release will feature a manpage in man format, rather than the current mdoc one which can’t actually be formatted on Solaris. Anyone who knows an automated method to convert from mdoc to man, please shout.

Anyway, the new release is available for download, knock yourselves out.

My 15th Snort Report titled Justifying Snort has been posted. I really like this post. The staff (Crystal Ferraro) at SearchSecurity did a great job editing my original submission, cutting the text but enhancing it too. Prospective book authors should judge their publishers by the quality of the editing and copyediting/proofing staffs. From the article:

Service provider takeaway: Service providers will learn how to communicate the value of Snort to customers.

There's a good chance that as a value-added reseller (VAR) or security service provider, you believe Snort and similar tools are valuable. However, there are plenty of technical folks that believe Snort is a waste of time. The goal of this Snort Report is to help you communicate the value of Snort to those customers whose IT departments are resistant to the open source tool. Although I focus on the value of Snort, you can apply this approach to any similar product.

IDS vs. IPS

I believe the majority of objections to the value of Snort stem from the fact that it's called an intrusion detection system (IDS). Looking closely at that label, we should assume that an IDS is a "system" that "detects" "intrusions." The ultimate IDS would be 100% accurate in its ability to perform that role. A simple question flows naturally from the perception that an IDS is supposed to detect intrusions: "If you can detect intrusions, why can't you prevent them?" At first glance this question makes sense. We should prevent activity that has been 100% identified as being an intrusion.

For more please read the article. It will take 5 mins or less. Debate here is welcome.

My 14th Snort Report titled Network session data analysis with Snort and Argus has been posted. The article doesn't talk about Snort (despite the title -- not mine!) but it does discuss Argus, the network session tool developed by Carter Bullard. From the start of the article:

This edition of the Snort Report departs from the standard format by introducing a data format and data collecting tool that can work alongside Snort. The data format is session data, and the tool is Argus 3.0.

Why session data?

The Snort intrusion detection system can identify suspicious and malicious activity by inspecting network traffic. Snort makes a judgment based on its analytical capabilities and notifies the operator of its decision by generating an alert. I call the output of this collect-inspect-report process "alert data."

While this is a good and necessary methodology, it has one important flaw. In most configurations, Snort is not told to report on what it sees if the traffic in question is deemed to be "normal." One might consider this aspect of Snort to be a benefit. Why generate an alert if the traffic is "normal" and not suspicious or malicious?

No alerting system can perfectly identify all suspicious or malicious activity. In many cases it's simply not possible -- especially on a packet-by-packet basis -- to identify a packet or stream as being worthy of an operator's attention. In those cases it makes sense to keep a log of the traffic. Recording traffic or characteristics of traffic for later analysis has recently been labeled retrospective network analysis (RNA), not to be confused with Sourcefire's Real-time Network Awareness. Others call recording traffic in this manner "network forensics," but that implies a degree of care and evidence handling that exceeds the methodology I present here.
When you collect data about traffic that Snort didn't consider to be suspicious or malicious, you have the opportunity to look back (hence the term "retrospective") to see what happened during an incident. How do you know to look back? Perhaps you receive a tip from law enforcement. Maybe a client reports odd activity. Or you perform a manual investigation and realize you'd like to know as much as possible about the network traffic of a certain host. In all of these situations, Snort might not have provided any clue that something was amiss.

Despite my attention to Snort in this series, I never deploy Snort as a stand-alone tool. I always supplement Snort with additional data sources. One of the most important supplementary data sources I collect is session data.

In my 15th Snort Report, already submitted to the publisher, I explain why IDS was never "dead." You might want to hear Marty Roesch's views on the subject in this video from RSA, where he also discusses Snort 3.0.

I've been searching for a del.icio.us client for windows, and its been driving me crazy!

I download one client, it requires dotnet framework 2, I download another its requires version 1.somethingortheother! What is Microsoft thinking? Why has it become so hard to run windows applications. Who thought this was a good idea for developing desktop applications?

Heck programming applications in Python is easier. At least you can use py2exe to create an executable that will work on any windows!

From loldebian, by Amaya:

So refreshing.

People who still can laugh about themselves…

“From a cost, functionality and non-thievary perspective, open-source software isn’t just the best game in town, it’s the only game.”

How true.

I love the new features in NetBeans 6.1. Groovy support and faster startup are my favorites. But the responsiveness in the IDE is worse than in 6.0. It is so slow that I am thinking about downgrading to 6.0.
Expanding a tree, moving from tab to tab in the editor, even getting a menu displayed is often dead slow.

I am surprised I haven’t seen this mentioned in the blogs. Is it only me who have this issue? I’m running on a first generation MacBook with 2 Gb of memory. Java 5.0 naturally (Thanks for nothing, Apple).

I guess I should be looking at upgrading my laptop…

Update:
I did and the slowness was probably all to blame on the old laptop falling a part. On this new shiny MacBook Pro 6.1 rocks. Java 6, at LAST.

Several months ago, I installed the latest and greatest version of Apache web server. In addition, I installed PHP and MySQL. Well, I found that effort a little trickier to tackle on my box. Fortunately, an acquaintance recommended using XAMPP from Apache Friends.

I found XAMPP easy to install, a time saver, and to use - just download, extract and start.

Available for the following platforms:

XAMPP for Linux
The distribution for Linux systems (tested for SuSE, RedHat, Mandrake and Debian) contains: Apache, MySQL, PHP & PEAR, Perl, ProFTPD, phpMyAdmin, OpenSSL, GD, Freetype2, libjpeg, libpng, gdbm, zlib, expat, Sablotron, libxml, Ming, Webalizer, pdf class, ncurses, mod_perl, FreeTDS, gettext, mcrypt, mhash, eAccelerator, SQLite and IMAP C-Client.

XAMPP for Windows
The distribution for Windows 98, NT, 2000, 2003, XP and Vista. This version contains: Apache, MySQL, PHP + PEAR, Perl, mod_php, mod_perl, mod_ssl, OpenSSL, phpMyAdmin, Webalizer, Mercury Mail Transport System for Win32 and NetWare Systems v3.32, Ming, JpGraph, FileZilla FTP Server, mcrypt, eAccelerator, SQLite, and WEB-DAV + mod_auth_mysql.

XAMPP for Mac OS X
The distribution for Mac OS X contains: Apache, MySQL, PHP & PEAR, SQLite, Perl, ProFTPD, phpMyAdmin, OpenSSL, GD, Freetype2, libjpeg, libpng, zlib, Ming, Webalizer, mod_perl, eAccelerator, phpSQLiteAdmin.
WARNING: This version of XAMPP is still in the first steps of development. Use at you own risk!

XAMPP for Solaris
The distribution for Solaris (developed and tested with Solaris 8, tested with Solaris 9) contains: Apache, MySQL, PHP & PEAR, Perl, ProFTPD, phpMyAdmin, OpenSSL, Freetype2, libjpeg, libpng, zlib, expat, Ming, Webalizer, pdf class.
WARNING: This version of XAMPP is still in the first steps of development. Use at you own risk!

Here is the download link for XAMPP and it is free of charge.

What does a query cost? In one recent case, about a quarter-million dollars.

The story (suitably anonymized): User requirements resulted in a new feature added to a busy application. Call it Widget 16. Widget 16 was complex enough that the database side of the widget put a measurable load on the server. That alone wouldn't have been a problem, but Widget 16 turned out the be a useful widget. So useful in fact, that users clicked on Widget 16 far more often have anticipated. Complex query + lots of clicks = load problem. How big was the load problem? In this case, the cost of the query (in CPU seconds, logical reads, etc.) and the number of times the query was executed were both relevant. After measuring the per-widget load on the server and multiplying by the number of widget clicks per second, we figured that widget cost at least a quarter million dollars per year in hardware and database license costs. Especially database licensing costs.

That was an interesting number.

Obviously we want to use that number to help make the application better, faster and cheaper.

The developers - who are faced with having to balance impossible user requirements, short deadlines, long bug lists, and whiny hosting teams complaining about performance - likely will favor the former over the latter. We expect that to happen. Unfortunately if that happens too often, the hosting team is stuck with either poor performance or a large hardware and software bill. To properly prioritize the development work effort, some rational measurement must be made of the cost of re-working existing functionality to reduce load verses the value of using that same work effort to add user requested features. Calculating the cost of running a feature or widget makes the prioritization determination possible. In this case, the cost of running the query compared to the person-time required to design, code, test and deploy a solution made the decision to optimize the widget (or cache it) pretty easy to make.

DBA's already have the tools (Profiler, AWR) to determine the utilization of a feature as measured in CPU, Memory and I/O. Hosting mangers have the dollar cost of the CPU's and I/O figured out. What the DBA's and managers need to do is merge the data, format it into something readable and feed it back to the developers. Closing the loop via feedback to developers is essential.

The relevant data may vary depending on your application, but the data that almost certainly will be interesting will include:

• Number of executions per time period (second, minute, hour)
• CPU cycles per execution
• Logical and Physical I/O's per execution.

The rough approximation of CPU load on the database server will be # of executions x CPU cycles per execution. The I/O's per execution x number of executions will give you a rough estimate of SAN or disk load. Obviously you only have a finite number of CPU cycles & I/O's available each second, and those CPU's and related database licenses have dollar costs associated with them. The actual application CPU and I/O data, measured against the total CPU and I/O available from your system and the annual hardware and software cost of the system will give you an estimate of the overall cost in dollars to run the query.

Notice that I didn't mention query response time. From the point of view of the developer, response time is interesting. It's what the user will feel when they navigate the site, and it is easy to measure. From a capacity/load point of view however, response time itself doesn't indicate the back-end CPU & I/O cost to achieve the response time. If the query or stored procedure returned in 200ms, but during that 200ms it paralleled out across all your CPU's and used up all available CPU time, you'll obviously only be able to do a handful of those each second, and if you try to do more than a handful per second, your application isn't going to be happy. Or if in that 200ms, it used 200ms of CPU time, you'll only be able to execute a handful of that query per CPU each second. In other words, focusing only on response time isn't adequate because it doesn't relate back to load related bottlenecks on the application and database servers.

For those who haven't seen an AWR report, Oracle has an example here. An AWR report allows your DBA's and dev team to slice, dice sort and analyze the heck out of an application. For SQL server we built a system that runs periodic profiler traces, uploads the trace to a database, and dumps out reports similar to the Oracle AWR's.

The bottom line: In order for application developers to successfully design and build efficient, scalable applications, they need to have comprehensive performance related data. They need to be able to 'see' all the way through from the web interface to the database. If they do not have data, they cannot build efficient, scalable applications.

• Splitting VI windows

I've been maintaining my own repository(s) for years, and I've finally grown out of doing it.

My first major repository move was to merge all my CVS and Subversion repositories into a single Subversion repository. This move made me happy for a while, but from time to time the machine hosting the repository would go down, and I'd be out of Subversion access for a while. Additionally, the machine hosting this repository grants me only a small quota (500mb) and my subversion repository was occupying 10% of the space. Lastly, I couldn't be bothered to setup webdav+svn, so I couldn't grant arbitrary users (like you) proper read (and perhaps write) access.

To solve all of these problems, in part or in full, I created a new project on googlecode called 'semicomplete' for my repository. All of my projects will now live there.

I used svnsync to upload my local repository so as to keep all the change history, which took 5 hours, but was otherwise painless.

New repository: http://semicomplete.googlecode.com/

As a side bonus, Google Code Hosting allows you to publish "downloads", which means all of my releases can be put here, saving me 24 megs of used quota on the old machine. Further bonuses include an issue tracking system (so you and I can file bugs that won't get lost) and a project wiki. I don't know if I'll use the wiki yet.

Back in February, Jason Hiner, TechRepublic’s executive editor did me a favor. He posted the article, “Sanity check: The 10 best technology podcasts.” It may not seem like much to most people, but it had a huge impact on me. So many choices, I felt like a kid in a candy shop. Even today, I still refer to the list and tell everyone I can about all of the neat podcasts.

My primary interests are wireless networking and security. So when I noticed Steve Gibson (a hero of mine) had created Security Now, a podcast series with TechTV’s Leo Laporte, I had to check it out. I’m very glad I did as their podcasts cover many aspects of security that are of interest to me.

I’ve actually listened to all of the Security Now podcasts and have gleaned something new from just about every podcast. One thing I quickly learned to appreciate was the extra effort Mr. Gibson uses to explain normally overlooked details. Finally in hope of spreading the word, I thought I might point out some of the more significant podcasts.

Wi-Fi security

Episode 10: The security and privacy considerations of using non-encrypted or wireless access points at home and in public locations.

Episode 11: A detailed discussion of the lack of security from MAC address filtering, the futility of disabling SSID’s for security, and the extremely poor security offered by the first-generation WEP encryption system.

Episode 13: A discussion of Wi-Fi security, demystifying the many confusing flavors of WPA encryption and presenting several critical MUST DO tips for WPA users.

Episode 89: Review the operation of wireless network security and discuss in detail the operation of the latest attack on the increasingly insecure WEP encryption system.

VPN Series

Episodes 14, 15, 17, 18, and 19: Mr. Gibson and Mr. Laporte developed a series of podcasts pertaining to VPN theory and solutions. I felt the series was very informative, especially for road warriors who maybe trying to increase their security while using public Wi-Fi networks.

Cryptography Series

Episodes 30, 31, 33, 34, 35, and 37: This is a multi-episode discussion on how cryptography works. This series explains a complicated subject thoroughly and in a manner that allows everyone to understand crypto.

PayPal and SecurID

Episodes 103 and 119: A “must listen to” couple of podcasts if you’re a PayPal user or are thinking about using the service.

Final thoughts

I’ve personally learned several very useful tidbits from listening to the podcasts. For example, I wasn’t a big fan of PayPal, but now I am. I even have their securID key. Finally I’ve only mentioned a few of the podcasts that are available at the Security Now Web site. Check the rest out and let me know what you think.

The Internet Control Message Protocol (ICMP) is not TCP and it’s not UDP. However, ICMP is critical to the functionality of any IP network such as your corporate network or the Internet. While ICMP is required for IP network traffic redirection and pinging hosts on your LAN or WAN, ICMP can also pose a security concern. In this article, learn the basics of ICMP and how to filter it properly in the Cisco IOS.

What is ICMP?

The Internet Control Message Protocol (ICMP) is based on RFC 792 and is used to send IP network errors and diagnostic messages. I have heard of ICMP referred to as the management protocol for IP networks. That is basically true as ICMP is used to communicate things like “host unavailable” and other errors. ICMP is most well known as being used to ping a host on your network. Depending on the version of traceroute used, ICMP may or may not be used for traceroute as well.

While many of us just associate ICMP with “ping”, there is actually a lot to know about it. In fact, there are 42 types of ICMP traffic (you can view each of them at the IANA ICMP parameters site). For example, for a ping to work, your host needs to be able to send an ICMP echo (type 8); the host you are pinging needs to be able to receive the echo; that host needs to be able to send an ICMP echo reply (type 0); and your host needs to be able to receive it for your ping program to be able to respond that the host is alive (and a round trip time for that ping). Like UDP, ICMP traffic is an unreliable protocol with no guaranteed delivery.

What are the security issues with ICMP traffic?

On the typical LAN with a ‘soft core,’ ICMP traffic is typically unrestricted. Depending on the level of internal network security that you require, you may want to filter ICMP traffic on your LAN between subnets (regardless of the Internet). As ICMP traffic from a malicious attacker can be used to bring down your network, ICMP traffic needs to be strictly filtered when coming in from the Internet and, perhaps, when going out to the Internet.

ICMP traffic can be used not only to discover hosts on your network, but also to flood your network with traffic. By not restricting the type and flow of ICMP traffic from the Internet, you increase the potential for a denial of service (DoS) attack by allowing ICMP traffic to flood your network and affect service to all network traffic from the servers.

To prevent these types of attacks, there are various solutions. Commonly, ICMP traffic is filtered with a firewall. That firewall could be a Cisco PIX, ASA, or a Cisco IOS router. Now, let’s take a look at how Cisco IOS ACLs can be used to filter ICMP traffic.

How do I filter ICMP traffic?

When creating Cisco IOS ACLs, many admins start out with either:

access-list 101 deny ip …

Or

access-list 101 deny tcp …

While these may be the two most common ways to filter network traffic with Cisco IOS extended ACLs, neither of these will work to filter ICMP. Additionally, no standard access list will work for ICMP specifically.

To filter ICMP traffic, you need to use an extended access list and start with something like this:

access-list 101 deny icmp …

You can see all the ICMP filtering options that can be used with a Cisco IOS ACL by following the link.

Filtering ICMP inbound and outbound traffic both to your network and the Internet are important, but the most important of the two is to properly filter ICMP inbound to protect your network.

Protecting a network from attack isn’t as simple as adding a few network access-lists. In fact, there are entire books you can buy (like Cisco Press Network Security Technologies and Solutions); there are guides you can download (like the NSA Router Security Guide); and there are certifications you can pursue (like the Cisco CCSP). I say that because the following ICMP inbound filtering ACLs are examples of how to filter ICMP to block certain traffic - but not necessarily the only ones that w ill “secure your network.”

In the following inbound ACL filtering example, we are filtering ICMP echo, redirect, and mask-requests, while allowing other types:

Router(config)# access-list 100 deny icmp any any echo log

Router (config)# access-list 100 deny icmp any any redirect log

Router (config)# access-list 100 deny icmp any any mask-request log

Router (config)# access-list 100 permit icmp any 1.1.1.0 0.0.0.255

Of course, the ACL must be applied to your interface in the “in” direction.

You may also want to use an ICMP ACL to throttle ICMP traffic that could cause a DoS attack. Here is an example from the NSA Cisco Router Security Guide:

Router(config)# access-list 131 permit icmp any any echo

Router(config)# access-list 131 permit icmp any any echo-reply

Router(config)# interface eth0/0

Router(config-if)# rate-limit output access-group 131

16000 8000 8000 conform-action continue exceed-action drop

In this example, any ICMP echo or echo-reply traffic will be allowed until it exceeds 16K, at which point it will be dropped.

Be careful with ICMP

ICMP traffic is critical network traffic, but it can also cause security issues if used against your network by a malicious attacker. In this article, you learned what ICMP was, how it can help you, and how to filter ICMP to prevent security holes and DoS attacks.

Want to learn more about router and switch management? Automatically sign up for our free Cisco Routers and Switches newsletter, delivered each Friday!

James McLurkin's personal webpage about swarm robotics

I was interested to spot (via 3mobilebuzz) that Three is thinking about rolling out a line of home routers which, instead of connecting you to the Internet via a land line ADSL connection, instead provide your home broadband via their 3G network.

CREDIT: Electricpig

To be clear, we're not talking about a femtocell, which uses your existing landline broadband to provide a cell base station, and allows networks to extend their coverage indoors, or other places where their coverage is limited or just plain unavailable. In fact what we're talking about here is exactly the opposite of a femtocell. Such a device would have its own SIM card, and then share its 3G conneciton to the Internet over a local wired and wireless network.

This is interesting, and not an idea I've come across before. Of course, with only spotty coverage on any of the networks in the out of the way village I live in, such a device isn't going to be on my shopping list. I'd be much better off with a femtocell, and when they finally (if ever) come onto the market I'll be the first in line, no matter which of the UK networks are offering it. However as someone that does drop off grid in some weird and wonderful places from time to time, I can see the attraction of this anti-femtocell.

Of course if you have a 3G dongle, you can already share your internet connection with other devices fairly easily, at least you can if you have a Mac, I'll not speak for Windows. But it's really nice to see this wrapped in a box for ease of use...

John Resig does a nice article on Google Doctype.

Update: Here is video of Mark Pilgrim on Google Doctype:

Glassfish v3 ships with JavaDB (aka Apache Derby aka Cloudscape).
I’ll be using this for trying out Rails and JRuby, but it’s also handy
for things like authentication via JDBC Realms.

A JavaDB database is essentially a directory that only one process can access at a time. This can be Glassfish itself (an embedded database) or a standalone database process (that serves SQL clients over TCP/IP).

Both have pros and cons. I’ll take you through creating both.

option 1: standalone database server

The main benefit to running a network server process is that
it’s the only way for multiple clients to access the database simultaneously
(it’s also the only option that makes sense if you were clustering Glassfish - personally I’d use PostgreSQL in that case).

If you need to create a schema before you deploy a webapp (with NetBeans or ‘rake migrate’)
you’ll have to stop Glassfish first unless you go down this route.

hypnotoad:databases $ asadmin start-database --dbhost 127.0.0.1
Database started in Network Server mode on host 127.0.0.1 and port 1527.
Could not connect to Derby Network Server on host 127.0.0.1 port 1527.
Starting database in the background.
Log redirected to /Users/dick/Applications/glassfishv3-tp2/glassfish/databases/derby.log.
Command start-database executed successfully.

( –dbhost defaults to ‘0.0.0.0’ but this causes problems if you change IP . Stick to 127.0.0.1).

Next, create the connection pool (and associated database - see later).
It’s simplest to do this on the command line (partly due to bug 4889 ):

hypnotoad:databases $ asadmin create-jdbc-connection-pool \
--datasourceclassname=org.apache.derby.jdbc.ClientConnectionPoolDataSource\
--isconnectvalidatereq=true --validationmethod=meta-data \
--property user=GFv3:password=GFv3:databaseName=railsdb:\
connectionAttributes=\;create\\=true \
railspool
Command create-jdbc-connection-pool executed successfully.

• The ’;create=true’ option tells JavaDB to create the ‘railsdb’ database on demand
• host:port defaults to localhost:1527
• username and password can be anything, but are required

We now ‘ping’ the pool. This checks our network connection is good, and has the side-effect
of creating the ‘railsdb’ database:

hypnotoad:databases $ asadmin ping-connection-pool railspool
Command ping-connection-pool executed successfully.
hypnotoad:databases $ ls
derby.log railsdb

option 2. embed Derby in Glassfish

This is my preferred option for several reasons:

1. it saves having to run 2 JVMs
2. in development, I don’t mind stopping Glassfish
3. for production, I want webapps to create their own schema anyway
4. connection validation and authentication is no longer an issue

There’s no need to ‘start-database’ in this case – just go ahead and make the pool:

hypnotoad:glassfishv3-tp2 $ asadmin create-jdbc-connection-pool \
--datasourceclassname org.apache.derby.jdbc.EmbeddedDataSource \
--property databaseName=\$\{com.sun.aas.instanceRoot\}/databases/railsdb:\
connectionAttributes=\;create\\=true \
railspool
Command create-jdbc-connection-pool executed successfully.

• the different DataSource class is what makes it embedded
• provide a full path in the databaseName attribute to avoid current working directory hell
• since Glassfish is the database server, we can skip username,password and connection validation options

If we ping the pool, we can see Glassfish creates derby.log and the database dir

hypnotoad:glassfishv3-tp2 $ asadmin ping-connection-pool railspool
Command ping-connection-pool executed successfully.
hypnotoad:glassfishv3-tp2 $ tail derby.log
2008-05-15 11:12:48.770 GMT:
Booting Derby version The Apache Software Foundation – Apache Derby – 10.2.2.1 – (538595): instance c013800d-0119-ec48-424c-000001a39158
on database directory /Users/dick/Applications/glassfishv3-tp2/glassfish/domains/domain1/databases/railsdb
Database Class Loader started – derby.database.classpath=’‘
hypnotoad:glassfishv3-tp2 $

Butler Lampsons mamma didn’t raise no fools

Whichever option you choose, the command to give the pool a JNDI name is the same:

hypnotoad:glassfishv3-tp2 $ asadmin create-jdbc-resource --connectionpoolid=railspool jdbc/railspool
Command create-jdbc-resource executed successfully.

And we’re done. Next thing to do is write a webapp to use the damn thing. Stay tuned.

How do you enable Groovy support on Netbeans 6.1 fcs?

I read somewhere that you enable the development update center. So I did and then Groovy & Grails was available. I selected it and it said it wanted to install a bunch of other modules, among them Weblogic and WebSphere support (??).

After they where installed I restarted and Netbeans didn’t start. I had to reinstall from scratch. Not fun.

aallan posted a photo:

That's Ubuntu not Ubuntu, despite the similar colour schemes. Bets on how long until one sues the other for trademark infringement?

Muscle memory is great. Are there flexible, programmable tools which let you turn a set of potentially-complex actions into something muscle-memory trainable?

I suspect making a generic tool to do this would be difficult. keynav and xdotool aim to solve some of these problems, but what about some of the more complex ones? Is it worth trying to solve these edge cases with automation? Specifically, I mean solutions where programatically you'd be talking to two or more separate systems (or APIs).

One specific set of problems is because X11's default clipboard buffer is not the same thing as GTK's clipboard buffer. So, in firefox, using 'middle click' to paste gives me X11's clipboard while CTRL+V gives me GTK (firefox)'s clipboard contents. It's likely I'm calling this thing "X11's clipboard" when it's really the "X11 Selection". It seems simple to write a tool that would copy X11's current selection to GTK's clipboard.

You could have code that looked like this, but it wouldn't be efficient:

while true:
  if gtk_clipboard_changed:
    set_x11_clipboard(value)
  else if x11_clipboard_changed:
    set_gtk_clipboard(value)

You could make that not chew up cpu by adding a small sleep at the end of each iteration, but that still sucks. From what I can tell, GTK has a way to block for clipboard changes, but X11 may not.

If the X11 application uses cut buffers, then the root X window gets notifications about cut buffer changes. However, copying stuff in firefox doesn't show any cut buffers being used.

Alternately, we could hack our own "ctrl+v" functionaly by grabbing that keystroke, or by grabbing a different, unrelated keystroke, which would do:

1. copy primary selection to clipboard
2. send literal "ctrl+v"
3. restore clipboard

Update: An existing tool will keep your selection and clipboard buffers in sync: autocutsel. Looks like it uses the sleepy-loop approach I mentioned, but it does work. Awesome!

Tutorials on embedded devices and sample projects from SparkFun Electronics

Security researcher Sebastian Muniz of Core Security Technologies will be unveiling a malicious rootkit that he developed for Cisco’s routers at the EuSecWest conference come 22 of May.

Traditionally the domain of operating systems, rootkits are essentially malware that makes extraordinary efforts to hide themselves by subverting key processes or files on a target operating system.

Excerpt from Network World:

A Cisco rootkit is particularly worrisome because, like Microsoft’s Windows, Cisco’s routers are very widely used. Cisco owned nearly two-thirds of the router market in the fourth quarter of 2007, according to IDC.

If Muniz’s claim is true, this could also mark the first time that someone is presenting a rootkit specifically written for Cisco’s proprietary Internetwork Operating System, or IOS. Unlike specific “IOS patching shellcode” exploits that are custom-written with a specific version of IOS in mind, Muniz’s rootkit is particularly virulent as it would work on several different versions of IOS.

While a method of compromising a deployed router is still required, the door is now open for the router to be tempered prior to delivery, from which it can be used to covertly monitor and subvert the device as necessary.

In case you think tempering at the supply-chain level is unlikely, I have posted a story earlier this week on an FBI investigation that recovered $3.5 million worth of fake Cisco network equipment.

How do you think Cisco will do in reaction to the development of a rootkit for the Cisco IOS?

A TechRepublic reader wrote in with the following scenario:

“My company is currently running Exchange 2003 on Windows Server 2003.
We want to do a test run of Windows Server 2008/Exchange Server 2007 running together in a test environment. I have few concerns, I do not want to add Windows Server 2008 to the domain and run into problems but I want to flexibility to duplicate our current environment and run them simultaneously. ”

There are a number of different ways that the reader could accomplish his goals. In this posting, I’ll talk about two different options the reader could take.

The physical route - Plan A

This is perhaps the most painful and most obvious option, but will provide the reader with the best comparative baseline analysis. In short, our dear reader would need to replicate at least a chunk of his infrastructure in a lab environment. This lab would be physically separate from the primary network and each server would be individually reinstalled to match the production environment as closely as possible.

This is an extremely laborious option and introduces significant potential for error. For example, how likely is it that the full Exchange environment would be appropriately replicated? So, on to Plan B.

The physical or virtual route - Plan B

In this case, Plan B is a breeze compared to Plan A. Whereas Plan A would require massive staff time and would not guarantee an identical environment, Plan B corrects both of these deficiencies. One of my favorite products of all time is PowerConvert from PlateSpin. PowerConvert promises (and delivers!) what they call “anywhere-to-anywhere conversion.” In short, PowerConvert automatically moves a server workload from any physical or virtual machine to any other physical or virtual machine. I’ve used PowerConvert to perform a number of physical-to-virtual migrations and the product has saved countless hours and perfectly replicated my servers to VMware ESX hosts. PowerConvert, however, isn’t designed solely for physical-to-virtual migrations.

The reader in this scenario could in his lab deploy a bank of servers similarly configured to the production systems. Once replicated, the reader could run the lab on a separate network and perform his product evaluations in a safe environment complete with at least some level of performance analysis. Sure, this isn’t perfect since the lab network is still isolated and not accessible by all users, but it’s still better than testing in the production environment.

If the reader isn’t that concerned with performance baselines but is instead more concerned with how easily his environment can be migrated to Windows Server 2008 with Exchange Server 2007, a virtual environment — rather than a physical one — might be an ever better option. Although Microsoft doesn’t support Exchange Server 2007 on virtual machines, it’s still an appropriate platform for testing and will provide the reader with an adequate environmental replica on which to work.

Summary

You probably gathered that the key solution to this problem lies in PowerConvert. Although I am a huge fan of the product, it’s not cheap. Pricing starts at $200 per workload converted. So, if the reader converts just two servers — his Exchange 2003 server and his Windows Server 2003 domain controller — he’s still looking at a minimum outlay of $400 plus the cost of a lab server. An ESX license is not essential, however. PowerConvert also supports Microsoft Virtual Server 2005 virtual hosts. I’ve used PowerConvert with both Virtual Server and ESX Server with excellent results. There are also other products out there from companies such as Vizioncore. VMware also produces the VMware Converter. In any case, the reader will be able to safely test his migration plans.

The /usr/ucb/from UNIX command prints out the mail header lines in your mailbox file. It shows you who the mail is from. Here is an example run for this command.

Display mail header lines in your mailbox file
# /usr/ucb/from
From root Sun Mar 16 03:15:01 2008
From root Sun Mar 23 03:15:00 2008
From root Sun Mar 30 03:15:01 2008
From root Sun Apr 6 03:15:01 2008
From root Sun Apr 13 03:15:01 2008
From esoft Thu May 15 19:50:10 2008

Display mail header for mail sent by sender
# /usr/ucb/from -s esoft
From esoft Thu May 15 19:50:10 2008

Display mail header lines for a user's mailbox file
# /usr/ucb/from soft
From soft Sun Feb 10 03:10:41 2008
From soft Sun Feb 10 03:15:01 2008
From soft Sun Feb 17 03:10:41 2008
From soft Sun Feb 17 03:15:01 2008
From soft Sun Feb 24 03:10:41 2008

As Wi-Fi standards go, 802.11n has a lot to live up to. Especially after hearing how 802.11n’s advertised throughput, security, and reliability will allow Wi-Fi to replace existing wired networks. This means 802.11n’s RF technology needs to be rock-solid, just like Ethernet cables, while facing ever-changing environmental conditions.

Initially I felt it was entirely possible. 802.11n’s new RF technology was certainly enough to take on all real-world demands, but I’m not so sure now. I’d like to explain, but before doing so I feel it’s important to really understand what challenges 802.11n technology must overcome in order to become rock-solid. To begin with, Ethernet bits flow nicely through solid amorphous materials like copper. Whereas Wi-Fi bits travel through a variety of media and environments, which can affect the following parameters:

• Received signal strength is dependent on the distance between the transmitter and receiver. Physical obstructions along the link path that absorb or disperse the RF signal also affect signal strength. Ultimately, received signal strength must exceed the receiver’s noise floor by a certain amount; otherwise, the signal cannot be processed.
• In-band RF interference comes in two flavors. The first flavor is non-802.11 RF capable devices like cordless phones or microwaves, which happen to share the same frequency band as Wi-Fi networks. The second flavor pertains to co-channel and or adjacent channel interference from other Wi-Fi networks. Both types of interference if strong enough will create sufficient RF noise to make it difficult or impossible for the receiver to distinguish between the interference and real traffic.
• Out of band RF interference is something most people don’t think about. This interference emanates from devices that are not normally considered RF transmitters. Any electromagnetic (fluorescent light) or thermal (lightning) radiation has the potential to disrupt the RF link between two Wi-Fi devices.
• Multipath interference or fading occurs when a RF signal encounters objects on its way to the receiving antenna. These objects could reflect or refract the original RF signal, creating variations that have different timing and phase characteristics. When the original RF signal and variations reach the destination antenna, that receiver usually has a difficult time trying to sort out what’s what. I went into more detail about this subject in an article named “Multipath environments and how they affect Wi-Fi propagation“.

Many people will argue that the previously mentioned types of interference exist in both wired and wireless networks. I agree, with the exception of multipath interference or fading, which is unique to RF propagation. The TRSFC crew may disagree and bring up the topic of electron or photon barrier activity in a captive medium, but that’s another topic. The simple reality is Wi-Fi networks are much more susceptible to interference than wired networks.

The fallout from poor signal quality is the re-transmission of digital traffic to meet TCP/IP requirements of error free data transmission. With sufficient errors, the connected 802.11 devices will renegotiate the transmission rate incrementally until the error count is below a set level, which dominos into lower data throughput and decreased network efficiency. The following chart graphically shows the extent of signal reduction caused by interference. I’d like to thank Ruckus Wireless for use of the chart.

Pre-802.11n solutions

Prior to 802.11n there were various methods to reduce the affects created by interference. Most helped to a limited extent and I penned an article “How to make the best of 802.11 multipath environments” that looks at the different solutions.

Now that we are on the same page as to what a RF signal has to contend with on its way to the receiving antenna, let’s proceed to the next topic. 802.11n uses RF technology based on MIMO, antenna diversity, and spatial multiplexing to help deal with the above-mentioned challenges. I’d like to take a few moments to explain the inner workings of MIMO as a prelude to pointing out why MIMO in of itself is not the definitive answer.

MIMO: Antenna diversity

Antenna diversity isn’t new to Wi-Fi technology. It’s just becoming official as part of the 802.11n standard. Wikipedia does a great job of explaining antenna diversity:

“Antenna diversity is especially effective at mitigating multipath situations. This is because multiple antennas afford a receiver several observations of the same signal. Each antenna will experience a different interference environment. Thus, if one antenna is experiencing a deep fade, it is likely that another has a sufficient signal. Collectively such a system can provide a robust link. While this is primarily seen in receiving systems (diversity reception), the analog has also proven valuable for transmitting systems (transmit diversity) as well.

Antenna diversity can be simple as “receive selection combining”. Where a multi-antenna device transmits using the same antenna from which it just successfully received digital traffic. Or as complicated as equipment using “maximum ratio combining”, which allows multiple RF signals to be sent simultaneously between two proprietary devices. The following graphs from Ruckus Wireless show the difference in signal gain between the two different approaches.

MIMO: Spatial multiplexing

If you remember, earlier in the article I mentioned that RF signals will be altered as they traverse multipath environments. Well, spatial multiplexing is counting on that. As it’s the only way a receiving 802.11n device will be able to distinguish between the different RF signals. The Ruckus Wireless chart below depicting spatial multiplexing helps explain the process. As you can see in the first graph, the signals are similar enough to make it difficult to distinguish the two, whereas the second graph depicts two uncorrelated signals.

If everything is working correctly, one 802.11n device using spatial multiplexing will transmit a unique data stream using N (number of antennas) antennas. The receiving 802.11n device with at least N antennas will then receive N unique data streams. Therefore, the link’s total throughput capacity is equal to the individual data throughput multiplied by N antennas. If interested, I went into more detail about this in the article “802.11n, MIMO, and multipath environments“.

MIMO: kind of hit or miss

Now it’s easy to see how antenna diversity and spatial multiplexing theoretically improve throughput and the reliability of Wi-Fi networks. My concern is what happens when dealing with real-world environments that are constantly changing. For example, if there isn’t enough alteration to a RF signal, the receiver using spatial multiplexing will not be able to distinguish it from the rest. Another example pertains to antenna diversity. What if it’s a bad assumption to transmit using the same antenna that worked the best for receiving? Seems to me that too much is left to chance. 802.11n networks need to be more self-determining and less reliant on the RF environment if they are going to compete with wired networks.

Smart antennas and beamforming

It took awhile but with all that background information, we can now tackle smart antenna technology. The term smart antenna in reality is a misnomer as all of the intelligent signal conditioning takes place before the RF signal gets to the appropriate set of antennas. Beamforming is the technology that does all of the hard work. The following definition is from a University of Washington website. It’s the best explanation of beamforming I’ve come across. The site even has interactive models to help explain the technology.

“Beamforming is a general signal processing technique used to control the directionality of the reception or transmission of a signal on a transducer array.

Using beamforming you can direct the majority of signal energy you transmit from a group of transducers (like audio speakers or radio antennae) in a chosen angular direction. Or you can calibrate your group of transducers when receiving signals such that you predominantly receive from a chosen angular direction.”

Beamforming isn’t new, being a key component of both radar and sonar systems for many years. Recently, telco and Wi-Fi researchers have become interested in beamforming and the ability to steer signals to where they do the most good. Ruckus Wireless is one such company and has a great deal of research expertise in beamforming. Ruckus Wireless also has been instrumental in introducing products into the Wi-Fi market that have beamforming capabilities. BeamFlex is their interpretation of beamforming and the following description comes from one of their technical articles:

“Central to BeamFlex is an agile antenna system with multiple antenna elements that can be combined in real time to offer an exponential increase in diversity order. With N number of high-gain, directional antenna elements, a BeamFlex antenna array provides 2N-1 unique radiating patterns to maxi­mize range and coverage in a home.

A Diversity Combiner composed of low cost, software-controlled circuitry allows the BeamFlex software to manage antenna combining in real time. The core of the BeamFlex software is an expert system that constantly learns the environment - the RF conditions, communicating devices, network performance and application flows.

A Path Control module selects optimum antenna combinations on a per packet basis to ensure a quality signal path to each receiving device.

The Transmission Control module sets the transmission policies including data rate and queuing strategy based on application and station knowledge. The BeamFlex software interfaces to the 802.11 MAC layer and is compatible with standard 802.11 chipsets. Residing in the host processor, it adds minimal incremental CPU load and memory utilization.”

In my research on smart antenna systems and beam forming, the Ruckus Wireless approach has surfaced as a very elegant design. It has the potential to alleviate my concerns about the inability of MIMO and spatial multiplexing to be reliable enough. The individual advantages are as follows:

• BleamFlex antenna arrays can rapidly present many different antenna configurations. Which translates into significantly different RF signal patterns that will afford spatial multiplexing technology the best opportunity of success.
• BeamFlex antenna arrays use both horizontal and vertical polarized antenna elements. Once again, to create RF signal patterns with increased diversity and ensure recognition by the 802.11n receiver using spatial multiplexing.
• BeamFlex architecture uses application-level performance parameters when making decisions on how to optimize the signal quality. Rather than information from the PHY and MAC layer that doesn’t take into account QoS or application networking requirements.

The following diagram depicts current equipment from Ruckus Wireless, which include all of the above-mentioned features.

I’m more interested in a symbiotic relationship between the BeamFlex antenna and 802.11n technology so as to have the best of both worlds. Ruckus is continuing work on this front as shown in the following diagram.

Final thoughts

I remain very optimistic about 802.11n being a disruptive technology that will alter everyone’s perception of data networks. 802.11n’s antenna diversity and spatial multiplexing are vast improvements over what’s been available in previous standards. I’m just concerned that the required reliability will not be there until additional RF signal conditioning like that offered by Ruckus Wireless is used to combat environmental variables.

You may have heard of it: I’m speaking about DSA-1571-1. Read more about it on the pages “Key Rollover“, or “SSLkeys“.

And no, I don’t put it off lightly, like tuxchick did lately, nor do I blame any Debian people or anyone else - we’re only human, after all. But think about the consequences, like Erich did.

For me, that meant for instance that with fixing my setups on my local and remote Etch systems, I had to take care not to lock myself out of my older (and not vulnerable) Sarge servers with just generating new keys. The same applies if you made keys and used them for instance in your OpenWrt (or other) routers. Or for (SSL-) certificates. Or Tor. The possibilities are endless.

It’s even an issue if you set up a new Ubuntu Hardy system with the shiny new CDs which come fresh out of Canonical’s shop - the host keys are generated before you’ll get any updates over the network!

Maybe that is why Steinar explains us the maths, why Daniel calls it the “Worst Debian day ever“, or why Steve thinks that “Fixing this will take years, probably“. And it affects half of the world, tho most end users probably won’t be thinking about the large number of servers which run their services (I bet most people still don’t know that each and every email or chat or whatever runs through Debian servers somewhere out there).

But, like Michal said, “Everything bad is good for something” - so let’s roll up our sleeves and get to work. I’m halfway through already, I hope. Let’s see if I forgot something…

So - for all the sysadmins out there: think twice, and then again. And for the end users who rely on someone else (like an ISP or some “managed hosting”) to run their stuff: ask them if they heard about DSA-1571-1.

Cross-browser, cross-platform 3D globe uses Flash.

There are times when a user of a free and open source operating system like FreeBSD or Debian GNU/Linux might encounter a warning or error message that looks something like this:

“Warning: using insecure memory!”

A likely moment might be the first time one uses GnuPG, because it’s helpful with the scary warning messages like that:

Yes, I intentionally changed settings on a (secured) machine to use “insecure” memory just to show you how such a message might look. See the sacrifices I make for my readers?

One might ask, “What does this mean? Does my computer’s RAM need a therapist?” Of course, the second question would just be a joke. I hope so, anyway. The first question is somewhat more serious, however. That and nineteen other questions, have been answered below.

What does it mean?

So-called “insecure memory” is used when something stored in memory might be written to non-volatile storage, as in the case of data being swapped out of RAM to the hard drive because too much memory is being used to keep everything in RAM.

Why is it insecure?

Data in your swap partition or swapfile (what Microsoft calls the “pagefile” in its Windows OS) is not automatically lost when you shut down your computer. Something “swapped out” of RAM might thus be retrieved later by someone who has physical access to your computer, even if it has been turned off, while something that was only stored in RAM theoretically “goes away” irretrievably when there is no longer any power maintaining the data in memory.

Why does it matter?

When working with encrypted data, you need to decrypt it at some point so that you can do something meaningful with it. Similarly, passwords should hopefully never be written to disk when you enter them at a prompt for authentication. Such things should only exist in RAM for the minimum necessary time to do what needs to be done, while the encrypted form is all that ever gets stored on disk.

Does this mean that RAM is secure and the hard drive isn’t?

Not exactly. It means that there are some very real, immediate security issues related to storing sensitive data on the hard drive and other long-term storage devices without encrypting it that do not apply to volatile memory. RAM does have other problems, however:

1. For one thing, while people imagine that turning off the power immediately clears RAM, the truth is that data can persist for several minutes — retrievable by someone with the correct skills and forensic tools.
2. Another problem arises if you use solid-state storage devices (like a USB flash drive) to extend your memory capacity (such as via MS Windows Vista’s ReadyBoost), because such storage devices are non-volatile. That means they do not automatically clear after cutting power to the computer.
3. In addition, if a malicious security cracker gains access to your computer either via the network or in person while it is on, unencrypted data in RAM may be accessible.
4. Finally, more esoteric tricks (like van Eck phreaking — perhaps a subject for another article later) can be used to access data in RAM without having direct access to the computer, either physically or over the network.

Can’t we just encrypt RAM?

Theoretically, yes. On the other hand, that wouldn’t be very useful. If all your sensitive data in RAM was encrypted, you wouldn’t be able to read it. Security can’t be perfect — but if you’re smarter about it than the guy trying to