Planet SysAdmin

Links for 2012-05-17 [del.icio.us]

2012-05-18T07:00:00+00:00

HP Communities - Logging: Opening Pandora's Box - Part 4 (Awareness... - Enterprise Business Community

My Firefox 12 extensions and addons

2012-05-17T20:15:55+00:00

My Firefox 12 extensions and addons

In light of yesterday's entry about my failed Firefox Nightly experiment and the potential that some of my extensions are the root cause of my Firefox problems, I'm going to run down the current set of Firefox extensions that I use in my main browser (updating previous discussions from the Firefox 7 era, which alarmingly was less than a year ago). This time around I'm going to group them by purpose:

Safe browsing:

NoScript to disable JavaScript for almost everything. I browse with JS blocked and only enable it selectively on sites when I have to (and almost always temporarily). I consider this more an issue of safety than of performance; I simply don't trust most JavaScript from most sites to not do things that will make me unhappy.
(NoScript also takes care of blocking Flash, Java, and so on.)
CookieSafe 3.0.5, with the actual addon here. I browse through a filtering proxy and it blocks ordinary cookies, but it can't do anything about cookies I get over HTTPS or via JavaScript. I use CookieSafe to block those (there's some more explanation here).
(For me, CookieSafe 3.1a10 has an explosive interaction with NoScript that hangs Firefox in some sort of infinite JavaScript loop, so I am still on 3.0.5 aka the 2011-12-10 version of CookieSafe.)

User interface:

All-in-One Gestures (specifically my tweaked version of it). I turn off A-i-O autoscroll because the native Firefox autoscroll is better (and has been for years). A-i-O hasn't been updated in ages but still seems to be the best, most reliable gesture extension in my brief experimentation.
(FireGestures is actively developed but the last time I tried it there was an odd bug with changing font size settings; however, that was a while back. It would be my leading alternate here.)
Update: All-in-One Gestures seems to have been a major cause of my Firefox memory bloat problems. I've now replaced it with FireGestures; see this update. I can no longer recommend it.
Status-4-Evar restores the old Firefox bottom status bar so that I can see the full display of link targets and have a useful page load status display.

Fixing annoying websites, especially Google's:

GreaseMonkey combined with the Google Link Cleanup user script to remove Google's tracking links from search results. I hate these tracking links with a burning passion for two reasons; first, I have no interest in letting Google know what search results I've followed and second, Google's tracking links screw up my history so that I can't see which search results I've already read and which are new.
Stylish combined with a number of mostly personally written styles to fix various website misdesigns. The most important is a version of this user style to disable the left option sidebar in Google searches (because I hate it and I use Google all the time). I also have Compact Google Reader in the Firefox instance I use with Google Reader, for similar reasons.
(This entry and its comments have a bunch of discussion about ways to fix Google's layout issues.)
I could probably replace my use of Stylish with more GreaseMonkey user scripts, but I started with Stylish and I prefer fixing things with CSS alterations than with JavaScript (even if the JavaScript just inserts CSS alterations). Certainly there seem to be plenty of 'fix Google stuff' GreaseMonkey user scripts, eg this one for Google Reader (which I have not tried).

Improving my life:

It's All Text! handily deals with how browsers make bad editors. The more I have it available the more I use it (and the longer comments and so on I wind up leaving, because I can actually edit them sensibly; this may not be a plus, all things considered).

Modern versions of Firefox also give you a JavaScript based PDF viewer addon for free. I have not done much with this and in fact currently have it turned off.

Of these extensions, I consider NoScript, All-in-One Gestures, GreaseMonkey, and Stylish to be completely essential. I can sort of live without the others, so as an experiment I am trying that to see if it makes a difference in Firefox memory usage and the number of zombie compartments that build up. If I am serious about this, I probably should migrate away from Stylish to GreaseMonkey for everything on the grounds that the latter is probably more actively used and maintained and so any leaks it has are more likely to get fixed promptly.

(Unfortunately I suspect that A-i-O is a likely candidate to be a leaky extension, since it hasn't been updated in ages.)

(5 comments.)

My experiment with Firefox Nightly builds: a failure

2012-05-17T20:14:08+00:00

My experiment with Firefox Nightly builds: a failure

Ever since my old Firefox build started crashing and I was forced to update to current versions, I've had serious memory issues with Firefox. I used to be able to leave Firefox running for weeks (or months) with basically stable memory usage. Now, Firefox will steadily bloat up from under a GB of resident memory at its initial steady state to, say, 1.5 GB in a few days at most. Although my current machine has 16 GB of RAM, Firefox progressively gets slower and slower as its resident memory grows; by the time it reaches around 1.5 to 1.6 GB resident the performance is visibly dragging and I have to restart.

Recently I stumbled across this Mozilla blog entry on Firefox memory usage, which discusses how current Firefox builds have changes that reduce memory leaks, especially a drastic reduction in zombie compartments (see this entry for more). Ever since I discovered the verbose about:memory information, I've noticed that I have zombie compartments that linger from my ordinary browsing; the longer I browse, the more zombie compartments build up. A Firefox change that actually dropped zombie compartments seemed very promising, certainly promising enough to build a current version of Firefox and see what happened.

(Thus this is not quite an experiment with the literal Nightly builds, although it should be very close; as far as I understand, they're built from the same source repository (see also) that I was using.)

Unfortunately, the experiment turned out to be mostly a failure, although a sort of interesting one; in some ways Firefox improved but in other ways it got significantly worse. I tweeted a cryptic short form version, and I feel like elaborating on it now.

What improved was Firefox's responsiveness as its resident memory grew. Firefox 12 visibly starts slowing down with as little as 1.2 or 1.3 GB of resident memory; the current Firefox code was still running almost as well as at start when it reached 2 GB or more of resident memory, and it might have kept going even as it bloated more. What did not improve was everything else. I still saw zombie compartments (probably just as many as before) and if anything Firefox memory usage grew faster than under Firefox 12, reaching 2 GB resident in a day or two. But the worse thing was that at home, Firefox would soon get into a state where it was constantly using CPU (apparently talking with the X server). In this state it would not shut down gracefully; I could quit Firefox and it would close all its windows, but the process would not exit and would continue consuming the CPU talking with the X server.

(I had to use 'kill -9' to get it to exit, and this happened more than once with builds across several days. It was also odd CPU usage; it showed clearly in top but did not affect the load average and didn't lag the X server that I could tell.)

Unclean shutdowns aren't something that I considered acceptable in this situation so I am now back to Firefox 12, memory bloat slowdown and all.

It's possible that the current Firefox codebase will improve as it marches towards release, eliminating the memory bloat and 100% CPU usage while preserving responsiveness as its memory usage grows. I could live with that and it certainly would be an improvement over the status quo. (In some ways, simply eliminating the CPU usage would be a bit of an improvement over the status quo, although I don't like Firefox consuming several GB of my RAM for no good reason.)

(Despite the result, I don't regret doing this experiment; it was worth trying and it didn't particularly explode in my face.)

Update, May 17th: It seems that most of my Nightly memory problems were probably due to a single old extension I was using. See this update.

Sidebar: dealing with this with Chrome or by disabling extensions

Chrome is not something I consider an acceptable alternative to Firefox, so switching to it is not an option.

One piece of advice the Mozilla people give about this sort of memory bloat is 'disable unnecessary addons'. Well, I don't have any of those; all of the addons I have loaded are ones that I consider either absolutely necessary (to the point where I would not browse without them) or important for how I use Firefox.

(I suppose there's one or two that I don't use very often, like It's All Text!, but it would be actively painful periodically.)

(4 comments.)

My Firefox memory bloat was mostly from All-in-One Gestures

2012-05-17T20:12:07+00:00

My Firefox memory bloat was mostly from All-in-One Gestures

It's time for an update to my prior Firefox situation (one, two). After some experimentation it's become clear that most of my Firefox problems with constant memory growth and zombie compartments were due to my use of All-in-One Gestures (as I kind of suspected it might be). I've switched to FireGestures instead (initially as an experiment and now full time on all of my various Firefox instances on various different machines) and things have been much better; there are no zombie compartments at all and memory growth seems to have dropped significantly (although it's not clear yet if it's completely gone). And I haven't run into any problems or bugs this time around; everything has just worked the way I expected.

(A-i-O doesn't seem to have been the only problem I had; for example, it seems to be a bad idea to leave a tab or window sitting around with an embedded Youtube video. It's also not clear if Firefox Nightly behaves well for me in general because I haven't been able to leave it running for multiple days yet.)

In addition to less memory usage, FireGestures also seems to simply be more responsive and snappy than A-i-O. It certainly has more useful features, including the ability to add gestures without needing to hack the source code, a library of existing additional gestures (including the one that I wanted), and the ability to 'back up' and 'restore' your settings (which for me really means the ability to easily synchronize my gestures between multiple Firefox instances).

(See FireGesture's homepage for more information on all of this.)

So FireGestures is now one of my core extensions, replacing All-in-One Gestures in the previous list.

The one drawback of FireGestures is that it doesn't work in Firefox 3.6; my laptop is still running Fedora 14 with this Firefox release (because that's the last one with Gnome 2 instead of Gnome 3). I don't consider this a real drawback, but you may.

PS: people migrating from All-in-One Gestures to FireGestures might want to use Down-Right-Down to call up the A-i-O information display that shows all of your gestures and then save it (as an HTML page, which is what it is). You can then conveniently look at it later when you're using FireGestures.

(I am far too impatient to try to retrain years of reflexes to use the native FireGestures gestures for various actions; I just ruthlessly rewrote them to be the A-i-O gestures I'm used to.)

(One comment.)

The Go language's problem on 32-bit machines

2012-05-17T07:20:06+00:00

The Go language's problem on 32-bit machines

Recently (for my value of recently) there was somewhat of a commotion of people declaring that Go wasn't usable in production on 32-bit systems because its garbage collection was broken and it would eat all of your memory. Naturally I was interested in this and spent some time digging in to the reports and trying to understand the situation. Today I'm going to try to write down as much as I know about what's going on to get it straight in my head, which is going to involve a trip into the fun land of garbage collection.

To simplify a bit, the purpose of garbage collection is to automatically free up memory that's no longer used. The GC technique everyone starts with is reference counting but since it has various problems (including dealing with circular references) most people soon upgrade to more complex schemes based on inverting the problem: rather than noticing when something stops being used, the garbage collection system periodically finds all of the memory that's still actively used and then frees everything else. This is 'tracing garbage collection' (and garbage collectors), so called because the garbage collector 'traces' all live objects.

One deep but unsexy problem in garbage collection is how your GC system knows what fields in your objects refer to other objects and what fields are just primitive types like numbers, memory buffers, strings, or the like, and how it does this efficiently. This can be a particular issue for a system language where you probably want to have structures and objects that are as simple and dense as possible, with as little overhead from type annotations, inefficient 'boxed' representations, and so on as possible. One solution is to maintain a separate bitmap of what words in an allocated memory area are actually pointers (which the GC can then scan efficiently, and which can be set by the runtime when an object is allocated). Another solution is what gets called 'conservative garbage collection'. The fundamental idea is that in conservative GC, we are willing to over-estimate references (and thus wind up not freeing some unused memory); rather than insisting on knowing about references, the GC system simply scans through allocated memory looking anything that might be a pointer to an allocated object. If it finds one, it conservatively declares that the object is still alive and traces things from there.

Go was initially designed as a system language, although it's no longer described as one. As such, one of the tradeoffs the language designers made is that Go more or less uses conservative garbage collection, as far as I understand, at least for objects or at least memory areas that may contain pointers (some static data that's known to be pointer free may be skipped by the conservative GC). Although there's said to be the start of a more efficient word-bitmap implementation for Go objects, it's not currently usable by the GC (and may not be fully live).

(As far as I can tell from commentary, Go's garbage collector only scans Go's own memory areas; it doesn't make any attempt to scan memory used by outside libraries or code to find references to Go objects. Runtime code that passes a pointer to a Go object to an outside function is apparently required to keep the object alive inside Go, for example by hooking it into a global variable.)

The problem with conservative GC is that it over-estimates memory still in use because it finds false 'references', things that look like pointers to allocated objects that aren't actually that. There are a number of factors that make conservative GC worse:

the more of your address space is in use for language objects, the more random values can look like references to them. If half of the address space is your objects, half of all properly aligned N-bit patterns look like pointers to your objects (where N is the size of a pointer).
the smaller the address space is in general, the more of it you're going to fill up with your objects for the same amount of memory use. Two GB of objects is half of the 32-bit address space but a tiny fraction of the 64-bit address space.
the larger your individual objects are, the more memory a single 'reference' somewhere inside one will prevent from being freed.
similarly, the more other objects a single object refers to, the more memory will be held down by a single spurious reference to the top object.

Many of these factors are apparently quite bad for 32-bit Go programs that use a significant amount of memory, apparently especially for large objects and when they use objects that the garbage collector treats conservatively. They are drastically reduced on 64-bit machines, where you would generally have to be unlucky in order for the conservative GC to accidentally hold a significant amount of memory busy. However, the problem could still happen with 64-bit Go; it's just less likely.

(The general reference for this is Go language issue 909.)

At this point I have no articulate personal reactions to all of this. As a pragmatic matter I'm not exactly writing Go programs right now for various reasons (although I keep vaguely wanting to because I like Go in the abstract), so if I'm being honest it's all kind of theoretical.

(My problem with Go in practice is partly that I have nothing to really use it on. I need to find a project that calls out for it instead of anything else.)

Sidebar: the 32-bit Windows issue

There's also an issue on Windows machines due to memory fragmentation (via Hacker News). When it starts, the Go runtime tries to allocate a contiguous 512 Mbyte region of virtual address space. Sometimes on Windows machines enough DLLs have loaded in enough places by this point that there isn't such a contiguous chunk of address space left any more, the allocation fails, and the Go runtime immediately exits with an error.

(In theory this sort of address space fragmentation could happen on any 32-bit OS, but apparently Windows is uniquely susceptible for various reasons.)

City Misses $1.6M in Parking Tickets Because of Computer Glitch

2012-05-16T21:16:08+00:00

Computer Glitch Forces Johnson County Motor Vehicle Offices to Close

2012-05-16T21:16:08+00:00

Computer Glitch Gave Free Education To College Students

2012-05-16T21:16:08+00:00

Computer glitch hampers Alaska deer hunt reporting

2012-05-16T21:16:08+00:00

Computer glitch means NC jobless can't collect

2012-05-16T21:16:08+00:00

Hundreds of potential jurors mistakenly head to Placer County courthouse

2012-05-16T21:16:08+00:00

NJ toddler on no-fly list was mistakenly pulled from JetBlue flight

2012-05-16T21:16:08+00:00

Risks of financial models being gamed

2012-05-16T21:16:08+00:00

Top judge: ditching software patents a "bad solution"

2012-05-16T21:16:08+00:00

Computerized prescriptions to stop fraud -- what could go wrong?

2012-05-16T21:16:08+00:00

Facebook Shares More About How It Uses Your Data

2012-05-16T21:16:08+00:00

Microsoft Funded Startup Aims to Kill BitTorrent Traffic

2012-05-16T21:16:08+00:00

Comcast Wants You to Watch Commercials

2012-05-16T21:16:08+00:00

Slick new type of "password"

2012-05-16T21:16:08+00:00

DNSCrypt for Windows released: Does DNSSEC make it obsolete?

2012-05-16T19:50:51+00:00

OpenDNS just released a Windows version of DNSCrypt, but does the implementation of the DNSSEC protocol make it obsolete? Patrick Lambert takes a look at what the critics are saying.

Bringing the Head Committee to order

2012-05-16T15:54:49+00:00

A client has just asked to move a maintenance window to some other time due to a sudden emergency. The in-house Project Managers, being the customer-service oriented people that they are, have just asked if it is possible to move the window to something less likely to get in the way. You are staring at the reply screen to the email.

Scene

A few people clustered around a table. The Herald of Duty has just departed, having delivered the demand that a decision needs to be made. A gavel sounds.

Bleeding Heart: Really? Well then. We'll move it.

Grumpasarus: For ****** sake, we had contractual a maint-en-ance win-dow for a ****** reason! ******, we took a quarter of what we COULD have! Tell 'em to ***** off and lump it. They were warned!

Schemer: Yeah, really. We told them four weeks ago and now it's a problem?

Slacker: Dude, I planned my day around that. Moving the work totally ruins my schedule

Voice of Reason: Well, it's a problem they have now and didn't know about then.

BH: Think about it! They don't know their emergencies four weeks in advance. It's not fair to them to have a problem only to have us unavailable. We owe it to them, it's the humane thing to do.

Grumpasarus: So ****** what? Not OUR problem.

Schemer: You know.... if we play our cards right we just might get a special gift from the client. Hm.

Grumpasarus: Who ****** cares?? It'll just go to the PM's not us.

VoR: I point out that the PMs have been known to be effusive in their praise when we do favors.

Slacker: But moving the window means I'll lose sleep. It no workie for me.

Schemer: You know, if we push back a little now and then give in, the chances of something special coming our way get higher.

Grumpasarus: Really? Like what?

Schemer: Clients have been known to send us things like large boxes of chocolate, cases of beer, or comps to swank restaurants.

Grumpasarus: Looks thoughtful.

Slacker: DUUUUUDE! Loss. Of. Sleep. Hello!

VoR: You can take a nap later.

Slacker: But I was going to play Skyrim all day, if I nap I won't get as much!

Schemer: Did I mention the Chocolate?

Slacker: But that's like, work and stuff. No, dude.

VoR: You'll nap less than the sleep you lost, so you'll actually get more Skyrim in.

Slacker: ... good point.

Schemer: Looks to the awaiting Page, We have come to a decision! We will push...

BH: Wait! This is the wrong course of action. You have to work with these people, and pushing back now will make you seem like you're hard to work with.

Schemer: And that means they have to buy me off. I don't see the problem with this.

BH: Then they won't tell you things.

VoR: And when they don't tell you things, you get cut out of decision loops.

Grumpasarus: For ****** sake just tell them to ****** lump it already. Or deliver the ****** chocolate. It's not like this'll get done without us.

Slacker: But that's, like, conflict and stuff. I don't like that.

Grumpasarus: ****** ****** you too? *******! Grow a ****** backbone!

Slacker: Not my job, man.

BH: Going along with this now will make us look reasonable, and willing to work around our customer's needs. Be a team player.

Grumpasarus: ****** team player. We ****** told them ****** four weeks ago that we'd ****** take the ****** system down. They can ****** well bend over and take it. We notified them! This window is in the ****** contract they ****** signed and ****** well didn't read.

VoR: I remind you again. When the notice for this outage went out, they didn't know about this problem. And secondly, the contracted window is much larger than the one we gave.

Schemer: I still say we should push back a little. This is last minute, there should be some costs.

Grumpasarus: ****** right there should be!

BH: Pushing back makes us look hard to work with.

Schemer: I really have to council against blithely rolling over on this one. We don't want to get to a spot where others can dictate our schedule. It'll mean dark-of-night for everything.

Slacker: I hate that!

Schemer: I know. And as we're East Coast and have West Coast clients, it'll mean morning outages not late-night ones.

Slacker: Duuuuude. No.

Schemer: I know. They do need to be made aware that this is not a trivial sacrifice on our part.

Slacker: Sleeeeeep! Skyrim!

BH: They already are aware. Look at the email. See? They know.

Grumpasarus: That's just ****** 'customer service' weasel-words.

Schemer: Hm, I'm not so sure. That sentence could go either way. I'm willing to be persuaded this once and see how it goes.

VoR: Fred has usually played straight with us in the past.

Grumpasarus: You're ***** serious??

VoR: Yes. Always.

Slacker: No. Uh uh. We gotta push a little. Make 'em think twice before asking to totally blow our schedule.

Grumpasarus: Better! Tell them to ****** off!

Schemer: No, we can still make some hay with this. Push a little, not a lot. Just enough.

BH: I really wish we didn't have to push back.

Grumpasarus: We don't ****** have to. We just tell them ****** no. No pushing, just ****** standing our ****** ground.

Slacker: I, uh, don't want to say no outright. It'll get us in trouble later. So kinda no?

Grumpasarus: For ****** sake, grow a ****** backbone you ****** lazy ****** ****** ******. ***** this. Stomps off in a huff.

BH: Points at the retreating Grumpasarus. That's an abstention!

VoR: I agree.

Slacker: Turns to waiting page, We have a decision....

To: Fred Gerkin
Subject: RE Pushing the maintenance window

Hi Fred,

We warned 'em, but I do understand emergencies. I had plans, but I'm pretty sure I can get them moved. The new window will be 4 to 7am. I'll let you know if I can't flex my plans.

Solving Blank or Apache 2 Test Page Problems (and more) in Plesk 10 on Linux

2012-05-16T09:53:26+00:00

This won’t be my usual “Problem, Solution, Long Story” style troubleshooting post. There are a few complexities involved that don’t allow it to fit into that template so easily.

I have a client-facing server running CentOS 5.7 and Plesk 10.3. When clients need web space, I put them on my Plesk server so they have shiny buttons to click when managing their own web space. Recently I had a series of unfortunate events cause an outage on one client.

It starts with my craving to have things standardized. All client account domain directories are in lower case. All, that is, except for one: AmazingClient. Their main domain’s vhost directory is /var/www/vhosts/AmazingClient which, in Plesk-land means that any reference to that client’s domain is always in that case. It bugs me. More than it should. When I created the client account several months ago, for some inexplicable reason, I used CamelCase in their name. One recent evening I decided to change the capitalization for their account’s main domain. Simple, right?

I did say that I’m using Plesk, did I not?

Before I go any further, I know what you might be thinking. “Domains aren’t case sensitive! What nonsense are you on about?!” They’re not case sensitive when approaching domains from a DNS perspective. However, I’m looking at this from a filesystem and Plesk user account perspective.

To change something as simple as the case of a domain’s vhost directory, one cannot merely rename it. There are many configuration files to consider as well as Plesk-specific tasks that rely on the domain’s directory not being glibly swapped out from underneath it. To change a domain’s name in Plesk, one has to go into the client’s control panel, and click on the Websites & Domains tab.

From there you will find the domain that you want to change the case of (remember, this isn’t about “domain” in the DNS sense, but rather the representation of that domain within Plesk and on the filesystem) and click on its link. From there you will come to the Host Settings page for that domain. Once on the Host Settings page, you’ll have the option to change the domain name. Here comes the trouble: you can’t change the name merely based on case. Even though Plesk sees the client domain differently in the backend based on case, in this Host Settings interface case is not taken into account. Plesk will complain that the domain already exists. You need to change the domain name to something different, then change it back to the original domain name, minus the capitalization. (Plesk FAIL #1)

In my case, I wanted to swing it from AwesomeClient.com, to awesomeclienttemp.com, and then back to awesomeclient.com (sans the capital “A” and “C”).

Tipping Over the Edge of Doom

When trying to move from AwesomeClient.com to awesomeclienttemp.com I received this error:

Internal error: [domain path] is out of webspace
Message is out of webspace
File Webspace.php
Line 334
Type PleskFatalException

After that error, the Websites & Domains tab is no longer accessible to that client account. Trying to use it receives the same “Internal Error: [domain path] is out of webspace” error.

You see, it appears that Plesk, upon requesting a domain rename, copies the domain’s existing files and then deletes the old ones. It does not perform a mere rename action (Plesk FAIL #2). This client uses quite a bit of space and it apparently maxed out their quota. I say “apparently” because, by a strict accounting for free space and quotas on the server, it should have been allowed – but just barely. Perhaps there’s more space that Plesk needs than a simple doubling of existing files. (Plesk FAIL #3?) Plesk certainly didn’t perform any kind of filesystem or account limitation checking prior to attempting the move. (Plesk FAIL #4)

The client site was still responsive; there didn’t appear to be any negative effects. I needed to investigate further, but as the night wore on I decided to postpone a thorough examination until another day.

Ask Not For Whom Your Cell Phone Tolls

Bright and early the next morning, I got a call. It was from the client.

“Our website seems to be down, so… uhh… if you could look into that…”

Super.

Nothing was being served up in response to any page requests for this domain. Apache’s error logs were showing requests for this client’s files as hitting in the default vhost root, not their own. Then, it hit me.

Plesk does not use the standard Apache configuration files. I mean, it does, but not really. It auto-generates Apache configuration files based on the information that is stored in its own customer database within MySQL. That’s why the domain was just fine the evening before, but didn’t fail until the wee hours of the morning. The configuration files had been latently generated based on the failed attempt at changing the domain account name.

Silly me… I expected there to be rollback statements in any of the SQL DML statements made to the database. I expected that a fatal error would be caught and changes rolled back. They weren’t. (Plesk FAIL #5) Silly, silly me.

Of course, I wasn’t going to be able to change the domain information because the Websites & Domains tab bombed out permanently with an internal error. I couldn’t access the officially sanctioned means of modifying the domain account. This called for some database mangling.

Let Pry Through the Portage of the Database

I logged into mysql and dumped the psa database. From there, I used grep to scour the .sql file for any mention of awesomeclienttemp. Sure enough, the bad change was recorded in the database. There were dozens of records in several tables that pointed to the bad domain. That was causing Apache configuration files to be written with bad data, among other applications. There was also mention of the original, unsullied domain. I guess not all of the SQL statements that are part and parcel of a domain change were able to be executed before the error condition was achieved. (Side note: ROLLBACK!! ROLLBACK!! ROLLBACK!!)

Solving the problem was a simple as searching for and replacing the string awesomeclienttemp with AwesomeClient. I used mysql to perform that, but it could have been done on the dump file and then imported. For those interested, I used the replace() function and performed a select statement first just to make sure that I was changing the data that I expected to. Once satisfied with the results I performed an update statement also using the replace() function. Here’s an example of changing some values in the dns_recs table of Plesk’s psa database:

mysql&gt> SELECT REPLACE(displayVal,'clienttemp', 'Client') FROM dns_recs WHERE displayVal LIKE '%clienttemp%';
+--------------------------------------------+
| REPLACE(displayVal,'clienttemp', 'Client') |
+--------------------------------------------+
| mail.AwesomeClient.com.                    |
| AwesomeClient.com.                         |
| AwesomeClient.com.                         |
| AwesomeClient.com.                         |
+--------------------------------------------+
mysql> UPDATE dns_recs SET displayVal=REPLACE(displayVal,'clienttemp', 'Client') WHERE displayVal LIKE '%clienttemp%';

With the database in a better state, there is still one more thing left to do. Plesk doesn’t dynamically look to the database for configuration information. It looks to regular files that have been dynamically generated from the database’s information. That generation happens on a schedule, but can be expedited using the httpdmng command. Specifically, I used:

/usr/local/psa/admin/bin/httpdmng --reconfigure-domain AwesomeClient.com

You could also use the –reconfigure-all option to perform a regeneration of all domain configuration files. After running httpdmng the domain was up and running.

Apache Test Page or Blank Page Problems

I glossed over some of the troubleshooting techniques I used while tracing the problem to its root. If you’re having trouble with seeing the Apache test page, then search through your httpd.conf file and make sure that your DirectoryIndex directive is set to look for all of the variants of an index.html page that you use. For example, index.html, index.htm. index.php, etc.

Furthermore, just to reiterate, check all of your vhost conf files, such as yourdomain/conf/vhost.conf (or any conf files that reside in that directory) for the DocumentRoot directive and make sure that it’s pointed to what you want it to be pointed at. Do not edit the files that are named similar to 13279881860.14852200_httpd.include. Those are auto-generated by Plesk and at worst you could cause destruction of files in your domain; at best you will have to re-edit those files every time a new one is generated.

Of course, do a dummy check to make sure that the domain you are trying to access is really resolving to the IP address of your web server. Just… do it. It takes 5 seconds and you have the outside chance of being pleasantly surprised.

The Takeaway

Plesk is rickety. If anyone has used a better control panel for client-facing servers, let me know. I’ve worked with cPanel and Plesk, but never with any of the others that I’ve listed in this giant list of web based server control panels. Most people will shout “Just don’t use a control panel!” but that’s not a terribly client friendly option. I’m not categorically against control panels when used in the correct situations. I am, however, against misbehaving control panels.

Let me know your experiences in the comments below.

Upgrade paths to System Center 2012 Operations Manager

2012-05-15T17:55:59+00:00

John Joyner explains the various upgrade paths to System Center Operations Manager 2012 for current users of SCOM 2007 R2. Find out the options before you decide if it's worth the effort right now.

Some stuff on 'time since boot' timestamps

2012-05-15T16:21:26+00:00

Some stuff on 'time since boot' timestamps

From today on Twitter:

@standaloneSA: Is it just me, or does it seem silly that the #NetFlow timestamp field for the flow references "ms since the router booted". Seems obtuse.

@thatcks: @standaloneSA It's probably easy to implement in the router and it creates an absolute ordering w/o worries about time going backwards.

In the Twitter way, this is a little bit cryptic so I'm going to elaborate on my guess here.

Suppose that routers were supposed to generate an absolute timestamp for their events instead of this relative one, for example UTC in milliseconds. This would create two problems.

First, routers would somehow need to know or acquire the correct UTC time (with millisecond resolution) and then maintain it. This is to some degree a solved problem but it adds complexity to the router. It also leads to the second problem, because a router is unlikely to boot with the correct UTC time (down to the millisecond).

The second problem is that the moment you have a system generating an absolute timestamp you need to deal with the certainty that the correct time, as the system sees it, will jump around. The router will boot will some idea of the UTC time but it's quite likely to be a bit off (remember that we're calling for millisecond accuracy here), then over time it will converge on the correct UTC time. As it does so, its version of UTC time may go forward abruptly, go backwards abruptly, or go forward more slowly than UTC time is really advancing. Backwards time jumps screw up event ordering completely, and all of the options screw up the true relative time between events; if you have two events timestamped UTC1 and UTC2, you actually have only a weak idea how long it is between them.

The valuable property that milliseconds since boot has is that it is a clear monotonic timestamp. It only ever goes forward and it goes forward at what should be a very constant rate, which means that it creates a clear order of events and a clear duration between any two events (well, for events from the same stream of monotonic timestamps). Monotonic timestamps are not a substitute for absolute time but neither is absolute time a substitute for monotonic timestamps; you really need both, which means that you need a map between them.

There are two possible places to build such a map: each device can do its own or it can be done in a central aggregator. I believe that the right answer is to do it in the central aggregator because this means that you have only a single version of absolute time, the aggregator's view (each device, aggregator included, may have a slightly different view of the current 'correct' absolute time for the reasons outlined above). Using only a single version of absolute time means that you have a single coherent map of all of the monotonic timestamps to (some) absolute time.

(Of course you need devices that generate monotonic timestamps to tell you when they reset their timestamps, eg when they boot.)

My impression is that using elapsed time since boot is actually common in a number of environments. For example, Linux kernel messages are usually reported this way these days (which has its own issues if you're trying to work backwards to roughly when in absolute time something happened).

(2 comments.)

SEC Guidance Is a Really Big Deal

2012-05-14T17:19:03+00:00

In November I wrote SEC Guidance Emphasizes Materiality for Cyber Incidents, my thoughts after reading an article by Senator Jay Rockefeller and former DHS Secretary Michael Chertoff. They explained why the CF Disclosure Guidance: Topic No. 2, Cybersecurity issued by the SEC in October is a big deal.

Since then I attended a conference on Director's and Officer's insurance in Connecticut, and spoke on a panel about that SEC guidance. During the conference I learned that the SEC guidance isn't a big deal -- it's a really big deal. We're talking a game changer, potentially on three fronts. Here's what I heard at the conference.

First, lawyers who read the language in the SEC guidance treated it as a "stop whatever you're doing and read this" moment. The lawyers I spoke to said the SEC guidance absolutely defined new reporting duties for companies, despite talk of it being merely a "clarification" or restatement of existing guidance.
Clients bombarded insurance firms asking what language they should use in their SEC disclosure documents. They asked "what are other companies saying? What should we say?" The firms noted similar boiler plate shared among clients, most of which insufficiently met the SEC's requirements.
One lawyer I spoke with said she expects the SEC to give publicly traded firms a "one year pass" before bringing enforcement actions against them for insufficiently outlining digital risk, pre- and post-breach.
Second, the SEC language will encourage shareholder lawsuits against companies by disgruntled parties who believe boards are not disclosing risks and actual breach details to investors. This will probably not be the primary cause for a suit but it will likely be one of other factors a shareholder action uses to show that a board is not fulfilling their duties to investors.
Third, the SEC language may prompt whistleblower reports from dissatisfied IT and security staff to organizations like the SEC Office of the Whistleblower. (That is a real organization!) In the seven weeks beginning with this new office's launch in August 2011, parties reported 334 tips from 37 states and 11 countries, with successful enforcement actions in up to 30% of cases.
Although it doesn't appear that this new office has paid any whisteblowers yet, it is apparently gearing up to do so. Imagine a case where security staff believes that management is not treating a breach as the staff thinks it should be treated, and decides to report the incident to the SEC -- with the possibility of a payout waiting!

Right now Congress doesn't seem to think that the SEC rules are working. Joe Menn reported in Hacked companies still not telling investors the following:

At least a half-dozen major U.S. companies whose computers have been infiltrated by cyber criminals or international spies have not admitted to the incidents despite new guidance from securities regulators urging such disclosures.

Top U.S. cybersecurity officials believe corporate hacking is widespread, and the Securities and Exchange Commission issued a lengthy "guidance" document on October 13 outlining how and when publicly traded companies should report hacking incidents and cybersecurity risk.

But with one full quarter having elapsed since the SEC request, some major companies that are known to have had significant digital security breaches have said nothing about the incidents in their regulatory filings.

Now Senator Rockefeller is taking a closer look as reported by Jennifer Martinez of Politico this week:

Senate Commerce Chairman Jay Rockefeller thinks the SEC needs to ensure hacked companies are adequately informing their investors about when they suffer a security breach or cybersecurity risk that could jeopardize their financial standing.

The West Virginia Democrat wants the full commission to issue guidance for companies — right now they only have staff-level instructions — on when they have to report cyber breaches or threats and what steps they’re taking to minimize the risks.

“It’s crucial that companies are disclosing to investors how cybersecurity risks affect their bottom lines, and what they are doing to address those risks,” Rockefeller said in a statement to POLITICO.

Rockefeller will soon introduce an amendment that calls on the SEC to issue interpretive guidance on when companies must disclose cybersecurity risks and intrusions. Staffers for the Commerce Committee are finalizing the amendment and aim to introduce it before Sen. Joe Lieberman’s (I-Conn.) cybersecurity bill goes to the floor.

This is the sort of activity that I think is going to mark a sea change in digital security over the coming years. I don't expect engineering or technical developments to have anywhere near the same level of impact as issues that involve legislators, lawyers, insurers, and financiers. Stay tuned!

Malware poses as software updates: Why the FBI is warning travelers

2012-05-14T14:55:17+00:00

Those "critical update" notices you get, especially while traveling, may not be what you think. Michael Kassner gets the low-down on this serious threat as well as the Evilgrade platform.

Happy Blogiversary Standalone SysAdmin

2012-05-14T14:47:41+00:00

Today is May 14th! Four years ago today, I wrote my very first entry on the Standalone SysAdmin blog!

In Free advice is worth what you pay for it, I wrote about the genesis of my blogging – it all started with my LiveJournal, and my friends complaining that they didn’t “get” what I was writing…so I moved to blogspot, then to my own VPS.

The blog has really come a long way from this look!

Every once in a while, someone will ask me how to blog. My answer is almost always the same, and it sounds disingenuous, but it isn’t. The answer is to write. Write. Write. Write.

My theory behind blogging is that if you start a blog, you want to eventually be successful and have lots of people read it. If you want lots of people to read it, you want to have stuff for them to read, so you need to write a lot, especially in the beginning. Even if your writing is bad (especially if it’s bad, because then you’re getting practice, too!). Even if you don’t have anything amazing to say.

Just keep writing. Build a body of work, because some day, you’re GOING to write something amazing, and people are going to wonder, “gee, I wonder what else that person has written”, but if you don’t have a body of work, they’re not going to have anything to read, and they’ll leave. So you need to write.

If you look at the early posts on Standalone SysAdmin, there are a few gems among a lot of cruft. But I kept writing anyway. Eventually I wrote better stuff, and more people read it. But it took time to build that body of work, and it took time to build a group of readers.

When I celebrated my first blogiversary, I had a bit over 500 subscribers, and I was highly impressed. Given how my writing was at that time, I’m still impressed ;-)

As it stands right now, I have over 2,700 subscribers (thank you everyone who subscribes!), and in the past six months, I’ve had around 2 million page views (!) according to CloudFlare. That is simply amazing.

I want to thank all of my readers, whether they subscribe or just visit regularly (or even just find me on a Google search). Thanks to everyone who has emailed me over the years offering support, advice, and asking for help. I’m thankful for all of the opportunities you all have given me, and I look forward to blogging for years to come.

OK, enough navel gazing, let’s get down to business.

This past year at LISA’11, I won a drawing for a free pass to the Technical Sessions at LISA’12 this year in San Diego. Now, that’s kind of silly, because as a member of the LISA Blog Team, I get free admission anyway. So what should I do with it? I guess I could put it on Ebay…

Or I could give it away! To one of you!

So here’s how it’s going to go. You have to work for your entry, but only just. Fill out the following survey for a chance to win the pass to the LISA’12 Tech Sessions pass. This doesn’t get you into training, but it does include 3 days of tech sessions, plus the hallway track and BoFs. I don’t know the value because the prices aren’t up yet, but it’s worth a lot of money.

LISA’12 is in San Diego, California from Sunday, December 9th through Friday, December 14th. The Technical Sessions (what this pass gets you into) is from Wednesady, December 12th through Friday, December 14th. It covers the conference pass only. No airfare, no hotel, no food (except that which may be provided by the conference itself). It’s literally a pass to register for the tech sessions.

(note: you do not have to be a member of LOPSA to win the contest)
Loading…

So there you go. Fill it out to throw your hat in the ring. Also, I included a thing for “don’t include me in the contest”, since some of you may not want (or need) to win a free pass.

If you have any questions, please throw them in the comments or email me directly at standalone.sysadmin@gmail.com. Thanks for 4 great years!

Use vSphere Client to determine number of paths to a datastore

2012-05-14T13:00:31+00:00

Ensuring that storage is properly provisioned is a key to a successful vSphere installation. Rickatron shows how to check the active number of paths for a datastore in this blog.

“My Neckbeard Grew Three Sizes That Day” or How I Beat a GNU tool with Perl

2012-05-14T09:32:58+00:00

(Today is another guest post from security expert Scott Pack!)

I spend a lot of time doing text based data processing. A *lot* of time. During an analysis, I often want to do things like look at ‘Top Talkers’, ‘Most Frequent Visitors’, or really anything that comprises a list of unique identifiers sorted by count. As a result, I’ve translated two actions into a series of pipes:

What’s the count of events per thingy: “ | sort | uniq -c | sort -n“
Who has been doing whatever: “ | sort -u“

This tends to work pretty well in most cases. Today, however, was not one of those cases. While attempting get a list of unique MACs I started out with a source (i.e. non-uniqued) 16GB text file with one MAC per line. This is where things got annoying. Muscle memory kicked in and since this matched Action #2, I ran the following command: cat macs_all.txt | sort -u >; macs_unique.txt

I expected it to take a few minutes, so I went back to the other things I was doing and let it go. I checked back 15 minutes later, and it was still running. Waited 5 minutes…still running. When the command had been running for 45 minutes, I got fed up and decided that I could do better. Perl, being my go to tool, came to the rescue in the form of hashes. I won’t go into gritty detail, but a Perl hash is a data structure that consists of a list of key/value pairs. Whenever you assign a value to a key it will add an entry for the key if it doesn’t exist, or update the value if it does. Since a key cannot be in the same hash multiple times, it makes for a pretty good hack to generate a unique list. This is what I ended up doing:

#!/usr/bin/perl -w
 
use strict;
 
my %unique;
 
while( my $line =  )
{
  next unless $line;
  chomp $line;
  $unique{$line} = '';
}
 
for my $key ( keys %unique )
{
  print "$key\n";
}

This worked significantly better for me. The output was not sorted, but that’s fine, I didn’t need it sorted, only unique. The timing information looked a lot better too.

packs@ node1:~&gt; time cat macs_all.txt | sort -u &gt; macs_unique.txt
 
real    181m12.417s
user    176m13.926s
sys     1m42.335s
packs@ node1:~&gt; time cat macs_all.txt | ./fast_uniq.pl &gt; macs_fast_uniqed.txt
 
real    8m9.074s
user    7m28.176s
sys     0m46.271s

The times can’t really be directly compared, since output from fast_uniq.pl isn’t actually sorted. Given the pretty substantial difference I think we can reasonably accept the fact that fast_uniq.pl is better in this use case. After seeing this, I’m tempted to add some functionality so I stop using both sort and uniq entirely.

I’m interested to hear if anyone else has done something similar or explain to me how much my code sucks.

6 Disasters Caused by Poorly Designed User Interfaces

2012-05-13T05:16:27+00:00

Never Trust a Robot

2012-05-13T05:16:27+00:00

Robot Soldiers Will Be a Reality -- and a Threat

2012-05-13T05:16:27+00:00

Automatic cars? Not so fast..

2012-05-13T05:16:27+00:00

"Fire risk: Lenovo expands recall of ThinkCentre all-in-ones"

2012-05-13T05:16:27+00:00

Disruptions: Indiscreet Photos, Glimpsed Then Gone

2012-05-13T05:16:27+00:00

USPS curtailing international lithium battery shipments... no iPads, laptops, cameras...

2012-05-13T05:16:27+00:00

Man jailed for accepting call in court

2012-05-13T05:16:27+00:00

FBI issues warning on hotel Internet connections

2012-05-13T05:16:27+00:00

".secure" TLD proposed

2012-05-13T05:16:27+00:00

More details on the .secure TLD proposal -- and why I believe it is fundamentally flawed

2012-05-13T05:16:27+00:00

Re: The Campus Tsunami

2012-05-13T05:16:27+00:00

Re: The Power of Individual Voters to Transform Their Government

2012-05-13T05:16:27+00:00

A basic step in measuring and improving network performance

2012-05-13T04:40:44+00:00

A basic step in measuring and improving network performance

There is a mistake that I have seen people make over and over again when they attempt to improve, tune, or even check network performance under unusual circumstances. Although what set me off now is this well intentioned article, I've seen the same mistake in people setting off to improve their iSCSI performance, NFS performance, and probably any number of other things that I've forgotten by now.

The mistake is skipping the most important basic step of network performance testing: the first thing you have to do is make sure that your network is working right. Before you can start tuning to improve your particular case or start measuring the effects of different circumstances, you need to know that your base case is not suffering from performance issues of its own. If you skip this step, you are building all future results on a foundation of sand and none of them are terribly meaningful.

(They may be very meaningful for you in that they improve your system's performance right now, but if your baseline performance is not up to what it should be it's quite possible that you could do better by addressing that.)

In the very old days, the correct base performance level you could expect was somewhat uncertain and variable; getting networks to run fast was challenging for various reasons. Fortunately those days have long since passed. Today we have a very simple performance measure, one valid for any hardware and OS from at least the past half decade if not longer:

Any system can saturate a gigabit link with TCP traffic.

As I've written before in passing, if you have two machines with gigabit Ethernet talking directly to each other on a single subnet you should be able to get gigabit wire rates between them (approximately 110 MBytes/sec) with simple testing tools like ttcp. If you cannot get this rate between your two test machines, something is wrong somewhere and you need to fix it before there's any point in going further.

(There are any number of places where the problem could be, but one definitely exists.)

I don't have an answer for what the expected latency should be (as measured either by ping or by some user-level testing tool), beyond that it should be negligible. Our servers range from around 150 microseconds down to 10 microseconds, but there's other traffic going on, multiple switch hops, and so on. Bulk TCP tends to smooth all of that out, which is part of why I like it for this sort of basic tests.

As a side note, a properly functioning local network has basically no packet loss whatsoever. If you see any more than a trace amount, you have a problem (which may be that your network, switches, or switch uplinks are oversaturated).

The one area today where there's real uncertainty in the proper base performance is 10G networking; we have not yet mastered the art of casually saturating 10G networks and may not for a while. If you have 10G networks you are going to have to do your own tuning and measurements of basic network performance before you start with higher level issues, and you may have to deliberately tune for your specific protocol and situation in a way that makes other performance worse.

(2 comments.)

On working from home

2012-05-12T08:23:29+00:00

Recently an ex-colleague of mine changed jobs and suggested that I write something about the pros and cons of working from home. I've thought about this subject, off and on, for a few years and frustratingly I think most of the pros and the cons are the same:

When you work from home you're working from home.

I live in a two-bedroom flat in Edinburgh. (Having just spent thousands on a new bathroom I don't expect I'll be moving any time soon. A rough budget of £40-50,000 would let me convert my attic into two/three rooms. So there is growth potential!)

In my flat I have made one of the bedrooms an office. The office contains:

A huge desk with two PCs, and two telephones on it.
Several book-cases.
A wall-mounted fan.
Very little else.

One PC is for work. One PC is for me. One phone is for work. One phone is my own.

Every working day I switch on the work router, the work phone, the work PC around 09:30. I then work, taking a lunch-break between either 12:00-13:00 or 13:00-14:00, until 18:00 at which point I switch off the work toys.

I ignore my personal PC during the working day with the exception that it is the source of my music. I can reach across and hit the appropriate multi-media keys to select Play/Pause/Next Track/Previous Track/Volume Up/Volume Down. (When thenever the work-phone rings the first ring is ignored as I scramble to hit "Mute" or "Pause"..!)

So what are some of the advantages/drawbacks? Well I'm at home. So the environment is one that I've made myself, and enjoy. The music is mine. The colour-scheme is mine. The pictures on the walls are mine. I have a Steve-loving chair. There is no soulless air-conditioning, no horrible cubicles, and no noisy people talking.

The downside? No people talking. If I didn't leave my house at lunchtime I'd speak to zero people face to face in an average working day. That took a while for me to notice, but it is not nice.

Since I work from home "the commute" takes seconds. I tend to get out of bed and wander straight to the desk. I'll work non-stop, then get dressed around lunchtime so that I can go out for lunch. Hail, Rain, Snow, or Sunshine I leave the house for lunch every single day (unless waiting for an atypical delivery). Because if I didn't I'd have no human contact. In the afternoon if it is a nice day I'll get undressed again, because I can, so why the hell not?

Providing you're focussed working from home has several advantages that I can think of - I've no qualms about setting the washing machine going before I start work knowing that I can "spare" five minutes to empty it later in the day. Similarly I've no concern about ordering (even large) items, because I know what time the postman comes, and I know I'm never going to be out and miss a delivery.

When I first started working from home I had a laptop instead of a PC and there were mornings when I worked, lazily, from my bed, or from my sofa whilst watching TV. That didn't last for long because I just didn't do a good job. I think I got away with it in the sense that I don't think people noticed, but I expect if it had lasted for longer it would have been quickly apparent. I stopped because the line was blurring between "home life" and "work life".

Having a dedicated working area is essential in keeping me focussed. I don't do "home things" when I'm in "work time" - with very rare exceptions. Yes I wander around and pace if I'm thinking, yes I make more tea and coffee than I would in a real office, and yes I might open windows, phone a friend, read my gas meter, washup dishes, or similar as I'm "making coffee". But on the whole it only works if I work when I'm working.

I could save money by using my work-internet instead of paying for personal-internet, but keeping the two links separate is another way of being focussed. I don't do "dodgy" things on the internet, on the whole (haha), but if I do I'd want to be damn sure that that was via my link and not the work-link - and having two PCs and two network links I know that is the case. There have been times when the work link has broken and I've used my personal link + openvpn to continue working, or at the very least re-join our internal chatroom and say "Internet down, brb".

As a system administrator there are times when I have to do things either early in the morning, late at night, or even during a weekend. I guess a final advantage is that this is not a struggle - providing I don't schedule such operations at times when I'm in the pub, meeting friends, or taking pictures of cute strangers, it isn't a struggle to say "I'll do this after 8pm tonight", or set the alarm early. No long walk to an office, and if I've already got food cooking for my tea I can eat it nearby whilst still configuring things and testing sites/services/machines.

So pros: I'm in my own environment, I don't worry about receiving parcels, meter-readings, and have wonderfully pleasant music all day. Ancillory bonuses are really side-effects of being in my environment: I have my good coffee, my nice cups, I can eat food I enjoy. etc, etc.

Cons: You must be dedicated. You must be focussed lest you give in to temptation and cease working for minutes/hours at a time. You lose part of your home space - I can't turn this room into a childrens bedroom, for example.

Nothing earth-shattering. I've done this for five years now, and although I was a little skeptical initially I thought "Why not?" It has worked out well and I think if I ever did need to leave my current position I'd have no hesitation about working from home in the future.

Finally it has to be said that when I've had partners in my life they've traditionally been the type to wake up later than me. I get significant brownie points for being able to wake them up around 10/11AM with a cup of hot coffee & breakfast in bed every morning. By virtue of having a separate space I can close the door and not be disturbed by them walking around.

I'm sure I've forgotten things - but as an initial pass the benefits and disadvantages of working from home are the same: You're in your own house.

ObQuote: "Explorers in the further regions of experience." - HellRaiser

The death of paging on the web

2012-05-12T06:13:34+00:00

The death of paging on the web

I've written about the problem of permanent headers and footers before (around a year ago), but I'm seeing more and more of them these days. What this confirms for me is that paging is dead on the modern web.

By this I don't mean long pages; I'm not one of those people who think that all of your content has to be 'above the fold', immediately visible as what people see (and the available evidence from actual experimentation apparently says otherwise). What I mean is getting to that content by paging, advancing in nearly full page increments (usually by hitting the space bar in your browser). Given that permanent headers or footers (or both) screw this up, and given that permanent headers and footers are increasingly popular, I can only conclude that paging isn't really used any more; otherwise, header and footer based designs would be wretched experiences and test badly (and on the modern web, people do at least do A/B tests).

Instead, I think that on the modern web everyone has scroll wheels (or some other way of scrolling, for example on tablets) and they scroll through articles and pages with them. Only an insignificant number of people still navigate with paging.

Now I'll add a personal confession here: since I started my scroll wheel mouse experiment, I've found myself increasingly scrolling web pages instead of paging them. I don't know why, but there's just something about it that feels right (and this is on pages without obnoxious headers and footers). I think that part of it is that the boundaries of things on the web page often don't align naturally with what I'd get by paging; by partially scrolling the page I can make things line up right (this is especially visible to me if the page content includes images).

(Looking back, I've had middle mouse button based scrolling in my browser for years and have used it too instead of paging. So I should have seen this one coming.)

I don't know what this means for web page design going forward, but I suspect that it means something (I also suspect that current web designers do know what it implies; I am not exactly current on the field). There have to be things you design differently if you expect almost everyone to scroll your page around so that things can catch their eye as they move past.

(I probably won't ever put a permanent header or footer on a page I design (at least not a full-width one), but that's a personal thing. Also it would have to be something awfully important to the page to deserve a permanent full-time presence in front of the viewer. My bias is that almost all headers and footers I've seen aren't that important; in fact, they're often rather presumptuous that way, which is part of the reason I dislike them.)

Lesser-known but extremely handy Linux tools

2012-05-11T21:28:58+00:00

Kristóf Kovács has a fantastic post about some lesser-known Linux tools that can really come in handy in different situations.

If you haven't tried dstat (I hadn't until I saw Kristóf's post), this is a great one to try. You can keep a running tally on various server metrics including load average, network transfer, and disk operations.

Here is some sample output:

----total-cpu-usage---- ---paging-- ---load-avg--- ------memory-usage----- -net/total- ---procs--- --io/total- ---system-- ----tcp-sockets----
usr sys idl wai hiq siq|  in   out | 1m   5m  15m | used  buff  cach  free| recv  send|run blk new| read  writ| int   csw |lis act syn tim clo
  0   0 100   0   0   0|   0     0 |0.07 0.25 0.25| 866M  249M  537M  387M|1314B  180B|  0   0   0|   0     0 |  70    80 | 13   7   0   0   5
  0   0 100   0   0   0|   0     0 |0.07 0.25 0.25| 866M  249M  537M  387M|1779B 1004B|  0   0   0|   0     0 |  84    78 | 13   7   0   0   5
  0   0 100   0   0   0|   0     0 |0.07 0.25 0.25| 866M  249M  537M  387M| 904B  362B|1.0   0 1.0|   0     0 |  75    86 | 13   9   0   0   5
  0   0 100   0   0   0|   0     0 |0.07 0.25 0.25| 866M  249M  537M  386M|2203B 1559B|  0   0   0|   0     0 | 180   127 | 13   7   0   0   5
  0   0 100   0   0   0|   0     0 |0.07 0.25 0.25| 866M  249M  537M  386M| 260B  130B|  0   0   0|   0     0 |  53    66 | 13   7   0   0   5
  0   0 100   0   0   0|   0     0 |0.07 0.25 0.25| 866M  249M  537M  387M|  52B  114B|  0   0   0|   0     0 |  54    77 | 13   7   0   0   5
  0   0 100   0   0   0|   0     0 |0.07 0.25 0.25| 866M  249M  537M  387M|2271B  872B|  0   0   0|   0     0 |  94    79 | 13   7   0   0   5
  0   0 100   0   0   0|   0     0 |0.07 0.25 0.25| 866M  249M  537M  387M|  52B  130B|  0   0   0|   0     0 |  54    74 | 13   7   0   0   5
  0   0 100   0   0   0|   0     0 |0.07 0.25 0.25| 866M  249M  537M  387M|1126B 1254B|  0   0   0|   0  24.0 |  80    87 | 13   7   0   0   5
  0   0 100   0   0   0|   0     0 |0.07 0.25 0.25| 866M  249M  537M  387M|1030B  130B|  0   0   0|   0     0 |  88    82 | 13   7   0   0   5
  0   0 100   0   0   0|   0     0 |0.06 0.24 0.25| 866M  249M  537M  387M| 578B  114B|  0   0   0|   0     0 |  53    64 | 13   7   0   0   5
  0   0 100   0   0   0|   0     0 |0.06 0.24 0.25| 866M  249M  537M  387M|1597B  890B|  0   0   0|   0     0 |  85    79 | 13   7   0   0   5
  0   0 100   0   0   0|   0     0 |0.06 0.24 0.25| 866M  249M  537M  387M| 552B  114B|  0   0   0|   0     0 |  63    77 | 13   7   0   0   5
  0   0 100   0   0   0|   0     0 |0.06 0.24 0.25| 866M  249M  537M  387M|1624B 1254B|  0   0   0|   0     0 |  81    75 | 13   7   0   0   5
  0   0 100   0   0   0|   0     0 |0.06 0.24 0.25| 866M  249M  537M  387M| 478B  114B|  0   0   0|   0     0 |  67    73 | 13   7   0   0   5
  0   0 100   0   0   0|   0     0 |0.06 0.24 0.25| 866M  249M  537M  387M| 418B  114B|  0   0   0|   0     0 |  59    74 | 13   7   0   0   5
  0   0 100   0   0   0|   0     0 |0.06 0.24 0.25| 866M  249M  537M  387M|1265B  874B|  0   0   0|   0     0 |  82    73 | 13   7   0   0   5
  0   0 100   0   0   0|   0     0 |0.06 0.24 0.25| 866M  249M  537M  387M| 758B  114B|  0   0   0|   0     0 |  60    80 | 13   7   0   0   5
  0   0 100   0   0   0|   0     0 |0.06 0.24 0.25| 866M  249M  537M  387M|1236B 1255B|  0   0   0|   0  4.00 |  93    79 | 13   7   0   0   5
  0   0 100   0   0   0|   0     0 |0.06 0.24 0.25| 866M  249M  537M  387M|  52B  130B|  0   0   0|   0     0 |  71    70 | 13   7   0   0   5
  0   0 100   0   0   0|   0     0 |0.05 0.23 0.25| 866M  249M  537M  387M| 214B  114B|  0   0   0|   0     0 |  55    73 | 13   7   0   0   5
  0   0 100   0   0   0|   0     0 |0.05 0.23 0.25| 866M  249M  537M  387M|1201B  890B|  0   0   0|   0     0 |  80    80 | 13   7   0   0   5
  0   0 100   0   0   0|   0     0 |0.05 0.23 0.25| 866M  249M  537M  387M| 108B  114B|  0   0   0|   0     0 |  53    66 | 13   7   0   0   5
  0   0 100   0   0   0|   0     0 |0.05 0.23 0.25| 866M  249M  537M  387M|1344B 1254B|  0   0   0|   0  10.0 | 119    85 | 13   7   0   0   5
  0   0 100   0   0   0|   0     0 |0.05 0.23 0.25| 866M  249M  537M  387M| 172B  130B|  0   0   0|   0  8.00 |  80    82 | 13   7   0   0   5

Learn more about dstat on Dag Wieërs' site.

Lesser-known but extremely handy Linux tools is a post from: Major Hayden's Racker Hacker blog.

Thanks for following the blog via the RSS feed. Please don't copy my posts or quote portions of them without attribution.

Restricting and Automating User Commands Through SSH and the authorized_keys File

2012-05-11T10:45:19+00:00

Previously I explored how to limit a user’s ability to runs commands with sudo. As a tangential topic, I needed to restrict the commands that a user account had access to when they connected to the server via SSH. Specifically, I needed just a few commands to be strung together and executed every time this account connected.

The mechanism that I used to do this was with the authorized_keys file. For a thorough explanation of that file, take a peak at the man page for sshd. To explain it very simply, the authorized_keys file holds the public keys of other users/systems that are allowed to connect to that machine. For example, I place my main user account’s public RSA key into the authorized_keys file on the Linux servers that I manage. When I connect to the remote servers using SSH, it checks to see if I’m who I say I am by challenging me with the public key that it has stored. The user account on my laptop uses the private key to validate itself (yes, the private key is password protected) and I am then allowed to haxor on the servers to my heart’s content.

Here’s an example of a public key:

ssh-rsa AAAB3NzaC1yc2EAAAADAQABAAABAQDclBxY7lOaolHGaogdcc9GaTQLWMcn2PK4hnQfWlJgeeGqgS66jL4XJyiR9HcgaebBW88Z2sevUxd7g25WhuuRAazfOcElEaE+h6MMPZ94gHY+x+iVAdlNKxLT/bTvCUXLEft/yZFpnknnv7jX4ChfSiII9OiAiCzuSdyHt1/1LgEHgvDIwKMzvTgImm5X/3IhtOitjJY3Q6yhKQ6LdenQtG/v+ANqKe6opDuUKc3k9hRmj7aHROxL52paQTEgEMoVLbIoZY4/yGUzmrZQU45jNqMrbXdAxG4XexZxb7bpTLu91s0DJQGx43JNXwhJVinPgxHLmfyoCSqR1WPqn8E3 testuser@testserver

The public key, when placed in a system’s authorized_keys file, can have some extra tidbits added to it that sshd honors. An SSH protocol 2 public key follows this format:

options, keytype, base64-encoded key, comment

In the above public key, you see the keytype as ‘ssh-rsa’ followed by a space, then the key itself followed by a space and finally a comment, which in this case is a username and hostname combination. That’s a helpful hint to know who this key supposedly belongs to. Notice that there are no options included in the above key, which would come before the keytype.

Some of the options that are available to be parsed by sshd include:

environment= Changes an environmental variable for the user that is on the receiving end of the connection.
from= Only allows connections that use this public key to be initiated from certain hosts. Helpful for the extremely paranoid or the very security conscious (the only difference between the two being pay grade).
no-X11-forwarding Because we don’t need users installing xorg and then browsing the web on a remote instance of Chrome.

There are plenty of other options, however the final one that I’ll mention is the most crucial to this topic: command="command"

With the command= option, you can cause a command to be run immediately upon a successful connection to a remote host. Once the command is run, the connection is closed. Notice how that works. The command is immediately run and then once the command finishes, the connection is closed. This is not something that you’d want to do to a key that is intended to be used interactively by a human.

What could this be good for? In my specific scenario, I am using a backup tool that moves all of the data to stdout which is then piped to ssh for a secure transfer to remote storage. The remote connection would normally look like this: ssh remoteuser@remoteserver ” cat > backupfile.zip” However, if I edit the authorized keys file, I can restrict the incoming ssh connection to only be allowed to use that specific command.

It’s just another layer of security to keep people from doing things that they shouldn’t be doing. Have different ways of achieving a similar goal? Any caveats you know about? Let me know in the comments.

All your servers should have Linux's magic SysRq enabled

2012-05-10T20:29:04+00:00

All your servers should have Linux's magic SysRq enabled

This is effectively another lesson learned from our recent building power shutdown. I will put it simply:

All of your servers should have magic SysRq enabled.

There are reasons to not do this on client machines (but not necessarily very good ones), but none on your servers (which certainly should have their hardware and consoles in a secure location).

What magic SysRq is good for on servers (above everything else) is giving you a last ditch chance to shut down or reboot the machine in something approaching an orderly way. I'm not just talking about if the system goes crazy, because it's also quite possible for ordinary system shutdowns to hang, especially if you're shutting down a group of systems that have complex NFS filesystem relationships and something went down out of order. If this happens and you don't have magic SysRq support available, you're plain out of luck; all you can do is pull the power and hope that nothing is going to explode because it hasn't been killed, had its data synced to disk, or whatever.

With magic SysRq you have at least a chance of doing something about this. You can force a kernel level sync, a kernel level unmount of as many filesystems as possible, and even hit processes with signals if you think it's going to do any good. And then you can reboot the machine (and afterwards, possibly pull the power to keep the machine down).

PS: you should explicitly enabled magic SysRq in your standard server install setup, even if your distribution normally defaults to leaving it on; distribution defaults can change over time. Also, note that if you have a serial console you generally need a getty listening on it in order to make magic SysRq work.

(You can check to see if magic SysRq is enabled by looking at the value of /proc/sys/kernel/sysrq; a 1 means that it is, a 0 means that it isn't.)

(2 comments.)

Installing Windows 8 Consumer Preview

2012-05-10T06:50:00+00:00

I have a fresh workstation and am running through the windows 8 installer on USB. When choosing the drive to install to, I get an error:

We couldn't create a new partition or locate an existing one

Lots of googling and I didn't find any hints for windows 8, but windows 7 has a similar error and folks pointed at diskpart nonsense to fix it. So let's do that -

At the installer, choose "Repair your computer"
Choose "troubleshoot"
Choose "advanced options"
Choose "command prompt"
Run diskpart.

In diskpart, you'll want to make sure your target drive is formatted and active.

list disk

# now pick your disk
select disk 0
clear
create partition
format fs=ntfs compress quick
active

Now reboot and try the installer again, it worked for me.

Using rsync to pull a directory tree to client machines

2012-05-10T03:54:57+00:00

Using rsync to pull a directory tree to client machines

Suppose that you have a decent sized directory tree that you want some number of clients to mirror from a master server (with the clients pulling updates instead of the master pushing them), perhaps because you've just noticed undesired NFS dependencies. Things in the directory tree are potentially sensitive (so you want access control), it's updated at random, and it's not in a giant VCS tree or something; this is your typical medium-sized ball of local stuff. The straightforward brute force approach is to use rsync with SSH; give the clients special SSH identities, put them in the server's authorized_keys, and have them run 'rsync -a --delete' (or some close variant) to pull the directory tree over. However, this has the problem that normal rsync is symmetric; if you allow a client to pull from you, you also allow a client to push to you (assuming that the server side login has write access to the directory tree, and yes let's make that assumption for now).

(You also have to set the SSH access up so that the clients can't run arbitrary commands on the server.)

Rsync's solution to this is its daemon mode, which can restricted to operate in read only mode. Normally rsync wants to be run this way as an actual daemon (listening on a port and so on), but that requires us to use rsync's weaker and harder to manage authentication, access control, and other things. I would rather continue to run daemon mode rsync over plain SSH and take advantage of all of the existing, proven SSH features for various things.

(The rsync manpage suggests hacks like binding the rsync daemon to only listen on localhost on the server and then using SSH port forwarding to give clients access to it. But those are hacks and require making various assumptions.)

How to to do this is not obvious from the documentation, so here is the setup I have come up with for doing this on both the server and the clients. First, you need an rsyncd.conf configuration file on the server. Don't use the normal /etc/rsyncd.conf; it's much more controllable to use your own in a different place. It should look something like:

use chroot = no

[somepath]
comment = Replication module
path = /some/path
read only = true
# if necessary:
uid = 0
gid = 0

(The '[somepath]' bit is what rsync calls the module name and can be anything meaningful for you; you'll need it on the client later. The comment is optional but potentially useful. You need to explicitly specify uid and gid if the server login is UID 0 for access to the directory tree and you need to keep that; otherwise rsync will drop privileges to a default UID.)

Next, you need a script on the server that will force an incoming SSH login to run rsync in daemon mode against this configuration file and do nothing else. We will set this as the command= value in the server login's authorized_keys to restrict what the incoming SSH connection from clients can do. This looks like:

#!/bin/sh
exec /usr/bin/rsync --server --daemon --config=/your/rsyncd.conf .

Note that this completely ignores any arguments that the client attempts to supply. However, this doesn't matter; as far as I can tell, the command line that the clients send will always be 'rsync --server --daemon .', regardless of what command line options and paths you use on the clients. (Certainly this is the only command line that clients seem to send for requests that you actually want to pay attention to.)

On the server, the login that you're using for this should have a .ssh/authorized_keys file with entries for the client SSH identities. These entries should all force incoming logins to run the command above and block various other activities (especially port forwarding, which could otherwise be done without command execution at all as Dan Astoorian mentioned in a comment here):

command="/your/rsyncd-shell",no-port-forwarding,no-X11-forwarding,no-agent-forwarding,no-pty [...]

A from="..." restriction is optional but potentially recommended. Even a broad one may limit the fallout from problems.

Finally, on the client you need to run rsync with all of the necessary arguments. You probably want to put this in a script:

#!/bin/sh
rsync -a --delete --rsh="/usr/bin/ssh -i /client/identity" LOGIN@MASTER-HOST::somepath /some/path/

Potentially useful additional arguments for rsync are -q and --timeout=<something>. In a production script you probably also want an option to mirror the directory tree to somewhere other than /some/path on the client.

If you run this from cron, remember to add some locking to prevent two copies from running at once. If the directory tree is large and you have enough clients, you may want to add some amount of randomization of the start times for the replication in order to keep load down on the master server.

(There may be a better way to do this with rsync; if you know of one, let me know in the comments. For various reasons we're probably not interested in doing this with any other tool, partly because we already have rsync and not the other tools. Another tool would have to be very much better than rsync to really be worth switching to.)

Infographic: How cyber-secure are public companies?

2012-05-09T16:00:20+00:00

This infographic presents some less-than-stellar numbers illustrating how well public companies are securing their web applications.

Use ExDeploy to help plan your Exchange deployment

2012-05-09T13:00:19+00:00

Scott Lowe illustrates how the planning and deployment tool, ExDeploy, helps you get ready for your Exchange 2010 upgrade.

The Downfall of a Meta Server or “Why a Mac Mini Makes a Terrible NMS Server”

2012-05-09T10:51:55+00:00

In a recent post named The Making of a Meta Server or “Why I Bought a Mac Mini as a NMS” I explained why I had chosen a brand new, 2012 Mac Mini as my NMS hardware. After two weeks of mind numbing work, I have officially declared the Mac-Mini-as-a-NMS project a failure.

The main problem surrounded Apple’s custom EFI. Apple hardware does not use a BIOS, but instead uses EFI (note: not, specifically speaking, UEFI). Or rather, it uses an ancient, bastardized version of EFI 1.1. There is a BIOS compatibility layer that allows OSs that can only communicate with a BIOS to operate on the hardware. Most notably Windows. Apple’s OS also runs on a hard disk that has been partitioned using the GPT partitioning scheme, which isn’t itself a huge deal, but you might be surprised at the anemic support for GPT boot disks in even modern operating systems.

To use the Mac Mini to boot an OS that needs BIOS compatibility and a MBR disk should be relatively easy. Right? Right!

Unless Apple is involved.

There are several things that Apple has mutated away from the EFI standard, one of them being not using the EFI system partition for anything except firmware updates. Their custom EFI implementation has the boot process (as well as some extra filesystem drivers) baked in. The whole EFI experience just never worked like I expected it to. The other trouble is that Boot Camp has been changed in OS X Lion. If you wanted to be hand held through the partitioning process and the creation of a hybrid GPT/MBR disk, you’re invited to use Boot Camp. However the latest alterations only allow media with Windows images to be accepted. You can no longer (from my ability to understand) use Boot Camp to install non-Windows OSs. Of course, it was always unsupported, but at least it was doable.

During the whole process, I used the EFI boot manager rEFIt which apparently only recently works with OS X Lion. I read more about the GPT partitioning scheme than I ever have previously. I learned more about EFI than I ever wanted to know (although all of that information will come in very handy in the near future). I hand-rolled bootable USB thumbdrives. I tweaked partition tables. I did very nearly everything I could think of except rolling my own EFI boot partition. After the hours had steadily ticked away I decided it was no longer worth it.

After countless errors concerning boot media, partition problems, and blinking cursors, I concede that the latest Mac Mini has defeated me. It has been shipped back to Amazon and I can go back to my Apple-less existence. Speaking of Amazon, I believe that they deserve some praise in this.

Amazon made the returns process easier than any return I have ever made. Anywhere. I stated that the reason I returned it was because software I had intended to use with it was not compatible. As a result of the return not being their fault, I had to pay return shipping. Within just a few clicks, Amazon created a return label. I printed it out, boxed the mini up, taped the label to the box and handed it over to the man behind the UPS Store counter. Within 15 seconds I was walking out of the store. I have the fortune of living just a few hundred miles from an Amazon return center located in the Las Vegas area so the return was processed and money credited back within two days. Thank you, Amazon. You were the only bright spot in this debacle.

I am now investigating other pieces of hardware for this project based on the recommendations of several colleagues. If you have a recommendation, share it with me and the rest of my readers in the comments below. I’ll certainly write about my second attempt at this project as it happens.

In the end, I’m not mad. The Apple wasn’t designed to do what I was asking it to do. It was my fault. My only lingering frustration is that the Mac seems to take any standard technology that it uses and twists it in new and different ways so that your familiairty with a standard becomes more of a liability than an asset. Sound like another familiar company that SysAdmins like to pick on? Then again, Apple isn’t intended to be in the business market. Let us pause and mourn the passing of the Xserve (I handed my G5 Xserve over to Best Buy for free recycling last year. So, so sad…).

Any similar experiences with an Apple product? Have you managed to wedge an alternate OS on 2012 Apple hardware? Let me know in the comments below.

Links for 2012-05-08 [del.icio.us]

2012-05-09T07:00:00+00:00

Announcement - McAfee Has Acquired Insightix | McAfee

Things I will do differently in the next building power shutdown (part 2)

2012-05-09T04:37:47+00:00

Things I will do differently in the next building power shutdown (part 2)

Back at the start of last September, we had an overnight building wide power shutdown in the building with our machine room and I wrote a lessons-learned entry in the aftermath. Well, we just had another one and apparently I didn't learn all of the lessons that I needed to learn the first time around. So here's another set of things that I've now learned.

Next time around I will:

explicitly save the previous time's checklist. If nothing else, the 'power up' portion makes a handy guide for what to do if you abruptly lose building power some day.
(I sort of did this last time, not through active planning but just because I reflexively don't delete basically any of this sort of stuff. But I should do it deliberately and put it somewhere where I can easily find it, instead of just leaving it lying around.)
Having last time's list isn't the end of the work, because things have undoubtedly changed since then. But it's a starting point and a jog to the memory.
start preparing the checklist well in advance, like more than a day beforehand. Things worked out in the end but doing things at the last moment was a bit nerve wracking.
(There's always stuff to do around here and somehow it always felt like there was plenty of time right up until it was Friday and we had a Monday night shutdown.)
update and correct the checklist immediately afterwards to cover things that we missed. My entry from last time is kind of vague; I'm sure I knew the specifics I was thinking of at the time, but I didn't write them down so they slipped away. I was able to reconstruct a few things from notes and email in the wake of last time, but others I only realized in the aftermath of this one.
add explanatory notes about why things are being done in a certain order and what the dependencies are. Especially in the bustle of trying to get everything down or up as fast as possible, it's useful to have something to jog our minds about why something is the way it is and whether or not it's that important.
(Our checklists for this sort of thing are not fixed; they're more guidelines than requirements. We deviate from them on the fly and thus it's really useful to have some indication of how flexible or rigid things are.)
if any machines are being brought down and then deliberately not being brought back up, explicitly mention this so that people don't get potentially confused about a 'missing' machine.

My entry from last time was very useful in several ways. I reread it when I was preparing our checklist for this time and it jogged my memory about several important issues; as a result our checklist for this time around was (I think) significantly better than for last time (and also noticeably longer and more verbose). This time I at least made new mistakes, which is progress that I can live with.

I will also probably try to put more explanation into the checklist the next time around. I'm sure it's possible to put too much of it in, but I don't think that's been our problem so far. In the heat of the moment we're going to skim anyways, so the thing to do is to break the checklist up into skimmable blocks with actions and things to check off and then chunks of additional explanation after them.

(In a sense a checklist like this serves two purposes at once. During the power down or power up it is mostly a catalog of actions and ordering, but beforehand it's a discussion and a rationale for what needs to be done and why. Without the logic behind it being written out explicitly, you can't have that discussion; once you have that logic written out, you might as well leave it in to jog people's memories on the spot.)

On a side note, a full power up is an interesting and useful way to find problematic dependencies that have quietly worked their way into your overall network, ones that are not so noticeable when your systems are in their normal steady state. For example, DHCP service for several of our networks now depends on our core fileserver, which means that it can only come up fairly late in the power up process. We're going to be fixing that.

(There is a chain of dependencies that made this make sense in a steady state environment.)

(One comment.)

The Duck Always Bites Twice

2012-05-08T19:25:53+00:00

These days I’m noticing myself saying more and more frequently that Duck Typing is great, except when it’s not.

An amusing issue that briefly cropped up this afternoon was when we failed to correctly negotiate a data structure inside of a Rake task. Consider the following basic task:

desc "a test task"
task :test, :glob do |t,args|
  if args[:glob].nil?
    args[:glob] = 'some default value'
  end
  puts args[:glob]
end

What kind of output would you expect would happen if you ran rake test right now? If you said nil you’d be right! That’s odd, I wonder what is going on here?

...
puts args
...

Some debugging code later… what is the output? That’s right, it’s an empty hash – {}.

You could forgive us for thinking it might behave as one. Anyway, needless to say we then tried args.class and it turns out to be a Rake::TaskArguments, which evidently decides to make the arguments immutable but in such a way that you never know about it.

What usually happens?

$ irb
irb(main):001:0> class Foo
irb(main):002:1> attr_reader :bar
irb(main):003:1> def initialize(value)
irb(main):004:2> @bar = value
irb(main):005:2> end
irb(main):006:1> end
=> nil
irb(main):007:0> f = Foo.new(5)
=> #
irb(main):008:0> f.bar
=> 5
irb(main):009:0> f.bar = 6
NoMethodError: undefined method `bar=' for #
	from (irb):9

If you’ve seen the WAT video then you know what’s coming next:

    def method_missing(sym, *args, &block)
      lookup(sym.to_sym)
    end

...

    protected

    def lookup(name)
      if @hash.has_key?(name)
        @hash[name]
      elsif ENV.has_key?(name.to_s)
        ENV[name.to_s]
      elsif ENV.has_key?(name.to_s.upcase)
        ENV[name.to_s.upcase]
      elsif @parent
        @parent.lookup(name)
      end
    end

To be fair, this is actually kinda cool. Not only can you do something like args.glob you can also do args[:pwd] or args.term or args.USERNAME.

Unfortunately it lets you do completely unexpected things as in the above example, which is handily translated into the symbol :[]= (which I like to call the Cookie Monster symbol), which doesn’t exist, returns nothing and throws away the value you attempted to assign to it. Because it is handled by method_missing, the additional value we supplied was accepted but not used, unlike any typical situation where it will cause a compile error.

Preparing for the DNSChanger Internet outage

2012-05-08T13:42:45+00:00

Alfonso Barreiro tells all you need to know to clean up the DNSChanger malware that has affected millions of users. Make sure your organization is prepared for the July 9, 2012 deadline that the...