Planet SysAdmin

July 05, 2015

Chris Siebenmann

Sysadmin use of email is often necessarily more or less interrupt driven

One of the things that people commonly do with virtual screens is to put their email client off in a separate virtual screen so they can ignore it and avoid having it interrupt them. As I mentioned when I wrote up my virtual screen usage, I don't try to cordon off email this way. Fundamentally this is because as a sysadmin, I feel my use of email is necessarily interrupt driven.

Allowing email to interrupt me certainly can derail my chain of thought when I'm coding or working on a hard problem. But at the same time it's absolutely necessary, because that email may carry news of an emergency or a high priority issue that I need to handle more or less right away. I almost never have the option of ignoring even the possibility of such things, so almost all of the time I have to allow email to interrupt me. The best I can do is contrive low distraction email monitoring so that when I'm in a flow state it distracts me as little as possible.

So I can stop there, right? No, not so fast. What this really argues is that email is a bad way of receiving high priority information like alerts. Because it mixes high priority information with much less important messages, I have to allow even unimportant things to interrupt me at least a bit just so I can figure out whether or not I can ignore them. If alerts and priority items came in through another channel, I could readily ignore email during high focus times.

(There are always going to be days where all I do is fiddle around with stuff and swat things as they come up; on those days, I'd read and handle everything right away.)

Of course the problem is that there is no good other channel today, at least for us. Oh, with work you can build such a thing (possibly outsourcing parts of it to companies who specialize in it), but there's very little in the way of a canned out of the box solution. Plus there's the problem of getting people use your new 'urgent things' channel when they have an urgent thing and of course not using it when they don't have an urgent thing (with the associated issue of having people know whether or not their issue is urgent).

(Life is likely somewhat easier if you can assume that everyone has a smartphone, perhaps by issuing them one, but that is not something that's true in our environment.)

by cks at July 05, 2015 06:52 AM

July 04, 2015

Sarah Allen

the stealthy openness of the us digital services

Fast Company’s recent article “Inside Obama’s Stealth Startup” provides a nice overview of how industry experts have been steadily joining forces to transform how the United States government is using technology to provide services to its people. One of the key elements of this strategy is open data and open source — there’s little or no stealth in this “startup.”

One of my proudest moments after I joined 18F was when we announced our open source policy. Developing in the open creates an unprecedented level of transparency and offers new potential to engage members of the public in the operation of our democracy.

Before that time, most projects from the Presidential Innovation Fellows and the new 18F team were open source, but each project required specific sign off by agency leaders for it to be open. Creating a policy dramatically streamlined this sign-off process. Working in the open saves time and money:

  • streamlines communication
  • increases code reuse
  • reduces vendor lock-in

In 2013, the Open Data Executive Order set the stage for this work. By making it so that open data was the default expectation, it means that thousands of civil servants may provide open data as part of their process, without needing to get permission for each individual data set to be published.

It’s great to see industry press starting to take notice of this transformation happening inside the US Government. We’re really just getting started. If you want to read more, check out the 18F blog

The post the stealthy openness of the us digital services appeared first on the evolving ultrasaurus.

by sarah at July 04, 2015 05:58 PM

Chris Siebenmann

Googlebot and Feedfetcher are still aggressively grabbing syndication feeds

Somewhat more than a year ago I wrote about how I'd detected Googlebot aggressively crawling my syndication feeds, despite them being marked as 'stay away'. At the time I was contacted by someone from Google about this and forwarded various information about it.

Well, you can probably guess what happened next: nothing. It is now more than a year later and Googlebot is still determinedly attempting to pound away at fetching my syndication feed. In fact it made 25 requests for it yesterday, all of which got 403s as a result of me blocking it back then. In fact Googlebot is still trying on the order of 25 times a day despite getting 403s on all of its requests for this URL for literally more than a year.

(At least it seems to be down to only trying to fetch one feed URL.)

Also, because I was looking, back what is now more than a year and a half ago I discovered that Google Feedfetcher was still fetching feeds; as a result I blocked it. Well, that's still happening too. Based on the last 30 days or so, Google Feedfetcher is making anywhere between four and ten attempts a day. And yes, that's despite getting 403s for more than a year and a half. Apparently those don't really discourage Google's crawling activities if Google really wants your data.

I'd like to say that I'm surprised, but I'm not in the least bit. Google long ago stopped caring about being a good Internet citizen, regardless of what its propaganda may say. These days the only reason to tolerate it and its behavior is because you have no choice.

(As far as I can tell it remains the 800 pound gorilla of search traffic, although various things make it much harder for me to tell these days.)

Sidebar: The grumpy crazy idea of useless random content

If I was a real crazy person, it would be awfully tempting to divert Google's feed requests to something that fed them an endless or at least very large reply. It would probably want to be machine generated valid Atom feed entries full of more or less random content. There are of course all sorts of tricks that could be played here, like embedding honeypot URLs on a special web server and seeing if Google shows up to crawl them.

I don't care enough to do this, though. I have other fish to fry in my life, even if this stuff makes me very grumpy when I wind up looking at it.

by cks at July 04, 2015 04:56 AM

July 03, 2015

Chris Siebenmann

Wandering Thoughts is now ten years old

Because I am often terrible at scheduling, Wandering Thoughts' ten year anniversary was actually almost a month ago, on June 12th (for odd reasons). And as I noted four years ago, I'm not really for anniversaries. Still, ten years is something that feels significant, enough so to produce some words.

I'm a different person than I was ten years ago and four years ago, but then we almost all are. Some of the changes are welcome ones, some less welcome, and some just are. Wandering Thoughts too has undoubtedly changed over the ten years I've been writing at least one entry a day here, but those changes are usually less obvious to me. Hopefully they are overall for the better.

(When I go back to read old entries, especially very old entries, I feel somewhat ambivalent about the changes in my writing style that I think I see. I suspect that everyone does.)

If you'd told me at the start that I would still be writing Wandering Thoughts ten years later, well, honestly I might have believed you; I'm the sort of person who gets into habits and then sticks to them unless something big comes along to jar me out. Am I happy to have done this and to still be doing this? Yes, of course, or I wouldn't be doing it any more. Writing Wandering Thoughts has enriched my life in any number of ways, both in the writing itself and in the contacts and associations I've made through the blog, and I'd be a quite different person without WT.

(Sometimes I wonder a bit about what that other me would be like. It's kind of fun but also hard; WT's effects on me feel quite pervasive.)

I don't expect to stop writing here and I probably won't change how I do it; my one entry a day habit is quite well set by now (although I sometimes think about the potential merits of taking longer to develop and write entries; writing them in an hour or two has its limitations and drawbacks).

(The next vaguely significant waypoint will be 4,000 main entries. Don't expect a marker for it, though.)

(And yes, if I think about it, ten years of an entry a day is kind of a scary thing to contemplate. I don't even try to add up the total time and effort I've put into Wandering Thoughts in the past ten years; it's far too intimidating.)

by cks at July 03, 2015 05:51 PM

Michael Biven

The Concepts of John Boyd are much more than just the OODA Loop

Ever since John Boyd’s OODA Loop was introduced to the web community at Velocity, we’ve had an oversimplified and incomplete view of his ideas. One that has been reinforced by reusing a version of the same basic diagram that he never drew.

Each time that the OODA Loop has been mentioned in discussions, blog posts, books, and presentations we continue to reinforce a misunderstanding of what Boyd was after. I’m guilty of this as well. It wasn’t until this past weekend that I found out there was more to his ideas than just a basic logical process.

John used to say if his work is going to become dogma, if its going to stop you from thinking, then right now run out and burn it.

– Chet Richards

First, Boyd never drew anything that resembled what we think of for the OODA Loop. His view was more complex. It requires orienting yourself to a situation and having the adaptability to adjust to any changes in it. Reality isn’t a static thing. It’s fluid, it’s sly, and his loop accommodates for that.

The parts we’ve been missing from his work:

  • having a common culture or training

  • analyzing and synthesizing information to prevent us from using “the same mental recipes over and over again”

  • high level of trust between leaders and subordinates

  • implicit rather than explicit authority to give subordinates the freedom to achieve the outcomes asked of them

Without a common outlook superiors cannot give subordinates freedom-of-action and maintain coherency of ongoing action.

A common outlook possessed by “a body of officers” represents a unifying theme that can be used to simultaneously encourage subordinate initiative yet realize superior intent.

– Colonel John R. Boyd, USAF

One key part missing is the basic loop includes an amount of inertia or friction that has to be overcome to take any action. To address this, Boyd describes increasing the tempo by moving from Cheng (expected/passive) to Ch’i (unexpected/active) more quickly than the basic loop allows.

He calls this “Asymmetric Fast Transients”. In what I believe is the only diagram that he drew for the OODA Loop there is a way to immediately jump from Observe to Act. He called this Implicit Guidance and Control (IG&C). By keeping orientation closer to reality and having a common culture, organizations can quickly respond to situations they recognize as they come up.

The actions taken in the IG&C are the repertoire of organizations. These are the actions that make up the muscle memory of a team, because they are repeatable and predictable.

But there is a danger in becoming stale in both your cultural view and your repertoire. While it is important to have everyone on the same page, it is equally important to promote people with unusual or unconventional thinking (like Boyd himself). This helps push back against everyone thinking the same way and increasing the confirmation bias in a team over time.

And at no point should our repertoire become nothing more than a runbook that we flip through to react to events. Instead we continue to use it when we can while at the same time thinking and considering new methods to add to it or old ones that need to be removed.

This leads us to the analysis and synthesis process that should be taking place. Which allows us to work through events that cannot sidestep the process by using the IG&C. Instead we go through the basic OODA Loop (there are others like Plan-Do-Check-Act) that we’ve known letting us engineer new possibilities into our repertoire while at the same time improving our orientation to what is happening.

Boyd said there are three ways to insult him. The first is to call him an analyst, because you’re telling him that he is a halfwit and that he has half a brain. The second is to call him an expert, because you’re then saying he has it all figured out and can’t learn anything new. The third is to call him an analytical expert. Now that’s saying that not only is he a halfwit, but he thinks he has it all figured out.

don’t try to assume that something is wrong because it doesn’t fit into your orientation”

– Colonel John R. Boyd, USAF

His admiration of the Toyota Production Systems (TPS) can be seen directly in his lectures. By flipping the process from a top down approach to a bottoms-up Toyota was able to create a chain of customers and providers within its own assembly line. The customer tells the provider what they want and when resulting in what we now call Just-In-Time (JIT) manufacturing. This reduced time of production, the amount of inventory needing to be on hand and allowed for different cars to be made on the same assembly line.

Interesting to consider new tools like Mesos as a way to use TPS concepts for web operations. One assembly line or in this case infrastructure being able to retool the resources available to make different services available quicker than before.

During the twenty years that he was actively thinking, tinkering and sharing his ideas about these concepts he was always adjusting his own orientation to what he learned. In one lecture he mentions that instead of saying no, he wants to be listening instead. To have himself in receiving mode to learn from the other person instead of letting his “own orientation drown out the other view.”

The question you have to ask yourself, is what is the purpose of the question in the first place?

– Colonel John R. Boyd, USAF

Don’t be afraid to ask the dumb questions and if you don’t understand the answer then ask again. If you don’t you can’t really orient yourself to this new view.

Boyd drew from many different resources and his experience to come to his conclusions. From the 2nd Law of Thermodynamics, Werner Heisenberg’s Uncertainty Principle, Kurt Gödel’s two Incompleteness Theorems, to Taiichi Ohno and Shigeo Shingo from Toyota. The point being he never stopped learning and he seemed willing to keep asking the dumb questions to adjust his orientation when the facts supported it.

If your knowledge of Boyd was only the OODA Loop you can find all of his writings available online (links below). It’s also worth reading Chet Richards and Chuck Spinney who both worked with Boyd. If you’re involved with the infrastructure or architecture decisions for any web service the Toyota Production System is perfectly applicable.

John R. Boyd’s A Discourse on Winning and Losing


Conceptual Spiral

Abstract of the Discourse and Conceptual Spiral

Destruction and Creation

Patters of Conflict

The Strategic Game of ? and ?

Organic Design for Command and Control

The Essence of Winning and Losing


Aerial Attack Study

New Conception for Air-to-Air Combat

Fast Transients

All sourced from Defense and the National Interest with exception for “New Conception for Air-to-Air Combat” which was sourced from Chet Richards. You can also find updated and edited versions of Boyd’s work on Richards’ site. Both he and Spinney list several writings on Boyd’s work including some of their own.

For an introduction into the Toyota Production System and why going slower can make better products (read the last article for that one).

Ohno, Taiichi (March 1998), Toyota Production System: Beyond Large-Scale Production

Taiichi Ohno (2007), Workplace Management

Shigeo Shingo (1989), A study of the Toyota Production System

Allen Ward, Jeffrey K. Liker, John J. Cristiano and Durward K. Sobek II, “The Second Toyota Paradox: How Delaying Decisions Can Make Better Cars FasterMIT Sloan Management Review, April 15, 1995

July 03, 2015 10:07 AM

July 02, 2015

Errata Security

Some notes when ordering Google's Project Fi

I just ordered my "Project Fi" phone. You probably should, too. Here are some notes (especially near the bottom on getting a new phone number).

Project Fi is Google's MVNO. An "MVNO" is a virtual mobile phone company -- they don't have any of their own network backbone or cell towers, but just rent them from the real mobile phone companies (like AT&T or T-Mobile). Most mobile phone companies are actually MVNOs, because building a physical network is expensive.

What makes Google's MVNO interesting:
  • Straightforward pricing. It's $20 a month for unlimited calling/texting, plus $10 per gigabyte of data used during the month. It includes tethering.
  • No roaming charges, in 120 countries. I can fly to Japan, Australia, and France, and still use email, Google maps, texting -- for no extra charge.
The pricing is similar to other phone companies, a little less or a little more depending on exactly what you want. For around 3 gigs a month, Project Fi is cheaper than AT&T, but for 30 gigs, it's more expensive.

There are more and more MVNOs providing easy international roaming (like, and your own phone company is increasingly solving the problem. T-Mobile, for example, provides free roaming at 2G speeds, enough to check email and maybe enough to navigate.

In-country phone calls are free, but international phone calls still cost $0.20 a minute -- unless you are on WiFi, in which case it's free. Again, this is a feature provided by other mobile phone companies and MVNOs.

In short, Google is really doing nothing new. They are just providing what you'd expect of a 21st century phone service without all the pricing shenanigans that other companies go through in order to squeeze extra money out of you.

One of the big things is which number you will use. In the United States, you can now take your phone number with you when you switch phone companies. But there are other options. For example, you can get a new phone number with the phone in order to try out the service, then switch numbers later. Or, you can switch your current number to Google Voice, and then simply forward it to the new phone. I'm choosing the third option -- using both phones for a while, and if I decide to keep my new Google phone, switch my old number over using Google Voice.

If you plan on getting a new phone number, there is a trick to it. In most areas, you'll just get a recycled phone number that was previously used by somebody else. You'll spend the next several years getting phone calls for that person. In particular, you'll get phone calls from collection agencies trying to collect money from dead beats that used to have your number. That's because people with credit problems go through a lot of phone numbers, either because they run up phone debt they can't pay, or because they deliberately change phones to avoid creditors. Consequently, on average, any recycled phone number you get will have one time been used by somebody with credit problems. Collection firms will then aggressively go through all the former numbers of a target and call you many times, sometimes in the middle of the night.

The way to fix this is to choose an area code without recycled numbers. In the United States, several new area codes are created every year for areas of the country that are growing, when they exhaust their existing area codes. Since long distance is free in the US, it doesn't really matter which area code you have anymore, so pick one of these new area codes for your number.

The way I did this with Project Fi was to first go to this website that documents new area codes. I then went to Google Voice to create a new number. I had to go about 10 area codes down the list to find one that Google Voice supports. I chose a number in that area, and to be certain, Googled it to make sure nobody had used it before. When I get my new Project Fi phone, the number will transfer over, becoming a real phone number instead of a virtual Google Voice number.

Thus, I get a virgin telephone number, albeit one from another state, rather than a recycled number that has been used by somebody else.

The main reason I'm getting a Project Fi phone is to hack it. The WiFi calling looks interesting, so I want to see how much I can mess with it, such as fuzzing the WiFi stack, or intercepting and decrypting my own communications. I suppose the Nexus 6 is necessary for the WiFi calling feature, but otherwise it should be possible to just stick the SIM in an iPhone. If anybody has any suggestions on what to play with, please tweet me @ErrataRob.

by Robert Graham ( at July 02, 2015 06:18 PM

League of Professional System Administrators

Election Results for 2015

I'm pleased to announce our new board members! Our independent monitor Andrew Hume compiled the results of the election and the report. Welcome our new and returning board members:

read more

by warner at July 02, 2015 04:14 PM


The culture-shift of moving to public cloud

Public Cloud (AWS, Azure, etc) is a very different thing than on-prem infrastructures. The low orbit view of the sector is that this is entirely intentional: create a new way of doing things to enable businesses to focus on what they're good at. A lot of high executives get that message and embrace it... until it comes time to integrate this new way with the way things have always been done. Then we get some problems.

The view from 1000ft is much different than the one from 250 miles up.

From my point of view, there are two big ways that integrating public cloud will cause culture problems.

  • Black-box infrastructure.
  • Completely different cost-model.

I've already spoken on the second point so I won't spend much time on it here. In brief: AWS costing makes you pay for what you use every month with no way to defer it for a quarter or two, which is completely not the on-prem cost model.

Black-box infrastructure

You don't know how it works.

You don't know for sure that it's being run by competent professionals who have good working habits.

You don't know for sure if they have sufficient controls in place to keep your data absolutely out of the hands of anyone but you or nosy employees. SOC reports help, but still.

You may not get console access to your instances.

You're not big enough to warrant the white glove treatment of a service contract that addresses your specific needs. Or will accept any kind of penalties for non-delivery of service.

They'll turn your account off if you defer payment for a couple of months.

The SLA they offer on the service is all you're going to get. If you need more than that... well, you'll have to figure out how to re-engineer your own software to deal with that kind of failure.

Your monitoring system doesn't know how to handle the public cloud monitoring end-points.

These are all business items that you've taken for granted in running your own datacenter, or contracting for datacenter services with another company. Service levels aren't really negotiable, this throws some enterprises. You can't simply mandate higher redundancies in certain must-always-be-up single-system services, you have to re-engineer them to be multi-system or live with the risk. As any cloud integrator will tell you if asked, public cloud requires some changes to how you think about infrastructure and that includes how you ensure it behaves the way you need it to.

Having worked for a managed services provider and a SaaS site, I've heard of the ways companies try to lever contracts as well as lazy payment of bills. If you're big enough (AWS) you can afford to lose customers by being strict about on-time payment for services. Companies that habitually defer payment on bills for a month or two in order to game quarterly results will describe such services as, 'unfriendly to my business'. Companies that expect to get into protracted SLA negotiations will find not nearly enough wiggle room, and the lack of penalties for SLA failures to be contrary to internal best practices. These are abuses that can be levered at startup and mid-size businesses, quite effectively, but not so much at the big public cloud providers.

It really does require a new way of thinking about infrastructure, at all levels. From finance, to SLAs, to application engineering, and to staffing. That's a big hill to climb.

by SysAdmin1138 at July 02, 2015 02:08 PM

Everything Sysadmin

Google's "Labs" features are DevOps Third Way

Someone on Quora recently asked, Why did Google include the 'undo send' feature on Gmail?. They felt that adding the 30-second delay to email delivery was inefficient. However rather than answering the direct question, I explained the deeper issue. My (slightly edited) answer is below. NOTE: While I previously worked at Google, I was never part of the Gmail team, nor do I even know any of their developers or the product manager(s). What I wrote here is true for any software company.

Why did Google include this feature? Because the "Gmail Labs" system permits developers to override the decisions of product managers. This is what makes the "Labs" system so brilliant.

A product manager has to decide which features to implement and which not to. This is very difficult. Each new feature takes time to design (how will it work from the user perspective), architect (how will the internals work), implement (write the code that makes it all happen), and support (documentation, and so on). There are only so many hours in the day, and only so many developers assigned to Gmail. The product manager has to say "no" to a lot of good ideas.

If you were the product manager, would you select features that are obviously going to possibly attract millions of new users, or features that help a few existing users have a slightly nicer day? Obviously you'll select the first category. IMHO Google is typically is concerned with growth, not retention. New users are more valuable than slight improvements that will help a few existing users. Many of these minor features are called "fit and finish"... little things that help make the product sparkle, but aren't things you can put in an advertisement because they have benefits that are intangible or would only be understood by a few. Many of the best features can't be appreciated or understood until they are available for use. When they are "on paper", it is difficult to judge their value.

Another reason a product manager may reject a proposed feature is politics. Maybe the idea came from someone that the product manager doesn't like, or doesn't trust. (possibly for good reason)

The "Labs" framework of Google products is a framework that let's developers add features that have been rejected by the product manager. Google engineers can, in their own spare time or in the "20% time" they are allocated, implement features that the product manager hasn't approved. "Yes, Mr Product Manager, I understand that feature x-y-z seems stupid to you, but the few people that want it would love it, so I'm going to implement it anyway and don't worry, it won't be an official feature."

The Third Way of DevOps is about creating a culture that fosters two things: continual experimentation (taking risks and learning from failure) and understanding that repetition and practice is the prerequisite to mastery. Before the Labs framework, adding any experimental feature had a huge overhead. Now most of the overhead is factored out so that there is a lower bar to experimenting. Labs-like frameworks should be added to any software product where one wants to improve their Third Way culture.

Chapter 2 of The Practice of Cloud System Administration talks about many different software features that developers should consider to assure that the system can be efficiently managed. Having a "Labs" framework enables features to be added and removed with less operational hassle because it keeps experiments isolated and easy to switch off if they cause an unexpected problem. It is much easier to temporarily disable a feature that is advertised as experimental.

What makes the "Labs" framework brilliant is that it not only gives a safe framework for experimental features to be added, but it gathers usage statistics automatically. If the feature becomes widely adopted, the developer can present hard cold data to the product manager that says the feature should be promoted to become an official feature.

Of course, the usage statistics might also show that the feature isn't well-received and prove the product manager correct.

A better way of looking at it is that the "labs" feature provides a way to democratize the feature selection process and provides a data-driven way to determine which features should be promoted to a more "official" status. The data eliminates politically-driven decision making and "I'm right because my business card lists an important title"-business as usual. This is one of the ways that Google's management is so brilliant.

I apologize for explaining this as an "us vs. them" paradigm i.e. as if the product managers and developers are at odds with each other. However, the labs feature wouldn't be needed if there wasn't some friction between the two groups. In a perfect world there would be infinite time to implement every feature requested, but we don't live in that world. (Or maybe the "Labs" feature was invented by a brilliant product manager that hated to say "no" and wanted to add an 'escape hatch' that encouraged developers to experiment. I don't know, but I'm pessimistic and believe that Labs started as an appeasement.)

So, in summary: Why did Google include the 'undo send' feature on Gmail? Because someone thought it was important, took the time to implement it under the "labs" framework, users loved the feature, and product management promoted it to be an official Gmail feature.

I wish more products had a "labs" system. The only way it could be better is if non-Googlers had a way to add features under the "labs" system too.

Hey Google, when do we get that?

July 02, 2015 01:14 PM

Steve Kemp's Blog

My new fitness challenge

So recently I posted on twitter about a sudden gain in strength:

To put that more into context I should give a few more details. In the past I've been using an assisted pull-up machine, which offers a counterweight to make such things easier.

When I started the exercise I assumed I couldn't do it for real, so I used the machine and set it on 150lb. Over a few weeks I got as far as being able to use it with only 80lb. (Which means I was lifting my entire body-weight minus 80lb. With the assisted-pullup machine smaller numbers are best!)

One evening I was walking to the cinema with my wife and told her I thought I'd be getting close to doing one real pull-up soon, which sounds a little silly, but I guess is pretty common for random men who are 40 as I almost am. As it happens there were some climbing equipment nearby so I said "Here see how close I am", and I proceeded to do 1.5 pullups. (The second one was bad, and didn't count, as I got 90% of the way "up".)

Having had that success I knew I could do "almost two", and I set a goal for the next gym visit: 3 x 3-pullups. I did that. Then I did two more for fun on the way out (couldn't quite manage a complete set.)

So that's the story of how I went from doing 1.5 pullus to doing 11 in less than a week. These days I can easily do 3x3, but struggle with more. It'll come, slowly.

So pull-up vs. chin-up? This just relates to which way you place your hands: palm facing you (chin-up) and palm way from you (pull-up).

Some technical details here but chinups are easier, and more bicep-centric.

Anyway too much writing. My next challenge is the one-armed pushup. However long it takes, and I think it will take a while, that's what I'm working toward.

July 02, 2015 08:18 AM

League of Professional System Administrators

Elections are over - results coming soon

As several folks have said, the elections are over.  As soon as the results are verified they will be posted here and in the LOPSAgram.  Thanks to all the people on the LC who worked hard to make this election possible, to the candidates that stood for election, and to all the people who voted.

by ski at July 02, 2015 04:43 AM

July 01, 2015

toolsmith: Malware Analysis with REMnux Docker Containers

Docker, runs on Ubuntu, Mac OS X, and Windows

ISSA Journal’s theme of the month is “Malware and what to do with it”. This invites so many possible smart-alecky responses, including where you can stick it, means by which to smoke it, and a variety of other abuses for the plethora of malware authors whose handy work we so enjoy each and every day of our security professional lives. But alas, that won’t get us further than a few chuckles, so I’ll just share the best summary response I’ve read to date, courtesy of @infosecjerk, and move on.
“Security is easy:
1)      Don't install malicious software.
2)      Don't click bad stuff.
3)      Only trust pretty women you don't know.
4)      Do what Gartner says.”
Wait, now I’m not sure there’s even a reason to continue here. :-)

One of the true benefits of being a SANS Internet Storm Center Handler is working with top notch security industry experts, and one such person is Lenny Zeltser. I’ve enjoyed Lenny’s work for many years; if you’ve taken SANS training you’ve either heard of or attended his GIAC Reverse Engineering Malware course and likely learned a great deal. You’re hopefully also aware of Lenny’s Linux toolkit for reverse-engineering and analyzing malware, REMnux. I covered REMnux in September 2010, but it, and the landscape, have evolved so much in the five years since. Be sure to grab the latest OVA and revisit it, if you haven’t utilized it lately. Rather than revisit REMnux specifically this month, I’ll draw your attention to a really slick way to analyze malware with Docker and specific malware-analysis related REMnux project Docker containers that Lenny’s created. Lenny expressed that he is personally interested in packaging malware analysis apps as containers because it gives him the opportunity to learn about container technologies and understand how they might be related to his work, customers and hobbies. Lenny’s packaging tools that are “useful in a malware analysis lab, that like-minded security professionals who work with malware or forensics might also find an interesting starting point for experimenting with containers and assessing their applicability to other contexts.”
Docker can be utilized on Ubuntu, Mac OS X, and Windows, I ran it on the SANS SIFT 3.0 virtual machine distribution, as well as my Mac Mini. The advantage of Docker containers, per the What Is Docker page, is simple to understand. First, “Docker allows you to package an application with all of its dependencies into a standardized unit for software development.” Everything you need therefore resides in a container: “Containers have similar resource isolation and allocation benefits as virtual machines but a different architectural approach allows them to be much more portable and efficient.” The Docker Engine is just that, the source from whom all container blessings flow. It utilizes Linux-specific kernel features so to run it on Windows and Mac OS X, it will install VirtualBox and boot2docker to create a Linux VM for the containers to run on Windows and Mac OS X. Windows Server is soon adding direct support for Docker with Windows Server Containers. In the meantime, if you’re going to go this extent, rather than just run natively on Linux, you might as well treat yourself to Kitematic, the desktop GUI for Docker. Read up on Docker before proceeding if you aren’t already well informed. Most importantly, read Security Risks and Benefits of Docker Application Containers.
Lenny mentioned that he is not planning to use containers as the architecture for the REMnux distro, stating that “This distribution has lots of useful tools installed directly on the REMnux host alongside the OS. It's fine to run most tools this way. However, I like the idea of being able to run some applications as separate containers, which is certainly possible using Docker on top of a system running the REMnux distro.” As an example, he struggled to set up Maltrieve and JSDetox directly on REMnux without introducing dependencies and settings that might break other tools but “running these applications as Docker containers allows people to have access to these handy utilities without worrying about such issues.” Lenny started the Docker image repository under the REMnux project umbrella to provide people with “the opportunity to conveniently use the tools available via the REMnux Docker repository even if they are not running REMnux.”
Before we dig in to REMnux Docker containers, I wanted to treat you to a very cool idea I’ve implemented after reading it on the SANS Digital Forensics and Incident Response Blog as posted by Lenny. He describes methods to install REMnux on a SIFT workstation, or SIFT on a REMnux workstation. I opted for the former because Docker runs really cleanly and natively on SIFT as it is Ubuntu 14.04 x64 under the hood. Installing REMnux on SIFT is as easy as wget --quiet -O - | sudo bash, then wait a bit. The script will update APT repositories (yes, we’re talking about malware analysis but no, not that APT) and install all the REMnux packages. When finished you’ll have all the power of SIFT and REMnux on one glorious workstation. By the way, if you want to use the full REMnux distribution as your Docker host, Docker is already fully installed.

Docker setup

After you’ve squared away your preferred distribution, be sure to run sudo apt-get update && sudo apt-get upgrade, then run sudo apt-get install

REMnux Docker Containers

Included in the REMnux container collection as of this writing you will find the V8 JavaScript engine, the Thug low-interaction honeyclient, the Viper binary analysis framework, Rekall and Volatility memory forensic frameworks, the JSDetox JavaScript analysis tool, the Radare2 reverse engineering framework, the Pescanner static malware analysis tool, the MASTIFF static analysis framework, and the Maltrieve malware samples downloader. This may well give you everything you possibly need as a great start for malware reverse engineering and analysis in one collection of Docker containers. I won’t discuss the Rekall or Volatility containers as toolsmith readers should already be intimately familiar with, and happily using, those tools. But it is mighty convenient to know you can spin them up via Docker.
The first time you run a Docker container it will be automatically pulled down from the Docker Hub if you don’t already have a local copy. All the REMnux containers reside there, you can, as I did, start with @kylemaxwell’s wicked good Maltrieve by executing sudo docker run --rm -it remnux/maltrieve bash. Once the container is downloaded and ready, exit and rerun it with sudo docker run --rm -it -v ~/samples:/home/sansforensics/samples remnux/maltrieve bash after you build a samples directory in your home directory. Important note: the -v parameter defines a shared directory that the container and the supporting host can both access and utilized. Liken it to Shared Folders in VMWare. Be sure to run sudo chmod a+xwr against it so it’s world readable/writeable. When all said and done you should be dropped to a nonroot prompt (a good thing), simply run maltrieve -d /home/sansforensics/samples/ -l /home/sansforensics/samples/maltieve.logand wait again as it populates malware samples to your sample directory, as seen in Figure 1, from the likes of Malc0de, Malware Domain List, Malware URLs, VX Vault, URLquery, CleanMX, and ZeusTracker.

Figure 1 – Maltrieve completes its downloads, 780 delicious samples ready for REMnux
So nice to have a current local collection. The above mentioned sources update regularly so you can keep your sample farm fresh. You can also define your preferred DUMPDIR and log directories in maltrieve.cfg for ease of use.

Next up, a look at the REMnux MASTIFF container. “MASTIFF is a static analysis framework that automates the process of extracting key characteristics from a number of different file formats” from @SecShoggoth.  I ran it as follows: sudo docker run --dns=my.dns.server.ip --rm -it -v ~/samples:/home/sansforensics/samples remnux/mastiff bash. You may want or need to replace --dns=my.dns.server.ip with your preferred DNS server if you don’t want to use the default I found this ensured name resolution for me from inside the container. MASTIFF can call the VirusTotal API and submit malware if you configure it to do so with mastiff.conf, it will fail if DNS isn’t working properly. You need to edit mastiff.conf via vi with you API key and enable submit=yes. Also note that, when invoked with --rm parameters, the container will be ephemeral and all customization will disappear once the container exits. You can invoke the container differently to save the customization and the state.
You may want to also instruct the log_dirdirective to point at your shared samples directory so the results are written outside the container.
You can then run /your/working/directory/samplename with your correct preferences and the result should resemble Figure 2.

Figure 2 – Successful REMnux MASTIFF run
All of the results can be found in /workdir/log under a folder named for each sample analyzed. Checking the Yara results in yara.txt will inform you that the while the payload is a PE32 it exhibits malicious document attributes per Didier Steven’s (another brilliant Internet Storm Center handler) maldoc rules as seen in Figure 3.

Figure 3 – Yara results indicating a malicious document attributes
The peinfo-full and peinfo-quick results will provide further details, indicators, and behaviors necessary to complete your analysis.

Our last example is the REMnux JSDetox container. Per its website, courtesy of @sven_t, JSDetox “is a tool to support the manual analysis of malicious Javascript code.” To run it is as simple as sudo docker run --rm -p 3000:3000 remnux/jsdetox, then point your browser to http://localhost:3000on your container host system. One of my favorite obfuscated malicious JavaScipt examples comes courtesy of and is seen in its raw, hidden ugliness in Figure 4.

Figure 4 – Obfuscated malicious JavaScript
Feed said script to JSDetox under the Code Analysis tab, run Analyze, choose the Execution tab, then Show Code and you’ll quickly learn that the obfuscated code serves up a malicious script from, flagged by major browsers and as distributing malware and acting as a redirector. The results are evident in Figure 5.

Figure 5 – JSDetox results
All the malware analysis horsepower you can imagine in the convenience of Docker containers, running on top of SIFT with a full REMnux install too. Way to go, Lenny, my journey is complete. J

In Conclusion

Lenny’s plans for the future include maintaining and enhancing the REMnux distro with the help of the Debian package repository he set up for this purpose with Docker and containers part of his design. Independently, he will continue to build and catalog Docker containers for useful malware analysis tools, so they can be utilized with or without the REMnux distro. I am certain this is the best way possible for you readers to immerse yourself in both Docker technology and some of the best of the REMnux collection at the same time. Enjoy!
Ping me via email or Twitter if you have questions (russ at holisticinfosec dot org or @holisticinfosec).
Cheers…until next month.


Thanks again to Lenny Zeltser, @lennyzeltser, for years of REMnux, and these Docker containers.

by Russ McRee ( at July 01, 2015 10:13 PM

Anton Chuvakin - Security Warrior

Monthly Blog Round-Up – June 2015

Here is my next monthly "Security Warrior" blog round-up of top 5 popular posts/topics this month:
  1. Why No Open Source SIEM, EVER?” contains some of my SIEM thinking from 2009. Is it relevant now? Well, you be the judge.  Current popularity of open source log search tools, BTW, does not break the logic of that post. Succeeding with SIEM requires a lot of work, whether you paid for the software, or not. That – and developing a SIEM is much harder than most people think  [278 pageviews]
  2. Top 10 Criteria for a SIEM?” came from one of my last projects I did when running my SIEM consulting firm in 2009-2011 (for my recent work on evaluating SIEM, see this document) [198 pageviews]
  3. Simple Log Review Checklist Released!” is often at the top of this list – the checklist is still a very useful tool for many people. “On Free Log Management Tools” is a companion to the checklist (updated version) [114 pageviews]
  4. My classic PCI DSS Log Review series is always popular! The series of 18 posts cover a comprehensive log review approach (OK for PCI DSS 3.0 as well), useful for building log review processes and procedures , whether regulatory or not. It is also described in more detail in our Log Management book and mentioned in our PCI book (just out in its 4th edition!) [100+ pageviews to the main tag]
  5. On Choosing SIEM” is about the least wrong way of choosing a SIEM tool – as well as why the right way is so unpopular. [60 pageviews out of a total of 4941 pageviews to all blog pages]
In addition, I’d like to draw your attention to a few recent posts from my Gartner blog:

Current research on cloud security monitoring:
Past research on security analytics:
Miscellaneous fun posts:

(see all my published Gartner research here)
Also see my past monthly and annual “Top Popular Blog Posts” – 2007, 2008, 2009, 2010, 2011, 2012, 2013, 2014.
Disclaimer: most content at SecurityWarrior blog was written before I joined Gartner on Aug 1, 2011 and is solely my personal view at the time of writing. For my current security blogging, go here.

Previous post in this endless series:

by Anton Chuvakin ( at July 01, 2015 03:09 PM

LZone - Sysadmin

Screen tmux Cheat Sheet

Here is a side by side comparison of screen and tmux commands and hotkeys.
Function Screen tmux
Start instance screen screen -S <name> tmux
Attach to instance screen -r <name> screen -x <name> tmux attach
List instances screen -ls screen -ls <user name>/ tmux ls
New Window ^a c ^b c
Switch Window ^a n ^a p ^b n ^b p
List Windows ^a " ^b w
Name Window ^a A ^b ,
Split Horizontal ^a S ^b "
Split Vertical ^a | ^b %
Switch Pane ^a Tab ^b o
Kill Pane ^a x ^b x
Paging ^b PgUp ^b PgDown
Scrolling Mode ^a [ ^b [

July 01, 2015 02:19 PM

The Tech Teapot

A recovering “wunderkind”

If there is anything the IT industry loves above anything else it is youth.

I used to work in a company with an engineer who’d been programming since the early 1970s, who’d implemented operating systems on mainframes in assembly language, wrote OSI (up to session layer) and TCP/IP comms stacks from scratch on DOS based machines and made them all work together in the background. He wasn’t the only one either. There were a number of very talented, mature engineers at the company at the time. And yet, the guys the company held up on a pedestal were the snot nosed kids just out of college who hadn’t done anything even slightly comparable.

I was one of the snot nosed kids.

My “wunderkind” years hid a great deal of insecurity. Deep down most wunderkinds understand that it’s all just a charade. Genius in the computer industry is a much debased term, meaning you are moderately competent at your job but are young.

The fear is always there, the fear that somebody is going to find you out. Maybe, just maybe, your next project is going to show that you don’t know all of the answers. You may even be tempted to turn down projects that are challenging because of your fear.

The biggest danger is that you might start believing the bullshit. The worst thing you can do in this business is stop learning. If you start thinking you have some kind of innate ability, then sooner or later, you may think you can just rely on it. Stopping your learning in the wunderkind years is just going to mean you miss out on the early to mid career development you need to punch through into being a genuinely good engineer.

My advice if you are a current wunderkind? Ignore it like the bullshit it is. Keep developing your skills and take on challenging projects. Fail if you must. But above all, keep learning…

by Jack Hughes at July 01, 2015 12:30 PM

Racker Hacker

Stumbling into the world of 4K displays [UPDATED]

Samsung U28D590D 4K displayWoot suckered me into buying a 4K display at a fairly decent price and now I have a Samsung U28D590D sitting on my desk at home. I ordered a mini-DisplayPort to DisplayPort from Amazon and it arrived just before the monitor hit my doorstep. It’s time to enter the world of 4K displays.

The unboxing of the monitor was fairly uneventful and it powered up after small amount of assembly. I plugged my mini-DP to DP cable into the monitor and then into my X1 Carbon 3rd gen. After a bunch of flickering, the display sprang to life but the image looked fuzzy. After some hunting, I found that the resolution wasn’t at the monitor’s maximum:

$ xrandr -q
DP1 connected 2560x1440+2560+0 (normal left inverted right x axis y axis) 607mm x 345mm
   2560x1440     59.95*+
   1920x1080     60.00    59.94  
   1680x1050     59.95  
   1600x900      59.98

I bought this thing because it does 3840×2160. How confusing. After searching through the monitor settings, I found an option for “DisplayPort version”. It was set to version 1.1 but version 1.2 was available. I selected version 1.2 (which appears to come with something called HBR2) and then the display flickered for 5-10 seconds. There was no image on the display.

I adjusted GNOME’s Display settings back down to 2560×1440. The display sprang back to life, but it was fuzzy again. I pushed the settings back up to 3840×2160. The flickering came back and the monitor went to sleep.

My laptop has an HDMI port and I gave that a try. I had a 3840×2160 display up immediately! Hooray! But wait — that resolution runs at 30Hz over HDMI 1.4. HDMI 2.0 promises faster refresh rates but neither my laptop or the display support it. After trying to use the display at max resolution with a 30Hz refresh rate, I realized that it wasn’t going to work.

The adventure went on and I joined #intel-gfx on Freenode. This is apparently a common problem with many onboard graphics chips as many of them cannot support a 4K display at 60Hz. It turns out that the i5-5300U (that’s a Broadwell) can do it.

One of the knowledgeable folks in the channel suggested a new modeline. That had no effect. The monitor flickered and went back to sleep as it did before.

I picked up some education on the difference between SST and MST displays. MST displays essentially have two chips handling half of the display within the monitor. Both of those do the work to drive the entire display. SST monitors (the newer variety, like the one I bought) take a single stream and one single chip in the monitor figures out how to display the content.

At this point, I’m stuck with a non-working display at 4K resolution over DisplayPort. I can get lower resolutions working via DisplayPort, but that’s not ideal. 4K works over HDMI, but only at 30Hz. Again, not ideal. I’ll do my best to update this post as I come up with some other ideas.

UPDATE 2015-07-01: Thanks to Sandro Mathys for spotting a potential fix:

I found BIOS 1.08 waiting for me on Lenovo’s site. One of the last items fixed in the release notes was:

(New) Supported the 60Hz refresh rate of 4K (3840 x 2160) resolution monitor.

After a quick flash of a USB stick and a reboot to update the BIOS, the monitor sprang to life after logging into GNOME. It looks amazing! The graphics performance is still not amazing (but hey, this is Broadwell graphics we’re talking about) but it does 3840×2160 at 60Hz without a hiccup. I tried unplugging and replugging the DisplayPort cable several times and it never flickered.

The post Stumbling into the world of 4K displays [UPDATED] appeared first on

by Major Hayden at July 01, 2015 04:33 AM

LZone - Sysadmin

The damage of one second

Update: According to the AWS status page the incident was a problem related to BGP route leaking. AWS does not hint on a leap second related incident as originally suggested by this post!

Tonight we had another leap second and not without suffering at the same time. At the end of the post you can find two screenshots of outages suggested by The screenshots were taken shortly after midnight UTC and you can easily spot those sites with problems by the disting peak at the right site of the graph.

AWS Outage

What is common to many of the affected sites: them being hosted at AWS which had some problems.

[RESOLVED] Internet connectivity issues

Between 5:25 PM and 6:07 PM PDT we experienced an Internet connectivity issue with a provider outside of our network which affected traffic from some end-user networks. The issue has been resolved and the service is operating normally.

The root cause of this issue was an external Internet service provider incorrectly accepting a set of routes for some AWS addresses from a third-party who inadvertently advertised these routes. Providers should normally reject these routes by policy, but in this case the routes were accepted and propagated to other ISPs affecting some end-user’s ability to access AWS resources. Once we identified the provider and third-party network, we took action to route traffic around this incorrect routing configuration. We have worked with this external Internet service provider to ensure that this does not reoccur.

Incident Details

Graphs from

Note that those graphs indicate user reported issues:

July 01, 2015 04:20 AM

PHP ini_set() Examples

Syntax of ini_set()

The ini_set() syntax is simple:
string ini_set ( string $varname , string $newvalue )
it is just a key value setter. The important question is which values can be set. Below you find a list of working examples. Please note that you cannot change all php.ini options especially those that need to be set before PHP initializes.

Useful Working ini_set() Examples

1. Enabling error display

On production sites you typically do not show errors in the page for usability and security reasons. But when you when you debug something live you might want to enable it temporarily and just for you:
# Assuming is your IP...
if ( $_SERVER["REMOTE_ADDR"] == "") {
    ini_set('display_errors', '1');
Note: you may want to combine this with
error_reporting(E_ALL | E_STRICT);

2. Changing Memory Limit

When you need to increase memory from within the code:
Note though that this might be prevent by a Suhosin hardended PHP installation.

3. Adding include paths

Normally this shouldn't be necessary. It is way cleaner to do it in php.ini, but if you bundle libraries and you administrator doesn't know:
<?php ini_set('include_path',ini_get('include_path').':../my-libs:');  ?>

When You Cannot Use ini_set()

For most php.ini settings you can't use ini_set(). To workaround consider deploying a .htaccess along with your code as this .htaccess can provide all PHP options to overwrite the default php.ini settings.

For example to change the HTTP POST limit add this line to a .htaccess read by your webserver:
php_value post_max_size 2500000
Note how the "php_value" prefix indicates settings for PHP. So simple syntax is
php_value <key name> <value>

July 01, 2015 04:20 AM

Everything Sysadmin


[This is still only draft quality but I think it is worth publishing at this point.]

Internally at Stack Exchange, Inc. we've been debating the value of certain file formats: YAML, JSON, INI and the new TOML format just to name a few.

[If you are unfamiliar with TOML, it is Tom's Obvious, Minimal Language. "Tom", in this case, is Tom Preston-Werner, founder and former CEO of GitHub. The file format is still not reached version 1.0 and is still changing. However I do like it a lot. Also, the name of the format IS MY FREAKIN' NAME which is totally awesome. --Sincerely, Tom L.]

No one format is perfect for all situations. However while debating the pros and cons of these formats something did dawn on me: one group is for humans and another is for machines. The reason there will never be a "winner" in this debate is that you can't have a single format that is both human-friendly and machine-friendly.

Maybe this is obvious to everyone else but I just realized:

  1. The group that is human-friendly is easy to add comments to, and tolerant of ambiguity, is often weakly typed (only differentiating between ints and strings).

  2. The group that is machine-friendly is difficult (or impossible) to add comments, is less forgiving about formatting, and use often strongly typed.

As an example of being unforgiving about formatting, JSON doesn't permit a comma on the last line of a list.

This is valid JSON:

   "a": "apple", 
   "alpha": "bits", 
   "j": "jax"

This is NOT valid JSON:

   "a": "apple", 
   "alpha": "bits", 
   "j": "jax",

Can you see the difference? Don't worry if you missed it because it just proves you are a human being. The difference is the "j" line has a comma at the end. This is forbidden in JSON. This catches me all the time because, well, I'm human.

It also distracts me because diffs are a lot longer as a result. If I add a new value, such as "p": "pebbles" the diff looks very different:

$ diff x.json  xd.json 
<    "j": "jax"
>    "j": "jax",
>    "p": "pebbles"

However if JSON did permit a trailing comma (which it doesn't), the diffs would look shorter and be more obvious.

$ diff y.json yd.json 
>    "p": "pebbles",

This is not just a personal preference. This has serious human-factors consequences in an operational environment. It is difficult to safely operate a large complex system and one of the ways we protect ourselves if by diff'ing versions of configuration files. We don't want to be visually distracted by little things like having to mentally de-dup the "j" line.

The other difference is around comments. One camp permits them and another camp doesn't. In operations often we need to be able to temporarily comment out a few lines, or include ad hoc messages. Operations people communicate by leaving breadcrumbs and todo items in files. Rather than commenting out some lines I could delete them and use version control to bring them back, but that is much more work. Also, often I write code in comments for the future. For example, as part of preparation for a recent upgrade, we added the future configuration lines to a file but commented them out. By including them, they could be proofread by coworkers. It was suggested that if we used JSON we would simply add a key to the data structure called "ignore" and update the code to ignore any hashes with that key. That's a lot of code to change to support that. Another suggestion was that we add a key called "comment" with a value that is the comment. This is what a lot of JSON users end up doing. However the comments we needed to add don't fit into that paradigm. For example we wanted to add comments like, "Ask so-and-so to document the history of why this is set to false" and "Keep this list sorted alphabetically". Neither of those comments could be integrated into the JSON structures that existed.

On the other hand, strictly formatted formats like JSON are, in theory, faster to parse. Supporting ambiguity slows things down and leads to other problems. In the case of JSON, it is just plain so widely supported there are many reasons to use it just for that reason.

Some formats have typed data, others assume all data are strings, others distinguish between integer and string but go no further. YAML, if you implement the entire standard, has a complex way of representing specific types and even supports repetition with pointers. All of that turns YAML's beautifully simple format into a nightmare unsuitable for human editing.

I'm not going to say "format XYZ is the best and should be used in all cases" however I'd like to summarize the attributes of each format:

M Formal standard YES YES soon no
M Strongly typed YES YES string/int no
M Easy to implement
the entire standard
H Awesome name! no no YES no
H Permits comments no start of line only YES usually
H diffs neatly no YES (I think) YES YES
H Can be
updated without losing
format or comments
yes-ish NO soon NO

The * column indicates if this quality is important for machines (M) or humans (H). NOTE: This chart is by no means complete.

Personally I'm trying to narrow the file formats in our system down to two: one used for machine-to-machine communication (that is still human readable), and the other that is human-generated (or at least human-updated) for machine consumption (like configuration files). (Technically there's a 3rd need: Binary format for machine-to-machine communication, such as ProtoBufs or CapnProto.)

I'm very optimistic about TOML and look forward to seeing it get to a 1.0 standard. Of course, the fact that I am "Tom L." sure makes me favor this format. I mean, how could I not like that, eh?

Update: 2015-07-01: Updated table (TOML is typed), and added row for "Awesome name".

July 01, 2015 02:20 AM

June 30, 2015

Errata Security

CyberUL is a dumb idea

Peiter “mudge” Zatko is leaving Google, asked by the White House to create a sort of a cyber “Underwriter Laboratories” (UL) for the government. UL is the organization that certifies electrical devices, so that they don’t short out and zap you to death. But here’s the thing: a CyberUL is a dumb idea. It’s the Vogon approach to the problem. It imagines that security comes from a moral weakness that could be solved by getting “serious” about the problem.

It’s not the hacking problem

According to data-breach reports, 95% of all attacks are simple things, like phishing, SQL injection, and bad passwords – nothing related to software quality. The other 5% is because victims are using old, unpatched software. When exploits are used, it’s overwhelmingly for software that has remained unpatched for a year.

In other words, CyberUL addresses less than 0.1% of real-world attacks.

It’s not the same quality problem

UL is about accidental failures in electronics. CyberUL would be about intentional attacks against software. These are unrelated issues. Stopping accidental failures is a solved problem in many fields. Stopping attacks is something nobody has solved in any field.

In other words, the UL model of accidents is totally unrelated to the cyber problem of attacks.

Security is a tradeoff

Security experts ignore the costs of fixing security. They assume that it due to moral weakness, and that getting tough is all that’s needed.

That’s not true. Improving security comes at great cost, in terms of price, functionality, or usability. Insecurity happens not because people are weak, but because the tradeoffs aren’t worth it. That’s why you have an iPhone, which can get hacked, instead of a 1980s era feature-phone that can do little more than make phone calls – you find the added risk worth the tradeoffs.

The premise of a CyberUL is that people are wrong, that more tradeoffs must be imposed against their will in order to improve cybersecurity, such as increasing the price, removing features, or making products hard to use.

Rules have a cost

Government already has the “Common Criteria” rules. They are all for obviously good things, like masking a password with **** when users type it in. But here’s the thing: while the actual criteria are easy and straightforward, it’s buried in layers of bureaucracy. It costs at least $1 million to get a product certified with Common Criteria.

OPM invested millions in dealing with similar bureaucratic regulations. It’s not that they had no security – it’s that their security people spent all their time with bureaucracy. They ignored basic problems like SQLi, phishing, bad passwords, and patches because compliance consumed all their budget and time.

Do you even government?

People believe that wise CyberUL administrators will define what’s right based on their own expertise. This is nonsense – rules will be designed according to whoever spends the most on lobbyists. It’s same thing that happens in every industry.

As soon as the White House starts a CyberUL, Oracle, Microsoft, and Cisco will show up offering to help. Whatever rules are created will be those that favor those three companies at the expensive of smaller companies.

Government doesn’t follow the rules, anyways

Government agencies don’t follow the rules anyway. There are so many impossibly onerous rules in government anyway that complaining and getting an exception is the norm. That’s why, for example, the Navy just gave Microsoft $20 million to continue to support WinXP – a 15 year old operating-system – which is otherwise against the rules.


A CyberUL is an absurd idea, being unrelated to the problem it purports to solve. The only reason people take it seriously is that they are secretly fascist at heart. They aren’t interested in solving the problem of cybersecurity, because that’s hard. Instead, they want to tell other people what to do, because that’s easy.

SQLi, phishing, bad passwords, and lack of patches are the Four Horseman of the cybersecurity apocalypse, not software quality. Unless you are addressing those four things, then you are doing essentially nothing to solve the problem.

by Robert Graham ( at June 30, 2015 09:53 PM


My Security Strategy: The "Third Way"

Over the last two weeks I listened to and watched all of the hearings related to the OPM breach. During the exchanges between the witnesses and legislators, I noticed several themes. One presented the situation facing OPM (and other Federal agencies) as confronting the following choice:

You can either 1) "secure your network," which is very difficult and going to "take years," due to "years of insufficient investment," or 2) suffer intrusions and breaches, which is what happened to OPM.

This struck me as an odd dichotomy. The reasoning appeared to be that because OPM did not make "sufficient investment" in security, a breach was the result.

In other words, if OPM had "sufficiently invested" in security, they would not have suffered a breach.

I do not see the situation in this way, for two main reasons.

First, there is a difference between an "intrusion" and a "breach." An intrusion is unauthorized access to a computing resource. A breach is the theft, alteration, or destruction of that computing resource, following an intrusion.

It therefore follows that one can suffer an intrusion, but not suffer a breach.

One can avoid a breach following an intrusion if the security team can stop the adversary before he accomplishes his mission.

Second, there is no point at which any network is "secure," i.e., intrusion-proof. It is more likely one could operate a breach-proof network, but that is not completely attainable, either.

Still, the most effective strategy is a combination of preventing as many intrusions as possible, complemented by an aggressive detection and response operation that improves the chances of avoiding a breach, or at least minimizes the impact of a breach.

This is why I call "detection and response" the "third way" strategy. The first way, "secure your network" by making it "intrusion-proof," is not possible. The second way, suffer intrusions and breaches, is not acceptable. Therefore, organizations should implement a third way strategy that stops as many intrusions as possible, but detects and responds to those intrusions that do occur, prior to their progression to breach status.

by Richard Bejtlich ( at June 30, 2015 07:23 PM

My Prediction for Top Gun 2 Plot

We've known for about a year that Tom Cruise is returning to his iconic "Maverick" role from Top Gun, and that drone warfare would be involved. A few days ago we heard a few more details in this Collider story:

[Producer David Ellison]: There is an amazing role for Maverick in the movie and there is no Top Gun without Maverick, and it is going to be Maverick playing Maverick. It is I don’t think what people are going to expect, and we are very, very hopeful that we get to make the movie very soon. But like all things, it all comes down to the script, and Justin is writing as we speak.

[Interviewer]; You’re gonna do what a lot of sequels have been doing now which is incorporate real use of time from the first one to now.

ELLISON and DANA GOLDBERG: Absolutely...

ELLISON:  As everyone knows with Tom, he is 100% going to want to be in those airplanes shooting it practically. When you look at the world of dogfighting, what’s interesting about it is that it’s not a world that exists to the same degree when the original movie came out. This world has not been explored. It is very much a world we live in today where it’s drone technology and fifth generation fighters are really what the United States Navy is calling the last man-made fighter that we’re actually going to produce so it’s really exploring the end of an era of dogfighting and fighter pilots and what that culture is today are all fun things that we’re gonna get to dive into in this movie.

What could the plot involve?

First, who is the adversary? You can't have dogfighting without a foe. Consider the leading candidates:

  • Russia: Maybe. Nobody is fond of what President Putin is doing in Ukraine.
  • Iran: Possible, but Hollywood types are close to the Democrats, and they will not likely want to upset Iran if Secretary Kerry secures a nuclear deal.
  • China: No way. Studios want to release movies in China, and despite the possibility of aerial conflict in the East or South China Seas, no studio is going to make China the bad guy. In fact, the studio will want to promote China as a good guy to please that audience.
  • North Korea: No way. Prior to "The Interview," this was a possibility. Not anymore!
My money is on an Islamic terrorist group, either unnamed, or possibly Islamic State. They don't have an air force, you say? This is where the drone angle comes into play.

Here is my prediction for the Top Gun 2 plot.

Oil tankers are trying to pass through the Gulf of Aden, or maybe the Strait of Hormuz, carrying their precious cargo. Suddenly a swarm of small, yet armed, drones attack and destroy the convoy, setting the oil ablaze in a commercial and environmental disaster. The stock market suffers a huge drop and gas prices skyrocket.

The US Fifth Fleet, and its Chinese counterpart, performing counter-piracy duties nearby, rush to rescue the survivors. They set up joint patrols to guard other commercial sea traffic. Later the Islamic group sends another swarm of drones to attack the American and Chinese ships. This time the enemy includes some sort of electronic warfare-capable drones that jam US and Chinese GPS, communications, and computer equipment. (I'm seeing a modern "Battlestar Galactica" theme here.) American and Chinese pilots die, and their ships are heavily damaged. (By the way, this is Hollywood, not real life.)

The US Navy realizes that its "net-centric," "technologically superior" force can't compete with this new era of warfare. Cue the similarities with the pre-Fighter Weapons School, early Vietnam situation described in the first scenes at Miramar in the original movie. (Remember, a 12-1 kill ratio in Korea, 3-1 in early Vietnam due to reliance on missiles and atrophied dogfighting skills, back to 12-1 in Vietnam after Top Gun training?)

The US Navy decides it needs to bring back someone who thinks unconventionally in order to counter the drone threat and resume commercial traffic in the Gulf. They find Maverick, barely hanging on to a job teaching at a civilian flight school. His personal life is a mess, and he was kicked out of the Navy during the first Gulf War in 1991 for breaking too many rules. Now the Navy wants him to teach a new generation of pilots how to fight once their "net-centric crutches" disappear.

You know what happens next. Maverick returns to the Navy as a contractor. Top Gun is now the Naval Strike and Air Warfare Center (NSAWC) at NAS Fallon, Nevada. The Navy retired his beloved F-14 in 2006, so there is a choice to be made about what aircraft awaits him in Nevada. I see three possibilities:

1) The Navy resurrects the F-14 because it's "not vulnerable" to the drone electronic warfare. This would be cool, but they aren't going to be able to fly American F-14s due to their retirement. CGI maybe?

2) The Navy flies the new F-35, because it's new and cool. However, the Navy will probably not have any to fly. CGI again?

3) The Navy flies the F-18. This is most likely, because producers could film live operations as they did in the 1980s.

Beyond the aircraft issues, I expect themes involving relevance as one ages, re-integration with military culture, and possibly friction between members of the joint US-China task force created to counter the Islamic threat.

In the end, thanks to the ingenuity of Maverick's teaching and tactics, the Americans and Chinese prevail over the Islamic forces. It might require Maverick to make the ultimate sacrifice, showing he's learned that warfare is a team sport, and that he really misses Goose. The Chinese name their next aircraft carrier the "Pete Mitchell" in honor of Maverick's sacrifice. (Forget calling it the "Maverick" -- too much rebellion for the CCP.)

I'm looking forward to this movie.

by Richard Bejtlich ( at June 30, 2015 03:01 PM

Sean's IT Blog

The Approaching Backup (Hyper)Convergence #VFD5

When we talk about convergence in IT, it usually means bringing things together to make them easier to manage and use.  Network convergence, in the data center, is bringing together your storage and IP stacks, while hyperconverged is about bringing together compute and storage together in a platform that can easily scale as new capacity is needed.

One area where we haven’t seen a lot of convergence is the backup industry.  One new startup, fresh out of stealth mode, aims to change that by bringing together backup storage, compute, and virtualization backup software in a scalable and easy to use package.

I had the opportunity to hear from Rubrik, a new player in the backup space, at Virtualization Field Day 5.   My coworker, and fellow VFD5 delegate, Eric Shanks, has also written his thoughts on Rubrik.

Note: All travel and incidental expenses for attending Virtualization Field Day 5 were paid for by Gestalt IT.  This was the only compensation provided, and it did not influence the content of this post.

One of the challenges of architecting backup solutions for IT environments is that you need to bring together a number of disparate pieces, often from different vendors, and try to make them function as one.  Even if multiple components are from the same vendor, they’re often not integrated in a way to make them easy to deploy.

Rubrik’s goal is to be a “Time Machine for private cloud” and to make backup so simple that you can have the appliance racked and starting backups within 15 minutes.  Their product, which hit general availability in May, combines backup software, storage, and hardware in a package that is easy to deploy, use, and scale.

They front this with an HTML5 interface and advanced search capabilities for virtual machines and files within the virtual machine file system.  This works across both locally stored data and data that has been aged out to the cloud due to a local metadata cache.

Because they control the hardware and software for the entire platform, Rubrik is able to engineer everything for the best performance.  They utilize flash in each node to store backup metadata as well as ingest the inbound data streams to deduplicate and compress data.

Rubrik uses SLAs to determine how often virtual machines are protected and how long that data is saved.  Over time, that data can be aged out to Amazon S3.  They do not currently support replication to another Rubrik appliance in another location, but that is on the roadmap.

Although there are a lot of cool features in Rubrik, it is a version 1.0 product.  It is missing some things that more mature products have such as application-level item recovery and role-based access control.  They only support vSphere in this reslease.  However, the vendor has committed to adding many more features, and support for additional hypervisors, in future releases.

You can watch the introduction and technical deep dive for the Rubrik presentation on Youtube.  The links are below.

If you want to see a hands-on review of Rubrik, you can read Brian Suhr’s unboxing post here.

Rubrik has brought an innovative and exciting product to market, and I look forward to seeing more from them in the future.

by seanpmassey at June 30, 2015 01:00 PM

Standalone Sysadmin

Are you monitoring your switchports the right way?

Graphite might be the best thing I’ve rolled out here in my position at CCIS.

One of our graduate students has been working on a really interesting paper for a while. I can’t go into details, because he’s going to publish before too long, but he has been making good use of my network diagrams. Since he has a lot riding on the accuracy of the data, he’s been asking me very specific questions about how the data was obtained, and how the graphs are produced, and so on.

One of the questions he asked me had to do with a bandwidth graph, much like this one:

His question revolved around the actual amount of traffic each datapoint represented. I explained briefly that we were looking at Megabytes per second, and he asked for clarification – specifically, whether each point was the sum total of data sent per second between updates, or whether it was the average bandwidth used over the interval.

We did some calculations, and decided that if it were, in fact, the total number of bytes received since the previous data point, it would mean my network had basically no traffic, and I know that not to be the case. But still, these things need verified, so I dug in and re-determined the entire path that the metrics take.

These metrics are coming from a pair of Cisco Nexus Switches via SNMP. The data being pulled is a per-interface ifInOctets and ifOutOctets. As you can see from the linked pages, each of those are 32 bit counters, with “The total number of octets transmitted [in|out] of the interface, including framing characters”.

Practically speaking, what this gives you is an ever-increasing number. The idea behind this counter is that you query it, and receive a number of bytes (say, 100). This indicates that at the time you queried it, the interface has sent (in the case of ifOutOctets) 100 bytes. If you query it again ten seconds later, and you get 150, then you know that in the intervening ten seconds, the interface has sent 50 bytes, and since you queried it ten seconds apart, you determine that the interface has transmitted 5 bytes per second.

Having the counter work like this means that, in theory, you don’t have to worry about how frequently you query it. You could query it tomorrow, and if it went from 100 to 100000000, you could be able to figure out how many seconds it was since you asked before, divide the byte difference, and figure out the average bytes per second that way. Granted, the resolution on those stats isn’t stellar at that frequency, but it would still be a number.

Incidentally, you might wonder, “wait, didn’t you say it was 32 bits? That’s not huge. How big can it get?”. The answer is found in RFC 1155: Counter

This application-wide type represents a non-negative integer which monotonically increases until it reaches a maximum value, when it wraps around and starts increasing again from zero. This memo specifies a maximum value of 2^32-1 (4294967295 decimal) for counters.

In other words, 4.29 gigabytes (or just over 34 gigabits). It turns out that this is actually kind of an important facet to the whole “monitoring bandwith” thing, because in our modern networks, switch interfaces are routinely 1Gb/s, often 10Gb/s, and sometimes even more. If our standard network interfaces can transfer one gigabits per second, then a fully utilized network interface can roll over an entire counter in 35 seconds. If we’re only querying that interface once a minute, then we’re potentially losing a lot of data. Consider, then, a 10Gb/s interface. Are you pulling metrics more often than once every 4 seconds? If not, you may be losing data.

Fortunately, there’s an easy fix. Instead of ifInOctets and ifOutOctets, query ifHCInOctets and ifHCOutOctets.  They are 64 bit counters, and only roll over once every 18 exabytes. Even with a 100% utilized 100Gb/s interface, you’ll still only roll over a counter every 5.8 years or so.

I made this change to my collectd configuration as soon as I figured out what I was doing wrong, and fortunately, none of my metrics jumped, so I’m going to say I got lucky. Don’t be me – start out doing it the right way, and save yourself confusion and embarrassment later.  Use 64-bit counters from the start!

(Also, there are the equivalent HC versions for all of the other interface counters you’re interested in, like the UCast, Multicast, and broadcast packet stats – make sure to use the 64-bit version of all of them).

Thanks, I hope I managed to help someone!

by Matt Simmons at June 30, 2015 09:14 AM


Contributing to the Ubuntu Weekly Newsletter

Super star Ubuntu Weekly Newsletter contributor Paul White recently was reflecting upon his work with the newsletter and noted that he was approaching 100 issues that he’s contributed to. Wow!

That caused me to look at how long I’ve been involved. Back in 2011 the newsletter when on a 6 month hiatus when the former editor had to step down due to obligations elsewhere. After much pleading for the return of the newsletter, I spent a few weeks working with Nathan Handler to improve the scripts used in the release process and doing an analysis of the value of each section of the newsletter in relation to how much work it took to produce each week. The result was a slightly leaner, but hopefully just as valuable newsletter, which now took about 30 minutes for an experienced editor to release rather than 2+ hours. This change was transformational for the team, allowing me to be involved for a whopping 205 consecutive issues.

If you’re not familiar with the newsletter, every week we work to collect news from around our community and the Internet to bring together a snapshot of that week in Ubuntu. It helps people stay up to date with the latest in the world of Ubuntu and the Newsletter archive offers a fascinating glimpse back through history.

But we always need help putting the newsletter together. We especially need people who can take some time out of their weekend to help us write article summaries.

Summary writers. Summary writers receive an email every Friday evening (or early Saturday) US time with a link to the collaborative news links document for the past week which lists all the articles that need 2-3 sentence summaries. These people are vitally important to the newsletter. The time commitment is limited and it is easy to get started with from the first weekend you volunteer. No need to be shy about your writing skills, we have style guidelines to help you on your way and all summaries are reviewed before publishing so it’s easy to improve as you go on.

Interested? Email and we’ll get you added to the list of folks who are emailed each week.

I love working on the newsletter. As I’ve had to reduce my commitment to some volunteer projects I’m working on, I’ve held on to the newsletter because of how valuable and enjoyable I find it. We’re a friendly team and I hope you can join us!

Still just interested in reading? You have several options:

And everyone is welcome to drop by #ubuntu-news on Freenode to chat with us or share links to news we may found valuable for the newsletter.

June 30, 2015 02:29 AM

June 29, 2015

Sean's IT Blog

GPUs Should Be Optional for VDI

Note: I disabled comments on my blog in 2014 because of spammers. Please comment on this discussion on Twitter using the #VDIGPU hashtag.

Brian Madden recently published a blog arguing that GPU should not be considered optional for VDI.  This post stemmed from a conversation that he had with Dane Young about a BriForum 2015 London session on his podcast

Dane’s statement that kicked off this discussion was:
”I’m trying to convince people that GPUs should not be optional for VDI.”

The arguments that were laid out in Brian’s blog post were:

1. You don’t think of buying a desktop without a GPU
2. They’re not as expensive as people think

I think these are poor arguments for adopting a technology.  GPUs are not required for general purpose VDI, and they should only be used when the use case calls for it.  There are a couple of reasons why:

1. It doesn’t solve user experience issues: User experience is a big issue in VDI environments, and many of the complaints from users have to do with their experience.  From what I have seen, a good majority of those issues have resulted from a) IT doing a poor job of setting expectations, b) storage issues, and/or c) network issues.

Installing GPUs in virtual environments will not resolve any of those issues, and the best practices are to turn off or disable graphics intensive options like Aero to reduce the bandwidth used on wide-area network links.

Some modern applications, like Microsoft Office and Internet Explorer, will offload some processing to the GPU.  The software GPU in vSphere can easily handle these requirements with some additional CPU overhead.  CPU overhead, however, is rarely the bottleneck in VDI environments, so you’re not taking a huge performance hit by not having a dedicated hardware GPU.

2. It has serious impacts on consolidation ratios and user densities: There are three ways to do hardware graphics acceleration for virtual machines running on vSphere with discrete GPUs.

(Note: These methods only apply to VMware vSphere. Hyper-V and XenServer have their own methods of sharing GPUs that may be similar to this.)

  • Pass-Thru (vDGA): The physical GPU is passed directly through to the virtual machines on a 1 GPU:1 Virtual Desktop basis.  Density is limited to the number of GPUs installed on the host. The VM cannot be moved to another host unless the GPU is removed. The only video cards currently supported for this method are high-end NVIDIA Quadro and GRID cards.
  • Shared Virtual Graphics (vSGA): VMs share access to GPU resources through a driver that is installed at the host level, and the GPU is abstracted away from the VM. The software GPU driver is used, and the hypervisor-level driver acts as an interface to the physical GPU.  Density depends on configuration…and math is involved (note: PDF link) due to the allocated video memory being split between the host’s and GPU’s RAM. vSGA is the only 3D graphics type that can be vMotioned to another host while the VM is running, even if that host does not have a physical GPU installed. This method supports NVIDIA GRID cards along with select QUADRO, AMD FirePro, and Intel HD graphics cards.
  • vGPU: VMs share access to an NVIDIA GRID card.  A manager application is installed that controls the profiles and schedules access to GPU resources.  Profiles are assigned to virtual desktops that control resource allocation and number of virtual desktops that can utilize the card. A Shared PCI device is added to VMs that need to access the GPU, and VMs may not be live-migrated to a new host while running. VMs may not start up if there are no GPU resources available to use.

Figure 1: NVIDIA GRID Profiles and User Densities

There is a hard limit to the number of users that you can place on a host when you give every desktop access to a GPU, so it would require additional hosts to meet the needs of the VDI environment.  That also means that hardware could be sitting idle and not used to its optimal capacity because the GPU becomes the bottleneck.

The alternative is to try and load up servers with a large number of GPUs, but there are limits to the number of GPUs that a server can hold.  This is usually determined by the number of available PCIe x16 slots and available power, and the standard 2U rackmount server can usually only handle two cards.   This means I would still need to take on additional expenses to give all users a virtual desktop with some GPU support.

Either way, you are taking on unnecessary additional costs.

There are few use cases that currently benefit from 3D acceleration.  Those cases, such as CAD or medical imaging, often have other requirements that make high user consolidation ratios unlikely and are replacing expensive, high-end workstations.

Do I Need GPUs?

So do I need a GPU?  The answer to that question, like any other design question, is “It Depends.”

It greatly depends on your use case, and the decision to deploy GPUs will be determined by the applications in your use case.  Some of the applications where a GPU will be required are:

  • CAD and BIM
  • Medical Imaging
  • 3D Modeling
  • Computer Animation
  • Graphic Design

You’ll notice that these are all higher-end applications where 3D graphics are a core requirement.

But what about Office, Internet Explorer, and other basic apps?  Yes, more applications are offloading some things to the GPU, but these are often minor things to improve UI performance.  They can also be disabled, and the user usually won’t notice any performance difference.

Even if they aren’t disabled, the software GPU can handle these elements.  There would be some additional CPU overhead, but as I said above, VDI environments usually constrained by memory and have enough available CPU capacity to accommodate this.

But My Desktop Has a GPU…

So let’s wrap up by addressing the point that all business computers have GPUs and how that should be a justification for putting GPUs in the servers that host VDI environments.

It is true that all desktops and laptops come with some form of a GPU.  But there is a very good reason for this. Business desktops and laptops are designed to be general purpose computers that can handle a wide-range of use cases and needs.  The GPUs in these computers are usually integrated Intel graphics cards, and they lack the capabilities and horsepower of the professional grade NVIDIA and AMD products used in VDI environments. 

Virtual desktops are not general purpose computers.  They should be tailored to their use case and the applications that will be running in them.  Most users only need a few core applications, and if they do not require that GPU, it should not be there.

It’s also worth noting that adding NVIDIA GRID cards to servers is a non-trivial task.  Servers require special factory configurations to support GPUs that need to be certified by the graphics manufacturer.  There are two reasons for this – GPUs often draw more than the 75W that a PCIe x16 slot can provide and are passively cooled, requiring additional fans.  Aside from one vendor on Amazon, these cards can only be acquired from OEM vendors as part of the server build.

The argument that GPUs should be required for VDI will make much more sense when hypervisors have support for mid-range GPUs from multiple vendors. Until that happens, adding GPUs to your virtual desktops is a decision that needs to be made carefully, and it needs to fit your intended use cases.  While there are many use cases where they are required or would add significant value, there are also many use cases where they would add unneeded constraints and costs to the environment. 

by seanpmassey at June 29, 2015 01:26 PM

June 28, 2015

League of Professional System Administrators

IPv6 on a home Mac Server, a lesson

Trying to set up IPv6 each of my home nodes created an unusual challenge.  All my laptops Just Worked(tm), but the mac mini server wouldn't.  They were running the same OS version, Yosemite, but I couldn't get any IPv6 traffic to talk to my home server, running on an older mac mini.  I even went so far as to wonder if there was some strange hardware compatibility issue on the ethernet card used by my mac mini.  Every other node, auto-config worked fine.

read more

by alcourt at June 28, 2015 06:53 PM

June 27, 2015


My Federal Government Security Crash Program

In the wake of recent intrusions into government systems, multiple parties have been asking for my recommended courses of action.

In 2007, following public reporting on the 2006 State Department breach, I blogged When FISMA BitesInitial Thoughts on Digital Security Hearing. and What Should the Feds Do. These posts captured my thoughts on the government's response to the State Department intrusion.

The situation then mirrors the current one well: outrage over an intrusion affecting government systems, China suspected as the culprit, and questions regarding why the government's approach to security does not seem to be working.

Following that breach, the State Department hired a new CISO who pioneered the "continuous monitoring" program, now called "Continuous Diagnostic Monitoring" (CDM). That CISO eventually left State for DHS, and brought CDM to the rest of the Federal government. He is now retired from Federal service, but CDM remains. Years later we're reading about another breach at the State Department, plus the recent OPM intrusions. CDM is not working.

My last post, Continuous Diagnostic Monitoring Does Not Detect Hackers, explained that although CDM is a necessary part of a security program, it should not be the priority. CDM is at heart a "Find and Fix Flaws Faster" program. We should not prioritize closing and locking doors and windows while there are intruders in the house. Accordingly, I recommend a "Detect and Respond" strategy first and foremost.

To implement that strategy, I recommend the following, three-phase approach. All phases can run concurrently.

Phase 1: Compromise Assessment: Assuming the Federal government can muster the motivation, resources, and authority, the Office of Management and Budget (OMB), or another agency such as DHS, should implement a government-wide compromise assessment. The compromise assessment involves deploying teams across government networks to perform point-in-time "hunting" missions to find, and if possible, remove, intruders. I suspect the "remove" part will be more than these teams can handle, given the scope of what I expect they will find. Nevertheless, simply finding all of the intruders, or a decent sample, should inspire additional defensive activities, and give authorities a true "score of the game."

Phase 2: Improve Network Visibility: The following five points include actions to gain enhanced, enduring, network-centric visibility on Federal networks. While network-centric approaches are not a panacea, they represent one of the best balances between cost, effectiveness, and minimized disruption to business operations.

1. Accelerate the deployment of Einstein 3A, to instrument all Federal network gateways. Einstein is not the platform to solve the Federal government's network visibility problem, but given the current situation, some visibility is better than no visibility. If the inline, "intrusion prevention system" (IPS) nature of Einstein 3A is being used as an excuse for slowly deploying the platform, then the IPS capability should be disabled and the "intrusion detection system" (IDS) mode should be the default. Waiting until the end of 2016 is not acceptable. Equivalent technology should have been deployed in the late 1990s.

2. Ensure DHS and US-CERT have the authority to provide centralizing monitoring of all deployed Einstein sensors. I imagine bureaucratic turf battles may have slowed Einstein deployment. "Who can see the data" is probably foremost among agency worries. DHS and US-CERT should be the home for centralized analysis of Einstein data. Monitored agencies should also be given access to the data, and DHS, US-CERT, and agencies should begin a dialogue on whom should have ultimately responsibility for acting on Einstein discoveries.

3. Ensure DHS and US-CERT are appropriately staffed to operate and utilize Einstein. Collected security data is of marginal value if no one is able to analyze, escalate, and respond to the data. DHS and US-CERT should set expectations for the amount of time that should elapse from the time of collection to the time of analysis, and staff the IR team to meet those requirements.

4. Conduct hunting operations to identify and remove threat actors already present in Federal networks. Now we arrive at the heart of the counter-intrusion operation. The purpose of improving network visibility with Einstein (for lack of an alternative at the moment) is to find intruders and eliminate them. This operation should be conducted in a coordinated manner, not in a whack-a-mole fashion that facilitates adversary persistence. This should be coordinated with the "hunt" mission in Phase 1.

5. Collect metrics on the nature of the counter-intrusion campaign and devise follow-on actions based on lessons learned. This operation will teach Federal network owners lessons about adversary campaigns and the unfortunate realities of the state of their enterprise. They must learn how to improve the speed, accuracy, and effectiveness of their defensive campaign, and how to prioritize countermeasures that have the greatest impact on the opponent. I expect they would begin considering additional detection and response technologies and processes, such as enterprise log management, host-based sweeping, modern inspection platforms with virtual execution and detonation chambers, and related approaches.

Phase 3. Continuous Diagnostic Monitoring, and Related Ongoing Efforts: You may be surprised to see that I am not calling for an end to CDM. Rather, CDM should not be the focus of Federal security measures. It is important to improve Federal security through CDM practices, such that it becomes more difficult for adversaries to gain access to government computers. I am also a fan of the Trusted Internet Connection program, whereby the government is consolidating the number of gateways to the Internet.

Note: I recommend anyone interested in details on this matter see my latest book, The Practice of Network Security Monitoring, especially chapter 9. In that chapter I describe how to run a network security monitoring operation, based on my experiences since the late 1990s.

by Richard Bejtlich ( at June 27, 2015 04:58 PM

June 26, 2015

Everything Sysadmin

Marriage Equality becomes the law of the land in the US

I literally never thought I'd see this day arrive.

In 1991/1992 I was involved in passing the LGB anti-discrimination law in New Jersey. When it passed in January 1992, I remember a reporter quoting one of our leaders that marriage was next. At the time I thought Marriage Equality would be an impossible dream, something that wouldn't happen in my lifetime. Well, less than quarter-century later, it has finally happened.

In the last few years more than 50% of the states approved marriage equality and soon it became a foregone conclusion. States are the "laboratory of democracy" and with 26 states (IIRC) having marriage equality, its about time to declare that the experiment is a success.

There were always predictions that marriage equality would somehow "ruin marriage" but in the last decade of individual states having marriage equality not a single example has come forward. What has come forward has been example after example of problems from not having marriage equality. The Oscar winning documentary "Freeheld" is about one such example. Having different laws in different states don't just create confusion, it hurts families.

"Human progress is neither automatic nor inevitable", wrote Martin Luther King Jr. It is not automatic: it doesn't "just happen", it requires thousands of little steps.

This day only happened because of thousands of activists working for many years, plus hundreds of thousands of supporters, donors, and millions of "like" buttons clicked.

A lot of people make jokes about lawyers but I never do. No civil rights law or court decision ever happens without a lawyer writing legislation or arguing before a court. The legal presentations given in Obergefell v. Hodges were top notch. Implementing the decision requires operational changes that will require policy makers, legal experts, and community activists to work together.

This is really an amazing day.

June 26, 2015 03:51 PM

The Tech Teapot

If you haven’t made it by the time you reach age X, you never will

I forget where I read that, but for a while it made me very unhappy. I was approaching X at the time and I most certainly had not “made it“.

Still haven’t.

Probably never will by the definition of myself when I was aged X. And you know, it doesn’t bother me one bit now.

One of the nicer things about getting older is that your definition of “making it” changes. In my twenties it was about money for the most part. But, it is hard not to be sucked into thinking about your position at work.

There’s a lot of status tied up with managing people.

It does show the dangers of reading all of those articles on the web pontificating about success. It is your success, you get to define precisely what it is. Don’t let anybody else define it for you.

by Jack Hughes at June 26, 2015 02:19 PM

June 23, 2015

Racker Hacker

Fedora 22 and rotating GNOME wallpaper with systemd timers

My older post about rotating GNOME’s wallpaper with systemd timers doesn’t seem to work in Fedora 22. The DISPLAY=:0 environment variable isn’t sufficient to allow systemd to use gsettings.

Instead, the script run by the systemd timer must know a little bit more about dbus. More specifically, the script needs to know the address of the dbus session so it can communicate on the bus. That’s normally kept within the DBUS_SESSION_BUS_ADDRESS environment variable.

Open a shell and you can verify that yours is set:

$ env | grep ^DBUS_SESSION

That is actually set when gnome-session starts as your user on your machine. for the script to work, we need to add a few lines at the top:

# These three lines are new
PID=$(pgrep -u $USER gnome-session)
export DBUS_SESSION_BUS_ADDRESS=$(grep -z DBUS_SESSION_BUS_ADDRESS /proc/$PID/environ|cut -d= -f2-)
# These three lines are unchanged from the original script
selection=$(find $walls_dir -type f -name "*.jpg" -o -name "*.png" | shuf -n1)
gsettings set org.gnome.desktop.background picture-uri "file://$selection"

Let’s look at what the script is doing:

  • First, we get the username of the user running the script
  • We look for the gnome-session process that is running as that user
  • We pull out the dbus environment variable from gnome-session’s environment variables when it was first started

Go ahead and adjust your script. Once you’re done, test it by simply running the script manually and then using systemd to run it:

$ bash ~/bin/
$ systemctl --user start gnome-background-change

Both of those commands should now rotate your GNOME wallpaper in Fedora 22.

The post Fedora 22 and rotating GNOME wallpaper with systemd timers appeared first on

by Major Hayden at June 23, 2015 05:25 PM

June 22, 2015

Keep your home dir in Git with a detached working directory

logo@2xMany posts have been written on putting your homedir in git. Nearly everyone uses a different method of doing so. I've found the method I'm about to describe in this blog post to work the best for me. I've been using it for more than a year now, and it haven't failed me yet. My method was put together from different sources all over the web; long since gone or untracable. So I'm documenting my setup here.

The features

So, what makes my method better than the rest? What makes it better than the multitude of pre-made tools out there? The answer is: it depends. I've simply found that this methods suits me personally because:

  • It's simple to implement, simple to understand and simple to use.
  • It gets out of your way. It doesn't mess with repositories deeper in your home directory, or with any tools looking for a .git directory. In fact, your home directory won't be a git repository at all.
  • It's simple to see what's changed since you last committed. It's a little harder to see new files not yet in your repository . This is because by default everything is ignored unless you specifically add it.
  • No special tools required, other than Git itself. A tiny alias in your .profile takes care of all of it.
  • No fiddling with symlinks and other nonsense.

How does it work?

It's simple. We create what is called a "detached working tree". In a normal git repository, you've got your .git dir, which is basically your repository database. When you perform a checkout, the directory containing this .git dir is populated with files from the git database. This is problematic when you want to keep your home directory in Git, since many tools (including git itself) will scan upwards in the directory tree in order to find a .git dir. This creates crazy scenario's such as Vim's CtrlP plugin trying to scan your entire home directory for file completions. Not cool. A detached working tree means your .git dir lives somewhere else entirely. Only the actual checkout lives in your home dir. This means no more nasty .git directory.

An alias 'dgit' is added to your .profile that wraps around the git command. It understands this detached working directory and lets you use git like you would normally. The dgit alias looks like this:

alias dgit='git --git-dir ~/.dotfiles/.git --work-tree=$HOME'

Simple enough, isn't it? We simply tell git that our working tree doesn't reside in the same directory as the .git dir (~/.dotfiles), but rather it's our directory. We set the git-dir so git will always know where our actual git repository resides. Otherwise it would scan up from the curent directory your in and won't find the .git dir, since that's the whole point of this exercise.

Setting it up

Create a directory to hold your git database (the .git dir):

$ mkdir ~/.dotfiles/
$ cd ~/.dotfiles/
~/.dotfiles$ git init .

Create a .gitifnore file that will ignore everything. You can be more conservative here and only ignore things you don't want in git. I like to pick and choose exactly which things I'll add, so I ignore everything by default and then add it later.

~/.dotfiles$ echo "*" > .gitignore
~/.dotfiles$ git add -f .gitignore 
~/.dotfiles$ git commit -m "gitignore"

Now we've got a repository set up for our files. It's out of the way of our home directory, so the .git directory won't cause any conflicts with other repositories in your home directory. Here comes the magic part that lets us use this repository to keep our home directory in. Add the dgit alias to your .bashrc or .profile, whichever you prefer:

~/.dotfiles$ echo "alias dgit='git --git-dir ~/.dotfiles/.git --work-tree=\$HOME'" >> ~/.bashrc

​You'll have to log out and in again, or just copy-paste the alias defnition in your current shell. We can now the repository out in our home directory with the dgit command:

~/.dotfiles$ cd ~
$ dgit reset --hard
HEAD is now at 642d86f gitignore

Now the repository is checked out in our home directory, and it's ready to have stuff added to it. The dgit reset --hard command might seem spooky (and I do suggest you make a backup before running it), but since we're ignoring everything, it'll work just fine.

Using it

Everything we do now, we do with the dgit command instead of normal git. In case you forget to use dgit, it simply won't work, so don't worry about that.

A dgit status shows nothing, since we've gitignored everything:

$ dgit status
On branch master
nothing to commit, working directory clean

We add things by overriding the ignore with -f:

$ dgit add -f .profile 
$ dgit commit -m "Added .profile"
[master f437f9f] Added .profile
 1 file changed, 22 insertions(+)
 create mode 100644 .profile

We can push our configuration files to a remote repository:

$ dgit remote add origin ssh://
$ dgit push origin master
 * [new branch]      master -> master

And easily deploy them to a new machine:

$ ssh someothermachine
$ git clone ssh:// ./.dotfiles
$ alias dgit='git --git-dir ~/.dotfiles/.git --work-tree=$HOME'
$ dgit reset --hard
HEAD is now at f437f9f Added .profile

Please note that any files that exist in your home directory will be overwritten by the files from your repository if they're present.


This DIY method of keeping your homedir in git should be easy to understand. Although there are tools out there that are easier to use, this method requires no installing other than Git. As I've stated in the introduction, I've been using this method for more than a year, and have found it to be the best way of keeping my home directory in git. 

by admin at June 22, 2015 01:30 PM

Warren Guy

Deploy a Tor hidden service to Heroku in under a minute

Getting a Tor hidden service running doesn't have to be hard. I've just published an example Sinatra application demonstrating how to deploy a hidden service to Heroku (or Dokku, etc) in just a few lines. The app uses my ruby-hidden-service library with the multi and apt Heroku buildpacks to install and configure Tor. A deployed example is running at

Here are the complete steps required to deploy the sample app:

Read full post

June 22, 2015 01:09 PM

June 21, 2015

Steve Kemp's Blog

We're all about storing objects

Recently I've been experimenting with camlistore, which is yet another object storage system.

Camlistore gains immediate points because it is written in Go, and is a project initiated by Brad Fitzpatrick, the creator of Perlbal, memcached, and Livejournal of course.

Camlistore is designed exactly how I'd like to see an object storage-system - each server allows you to:

  • Upload a chunk of data, getting an ID in return.
  • Download a chunk of data, by ID.
  • Iterate over all available IDs.

It should be noted more is possible, there's a pretty web UI for example, but I'm simplifying. Do your own homework :)

With those primitives you can allow a client-library to upload a file once, then in the background a bunch of dumb servers can decide amongst themselves "Hey I have data with ID:33333 - Do you?". If nobody else does they can upload a second copy.

In short this kind of system allows the replication to be decoupled from the storage. The obvious risk is obvious though: if you upload a file the chunks might live on a host that dies 20 minutes later, just before the content was replicated. That risk is minimal, but valid.

There is also the risk that sudden rashes of uploads leave the system consuming all the internal-bandwith constantly comparing chunk-IDs, trying to see if data is replaced that has been copied numerous times in the past, or trying to play "catch-up" if the new-content is larger than the replica-bandwidth. I guess it should possible to detect those conditions, but they're things to be concerned about.

Anyway the biggest downside with camlistore is documentation about rebalancing, replication, or anything other than simple single-server setups. Some people have blogged about it, and I got it working between two nodes, but I didn't feel confident it was as robust as I wanted it to be.

I have a strong belief that Camlistore will become a project of joy and wonder, but it isn't quite there yet. I certainly don't want to stop watching it :)

On to the more personal .. I'm all about the object storage these days. Right now most of my objects are packed in a collection of boxes. On the 6th of next month a shipping container will come pick them up and take them to Finland.

For pretty much 20 days in a row we've been taking things to the skip, or the local charity-shops. I expect that by the time we've relocated the amount of possesions we'll maintain will be at least a fifth of our current levels.

We're working on the general rule of thumb: "If it is possible to replace an item we will not take it". That means chess-sets, mirrors, etc, will not be carried. DVDs, for example, have been slashed brutally such that we're only transferring 40 out of a starting collection of 500+.

Only personal, one-off, unique, or "significant" items will be transported. This includes things like personal photographs, family items, and similar. Clothes? Well I need to take one jacket, but more can be bought. The only place I put my foot down was books. Yes I'm a kindle-user these days, but I spent many years tracking down some rare volumes, and though it would be possible to repeat that effort I just don't want to.

I've also decided that I'm carrying my complete toolbox. Some of the tools I took with me when I left home at 18 have stayed with me for the past 20+ years. I don't need this specific crowbar, or axe, but I'm damned if I'm going to lose them now. So they stay. Object storage - some objects are more important than they should be!

June 21, 2015 04:10 PM

Racker Hacker

Book Review: Linux Kernel Development

Linux Kernel Development book coverI picked up a copy of Robert Love’s book, Linux Kernel Development, earlier this year and I’ve worked my way through it over the past several weeks. A few people recommended the book to me on Twitter and I’m so glad they did. This book totally changed how I look at a system running Linux.

You must be this tall to ride this ride

I’ve never had formal education in computer science or software development in the past. After all, my degree was in Biology and I was on the path to becoming a phyisician when this other extremely rewarding career came into play. (That’s a whole separate blog post in itself.)

Just to level-set: I can read C and make small patches when I spot problems. However, I’ve never set out and started a project in C on my own and I haven’t really made any large contributions to projects written in C. However, I’m well-versed in Perl, Ruby, and Python mainly from job experience and leaning on some much more skilled colleagues.

The book recommends that you have a basic grasp of C and some knowledge around memory management and process handling. I found that I was able to fully understand about 70% of the book immediately, another 20% or so required some additional research and practice, while about 10% was mind-blowing. Obviously, that leaves me with plenty of room to grow.

Honestly, if you understand how most kernel tunables work and you know at least one language that runs on your average Linux box, you should be able to understand the majority of the material. Some sections might require some re-reading and you might need to go back and read a section when a later chapter sheds more light on the subject.

Moving through the content

I won’t go into a lot of detail around the content itself other than to say it’s extremely comprehensive. After all, you wouldn’t be reading a book about something as complex as the Linux kernel if you weren’t ready for an onslaught of information.

The information is organized in an effective way. Initial concepts are familiar to someone who has worked in user space for quite some time. If you’ve dealt with oom-killer, loaded kernel modules, or written some horrible code that later needed to be optimized, you’ll find the beginning of the book to be very useful. Robert draws plenty of distinctions around kernel space, user space, and how they interact. He take special care to cover SMP-safe code and how to take non-SMP-safe code and improve it.

I found a ton of value in the memory management, locking, and the I/O chapters. I didn’t fully understand the blocks of C code within the text but there was a ton of value in the deep explanations of how data flows (and doesn’t flow) from memory to disk and back again.

The best part

If I had to pick one thing to entice more people to read the book, it would be the way Robert explains every concept in the book. He has a good formula that helps you understand the how, the what, and the why. So many books forget the why.

He takes the time to explain what frustrated the kernel developers that made them write a feature in the first place and then goes into detail about how they fixed it. He also talks about differences between other operating systems (like Unix, Windows, and others) and other hardware types (like ARM and Alpha). So many books leave this part out but it’s often critical for understanding difficult topics. I learned this the hard way in my biology classes when I tried to memorize concepts rather than trying to understand the evolutionary or chemical reasons for why it occurs.

Robert also rounds out the book with plenty of debugging tips that allow readers to trudge through bug hunts with better chances of success. He helps open the doors to the Linux kernel community and gives tips on how to get the best interactions from the community.


This book is worth it for anyone who wants to learn more about how their Linux systems operate or who want to actually write code for the kernel. Much of the deep workings of the kernel was a mystery to me before and I really only knew how to interact with a few interfaces.

Reading this book was like watching a cover being taken off of a big machine and listening to an expert explain how it works. It’s definitely worth reading.

The post Book Review: Linux Kernel Development appeared first on

by Major Hayden at June 21, 2015 03:26 PM

June 20, 2015

Warren Guy

Rack::DetectTor: Rack middleware for detecting Tor exits

I've just released Rack::DetectTor, Rack middleware for detecting Tor users. It adds an environment varliable tor_exit_user with a value of true or false to the Rack request object. I've previously blogged about detecting Tor users in nginx using iptables, however Rack::DetectTor is a much neater and more self contained solution for Ruby/Rack based web apps (built on Ruby on Rails, Sinatra, Padrino, etc).

More info on the Github project page:

Read full post

June 20, 2015 09:37 PM

June 18, 2015

Warren Guy

BTC Watch - Instant Bitcoin address monitoring by email

BTC Watch - Instant Bitcoin address monitoring by email

I've just launched BTC Watch. It's a simple service that monitors the Bitcoin network in realtime, allowing you to subscribe to real-time e-mail updates of transactions occuring on addresses you wish to monitor. You can opt to receive notification immediately when the transaction first appears on the network, and/or when the transaction has been confirmed between 1 and 120 times and at several steps inbetween.

You might find this useful for monitoring addresses you publish for receiving funds or donations, or for keeping track of your own addresses and transactions. No sign up is required. Just enter your email address, and the Bitcoin address you want to monitor, and you're set. Check it out at Some screenshots of sample email notifications are below. The emails have both text and HTML parts, and also include an attached file with a JSON object containing the complete transaction information.

Read full post

June 18, 2015 12:29 AM

June 17, 2015

Evaggelos Balaskas

fluxbox keys

I am using fluxbox as my primary window manager on both my laptop and home desktop. If you want a non distractive environment to work with, I strongly suggest to take a look.


On the laptop, I had a problem to configure the backlight. It was always on 100%, fixed without the ability to change it. If you run on battery, then you need to lower the brightness of your display.

After Linux kernel v3.16, things got a lot easier and better for newest models of laptops that had problems with the backlight and brightness.

You can find a lot of blog/site & wiki pages that suggest to append something of the below to your grub menu entry:


or something similar.


Note: On Dell XPS13 laptops a firmware bug exists when disabling legacy boot or switching through UEFI & legacy. That can break the backlight support and the result is a blank screen. This is a stupid manufacture error of Dell cause they used a different firmware module for backlight that can send different acpi events!

For me that’s irrelevant now. I am using UEFI and Linux kernel v4.0.5 and I have disabled legacy boot from my laptop a long time ago.
My grub menu doesnt have any of the above settings.


Ok, so now it’s time to explain how you can use fluxbox keys to control the brightness on your laptop.

Open a terminal and type:


With this program you can capture the keycode of the keys your are pressing.

I want to use the same keys that I would normally use for adjusting the display brightness on my laptop.
So on my machine, FN+F4 returns 232 and FN+F5 233.

Edit your ~/.fluxbox/startup file to add the below lines:

exec xmodmap -e "keycode 232 = F14 " &
exec xmodmap -e "keycode 233 = F15 " &

somewhere before

exec fluxbox

With the above commands, you are telling xorg to map the keycodes to a new key (even if that key doesnt exist on our keyboard). From now on, fluxbox will recognize FN+F4 (keycode 232) as F14 and FN+F5 (keycode 233) as F15.

At this point, if you have not already installed xorg-xbacklight, do it now.

The final step is to map our new keys to specific commands. Edit your ~/.fluxbox/keys so that you can add the below:

None F14 : ExecCommand xbacklight -dec 5
None F15 : ExecCommand xbacklight -inc 5

and restart your fluxbox !

June 17, 2015 09:55 PM

Errata Security

How would you use Lua scripting in a DNS server?

I'm currently putting Lua into a DNS server, and I'm trying to figure out how people would use it.

A typical application would be load-balancing. How I would do this is to create a background Lua thread that frequently (many times a second) queried an external resource to discover current server utilitzation, then rewrote the RRset for that server to put the least utilized server first. This would technically change the zone, but wouldn't be handled as such (i.e. wouldn't trigger serial number changes, wouldn't trigger notification of slave zones).

Such a thread could be used for zone backends. Right now, DNS servers support complex backends like SQL servers and LDAP servers. Instead of making the server code complex, this could easily be done with a Lua thread, that regularly scans an SQL/LDAP server for changes and updates the zone in memory with the changes.

Both these examples are updating static information. One possible alternative is to execute a Lua script on each and every DNS query, such as adding a resource record to a zone that would look like this:

* TXT $LUA:my_script

Every query would cause the script to be executed. There are some issues with this, of course, but for a lot of typical uses, such limitations wouldn't matter. For example, there's complex thread synchronization issues, but I could simply force any use of this feature to go into single threaded mode -- whatever narrow use you'd have for this feature could probably accept the performance hit.

The specific use for this would be, of course, to setup a DNS communication channel. Captive portals forward DNS, but redirect other TCP/UDP packet. Sending messages back and forth through DNS would allow you to do things like tunnel Twitter messages through even without "real" Internet access. As well know, people in the past have written entire VPNs through DNS this way, with custom DNS stacks.

These are my ideas. Maybe you could post some other ideas. I'm looking for either a problem you want solved (without necessarily dictating the precise solution), or a nifty way of integrating Lua (without necessarily any specific problem in mind).

by Robert Graham ( at June 17, 2015 05:57 PM

Michael Biven

Are We Becoming just the Caretakers of Technology?

Even before the first opening talk at Monitorama I started to feel that some of the technology we use have become more magic than science to many of us, including myself. And that the tempo of work we feel either from ourselves or others prevents our chance to adequately become confident in how these things work.

As soon as I started done this path I saw similarities between it and the decline of knowledge described in Isacc Asimov’s The Foundation series. That decline led to the people running the machines only being able to run simple maintenances on them and not fully understanding or being able to reproduce them.

The machines work from generation to generation automatically, and the caretakers are a hereditary caste who would be helpless if a single D-tube in all that vast structure burned out.

– Isaac Asimov, Foundation

We are all vulnerable to the fear that we’re going to fall behind if we don’t try and stay current with advances in our field of work. But if you’re holding yourself accountable to those fears instead of just acknowledging them and placing them aside how will you ever get past just simple maintenances?

Writing this is a reminder to myself to take the time I need and not settle on being a caretaker.

June 17, 2015 10:48 AM

June 16, 2015

Evaggelos Balaskas

vim modeline

a back to vim basics post !


It’s quite obvious that we need different vimrc settings for different files/languages. This is mostly about tabs and characters but it can be extended to more. For example in pytnon (ansible etc) you need to replace the tab to four or eight characters. Most of us are using something like this:

:set tabstop=4 shiftwidth=4  softtabstop=4 expandtab

every time we open a python related file.

But you can set your own options in every file using a comment in the end or in the begging of the file. Like this one:


# vim: tabstop=4 expandtab shiftwidth=4 softtabstop=4

... (awesome code) ...

This is called: modeline and is something magic!
Just add the below line to your ~./vimrc file or if you need a more global setting append it to your /etc/vimrc

set modeline
Tag(s): vim, modeline

June 16, 2015 03:08 PM

June 14, 2015


Impostor syndrome and applying for jobs

Over the weekend I ran into an article on the Safari Books blog about writing job postings. The big recommendation they put forth was a simple one: the 'requirements' section on every job-posting is where you lose most of the qualified candidates for the job who then elect not to apply. Wouldn't it be nice if you rephrased it to be inclusive? I tweeted about it.

Responses came in two versions:

  1. Favorites.
  2. People telling me I'm senior enough to know better about how the requirements game works.

Yes, I'm pretty senior now. However, in the 15+ years I've been a sysadmin I've job-hunted as one only three times.

Time 1: One app, one offer. It wasn't really a job-hunt so much as a pounce-kill.
Time 2: Was in the middle of a huge recession, I had a stale skill-set, and even in the technology I had skills in I didn't have experience with the hot newness. A startup took a chance on me. That search took better than two years.
Time 3: I was terminated and needed another job RIGHT NOW. It was also a hot market, and I had relevant skills. The firms that gave me offers (I had three) all were applied to in the first week of my search. It only took as long as it did to start working due to the Thanksgiving and Christmas holidays getting in the way of everything. That search took six weeks, of which only three were active search/apply/interview focused weeks.

One of the replies is very good at capturing the spirit of the requirements game as it exists now:

True, that's what the startup that hired me did. They needed a blend of Windows and Linux, and apparently I talked a good enough game they hired me even though I didn't exactly have much being-paid-for-it Linux experience at the time (this was one of the big reasons I left WWU, by the way; they wouldn't let me get out of the Windows box). That job posting? It had a 'Qualifications' section, and didn't mention operating system! This was my in!

I haven't done enough hiring or searching to know how flexible 'requirements' are. If I hit every point, I make sure to mention that in my cover-letter. If I hit all but one, I'm worried but if the one doesn't sound mission-critical I'll still drop an app on it. If I'm missing two... I'll apply only if I really want to work there, or I have some other clue that the 'requirements' are actually a wish-list (big hint: 20 items in the requirements list).

Here's the thing though. If you're suffering impostor syndrome, and I sure as hell was during the Time 2 search, it's extremely easy to talk yourself out of thinking you're good enough for a position.

Do not bother applying unless you have...

  • 6 years of Enterprise Linux administration.
  • 5 years of Python scripting development.
  • 5 years of Postgres administration.
  • 3 years of Chef.

That's what a 'Requirements' section looks like. It takes a sense of entitlement to see that list and add, "...or can talk us into taking you anyway" to the bolded text.

I have one of those bullet-points. However, I do have Ruby, MySQL, and Puppet. Is that enough to take a chance on the position, or are they dead set on not having to train in a new-hire on those things? Can't tell, not going to bother going to all the effort of crafting a resume and coverletter just to be told 'Go away'.

Or maybe I tell my impostor syndrome to go hide in a corner somewhere and trust in my interview skills to win me a chance.

By changing the 'Requirements' away from a checklist of skills, and towards the qualities you need in a new-hire, you remove a big barrier to application. You'll get apps from people in non-traditional career-paths, like the person with a Masters in History but spent the last six years doing statistical analysis automation in AWS on a grant, and kept getting consults from other academic computation people for automating things.

I keep hearing there is a real talent crunch for senior people these days. Doesn't it make sense to encourage applications, rather than discourage them?

by SysAdmin1138 at June 14, 2015 02:40 PM

Strong SSL Security on nginx

This tutorial shows you how to set up strong SSL security on the nginx webserver. We do this by updating OpenSSL to the latest version to mitigate attacks like Heartbleed, disabling SSL Compression and EXPORT ciphers to mitigate attacks like FREAK, CRIME and LogJAM, disabling SSLv3 and below because of vulnerabilities in the protocol and we will set up a strong ciphersuite that enables Forward Secrecy when possible. We also enable HSTS and HPKP. This way we have a strong and future proof ssl configuration and we get an A+ on the Qually Labs SSL Test.

June 14, 2015 12:00 AM

Stong SSL Security on lighttpd

This tutorial shows you how to set up strong SSL security on the lighttpd webserver. We do this by updating OpenSSL to the latest version to mitigate attacks like Heartbleed, disabling SSL Compression and EXPORT ciphers to mitigate attacks like FREAK, CRIME and LogJAM, disabling SSLv3 and below because of vulnerabilities in the protocol and we will set up a strong ciphersuite that enables Forward Secrecy when possible. We also enable HSTS and HPKP. This way we have a strong and future proof ssl configuration and we get an A+ on the Qually Labs SSL Test.

June 14, 2015 12:00 AM

Strong SSL Security on Apache2

This tutorial shows you how to set up strong SSL security on the Apache2 webserver. We do this by updating OpenSSL to the latest version to mitigate attacks like Heartbleed, disabling SSL Compression and EXPORT ciphers to mitigate attacks like FREAK, CRIME and LogJAM, disabling SSLv3 and below because of vulnerabilities in the protocol and we will set up a strong ciphersuite that enables Forward Secrecy when possible. We also enable HSTS and HPKP. This way we have a strong and future proof ssl configuration and we get an A+ on the Qually Labs SSL Test.

June 14, 2015 12:00 AM

June 13, 2015

Steve Kemp's Blog

I'm still moving, but ..

Previously I'd mentioned that we were moving from Edinburgh to Newcastle, such that my wife could accept a position in a training-program, and become a more specialized (medical) doctor.

Now the inevitable update: We're still moving, but we're no longer moving to Newcastle, instead we're moving to Helsinki, Finland.

Me? I care very little about where I end up. I love Edinburgh, I always have, and I never expected to leave here, but once the decision was made that we needed to be elsewhere the actual destination does/didn't matter too much to me.

Sure Newcastle is the home of Newcastle Brown Ale, and has the kind of proper-Northern accents I both love and miss but Finland has Leipäjuusto, Saunas, and lovely people.

Given the alternative - My wife moves to Finland, and I do not - Moving to Helsinki is a no-brainer.

I'm working on the assumption that I can keep my job and work more-remotely. If that turns out not to be the case that'll be a real shame given the way the past two years have worked out.

So .. 60 days or so left in the UK. Fun.

June 13, 2015 11:51 AM

June 12, 2015

Evaggelos Balaskas

Changing SSH Host keys

The inspiration for this post comes from Kees Cook’s tweet about having

VisualHostKey yes

on his ~/.ssh/config file.

I’ve played with this option in the past, but having some scripts running over ssh, I was afraid about parsing the “wrong” things on the output.

I’ve enabled this option again this evening, so the head of my ~/.ssh/config looks like:

Host *
	VisualHostKey yes
	Compression yes

I started to ssh login in to a few machines, just to see the output.

A sample output, looks like this:

+---[RSA 2048]----+
|.E       . . o   |
|= . .   . . o o  |
| +   o .  ..o. . |
|  o . o . .*.    |
|   .   +S...*. o |
|      . ...+o.+oo|
|        . +o  +.B|
|       . + oo+ +=|
|        . o.=o. .|

RSA 2048 is the size of the servers public key and you can check the size of the servers key -as of course yours too- with this command:

# ssh-keygen -l -f /etc/ssh/

on your local machine:

> ssh-keygen -l -f ~/.ssh/

I have changed a few times my ssh key pair (you must remember to append your new public key to your server authorized_keys, before removing your old key) but I never changed the servers key pairs.

After searching online for a few minutes to educate my self on the matter, seems that when your (in this case) centos machine is starting ssh daemon for the first time, it creates new ssh key pairs.

The procedure is really easy, but before doing anything, we need to edit

/etc/init.d/sshd to add a 4096 bit keysize for SSHv2 RSA:

echo -n $"Generating SSH2 RSA host key: "
rm -f $RSA_KEY
if test ! -f $RSA_KEY && $KEYGEN -q -b 4096 -t rsa -f $RSA_KEY -C '' -N '' >&/dev/null; then

The final steps are these:

> ssh linuxbox
> sudo -s
# cd /etc/
# tar cf ssh.tar ssh
# cd ssh
# rm -f ssh_host_*
# service sshd restart

If you test your ssh connection, you will get an ugly message:


BUT you have to see this error msg.

If you dont, then you have a very serious problem !

Edit your ~/.ssh/known_hosts to get rid of this message and delete the previous record about the old server ssh public key.

Try again, now it should show you something like:

Are you sure you want to continue connecting (yes/no)? yes

Type yes and memorize your new VisualHostKey !

Tag(s): ssh, centos

June 12, 2015 09:54 PM

June 11, 2015

Joe Topjian




Waffles is a simple configuration management system written in Bash. It’s small, simple, and has allowed me to easily sketch out various environments that I want to experiment with. If you want to jump right in, head over to the Github page, particularly the docs section. The rest of this article will cover the history and some personal thoughts on why I created it.

Update: I apologize if Jekyll flags this as a new post in the RSS feed. After originally publishing this article, I thought of a few other items I wanted to mention. In addition, I noticed Waffles was posted to Reddit, so I wanted to address a few of the comments made.

Defining the Problem

The last article I wrote, Puppet Infrastructure 2015, was quite a beast. In a way, it was a cathartic exercise to write down all of my practices to sanely work with Puppet every day. After I published the article, I couldn’t get over the fact that all of those practices are just for Puppet – not any of the services Puppet sets up and configures. It didn’t sit right with me, so I began to look into why I found configuration management systems so complex.

“Complex” and “Simple” are subjective terms. One core focus was the fact that Omnibus and all-in-one packages are becoming a popular way to ease the installation and configuration of the configuration management system. In my opinion, when configuring your configuration management system becomes so involved that it’s easier to just have an all-in-one installation, that’s “complex”. I wanted to see if it was possible to escape that. I wanted to see if it was possible to create a configuration management system that was able to work out of the box on modern Linux systems and able to be installed by simply cloning a git repository.

Secondly, I wanted to see if it was possible to strip “resource models” down to more simple components. Some systems have modeled resources in a specific language, like Python, while others have chosen to use a subprocess to interact with the best native command available. Both methods have their merits, though I favor the latter more. I wanted to take that method further: why not just use a Unix shell and related tools? After all, for decades, the standard way of interacting with a Unix-based system has been through the command-line.

That’s not to say that resource models in configuration management systems are the way they are for no reason. Take Puppet’s Types and Providers system, for example. Decoupling the “type” and “provider” has enabled Puppet to provide a common resource interface for a variety of platforms and backends. In addition, the Types and Providers system provides the user with a way to easily manage all resources of a given type on a system, provide input validation, and a lot of other features.

I have nothing but respect for the Types and Providers system. But I still wanted to see if it was possible to create a tool that provided the same core idea (abstract a resource into a simple model that allowed for easy management) in a more simple way. It’s an experiment to see what happens when a robust catalog system, input validation system, etc were removed for something more bare. Would chaos ensue?

Third, I wanted to break out of the “compiled manifest” and “workflow” views of configuration management systems. Every configuration management system has some sort of DSL, and for a good reason. You can read about the decision to use a DSL with Puppet here. I have nothing against DSLs (you can see how plain Bash evolved into the Bash-based DSL in Waffles here), but I wanted to see if it was possible to not use a compiled manifest or workflow, and instead use a plain old top-down execution sequence.

With these thoughts in mind, I set off to see what I could build.

As a real-world test to validate my solution, I created a detailed list of steps that are required to build a simple, yet robust, Memcached service (this is the reason why there are so many references to Memcached in the Waffles documentation).

You can easily get away with installing Memcached by doing:

$ sudo apt-get install -y memcached

But what if you also need to:

  • Edit /etc/memcached.conf
  • Have other common packages installed (logwatch, Postfix, fail2ban, Sensu, Nagios, etc)
  • Have some standard users or SSH keys installed
  • Have some standard firewall rules installed

You can see how the idea of a “simple” Memcached service quickly becomes a first-class service in your environment. But does that mean the configuration management system must be complex to satisfy this?

Attempted Solutions

First Version

My initial solution was written in Golang. This was because I was doing a lot of work with OpenStack and Terraform and it minimized context switching to stay with Go. I ended up with a working system that compiled shell scripts together into a master shell script “role” (my term for a fully-defined node) and referenced data stored in a hierarchical YAML system (a very watered down Hiera).

It looked like this:

    server_types: [lxc]
    environments: [production, testing]

The program would then search a structured directory for shell scripts that contained memcached, lxc, production, and testing and create two master scripts:


I didn’t like it at all. The first thing I removed was the YAML data system. I felt that it carried too much overhead. Then I decided to get rid of Go altogether. The Go component was only being used to organize shell scripts. Why not just make everything in shell?

Second Version

The next major iteration was completely written in Bash. It used Bash variables to store data and Bash arrays to store a list of scripts that should be executed. Here’s what a “role” looked like:



The ATTRIBUTES hash stored key/value pairs that described unique features of the role. The RUN array stored a list of scripts that would be executed in a top-down order. This was working very well and it enabled me to easily deploy all sorts of environments. I wasn’t totally happy with the design, but I couldn’t figure out what exactly I didn’t like.


That got put on hold when I ran into a major roadblock: I was deploying a Consul cluster when I ran into the need to use a JSON-based configuration file. What would be the best way to handle it?

  • Static files meant data embedded inside the file and outside of the data system
  • Templates meant another layer of programming logic
  • External languages broke the Bash-only feature

I was stumped for a few days until I realized: Augeas! I’ve always had a lot of respect for the Augeas project but never had a reason to use it – until now. Even better, Augeas was able to cleanly parse and write JSON files. So I made Augeas an optional, but encouraged, component of Waffles.

Third Version

Now back to figuring out why I didn’t like the current iteration. I realized I didn’t like the format of the “role”. I wanted the “role” to look more like a shell script and not like a metadata file that describes a system.

So I made some changes and was happier. The above RabbitMQ role now looked like: common openstack/rabbitmq

stdlib.profile site/acng
stdlib.profile site/packages
stdlib.module rabbitmq
stdlib.module rabbitmq/repo
stdlib.module rabbitmq/server
stdlib.module openstack/rabbitmq

Fourth and Final Version

There was still one part that bugged me: modules. In the above example, rabbitmq and openstack were both modules. I didn’t like how I had profiles and modules mixed in the role. I refactored the above so that profiles were just an abstraction of modules, like in Puppet, but I didn’t like that either. Finally, I decided to get rid of modules altogether. You can read more about this decision here.

So modules went away and the above turned into: common openstack/rabbitmq

stdlib.profile common/acng
stdlib.profile common/packages
stdlib.profile rabbitmq/repo
stdlib.profile rabbitmq/server
stdlib.profile openstack/rabbitmq

At that point I was very satisfied with how it looked. I used it for a week or two and was still happy and decided to finally release it publicly.

Current Status

Now I have a tool that I find fits my definition of “simple” and “intuitive”.

But Waffles is nowhere near complete. The only milestone that has been reached is that I’m happy with its core design. Waffles certainly isn’t at feature parity with other configuration management systems. For some features, this is only a matter of time. For other features, they just aren’t applicable to Waffles’s core ideas. For example, I want to have some kind of pull-based mechanism so clients can contact a central Waffles server. At the same time, I don’t feel a PuppetDB or Etcd database belongs in Waffles.

Is Waffles Competing?

Waffles is just another tool. I have no intention to market Waffles as a competitor to other configuration management systems. The only reason Puppet has been called out throughout this article is because it’s the system that I’m most knowledgeable with.

Is Waffles Better Than X?

Again, Waffles is just another tool that does a certain action in a certain way. The only thing that matters is what tool you find most useful. I have nothing but respect for every other configuration management system available today. If it wasn’t for those projects, I wouldn’t have been able to learn what I like and don’t like in order to create my own.

As mentioned above, Puppet is called out a lot because it’s the system I’m most knowledgeable with. Puppet is a great tool and I still use it everyday. I’ve also spent time with Chef, Ansible, Juju, and Salt and find them to be great tools, too.

I will make one comment about Ansible, though, because comparisons will be made (I’m assuming because Ansible is known to be a small and simple configuration management system): I feel Waffles and Ansible are two separate tools that do things very differently. I feel the biggest difference is that Ansible is a workflow-based system while Waffles is just a bunch of shell functions that execute from top to bottom.

I take no offence if you don’t like Waffles. You can see from a lot of the other tools and code I’ve created that I do things very differently sometimes – that’s just me.


This article gave the history and thought process behind a new project of mine: Waffles. Try it out and tell your friends. Feedback is welcome!

June 11, 2015 06:00 AM

June 10, 2015

Sean's IT Blog

Countdown to Virtualization Field Day 5–#VFD5

In two weeks, I get the pleasure of joining some awesome members of the virtualization community in Boston for Virtualization Field Day 5. 

If you’re not familiar with Virtualization Field Day, it is one of the many Tech Field Day events put on by Stephen Foskett (@sfoskett) and the crew at Gestalt IT.  These events bring together vendors and members from the community to have technical discussions about the vendor’s products and offerings.   These events are streamed live on the Tech Field Day website, and there are many opportunities to interact with the delegates via Twitter by following the #VFD5 hashtag.

The vendors that will be sponsoring and presenting at Virtualization Field Day 5 are:



logo-wpcf_100x21        med-vert-notag-wpcf_93x60 PernixData_Logo_Color
Scale_Logo_High_Res-wpcf_100x38 v2Ravello_Logo_large-wpcf_100x27 VMTurboLogoSm

I will be joining an awesome group of delegates:

This will be my first time attending Virtualization Field Day as a delegate.  I’ve previously watched the events online and interacted with the delegates on Twitter. 

Keep watching this space, and the #VFD5 hashtag on Twitter, as there will be a lot more exciting stuff.

by seanpmassey at June 10, 2015 06:22 PM


xhyve – Lightweight Virtualization on OS X Based on bhyve

The Hypervisor.framework user mode virtualization API introduced in Mac OS X 10.10 (Yosemite) cannot only be used for toy projects like the hvdos DOS Emulator, but is full-featured enough to support a full virtualization solution that can for example run Linux.

xhyve is a lightweight virtualization solution for OS X that is capable of running Linux. It is a port of FreeBSD’s bhyve, a KVM+QEMU alternative written by Peter Grehan and Neel Natu.

  • super lightweight, only 230 KB in size
  • completely standalone, no dependencies
  • the only BSD-licensed virtualizer on OS X
  • does not require a kernel extension (bhyve’s kernel code was ported to user mode code calling into Hypervisor.framework)
  • multi-CPU support
  • networking support
  • can run off-the-shelf Linux distributions (and could be extended to run other operating systems)

xhyve may make a good solution for running Docker on your Mac, for instance.

Running Tiny Core Linux on xhyve

The xhyve repository already contains a small Linux system for testing, so you can try out xhyve by just typing these few lines:

$ git clone
$ cd xhyve
$ make
$ ./

And you will see Tiny Core Linux booting in your terminal window:

Initializing cgroup subsys cpuset
Initializing cgroup subsys cpu
Initializing cgroup subsys cpuacct
Linux version 3.16.6-tinycore64 (tc@box) (gcc version 4.9.1 (GCC) ) #777 SMP Thu Oct 16 10:21:00 UTC 2014
Command line: earlyprintk=serial console=ttyS0 acpi=off
e820: BIOS-provided physical RAM map:
BIOS-e820: [mem 0x0000000000000000-0x000000000009fbff] usable
BIOS-e820: [mem 0x0000000000100000-0x000000003fffffff] usable
NX (Execute Disable) protection: active
SMBIOS 2.6 present.
AGP: No AGP bridge found
e820: last_pfn = 0x40000 max_arch_pfn = 0x400000000
x86 PAT enabled: cpu 0, old 0x7040600070406, new 0x7010600070106
CPU MTRRs all blank - virtualized system.


 //\   Core is distributed with ABSOLUTELY NO WARRANTY.


To shut down the VM and exit to the Mac’s command line, enter:

$ sudo halt

Running Ubuntu on xhyve

You can also install a more complete Linux distribution on xhyve. The tricky bit is that xhyve doesn’t come with a BIOS or EFI booter, so it is necessary to extract the kernel and initrd from the Linux image and pass them to xhyve manually.

First download Ubuntu Server (the desktop version doesn’t support the text mode installer) into the directory “ubuntu” inside the “xhyve” directory:

$ ls -l
total 1218560
-rw-r--r--@ 1 mist  staff  623902720  6 Jun 22:14 ubuntu-14.04.2-server-amd64.iso

We need to extract the kernel and initrd, which is a little tricky, because OS X doesn’t recognize the hybrid file system on the image without a little hack:

$ dd if=/dev/zero bs=2k count=1 of=/tmp/tmp.iso
$ dd if=ubuntu-14.04.2-server-amd64.iso bs=2k skip=1 >> /tmp/tmp.iso
$ hdiutil attach /tmp/tmp.iso
$ cp /Volumes/Ubuntu-Server\ 14/install/vmlinuz .
$ cp /Volumes/Ubuntu-Server\ 14/install/initrd.gz .

Create a virtual hard disk image (8 GB in the example):

$ dd if=/dev/zero of=hdd.img bs=1g count=8

Then create a script to run xhyve with the correct arguments for the installer:


CMDLINE="earlyprintk=serial console=ttyS0 acpi=off"

MEM="-m 1G"
#SMP="-c 2"
NET="-s 2:0,virtio-net"
IMG_CD="-s 3,ahci-cd,ubuntu/ubuntu-14.04.2-server-amd64.iso"
IMG_HDD="-s 4,virtio-blk,ubuntu/hdd.img"
PCI_DEV="-s 0:0,hostbridge -s 31,lpc"
LPC_DEV="-l com1,stdio"


You will want networking enabled, so it’s easiest to run the script as root (this requirement is lifted if you codesign the binary):

$ sudo ./

You will see the Ubuntu text mode installer:

  ┌───────────────────────┤ [!!] Select a language ├────────────────────────┐
  │                                                                         │
  │ Choose the language to be used for the installation process. The        │
  │ selected language will also be the default language for the installed   │
  │ system.                                                                 │
  │                                                                         │
  │ Language:                                                               │
  │                                                                         │
  │                               C                                         │
  │                               English                                   │
  │                                                                         │
  │     <Go Back>                                                           │
  │                                                                         │

<Tab> moves; <Space> selects; <Enter> activates buttons

All answers should be straightforward, and the defaults are usually fine. Make sure to select “Yes” when asked “Install the GRUB boot loader to the master boot record”.

At the very end, on the “Installation complete” screen, select “Go back” and “Execute a shell”, so you can copy the installed kernel and initrd to the Mac side. In the VM, type this:

# cd /target
# sbin/ifconfig
# tar c boot | nc -l -p 1234

On the Mac, type this, replacing the IP with the output from ifconfig before:

$ cd ubuntu
$ nc 1234 | tar x

In the VM, exit the shell:

# exit

Then select “Finish the installation”.

To run the Ubuntu installation from the virtual hard disk, create the following script, fixing up the kernel and initrd version numbers:


CMDLINE="earlyprintk=serial console=ttyS0 acpi=off root=/dev/vda1 ro"

MEM="-m 1G"
#SMP="-c 2"
NET="-s 2:0,virtio-net"
IMG_HDD="-s 4,virtio-blk,ubuntu/hdd.img"
PCI_DEV="-s 0:0,hostbridge -s 31,lpc"
LPC_DEV="-l com1,stdio"


Then run the script:

$ sudo ./

To make your Linux installation useful, you may want to install an SSH server:

$ sudo apt-get install openssh-server

Or install a full UI that you can access using VNC:

$ sudo apt-get install xubuntu-desktop vnc4server

Then run the VNC server:

$ vnc4server :0 -geometry 1024x768

And conntect to it by pasting this into Finder’s Cmd+K “Connect to Server” dialog:


If you also follow skerit’s Ubuntu VNC tutorial, you’ll get to a desktop like this.

Next Steps

xhyve is very basic and lightweight, but it has a lot of potential. If you are a developer, you are welcome to contribute to it. A list of current TODOs and ideas is part of the README file in the repository.

by Michael Steil at June 10, 2015 05:09 AM

June 09, 2015


High available NFS server: Setup Corosync & Pacemaker


This post is the continuation of the series of posts for setup a High available NFS server, check the first post to setup the iSCSI storage part:
On this post I’ll explain how to setup the NFS cluster and the failover between two servers setup on the first post, using Corosync as the cluster engine and Pacemaker as the resource manager of the cluster.


Corosync is an open source cluster engine which allows to share messages between the different servers of the cluster to check the health status and inform the other components of the cluster in case one of the servers goes down and starts the failover process.


Pacemaker is an open source high availability resource manager. The task of Pacemaker is to keep the configuration of all the resources of the cluster and the relations between the servers and resources. For example if we need to setup a VIP (virtual IP), mount a filesystem or start a service on the active node of the cluster, pacemaker will setup all the resources assigned to the server in the order we specify on the configuration to ensure all the services will be started correctly.

Resource Agents

They’re just Scripts that manages different services. That scripts are based on the OCF standard: The system comes already with some scripts, where most of the time will be enough for typical cluster setups, but of course that’s possible to develop a new one depending on your needs and requirements.


So after this small introduction about the cluster components, let’s get started with the configuration:

Corosync configuration

– Install package dependencies:

# aptitude install corosync pacemaker

– Generate a private key to ensure the authenticity and privacy of the messages sent between the nodes of the cluster:

# corosync-keygen –l

NOTE: This command will generate the private key on the path: /etc/corosync/authkey copy the key file to the other server.

– Edit /etc/corosync/corosync.conf:

# Please read the openais.conf.5 manual page

totem {
version: 2

# How long before declaring a token lost (ms)
token: 3000

# How many token retransmits before forming a new configuration
token_retransmits_before_loss_const: 10

# How long to wait for join messages in the membership protocol (ms)
join: 60

# How long to wait for consensus to be achieved before starting a new round of membership configuration (ms)
consensus: 3600

# Turn off the virtual synchrony filter
vsftype: none

# Number of messages that may be sent by one processor on receipt of the token
max_messages: 20

# Limit generated nodeids to 31-bits (positive signed integers)
clear_node_high_bit: yes

# Enable encryption
secauth: on

# How many threads to use for encryption/decryption
threads: 0

# This specifies the mode of redundant ring, which may be none, active, or passive.
rrp_mode: active

interface {
# The following values need to be set based on your environment
ringnumber: 0
mcastport: 5405

nodelist {
node {
ring0_addr: nfs1-srv
nodeid: 1
node {
ring0_addr: nfs2-srv
nodeid: 2

amf {
mode: disabled

quorum {
# Quorum for the Pacemaker Cluster Resource Manager
provider: corosync_votequorum
expected_votes: 1

service {
# Load the Pacemaker Cluster Resource Manager
ver: 0
name: pacemaker

aisexec {
user: root
group: root

logging {
fileline: off
to_stderr: yes
to_logfile: no
to_syslog: yes
syslog_facility: daemon
debug: off
timestamp: on
logger_subsys {
subsys: AMF
debug: off
tags: enter|leave|trace1|trace2|trace3|trace4|trace6


Pacemaker configuration

– Disable the quorum policy, since we need to deploy a 2-node configuration:

# crm configure property no-quorum-policy=ignore

– Setup the VIP resource of the cluster:

# crm configure primitive p_ip_nfs ocf:heartbeat:IPaddr2 params ip="" cidr_netmask="24" nic="eth0" op monitor interval="30s"

– Setup the init script for the NFS server:

# crm configure primitive p_lsb_nfsserver lsb:nfs-kernel-server op monitor interval="30s"

NOTE: The nfs-kernel-server init script will be managed by the cluster, so disable the service to start it at boot time using update-rc.d utility:

# update-rc.d -f nfs-kernel-server remove


– Configure the mount point for the NFS export:

# crm configure primitive p_fs_nfs ocf:heartbeat:Filesystem params device="/dev/mapper/nfs1" directory="/mnt/nfs" fstype="ext3" op start interval="0" timeout="120" op monitor interval="60" timeout="60" OCF_CHECK_LEVEL="20" op stop interval="0" timeout="240"

– Configure a resource group with the nfs service, the mountpoint and the VIP:

# crm configure group g_nfs p_fs_nfs p_lsb_nfsserver p_ip_nfs meta target-role="Started"


– Prevent healthy resources from being moved around the cluster configuring a resource stickiness:

# crm configure rsc_defaults resource-stickiness=200


Check cluster status

– Check the status of the resources of the cluster:

# crm status
Last updated: Wed Jun 3 21:44:29 2015
Last change: Wed Jun 3 16:56:15 2015 via crm_resource on nfs1-srv
Stack: corosync
Current DC: nfs1-srv (1) - partition with quorum
Version: 1.1.10-42f2063
2 Nodes configured
3 Resources configured

Online: [ nfs1-srv nfs2-srv ]

Resource Group: g_nfs
p_lsb_nfsserver (lsb:nfs-kernel-server): Started nfs2-srv
p_ip_nfs (ocf::heartbeat:IPaddr2): Started nfs2-srv
p_fs_nfs (ocf::heartbeat:Filesystem): Started nfs2-srv


Cluster failover

– If resources are in nfs2-srv and we want to failover to nfs1-srv:

# crm resource move g_nfs nfs1-srv

– Remove all constraints created by the move command:

# crm resource unmove g_nfs


Resulting configuration

# crm configure show
node $id="1" nfs1-srv
node $id="2" nfs2-srv
primitive p_fs_nfs ocf:heartbeat:Filesystem \
params device="/dev/mapper/nfs-part1" directory="/mnt/nfs" fstype="ext3" options="_netdev" \
op start interval="0" timeout="120" \
op monitor interval="60" timeout="60" OCF_CHECK_LEVEL="20" \
op stop interval="0" timeout="240"
primitive p_ip_nfs ocf:heartbeat:IPaddr2 \
params ip="" cidr_netmask="24" nic="eth0" \
op monitor interval="30s"
primitive p_lsb_nfsserver lsb:nfs-kernel-server \
op monitor interval="30s"
group g_nfs p_lsb_nfsserver p_ip_nfs \
meta target-role="Started"
colocation c_nfs_on_fs inf: p_lsb_nfsserver p_fs_nfs
order o_volume_before_nfs inf: p_fs_nfs g_nfs:start
property $id="cib-bootstrap-options" \
dc-version="1.1.10-42f2063" \
cluster-infrastructure="corosync" \
rsc_defaults $id="rsc-options" \



This post is a second part of the series of post High available NFS server, find the first part here.

by ivanmp91 at June 09, 2015 07:43 PM

SquashFS: Mountable compressed read-only filesystem

SquashFS is generally used for LiveCDs or embedded devices to store a compressed read-only version of a file system. This saves space at the expense of slightly slower access times from the media. There's another use for SquashFS: keeping an easily accessible compressed mounted image available. This is particularly useful for archival purposes such as keeping a full copy of an old server or directory around.

Usage is quite easy under Debian-derived systems. First we install the squashfs-tools package

$ sudo apt-get install squashfs-tools

Create an compressed version of a directory:

$ sudo mksquashfs /home/fboender/old-server_20150608/ old-server_20150608.sqsh

Remove the original archive:

$ sudo rm -rf /home/fboender/old-server_20150608

Finally, mount the compressed archive:

$ sudo mkdir /home/fboender/old-server_2015060
$ sudo mount -t squashfs -o loop old-server_20150608.sqsh /home/fboender/old-server_2015060

Now you can directly access files in the compressed archive:

$ sudo ls /home/fboender/old-server_2015060

The space savings are considerable too.

$ sudo du -b -s /home/fboender/old-server_2015060
17329519042	/home/fboender/old-server_2015060
$ sudo ls -l old-server_20150608.sqsh
-rw-r--r-- 1 root root 1530535936 Jun  8 12:45

17 Gb for the full uncompressed archive versus only 1.5 Gb for the compressed archive. We just saved 15.5 Gb of diskspace. .

 Optionally, you may want to have it mounted automatically at boottime:

$ sudo vi /etc/fstab
/home/fboender/old-server_20150608.sqsh   /home/fboender/old-server_2015060        squashfs        ro,loop 0       0

If the server starts up, the archive directory will be automatically mounted.

by admin at June 09, 2015 07:41 PM

June 07, 2015

Sarah Allen

the transformative power of games

boy's face lit by the light of a video game -- sense of urgency, a little bit of fear, but intense concentrationJane McGonigal’s TED talk “Gaming can make a better world” has some highlights from her research on what games make us good at. She talks about the “epic win,” an extraordinary outcome that you didn’t believe was even possible until you achieved it — almost beyond your threshold of imagination, something that teaches you what you’re truly capable of.

“Gamers always believe that an epic win is possible, and that it’s always worth trying, and trying now.”

What capabilities does gaming create? what are their superpowers?

  • Urgent optimism The desire to act immediately to tackle an obstacle, combined with the belief that we have a reasonable hope of success.
  • Tight social fabric Playing a game together builds trust and cooperation. Playing creates strong social bonds. “We trust that they will spend their time with us, that they will play by the same rules, value the same goal, stay with the game until it’s over.”
  • Blissful productivity Humans are optimized to do hard and meaningful work. The average World of Warcraft gamer plays for 22 hours a week hours per week — that’s like a part time job. Gamers are willing to “work” really hard, given the right kind of work.
  • Epic meaning awe-inspiring missions, planetary scale stories. World of Warcraft has the 2nd largest wiki in the world, with almost 80,000 articles. McGonigal describes this as building an epic story.

10,000 hours

The average young person in a country with a “strong gamer culture” will have spent 10,000 hours playing online games by the age of 21. This is an interesting number:

  • 10,080 hours of school from 5th – 12th grade (with perfect attendance)
  • Malcolm Gladwell’s theory of success based on cognitive science research that with 10,000 hours of effortful study anyone could become a virtuoso by age 21

She challenges us to think about what we might do with this incredible human resource. At the institute of the future, she has invented a few games, which have been played by thousands of people, focused on solving serious real, world problems. I wonder how a new generation with these problem-solving skills and ability for extended focus will transform our society. With any luck, we’ll successfully game-ify the real world, rather than creating ever-increasingly delicious virtual escapes.

An Epic Mission

I love the way she describes the elements of online games that make it so compelling:

  • Lots and lots of different characters who are willing to trust you with a world-saving mission, right away. B
  • You get a mission that is perfectly matched with your current level in the game
  • They never give you a challenge you can’t achieve.
  • You are challenged with what you are on the verge of what you’re capable of, so you have to try hard.

“There’s no unemployment in World of Warcraft; no sitting around, wringing your hands — there’s always something specific and important to be done. There are also tons of collaborators. Everywhere you go, hundreds of thousands of people ready to work with you to achieve your epic mission.”

How can we apply these ideas to make our real lives and real challenges more engaging?

Watch Jane McGonigal’s whole talk: Gaming can make a better world

The post the transformative power of games appeared first on the evolving ultrasaurus.

by sarah at June 07, 2015 10:36 PM

June 04, 2015


High available NFS server: Setup iSCSI & multipath


On this series of post I’ll explain how to setup a high available and redundant NFS cluster using iSCSI with DM-Multipath and Corosync & Pacemaker to manage the cluster and the resources associated. The objective of this scenario it’s create a redundant and fault tolerant NFS storage with automatic failover, to ensure the maximum availability of the NFS exports most of the time.

For this environment I’ve used two servers running Ubuntu 14.04.2 LTS with two NICs configured on each server, one to provide the NFS service to the clients and another one to connect with the iSCSI SAN network. For the iSCSI SAN storage device, I’ve already setup two physical adapters and two network interfaces for each adapter for redundant network access and provide two physical paths to the storage system. Both NFS servers will have attached the LUN device using a different InitiatorName and will have setup the device mapper multipathing (DM-Multipath), which allows you to configure multiple I/O paths between server nodes and storage arrays into a single device. These I/O paths are physical SAN connections that can include separate cables, switches, and controllers, so basically It is as if the NFS servers had a single block device.


The cluster software used is Corosync and the resource manager Pacemaker, where Pacemaker will be the responsible to assign a VIP (virtual ip address), mount the file system from the block device and starts the nfs service with the specific exports for the clients on the active node of the cluster. In case of failure of the active node of the cluster the resources will be migrated to the passive node and the services will continue to operate as if nothing had happened.

This post specifically will cover the configuration part of the iSCSI initiator for both NFS servers and the configuration for the device mapper multipathing, to see the configuration for the cluster with corosync and pacemaker check the second part:

So let’s get started with the setup!

iSCSI initiator configuration

– Install dependencies:

# aptitude install multipath-tools open-iscsi

Server 1

– Edit configuration file /etc/iscsi/initiatorname.iscsi:

Server 2

– Edit configuration file /etc/iscsi/initiatorname.iscsi:

NOTE: initiator identifiers on both servers are different but they are associated with the same LUN device.

– Runs a discovery on iSCSI targets:

# iscsiadm -m discovery -t sendtargets -p
# iscsiadm -m discovery -t sendtargets -p
# iscsiadm -m discovery -t sendtargets -p
# iscsiadm -m discovery -t sendtargets -p

– Connect and login with the iSCSI target:

# iscsiadm -m node -T -p --login
# iscsiadm -m node -T -p --login
# iscsiadm -m node -T -p --login
# iscsiadm -m node -T -p --login

– Check the sessions established with the iSCSI SAN device:

# iscsiadm -m node,1,2,1,2

– At this point the block devices should be available on both servers like a local attached devices, you can check it simply running fdisk:

# fdisk -l

Disk /dev/sdb: 1000.0 GB, 1000000716800 bytes
255 heads, 63 sectors/track, 121576 cylinders, total 1953126400 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000

   Device Boot      Start         End      Blocks   Id  System
/dev/sdb1              63  1953118439   976559188+  83  Linux

Disk /dev/sdc: 1000.0 GB, 1000000716800 bytes
255 heads, 63 sectors/track, 121576 cylinders, total 1953126400 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000

   Device Boot      Start         End      Blocks   Id  System
/dev/sdc1              63  1953118439   976559188+  83  Linux

In my case /dev/sda is the local disk for the server and /dev/sdb and /dev/sdc corresponds to the iSCSI block devices (one device for each adapter). Now We need to setup a device mapper multipath for these two devices, /dev/sdb and /dev/sdc, so in case one of the adapter fails the LUN device will continue working in our system and multipath will switch the used disk for our block device.

Multipath configuration

– We need first to retrieve and generate a unique SCSI identifier to configure on the multipath configuration, running the following command for one of the iSCSI devices:

# /lib/udev/scsi_id --whitelisted --device=/dev/sdb

– Create the multipath configuration file /etc/multipath.conf with the following content:

## This is a template multipath-tools configuration file
## Uncomment the lines relevent to your environment
defaults {
       user_friendly_names yes
       polling_interval        3
       selector                "round-robin 0"
       path_grouping_policy    multibus
       path_checker            directio
       failback                immediate
       no_path_retry           fail
blacklist {
        devnode "^(ram|raw|loop|fd|md|dm-|sr|scd|st)[0-9]*"
        devnode "^hd[a-z][[0-9]*]"

        multipath {
	        # id retrieved with the utility /lib/udev/scsi_id
                wwid                    3600c0ff000d823e5ed6a0a4b01000000
                alias                   nfs

– Restart multipath-tools service:

# service multipath-tools restart

– Check again the disks available in the system:

# fdisk -l

Disk /dev/sdb: 1000.0 GB, 1000000716800 bytes
255 heads, 63 sectors/track, 121576 cylinders, total 1953126400 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000

   Device Boot      Start         End      Blocks   Id  System
/dev/sdb1              63  1953118439   976559188+  83  Linux

Disk /dev/sdc: 1000.0 GB, 1000000716800 bytes
255 heads, 63 sectors/track, 121576 cylinders, total 1953126400 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000

   Device Boot      Start         End      Blocks   Id  System
/dev/sdc1              63  1953118439   976559188+  83  Linux

Disk /dev/mapper/nfs: 1000.0 GB, 1000000716800 bytes
255 heads, 63 sectors/track, 121576 cylinders, total 1953126400 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000

          Device Boot      Start         End      Blocks   Id  System
/dev/mapper/nfs1              63  1953118439   976559188+  83  Linux

Disk /dev/mapper/nfs-part1: 1000.0 GB, 999996609024 bytes
255 heads, 63 sectors/track, 121575 cylinders, total 1953118377 sectors
Units = sectors of 1 * 512 = 512 bytes
Sector size (logical/physical): 512 bytes / 512 bytes
I/O size (minimum/optimal): 512 bytes / 512 bytes
Disk identifier: 0x00000000

Now as you can see we’ve a new block device using the alias setup on the multipath configuration file /dev/mapper/nfs. The disk I’ve partitioned and implemented the filesystem is the block device /dev/mapper/nfs-part1, so you can mount it in your system with the mount utility.

– You can check the health of the multipath block device and check if both devices are operational, running the following command:

# multipath -ll
nfs (3600c0ff000d823e5ed6a0a4b01000000) dm-3 HP,MSA2012i
size=931G features='1 queue_if_no_path' hwhandler='0' wp=rw
`-+- policy='round-robin 0' prio=1 status=active
  |- 6:0:0:0 sdb 8:16 active ready running
  `- 5:0:0:0 sdc 8:32 active ready running


This post is a second part of the series of post High available NFS server, find the second part here.

by ivanmp91 at June 04, 2015 08:09 PM

June 03, 2015

I've released cfgtrack v1.0: Get notified of changes on your server

I needed a simple way of being notified when configuration files had been changed on some servers. Nothing fancy. No configuration management, no intrusion detection, no centralised versioning control repositories. Just a simple email saying what's been changed. I couldn't find a tool that did just that, and didn't require massive amounts of configuration, so I wrote one myself.

I've just released version 1.0 of the tool, which is available in source, Debian, Redhat and zip packages.

Here's how simple it is:

$ sudo cfgtrack track /etc/
Now tracking /etc/

# Make some changes in a file

$ sudo cfgtrack -a -m compare

And I'll get an email in my mailbox if anything's been changed since the last time I ran compare. A diff is included to easily spot what has changed.

Add the above to a daily cronjob and you'll be kept up-to-date about changes to your configuration files. Now you'll have a heads-up if automatic package upgrades modify configuration files or a co-administrator decided to make some changes.

More information is available on the Github project page.



by admin at June 03, 2015 02:00 PM

June 01, 2015

Anton Chuvakin - Security Warrior

Monthly Blog Round-Up – May 2015

Here is my next monthly "Security Warrior" blog round-up of top 5 popular posts/topics this month:
  1. Why No Open Source SIEM, EVER?” contains some of my SIEM thinking from 2009. Is it relevant now? Well, you be the judge.  Current emergence of open source log search tools, BTW, does not break the logic of that post. SIEM requires a lot of work, whether you paid for the software, or not. [179 pageviews]
  2. Simple Log Review Checklist Released!” is often at the top of this list – the checklist is still a very useful tool for many people. “On Free Log Management Tools” is a companion to the checklist (updated version) [136 pageviews]
  3. Top 10 Criteria for a SIEM?” came from one of my last projects I did when running my SIEM consulting firm in 2009-2011 (for my recent work on evaluating SIEM, see this document) [94 pageviews]
  4. My classic PCI DSS Log Review series is always popular! The series of 18 posts cover a comprehensive log review approach (OK for PCI DSS 3.0 as well), useful for building log review processes and procedures , whether regulatory or not. It is also described in more detail in our Log Management book and mentioned in our PCI book (just out in its 4th edition!) [80+ pageviews to the main tag]
  5. “New SIEM Whitepaper on Use Cases In-Depth OUT!” (dated 2010) presents a whitepaper on select SIEM use cases described in depth with rules and reports [using now-defunct SIEM product]; also see this SIEM use case in depth and this for a more current list of popular SIEM use cases. [55 pageviews of total 4310 pageviews to all blog pages]
In addition, I’d like to draw your attention to a few recent posts from my Gartner blog:

Current research on cloud security monitoring:
Past research on security analytics:
Miscellaneous fun posts:

(see all my published Gartner research here)
Also see my past monthly and annual “Top Popular Blog Posts” – 2007, 2008, 2009, 2010, 2011, 2012, 2013, 2014.
Disclaimer: most content at SecurityWarrior blog was written before I joined Gartner on Aug 1, 2011 and is solely my personal view at the time of writing. For my current security blogging, go here.
Previous post in this endless series:

by Anton Chuvakin ( at June 01, 2015 03:03 PM

toolsmith: IoT Fruit - Pineapple and Raspberry

Wifi Pineapple
Raspberry Pi 2

You could call this particular column the Internet of Toolsmith. As much as I am a curmudgeonly buzzword, catch-phrase hater (I lose my mind at RSA and refuse to go any more), the Internet of Things, or IoT is all the rage for good reason. Once obscure items are now connected and as such, at risk. The ability to load a full operating system and a plethora of functionality on a micro device has become trivial thanks to the likes of Raspberry Pi and Arduino. I’d like to point out that the Pwnie Express PwnPlug Elite, built on a Sheevaplug, as discussed in March 2012’s toolsmith, was amongst those devices that met the IoT bar before IoT was all the rage. Kudos to that crazy pack o’ hackers for seeing the imminent future of security challenges with smart devices. In 2013 Chris Clearfield wrote Rethinking Security for the Internet ofThings wherein he stated that “the growing Internet of Things, the connection of physical devices to the internet, will rapidly expand the number of connected devices integrated into our everyday lives. They also have the potential to allow cyber attackers into the physical world in which we live as they seize on security holes in these new systems.” It is in that mindset that we’ll converge security assessment tools and services, as implemented on a couple of tiny devices I’m fond of, with ISSA Journal’s topic of the month. Normally, toolsmith focuses on free and open source tools, and the software we’ll discuss this month continues to meet that bar. That said, it’s impossible to explore IoT without some related “things”, so you’ll need to make a small investment in one or both of the devices we’ll discuss, or experiment similarly on related platforms. If you were to purchase the Wifi Pineapple and the Raspberry Pi 2 (RPI2) kits I own, you’d spend a grand total of $229. Much as the Pwnie Express crew did, the hak5 team started building WiFi penetration testing platforms on tiny hardware as early as 2008. The Raspberry Pi project has enabled all sorts of makers to build miniature attack or assessment systems on devices the size of a pack of playing cards. We’ll drop Kali Linux on a Raspberry Pi 2 here. I chuckled a bit as I wrote this as I was reminded that WiFi Pineapple, intended for WFi hacking, was itself popped at Defcon 22. The language in the resulting message is too salty to print here but it starts with “Dear Lamer” and ends with “criminally insecure” which should convey the general concepts. ;-) That said, the Hak5 team addressed the issues quickly, and the device really is a sound, capable investment; let’s start there.

WiFi Pineapple

Figure 1 – WiFi Pineapple
Wifi Pineapple use is about as easy as plugging in, connecting the included Cat5 cable to a DHCP-enabled NIC, and browsing to “The WiFi Pineapple firmware is a heavily modified version of OpenWRT, packed with tools to aid your pen testing.” Initial username is root, you’ll assign a password during initial setup. I did flash my Pineapple to the latest firmware, 2.3.0 as this was written, using the WiFi Pineapple MK5 Infusion. Using the Network Infusion, I put my Pineapple in Client Mode, so I could connect to the Internet for updates and install additional Infusions. Using the AutoSSG Infusion I setup the AutoSSH service so I could interact with a remote shell and download/upload file via SCP. Real fun with a Wifi Pineapple can be had when you add Infusions. I immediately added sitesurvey, status, monitor, logcheck, connectedclients, notify, and wifimanager as seen in Figure 2.

Figure 2 – Installing Infusions
Make sure you install all Infusions to SD storage as there is much more available in the SD card, you’ll quickly clog internal storage if you’re not careful.
While WiFI Pineapple is first and foremost a Wifi attack platform, I believe it can be used as a defensive platform as well, in particular a monitoring sensor particularly in an area where many WiFi connected devices are in play and you’d like to monitor the local IoT.
In the Logs Infusion I followed the /tmp/pineap.log which logs probes for SSIDs by MAC addresses.
The PineAP Infusion, with MK5 Karma enabled, will allow you to filter under the Log tab as well. From the Pineapple information content under the PineAP Infusion states that “MK5 Karma is a module of the PineAP suite intended to host spoofed Access Points, or honeypots. This is achieved by replying to probe requests with appropriately crafted probe responses.” You can tweak MK5 Karma and Pine AP as a honeypot to ensure only trusted, known devices connect in your environment. You can then blacklist and whitelist both clients and SSIDs, then send notifications via email or Pushover based on specific rules if you so choose. All the related Infusions are noted in Figure 3.

Figure 3 – Monitor and notify with Pineapple Infusions
As a result, WiFi Pineapple, while a fantastic red team tool, can also be used for defensive monitoring in a highly connected environment where only trusted devices are a requirement.

Raspberry Pi 2

Loading Kali on a Raspberry Pi 2 is also quite simple and is spelled out nicely on Grab a Class 10 SD card and DD the latest image to the card from a *nix host. I ran dd if= kali-1.1.0-rpi2.img of=/dev/sdb bs=512k, used gparted to allocate (resize) all the available storage on my 32GB SD, popped the SD card in my RPI2, and powered it up. You’ll login as root, initial password is toor as expected (change it), then execute startx. Follow the steps in the guidance to change your SSH keys as all ARM images are pre-configured with the same keys. Initially, this installation is missing almost all of the Kali packages, easily fixed as follows:

1)  apt-get update
2)  apt-get upgrade
3)  apt-get install kali-linux-full

A bit of patience as kali-linux-full exceeds 3GB, and voila, you’re running Kali on a kick@$$ wallet-sized computer!
Here’s a scenario I imagine a RPI2 being useful in for a penetration test/red team exercises, given that it is both inexpensive and concealable. You’re assessing an organization that has a significant public area (lobby, customer services offices, conference rooms, and auditorium). The organization offers guest WiFi and does not lock down numerous Cat5 wall jacks. Your recon determines that:
1)      There is a keys-to-the-castle health services database on the internal organization network that is your ultimate goal and primary agenda for the assessment
2)      There is a location in the public space near a cabinet and a large plant where a WiFi enabled RPI2 (Figure 4) can be plugged into both power and one of the unregulated wall jacks. Even if discovered in a day or two, you only need a few hours.

Figure 4 – Raspberry Pi 2 (in camera support case)
After “installing” your device, you can access it over the public WiFI as wlan0 is serving up SSH in the same IP range as your laptop. You’re simply sitting in the organizations public café, seemingly browsing the Intarwebs during lunch. As an added bonus, you find that the wired connection to your RPI2 enjoys unfettered access to the internal (Intranet) environment. You nmap it accordingly and discover a few hosts offering up HTTP and HTTPS. You can kick in a little X11 forwarding on your RPI2 or tunnel through your RPI2 and browse the sites directly from your laptop in the café. Sure enough, after a bit of review, you discover that one of these web servers hosts the front end for that health services database you seek. You quickly recognize that the Security Development Lifecycle long ago left the building (may never have entered) and that this front end is rampant with SQL injections vulns. You ready SQLmap and strike while the iron is hot. You run the following from your RPI2 and in four quick steps have dumped the patient db. Great, now you have to write the report.

1) --url="" --data="bill_month" --banner
2) --url="" --data="bill_month" --dbs
3) --url="" --data="bill_month" -D db337433205 --tables
4) --url="" --data="bill_month" --dump -D db337433205 -T dbo337433205.PATIENTS

The above gives you the database banner, the populated databases, the tables in the db337433205 database, and then,yep, there’s the proverbial gold in that dump (Figure 5).

Figure 5 – SQLmap strikes gold from Kali on Raspberry Pi 2
 By the way, if want to take screenshots in Kali on and RPI2, you’ll need to run apt-get install xfce4-screenshooter-plugin to enable the app, you’ll find it under Accessories thereafter.
This is but one example of an endless slew of opportunities running Kali and other distros from this credit card-sized device. Grab some spare SD cards and build out a few of your favorites, then swap them in as you want to boot them up. Some RPI2 kits come with NOOBS on an 8GB SD card as well, which will help get you started and your feet wet. Hackers/makers rejoice! I’m going to add sensors and a camera to my kit so I can implement specific scripted actions when movement initiated. 
In Conclusion

Working with the Raspberry Pi 2 or earlier versions allows you so many options. You’ll recall that FruityWifi, as discussed in November 2014, is specifically tailored to Raspberry Pi, and there are Pwn Pi, Raspberry Pwn (from Pwnie Express), and MyLittlePwny, amongst others. Grab a kit today and get started, it’ll be great for your Linux skills development, and can be used for attack or defense; the options are literally endless. I’d also be remiss if I didn’t mention that Microsoft is releasing Windows 10 for IoT (Windows 10 IoT Core), currently in Insider Preview mode, so you can play on the Windows front as well.
Ping me via email or Twitter if you have questions (russ at holisticinfosec dot org or @holisticinfosec).
Cheers…until next month.

by Russ McRee ( at June 01, 2015 02:53 PM

May 30, 2015


Varnish – Handling bad responses from backend

Sometimes servers and applications fails, sooner rather than later, that’s because part of our work is to design and continuously planning a robust and highest reliable architecture for our application to ensure his availability most of the time. Depending on the application, software used, architecture, environment we’ve different challenges and strategies, so we’re depending on multiple factors, but in that post I’ll focus in one particular piece for architectures using web servers with Varnish that can help us to improve a bit more the available with a couple of tweaks. As you know Varnish is an HTTP accelerator which is used to cache dynamic content from web servers, acting as a proxy between the client and the original web server. That’s not the objective of this post to focus on the functionality and configuration of Varnish, so you can find very well documentation on his website:

One of the features of Varnish is the support of Saint and Grace mode, both features will allow us to handle troubles with our web servers and keep the service online even if our backend servers goes down. Well this is in part true, of course we cannot guarantee the entire service will continue working just with Varnish, but unless we can keep working part of our application.
So imagine a website or API service with thousands of requests per second, some of them may be POST, DELETE or PUT requests to submit changes for the application, which that kind of requests cannot be handled in the situations of the backend servers goes down, but in the case of GET requests where the clients wants to obtain the information from the service and Varnish have that particular content on his memory cache, this can be handled perfectly and returning the content to the client even if the backends are not working. Of course this has two things to bear in mind, this requests has to be cached before from Varnish and we’ll have outdated responses to the clients, but that’s better than reply with an error page! As always this behavior it’s useful depending on the requirements and the type of application, but most of the time can can save us requests and keep part of the service working in case of failure, so in my opinion highly recommendable if you can use that.

So let’s get started with the configuration of Varnish:

Edit the VCL definition file, usually located in /etc/varnish/default.vcl and edit the following directives:

sub vcl_recv {

if (req.backend.healthy) {
set req.grace = 30s;
} else {
set req.grace = 6h;


sub vcl_fetch {

if (beresp.status == 500 || beresp.status == 502 || beresp.status == 503) {
set beresp.saintmode = 10s;

if (req.request != "POST") {
set beresp.grace = 6h;


Let’s see with a bit more in depth what this configuration does. The “vcl_recv” is called when at a request comes from the client, and the purpose of this method is decide what to do with that request. In that configuration we’re saying if the backend servers are alive we’ll keep the content for 30 seconds beyond their TTL “set req.grace = 30s;”, in case the backends becomes unavailable we’ll keep the content for 6h to serve to the clients “req.grace = 6h;”.

The “vcl_fetch” is called when a document has been successfully retrieved from the backend. In case the backend server returns a HTTP error code of 500, 502 or 503, Varnish will not ask that backend again for this object for 10 seconds “set beresp.saintmode = 10s;”.  “return(restart);” restart the HTTP request, this will automatically happen on the next available server, except for POST requests to avoid duplicate form submits, etc… The max_restarts parameter defines the maximum number of restarts that can be issued in VCL before an error is triggered, thus avoiding infinite looping.

set beresp.grace = 6h;” Keep all objects for 6h longer in the cache than their TTL specifies, so even if HTTP objects are expired (they’ve passed their TTL), we can still use them in case all backends goes down.



by ivanmp91 at May 30, 2015 11:06 PM

May 29, 2015

Michael Biven

The Almost Offical Package Manager for Mac OS X

I recently watched the video of Mike McQuaid’s “Homebrew – The Good, the Bad, and the Ugly of OS X Packaging” talk from FOSDEM ’15 and it reminded me of the early days of package management for OS X.

Back then it was an exciting time. You had this new operating system from Apple with its roots coming from NeXTSTEP and a surge of open source software being ported over to the new OS. And at the same time you had Gentoo coming into existence which had influence on how people thought package managers should function.

Once Mac OS X came out as a public beta and after the release of 10.0 (Cheetah) there wasn’t a package manager to make installing OSS any easier on OS X. This led to Fink and DarwinPorts (later renamed to MacPorts). The work of these projects gave the Mac community a set of tools that were missing in the OS.

Remember Apple was still using PowerPC processors; the announcement of the Intel transition wouldn’t come till five years later.

Late 2006 I started working on an article covering the package managers available for OS X, the Metapkg Alliance and a rumor I heard that Apple considered adopting one of the package manager projects as the official system for OS X. For a long time I had all of the emails and notes sitting around and then I wrote up a short blog post around the middle of 2009 to share them. After starting writing again I gave myself a clean slate and didn’t import any of the old material, but after watching McQuaid’s talk I thought it’s still worth sharing.

Shortly after the release of the Mac OS X public beta in 2000 Christoph Pfisterer started the Fink project giving OS X it’s first package management tool.

Two years later a number of Apple employees including Landon Fuller, Kevin Van Vechten, and Jordan Hubbard (co-founder of FreeBSD) created DarwinPorts (now MacPorts).

Gentoo was working on support for Mac OS X in their package manager called Portage. And then in 2003 DarwinPorts, Fink and Gentoo formed the MetaPkg Alliance as a way to share information on porting software to OS X. These three projects represented the majority of people working towards making packages for the Mac.

Around this time a rumor was floating around that Apple was set to announce that Fink would become the official package management system at the next WWDC. What prevented this was their reluctance of using anything licensed under the GPL.

Below are a few quotes from the various email exchanges I had with people from Fink, MacPorts, and Gentoo. They give what I think is a look into an interesting bit of history from the early Mac OS X community.

I suggest you talk to Landon Fuller. Metapkg was his personal project from the beginning and I don’t think anyone from OpenDarwin (or Apple) ever got involved at all. Landon would know the details - I was one of those who never got involved with it and probably have as many questions about it as you do!

– Jordan K. Hubbard, former director of engineering for Unix Technologies at Apple 10/24/2006

When Mac OS X was originally released, porting software to the platform was considerably more difficult than it was today. It was highly unlikely that any mildly complex piece of software would work without (extensive) patching, and MetaPkg was intended to provide a venue for sharing the load of that porting work between Fink, DarwinPorts, and Gentoo.

However, MetaPkg arrived just in time to solve a problem that was disappearing:

– Apple improved Mac OS X’s support for common APIs (e.g., dlsym, poll, nls, etc.)

– As the UNIX Mac OS X developer community grew, projects were ported to the platform by the upstream developers, and fewer changes were required.

Simply put, the success of Mac OS X, and porting software to Mac OS X, quickly obsolesced the cooperative project. Good news, really.

– Landon Fuller, former Apple BSD Technology Group engineer and MacPorts developer 10/26/2008

It has also helped that libtool’s support for OS X has greatly matured (thanks in large part to Peter O’Gorman, formerly a Fink core team member).

– David R. Morrison, Fink 10/25/2006

Wow… a looooong time ago. As I remember it, Apple decided to name one of the package formats/repositories the “official” format at WWDC, very shortly after metapkg was announced, which seemed to go against the entire spirit of the metapkg alliance.

Daniel Robbins, Gentoo Founder 10/22/2006

Was there any influence from Apple after the Alliance was announced to consolidate to one format or any other changes to have an “official” repository?

Not especially. Apple had been struggling with the packaging question long before Kevin Van Vechten, JKH, and I wrote DarwinPorts, and until Apple selected a single packaging format there was little room for them to push for an official repository. DarwinPorts (now MacPorts) has never solved the packaging format question, though we did implement support for a variety of formats everything from rpm, to dpkg, to Apple’s .pkg.

– Landon Fuller 10/28/2006

We got an email from some guy at Apple telling us that one of the package formats was going to be declared officially supported by Apple at WWDC. My impression was that Apple didn’t want more than one package format/package repository to succeed on OS X and weeks after metapkg was launched tried to do some preemptive selection of format via announcement at WWDC that chose one format to be “official” and “blessed.”

– Daniel Robbins 10/22/2006

There were indications that Apple was seriously considering adopting one of the open source package management tools as the official one. (I don’t remember the timing too well, but it may have been right around the time that Metapkg started.)

– David R. Morrison 10/25/2006

Well, Apple never ended up doing anything about package management… The closest they came was creating dports, and it actually made it into one of the seeds of panther, but was quietly removed again, and as we all know by this point, opendarwin and dports got pretty much ignored except by the few people on the BSD team that spent their own time on it.

I’ve heard some rumors of RPM support in leopard server but hadn’t looked into it. I helped out on the early dports RPM-binary-generation port, RPM works pretty well on OS X and has the advantage of having an architecture designed for platform “variants” that should work reasonably well with universal binaries, so it seems a plausible rumor, if not likely.

– Benjamin Reed, Project Admin Fink 10/25/2006

The most promising code to come out of Apple in regards to a packaging format was Xar, originally written by Rob Braun and contributed to heavily by Kevin Van Vechten, who was also a member of the BSD team at Apple, as well as the fellow responsible for the creation of the OpenDarwin project.

– Landon Fuller 10/28/2006 in Mac OS 10.5 started to use xarchives from xar for installing software which replaced the previous method of using gzipped pax files.

Metapkg post-date’s any official Apple discussions along the lines of package management, by the time we formed it, it was for our own edification—internal politics had made them punt “real” package management in OS X until some future release, and the team “won” for the time being. We mostly did metapkg to save ourselves some time, but it turned out to not really work out that way, so it fell by the wayside, and we all continued on as we had been.

– Benjamin Reed

In the end Apple’s focus on the consumer, the on-again, off-again criticism from the Open Source community for Apple’s lack of participation, and Apple’s eventual abandonment (again) of the server market created the environment where a project like Homebrew was needed.

The last quote from Benjamin Reed saying that the “ team won” clearly shows that the OSS side of package management wasn’t a priority. And I’m pretty sure that was a good thing. That lack of attention gave the development community the room to create the tools we wanted and needed. And I think unintentionally lead to the wide spread adoption of Mac’s and Mac OS X as the computers that most of the modern web was built on.

May 29, 2015 08:05 AM

May 28, 2015

Anton Chuvakin - Security Warrior