Planet SysAdmin

October 27, 2016

Chris Siebenmann

Why I'm unhappy with how Debian builds from source packages right now

A few years ago, I wrote about a problem with how Debian builds from source packages, namely that it builds your binary packages in the source package working area and this can lead to contamination of your source package working area. Fast-forward to today:

@thatcks: Current status: really hating the Debian approach to building packages because it commingling source & build areas has blown up in my face.

@thatcks: I think I'm going to have to extract my change, nuke the entire package area, and start over completely from scratch, testing included.

I'd been making a change to a quilt-based Ubuntu source package. It was a simple change, I had it in a quilt patch since we've done this drill before, and I'd built and tested the updated binary packages. As one of our pre-deployment steps, an obvious thing to do was build a source package too as a nice encapsulated artifact of our changes.


$ debuild -S -us -uc
[... natter natter ...]
dpkg-source: info: local changes detected, the modified files are:
dpkg-source: error: aborting due to unexpected upstream changes, see [...]

There is only one problem: I hadn't changed configure. At all.

The upstream tarball of this package ships a as well as the straight configure. My best guess is that building the package from source the way Ubuntu does it re-runs autoconf, and the autoconf-generated configure is different from the upstream configure in minor ways (the actual diffs are small but plausible for this). This means that after a single build, your source package working area is contaminated unless you carefully save or re-extract the original configure in order to put it back at the start of any rebuild.

(Indeed I just did a test 'debuild -b -us -uc' on yet another scratch copy of the source, and it seems to have run autoconf.)

I assume that the people who maintain this Ubuntu package don't notice this because the normal Debian source package is not how they actually work with it. Instead I suspec that they have it in git, build it with automated builders that always start clean from a source package, and so on. In other words, they have a different industrial production system and the source package is a lie.

It's difficult for me to find words to express how this makes me feel. Angry is certainly part of it, not simply for what this means for us making changes to the package but also for what it means about how much the package maintainers and Ubuntu in general care about the 'standard' way of building packages. At the best, practices have shifted once again without anyone making it official and clear; at the worst, outside people like me are simply second class citizens when it comes to package work.

(This is of course par for the course for Ubuntu, where Canonical has been extending various digits in people's directions for some time, but I don't have to like it.)

I have no idea what this means for how we want to maintain our local changes here. I have at best vague theories. Probably I'm going to have to try to find unofficial information online about how other people do it.

(This is definitely going to slow down our work with the package, too, and I'll leave things at that rather than going off at length here.)

PS: One thing that this means is that the Ubuntu package build process lacks consistency checks to detect this sort of thing. Given an aphorism, I expect that there are plenty of Ubuntu source packages that have this problem.

Sidebar: Some actual build-time output

 debian/rules build
# <pkg>
[runs configure]

And debian/rules has a 'config-stamp: configure' section that does the dh_testdir stuff et al. The earlier output is from the 'clean:' section of debian/rules for this package. So I believe I have the root cause identified.

Whoever thought that re-running autoconf was a good idea, well, I have no words.

by cks at October 27, 2016 03:40 AM

October 26, 2016

Evaggelos Balaskas

Thunderbird Enigmail

A Beginner’s Guide on How to use Thunderbird with Enigmail (Gpg4win) with their gmail account in 10 minutes on a windows machine

Thunderbird Enigmail - Gmail, Windows from vimeouser on Vimeo.

October 26, 2016 10:26 PM

Chris Siebenmann

What I'm doing to use a Yubikey in Fedora 24's Cinnamon desktop environment

On my desktop machines, I run a fully custom window manager environment where I had almost no problems integrating my Yubikey into my regular environment; most of what it took was some mild hacks to my special screensaver stuff. However, I also have an office laptop that uses a Cinnamon-based environment and I wanted to be able to use the Yubikey there. That turned out to be rather more difficult, because Cinnamon normally handles SSH agent stuff through gnome-keyring-daemon.

The problem with using g-k-d is simple; gnome-keyring-daemon is pretty bad at dealing with SSH keys. Well, perhaps 'limited' is a more charitable label. It still doesn't support ED25519 keys, for example. But in this case the really fatal limitation is that it appears to have no support for using PKCS#11 shared library providers. Since Yubikey-hosted SSH keys can only be accessed through just such a shared library, gnome-keyring-daemon is unable to host them. This means that if you want to have Yubikeys managed through your SSH agent instead of having to unlock them with your PIN every time you use them, you can't let gnome-keyring-daemon be your SSH agent; instead you need to be using the real ssh-agent.

At least on my Fedora 24 install, the session stuff was already starting a ssh-agent process. It was just being preempted by part of the gnome-keyring-daemon setup. Perhaps in a normal Gnome 3 environment there is some graphical way of turning this off, but I couldn't find anything in my Cinnamon settings. Instead you have to do this by hand, like so:

  1. Make a ~/.config/autostart directory if you don't already have one.
  2. Copy /etc/xdg/autostart/gnome-keyring-ssh.desktop to it.
  3. Edit your copy to delete or rename the X-GNOME-Autostart, X-GNOME-AutoRestart, and X-GNOME-Autostart-Notify settings. Possibly you only need to get the first one.

Now log out and log back in again, and you should be using the real ssh-agent instead of gnome-keyring-daemon's emulation of it. You can see which one you're using based on the value of the $SSH_AUTH_SOCK environment variable; if it's of the form /run/user/<uid>/keyring/ssh, you're still using gnome-keyring-agent. The real ssh-agent gives you values that look like /tmp/ssh-<blah>/agent.<pid>.

(The ssh-agent process is started in /etc/X11/xinit/xinitrc-common, which in Fedora comes from the xorg-x11-xinit package. You probably have it installed by default, at least in a Cinnamon-based environment.)

Once you're using the real ssh-agent, you can add the Yubikey hosted SSH keys to it with the usual 'ssh-add -s /usr/lib64/' and so on. I wrote a script for this, which I can easily run through my Cinnamon customizations. Since I only use my laptop infrequently, I haven't explored things like integration into screen locking; it's enough for me that I can use my Yubikey through the laptop if I really need to, it doesn't have to be as convenient or polished as I want it to be on my desktop.

Information for this entry comes from Using ssh-agent and ED25519 keys on GNOME, the Arch wiki entry on the GNOME keyring daemon, and this entry on using gpg-agent with GNOME. Sadly, the gconftool-2 stuff from this askubuntu question and its answers doesn't work for me.

(It turns out that lots of people are unhappy with having gnome-keyring-daemon act as their SSH agent. Possibly I should have hunted this down years ago just so I could use ED25519 keys on my laptop.)

by cks at October 26, 2016 05:37 AM

October 25, 2016

LZone - Sysadmin

More Changes after the Dyn DDoS Attack

Looking at NS records again of all scanned top 500 Alexa domains after the recent Dyn DDoS attack now 13 of the previously 14 Dyn customers which previously relied solely on Dyn now switched away entirely or added additional non-Dyn DNS servers.

Who Switched Entirely,,,,,,

Who Switched to "Multi"-DNS,,,,


Here is a summary of the NS record changes. To automatically compare the server names all numbers in the DNS server names having been stripped in the following table:

SiteBefore (15.10.)After (24.10.)
The note-worthy non-changer is Twitter which is still exclusively at Dyn, eryone else seems to have mitigated. Some to using two providers, most of them switching to AWS DNS some to UltraDNS exclusively.

October 25, 2016 10:08 AM

Errata Security

Lamers: the problem with bounties

In my last two posts, I pointed out that the anti-spam technique known as "DKIM" cryptographically verifies emails. This can be used to verify that some of the newsworthy emails are, indeed, correct and haven't been doctored. I offer a 1 btc (one bitcoin, around ~$600 at current exchange rates) bounty if anybody can challenge this assertion.

Unfortunately, bounties attract lamers who think they deserve the bounty. 

This guy insists he wins the bounty because he can add spaces to the email, and add fields like "Cc:" that DKIM doesn't check. Since DKIM ignores extra spaces and only checks important fields, these changes pass. The guy claims it's "doctored" because technically, he has changed things, even though he hasn't actually changed any of the important things (From, Date, Subject, and body content).

No. This doesn't qualify for the bounty. It doesn't call into question whether the Wikileaks emails say what they appear to say. It's so obvious that people have already contacted me and passed on it, knowing it wouldn't win the bounty. If I'd pay out this bounty for this lameness, one of the 10 people who came up with the idea before this lamer would get this bounty, not him. It'd probably go to this guy -- who knows it's lame, who isn't seeking the bounty, but would get it anyway before the lamer above:

Let me get ahead of the lamers and point to more sophisticated stuff that also doesn't count. The following DKIM verified email appears to say that Hillary admitting she eats kittens. This would be newsworthy if true, and a winner of this bounty if indeed it could trick people.
This is in fact also very lame. I mean, it's damn convincing, but only to lamers. You can see my trick by looking at the email on pastebin ( and comparing it to the original (

The trick is that I've added extra From/Subject fields before the DKIM header, so DKIM doesn't see them. DKIM only sees the fields after. It tricks other validation tools, such as this online validator. However, email readers (Thunderbird, Outlook, Apple Mail) see the first headers, and display something that DKIM hasn't checked.

I've taken a screenshot of the raw email to show both From fields:

Since I don't rely upon the "magic" of tools to verify DKIM, but look at the whole package, I'll see the added From/Subject fields. Far from fooling anybody, such modifications will be smoking gun that somebody has attempted illicit modifications. Not just me, mostly anybody viewing the "raw source" that Wikileaks provides would instantly see shenanigans.

The Wikileaks emails can be verified with crypto, using DKIM. Anybody who can doctor an email in such a way that calls this into question, such that they could pass something incriminating through ("I eat kittens"), they win the full bounty. Any good attempts, with something interesting and innovative, wins partial bounty.

Lamers, though, are unwelcome.

BTW, the same is true for bug bounties. Bounties are becoming standard in the infosec industry, whereby companies give people money if they can show ways hackers might hack products. Paying bounties allows companies to fix products before they actually get hacked. Even the U.S. military offers bounties if you can show ways to hack their computers. Lamers are a pox on bug bounty systems -- administrators must spend more time telling lamers to go away than they spend on dealing with real issues. No bounties rules are so tight that lamers can't find a way to subvert the rules without finding anything that matches the intent.

by Robert Graham ( at October 25, 2016 04:54 AM

Politifact: Yes we can fact check Kaine's email

This Politifact post muddles over whether the Wikileaks leaked emails have been doctored, specifically the one about Tim Kaine being picked a year ago. The post is wrong -- we can verify this email and most of the rest.

In order to bloc spam, emails nowadays contain a form of digital signatures that verify their authenticity. This is automatic, it happens on most modern email systems, without users being aware of it.

This means we can indeed validate most of the Wikileaks leaked DNC/Clinton/Podesta emails. There are many ways to do this, but the easiest is to install the popular Thunderbird email app along with the DKIM Verifier addon. Then go to the Wikileaks site and download the raw source of the email

As you see in the screenshot below, the DKIM signature verifies as true.

If somebody doctored the email, such as changing the date, then the signature would not verify. I try this in the email below, changing the date from 2015 to 2016. This causes the signature to fail.

There are some ways to forge DKIM-signed emails, specifically if the sender uses short keys. When short keys are used, hackers can "crack" them, and sign fraudulent emails. This doesn't apply to GMail, which uses strong 2048 bit keys, as demonstrated in the following screenshot. (No, the average person isn't supposed to understand this screen shot, but experts can).

What this means is that the only way this email could've been doctored is if there has been an enormous, nation-state level hack of Google to steal their signing key. It's possible, of course, but extraordinarily improbable. It's conspiracy-theory level thinking. Google GMail has logs of which emails went through its systems -- if there was a nation-state attack able to forge them, Google would know, and they'd be telling us. (For one thing, they'd be forcing password resets on all our accounts).

Since DKIM verifies this email and most of the others, we conclude that Kaine is "pants on fire" lying about this specific email, and "mostly untrue" in his claim that the Wikileaks emails have been doctored.

On the other hand, Wikileaks only shows us some of the emails. We don't see context. We don't see other staffers certain it's going to be somebody else for VP. We don't see related email discusses that cast this one in a different light. So of course whether this (verified) email means they'd firmly chosen Kaine is "mostly unproven". The purpose of this document isn't diagnosing what the emails mean, only the claims by Hillary's people that these emails have been "doctored".

As a side note, I offer a 1-BTC (one bit coin, ~$600 at today's exchange rate) bounty to anybody who can prove me wrong. If you can doctor the above email, then you win the bounty. Some rules apply (i.e. it needs to be a real doctored email, not a trick). I offer this bounty because already people are trying to cast doubt on whether DKIM works, without offering any evidence. Put up or shut up. Lamers aren't welcome.

by Robert Graham ( at October 25, 2016 04:02 AM

Chris Siebenmann

On classifying phish spam as malware, an update

Back a number of years ago I noted that our commercial anti-spam filter was counting some varieties of phish spam as 'viruses', and I wrote some thoughts on why this might make sense. I now think that I was partly wrong about some of why the filter was acting this way. What's happened since then is that we now log some information about the structure of incoming messages as part of logging MIME attachment type information, which has given me the opportunity to see more information about the structure of many of these messages.

So here is a typical entry from our logs of the rejection, giving the information the anti-spam filter gave us:

rejected 1byktC-00047A-8j from to <redacted>: identified virus: Mal/Phish-A

And here is the MIME attachment type information for the same message:

1byktC-00047A-8j attachment application/octet-stream; MIME file ext: .html

That's right: as far as I can tell, all of the phish spam being rejected this way has had a .html attachment. This sample was in a MIME multipart/mixed structure; the other part of parts of the structure were something we consider uninteresting and didn't log.

To me, this puts a somewhat different spin on our commercial anti-spam filter detecting phish spam. The entire purpose of its virus detecting side of things is to look at attachments and detect bad stuff (and then strip it out). Should it pass up detecting phish stuff in attachments, just because that's a different sort of bad stuff than it normally looks for?

(Since you can embed other sorts of malware in .html attachments (and people do), the virus detecting side already has to look at such attachments.)

There's still a conscious choice here to include phish as part of the 'malware' that the anti-virus detection looks for, but I think it's a more natural thing to do this to attachments that the software is already scanning for other things. It's less of a special case for both detection and, presumably, for stripping out these attachments as it does for other virus-contaminated attachments.

PS: Sophos's detailed information page on this label does specifically mention that these web pages are often sent as (spam) attachments.

by cks at October 25, 2016 01:44 AM

October 24, 2016


Face to Face: Roadmap and Platform Updates

This is another in the series of posts about decisions we made at our face-to-face meeting a couple of weeks ago.

We updated the project roadmap.

I think the most important news here, is that our next release will include TLS 1.3. Our current plan is that this will be 1.1.1, which means that it is API-compatible with the current 1.1.0 release. This is really only possible because of the work we did on making most of the structure internals opaque. Also, since we are doing all of our work in public on our GitHub repository, we hope that the entire community will be able to “follow along at home” and help us improve the code. There will be more, much more, to say about this later.

We have also set ourselves major goals of increasing our code coverage and protocol-level testing. In addition, all new API’s must be documented, and the entire set of SSL API’s will also be documented. The script util/ lists them. If you’re inclined to help write some documentation, please feel free!

There are a variety of other crypto-nerd things we are looking at, so please see the roadmap for details.

The second point I want to mention in this post is that we have sketched out a platform policy. This is, admittedly, an overdue item from our first meeting two years ago. The first step will be to review all targets in the configuration scripts, and classify them according to the categories mentioned in the platform policy by our next release.

Which, in case I failed to mention it, will include TLS 1.3 :)

October 24, 2016 01:00 AM

October 23, 2016

Pi-Hole: A DNS-based blacklist for ads and tracking for Raspberry Pi

The post Pi-Hole: A DNS-based blacklist for ads and tracking for Raspberry Pi appeared first on

I just "upgraded" our home network with a Pi-Hole, an interesting project that implements a DNS server with a known-list of ad- and privacy trackers. The result is that everyone on your network that uses that DNS server gets an adblocker for free, without configuration work.

How DNS-based blacklisting works

The idea is simple and powerful: the project acts as your DNS server, so every time your smartphone/tablet/laptop/pc requests the IP for known adtrackers, the Pi-Hole just responds with a bogus IP address.

Instead of connecting to the server for the adtrackers, you connect to your Pi. Since that's running a webserver, it instantly connects and returns an empty reply.

There's no delay in waiting to connect to a non-existent IP address, there's just an instant "blackholed" answer.

Pi-Hole configuration

While I usually don't like curl | bash installers, this one seems pretty safe.

I installed it on a Raspberry Pi, but it actually runs on any Ubuntu/Debian based Linux server. The Pi is just convenient as it's a low-power, easy-to-hide-away little server for your home.

On your Pi, run this;

$ curl -L | bash

And you're good to go.

Your entire home gets an adblocker, without the hassle

If you want to protect the members of your home (or: anyone connected to your WiFi), it's usually a bit of a hassle: every device needs an adblocker/privacy blocker, it's a pain to get it to work on iOS, ...

Since Pi-Hole acts as a DNS server, all you need to do is to configure your router to stop handing out the DNS server(s) from your ISP, but to configure it to point to the IP of your Raspberry Pi running the Pi-Hole.

Added bonus: since that Pi-Hole can be configured to use Google's upstream namservers ( & you automatically bypass DNS based blocks implemented by your provider.

So if your ISP is preventing you from visiting certain websites, that's usually DNS based and the Pi-Hole bypasses that by not using your ISPs' nameservers.

Web interface

This is actually what made me try out Pi-Hole in the first place: their reporting dashboard!

Every DNS query made gets logged: you get graphs, a list of DNS queries, you can add white- or blacklist entries, ...





Top DNS & advertisers


Full DNS query log


Control over convenience

There are already DNS-based blacklist providers you can use (like OpenDNS), but having the DNS server run on your own Raspberry Pi gives you more control and privacy. All the blocking & tracking happens in the comfort of your own home, not by a corporation you don't know.

The downside; you need to configure a Pi and you'll have to troubleshoot why DNS isn't working one day, if the service is stopped/crashed or your Pi has stopped working.

But as a sysadmin, I don't mind -- I'm happy my home gets an adblocker for everyone connected to our WiFi for free.

Give it a try: Pi-Hole, a Black Hole for Internet Advertisers for your Raspberry Pi

The post Pi-Hole: A DNS-based blacklist for ads and tracking for Raspberry Pi appeared first on

by Mattias Geniar at October 23, 2016 10:08 AM

Errata Security

Yes, we can validate the Wikileaks emails

Recently, WikiLeaks has released emails from Democrats. Many have repeatedly claimed that some of these emails are fake or have been modified, that there's no way to validate each and every one of them as being true. Actually, there is, using a mechanism called DKIM.

DKIM is a system designed to stop spam. It works by verifying the sender of the email. Moreover, as a side effect, it verifies that the email has not been altered.

Hillary's team uses "", which as DKIM enabled. Thus, we can verify whether some of these emails are true.

Recently, in response to a leaked email suggesting Donna Brazile gave Hillary's team early access to debate questions, she defended herself by suggesting the email had been "doctored" or "falsified". That's not true. We can use DKIM to verify it.

You can see the email in question at the WikiLeaks site: The title suggests they have early access to debate questions, and includes one specifically on the death penalty, with the text:
since 1973, 156 people have been on death row and later set free. Since 1976, 1,414 people have been executed in the U.S
Indeed, during the debate the next day, they asked the question:
Secretary Clinton, since 1976, we have executed 1,414 people in this country.  Since 1973, 156 who were convicted have been exonerated from the death row.
It's not a smoking gun, but at the same time, it both claims they got questions in advance while having a question in advance. Trump gets hung on similar chains of evidence, so it's not something we can easily ignore.

Anyway, this post isn't about the controversy, but the fact that we can validate the email. When an email server sends a message, it'll include an invisible "header". They aren't especially hidden, most email programs allow you to view them, it's just that they are boring, so hidden by default. The DKIM header in this email looks like:

DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed;; s=google;

How do you verify this is true. There are a zillion ways with various "DKIM verifiers". I use the popular Thunderbird email reader (from the Mozilla Firefox team). They have an addon designed specifically to verify DKIM. Normally, email readers don't care, because it's the email server's job to verify DKIM, not the client. So we need a client addon to enable verification.

Downloading the raw email from WikiLeaks and opening in Thunderbird, with the addon, I get the following verification that the email is valid. Specifically, it validates that the sent precisely this content, with this subject, on that date.

Let's see what happens when somebody tries to doctor the email. In the following, I added "MAKE AMERICA GREAT AGAIN" to the top of the email.

As you can see, we've proven that DKIM will indeed detect if anybody has "doctored" or "falsified" this email.

I was just listening to ABC News about this story. It repeated Democrat talking points that the WikiLeaks emails weren't validated. That's a lie. This email in particular has been validated. I just did it, and shown you how you can validate it, too.

Btw, if you can forge an email that validates correctly as I've shown, I'll give you 1-bitcoin. It's the easiest way of solving arguments whether this really validates the email -- if somebody tells you this blogpost is invalid, then tell them they can earn about $600 (current value of BTC) proving it. Otherwise, no.

Update: I'm a bit late writing this blog post. Apparently, others have validated these, too.

Update: In the future, when changes their DKIM key, it will no longer be able to verify. Thus, I'm recording the domain key here: type TXT, class IN
v=DKIM1; k=rsa; p=MIGfMA0GCSqGSIb3DQEBAQUAA4GNADCBiQKBgQCJdAYdE2z61YpUMFqFTFJqlFomm7C4Kk97nzJmR4YZuJ8SUy9CF35UVPQzh3EMLhP+yOqEl29Ax2hA/h7vayr/f/a19x2jrFCwxVry+nACH1FVmIwV3b5FCNEkNeAIqjbY8K9PeTmpqNhWDbvXeKgFbIDwhWq0HP2PbySkOe4tTQIDAQAB

by Robert Graham ( at October 23, 2016 03:52 AM

October 22, 2016

LZone - Sysadmin

Changes after the Dyn DNS Outage

Looking at NS records today some of yesterdays affected companies decided to change things after the DoS on Dyn. As the NS record is not really a customer facing feature this is more an indication of Dyn's customers expectations. One could argue switching away from Dyn could mean fear of more downtimes to come.

Here is a summary of changed NS records so far:
SiteBefore (15.10.)After (22.10.)
awsdns awsdns anycastns*.org ultradns.* awsdns
I did only check some NS records for changes, just several of the top sites. There are two note-worthy non-changers being Twitter and Github though, everyone else seems to have mitigated. Some to using two providers, several switching to AWS DNS or UltraDNS exclusively.

October 22, 2016 07:45 PM

October 21, 2016

LZone - Sysadmin

Whom you can DDoS via DynDNS

After todays various affected major websites you might ask who is actually also affected together with the well known sites as Amazon, Twitter or Github.

Using the results of a monthly scan I automatically run on the top 500 Alexa sites it is easy to find out. The only thing you need to know is that the Dyn DNS server domain is (detailed results).

Top Affected Sites

Probably Different Impact

Note that not all of these sites were equally affected as some of them like Amazon are using multiple DNS providers. The Amazon main domains NS records do point to Dyn and UltraDNS. The same way probably none of the major adult sites was down as the also relied on at least two providers.

So while Amazon users probably got to the website after one DNS timeout and switching over to UltraDNS, Twitter and Github users were not so lucky and had to hope for Dyn to respond. It will be interesting to see if Twitter and Github will add a second DNS provider in the next time as a result of this.

The Need For "Multi-DNS"

Reading different reports on this incident it seems to me the headlines are focussing on those sites using just Dyns DNS and not on those having a "Multi-DNS".

Detailed results on who is using which DNS domain can be found in the monthly DNS usage index.

October 21, 2016 09:16 PM

Everything Sysadmin

Watch my Punk Rock DevOps talk live on Friday at 2:30pm PT

My talk "DevOps Where You Wouldn't Have Expected" will be live-streamed from PuppetConf on Friday, Oct 21 at 2:30pm PT / 5:30 ET.

The talk happens to summarize the major points of Chapter 1-4 of the new edition of The Practice of System and Network Administration, which is due out on November 4, 2016.

You need to pre-register, which takes time. Please preregister early.

A full description of the talk is at

To register and watch the live-steam visit

P.S. The unofficial title of the talk is "Punk Rock DevOps".

by Tom Limoncelli at October 21, 2016 12:10 AM

October 20, 2016

That grumpy BSD guy

Is SPF Simply Too Hard For Application Developers?

The Sender Policy Framework (SPF) is unloved by some, because it conflicts with some long-established SMTP email use cases. But is it also just too hard to understand and to use correctly for application developers?

Here is a story involving a web-based contact form that indicates that this may be the case.

A wise man once noted that only two things in life are inevitable: death and taxes.

Just how taxes and interactions with the tax authorities are handled vary widely from jurisdiction to jurisdiction, but in Norway, where I live, the default mode of contact with the tax authorities for most of us is via web forms protected by sometimes cumbersome authentication procedures and the occasional alert via SMS text message to your phone.

And we're used to that the things just work with only occasional and very minor technical embarrassments for the people who operate the thing.

Then in August 2016, I tried to report a bug via the contact form at, the main tax authorities web site.

The report in itself was fairly trivial: The SMS alert I had just received about an invoice for taxes due contained one date, which turned out to be my birth date rather than the invoice due date. Not a major issue, but potentially confusing to the recipient until you actually log in and download the invoice as PDF and read the actual due date and other specifics.

The next time I checked my mail at, I found this bounce.

The meat of the message is
    SMTP error from remote mail server after RCPT TO::
    host []: 550 5.7.23 SPF validation failed   

which means that somebody, somewhere tried to send a message to, but the message could not be delivered because the sending machine did not match the published SPF data for the sender domain.

SPF expands to "Sender Policy Framework", which is one of the early and still valid ways for a domain to publish which hosts are supposed to be sending mail on the domain's behalf.

There is no requirement to refuse delivery of mail from a host that is not in a domain's SPF record, and emphatically so when no such record exists. On the other hand, when a domain does publish SPF data, rejecting mail from hosts that are not in the set published is a valid and accepted action.

What happened is actually quite clear even from the part quoted above: the host [] tried to deliver mail on my behalf (I received the bounce, remember), and since I have no agreement for mail delivery with the owners and operators of that host, it is not in's SPF record either, and the delivery fails.

The bounce message contained one Received: header which tells part of the story, and after decoding the MIME-encoded part it becomes clear that it contains my bug report with some slightly odd markup.

So it's clear that what produced the bounce was that the contact form had tried to deliver mail using the address I had given the contact form.

I can hear your groans all the way from there to here.

My original bug report had not been delivered, but I thought perhaps I could help them with the other problem, so I sent off a new message to the addresses that were given as contacts in the domain's whois info, taking care to send it from a machine that was properly authorized to send mail for my domain.

The full text of the message is available here, in Norwegian. The message includes the bounce with a short introduction that said essentially "this is the result of trying to submit a bug report via your web contact form. This is clearly an SPF problem, please look into that".

Then that message bounced, too.

The exact reason is likely buried in the configuration files for's MX, but it could be that it has some reservations against accepting mail from what it sees as a subdomain that doesn't have an MX record of its own.

Anyway, by then I had spent far too much time on this rather trivial matter while I was supposed to be doing other things, so I quickly wrote up a summary and sent it off to the same contact addresses, this time from my account, with the various accumulated data as attachments.

Then, as often happens when dealing with the authorities, nothing happened. For quite a while.

It was only on October 18, 2016 that my account received a reply from the support address, which quoted my last message in full, and added their resolution text saying:

Det kan se ut som du har Sender Policy Framework (SPF) aktivert på din mailserver, Det er en kjent svakhet ved vårt kontaktskjema at mailservere med SPF ikke støttes.


It looks like you have Sender Policy Framework (SPF) enabled on your mailserver, It is a known weakness of our contact form that mailervers with SPF are not supported.

Once again, I can hear and fully sympathize with your groans.

This is as perfectly wrong as can be, in fact the exact opposite of right.

The obvious answer should be, as you will agree if you're still reading: The form's developer should place the user's email address in the Reply-To: field, and send the message as its own, valid local user. That would solve the problem.

So I put it to you: Is SPF, the Sender Policy Framework, simply too hard for application developers to understand?

Yes, I'm well aware that SPF also breaks traditional forwarding of the type generally used by mailing lists and a few other use cases. Just how afraid should we be when those same developers come to do battle with the followup specifications such as DKIM and (shudder) the full DMARC specification?

I anticipate your feedback in comments, or if you have things you really want me to only paraphrase in public, email.

by Peter N. M. Hansteen ( at October 20, 2016 07:10 PM

Varnish Explained

The post Varnish Explained appeared first on

A few months ago, I gave a presentation at LaraconEU in Amsterdam titled "Varnish for PHP developers". The generic title of that presentation is actually Varnish Explained and this is a write-up of that presentation, the video and the slides.

The simplest way to get a grip of the basics of Varnish would be to watch the video recording of that presentation.

Further down you'll find the slides and a transcript of each slide, with accompanying links.

Varnish Explained: presentation video

This is the recording of the talk at LaraconEU on August 2016.

Varnish Explained

Here's a write-up per slide, with the most important topics.

(Alternatively, if you prefer to just skim through the slides, they are up on Speakerdeck too.)


In order to fully understand Varnish, it's vital that you understand the HTTP protocol (at least the basics) and how its HTTP headers are formed and which purpose they serve.

Later on, I'll cover the Varnish internals, cool acronyms like ESI, Grace, TTL & more.


A quick introduction: I'm Mattias Geniar, I'm the Support/Operations manager at Nucleus Hosting in Belgium.

I run the cron.weekly linux & open source newsletter and, when I find the time, host the SysCast podcast.

Part One: What's Varnish?


Let's go.


Varnish can do a lot of things, but it's mostly known as a reverse HTTP proxy. It brands itself as an HTTP accelerator, making HTTP requests faster by caching them.

It can also serve as a TCP load balancer (using pipe in the VCL code, more on that later), an HTTP load balancer or a generic reverse proxy.

It uses the Varnish Configuration Language (VCL) to allow you as a developer or sysadmin to modify the behaviour of Varnish and to play with the HTTP protocol to shape it so it behaves the way you want.

Varnish is also extendable with VMODs, modules that can be loaded by varnish and provide additional functions/methods and functionality you can call upon in your VCL configurations. Think of these like PHP extensions; binary files that are loaded by Varnish that extend the userspace in which you can create VCL configurations.


A couple of important versions: Varnish 3.x is probably the most widely used, but is now EOL (so: no more security updates). Varnish 4.1 is still supported and Varnish 5 came out a few weeks ago (so the slides are slightly outdated).

If you're upgrading from Varnish 3.x to later: beware, the configurations changed drastically. You'll need to update your syntax/methods in Varnish, or Varnish won't start.


Using Varnish in your infrastructure gives you a couple of advantages: performance, scalability, control & security.

Varnish is usually associated with performance, but it greatly increases your options to scale your infrastructure (load balancing, failover backends etc) and adds a security layer right out of the box: you can easily let Varnish protect you from the httpoxy vulnerability or slowloris type attacks.

Part Deux: HTTP Headers


Let's talk HTTP headers.


A quick summary of HTTP headers: there are request and response headers (basically what your User Agent (=browser) requests and what the Server (Apache/Nginx) responds) and Varnish listens to these by default to determine if requests can or should be cached.


Whenever your browser goes to a certain website, it'll a do a few low-level things: it'll resolve the DNS of the hostname you want to browse to, open a TCP connection to that IP and start sending the HTTP request headers.


They are basically a new-line separated key/value pair. The concepts are the same in HTTP/2, except it's binary and more structured.

The browser describes the kind of response it can receive (plain text, compressed), what site it wants to load (because remember, the browser just connected to the IP, it now needs to tell the webserver which website at that IP address it wants to load).


If all goes well, the server will respond with a similar set of headers: the response headers. It will confirm the HTTP protocol (or down/up-grade the protocol), give a status code and describe the response (plain text / compressed).

Good, that was the HTTP reminder -- now let's get to Varnish.

Part Three: How does Varnish work?




Varnish is a reverse proxy, so it sits between the User Agent (the browser) and the webserver. It makes decisions to either deliver a cached version of the page, or send the request to the backend webserver for processing.

By default, it'll do so by listening to the HTTP headers the client sends as well as the ones the server responds with. But, those usually suck -- creating the need for proper Varnish templates to avoid boilerplate code being re-created all the time.


In a normal scenario, the browser probably connects directly to the webserver, which in turn will let PHP execute the request. The simplest possible setup.


When Varnish is introduced, a few things happen: the webserver is hidden to the user, as that user now only connects to Varnish. Varnish has bound itself on port :80 and the webserver is either running on a different server or on an alternative port. It's Varnish that will make the connection to the webserver if needed, the user has no idea.


If you follow the official Varnish book, the "internals of Varnish" are described as this. While it's 100% correct, if you're new to Varnish, that image does you no good: it's scary, confusing and it teaches you nothing.

Instead of such a flowchart, I prefer to visualise Varnish slightly different.


Let's start with a basic example: a user tries to connect to a server running Varnish. It'll make a connection to the port where Varnish is running and Varnish can start to process the request.


It'll trigger a routine in Varnish called vcl_recv().

It's a routine where you can write custom VCL code to manipulate requests, determine backends, redirect users, ... it gives you full control over the HTTP request of the user.


After that routine, Varnish can do a cache lookup: the page the user is requesting, does it already exist in the cache?


If it does, it can serve that page to the user. Before it does that, it triggers a new internal routine called vcl_deliver(). It's another place in Varnish where you can manipulate the request, change the HTTP headers etc.


Once that request is finished, Varnish sends the response to the user so he can render it on screen.


Of course, not everything is a cache hit: if Varnish does a cache lookup but finds it doesn't have that object/request in the cache, it has to go and fetch that content from its own backend.

To do so, it'll trigger another internal routine called vcl_backend_fetcht().

You guessed it, it's yet another place where you can further manipulate the request before it gets sent to the backend webserver(s).


That routine will eventually call out to another webserver and have it process the request the user made.


Once the response was created on the backend webserver, Varnish will receive it and fire another routine: vcl_backend_response().

Hey, yet another place to play with the HTTP protocol and change the headers etc. before Varnish will process the request internally.


Varnish can then store the item in its cache (if it's cacheable) and deliver it to the client, via the same vcl_deliver() routine.

Pfew, what a ride!

Part Four: The Varnish Configuration Language


Now that you have a visual of the flows within Varnish, let's see what the configurations look like.


The Varnish Configuration Language is a C-like language, which should be familiar to those writing PHP or JavaScript code.

It looks like the screenshot above, where routines are described using the sub keyword. For a complete example, have a look at my varnish 4.x configuration templates.


Each of those routines I described earlier allow you to customise the HTTP request with that VCL language.

To do so, it offers you a few useful objects: req, req.http, ... Each of those objects represent a part of the HTTP protocol. There's req.method to indicate a GET, POST, PUT, PURGE, ...

There's req.http which contains all the raw HTTP headers sent by the client, so there's for the Host-header, req.http.User-Agent for the User-Agent (Chrome/Firefox) of the client, ...

Just like the HTTP protocol itself, the HTTP headers are case insensitive. So req.http.HOST is the same as But do yourself a favour, pick a coding/style standard you like and stick to it.

In the example above we tell Varnish, in the vcl_recv() routine (the very first routine to be called by Varnish) to only deal with GET or HEAD requests. Anything with a different method (a POST, PUT, ...) will be sent to the backend webserver(s) via the return (pass); command, indicating that the request should be passed on to the backend.

Further down, we manipulate the cookies the client has set. This is probably the part where Varnish has thought me the most about the HTTP protocol! As a user, every cookie that is set on a particular domain gets sent to the server for every request. It's a single string, semi-colon delimited. You can change the value of cookies in Varnish, remove or add cookies, etc.


You can also use Varnish to be a "dumb load balancer": it can look at the Host header sent by the user and redirect requests to different backends.

This would allow you to set up a single Varnish server which can send requests to multiple backend servers, depending on the site the user is requesting.

The set req.backend = xxx; code instructs varnish to use a particular backend for that HTTP request.


A common task in Varnish is dealing with Cookies (more tips on that later by the way) and removing cookies for objects or HTTP requests that don't need them.

A typical example is content like CSS, JavaScript, images, ... whether cookies are set or not, chances are that the image won't change. So we remove cookies altogether to determine the cacheability of such an object.


Alternatively, we could also modify the response the webserver has sent us, in case it was a cache miss.

If a webserver wants to set a new cookie, it'll send along one or multiple Set-Cookie headers. In Varnish, we can remove those headers by removing the bereq.http.Set-Cookie header, making it look like the webserver never even sent those requests in the first place!


Another interesting trick is to tell Varnish to never cache a response when it's an server error, something with a status code in the HTTP/500 range. By telling it to reply with return (abandon); it'll bail out that routine and exit the flow.


In the vcl_deliver(); routine you have a final place, right before the HTTP response is sent to your user, to add or remove HTTP headers.

I commonly use this to clean up the request: remove sensitive headers (like Apache or PHP version numbers), add some debug headers to indicate if it was a cache miss or hit, ...


Having so many places to tinker with the request and response can make it hard to visualise where changes were happening and what the outcome would/could be. It can also make you question what's the best place to even begin to manipulate such a request.

To be honest: there's no right or wrong answer here. It depends on your setup. But, there are a couple of tricks to help work with Varnish.


For example, let's run through a request in Varnish and see where/how we can play with it.


In this example, a user wants to connect to our server and load the homepage. It sends a request to load the / page (homepage) indicating that it's the Chrome User-Agent.

When Varnish receives that request, it'll start its vcl_recv() routine.


In that vcl_recv routine, we can manipulate the request.

For instance, we can change the request so that it no longer wants to load but, by setting a new value in the object.

Important to know here: the user/client has no idea we're doing this. This is purely internally in Varnish. For the user, it still thinks it's going to receive a response for!

Varnish will use that new information (the modified and req.http.User-Agent) to check its cache if it has a response it can serve to the client. In this example, we'll assume it was a cache miss.


So now Varnish has to fetch the content from its own backend webserver. But because we modified the request in vcl_recv(), Varnish will ask the backend for the website behind, claiming to be an IE5 User-Agent.


The webserver does its thing, and if all goes well it can generate the homeapge for

It'll respond with an HTTP/200 status code, indicating everything was OK. It will also set a PHP session ID, tell that it was generated with the Apache Webserver and indicate that this HTTP response should not be cached, by setting a Cache-Control: No-Cache header.


Varnish receives that response and triggers its routine vcl_backend_response(), where we can modify what the webserver sent us.

Even though the webserver said we should not cache the page (remember: Cache-Control: No-Cache), and Varnish would by default listen to that header and not cache the page, we can overwrite it by setting the beresp.ttl object. The TTL -- or Time To Live -- determines how long we can cache that object.

Additionally, we will overwrite the status code that the webserver sent us (HTTP/200) with one that we invented (HTTP/123).

And finally, we'll remove that Laravel session ID that was generated server side by removing the beresp.http.Set-Cookie header.

After all that fiddling with the HTTP response, Varnish will store the object in its cache using our modified parameters. So it'll store that object as an HTTP/123 response without cookies, even though the server generated an HTTP/200, with cookies and clearly indicated the page shouldn't be cached.


So what eventually gets delivered to the client is a page for (our client requested with an HTTP status code of HTTP/123 (the server actually said it was an HTTP/200), without cookies being set.

Such Varnish power. Much amaze. So many places to fuck things up.


There's another important Varnish routine though, it's called vcl_hash(). This is where Varnish decides which parameters it will use to determine its cache/hash key. If the key can match an object already in the cache, it will deliver that object.

By default, Varnish looks at 3 key parameters: the Host-header, the URL being requested and the cookies sent by the client.

Behind the scenes, it will take care of the Content-Encoding that clients can request: it'll store a compressed version of every page in memory, but if a client requests a plain text -- non-compressed -- version, it'll decompress it on the way out and deliver a plain text version to the client.

If any of the parameters inside the vcl_hash() routine changes, Varnish will consider it a new HTTP request and the cache lookup will fail.


If you were thinking ahead, you can see a problem with that kind of hash-key determination.

For instance, the URL /?page=1&languange=nl will show the same content as /?language=nl&page=1, but since the order of the parameters is different, the hash will be different. To resolve this, we normalise the incoming requests as much as possible.

Varnish offers a good tool for this called std.querysort(), which will sort the query parameters alphabetically.

Additionally, you might want to strip query parameters that are irrelevant to the backend, like Google Analytics campaign trackers.


Once you normalised the URL, cookies are the Next Big Thing.

Because they are by default used in the hash-key lookup, any change in the cookies client-side will result in a different hash and a new object in the cache.

If every one of your users has a unique PHP session ID, every Cookie set will be different and thus every hash-key will be unique. Just setting a unique PHP session ID for all your users would effectively destroy caching for your site, as every request is unique.

There are a couple of fixes for this, but the biggest one is to only initialise a PHP session when you need it (ie: on the login page), and not for every page. This is called lazy session initialisation.

Alternatively, some XSRF implementations rely on cookies and should be unique per user too.


Varnish offers an interesting VMOD (extension to Varnish) to help manipulate Cookies.

The cookie.parse object allows to you do interesting things like whitelisting cookies in Varnish, without crazy mind-breaking regular expressions.


In addition, the Host header is also a key in your hashing. So a request for and, even though they're probably the same site, would result in 2 different cached objects in Varnish.

To help Varnish, it's best to redirect users to a single domain and not keep multiple possible domains that would result in the same content being generated.

There are plenty of SEO & technical reasons this is a good move too.

The code example above detects a page without the www-subdomain and triggers a synthetic response in Varnish, via the new vcl_synth() routine.

In that routine, you can detect the resp.status object and create logic handling that differently. In this case, we set a custom error code of 720 in order to replace the Location block in the HTTP response and force a 301 redirect.


That same vcl_hash() can be expended to take any HTTP header into account when determining the cache key. It can add Authorization headers for basic HTTP authentication, custom headers for your API, ...

We also use it at Nucleus to separate cache responses for HTTP vs. HTTPS content, since we often use an Nginx TLS proxy in front of Varnish.

Part Five: Flushing the Cache


There are actually more than 2 hard things in computer science: naming things, cache invalidation, off-by-one errors, DNS, Docker, ...


There are basically 3 ways to flush a cache: the old PURGE HTTP method (which is basically obsoleted), the newer "ban" method (suggested) or the absolute lazy way: restart your varnish service.


The HTTP PURGE method is a "special" HTTP request where you can set the method to PURGE. That in and of itself doesn't actually do anything, but it allows you to catch that special method in varnish's vcl_recv() routine (the very first routine Varnish will call when it receives a request) and trigger an internal cache flush via the return (purge); call.


The preferred way is actually to use "bans": this uses the Varnish Administration CLI tool called varnishadm to connect to a running Varnish daemon on its administrative port and send it control commands.

This syntax gives you more flexibility to flush content based on regex's, either on the host, the URL or arbitrary headers.

Part Six: ESI, TTL's, Grace & Saint


More acronyms! More!


Edge Side Includes are a very powerful but tricky feature of Varnish. By default, Varnish will either cache an entire page, or not cache an entire page.

With ESI, Varnish can cache parts of a page. Funky, right?

In a way, this works very similar to PHP's include or require method or the old Server Side Includes in webservers.


Take for instance a default layout for a news website: it'll contain a personal greeting at the top, some ads, some navigation and the main body.

These different "fragments" that usually build a page are what Varnish needs to be able to do partial caching.


Instead of generating the content for each partial directly in the HTML/DOM of the page, you generate special esi:include tags instead.


Eventually, your server-side generated HTML looks like this. It has several esi:include blocks, where it would normally have either include/require statements from PHP. But instead of having PHP do the combining of the page, those esi:include tags are going to be interpreted by Varnish.


When Varnish processes a vcl_backend_response(), it can look inside the body and detect those esi:include tags.

For each of those ESI tags, it will -- on its own -- send a new request to the backend and request those URLs. The backend will in turn process them, reply them back to Varnish, and Varnish will build the page by glueing all the fragments together.

The biggest benefit is that each fragment can have its own caching policy: in the best case scenario, all those esi:include calls were already in the Varnish cache and it could assemble the page entirely from memory.

The magic of ESI is that the user just requested a single page, it had no idea that it would in turn fire of multiple small requests to the backend to assemble that page before getting it delivered.


Now, by default, Varnish will look at the Cache-Control headers (or ETAG, Expires, ...) to determine how long it can cache an object. In the absolute best case scenario, the HTTP response headers determine the lifetime of each HTTP response.

But you can artificially overwrite that response by setting a new value in the beresp.ttl object.


The TTL or cache lifetime can be modified even further by introducing the concept of Grace Mode.

You can let Varnish keep objects in its cache that have actually surpassed their lifetime. This is useful in case your backends are down; it's better to serve old content, than no content.

It's also of value when there are HTTP requests coming in for expired objects: Varnish can reply with a slightly outdated response, while asynchronously fetching a new version from the backend. Your user never needs to wait for that response (it got a slightly outdated response), while your webserver can generate the response.

This is absolutely a lifesaver in cache stampedes scenario's, where Varnish will only send a single request to the backend, even though 100+ users are requesting the same page.

Bonus: CLI tools


Quick demo: some CLI tools.


varnishhist shows a histogram of the requests currently being served by Varnish: the more to the left they are shown, the faster they have been served. These are generally more cache hits, as they're served the fastest.

The more items to the right, the slower each request is.

A pipe sign | is a cache hit, a hashtag # is a cache miss (can also be a cache miss on purpose, if the VCL code says return (pass);).

varnishhist is a good quick glance to see if Varnish is caching anything and if so, what the overal ratio is: lots of cache hits or misses. But beyond that, the tool probably won't tell you that much.


varnishlog on the other hand is the real gem: it will show in great detail the request headers of a user and the response headers returned by Varnish. If you really need to debug Varnish, this'll be the tool that tells you the most.

There are filters you can apply to only log requests from a certain IP (ie: your source IP), only log requests matching a certain URL pattern, you can see which cookies are being stripped in the VCL and which aren't, etc.

The only downside: varnishlog can be a bit daunting at first with lots of output, it takes a while to grok the format.


Another interesting one, especially when just getting started with Varnish, is varnishncsa: it takes the complicated varnishlog and shows the output just like the Apache or Nginx access logs.

It won't tell you many details (like which cookies get stripped, which VCL routines got triggered etc.), but it will tell you in a more readable format (or at least: a format you're more used to) which requests Varnish is serving.

You can also run varnishncsa as a daemon/service to log all requests made by Varnish in this format, a bit like the default behaviour in Apache or Nginx. For debugging purposes, this could be useful.

Part Seven: getting started with Varnish


Let's get started.


This is the really short version of getting started with Varnish;

  • Download my Varnish templates from Github
  • Install varnish, by default it'll run on a custom port (check /etc/sysconfig/varnish)
  • Configure your backend so that it points to your existing Apache/Nginx
  • Test your site on the custom port: domain.tld:8080
  • If everything still works, swap the ports: set your Apache/Nginx on port :8080, move Varnish to port :80 and change the backends in Varnish so that it sends its requests to port :8080.

After that, you're done: Varnish now serves all requests on port :80 and will proxy everything it can't serve to port :8080.

Chances are, it'll take longer to implement Varnish than just those couple of steps, but that's basically the gist of it. Run Varnish on an alternative port, so that you can test it safely without interfering with your site, and once it's tested and approved, swap out the ports and let it run on port :80.

A couple of tips, that might help you get started quicker;

As with everything: testing, testing & testing is what will make Varnish deployment a success!


The end!

For any questions, leave a comment on this blog, poke me on Twitter, send me an e-mail, ...

The post Varnish Explained appeared first on

by Mattias Geniar at October 20, 2016 01:33 AM

October 19, 2016

macOS: Convert PDF to PNG files for each page

The post macOS: Convert PDF to PNG files for each page appeared first on

I had to convert a long PDF to multiple PNG files, and it's pretty easy with ghostscript at the terminal.

First, install Ghostscript using Homebrew.

$ brew install ghostscript

Next up, convert the PDF to PNG files.

$ gs -dNOPAUSE -sDEVICE=png16m -r256 -sOutputFile=page%03d.png input.pdf

A quick rundown:

  • -sDEVICE=png16m: the convertion to use, in this case convert to a 24-bit RGB color png file. If you value transparancy, use -sDEVICE=pngalpha.
  • -r256: the dimensions of your png file, the smaller the number, the smaller the size of your png's. To give you an idea, -r256 comes down to 2816×2176 pixels.
  • -sOutputFile=page%03d.png: number each page of the PDF as "pageXXXX.png", where XXXX is the number of the page.
  • input.pdf: the PDF you want to convert.

Good ol' Ghostscript saves the day, yet again.

The post macOS: Convert PDF to PNG files for each page appeared first on

by Mattias Geniar at October 19, 2016 07:39 PM

October 18, 2016


Five Ways That Good Guys Share More Than Bad Guys

It takes a lot for me to write a cybersecurity blog post these days. I spend most of my writing time working on my PhD. Articles like Nothing Brings Banks Together Like A Good Hack drive me up the wall, however, and a Tweet rant is insufficient. What fired me up, you might ask? Please read the following excerpt:

[Troels] Oerting, with no small dose of grudging admiration, says his adversaries excel at something that can’t be addressed with deep pockets or killer software: They’re superb networkers. “The organized crime groups in cyber are sharing much better than we are at the moment,” says Oerting, a Dane with a square jaw and the watchful eyes of a cop who’s investigated the underworld for 35 years. “They are sharing methodologies, knowledge, tools, practices—what works and what doesn’t.”

Statements like these are regularly submitted without evidence. In response, I provide five sources of evidence why organized crime groups do not share more than defenders.

1. Solution providers share. Both commercial and not-for-profit solution providers share enormous amounts of information on the security landscape. Some of it is free, and some of it is sold as products or consulting. Thousands of security companies and not-for-profit providers compete for your attention, producing white papers, Webinars, and other resources. You might argue that all of them claim to be the answer to your problem. However, this situation is infinitely better than the 1980s and early 1990s. Back then, hardly any solutions, or even security companies and organizations, existed at all.

Criminal solution providers share, but they do so by selling their wares. This is true for the open world as well, but the volume of the open world is orders of magnitude greater.

2. Government agencies share. My fellow Americans, has your organization you been visited by the FBI? Federal agents notified more than 3,000 U.S. companies [in 2013] that their computer systems had been hacked. The agents didn't just walk in, drop a letter, and leave. If a relationship did not exist previously, it will now be developed.

Beyond third party breach notifications, agencies such as NIST, DHS, and others regularly share information with organizations. They may not share as much as we would like, but again, historical perspective reveals great progress.

3. Books, articles, and social media share. The amount of readable material on security is astounding. Again, in the late 1980s and early 1990s hardly any books or articles were available. Now, thousands of resources exist, with new material from publishers like No Starch arriving monthly. Where are the books written by the underground?

4. Security conferences share. You could spend every week of the year at a security conference. If you happen to miss a talk, it's likely the incomparable Iron Geek recorded it. Does the underground offer similar opportunities?

5. Private groups and limited information exchange groups share. A final category of defender sharing takes place in more controlled settings. These involve well-established Information Sharing and Analysis Centers (ISACs), developing Information Sharing and Analysis Organizations (ISAOs), and private mailing lists and forums with limited membership. These could possibly be the closest analogue to the much-esteemed underground. Even if you disregard points 1-4 above, the quality of information shared in this final category absolutely equals, if not exceeds, anything you would find in the criminal world.

If you disagree with this analysis, and continue to lament that bad guys share more than the good guys, what evidence can you provide?

by Richard Bejtlich ( at October 18, 2016 03:17 PM

October 17, 2016

Evaggelos Balaskas

Linux Raid mdadm md0

Linux Raid

This blog post is created as a mental note for future reference

Linux Raid is the de-facto way for decades in the linux-world on how to create and use a software raid. RAID stands for: Redundant Array of Independent Disks. Some people use the I for inexpensive disks, I guess that works too!

In simple terms, you can use a lot of hard disks to behave as one disk with special capabilities!

You can use your own inexpensive/independent hard disks as long as they have the same geometry and you can do almost everything. Also it’s pretty easy to learn and use linux raid. If you dont have the same geometry, then linux raid will use the smallest one from your disks. Modern methods, like LVM and BTRFS can provide an abstract layer with more capabilities to their users, but some times (or because something you have built a loooong time ago) you need to go back to basics.

And every time -EVERY time- I am searching online for all these cool commands that those cool kids are using. Cause what’s more exciting than replacing your -a decade ago- linux raid setup this typical Saturday night?

Identify your Hard Disks

% find /sys/devices/ -type f -name model -exec cat {} \;

% lsblk
sda      8:0    0 931.5G  0 disk
sdb      8:16   0 931.5G  0 disk
sdc      8:32   0 931.5G  0 disk  


sda   disk  931.5G ST1000DX001-1CM1
sdb   disk  931.5G ST1000DX001-1CM1
sdc   disk  931.5G ST1000DX001-1CM1

Create a RAID-5 with 3 Disks

Having 3 hard disks of 1T size, we are going to use the raid-5 Level . That means that we have 2T of disk usage and the third disk with keep the parity of the first two disks. Raid5 provides us with the benefit of loosing one hard disk without loosing any data from our hard disk scheme.


% mdadm -C -v /dev/md0 --level=5 --raid-devices=3 /dev/sda /dev/sdb /dev/sdc 

mdadm: layout defaults to left-symmetric
mdadm: layout defaults to left-symmetric
mdadm: chunk size defaults to 512K
mdadm: sze set to 5238784K
mdadm: Defaulting to version 1.2 metadata
md/raid:md0 raid level 5 active with 2 our of 3 devices, algorithm 2
mdadm: array /dev/md0 started.

% cat /proc/mdstat

Personalities : [raid6] [raid5] [raid4]
md0: active raid5 sdc[3] sdb[2] sda[1]
        10477568 blocks super 1.2 level 5, 512k chink, algorith 2 [3/3] [UUU]

unused devices: <none>

running lsblk will show us our new scheme:

sda   disk  931.5G ST1000DX001-1CM1
md0   raid5   1.8T
sdb   disk  931.5G ST1000DX001-1CM1
md0   raid5   1.8T
sdc   disk  931.5G ST1000DX001-1CM1
md0   raid5   1.8T

Save the Linux Raid configuration into a file

Software linux raid means that the raid configuration is actually ON the hard disks. You can take those 3 disks and put them to another linux box and everything will be there!! If you are keeping your operating system to another harddisk, you can also change your linux distro from one to another and your data will be on your linux raid5 and you can access them without any extra software from your new linux distro.

But it is a good idea to keep the basic conf to a specific configuration file, so if you have hardware problems your machine could understand what type of linux raid level you need to have on those broken disks!

% mdadm --detail --scan >> /etc/mdadm.conf

% cat /etc/mdadm.conf
ARRAY /dev/md0 metadata=1.2 name=MyServer:0 UUID=ef5da4df:3e53572e:c3fe1191:925b24cf

UUID - Universally Unique IDentifier

Be very careful that the above UUID is the UUID of the linux raid on your disks.
We have not yet created a filesystem over this new disk /dev/md0 and if you need to add this filesystem under your fstab file you can not use the UUID of the linux raid md0 disk.

Below there is an example on my system:

% blkid
/dev/sda: UUID="ef5da4df-3e53-572e-c3fe-1191925b24cf" UUID_SUB="f4e1da17-e4ff-74f0-b1cf-6ec86eca3df1" LABEL="MyServer:0" TYPE="linux_raid_member"
/dev/sdb: UUID="ef5da4df-3e53-572e-c3fe-1191925b24cf" UUID_SUB="ad7315e5-56ce-bd8c-75c5-0a72893a63db" LABEL="MyServer:0" TYPE="linux_raid_member"
/dev/sdc: UUID="ef5da4df-3e53-572e-c3fe-1191925b24cf" UUID_SUB="a90e317e-4384-8f30-0de1-ee77f8912610" LABEL="MyServer:0" TYPE="linux_raid_member" 

/dev/md0: LABEL="data" UUID="48fc963a-2128-4d35-85fb-b79e2546dce7" TYPE="ext4" 

% cat /etc/fstab

UUID=48fc963a-2128-4d35-85fb-b79e2546dce7   /backup auto    defaults    0   0

Replacing a hard disk

Hard disks will fail you. This is a fact that every sysadmin knows from day one. Systems will fail at some point in the future. So be prepared and keep backups !!

Failing a disk

Now it’s time to fail (if not already) the disk we want to replace:

% mdadm --manage /dev/md0 --fail /dev/sdb
mdadm: set /dev/sdb faulty in /dev/md0

Remove a broken disk

Here is a simple way to remove a broken disk from your linux raid configuration. Remember with raid5 level we can manage with 2 hard disks.

% mdadm --manage /dev/md0 --remove /dev/sdb
mdadm: hot removed /dev/sdb from /dev/md0

% cat /proc/mdstat
Personalities : [raid6] [raid5] [raid4]
md0 : active raid5 sda[1] sdc[3]
      1953262592 blocks super 1.2 level 5, 512k chunk, algorithm 2 [3/2] [_UU]

unused devices: <none>

dmesg shows:

% dmesg | tail

md: data-check of RAID array md0
md: minimum _guaranteed_  speed: 1000 KB/sec/disk.
md: using maximum available idle IO bandwidth (but not more than 200000 KB/sec) for data-check.
md: using 128k window, over a total of 976631296k.
md: md0: data-check done.
md/raid:md0: Disk failure on sdb, disabling device.
md/raid:md0: Operation continuing on 2 devices.
RAID conf printout:
 --- level:5 rd:3 wd:2
 disk 0, o:0, dev:sda
 disk 1, o:1, dev:sdb
 disk 2, o:1, dev:sdc
RAID conf printout:
 --- level:5 rd:3 wd:2
 disk 0, o:0, dev:sda
 disk 2, o:1, dev:sdc
md: unbind<sdb>
md: export_rdev(sdb)

Adding a new disk - replacing a broken one

Now it’s time to add a new and (if possible) clean hard disk. Just to be sure, I always wipe with dd the first few kilobytes of every disk with zeros.

Using mdadm to add this new disk:

# mdadm --manage /dev/md0 --add /dev/sdb
mdadm: added /dev/sdb

% cat /proc/mdstat
Personalities : [raid6] [raid5] [raid4]
md0 : active raid5 sdb[4] sda[1] sdc[3]
      1953262592 blocks super 1.2 level 5, 512k chunk, algorithm 2 [3/2] [_UU]
      [>....................]  recovery =  0.2% (2753372/976631296) finish=189.9min speed=85436K/sec

unused devices: <none>

For a 1T Hard Disk is about 3h of recovering data. Keep that in mind on scheduling the maintenance window.

after a few minutes:

% cat /proc/mdstat
Personalities : [raid6] [raid5] [raid4]
md0 : active raid5 sdb[4] sda[1] sdc[3]
      1953262592 blocks super 1.2 level 5, 512k chunk, algorithm 2 [3/2] [_UU]
      [>....................]  recovery =  4.8% (47825800/976631296) finish=158.3min speed=97781K/sec

unused devices: <none>

mdadm shows:

% mdadm --detail /dev/md0 

        Version : 1.2
  Creation Time : Wed Feb 26 21:00:17 2014
     Raid Level : raid5
     Array Size : 1953262592 (1862.78 GiB 2000.14 GB)
  Used Dev Size : 976631296 (931.39 GiB 1000.07 GB)
   Raid Devices : 3
  Total Devices : 3
    Persistence : Superblock is persistent

    Update Time : Mon Oct 17 21:52:05 2016
          State : clean, degraded, recovering
 Active Devices : 2
Working Devices : 3
 Failed Devices : 0
  Spare Devices : 1

         Layout : left-symmetric
     Chunk Size : 512K

 Rebuild Status : 58% complete

           Name : MyServer:0  (local to host MyServer)
           UUID : ef5da4df:3e53572e:c3fe1191:925b24cf
         Events : 554

    Number   Major   Minor   RaidDevice State
       1       8       16        1      active sync   /dev/sda
       4       8       32        0      spare rebuilding   /dev/sdb
       3       8       48        2      active sync   /dev/sdc

You can use watch command that refreshes every two seconds your terminal with the output :

# watch cat /proc/mdstat

Every 2.0s: cat /proc/mdstat                                                                                               Mon Oct 17 21:53:34 2016

Personalities : [raid6] [raid5] [raid4]
md0 : active raid5 sdb[4] sda[1] sdc[3]
      1953262592 blocks super 1.2 level 5, 512k chunk, algorithm 2 [3/2] [_UU]
      [===========>.........]  recovery = 59.4% (580918844/976631296) finish=69.2min speed=95229K/sec

unused devices: <none>

Growing a Linux Raid

Even so … 2T is not a lot of disk usage these days! If you need to grow-extend your linux raid, then you need hard disks with the same geometry (or larger).

Steps on growing your linux raid are also simply:

# Umount the linux raid device:
% umount /dev/md0

# Add the new disk
% mdadm --add /dev/md0 /dev/sdd

# Check mdstat
% cat /proc/mdstat

# Grow linux raid by one device
%  mdadm --grow /dev/md0 --raid-devices=4

# watch mdstat for reshaping to complete - also 3h+ something
% watch cat /proc/mdstat

# Filesystem check your linux raid device
% fsck -y /dev/md0

# Resize - Important
% resize2fs /dev/md0

But sometimes life happens …

Need 1 spare to avoid degraded array, and only have 0.

mdadm: Need 1 spare to avoid degraded array, and only have 0.


mdadm: Failed to initiate reshape!

Sometimes you get an error that informs you that you can not grow your linux raid device! It’s not time to panic or flee the scene. You’ve got this. You have already kept a recent backup before you started and you also reading this blog post!

You need a (an extra) backup-file !

% mdadm --grow --raid-devices=4 --backup-file=/tmp/backup.file /dev/md0
mdadm: Need to backup 3072K of critical section..

% cat /proc/mdstat
Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10]
md0 : active raid5 sda[4] sdb[0] sdd[3] sdc[1]
      1953262592 blocks super 1.2 level 5, 512k chunk, algorithm 2 [4/4] [UUUU]
      [>....................]  reshape =  0.0

% (66460/976631296) finish=1224.4min speed=13292K/sec

unused devices: <none>

1224.4min seems a lot !!!

dmesg shows:

% dmesg
[   36.477638] md: Autodetecting RAID arrays.
[   36.477649] md: Scanned 0 and added 0 devices.
[   36.477654] md: autorun ...
[   36.477658] md: ... autorun DONE.

[  602.987144] md: bind<sda>
[  603.219025] RAID conf printout:
[  603.219036]  --- level:5 rd:3 wd:3
[  603.219044]  disk 0, o:1, dev:sdb
[  603.219050]  disk 1, o:1, dev:sdc
[  603.219055]  disk 2, o:1, dev:sdd
[  608.650884] RAID conf printout:
[  608.650896]  --- level:5 rd:3 wd:3
[  608.650903]  disk 0, o:1, dev:sdb
[  608.650910]  disk 1, o:1, dev:sdc
[  608.650915]  disk 2, o:1, dev:sdd
[  684.308820] RAID conf printout:
[  684.308832]  --- level:5 rd:4 wd:4
[  684.308840]  disk 0, o:1, dev:sdb
[  684.308846]  disk 1, o:1, dev:sdc
[  684.308851]  disk 2, o:1, dev:sdd
[  684.308855]  disk 3, o:1, dev:sda
[  684.309079] md: reshape of RAID array md0
[  684.309089] md: minimum _guaranteed_  speed: 1000 KB/sec/disk.
[  684.309094] md: using maximum available idle IO bandwidth (but not more than 200000 KB/sec) for reshape.
[  684.309105] md: using 128k window, over a total of 976631296k.


% cat /proc/mdstat
Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10]
md0 : active raid5 sda[4] sdb[0] sdd[3] sdc[1]
      1953262592 blocks super 1.2 level 5, 512k chunk, algorithm 2 [4/4] [UUUU]
      [>....................]  reshape =  0.0

% (349696/976631296) finish=697.9min speed=23313K/sec

unused devices: <none>

ok it’s now 670minutes

Time to use watch:

(after a while)

% watch cat /proc/mdstat

Personalities : [linear] [multipath] [raid0] [raid1] [raid6] [raid5] [raid4] [raid10]
md0 : active raid5 sda[4] sdb[0] sdd[3] sdc[1]
      1953262592 blocks super 1.2 level 5, 512k chunk, algorithm 2 [4/4] [UUUU]
      [===========>......]  reshape = 66.1% (646514752/976631296) finish=157.4min speed=60171K/sec

unused devices: <none>

mdadm shows:

% mdadm --detail /dev/md0

        Version : 1.2
  Creation Time : Thu Feb  6 13:06:34 2014
     Raid Level : raid5
     Array Size : 1953262592 (1862.78 GiB 2000.14 GB)
  Used Dev Size : 976631296 (931.39 GiB 1000.07 GB)
   Raid Devices : 4
  Total Devices : 4
    Persistence : Superblock is persistent

    Update Time : Sat Oct 22 14:59:33 2016
          State : clean, reshaping
 Active Devices : 4
Working Devices : 4
 Failed Devices : 0
  Spare Devices : 0

         Layout : left-symmetric
     Chunk Size : 512K

 Reshape Status : 66% complete
  Delta Devices : 1, (3->4)

           Name : MyServer:0
           UUID : d635095e:50457059:7e6ccdaf:7da91c9b
         Events : 1536

    Number   Major   Minor   RaidDevice State
       0       8       16        0      active sync   /dev/sdb
       1       8       32        1      active sync   /dev/sdc
       3       8       48        2      active sync   /dev/sdd
       4       8        0        3      active sync   /dev/sda

be patient and keep an aye on mdstat under proc.

So basically those are the steps, hopefuly you will find them useful.

Tag(s): md0, mdadm, linux, raid

October 17, 2016 03:38 PM

October 12, 2016


Face to Face: Goodbye RT, Hello GitHub

Last week, the OpenSSL dev team had another face-to-face meeting. It was a week of “mosts”: most of the team flew in for most of the week, and most of it was funded by the CII/LF

We got a great deal accomplished during that week. We do many things by vote, and having everyone in the room to talk not only beats email all to hell, but it ensures that we’re all on the same page for the decisions we make. Sure, not everything was a unanimous decision, but none were decided by narrow margins.

In this post I’m going to talk about two important decisions.

But first, here’s a team picture, so you can get an idea of the faces behind the code:

The first issue is about RT our old issue/ticket tracking system. After many fine years of service, frankly interspersed with periods of neglect, we are putting the old nag out to pasture and retiring the service. We’ll soon be updating the website, and the distribution files, but starting immediately we ask that you open GitHub issues instead.

One of the nice things about RT was its email integration. Conversation could happen on openssl-dev and it would get collected into the ticket. We know similar things are possible with various GitHub hooks, so please be patient as we make it more seamless. The general pattern of open an issue and then post a pointer and the text on openssl-dev should still be good, however.

The obvious question is what do we do with the existing tickets? We have been making pretty excellent progress over the past two years (the red line represents the open tickets and the blue line is the total number of tickets closed):

Last week we came to the painful decision to declare a “bug bankruptcy” and close any tickets that are at least two years old. A detailed list of the RT tickets will be posted to openssl-dev, and we will try to send email to everyone mentioned in a ticket (without spamming the dev list), and if their issue is still important, they should open a GitHub issue.

This will result in about half of the 326 open tickets being closed. It was a tough decision; we all had a couple of favorites we wanted to fix “real soon now.” :) But it is the most honest and transparent thing to do. And we are well aware that you don’t get to do this more than once.

Without RT, how is the community supposed to submit patches? For some time we encouraged the use of GitHub pull requests. Now it’s pretty much required. And it’s not right that we don’t do the same thing ourselves. Effectively immediately, all of the team changes will be done via public pull requests. Some of us have been doing this for awhile, but last week we made it official policy that all public team work must be done in public branches. Our policy of every submission requiring another team member to review, or outside code requiring two approvals, has not changed. The new GitHub review tools help make this easier, thanks GitHub!

Of course, security fixes will still be developed in private and not disclosed early. But with this new policy, the community should have the chance to make an earlier impact on improving OpenSSL, which is what this is all about.

October 12, 2016 01:00 AM

October 11, 2016

Evaggelos Balaskas

Find your hard disk model

# /bin/find /sys/devices -type f -name model -exec cat {} \; 


# udisksctl status

MODEL                     REVISION  SERIAL               DEVICE
KINGSTON SV300S37A120G    527ABBF0  50026B774902D7E5     sda     
WDC WD10EURX-63FH1Y0      01.01A01  WD-WMC1U5525831      sdb     
WDC WD10EZRX-00A8LB0      01.01A01  WD-WCC1U2715714      sdc     
VB0250EAVER               HPG9      Z3TLRVYK             sdd     
VB0250EAVER               HPG9      Z3TLRRKM             sde   


sda   disk  111,8G KINGSTON SV300S3
sdb   disk  931,5G WDC WD10EURX-63F
sdc   disk  931,5G WDC WD10EZRX-00A
sdd   disk  232,9G VB0250EAVER     
sde   disk  232,9G VB0250EAVER     


# smartctl -a -i /dev/sda

# hdparm -i /dev/sda

# lshw -class disk

# hwinfo --disk

Tag(s): hard disk

October 11, 2016 09:30 AM

Racker Hacker

Why should students learn to write code?

Classroom computersThere are lots of efforts underway to get students (young and old) to learn to write code. There are far-reaching efforts, like the Hour of Code, and plenty of smaller, more focused projects, such as the Design and Technology Academy (part of Northeast ISD here in San Antonio, Texas). These are excellent programs that enrich the education of many students.

I often hear a question from various people about these programs:

Why should a student learn to write code if they aren’t going to become a software developer or IT professional?

It’s a completely legitimate question and I hope to provide a helpful response in this post.

Some students will actually enter the IT field

This may seem obvious, but it’s important to note that many students may choose to enter the IT field. They may not become full-time software developers, but the experience is useful for many different IT jobs. For example, knowing some basic principles about how software works is critical for system administrators, network administrators, project managers, and people managers. These skills could give students an edge later in their IT career.

Students learn to measure twice and cut once

The concept of thorough planning before execution shows up in many different fields. You can find it in general engineering, architecture, medicine, and criminal justice. A failure to plan often leads to bigger challenges down the line.

In software development, planning is key. Where will you store your data? How much data will you need to store? Does the user need a graphical interface? What if the amount of users on the system increases by a factor of ten? How do we keep the system secure?

I talk to computer science students at UTSA on a regular basis. One of their most challenging courses involves a group project where the students must build a fully functional application that solves a need. The students run head-first into a myriad of problems during development. They almost always learn that it’s much easier to talk through how everything will fit together before they write one line of software.

Students learn to think logically

Dale Carnegie said:

When dealing with people, remember you are not dealing with creatures of logic, but creatures of emotion.

Humans are swayed easily by emotional appeals and logic is often tossed aside. Computers are the total opposite. You can scream for joy, cry uncontrollably, or yell angrily at a computer and it keeps doing the same thing. It doesn’t understand that you just told it to eat your data. It doesn’t understand that you can’t figure out why an error is occurring.

Students can learn a lot from the way computers work. As one of my journalism professors said in school:

You put garbage in? You’ll get garbage out.

Students will learn to be explicit in their instructions and contemplate how to handle treacherous situations in software. Some smaller failures might result in an error or a poor user experience. Others could result in a buffer overflow that leads to a costly security breach. In the end, computers can only do what they’re told and it’s up to the developer to tell to the computer — in the computer’s language — in all of these situations.

Outside of the world of computers, learning to be explicit has its benefits. It reduces confusion in conversations, leads to better results in group projects, and it encourage students to structure their thoughts into more organized communication.

There’s more to IT than writing code

Software doesn’t run without computer hardware. We live in the age of easily accessible, inexpensive cloud infrastructure where you can have a server online in seconds, but it’s still someone’s computer. Within large businesses, software developers are often asked to justify the resources they need to deploy their applications.

There is obviously some technical knowledge involved here, especially around the topic of sizing a server for a particular software workload. However, there are plenty of non-technical questions to ask.

Can we re-use depreciated hardware to reduce capital expenditures? Is our datacenter space limited by power, space, or something else? Can we select a lower-wattage server to reduce power consumption (if that’s the bigger expense)? Will ordering in a larger batch allow us to drive down the hardware cost?

Software developers wield significant power if they can use their technical skills to branch into the world of accounting, finance, and basic engineering. A well-rounded approach could also allow developers to get more hardware than they planned to get if they make the purchases in a smarter way.

Basic understanding of computers is useful

Almost every technical person has fielded that awkward question from a family member at a gathering or during the holidays:

I heard you do computer stuff? I think I have a virus — could you look at it?

All students should have a basic understanding of how a computer works, even if they never write software or work in IT. This knowledge helps keep all of us a bit safer online and it helps to diagnose some issues before they become a serious problem. Approaching computers with an observant and inquisitive mind will reduce security breaches and increase confidence. We will also flood the market with people who can teach others the basics about their technology


All students could learn some important life lessons simply from learning how to write some code. Does this mean that all students must write some software before they graduate? Definitely not.

Many of the lessons learned from writing software will easily transfer into other fields and disciplines. Gaining these skills is a lot like learning a foreign language. If you use them frequently, you’ll have a strong foundation of knowledge to build upon over time. Even if you don’t use them frequently, they could give you that small edge that you need later on in your professional career.

Photo credit: Pixabay

The post Why should students learn to write code? appeared first on

by Major Hayden at October 11, 2016 04:08 AM

Nitrokey gnuk firmware update via DFU

The Nitrokey (start, all of them) can be upgraded to a newer GNUK firmware. However, this can only be done via ST Link or DFU, if you use the Gnuk USB firmware upgrade you will brick the device. This guide shows you how to attach a DFU adapter and how to flash firmware to a Nitrokey, both for upgrading or unbricking an USB upgraded one.

October 11, 2016 12:00 AM

October 10, 2016

MySQL restore after a crash and disk issues

Recently I had to restore a MySQL server. The hardware had issues with the storage and required some FSCK's, disk replacements and a lot of RAID and LVM love to get working again. Which was the easy part. MySQL was a bit harder to fix. This post describes the proces I used to get MySQL working again with a recent backup. In this case it was a replicated setup so the client had no actual downtime.

October 10, 2016 12:00 AM

October 09, 2016

Colin Percival

EC2's most dangerous feature

As a FreeBSD developer — and someone who writes in C — I believe strongly in the idea of "tools, not policy". If you want to shoot yourself in the foot, I'll help you deliver the bullet to your foot as efficiently and reliably as possible. UNIX has always been built around the idea that systems administrators are better equipped to figure out what they want than the developers of the OS, and it's almost impossible to prevent foot-shooting without also limiting useful functionality. The most powerful tools are inevitably dangerous, and often the best solution is to simply ensure that they come with sufficient warning labels attached; but occasionally I see tools which not only lack important warning labels, but are also designed in a way which makes them far more dangerous than necessary. Such a case is IAM Roles for Amazon EC2.

October 09, 2016 08:30 AM

October 08, 2016

Toolsmith Release Advisory: Malware Information Sharing Platform (MISP) 2.4.52

7 OCT 2016 saw the release of MISP 2.4.52.
MISP, Malware Information Sharing Platform and Threat Sharing, is free and open source software to aid in sharing of threat and cyber security indicators.
An overview of MISP as derived from the project home page:
  • Automation:  Store IOCs in a structured manner, and benefit from correlation, automated exports for IDS, or SIEM, in STIX or OpenIOC and even to other MISPs.
  • Simplicity: the driving force behind the project. Storing and using information about threats and malware should not be difficult. MISP allows getting the maximum out of your data without unmanageable complexity.
  • Sharing: the key to fast and effective detection of attacks. Often organizations are targeted by the same Threat Actor, in the same or different Campaign. MISP makes it easier to share with and receive from trusted partners and trust-groups. Sharing also enables collaborative analysis, preventing redundant work.
The MISP 2.4.52 release includes the following new features:
  • Freetext feed import: a flexible scheme to import any feed available on Internet and incorporate them automatically in MISP. The feed imported can create new event or update an existing event. The freetext feed feature permits to preview the import and quickly integrates external sources.
  • Bro NIDS export added in MISP in addition to Snort and Suricata.
  • A default role can be set allowing flexible role policy.
  • Functionality to allow merging of attributes from a different event.
  • Many updates and improvement in the MISP user-interface including filtering of proposals at index level.
Bug fixes and improvements include:
  • XML STIX export has been significantly improved to ensure enhanced compatibility with other platforms.
  • Bruteforce protection has been fixed.
  • OpenIOC export via the API is now possible.
  • Various bugs at the API level were fixed.
This is an outstanding project that will be the topic of my next Toolsmith In-depth Analysis.

Cheers...until next time.

by Russ McRee ( at October 08, 2016 09:21 PM

Ansible-cmdb v1.17: Generate a host overview of Ansible facts.

I've just released ansible-cmdb v1.17. Ansible-cmdb takes the output of Ansible's fact gathering and converts it into a static HTML overview page containing system configuration information. It supports multiple templates (fancy html, txt, markdown, json and sql) and extending information gathered by Ansible with custom data.

This release includes the following bugfixes;:

  • A bug that caused whitespace to be added each time a character was typed into the search box was fixed.
  • Fix for HP-UX missing netmask (patch by Randy Rasmussen)
  • SQL template improvements for BSD.

As always, packages are available for Debian, Ubuntu, Redhat, Centos and other systems. Get the new release from the Github releases page.

by admin at October 08, 2016 03:35 PM

October 06, 2016

Carl Chenet

Crypto-monnaies : le bitcoin

Dans cette série d’articles consacrés aux crypto-monnaies et après Quelques généralités au sujet des  crypto-monnaies, nous entrons aujourd’hui dans le vif du sujet avec l’installation du porte-feuille Bitcoin sous GNU/Linux Debian.




Introduction au Bitcoin

Le Bitcoin est la première crypto-monnaie a rencontré une large adhésion, inventé en 2008 par un dénommé Satoshi Nakamoto, personnage à ce jour non-identifié avec certitude. Le Bitcoin est rendu libre en 2009. Quelques points techniques importants à retenir au sujet de cette monnaie :

  • la masse monétaire est limitée à 21 millions d’unités
  • chaque unité est divisible jusqu’à la huitième décimale
  • de par son ancienneté et sa popularité c’est la crypto-monnaie la plus facilement échangeable, jouant souvent le rôle de monnaie intermédiaire entre les monnaies courantes (euro, dollar, etc) et les crypto-monnaies moins connues

Le but de cet article n’est pas d’entrer dans le détail des spécificités du Bitcoin, mais plutôt de fournir une introduction pratique à son utilisation. Mettons donc dès maintenant les mains dans le cambouis.

Installation du porte-feuille Bitcoin

Le porte-feuille Bitcoin est téléchargeable sur le site web officiel du projet. Nous choisissons de télécharger ce dernier directement afin de ne pas dépendre d’un mainteneur intermédiaire et d’être le plus à jour possible quant à la version du porte-feuille Bitcoin installée sur notre poste.

Vous pouvez installer le porte-feuille Bitcoin à l’aide des commandes suivantes (version Linux 64 bits) :

$ mkdir -p ~/wallets/bitcoin
$ cd ~/wallets/bitcoin
$ wget
$ sha256sum bitcoin-0.13.0-x86_64-linux-gnu.tar.gz |grep bcc1e42d61f88621301bbb00512376287f9df4568255f8b98bc10547dced96c8
bcc1e42d61f88621301bbb00512376287f9df4568255f8b98bc10547dced96c8 bitcoin-0.13.0-x86_64-linux-gnu.tar.gz
$ tar zxvf bitcoin-0.13.0-x86_64-linux-gnu.tar.gz

Lors de cette installation très standard, nous vérifions la somme de hachage de l’archive afin de vérifier sa conformité par rapport à celle fournie par le projet Bitcoin.

Utilisation du porte-feuille Bitcoin

Nous allons maintenant lancer le porte-feuille Bitcoin dans sa version graphique. Attention, par défaut ce dernier va vous créer un répertoire ~/.bitcoin et commencer à stocker la chaîne de bloc Bitcoin dans le répertoire ~/.bitcoin/blocks . Il vous faut prévoir 91 gigaoctets d’espace disque, oui je confirme 91 gigaoctets d’espace disque).

$ ./bin/bitcoin-qt

Le porte-feuille bitcoin-qt

Voici une capture d’écran du porte-feuille Bitcoin que vous obtiendrez rapidement (désolé pour les nombreux edits de l’image).

Nous constatons tout de suite tout en bas que la synchronisation avec le réseau Bitcoin est toujours en cours et que nous avons encore 7 semaines de retard. Au bout à droite, le sigle BTC indique l’unité de la monnaie en cours d’utilisation. Le petit cadenas indique quant à lui que le porte-feuille est chiffré et actuellement verrouillé (nous y reviendrons plus bas dans l’article), l’icône encore à droite indique le nombre de connexions actives au réseau Bitcoin et enfin l’icône de synchronisation indique combien de blocs il reste à traiter avant d’être synchronisé avec le réseau Bitcoin.

Première action avec votre porte-feuille Bitcoin

Chiffrez votre porte-feuille ! Cela doit être votre première action. Cela paraîtra évident à tout le monde mais choisissez un mot de passe certes complexe mais surtout que vous ne pouvez pas oublier ! Si vous lancez votre porte-feuille tous les 6 mois, ce détail a son importance (je parle d’expérience).

Recevoir de l’argent avec votre porte-feuille Bitcoin

Il va sans dire que votre première action consistera à recevoir des Bitcoins. En effet, à quoi sert à un porte-feuille vide ? À pas grand-chose. Nous allons donc générer une adresse Bitcoin pour recevoir nos premier Bitcoins que nous aurons acheté par exemple sur une plateforme d’échange Euro vers Bitcoin ou Dollar vers Bitcoin. Deux exemples parmi les plus anciennes, stables et non piratées à ce jour : Kraken et Poloniex.

Vous pouvez également échanger des Bitcoins de la main à la main ou passer par une plateforme décentralisée d’échange comme Bitsquare.

Petit rappel pour les curieux que l’idée d’investir de l’argent commence à rebuter : se procurer des Bitcoins s’apparente à vouloir se procurer des dollars : il n’y a pas de secret et il faut soit échanger de l’argent, soit offrir un service en échange de Bitcoins. Méfiez-vous des sites offrant de vous faire réaliser des tâches simples en contrepartie de Bitcoins, comme dit le proverbe l’argent ne tombe jamais du ciel.

La première étape consiste à définir une adresse de réception pour la transaction. Il faut pour cela cliquer sur l’onglet réception en haut de notre porte-feuille.


Puis nous entrons simplement un étiquette arbitraire correspondant à l’action que nous allons effectuer, nous permettant de nous rappeler de quoi il s’agissait dans quelques mois (un peu comme le talon d’un chèque).


L’adresse de réception apparaît, ainsi que le QR code associé à la transaction. Nous sommes maintenant armés pour recevoir nos premier Bitcoins, ou plus précisément au cours élevé du Bitcoin aujourd’hui, vos premier mBTC, pour milli-Bitcoin.


Cette étape de créer une étiquette pour votre transaction est complètement facultative, mais c’est une bonne pratique de remplir cette étiquette afin de garder une trace de vos différentes transactions.

Envoyer de l’argent avec votre porte-feuille Bitcoin

Mettons maintenant que vous souhaitiez envoyer des Bitcoins à un tiers, vous allez pour cela générer une adresse d’émission.


Nous entrons dans le champ Payer à l’adresse du destinataire. Puis nous précisons le contexte de la transaction dans le champ Étiquette, avant de définir dans le champ Montant à transmettre (ici en mBTC, donc 0,005 BTC). Au moment de l’envoi, la phrase de passe de votre porte-feuille Bitcoin vous est demandé afin de confirmer l’envoi.

Vous pouvez constater que des frais sont associés au paiement, ces frais permettant de rémunérer les mineurs qui vont traiter votre transaction et la certifier.

Émettre un paiement avec des frais de transaction au tarif recommandé vous assure un traitement normal de votre transaction. Émettre une transaction et en particulier à la valeur plancher de 0,00001BTC peut faire que votre transaction ne soit tout simplement vérifiée par personne et donc qu’elle échoue.

Sauvegardez votre porte-feuille Bitcoin

Si vous souhaitez utiliser régulièrement votre porte-feuille Bitcoin, n’oubliez pas de le sauvegarder. Rien de plus facile via le menu Fichier->Sauvegarder. En 2013, 7500BTC sont perdus par un utilisateur ayant jeté le disque dur contenant la clé privée de son porte-feuille Bitcoin. Au moins il n’a pas fait les choses à moitié 🙁 Une perte de données et l’impossibilité pour vous d’accéder à votre porte-feuille peut arriver très vite.


Nous avons vu au cours de cet article comment mettre en place votre porte-feuille Bitcoin et comment commencer à l’utiliser. Le but de cet article est de mettre les mains dans le cambouis et de commencer à manipuler le Bitcoin, pas d’être exhaustif sur les concepts ou le fonctionnement du Bitcoin. N’hésitez pas à consulter les liens ci-dessous pour aller plus loin dans la connaissance et l’actualité du Bitcoin.

Et de votre côté ? Qu’avez-vous pensé de cet article ? Est-il assez didactique ? Convient-il à vos attentes ? N’hésitez pas à réagir dans les commentaires.

Quelques liens pour finir


by Carl Chenet at October 06, 2016 10:00 PM

October 05, 2016

Everything Sysadmin

DevOps Handbook hitting stores Oct 6!

One of the most anticipated DevOps books in years is about to start shipping! DevOps Handbook: How to Create World-Class Agility, Reliability, & Security in Technology Organizations by Gene Kim, Jez Humble, Patrick Debois, and John Willis is the practical guide to doing all the wonderful things that The Phoenix Project talks about.

I've received an early copy of the book and it is excellent. It is very down-to-earth, practical, advice. I'll write more next week when I've had time to read the entire thing.

You can pre-order it directly from IT Revolution or via Amazon.

Check it out!

by Tom Limoncelli at October 05, 2016 03:00 PM

Sarah Allen

communication is a technical skill

For Rocky Mountain Ruby last week, I opened my talk with inspiration from Jim Weirich, a legendary developer, inspiring teacher and a great communicator. The short video clip was from a prior Rocky Mountain Ruby, where Jim Weirich shared a song he had composed for _why day:

Here are my notes from the talk which aren’t exactly what I said, but provide a bit of the story…

The ruby community has a lovely culture of openness where people share their work publicly with permissive open source licenses, providing the opportunity for serendipitous collaboration. We actively invite people to use our gems with easy installation steps, documented on clean easy-to-read home pages. We showcase whimsical code snippets hoping to inspire developers, peak the curiosity of someone new or just share our joy in our new creation. As with Jim’s song, we love sharing our code, and we like to read and learn from code written by other developers.

Just in case you think this kind of effort only makes sense for mature projects like Bundler and Sinatra, I picked a random repo from one of my favorite developers, Sarah Mei: a small app embloggen with 33 commits, developed over just a few days. The application took shape in the first day with 14 commits, and we can see that she wrote a README on that same day.

And this isn’t a unique process. I took a look at Jekyll, a popular static website generator, written in Ruby that now powers github pages. It has over 8000 commits and 625 contributors. I went back in time to 2008. There was no README that first day, but a couple of weeks later, we see a description of AutoBlog. It is still something that is only used for this developer’s own use. I’m not sure what “blog like a developer” is supposed to mean, but its a start, and there are install instructions and a license. That starting point must have made it easier to iterate a month later, with a better explanation of what it is and how it works. The README is starting to be friendlier, and more clear, but still geared mostly for a developer audience, which communicates something about the state of the project as well.

We usually think of software as being made of code, but it is really made of people. We imagine what the software should do, and through our code, we turn ideas into reality.

I chose three example of what I think of as good communication — some that I have participated in or that I merely witnessed. I believe the best communication has three parts:

  1. Big Vision
  2. Concrete Step
  3. The Path

Don’t be afraid to communicate your big vision. How will the future change because what you are creating exists? What will it feel like when you get there? Then… what are the small concrete parts that people can experience now which make them believe, and cause them to understand that it is possible. Finally, connect the dots. Describe the path where all the small parts, the ones that exist now and the ones yet to come add up to changing the world!

Bridge Foundry

Bridge Foundry has a vision of the future where the makers of technology are reflective of our society, where all people have equal access. This is not only the right thing to do from a fairness perspective; we also believe that the only way we will solve the very real and critical problems of our time is to leverage all of the talent and potential available on this planet.

One of the key activities of this organization are workshops where we outreach to women and underrepresented minorities and teach coding, with a specific language or framework.

It seems simple, but it is a carefully designed experience that was co-created by our early volunteers and students. We always seek to involve a diverse teaching and organizing team. We offer childcare, and make sure there is plenty of good food. If installation is needed, we get that out of the way the night before.

Then students can focus on learning to code in a full day, where every need is met, and they feel supported. Students and volunteers alike often think this is about learning the process of coding, the syntax and the patterns, and it is, but the primary thing we are doing is sharing this culture: our joy of coding and working together collaboratively. This is the culture that most of us live every day at our jobs, but it is invisible to those outside the industry, and is sometimes surprising to those who only know tech culture from headlines or TV.

Through the workshops and participation in our local meetup, we demonstrated results at a small scale, we iterated, and measured impact. We found that individuals taking the small step of teaching weekend workshops could see it happen. They were inspired by making an impact on just a few people, and we shared the information about how the effects were adding up. In just 6 months, our local SF Ruby meetup went from 2% to 18% women. We illustrated the path from the simple step of volunteering, teaching what you know in a fun weekend experience, to changing the industry.


Next I will share an example from the business of building a software product. In my new job, I work on the Firebase team.

The Firebase Product Manager, James Tamplin, gets up every other week in front of the whole team to give us product and customer updates. He starts every meeting by saying that we are working to “help developers build better apps and grow successful businesses.” And that’s great, it keeps us aligned internally, but I wondered… now that we’re inside Google can we communicate that mission as effectively? So I googled “Firebase mission” and could only find pre-acquisition statements… and then… I noticed on our home page… there it is:

App success made simple The tools and infrastructure you need to build better apps and grow successful businesses”

For an outsider, it looks like marketing fluff, but it’s not. This is our mission. It is not enough to just say this words. We need small things we can do to align the team in building toward this vision. You can’t see it from the outside, but I think it adds up to a difference you can feel. We have these practices, which I believe are key to making this vision a reality.

Our whole team has this relentless focus on the first fifteen minutes of the developer experience. Throughout the development process, in meetings or spontaneous conversations, someone will ask “how does this affect the initial experience?”

One of my colleagues, Joey Yang, told me a bit more detail about this developer experience that is so important to the Firebase team. He described the user experience as the “sum total of every interaction the user has about the product.”

It’s not just our code and documentation, it’s social media and stack overflow. It’s not jut your blog, it’s other people’s blog posts. Someone’s first fifteen minutes might be interacting with another person in the community who is not you! Or it might be through your github repo, conference talks, meetups or hackathons.

“Your product is not just your code — if it takes you more than five minutes to get up and running, then you will spend your time elsewhere” — Joey Yang

I first met Firebase team in May 2012 at a hackathon. I had no idea at the time that I was experiencing a company philosophy! I later learned that they had a policy that “even if you weren’t using Firebase, we would help you out… even if you were using a competing product. Generating good will from the interaction was important to us.”

So my team was at AngelHack back in 2012 making an app that let people sketch 3D shapes in the air. Firebase helped us send paths to the server instantly, to be rendered on a webpage with WebGL. This was just a hobby app, something delightful and fun, a way to stretch our developer skills and our imagination. Firebase treated us with as much reverence and excitement as they did people who were building a business on their product. They invited us to their launch. They showcased our success. Of course, we knew that this was their success as well, but we were center stage.


I started learning Ruby on Rails in December 2008. Rails 2.2 had just been released. I witnessed the adoption of Rack by Rails and by all of the Ruby frameworks. As a developer in the community, it felt like one day, everyone decided that Rack was a good idea, and then everything just got easier. I don’t know the actual history, but I did a little research and was intrigued by what I found.

In February 2007, Chris Neukirchen wrote a blog post introducing rack.

I noticed that there is a lot of code duplication among frameworks since they essentially all do the same things. And still, every Ruby web framework developer is writing his own handlers for every webserver he wants to use.

He observed that HTTP is actually quite simple and expressed it as a simple API:

class HelloWorld
  def call(env)
    [200, {"Content-Type" => "text/plain"}, ["Hello world!"]]

This is actually a very simple Rack application. Chris presented something which could be adopted rather easily, and when adopted by a single web server was not that revolutionary, but with each server and framework that adopted it, it became easier for everyone. It didn’t need to be adopted all at once to save time and effort. It was a good idea. It was really only an idea. He did write some helper libraries, but no one had to use them.

Rack was just an idea, a thought, that simplified the approach to connecting different pieces of web software.

Communication is What We Do

I find it confusing that anyone could even suggest that communication is a different skill that is optional for programmers. This is what we do. We write our code in programming languages. We define things and then they exist. We make real things using language.

When I was a kid, I wanted to be a wizard, with magic where you could know the name of a thing, sketch it in the air and make it real. We do that with software. When we write code, we simply define a thing and run the code, to make it real.

“Wizards channel arcane forces through extensive study, hidden knowledge, and intricate preparation. To a wizard, magic is an art form, an expressive and powerful method by which he or she seeks to control the world around them” —

New Languages

There seems to be an explosion of new languages or old ones that people are suddenly using more often. This seems driven by combination of fast processors and new problems of building distributed systems that require better handling of concurrency, and new user experience patterns with mobile and all sorts of connected devices.

To understand what these new languages are for, it helps me to think of them in context of history and language design trends. This invention is a conversation between the past and the future we’re creating. Software developers exchange ideas through experimentation.

Closing Thoughts

Learn a new language. Make a new language. Code is communication. Figure out what you want to say: make things that communicate your intention. Experiment! Maybe you can make the thing that will give us all new magical powers.


The post communication is a technical skill appeared first on the evolving ultrasaurus.

by sarah at October 05, 2016 03:57 AM

October 03, 2016

Anton Chuvakin - Security Warrior

Monthly Blog Round-Up – September 2016

Here is my next monthly "Security Warrior" blog round-up of top 5 popular posts/topics this month:
  1. Why No Open Source SIEM, EVER?” contains some of my SIEM thinking from 2009. Is it relevant now? You be the judge.  Succeeding with SIEM requires a lot of work, whether you paid for the software, or not. BTW, this post has an amazing “staying power” that is hard to explain – I suspect it has to do with people wanting “free stuff” and googling for “open source SIEM” … 
  2. “New SIEM Whitepaper on Use Cases In-Depth OUT!” (dated 2010) presents a whitepaper on select SIEM use cases described in depth with rules and reports [using now-defunct SIEM product]; also see this SIEM use case in depth and this for a more current list of popular SIEM use cases. Finally, see our 2016 research on security monitoring use cases here!
  3. “SIEM Resourcing or How Much the Friggin’ Thing Would REALLY Cost Me?” is a quick framework for assessing the costs of a SIEM project (well, a program, really) at an organization (much more details on this here in this paper).
  4. My classic PCI DSS Log Review series is always popular! The series of 18 posts cover a comprehensive log review approach (OK for PCI DSS 3+ as well), useful for building log review processes and procedures , whether regulatory or not. It is also described in more detail in our Log Management book and mentioned in our PCI book (now in its 4th edition!)
  5. Simple Log Review Checklist Released!” is often at the top of this list – this aging checklist is still a very useful tool for many people. “On Free Log Management Tools” is a companion to the checklist (updated version)
In addition, I’d like to draw your attention to a few recent posts from my Gartner blog [which, BTW, now has about 5X of the traffic of this blog]: 
Currect research on deception:
Recent research on SOC:
Recent research on threat intelligence:
Miscellaneous fun posts:

(see all my published Gartner research here)
Also see my past monthly and annual “Top Popular Blog Posts” – 2007, 2008, 2009, 2010, 2011, 2012, 2013, 2014, 2015.

Disclaimer: most content at SecurityWarrior blog was written before I joined Gartner on Aug 1, 2011 and is solely my personal view at the time of writing. For my current security blogging, go here.

Previous post in this endless series:

by Anton Chuvakin ( at October 03, 2016 02:51 PM

Colin Percival


FreeBSD 11.0-RELEASE is just around the corner, and it will be bringing a long list of new features and improvements — far too many for me to list here. But as part of the release process, the FreeBSD release engineering team has built images for Amazon EC2, and as semi-official maintainer of that platform (I've never been appointed to this role, but I've been doing it for years and nobody has told me to stop...) I think there are some improvements in FreeBSD 11.0 which are particularly noteworthy for EC2 users.

October 03, 2016 06:00 AM

October 01, 2016

Simon Lyall

DevOpsDays Wellington 2016 – Day 2, Session 3


Mrinal Mukherjee – How to choose a DevOps tool

Right Tool
– Does the job
– People will accept

Wrong tool
– Never ending Poc
– Doesn’t do the job

How to pick
– Budget / Licensing
– does it address your pain points
– Learning cliff
– Community support
– Enterprise acceptability
– Config in version control?

Central tooling team
– Pro standardize, educate, education
– Constant Bottleneck, delays, stifles innovation, not in sync with teams

DevOps != Tool
Tools != DevOps

Tools facilitate it not define it.

Howard Duff – Eric and his blue boxes

Physical example of KanBan in an underwear factory

Lindsey Holmwood – Deepening people to weather the organisation

Note: Lindsey presents really fast so I missed recording a lot from the talk

His Happy, High performing Team -> He left -> 6 months later half of team had left

How do you create a resilient culture?

What is culture?
– Lots of research in organisation psychology
– Edgar Schein – 3 levels of culture
– Artefacts, Values, Assumptions

– Physical manifestations of our culture
– Standups, Org charts, desk layout, documentation
– actual software written
– Easiest to see and adopt

– Goals, strategies and philosophise
– “we will dominate the market”
– “Management if available”
– “nobody is going to be fired for making a mistake”
– lived values vs aspiration values (People have good nose for bullshit)
– Example, cores values of Enron vs reality
– Work as imagined vs Work is actually done

– beliefs, perceptions, thoughts and feelings
– exist on an unconscious level
– hard to discern
– “bad outcomes come from bad people”
– “it is okay to withhold information”
– “we can’t trust that team”
– “profits over people”

If we can change our people, we can change our culture

What makes a good team member?

– Vulnerability
– Assume the best of others
– Aware of their cognitive bias
– Aware of the fundamental attribution error (judge others by actions, judge ourselves by our intentions)
– Aware of hindsight bias. Hindsight bias is your culture killer
– When bad things happen explain in terms of foresight
– Regular 1:1s
Eliminate performance reviews
Willing to play devils advocate

Commit and acting
– Shared goal settings
– Don’t solutioneer
– Provide context about strategy, about desired outcome
What makes a good team?

Influence of hiring process
– Willingness to adapt and adopt working in new team
– Qualify team fit, tech talent then rubber stamp from team lead
– have a consistent script, but be prepared to improvise
– Everyone has the veto power
– Leadership is vetoing at the last minute, thats a systemic problem with team alignment not the system
– Benefit: team talks to candidate (without leadership present)
– Many different perspectives
– unblock management bottlenecks
– Risk: uncovering dysfunctions and misalignment in your teams
– Hire good people, get out of their way

Diversity and inclusion
– includes: race, gender, sexual orientation, location, disability, level of experience, work hours
– Seek out diverse candidates.
– Sponsor events and meetups
– Make job description clear you are looking for diverse background
– Must include and embrace differences once they actually join
– Safe mechanism for people to raise criticisms, and acting on them

Leadership and Absence of leadership
– Having a title isn’t required
– If leader steps aware things should continue working right
– Team is their own shit umbrella
– empowerment vs authority
– empowerment is giving permission from above (potentially temporary)
– authority is giving power (granting autonomy)

Part of something bigger than the team
– help people build up for the next job
– Guilds in the Spotify model
– Run them like meetups
– Get senior management to come and observe
– What we’re talking about is tech culture

We can change tech culture
– How to make it resist the culture of the rest of the organisation
– Artefacts influence behaviour
– Artifact fast builds -> value: make better quality
– Artifact: post incident reviews -> Value: Failure is an opportunity for learning

Q: What is a pre-incident review
A: Brainstorm beforehand (eg before a big rollout) what you think might go wrong if something is coming up
then afterwards do another review of what just went wrong

Q: what replaces performance reviews
A: One on ones

Q: Overcoming Resistance
A: Do it and point back at the evidence. Hard to argue with an artifact

Q: First step?
A: One on 1s

Getting started, reading books by Patrick Lencioni:
– Solos, Politics and turf wars
– 5 Dysfunctions of a team


by simon at October 01, 2016 12:54 AM

September 30, 2016

Simon Lyall

DevOpsDays Wellington 2016 – Day 2, Session 2

Troy Cornwall & Alex Corkin – Health is hard: A Story about making healthcare less hard, and faster!

Maybe title should be “Culture is Hard”

@devtroy @4lexNZ

Working at HealthLink
– Windows running Java stuff
– Out of date and poorly managed
– Deployments manual, thrown over the wall by devs to ops

Team Death Star
– Destroy bad processes
– Change deployment process

Existing Stack
– VMware
– Windows
– Puppet

CD and CI Requirements
– Goal: Time to regression test under 2 mins, time to deploy under 2 mins (from 2 weeks each)
– Puppet too slow to deploy code in a minute or two. App deply vs Conf mngt
– Can’t use (then) containers on Windows so not an option

New Stack
– VMware
– Ubuntu
– Puppet for Server config
– Docker
– rancher

Smashed the 2 minute target!

– We focused on the tech side and let the people side slip
– Windows shop, hard work even to get a Linux VM at the start
– Devs scared to run on Linux. Some initial deploy problems burnt people
– Lots of different new technologies at once all pushed to devs, no pull from them.

Blackout where we weren’t allowed to talk to them for four weeks
– Should have been a warning sign…

We thought we were ready.
– Ops was not ready

“5 dysfunctions of a team”
– Trust as at the bottom, we didn’t have that

– We were aware of this, but didn’t follow though
– We were used to disruption but other teams were not

Note: I’m not sure how the story ended up, they sort of left it hanging.

Pavel Jelinek – Kubernetes in production

Works at Movio
– Software for Cinema chains (eg Loyalty cards)
– 100million emails per month. million of SMS and push notifications (less push cause ppl hate those)

Old Stack
– Started with mysql and php application
– AWS from the beginning
– On largest aws instance but still slow.

Decided to go with Microservices
– Put stuff in Docker
– Used Jenkins, puppet, own docker registery, rundeck (see blog post)
– Devs didn’t like writing puppet code and other manual setup

Decided to go to new container management at start of 2016
– Was pushing for Nomad but devs liked Kubernetes

– Built in ports, HA, LB, Health-checks

Concepts in Kub
– POD – one or more containers
– Deployment, Daemon, Pet Set – Scaling of a POD
– Service- resolvable name, load balancing
– ConfigMap, Volume, Secret – Extended Docker Volume

Devs look after some kub config files
– Brings them closer to how stuff is really working

– Using kubectl to create pod in his work’s lab env
– Add load balancer in front of it
– Add a configmap to update the container’s nginx config
– Make it public
– LB replicas, Rolling updates

Best Practices
– lots of small containers are better
– log on container stdout, preferable via json
– Test and know your resource requirements (at movio devs teams specify, check and adjust)
– Be aware of the node sizes
– Stateless please
– if not stateless than clustered please
– Must handle unexpected immediate restarts


by simon at September 30, 2016 09:43 PM

Racker Hacker

What’s Happening in OpenStack-Ansible (WHOA) – September 2016

OpenStackWelcome to the fourth post in the series of What’s Happening in OpenStack-Ansible (WHOA) posts that I’m assembling each month. OpenStack-Ansible is a flexible framework for deploying enterprise-grade OpenStack clouds. In fact, I use OpenStack-Ansible to deploy the OpenStack cloud underneath the virtual machine that runs this blog!

My goal with these posts is to inform more people about what we’re doing in the OpenStack-Ansible community and bring on more contributors to the project.

There are plenty of updates since the last post from August. The race is on to finish up the Newton release and start new developments for the Ocata release! We hope to see lots of contributors in Barcelona!

Arc de Triomf Barcelona Spain

New releases

The OpenStack-Ansible releases are announced on the OpenStack development mailing list. Here are the things you need to know:


The OpenStack-Ansible Newton release is still being finalized this week. The stable/newton branches were created yesterday and stabilization work is ongoing.


Mitaka is the latest stable release available and the latest version is 13.3.4. This release includes some upgrade improvements for ml2 ports and container hostname adjustments.


The latest Liberty release, 12.2.4, contains lots of updates and fixes. The updates include a fix for picking up where you left off on a failed upgrade and a fix for duplicated log lines. The security role received some updates to improve performance and reduce unnecessary logging.

Notable discussions

This section covers discussions from the OpenStack-Ansible weekly meetings, IRC channels, mailing lists, or in-person events.

Newton branch

As mentioned earlier, the stable/newton branches have arrived for OpenStack-Ansible! This will allow us to finish stabilizing the Newton release and look ahead to Ocata.


Michael Johnson and Jorge Miramontes stopped by our weekly meeting to talk about how Octavia could be implemented in OpenStack-Ansible. Recent Octavia releases have some new features that should be valuable to OpenStack-Ansible deployers.

There is a spec from the Liberty release for deploying Octavia, but we were only able to get LBaaSv2 with the agent deployed. Jorge and Michael are working on a new spec to get Octavia deployed with OpenStack-Ansible.

Testing repo

There’s now a centralized testing repository for all OpenStack-Ansible roles. This allows the developers to share variables, scripts, and test cases between multiple roles. Developers can begin testing new roles with much less effort since the scaffolding for a basic test environment is available in the repository.

You can follow along with the development by watching the central-test-config topic in Gerrit.

Mailing list

The OpenStack-Ansible tag was fairly quiet on the OpenStack Development mailing list during the time frame of this report, but there were a few threads:

Notable developments

This section covers some of the improvements coming to Newton, the upcoming OpenStack release.

OpenStack-Ansible Training

Hastexo logoHastexo is now offering a self-paced course for learning OpenStack-Ansible! It’s called HX201 Cloud Fundamentals for OpenStack-Ansible and it is available now.

Thanks to Florian Haas and Adolfo Brandes for assembling this course!

OpenStack-Ansible powers the OSIC cloud

OpenStack Innovation CenterOne of the clouds operated by the OpenStack Innovation Center (OSIC) is powered by OpenStack-Ansible. It’s a dual-stack (IPv4 and IPv6) environment and it provides the most nodes for the OpenStack CI service! If you need to test an application on a large OpenStack cloud, apply for access to the OSIC cluster.

Inventory improvements

The backbone of OpenStack-Ansible is its inventory. The dynamic inventory defines where each service should be deployed, configured and managed. Some recent improvements include exporting inventory for use by other scripts or applications. Ocata should bring even more improvements to the dynamic inventory.

Thanks to Nolan Brubaker for leading this effort!

Install guide

The installation guide has been completely overhauled! It has a more concise, opinionated approach to deployments and this should make the first deployment a little easier for newcomers. OpenStack can be a complex system to deploy and our goal is to provide the cleanest path to a successful deployment.

Thanks to Alex Settle for leading this effort!


The goal of this newsletter is three fold:

  • Keep OpenStack-Ansible developers updated with new changes
  • Inform operators about new features, fixes, and long-term goals
  • Bring more people into the OpenStack-Ansible community to share their use
    cases, bugs, and code

Please let me know if you spot any errors, areas for improvement, or items that I missed altogether. I’m mhayden on Freenode IRC and you can find me on Twitter anytime.

Photo credit: Mattia Felice Palermo (Own work) CC BY-SA 3.0 es, via Wikimedia Commons

The post What’s Happening in OpenStack-Ansible (WHOA) – September 2016 appeared first on

by Major Hayden at September 30, 2016 02:33 PM

Simon Lyall

DevOpsDays Wellington 2016 – Day 2, Session 1

Jethro Carr – Powering with DevOps goodness
– “News” Website
– 5 person DevOps team

– “Something you do because Gartner said it’s cool”
– Sysadmin -> InfraCoder/SRE -> Dev Shepherd -> Dev
– Stuff in the middle somewhere
– DevSecOps

Company Structure drives DevOps structure
– Lots of products – one team != one product
– Dev teams with very specific focus
– Scale – too big, yet to small

About our team
– Mainly Ops focus
– small number compared to developers
– Operate like an agency model for developers
– “If you buy the Dom Post it would help us grow our team”
– Lots of different vendors with different skill levels and technology

Work process
– Use KanBan with Jira
– Works for Ops focussed team
– Not so great for long running projects

War Against OnCall
– Biggest cause of burnout
– focus on minimising callouts
– Zero alarm target
– Love pagerduty

Commonalities across platforms
– Everyone using compute
– Most Java and javascript
– Using Public Cloud
– Using off the shelf version control, deployment solutions
– Don’t get overly creative and make things too complex
– Proven technology that is well tried and tested and skills available in marketplace
– Classic technologist like Nginx, Java, Varnish still have their place. Don’t always need latest fashion

– Linux, ubuntu
– Adobe AEM Java CMS
– AWS 14x c4.2xlarge
– Varnish in front, used by everybody else. Makes ELB and ALB look like toys

How use Varnish
– Retries against backends if 500 replies, serve old copies
– split routes to various backends
– Control CDN via header
– Dynamic Configuration via puppet

– Akamai
– Keeps online during breaking load
– 90% cache offload
– Management is a bit slow and manual

– Small batch jobs
– Check mail reputation score
– “Download file from a vendor” type stuff
– Purge cache when static file changes
– Lamda webapps – Hopefully soon, a bit immature

Increasing number of microservices

Standards are vital for microservices
– Simple and reasonable
– Shareable vendors and internal
– flexible
– grow organicly
– Needs to be detail
– 12 factor App
– 3 languages Node, Java, Ruby
– Common deps (SQL, varnish, memcache, Redis)
– Build pipeline standardise. Using Codeship
– Standardise server builds
– Everything Automated with puppet
– Puppet building docker containers (w puppet + puppetstry)
– Std Application deployment

Init systems
– Had proliferation
– pm2, god, supervisord, systemvinit are out
– systemd and upstart are in

Always exceptions
– “Enterprise ___” is always bad
– Educating the business is a forever job
– Be reasonable, set boundaries

More Stuff at

Q: Pull request workflow
A: Largely replaced traditional review

Q: DR eg AWS outage
A: Documented process if codeship dies can manually push, Rest in 2*AZs, Snapshots

Q: Dev teams structure
A: Project specific rather than product specific.

Q: Puppet code tested?
A: Not really, Kinda tested via the pre-prod environment, Would prefer result (server spec) testing rather than low level testing of each line
A: Code team have good test coverage though. 80-90% in many cases.

Q: Load testing, APM
A: Use New Relic. Not much luck with external load testing companies

Q: What is somebody wants something non-standard?
A: Case-by-case. Allowed if needed but needs a good reason.

Q: What happens when automation breaks?
A: Documentation is actually pretty good.


by simon at September 30, 2016 09:42 AM

September 25, 2016

Firefox History stats with Bash

This is a small script to gather some statistics from your Firefox history. First we use sqlite3 to parse the Firefox history database and get the last three months, then we remove all the IP addresses and port numbers and finally we sort and count it.

September 25, 2016 12:00 AM

September 23, 2016

Everything Sysadmin

Why Google should buy Twitter

[Disclaimer: I do not work for Google or Twitter; I have no investments in Google or Twitter. ]

Update: 2016-09-25: Someone pointed out that a better title considering what I'm saying is, "Google should save Twitter as an act of charity".

Google should buy Twitter. (link to rumors here)

Twitter isn't a good "MBA runs the numbers" acquisition. However could be used as a showcase for GCE. It would more than justify itself. In fact, the financial losses might be off-set by the marketing value it provides to GCE.

As part of integrating it into the internal Google stack, they should require their engineers to rebuild it on the Google Cloud Engine platform. GCE scales crazy-good. Twitter has a history of scaling problems. If Google could run it on the Google Cloud Engine, and show that it scales, it would be great advertising.

Google needs GCE to succeed (but that's for another blog post... or you can read Appendix B of .. especially the last few paragraphs.)

How difficult would it be to rebuild Twitter on GCE? I think it would be easier than you'd imagine. Every talk I've seen by Twitter engineers at conferences is about technology that (and I don't mean this with any disrespect) is reproducing something in Google's stack. Most of those technologies being re-invented are available in GCE now, and the rest really should be. In fact, if they aren't available in GCE they should be. The project of porting Twitter to GCE would generate a list of high-quality feature requests. Interestingly enough, the re-invented technologies don't seem to be as scalable as Google's original. Oh, and it seems like a lot of people re-implementing those technologies at Twitter are ex-Google employee so ... you have that.

Sadly the few Google executives that I know think that Twitter is a joke, nobody uses it, and isn't worth saving. I disagree. I think it is "the world's chatroom". If you think Twitter "doesn't matter" then why does every news program, TV show, and billboard list a Twitter handle? (Hint: they don't list G+ handles... does G+ even have handles?)

So, in summary:

  • Google should buy Twitter.
  • It would help save this important resource that the world finds very useful.
  • It would be the best showcase of GCE evah.... which is something Google needs more than the revenue of Twitter.
  • Sadly Google executives dis Twitter as a niche application that a very small number of people find compelling. (Spoiler alert: I think they're wrong)

I wonder what will happen.


NOTE: This article was written by Thomas Limoncelli and included no involvement by current or past co-authors or friends.

by Tom Limoncelli at September 23, 2016 07:14 PM

September 22, 2016

Racker Hacker

Power 8 to the people

IBM Edge 2016 is almost over and I’ve learned a lot about Power 8 this week. I’ve talked about some of the learnings in my recaps of days one and two. The performance arguments sound really interesting and some of the choices in AIX’s design seem to make a lot of sense.

However, there’s one remaining barrier for me: Power 8 isn’t really accessible for a tinkerer.

Power 8 E870-2


Google defines tinkering as:

attempt to repair or improve something in a casual or desultory way,
often to no useful effect.
“he spent hours tinkering with the car”

When I come across a new piece of technology, I really enjoy learning how it works. I like to find its strengths and its limitations. I use that information to figure out how I might use the technology later and when I would recommend the technology for someone else to use it.

To me, tinkering is simply messing around with something until I have a better understanding of how it works. Tinkering doesn’t have a finish line. Tinkering may not have a well-defined goal. However, it’s tinkering that leads to a more robust community around a particular technology.

For example, take a look at the Raspberry Pi. There were plenty of other ARM systems on the market before the Pi and there are still a lot of them now. What makes the Pi different is that it’s highly accessible. You can get the newest model for $35 and there are tons of guides for running various operating systems on it. There are even more guides for how to integrate it with other items, such as sprinkler systems, webcams, door locks, and automobiles.

Another example is the Intel NUC. Although the NUC isn’t the most cost-effective way to get an Intel chip on your desk, it’s powerful enough to be a small portable server that you can take with you. This opens up the door for software developers to test code wherever they are (we use them for OpenStack development), run demos at a customer location, or make multi-node clusters that fit in a laptop bag.

What makes Power 8 inaccessible to tinkerers?

One of the first aspects that most people notice is the cost. The S821LC currently starts at around $6,000 on IBM’s site, which is a bit steep for someone who wants to learn a platform.

I’m not saying this server should cost less — the pricing seems quite reasonable when you consider that it comes with dual 8-core Power 8 processors in a 1U form factor. It also has plenty of high speed interconnects ready for GPUs and CAPI chips. With all of that considered, $6,000 for a server like this sounds very reasonable.

There are other considerations as well. A stripped down S821LC with two 8-core CPUs will consume about 406 Watts at 50% utilization. That’s a fair amount of power draw for a tinkerer and I’d definitely think twice about running something like that at home. When you consider the cooling that’s required, it’s even more difficult to justify.

What about AIX?

AIX provides some nice benefits on Power 8 systems, but it’s difficult to access as well. Put “learning AIX” into a Google search and look at the results. The first link is a thread on where the original poster is given a few options:

  • Buy some IBM hardware
  • Get in some legal/EULA gray areas with VMware
  • Find an old Power 5/6 server that is coming offline at a business that is doing a refresh

Having access to AIX is definitely useful for tinkering, but it could be very useful for software developers. For example, if I write a script in Python and I want to add AIX support, I’ll need access to a system running AIX. It wouldn’t necessarily need to be a system with tons of performance, but it would need the functionality of a basic AIX environment.

Potential solutions

I’d suggest two solutions:

  1. Get AIX into an accessible format, perhaps on a public cloud
  2. Make a more tinker-friendly Power 8 hardware platform

Let’s start with AIX. I’d gladly work with AIX in a public cloud environment where I pay some amount for the virtual machine itself plus additional licensing for AIX. It would still be valuable even if the version of AIX had limiters so that it couldn’t be used for production workloads. I would be able to access the full functionality of a running AIX environment.

The hardware side leads to challenges. However, if it’s possible to do a single Power 8 SMT2 CPU in a smaller form factor, this could become possible. Perhaps these could even be CPUs with some type of defect where one or more cores are disabled. That could reduce cost while still providing the full functionality to someone who wants to tinker with Power 8.

Some might argue that this defeats the point of Power 8 since it’s a high performance, purpose-built chip that crunches through some of the world’s biggest workloads. That’s a totally valid argument.

However, that’s not the point.

The point is to get a fully-functional Power 8 CPU — even if it has serious performance limitations — into the hands of developers who want to do amazing things with it. My hope would be that these small tests will later turn into new ways to utilize POWER systems.

It could also be a way for more system administrators and developers to get experience with AIX. Companies would be able to find more people with a base level of AIX knowledge as well.

Final thoughts

IBM has something truly unique with Power 8. The raw performance of the chip itself is great and the door is open for even more performance through NVlink and CAPI accelerators. These features are game changers for businesses that are struggling to keep up with customer demands. A wider audience could learn about this game-changing technology if it becomes more accessible for tinkering.

Photo credit: Wikipedia

The post Power 8 to the people appeared first on

by Major Hayden at September 22, 2016 12:00 AM

September 19, 2016


The origins of on-call work

On September 6th, Susan Fowler posted an article titled, "Who's on-call?", talking about evolving on-call duties between development teams and SRE teams. She has this quote at the top:

I'm not sure when in the history of software engineering separate operations organizations were built and run to take on the so-called "operational" duties associated with running software applications and systems, but they've been around for quite some time now (by my research, at least the past twenty years - and that's a long time in the software world).

My first job was with a city government, and many of the people I was working with started at that city when they decided to computerize in 1978. Most of them have retired or died off by now. In 1996, when I started there, the original dot-com boom was very much on the upswing, and that city was still doing things the way they'd been done for years.

I got into the market in time to see the tail end of that era. One of the things I learned there was the origins of many of the patterns we see today. To understand the origins of on-call in IT systems, you have to go back to the era of serial networking, when the term 'minicomputer' was distinct from 'microcomputer', which were marketing terms to differentiate from 'mainframe'.

IT systems of the era employed people to do things we wouldn't even consider today, or would work our damnedest to automate out of existence. There were people who had, as their main job, duties such as:

  • Entering data into the computer from paper forms.
    • Really. All you did all day was punch in codes. Computer terminals were not on every desk, so specialists were hired to do it.
    • The worst part is: there are people still doing this today.
  • Kick off backups.
  • Change backup tapes when the computer told them to.
  • Load data-tapes when the computer told them to.
    • Tape stored more than spinning rust, so it was used as a primary storage medium. Disk was for temp-space.
    • I spent a summer being a Tape Librarian. My job was roboticized away.
  • Kick off the overnight print-runs.
  • Colate printer output into reports, for delivery to the mailroom.
  • Execute the overnight batch processes.
    • Your crontab was named 'Stephen,' and you saw him once a quarter at the office parties. Usually very tired-looking.
  • Monitor by hand system usage indicators, and log them in a paper logbook.
  • Keep an Operations Log of events that happened overnight, for review by the Systems Programmers in the morning.
  • Follow runbooks given to them by Systems Programming for performing updates overnight.
  • Be familiar with emergency procedures, and follow them when required.

Many of these things were only done by people working third shift. Which meant computer-rooms had a human on-staff 24/7. Sometimes many of them.

There was a side-effect to all of this, though. What if the overnight Operator had an emergency they couldn't handle? They had to call a Systems Programmer to advise a fix, or come in to fix it. In the 80's, when telephone modem came into their own, they may even be able to dial in and fix it from home.

On-Call was born.

There was another side-effect to all of this: it happened before the great CompSci shift in the colleges, so most Operators were women. And many Systems Programmers were too. This was why my first job was mostly women in IT management and senior technical roles. This was awesome.

A Systems Programmer, as they were called at the time, is less of a Software Engineering role as we would define it today. They were more DevOps, if not outright SysAdmin. They had coding chops, because much of systems management at the time required that. Their goal was more wiring together purchased software packages to work coherently, or modifying purchased software to work appropriately.

Time passed, and more and more of the overnight Operator's job was automated away. Eventually, the need for an overnight Operator exceeded requirements. Or you simply couldn't hire one to replace the Operator that just quit. However, the systems were still running 24/7, and you needed someone ready to respond to disasters. On-call got more intense, since you no longer had an experienced hand in the room at all times.

The Systems Programmers earned new job-titles. Software Engineering started to be a distinct skill-path and career, so was firewalled off in a department called Development. In those days, Development and Systems people spoke often; something you'll hear old hands grumble about with DevOps not being anything actually new. Systems was on-call, and sometimes Development was if there was a big thing rolling out.

Time passed again. Management culture changed, realizing that development people needed to be treated and managed differently than systems people. Software Engineering became known as Software Engineering, and became its own career-track. The new kids getting into the game never knew the close coordination with Systems that the old hands had, and assumed this separation was the way it's always been. Systems became known as Operations; to some chagrin of the old Systems hands who resented being called an 'Operator', which was typically very junior. Operations remained on-call, and kept informal lists of developers who could be relied on to answer the phone at o-dark-thirty in case things went deeply wrong.

More time, and the separation between Operations and Software Engineering became deeply entrenched. Some bright sparks realized that there were an awful lot of synergies to be had with close coordination between Ops and SE. And thus, DevOps was (re)born in the modern context.

Operations was still on-call, but now it was open for debate about how much of Software Engineering needed to be put on the Wake At 3AM In Case Of Emergency list.

And that is how on-call evolved from the minicomputer era, to the modern era of cloud computing.

You're welcome.

by SysAdmin1138 at September 19, 2016 05:48 PM

September 17, 2016

Toolsmith In-depth Analysis: motionEyeOS for Security Makers

It's rather hard to believe, unimaginable even, but here we are. This is the 120th consecutive edition of toolsmith; every month for the last ten years, I've been proud to bring you insights and analysis on free and open source security tools. I hope you've enjoyed the journey as much as I have, I've learned a ton and certainly hope you have too. If you want a journey through the past, October 2006 through August 2015 are available on my web site here, in PDF form, and many year's worth have been published here on the blog as well.
I labored a bit on what to write about for this 10th Anniversary Edition and settled on something I have yet to cover, a physical security topic. To that end I opted for a very slick, maker project, using a Raspberry Pi 2, a USB web cam, and motionEyeOS. Per Calin Crisan, the project developer, motionEyeOS is a Linux distribution that turns a single-board computer into a video surveillance system. The OS is based on BuildRoot and uses motion as a backend and motionEye for the frontend.
  • Buildroot "is a simple, efficient and easy-to-use tool to generate embedded Linux systems through cross-compilation."
  • Motion (wait for it) is a program that monitors the video signal from cameras and is able to detect if a significant part of the picture has changed; in other words, it can detect motion.
  • motionEye is also Calin's project and is web frontend for the motion daemon.

Installation was insanely easy, I followed Calin's installation guidelines and used Win32DiskImager to write the image to the SD card. Here's how straightforward it was in summary.
1) Download the latest motionEyeOS image. I used build 20160828 for Raspberry Pi 2.
2) Write the image to SD card, insert the SD into your Pi.
3) Plug a supported web camera in to your Pi, power up the Pi. Give it a couple minutes after first boot per the guidelines: do not disconnect or reboot your board during these first two minutes. The initialization steps:
  • prepare the data partition on the SD card
  • configure SSH remote access
  • auto-configure any detected camera devices
4) Determine the IP addressed assigned to the Pi, DHCP is default. You can do this with a monitor plugged in the the Pi's HDMI port, via your router's connected devices list, or with a network scan.
For detailed installation instructions, refer to PiMyLifeUp's Build a Raspberry Pi Security Camera Network. It refers to a dated, differently named (motionPie) version of motionEyeOS, but provides great detail if you need it. There are a number of YouTube videos too, just search motionEyeOS.

Configuration is also ridiculously simple. Point your browser to the IP address for the Pi, for me on my wired network, and once I configured motionEyeOS to use my WiFi dongle.
The first time you login, the password is blank so change that first. In the upper left corner of the UI you'll see a round icon with three lines, that's the setting menu. Click it, change your admin and user (viewer) passwords STAT. Then immediately enable Advanced Settings.
Figure 1: Preferences

You'll definitely want to add a camera, and keep in mind, you can manage multiple cameras with on motionEyeOS devices, and even multiple motionEyeOS systems with one master controller. Check out Usage Scenarios for more.
Figure 2: Add a camera

Once your camera is enabled, you'll see its feed in the UI. Note that there are unique URLs for snapshots, streaming and embedding.

Figure 3: Active camera and URLs
When motion detection has enabled the camera, the video frame in the UI will be wrapped in orange-red. You can also hover over the video frame for additional controls such as full screen and immediate access to stored video.

There are an absolute plethora of settings options, the most important of which, after camera configuration, is storage. You can write to local storage or a network share, this quickly matters if you choose and always-on scenario versus motion enabled.
Figure 4: Configure file storage
You can configure text overlay, video streaming, still images, schedules, and more.
Figure 5: Options, options, options
The most important variable of all us how you want to be notified. 
There are configuration options that allow you to run commands so you script up a preferred process or use one already devised.
Figure 6: Run a command for notification

Best of all, you can make uses of a variety of notification services including email, as well as Pushover, and IFTTT via Web Hooks.
Figure 7: Web Hook notifications
There is an outstanding article on using Pushover and IFTTT on Pi Supply's Maker Zone. It makes it easy to leverage such services even if you haven't done so before.
The net result, after easy installation, and a little bit of configuration is your on motion-enabled CCTV system that costs very little compared to its commercial counterparts.
Figure 8: Your author entering his office under the watchful eye of Camera1
Purists will find image quality a bit lacking perhaps, but with the right camera you can use Fast Network Camera. Do be aware of the drawbacks though (lost functionality).

In closing, I love this project. Kudos to Calin Crisan for this project. Makers and absolute beginners alike can easily create a great motion enabled video/still camera setup, or a network of managed cameras with always on video. The hardware is inexpensive and readily available. If you've not explored Raspberry Pi this is a great way to get started. If you're looking for a totally viable security video monitoring implementation, motionEyeOS and your favorite IoT hardware (the project supports other boards too) are a perfect combo. Remember too that there are Raspberry Pi board-specific camera modules available.

Ping me via email or Twitter if you have questions (russ at holisticinfosec dot org or @holisticinfosec).
Cheers…until next time.

by Russ McRee ( at September 17, 2016 02:10 AM

September 12, 2016

Steve Kemp's Blog

If your code accepts URIs as input..

There are many online sites that accept reading input from remote locations. For example a site might try to extract all the text from a webpage, or show you the HTTP-headers a given server sends back in response to a request.

If you run such a site you must make sure you validate the schema you're given - also remembering to do that if you're sent any HTTP-redirects.

Really the issue here is a confusion between URL & URI.

The only time I ever communicated with Aaron Swartz was unfortunately after his death, because I didn't make the connection. I randomly stumbled upon the html2text software he put together, which had an online demo containing a form for entering a location. I tried the obvious input:


The software was vulnerable, read the file, and showed it to me.

The site gives errors on all inputs now, so it cannot be used to demonstrate the problem, but on Friday I saw another site on Hacker News with the very same input-issue, and it reminded me that there's a very real class of security problems here.

The site in question was and allows you to enter a URL to convert to markdown - I found this via the hacker news submission.

The following link shows the contents of /etc/hosts, and demonstrates the problem:

The output looked like this:

.. localhost broadcasthost
::1 localhost
fe80::1%lo0 localhost stage files brettt..

In the actual output of '/etc/passwd' all newlines had been stripped. (Which I now recognize as being an artifact of the markdown processing.)

UPDATE: The problem is fixed now.

September 12, 2016 12:25 PM

September 10, 2016

Best toolsmith tool of the last ten years

As we celebrate Ten Years of Toolsmith and 120 individual tools covered in detail with the attention they deserve, I thought it'd be revealing to see who comes to the very top of the list for readers/voters.
I've built a poll from the last eight Toolsmith Tools of the Year to help you decide, and it's a hell of a list.
 Amazing, right? The best of the best.

You can vote in the poll to your right, it'll be open for two weeks.

by Russ McRee ( at September 10, 2016 07:10 PM

September 09, 2016

Sarah Allen

five dysfunctions of a team

The Five Dysfunctions of a Team by Patrick Lencioni tells the story of a fictional team leading a believable tech company. It illustrates how we bring our humanity and our flaws to work, even as good leaders who contribute exceptionally well, we can still fail if we can’t collaborate effectively on a shared goal.

The storytelling was particularly good at showing how it can be so hard to facilitate change. Everyone won’t be in the same place in the same moment. Someone gets it, while others are struggling. The best teams have a culture of helping each other, and we need to create that culture intentionally.

I tend to prefer non-fiction that conveys research when I read books about work, but decided to read it after it was recommended twice in one week (thx Luna Comerford and Adria Richards). I found this corporate fable to be compelling and thought-provoking. The audio book was particularly entertaining and was surprised to finish it in just a day mostly during dog walking, muni rides, and evening errands.

I enjoyed the book and agree with the five dysfunctions presented as a pyramid that illustrates how each weakness leads to the next. In thinking through my reflections on the book, I found it helpful to review Abi Noda’s book notes. I liked his re-creation of the pyramid of dysfunctions, along with annotations, which inspired me to create my own alternate annotated pyramid:

Absence of Trust

At the top of the pyramid, the book talks about “inattention to results,” though the discussion is really about focusing on the right results, which are based on shared priorities and shared decisions. Individuals on the executive team might inadvertently jeopardize the company’s success because of their own status and ego. No one wants the company to fail because of their own functional area, and thus, well-meaning people can become defensive and fail to deliver on shared goals because they are too focused on their specific deliverable. Certainly we all need to deliver great results within our area of expertise, but that must be in support of a primary shared goal.

At the base of the pyramid is absence of trust. The book had an interesting anecdote on how “politics” is common on teams when people don’t trust each other. People use that word to describe different behaviors, but the CEOs definition works well in this setting: “Politics is when people choose their words and actions based on how they want others to react rather than based on what they really think.” When we work in an environment where we can just say what we think, trusting that people will believe we have good intentions and are working toward our shared goals, makes everything go faster and leads to better solutions.

Psychological Safety and Vulnerability, Healthy Conflict and Speaking Up, Shared Decisions with Buy In, Accountability supported by collaboration, Alignment of key results, shared goal and a shared understanding of reality

I felt like I wanted to remember the functioning team and the positive state we’re striving for when we work well together. As the story progressed, I imagined the opposite pyramid. At its base is psychological safety, which allows people to be vulnerable; this creates an environment where people can speak up, allowing for and supporting healthy conflict. If people can have productive discussions and have respectful arguments, the team can make decisions that the each individual buys into. People hold each other accountable, but also support each other and collaborate, consistently seeking alignment on delivering on shared priorities. Working as a team requires repeated course corrections. We need to make time to help each other and if we’re all focused on different things, we just won’t have time to do it all. If we’re all focused on the same goal, supporting parts of the same deliverable, then collaboration helps each team member deliver on their individual work and the shared goal.

One key element to making good teams work well is what I call a shared understanding of reality. In the book, the new CEO starts each day of their executive offsite by saying: “We have more money, better technology, more talented and experienced executives, and yet we’re behind our competitors.” She was creating a shared understanding of reality. She was making it clear to the team that she believed in them and in the business opportunity. A lot of what we do when we create alignment on a team is to build that shared reality. Shared goals are built on shared information and shared conclusions, which are based on shared values.

There are lots of ways to think about how to create a healthy team. I like how Abi Noda reflects on the pyramid of dysfunctions presented in this book: “Teamwork deteriorates if even a single dysfunction is allowed to flourish, like a chain with just one link broken.”

The post five dysfunctions of a team appeared first on the evolving ultrasaurus.

by sarah at September 09, 2016 01:10 PM

Sean's IT Blog

Horizon 7.0 Part 10–Building Your Desktop Golden Images

A virtual desktop environment is nothing without virtual desktops.  Poorly performing virtual desktops and applications, or virtual desktops and remote desktop session hosts that aren’t configured properly for the applications that are being deployed, can turn users off to modern end user computing solutions and sink the project.

How you configure your desktop base image can depend on the type of desktop pools that you plan to deploy.  The type of desktop pools that you deploy can depend on the applications and how you intend to deploy them.  This part will cover how to configure a desktop base image for non-persistent desktop pools, and the next part in this series will cover how to set up both linked and instant clone desktop pools.

Before You Begin, Understand Your Applications

Before we begin talking about how to configure the desktop base image and setting up the desktop pools, its very important to understand the applications that you will be deploying to your virtual desktops.  The types of applications and how they can be deployed will determine the types of desktop pools that can be used.

A few factors to keep in mind are:

  • Application Delivery – How are the applications going to be delivered to the desktop or RDSH host?
  • User Installed Applications – Will users be able to install their own applications?  If so, how are applications managed on the desktop?
  • User Profiles – How are the user profiles and settings being managed?  Is there any application data or setting that you want to manage or make portable across platforms?
  • Licensing – How are the applications licensed?  Are the licenses locked to the computer in some way, such as by computer name or MAC address?  Is a hardware key required?
  • Hardware – Does the application require specific hardware in order to function, or does it have high resource requirements?  This is usually a consideration for high-end CAD or engineering applications that require a 3D card, but it could also apply to applications that need older hardware or access to a serial port.

Application management and delivery has changed significantly since I wrote the Horizon 6.0 series.  When that series was written, VMware had just purchased Cloud Volumes, and it hadn’t been added into the product suite.  Today, App Volumes is available in the Horizon Suite Enterprise SKU, and it provides application layering capabilities in Horizon.  Application layering allows administrators to place applications into virtual disk files that get attached at logon, and this allows you to create a single master golden image that has applications added when the user logs in.  If you don’t have the Horizon Suite Enterprise SKU, there are a few other players in the application layering space such as Liquidware Labs FlexApp and Unidesk, and these tools also provide the ability to abstract your applications from the underlying operating system.

Application layering isn’t the only delivery mechanism.  App Virtualization, using tools like ThinApp, Microsoft AppV, or Turbo, is one option for providing isolated applications.  Reverse layering has all applications installed into the golden template, and applications are exposed on a per-user basis. This is the concept behind tools like FSLogix.  Publishing applications to virtual desktops using XenApp or Horizon Published Applications is an option that places the applications on a centralized server, or you could just install some or all of your applications into the golden image and manage them with tools like Altiris or SCCM.

All of these options are valid ways to deliver applications to virtual desktops, and you need to decide on which methods you will use when designing your desktop golden images and desktop pools.  There may not be a single solution for delivering all of your applications, and you may need to rely on multiple methods to meet the needs of your users.

Supported Desktop Operating Systems

Horizon 7.0 supports desktops running Windows and Linux.  The versions of Windows that are supported for full clone and linked clone desktops are:

  • Windows 10 Enterprise (including the Long Term Servicing Branch and Anniversary Update in Horizon 7.0.2)
  • Windows 8.1 Enterprise or Professional
  • Windows 8 Enterprise or Professional
  • Windows 7 SP1 Enterprise or Professional
  • Windows Server 2008 R2 (RDSH and Server-based Desktop)
  • Windows Server 2012 R2 (RDSH and Server-based Desktop)

In order to run desktops on Windows Server-based OSes, you need to enable the “Enable Windows Server desktops” setting under View Configuration –> Global Settings and install the Desktop Experience feature after installing the OS.  There are some benefits to using Windows Server for your desktop OS including avoiding the Microsoft VDA tax on desktop VDI.  The choice to use a server OS vs. a desktop OS must be weighed carefully, however, as this can impact management and application compatibility.

Instant clone desktops are supported on the following operating systems:

  • Windows 10 Enterprise
  • Windows 7 SP1 Enterprise or Professional

The Horizon Linux agent is supported on the following 64-bit versions:

  • Ubuntu 14.04 (note: VMware recommends disabling Compviz due to performance issues)
  • Ubuntu 12.04
  • RHEL and CentOS 6.6
  • RHEL and CentOS 7.2
  • NeoKylin 6 Update 1
  • SLES 11 SP3/SP4
  • SLES 12 SP1

The Linux component supports both full clone and linked clone desktops in Horizon 7.0.1.  However, there are a number of additional requirements for Linux desktops, so I would recommend reading the Setting Up Horizon 7 Version 7.0.1 for Linux Desktops guide.

For this part, we’re going to assume that we’re building a template running a desktop version of Windows.  This will be more of a high-level overview of creating a desktop template for Horizon, and I won’t be doing a step-by-step walkthrough of any of the steps for this section.  Once the desktop image is set up, I’ll cover some of the ways to optimize the desktop templates.

Configure the VM

Building a desktop VM isn’t much different than building a server VM.  The basic process is create the VM, configure the hardware, install the operating system, and then install your applications.  Although there are a few additional steps, building a desktop VM doesn’t deviate from this.

You should base the number of vCPUs and the amount of RAM assigned to your virtual desktops on the requirements for of the applications that you plan to run and fine tune based on user performance and resource utilization.   Horizon doesn’t allow you to set the CPU and RAM allocation when deploying desktop pools, so these need to be set on the template itself.

The recommended hardware for a virtual desktop is:

  • SCSI Controller – LSI SAS
  • Hard Disk – At least 40GB Thin Provisioned
  • Remove Floppy Drive, and disable parallel and serial ports in BIOS
  • Remove the CD-ROM drive if you do not have an alternative method for installing Windows.

Note: You cannot remove the CD-ROM drive until after Windows has been installed if you are installing from an ISO.

BIOS Settings
BIOS screen for disabling Serial and Parallel ports and floppy controller

You’ll notice that I didn’t put minimums for vCPUs and RAM.  Sizing these really depends on the requirements of your user’s applications.  I’ve had Windows 7 64-bit desktops deployed with as little as 1GB of RAM for general office workers up to 4GB of RAM for users running the Adobe Suite.  Generally speaking, customers are deploying knowledge or task worker desktops with at least 2 vCPUs and between 2-4 GB of RAM, however the actual sizing depends on your applications.

Install Windows

After you have created a VM and configured the VM’s settings, you need to install Windows.  Again, it’s not much different than installing Windows Server into a VM or installing a fresh copy of Windows onto physical hardware.  You can install Windows using the ISO of the disk or by using the Microsoft Deployment Toolkit and PXE boot to push down an image that you’ve already created.

When installing Windows for your desktop template, you’ll want to make sure that the default 100 MB system partition is not created.  This partition is used by Windows to store the files used for Bitlocker.  Since Bitlocker is not supported on virtual machines by either Microsoft or VMware, there is no reason to create this partition.  This will require bypassing the installer and manually partitioning the boot drive.  The steps for doing this when installing from the DVD/ISO are:

1. Boot the computer to the installer
2. Press Shift-F10 to bring up the command prompt
3. Type DiskPart
4. Type Select Disk 0
5. Type Create Partition Primary
6. Type Exit twice.


Once you’ve set up the partition, you can install Windows normally.  If you’re using something like the Microsoft Deployment Toolkit, you will need to configure your answer file to set up the proper hard drive partition configuration.

Install VMware Tools and Join the Template to a Domain

After you have installed Windows, you will need to install the VMware tools package.  The tools package is required to install the View Agent.  VMware Tools also includes the VMXNET3 driver, and your template will not have network access until this is installed.   The typical installation is generally all that you will need unless you’re using Guest Introspection as part of  NSX or your antivirus solution.

After you have installed VMware Tools and rebooted the template, you should join it to your Active Directory domain.  The template doesn’t need to be joined to a domain, but it makes it easier to manage and install software from network shares.  I’ve also heard that there are some best practices around removing the computer from the domain before deploying desktop pools.  This is an optional task, and it’s not required.  I’ve never removed the machines from the domain before provisioning, and I haven’t experienced any issues.

Install The Horizon Agent

After you have installed the VMware tools package and joined your computer to the domain, you will need to install the VMware Horizon Agent.  There are a number of new features in the Horizon 7 Agent install, and not all features are enabled by default.  Be careful when enabling or disabling features as this can have security implications.

One thing to note about the Horizon 7 agent is that there is a Composer component and an Instant Clones component.  These items cannot be installed together.  A desktop template can only be used for Linked Clones or Instant Clones.

Installing Applications on the Template

After you install the Horizon Agent, you can begin to install the applications that your users will need when they log into Horizon View.

With tools like Thinapp available to virtualize Windows applications or layering software like FlexApp, Unidesk and App Volumes, it is not necessary to install all of your applications in your template or to create templates for all of the different application combinations.  You can create a base template with your common applications that all users receive and then either virtualize or layer your other applications so they can be delivered on demand.

“Finalizing” the Image

Once you have the applications installed, it is time to finalize the image to prepare it for Horizon.  This step involves disabling unneeded services and making configuration settings changes to ensure a good user experience.   This may also involve running antivirus or other malware scans to ensure that only new or changed files are scanned after the image is deployed (Symantec…I’m looking at you for this one).

VMware has released a white paper that covers how to optimize a Windows-based virtual desktop or RDSH server.  Previous versions of this white paper have focused on making changes using a PowerShell or batch script.   VMware has also released a fling, the OS Optimization Tool, with a graphical interface that can simplify the optimization process.  Whenever possible, I recommend using the fling to optimize virtual desktop templates.  It not only provides an easy way to select which settings to apply, but it contains templates for different operating systems.  It also provides a way to log which changes are made and to roll back unwanted changes.

Prior to optimizing your desktops, I recommend taking a snapshot of the virtual machine.  This provides a quick way to roll back to a clean slate.  I recommend applying most of the defaults, but I also recommend reading through each change to understand what changes are being made.  I do not recommend disabling the Windows Firewall at all, and I don’t recommend disabling Windows Update as this can be controlled by Group Policy.

Before you shut the virtual machine down to snapshot it, verify that any services required for applications are enabled.  This includes the Windows Firewall service which is required for the Horizon Agent to function properly.

Shutdown and Snapshot

After you have your applications installed, you need to shut down your desktop template and take a snapshot of it.  If you are using linked clones, the linked clone replica will be based on the snapshot you select.

That’s a quick rundown of setting up a desktop template to be used with Horizon desktops.

In the next part of this series, I’ll cover how to create desktop pools.

by seanpmassey at September 09, 2016 02:33 AM

September 07, 2016

Sean's IT Blog

Horizon 7.0 Part 9–Configuring Horizon for the First Time

Now that the Connection Server and Composer are installed, it’s time to configure the components to actually work together with vCenter to provision and manage desktop pools.

Logging into the Horizon Administrator

Before anything can be configured, though, we need to first log into the Horizon Administrator management interface.  This management interface is based on the Adobe Flex platform, so Flash will need to be installed on any endpoints you use to administer the environment.

The web browsers that VMware currently supports, with Adobe Flash 10.1 or later are:

  • Internet Explorer 9-11
  • Firefox
  • Chrome
  • Safari 6
  • Microsoft Edge

To log in, take the following steps:

1. Open your web browser.

2. Navigate to https://<FQDN of connection server>/admin

3. Log in with the Administrator Account you designated (or with an account that is a member of the administrator group you selected) when you installed the Connection Server.

1. Login

4. After you log in, you will be prompted for a license key.

2. License pt 1

Note:  The license keys are retrieved from your MyVMware site.  If you do not input a license key, you will not be able to connect to desktops or published applications after they are provisioned.  You can add or change a license key later under View Configuration –> Product Licensing and Usage.

5. Click Edit License.  Paste your license key from the MyVMware site into the license key box and click OK.

3. License pt 2

6. After your license key is installed, the Licensing area will show when your license expires and the features that are licensed in your deployment.

4. License pt 3

Configuring Horizon for the First Time

Once you’ve logged in and configured your license, you can start setting up the Horizon environment.  In this step, the Connection Server will be configured to talk to vCenter and Composer.

1.   Expand View Configuration and select Servers.


2.  Select the vCenter Servers tab and select Add…


3, Enter your vCenter server information.  The service account that you use in this section should be the vCenter Service Account that you created in Part 6.

Note: If you are using vCenter 5.5 or later, the username should be entered in User Principal Name format – username@fqdn.


4. If you have not updated the certificates on your vCenter Server, you will receive an Invalid Certificate Warning.  Click View Certificate to view and accept the certificate.


5.  Select the View Composer option that you plan to use with this vCenter.  The options are:

A. Do not use View Composer – View Composer and Linked Clones will not be available for desktop pools that use this vCenter.

B. View Composer is co-installed with vCenter Server – View Composer is installed on the vCenter Server, and the vCenter Server credentials entered on the previous screen will be used for connecting.  This option is only available with the Windows vCenter Server.

C. Standalone View Composer Server – View Composer is installed on a standalone Windows Server, and credentials will be required to connect to the Composer instance.  This option will work with both the Windows vCenter Server and the vCenter Server virtual appliance.

Note: The account credentials used to connect to the View Composer server must have local administrator rights on the machine where Composer is installed.  If they account does not have local administrator rights, you will get an error that you cannot connect.


6. If Composer is using an untrusted SSL certificate, you will receive a prompt that the certificate is invalid.  Click View Certificate and then accept.

For more information on installing a trusted certificate on your Composer server, please see Part 5.


7. The next step is to set up the Active Directory domains that Composer will connect to when provisioning desktops.  Click Add to add a new domain.


8. Enter the domain name, user account with rights to Active Directory, and the password and click OK.  The user account used for this step should be the account that was set up in Part 6.

Once all the domains have been added, click Next to continue.


9. The next step is to configure the advanced storage settings used by Horizon.  The two options to select on this screen are:

  • Reclaim VM Disk Space – Allows Horizon to reclaim disk space allocated to linked-clone virtual machines.
  • Enable View Storage Accelerator – View Storage Accelerator is a RAMDISK cache that can be used to offload some storage requests to the local system.  Regenerating the cache can impact IO operations on the storage array, and maintenance blackout windows can be configured to avoid a long train of witnesses.  The max cache size is 2GB.

After you have made your selections, click Next to continue.


10. Review the settings and click finish.


Configuring the Horizon Events Database

The last thing that we need to configure is the Horizon Events Database.  As the name implies, the Events Database is a repository for events that happen with the View environment.  Some examples of events that are recorded include logon and logoff activity and Composer errors.

Part 6 described the steps for creating the database and the database user account.

1. In the View Configuration section, select Event Configuration.

4. Event Configuration

2. In the Event Database section, click Edit.

5. View Events Database Section

3. Enter the following information to set up the connection:

  • Database Server (if not installed to the default instance, enter as servername\instance)
  • Database Type
  • Port
  • Database name
  • Username
  • Password
  • Table Prefix (not needed unless you have multiple Connection Server environments that use the same events database – IE large “pod” environments)

6. Edit Events Database Settings

Note: The only SQL Server instance that uses port 1433 is the default instance.  Named instances use dynamic port assignment that assigns a random port number to the service upon startup.  If the Events database is installed to a named instance, it will need to have a static port number.  You can set up SQL Server to listen on a static port by using this TechNet article.  For the above example, I assigned the port 1433 to the Composer instance since I will not have a named instance on that server.

If you do not configure a static port assignment and try to connect to a named instance on port 1433, you may receive the error below.

7a. Bad Username or Password

5. If setup is successful, you should see a screen similar to the one below.  At this point, you can change your event retention settings by editing the event settings.

7b. Success!

by seanpmassey at September 07, 2016 01:00 PM

September 06, 2016

Sean's IT Blog

Horizon 7.0 Part 8 – Installing The First Connection Server

Connection Servers are one of the most important components in any Horizon environment, and they come in three flavors – the standard Connection Server, the Replica Connection Server, and the Security Server. 

You may have noticed that I listed two types of connection servers.  The Standard and Replica Connection Servers have the same feature set, and the only difference between the two is that the standard connection server is the first server installed in the pod.  Both connection server types handle multiple roles in the Horizon infrastructure.   They handle primary user authentication against Active Directory, management of desktop pools, provide a portal to access desktop pools and published applications, and broker connections to desktops, terminal servers, and applications.  The connection server’s analog in Citrix environments would be a combination of Storefront and the Delivery Controller.

The Security Server is a stripped down version of the regular Connection Server designed to provide secure remote access.  It is designed to operate in a DMZ network and tunnel connections back to the Connection server, and it must be paired with a specific Connection Server in order for the installation to complete successfully.  Unlike previous versions of this walkthrough, I won’t be focusing on the Security Server in the remote access section as VMware now provides better tools.

Installing the First Connection Server

Before you can begin installing the Horizon View, you will need to have a server prepared that meets the minimum requirements for the Horizon View Connection Server instance.  The basic requirements, which are described in Part 2, are a server running Windows Server 2008 R2 or Server 2012 R2 with 2 CPUs and at least 4GB of RAM.

Note:  If you are going have more than 50 virtual desktop sessions on a Connection Server, it should be provisioned with at least 10GB of RAM.

Once the server is provisioned, and the Connection Server installer has been copied over, the steps for configuring the first Connection Server are:

1. Launch the Connection Server installation wizard by double-clicking on VMware-viewconnectionserver-x86_64-7.x.x-xxxxxxx.exe.

2. Click Next on the first screen to continue.


3.  Accept the license agreement and click Next to continue.


4.  If required, change the location where the Connection Server files will be installed and click Next.


5. Select the type of Connection Server that you’ll be installing.  For this section, we’ll select the Horizon 7 Standard Server.  If you plan on allowing access to desktops through an HTML5 compatible web browser, select “Install HTML Access.”  Select the IP protocol that will be used to configure the Horizon environment.  Click Next to continue.


6. Enter a strong password for data recovery.  This will be used if you need to restore the Connection Server’s LDAP database from backup.  Make sure you store this password in a secure place.  You can also enter a password reminder or hint, but this is not required.


7. Horizon View requires a number of ports to be opened on the local Windows Server firewall, and the installer will prompt you to configure these ports as part of the installation.  Select the “Configure Windows Firewall Automatically” to have this done as part of the installation.


Note: Disabling the Windows Firewall is not recommended.  If you plan to use Security Servers to provide remote access, the Windows Firewall must be enabled on the Connection Servers to use IPSEC to secure communications between the Connection Server and the Security Server.  The Windows Firewall should not be disabled even if Security Servers and IPSEC are not required.

8. The installer will prompt you to select the default Horizon environment administrator.  The options that can be selected are the local server Administrator group, which will grant administrator privileges to all local admins on the server, or to select a specific domain user or group.  The option you select will depend on your environment, your security policies, and/or other requirements.

If you plan to use a specific domain user or group, select the “Authorize a specific domain user or domain group” option and enter the user or group name in the “domainname\usergroupname” format.


Note: If you plan to use a custom domain group as the default Horizon View administrator group, make sure you create it and allow it to replicate before you start the installation. 

9.  Chose whether you want to participate in the User Experience Improvement program.  If you do not wish to participate, just click Next to continue.


10. Click Install to begin the installation.


11. The installer will install and configure the application and any additional windows roles or features that are needed to support Horizon View. 


12. Once the install completes, click Finish.  You may be prompted to reboot the server after the installation completes.

Now that the Connection Server and Composer are installed, it’s time to begin configuring the Horizon application so the Connection Server can talk to both vCenter and Composer as well as setting up any required license keys and the events database.  Those steps will be covered in the next part.

by seanpmassey at September 06, 2016 01:00 PM

September 05, 2016


Puppet 4 Sensitive Data Types

You often need to handle sensitive data in manifests when using Puppet. Private keys, passwords, etc. There has not been a native way to deal with these and so a cottage industry of community tools have spring up.

To deal with data at rest various Hiera backends like the popular hiera-eyaml exist, to deal with data on nodes a rather interesting solution called binford2k-node_encrypt exist. There are many more but less is more, these are good and widely used.

The problem is data leaks all over the show in Puppet – diffs, logs, reports, catalogs, PuppetDB – it’s not uncommon for this trusted data to show up all over the place. And dealing with this problem is a huge scope issue that will require adjustments to every component – Puppet, Hiera / Lookup, PuppetDB, etc.

But you have to start somewhere and Puppet is the right place, lets look at the first step.


Puppet 4.6.0 introduce – and 4.6.1 fixed – a new data type that decorates other data telling the system it’s sensitive. And this data cannot by accident become logged or leaked since the type will only return a string indicating it’s redacted.

It’s important to note this is step one of a many step process towords having a unified blessed way of dealing with Sensitive data all over. But lets take a quick look at them. The official specification for this feature lives here.

In the most basic case we can see how to make sensitive data, how it looks when logged or leaked by accident:

$secret = Sensitive("My Socrates Note")

This prints out the following:

Notice: Scope(Class[main]): Sensitive [value redacted]

To unwrap this and gain access to the real original data:

$secret = Sensitive(hiera("secret"))
$unwrapped = $secret.unwrap |$sensitive| { $sensitive }
notice("Unwrapped: ${unwrapped}")
$secret.unwrap |$sensitive| { notice("Lambda: ${sensitive}") }

Here you can see how to assign it unwrapped to a new variable or just use it in a block. Important to note you should never print these values like this and ideally you’d only ever use them inside a lambda if you have to use them in .pp code. Puppet has no concept of private variables so this $unwrapped variable could be accessed from outside of your classes. A lambda scope is temporary and private.

The output of above is:

Notice: Scope(Class[main]): Unwrapped: Too Many Secrets
Notice: Scope(Class[main]): Lambda: Too Many Secrets

So these are the basic operations, you can now of course pass the data around classes.

class mysql (
  Sensitive[String] $root_pass
) {
  # somehow set the password
  root_pass => Sensitive(hiera("mysql_root"))

Note here you can see the class specifically wants a String that is sensitive and not lets say a Number using the Sensitive[String] markup. And if you attempted to pass Sensitive(1) into it you’d get a type missmatch error.


So this appears to be quite handy, you can see down the line that lookup() might have a eyaml like system and emit Sensitive data directly and perhaps some providers and types will support this. But as I said it’s early days so I doubt this is actually useful yet.

I mentioned how other systems like PuppetDB and so forth also need updates before this is useful and indeed today PuppetDB is oblivious to these types and stores the real values:

$ puppet query 'resources[parameters] { type = "Class" and title = "Test" }'
    "parameters": {
      "string": "My Socrates Note"

So this really does not yet serve any purpose but as a step one it’s an interesting look at what will come.

by R.I. Pienaar at September 05, 2016 08:38 AM

September 04, 2016

Carl Chenet

Retweet 0.9: Automatically retweet & like

Retweet 0.9, a self-hosted Python app to automatically retweet and like tweets from another user-defined Twitter account, was released this September, 2nd.

Retweet 0.9 is already in production for Le Journal du hacker, a French-speaking Hacker News-like website,, a French-speaking job board and this very blog.


What’s the purpose of Retweet?

Let’s face it, it’s more and more difficult to communicate about our projects. Even writing an awesome app is not enough any more. If you don’t appear on a regular basis on social networks, everybody thinks you quit or that the project is stalled.

But what if you already have built an audience on Twitter for, let’s say, your personal account. Now you want to automatically retweet and like all tweets from the account of your new project, to push it forward.

Sure, you can do it manually, like in the old good 90’s… or you can use Retweet!

Twitter Out Of The Browser

Have a look at my Github account for my other Twitter automation tools:

  • Feed2tweet, a RSS-to-Twitter command-line tool
  • db2twitter, get data from SQL database (several supported), build tweets and send them to Twitter
  • Twitterwatch, monitor the activity of your Twitter timeline and warn you if no new tweet appears

What about you? Do you use tools to automate the management of your Twitter account? Feel free to give me feedback in the comments below.

by Carl Chenet at September 04, 2016 10:00 PM

September 01, 2016

Carl Chenet

Dans le Libre : manger ce que l’on prépare

Dans le précédent article de cette série Dans le Libre : gratter ses propres démangeaisons, nous avons vu pourquoi dans une démarche de Libriste visant à régler un problème par l’utilisation ou l’écriture d’un programme, il était important d’identifier les tâches qui pouvaient être automatisées, d’établir un plan d’action, puis de rechercher des solutions existantes avant d’éventuellement s’attaquer soi-même à écrire un programme pouvant régler le problème.

Nous verrons aujourd’hui pourquoi il est important d’utiliser soi-même son propre logiciel.

L’usage créé de nouveaux besoins

Ce faisant, il s’est avéré qu’utiliser notre propre programme levait peu à peu des nouveaux besoins. La première version de notre programme réglait le principal problème que nous avions, mais nous avons rapidement eu envie de rajouter des fonctionnalités pour répondre encore plus précisément à notre besoin et ce de manière complètement automatisée.

Manger ce que l’on prépare (Eat Your Own Dog Food en anglais, d’où l’image ci-dessous) – dans notre cas utiliser notre propre programme – nous a permis de mieux cerner le domaine des possibles et les évolutions futures à réaliser pour ce programme.

Manger votre propre nourriture (Eat Your Own Dog Food en anglais), c'est s'assurer un bon logiciel

Manger ce que l’on prépare (librement traduit de Eat Your Own Dog Food en anglais), c’est s’assurer la résolution de son problème et peut-être d’autres utilisateurs

Un exemple de ce processus est l’application db2twitter, un logiciel capable de piocher des données dans une base de données SQL avant de créer des tweets avec et de les poster sur le réseau social Twitter.

Arrivé assez rapidement à une version 0.1, le besoin d’identifier les nouvelles entrées dans la base SQL et de tweeter en conséquence m’a rapidement poussé à écrire la 0.2. Puis le besoin de modifier la requête SQL à la source des interrogations vers la base m’a orienté vers une nouvelle fonctionnalité pour la version 0.3. La version 0.4 a apporté la possibilité de ré-émettre régulièrement des tweets déjà envoyés et ainsi de suite  jusqu’à la version actuelle 0.6.

db2twitter gère MySQL, mais aussi PostgreSQL, SQLite et plusieurs bases propriétaires

db2twitter gère MySQL, mais aussi PostgreSQL, SQLite et plusieurs bases de données propriétaires

Avec une description systématique des changements des nouvelles versions, l’histoire des besoins du projet est simple à suivre. L’usage de votre propre programme amène donc à des nouveaux besoins.

Tout est simple les premiers jours

Les premiers jours, tout est simple. Vous identifiez un nouveau besoin, vous mettez à jour vos sources dans votre dépôt, vous clonez  le code depuis votre compte  sur votre serveur, une petite ligne dans la crontab et hop, c’est en production. C’est en effet le moment d’itérer très rapidement dans votre développement. Vous n’être pas encore ralenti par l’existant. Mais parfois aller trop vite peut vous jouer des tours. Quelques règles à retenir :

  • vous n’avez pas créé dès la version 0.1 de votre projet un dépôt de sources pour rien.  Contraignez-vous à sortir de nouvelles versions complètes et pas seulement de pousser des nouveaux commits dans votre dépôt
  • ouvrez systématiquement des rapports de bugs dès que vous avez une idée, quitte à fermer le rapport si vous trouvez ça inutile ensuite. Cela vous permet de suivre le développement de vos nouvelles fonctionnalités et de potentiellement communiquer dessus, en particulier auprès de vos utilisateurs ou potentiels utilisateurs
  • lorsque vous souhaitez sortir une nouvelle version, générez une marque (tag) avec le numéro de la version en question, signez-le avec GPG éventuellement, puis poussez le vers votre dépôt
  • au niveau du dépôt, avec la marque précédemment créée, générez une archive que vos utilisateurs pourront télécharger, facilitant ainsi la vie des empaqueteurs des différentes distributions intéressés par votre programme
  • N’oubliez pas de définir une liste des changements qu’apporte la nouvelle version. Concentré dans un fichier appelé CHANGELOG, les nouveautés de votre nouvelle version peuvent prendre la forme d’un fichier dans vos sources ou vous pouvez remplir la description de la nouvelle version sur votre gestionnaire de sources. Attention toutefois si vous utilisez un service en ligne pour faire cela. Si le service s’arrête un jour, vous risquez de perdre cette liste des changements depuis la création de votre logiciel
Le site donne de très bonne lignes directrices pour créer votre fichier CHANGELOG

Le site donne de très bonne lignes directrices pour créer votre fichier CHANGELOG

  • Communiquez autour de vos nouvelles versions ! Ces communications autour de vos nouvelles versions sont la ligne de vie de votre existence publique. Cela inclue une dépêche sur un ou plusieurs sites web communautaires, un billet sur votre blog, des communications sur les réseaux sociaux et contacter vos empaqueteurs des différentes distributions Linux et/ou BSD si vous en avez.


Si ces bonnes pratiques peuvent paraître un peu lourdes pour un projet dont vous êtes le seul utilisateur au début, elles vont vous donner un cadre de publication qu’il faudra impérativement respecter dans le futur sous peine de rater des publications, d’être incompris ou ignoré de vos éventuels utilisateurs.

Un excellent exemple de cette évolution est Retweet, un outil pour retweeter automatiquement selon de nombreux critères. Dès la première version, une marque a été utilisée pour marquer la sortie d’une nouvelle version, l’archive et le CHANGELOG étant générés via l’interface web du dépôt Github de Retweet. On peut ainsi fournir le maximum d’informations aux utilisateurs dans le temps, et éventuellement fournir des repères chronologiques pour une chasse aux bugs ou de l’aide à de nouveaux contributeurs.

Github présente des outils pour générer les archives de versions et tracer les nouveautés associées

Github présente des outils pour générer les archives de versions et tracer les nouveautés associées, outils malheureusement propriétaires. Préférez-lui Gitlab.

… mais avec plusieurs instances d’installées en production, ça se complique

Vous avez maintenant plusieurs versions à votre actif et une dizaine d’instances installées sur deux ou trois serveurs. Le temps entre le développement d’une nouvelle fonctionnalité et son arrivée en production a augmenté. Vous devez en effet coder la nouvelle fonctionnalité, la publier, la mettre en place sur différents serveurs, donc dans différents contextes servant des buts différents.

Dans l’idée principale de cet article et toujours en suivant notre exemple, utiliser votre propre programme a amené à un besoin de flexibilité. Très certainement le besoin d’un fichier de configuration s’est fait sentir, pour s’adapter à différentes configurations. Un besoin de stockage avec la gestion de fichiers ou d’une base de données s’est ajouté à votre projet, pour stocker les informations d’une version à l’autre.

Prenons un exemple : le projet Feed2tweet, petit projet qui récupère un flux RSS pour l’envoyer vers Twitter, est en production sur le Journal du hacker mais aussi, ce blog et bien d’autres aujourd’hui. Il a donc été indispensable au bout d’un moment d’automatiser le déploiement de nouvelles versions. Il est important de comprendre qu’utiliser son propre produit créé de nouveaux besoins, mais qu’une fois satisfaits, les nouvelles fonctionnalités répondant aux différents besoins permettent de pousser la flexibilité et le confort d’utilisation et de toucher ainsi des nouveaux cas d’utilisation et un public plus large., le site d'emploi de la communauté du Logiciel Libre, contribue au Logiciel Libre, le site d’emploi de la communauté du Logiciel Libre, contribue au Logiciel Libre et à Feed2tweet

Manger ce que l’on prépare mène à tout automatiser

Comme nous l’avons vu, notre petit projet devient s’étoffe et devient de plus en plus complexe. Il couvre désormais différents cas d’utilisation. Il devient difficile de le tester manuellement et de prendre en cas tous les cas d’utilisation. Mais aussi d’assurer un système fiable de publication des nouvelles versions ainsi qu’un système de déploiement assurant un minimum d’impact sur nos différents systèmes désormais en production.

Pour nous protéger de tout ces risques et continuer à fournir le meilleur service tout d’abord à nous, mais aussi désormais aux différents contributeurs/utilisateurs de notre programmeur, il est nécessaire d’automatiser les différents points vus ci-dessus. Nous aborderons différents moyens d’y arriver dans le prochain article de notre série 😉

Et de votre côté ? Avez-vous amélioré votre propre programme par l’identification de nouveaux besoins en l’utilisant régulièrement ? N’hésitez pas à en parler dans les commentaires !

by Carl Chenet at September 01, 2016 10:00 PM

Anton Chuvakin - Security Warrior

Monthly Blog Round-Up – August 2016

Here is my next monthly "Security Warrior" blog round-up of top 5 popular posts/topics this month:
  1. Why No Open Source SIEM, EVER?” contains some of my SIEM thinking from 2009. Is it relevant now? You be the judge.  Succeeding with SIEM requires a lot of work, whether you paid for the software, or not. BTW, this post has an amazing “staying power” that is hard to explain – I suspect it has to do with people wanting “free stuff” and googling for “open source SIEM” … 
  2. “New SIEM Whitepaper on Use Cases In-Depth OUT!” (dated 2010) presents a whitepaper on select SIEM use cases described in depth with rules and reports [using now-defunct SIEM product]; also see this SIEM use case in depth and this for a more current list of popular SIEM use cases. Finally, see our 2016 research on security monitoring use cases here!
  3. Simple Log Review Checklist Released!” is often at the top of this list – this aging checklist is still a very useful tool for many people. “On Free Log Management Tools” is a companion to the checklist (updated version)
  4. My classic PCI DSS Log Review series is always popular! The series of 18 posts cover a comprehensive log review approach (OK for PCI DSS 3+ as well), useful for building log review processes and procedures , whether regulatory or not. It is also described in more detail in our Log Management book and mentioned in our PCI book (now in its 4th edition!)
  5. “SIEM Resourcing or How Much the Friggin’ Thing Would REALLY Cost Me?” is a quick framework for assessing the costs of a SIEM project (well, a program, really) at an organization (much more details on this here in this paper).
In addition, I’d like to draw your attention to a few recent posts from my Gartner blog [which, BTW, now has about 5X of the traffic of this blog]: 
Current research on SOC:
Current research on threat intelligence:
Miscellaneous fun posts:

(see all my published Gartner research here)
Also see my past monthly and annual “Top Popular Blog Posts” – 2007, 2008, 2009, 2010, 2011, 2012, 2013, 2014, 2015.
Disclaimer: most content at SecurityWarrior blog was written before I joined Gartner on Aug 1, 2011 and is solely my personal view at the time of writing. For my current security blogging, go here.

Previous post in this endless series:

by Anton Chuvakin ( at September 01, 2016 12:55 PM

August 29, 2016

That grumpy BSD guy

The Voicemail Scammers Never Got Past Our OpenBSD Greylisting

We usually don't see much of the scammy spam and malware. But that one time we went looking for them, we found a campaign where our OpenBSD greylisting setup was 100% effective in stopping the miscreants' messages.

During August 23rd to August 24th 2016, a spam campaign was executed with what appears to have been a ransomware payload. I had not noticed anything particularly unusual about the and friends setup that morning, but then Xavier Mertens' post at Voice Message Notifications Deliver Ransomware caught my attention in the tweetstream, and I decided to have a look.

The first step was, as always, to grep the spamd logs, and sure, there were entries with from: addresses of voicemail@ in several of the domains my rigs are somehow involved in handling mail for.

But no message from had yet reached any mailbox within my reach at that point. However, a colleague checked the quarantine at one of his private mail servers, and found several messsages from voicemail@ aimed at users in his domains.

Dissecting a random sample confirmed that the message came with an attachment with a filename that was actually a somewhat obfuscated bit of javascript, and I take others at their word that this code, if executed on your Microsoft system, would wreak havoc of some sort.

At this point, before I start presenting actual log file evidence, it is probably useful to sketch how the systems here work and interact. The three machines skapet, deliah and portal are all OpenBSD systems that run spamd in greylisting mode, and they sync their spamd data with each other via spamd's own synchronization mechanism.

All of those machines do greytrapping based on the list of spamtraps, and skapet has the additional duty of dumping the contents of its greytrapping generated blacklist to a downloadable text file once per hour. Any message that makes it past spamd is then fed to a real mail server that performs content filtering before handing the messages over a user's mailbox or, in the case of domains we only do the filtering for, forwards the message to the target domain's mail server.

The results of several rounds of 'grep voicemail $logfile' over the three spamd machines are collected here, or with the relatively uninteresting "queueing deletion of ..." messages removed, here.

From those sources we can see that there were a total of 386 hosts that attempted delivery, to a total of 396 host and target email pairs (annotated here in a .csv file with geographic origin according to whois).

The interesting part came when I started looking at the mail server logs to see how many had reached the content filtering or had even been passed on in the direction of users' mailboxes.

There were none.

The number of messages purportedly from voicemail@ in any of the domains we handle that made it even to the content filtering stage was 0.

Zero. Not a single one made it through even to content filtering.

That shouldn't have been a surprise.

After all I've spent significant time over the years telling people how effective greylisting is, and that the OpenBSD spamd version is the best of the breed.

You could take this episode as a recent data point that you are free to refer to in your own marketing pushes if you're doing serious business involving OpenBSD.

And if you're into those things, you will probably be delighted to learn, if you hadn't figured that out already, that a largish subset of the attempted deliveries were to addresses that were already in our published list of spamtrap addresses.

That means our miscreants automatically had themselves added to the list of trapped spammer IP addresses as intended.

If you're interested in how this works and why, I would suggest taking a peek at the OpenBSD web site, and of course I have a book out (available at that link and via better bookstores everywhere) that explains those things as well.

Relevant blog posts of mine include Keep smiling, waste spammers' time, Maintaining A Publicly Available Blacklist - Mechanisms And Principles, In The Name Of Sane Email: Setting Up OpenBSD's spamd(8) With Secondary MXes In Play - A Full Recipe and a few others, including the somewhat lengty Effective Spam and Malware Countermeasures - Network Noise Reduction Using Free Tools . To fully enjoy the experience of what these articles describe, you may want to get hold of your own CD set from the OpenBSD store.

And again, if you're doing business involving OpenBSD, please head over to the project's donations page and use one or more of the methods there to send the developers some much needed cash.

In addition to the files directly referenced in this article, some related files are  available from this directory. I'll be happy to answer any reasonable queries related to this material.

Good night and good luck.

Update 2016-08-30: I've been getting questions about the currently active campaign that has document@ as its sender. The same story there: I see them in the greylist and spamd logs, no trace whatsoever in later steps. Which means they're not getting anyhwere.

Update 2016-09-13: A quick glance at a tail -f'ed spamd log file reveals that today's fake sender of choice is CreditControl@. Otherwise same story as before, no variations. And of course, there may have been dozens I haven't noticed in the meantime.

by Peter N. M. Hansteen ( at August 29, 2016 05:28 PM

August 24, 2016


The SWEET32 Issue, CVE-2016-2183

Today, Karthik Bhargavan and Gaetan Leurent from Inria have unveiled a new attack on Triple-DES, SWEET32, Birthday attacks on 64-bit block ciphers in TLS and OpenVPN. It has been assigned CVE-2016-2183.

This post gives a bit of background and describes what OpenSSL is doing. For more details, see their website.

Because DES (and triple-DES) has only a 64-bit block size, birthday attacks are a real concern. With the ability to run Javascript in a browser, it is possible to send enough traffic to cause a collision, and then use that information to recover something like a session Cookie. Their experiments have been able to recover a cookie in under two days. More details are available at their website. But the take-away is this: triple-DES should now be considered as “bad” as RC4.

Triple-DES, which shows up as “DES-CBC3” in an OpenSSL cipher string, is still used on the Web, and major browsers are not yet willing to completely disable it.

If you run a server, you should disable triple-DES. This is generally a configuration issue. If you run an old server that doesn’t support any better ciphers than DES or RC4, you should upgrade.

Within the OpenSSL team, we discussed how to classify this, using our security policy, and we decided to rate it LOW. This means that we just pushed the fix into our repositories. Here is what we did:

  • For 1.0.2 and 1.0.1, we removed the triple-DES ciphers from the “HIGH” keyword and put them into “MEDIUM.” Note that we did not remove them from the “DEFAULT” keyword.

  • For the 1.1.0 release, which we expect to release tomorrow, we will treat triple-DES just like we are treating RC4. It is not compiled by default; you have to use “enable-weak-ssl-ciphers” as a config option. Even when those ciphers are compiled, triple-DES is only in the “MEDIUM” keyword. In addition, because this is a new release, we also removed it from the “DEFAULT” keyword.

When you have a large installed base, it is hard to move forward in a way that will please everyone. Leaving triple-DES in “DEFAULT” for 1.0.x and removing it from 1.1.0 is admittedly a compromise. We hope the changes above make sense, and even if you disagree and you run a server, you can explicitly protect your users through configuration.

Finally, we would like to thank Karthik and Gaeten for reaching out to us, and working closely to coordinate our releases with their disclosure.

August 24, 2016 11:16 PM

Geek and Artist - Tech

Adding Meaning to Code

This is the product of only about 5 minutes worth of thought, so take it with a grain of salt. When it comes to how to write maintainable, understandable code, there are as many opinions out there as there are developers. Personally I favour simple, understandable, even “boring” method bodies that don’t try to be flashy or use fancy language features. Method and class names should clearly signal intent and what the thing is or does. And, code should (IMHO) include good comments.

This last part is probably an area I’ve seen the most dissent. For some reason people hate writing comments, and think that the code should be “self-documenting”. I’ve rarely, perhaps never seen this in practice. While perhaps the intent was for it to be self-documenting, that never arose in practice.

Recently (and this is related, I promise), I watched a lot of talks (one, in person) and read a lot about the Zalando engineering principles. They base their engineering organisation around three pillars of How, What and Why. I think the same thing can be said for how you should write code and document it:

class Widget
  def initialize
    @expires_at = + 86400

  # Customer X was asking for the ability to expire     #  <--- Why
  # widgets, but some may not have an expiry date or
  # do not expire at all. This method handles these
  # edge cases safely.
  def is_expired?()                                     #  <--- What
    return !!@expires_at && > @expires_at    #  <--- How

This very simple example shows what I mean (in Ruby, since it's flexible and lends itself well to artificial examples like this). The method body itself should convey the How of the equation. The method name itself should convey the intent of the method - What does this do? Ultimately, the How and What can probably never fully explain the history and reasoning for their own existence. Therefore I find it helpful to accompany these with the Why in a method comment to this effect (and a comment above the method could also be within the method, or distributed across the method - it's not really important).

You could argue that history and reasoning for having the method can be determined from version control history. This turns coding from what should be a straightforward exercise into some bizarre trip through the Wheel of Time novels, cross-referencing back to earlier volumes in order to try to find some obscure fact that may or may not actually exist, so that you can figure out the reference you are currently reading. Why make the future maintainer of your code go through that? Once again, it relies entirely on the original committer having left a comprehensive and thoughtful message that is also easy to find.

The other counter argument is that no comments are better than out of date or incorrect comments. Again, personally I haven't run into this (or at least, not nearly as frequently as comments missing completely). Usually it will be pretty obvious where the comment does not match up with the code, and in this (hopefully outlier) case you can then go version control diving to find out when they diverged. Assessing contents of the code itself is usually far easier than searching for an original comment on the first commit of that method, so it seems like this should be an easier exercise.

Writing understandable code (and let's face it, most of the code written in the world is probably doing menial things like checking if statements, manipulating strings or adding/removing items from arrays) and comments is less fun than hacking out stuff that just works when you are feeling inspired, so no wonder we've invented an assortment of excuses to avoid doing it. So if you are one of the few actually doing this, thank you.

by oliver at August 24, 2016 06:45 PM

Cryptography Engineering

Attack of the week: 64-bit ciphers in TLS

A few months ago it was starting to seem like you couldn’t go a week without a new attack on TLS. In that context, this summer has been a blessed relief. Sadly, it looks like our vacation is over, and it’s time to go back to school.

Today brings the news that Karthikeyan Bhargavan and Gaëtan Leurent out of INRIA have a new paper that demonstrates a practical attack on legacy ciphersuites in TLS (it’s called “Sweet32”, website here). What they show is that ciphersuites that use 64-bit blocklength ciphers — notably 3DES — are vulnerable to plaintext recovery attacks that work even if the attacker cannot recover the encryption key.

While the principles behind this attack are well known, there’s always a difference between attacks in principle and attacks in practice. What this paper shows is that we really need to start paying attention to the practice.

So what’s the matter with 64-bit block ciphers?

Block ciphers are one of the most widely-used cryptographic primitives. As the nameimplies, these are schemes designed to encipher data in blocks, rather than a single bit at a time.

The two main parameters that define a block cipher are its block size (the number of bits it processes in one go), and its key size. The two parameters need not be related. So for example, DES has a 56-bit key and a 64-bit block. Whereas 3DES (which is built from DES) can use up to a 168-bit key and yet still has the same 64-bit block. More recent ciphers have opted for both larger blocks and larger keys.

When it comes to the security provided by a block cipher, the most important parameter is generally the key size. A cipher like DES, with its tiny 56-bit key, is trivially vulnerable to brute force attacks that attempt decryption with every possible key (often using specialized hardware). A cipher like AES or 3DES is generally not vulnerable to this sort of attack, since the keys are much longer.

However, as they say: key size is not everything. Sometimes the block size matters too.

You see, in practice, we often need to encrypt messages that are longer than a single block. We also tend to want our encryption to be randomized. To accomplish this, most protocols use a block cipher in a scheme called a mode of operation. The most popular mode used in TLS is CBC mode. Encryption in CBC looks like this:

Source: Wikipedia

The nice thing about CBC is that (leaving aside authentication issues) it can be proven (semantically) secure if we make various assumptions about the security of the underlying block cipher. Yet these security proofs have one important requirement. Namely, the attacker must not receive too much data encrypted with a single key.

The reason for this can be illustrated via the following simple attack.

Imagine that an honest encryptor is encrypting a bunch of messages using CBC mode. Following the diagram above, this involves selecting a random Initialization Vector (IV) of size equal to the block size of the cipher, then XORing IV with the first plaintext block (P), and enciphering the result (P \oplus IV). The IV is sent (in the clear) along with the ciphertext.

Most of the time, the resulting ciphertext block will be unique — that is, it won’t match any previous ciphertext block that an attacker may have seen. However, if the encryptor processes enough messages, sooner or later the attacker will see a collision. That is, it will see a ciphertext block that is the same as some previous ciphertext block. Since the cipher is deterministic, this means the cipher’s input (P \oplus IV) must be identical to the cipher’s previous input (P' \oplus IV') that created the previous block.

In other words, we have (P \oplus IV) = (P' \oplus IV'), which can be rearranged as (P \oplus P') = (IV \oplus IV'). Since the IVs are random and known to the attacker, the attacker has (with high probability) learned the XOR of two (unknown) plaintexts!

What can you do with the XOR of two unknown plaintexts? Well, if you happen to know one of those two plaintext blocks — as you might if you were able to choose some of the plaintexts the encryptor was processing — then you can easily recover the other plaintext. Alternatively, there are known techniques that can sometimes recover useful data even when you don’t know both blocks.

The main lesson here is that this entire mess only occurs if the attacker sees a collision. And the probability of such a collision is entirely dependent on the size of the cipher block. Worse, thanks to the (non-intuitive) nature of the birthday bound, this happens much more quickly than you might think it would. Roughly speaking, if the cipher block is b bits long, then we should expect a collision after roughly 2^{b/2} encrypted blocks.

In the case of a 64-bit blocksize cipher like 3DES, this is somewhere in the vicinity of 2^{32}, or around 4 billion enciphered blocks.

(As a note, the collision does not really need to occur in the first block. Since all blocks in CBC are calculated in the same way, it could be a collision anywhere within the messages.)

Whew. I thought this was a practical attack. 4 billion is a big number!

It’s true that 4 billion blocks seems like an awfully large number. In a practical attack, the requirements would be even larger — since the most efficient attack is for the attacker to know a lot of the plaintexts, in the hope that she will be able to recover one unknown plaintext when she learns the value (P ⊕ P’).

However, it’s worth keeping in mind that these traffic numbers aren’t absurd for TLS. In practice, 4 billion 3DES blocks works out to 32GB of raw ciphertext. A lot to be sure, but not impossible. If, as the Sweet32 authors do, we assume that half of the plaintext blocks are known to the attacker, we’d need to increase the amount of ciphertext to about 64GB. This is a lot, but not impossible.

The Sweet32 authors take this one step further. They imagine that the ciphertext consists of many HTTPS connections, consisting of 512 bytes of plaintext, in each of which is embedded the same secret 8-byte cookie — and the rest of the session plaintext is known. Calculating from these values, they obtain a requirement of approximately 256GB of ciphertext needed to recover the cookie with high probability.

That is really a lot.

But keep in mind that TLS connections are being used to encipher increasingly more data. Moreover, a single open browser frame running attacker-controlled Javascript can produce many gigabytes of ciphertext in a single hour. So these attacks are not outside of the realm of what we can run today, and presumably will be very feasible in the future.

How does the TLS attack work?

While the cryptographic community has been largely pushing TLS away from ciphersuites like CBC, in favor of modern authenticated modes of operation, these modes still exist in TLS. And they exist not only for use not only with modern ciphers like AES, but they are often available for older ciphersuites like 3DES. For example, here’s a connection I just made to Google:

Of course, just because a server supports 3DES does not mean that it’s vulnerable to this attack. In order for a particular connection to be vulnerable, both the client and server must satisfy three main requirements:

    1. The client and server must negotiate a 64-bit cipher. This is a relatively rare occurrence, but can happen in cases where one of the two sides is using an out-of-date client. For example, stock Windows XP does not support any of the AES-based ciphersuites. Similarly, SSL3 connections may negotiate 3DES ciphersuites.
    2. The server and client must support long-lived TLS sessions, i.e., encrypting a great deal of data with the same key. Unfortunately, most web browsers place no limit on the length of an HTTPS session if Keep-Alive is used, provided that the server allows the session. The Sweet32 authors scanned and discovered that many servers (including IIS) will allow sessions long enough to run their attack. Across the Internet, the percentage of vulnerable servers is small (less than 1%), but includes some important sites.
    3. The client must encipher a great deal of known data, including a secret session cookie. This is generally achieved by running adversarial Javascript code in the browser, although it could be done using standard HTML as well.

      Sites vulnerable to Sweet32. (source)

These caveats aside, the authors were able to run their attack using Firefox, sending at a rate of about 1500 connections per second. With a few optimizations, they were able to recover a 16-byte secret cookie in about 30 hours (a lucky result, given an expected 38 hour run time).The client must encipher a great deal of known data, including a secret session cookie. This is generally achieved by running adversarial Javascript code in the browser, although it could be done using standard HTML as well.

So what do we do now?

While this is not an earthshaking result, it’s roughly comparable to previous results we’ve seen with legacy ciphers like RC4.

In short, while these are not the easiest attacks to run, it’s a big problem that there even exist semi-practical attacks that undo the encryption used in standard encryption protocols. This is a problem that we should address, and these attack papers help to make those problems more clear.

by Matthew Green at August 24, 2016 04:55 PM

Anton Chuvakin - Security Warrior

August 22, 2016

cfengine-tap now on GitHub

github-logo Back from the holiday season, I have finally found the time to publish a small library on GitHub. It’s called cfengine-tap and can help you writing TAP-compatible tests for your CFEngine policies.

TAP is the test anything protocol. It is a simple text format that test scripts can use to print out the results and test suites can consume. Originally born in the Perl world, it is now supported in many other languages.

Using this library it’s easier to write test suites for your CFEngine policies. Since it’s publicly available on GitHub and published under a GPL license, you are free to use it and welcome to contribute and make it better (please do).


Tagged: cfengine, Configuration management, DevOps, Github, TAP, testing

by bronto at August 22, 2016 01:00 PM

August 20, 2016

Feeding the Cloud

Remplacer un disque RAID défectueux

Traduction de l'article original anglais à

Voici la procédure que j'ai suivi pour remplacer un disque RAID défectueux sur une machine Debian.

Remplacer le disque

Après avoir remarqué que /dev/sdb a été expulsé de mon RAID, j'ai utilisé smartmontools pour identifier le numéro de série du disque à retirer :

smartctl -a /dev/sdb

Cette information en main, j'ai fermé l'ordinateur, retiré le disque défectueux et mis un nouveau disque vide à la place.

Initialiser le nouveau disque

Après avoir démarré avec le nouveau disque vide, j'ai copié la table de partitions avec parted.

Premièrement, j'ai examiné la table de partitions sur le disque dur non-défectueux :

$ parted /dev/sda
unit s

et créé une nouvelle table de partitions sur le disque de remplacement :

$ parted /dev/sdb
unit s
mktable gpt

Ensuite j'ai utilisé la commande mkpart pour mes 4 partitions et je leur ai toutes donné la même taille que les partitions équivalentes sur /dev/sda.

Finalement, j'ai utilisé les commandes toggle 1 bios_grub (partition d'amorce) et toggle X raid (où X est le numéro de la partition) pour toutes les partitions RAID, avant de vérifier avec la commande print que les deux tables de partitions sont maintenant identiques.

Resynchroniser/recréer les RAID

Pour synchroniser les données du bon disque (/dev/sda) vers celui de remplacement (/dev/sdb), j'ai exécuté les commandes suivantes sur mes partitions RAID1 :

mdadm /dev/md0 -a /dev/sdb2
mdadm /dev/md2 -a /dev/sdb4

et j'ai gardé un oeil sur le statut de la synchronisation avec :

watch -n 2 cat /proc/mdstat

Pour accélérer le processus, j'ai utilisé le truc suivant :

blockdev --setra 65536 "/dev/md0"
blockdev --setra 65536 "/dev/md2"
echo 300000 > /proc/sys/dev/raid/speed_limit_min
echo 1000000 > /proc/sys/dev/raid/speed_limit_max

Ensuite, j'ai recréé ma partition swap RAID0 comme suit :

mdadm /dev/md1 --create --level=0 --raid-devices=2 /dev/sda3 /dev/sdb3
mkswap /dev/md1

Par que la partition swap est toute neuve (il n'est pas possible de restorer une partition RAID0, il faut la re-créer complètement), j'ai dû faire deux choses:

  • remplacer le UUID pour swap dans /etc/fstab, avec le UUID donné par la commande mkswap (ou bien en utilisant la command blkid et en prenant le UUID pour /dev/md1)
  • remplacer le UUID de /dev/md1 dans /etc/mdadm/mdadm.conf avec celui retourné pour /dev/md1 par la commande mdadm --detail --scan

S'assurer que l'on peut démarrer avec le disque de remplacement

Pour être certain de bien pouvoir démarrer la machine avec n'importe quel des deux disques, j'ai réinstallé le boot loader grub sur le nouveau disque :

grub-install /dev/sdb

avant de redémarrer avec les deux disques connectés. Ceci confirme que ma configuration fonctionne bien.

Ensuite, j'ai démarré sans le disque /dev/sda pour m'assurer que tout fonctionnerait bien si ce disque décidait de mourir et de me laisser seulement avec le nouveau (/dev/sdb).

Ce test brise évidemment la synchronisation entre les deux disques, donc j'ai dû redémarrer avec les deux disques connectés et puis ré-ajouter /dev/sda à tous les RAID1 :

mdadm /dev/md0 -a /dev/sda2
mdadm /dev/md2 -a /dev/sda4

Une fois le tout fini, j'ai redémarrer à nouveau avec les deux disques pour confirmer que tout fonctionne bien :

cat /proc/mdstat

et j'ai ensuite exécuter un test SMART complet sur le nouveau disque :

smartctl -t long /dev/sdb

August 20, 2016 10:00 PM


Reverse-Engineered GEOS 2.0 for C64 Source Code

The GEOS operating system managed to clone the Macintosh GUI on the Commodore 64, a computer with an 8 bit CPU and 64 KB of RAM. Based on Maciej Witkowiak's work, I created a reverse-engineered source version of the C64 GEOS 2.0 KERNAL for the cc65 compiler suite:

  • The source compiles into the exact same binary as shipped with GEOS 2.0.
  • The source is well-structured and split up into 31 source files.
  • Machine-specific code is marked up.
  • Copy protection/trap mechanisms can be disabled.
  • The build system makes sure binary layout requirements are met.

This makes the source a great starting point for

  • adding (optional) optimized code paths or features
  • integrating existing patches from various sources
  • integrating versions for other computers
  • porting it to different 6502-based computers

Just fork the project and send pull requests!

by Michael Steil at August 20, 2016 01:11 AM

August 18, 2016

Michael Biven

The Loss of a Sense Doesn’t Always Heighten the Others

Over a two week break my wife and I were talking about a statement she read where someone was called the “long time embedded photojournalist for Burning Man” and how she disagreed with this. This person wasn’t shooting for any news organization. They’re known to be one of the Burning Man faithful which removes some impartiality they may have. In essence they’re a PR photographer for Burning Man.

This made me consider most of the output from social media is in one of two different camps. The first is “Free PR” and the second is “Critics”. You’re either giving away free material to promote an idea, organization, product, or a person or you’re criticizing them.

There is a third camp (Journalism), but so few people have the patience to provide an accurate and objective comment. And so few organization have the integrity to support those ideals. It’s like the goals of our past that helped drive society to do better has been toppled by a rush of individuals trying to create a better self-image.

After mentioning this idea she told me about a piece she heard recently on the radio where the guest was referencing an article from the New York Times about Trump. The host interrupted them to mention that it was an Op-Ed and not an article. This baffled the guest who didn’t understand that it was an opinion piece and not journalism.

The flood of sources of information, both accurate and inaccurate, provided on the internet hasn’t lead people to judge the validity of something better. Instead we have seen a rise in lies, ignorance, and the absurd being reported as fact. This phenomena even has a name… post-fact. As in we now live in a post-fact world. Think birthers, anti-vaxxers, or any other conspiracy theory movement.

Progress has been delayed or reversed by having to debunk ignorance being held up as fact. The time and energy being wasted by these distractions makes me wonder if this period of time will be know as the Age of Misfeasance.

P.S. After reading this post over and fixing one too many mistakes before I hit publish, I thought are people just writing so quickly and then having faith in a technology to fix their mistakes? Is autocomplete damaging our ability to think clearly, because we’re not reading and editing what we’re writing as much as in the past?

August 18, 2016 01:34 PM

August 15, 2016

Disable "New release available" emails on Ubuntu

We have our Ubuntu machines set up to mail us the output of cron jobs like so:

$ cat /etc/crontab 

# m h dom mon dow user    command

This is encrible useful, since cronjobs should never output anything unless something is wrong.

Unfortunately, this means we also get emails like this:

New release '16.04.1 LTS' available.
Run 'do-release-upgrade' to upgrade to it.

You can fairly easily disable these by modifying the corresponding cronjob /etc/cron.weekly/update-notifier-common:

 set -e
 [ -x /usr/lib/ubuntu-release-upgrader/release-upgrade-motd ] || exit 0
 # Check to see whether there is a new version of Ubuntu available
+/usr/lib/ubuntu-release-upgrader/release-upgrade-motd > /dev/null

Now you'll no longer receive these emails. It's also possible to remove the cronjob entirely, but then an upgrade is likely to put it back, and I have no idea if the cronjob has any other side effects besides emailing.

by admin at August 15, 2016 09:15 AM