Planet SysAdmin

May 25, 2017

Chris Siebenmann

Using Linux's Magic Sysrq on modern keyboards without a dedicated Syrq key

In the old days, more or less all PC keyboards had a dedicated 'Print Screen/Sysrq' key and using Linux's Magic Sysrq was easy: you held down Alt, PrintScrn/Sysrq, and an appropriate key at the same time. However, this is becoming less and less common all the time, and in particular my current keyboard doesn't have a dedicated Sysrq key. Instead SysRq is overloaded on F9, and you get it by holding the keyboard's Fn key down when you push F9/PrintScrn. This presents a small difficulty in hitting Sysrq key combos, because holding that Fn key down while you hit regular keys often produces absolutely nothing. So if you just press, say, Fn + Alt + F9 + s to try to force a sync, nothing happens.

After some flailing around and unpredictable, intermittent successes, I think I have finally figured out how to use magic SysRq reliably on my keyboard and on similar keyboards without dedicated SysRq keys. The trick is this:

  1. press and hold Fn + Alt + your SysRq key
  2. release your SysRq key and Fn, while still holding Alt
  3. press your desired SysRq action key, such as s.

The entire sequence has proven to be important. Releasing Fn while still holding SysRq/F9 down has had a tendency to make the kernel see an Alt+F9 sequence (which switches virtual consoles and in the process wipes away a bunch of the messages I want to keep seeing, and which obviously aborts entering magic Sysrq stuff). Releasing all three keys ends the whole magic SysRq sequence, which means my s does nothing.

This turns out to be the suggested approach in the guide to magic Sysrq, although their advice is about keyboards that don't like having too many keys down at once. My keyboard specifically appears to do nothing even with just Fn + s, so I don't think it's an issue of the number of keys held down at once. And yes, I used usbmon to verify that my keyboard sends no USB events for Fn + s.

(This is perhaps trivial but I want to documented it for my own future use, because I'm sure I'm going to forget it at some point.)

by cks at May 25, 2017 08:15 PM

URLs are terrible permanent identifiers for things

I was recently reading the JSON Feed version 1 specification (via Trivium, among other places). I have a number of opinions on it as a syndication feed format, but that's not the subject of today's entry, because in the middle of the specification I ran into the following bit (which is specifically talking about the elements of feed entries, ie posts):

  • id (required, string) is unique for that item for that feed over time. [...] Ideally, the id is the full URL of the resource described by the item, since URLs make great unique identifiers.

(Emphasis mine.)

When I read this bit, I had an immediate pained reaction. As someone who has been running a blog for more than ten years and has made this exact mistake, let me assure you that URLs make terrible permanent unique identifiers for things. Yes, yes, cool URLs don't change, as the famous writeup says. Unfortunately in the real world, URLs change all of the time. One reason for this that is especially relevant right now is that URLs include the protocol, and right now the web is in the process of a major shift from HTTP to HTTPS. That shift just changed all your URLs.

(I think that over the next ten years the web will wind up being almost entirely HTTPS, even though much of it is not HTTPS today, so quite a lot of people will be going through this URL transition in the future.)

This is not the only case that may force your hand. And beyond more or less forced changes, you may someday move your blog from one domain to another or change the real URLs of all of your entries because you changed the blog system that you use (both of which has happened). In theory you can create a system to generate syndication feeds that deals with all of that, by having a 'URL for id' field of some sort (perhaps automatically derived from your configuration of URL redirections), but if you're going to wind up detaching what you put in the id field from the actual canonical URL of the entry, why not make it arbitrary in the first place? It will save you a bunch of pain to do this from the start.

(Please trust me on this one, seeing as this general issue has caused me pain. As my example illustrates, using any part of the URL as part of your 'permanent identifier' is going to cause you heartburn sooner or later.)

There are excellent reasons why the Atom syndication format both explicitly allows for and more importantly encourages various forms of permanent identifiers for feed entries that are not URLs. For example, you can use UUIDs (as 'urn:uuid:<uuid>') or your own arbitrary but unique identifier in your own namespace (as tag: URNs). The Atom format does this because the people who created it had already run into various problems with the widespread use of URLs as theoretically permanent entry identifiers in RSS feeds.

by cks at May 25, 2017 05:27 AM

Openstack Horizon, remove the loading modal with uBlock Origin

The OpenStack dashboard, Horizon, is a great piece of software to manage your OpenStack resources via the web. However, it has, in my opinion, a very big usability issue. The loading dialog that appears after you click a link. It blocks the entire page and all other links. So, whenever I click, I have to wait three to five seconds before I can do anything else. Clicked the wrong menu item? Sucks to be you, here have some loading. Clicked a link and quickly want to open something in a new tab while the page is still loading? Nope, not today. It's not that browsers have had a function to show that a page is loading, no, of course, the loading indication that has been there forever is not good enough. Let's re-invent the wheel and significantly impact the user experience. With two rules in uBlock Origin this loading modal is removed and you can work normally again in Horizon

May 25, 2017 12:00 AM

May 24, 2017

Steve Kemp's Blog

Getting ready for Stretch

I run about 17 servers. Of those about six are very personal and the rest are a small cluster which are used for a single website. (Partly because the code is old and in some ways a bit badly designed, partly because "clustering!", "high availability!", "learning!", "fun!" - seriously I had a lot of fun putting together a fault-tolerant deployment with haproxy, ucarp, etc, etc. If I were paying for it the site would be both retired and static!)

I've started the process of upgrading to stretch by picking a bunch of hosts that do things I could live without for a few days - in case there were big problems, or I needed to restore from backups.

So far I've upgraded:

  • master.steve
    • This is a puppet-master, so while it is important killing it wouldn't be too bad - after all my nodes are currently setup properly, right?
    • Upgrading this host changed the puppet-server from 3.x to 4.x.
    • That meant I had to upgrade all my client-systems, because puppet 3.x won't talk to a 4.x master.
    • Happily jessie-backports contains a recent puppet-client.
    • It also meant I had to rework a lot of my recipes, in small ways.
  • builder.steve
    • This is a host I use to build packages upon, via pbuilder.
    • I have chroots setup for wheezy, jessie, and stretch, each in i386 and amd64 flavours.
  • git.steve
    • This is a host which stores my git-repositories, via gitbucket.
    • While it is an important host in terms of functionality, the software it needs is very basic: nginx proxies to a java application which runs on localhost:XXXX, with some caching magic happening to deal with abusive clients.
    • I do keep considering using gitlab, because I like its runners, etc. But that is pretty resource intensive.
    • On the other hand If I did switch I could drop my builder.steve host, which might mean I'd come out ahead in terms of used resources.
  • leave.steve
    • Torrent-box.
    • Upgrading was painless, I only run rtorrent, and a simple object storage system of my own devising.

All upgrades were painless, with only one real surprise - the attic-backup software was removed from Debian.

Although I do intend to retry using Larss' excellent obnum in the near future pragmatically I wanted to stick with what I'm familiar with. Borg backup is a fork of attic I've been aware of for a long time, but I never quite had a reason to try it out. Setting it up pretty much just meant editing my backup-script:


Once I did that, and created some new destinations all was good: ~ $ borg init /backups/ ~ $ borg init /backups/ ~ $ ..

Upgrading other hosts, for example my website(s), and my email-box, will be more complex and fiddly. On that basis they will definitely wait for the formal stretch release.

But having a couple of hosts running the frozen distribution is good for testing, and to let me see what is new.

May 24, 2017 09:00 PM

Samba CVE-2017-7494: Remote Code Execution in Samba 3.5.0 and upwards

The post Samba CVE-2017-7494: Remote Code Execution in Samba 3.5.0 and upwards appeared first on

If you run Samba, get patching.

CVE-2017-7494: All versions of Samba from 3.5.0 onwards are vulnerable to a remote code execution vulnerability, allowing a malicious client to upload a shared library to a writable share, and then cause the server to load and execute it.

Source: [Announce] Samba 4.6.4, 4.5.10 and 4.4.14 Available for Download

The post Samba CVE-2017-7494: Remote Code Execution in Samba 3.5.0 and upwards appeared first on

by Mattias Geniar at May 24, 2017 12:15 PM

Chris Siebenmann

Exploiting Python's Global Interpreter Lock for atomic operations is fun

Yesterday I wrote a sober and respectable article on how using the GIL safely has various tricky traps and you probably shouldn't try to do it. This is indeed the sensible, pragmatic approach to exploiting what the GIL protects for threaded Python code; you probably don't need that much performance and so you should use some form of explicit locking instead of trying to be clever.

The honest truth is that I've exploited the GIL this way before and I'll probably write more Python code that does it in the future. This isn't because it's necessarily a good idea; it's because the GIL is fun. Like many other complicated parts of CPython, the GIL is simply neat all by itself (at least if you're the kind of person who likes to peer behind the curtain and know how things work). And in general it's an important part of threaded CPython, with a lot of complexity that is important to understand if you want to write well-performing threaded Python code. You can't avoid learning something about the GIL's technicalities if you care about this area.

Once you have learned enough about the GIL, you have a Turing tarpit style puzzle of figuring out how to map your problem onto Python operations that are what I'll call GIL atomic, things that will never have races because the GIL is held across them. This sort of thing is like catnip for me and many programmers, especially since it requires deep domain knowledge. It make us feel smart, we're solving tough problems creatively and with real benefits (after all, this is the high performance option), and it means all of our domain knowledge gets put to nominally good use. It's simply fun to make CPython do what I want here, without the overhead and typing and work of setting up mutexes or managing locks or similar measures. It's magic; you do ordinary operations and things just work because you designed everything right.

(Then I can make myself feel even more clever by writing a long comment about how and why what I'm doing is perfectly safe.)

So that's my awkward confession about exploiting the GIL. It would be completely sensible to avoid doing so, but I'm probably not going to. Either I'll justify it on the grounds that this is a personal project and I'm allowed to have fun, or I'll justify it on the grounds of performance, but the truth is that I want to do it because it's neat and fun.

(If it was boring and tedious and took more work to arrange to exploit the GIL this way, you can bet that I'd be using mutexes and locks all the time.)

by cks at May 24, 2017 05:00 AM

May 23, 2017

Errata Security

Houston we have a problem!

Of the many undesirable results of the Space Program is the fetishization of the "mission control center", with it's rows of workstations facing a common central screen. Ever since, anybody with any sort of mission now has a similar control center.

It's a pain for us in the cybersecurity community because every organization wants a "security operations center" laid out the same way. The point of he room isn't to create something that's efficient for working, but one that will impress visitors. The things done to impress customers can often make an already difficult job even more difficult.

I point this out because of the "glowing globe" picture from President Trump's visit to Saudi Arabia. It's supposed to celebrate the opening of the "Global Center for Combating Extremist Ideology" ( Zoom the camera out a bit, and you can see it's the mission control center from hell.

Manually counting, I see three sides, each with slightly more than 100 workstations/employees, or more than 300 in total. I don't know if they intend all three sections to focus on the same sets of problems, or if they are split into three different tasks (e.g. broadcast TV vs. Internet content). Their brochure is unclear. I suspect in the long it'll be full of third country nations from a broad swath of Muslim nations who can speak the local languages and dialects, working in a sweat-shop manner.

In any case, it's clear that the desire for show/spectacle has far outstripped any practical use.

The more I read about this, the more Orwellian it seems. Rather than opposing ISIS's violence, it seems more intent on promoting a Saudi ideology. The whole spectacle seems intent on tricking the Trump administration into supporting something it really should be opposing.

by Robert Graham ( at May 23, 2017 01:46 AM

May 22, 2017

Everything Sysadmin

How difficult is it to enable HTTPS on a large website?

Very. The larger the site, the harder it becomes.

My co-worker Nick Craver just blogged about the multi-year journey to enable HTTPS at and all our sub-properties (spoiler alert: there is still more work to do!) It is probably the best detailed description of the process. If your boss ever asks you why HTTPS can't be enabled in just a few hours, this is a good resource.

by Tom Limoncelli at May 22, 2017 03:00 PM

Toolsmith #125: ZAPR - OWASP ZAP API R Interface

It is my sincere hope that when I say OWASP Zed Attack Proxy (ZAP), you say "Hell, yeah!" rather than "What's that?". This publication has been a longtime supporter, and so many brilliant contibutors and practitioners have lent to OWASP ZAPs growth, in addition to @psiinon's extraordinary project leadership. OWASP ZAP has been 1st or 2nd in the last four years of @ToolsWatch best tool survey's for a damned good reason. OWASP ZAP usage has been well documented and presented over the years, and the wiki gives you tons to consider as you explore OWASP ZAP user scenarios.
One of the more recent scenarios I've sought to explore recently is use of the OWASP ZAP API. The OWASP ZAP API is also well documented, more than enough detail to get you started, but consider a few use case scenarios.
First, there is a functional, clean OWASP ZAP API UI, that gives you a viewer's perspective as you contemplate programmatic opportunities. OWASP ZAP API interaction is URL based, and you can invoke both access views and invoke actions. Explore any component and you'll immediately find related views or actions. Drilling into to core via http://localhost:8067/UI/core/ (I run OWASP ZAP on 8067, your install will likely be different), gives me a ton to choose from.
You'll need your API key in order to build queries. You can find yours via Tools | Options | API | API Key. As an example, drill into numberOfAlerts (baseurl ), which gets the number of alerts, optionally filtering by URL. You'll then be presented with the query builder, where you can enter you key, define specific parameter, and decide your preferred output format including JSON, HTML, and XML.
Sure, you'll receive results in your browser, this query will provide answers in HTML tables, but these aren't necessarily optimal for programmatic data consumption and manipulation. That said, you learn the most important part of this lesson, a fully populated OWASP ZAP API GET URL: http://localhost:8067/HTML/core/view/numberOfAlerts/?zapapiformat=HTML&apikey=2v3tebdgojtcq3503kuoq2lq5g&formMethod=GET&baseurl=.
This request would return

in HTML. Very straightforward and easy to modify per your preferences, but HTML results aren't very machine friendly. Want JSON results instead? Just swap  out HTML with JSON in the URL, or just choose JSON in the builder. I'll tell you than I prefer working with JSON when I use the OWASP ZAP API via the likes of R. It's certainly the cleanest, machine-consumable option, though others may argue with me in favor of XML.
Allow me to provide you an example with which you can experiment, one I'll likely continue to develop against as it's kind of cool for active reporting on OWASP ZAP scans in flight or on results when session complete. Note, all my code, crappy as it may be, is available for you on GitHub. I mean to say, this is really v0.1 stuff, so contribute and debug as you see fit. It's also important to note that OWASP ZAP needs to be running, either with an active scanning session, or a stored session you saved earlier. I scanned my website,, and saved the session for regular use as I wrote this. You can even see reference to the saved session by location below.
R users are likely aware of Shiny, a web application framework for R, and its dashboard capabilities. I also discovered that rCharts are designed to work interactively and beautifully within Shiny.
R includes packages that make parsing from JSON rather straightforward, as I learned from Zev Ross. RJSONIO makes it as easy as fromJSON("http://localhost:8067/JSON/core/view/alerts/?zapapiformat=JSON&apikey=2v3tebdgojtcq3503kuoq2lq5g&formMethod=GET&baseurl=&start=&count=")
to pull data from the OWASP ZAP API. We use the fromJSON "function and its methods to read content in JSON format and de-serializes it into R objects", where the ZAP API URL is that content.
I further parsed alert data using Zev's grabInfo function and organized the results into a data frame (ZapDataDF). I then further sorted the alert content from ZapDataDF into objects useful for reporting and visualization. Within each alert objects are values such as the risk level, the alert message, the CWE ID, the WASC ID, and the Plugin ID. Defining each of these values into parameter useful to R is completed with the likes of:
I then combined all those results into another data frame I called reportDF, the results of which are seen in the figure below.
reportDF results
Now we've got some content we can pivot on.
First, let's summarize the findings and present them in their resplendent glory via ZAPR: OWASP ZAP API R Interface.
Code first, truly simple stuff it is:
Summary overview API calls

You can see that we're simply using RJSONIO's fromJSON to make specific ZAP API call. The results are quite tidy, as seen below.
ZAPR Overview
One of my favorite features in Shiny is the renderDataTable function. When utilized in a Shiny dashboard, it makes filtering results a breeze, and thus is utilized as the first real feature in ZAPR. The code is tedious, review or play with it from GitHub, but the results should speak for themselves. I filtered the view by CWE ID 89, which in this case is a bit of a false positive, I have a very flat web site, no database, thank you very much. Nonetheless, good to have an example of what would definitely be a high risk finding.

Alert filtering

Alert filtering is nice, I'll add more results capabilities as I develop this further, but visualizations are important too. This is where rCharts really come to bear in Shiny as they are interactive. I've used the simplest examples, but you'll get the point. First, a few, wee lines of R as seen below.
Chart code
The results are much more satisfying to look at, and allow interactivity. Ramnath Vaidyanathan has done really nice work here. First, OWASP ZAP alerts pulled via the API are counted by risk in a bar chart.
Alert counts

As I moused over Medium, we can see that there were specifically 17 results from my OWASP ZAP scan of
Our second visualization are the CWE ID results by count, in an oft disdained but interactive pie chart (yes, I have some work to do on layout).

CWE IDs by count

As we learned earlier, I only had one CWE ID 89 hit during the session, and the visualization supports what we saw in the data table.
The possibilities are endless to pull data from the OWASP ZAP API and incorporate the results into any number of applications or report scenarios. I have a feeling there is a rich opportunity here with PowerBI, which I intend to explore. All the code is here, along with the OWASP ZAP session I refer to, so you can play with it for yourself. You'll need OWASP ZAP, R, and RStudio to make it all work together, let me know if you have questions or suggestions.
Cheers, until next time.

by Russ McRee ( at May 22, 2017 06:35 AM

May 20, 2017

Sarah Allen

serverless patterns at google i/o 2017

At Google I/O last week, we presented how to build robust mobile applications for the distributed cloud about building mobile apps in this new world of “serverless development.” When people hear “serverless” sometimes they think we can write code on the server that is just like client-side code, but that’s not really the point.

We have many of the same concerns in developing the code that we always have when we write client-server sofware — we just don’t need to manage servers directly, which drastically reduces operational challenges. In this talk we dive into some specific software development patterns that help us write robust code that scales well.


  • Sarah Allen (me!) I lead the engineering team that works at the intersection of Firebase and Google Cloud on the server side.
  • Judy Tuan introduced me to Firebase over 5 years ago, when she led our team at AngelHack SF (on May 3, 2012) to build an iPhone app that would paint 3D shapes by waving your phone around using the accelerometer. That event happened to be Firebase’s first public launch, and she met Andrew Lee who convinced her to use Firebase in our app. She’s an independent software developer, and also working with Tech Workers Coalition.
  • Jen-Mei Wu is a software architect at Indiegogo, and also volunteers at Liberating Ourselves Locally, a people-of-color-led, gender-diverse, queer and trans inclusive hacker/maker space in East Oakland.

Jen-Mei kicked off the talk by introducing a use case for serverless development based on her work at the maker space, which often works to help non-profits. They are limited in their ability to deploy custom software because they work with small organizations who are staffed by non-technical folk. It doesn’t make sense to set them up with a need to devote resources to updating to the underlying software and operating systems needed to run a web application. With Firebase, the server software is automatically kept up to date with all the needed security patches for all of the required dependencies, it scales up when needed, and scales down to zero cost when not in active use.

The primary use case that motivated the talk from my perspective is for businesses that need to get started quickly, then scale up as usage grows. I found it inspiring to hear about how Firebase supports very small organizations with the same products and infrastructure that auto-scale to a global, distributed network supporting businesses with billions of users.

The concept for this talk was for some real-world developers (Judy and Jen-Mei) to envision an application that would illustrate common patterns in mobile application development, then I recruited a few Firebase engineers to join their open source team and we built the app together.

Severless Patterns

We identified some key needs that are common to mobile applications:

  • People use the app
  • The app stores data
  • The app show the data to people
  • There is some core business logic (“secret sauce”)
  • People interact with each other in some way

The talk details some of the core development patterns:

  • Server-side “special sauce”
  • Authentication & access control
  • Databinding to simplify UI development

The App: Hubbub

Ever go to a conference and feel like you haven’t connected to the people you would have liked to meet? This app seeks to solve that! Using your public GitHub data, we might be able to connect you with other people who share your technical interests. Maybe not, but at least you’ll have lunch with somebody.

You authenticate with GitHub, then the app creates a profile for you by pulling a lot of data from the GitHub API. Your profile might include languages you code in, a list of connections based on your pull request history or the other committers on projects that you have contributed to. Then there’s a list of events, which have a time and place, which people can sign up for. Before the event, people will be placed in groups with a topic they might be interested in. They show up at the appointed time and place, and then to find their assigned group, they open a beacon screen in the app which shows an image that is unique to their group (a pattern of one or more hubbubs with their topic name) and they find each other by holding up their phones.

We built most of the app, enough to really work through the key development patterns, but didn’t have time to hook up the profile generation data collection and implement a good matching algorithm. We ended up using a simple random grouping and were able to test the app at Google I/O for lunch on Friday. Some people showed up and it worked!

You can see the code at:

Many thanks to all the people who joined our open source team to develop the app:

and who helped with the presentation:

  • Virginia Poltrack, graphics
  • Yael Kazaz, public speaking, story-telling coach
  • Puf and Megan, rehearsal feedback
  • Todd Kerpelman our Google I/O mentor
  • Laura Nozay and all of the wonderful Google I/O staff who made the event possible

by sarah at May 20, 2017 05:01 PM

May 19, 2017

CentOS 7.4 to ship with TLS 1.2 + ALPN

The post CentOS 7.4 to ship with TLS 1.2 + ALPN appeared first on

Oh happy days!

I've long been tracking the "Bug 1276310 -- (rhel7-openssl1.0.2) RFE: Need OpenSSL 1.0.2" issue, where Red Hat users are asking for an updated version of the OpenSSL package. Mainly to get TLS 1.2 and ALPN.

_openssl_ rebased to version 1.0.2k

The _openssl_ package has been updated to upstream version 1.0.2k, which provides a number of enhancements, new features, and bug fixes, including:

* Added support for the datagram TLS (DTLS) protocol version 1.2.

* Added support for the TLS automatic elliptic curve selection.

* Added support for the Application-Layer Protocol Negotiation (ALPN).

* Added Cryptographic Message Syntax (CMS) support for the following schemes: RSA-PSS, RSA-OAEP, ECDH, and X9.42 DH.

Note that this version is compatible with the API and ABI in the *OpenSSL* library version in previous releases of Red Hat Enterprise Linux 7.
RFE: Need OpenSSL 1.0.2

The ALPN support is needed because in the Chrome browser, server-side ALPN support is a dependency to support HTTP/2. Without it, Chrome users don't get to use HTTP/2 on your servers.

The newly updated packages for OpenSSL are targeting the RHEL 7.4 release, which -- as far as I'm aware -- has no scheduled release date yet. But I'll be waiting for it!

As soon as RHEL 7.4 is released, we should expect a CentOS 7.4 release soon after.

The post CentOS 7.4 to ship with TLS 1.2 + ALPN appeared first on

by Mattias Geniar at May 19, 2017 01:56 PM

(Dutch) Tech45 podcast #341: Technologica & WannaCry ransomware

The post (Dutch) Tech45 podcast #341: Technologica & WannaCry ransomware appeared first on

I came as a guest on the tech45 podcast earlier this week, to talk about the WannaCry ransomware virus doing the rounds.

It's a Dutch podcast, so is the abstract below. The episode is available on the website or wherever you get your podcasts.

Mattias Geniar komt ons helpen om het Wannacry ransomware-verhaal uit te pluizen.

Alles over de Wannacry ransomware-aanval. "Wannacry is ransomware die een (recent gepatcht) lek in Microsoft Windows gebruikt om zich razendsnel te verspreiden via interne netwerken. Hij lijkt te worden verspreid via een massale mailingcampagne. Eens een machine is geïnfecteerd, scant ze het interne netwerk om collega's aan te steken. Die machines moeten daarvoor niet zelf met het internet verbonden zijn.", schrijven ze bij Datanews.

Source: Tech45 #341: Technologica

The post (Dutch) Tech45 podcast #341: Technologica & WannaCry ransomware appeared first on

by Mattias Geniar at May 19, 2017 07:45 AM

May 17, 2017

Everything Sysadmin

May 2017 NYCDevOps Meeting Notes

Here are my notes from last night's NYCDevOps meeting.

Title: Using innersourcing to break down organizational barriers
Speaker: Aroon Gursahaney, Verizon

This was one of the best talks I've seen in a while because it was entirely new material for me. I haven't heard of a company doing anything like this.

Last night I learned:

  • You can replace legacy systems by crowdsourcing parts of the project to people around the company in exchange for giving them the opportunity to learn new technologies, tools, and devops practices
  • You can gamify culture change in an organization
  • You cam make crowdsourcing the norm, not the exception.
  • Money motivates but a bigger motivation is interesting projects and the opportunity to learn, and to do something new.
  • 21 teams. 12 outside his VP's organization, 106 people participated.
  • In 2 days of the game, 2 months of progress was made on the project.
  • People took tasks not in their expertise areas, but where they could learn the most.
  • Seeing your code in production in other people's projects is a big source of pride, which is a big source of motivation.

Next month's speaker will be announced soon on the NYCDevOps meetup page.

by Tom Limoncelli at May 17, 2017 03:30 PM

May 15, 2017

Sean's IT Blog

Integrating Duo MFA and Amazon WorkSpaces

I recently did some work integrating Duo MFA with Amazon WorkSpaces.  The integration work ran into a few challenges, and I wanted to blog about those challenges to help others in the future.

Understanding Amazon WorkSpaces

If you couldn’t tell by the name, Amazon WorkSpaces is a cloud-hosted Desktop-as-a-Service offering that runs on Amazon Web Services (AWS).  It utilizes AWS’s underlying infrastructure to deploy desktop workloads, either using licensing provided by Amazon or the customer.  The benefit of this over on-premises VDI solutions is that Amazon manages the management infrastructure.  Amazon also provides multiple methods to integrate with the customer’s Active Directory environment, so users can continue to use their existing credentials to access the deployed desktops.

WorkSpaces uses an implementation of the PCoIP protocol for accessing desktops from the client application, and the client application is available on Windows, Linux, MacOS, iOS, and Android.  Amazon also has a web-based client that allows users to access their desktop from a web browser.  This feature is not enabled by default.

Anyway, that’s a very high-level overview of WorkSpaces.

Understanding How Users Connect to WorkSpaces

Before we can talk about how to integrate any multi-factor authentication solution into WorkSpaces, let’s go through the connection path and how that impacts how MFA is used.  WorkSpaces differs from traditional on-premises VDI in that all user connections to the desktop go through a public-facing service.  This happens even if I have a VPN tunnel or use DirectConnect.

The following image illustrates the connection path between the users and the desktop when a VPN connection is in place between the on-premises environment and AWS:


Image courtesy of AWS, retrieved from

As you can see from the diagram, on-premises users don’t connect to the WorkSpaces over the VPN tunnel – they do it over the Internet.  They are coming into their desktop from an untrusted connection, even if the endpoint they are connecting from is on a trusted network.

Note: The page that I retrieved this diagram from shows users connecting to WorkSpaces over a DirectConnect link when one is present.  This diagram shows a DirectConnect link that has a public endpoint, and all WorkSpaces connections are routed through this public endpoint.  In this configuration, logins are treated the same as logins coming in over the Internet.

When multi-factor authentication is configured for WorkSpaces, it is required for all user logins.  There is no way to configure rules to apply MFA to users connecting from untrusted networks because WorkSpaces sees all connections coming in on a public, Internet-facing service.

WorkSpaces MFA Requirements

WorkSpaces only works with MFA providers that support RADIUS.  The RADIUS servers can be located on-premises or in AWS.  I recommend placing the RADIUS proxies in another VPC in AWS where other supporting infrastructure resources, like AD Domain Controllers, are located.  These servers can be placed in the WorkSpaces VPC, but I wouldn’t recommend it.

WorkSpaces can use multiple RADIUS servers,and it will load-balance requests across all configured RADIUS servers.  RADIUS can be configured on both the Active Directory Connectors and the managed Microsoft AD directory services options. 

Note: The Duo documentation states that RADIUS can only be configured on the Active Directory Connector. Testing shows that RADIUS works on both enterprise WorkSpaces Directory Services options.

Configuring Duo

Before we can enable MFA in WorkSpaces, Duo needs to be configured.  At least one Duo proxy needs to be reachable from the WorkSpaces VPC.  This could be on a Windows or Linux EC2 instance in the WorkSpaces VPC, but it will most likely be an EC2 instance in a shared services or management VPC or in an on-premises datacenter.  The steps for installing the Duo Proxies are beyond the scope of this article, but they can be found in the Duo documentation.

Before the Duo Authentication Proxies are configured, we need to configure a new application in the Duo Admin console.  This step generates an integration key and secret key that will be used by the proxy and configure how new users will be handled.  The steps for creating the new application in Duo are:

1. Log into your Duo Admin Console at

2. Login as  your user and select the two-factor authentication method for your account.  Once you’ve completed the two-factor sign-on, you will have access to your Duo Admin panel.

3. Click on Applications in the upper left-hand corner.

4. Click on Protect an Application

5. Search for RADIUS.

6. Click Protect this Application  underneath the RADIUS application.


7. Record the Integration Key, Secret Key, and API Hostname.  These will be used when configuring the Duo Proxy in a few steps.


8. Provide a Name for the Application.  Users will see this name in their mobile device if they use Duo Push.


9. Set the New User Policy to Deny Access.  Deny Access is required because the WorkSpaces setup process will fail if Duo is configured to fail open or auto-enroll.


10. Click Save Changes to save the new application.

Get the AWS Directory Services IP Addresses

The RADIUS Servers need to know which endpoints to accept RADIUS requests from.  When WorkSpaces is configured for MFA, these requests come from the Directory Services instance.  This can either be the Active Directory Connectors or the Managed Active Directory domain controllers.

The IP addresses that will be required can be found in the AWS Console under WorkSpaces –> Directories.  The field to use depends on the directory service type you’re using.  If you’re using Active Directory Connectors, the IP addresses are in the Directory IP Address field.  If you’re using the managed Microsoft AD service, the Directory IP address field has a value of “None,” so you will need to use the  IP addresses in the DNS Address field.

Configure AWS Security Group Rules for RADIUS

By default, the AWS security group for the Directory Services instance heavily restrict the inbound and outbound traffic.  In order to enable RADIUS, you’ll need to add inbound and outbound UDP 1812 to the destination destination IPs or subnet where the proxies are located.

The steps for updating the AWS security group rules are:

1. Log into the AWS Console.

2. Click on the Services menu and select WorkSpaces.

3. Click on Directories.

4. Copy the value in the Directory ID field of the directory that will have RADIUS configured.

5. Click on the Services menu and select VPC.

6. Click on Security Groups.

7. Paste the Directory ID into the Search field to find the Security Group attached to the Directory Services instance.

8. Select the Security Group.

9. Click on the Inbound Rules tab.

10. Click Edit.

11. Click Add Another Rule.

12. Select Custom UDP Rule in the Type dropdown box.

13. Enter 1812 in the port range field.

14. Enter the IP Address of the RADIUS Server in the Source field.

15. Repeat Steps 11 through 14 as necessary to create inbound rules for each Duo Proxy.

16. Click Save.

17. Click on the Outbound Rules tab.

18. Click Edit.

19. Click Add Another Rule.

20. Select Custom UDP Rule in the Type dropdown box.

21. Enter 1812 in the port range field.

22. Enter the IP Address of the RADIUS Server in the Source field.

23. Repeat Steps 11 through 14 as necessary to create inbound rules for each Duo Proxy.

24. Click Save.

Configure the Duo Authentication Proxies

Once the Duo side is configured and the AWS security rules are set up, we need to configure our Authentication Proxy or Proxies.  This needs to be done locally on each Authentication Proxy.  The steps for configuring the Authentication Proxy are:

Note:  This step assumes the authentication proxy is running on Windows.The steps for installing the authentication proxy are beyond the scope of this article.  You can find the directions for installing the proxy at:

1. Log into your Authentication Proxy.

2. Open the authproxy.cfg file located in C:\Program Files (x86)\Duo Security Authentication Proxy\conf.

Note: The authproxy.cfg file uses the Unix newline setup, and it may require a different text editor than Notepad.  If you can install it on your server, I recommend Notepad++ or Sublime Text. 

3. Add the following lines to your config file.  This requires the Integration Key, Secret Key, and API Hostname from Duo and the IP Addresses of the WorkSpaces Active Directory Connectors or Managed Active Directory domain controllers.  Both RADIUS servers need to use the same RADIUS shared secret, and it should be a complex string.


ikey=<Integration Key>
skey=<Secret Key>
api_host=<API Hostname>
radius_ip_1=Directory Service IP 1
radius_secret_1=Secret Key
radius_ip_2=Directory Service IP 2
radius_secret_2=Secret Key

4. Save the configuration file.

5. Restart the Duo Authentication Proxy Service to apply the new configuration.

Integrating WorkSpaces with Duo

Now that our base infrastructure is configured, it’s time to set up Multi-Factor Authentication in WorkSpaces.  MFA is configured on the Directory Services instance.  If multiple directory services instances are required to meet different WorkSpaces use cases, then this configuration will need to be performed on each directory service instance. 

The steps for enabling WorkSpaces with Duo are:

1. Log into the AWS Console.

2. Click on the Services menu and select Directory Service.

3. Click on the Directory ID of the directory where MFA will be enabled.

4. Click the Multi-Factor authentication tab.

5. Click the Enable Multi-Factor Authentication checkbox.

6. Enter the IP address or addresses of your Duo Authentication Proxies in the RADIUS Server IP Address(es) field.

7. Enter the RADIUS Shared Secret in the Shared Secret Code and Confirm Shared Secret Code fields.

8. Change the Protocol to PAP.

9. Change the Server Timeout to 20 seconds.

10. Change the Max Retries to 3.

11. Click Update Directory.

12. If RADIUS setup is successful, the RADIUS Status should say Completed.


Using WorkSpaces with Duo

Once RADIUS is configured, the WorkSpaces client login prompt will change.  Users will be prompted for an MFA code along with their username and password.  This can be the one-time password generated in the mobile app, a push, or a text message with a one-time password.  If the SMS option is used, the user’s login will fail, but they will receive a text message with a one-time password. They will need to log in again using one of the codes received in the text message.



If the RADIUS setup does not complete successfully, there are a few things that you can check.  The first step is to verify that port 1812 is open on the Duo Authentication Proxies.  Also verify that the security group the directory services instance is configured to allow port 1812 inbound and outbound. 

One issue that I ran into in previous setups was the New User Policy in the Duo settings.  If it is not set to Deny Access, the process setup process will fail.  WorkSpaces is expecting a response code of Access Reject.  If it receives anything else, the process stalls out and fails about 15 minutes later. 

Finally, review the Duo Authentication Proxy logs.  These can be found in C:\Program Files (x86)\Duo Security Authentication Proxy\log on Authentication Proxies running Windows.  A tool like WinTail can be useful for watching the logs in real-time when trying to troubleshoot issues or verify that the RADIUS Services are functioning correctly.

by seanpmassey at May 15, 2017 06:08 PM

Everything Sysadmin

The Blog March: A personal drumbeat of #resistance

This post is part of the Blog March 2017. Every day this month a different blogger is writing about politics. Please march along with us!

A personal drumbeat of #resistance

A friend of mine recently commented that she'd like to get more involved in the resistance. She had attended a march or two, but those were occasional and spur of the moment activities. She asked if I had advice about how to turn her occasional activism into something more sustained? In other words, how could she make more of a difference but not quit her job and dedicate her life to the resistance?

I thought back to the 1990s when I was much more involved in politics. I was involved in multiple organizations but I still had a full time job. Was there something I could suggest? Activism is very different today than it was back then (we met in person and sometimes sent faxes!)

Times have changed but what works is still the same: Create a personal drumbeat of activism in your life.

A personal drumbeat sets that cadence for your work. It sets you up to do a little work each week. Over time this amounts to a lot of effort. However it doesn't feel like a lot because you've done a little at a time. More importantly, whatever you do becomes easier the more you repeat it.

For example, suppose you decided that your personal activism cadence would be to call 1 representative every Tuesday morning at 10am. The first time might be difficult, but the second time it a lot easier because you know who to call, and what to say. The third time you are a pro. Soon you can make two calls in the same amount of time. A call each week is going to generate more calls than if you make calls... when you think of it. Even a single call each week is better than, say, occasionally calling or planning to call but not actually doing it.

If everyone contributed 1 more hour per week to the resistance, we could achieve anything! This is very different than the old days before social networks. Back then efforts had to be more centralized (i.e. committees) because we didn't have mass communication tools like Facebook and Twitter. Now we can get more done with everyone doing little bits of work because it is easier to reach more people.

When has the resistance been successful?

If you look at Trump's first 100 days, you'll see that 100 percent of the resistance's victories involved an outpouring of people.

Public support is like oxygen for an issue. Without it the movement dies, with it the movement has a chance to see another day.

The huge numbers at The Women's March told Democrats in D.C. "we've got your back!" and gave them the confidence to put up a fight. Without their new-found backbones, they would not have stood up to Trump's policies at all.

The courts had the confidence to knock down the hate-fueled Muslim ban because crowds of people flooded airports and protested.

Trump failed to kill the ACA because of the outpouring of Democrats that showed up at townhalls and other venues as recommended by the Indivisible Handbook. Blue-state Republicans were terrified of not being reelected (basically the only thing that motivates them). In fact, the ACA repeat didn't even make it out of committee because the chair, NJ-11 Representative Rodney Frelinghuysen, has been dogged by constituants relentlessly (and by "relentlessly" I mean appropriately, politely, and never backing down). Keep up the good work, NJ11!

This brings me to 3 suggestions for setting up your personal drumbeat:

Technique 1: Set aside a specific time and place

Set aside the amount of time that is right for you. It can be five minutes each morning, once a week, or once a month. Something is better than nothing. Pick what is comfortable for you.

One day I realized that I wasn't making any progress on an important project. There was always a reason to put it off. I'd be invited to do something Monday night, and I'd say to myself, "It is ok. I'll work on that project on Tuesday." Tuesday I'd get home from work and be too tired to work on it. That's ok, I'll work on it Wednesday. You can guess where this is going. Soon it is Monday again and I've missed another week.

To fix this I set aside one night as my designated night just to work on that project. I picked Tuesday. If someone invited me out on Tuesday night, or if my boss asked if I could work late, I could honestly say, "no, I have a prior committment."

Nobody has time for big projects. People only make time for big projects. You have to say "no" to a lot of things to create the time for a big project. Creating an excuse (like, "Tuesday is my activism night") is one way to force yourself to create that time.

Think about committee meetings. If they meet weekly at the same time each week, it sets a cadence. It is difficult to find a night that everyone is available, but if the meeting repeats at a predictable cadence, everyone can make time for it.

If you are a procrastinator, the tasks you promise to do at one meeting tend to get done the night before the next meeting. By having the meetings weekly you get more work done than if they met monthly. If you have trouble setting aside time, maybe you need to join the organizing committee.

Technique 2: Be a body at events

Anti-Trump rallies and protests that have been very successful. The recipe is simple. A few people do a lot of planning and a lot of people show up. The goal is to create a big spectacle that looks impressive in photos and videos. That requires having a lot of people show up.

So... show up.

Believe it or not, they need a lot of people to "be a body". If you think of it like a movie, you are an "extra". I'm not saying that to belittle extras nor protesters. A movie with a big crowd scene would look silly if there was no crowd in the scene.

Rallies need a lot of bodies. Just show up and stand there. You don't need to prepare what you are going to say because you are part of the crowd... you don't need to talk! Heck, you don't even need to listen... just applaud when others do! (I'm only half serious there.)

Some townhalls do require RSVPs or have a limit on the number of people that can attend. RSVP if you can, but show up even if they run out of tickets. Just show up. If you don't get in, you can be part of the crowd that stands outside. Now the reporters have a second story to write about: the event plus the fact that there was an overflow crowd. ("Hey, why is that politician intentionally booking small rooms so that he doesn't have to face all of his constitutients! What is he trying to hide!?!"). That is the kind of spectacle you want to create.

"Being a body" is awesomely low-commitment. You don't need to get dressed up, do homework in advance, or nothin'. You don't have to make a sign. The people that do make signs probably have extras, or will be very glad if you hold their sign when their arms are tired.

If setting aside a specific time each week sounds like too big of a commitment, set a goal of "being a body" at every rally and meeting. Find your local indivisible group, put their events on your calendar, and just show up.

It is amazingly low-effort yet urgently needed. Oh and you might meet a neighbor or make a new friend.

Technique 3: Donate

If public support is a movement's oxygen, then money is the fuel. While grassroots organizing is less expensive than national activism, money keeps the movement moving.

Nobody can afford to donate in response to every request they get in email. It is much better to set a budget and stick to it.

Here's a strategy that I like. I donate the equivalent of 1 hour's wages each week. 1 hour out of 40 is 2.5 percent. Giving 2.5 percent to good causes is completely reasonable. (After taxes that's like 3.6 percent.)

How much do you make in one hour? If you are paid yearly instead of hourly, the math is quite simple. There are approximately 2,000 work hours in the year. Dividing by 2,000 is like lopping off the 3 zeros at the end of your salary and dividing by two. So, if you make $40,000 per year, take off the 3 zeros (leaving you with "40") and divide by 2 (leaving you with "20"). So, if you donate $20/week to a cause that's one hour's of wages.

Because I'm lazy, I don't want to have to click on a group's website once a week. Instead, I pick 4 charities and set the all up for automatic monthly donations.

Organizations love automatic donations. They can budget around it. They can make big, long-term, plans. Imagine if your car had a 1-gallon gas tank and you had to spend all your time worrying about whether or not you'll see a gas station in the next 20-30 miles. You would never plan big trips. Organizations need monthly donors so they have the confidence to make big plans... like taking back the house and senate. The right-wing knows this and uses long-term funding strategies from billionaries to fund things like national gerrymandering strategies and ALEC. We won't have a left-wing response to either without a guarantee of long-term funding by millions of grassroots people like yourself.

I also love automatic donations because I'm lazy and, to be honest, if I can "set it and forget it" then I'm totally happy. (And I get to delete all other fundraising emails without guilt!)

There's another trick I use related to money. Remember how I mentioned I donate the equivalent of 1 hour's wages each week? I actually have a specific hour! It's Wednesday from 10am to 11am! The money I earn during that time is what I donate.

Ok, that's totally not true. Money is money and I don't know which specific George Washingtons I earn during that hour. However I pretend that it is the money I earn on Wednesday from 10am to 11am!

Why? Because it is fun. If I'm working hard during that hour, or attending a painfully boring meeting, or even being yelled at by my boss (not really... my boss doesn't yell at me), I can smile and know that this is the hour that I'm helping save the world.

It is also a great conversation starter. "Guess what I'm doing right now? I'm saving the world."

And that's pretty cool.

(I won't reveal which organizations my 4 donations go to, but here are a few very worthy causes: NDRC (gerrymandering), ACLU, DNC LGBT Caucus, Planned Parenthood Action Fund, NGLTF, BiNetUSA,

"A personal drumbeat of #resistance" is a part of Blog March 2017, a movement for Raising Voices for Freedom of Expression, Knowledge, and Information. We will be broadcasting voices throughout the month of May. Follow up on May 16th with Diana Adams!

by Tom Limoncelli at May 15, 2017 10:00 AM

May 14, 2017

Steve Kemp's Blog

Some minor updates ..

The past few weeks have been randomly busy, nothing huge has happened, but several minor diversions.


I made a new release of my console-based mail-client, with integrated Lua scripting, this is available for download over at

I've also given a talk (!!) on using a literate/markdown configuration for GNU Emacs. In brief I created two files:


This contains both my configuration of GNU Emacs as well as documentation for the same. Neat.


This parse the previous file, specifically looking for "code blocks" which are then extracted and evaluated.

This system is easy to maintain, and I'm quite happy with it :)


Somebody nice took the time to report a couple of bugs against my simple bytecode-intepretting virtual-machine project - all found via fuzzing.

I've done some fun fuzzing of my own in the past, so this was nice to see. I've now resolved those bugs, and updated the file to include instructions on fuzzing it. (Which I started doing myself, after receiving the first of the reports )

Finally I have more personal news too: I had a pair of CT-scans carried out recently, and apparently here in sunny Finland (that's me being ironic, it was snowing in the first week of May) when you undergo a CT-scan you can pay to obtain your data on CD-ROM.

I'm 100% definitely going to get a copy of my brain-scan data. I'll be able to view a 3d-rendered model of my own brain on my desktop. (Once upon a time I worked for a company that produced software, sold to doctors/surgeons, for creating 3d-rendered volumes from individual slices. I confirmed with the radiologist that handled my tests that they do indeed use the standard DICOM format. Small world.)

May 14, 2017 09:00 PM


Estimates and compounding margin of error

The two sides of this story are:

  • Software requirements are often... mutable over time.
  • Developer work-estimates are kind of iffy.

Here is a true story about the later. Many have similar tales to tell, but this one is mind. It's about metrics, agile, and large organizations. If you're expecting this to turn into a frAgile post, you're not wrong. But it's about failure-modes.

The setup

My employer at the time decided to go all-in on agile/scrum. It took several months, but every department in the company was moved to it. As they were an analytics company, their first reflex was to try to capture as much workflow data as possible so they can do magic data-analysis things on it. Which meant purchasing seats in an Agile SaaS product so everyone could track their work in it.

The details

By fiat from on-high, story points were effectively 'days'.

Due to the size of the development organization, there were three levels of Product Managers over the Individual Contributors I counted myself a part of.

The apex Product Manager, we had two, were for the two products we sold.

Marketing was also in Agile, as was Sales.

The killer feature

Because we were an analytics company, the CEO wanted a "single pane of glass" to give a snapshot of how close we were to achieving our product goals. Gathering metrics on all of our sprint-velocities, story/epic completion percentages, and story estimates, allowed us to give him that pane of glass. It had:

  • Progress bars for how close our products were to their next major milestones.
  • How many sprints it will take to get there.
  • How many sprints it would take to get to the milestone beyond it.


The failure

That pane of glass was a lying piece of shit.

The dashboard we had to build was based on so many fuzzy measurements that it was a thumb in the wind approximation for how fast we were going, and in what direction. The human bias to trust numbers derived using Science! is a strong one, and they were inappropriately trusted. Which lead to pressure from On High for highly accurate estimates, as the various Product Managers realized what was going on and attempted to compensate (remove uncertainty from one of the biggest sources of it).

Anyone who has built software knows that problems come in three types:

  1. Stuff that was a lot easier than you thought
  2. Stuff that was pretty much as bad as you thought.
  3. Hell-projects of tar-pits, quicksand, ambush-yaks, and misery.

In the absence of outside pressures, story estimates usually are pitched at type 2 efforts; the honest estimate. Workplace culture introduces biases to this, urging devs to skew one way or the other. Skew 'easier', and you'll end up overshooting your estimates a lot. Skew 'harder' and your velocity will look great, but capacity planning will suffer.

This leads to an interesting back and forth! Dev-team skews harder for estimates. PM sees that team typically exceeds its capacity in productivity, so adds more capacity in later sprints. In theory equilibrium is reached between work-estimation and work-completion-rate. In reality, it means that the trustability of number is kind of low and always will be.

The irreducible complexity

See, the thing is, marketing and sales both need to know when a product will be released so they can kick off marketing campaigns and start warming up the sales funnel. Some kinds of ad-buys are done weeks or more in advance, so slipping product-ship at the last minute can throw off the whole marketing cadence. Trusting in (faulty) numbers means it may look like release will be in 8 weeks, so its safe to start baking that in.

Except those numbers aren't etched in stone. They're graven in the finest of morning dew.

As that 8 week number turns into 6, then 4, then 2, pressure to hit the mark increases. For a company selling on-prem software, you can afford to miss your delivery deadline so long as you have a hotfix/service-pack process in place to deliver stability updates quickly. You see this a lot with game-dev: the shipping installer is 8GB, but there are 2GB of day-1 patches to download before you can play. SaaS products need to work immediately on release, so all-nighters may become the norm for major features tied to marketing campaigns.

Better estimates would make this process a lot more trustable. But, there is little you can do to actually improve estimate quality.

by SysAdmin1138 at May 14, 2017 03:30 PM

May 13, 2017

Anton Chuvakin - Security Warrior

Monthly Blog Round-Up – April 2017

Here is my next monthly "Security Warrior" blog round-up of top 5 popular posts/topics this month:
  1. “New SIEM Whitepaper on Use Cases In-Depth OUT!” (dated 2010) presents a whitepaper on select SIEM use cases described in depth with rules and reports [using now-defunct SIEM product]; also see this SIEM use case in depth and this for a more current list of popular SIEM use cases. Finally, see our 2016 research on developing security monitoring use cases here!
  2. Why No Open Source SIEM, EVER?” contains some of my SIEM thinking from 2009. Is it relevant now? You be the judge.  Succeeding with SIEM requires a lot of work, whether you paid for the software, or not. BTW, this post has an amazing “staying power” that is hard to explain – I suspect it has to do with people wanting “free stuff” and googling for “open source SIEM” … 
  3. This month, my classic PCI DSS Log Review series is extra popular! The series of 18 posts cover a comprehensive log review approach (OK for PCI DSS 3+ even though it predates it), useful for building log review processes and procedures, whether regulatory or not. It is also described in more detail in our Log Management book and mentioned in our PCI book (now in its 4th edition!) – note that this series is mentioned in some PCI Council materials.
  4. Simple Log Review Checklist Released!” is often at the top of this list – this aging checklist is still a very useful tool for many people. “On Free Log Management Tools” (also aged a bit by now) is a companion to the checklist (updated version)
  5. “SIEM Resourcing or How Much the Friggin’ Thing Would REALLY Cost Me?” is a quick framework for assessing the SIEM project (well, a program, really) costs at an organization (a lot more details on this here in this paper).
In addition, I’d like to draw your attention to a few recent posts from my Gartner blog [which, BTW, now has about 5X of the traffic of this blog]: 
Current research on cloud security monitoring:
Recent research on security analytics and UBA / UEBA:
EPIC WIN post:
Miscellaneous fun posts:

(see all my published Gartner research here)
Also see my past monthly and annual “Top Popular Blog Posts” – 2007, 2008, 2009, 2010, 2011, 2012, 2013, 2014, 2015, 2016.

Disclaimer: most content at SecurityWarrior blog was written before I joined Gartner on August 1, 2011 and is solely my personal view at the time of writing. For my current security blogging, go here.

Previous post in this endless series:

by Anton Chuvakin ( at May 13, 2017 12:16 AM

Monthly Blog Round-Up – February 2017

Here is my next monthly "Security Warrior" blog round-up of top 5 popular posts/topics this month:
  1. “New SIEM Whitepaper on Use Cases In-Depth OUT!” (dated 2010, so I have no idea why it tops the charts now!) presents a whitepaper on select SIEM use cases described in depth with rules and reports [using now-defunct SIEM product]; also see this SIEM use case in depth and this for a more current list of popular SIEM use cases. Finally, see our 2016 research on developing security monitoring use cases here!
  2. Why No Open Source SIEM, EVER?” contains some of my SIEM thinking from 2009. Is it relevant now? You be the judge.  Succeeding with SIEM requires a lot of work, whether you paid for the software, or not. BTW, this post has an amazing “staying power” that is hard to explain – I suspect it has to do with people wanting “free stuff” and googling for “open source SIEM” … 
  3. Simple Log Review Checklist Released!” is often at the top of this list – this aging checklist is still a very useful tool for many people. “On Free Log Management Tools” (also aged a bit by now) is a companion to the checklist (updated version)
  4. This month, my classic PCI DSS Log Review series is extra popular! The series of 18 posts cover a comprehensive log review approach (OK for PCI DSS 3+ even though it predates it), useful for building log review processes and procedures, whether regulatory or not. It is also described in more detail in our Log Management book and mentioned in our PCI book (now in its 4th edition!) – note that this series is mentioned in some PCI Council materials.
  5. “SIEM Resourcing or How Much the Friggin’ Thing Would REALLY Cost Me?” is a quick framework for assessing the SIEM project (well, a program, really) costs at an organization (a lot more details on this here in this paper).
In addition, I’d like to draw your attention to a few recent posts from my Gartner blog [which, BTW, now has about 5X of the traffic of this blog]: 
Recent research on security analytics and UBA / UEBA:
Miscellaneous fun posts:
(see all my published Gartner research here)
Also see my past monthly and annual “Top Popular Blog Posts” – 2007, 2008, 2009, 2010, 2011, 2012, 2013, 2014, 2015, 2016.
Disclaimer: most content at SecurityWarrior blog was written before I joined Gartner on August 1, 2011 and is solely my personal view at the time of writing. For my current security blogging, go here.
Previous post in this endless series:

by Anton Chuvakin ( at May 13, 2017 12:08 AM

May 12, 2017

Errata Security

Some notes on Trump's cybersecurity Executive Order

President Trump has finally signed an executive order on "cybersecurity". The first draft during his first weeks in power were hilariously ignorant. The current draft, though, is pretty reasonable as such things go. I'm just reading the plain language of the draft as a cybersecurity expert, picking out the bits that interest me. In reality, there's probably all sorts of politics in the background that I'm missing, so I may be wildly off-base.

Holding managers accountable

This is a great idea in theory. But government heads are rarely accountable for anything, so it's hard to see if they'll have the nerve to implement this in practice. When the next breech happens, we'll see if anybody gets fired.

"antiquated and difficult to defend Information Technology"

The government uses laughably old computers sometimes. Forces in government wants to upgrade them. This won't work. Instead of replacing old computers, the budget will simply be used to add new computers. The old computers will still stick around.

"Legacy" is a problem that money can't solve. Programmers know how to build small things, but not big things. Everything starts out small, then becomes big gradually over time through constant small additions. What you have now is big legacy systems. Attempts to replace a big system with a built-from-scratch big system will fail, because engineers don't know how to build big systems. This will suck down any amount of budget you have with failed multi-million dollar projects.

It's not the antiquated systems that are usually the problem, but more modern systems. Antiquated systems can usually be protected by simply sticking a firewall or proxy in front of them.

"address immediate unmet budgetary needs necessary to manage risk"

Nobody cares about cybersecurity. Instead, it's a thing people exploit in order to increase their budget. Instead of doing the best security with the budget they have, they insist they can't secure the network without more money.

An alternate way to address gaps in cybersecurity is instead to do less. Reduce exposure to the web, provide fewer services, reduce functionality of desktop computers, and so on. Insisting that more money is the only way to address unmet needs is the strategy of the incompetent.

Use the NIST framework

Probably the biggest thing in the EO is that it forces everyone to use the NIST cybersecurity framework.

The NIST Framework simply documents all the things that organizations commonly do to secure themselves, such run intrusion-detection systems or impose rules for good passwords.

There are two problems with the NIST Framework. The first is that no organization does all the things listed. The second is that many organizations don't do the things well.

Password rules are a good example. Organizations typically had bad rules, such as frequent changes and complexity standards. So the NIST Framework documented them. But cybersecurity experts have long opposed those complex rules, so have been fighting NIST on them.

Another good example is intrusion-detection. These days, I scan the entire Internet, setting off everyone's intrusion-detection systems. I can see first hand that they are doing intrusion-detection wrong. But the NIST Framework recommends they do it, because many organizations do it, but the NIST Framework doesn't demand they do it well.

When this EO forces everyone to follow the NIST Framework, then, it's likely just going to increase the amount of money spent on cybersecurity without increasing effectiveness. That's not necessarily a bad thing: while probably ineffective or counterproductive in the short run, there might be long-term benefit aligning everyone to thinking about the problem the same way.

Note that "following" the NIST Framework doesn't mean "doing" everything. Instead, it means documented how you do everything, a reason why you aren't doing anything, or (most often) your plan to eventually do the thing.

preference for shared IT services for email, cloud, and cybersecurity

Different departments are hostile toward each other, with each doing things their own way. Obviously, the thinking goes, that if more departments shared resources, they could cut costs with economies of scale. Also obviously, it'll stop the many home-grown wrong solutions that individual departments come up with.

In other words, there should be a single government GMail-type service that does e-mail both securely and reliably.

But it won't turn out this way. Government does not have "economies of scale" but "incompetence at scale". It means a single GMail-like service that is expensive, unreliable, and in the end, probably insecure. It means we can look forward to government breaches that instead of affecting one department affecting all departments.

Yes, you can point to individual organizations that do things poorly, but what you are ignoring is the organizations that do it well. When you make them all share a solution, it's going to be the average of all these things -- meaning those who do something well are going to move to a worse solution.

I suppose this was inserted in there so that big government cybersecurity companies can now walk into agencies, point to where they are deficient on the NIST Framework, and say "sign here to do this with our shared cybersecurity service".

"identify authorities and capabilities that agencies could employ to support the cybersecurity efforts of critical infrastructure entities"

What this means is "how can we help secure the power grid?".

What it means in practice is that fiasco in the Vermont power grid. The DHS produced a report containing IoCs ("indicators of compromise") of Russian hackers in the DNC hack. Among the things it identified was that the hackers used Yahoo! email. They pushed these IoCs out as signatures in their "Einstein" intrusion-detection system located at many power grid locations. The next person that logged into their Yahoo! email was then flagged as a Russian hacker, causing all sorts of hilarity to ensue, such as still uncorrected stories by the Washington Post how the Russians hacked our power-grid.

The upshot is that federal government help is also going to include much government hindrance. They really are this stupid sometimes and there is no way to fix this stupid. (Seriously, the DHS still insists it did the right thing pushing out the Yahoo IoCs).

Resilience Against Botnets and Other Automated, Distributed Threats

The government wants to address botnets because it's just the sort of problem they love, mass outages across the entire Internet caused by a million machines.

But frankly, botnets don't even make the top 10 list of problems they should be addressing. Number #1 is clearly "phishing" -- you know, the attack that's been getting into the DNC and Podesta e-mails, influencing the election. You know, the attack that Gizmodo recently showed the Trump administration is partially vulnerable to. You know, the attack that most people blame as what probably led to that huge OPM hack. Replace the entire Executive Order with "stop phishing", and you'd go further fixing federal government security.

But solving phishing is tough. To begin with, it requires a rethink how the government does email, and how how desktop systems should be managed. So the government avoids complex problems it can't understand to focus on the simple things it can -- botnets.

Dealing with "prolonged power outage associated with a significant cyber incident"

The government has had the hots for this since 2001, even though there's really been no attack on the American grid. After the Russian attacks against the Ukraine power grid, the issue is heating up.

Nation-wide attacks aren't really a threat, yet, in America. We have 10,000 different companies involved with different systems throughout the country. Trying to hack them all at once is unlikely. What's funny is that it's the government's attempts to standardize everything that's likely to be our downfall, such as sticking Einstein sensors everywhere.

What they should be doing is instead of trying to make the grid unhackable, they should be trying to lessen the reliance upon the grid. They should be encouraging things like Tesla PowerWalls, solar panels on roofs, backup generators, and so on. Indeed, rather than industrial system blackout, industry backup power generation should be considered as a source of grid backup. Factories and even ships were used to supplant the electric power grid in Japan after the 2011 tsunami, for example. The less we rely on the grid, the less a blackout will hurt us.

"cybersecurity risks facing the defense industrial base, including its supply chain"

So "supply chain" cybersecurity is increasingly becoming a thing. Almost anything electronic comes with millions of lines of code, silicon chips, and other things that affect the security of the system. In this context, they may be worried about intentional subversion of systems, such as that recent article worried about Kaspersky anti-virus in government systems. However, the bigger concern is the zillions of accidental vulnerabilities waiting to be discovered. It's impractical for a vendor to secure a product, because it's built from so many components the vendor doesn't understand.

"strategic options for deterring adversaries and better protecting the American people from cyber threats"

Deterrence is a funny word.

Rumor has it that we forced China to backoff on hacking by impressing them with our own hacking ability, such as reaching into China and blowing stuff up. This works because the Chinese governments remains in power because things are going well in China. If there's a hiccup in economic growth, there will be mass actions against the government.

But for our other cyber adversaries (Russian, Iran, North Korea), things already suck in their countries. It's hard to see how we can make things worse by hacking them. They also have a strangle hold on the media, so hacking in and publicizing their leader's weird sex fetishes and offshore accounts isn't going to work either.

Also, deterrence relies upon "attribution", which is hard. While news stories claim last year's expulsion of Russian diplomats was due to election hacking, that wasn't the stated reason. Instead, the claimed reason was Russia's interference with diplomats in Europe, such as breaking into diplomat's homes and pooping on their dining room table. We know it's them when they are brazen (as was the case with Chinese hacking), but other hacks are harder to attribute.

Deterrence of nation states ignores the reality that much of the hacking against our government comes from non-state actors. It's not clear how much of all this Russian hacking is actually directed by the government. Deterrence polices may be better directed at individuals, such as the recent arrest of a Russian hacker while they were traveling in Spain. We can't get Russian or Chinese hackers in their own countries, so we have to wait until they leave.

Anyway, "deterrence" is one of those real-world concepts that hard to shoe-horn into a cyber ("cyber-deterrence") equivalent. It encourages lots of bad thinking, such as export controls on "cyber-weapons" to deter foreign countries from using them.

"educate and train the American cybersecurity workforce of the future"

The problem isn't that we lack CISSPs. Such blanket certifications devalue the technical expertise of the real experts. The solution is to empower the technical experts we already have.

In other words, mandate that whoever is the "cyberczar" is a technical expert, like how the Surgeon General must be a medical expert, or how an economic adviser must be an economic expert. For over 15 years, we've had a parade of non-technical people named "cyberczar" who haven't been experts.

Once you tell people technical expertise is valued, then by nature more students will become technical experts.

BTW, the best technical experts are software engineers and sysadmins. The best cybersecurity for Windows is already built into Windows, whose sysadmins need to be empowered to use those solutions. Instead, they are often overridden by a clueless cybersecurity consultant who insists on making the organization buy a third-party product instead that does a poorer job. We need more technical expertise in our organizations, sure, but not necessarily more cybersecurity professionals.


This is really a government document, and government people will be able to explain it better than I. These are just how I see it as a technical-expert who is a government-outsider.

My guess is the most lasting consequential thing will be making everyone following the NIST Framework, and the rest will just be a lot of aspirational stuff that'll be ignored.

by Robert Graham ( at May 12, 2017 06:51 AM

May 11, 2017

Evaggelos Balaskas

time offset and nagios nrpe check

What is the time?

Time offset is the amount of time that is off (or drift) from a specific value. In Linux systems, date is been calculating from the beginning of time. That is 00:00:00 1 January 1970 or as it called Unix Time and systems define date (time) as the number of seconds that have elapsed from 01.01.1970.

It is so important that even a few seconds off can cause tremendous disaster in data centers and applications.

Network Time

To avoid problems with time, systems must and should synchronize their time over the Internet every now and then. This is being done by asking a central NTP server via Network Time Protocol. The most common scenario for infrastructures is to have one or two NTP servers and then all the systems inside this infrastructure can synchronize their time from those machines.

Nagios - NRPE

In my case, I have a centralized NTP Daemon that runs on the Nagios Linux machine. That gives me the opportunity to check the EPOCH time of any system in my infrastructure against the time that the Nagios Server has.

Nagios Check

This is the script I am using:

# ebal, Thu, 11 May 2017 12:08:50 +0000


# seconds
OFFSET=$( echo $(( $(date -d 'now ' +%s) - ${TIME} )) | sed -e 's#-##g' )

if [ "${OFFSET}" -lt "${WARN}" ]; then
        echo "OK"
        exit 0
elif [ "${OFFSET}" -ge "${CRIT}" ]; then
        echo "CRITICAL- ${OFFSET}"
        exit 2
elif [ "${OFFSET}" -lt "${CRIT}" ]; then
        echo "WARNING- ${OFFSET}"
        exit 1
        echo "UNKNOWN- ${OFFSET}"
        exit 3

In a nutshell the script gets as the first argument an epoch time and calculate the diff between it’s own epoch time and that.


./check_time_offset $(date -d 'now + 1 min' +%s)

The output is this:


Nrpe Configuration

This is the configuration for nrpe to run the check_time_offset

# tail -1 /etc/nrpe.d/time_offset.cfg

command[check_time_offset]=/usr/lib64/nagios/plugins/check_time_offset $ARG1$

Nagios Configuration

and this is my nagios configuration setup to use a remote nrpe :

define service{
        use                             service-critical
        hostgroup_name                  lnxserver01
        service_description             time_offset
        check_command                   check_nrpe!check_time_offset!$TIMET$

Take a minute to observer a little better the nrpe command.



I was having problems passing the nagios epoch time as an argument on the definition of the above service.

Testing the nrpe command as below, I was getting the results I was looking for:

./check_nrpe -H lnxserver01 -c check_time_offset -a $(date -d 'now + 6 sec' +%s)

But is there a way to pass as a nagios argument the output of a command ?

  • No

A dear colleague of mine mentioned nagios macros:

Standard Macros in Nagios

$TIMET$     Current time stamp in time_t format (seconds since the UNIX epoch)

Perfect !!!

Tag(s): nagios, nrpe, ntp, time

May 11, 2017 05:18 PM

Colin Percival

A plan for open source software maintainers

I've been writing open source software for about 15 years now; while I'm still wet behind the ears compared to FreeBSD greybeards like Kirk McKusick and Poul-Henning Kamp, I've been around for long enough to start noticing some patterns. In particular:
  • Free software is expensive. Software is expensive to begin with; but good quality open source software tends to be written by people who are recognized as experts in their fields (partly thanks to that very software) and can demand commensurate salaries.
  • While that expensive developer time is donated (either by the developers themselves or by their employers), this influences what their time is used for: Individual developers like doing things which are fun or high-status, while companies usually pay developers to work specifically on the features those companies need. Maintaining existing code is important, but it is neither fun nor high-status; and it tends to get underweighted by companies as well, since maintenance is inherently unlikely to be the most urgent issue at any given time.
  • Open source software is largely a "throw code over the fence and walk away" exercise. Over the past 15 years I've written freebsd-update, bsdiff, portsnap, scrypt, spiped, and kivaloo, and done a lot of work on the FreeBSD/EC2 platform. Of these, I know bsdiff and scrypt are very widely used and I suspect that kivaloo is not; but beyond that I have very little knowledge of how widely or where my work is being used. Anecdotally it seems that other developers are in similar positions: At conferences I've heard variations on "you're using my code? Wow, that's awesome; I had no idea" many times.
  • I have even less knowledge of what people are doing with my work or what problems or limitations they're running into. Occasionally I get bug reports or feature requests; but I know I only hear from a very small proportion of the users of my work. I have a long list of feature ideas which are sitting in limbo simply because I don't know if anyone would ever use them — I suspect the answer is yes, but I'm not going to spend time implementing these until I have some confirmation of that.
  • A lot of mid-size companies would like to be able to pay for support for the software they're using, but can't find anyone to provide it. For larger companies, it's often easier — they can simply hire the author of the software (and many developers who do ongoing maintenance work on open source software were in fact hired for this sort of "in-house expertise" role) — but there's very little available for a company which needs a few minutes per month of expertise. In many cases, the best support they can find is sending an email to the developer of the software they're using and not paying anything at all — we've all received "can you help me figure out how to use this" emails, and most of us are happy to help when we have time — but relying on developer generosity is not a good long-term solution.
  • Every few months, I receive email from people asking if there's any way for them to support my open source software contributions. (Usually I encourage them to donate to the FreeBSD Foundation.) Conversely, there are developers whose work I would like to support (e.g., people working on FreeBSD wifi and video drivers), but there isn't any straightforward way to do this. Patreon has demonstrated that there are a lot of people willing to pay to support what they see as worthwhile work, even if they don't get anything directly in exchange for their patronage.

May 11, 2017 04:10 AM

May 10, 2017

Villagers with Pitchforks

What You Should Know – Air Conditioning Repair 101

To so many homeowners, owning an air conditioner is not only a luxury but also a huge necessity. This important unit keeps your home cool and comfortable during the extremely hot season. It is, therefore, your responsibility as a homeowner, of ensuring that the air circulating your house remains cool and is of good quality always, by watching out for your air conditioning system. When your AC unit starts to show signs of damage, inefficiency or malfunction, the appropriate repairs ought to be done before the problem escalates into something bigger and much more costly. As a matter of fact, delaying air conditioning repairs will only lead to more problems, costlier damages, jeopardized comfort, higher energy bills, and reduced equipment lifetime. If you own an AC system in your home, below are some pointers you might want to have a look at in our article, air conditioning repair 101: what homeowners should have in mind.

How Your Air Conditioner Works

Your air conditioner cools the air around your home by removing the heat from the warm air. Warm air is sucked from the indoor space and blown through a set of pipes in the AC called the evaporator coils. The evaporator coils are filled with a liquid called a refrigerant. After absorbing heat from the air, this liquid changes state to gas and is then pumped to the condenser component of the air conditioning system. In the condenser, the absorbed heat is released into the outdoor air while the refrigerant goes back into liquid state and the cycle continues. Most air conditioners have a thermostat, which is the device that regulates the amount and temperature of cool air supplied throughout your home.

A Word about Air Filters And Your Air Conditioner

These are found in the ductwork system or other strategic points within the HVAC system. They help in removing particles from the air both to keep the air conditioner system clean and the air supplied from the unit. As the process causes the air filters to get loaded with particles, the air filters may become loaded clogged with debris, which may consequently lead to a reduction in air flow. This can, in turn, pave the way to energy inefficiency, potentially leading to high electricity bills, increased frequency of repairs, and a reduction in the equipment’s lifetime. This means that regular air filter replacement is not only an important aspect of central air conditioning system maintenance but also plays a huge part in preventing the need for air conditioning repairs in the first place.

Costly Air Conditioner Repairs Are Usually Preventable

Some air conditioning repairs such as condenser replacement can be highly expensive. One of the ways you can prevent costly AC repairs is by regularly cleaning your filters, cleaning your condensate drains, and checking out for warning signs of a bad or malfunctioning air conditioning unit. Scheduling annual professional maintenance can also be a helpful way, especially with contractors that a reputed for preventive maintenance services.

Choose Your AC Repair Contractor Wisely

Having as much information as possible on DYI tips and what to do when something goes wrong is overly important. However, knowing how to hire an experienced, professional air conditioning repair contractor is even more important. This is because a reputed HVAC expert is well trained and qualified to handle the specific job without hitches and will be able to detect the most complex and hidden problems. When choosing an HVAC service contractor, always be on the lookout for common scams in the air HVAC industry and be prepared on how to avoid them.

According to the manager of the Altitude Comfort Heating and Air – air conditioning repair Denver division, choosing the right HVAC contractor is the absolute most important thing to focus on when you need HVAC service. “Everything else will take care of itself if you choose a really good company to take care of your problem.”

The post What You Should Know – Air Conditioning Repair 101 appeared first on The Keeping Room.

by Level at May 10, 2017 08:56 PM

How To Become An Air Conditioning Repair Tech In Colorado

Colorado is one of the few states that doesn’t issue licenses for HVAC work. While there’s no state-wide license you need to attain, it’s still a good idea to become certified by a reputable trade school before starting work. Not only will you meet national, city, and county requirements, you’ll also have the ability to prove to your employer and clients that you’ve got the skills necessary to work as an HVAC technician.

Local Requirements

Colorado doesn’t have a state-wide license requirement. This doesn’t mean you can legally perform HVAC work without any certification anywhere in the state. It’s up to the local government of each city and county to determine what licenses are required for aspiring technicians.

Notably, Denver, CO has fairly strict requirements when it comes to HVAC licenses. All journeyman licenses require passing an ICC test in a related subject and several years of work in the field. While these licenses aren’t required to perform work, they’re very highly sought after by employers and potential clients.

Licenses are required to perform work in Colorado Springs and the surrounding parts of El Paso County. Applicants have to pass a background check and demonstrate proficiency by passing an exam. Unlike Denver, you can be fined or jailed if you’re caught performing work without an El Paso license. You can find the specific requirements for the Denver, CO Heating, ventilating, A/C Only Supervisor’s Certificate online.

National Qualifications

Students at trade schools in Colorado often focus on getting a certificate that holds value across the country. Often, the first step involves getting certified under section 608 of the EPA. This certification is mandatory for anyone who works with appliances that contain refrigerants — in other words, you need it to do maintenance on air conditioners, and you’ll definitely have to have it to perform air conditioning repair service.

Additionally, many students prepare for various ICC exams for HVAc technicians. These nationwide standards are the basis for many local, state and national certificates. Mastering the material covered by ICC exams gives you the freedom to become certified almost anywhere with just a little bit of review.

Job Training

If you’ve applied for a job recently, you’ve noticed that more and more companies want to hire people with experience. Trade schools like the Lincoln School of Technology help give you some of that experience. It’s much easier to get your foot in the door and find your first job as an HVAC technician when you’ve spent time in a classroom practicing the skills necessary for basic maintenance and repair. Not only will you get a paper certification, you’ll also be able to prove your skills when you ace tricky interview questions.

Further Experience

Most advanced licenses, like Denver’s journeyman licenses, require that you keep a log detailing your employer, your duties, and roughly how many hours you work. You need a minimum of two to four years of on-the-job experience to apply for these licenses. Once you’ve put in the time, however, these expert certificates will make getting a job easy. Potential employers will know that you’ve put in the time and acquired the skills to truly master your craft.

A Rewarding Career

HVAC technicians enjoy a challenging career with lots of opportunities for advancement. By starting your journey at a technical school, you’ll prove to your employer that you’re serious and enter the job with the skills that you need. This often means higher starting pay, faster advancement, and more freedom when it comes to finding the job you want. If you want to earn good money while helping local community members solve their problems, enroll in a reputable trade school like Lincoln Tech today.

The post How To Become An Air Conditioning Repair Tech In Colorado appeared first on The Keeping Room.

by Level at May 10, 2017 08:55 PM

How To Troubleshoot Your Air Conditioner

Air conditioning units are very much sought after especially throughout summertime when it gets very humid. It can get difficult to rest at night when the temperature rises beyond comfort. But like other home appliances, they can easily break down because of excessive use. Below are ways on how to troubleshoot your air conditioner.

Step 1

If you want to troubleshoot your air conditioner you need to find the best, surest and simplest ways to fix most overlooked problems. If your air conditioner is not working, this is the first step. If your AC is not turning on, you need to check on the thermostat. You may find out that the thermostat as been turned off or even set at a very low temperature that cannot allow your AC to turn on. You must also ensure that your electrical breaker hasn’t been tripped. At this point, you can trip the breaker on and off again to ensure that is in the ON position. There are several good air conditioner troubleshooting charts that can be found online.

Step 2

This is where you inspect the condition of the blower unit to ensure that it is in good working condition. At this point, you check whether the AC is producing excess air flow in the dwelling. The unit is perfectly located in the main AC housing outside of your house and looks exactly like a hamster wheel. Clean the blower unit and check if there is any damaged or bent blade. Any bent or damaged blades can cause inconvenience and ineffectiveness. If you find that you need to buy parts for your air conditioner, you can enter your model and serial number into some websites, and search for parts for the most common repair problems.

Check the duct system that comes from the AC to the vent in your house. These ducts are mostly located at the basement or attic. Ensure that there is no kink, pinched or disconnected duct. Depending on the condition of the AC filter, you can either decide to clean or replace every month. A clogged or a dirty filter will affect the air flow. This is why they require regular maintenance. This AC filter is perfectly located along the duct that returns air form home to the AC.

Step 3

This is a very important step if you want to effectively troubleshoot your air conditioner. Check whether your condenser fan is properly functioning. If it is not functioning, you can try spin the fan by your hand. The AC blades should spin on their own for 5-10 seconds. If blades do not spin on their own, there must be something wrong with the bearings and you’ll be required to replace the motor. Some manufacturers also have troubleshooting guides online for their specific air conditioner models.

Step 4

If your AC’s air isn’t cold enough, you can turn on the unit ad allow the unit to run for 10-15 minutes. Shut the unit off and go to AC unit and locate the copper suction line. Feel the line with your hand, the line should be ice cold. If the suction line is not cold, your system may require more refrigerant. You can contact an expert who can repair and install all the necessary refrigerants and seal all air conditioner leaks. He must be a reputable and a licensed air conditioning repair contractor.

Step 5

It is at this step where you should open the AC housing and check if the compressor and fan are functioning/running. The compressor is found in the main AC and looks like a round canister with a tubing coming out. If the fan is running and the compressor is not, air must be flowing via your home’s vent and it won’t be cold. If the compressor is hot and not running, this means the compressor has broken down, and you require a professional.

The post How To Troubleshoot Your Air Conditioner appeared first on The Keeping Room.

by Level at May 10, 2017 08:55 PM

Errata Security

John Oliver is wrong about Net Neutrality

People keep linking to John Oliver bits. We should stop doing this. This is comedy, but people are confused into thinking Oliver is engaging in rational political debate:

Enlightened people know that reasonable people disagree, that there's two sides to any debate. John Oliver's bit erodes that belief, making one side (your side) sound smart, and the other side sound unreasonable.

The #1 thing you should know about Net Neutrality is that reasonable people disagree. It doesn't mean they are right, only that they are reasonable. They aren't stupid. They aren't shills for the telcom lobby, or confused by the telcom lobby. Indeed, those opposed to Net Neutrality are the tech experts who know how packets are routed, whereas the supporters tend only to be lawyers, academics, and activists. If you think that the anti-NetNeutrality crowd is unreasonable, then you are in a dangerous filter bubble.

Most everything in John Oliver's piece is incorrect.

For example, he says that without Net Neutrality, Comcast can prefer original shows it produces, and slow down competing original shows by Netflix. This is silly: Comcast already does that, even with NetNeutrality rules.

Comcast owns NBC, which produces a lot of original shows. During prime time (8pm to 11pm), Comcast delivers those shows at 6-mbps to its customers, while Netflix is throttled to around 3-mbps. Because of this, Comcast original shows are seen at higher quality than Netflix shows.

Comcast can do this, even with NetNeutrality rules, because it separates its cables into "channels". One channel carries public Internet traffic, like Netflix. The other channels carry private Internet traffic, for broadcast TV shows and pay-per-view.

All NetNeutrality means is that if Comcast wants to give preference to its own contents/services, it has to do so using separate channels on the wire, rather than pushing everything over the same channel. This is a detail nobody tells you because NetNeutrality proponents aren't techies. They are lawyers and academics. They maximize moral outrage, while ignoring technical details.

Another example in Oliver's show is whether search engines like Google or the (hypothetical) Bing can pay to get faster access to customers. They already do that. The average distance a packet travels on the web is less than 100-miles. That's because the biggest companies (Google, Facebook, Netflix, etc.) pay to put servers in your city close to you. Smaller companies, such as search engine, also pay third-party companies like Akamai or Amazon Web Services to get closer to you. The smallest companies, however, get poor performance, being a thousand miles away.

You can test this out for yourself. Run a packet-sniffer on your home network for a week, then for each address, use mapping tools like ping and traceroute to figure out how far away things are.

The Oliver bit mentioned how Verizon banned Google Wallet. Again, technical details are important here. It had nothing to do with Net Neutrality issues blocking network packets, but only had to do with Verizon-branded phones blocking access to the encrypted enclave. You could use Google Wallet on unlocked phones you bought separately. Moreover, market forces won in the end, with Google Wallet (aka. Android Wallet) now the preferred wallet on their network. In other words, this incident shows that the "free market" fixes things in the long run without the heavy hand of government.

Oliver shows a piece where FCC chief Ajit Pai points out that Internet companies didn't do evil without Net Neutrality rules, and thus NetNeutrality rules were unneeded. Oliver claimed this was a "disingenuous" argument. No, it's not "disingenuous", it entirely the point of why Net Neutrality is bad. It's chasing theoretical possibility of abuse, not the real thing. Sure, Internet companies will occasionally go down misguided paths. If it's truly bad, customers will rebel. In some cases, it's not actually a bad thing, and will end up being a benefit to customers (e.g. throttling BitTorrent during primetime would benefit most BitTorrent users). It's the pro-NetNeutrality side that's being disingenuous, knowingly trumping up things as problems that really aren't.

The point is this. The argument here is a complicated one, between reasonable sides. For humor, John Oliver has created a one-sided debate that falls apart under any serious analysis. Those like the EFF should not mistake such humor for intelligent technical debate.

by Robert Graham ( at May 10, 2017 05:52 AM

May 08, 2017


Latest Book Inducted into Cybersecurity Canon

Thursday evening Mrs B and I were pleased to attend an awards seminar for the Cybersecurity Canon. This is a project sponsored by Palo Alto Networks and led by Rick Howard. The goal is "identify a list of must-read books for all cybersecurity practitioners."

Rick reviewed my fourth book The Practice of Network Security Monitoring in 2014 and someone nominated it for consideration in 2016. I was unaware earlier this year that my book was part of a 32-title "March Madness" style competition. My book won the five rounds, resulting in its conclusion in the 2017 inductee list! Thank you to all those that voted for my book.

Ben Rothke awarded me the Canon trophy.
Ben Rothke interviewed me prior to the induction ceremony. We discussed some current trends in security and some lessons from the book. I hope to see that interviewed published by Palo Alto Networks and/or the Cybersecurity canon project in the near future.

In my acceptance speech I explained how I wrote the book because I had not yet dedicated a book to my youngest daughter, since she was born after my third book was published.

A teaching moment at Black Hat Abu Dhabi in December 2012 inspired me to write the book. While teaching network security monitoring, one of the students asked "but where do I install the .exe on the server?"

I realized this student had no idea of physical access to a wire, or using a system to collect and store network traffic, or any of the other fundamental concepts inherent to NSM. He thought NSM was another magical software package to install on his domain controller.

Four foreign language editions.
Thanks to the interpretation assistance of a local Arabic speaker, I was able to get through to him. However, the experience convinced me that I needed to write a new book that built NSM from the ground up, hence the selection of topics and the order in which I presented them.

While my book has not (yet?) been translated into Arabic, there are two Chinese language editions, a Korean edition, and a Polish edition! I also know of several SOCs who provide a copy of the book to all incoming analysts. The book is also a text in several college courses.

I believe the book remains relevant for anyone who wants to learn the NSM methodology to detect and respond to intrusions. While network traffic is the example data source used in the book, the NSM methodology is data source agnostic.

In 2002 Bamm Visscher and I defined NSM as "the collection, analysis, and escalation of indications and warnings to detect and respond to intrusions." This definition makes no reference to network traffic.

It is the collection-analysis-escalation framework that matters. You could perform NSM using log files, or host-centric data, or whatever else you use for indications and warning.

I have no plans for another cybersecurity book. I am currently editing a book about combat mindset written by the head instructor of my Krav Maga style and his colleague.
Thanks for asking for an autograph!

Palo Alto hosted a book signing and offered free books for attendees. I got a chance to speak with Steven Levy, whose book Hackers was also inducted. I sat next to him during the book signing, as shown in the picture at right.

Thank you to Palo Alto Networks, Rick Howard, Ben Rothke, and my family for making inclusion in the Cybersecurity Canon possible. The awards dinner was a top-notch event. Mrs B and I enjoyed meeting a variety of people, including students in local cybersecurity degree programs.

I closed my acceptance speech with the following from the end of the Old Testament, at the very end of 2nd Maccabees. It captures my goal when writing books:

"So I too will here end my story. If it is well told and to the point, that is what I myself desired; if it is poorly done and mediocre, that was the best I could do."

If you'd like a copy of The Practice of Network Security Monitoring the best deal is to buy print and electronic editions from the publisher's Web site. Use code NSM101 to save 30%. I like having the print version for easy review, and I carry the digital copy on my tablet and phone.

Thank you to everyone who voted and who also bought a copy of my book!

Update: I forgot to thank Doug Burks, who created Security Onion, the software used to demonstrate NSM in the book. Doug also contributed the appendix explaining certain SO commands. Thank you Doug! Also thank you to Bill Pollack and his team at No Starch Press, who edited and published the book!

by Richard Bejtlich ( at May 08, 2017 03:20 PM

May 07, 2017

Merging two Python dictionaries by deep-updating

Say we have two Python dictionaries:

    'name': 'Ferry',
    'hobbies': ['programming', 'sci-fi']


    'hobbies': ['gaming']

What if we want to merge these two dictionaries such that "gaming" is added to the "hobbies" key of the first dictionary? I couldn't find anything online that did this already, so I wrote the following function for it:

# Copyright Ferry Boender, released under the MIT license.
def deepupdate(target, src):
    """Deep update target dict with src
    For each k,v in src: if k doesn't exist in target, it is deep copied from
    src to target. Otherwise, if v is a list, target[k] is extended with
    src[k]. If v is a set, target[k] is updated with v, If v is a dict,
    recursively deep-update it.

    >>> t = {'name': 'Ferry', 'hobbies': ['programming', 'sci-fi']}
    >>> deepupdate(t, {'hobbies': ['gaming']})
    >>> print t
    {'name': 'Ferry', 'hobbies': ['programming', 'sci-fi', 'gaming']}
    for k, v in src.items():
        if type(v) == list:
            if not k in target:
                target[k] = copy.deepcopy(v)
        elif type(v) == dict:
            if not k in target:
                target[k] = copy.deepcopy(v)
                deepupdate(target[k], v)
        elif type(v) == set:
            if not k in target:
                target[k] = v.copy()
            target[k] = copy.copy(v)

It uses a combination of deepcopy(), updating and self recursion to perform a complete merger of the two dictionaries.

As mentioned in the comment, the above function is released under the MIT license, so feel free to use it any of your programs.

by admin at May 07, 2017 02:56 PM

Evaggelos Balaskas

How a slow disk affects your system

The problem

The last couple weeks, a backup server I am managing is failing to make backups!

The backup procedure (a script via cron daemon) is to rsync data from a primary server to it’s /backup directory. I was getting cron errors via email, informing me that the previous rsync script hasnt already finished when the new one was starting (by checking a lock file). This was strange as the time duration is 12hours. 12 hours werent enough to perform a ~200M data transfer over a 100Mb/s network port. That was really strange.

This is the second time in less than a year that this server is making problems. A couple months ago I had to remove a faulty disk from the software raid setup and check the system again. My notes on the matter, can be found here:

Identify the problem

So let us start to identify the problem. A slow rsync can mean a lot of things, especially over ssh. Replacing network cables, viewing dmesg messages, rebooting servers or even changing the filesystem werent changing any things for the better. Time to move on the disks.

Manage and Monitor software RAID devices

On this server, I use raid5 with four hard disks:

# mdadm --verbose --detail /dev/md0

        Version : 1.2
  Creation Time : Wed Feb 26 21:00:17 2014
     Raid Level : raid5
     Array Size : 2929893888 (2794.16 GiB 3000.21 GB)
  Used Dev Size : 976631296 (931.39 GiB 1000.07 GB)
   Raid Devices : 4
  Total Devices : 4
    Persistence : Superblock is persistent

    Update Time : Sun May  7 11:00:32 2017
          State : clean
 Active Devices : 4
Working Devices : 4
 Failed Devices : 0
  Spare Devices : 0

         Layout : left-symmetric
     Chunk Size : 512K

           Name : ServerTwo:0  (local to host ServerTwo)
           UUID : ef5da4df:3e53572e:c3fe1191:925b24cf
         Events : 10496

    Number   Major   Minor   RaidDevice State
       4       8       16        0      active sync   /dev/sdb
       1       8       32        1      active sync   /dev/sdc
       6       8       48        2      active sync   /dev/sdd
       5       8        0        3      active sync   /dev/sda

View hardware parameters of hard disk drive

aka test the hard disks:

# hdparm -Tt /dev/sda

 Timing cached reads:   2490 MB in  2.00 seconds = 1245.06 MB/sec
 Timing buffered disk reads: 580 MB in  3.01 seconds = 192.93 MB/sec

# hdparm -Tt /dev/sdb

 Timing cached reads:   2520 MB in  2.00 seconds = 1259.76 MB/sec
 Timing buffered disk reads: 610 MB in  3.00 seconds = 203.07 MB/sec

# hdparm -Tt /dev/sdc

 Timing cached reads:   2512 MB in  2.00 seconds = 1255.43 MB/sec
 Timing buffered disk reads: 570 MB in  3.01 seconds = 189.60 MB/sec

# hdparm -Tt /dev/sdd

 Timing cached reads:     2 MB in  7.19 seconds = 285.00 kB/sec
 Timing buffered disk reads:   2 MB in  5.73 seconds = 357.18 kB/sec

Root Cause

Seems that one of the disks (/dev/sdd) in raid5 setup, is not performing as well as the others. The same hard disk had a problem a few months ago.

What I did the previous time, was to remove the disk, reformatting it in Low Level Format and add it again in the same setup. The system rebuild the raid5 and after 24hours everything was performing fine.

However the same hard disk seems that still has some issues . Now it is time for me to remove it and find a replacement disk.

Remove Faulty disk

I need to manually fail and then remove the faulty disk from the raid setup.

Failing the disk

Failing the disk manually, means that mdadm is not recognizing the disk as failed (as it did previously). I need to tell mdadm that this specific disk is a faulty one:

# mdadm --manage /dev/md0 --fail /dev/sdd
mdadm: set /dev/sdd faulty in /dev/md0

Removing the disk

now it is time to remove the faulty disk from our raid setup:

# mdadm --manage /dev/md0 --remove  /dev/sdd
mdadm: hot removed /dev/sdd from /dev/md0

Show details

# mdadm --verbose --detail /dev/md0

        Version : 1.2
  Creation Time : Wed Feb 26 21:00:17 2014
     Raid Level : raid5
     Array Size : 2929893888 (2794.16 GiB 3000.21 GB)
  Used Dev Size : 976631296 (931.39 GiB 1000.07 GB)
   Raid Devices : 4
  Total Devices : 3
    Persistence : Superblock is persistent

    Update Time : Sun May  7 11:08:44 2017
          State : clean, degraded
 Active Devices : 3
Working Devices : 3
 Failed Devices : 0
  Spare Devices : 0

         Layout : left-symmetric
     Chunk Size : 512K

           Name : ServerTwo:0  (local to host ServerTwo)
           UUID : ef5da4df:3e53572e:c3fe1191:925b24cf
         Events : 10499

    Number   Major   Minor   RaidDevice State
       4       8       16        0      active sync   /dev/sdb
       1       8       32        1      active sync   /dev/sdc
       4       0        0        4      removed
       5       8        0        3      active sync   /dev/sda

Mounting the Backup

Now it’s time to re-mount the backup directory and re-run the rsync script

mount /backup/

and run the rsync with verbose and progress parameters to review the status of syncing

/usr/bin/rsync -zravxP --safe-links --delete-before --partial --protect-args -e ssh /backup/

Everything seems ok.

A replacement order has already been placed.

Rsync times manage to hit ~ 10.27MB/s again!

rsync time for a daily (12h) diff is now again in normal rates:

real    15m18.112s
user    0m34.414s
sys     0m36.850s
Tag(s): linux, mdadm, md0

May 07, 2017 10:26 AM

May 05, 2017


Talk about Choria and NATS

Recently I was given the opportunity by the folk to talk about Choria and NATS on one of their community events. The recording of the talk as well as the slide deck can be found below.

Thanks again for having me team!

by R.I. Pienaar at May 05, 2017 05:54 AM

May 04, 2017


Using TLS1.3 With OpenSSL

The forthcoming OpenSSL 1.1.1 release will include support for TLSv1.3. The new release will be binary and API compatible with OpenSSL 1.1.0. In theory, if your application supports OpenSSL 1.1.0, then all you need to do to upgrade is to drop in the new version of OpenSSL when it becomes available and you will automatically start being able to use TLSv1.3. However there are some issues that application developers and deployers need to be aware of. In this blog post I am going to cover some of those things.

Differences with TLS1.2 and below

TLSv1.3 is a major rewrite of the specification. There was some debate as to whether it should really be called TLSv2.0 - but TLSv1.3 it is. There are major changes and some things work very differently. A brief, incomplete, summary of some things that you are likely to notice follows:

  • There are new ciphersuites that only work in TLSv1.3. The old ciphersuites cannot be used for TLSv1.3 connections.
  • The new ciphersuites are defined differently and do not specify the certificate type (e.g. RSA, DSA, ECDSA) or the key exchange mechanism (e.g. DHE or ECHDE). This has implications for ciphersuite configuration.
  • Clients provide a “key_share” in the ClientHello. This has consequences for “group” configuration.
  • Sessions are not established until after the main handshake has been completed. There may be a gap between the end of the handshake and the establishment of a session (or, in theory, a session may not be established at all). This could have impacts on session resumption code.
  • Renegotiation is not possible in a TLSv1.3 connection
  • More of the handshake is now encrypted.
  • More types of messages can now have extensions (this has an impact on the custom extension APIs and Certificate Transparency)
  • DSA certificates are no longer allowed in TLSv1.3 connections

Note that at this stage only TLSv1.3 is supported. DTLSv1.3 is still in the early days of specification and there is no OpenSSL support for it at this time.

Current status of the TLSv1.3 standard

As of the time of writing TLSv1.3 is still in draft. Periodically a new version of the draft standard is published by the TLS Working Group. Implementations of the draft are required to identify the specific draft version that they are using. This means that implementations based on different draft versions do not interoperate with each other.

OpenSSL 1.1.1 will not be released until (at least) TLSv1.3 is finalised. In the meantime the OpenSSL git master branch contains our development TLSv1.3 code which can be used for testing purposes (i.e. it is not for production use). You can check which draft TLSv1.3 version is implemented in any particular OpenSSL checkout by examining the value of the TLS1_3_VERSION_DRAFT_TXT macro in the tls1.h header file. This macro will be removed when the final version of the standard is released.

In order to compile OpenSSL with TLSv1.3 support you must use the “enable-tls1_3” option to “config” or “Configure”.

Currently OpenSSL has implemented the “draft-20” version of TLSv1.3. Many other libraries are still using older draft versions in their implementations. Notably many popular browsers are using “draft-18”. This is a common source of interoperability problems. Interoperability of the draft-18 version has been tested with BoringSSL, NSS and picotls.

Within the OpenSSL git source code repository there are two branches: “tls1.3-draft-18” and “tls1.3-draft-19”, which implement the older TLSv1.3 draft versions. In order to test interoperability with other TLSv1.3 implementations you may need to use one of those branches. Note that those branches are considered temporary and are likely to be removed in the future when they are no longer needed.


OpenSSL has implemented support for five TLSv1.3 ciphersuites as follows:

  • TLS13-AES-256-GCM-SHA384
  • TLS13-CHACHA20-POLY1305-SHA256
  • TLS13-AES-128-GCM-SHA256
  • TLS13-AES-128-CCM-8-SHA256
  • TLS13-AES-128-CCM-SHA256

Of these the first three are in the DEFAULT ciphersuite group. This means that if you have no explicit ciphersuite configuration then you will automatically use those three and will be able to negotiate TLSv1.3.

All the TLSv1.3 ciphersuites also appear in the HIGH ciphersuite alias. The CHACHA20, AES, AES128, AES256, AESGCM, AESCCM and AESCCM8 ciphersuite aliases include a subset of these ciphersuites as you would expect based on their names. Key exchange and authentication properties were part of the ciphersuite definition in TLSv1.2 and below. This is no longer the case in TLSv1.3 so ciphersuite aliases such as ECDHE, ECDSA, RSA and other similar aliases do not contain any TLSv1.3 ciphersuites.

If you explicitly configure your ciphersuites then care should be taken to ensure that you are not inadvertently excluding all TLSv1.3 compatible ciphersuites. If a client has TLSv1.3 enabled but no TLSv1.3 ciphersuites configured then it will immediately fail (even if the server does not support TLSv1.3) with an error message like this:

140460646909376:error:141A90B5:SSL routines:ssl_cipher_list_to_bytes:no ciphers available:ssl/statem/statem_clnt.c:3562:No ciphers enabled for max supported SSL/TLS version

Similarly if a server has TLSv1.3 enabled but no TLSv1.3 ciphersuites it will also immediately fail, even if the client does not support TLSv1.3, with an error message like this:

140547390854592:error:141FC0B5:SSL routines:tls_setup_handshake:no ciphers available:ssl/statem/statem_lib.c:108:No ciphers enabled for max supported SSL/TLS version

For example, setting a ciphersuite selection string of ECDHE:!COMPLEMENTOFDEFAULT will work in OpenSSL 1.1.0 and will only select those ciphersuites that are in DEFAULT and also use ECDHE for key exchange. However no TLSv1.3 ciphersuites are in the ECDHE group so this ciphersuite configuration will fail in OpenSSL 1.1.1 if TLSv1.3 is enabled.

You may want to explicitly list the TLSv1.3 ciphersuites you want to use to avoid problems. For example:


You can test which ciphersuites are included in a given ciphersuite selection string using the openssl ciphers -s -v command:

$ openssl ciphers -s -v "ECDHE:!COMPLEMENTOFDEFAULT"

Ensure that at least one ciphersuite supports TLSv1.3


In TLSv1.3 the client selects a “group” that it will use for key exchange. At the time of writing, OpenSSL only supports ECDHE groups for this (it is possible that DHE groups will also be supported by the time OpenSSL 1.1.1 is actually released). The client then sends “key_share” information to the server for its selected group in the ClientHello.

The list of supported groups is configurable. It is possible for a client to select a group that the server does not support. In this case the server requests that the client sends a new key_share that it does support. While this means a connection will still be established (assuming a mutually supported group exists), it does introduce an extra server round trip - so this has implications for performance. In the ideal scenario the client will select a group that the server supports in the first instance.

In practice most clients will use X25519 or P-256 for their initial key_share. For maximum performance it is recommended that servers are configured to support at least those two groups and clients use one of those two for its initial key_share. This is the default case (OpenSSL clients will use X25519).

The group configuration also controls the allowed groups in TLSv1.2 and below. If applications have previously configured their groups in OpenSSL 1.1.0 then you should review that configuration to ensure that it still makes sense for TLSv1.3. The first named (i.e. most preferred group) will be the one used by an OpenSSL client in its intial key_share.

Applications can configure the group list by using SSL_CTX_set1_groups() or a similar function (see here for further details). Alternatively, if applications use SSL_CONF style configuration files then this can be configured using the Groups or Curves command (see here).


In TLSv1.2 and below a session is established as part of the handshake. This session can then be used in a subsequent connection to achieve an abbreviated handshake. Applications might typically obtain a handle on the session after a handshake has completed using the SSL_get1_session() function (or similar). See here for further details.

In TLSv1.3 sessions are not established until after the main handshake has completed. The server sends a separate post-handshake message to the client containing the session details. Typically this will happen soon after the handshake has completed, but it could be sometime later (or not at all).

The specification recommends that applications only use a session once (although this is not enforced). For this reason some servers send multiple session messages to a client. To enforce the “use once” recommendation applications could use SSL_CTX_remove_session() to mark a session as non-resumable (and remove it from the cache) once it has been used.

The old SSL_get1_session() and similar APIs may not operate as expected for client applications written for TLSv1.2 and below. Specifically if a client application calls SSL_get1_session() before the server message containing session details has been received then an SSL_SESSION object will still be returned, but any attempt to resume with it will not succeed and a full handshake will occur instead. In the case where multiple sessions have been sent by the server then only the last session will be returned by SSL_get1_session().

Client application developers should consider using the SSL_CTX_sess_set_new_cb() API instead (see here). This provides a callback mechanism which gets invoked every time a new session is established. This can get invoked multiple times for a single connection if a server sends multiple session messages.

Note that SSL_CTX_sess_set_new_cb() was also available in OpenSSL 1.1.0. Applications that already used that API will still work, but they may find that the callback is invoked at unexpected times, i.e. post-handshake.

An OpenSSL server will immediately attempt to send session details to a client after the main handshake has completed. To server applications this post-handshake stage will appear to be part of the main handshake, so calls to SSL_get1_session() should continue to work as before.

Custom Extensions and Certificate Transparency

In TLSv1.2 and below the initial ClientHello and ServerHello messages can contain “extensions”. This allows the base specifications to be extended with additional features and capabilities that may not be applicable in all scenarios or could not be foreseen at the time that the base specifications were written. OpenSSL provides support for a number of “built-in” extensions.

Additionally the custom extensions API provides some basic capabilities for application developers to add support for new extensions that are not built-in to OpenSSL.

Built on top of the custom extensions API is the “serverinfo” API. This provides an even more basic interface that can be configured at run time. One use case for this is Certificate Transparency. OpenSSL provides built-in support for the client side of Certificate Transparency but there is no built-in server side support. However this can easily be achieved using “serverinfo” files. A serverinfo file containing the Certificate Transparency information can be configured within OpenSSL and it will then be sent back to the client as appropriate.

In TLSv1.3 the use of extensions is expanded significantly and there are many more messages that can include them. Additionally some extensions that were applicable to TLSv1.2 and below are no longer applicable in TLSv1.3 and some extensions are moved from the ServerHello message to the EncryptedExtensions message. The old custom extensions API does not have the ability to specify which messages the extensions should be associated with. For that reason a new custom extensions API was required.

The old API will still work, but the custom extensions will only be added where TLSv1.2 or below is negotiated. To add custom extensions that work for all TLS versions application developers will need to update their applications to the new API (see here for details).

The “serverinfo” data format has also been updated to include additional information about which messages the extensions are relevant to. Applications using “serverinfo” files may need to update to the “version 2” file format to be able to operate in TLSv1.3 (see here and here for details).


TLSv1.3 does not have renegotiation so calls to SSL_renegotiate() or SSL_renegotiate_abbreviated() will immediately fail if invoked on a connection that has negotiated TLSv1.3.

The most common use case for renegotiation is to update the connection keys. The function SSL_key_update() can be used for this purpose in TLSv1.3 (see

DSA certificates

DSA certificates are no longer allowed in TLSv1.3. If your server application is using a DSA certificate then TLSv1.3 connections will fail with an error message similar to the following:

140348850206144:error:14201076:SSL routines:tls_choose_sigalg:no suitable signature algorithm:ssl/t1_lib.c:2308:

Please use an ECDSA or RSA certificate instead.


TLSv1.3 represents a significant step forward and has some exciting new features but there are some hazards for the unwary when upgrading. Mostly these issues have relatively straight forward solutions. Application developers should review their code and consider whether anything should be updated in order to work more effectively with TLSv1.3. Similarly application deployers should review their configuration.

May 04, 2017 11:00 AM

Michael Biven

Chai 2000 - 2017

I lost my little buddy of 17 years this week.

Chai, I’m happy you had a peaceful morning in the patio at our new house doing what you loved. Sitting under bushes and in the sun. Even in your last day you still comforted us. You were relaxed throughout the morning, when I picked you up and held you as we drove to the vet till the end.

Chai, thank you for everything, we love you and you will be missed.

May 04, 2017 08:04 AM

May 03, 2017

Marios Zindilis

Back-end Validation for Django Model Field Choices

In Django, you can provide a list of values that a field can have when creating a model. For example, here's a minimalistic model for storing musicians:

from django.db import models

class Artist(models.Model):

        ('Person', 'Person'),
        ('Group', 'Group'),
        ('Other', 'Other'),)

    name = models.CharField(max_length=100)
    type = models.CharField(max_length=20, choices=TYPE_CHOICES)

Using this code in the Admin interface, as well as in Forms, will make the "Type" field be represented as a drop-down box. Therefore, if all input comes from Admin and Forms, the value of "Type" is expected to be one of those defined in the TYPE_CHOICES tuple. There are 2 things worth noting:

  1. Using choices is a presentation convenience. There is no restriction of the value that can be sent to the database, even coming from the front-end. For example, if you use the browser's inspect feature, you can edit the drop-down list, change one of the values and submit it. The back-end view will then happily consume it and save it to the database.

  2. There is no validation at any stage of the input lifecycle. You may have a specified list of values in the Admin or Forms front-end, but your application may accept input from other sources as well, such as management commands, or bulk imports from fixtures. In none of those cases will a value that is not in the TYPE_CHOICES structure be rejected.


So let's find a way to fix this by adding some validation in the back-end.

Django has this amazing feature called signals. They allow you to attach extra functionality to some actions, by emitting a signal from that action. For example, in this case we will use the pre_save signal, which gets executed just before a model instance is saved to the database.

Here's the example model, with the signal connected to it:

from django.db import models

class Artist(models.Model):

        ('Person', 'Person'),
        ('Group', 'Group'),
        ('Other', 'Other'),)

    name = models.CharField(max_length=100)
    type = models.CharField(max_length=20, choices=TYPE_CHOICES)

models.signals.pre_save.connect(validate_artist_name_choice, sender=Artist)

That last line will call a function named validate_artist_name_choice just before an Artist is saved in the database. Let's write that function:

def validate_artist_name_choice(sender, instance, **kwargs):
    valid_types = [t[0] for t in sender.TYPE_CHOICES]
    if instance.type not in valid_types:
        from django.core.exceptions import ValidationError
        raise ValidationError(
            'Artist Type "{}" is not one of the permitted values: {}'.format(
               ', '.join(valid_types)))

This function can be anywhere in your code, as long as you can import it in the model. Personally, if a pre_save function is specific to one model, I like to put it in the same module as the model itself, but that's just my preference.

So let's try to create a couple of Artists and see what happens. In this example, my App is called "v".

>>> from v.models import Artist
>>> some_artist = Artist(name='Some Artist', type='Person')
>>> some_other_artist = Artist(name='Some Other Artist', type='Alien Lifeform')
Traceback (most recent call last):
  File "/usr/lib/python3.5/", line 91, in runcode
    exec(code, self.locals)
  File "<console>", line 1, in <module>
  File "/home/vs/.local/lib/python3.5/site-packages/django/db/models/", line 796, in save
    force_update=force_update, update_fields=update_fields)
  File "/home/vs/.local/lib/python3.5/site-packages/django/db/models/", line 820, in save_base
  File "/home/vs/.local/lib/python3.5/site-packages/django/dispatch/", line 191, in send
    response = receiver(signal=self, sender=sender, **named)
  File "/home/vs/Tests/validator/v/", line 14, in validate_artist_name_choice
    ', '.join(valid_types)))
django.core.exceptions.ValidationError: ['Artist Type "Alien Lifeform" is not one of the permitted values: Person, Group, Other']

Brilliant! We now have a expressive exception that we can catch in our views or scripts or whatever else we want to use to create model instances. This exception is thrown either when we call the .save() method of a model instance, or when we use the .create() method of a model class. Using the example model, we would get the same exception if we ran:

Artist.objects.create(name='Some Other Artist', type='Alien Lifeform')

Or even:

Artist.objects.get_or_create(name='Some Other Artist', type='Alien Lifeform')

Test It!

Here's a test that triggers the exception:

from django.test import TestCase
from django.core.exceptions import ValidationError
from .models import Artist

class TestArtist(TestCase):
    def setUp(self):
        self.test_artist = Artist(name='Some Artist', type='Some Type')

    def test_artist_type_choice(self):
        with self.assertRaises(ValidationError):


Using Django's pre_save functionality, you can validate your application's input in the back-end, and avoid any surprises. Go forth and validate!

May 03, 2017 11:00 PM

Vincent Bernat

VXLAN: BGP EVPN with Cumulus Quagga

VXLAN is an overlay network to encapsulate Ethernet traffic over an existing (highly available and scalable, possibly the Internet) IP network while accomodating a very large number of tenants. It is defined in RFC 7348. For an uncut introduction on its use with Linux, have a look at my “VXLAN & Linux” post.

VXLAN deployment

In the above example, we have hypervisors hosting a virtual machines from different tenants. Each virtual machine is given access to a tenant-specific virtual Ethernet segment. Users are expecting classic Ethernet segments: no MAC restrictions1, total control over the IP addressing scheme they use and availability of multicast.

In a large VXLAN deployment, two aspects need attention:

  1. discovery of other endpoints (VTEPs) sharing the same VXLAN segments, and
  2. avoidance of BUM frames (broadcast, unknown unicast and multicast) as they have to be forwarded to all VTEPs.

A typical solution for the first point is using multicast. For the second point, this is source-address learning.

Introduction to BGP EVPN

BGP EVPN (RFC 7432 and draft-ietf-bess-evpn-overlay for its application to VXLAN) is a standard control protocol to efficiently solves those two aspects without relying on multicast nor source-address learning.

BGP EVPN relies on BGP (RFC 4271) and its MP-BGP extensions (RFC 4760). BGP is the routing protocol powering the Internet. It is highly scalable and interoperable. It is also extensible and one of its extension is MP-BGP. This extension can carry reachability information (NLRI) for multiple protocols (IPv4, IPv6, L3VPN and in our case EVPN). EVPN is a special family to advertise MAC addresses and the remote equipments they are attached to.

There are basically two kinds of reachability information a VTEP sends through BGP EVPN:

  1. the VNIs they have interest in (type 3 routes), and
  2. for each VNI, the local MAC addresses (type 2 routes).

The protocol also covers other aspects of virtual Ethernet segments (L3 reachability information from ARP/ND caches, MAC mobility and multi-homing2) but we won’t describe them here.

To deploy BGP EVPN, a typical solution is to use several route reflectors (both for redundancy and scalability), like in the picture below. Each VTEP opens a BGP session to at least two route reflectors, sends its information (MACs and VNIs) and receives others’. This reduces the number of BGP sessions to configure.

VXLAN deployment with route reflectors

Compared to other solutions to deploy VXLAN, BGP EVPN has three main advantages:

  • interoperability with other vendors (notably Juniper and Cisco),
  • proven scalability (a typical BGP routers handle several millions of routes), and
  • possibility to enforce fine-grained policies.

On Linux, Cumulus Quagga is a fairly complete implementation of BGP EVPN (type 3 routes for VTEP discovery, type 2 routes with MAC or IP addresses, MAC mobility when a host changes from one VTEP to another one) which requires very little configuration.

This is a fork of Quagga and currently used in Cumulus Linux, a network operating system based on Debian powering switches from various brands. At some point, BGP EVPN support will be contributed back to FRR, a community-maintained fork of Quagga3.

It should be noted the BGP EVPN implementation of Cumulus Quagga currently only supports IPv4.

Route reflector setup

Before configuring each VTEP, we need to configure two or more route reflectors. There are many solutions. I will present three of them:

  • using Cumulus Quagga,
  • using GoBGP, an implementation of BGP in Go,
  • using Juniper JunOS.

For reliability purpose, it’s possible (and easy) to use one implementation for some route reflectors and another implementation for the other ones.

The proposed configurations are quite minimal. However, it is possible to centralize policies on the route reflectors (e.g. routes tagged with some community can only be readvertised to some group of VTEPs).

Using Quagga

The configuration is pretty simple. We suppose the configured route reflector has configured as a loopback IP.

router bgp 65000
  bgp router-id
  bgp cluster-id
  bgp log-neighbor-changes
  no bgp default ipv4-unicast
  neighbor fabric peer-group
  neighbor fabric remote-as 65000
  neighbor fabric capability extended-nexthop
  neighbor fabric update-source
  bgp listen range peer-group fabric
  address-family evpn
   neighbor fabric activate
   neighbor fabric route-reflector-client

A peer group fabric is defined and we leverage the dynamic neighbor feature of Cumulus Quagga: we don’t have to explicitely define each neighbor. Any client from and presenting itself as part of AS 65000 can connect. All sent EVPN routes will be accepted and reflected to the other clients.

You don’t need to run Zebra, the route engine talking with the kernel. Instead, start bgpd with the --no_kernel flag.

Using GoBGP

GoBGP is a clean implementation of BGP in Go4. It exposes an RPC API for configuration (but accepts a configuration file and comes with a command-line client).

It doesn’t support dynamic neighbors, so you’ll have to use the API, the command-line client or some templating language to automate their declaration. A configuration with only one neighbor is like this:

    as: 65000
  - config:
      peer-as: 65000
      - config:
          afi-safi-name: l2vpn-evpn
        route-reflector-client: true

More neighbors can be added from the command line:

$ gobgp neighbor add as 65000 \
>         route-reflector-client \
>         --address-family evpn

GoBGP won’t try to interact with the kernel which is fine as a route reflector.

Using Juniper JunOS

A variety of Juniper products can be a BGP route reflector, notably:

The main factor is the CPU and the memory. The QFX5100 is low on memory and won’t support large deployments without some additional policing.

Here is a configuration similar to the Quagga one:

interfaces {
    lo0 {
        unit 0 {
            family inet {

protocols {
    bgp {
        group fabric {
            family evpn {
                signaling {
                    /* Do not try to install EVPN routes */
            type internal;

routing-options {
    autonomous-system 65000;

VTEP setup

The next step is to configure each VTEP/hypervisor. Each VXLAN is locally configured using a bridge for local virtual interfaces, like illustrated in the below schema. The bridge is taking care of the local MAC addresses (notably, using source-address learning) and the VXLAN interface takes care of the remote MAC addresses (received with BGP EVPN).

Bridged VXLAN device

VXLANs can be provisioned with the following script. Source-address learning is disabled as we will rely solely on BGP EVPN to synchronize FDBs between the hypervisors.

for vni in 100 200; do
    # Create VXLAN interface
    ip link add vxlan${vni} type vxlan
        id ${vni} \
        dstport 4789 \
        local \
    # Create companion bridge
    brctl addbr br${vni}
    brctl addif br${vni} vxlan${vni}
    brctl stp br${vni} off
    ip link set up dev br${vni}
    ip link set up dev vxlan${vni}
# Attach each VM to the appropriate segment
brctl addif br100 vnet10
brctl addif br100 vnet11
brctl addif br200 vnet12

The configuration of Cumulus Quagga is similar to the one used for a route reflector, except we use the advertise-all-vni directive to publish all local VNIs.

router bgp 65000
  bgp router-id
  no bgp default ipv4-unicast
  neighbor fabric peer-group
  neighbor fabric remote-as 65000
  neighbor fabric capability extended-nexthop
  neighbor fabric update-source dummy0
  ! BGP sessions with route reflectors
  neighbor peer-group fabric
  neighbor peer-group fabric
  address-family evpn
   neighbor fabric activate

If everything works as expected, the instances sharing the same VNI should be able to ping each other. If IPv6 is enabled on the VMs, the ping command shows if everything is in order:

$ ping -c10 -w1 -t1 ff02::1%eth0
PING ff02::1%eth0(ff02::1%eth0) 56 data bytes
64 bytes from fe80::5254:33ff:fe00:8%eth0: icmp_seq=1 ttl=64 time=0.016 ms
64 bytes from fe80::5254:33ff:fe00:b%eth0: icmp_seq=1 ttl=64 time=4.98 ms (DUP!)
64 bytes from fe80::5254:33ff:fe00:9%eth0: icmp_seq=1 ttl=64 time=4.99 ms (DUP!)
64 bytes from fe80::5254:33ff:fe00:a%eth0: icmp_seq=1 ttl=64 time=4.99 ms (DUP!)

--- ff02::1%eth0 ping statistics ---
1 packets transmitted, 1 received, +3 duplicates, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.016/3.745/4.991/2.152 ms


Step by step, let’s check how everything comes together.

Getting VXLAN information from the kernel

On each VTEP, Quagga should be able to retrieve the information about configured VXLANs. This can be checked with vtysh:

# show interface vxlan100
Interface vxlan100 is up, line protocol is up
  Link ups:       1    last: 2017/04/29 20:01:33.43
  Link downs:     0    last: (never)
  PTM status: disabled
  vrf: Default-IP-Routing-Table
  index 11 metric 0 mtu 1500
  Type: Ethernet
  HWaddr: 62:42:7a:86:44:01
  inet6 fe80::6042:7aff:fe86:4401/64
  Interface Type Vxlan
  VxLAN Id 100
  Access VLAN Id 1
  Master (bridge) ifindex 9 ifp 0x56536e3f3470

The important points are:

  • the VNI is 100, and
  • the bridge device was correctly detected.

Quagga should also be able to retrieve information about the local MAC addresses :

# show evpn mac vni 100
Number of MACs (local and remote) known for this VNI: 2
MAC               Type   Intf/Remote VTEP      VLAN
50:54:33:00:00:0a local  eth1.100
50:54:33:00:00:0b local  eth2.100

BGP sessions

Each VTEP has to establish a BGP session to the route reflectors. On the VTEP, this can be checked by running vtysh:

# show bgp neighbors
BGP neighbor is, remote AS 65000, local AS 65000, internal link
 Member of peer-group fabric for session parameters
  BGP version 4, remote router ID
  BGP state = Established, up for 00:00:45
  Neighbor capabilities:
    4 Byte AS: advertised and received
      L2VPN EVPN: RX advertised L2VPN EVPN
    Route refresh: advertised and received(new)
    Address family L2VPN EVPN: advertised and received
    Hostname Capability: advertised
    Graceful Restart Capabilty: advertised
 For address family: L2VPN EVPN
  fabric peer-group member
  Update group 1, subgroup 1
  Packet Queue length 0
  Community attribute sent to this neighbor(both)
  8 accepted prefixes

  Connections established 1; dropped 0
  Last reset never
Local host:, Local port: 37603
Foreign host:, Foreign port: 179

The output includes the following information:

  • the BGP state is Established,
  • the address family L2VPN EVPN is correctly advertised, and
  • 8 routes are received from this route reflector.

The state of the BGP sessions can also be checked from the route reflectors. With GoBGP, use the following command:

# gobgp neighbor
BGP neighbor is, remote AS 65000, route-reflector-client
  BGP version 4, remote router ID
  BGP state = established, up for 00:04:30
  BGP OutQ = 0, Flops = 0
  Hold time is 9, keepalive interval is 3 seconds
  Configured hold time is 90, keepalive interval is 30 seconds
  Neighbor capabilities:
        l2vpn-evpn:     advertised and received
    route-refresh:      advertised and received
    graceful-restart:   received
    4-octet-as: advertised and received
    add-path:   received
    UnknownCapability(73):      received
    cisco-route-refresh:        received
  Route statistics:
    Advertised:             8
    Received:               5
    Accepted:               5

With JunOS, use the below command:

> show bgp neighbor
Peer: AS 65000 Local: AS 65000
  Group: fabric                Routing-Instance: master
  Forwarding routing-instance: master
  Type: Internal    State: Established
  Last State: OpenConfirm   Last Event: RecvKeepAlive
  Last Error: None
  Options: <Preference LocalAddress Cluster AddressFamily Rib-group Refresh>
  Address families configured: evpn
  Local Address: Holdtime: 90 Preference: 170
  NLRI evpn: NoInstallForwarding
  Number of flaps: 0
  Peer ID:     Local ID:     Active Holdtime: 9
  Keepalive Interval: 3          Group index: 0    Peer index: 2
  I/O Session Thread: bgpio-0 State: Enabled
  BFD: disabled, down
  NLRI for restart configured on peer: evpn
  NLRI advertised by peer: evpn
  NLRI for this session: evpn
  Peer supports Refresh capability (2)
  Stale routes from peer are kept for: 300
  Peer does not support Restarter functionality
  NLRI that restart is negotiated for: evpn
  NLRI of received end-of-rib markers: evpn
  NLRI of all end-of-rib markers sent: evpn
  Peer does not support LLGR Restarter or Receiver functionality
  Peer supports 4 byte AS extension (peer-as 65000)
  NLRI's for which peer can receive multiple paths: evpn
  Table bgp.evpn.0 Bit: 20000
    RIB State: BGP restart is complete
    RIB State: VPN restart is complete
    Send state: in sync
    Active prefixes:              5
    Received prefixes:            5
    Accepted prefixes:            5
    Suppressed due to damping:    0
    Advertised prefixes:          8
  Last traffic (seconds): Received 276  Sent 170  Checked 276
  Input messages:  Total 61     Updates 3       Refreshes 0     Octets 1470
  Output messages: Total 62     Updates 4       Refreshes 0     Octets 1775
  Output Queue[1]: 0            (bgp.evpn.0, evpn)

If a BGP session cannot be established, the logs of each BGP daemon should mention the cause.

Sent routes

From each VTEP, Quagga needs to send:

  • one type 3 route for each local VNI, and
  • one type 2 route for each local MAC address.

The best place to check the received routes is on one of the route reflectors. If you are using JunOS, the following command will display the received routes from the provided VTEP:

> show route table bgp.evpn.0 receive-protocol bgp

bgp.evpn.0: 10 destinations, 10 routes (10 active, 0 holddown, 0 hidden)
  Prefix                  Nexthop              MED     Lclpref    AS path
  2: MAC/IP
*                                 100        I
  2: MAC/IP
*                                 100        I
  3: IM
*                                 100        I
  3: IM
*                                 100        I

There is one type 3 route for VNI 100 and another one for VNI 200. There are also two type 2 routes for two MAC addresses on VNI 100. To get more information, you can add the keyword extensive. Here is a type 3 route advertising as a VTEP for VNI 1008:

> show route table bgp.evpn.0 receive-protocol bgp extensive

bgp.evpn.0: 11 destinations, 11 routes (11 active, 0 holddown, 0 hidden)
* 3: IM (1 entry, 1 announced)
     Route Distinguisher:
     Localpref: 100
     AS path: I
     Communities: target:65000:268435556 encapsulation:vxlan(0x8)

Here is a type 2 route announcing the location of the 50:54:33:00:00:0a MAC address for VNI 100:

> show route table bgp.evpn.0 receive-protocol bgp extensive

bgp.evpn.0: 11 destinations, 11 routes (11 active, 0 holddown, 0 hidden)
* 2: MAC/IP (1 entry, 1 announced)
     Route Distinguisher:
     Route Label: 100
     ESI: 00:00:00:00:00:00:00:00:00:00
     Localpref: 100
     AS path: I
     Communities: target:65000:268435556 encapsulation:vxlan(0x8)

With Quagga, you can get a similar output with vtysh:

# show bgp evpn route
BGP table version is 0, local router ID is
Status codes: s suppressed, d damped, h history, * valid, > best, i - internal
Origin codes: i - IGP, e - EGP, ? - incomplete
EVPN type-2 prefix: [2]:[ESI]:[EthTag]:[MAClen]:[MAC]
EVPN type-3 prefix: [3]:[EthTag]:[IPlen]:[OrigIP]

   Network          Next Hop            Metric LocPrf Weight Path
Route Distinguisher:
                             100      0 i
                             100      0 i
                             100      0 i
Route Distinguisher:
                             100      0 i

With GoBGP, use the following command:

# gobgp global rib -a evpn | grep rd:
    Network  Next Hop             AS_PATH              Age        Attrs
*>  [type:macadv][rd:][esi:single-homed][etag:0][mac:50:54:33:00:00:0a][ip:<nil>][labels:[100]]                               00:00:17   [{Origin: i} {LocalPref: 100} {Extcomms: [VXLAN], [65000:268435556]}]
*>  [type:macadv][rd:][esi:single-homed][etag:0][mac:50:54:33:00:00:0b][ip:<nil>][labels:[100]]                               00:00:17   [{Origin: i} {LocalPref: 100} {Extcomms: [VXLAN], [65000:268435556]}]
*>  [type:macadv][rd:][esi:single-homed][etag:0][mac:50:54:33:00:00:0a][ip:<nil>][labels:[200]]                               00:00:17   [{Origin: i} {LocalPref: 100} {Extcomms: [VXLAN], [65000:268435656]}]
*>  [type:multicast][rd:][etag:0][ip:]                               00:00:17   [{Origin: i} {LocalPref: 100} {Extcomms: [VXLAN], [65000:268435556]}]
*>  [type:multicast][rd:][etag:0][ip:]                               00:00:17   [{Origin: i} {LocalPref: 100} {Extcomms: [VXLAN], [65000:268435656]}]

Received routes

Each VTEP should have received the type 2 and type 3 routes from its fellow VTEPs, through the route reflectors. You can check with the show bgp evpn route command of vtysh.

Does Quagga correctly understand the received routes? The type 3 routes are translated to an assocation between the remote VTEPs and the VNIs:

# show evpn vni
Number of VNIs: 2
VNI        VxLAN IF              VTEP IP         # MACs   # ARPs   Remote VTEPs
100        vxlan100         4        0
200        vxlan200         3        0

The type 2 routes are translated to an association between the remote MACs and the remote VTEPs:

# show evpn mac vni 100
Number of MACs (local and remote) known for this VNI: 4
MAC               Type   Intf/Remote VTEP      VLAN
50:54:33:00:00:09 remote
50:54:33:00:00:0a local  eth1.100
50:54:33:00:00:0b local  eth2.100
50:54:33:00:00:0c remote

FDB configuration

The last step is to ensure Quagga has correctly provided the received information to the kernel. This can be checked with the bridge command:

# bridge fdb show dev vxlan100 | grep dst
00:00:00:00:00:00 dst self permanent
00:00:00:00:00:00 dst self permanent
50:54:33:00:00:0c dst self
50:54:33:00:00:09 dst self

All good! The two first lines are the translation of the type 3 routes (any BUM frame will be sent to both and and the two last ones are the translation of the type 2 routes.


One of the strength of BGP EVPN is the interoperability with other network vendors. To demonstrate it works as expected, we will configure a Juniper vMX to act as a VTEP.

First, we need to configure the physical bridge9. This is similar to the use of ip link and brctl with Linux. We only configure one physical interface with two old-school VLANs paired with matching VNIs.

interfaces {
    ge-0/0/1 {
        unit 0 {
            family bridge {
                interface-mode trunk;
                vlan-id-list [ 100 200 ];
routing-instances {
    switch {
        instance-type virtual-switch;
        interface ge-0/0/1.0;
        bridge-domains {
            vlan100 {
                domain-type bridge;
                vlan-id 100;
                vxlan {
                    vni 100;
            vlan200 {
                domain-type bridge;
                vlan-id 200;
                vxlan {
                    vni 200;

Then, we configure BGP EVPN to advertise all known VNIs. The configuration is quite similar to the one we did with Quagga:

protocols {
    bgp {
        group fabric {
            type internal;
            family evpn signaling;

routing-instances {
    switch {
        vtep-source-interface lo0.0;
        route-distinguisher; # ❶
        vrf-import EVPN-VRF-VXLAN;
        vrf-target {
        protocols {
            evpn {
                encapsulation vxlan;
                extended-vni-list all;
                multicast-mode ingress-replication;

routing-options {
    autonomous-system 65000;

policy-options {
    policy-statement EVPN-VRF-VXLAN {
        then accept;

We also need a small compatibility patch for Cumulus Quagga10.

The routes sent by this configuration are very similar to the routes sent by Quagga. The main differences are:

  • on JunOS, the route distinguisher is configured statically (in ❶), and
  • on JunOS, the VNI is also encoded as an Ethernet tag ID.

Here is a type 3 route, as sent by JunOS:

> show route table bgp.evpn.0 receive-protocol bgp extensive

bgp.evpn.0: 13 destinations, 13 routes (13 active, 0 holddown, 0 hidden)
* 3: IM (1 entry, 1 announced)
     Route Distinguisher:
     Localpref: 100
     AS path: I
     Communities: target:65000:268435556 encapsulation:vxlan(0x8)
     PMSI: Flags 0x0: Label 6: Type INGRESS-REPLICATION

Here is a type 2 route:

> show route table bgp.evpn.0 receive-protocol bgp extensive

bgp.evpn.0: 13 destinations, 13 routes (13 active, 0 holddown, 0 hidden)
* 2: MAC/IP (1 entry, 1 announced)
     Route Distinguisher:
     Route Label: 200
     ESI: 00:00:00:00:00:00:00:00:00:00
     Localpref: 100
     AS path: I
     Communities: target:65000:268435656 encapsulation:vxlan(0x8)

We can check that the vMX is able to make sense of the routes it receives from its peers running Quagga:

> show evpn database l2-domain-id 100
Instance: switch
VLAN  DomainId  MAC address        Active source                  Timestamp        IP address
     100        50:54:33:00:00:0c                    Apr 30 12:46:20
     100        50:54:33:00:00:0d                    Apr 30 12:32:42
     100        50:54:33:00:00:0e                    Apr 30 12:46:20
     100        50:54:33:00:00:0f  ge-0/0/1.0                     Apr 30 12:45:55

On the other end, if we look at one of the Quagga-based VTEP, we can check the received routes are correctly understood:

# show evpn vni 100
VNI: 100
 VxLAN interface: vxlan100 ifIndex: 9 VTEP IP:
 Remote VTEPs for this VNI:
 Number of MACs (local and remote) known for this VNI: 4
 Number of ARPs (IPv4 and IPv6, local and remote) known for this VNI: 0
# show evpn mac vni 100
Number of MACs (local and remote) known for this VNI: 4
MAC               Type   Intf/Remote VTEP      VLAN
50:54:33:00:00:0c local  eth1.100
50:54:33:00:00:0d remote
50:54:33:00:00:0e remote
50:54:33:00:00:0f remote

Get in touch if you have some success with other vendors!

  1. For example, they may use bridges to connect containers together. 

  2. Such a feature can replace proprietary implementations of MC-LAG allowing several VTEPs to act as a endpoint for a single link aggregation group. This is not needed on our scenario where hypervisors act as VTEPs

  3. The development of Quagga is slow and “closed”. New features are often stalled. FRR is placed under the umbrella of the Linux Foundation, has a GitHub-centered development model and an election process. It already has several interesting enhancements (notably, BGP add-path, BGP unnumbered, MPLS and LDP). 

  4. I am unenthusiastic about projects whose the sole purpose is to rewrite something in Go. However, while being quite young, GoBGP is quite valuable on its own (good architecture, good performance). 

  5. The 48-port version is around $10,000 with the BGP license. 

  6. An empty chassis with a dual routing engine (RE-S-1800X4-16G) is around $30,000. 

  7. I don’t know how pricey the vRR is. For evaluation purposes, it can be downloaded for free if you are a customer. 

  8. The value 100 used in the route distinguishier ( is not the one used to encode the VNI. The VNI is encoded in the route target (65000:268435556), in the 24 least signifiant bits (268435556 & 0xffffff equals 100). As long as VNIs are unique, we don’t have to understand those details. 

  9. For some reason, the use of a virtual switch is mandatory. This is specific to this platform: a QFX doesn’t require this. 

  10. The encoding of the VNI into the route target is being standardized in draft-ietf-bess-evpn-overlay. Juniper already implements this draft. 

by Vincent Bernat at May 03, 2017 09:56 PM

VXLAN & Linux

VXLAN is an overlay network to carry Ethernet traffic over an existing (highly available and scalable) IP network while accommodating a very large number of tenants. It is defined in RFC 7348.

Starting from Linux 3.12, the VXLAN implementation is quite complete as both multicast and unicast are supported as well as IPv6 and IPv4. Let’s explore the various methods to configure it.

VXLAN setup

To illustrate our examples, we use the following setup:

  • an underlay IP network (highly available and scalable, possibly the Internet),
  • three Linux bridges acting as VXLAN tunnel endpoints (VTEP),
  • four servers believing they share a common Ethernet segment.

A VXLAN tunnel extends the individual Ethernet segments accross the three bridges, providing a unique (virtual) Ethernet segment. From one host (e.g. H1), we can reach directly all the other hosts in the virtual segment:

$ ping -c10 -w1 -t1 ff02::1%eth0
PING ff02::1%eth0(ff02::1%eth0) 56 data bytes
64 bytes from fe80::5254:33ff:fe00:8%eth0: icmp_seq=1 ttl=64 time=0.016 ms
64 bytes from fe80::5254:33ff:fe00:b%eth0: icmp_seq=1 ttl=64 time=4.98 ms (DUP!)
64 bytes from fe80::5254:33ff:fe00:9%eth0: icmp_seq=1 ttl=64 time=4.99 ms (DUP!)
64 bytes from fe80::5254:33ff:fe00:a%eth0: icmp_seq=1 ttl=64 time=4.99 ms (DUP!)

--- ff02::1%eth0 ping statistics ---
1 packets transmitted, 1 received, +3 duplicates, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.016/3.745/4.991/2.152 ms

Basic usage

The reference deployment for VXLAN is to use an IP multicast group to join the other VTEPs:

# ip -6 link add vxlan100 type vxlan \
>   id 100 \
>   dstport 4789 \
>   local 2001:db8:1::1 \
>   group ff05::100 \
>   dev eth0 \
>   ttl 5
# brctl addbr br100
# brctl addif br100 vxlan100
# brctl addif br100 vnet22
# brctl addif br100 vnet25
# brctl stp br100 off
# ip link set up dev br100
# ip link set up dev vxlan100

The above commands create a new interface acting as a VXLAN tunnel endpoint, named vxlan100 and put it in a bridge with some regular interfaces1. Each VXLAN segment is associated to a 24-bit segment ID, the VXLAN Network Identifier (VNI). In our example, the default VNI is specified with id 100.

When VXLAN was first implemented in Linux 3.7, the UDP port to use was not defined. Several vendors were using 8472 and Linux took the same value. To avoid breaking existing deployments, this is still the default value. Therefore, if you want to use the IANA-assigned port, you need to explicitely set it with dstport 4789.

As we want to use multicast, we have to specify a multicast group to join (group ff05::100), as well as a physical device (dev eth0). With multicast, the default TTL is 1. If your multicast network leverages some routing, you’ll have to increase the value a bit, like here with ttl 5.

The vxlan100 device acts as a bridge device with remote VTEPs as virtual ports:

  • it sends broadcast, unknown unicast and multicast (BUM) frames to all VTEPs using the multicast group, and
  • it discovers the association from Ethernet MAC addresses to VTEP IP addresses using source-address learning.

The following figure summarizes the configuration, with the FDB of the Linux bridge (learning local MAC addresses) and the FDB of the VXLAN device (learning distant MAC addresses):

Bridged VXLAN device

The FDB of the VXLAN device can be observed with the bridge command. If the destination MAC is present, the frame is sent to the associated VTEP (unicast). The all-zero address is only used when a lookup for the destination MAC fails.

# bridge fdb show dev vxlan100 | grep dst
00:00:00:00:00:00 dst ff05::100 via eth0 self permanent
50:54:33:00:00:0b dst 2001:db8:3::1 self
50:54:33:00:00:08 dst 2001:db8:1::1 self

If you are interested to get more details on how to setup a multicast network and build VXLAN segments on top of it, see my “Network virtualization with VXLAN” article.

Without multicast

Using VXLAN over a multicast IP network has several benefits:

  • automatic discovery of other VTEPs sharing the same multicast group,
  • good bandwidth usage (packets are replicated as late as possible),
  • decentralized and controller-less design2.

However, multicast is not available everywhere and managing it at scale can be difficult. In Linux 3.8, the DOVE extensions have been added to the VXLAN implementation, removing the dependency on multicast.

Unicast with static flooding

We can replace multicast by head-end replication of BUM frames to a statically configured lists of remote VTEPs3:

# ip -6 link add vxlan100 type vxlan \
>   id 100 \
>   dstport 4789 \
>   local 2001:db8:1::1
# bridge fdb append 00:00:00:00:00:00 dev vxlan100 dst 2001:db8:2::1
# bridge fdb append 00:00:00:00:00:00 dev vxlan100 dst 2001:db8:3::1

The VXLAN is defined without a remote multicast group. Instead, all the remote VTEPs are associated with the all-zero address: a BUM frame will be duplicated to all those destinations. The VXLAN device will still learn remote addresses automatically using source-address learning.

It is a very simple solution. With a bit of automation, you can keep the default FDB entries up-to-date easily. However, the host will have to duplicate each BUM frame (head-end replication) as many times as there are remote VTEPs. This is quite reasonable if you have a dozen of them. This may become out-of-hand if you have thousands of them.

Cumulus vxfld daemon is an example of use of this strategy (in the head-end replication mode).

Unicast with static L2 entries

When the associations of MAC addresses and VTEPs are known, it is possible to pre-populate the FDB and disable learning:

# ip -6 link add vxlan100 type vxlan \
>   id 100 \
>   dstport 4789 \
>   local 2001:db8:1::1 \
>   nolearning
# bridge fdb append 00:00:00:00:00:00 dev vxlan100 dst 2001:db8:2::1
# bridge fdb append 00:00:00:00:00:00 dev vxlan100 dst 2001:db8:3::1
# bridge fdb append 50:54:33:00:00:09 dev vxlan100 dst 2001:db8:2::1
# bridge fdb append 50:54:33:00:00:0a dev vxlan100 dst 2001:db8:2::1
# bridge fdb append 50:54:33:00:00:0b dev vxlan100 dst 2001:db8:3::1

Thanks to the nolearning flag, source-address learning is disabled. Therefore, if a MAC is missing, the frame will always be sent using the all-zero entries.

The all-zero entries are still needed for broadcast and multicast traffic (e.g. ARP and IPv6 neighbor discovery). This kind of setup works well to provide virtual L2 networks to virtual machines (no L3 information available). You need some glue to update the FDB entries.

BGP EVPN with Cumulus Quagga is an example of use of this strategy (see “VXLAN: BGP EVPN with Cumulus Quagga” for additional information).

Unicast with static L3 entries

In the previous example, we had to keep the all-zero entries for ARP and IPv6 neighbor discovery to work correctly. However, Linux can answer to neighbor requests on behalf of the remote nodes4. When this feature is enabled, the default entries are not needed anymore (but you could keep them):

# ip -6 link add vxlan100 type vxlan \
>   id 100 \
>   dstport 4789 \
>   local 2001:db8:1::1 \
>   nolearning \
>   proxy
# ip -6 neigh add 2001:db8:ff::11 lladdr 50:54:33:00:00:09 dev vxlan100
# ip -6 neigh add 2001:db8:ff::12 lladdr 50:54:33:00:00:0a dev vxlan100
# ip -6 neigh add 2001:db8:ff::13 lladdr 50:54:33:00:00:0b dev vxlan100
# bridge fdb append 50:54:33:00:00:09 dev vxlan100 dst 2001:db8:2::1
# bridge fdb append 50:54:33:00:00:0a dev vxlan100 dst 2001:db8:2::1
# bridge fdb append 50:54:33:00:00:0b dev vxlan100 dst 2001:db8:3::1

This setup totally eliminates head-end replication. However, protocols relying on multicast won’t work either. With some automation, this is a setup that should work well with containers: if there is a registry keeping a list of all IP and MAC addresses in use, a program could listen to it and adjust the FDB and the neighbor tables.

The VXLAN backend of Docker’s libnetwork is an example of use of this strategy (but it also uses the next method).

Unicast with dynamic L3 entries

Linux can also notify a program an (L2 or L3) entry is missing. The program queries some central registry and dynamically adds the requested entry. However, for L2 entries, notifications are issued only if:

  • the destination MAC address is not known,
  • there is no all-zero entry in the FDB, and
  • the destination MAC address is not a multicast or broadcast one.

Those limitations prevent us to do a “unicast with dynamic L2 entries” scenario.

First, let’s create the VXLAN device with the l2miss and l3miss options5:

ip -6 link add vxlan100 type vxlan \
   id 100 \
   dstport 4789 \
   local 2001:db8:1::1 \
   nolearning \
   l2miss \
   l3miss \

Notifications are sent to programs listening to an AF_NETLINK socket using the NETLINK_ROUTE protocol. This socket needs to be bound to the RTNLGRP_NEIGH group. The following is doing exactly that and decodes the received notifications:

# ip monitor neigh dev vxlan100
miss 2001:db8:ff::12 STALE
miss lladdr 50:54:33:00:00:0a STALE

The first notification is about a missing neighbor entry for the requested IP address. We can add it with the following command:

ip -6 neigh replace 2001:db8:ff::12 \
    lladdr 50:54:33:00:00:0a \
    dev vxlan100 \
    nud reachable

The entry is not permanent so that we don’t need to delete it when it expires. If the address becomes stale, we will get another notification to refresh it.

Once the host receives our proxy answer for the neighbor discovery request, it can send a frame with the MAC we gave as destination. The second notification is about the missing FDB entry for this MAC address. We add the appropriate entry with the following command6:

bridge fdb replace 50:54:33:00:00:0a \
    dst 2001:db8:2::1 \
    dev vxlan100 dynamic

The entry is not permanent either as it would prevent the MAC to migrate to the local VTEP (a dynamic entry cannot override a permanent entry).

This setup works well with containers and a global registry. However, there is small latency penalty for the first connections. Moreover, multicast and broadcast won’t be available in the underlay network. The VXLAN backend for flannel, a network fabric for Kubernetes, is an example of this strategy.


There is no one-size-fits-all solution.

You should consider the multicast solution if:

  • you are in an environment where multicast is available,
  • you are ready to operate (and scale) a multicast network,
  • you need multicast and broadcast inside the virtual segments,
  • you don’t have L2/L3 addresses available beforehand.

The scalability of such a solution is pretty good if you take care of not putting all VXLAN interfaces into the same multicast group (e.g. use the last byte of the VNI as the last byte of the multicast group).

When multicast is not available, another generic solution is BGP EVPN: BGP is used as a controller to ensure distribution of the list of VTEPs and their respective FDBs. As mentioned earlier, an implementation of this solution is Cumulus Quagga. I explore this option in a separate post: VXLAN: BGP EVPN with Cumulus Quagga.

If you operate in a container-like environment where L2/L3 addresses are known beforehand, a solution using static and/or dynamic L2 and L3 entries based on a central registry and no source-address learning would also fit the bill. This provides a more security-tight solution (bound resources, MiTM attacks dampened down, inability to amplify bandwidth usage through excessive broadcast). Various environment-specific solutions are available7 or you can build your own.

Other considerations

Independently of the chosen strategy, here are a few important points to keep in mind when implementing a VXLAN overlay.


While you may expect VXLAN interfaces to only carry L2 traffic, Linux doesn’t disable IP processing. If the destination MAC is a local one, Linux will route or deliver the encapsulated IP packet. Check my post about the proper isolation of a Linux bridge.


VXLAN enforces isolation between tenants, but the traffic is totally unencrypted. The most direct solution to provide encryption is to use IPsec. Some container-based solutions may come with IPsec support out-of-the box (notably Docker’s libnetwork, but flannel has plan for it too). This is quite important for a deployment over a public cloud.


The format of a VXLAN-encapsulated frame is the following:

VXLAN encapsulation

VXLAN adds a fixed overhead of 50 bytes. If you also use IPsec, the overhead depends on many factors. In transport mode, with AES and SHA256, the overhead is 56 bytes. With NAT traversal, this is 64 bytes (additional UDP header). In tunnel mode, this is 72 bytes. See Cisco IPsec Overhead Calculator Tool.

Some users will expect to be able to use an Ethernet MTU of 1500 for the overlay network. Therefore, the underlay MTU should be increased. If it is not possible, ensure the inner MTU (inside the containers or the virtual machines) is correctly decreased8.


While all the examples above are using IPv6, the ecosystem is not quite ready yet. The multicast L2-only strategy works fine with IPv6 but every other scenario currently needs some patches (1, 2, 3).

On top of that, IPv6 may not have been implemented in VXLAN-related tools:


Linux VXLAN implementation doesn’t support IGMP snooping. Multicast traffic will be broadcasted to all VTEPs unless multicast MAC addresses are inserted into the FDB.

  1. This is one possible implementation. The bridge is only needed if you require some form of source-address learning for local interfaces. Another strategy is to use MACVLAN interfaces. 

  2. The underlay multicast network may still need some central components, like rendez-vous points for PIM-SM protocol. Fortunately, it’s possible to make them highly available and scalable (e.g. with Anycast-RP, RFC 4610). 

  3. For this example and the following ones, a patch is needed for the ip command (to be included in 4.11) to use IPv6 for transport. In the meantime, here is a quick workaround:

    # ip -6 link add vxlan100 type vxlan \
    >   id 100 \
    >   dstport 4789 \
    >   local 2001:db8:1::1 \
    >   remote 2001:db8:2::1
    # bridge fdb append 00:00:00:00:00:00 \
    >   dev vxlan100 dst 2001:db8:3::1

  4. You may have to apply an IPv6-related patch to the kernel (to be included in 4.12). 

  5. You have to apply an IPv6-related patch to the kernel (to be included in 4.12) to get appropriate notifications for missing IPv6 addresses. 

  6. Directly adding the entry after the first notification would have been smarter to avoid unnecessary retransmissions. 

  7. flannel and Docker’s libnetwork were already mentioned as they both feature a VXLAN backend. There are also some interesting experiments like BaGPipe BGP for Kubernetes which leverages BGP EVPN and is therefore interoperable with other vendors. 

  8. There is no such thing as MTU discovery on an Ethernet segment. 

by Vincent Bernat at May 03, 2017 09:56 PM

May 01, 2017

Ben's Practical Admin Blog

It’s Nice to be Appreciated

Quite out of the blue I had a parcel arrive on my doorstep while I was on leave. It was notable in that I had received all the parcels I thought I was expecting.

As it turns out the fine administration over at ITPA felt that I do quite a bit to help out this organisation and wanted to thank me with a signed copy of Tom Limoncelli’s tome of The Practice of System and Network Administration.

I feel suitably humble. It’s nice to be appreciated 🙂

by Ben at May 01, 2017 09:20 AM

ScriptForm v1.3: Webserver that automatically generates forms to serve as frontends to scripts

I've just released v1.3 of Scriptform.

ScriptForm is a stand-alone webserver that automatically generates forms from JSON to serve as frontends to scripts. It takes a JSON file which contains form definitions, constructs web forms from this JSON and serves these to users over HTTP. The user can select a form and fill it out. When the user submits the form, it is validated and the associated script is called. Data entered in the form is passed to the script through the environment.

You can get the new release from the Releases page.

More information, including screenshots, can be found on the Github page.

by admin at May 01, 2017 06:51 AM

April 29, 2017

Evaggelos Balaskas

Protecting your Authoritative PowerDNS Server with dnsdist


My Authoritative PowerDNS configuration, is relatively simple:


Here is my configuration:

# egrep -v '^($|#)' pdns.conf


Bind Backend

I am using a bind backend because I used to run a bind dns server and I am too lazy to change it.

the named.conf doesnt have much:

zone "" IN {
    type master;
    file "/etc/pdns/var/";


Today, I’ve noticed some unusual traffic to my server, so I’ve enabled the logging features:



The horror !!!

In less than 10minutes or so, almost 2500 “unique” IPs were “attacking” my auth-dns with random queries.

Let me give you an example:


iptables to the rescue:

iptables -I INPUT -m string --algo bm --string "madingyule" -j DROP

Any dns query with the string madingyule will be blocked in INPUT chain with Boyer–Moore string search algorithm.


I need a more permanent solution than reading logs and block attacks with iptables, so I’ve asked the IRC about it. They pointed me to dnsdist.

I’ve already knew about dnsdist but I always thought it was a solution for recursors and not for auth-ns.

I was wrong! dnsdist is a highly DNS-, DoS- and abuse-aware loadbalancer and works fine for auth-ns setup too.

pdns configuration

My auth-ns configuration had to change to something like this:

launch = bind
bind-config = /etc/pdns/named.conf

Disabling any global listener and tcp.

dnsdist configuration

here is my dnsdist configuration:


-- accept DNS queries on UDP and TCP

-- fwd queries to localhost

-- resets the list to this array

I am not 100% sure about the ACL but everything seems ok.

Thats it !!!! - Finished

dnsdist - client

To connect to the dnsdist daemon, you need to add the below configuration:


That means, after reloading the daemon, you can connect on it with:

# dnsdist -c



 -- log everything
 addAction(AllRule(), LogAction("/var/log/dnsdist.log", false, true, false))

Domain Blocking

Let’s start with the above iptables example:



You can connect to dnsdist client (see above) and and any domain you wan to block without restarting your dnsdist service.

Allow Action

Another trick you can do, is to create some custom rules by allowing any DNS queries for your domains and drop any other dns query. You can do this with something like that:

addAction(makeRule("")   , AllowAction())
addAction(makeRule("")  , AllowAction())

addAction(AllRule()                  , DropAction())

Rule Order

Just remember, that the rules will be processed in line order of the file.

Block ANY

You can drop all ANY queries with:

addAction(QTypeRule(dnsdist.ANY), DropAction())

although I dont recommend it.

Rate-Limiting - QPS (Queries Per Second)

Now to the good stuff: rate limiting

A simple rule is something like the below:

-- drop queries exceeding 5 qps, grouped by /24 for IPv4 and /64 for IPv6
addAction(MaxQPSIPRule(5, 24, 64), DropAction())

If you want to drop everything when they pass the 5qps:

addAction(MaxQPSIPRule(5), DropAction())


An alternative approach is to delay everything for more than 5qps (rate limiting), this may make the bot (ddos) to overlook you.

-- Delay for 1000ms aka 1s for 5qps
addDelay(MaxQPSIPRule(5), 1000)

File Descriptors

Working on a VPS (virtual private server), I’ve troubled with file descriptors.
Message in logs from dnsdist is:

Warning, this configuration can use more than 1057 file descriptors, web server and console connections not included, and the current limit is 1024

From the command line you can tweak it to 2048 like this:

# ulimit -n 2048

If you need to make it permanent:

vim /etc/security/limits.conf

*        -        nofile        2048


okei, it’s time to see what’s the traffic:


will report the domains that are reaching to our dnsdsist.

topQueries() will report everything


will report TLD (Top Level Domains)

Identify your traffic:



So dnsdist is now in front of my powerdns auth-ns setup and handles everything, blocking what is necessary.

To be sure that the daemon is up and running:


check process dnsdist with pidfile /var/run/
    alert evaggelos_AT_balaskas_DOT_gr only on { timeout, nonexist }
    start program = "/etc/init.d/dnsdist start"
    stop program  = "/etc/init.d/dnsdist stop"

dnsdist - basics

Some basic commands about dnsdist (when connecting to the client):


addAction(                         addAnyTCRule()                     addDelay(
addDisableValidationRule(          addDNSCryptBind(                   addDomainBlock(
addDomainSpoof(                    addDynBlocks(                      addLocal(
addLuaAction(                      addNoRecurseRule(                  addPoolRule(
addQPSLimit(                       addQPSPoolRule(                    addResponseAction(
AllowAction()                      AllowResponseAction()              AllRule()
AndRule(                           benchRule(                         carbonServer(
clearDynBlocks()                   clearQueryCounters()               clearRules()
controlSocket(                     DelayAction(                       DelayResponseAction(
delta()                            DisableValidationAction()          DropAction()
DropResponseAction()               dumpStats()                        exceedNXDOMAINs(
exceedQRate(                       exceedQTypeRate(                   exceedRespByterate(
exceedServFails(                   firstAvailable                     fixupCase(
generateDNSCryptCertificate(       generateDNSCryptProviderKeys(      getPoolServers(
getQueryCounters(                  getResponseRing()                  getServer(
getServers()                       grepq(                             leastOutstanding
LogAction(                         makeKey()                          MaxQPSIPRule(
MaxQPSRule(                        mvResponseRule(                    mvRule(
newDNSName(                        newQPSLimiter(                     newRemoteLogger(
newRuleAction(                     newServer(                         newServerPolicy(
newSuffixMatchNode()               NoRecurseAction()                  PoolAction(
printDNSCryptProviderFingerprint(  QNameLabelsCountRule(              QNameWireLengthRule(
QTypeRule(                         RCodeRule(                         RegexRule(
registerDynBPFFilter(              RemoteLogAction(                   RemoteLogResponseAction(
rmResponseRule(                    rmRule(                            rmServer(
roundrobin                         setACL(                            setAPIWritable(
setDNSSECPool(                     setECSOverride(                    setECSSourcePrefixV4(
setECSSourcePrefixV6(              setKey(                            setLocal(
setMaxTCPClientThreads(            setMaxTCPQueuedConnections(        setMaxUDPOutstanding(
setQueryCount(                     setQueryCountFilter(               setRules(
setServerPolicy(                   setServerPolicyLua(                setServFailWhenNoServer(
setTCPRecvTimeout(                 setTCPSendTimeout(                 setUDPTimeout(
setVerboseHealthChecks(            show(                              showACL()
showDNSCryptBinds()                showDynBlocks()                    showResponseLatency()
showResponseRules()                showRules()                        showServerPolicy()
showServers()                      showTCPStats()                     showVersion()
shutdown()                         SpoofAction(                       TCAction()
testCrypto()                       topBandwidth(                      topClients(
topQueries(                        topResponseRule()                  topResponses(
topRule()                          topSlow(                           truncateTC(
unregisterDynBPFFilter(            webserver(                         whashed
wrandom                            addACL(                            

dnsdist - ACL

Keep in mind that the default ACL is:

> showACL()
Tag(s): dnsdist, powerdns

April 29, 2017 11:27 PM

April 28, 2017

The Lone Sysadmin

Intel’s Memory Drive Implementation for Optane Guarantees its Doom

A few weeks ago Intel started releasing their Optane product, a commercialization of the 3D Xpoint (Crosspoint) technology they’ve been talking about for a few years. Predictably, there has been a lot of commentary in all directions. Did you know it’s game changing, or that it’s a solution looking for a problem? It’s storage. It […]

The post Intel’s Memory Drive Implementation for Optane Guarantees its Doom appeared first on The Lone Sysadmin. Head over to the source to read the full post!

by Bob Plankers at April 28, 2017 08:26 PM

April 22, 2017

DNS over TLS. Secure the DNS!

Secure the Web

A while back we had the "secure the web" initiative, where everyone was inspired to enable encryption (https) on their websites. This was so we could thwart things like eavesdropping and content hijacking. In 2016 about about half of website visits where https. This is great and things seem to be only getting better. ISP's can not see the content in https traffic. Not seeing your traffic content anymore makes them sad. What makes them happy? They can still see all of your DNS requests.

Your ISP can see the websites you visit

Every ISP assigns you some of their DNS servers for you to use when you connect to them for your internet connection. Every time you type in a website name in your browser bar, this request goes to their DNS servers to look up an number called an IP address. After this happens an IP address is returned to your computer, and the connection to the website is made. Your ISP now has a log of the website you requested attached to the rest of the information they have about you. Then they build profiles about you, and sell that info to 3rd parties to target advertising to you in many different ways. Think you'll be slick and switch out their DNS servers with someone else's like Google's free DNS servers ( Think again. Any request through your ISP to any DNS server on the internet is unencrypted. Your ISP can slurp up all same requests and get the same info they did just like when you were using their DNS servers. Just like when they could see all of your content with http before https. This also means that your DNS traffic could possibly be intercepted and hijacked by other people or entities. TLS prevents this.

Securing the DNS

The thing that secures https is Transport Layer Security (TLS). It is a set of cryptographic protocols that provides communications security over a computer network. Now that we are beginning to secure the websites I think it is high time we secure the DNS. Others seem to agree. In 2016 rfc7858 and rfc8094 was submitted to Internet Engineering Task Force (IETF) which describes the use of DNS over TLS and DNS over DTLS. Hopefully these will eventually become a standard and all DNS traffic will be more secure in transit. Can you have DNS over TLS today? Yes you can!

Trying DNS over TLS now

DNS over TLS is in its infancy currently, but there are ways to try it out now. You could try using Stubby, a program that acts as a local DNS Privacy stub resolver (using DNS-over-TLS). You will have compile Stubby on Linux or OS X to use it. You could also setup your own DNS server at home and point it to some upstream forwarders that support DNS over TLS. This is what I have done to start testing this. I use Unbound as my local DNS server on my lan for all of my client machines. I used the configuration settings provided by our friends over at to setup my Ubound server to use DNS over TLS. Here is a list of other open source DNS software that can use DNS over TLS. With my Unbound setup, all of my DNS traffic is secured from interception and modification. So how is my testing going?

The early days

Since this is not a IETF standard yet, there are not a lot of providers of DNS over TLS resolvers. I have had to rearrange my list of DNS over TLS providers a few times when some of the servers were just not resolving hostnames. The latency is also higher than using your local ISP's DNS servers or using someones like Googles DNS servers. This is not very noticeable since my local DNS server caches the lookups. I have a feeling the generous providers of these DNS over TLS services are being overwhelmed and can not handle the load of the requests. This is where bigger companies come into play.

Places like Google or OpenDNS do not support DNS over TLS yet, but I'm hoping that they will get on board with this. Google especially since they have been big proponents of making all websites https. They also have the infrastructure to pull this off. Even if someone like Google turned this on, that means they get your DNS traffic instead of your ISP. Will this ever end?

Uggh, people can still see my traffic

Let's face it, if your connected to the internet, at some point someone gets to see where your going. You just have to choose who you want/trust to give this information to. If you point your DNS servers to Google they get to see your DNS requests. If I point my DNS at these test DNS over TLS servers then they get to see my DNS traffic. It seems like the lesser of 2 evils to send your DNS to 2nd party DNS servers then to your ISP. If you use your ISP's DNS servers they know the exact name attached to the IP address they assigned you and the customer that is making the query. I have been holding off telling you the bad news. https SNI will still give up the domain names you visit.

Through all of this even if you point your DNS traffic to a DNS over TLS server your ISP can still see many of the sites you go to. This is thanks to something in https called Server Name Indication (SNI). When you make a connection to an https enabled website there is a process called a handshake. This is the exchange of information before the encryption starts. During this unencrypted handshake (ClientHello) one of the things that is sent by you is a remote host name. This allows the server on the other end to choose appropriate certificate based on the requested host name. This happens when multiple virtual hosts reside on the same server, and this a very common setup. Unfortunately, your ISP can see this, slurp it up, and log this to your account/profile. So now what?

Would a VPN help? Yes, but remember now your DNS queries go to your VPN provider. What is nice is your ISP will not see any of your traffic anymore. That pesky SNI issue mentioned above goes away when using a VPN. But now your trusted endpoint is your VPN provider. They now can log all the sites you go to. So choose wisely when picking a VPN provider. Read their policy on saving logs, choose one that will allow you to pay with Bitcoin so you will be anonymous as possible. With a VPN provider you also have to be careful about DNS leaking. If your VPN client is not configured right, or you forget to turn it on or, any other myriad of ways a VPN can fail, your traffic will go right back to your ISP.

Even VPN's don't make you anonymous

So you have encrypted your DNS and web traffic with TLS and your using a VPN. Good for you, now your privacy is a bit better, but not anonymity. Your still being tracked. This time it is AD networks and services you use. I'm not going to go into this as many other people have written on this topic. Just know that your being tracked one way or another.

I know this all seems hopeless, but securing the web's infrasturcture bit by bit helps improve privacy just a little more. DNS like http is unencrypted. There was a big push to get websites to encrypt their data, now there needs to be the same attention given to DNS.

by at April 22, 2017 09:59 PM

April 19, 2017

That grumpy BSD guy

Forcing the password gropers through a smaller hole with OpenBSD's PF queues

While preparing material for the upcoming BSDCan PF and networking tutorial, I realized that the pop3 gropers were actually not much fun to watch anymore. So I used the traffic shaping features of my OpenBSD firewall to let the miscreants inflict some pain on themselves. Watching logs became fun again.

Yes, in between a number of other things I am currently in the process of creating material for new and hopefully better PF and networking session.

I've been fishing for suggestions for topics to include in the tutorials on relevant mailing lists, and one suggestion that keeps coming up (even though it's actually covered in the existling slides as well as The Book of PF) is using traffic shaping features to punish undesirable activity, such as

What Dan had in mind here may very well end up in the new slides, but in the meantime I will show you how to punish abusers of essentially any service with the tools at hand in your OpenBSD firewall.

Regular readers will know that I'm responsible for maintaining a set of mail services including a pop3 service, and that our site sees pretty much round-the-clock attempts at logging on to that service with user names that come mainly from the local part of the spamtrap addresses that are part of the system to produce our hourly list of greytrapped IP addresses.

But do not let yourself be distracted by this bizarre collection of items that I've maintained and described in earlier columns. The actual useful parts of this article follow - take this as a walkthrough of how to mitigate a wide range of threats and annoyances.

First, analyze the behavior that you want to defend against. In our case that's fairly obvious: We have a service that's getting a volume of unwanted traffic, and looking at our logs the attempts come fairly quickly with a number of repeated attempts from each source address. This similar enough to both the traditional ssh bruteforce attacks and for that matter to Dan's website scenario that we can reuse some of the same techniques in all of the configurations.

I've written about the rapid-fire ssh bruteforce attacks and their mitigation before (and of course it's in The Book of PF) as well as the slower kind where those techniques actually come up short. The traditional approach to ssh bruteforcers has been to simply block their traffic, and the state-tracking features of PF let you set up overload criteria that add the source addresses to the table that holds the addresses you want to block.

I have rules much like the ones in the example in place where there I have a SSH service running, and those bruteforce tables are never totally empty.

For the system that runs our pop3 service, we also have a PF ruleset in place with queues for traffic shaping. For some odd reason that ruleset is fairly close to the HFSC traffic shaper example in The Book of PF, and it contains a queue that I set up mainly as an experiment to annoy spammers (as in, the ones that are already for one reason or the other blacklisted by our spamd).

The queue is defined like this:

   queue spamd parent rootq bandwidth 1K min 0K max 1K qlimit 300

yes, that's right. A queue with a maximum throughput of 1 kilobit per second. I have been warned that this is small enough that the code may be unable to strictly enforce that limit due to the timer resolution in the HFSC code. But that didn't keep me from trying.

And now that I had another group of hosts that I wanted to just be a little evil to, why not let the password gropers and the spammers share the same small patch of bandwidth?

Now a few small additions to the ruleset are needed for the good to put the evil to the task. We start with a table to hold the addresses we want to mess with. Actually, I'll add two, for reasons that will become clear later:

table <longterm> persist counters
table <popflooders> persist counters 

The rules that use those tables are:

block drop log (all) quick from <longterm> 

pass in quick log (all) on egress proto tcp from <popflooders> to port pop3 flags S/SA keep state \ 
(max-src-conn 2, max-src-conn-rate 3/3, overload <longterm> flush global, pflow) set queue spamd 

pass in log (all) on egress proto tcp to port pop3 flags S/SA keep state \ 
(max-src-conn 5, max-src-conn-rate 6/3, overload <popflooders> flush global, pflow) 
The last one lets anybody connect to the pop3 service, but any one source address can have only open five simultaneous connections and at a rate of six over three seconds.

Any source that trips up one of these restrictions is overloaded into the popflooders table, the flush global part means any existing connections that source has are terminated, and when they get to try again, they will instead match the quick rule that assigns the new traffic to the 1 kilobyte queue.

The quick rule here has even stricter limits on the number of allowed simultaneous connections, and this time any breach will lead to membership of the longterm table and the block drop treatment.

The for the longterm table I already had in place a four week expiry (see man pfctl for detail on how to do that), and I haven't gotten around to deciding what, if any, expiry I will set up for the popflooders.

The results were immediately visible. Monitoring the queues using pfctl -vvsq shows the tiny queue works as expected:

 queue spamd parent rootq bandwidth 1K, max 1K qlimit 300
  [ pkts:     196136  bytes:   12157940  dropped pkts: 398350 bytes: 24692564 ]
  [ qlength: 300/300 ]
  [ measured:     2.0 packets/s, 999.13 b/s ]

and looking at the pop3 daemon's log entries, a typical encounter looks like this:

Apr 19 22:39:33 skapet spop3d[44875]: connect from
Apr 19 22:39:33 skapet spop3d[75112]: connect from
Apr 19 22:39:34 skapet spop3d[57116]: connect from
Apr 19 22:39:34 skapet spop3d[65982]: connect from
Apr 19 22:39:34 skapet spop3d[58964]: connect from
Apr 19 22:40:34 skapet spop3d[12410]: autologout time elapsed -
Apr 19 22:40:34 skapet spop3d[63573]: autologout time elapsed -
Apr 19 22:40:34 skapet spop3d[76113]: autologout time elapsed -
Apr 19 22:40:34 skapet spop3d[23524]: autologout time elapsed -
Apr 19 22:40:34 skapet spop3d[16916]: autologout time elapsed -

here the miscreant comes in way too fast and only manages to get five connections going before they're shunted to the tiny queue to fight it out with known spammers for a share of bandwidth.

I've been running with this particular setup since Monday evening around 20:00 CEST, and by late Wednesday evening the number of entries in the popflooders table had reached approximately 300.

I will decide on an expiry policy at some point, I promise. In fact, I welcome your input on what the expiry period should be.

One important takeaway from this, and possibly the most important point of this article, is that it does not take a lot of imagination to retool this setup to watch for and protect against undesirable activity directed at essentially any network service.

You pick the service and the ports it uses, then figure out what are the parameters that determine what is acceptable behavior. Once you have those parameters defined, you can choose to assign to a minimal queue like in this example, block outright, redirect to something unpleasant or even pass with a low probability.

All of those possibilities are part of the normal pf.conf toolset on your OpenBSD system. If you want, you can supplement these mechanisms with a bit of log file parsing that produces output suitable for feeding to pfctl to add to the table of miscreants. The only limits are, as always, the limits of your imagination (and possibly your programming abilities). If you're wondering why I like OpenBSD so much, you can find at least a partial answer in my OpenBSD and you presentation.

FreeBSD users will be pleased to know that something similar is possible on their systems too, only substituting the legacy ALTQ traffic shaping with its somewhat arcane syntax for the modern queues rules in this article.

Will you be attending our PF and networking session in Ottawa, or will you want to attend one elsewhere later? Please let us know at the email address in the tutorial description.

Update 2017-04-23: A truly unexpiring table, and downloadable datasets made available

Soon after publishing this article I realized that what I had written could easily be taken as a promise to keep a collection of POP3 gropers' IP addresses around indefinitely, in a table where the entries never expire.

Table entries do not expire unless you use a pfctl(8) command like the ones mentioned in the book and other resources I referenced earlier in the article, but on the other hand table entries will not survive a reboot either unless you arrange to have table contents stored to somewhere more permanent and restored from there. Fortunately our favorite toolset has a feature that implements at least the restoring part.

Changing the table definition quoted earler to read

 table <popflooders> persist counters file "/var/tmp/popflooders"

takes part of the restoring, and the backing up is a matter of setting up a cron(8) job to dump current contents of the table to the file that will be loaded into the table at ruleset load.

Then today I made another tiny change and made the data available for download. The popflooders table is dumped at five past every full hour to pop3gropers.txt, a file desiged to be read by anything that takes a list of IP addresses and ignores lines starting with the # comment character. I am sure you can think of suitable applications.

In addition, the same script does a verbose dump, including table statistiscs for each entry, to pop3gropers_full.txt for readers who are interested in such things as when an entry was created and how much traffic those hosts produced, keeping in mind that those hosts are not actually blocked here, only subjected to a tiny bandwidth.

As it says in the comment at the top of both files, you may use the data as you please for your own purposes, for any re-publishing or integration into other data sets please contact me via the means listed in the whois record.

As usual I will answer any reasonable requests for further data such as log files, but do not expect prompt service and keep in mind that I am usually in the Central European time zone (CEST at the moment).

I suppose we should see this as a tiny, incremental evolution of the "Cybercrime Robot Torture As A Service" (CRTAAS) concept.

Update 2017-04-29: While the world was not looking, I supplemented the IP address dumps with versions including one with geoiplocation data added and a per country summary based on the geoiplocation data.

Spending a few minutes with an IP address dump like the one described here and whois data is a useful excersise for anyone investigating incidents of this type. This .csv file is based on the 2017-04-29T1105 dump (preserved for reference), and reveals that not only is the majority of attempts from one country but also a very limited number of organizations within that country are responsible for the most active networks.

The spammer blacklist (see this post for background) was of course ripe for the same treatment, so now in addition to the familiar blacklist, that too comes with a geoiplocation annotated version and a per country summary.

Note that all of those files except the .csv file with whois data are products of automatic processes. Please contact me (the email address in the files works) if you have any questions or concerns.

Update 2017-05-17: After running with the autofilling tables for approximately a month, and I must confess, extracting bad login attempts that didn't actually trigger the overload at semi-random but roughly daily intervals, I thought I'd check a few things about the catch. I already knew roughly how many hosts total, but how many were contactin us via IPv6? Let's see:

[Wed May 17 19:38:02] peter@skapet:~$ doas pfctl -t popflooders -T show | wc -l
[Wed May 17 19:38:42] peter@skapet:~$ doas pfctl -t popflooders -T show | grep -c \:

Meaning, that of a total 5239 miscreant trapped, only 77, or just sort of 1.5 per ent tried contacting us via IPv6. The cybercriminals, or at least the literal bottom feeders like the pop3 password gropers, are still behind the times in a number of ways.

by Peter N. M. Hansteen ( at April 19, 2017 09:45 PM

Steve Kemp's Blog

3d-Printing is cool

I've heard about 3d-printing a lot in the past, although the hype seems to have mostly died down. My view has always been "That seems cool", coupled with "Everybody says making the models is very hard", and "the process itself is fiddly & time-consuming".

I've been sporadically working on a project for a few months now which displays tram-departure times, this is part of my drive to "hardware" things with Arduino/ESP8266 devices . Most visitors to our flat have commented on it, at least once, and over time it has become gradually more and more user-friendly. Initially it was just a toy-project for myself, so everything was hard-coded in the source but over time that changed - which I mentioned here, (specifically the Access-point setup):

  • When it boots up, unconfigured, it starts as an access-point.
    • So you can connect and configure the WiFi network it should join.
  • Once it's up and running you can point a web-browser at it.
    • This lets you toggle the backlight, change the timezone, and the tram-stop.
    • These values are persisted to flash so reboots will remember everything.

I've now wired up an input-button to the device too, experimenting with the different ways that a single button can carry out multiple actions:

  • Press & release - toggle the backlight.
  • Press & release twice - a double-click if you like - show a message.
  • Press, hold for 1 second, then release - re-sync the date/time & tram-data.

Anyway the software is neat, and I can't think of anything obvious to change. So lets move onto the real topic of this post: 3D Printing.

I randomly remembered that I'd heard about an online site holding 3D-models, and on a whim I searched for "4x20 LCD". That lead me to this design, which is exactly what I was looking for. Just like open-source software we're now living in a world where you can get open-source hardware! How cool is that?

I had to trust the dimensions of the model, and obviously I was going to mount my new button into the box, rather than the knob shown. But having a model was great. I could download it, for free, and I could view it online at

But with a model obtained the next step was getting it printed. I found a bunch of commercial companies, here in Europe, who would print a model, and ship it to me, but when I uploaded the model they priced it at €90+. Too much. I'd almost lost interest when I stumbled across a site which provides a gateway into a series of individual/companies who will print things for you, on-demand: 3dhubs.

Once again I uploaded my model, and this time I was able to select a guy in the same city as me. He printed my model for 1/3-1/4 of the price of the companies I'd found, and sent me fun pictures of the object while it was in the process of being printed.

To recap I started like this:


Then I boxed it in cardboard which looked better than nothing, but still not terribly great:


Now I've found an online case-design for free, got it printed cheaply by a volunteer (feels like the wrong word, after-all I did pay him), and I have something which look significantly more professional:


Inside it looks as neat as you would expect:

case internals

Of course the case still cost 5 times as much as the actual hardware involved (button: €0.05, processor-board €2.00 and LCD I2C display €3.00). But I've gone from being somebody who had zero experience with hardware-based projects 4 months ago, to somebody who has built a project which is functional and "pretty".

The internet really is a glorious thing. Using it for learning, and coding is good, using it for building actual physical parts too? That's something I never could have predicted a few years ago and I can see myself doing it more in the future.

Sure the case is a little rough around the edges, but I suspect it is now only a matter of time until I learn how to design my own models. An obvious extension is to add a status-LED above the switch, for example. How hard can it be to add a new hole to a model? (Hell I could just drill it!)

April 19, 2017 09:00 PM

April 15, 2017


Using stunnel and TinyProxy to obfuscate HTTP traffic

Recently there has been a lot of coverage in both tech and non-tech news outlets about internet privacy and how to prevent snooping both from service providers and governments. In this article I am going to show one method of anonymizing internet traffic; using a TLS enabled HTTP/HTTPS Proxy.

In this article we will walk through using stunnel to create a TLS tunnel with an instance of TinyProxy on the other side.

How does this help anonymize internet traffic

TinyProxy is an HTTP & HTTPS proxy server. By setting up a TinyProxy instance on a remote server and configuring our HTTP client to use this proxy. We can route all of our HTTP & HTTPS traffic through that remote server. This is a useful technique for getting around network restrictions that might be imposed by ISP's or Governments.

However, it's not enough to simply route HTTP/HTTPS traffic to a remote server. This in itself does not add any additional protection to the traffic. In fact, with an out of the box TinyProxy setup, all of the HTTP traffic to TinyProxy would still be unencrypted, leaving it open to packet capture and inspection. This is where stunnel comes into play.

I've featured it in earlier articles but for those who are new to stunnel, stunnel is a proxy that allows you to create a TLS tunnel between two or more systems. In this article we will use stunnel to create a TLS tunnel between the HTTP client system and TinyProxy.

By using a TLS tunnel between the HTTP client and TinyProxy our HTTP traffic will be encrypted between the local system and the proxy server. This means anyone trying to inspect HTTP traffic will be unable to see the contents of our HTTP traffic.

This technique is also useful for reducing the chances of a man-in-the-middle attack to HTTPS sites.

I say reducing because one of the caveats of this article is, while routing our HTTP & HTTPS traffic through a TLS tunneled HTTP proxy will help obfuscate and anonymize our traffic. The system running TinyProxy is still susceptible to man-in-the-middle attacks and HTTP traffic snooping.

Essentially, with this article, we are not focused on solving the problem of unencrypted traffic, we are simply moving our problem to a network where no one is looking. This is essentially the same approach as VPN service providers, the advantage of running your own proxy is that you control the proxy.

Now that we understand the end goal, let's go ahead and get started with the installation of TinyProxy.

Installing TinyProxy

The installation of TinyProxy is fairly easy and can be accomplished using the apt-get command on Ubuntu systems. Let's go ahead and install TinyProxy on our future proxy server.

server: $ sudo apt-get install tinyproxy

Once the apt-get command finishes, we can move to configuring TinyProxy.

Configuring TinyProxy

By default TinyProxy starts up listening on all interfaces for a connection to port 8888. Since we don't want to leave our proxy open to the world, let's change this by configuring TinyProxy to listen to the localhost interface only. We can do this by modifying the Listen parameter within the /etc/tinyproxy.conf file.



Replace With:


Once complete, we will need to restart the TinyProxy service in order for our change to take effect. We can do this using the systemctl command.

server: $ sudo systemctl restart tinyproxy

After systemctl completes, we can validate that our change is in place by checking whether port 8888 is bound correctly using the netstat command.

server: $ netstat -na
Active Internet connections (servers and established)
Proto Recv-Q Send-Q Local Address           Foreign Address         State      
tcp        0      0*               LISTEN

Based on the netstat output it appears TinyProxy is setup correctly. With that done, let's go ahead and setup stunnel.

Installing stunnel

Just like TinyProxy, the installation of stunnel is as easy as executing the apt-get command.

server: $ sudo apt-get install stunnel

Once apt-get has finished we will need to enable stunnel by editing the /etc/default/stunnel4 configuration file.


# Change to one to enable stunnel automatic startup


# Change to one to enable stunnel automatic startup

By default on Ubuntu, stunnel is installed in a disabled mode. By changing the ENABLED flag from 0 to 1 within /etc/default/stunnel4, we are enabling stunnel to start automatically. However, our configuration of stunnel does not stop there.

Our next step with stunnel, will involve defining our TLS tunnel.

TLS Tunnel Configuration (Server)

By default stunnel will look in /etc/stunnel for any files that end in .conf. In order to configure our TLS stunnel we will be creating a file named /etc/stunnel/stunnel.conf. Once created, we will insert the following content.

accept =
connect =
cert = /etc/ssl/cert.pem
key = /etc/ssl/key.pem

The contents of this configuration file are fairly straight forward, but let's go ahead and break down what each of these items mean. We will start with the accept option.

The accept option is similar to the listen option from TinyProxy. This setting will define what interface and port stunnel will listen to for incoming connections. By setting this to we are defining that stunnel should listen on all interfaces on port 3128.

The connect option is used to tell stunnel what IP and port to connect to. In our case this needs to be the IP and port that TinyProxy is listening on;

An easy way to remember how accept and connect should be configured is that accept is where incoming connections should come from, and connect is where they should go to.

Our next two configuration options are closely related, cert & key. The cert option is used to define the location of an SSL certificate that will be used to establish our TLS session. The key option is used to define the location of the key used to create the SSL certificate.

We will set these to be located in /etc/ssl and in the next step, we will go ahead and create both the key and certificate.

Creating a self-signed certificate

The first step in creating a self-signed certificate is to create an private key. To do this we will use the following openssl command.

server: $ sudo openssl genrsa -out /etc/ssl/key.pem 4096

The above will create a 4096 bit RSA key. From this key, we will create a public certificate using another openssl command. During the execution of this command there will be a series of questions. These questions are used to populate the key with organization information. Since we will be using this for our own purposes we will not worry too much about the answers to these questions.

server: $ sudo openssl req -new -x509 -key /etc/ssl/key.pem -out /etc/ssl/cert.pem -days 1826
You are about to be asked to enter information that will be incorporated
into your certificate request.
What you are about to enter is what is called a Distinguished Name or a DN.
There are quite a few fields but you can leave some blank
For some fields there will be a default value,
If you enter '.', the field will be left blank.
Country Name (2 letter code) [AU]:US
State or Province Name (full name) [Some-State]:Arizona
Locality Name (eg, city) []:Phoenix
Organization Name (eg, company) [Internet Widgits Pty Ltd]
Organizational Unit Name (eg, section) []:
Common Name (e.g. server FQDN or YOUR name) []
Email Address []

Once the questions have been answered the openssl command will create our certificate file.

After both the certificate and key have been created, we will need to restart stunnel in order for our changes to take effect.

server: $ sudo systemctl restart stunnel4

At this point we have finished configuring the proxy server. We now need to setup our client.

Setting up stunnel (client)

Like the proxy server, our first step in setting up stunnel on our client is installing it with the apt-get command.

client: $ sudo apt-get install stunnel

We will also once again need to enabling stunnel within the /etc/default/stunnel4 configuration file.


# Change to one to enable stunnel automatic startup


# Change to one to enable stunnel automatic startup

After enabling stunnel we will need to restart the service with the systemctl command.

client: $ sudo systemctl restart stunnel4

From here we can move on to configuring the stunnel client.

TLS Tunnel Configuration (Client)

The configuration of stunnel in "client-mode" is a little different than the "server-mode" configuration we set earlier. Let's go ahead and add our client configuration into the /etc/stunnel/stunnel.conf file.

client = yes

accept =
connect =
verify = 4
CAFile = /etc/ssl/cert.pem

As we did before, let's break down the configuration options shown above.

The first option is client, this option is simple, as it defines whether stunnel should be operating in a client or server mode. By setting this to yes, we are defining that we would like to use client mode.

We covered accept and connect before and if we go back to our description above we can see that stunnel will accept connections on and then tunnel them to, which is the IP and port that our stunnel proxy server is listening on.

The verify option is used to define what level of certificate validation should be performed. The option of 4 will cause stunnel to verify the remote certificate with a local certificate defined with the CAFile option. In the above example, I copied the /etc/ssl/cert.pem we generated on the server to the client and set this as the CAFile.

These last two options are important, without setting verify and CAFile stunnel will open an TLS connection without necessarily checking the validity of the certificate. By setting verify to 4 and CAFile to the same cert.pem we generated earlier, we are giving stunnel a way to validate the identity of our proxy server. This will prevent our client from being hit with a man-in-the-middle attack.

Once again, let's restart stunnel to make our configurations take effect.

client: $ sudo systemctl restart stunnel4

With our configurations complete, let's go ahead and test our proxy.

Testing our TLS tunneled HTTP Proxy

In order to test our proxy settings we will use the curl command. While we are using a command line web client in this article, it is possible to use this same type of configuration with GUI based browsers such as Chrome or Firefox.

Before testing however, I will need to set the http_proxy and https_proxy environmental variables. These will tell curl to leverage our proxy server.

client: $ export http_proxy="http://localhost:3128"
client: $ export https_proxy="https://localhost:3128"

With our proxy server settings in place, let's go ahead and execute our curl command.

client: $ curl -v
* Rebuilt URL to:
*   Trying
* Connected to localhost ( port 3128 (#0)
* Establish HTTP proxy tunnel to
> Host:
> User-Agent: curl/7.47.0
> Proxy-Connection: Keep-Alive
< HTTP/1.0 200 Connection established
< Proxy-agent: tinyproxy/1.8.3
* Proxy replied OK to CONNECT request
* found 173 certificates in /etc/ssl/certs/ca-certificates.crt
* found 692 certificates in /etc/ssl/certs
* ALPN, offering http/1.1
* SSL connection using TLS1.2 / ECDHE_ECDSA_AES_128_GCM_SHA256
*    server certificate verification OK
*    server certificate status verification SKIPPED
*    common name: * (matched)
*    server certificate expiration date OK
*    server certificate activation date OK
*    certificate public key: EC
*    certificate version: #3
*    subject: C=US,ST=California,L=Mountain View,O=Google Inc,CN=*
*    start date: Wed, 05 Apr 2017 17:47:49 GMT
*    expire date: Wed, 28 Jun 2017 16:57:00 GMT
*    issuer: C=US,O=Google Inc,CN=Google Internet Authority G2
*    compression: NULL
* ALPN, server accepted to use http/1.1
> GET / HTTP/1.1
> Host:
> User-Agent: curl/7.47.0
> Accept: */*
< HTTP/1.1 301 Moved Permanently
< Location:
< Content-Type: text/html; charset=UTF-8
< Date: Fri, 14 Apr 2017 22:37:01 GMT
< Expires: Sun, 14 May 2017 22:37:01 GMT
< Cache-Control: public, max-age=2592000
< Server: gws
< Content-Length: 220
< X-XSS-Protection: 1; mode=block
< X-Frame-Options: SAMEORIGIN
< Alt-Svc: quic=":443"; ma=2592000; v="37,36,35"
<HTML><HEAD><meta http-equiv="content-type" content="text/html;charset=utf-8">
<H1>301 Moved</H1>
The document has moved
<A HREF="">here</A>.
* Connection #0 to host localhost left intact

From the above output we can see that our connection was routed through TinyProxy.

< Proxy-agent: tinyproxy/1.8.3

And we were able to connect to Google.

<HTML><HEAD><meta http-equiv="content-type" content="text/html;charset=utf-8">
<H1>301 Moved</H1>
The document has moved
<A HREF="">here</A>.

Given these results, it seems we have a working TLS based HTTP/HTTPS proxy. Since this proxy is exposed on the internet what would happen if this proxy was found by someone simply scanning subnets for nefarious purposes.

Securing our tunnel further with PreShared Keys

In theory as it stands today they could use our proxy for their own purposes. This means we need some way to ensure that only our client can use this proxy; enter PreShared Keys.

Much like an API token, stunnel supports an authentication method called PSK or PreShared Keys. This is essentially what it sounds like. A token that has been shared between the client and the server in advance and used for authentication. To enable PSK authentication we simply need to add the following two lines to the /etc/stunnel/stunnel.conf file on both the client and the server.

ciphers = PSK
PSKsecrets = /etc/stunnel/secrets

By setting ciphers to PSK we are telling stunnel to use PSK based authentication. The PSKsecrets option is used to provide stunnel a file that contains the secrets in a clientname:token format.

In the above we specified the /etc/stunnel/secrets file. Let's go ahead and create that file and enter a sample PreShared Key.


Since the /etc/stunnel/secrets file contains sensitive information. Let's ensure that the permissions on the file are set appropriately.

$ sudo chmod 600 /etc/stunnel/secrets

By setting the permissions to 600 we are ensuring only the root user (the owner of the file) can read this file. These permissions will prevent other users from accessing the file and stumbling across our authentication credentials.

After setting permissions we will need to once again restart the stunnel service.

$ sudo systemctl restart stunnel4

With our settings are complete on both the client and server, we now have a reasonably secure TLS Proxy service.


In this article we covered setting up both a TLS and HTTP Proxy. As I mentioned before this setup can be used to help hide HTTP and HTTPS traffic on a given network. However, it is important to remember that while this setup makes the client system a much more tricky target, the proxy server itself could still be targeted for packet sniffing and man-in-the-middle attacks.

The key is to wisely select the location of where the proxy server is hosted.

Have feedback on the article? Want to add your 2 cents? Shoot a tweet out about it and tag @madflojo.

Posted by Benjamin Cane

April 15, 2017 07:30 AM

April 13, 2017

Feeding the Cloud

Automatically renewing Let's Encrypt TLS certificates on Debian using Certbot

I use Let's Encrypt TLS certificates on my Debian servers along with the Certbot tool. Since I use the "temporary webserver" method of proving domain ownership via the ACME protocol, I cannot use the cert renewal cronjob built into Certbot.

Instead, this is the script I put in /etc/cron.daily/certbot-renew:


/usr/bin/certbot renew --quiet --pre-hook "/bin/systemctl stop apache2.service" --post-hook "/bin/systemctl start apache2.service"

pushd /etc/ > /dev/null
/usr/bin/git add letsencrypt
DIFFSTAT="$(/usr/bin/git diff --cached --stat)"
if [ -n "$DIFFSTAT" ] ; then
    /usr/bin/git commit --quiet -m "Renewed letsencrypt certs"
    echo "$DIFFSTAT"
popd > /dev/null

# Generate the right certs for ejabberd and znc
if test /etc/letsencrypt/live/ -nt /etc/ejabberd/ejabberd.pem ; then
    cat /etc/letsencrypt/live/ /etc/letsencrypt/live/ > /etc/ejabberd/ejabberd.pem
cat /etc/letsencrypt/live/ /etc/letsencrypt/live/ > /home/francois/.znc/znc.pem

It temporarily disables my Apache webserver while it renews the certificates and then only outputs something to STDOUT (since my cronjob will email me any output) if certs have been renewed.

Since I'm using etckeeper to keep track of config changes on my servers, my renewal script also commits to the repository if any certs have changed.

Finally, since my XMPP server and IRC bouncer need the private key and the full certificate chain to be in the same file, so I regenerate these files at the end of the script. In the case of ejabberd, I only do so if the certificates have actually changed since overwriting ejabberd.pem changes its timestamp and triggers an fcheck notification (since it watches all files under /etc).

External Monitoring

In order to catch mistakes or oversights, I use ssl-cert-check to monitor my domains once a day:

ssl-cert-check -s -p 443 -q -a -e

I also signed up with Cert Spotter which watches the Certificate Transparency log and notifies me of any newly-issued certificates for my domains.

In other words, I get notified:

  • if my cronjob fails and a cert is about to expire, or
  • as soon as a new cert is issued.

The whole thing seems to work well, but if there's anything I could be doing better, feel free to leave a comment!

April 13, 2017 03:00 PM

Distributed load testing with Tsung

At $dayjob I manage a large OpenStack Cloud. Next to that I also build high-performance and redundant clusters for customers. Think multiple datacenters, haproxy, galera or postgres or mysql replication, drbd with nfs or glusterfs and all sorts of software that can (and sometimes cannot) be clustered (redis, rabbitmq etc.). Our customers deploy their application on there and when one or a few components fail, their application stays up. Hypervisors, disks, switches, routers, all can fail without actual service downtime. Next to building such clusters, we also monitor and manage them. When we build such a cluster (fully automated with Ansible) we do a basic load test. We do this not for benchmarking or application flow testing, but to optimize the cluster components. Simple things like the mpm workers or threads in Apache or more advanced topics like MySQL or DRBD. Optimization there depends on the specifications of the servers used and the load patterns. Tsung is a high-performance but simple to configure and use piece of software written in Erlang. Configuration is done in a simple readable XML file. Tsung can be run distributed as well for large setups. It has good reporting and a live web interface for status and reports during a test.

April 13, 2017 12:00 AM

April 12, 2017

Vincent Bernat

Proper isolation of a Linux bridge

TL;DR: when configuring a Linux bridge, use the following commands to enforce isolation:

# bridge vlan del dev br0 vid 1 self
# echo 1 > /sys/class/net/br0/bridge/vlan_filtering

A network bridge (also commonly called a “switch”) brings several Ethernet segments together. It is a common element in most infrastructures. Linux provides its own implementation.

A typical use of a Linux bridge is shown below. The hypervisor is running three virtual hosts. Each virtual host is attached to the br0 bridge (represented by the horizontal segment). The hypervisor has two physical network interfaces:

  • eth0 is attached to a public network providing various services for the virtual hosts (DHCP, DNS, NTP, routers to Internet, …). It is also part of the br0 bridge.
  • eth1 is attached to an infrastructure network providing various services to the hypervisor (DNS, NTP, configuration management, routers to Internet, …). It is not part of the br0 bridge.

Typical use of Linux bridging with virtual machines

The main expectation of such a setup is that while the virtual hosts should be able to use resources from the public network, they should not be able to access resources from the infrastructure network (including resources hosted on the hypervisor itself, like a SSH server). In other words, we expect a total isolation between the green domain and the purple one.

That’s not the case. From any virtual host:

# ip route add dev eth0
# ping -c 3
PING ( 56(84) bytes of data.
64 bytes from icmp_seq=1 ttl=59 time=0.644 ms
64 bytes from icmp_seq=2 ttl=59 time=0.829 ms
64 bytes from icmp_seq=3 ttl=59 time=0.894 ms

--- ping statistics ---
3 packets transmitted, 3 received, 0% packet loss, time 2033ms
rtt min/avg/max/mdev = 0.644/0.789/0.894/0.105 ms


There are two main factors of this behavior:

  1. A bridge can accept IP traffic. This is a useful feature if you want Linux to act as a bridge and provide some IP services to bridge users (a DHCP relay or a default gateway). This is usually done by configuring the IP address on the bridge device: ip addr add dev br0.
  2. An interface doesn’t need an IP address to process incoming IP traffic. Additionally, by default, Linux accepts to answer ARP requests independently from the incoming interface.

Bridge processing

After turning an incoming Ethernet frame into a socket buffer, the network driver transfers the buffer to the netif_receive_skb() function. The following actions are executed:

  1. copy the frame to any registered global or per-device taps (e.g. tcpdump),
  2. evaluate the ingress policy (configured with tc),
  3. hand over the frame to the device-specific receive handler, if any,
  4. hand over the frame to a global or device-specific protocol handler (e.g. IPv4, ARP, IPv6).

For a bridged interface, the kernel has configured a device-specific receive handler, br_handle_frame(). This function won’t allow any additional processing in the context of the incoming interface, except for STP and LLDP frames or if “brouting” is enabled1. Therefore, the protocol handlers are never executed in this case.

After a few additional checks, Linux will decide if the frame has to be locally delivered:

  • the entry for the target MAC in the FDB is marked for local delivery, or
  • the target MAC is a broadcast or a multicast address.

In this case, the frame is passed to the br_pass_frame_up() function. A VLAN-related check is optionally performed. The socket buffer is attached to the bridge interface (br0) instead of the physical interface (eth0), is evaluated by Netfilter and sent back to netif_receive_skb(). It will go through the four steps a second time.

IPv4 processing

When a device doesn’t have a protocol-independent receive handler, a protocol-specific handler will be used:

# cat /proc/net/ptype
Type Device      Function
0800          ip_rcv
0011          llc_rcv [llc]
0004          llc_rcv [llc]
0806          arp_rcv
86dd          ipv6_rcv

Therefore, if the Ethernet type of the incoming frame is 0x800, the socket buffer is handled by ip_rcv(). Among other things, the three following steps will happen:

  • If the frame destination address is not the MAC address of the incoming interface, not a multicast one and not a broadcast one, the frame is dropped (“not for us”).
  • Netfilter gets a chance to evaluate the packet (in a PREROUTING chain).
  • The routing subsystem will decide the destination of the packet in ip_route_input_slow(): is it a local packet, should it be forwarded, should it be dropped, should it be encapsulated? Notably, the reverse-path filtering is done during this evaluation in fib_validate_source().

Reverse-path filtering (also known as uRPF, or unicast reverse-path forwarding, RFC 3704) enables Linux to reject traffic on interfaces which it should never have originated: the source address is looked up in the routing tables and if the outgoing interface is different from the current incoming one, the packet is rejected.

ARP processing

When the Ethernet type of the incoming frame is 0x806, the socket buffer is handled by arp_rcv().

  • Like for IPv4, if the frame is not for us, it is dropped.
  • If the incoming device has the NOARP flag, the frame is dropped.
  • Netfilter gets a chance to evaluate the packet (configuration is done with arptables).
  • For an ARP request, the values of arp_ignore and arp_filter may trigger a drop of the packet.

IPv6 processing

When the Ethernet type of the incoming frame is 0x86dd, the socket buffer is handled by ipv6_rcv().

  • Like for IPv4, if the frame is not for us, it is dropped.
  • If IPv6 is disabled on the interface, the packet is dropped.
  • Netfilter gets a chance to evaluate the packet (in a PREROUTING chain).
  • The routing subsystem will decide the destination of the packet. However, unlike IPv4, there is no reverse-path filtering2.


There are various methods to fix the situation.

We can completely ignore the bridged interfaces: as long as they are attached to the bridge, they cannot process any upper layer protocol (IPv4, IPv6, ARP). Therefore, we can focus on filtering incoming traffic from br0.

It should be noted that for IPv4, IPv6 and ARP protocols, the MAC address check can be circumvented by using the broadcast MAC address.

Protocol-independent workarounds

The four following fixes will indistinctly drop IPv4, ARP and IPv6 packets.

Using VLAN-aware bridge

Linux 3.9 introduced the ability to use VLAN filtering on bridge ports. This can be used to prevent any local traffic:

# echo 1 > /sys/class/net/br0/bridge/vlan_filtering
# bridge vlan del dev br0 vid 1 self
# bridge vlan show
port    vlan ids
eth0     1 PVID Egress Untagged
eth2     1 PVID Egress Untagged
eth3     1 PVID Egress Untagged
eth4     1 PVID Egress Untagged
br0     None

This is the most efficient method since the frame is dropped directly in br_pass_frame_up().

Using ingress policy

It’s also possible to drop the bridged frame early after it has been re-delivered to netif_receive_skb() by br_pass_frame_up(). The ingress policy of an interface is evaluated before any handler. Therefore, the following commands will ensure no local delivery (the source interface of the packet is the bridge interface) happens:

# tc qdisc add dev br0 handle ffff: ingress
# tc filter add dev br0 parent ffff: u32 match u8 0 0 action drop

In my opinion, this is the second most efficient method.

Using ebtables

Just before re-delivering the frame to netif_receive_skb(), Netfilter gets a chance to issue a decision. It’s easy to configure it to drop the frame:

# ebtables -A INPUT --logical-in br0 -j DROP

However, to the best of my knowledge, this part of Netfilter is known to be inefficient.

Using namespaces

Isolation can also be obtained by moving all the bridged interfaces into a dedicated network namespace and configure the bridge inside this namespace:

# ip netns add bridge0
# ip link set netns bridge0 eth0
# ip link set netns bridge0 eth2
# ip link set netns bridge0 eth3
# ip link set netns bridge0 eth4
# ip link del dev br0
# ip netns exec bridge0 brctl addbr br0
# for i in 0 2 3 4; do
>    ip netns exec bridge0 brctl addif br0 eth$i
>    ip netns exec bridge0 ip link set up dev eth$i
> done
# ip netns exec bridge0 ip link set up dev br0

The frame will still wander a bit inside the IP stack, wasting some CPU cycles and increasing the possible attack surface. But ultimately, it will be dropped.

Protocol-dependent workarounds

Unless you require multiple layers of security, if one of the previous workarounds is already applied, there is no need to apply one of the protocol-dependent fix below. It’s still interesting to know them because it is not uncommon to already have them in place.


The easiest way to disable ARP processing on a bridge is to set the NOARP flag on the device. The ARP packet will be dropped as the very first step of the ARP handler.

# ip link set arp off dev br0
# ip l l dev br0
8: br0: <BROADCAST,MULTICAST,NOARP,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP mode DEFAULT group default qlen 1000
    link/ether 50:54:33:00:00:04 brd ff:ff:ff:ff:ff:ff

arptables can also drop the packet quite early:

# arptables -A INPUT -i br0 -j DROP

Another way is to set arp_ignore to 2 for the given interface. The kernel will only answer to ARP requests whose target IP address is configured on the incoming interface. Since the bridge interface doesn’t have any IP address, no ARP requests will be answered.

# sysctl -qw net.ipv4.conf.br0.arp_ignore=2

Disabling ARP processing is not a sufficient workaround for IPv4. A user can still insert the appropriate entry in its neighbor cache:

# ip neigh replace lladdr 50:54:33:00:00:04 dev eth0
# ping -c 1
PING ( 56(84) bytes of data.
64 bytes from icmp_seq=1 ttl=49 time=1.30 ms

--- ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 1.309/1.309/1.309/0.000 ms

As the check on the target MAC address is quite loose, they don’t even need to guess the MAC address:

# ip neigh replace lladdr ff:ff:ff:ff:ff:ff dev eth0
# ping -c 1
PING ( 56(84) bytes of data.
64 bytes from icmp_seq=1 ttl=49 time=1.12 ms

--- ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 1.129/1.129/1.129/0.000 ms


The earliest place to drop an IPv4 packet is with Netfilter3:

# iptables -t raw -I PREROUTING -i br0 -j DROP

If Netfilter is disabled, another possibility is to enable strict reverse-path filtering for the interface. In this case, since there is no IP address configured on the interface, the packet will be dropped during the route lookup:

# sysctl -qw net.ipv4.conf.br0.rp_filter=1

Another option is the use of a dedicated routing rule. Compared to the reverse-path filtering option, the packet will be dropped a bit earlier, still during the route lookup.

# ip rule add iif br0 blackhole


Linux provides a way to completely disable IPv6 on a given interface. The packet will be dropped as the very first step of the IPv6 handler:

# sysctl -qw net.ipv6.conf.br0.disable_ipv6=1

Like for IPv4, it’s possible to use Netfilter or a dedicated routing rule.

About the example

In the above example, the virtual host get ICMP replies because they are routed through the infrastructure network to Internet (e.g. the hypervisor has a default gateway which also acts as a NAT router to Internet). This may not be the case.

If you want to check if you are “vulnerable” despite not getting an ICMP reply, look at the guest neighbor table to check if you got an ARP reply from the host:

# ip route add dev eth0
# ip neigh show dev eth0 lladdr 50:54:33:00:00:04 REACHABLE

If you didn’t get a reply, you could still have issues with IP processing. Add a static neighbor entry before checking the next step:

# ip neigh replace lladdr ff:ff:ff:ff:ff:ff dev eth0

To check if IP processing is enabled, check the bridge host’s network statistics:

# netstat -s | grep "ICMP messages"
    15 ICMP messages received
    15 ICMP messages sent
    0 ICMP messages failed

If the counters are increasing, it is processing incoming IP packets.

One-way communication still allows a lot of bad things, like DoS attacks. Additionally, if the hypervisor happens to also act as a router, the reach is extended to the whole infrastructure network, potentially exposing weak devices (e.g. PDU) exposing an SNMP agent. If one-way communication is all that’s needed, the attacker can also spoof its source IP address, bypassing IP-based authentication.

About MACVLAN interfaces

When source-address learning is not needed, bridges can be replaced by MACVLAN or MACVTAP interfaces. Have a look at the following article for more details: “Bridge vs Macvlan”.

In this case, the packet processing is different and asymmetric. When a frame exits the MACVLAN interface, it is directly transmitted to the appropriate interface (either another MACVLAN interface or the lower device). In this direction, communication with the host is not possible.

However, when a frame enters the lower device, if the destination MAC is not one of the sub-interfaces, the frame is processed by the host using the appropriate protocol handler (ARP, IPv4, IPv6). Therefore, isolation can only be obtained by using the protocol-dependent methods.

  1. A frame can be forcibly routed (L3) instead of bridged (L2) by “brouting” the packet. This action can be triggered using ebtables

  2. For IPv6, reverse-path filtering needs to be implemented with Netfilter, using the rpfilter match

  3. If the br_netfilter module is loaded, net.bridge.bridge-nf-call-ipatbles sysctl has to be set to 0. Otherwise, you also need to use the physdev match to not drop IPv4 packets going through the bridge. 

by Vincent Bernat at April 12, 2017 07:58 AM

April 11, 2017

Exploring Docker overlay networks

Docker In the past months I have made several attempts to explore Docker overlay networks, but there were a few pieces to set up before I could really experiment and… well, let’s say that I have probably approached the problem the wrong way and wasted some time along the way. Not again. I have set aside some time and worked agile enough to do the whole job, from start to finish. Nowadays there is little point in creating overlay networks by hand, except that it’s still a good learning experience. And a learning experience with Docker and networking was exactly what I was after.

When I started exploring multi-host Docker networks, Docker was quite different than it is now. In particular, Docker Swarm didn’t exist yet, and there was a certain amount of manual work required in order to create an overlay network, so that containers located in different hosts can communicate.

Before Swarm, in order to set up an overlay network one needed to:

  • have at least two docker hosts to establish an overlay network;
  • have a supported key/value store available for the docker hosts to sync information;
  • configure the docker hosts to use the key/value store;
  • create an overlay network on one of the docker host; if everything worked well, the network will “propagate” to the other docker hosts that had registered with the key/value store;
  • create named containers on different hosts; then try and ping each other using the names: if everything was done correctly, you would be able to ping the containers through the overlay network.

This looks like simple high-level checklist. I’ll now describe the actual steps needed to get this working, leaving the details of my failuers to the last section of this post.

Overlay networks for the impatient

A word of caution

THIS SETUP IS COMPLETELY INSECURE. Consul interfaces are exposed on the public interface of the host with no authentication. Consul communication is not encrypted. The traffic over the overlay network itself is not encrypted nor protected in any way. Using this set-up in any real-life environment is strongly discouraged.

Hardware set-up

I used three hardware machines, all running Debian Linux “Jessie”; all machines got a dynamic IPv4 address from DHCP

  • murray:, running consul in server mode;
  • tyrrell:, docker host on a 4.9 Linux kernel installed from backports;
  • brabham:, docker host, same configuration as tyrrell.

Running the key-value store

create a directory structure, e.g.:

mkdir consul
cd consul
mkdir bin config data

Download consul and unpack it in the bin subdirectory.

Finally, download the script from my github account into the consul directory you created, make it executable and run it:

chmod a+x

If everything goes well, you’ll now have a consul server running on your machine. If not, you may have something to tweak in the script. In that case, please refer to the comments in the script itself.

On murray, the script started consul successfully. These are the messages I got on screen:

bronto@murray:~/Downloads/consul$ ./ 
Default route on interface wlan0
The IPv4 address on wlan0 is
The data directory for consul is /home/bronto/Downloads/consul/data
The config directory for consul is /home/bronto/Downloads/consul/config
Web UI available at
DNS server available at port 8600 (TCP/UDP)
==> WARNING: Bootstrap mode enabled! Do not enable unless necessary
==> Starting Consul agent...
==> Consul agent running!
           Version: 'v0.8.0'
           Node ID: 'eeaf4121-262a-4799-a175-d7037d20695f'
         Node name: 'consul-master-murray'
        Datacenter: 'dc1'
            Server: true (bootstrap: true)
       Client Addr: (HTTP: 8500, HTTPS: -1, DNS: 8600)
      Cluster Addr: (LAN: 8301, WAN: 8302)
    Gossip encrypt: false, RPC-TLS: false, TLS-Incoming: false

==> Log data will now stream in as it occurs:

    2017/04/11 14:53:56 [INFO] raft: Initial configuration (index=1): [{Suffrage:Voter ID: Address:}]
    2017/04/11 14:53:56 [INFO] raft: Node at [Follower] entering Follower state (Leader: "")
    2017/04/11 14:53:56 [INFO] serf: EventMemberJoin: consul-master-murray
    2017/04/11 14:53:56 [WARN] serf: Failed to re-join any previously known node
    2017/04/11 14:53:56 [INFO] consul: Adding LAN server consul-master-murray (Addr: tcp/ (DC: dc1)
    2017/04/11 14:53:56 [INFO] serf: EventMemberJoin: consul-master-murray.dc1
    2017/04/11 14:53:56 [WARN] serf: Failed to re-join any previously known node
    2017/04/11 14:53:56 [INFO] consul: Handled member-join event for server "consul-master-murray.dc1" in area "wan"
    2017/04/11 14:54:03 [ERR] agent: failed to sync remote state: No cluster leader
    2017/04/11 14:54:06 [WARN] raft: Heartbeat timeout from "" reached, starting election
    2017/04/11 14:54:06 [INFO] raft: Node at [Candidate] entering Candidate state in term 3
    2017/04/11 14:54:06 [INFO] raft: Election won. Tally: 1
    2017/04/11 14:54:06 [INFO] raft: Node at [Leader] entering Leader state
    2017/04/11 14:54:06 [INFO] consul: cluster leadership acquired
    2017/04/11 14:54:06 [INFO] consul: New leader elected: consul-master-murray
    2017/04/11 14:54:07 [INFO] agent: Synced node info

Note that the consul web UI is available (on murray, it was on as you can read from the output of the script). I suggest that you take a look at it while you progress in this procedure; in particular, look at how the objects in the KEY/VALUE section change when the docker hosts join the cluster and when the overlay network is created.

Using the key/value store in docker engine

Since my set-up was not permanent (I just wanted to test out the overlay networks but I don’t mean to have a node always on and exposing the key/value store), I did only a temporary change in Docker Engine’s configuration by stopping the system service and then running the docker daemon by hand. That’s not something you would do in production. If you are interested in making the change permanent, please look at the wonderful article in the “Technical scratchpad” blog.

On both docker hosts I ran:

systemctl stop docker.service

Then on brabham I ran:

/usr/bin/dockerd -H unix:///var/run/docker.sock --cluster-store consul:// --cluster-advertise=

while on tyrrell I ran:

/usr/bin/dockerd -H unix:///var/run/docker.sock --cluster-store consul:// --cluster-advertise=

If everything goes well, messages on screen will confirm that you have successully joined consul. E.g.: this was on brabham:

INFO[0001] 2017/04/10 19:55:44 [INFO] serf: EventMemberJoin: brabham

This one was also on brabham, letting us know that tyrrell had also joined the family:

INFO[0058] 2017/04/10 19:56:41 [INFO] serf: EventMemberJoin: tyrrell

The parameter --cluster-advertise is quite important. If you don’t include it, the creation of the overlay network at the following step will succeed, but containers in different hosts will fail to communicate through the overlay network, which will make the network itself pretty pointless.

Creating the overlay network

On one of the docker hosts, create an overlay network. In my case, I ran the following command on tyrrell to create the network to span both hosts, and name it bronto-overlay:

$ docker network create -d overlay --subnet= bronto-overlay

The output, a long network ID, looks promising. Let’s verify that the network is actually created:

$ docker network ls
NETWORK ID          NAME                DRIVER              SCOPE
3277557c8f42        bridge              bridge              local
80dfa291f573        bronto-bridged      bridge              local
c4bcd49e4c02        bronto-overlay      overlay             global
dee1b78004ef        host                host                local
f1fb0c7be326        none                null                local

Promising indeed: there is a bronto-overlay network and the scope is global. If the command really succeeded, we should find it on brabham, too:

$ docker network ls
NETWORK ID          NAME                DRIVER              SCOPE
90221b0a57e1        bridge              bridge              local
c4bcd49e4c02        bronto-overlay      overlay             global
3565b28a327b        docker_gwbridge     bridge              local
f4691f3066cb        host                host                local
86bdd8ab9b90        none                null                local

There it is. Good.

Test that everything works

With the network set up, I should be able to create one container on each host, bound to the overlay network, and they should be able to ping each other. Not only that: they should be able to resolve each other’s names through Docker’s DNS service and despite being on different machines. Let’s see.

On each machine I ran:

docker run --rm -it --network bronto-overlay --name in-$HOSTNAME debian /bin/bash

That will create a container named in-tyrrell on tyrrell and a container named in-brabham on brabham. Their main network interface will appear to be bound to the overlay network. E.g. on in-tyrrell:

root@d86df4662d7e:/# ip addr show dev eth0
68: eth0@if69: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1450 qdisc noqueue state UP group default 
    link/ether 02:42:0a:11:00:03 brd ff:ff:ff:ff:ff:ff
    inet scope global eth0
       valid_lft forever preferred_lft forever

We now try to ping the other container:

root@d86df4662d7e:/# ping in-brabham
PING in-brabham ( 56 data bytes
64 bytes from icmp_seq=0 ttl=64 time=0.857 ms
64 bytes from icmp_seq=1 ttl=64 time=0.618 ms
^C--- in-brabham ping statistics ---
2 packets transmitted, 2 packets received, 0% packet loss
round-trip min/avg/max/stddev = 0.618/0.738/0.857/0.120 ms


How things really went

Things weren’t so straightforward in reality.

I will use a consul container. Or not.

When I was designing my experiment I thought I would run consul through a consul container in one of the two hosts. I thus spent some time playing with the progrium/consul container, as indicated by Docker’s documentation, until I found out that it wasn’t being updated since two years… You can find the outcast of my experiment with progrium/consul on github.

I thus switched to Docker’s official consul image. It was similar enough to progrium/consul to seem an easy port, and different enough that it made me hit the wall a few times before I got things right with that image, too. With the help of a couple of scripts I could now easily start a server container and a client container. You can find the outcast of my experiments with docker’s consul image on github, too.

Then, I understood that the whole thing was pointless. You don’t need to have a consul client running on the docker host, because the docker engine itself will be the client. And you can’t have the consul server running as a container on one of the docker hosts. In fact, the docker engine itself needs to register at start-up with the consul server, but the consul server won’t be running because docker hasn’t started… ouch!

That wasn’t a complete waste of time anyway. In fact, with the experience acquired in making consul containers work and the scripts that I had already written, it was an easy task to configure a consul server on a separate laptop (murray). The script that I used to run the consul server is also available on github.

The documentation gap

When I started playing with docker, Docker Swarm didn’t exist yet, or was still in a very primitive stage. The only way to make containers on different hosts communicate were overlay networks and there was no automated procedure to create them, therefore Docker’s documentation explained in some details how they should be set up. With the release of Swarm, overlay networks are created automatically by Swarm itself and not much about the “traditional” set up of the overlay networks was left in the documentation. Walking through the “time machine” in the docs web page proved to be problematic, but luckily enough I found this article on the Technical Scratchpad blog that helped me to connect the dots.

Don’t believe in magic

When I initially started the docker engine and bound it to the consul server I left out the --cluster-advertise option. All in all, docker must be smart enough to guess the right interface for that, right?

No, wrong.

When I did leave the option out, everything seemed to work. In particular, when I created the overlay network on tyrrell I could also see it on brabham. And when I created a container on each host, docker network inspect bronto-overlay showed that both containers were registered on the network. Except that they couldn’t communicate. The problem seems to be known, and things started to work only when I added the --cluster-advertise option with a proper value for each host.

Tagged: consul, Debian, docker, linux, networking

by bronto at April 11, 2017 02:42 PM

Sarah Allen

the making of a fireplace

I built a fake fireplace in celebration of our team shipping Cloud Functions for Firebase. The codename was “hearth” — the product was almost ready for public beta by the time I joined the team, yet I know it was an epic journey to create a long-awaited feature and I felt the milestone deserved dramatic punctuation.

Red brick surrounds a monitor with video of log fire, wooden top has rows of whiskey bottles and a card

If you would like to build your very own fake fireplace, here’s how…

Initial Research

I reached out to our helpful facilities staff to make sure that I understood the constraints of the space. (I didn’t want to buy or create something that would be immediately taken down!) James Miller from facilities investigated some options:

“Since a hearth is technically the flat stone floor or the part outside of the fireplace, putting up just a fireplace and mantel leaves out the most important part, but including it presents a tripping hazard.” He suggested a quick and easy solution would be to temporarily change the display play a 8hr yule log video or on extra computer monitor with some fake foam bricks. He noted that some prop fireplaces which look pretty good in the photos, but are just cardboard. There are more expensive electronic fireplaces, but we can’t have anything that includes heating features similar to a space heater, and he reported “most that I’ve seen online are also missing the most important part… the actual hearth.”

Initially I felt that a literal interpretation of the “hearth” code name was not a critical element. We explored buying a real fake fireplace, but the ones we found were very small and we felt they were very expensive relative to the potential impact. However, his thoughts on the meaning of “hearth” later led to some important design elements. I love being involved in a collaborative creative process!

James suggested that something in a stage prop style would have dramatic effect, perhaps using styrofoam so that it would be light and easy to move, if we needed to shift it in the future around team moves. In brainstorming location, he helped me remember to pick out a spot with an electrical outlet.

Measure Twice, Cut Once

an aisle with people at desks on either side, at the end is a small bookshelf with bottles of alcohol and above that a large monitor on the wallI surreptitiously took a photo of the area, and arrived early one morning with a tape measure to determine its maximum dimension.I then measured the computer monitor I planned to use to determine the minimum internal dimension.

There was actually a third, very important measurement: the internal size of my car, which later became a critical optimization. I designed the fireplace in three pieces, so it could fit into the car, which also made it much lighter and easier to transport. Styrofoam is rather fragile, and the smaller pieces fit easily through doors.

Tools & Supplies

Tools: tape measure, hammer, screw driver, propane torch, safety googles, mask, electric drill, electric sander, garden shears, mat knife, level, steel square

Supplies: small nails, wood, styrofoam panels, gorilla glue, sandpaper, acrylic paint (red, yellow, white, black), gloss latex paint (off-white), sharpie, cardboard

I had never worked with styrofoam before and read that it could be carved with a sharp knife, acetone or a hot knife. If it were a small project, a regular soldering iron would likely have been fine, but I wanted something that would be fast and really liked that the propane torch I found had two different tips as well as an open flame option. The small hand-held propane filled torch is also easier to work with than my electric soldering iron which requires an extension cord.

Construction Details

I tested carving and painting the styrofoam very early, including figuring out how long the paint took to dry. I found that painting the white cracks before painting the red bricks made it come out a bit better. It was also good to test carving to create an effect that looks like mortar.

See key steps in photos

  1. Wooden frame: I uses 1″ thick boards for the frame. Discount Builders in San Francisco has a friendly staff and cut long 12′ or 8′ boards into the lengths I needed. The hearth is a separate component, which is like a hollow box with one side missing, where the missing side is actually attached to the larger frame, so the top can rest on it for stability and is not actually physically attached. I used a level and steel square for an even frame and wood screws.
  2. Top: I used a single 1″ thick board and drilled small holes for pegs to attach the top to the frame in a way that could be easily removed, yet fit snugly. I used garden shears to cut a long wooden dowel into 1″ lengths to create the pegs. I used an electric sander for rounded edges on three sides, and finished with fine sandpaper. I then coated with a few coats of gloss paint (which would have been easier if I had remembered primer!).
  3. Styrofoam: for the large panels I found that using small nails to hold the styrofoam in place made it easier to keep everything steady while gluing it all together. I attached all of the styrofoam panels before carving bricks with the torch. Large cuts were most effective with a mat knife.
  4. Carving the bricks: I found a few inspirational Fireplace images via Google Image Search and drew bricks with a Sharpie, then traced the lines with the torch using the soldering gun tip. Then I went back and made the brick edges a bit irregular and created a few cracks in random bricks. I also used an open flame in areas to create more texture.
  5. Painting the bricks: I mixed acrylic paint with a little dish soap and water, which makes it easier to coat the styrofoam. I noticed that most fireplaces have irregular brick colors, so I varied the colors a bit with yellow and brown highlights, as well as black “sooty” areas.
  6. The inside is painted black with a piece of cardboard as backing.

Additional Resources

I studied sculpture in college, so I had practical experience with wood construction, and learned a bit more from the amazing people who publish about their experience on the web. Here are some links to resources that I found particularly helpful.



by sarah at April 11, 2017 03:59 AM

April 10, 2017

Run software on the tty1 console instead of getty login on Ubuntu 14.04 and 16.04

Recently I wanted to change the default login prompt on the tty1 console on an OpenStack instance to automatically run htop. Instead of logging in via the console, I wanted it to start up htop right away and nothing else. Ubuntu 14.04 uses init and Ubuntu 16.04 uses systemd. Both ways are shown in this tutorial.

April 10, 2017 12:00 AM

April 01, 2017

Improving your services, the DevOps way

devops-italiaOn March 10th I was in Bologna for Incontro DevOps Italia 2017, the Italian DevOps meeting organized by the great people at BioDec. The three tracks featured several talks in both Italian and English, and first-class international speakers. And, being a conference in Bologna, it also featured first-class local food that no other conference around the world will ever be able to match.

In my presentation, “Improving your services, the DevOps way”, I illustrated the methodology we used in Opera to improve our mail service with a rolling approach, where the new infrastructure was built alongside the existing one through collaboration and a mix of agile and DevOps techniques, and progressively replaced it. The slides are available on SpeakerDeck.

I understand from the organizers that the recordings will be available some time in April. However, they will be of little use for non-Italian speakers, since at the very last minute I decided to hold the speech in Italian — a choice that I deeply regretted. I have a plan to do the presentation again for the DevOps Norway meetup group, though I haven’t set a date yet. When I do, I’ll see if I can at least provide a Periscope stream.

Tagged: conference, DevOps, Opera, Sysadmin

by bronto at April 01, 2017 03:07 PM

Feeding the Cloud

Manually expanding a RAID1 array on Ubuntu

Here are the notes I took while manually expanding an non-LVM encrypted RAID1 array on an Ubuntu machine.

My original setup consisted of a 1 TB drive along with a 2 TB drive, which meant that the RAID1 array was 1 TB in size and the second drive had 1 TB of unused capacity. This is how I replaced the old 1 TB drive with a new 3 TB drive and expanded the RAID1 array to 2 TB (leaving 1 TB unused on the new 3 TB drive).

Partition the new drive

In order to partition the new 3 TB drive, I started by creating a temporary partition on the old 2 TB drive (/dev/sdc) to use up all of the capacity on that drive:

$ parted /dev/sdc
unit s

Then I initialized the partition table and creating the EFI partition partition on the new drive (/dev/sdd):

$ parted /dev/sdd
unit s
mktable gpt

Since I want to have the RAID1 array be as large as the smaller of the two drives, I made sure that the second partition (/home) on the new 3 TB drive had:

  • the same start position as the second partition on the old drive
  • the end position of the third partition (the temporary one I just created) on the old drive

I created the partition and flagged it as a RAID one:

toggle 2 raid

and then deleted the temporary partition on the old 2 TB drive:

$ parted /dev/sdc
rm 3

Create a temporary RAID1 array on the new drive

With the new drive properly partitioned, I created a new RAID array for it:

mdadm /dev/md10 --create --level=1 --raid-devices=2 /dev/sdd1 missing

and added it to /etc/mdadm/mdadm.conf:

mdadm --detail --scan >> /etc/mdadm/mdadm.conf

which required manual editing of that file to remove duplicate entries.

Create the encrypted partition

With the new RAID device in place, I created the encrypted LUKS partition:

cryptsetup -h sha256 -c aes-xts-plain64 -s 512 luksFormat /dev/md10
cryptsetup luksOpen /dev/md10 chome2

I took the UUID for the temporary RAID partition:

blkid /dev/md10

and put it in /etc/crypttab as chome2.

Then, I formatted the new LUKS partition and mounted it:

mkfs.ext4 -m 0 /dev/mapper/chome2
mkdir /home2
mount /dev/mapper/chome2 /home2

Copy the data from the old drive

With the home paritions of both drives mounted, I copied the files over to the new drive:

eatmydata nice ionice -c3 rsync -axHAX --progress /home/* /home2/

making use of wrappers that preserve system reponsiveness during I/O-intensive operations.

Switch over to the new drive

After the copy, I switched over to the new drive in a step-by-step way:

  1. Changed the UUID of chome in /etc/crypttab.
  2. Changed the UUID and name of /dev/md1 in /etc/mdadm/mdadm.conf.
  3. Rebooted with both drives.
  4. Checked that the new drive was the one used in the encrypted /home mount using: df -h.

Add the old drive to the new RAID array

With all of this working, it was time to clear the mdadm superblock from the old drive:

mdadm --zero-superblock /dev/sdc1

and then change the second partition of the old drive to make it the same size as the one on the new drive:

$ parted /dev/sdc
rm 2
toggle 2 raid

before adding it to the new array:

mdadm /dev/md1 -a /dev/sdc1

Rename the new array

To change the name of the new RAID array back to what it was on the old drive, I first had to stop both the old and the new RAID arrays:

umount /home
cryptsetup luksClose chome
mdadm --stop /dev/md10
mdadm --stop /dev/md1

before running this command:

mdadm --assemble /dev/md1 --name=mymachinename:1 --update=name /dev/sdd2

and updating the name in /etc/mdadm/mdadm.conf.

The last step was to regenerate the initramfs:

update-initramfs -u

before rebooting into something that looks exactly like the original RAID1 array but with twice the size.

April 01, 2017 06:00 AM

The Lone Sysadmin

Install the vCenter Server Appliance (VCSA) Without Ephemeral Port Groups

Trying to install VMware vCenter in appliance/VCSA form straight to a new ESXi host? Having a problem where it isn’t listing any networks, and it’s telling you that “Non-ephemeral distributed virtual port groups are not supported” in the little informational bubble next to it? Thinking this is Chicken & Egg 101, because you can’t have an […]

The post Install the vCenter Server Appliance (VCSA) Without Ephemeral Port Groups appeared first on The Lone Sysadmin. Head over to the source to read the full post!

by Bob Plankers at April 01, 2017 04:07 AM

March 28, 2017

Sean's IT Blog

Configuring Duo Security MFA for Horizon Unified Access Gateway

Note: After publishing, I decided to rework this blog post a bit to separate the AD-integrated Duo configuration from the Duo-only configuration.  This should make the post easier to follow.

In my last post, I went through the steps for deploying a Horizon Access Point/Unified Access Gateway using the PowerShell deployment script.  That post walked through the basic deployment steps.

The Unified Access Gateway supports multiple options for two-factor authentication, and many real-world deployments will use some form of two-factor when granting users access to their desktops and applications remotely.  The Unified Access Gateway supports the following two-factor authentication technologies:

  • RSA Secur-ID
  • Certificate Based/Smart-Cards

Because I’m doing this in a lab environment, I decided to use a RADIUS-based technology for this post.  I’ve been using Duo Security for a while because they support RADIUS, have a mobile app, and have a free tier.  Duo also supports VMware Horizon, although they do not currently have any documentation on integrating with the Access Point/Unified Access Gateway.

Duo Security for Multi-factor Authentication

Duo Security is a cloud-based MFA provider.  Duo utilizes an on-premises Authentication Proxy to integrate with customer systems.  In addition to providing their own authentication source, they can also integrate into existing Active Directory environments or RADIUS servers.

When using Active Directory as the authentication source, Duo will utilize the same username and password as the user’s AD account.  It will validate these against Active Directory before prompting the user for their second authentication factor.

Understanding Unified Access Gateway Authentication Path

Before I configure the Unified Access Gateway for two-factor authentication with Duo, let’s walk through how the appliance handles authentication for Horizon environments and how it compares to the Security Server.  There are some key differences between how these two technologies work.

When the Security Server was the only option, two-factor authentication was enabled on the Connection Servers.  When users signed in remotely, the security server would proxy all authentication traffic back to the connection server that it was paired with.  The user would first be prompted for their username and their one-time password, and if that validated successfully, they would be prompted to log in with their Active Directory credentials.  The reliance on connection servers meant that two sets of connection servers needed to be maintained – one internal facing without multi-factor authentication configured and one external facing with multi-factor configured.

Note: When using Duo in this setup, I can configure Duo to use the same initial credentials as the user’s AD account and then present the user with options for how they want to validate their identity – a phone call, push to a mobile device, or passcode.  I will discuss that configuration below.

The Unified Access Gateway setup is, in some ways, similar to the old Security Server/Connection Server setup.  If 2FA is used, the user is prompted for the one-time password first, and if that is successful, the user is prompted for their Active Directory credentials.  But there are two key differences here.  First, the Unified Access Gateway is not tied to a specific Connection Server, and it can be pointed at a load-balanced pool of connection servers.  The other difference is 2FA is validated in the DMZ by the appliance, so 2FA does not need to be configured on the Connection Servers.

Horizon has a couple of multi-factor authentication options that we need to know about when doing the initial configuration.  These two settings are:

  • Enforce 2-factor and Windows user name matching (matchWindowsUserName in UAG) – enabled when the MFA and Windows user names match.
  • Use the same username and password for RADIUS and Windows authentication (WindowsSSOEnabled in UAG) – Used when the RADIUS server and Windows have the same password.  When this setting is enabled, the domain sign-on prompt is skipped.

If my two-factor authentication system supports Active Directory authentication, I can use my Windows Username and Password to be authenticated against it and then receive a challenge for a one-time password (or device push).  If this second factor of authentication is successful, I will be automatically signed into Horizon.

Configuring Duo

The Duo environment needs to be configured before Horizon MFA can be set up.  Duo requires an on-premises authentication proxy.  This proxy acts as a RADIUS server, and it can run on Windows or Linux.  The steps for installing the Duo authentication proxy are beyond the scope of this article.

Once the authentication proxy is installed, it needs to be configured.  Duo has a number of options for configuration, and it’s important to review the documentation.  I’ll highlight a few of these options below.

The first thing we need to configure is the Duo client.  The client determines what type of primary authentication is performed.  Duo supports three methods:

  • ad_client: Duo uses Active Directory for primary authentication.  Users sign in with their Active Directory username and password.
  • radius_client: Duo uses the specified RADIUS server, such as Microsoft NPS or Cisco ACS, for primary authentication.
  • duo_only_client: Duo does not perform primary authentication.  The authentication proxy only performs secondary authentication, and primary authentication is handled by the configured system.

I don’t have a RADIUS server configured in my lab, so I did testing with both the ad_client and the duo_only_client. I prefer a single sign-on solution in my lab, so this setup will be configured with the ad_client.

The authentication proxy configuration is contained in the authproxy.cfg file located in C:\Program Files (x86)\Duo Security Authentication Proxy\conf.  Open this file in a text editor and add the following lines:


Duo includes a number of options for configuring an Active Directory client, including encrypted passwords, LDAPS support, and other security features.  The configuration above is a simple configuration meant for a lab or PoC environment.

Before the RADIUS server settings can be configured, a new application needs to be created in the Duo Administration Console.  The steps for this are:

1. Log into the Duo Admin Console.

2. Click Applications

3. Click Protect an Application

4. Scroll down to VMware View and select “Protect this Application.”

5. Copy the Integration Key, Secret Key, and API Hostname.

6. Change the username normalization option to “Simple.”

7. Click “Save Changes.”

The next step is to configure the authentication proxy as a RADIUS service.  Duo also has a number of options for this.  These determine how Duo interacts with the client service.  These options include:

  • RADIUS_Auto: The user’s authentication factor, and the device that receives the factor, is automatically selected by Duo.
  • RADIUS_Challenge: The user receives a textual challenge after primary authentication is complete.   The user then selects the authentication factor and device that it is received on.
  • RADIUS_Duo_Only: The RADIUS service does not handle primary authentication, and the user’s passcode or factor choice is used as the RADIUS password.

Duo also has multiple types of authentication factors.  These options are:

  • Passcode: A time-based one-time password that is generated by the mobile app.
  • Push: A challenge is pushed to the user’s mobile device with the Duo mobile app installed.  The user approves the access request to continue sign-in.
  • Phone Call: Users can opt to receive a phone call with their one-time passcode.
  • SMS: Users can opt to receive a text message with their one-time passcode.

RADIUS_Challenge will be used for this setup.  In my testing, the combination of an ad_client and RADIUS_Auto meant that my second authentication factor was a push to the mobile app.  In most cases, this was fine, but in situations where my laptop was online but my phone was not (such as on an airplane), meant that I was unable to access the environment.  The other option is to use RADIUS_Duo_Only with the Duo_Only_Client, but I wanted a single sign-on experience.

The RADIUS server configuration for the Horizon Unified Access Gateway is:

ikey=[random key generated by the Duo Admin Console]
skey=[random key generated by the Duo Admin Console] [generated by the Duo Admin Console]
radius_ip_1=IP Address or Range

The above configuration is set to fail secure.  This means that if the Duo service is not available, users will be unable to log in.  The other option that was selected was a short prompt format.  This displays a simple text message with the options that are available and prompts the user to select one.

Save the authproxy.cfg file and restart the Duo Authentication Proxy service for the new settings to take effect.

The next step is to configure the Unified Access Gateway to use RADIUS.  The following lines need to be added to the [Horizon] section:




A new section needs to be added to handle the RADIUS configuration.  The following lines need to be added to create the RADIUS section and configure the RADIUS client on the Unified Access Gateway.  If more than one RADIUS server exists in the environment, you can add an _2 to the end of hostname and auth_port.

[RADIUSAuth] [IP of the primary RADIUS server]



radiusDisplayHint=XXX Token

Save the UAG configuration file and deploy, or redeploy, your Unified Access Gateway.  When the deployment completes, and you log in externally, you should see the following screens:



Duo-Only Authentication

So what if I don’t want an AD-integrated single sign-on experience?  What if I just want users to enter a code before signing in with their AD credentials.  Duo supports that configuration as well.  There are a couple of differences compared to the configuration for the AD-enabled Duo environment.

The first change is in the Duo authentication proxy config file.  No AD client configuration is required.  Instead of an [ad_client] section, a single line entry of [duo_only_client] is required.  The RADIUS Server configuration can also mostly stay the same.  The only required changes are to change the client from ad_client to duo_only_client and the section name from [radius_server_challenge] to [radius_server_duo_only].

The Access Point configuration will change.  The following configuration items should be used instead:

authMethods=radius-auth && sp-auth




by seanpmassey at March 28, 2017 01:30 PM

LZone - Sysadmin

RabbitMQ Does Not Start: init terminating in do_boot

If you have a RabbitMQ cluster and a crashed node fails to start again with
{"init terminating in do_boot",{undef,[{rabbit_prelaunch,start,[]},{init,start_it,1},{init,start_em,1}]}}
in /var/log/rabbitmq/startup_log and something like
Error description:

Log files (may contain more information): /var/log/rabbitmq/rabbit@rabbit-01.log /var/log/rabbitmq/rabbit@rabbit-01-sasl.log
in /var/log/rabbitmq/rabbit@rabbit-01.log then you might want to try to drop the node from the cluster by running
rabbitmqctl forget_cluster_node rabbit@rabbit-01
one a working cluster node and rejoin the node by running
rabbitmqctl join_cluster rabbit@rabbit-02
on the disconnected node (given rabbit-02 is a working cluster member).

Note: Doing this might make you lose messages!

March 28, 2017 10:46 AM

March 27, 2017

Sean's IT Blog

Horizon 7.0 Part 13 – Deploying The Horizon Access Point

In the last part of this series, I walked through the different remote access options for a Horizon 7 environment.  In this post, I’ll cover how to install and configure an Access Point appliance for a Horizon environment.

Note: The Access Point appliance has been renamed to the Unified Access Gateway as of Horizon 7.1.  This post began before the product was renamed, and the old naming convention will be used.

Before we go into deploying the appliance, let’s dive into what the appliance does and how it’s built.

As I said in the previous post, the Access Point is a purpose built virtual appliance that is designed to be the remote access component for VMware Horizon, VMware Identity Manager, and Airwatch.  The appliance is hardened for deployment in a DMZ scenario, and it is designed to only pass authorized traffic from authenticated users into a secure network.  In some ways, the Access Point is designed to replace VPNs, but it doesn’t provide full access to an internal network like a VPN would.

When deploying an Access Point, I highly recommend using the PowerShell Deployment Script.  This script was written by Mark Benson, the lead developer of the Access Point.  The script uses an INI configuration file that can be customized for each appliance that is deployed.  I like the PowerShell script over deploying the appliance through vCenter because the appliance is ready to use on first boot, it allows administrators to track all configurations in a source control system such as Github or Bitbucket Server, and this provides both documentation for the configuration and change tracking.  It also makes it easy to redeploy or upgrade the access point because I rerun the script with my config file and the new OVA file.

The PowerShell script requires the OVF Tool to be installed on the server or desktop where the PowerShell script will be executed.  The latest version of the OVF Tool can be downloaded from the My VMware site.  PowerCLI is not required when deploying the Access Point as OVF Tool will be deploying the Access Point and injecting the configuration.

The steps for deploying the Access Point are:

1. Download the PowerShell deployment script for the version of the Access Point you will be deploying.  You can download the script from here.

2.  Right click on the downloaded zip file and select Properties.

3. Click Unblock.  This step is required because the file was downloaded from the Internet, and is untrusted by default, and this can prevent the script from executing.

4. Extract the contents of the downloaded ZIP file to a folder on the system where the deployment script will be run.  The ZIP file contains the apdeploy.ps1 script file and five INI template files.  As of January 2017, four of the template files are example configurations for Horizon, and one is a sample configuration for vIDM. 

When deploying the access points for Horizon, I recommend starting with the AP2-Advanced.ini template.  This template provides the most options for configuring Horizon remote access and networking.  Once you have the AP deployed successfully, I recommend copying the relevant portions of the SecurID or RADIUS auth templates into your working AP template.  This allows you to test remote access and your DMZ networking and routing before adding in MFA.

5. Before we start filling out the template for our first access point, there are some things we’ll need to do to ensure a successful deployment. These steps are:

A. Ensure that the OVF Tool is installed on your deployment machine.

B. Locate the Access Point’s OVA file and record the full file path.  The OVA file can be placed on a network share.

C. We will need a copy of the certificate, including any intermediate and root CA certificates, and the private key in PEM format.  The certificate files should be concatenated so that the certificate and any CA certificates in the chain are in one file, and the private key should not have a password on it.  Place these files into a folder on the local or network folder and record the full path.

D. We need to create the path to the vSphere resources that OVF Tool will use when deploying the appliance.  This path looks like: vi://user@PASSWORD:vcenter.fqdn.orIP/DataCenter Name/host/Host or Cluster Name/

Do not replace the uppercase PASSWORD with any value.  This is an OVF Tool variable that prompts the user for a password before deploying the appliance.  OVF Tool is case sensitive, so make sure that the datacenter name and host or cluster names are entered as they are displayed in vCenter.

E. Generate the passwords that  you will use for the appliance Root and Admin passwords.

F. Get the SSL Thumbprint for the certificate on your Connection Server or load balancer that is in front of the connection servers.

6. Fill out the template file.  The file has comments for documentation, so it should be pretty easy to fill out.  There are a couple of things that I’ve noticed when deploying the access point using this method.  You need to have a valid port group for all three networks, even if you are only using the OneNic deployment option. 

7. Save your INI file as <APName>.ini in the same directory as the deployment scripts.

8. Open PowerShell and change to the directory where the deployment scripts are stored.

9. Run the deployment script.  The syntax is .\APDeploy.ps1 –inifile <apname>.ini

10. Enter the appliance root password twice.

11.  Enter the admin password twice.  This password is optional, however, if one is not configured, the REST API and Admin interface will not be available.

12.  If RADIUS is configured in the INI file, you will be prompted for the RADIUS shared secret.

13. After the script opens the OVA and validates the manifest, it will prompt you for the password for accessing vCenter.  Enter it here.

14. If an access point with the same name is already deployed, it will be powered off and deleted.

15. The appliance OVA will be deployed.  When the deployment is complete, the appliance will be powered on and get an IP address from DHCP.

16. The appliance configuration defined in the INI file will be injected into the appliance.  It may take a few minutes for configuration to be completed.


Testing the Access Point

Once the appliance has finished it’s deployment and self-configuration, it needs to be tested to ensure that it is operating properly. The best way that I’ve found for doing this is to use a mobile device, such as a smartphone or cellular-enabled tablet, to access the environment using the Horizon mobile app.  If everything is working properly, you should be prompted to sign in, and desktop pool connections should be successful.

If you are not able to sign in, or you can sign in but not connect to a desktop pool, the first thing to check is your firewall rules.  Validate that TCP and UDP ports 443 and 4172 are open between the Internet and your Access Point.  You may also want to check your network routes in your configuration file.  If your desktops live in a different subnet than your access points and/or your connection servers, you may need to statically define your routes.  An example of a route configuration may look like the following:

routes1 =,

If you need to make a routing change, the best way to handle it is to update the ini file and then redeploy the appliance.

by seanpmassey at March 27, 2017 01:30 PM

Toolsmith #124: Dripcap - Caffeinated Packet Analyzer

Dripcap is a modern, graphical packet analyzer based on Electron.
Electron, you say? "Electron is a framework for creating native applications with web technologies like JavaScript, HTML, and CSS. It takes care of the hard parts so you can focus on the core of your application."
We should all be deeply familiar with the venerable Wireshark, as it has long been the forerunner for packet analysts seeking a graphical interface to their PCAPs. Occasionally though, it's interesting to explore alternatives. I've long loved NetworkMiner, and the likes of Microsoft Message Analyzer and Xplico each have unique benefits.
For basic users comfortabel with Wireshark, you'll likely find Dripcap somewhat rudimentary at this stage, but it does give you opportunities to explore packet captures at fundamental levels and learn without some of the feature crutches more robust tools offer.
However, for JavaScript developers,  Dripcap opens up a whole other world of possibilities. Give the Create NTP dissector package tutorial a read, you can create, then publish and load dissector (and others) packages of your choosing.

I built Dripcap from source on Windows as follows, using Chocolatey.
From a administrator PowerShell prompt (ensure Get-ExecutionPolicy is not Restricted), execute the following (restart your admin PS prompt after #2):
  1. iwr -UseBasicParsing | iex
  2. choco install git make jq nodejs
  3. git clone
  4. cd dripcap
  5. npm install -g gulp node-gyp babel-cli
  6. npm install
  7. gulp
Step 1 installs Chocolatey, step 2 uses Chocolatey to install tools, step 3 clones Dripcap, steps 5 & 6 install packages, and step 7 builds it all.    
Execute dripcap, and you should be up and running.
You can also  use npm, part of Node.js' package ecosystem to install Dripcap CLI with npm install -g dripcap, or just download dripcap-windows-amd64.exe from Dripcap Releases.


I'll walk you through packet carving of sorts with Dripcap. One of Dripcap's strongest features is its filtering capabilities. I used an old PCAP with an Operation Aurora Internet Explorer exploit (CVE-2010-0249) payload for this tool test.
Ctrl+O will Import Pcap File for you.

Click Developer, then Toggle Log Panel for full logging.

Figure 1: Dripcap
You'll note four packets with lengths of 1514, as seen in Figure 1. Exploring the first of these packets indicates just what we'd expect: an Ethernet MTU (maximum transmission unit) of 1500 bytes, and a TCP payload of 1460 bytes, leaving 40 bytes for our header (20 byte IP and 20 byte TCP).

Figure 2: First large packet
 Hovering your mouse over the TCP details in the UI will highlight all the TCP specific data, but you can take such actions a step further. First, let's filter down to just the large packets with tcp.payload.length == 1460.
Figure 3: Filtered packets
 With our view reduced we can do some down and dirty carving pretty easily with Dripcap. In each of the four filtered packets I hovered over Payload 1460 bytes as seen in Figure 4, which highlighted the payload-specific hex. I then used the mouse to capture the highlighted content and, using Dripcap's Edit and Copy, grabbed only that payload-specific hex and pasted it to a text file.
Figure 4: Hex payload
I did this with each of these four packets and copied content, one hex blob after the other, into my text file, in tight, seamless sequence. I then used Python Tools for Visual Studio to do a quick hexadecimal to ASCII translation as easily as bytearray.fromhex("my hex snippet here").decode(). The result, in Figure 5, shows the resulting JavaScript payload the exploits CVE-2010-0249.
Figure 5: ASCII results
You can just as easily use online converters as well. I saved the ASCII results to a text file in a directory which I had excluded from my anti-malware protection. After uploading the file to VirusTotal as payload.txt, my expectations were confirmed: 32 of 56 AV providers detected the file as the likes of Exploit:JS/Elecom.D or, more to the point, Exploit.JS.Aurora.a.

In closing
Perhaps not the most elegant method, but it worked quickly and easily with Dripcap's filtering and editing functions. I hope to see this tool, and its community, continue to grow. Build dissector packages, create themes, become part of the process, it's always good to see alternatives in available to security practitioners.
Cheers...until next time.

by Russ McRee ( at March 27, 2017 04:49 AM

March 23, 2017

Carl Chenet

Feed2tweet 1.0, tool to post RSS feeds to Twitter, released

Feed2tweet 1.0, a self-hosted Python app to automatically post RSS feeds to the Twitter social network, was released March 2017, 23th.

The main new feature of this release allows to create filters for each RSS feed, because before you could only define global filters. Contributed by Antoine Beaupré, Feed2tweet is also able to use syslog, starting from this release.

fiestaWhat’s the purpose of Feed2tweet?

Some online services offer to convert your RSS entries into Twitter posts. Theses services are usually not reliable, slow and don’t respect your privacy. Feed2tweet is Python self-hosted app, the source code is easy to read and you can enjoy the official documentation online with lots of examples.

Twitter Out Of The Browser

Have a look at my Github account for my other Twitter automation tools:

  • Retweet , retweet all (or using some filters) tweets from a Twitter account to another one to spread content.
  • db2twitter, get data from SQL database (several supported), build tweets and send them to Twitter
  • Twitterwatch, monitor the activity of your Twitter timeline and warn you if no new tweet appears

What about you? Do you use tools to automate the management of your Twitter account? Feel free to give me feedback in the comments below.

… and finally

You can help Feed2tweet by donating anything through Liberaypay (also possible with cryptocurrencies). That’s a big factor motivation 😉


by Carl Chenet at March 23, 2017 11:00 PM


Five Reasons I Want China Running Its Own Software

Periodically I read about efforts by China, or Russia, or North Korea, or other countries to replace American software with indigenous or semi-indigenous alternatives. I then reply via Twitter that I love the idea, with a short reason why. This post will list the top five reasons why I want China and other likely targets of American foreign intelligence collection to run their own software.

1. Many (most?) non-US software companies write lousy code. The US is by no means perfect, but our developers and processes generally appear to be superior to foreign indigenous efforts. Cisco vs Huawei is a good example. Cisco has plenty of problems, but it has processes in place to manage them, plus secure code development practices. Lousy indigenous code means it is easier for American intelligence agencies to penetrate foreign targets. (An example of a foreign country that excels in writing code is Israel, but thankfully it is not the same sort of priority target like China, Russia, or North Korea.)

2. Many (most?) non-US enterprises are 5-10 years behind US security practices. Even if a foreign target runs decent native code, the IT processes maintaining that code are lagging compared to American counterparts. Again, the US has not solved this problem by any stretch of the imagination. However, relatively speaking, American inventory management, patch management, and security operations have the edge over foreign intelligence targets. Because non-US enterprises running indigenous code will not necessarily be able to benefit from American expertise (as they might if they were running American code), these deficiencies will make them easier targets for foreign exploitation.

3. Foreign targets running foreign code is win-win for American intel and enterprises. The current vulnerability equities process (VEP) puts American intelligence agencies in a quandary. The IC develops a zero-day exploit for a vulnerability, say for use against Cisco routers. American and Chinese organizations use Cisco routers. Should the IC sit on the vulnerability in order to maintain access to foreign targets, or should it release the vulnerability to Cisco to enable patching and thereby protect American and foreign systems?

This dilemma disappears in a world where foreign targets run indigenous software. If the IC identifies a vulnerability in Cisco software, and the majority of its targets run non-Cisco software, then the IC is more likely (or should be pushed to be more likely) to assist with patching the vulnerable software. Meanwhile, the IC continues to exploit Huawei or other products at its leisure.

4. Writing and running indigenous code is the fastest way to improve. When foreign countries essentially outsource their IT to vendors, they become program managers. They lose or never develop any ability to write and run quality software. Writing and running your own code will enroll foreign organizations in the security school of hard knocks. American intel will have a field day for 3-5 years against these targets, as they flail around in a perpetual state of compromise. However, if they devote the proper native resources and attention, they will learn from their mistakes. They will write and run better software. Now, this means they will become harder targets for American intel, but American intel will retain the advantage of point 3.

5. Trustworthy indigenous code will promote international stability. Countries like China feel especially vulnerable to American exploitation. They have every reason to be scared. They run code written by other organizations. They don't patch it or manage it well. Their security operations stink. The American intel community could initiate a complete moratorium on hacking China, and the Chinese would still be ravaged by other countries or criminal hackers, all the while likely blaming American intel. They would not be able to assess the situation. This makes for a very unstable situation.

Therefore, countries like China and others are going down the indigenous software path. They understand that software, not oil as Daniel Yergen once wrote, is now the "commanding heights" of the economy. Pursuing this course will subject these countries to many years of pain. However, in the end I believe it will yield a more stable situation. These countries should begin to perceive that they are less vulnerable. They will experience their own vulnerability equity process. They will be more aware and less paranoid.

In this respect, indigenous software is a win for global politics. The losers, of course, are global software companies. Foreign countries will continue to make short-term deals to suck intellectual property and expertise from American software companies, before discarding them on the side of Al Gore's information highway.

One final point -- a way foreign companies could jump-start their indigenous efforts would be to leverage open source software. I doubt they would necessarily honor licenses which require sharing improvements with the open source community. However, open source would give foreign organizations the visibility they need and access to expertise that they lack. Microsoft's shared source and similar programs were a step in this direction, but I suggest foreign organizations adopt open source instead.

Now, widespread open source adoption by foreign intelligence targets would erode the advantages for American intel that I explained in point 3. I'm betting that foreign leaders are likely similar to Americans in that they tend to not trust open source, and prefer to roll their own and hold vendors accountable. Therefore I'm not that worried, from an American intel perspective, about point 3 being vastly eroded by widespread foreign open source adoption.

TeePublic is running a sale until midnight ET Thursday! Get a TaoSecurity Milnet T-shirt for yourself and a friend!

by Richard Bejtlich ( at March 23, 2017 02:26 PM

The Missing Trends in M-Trends 2017

FireEye released the 2017 edition of the Mandiant M-Trends report yesterday. I've been a fan of this report since the 2010 edition, before I worked at the company.

Curiously for a report with the name "trends" in the title, this and all other editions do not publish the sorts of yearly trends I would expect. This post will address that limitation.

The report is most famous for its "dwell time" metric, which is the median (not average, or "mean") number of days an intruder spends inside a target company until he is discovered.

Each report lists the statistic for the year in consideration, and compares it to the previous year. For example, the 2017 report, covering incidents from 2016, notes the dwell time has dropped from 146 days in 2015, to 99 days in 2016.

The second most interesting metric (for me) is the split between internal and external notification. Internal notification means that the target organization found the intrusion on its own. External notification means that someone else informed the target organization. The external party is often a law enforcement or intelligence agency, or a managed security services provider. The 2016 split was 53% internal vs 47% external.

How do these numbers look over the years that the M-Trends report has been published? Inquiring minds want to know.

The 2012 M-Trends report was the first edition to include these statistics. I have included them for that report and all subsequent editions in the table below.

Year Days Internal External
2011 416 6 94
2012 243 37 63
2013 229 33 67
2014 205 31 69
2015 146 47 53
2016 99 53 47

As you can see, all of the numbers are heading in the right direction. We are finally into double digits for dwell time, but over 3 months is still far too long. Internal detection continues to rise as well. This is a proxy for the maturity of a security organization, in my opinion.

Hopefully future M-Trends reports will include tables like this.

by Richard Bejtlich ( at March 23, 2017 12:43 PM

March 22, 2017


Licensing Update

The following is a press release that we just released, with the cooperation and financial support of the Core Infrastructure Initiative and the Linux Foundation.

In the next few days we’ll start sending out email to all contributors asking them to approve the change. In the meantime, you can visit the licensing website and search for your name and request the email. If you have changed email addresses, or want to raise other issues about the license change, please email You can also post general issues to

We are grateful to all the contributors who have contributed to OpenSSL and look forward to their help and support in this effort.

The official press release can be found at the CII website. The rest of this post is a copy:

OpenSSL Re-licensing to Apache License v. 2.0 To Encourage Broader Use with Other FOSS Projects and Products

OpenSSL Launches New Website to Organize Process, Seeks to Contact All Contributors

SAN FRANCISCO, March 16, 2017 – The OpenSSL project, home of the world’s most popular SSL/TLS and cryptographic toolkit, is changing its license to the Apache License v 2.0 (ASL v2). As part of this effort, the OpenSSL team launched a new website and has been working with various corporate collaborators to facilitate the re-licensing process.

“This re-licensing activity will make OpenSSL, already the world’s most widely-used FOSS encryption software, more convenient to incorporate in the widest possible range of free and open source software,” said Mishi Choudhary, Legal Director of Software Freedom Law Center (SFLC) and counsel to OpenSSL. “OpenSSL’s team has carefully prepared for this re-licensing, and their process will be an outstanding example of ‘how to do it right.’ SFLC is pleased to have been able to help the team bring this process to this point, and looks forward to its successful and timely completion.”

The website will aid the OpenSSL team’s efforts to contact everyone who has contributed to the project so far, which includes nearly 400 individuals with a total of more than 31,000 commits. The current license dates back to the 1990’s and is more than 20 years old. The open source community has grown and changed since then, and has mostly settled on a small number of standard licenses.

After careful review, consultation with other projects, and input from the Core Infrastructure Initiative and legal counsel from the SFLC, the OpenSSL team decided to relicense the code under the widely-used ASLv2.

“The Linux Foundation is excited to see the OpenSSL project re-licensing under the Apache License,” said Nicko van Someren, Chief Technology Officer, the Linux Foundation. “Using a standard and well-understood license is a huge benefit when incorporating a FOSS project into other projects and products. OpenSSL has made huge progress in recent years, in part through support from the Linux Foundation’s Core Infrastructure Initiative, and this license move will further help to ensure it remains one of the most important and relied-upon open source projects in the world.”

The website contains a list of every email address mentioned in every single commit, a searchable database of authors, and the ability to send email and approve the license change. Because email addresses change, the website will also be updated over time to record email bounces and the names of people the project is still trying to reach.

“Oracle is proud to extend its collaboration with the OpenSSL Foundation by relicensing its contributions of elliptic curve cryptography,” said Jim Wright, Chief Architect of Open Source Policy, Strategy, Compliance and Alliances, Oracle. “OpenSSL is a critical component in both Oracle products and the infrastructure of the Internet, and we strongly believe the increased use of cryptography fostered by OpenSSL will benefit the entire enterprise software community.”

“Intel is thrilled to see OpenSSL moving to the standard Apache 2.0 license, improving license compatibility within the Open Source ecosystem,” said Imad Sousou, Vice President and General Manager of the Open Source Technology Center, Intel. “This will help defragment the open source cryptography ecosystem, leading to stronger and more pervasive use of crypto to improve privacy and security in the global technology infrastructure.”

Additional details on the decision to move to ASL v. 2.0 are available here. For progress updates on re-licensing, which is expected to take several months, check the website and project mailing lists.

To reach the OpenSSL team involved in this effort, email The team also asks that anyone who knows of other people that should be contacted, such as “silent collaborators” on code contributions, to also send email.

March 22, 2017 12:00 PM