Planet SysAdmin

December 08, 2016

Chris Siebenmann

Trailing text, a subtle gotcha with Go's fmt.Sscanf

I've written some Go code that wanted to do some simple scanf-like parsing of strings. When I did this, I peered at the fmt package documentation for the Sscanf function and confidently wrote something like the following code:

_, e := fmt.Sscanf(input, "%d:%d", &hr, &min)
if e != nil {
   return ..., e

This code has a bug, or perhaps some people would call it an unintended feature (for me it's definitely a bug). Namely, if you feed this code the input string '10:15 this is trailing text', you will not get an error message. Your code will parse the 10:15 part of the input string and silently ignore the rest, or more exactly Sscanf() will.

At this point you might wonder how to either force Sscanf to produce an error on trailing text or detect that you have trailing text. As far as I can tell there is no straightforward way, but there are two options depending on how paranoid you want to be (and where you get your input string from). The simple option is to add an explicit newline to your format string:

_, e := fmt.Sscanf(input, "%d:%d\n", &hr, &min)

This will parse an input string of '10:15' (with no trailing newline) without raising an error, and will detect most cases of trailing input by raising an error. It won't detect the relatively perverse case of something such as '10:15\n and more', because the '\n' in the input matches the expected newline and then Sscanf stops looking.

(At the moment you can stack more than one \n on the end of your format string and still parse a plain '10:15', so you can add some more caution and/or paranoia if you want. Sufficiently perverse input can always get past you, though, because as far as I can see there is no way to tell Sscanf that what you really mean is an EOF.)

The complicated hack is to add an extra string match to your format string and look at how many items were successfully parsed:

n, _ := fmt.Sscanf(input, "%d:%d%s", &hr, &min, &junk)
if n != 2 {
   return ..., error("Bad input")

Among other drawbacks, we have to ignore the error that Sscanf returns; it doesn't tell us whether or not the input was good, and when it has an error value it may be meaningless for our caller.

My suspicion is that in cases like this I am probably pushing Sscanf too far and it's actually the wrong tool for the job. In most cases the right answer is probably matching things with regular expressions so that I can directly say what I mean. Or, in this case, just using time.ParseInLocation even though it's less convenient and I'd have to do a bunch of manipulation on the result.

(Regular expressions are probably slower than Sscanf and I'd have to use strconv to turn the results into numbers, but my code here is not exactly performance critical.)

by cks at December 08, 2016 05:38 AM

System Administration Advent Calendar

Day 8 - Building Robust Jenkins Pipelines

Written By: Michael Heap (@mheap)
Edited By: Daniel Maher (@phrawzty)

For many years continuous integration was synonymous with the name Jenkins, but as time went on it fell out of favour and was replaced by newer kids on the block such as Travis, GoCD and Concourse. Users became frustrated with Jenkins’ way of defining build jobs and looked to services that allowed you to define your builds as code alongside your project, building real pipelines with fan-in/fan-out capabilities.

Jenkins recently hit version 2, and with it came a whole host of new features! There were some small additions to the base install (such as the git plugin being shipped by default), but true to its roots, Jenkins still ships most of it’s functionality as plugins. Functionality such as defining your job configuration in code is provided by plugins that are maintained by the Jenkins core team, which means they’re held to the same high standards as Jenkins itself.

In this article we’ll be building an Electron application and adding it to Jenkins. Whilst we’ll be using the Github organization plugin, it’s not essential - you can configure repositories by hand if you prefer.

The majority of the code in this article is simple calls to shell commands, but a passing familiarity with either Java or Groovy may help later on.

Getting Started

In this article, we’re going to be building a pipeline that builds, tests, and packages an Electron application for deployment on Linux. We’ll start with a basic Jenkins installation and add all of the plugins required, before writing a Jenkinsfile that builds our project.

Let’s create a Vagrant virtual machine to run Jenkins inside. This isn’t a requirement to run Jenkins, but helps us get up and running slightly easier.

Creating a VM

vagrant init ubuntu/xenial64
sed -i &apos/#.*private_network*/ s/^  #/ /&apos Vagrantfile # Enable private networking
vagrant up
vagrant ssh
wget -q -O - | sudo apt-key add -
sudo sh -c &aposecho deb binary/ > /etc/apt/sources.list.d/jenkins.list&apos
sudo apt-get update
sudo apt-get install -y jenkins
sudo cat /var/lib/jenkins/secrets/initialAdminPassword

Visit on your local machine and continue the setup.

Bootstrapping Jenkins

You’ll need two plugins to get started: the Pipeline plugin and the GitHub Organization Folder plugin. These form the cornerstone of your Jenkins install. Click on None at the top then type pipeline into the search box. Select Pipeline and GitHub Organization Folder before clicking Install.

You’ll be prompted to create a user once the plugins are installed. Do this now, then click on Start using Jenkins to get underway.

Configuring Jenkins

Now that you’re logged in, there’s a little bit of work to do to configure Jenkins so that we can start using it. We’ll need to provide access credentials and set up our GitHub Organization job so that Jenkins knows what to build. Sadly, we need to do this even if we’re only working with public repositories as certain endpoints on the GitHub API require authentication.

  • Click Credentials on the left hand side.

  • Click on (global) next to the Jenkins store.

  • Click Add Credentials on the left.

  • Decide what kind of credentials you want to add. I’m using username & password authentication for HTTPS clones from GitHub, so I visited and generated a token that had only the repo OAuth scope.

  • Provide the credentials by filling in the username and password fields. (Note that the fields are different if you’re using SSH authentication.)

  • Click Jenkins in the top left to go back to the homepage.

  • Click New Item on the left.

  • Choose GitHub Organization and give it a name (I’ve called mine michael-test), then click OK.

  • Under Repository sources, set the Owner field to be either your organisation name or your personal account name.

  • Select the credentials you just set up using the Scan Credentials option.

  • Click Advanced under Repository name pattern and make sure that Build origin PRs merged with base branch is enabled. This will make Jenkins build any incoming pull requests.

It’s worth noting at this point that under the Project Recognizers section it says Pipeline Jenkinsfile. Jenkins will scan an organisation or account for any repos with branches that contain a “Jenkinsfile” at their root and create a build job for that branch.

Let’s create your first Jenkinsfile and kick off a build.

Writing your first Jenkinsfile

We need a project to build, so let’s create one. For now it’ll be an empty project that only contains a Jenkinsfile.

mkdir demo-electron
cd demo-electron
git init
touch Jenkinsfile

Edit Jenkinsfile and add the following content:

echo "Hello World"

Commit and push this up to GitHub (you’ll need to create a repository first). I’ve called my repo demo-electron.

git add .
git commit -m "Initial Jenkinsfile"
git remote add origin
git push -u origin master

Go back to Jenkins, and you’ll see an option on the left called Re-scan organization. Click it, then click on the Run now option that appears. You’ll see some log output where it detects the Jenkinsfile and creates a job for you.

Within michael-test there’s now a job called demo-electron, which matches our repository name. Clicking through, we can now see a master job within. By default, Jenkins will build all existing branches. Click into the master job and then click on #1 on the left hand side to see the results from the first build that was performed when the repository was detected. If you click on Console Output on the left, you’ll see Jenkins cloning our repo before outputting “Hello World”.

Congratulations, you just wrote your first Jenkinsfile! It doesn’t do much, but it proves that our Jenkins instance is up and running, detecting pipelines, and automatically executing them.

Throughout the rest of this post, we’ll be building a small Electron based desktop application. This means that our build machine will need some additional dependencies installed. If you want to work along with this tutorial, your Vagrant machine will need a little bit of configuration. Log in to your machine with vagrant ssh and run the following command to install the necessary dependencies to build an Electron application:

sudo apt-get install -y nodejs nodejs-legacy npm jq

Adding Electron

We’re going to add Electron to the project at this point. Instead of working directly on master, we’re going to work on a new branch called add-electron.

Create a new branch to work on add-electron by running the following command:

git checkout -b add-electron

The first thing we need to do is create a standalone Electron application. To do this, we follow the Electron Quick Start document. Go ahead and follow that - once you’re done you should have a package.json, main.js and index.html. We also need to install Electron as a dependency, so run the following command on your local machine in the same directory as package.json.

npm install electron --save-dev

This will make Electron available to run our app. You can test it by running ./node_modules/.bin/electron . in the same folder as your main.js file. This will show a “Hello World” page.

This gives us our first real action to run in Jenkins! Although running npm install doesn’t feel like a lot, ensuring that we can install our dependencies is a great first step towards continuous builds.


It’s time to change our Jenkinsfile! It’ll be a nice simple one to start with: first make sure that we’re on the right branch, then run npm install.

The first thing to do is delete our “Hello World” line and add the following contents:

node {

This declaration tells Jenkins that anything inside the braces must run on a build agent. Our old echo statement could run without a workspace (a copy of our repository on disk) but as we’re about to interact with the repository we need a workspace.

The next thing to do is to tell Jenkins to make sure that our repository is in the correct state based on the job that we’re running. To do this, we run a special checkout scm command that makes sure we’re on the correct branch. Finally, we run npm install using the sh helper to run a shell command. Our entire Jenkinsfile looks like this after the changes:

node {
  checkout scm
  sh &aposnpm install&apos

If we commit and push this, Jenkins will build the project using our new Jenkinsfile and run npm install each time we commit.

git add Jenkinsfile index.html main.js package.json
git commit -m "Create Electron app"
git push origin add-electron

While it is possible for Jenkins to be notified from GitHub each time there is a commit to build, this won’t work for us as we’re testing with a local Jenkins instance. Instead, we need to tell Jenkins to search for changes by visiting the main page for our job and clicking on Branch Indexing then Run Now. Jenkins will then rescan the repo for new branches, add our add-electron branch, then run the new Jenkinsfile.

Just like we could view the build for the master branch, we can click in to add-electron, click on Console Output on the left and watch as Jenkins runs npm install for us.

At this point, we can raise a pull request to merge our branch into master on GitHub. Once Jenkins rescans the repository (manually triggered, see above), it will detect this pull request and automatically update the pull request’s commit status with the outcome of the build.

Once the request is merged and our add-electron branch is deleted, Jenkins will go back to just building the master branch on its own.

Creating a build

We now have an application that’s building, but we need a way to package it up and ship it to customers. Let’s create a new working branch called build-app by running the following command:

git checkout -b build-app

There’s a project called electron-builder which can help us build our application. Just like with Electron, we need to require it in our package.json by running the following on your local machine in the same directory as package.json:

npm install electron-builder --save-dev

Next, we need to add some details to our package.json for the builder to use, including configuration values such as our application’s description and application identifier. We’ll also add a few npm scripts that build the application for different operating systems.

Update your package.json to like the following, which contains all of the required information and build scripts:

  "name": "sysadvent",
  "version": "0.1.0",
  "main": "main.js",
  "devDependencies": {
    "electron": "^1.4.10",
    "electron-builder": "^10.4.1"
  "description": "SysAdvent",
  "author": "Michael Heap <>",
  "license": "MIT",
  "build": {
    "appId": "com.michaelheap.sysadvent",
    "category": "app.category.type"
  "scripts": {
      "start": "electron main.js",
      "build": "build --mwl --x64 --ia32",
      "build-windows": "build --windows --x64 --ia32",
      "build-mac": "build --mac --x64 --ia32",
      "build-linux": "build --linux --x64 --ia32"

At this point, you can test the build on your local machine by running npm run build-<os>; in my case I run npm run build-linux and the build applications show up in the dist folder. I can run ./dist/sysadvent-0.1.0-x86_64.AppImage to run the application. If you’re on OSX, there should be a .app file in the dist folder. If you’re on Windows there should be an installer .exe

This is another huge step in our build process. We now have a tangible build that we can distribute to people! Let’s update our build process so that it builds the application. Edit your Jenkinsfile and add a call to npm run build-linux, as we’re doing to be building on a Linux machine.

node {
    checkout scm
    sh &aposnpm install&apos
    sh &aposnpm run build-linux&apos

This is almost enough to have our build automated, but we somehow have to get the build artefacts out of our workspace and make them available for people to download. Jenkins has built in support for artefacts, which means there’s a built in archiveArtifacts step that we can use. We tell Jenkins to step in to the dist folder with the dir command and tell it to archive all artefacts that end with .AppImage in that directory. Your final Jenkinsfile will look like the following:

node {
    checkout scm
    sh &aposnpm install&apos
    sh &aposnpm run build-linux&apos
    dir(&aposdist&apos) {
      archiveArtifacts artifacts: &apos*.AppImage&apos, fingerprint: true;

If we commit and push this up to GitHub and retrigger the branch indexing, Jenkins will pick up the new branch, build our application, and publish an artefact.

git add Jenkinsfile package.json
git commit -m "Build packaged application"
git push origin build-app

Once the branch has built and is green, we can raise a pull request and get our changes merged into master. At this point, we have an application that has a build process and creates a packaged application that we can send out to users.

A more complex Jenkinsfile

What we’ve put together so far is a good introduction to building applications with Jenkins, but it’s not really representative of a real world project that would contain tests, code style linting, and various other build steps.

On the full-application branch, you’ll find a build of the Electron application that has basic functionality added, tests for that functionality, and code style linting added. With the addition of these common steps, the Jenkinsfile is already starting to grow.

node {
    checkout scm
    sh &aposnpm install&apos
    sh &aposnpm test&apos
    sh &aposnpm run lint&apos
    sh &aposnpm run build-linux&apos
    dir(&aposdist&apos) {
      archiveArtifacts artifacts: &apos*.AppImage&apos, fingerprint: true;

It’s still relatively easy to understand this Jenkinsfile, but imagine that we’ve copied and pasted it in to half a dozen different applications that need to build an Electron app for distribution. If we wanted to update our application and offer an OS X build too, we’d have to update every project and edit every Jenkinsfile one by one. Fortunately, Jenkins has a solution for this too: the Jenkins Global Library.

Creating a Global Library

When writing software applications we frown upon copy and pasting code between files, so why is it accepted when building deployment pipelines? Just like you’d wrap your code up in a library to reuse in your application, we can do the same with our Jenkins build steps.

Just as the Jenkinsfile is written in the Groovy language, our global library will be written in Groovy too. Groovy runs on the JVM, so the namespace structure is very similar to Java. Create a new folder and bootstrap the required files:

mkdir global-lib
cd global-lib
mkdir -p src/com/michaelheap
touch src/com/michaelheap/ElectronApplication.groovy

The ElectronApplication file is where our build logic will live. Take everything in your Jenkinsfile and paste it in to ElectronApplication.groovy inside an execute function:

#!/usr/bin/env groovy

package com.michaelheap;

def execute() {
  node {
    checkout scm
    sh &aposnpm install&apos
    sh &aposnpm test&apos
    sh &aposnpm run lint&apos
    sh &aposnpm run build-linux&apos
    dir(&aposdist&apos) {
      archiveArtifacts artifacts: &apos*.AppImage&apos, fingerprint: true;

return this;

Then update your Jenkinsfile so that it calls just this file. Groovy files are automatically compiled into a class that has the same name as the file, so this would be available as new ElectronApplication(). Your Jenkinsfile should look like the following:

def pipe = new com.michaelheap.ElectronApplication()

Once we update all of our applications to run this class rather than having the same thing in multiple places, any time we need to update the build pipeline we only need to update it in our Global Library and it will automatically be used in the next build of any job that runs.

There is one final step we need to perform before Jenkins can start using our global library. We need to publish it to a repository somewhere (mine’s on GitHub and load it into the Jenkins configuration. Click on Manage Jenkins on the homepage and then Configure System before scrolling down to Global Pipeline Libraries.

To make Jenkins load your library, follow these steps:

  • Click Add.
  • Provide a name.
  • Set the default version to master.
  • Make sure Load Implicitly is checked, so that we don’t need to declare @Library in every Jenkinsfile.
  • Click Modern SCM.
  • Enter your organisation/account username into the Owner box.
  • Select the credentials to use for the scan.
  • Select your Global Library repository.
  • Click Save at the bottom.

The next time any job that references our new com.michaelheap.ElectronApplication definition runs, the global library will automatically be downloaded and imported so that any functions defined in the file can be used. In this case we call execute which runs everything else we needed.

Making the library DRY

Having a Global Library is a huge step towards re-usability, but if we ever needed to build an Electron application that didn’t have any tests to run, or it needed some additional steps running (or even the same steps in a different order) we’d need to copy and paste our ElectronApplication definition and make the changes in a new file. Isn’t this what we were trying to get away from?

Fortunately, I found an awesome example of how you can build your jobs as a pipeline of steps to be executed in the TYPO3-infrastructure project on Github. It’s quite tough to explain, so instead let’s work through an example. Let’s take our existing ElectronApplication and break it down in to five different steps:

  • npm install
  • npm test
  • npm run lint
  • npm run build-linux
  • archiveArtifacts

Each of these is a command that could be run independently, so instead of having them all in a single file, let’s give them each their own class each by creating some new files:

cd src/com/michaelheap
touch Checkout.groovy InstallDeps.groovy Test.groovy Lint.groovy Build.groovy ArchiveArtifacts.groovy

We move each command out of ElectronApplication and in to an execute function in each of these files. It’s important to ensure that they’re in the correct package namespace, and that they’re inside a node block, as we need a workspace:


package com.michaelheap;
def execute() {
  node {
    sh &aposnpm install&apos


package com.michaelheap;
def execute() {
  node {
    dir(&aposdist&apos) {
      archiveArtifacts artifacts: &apos*.AppImage&apos, fingerprint: true;

At this point, your ElectronApplication file will look pretty empty - just an empty execute function and a return this. We need to instruct Jenkins which steps to run. To do this, we’ll add a new run method that tries to execute a step and handles the error if anything fails:

#!/usr/bin/env groovy

package com.michaelheap;

def run(Object step){
    try {
    } catch (err) {
        currentBuild.result = "FAILURE"

def execute() {

return this;

Finally, we have to fill in the execute method with all of the steps we want to run:

def execute() { Checkout()); InstallDeps()); Test()); Lint()); Build()); ArchiveArtifacts());

Once we commit all of our changes and new files to Github, the next time an ElectronApplication pipeline runs, it’ll use our new code.

For example, if we ever needed to set up a pipeline that automatically tested and published new modules to NPM, we wouldn’t have to reimplement all of those steps. We may have to create a new Publish.groovy that runs npm publish, but we can reuse all of our existing steps by creating an NpmModule.groovy file that has the following execute function:

def execute() { Checkout()); InstallDeps()); Test()); Lint()); Publish());

Once that’s added to our global library, we can use it in any project by adding a Jenkinsfile with the following contents:

def pipe = new com.michaelheap.NpmModule();

This will reuse all of our existing steps alongside the new Publish step to test, lint, and publish an NPM module.

Conditional builds

One of the awesome things about the GitHub Organization plugin is that it automatically detects new branches and pull requests and runs builds for them. This is great for things like testing and linting, but we don’t want to publish every branch we create. Generally, we want to run all of our tests on every branch but only publish from the master branch. Fortunately, Jenkins provides the branch name as an environment variable called env.BRANCH_NAME. We can therefore add a conditional in NpmModule.groovy so that we only publish when the pipeline is running against master:

def execute() { Checkout()); InstallDeps()); Test()); Lint());
    if (env.BRANCH_NAME == "master") { Publish());

This works great for teams that are working towards a continuous delivery goal, but in the real world we do sometimes have to deploy from other branches too - whether it’s a legacy branch that receives security fixes or multiple active versions. Jenkins lets you do this too - the Jenkinsfile is just code after all!

If we wanted to publish everything on master, but also everything where the branch name starts with publish- we could change our if statement to look like the following:

if (env.BRANCH_NAME == "master" || (env.BRANCH_NAME.length() >= 8 && env.BRANCH_NAME.substring(0, 8) == "publish-")) { Publish());

It’s a bit long, but it gets the job done. Now, only commits to either master or publish-* will be published.

Using 3rd Party Plugins in a Jenkinsfile

The final part of this post is a section on calling 3rd party plugins from your Jenkinsfile. Plugins may specifically support the Jenkinsfile (you can find a list here), in which case they’re nice and easy to use. Take the Slack plugin for example - it’s been updated with Jenkinsfile compatibility, so it’s as simple as calling slackSend "string to send" in your Jenkinsfile.

Sadly, not all plugins have been updated with Jenkinsfile friendly syntax. In this situation, you need to know which class you want to call and use the special step method. The ws-cleanup plugin is one that I use all the time that hasn’t been updated, so I have to call it via step([$class: 'WsCleanup']). ws-cleanup doesn’t accept any parameters, but you can pass parameters via the step method. For example, the JUnitResultArchiver can be called via step([$class: 'JUnitResultArchiver', testResults: '**/target/surefire-reports/TEST-*.xml']) (as seen in the pipeline plugin documentation).

If we wrap these calls to step up in a custom step like we did with InstallDeps, Test etc then we can start working with the abstractions in our build pipelines. If the plugin ever updates to provide a Jenkinsfile friendly interface we only have a single place to edit rather than dozens of different projects.


package com.michaelheap;
def execute() {
  node {
    step([$class: &aposWsCleanup&apos])

Hopefully you won’t find too many plugins that aren’t pipeline-friendly. As more and more people are starting to use the Jenkinsfile, plugin authors are making it easier to work with their addons.

There’s more!

We’re only scratching the surface, and there’s so much more that Jenkins’ new pipeline support can do; we didn’t even get around to building on slave machines, running jobs in parallel, or stage support! If you’d like to learn more about any of these topics (or Jenkinsfiles in general) feel free to ask! You can find me on Twitter as @mheap or you can email me at

If you’re doing something really cool with Jenkins and the Jenkinsfile, I’d love to hear from you too. I bet I could definitely learn something from you, too!

by Christopher Webber ( at December 08, 2016 05:00 AM


Announcement: “The Ultimate Game Boy Talk” at 33C3

I will present “The Ultimate Game Boy Talk” at the 33rd Chaos Communication Congress in Hamburg later in December.

If you are interested in attending the talk, please go to, select it and press submit, so the organizers can reserve a big enough room.

The talk continues the spirit of The Ultimate Commodore 64 Talk, which I presented at the same conference eight years ago, as well as several other talks in the series done by others: Atari 2600 (Svolli), Galaksija (Tomaž Šolc), Amiga 500 (rahra).

Here’s the abstract:

The 8-bit Game Boy was sold between 1989 and 2003, but its architecture more closely resembles machines from the early 1980s, like the Commodore 64 or the NES. This talk attempts to communicate “everything about the Game Boy” to the listener, including its internals and quirks, as well as the tricks that have been used by games and modern demos, reviving once more the spirit of times when programmers counted clock cycles and hardware limitations were seen as a challenge.

by Michael Steil at December 08, 2016 02:49 AM

December 07, 2016

Errata Security

Orin's flawed argument on IP address privacy

In the PlayPen cases, judges have ruled that if you use the Tor network, then you don't have a reasonable expectation of privacy. It's a silly demonstration of how the law is out of sync with reality, since the entire point of using Tor is privacy.

Law prof Orin Kerr has a post discussing it. His conclusion is correct, that when the FBI exploits 0day and runs malware on your computer, then it's a search under the Fourth Amendment, requiring a warrant upon probable cause.

However, his reasoning is partly flawed. The title of his piece, "Remotely accessing an IP address inside a target computer is a search", is factually wrong. The IP address in question is not inside a target computer. This may be meaningful.

First, let's discuss how the judge reasons that there's no expectation of privacy with Tor. This is a straightforward application if the Third Party Doctrine, that as soon as you give something to a third party, your privacy rights are lost. Since you give your IP address to Tor, you lose privacy rights over it. You don't have a reasonable expectation of privacy: yes, you have an expectation of privacy, but it's not a reasonable one, and thus it's not protected.

The same is true of all your other digital information. Your credit card receipts, phone metadata, email archive, and all the rest of things you want to keep private on the Internet are not (currently) covered by the Fourth Amendment.

If you are thinking this is bullcrap, then you'd be right. Everyone knows the Third Party Doctrine doesn't fit the Internet. We want these things to be private from the government, meaning, that they must get a warrant to access them. But it's going to take a clueful Supreme Court overturning past precedence or an armed revolution in order to change things.

But that doesn't necessarily fit this case.  As Orin Kerr's post points out:
Fourth Amendment law regulates how the government learns information, not what information it learns
In other words, it doesn't matter if the FBI is allowed to get your IP address, they still need a warrant to search your computer. If you've got public information in your house, the FBI still needs a warrant to enter your house in order to get it.

Where Orin's argument is flawed is the fact that the IP address isn't on the computer being searched by the FBI's "NIT" malware. In other cases, the FBI will be able to discover a target's IP address without a search of their computer. His post would be better entitled something like "Infecting with malware is always a search" instead.

The way the Internet works is that computers have a local IP address that's meaningful only on the local network (like the one inside your home). For example, my laptop currently has the address This may, in fact, be the same address as your laptop. That's because the addresses starting with 192,168.x.x is extremely popular for home networks (along with 10.x.x.x). It's like how we both can have the address 1079 Elm Str, just in different cities, since every city has "Elm Street" somewhere.

As data leaves your computer, the local address is translated (network address translation) into a public IP address. Google "what's my ip address", and it will tell you your public IP address. Google knows it, but your computer doesn't.

Instead, it's your home router that knows your public IP address, using your public IP on the Internet and local IP on your home network.

This Cisco router knows my public IP address
It can get even more complicated. When I travel, I use my iPhone as a wifi hotspot. But my iPhone is given a local IP address within the cellphone company's network. This address is shared with hundreds of other cellphone customers. Thus, it's AT&T's routers which knows my public IP address, neither my phone nor my laptop knows it.

Phone doesn't know public IP, only local 10.x.x.x local IP

In the PlayPen case, the FBI discovers the target's public IP address by causing it to transmit information to the FBI. This information goes through the network address translator, and when it arrives on the FBI server, has the public IP address associated with it. In other words, the point where it's discovered is on the FBI's server located in Quantico, not within the "NIT" malware running on the person's computer. The malware on the computer does not "access" the IP address in any fashion --- but by generating traffic from inside the home, it causes the IP address to be revealed outside the home.

Rather than using malware to infect a computer, the FBI might try other ways to discover a suspect's IP address. They might host a PDF or Word document on the server that has a simple image tag pointing to the FBI's server. When the user opens the document, their Acrobat/Word program isn't protected by Tor. There computer will then contact the FBI's server looking for the image, revealing their public IP address. In this example, no exploit or malware is being used. In fact, Tor warns users about this problem. The target is willfully revealing their public IP address purely because they are unaware of the meaning of their actions.

If this were how the FBI were discovering the IP address, rather than using malware, then the judge's reasoning would (probably) be correct. Since the FBI relied upon user stupidity rather than malware, no search was done.

I'd like to see Orin update his post. Either to clarify, contrary to what his title says, that what he really means is "Running malware on a target is always a search". Or conversely, describe how this "image tag" example is, despite my feelings, a search.

As a wholly separate note, I'd like to point out a different flaw in the judge's reasoning. Yes, the entry Tor node knows your IP address, but it doesn't know it belongs to you or is associated with your traffic. Yes, the exit Tor knows your traffic, but it doesn't know your IP address.

Technically, both your traffic and IP address are public (according to the Third Party Doctrine), but the private bit is the fact that the two are related. The "Tor network" isn't a single entity, but a protocol for how various different entities work together. No single entity in the Tor network sees your IP address combined with your activity or identity. Even when the FBI and NSA themselves run Tor nodes, they still can't piece it together. It is a private piece of information.

In other words, the 4 digit PIN number for your ATM card is located in this document, so it's a public number. But which PIN belongs to you is still a secret.

Thus, the judge is wrong. The private information is not the public IP address. The private information is the public IP address combined with the traffic. The person isn't trying to keep their public IP address private, what they are trying to keep private is the fact that this IP address access the PlayPen servers.


This is a stupid post, because it doesn't disagree with Orin's conclusion: FBI running malware always needs a warrant, even if the information they are after is public. However, the technical details are wrong -- the IP address the FBI is after is located nowhere inside the computer they are searching.

by Robert Graham ( at December 07, 2016 10:02 PM

Chris Siebenmann

One reason why user namespaces keep enabling Linux kernel security issues

If you've been following Linux kernel security issues over the past few years, you've probably noticed that one common element that keeps reappearing over and over is 'user namespaces' (most recently in CVE-2016-8655). At this point you may reasonably wonder not just what user namespaces are but why they keep being involved in so many security problems.

The general idea of kernel namespaces is covered in the namespaces(7) manpage and user namespaces have their own specific manpage, user_namespaces(7) . From our perspective, the simple version of what user namespaces do is that they give a process or a group of them an isolated world in which they can effectively be root.

(The manpage has more details and a better explanation.)

By itself this is not a security problem, at least in theory. While you may have elevated capabilities in your user namespace, you're only supposed to be able to use them on things in the kernel that are safe for regular users to manipulate in an isolated environment. For example, you can't use your nominal root permissions to mount any old filesystem with any old mount options; only a very few filesystems will let themselves be mounted this way.

(Sometimes this isn't quite what happens in practice.)

The problem is that even when everything works the way the theory has it, user namespaces still expose a lot of code in the kernel to normal users that was previously only really accessible to root. That code has not necessarily been thoroughly hardened against various sorts of nasty things, so if you can find some issue with some part of it you can then leverage access to it from a user namespace to turn what was a 'root can damage the system, big deal' issue into an 'anyone can do nasty stuff to the system' big deal issue. The actual bugs are not in user namespaces, but user namespaces enable access to them by making a lot more code in the kernel reachable by ordinary users.

(It's my view that these bugs are going to keep surfacing for years to come. User namespaces expose a lot more kernel code to users, that code's not going to be thoroughly audited any time soon, and the user namespace people seem to like adding more and more things that are nominally safe for them to affect.)

Another sort of kernel security issue created by the existence of user namespaces is interactions between operations that are now allowed in user namespaces and other things in the system. CVE-2015-8709 is an example of this; the root cause was an assumption about what a particular security check covered (see the discussion in the kernel fix). Security checks in general are a complex system, with all that that implies.

(See also Anatomy of a user namespaces vulnerability and this overview of user namespaces and namespaces in general.)

by cks at December 07, 2016 07:19 AM

System Administration Advent Calendar

Day 7 - What we can learn about Change from the CIA

Written By: Michael Stahnke (@stahnma)
Edited By: Michelle Carroll (@miiiiiche)

I’ve recently spent some time on the road, working with customers and potential customers, as well as speaking at conferences. It’s been great. During the discussion with customers and prospects, I’m fascinated by the organizational descriptions, behaviors, and policies.

I was reflecting on one of those discussions one week, when I was preparing a lightning talk for our Madison DevOps Meetup. I looked through my chicken scratch notes I keep on talk ideas to see what I could whip up, and found a note about the CIA Simple Sabotage Field Manual. This guide was written in 1944, and declassified in 2008. It’s a collection of worst-practices to run a business. The intent of the guide is to have CIA assets, or citizens of occupied countries, slow the output of companies they are placed in, and thus reducing their effectiveness in supplying enemies. Half of these tips and tricks describe ITIL.

ITIL comes from the Latin word Itilus which means give up.

ITIL and change control processes came up a lot over my recent trips. I’ve never been a big fan of ITIL, but I do think the goals it set out to achieve were perfectly reasonable. I, too, would like to communicate about change before it happens, and have good processes around change. Shortly thereafter is where my ability to work within ITIL starts to deviate.

Let’s take a typical change scenario.

I am a system administrator looking to add a new disk into a volume group on a production system.

First, I open up some terrible ticket-tracking tool. Even if you’re using the best ticket tracking tool out there, you hate it. You’ve never used a ticket tracking tool and thought, “Wow, that was fun.” So I open a ticket of type “change.” Then I add in my plan, which includes scanning the bus for the new disk, running some lvm commands, and creating or expanding a filesystem. I add a backout plan, because that’s required. I guess the backout plan would be to not create the fileystem or expand it. I can’t unscan a bus. Then I have myriad of other fields to fill out, some required by the tool, some required by your company’s process folks (but not enforced at the form level). I save my work.

Now this change request is routed for approvals. It’s likely that somewhere between one and eight people review the ticket, approve the ticket and move it state into ready for review, or ready to be talked about or the like.

From there, I enter my favorite (and by favorite, I mean least favorite), part of the process: the Change Advisory Board (CAB). This is a required meeting that you have to go, or send a representative. They will talk about all changes, make sure all the approvals are in, make sure a backout plan is filled out, make sure the ticket is in the ready to implement phase. This meeting will hardly discuss the technical process of the change. It will barely scratch the surface of an impact analysis for the work. It might ask what time that change will occur. All in all, the CAB meeting is made up of something like eight managers, four project managers, two production control people, and a slew of engineers who just want to get some work done.

Oh, and because reasons, all changes must be open for at least two weeks before implementation, unless it’s an emergency.

Does this sound typical to you? It matches up fairly well with several customers I met with over the last several months.

Let’s recap:

Effort: 30 minutes at the most.

Lag Time: 2+ weeks.

Customer: unhappy.

If I dig into each of these steps, I’m sure we can make this more efficient.

Ticket creation:

If you have required fields for your process, make them required in the tool. Don’t make a human audit tickets and figure out if you forgot to fill out Custom Field 3 with the correct info.

Have a backout plan when it makes sense, recognize when it doesn’t. Rollback, without time-travel, is basically a myth.

Approval Routing:

Who is approving this? Why? Is it the business owner of the system? The technical owner? The manager of the infra team? The manager of the business team? All of them?

Is this adding any value to the process or is it simply a process in place so that if something goes wrong (anything, anything at all) there’s a clear chain of “It’s not my fault” to show? Too many approvals may indicate you have a buck-passing culture (you’re not driving accountability).

Do you have to approve this exact change, or could you get mass approval on this type of change and skip this step in the future? I’ve had success getting DNS changes, backup policy modifications, disk maintenance, account additions/removals, and library installations added to this bucket.


How much does this meeting cost? 12-20 people: if the average rate is $50 an hour per person, you’re looking at $600-$1000 just to talk about things in a spreadsheet or PDF. Is this cost in line with the value of the meeting?
What’s the most valuable thing that setup provides? Could it be done asynchronously?

Waiting Period:

Seriously, why? What good does a ticket do by sitting around for 2 weeks? Somebody could happen to stumble upon it while they browse your change tickets in their free time, and then ask an ever-so-important question. However, I don’t have stories or evidence that confirm this possibility.

CIA Sabotage Manual

Let’s see which of the CIA worst-practices to implement in an org (or perhaps best practices to ensure failure) this process hits:

Employees: Work slowly. Think of ways to increase the number of movements needed to do your job: use a light hammer instead of a heavy one; try to make a small wrench do instead of a big one.

This slowness is built into the system with a required duration of 2 weeks. It also requires lots of movement in the approval process. What if approver #3 is on holiday? Can the ticket move into the next phase?

When possible, refer all matters to committees, for “further study and consideration.” Attempt to make the committees as large and bureaucratic as possible. Hold conferences when there is more critical work to be done.

This just described a CAB meeting to the letter. Why make a decision about moving forward when we could simply talk about it and use up as many people’s time as possible?

Maybe, you think I’m being hyperbolic. I don’t think I am. I am certainly attempting to make a point, and to make it entertaining, but this is a very real-world scenario.

Now, if we apply some better practices here, what can we do? I see two ways forward. You can work within a fairly stringent ITIL-backed system. If you choose this path, the goal is to keep the processes forced upon you as out of your way as possible. The other path is to create a new process that works for you.

Working within the process

To work within a process structured with a CAB, a review, and waiting period, you’ll need to be aggressive. Most CAB process have a standard change flow, or pre-approved change flow for things that just happen all the time. Often you have to demonstrate a number of successful changes of a type to be considered for this type of change categorization.

If you have an option like that, drive toward it. When I last ran an operations team, we had dozens (I’m not even sure of the final tally) of standard, pre-approved change types set up. We kept working to get more and more of our work into this category.

The pre-approved designation meant it didn’t have to wait two weeks, and rarely needed to get approval. In cases where it did, it was the technical approver of the service/system who did the approval, which bypassed layers of management and production control processes.

That’s not to say we always ran things through this way. Sometimes, more eyes on a change is a good thing. We’d add approvers if it made sense. We’d change the type of change to normal or high impact if we had less confidence this one would go well. One of the golden rules was, don’t be the person who has an unsuccessful pre-approved change. When that happened, that change type was no longer pre-approved.

To get things into the pre-approved bucket, there was a bit of paperwork, but mostly, it was a matter of process. We couldn’t afford variability. I needed to have the same level of confidence that a change would work, no matter the experience of the person making the change. This is where automation comes in.

Most people think you automate things for speed, and you certainly can, but consistency was a much larger driver around automation for us. We’d look at a poorly-defined process, clean it up, and automate.

After getting 60%+ of the normal changes we made into the pre-approved category, our involvement in the ITIL work displacement activities shrunk dramatically. Since things were automated, our confidence level in the changes was high. I still didn’t love our change process, but we were able to remove much of its impact on our work.

Automating a bad proess just outputs crap...faster

Have a different process

At my current employer, we don’t have a strong ITIL practice, a change advisory board, or mandatory approvers on tickets. We still get stuff done.

Basically, when somebody needs to make a change, they’re responsible for figuring out the impact analysis of it. Sometimes, it’s easy and you know it off the top of your head. Sometimes, you need to ask other folks. We do this primarily on a voluntary mailing list — people who care about infrastructure stuff subscribe to it.

We broadcast proposed changes on that list. From there, impact information can be discovered and added. We can decide timing. We also sometimes defer changes if something critical is happening, such as release hardening.

In general, this has worked well. We’ve certainly had changes that had a larger impact than we originally planned, but I saw that with a full change control board and 3–5 approvers from management as well. We’ve also had changes sneak in that didn’t get the level of broadcast we’d like to see ahead of time. That normally only happens once for that type of change. We also see many changes not hit our discussion list because they’re just very trivial. That’s a feature.


If you work in an environment with lots of regulations preventing a more collaborative and iterative process, the first thing I encourage you to do is question those constraints. Are they in place for legal coverage, or are they simply “the way we’ve always worked?” If you’re not sure, dig in a little bit with the folks enforcing regulations. Sometimes a simple discussion about incentives and what behaviors you’re attempting to drive can cause people to rethink a process or remove a few pieces of red tape.

If you have regulations and constraints due to government policies, such as PCI or HIPAA, then you may have to conform. One of the common control in those types of environments is people who work in development environment may not have access or push code into production. If this is the case, dig into what that really means. I’ve seen several organization determine those constraints based on how they were currently operating, instead of what they could be doing.

A common rule is developers should not have uncontrolled access to production. often times companies see that mean they must restrict all access to production for developers. Instead however, if you focus on the uncontrolled part, you may find different incentives for the control. Could you mitigate risks by allowing developers to perform automated deployments and by having read-access for logs, but not have a shell prompt on the systems? If so, you’re still enabling collaboration and rapid movement, without creating a specific handover from development to a production control team.


The way things have always been done probably isn’t the best way. It’s a way. I encourage you to dig in, and play devil’s advocate for your current processes. Read a bit of the CIA sabotage manual, and if starts to hit too close to home, look at your processes, and rethink the approach. Even if you’re a line-level administrator or engineer, your questions and process concerns should be welcome. You should be able to receive justification for why things are the way they are. Dig in and fight that bureaucracy. Make change happen, either to the computers or to the process.


by Christopher Webber ( at December 07, 2016 05:00 AM

Create a PDP-8 OS8 RK05 system disk from RX01 floppies with SIMH (and get text files in and out of the PDP-8)

This guide shows you how to build an RK05 bootable system disk with OS/8 on it for the PDP-8, in the SIMH emulator. We will use two RX01 floppies as the build source, copy over all the files and set up the LPT printer and the PTR/PIP paper tape punch/readers. As an added bonus the article also shows you how to get text files in and out of the PDP-8 sytem using the printer and papertape reader / puncher.

December 07, 2016 12:00 AM

December 06, 2016

Errata Security

That "Commission on Enhancing Cybersecurity" is absurd

An Obama commission has publish a report on how to "Enhance Cybersecurity". It's promoted as having been written by neutral, bipartisan, technical experts. Instead, it's almost entirely dominated by special interests and the Democrat politics of the outgoing administration.

In this post, I'm going through a random list of some of the 53 "action items" proposed by the documents. I show how they are policy issues, not technical issues. Indeed, much of the time the technical details are warped to conform to special interests.

IoT passwords

The recommendations include such things as Action Item 2.1.4:
Initial best practices should include requirements to mandate that IoT devices be rendered unusable until users first change default usernames and passwords. 
This recommendation for changing default passwords is repeated many times. It comes from the way the Mirai worm exploits devices by using hardcoded/default passwords.

But this is a misunderstanding of how these devices work. Take, for example, the infamous Xiongmai camera. It has user accounts on the web server to control the camera. If the user forgets the password, the camera can be reset to factory defaults by pressing a button on the outside of the camera.

But here's the deal with security cameras. They are placed at remote sites miles away, up on the second story where people can't mess with them. In order to reset them, you need to put a ladder in your truck and drive 30 minutes out to the site, then climb the ladder (an inherently dangerous activity). Therefore, Xiongmai provides a RESET.EXE utility for remotely resetting them. That utility happens to connect via Telnet using a hardcoded password.

The above report misunderstands what's going on here. It sees Telnet and a hardcoded password, and makes assumptions. Some people assume that this is the normal user account -- it's not, it's unrelated to the user accounts on the web server portion of the device. Requiring the user to change the password on the web service would have no effect on the Telnet service. Other people assume the Telnet service is accidental, that good security hygiene would remove it. Instead, it's an intended feature of the product, to remotely reset the device. Fixing the "password" issue as described in the above recommendations would simply mean the manufacturer would create a different, custom backdoor that hackers would eventually reverse engineer, creating MiraiV2 botnet. Instead of security guides banning backdoors, they need to come up with standard for remote reset.

That characterization of Mirai as an IoT botnet is wrong. Mirai is a botnet of security cameras. Security cameras are fundamentally different from IoT devices like toasters and fridges because they are often exposed to the public Internet. To stream video on your phone from your security camera, you need a port open on the Internet. Non-camera IoT devices, however, are overwhelmingly protected by a firewall, with no exposure to the public Internet. While you can create a botnet of Internet cameras, you cannot create a botnet of Internet toasters.

The point I'm trying to demonstrate here is that the above report was written by policy folks with little grasp of the technical details of what's going on. They use Mirai to justify several of their "Action Items", none of which actually apply to the technical details of Mirai. It has little to do with IoT, passwords, or hygiene.

Public-private partnerships
Action Item 1.2.1: The President should create, through executive order, the National Cybersecurity Private–Public Program (NCP 3 ) as a forum for addressing cybersecurity issues through a high-level, joint public–private collaboration.
We've had public-private partnerships to secure cyberspace for over 20 years, such as the FBI InfraGuard partnership. President Clinton's had a plan in 1998 to create a public-private partnership to address cyber vulnerabilities. President Bush declared public-private partnerships the "cornerstone of his 2003 plan to secure cyberspace.

Here we are 20 years later, and this document is full of new naive proposals for public-private partnerships There's no analysis of why they have failed in the past, or a discussion of which ones have succeeded.

The many calls for public-private programs reflects the left-wing nature of this supposed "bipartisan" document, that sees government as a paternalistic entity that can help. The right-wing doesn't believe the government provides any value in these partnerships. In my 20 years of experience with government private-partnerships in cybersecurity, I've found them to be a time waster at best and at worst, a way to coerce "voluntary measures" out of companies that hurt the public's interest.

Build a wall and make China pay for it
Action Item 1.3.1: The next Administration should require that all Internet-based federal government services provided directly to citizens require the use of appropriately strong authentication.
This would cost at least $100 per person, for 300 million people, or $30 billion. In other words, it'll cost more than Trump's wall with Mexico.

Hardware tokens are cheap. Blizzard (a popular gaming company) must deal with widespread account hacking from "gold sellers", and provides second factor authentication to its gamers for $6 each. But that ignores the enormous support costs involved. How does a person prove their identity to the government in order to get such a token? To replace a lost token? When old tokens break? What happens if somebody's token is stolen?

And that's the best case scenario. Other options, like using cellphones as a second factor, are non-starters.

This is actually not a bad recommendation, as far as government services are involved, but it ignores the costs and difficulties involved.

But then the recommendations go on to suggest this for private sector as well:
Specifically, private-sector organizations, including top online retailers, large health insurers, social media companies, and major financial institutions, should use strong authentication solutions as the default for major online applications.
No, no, no. There is no reason for a "top online retailer" to know your identity. I lie about my identity. thinks my name is "Edward Williams", for example.

They get worse with:
Action Item 1.3.3: The government should serve as a source to validate identity attributes to address online identity challenges.
In other words, they are advocating a cyber-dystopic police-state wet-dream where the government controls everyone's identity. We already see how this fails with Facebook's "real name" policy, where everyone from political activists in other countries to LGBTQ in this country get harassed for revealing their real names.

Anonymity and pseudonymity are precious rights on the Internet that we now enjoy -- rights endangered by the radical policies in this document. This document frequently claims to promote security "while protecting privacy". But the government doesn't protect privacy -- much of what we want from cybersecurity is to protect our privacy from government intrusion. This is nothing new, you've heard this privacy debate before. What I'm trying to show here is that the one-side view of privacy in this document demonstrates how it's dominated by special interests.

Cybersecurity Framework
Action Item 1.4.2: All federal agencies should be required to use the Cybersecurity Framework. 
The "Cybersecurity Framework" is a bunch of a nonsense that would require another long blogpost to debunk. It requires months of training and years of experience to understand. It contains things like "DE.CM-4: Malicious code is detected", as if that's a thing organizations are able to do.

All the while it ignores the most common cyber attacks (SQL/web injections, phishing, password reuse, DDoS). It's a typical example where organizations spend enormous amounts of money following process while getting no closer to solving what the processes are attempting to solve. Federal agencies using the Cybersecurity Framework are no safer from my pentests than those who don't use it.

It gets even crazier:
Action Item 1.5.1: The National Institute of Standards and Technology (NIST) should expand its support of SMBs in using the Cybersecurity Framework and should assess its cost-effectiveness specifically for SMBs.
Small businesses can't even afford to even read the "Cybersecurity Framework". Simply reading the doc, trying to understand it, would exceed their entire IT/computer budget for the year. It would take a high-priced consultant earning $500/hour to tell them that "DE.CM-4: Malicious code is detected" means "buy antivirus and keep it up to date".

Software liability is a hoax invented by the Chinese to make our IoT less competitive
Action Item 2.1.3: The Department of Justice should lead an interagency study with the Departments of Commerce and Homeland Security and work with the Federal Trade Commission, the Consumer Product Safety Commission, and interested private sector parties to assess the current state of the law with regard to liability for harm caused by faulty IoT devices and provide recommendations within 180 days. 
For over a decade, leftists in the cybersecurity industry have been pushing the concept of "software liability". Every time there is a major new development in hacking, such as the worms around 2003, they come out with documents explaining why there's a "market failure" and that we need liability to punish companies to fix the problem. Then the problem is fixed, without software liability, and the leftists wait for some new development to push the theory yet again.

It's especially absurd for the IoT marketspace. The harm, as they imagine, is DDoS. But the majority of devices in Mirai were sold by non-US companies to non-US customers. There's no way US regulations can stop that.

What US regulations will stop is IoT innovation in the United States. Regulations are so burdensome, and liability lawsuits so punishing, that it will kill all innovation within the United States. If you want to get rich with a clever IoT Kickstarter project, forget about it: you entire development budget will go to cybersecurity. The only companies that will be able to afford to ship IoT products in the United States will be large industrial concerns like GE that can afford the overhead of regulation/liability.

Liability is a left-wing policy issue, not one supported by technical analysis. Software liability has proven to be immaterial in any past problem and current proponents are distorting the IoT market to promote it now.

Cybersecurity workforce
Action Item 4.1.1: The next President should initiate a national cybersecurity workforce program to train 100,000 new cybersecurity practitioners by 2020. 
The problem in our industry isn't the lack of "cybersecurity practitioners", but the overabundance of "insecurity practitioners".

Take "SQL injection" as an example. It's been the most common way hackers break into websites for 15 years. It happens because programmers, those building web-apps, blinding paste input into SQL queries. They do that because they've been trained to do it that way. All the textbooks on how to build webapps teach them this. All the examples show them this.

So you have government programs on one hand pushing tech education, teaching kids to build web-apps with SQL injection. Then you propose to train a second group of people to fix the broken stuff the first group produced.

The solution to SQL/website injections is not more practitioners, but stopping programmers from creating the problems in the first place. The solution to phishing is to use the tools already built into Windows and networks that sysadmins use, not adding new products/practitioners. These are the two most common problems, and they happen not because of a lack of cybersecurity practitioners, but because the lack of cybersecurity as part of normal IT/computers.

I point this to demonstrate yet against that the document was written by policy people with little or no technical understanding of the problem.

Nutritional label
Action Item 3.1.1: To improve consumers’ purchasing decisions, an independent organization should develop the equivalent of a cybersecurity “nutritional label” for technology products and services—ideally linked to a rating system of understandable, impartial, third-party assessment that consumers will intuitively trust and understand. 
This can't be done. Grab some IoT devices, like my thermostat, my car, or a Xiongmai security camera used in the Mirai botnet. These devices are so complex that no "nutritional label" can be made from them.

One of the things you'd like to know is all the software dependencies, so that if there's a bug in OpenSSL, for example, then you know your device is vulnerable. Unfortunately, that requires a nutritional label with 10,000 items on it.

Or, one thing you'd want to know is that the device has no backdoor passwords. But that would miss the Xiongmai devices. The web service has no backdoor passwords. If you caught the Telnet backdoor password and removed it, then you'd miss the special secret backdoor that hackers would later reverse engineer.

This is a policy position chasing a non-existent technical issue push by Pieter Zatko, who has gotten hundreds of thousands of dollars from government grants to push the issue. It's his way of getting rich and has nothing to do with sound policy.

Cyberczars and ambassadors

Various recommendations call for the appointment of various CISOs, Assistant to the President for Cybersecurity, and an Ambassador for Cybersecurity. But nowhere does it mention these should be technical posts. This is like appointing a Surgeon General who is not a doctor.

Government's problems with cybersecurity stems from the way technical knowledge is so disrespected. The current cyberczar prides himself on his lack of technical knowledge, because that helps him see the bigger picture.

Ironically, many of the other Action Items are about training cybersecurity practitioners, employees, and managers. None of this can happen as long as leadership is clueless. Technical details matter, as I show above with the Mirai botnet. Subtlety and nuance in technical details can call for opposite policy responses.


This document is promoted as being written by technical experts. However, nothing in the document is neutral technical expertise. Instead, it's almost entirely a policy document dominated by special interests and left-wing politics. In many places it makes recommendations to the incoming Republican president. His response should be to round-file it immediately.

I only chose a few items, as this blogpost is long enough as it is. I could pick almost any of of the 53 Action Items to demonstrate how they are policy, special-interest driven rather than reflecting technical expertise.

by Robert Graham ( at December 06, 2016 09:11 PM

Geek and Artist - Tech

What’s Missing From Online Javascript Courses

Perhaps the title is somewhat excessive, but it expresses how I feel about this particular topic. I’m not a “front-end person” (whatever that is) and feel much more comfortable behind an API where you don’t have to worry about design, markup, logic, styling as well as how to put them all together. That being said, I feel it’s an area where I should face my fears head-on, and so I’m doing some small side-projects on the web.

One thing that I realised I had no idea about, is how you actually get a web app running. I don’t mean starting a server, or retrieving a web page, or even which line of javascript is executed first. I’m talking about how you put all the scripts and bits of code in the right place that the browser knows when to run them, and you don’t make an awful mess in the process.

This can only be shown by example, so here’s what I inevitably start with:

    <script src="//" />
    <div id="data" />
    <script type="text/javascript">
        success: function(data) {
          $("data").text = data

Yes, I’m using jQuery and no that code example is probably not entirely correct. I still find there is a reasonable period of experimentation involved before even the simple things like an AJAX call to get some data from an API are working. In any case, here we are with some in-line Javascript and things are generally working as expected. But of course we know that in-lining Javascript is not the way to a working, maintainable application, so as soon as we have something working, we should pull it into its own external script.

    <script src="//" />
    <script src="/javascripts/main.js" />
    <div id="data" />

Uh-oh, it stopped working. The code in main.js is the exact same as what we had in the document before but it is no longer functioning. Already, we are beyond what I’ve seen in most beginner Javascript online courses, yet this seems like a pretty fundamental issue. Of course, the reason is that the script has been loaded and executed in the same order as the script tags and before the HTML elements (including the div we are adding the data to) were present in the DOM.

So naturally we exercise jQuery and fix the problem, by only executing the code once the document is ready and the relevant event handler is fired:

    success: function(data) {
      $("data").text = data

But now we have another problem. We’ve heard from more experienced developers that using jQuery is frowned upon, and although figuring out when the document is loaded seems simple enough to do without using a library, we’re not sure that there is a single cross-browser way of doing it reliably. So jQuery it is.

Actually there is another way, well explained here and seems to be well supported without relying on Javascript functions. You simply drop the “defer” keyword into the script tag you want to execute after parsing of the page, and it will now only run at the right time for our purposes:

    <script src="/javascripts/main.js" defer/>

I had never seen that before, but it was so simple. Many thanks to my coworkers Maria and Thomas for shedding a bit of light on this corner of the browser world for me. Of course, they also mentioned correctly that using jQuery is not an unforgivable sin, nor are some cases of in-line Javascript snippets (look at some of your favourite websites, even those from respected tech giants, and you will undoubtedly see some). But for a novice to web development it is sometimes hard to look beyond Hackernews and figure out what you are meant to be doing.

On to the next web challenge – mastering D3!

by oliver at December 06, 2016 08:54 PM

Multi-repo Git status checking script

I've got a whole bunch of Git repositories in my ~/Projects/ directory. All of those may have unstaged, uncommitted or unpushed changes. I find this hard to keep track of properly, so I wrote a script to do this for me. The output looks like this:


As you can see, it shows:

  • Untracked files: File that are new, are unknown to git and have not been ignored.
  • Uncommitted changes: Files that are known to git and have changes which are not committed.
  • Needs push: New local commits which have not been pushed to the remove origin. 

The script scans for .git directories under the given path. It will only scan a certain level deep. The default for this is 2, which means "all directories directly under this directory". A value of '3' would scan two directories deep.

Full usage:

Usage: <DIR> [DEPTH=2]
​Scan for .git dirs under DIR (up to DEPTH dirs deep) and show git status

Get is from the Github project page.

by admin at December 06, 2016 04:01 PM

System Administration Advent Calendar

Day 6 - No More On-Call Martyrs

Written By: Alice Goldfuss (@alicegoldfuss)
Edited By: Justin Garrison (@rothgar)

Ops and on-call go together like peanut butter and jelly. It folds into the batter of our personalities and gives it that signature crunch. It’s the gallows from which our humor hangs.

Taking up the pager is an ops rite-of-passage, a sign that you are needed and competent. Being on-call is important because it entrusts you with critical systems. Being on-call is the best way to ensure your infrastructure maintains its integrity and stability.

Except that’s bullshit.

The best way to ensure the stability and safety of your systems is to make them self-healing. Machines are best cared for by other machines, and humans are only human. Why waste time with a late night page and the fumblings of a sleep-deprived person when a failure could be corrected automatically? Why make a human push a button when a machine could do it instead?

If a company was truly invested in the integrity of its systems, it would build simple, scalable ones that could be shepherded by such automation. Simple systems are key, because they reduce the possible failure vectors you need to automate against. You can’t slap self-healing scripts onto a spaghetti architecture and expect them to work. The more complex your systems become, the more you need a human to look at the bigger picture and make decisions. Hooking up restart and failover scripts might save yourself some sleepless nights, but it wouldn’t guard against them entirely.

That being said, I’m not aware of any company with such an ideal architecture. So, if not self-healing systems, why not shorter on-call rotations? Or more people on-call at once? After all, going 17 hours without sleep can be equivalent to a blood alcohol concentration of 0.05%, and interrupted sleep causes a marked decline in positive mood. Why trust a single impaired person with the integrity of your system? And why make them responsible for it a week at a time?

Because system integrity is only important when it impacts the bottom line. If a single engineer works herself half-to-death but keeps the lights on, everything is fine.

And from this void, a decades-old culture has arisen.

There is a cult of masochism around on-call, a pride in the pain and of conquering the rotating gauntlet. These martyrs are mostly found in ops teams, who spend sleepless nights patching deploys and rebuilding arrays. It’s expected and almost heralded. Every on-call sysadmin has war stories to share. Calling them war stories is part of the pride.

This is the language of the disenfranchised. This is the reaction of the unappreciated.

On-call is glorified when it’s all you’re allowed to have. And, historically, ops folk are allowed to have very little. Developers are empowered to create and build, while ops engineers are only allowed to maintain and patch. Developers are expected to be smart; ops engineers are expected to be strong.

No wonder so many ops organizations identify with military institutions and use phrases such as “firefighting” to describe their daily grind. No wonder they craft coats of arms and patches and nod to each other with tales of horrendous outages. We redefine what it means to be a hero and we revel in our brave deeds.

But, at what cost? Not only do we miss out on life events and much-needed sleep, but we also miss out on career progression. Classic sysadmin tasks are swiftly being automated away, and if you’re only allowed to fix what’s broken, you’ll never get out of that hole. Furthermore, you’ll burn out by bashing yourself against rocks that will never move. No job is worth that.

There is only one real benefit to being on-call: you learn a lot about your systems by watching them break. But if you’re only learning, never building, you and your systems will stagnate.

Consider the following:

  1. When you get paged, is it a new problem? Do you learn something, or is it the same issue with the same solution you’ve seen half a dozen times?
  2. When you tell coworkers you were paged last night, how do they react? Do they laugh and share stories, or are they concerned?
  3. When you tell your manager your on-call shift has been rough, do they try to rotate someone else in? Do they make you feel guilty?
  4. Is your manager on-call too? Do they cover shifts over holidays or offer to take an override? Do they understand your burden?

It’s possible you’re working in a toxic on-call culture, one that you glorify because it’s what you know. But it doesn’t have to be this way. Gilded self-healing systems aside, there are healthier ways to approach on-call rotations:

  1. Improve your monitoring and alerting. Only get paged for actionable things in an intelligent way. The Art of Monitoring is a good place to start.
  2. Have rules in place regarding alert fatigue. The Google SRE book considers more than two pages per 12 hour shift too many.
  3. Make sure you’re compensated for on-call work, either financially or with time-off, and make sure that’s publicly supported by management.
  4. Put your developers on-call. You’ll be surprised what stops breaking.

For those of you who read these steps and think, “that’s impossible,” I have one piece of advice: get another job. You are not appreciated where you are and you can do much better.

On-call may be a necessary evil, but it shouldn’t be your whole life. In the age of cloud platforms and infrastructure as code, your worth should be much more than editing Puppet manifests. Ops engineers are intelligent, scrappy, and capable of building great things. You should be allowed to take pride in making yourself and your systems better, and not just stomping out yet another fire.

Work on introducing healthier on-call processes in your company, so you can focus on developing your career and enjoying your life.

In the meantime, there is support for weathering rough rotations. I started a hashtag called #oncallselfie to share the ridiculous circumstances I’m in when paged. There’s also The On-Call Handbook as a primer for on-call shifts and a way to share what you’ve learned along the way. And if you’re burned out, I suggest this article as a good first step toward getting back on your feet.

You’re not alone and you don’t have to be a martyr. Be a real hero and let the pager rest.

by Christopher Webber ( at December 06, 2016 12:28 PM

USB 3 interferes with wireless radio-controlled keyboards and mice

I bought an Intel NUC Kit NUC5CPYH as a replacement for my Raspberry Pi 2b media center and file server. One of the main advantages of the NUC over the Raspberry is that it has USB v3 ports, which greatly increases the throughput speeds to the external USB disks and doesn't require me to have external power on some of them.

I use a radio-controlled keyboard as my remote control my media center. When I plugged it into the NUC, I suddenly found the range of the keyboard greatly reduced. Even when I placed it in exactly the same place as the Raspberry Pi, the range was still horrible further than about 1.5 meters.

Turns out USB 3.x ports interfere with 2.4 GHz devices, which the radio-controlled wireless keyboard operates on.

The solution was to plug in a USB extension cord and putting the RC receiver in that and placing it about 30 cm from the NUC.

On a side-note: Disable UEFI boot if you can't seem to install any Linux distro on your NUC.

by admin at December 06, 2016 10:19 AM

Chris Siebenmann

My RPM build setup no longer works on Fedora for some packages

A decade ago I wrote up how I set up to build RPMs, with sources for each RPM segregated into their own subdirectory under your RPM SOURCES directory instead of everything piled into SOURCES the way the default setup wants you to do it. I've used that RPM building setup ever since and it's worked for me over all of these years. The short version of what this looks like is:

%_topdir /some/where
%_sourcedir %{_topdir}/SOURCES/%{name}-%{version}

%_specdir %{_sourcedir}

This results in source paths like /some/where/SOURCES/openssh-7.2p2.

If you do this and you try to build the current Fedora 24 version of OpenSSH, what you will get is:

$ rpmbuild -bp openssh.spec
error: File /u/cks/src/RPM/SOURCES/openssh-0.10.2/pam_ssh_agent_auth-0.10.2.tar.bz2: No such file or directory

The direct problem is that the Fedora OpenSSH RPM essentially contains two different packages (OpenSSH itself and pam_ssh_agent_auth) and they have two completely separate versions. When rpmbuild goes to unpack the source for the latter, it uses the latter's version number of 0.10.2 as %{version} in the expansion of %{_sourcedir} instead of the version number of the main OpenSSH source package and everything goes off the rails. When I asked on the Fedora developers' IRC channel, no one had any suggestions for how to fix this and no one seemed to expect that there were any (for instance, no one knew of a magic equivalent of %{version} that meant 'the version of the source RPM, no really I mean it').

(At this point I will pause to mutter things about RPM's documentation or lack thereof, part of which is RPM's fault and part of which is the fault of distributions, who don't seem to document their distro-specific macros and so on very much. If RPM has good, current, accessible documentation it is not easy to find.)

The bigger problem is three-fold. First, I have no idea how to actually fix this, although I can probably work around it in various hack ways to work on this specific OpenSSH package (for example, I can temporarily take the %{version} out of my definition). That assumes that there's even a way to really fix this, which there may not be. Second, clearly my long standing RPM build configuration doesn't work completely reliably any more. This is the first RPM I've run across with problems (and I don't have a strong need to change the OpenSSH RPM at the moment), but there are probably others that are going to blow up on me in the future in some way.

Third and largest, I've clearly wound up in a situation where I'm basically setting up my RPM build environment based on superstition instead of actual knowledge. My settings may have had a sound, well researched basis ten years ago, but that was ten years ago, I haven't kept up with the changes in RPM build practices, and my RPM specfile knowledge has decayed. Using a superstition is sustainable when it's at least a common one, but it also seems like this is very much not and that I'm probably well outside how RPM building and package modification is now done.

Does it matter? I don't know. I could likely stumble on for quite a while before things fell totally apart, because these days I only build or rebuild a small handful of RPMs and they'll probably keep on working (and if it really matters, I have hack workarounds). I don't like knowing that my environment is superstition and partly broken, but I'm also lazy; learning enough to do it the reasonably right way would almost certainly be a bunch of work.

(Modern Fedora RPM building appears to involve special build systems and build environments. This is great if you're submitting a finished source RPM and want a binary package, but it doesn't really make it clear how to do the equivalent of modifying RPMs with quilt, for which you want the unpacked, already patched source in some area where you can look at the source, play around, and so on. I assume that there is some way to do this, and probably some documentation about it somewhere that I'm either overlooking or not understanding.)

PS: The pam_ssh_agent_auth (sub)package seems to be integrated into the OpenSSH source RPM because it uses OpenSSH source code as part of building itself. I think including it in this situation makes sense.

by cks at December 06, 2016 04:10 AM

December 03, 2016

Everything Sysadmin

Solaris being canned.

"Solaris being canned, at least 50% of teams to be RIF'd in short term. Hardware teams being told to cease development. There will be no Solaris 12, final release will be 11.4. Orders coming straight from Larry."

The crazy people that said that Oracle bought Sun just to shut it down and use the patents to sue Google are looking pretty non-crazy now. I guess once the lawsuit was lost (and boy was it expensive) the shutdown was inevitable.

Of course, not funding SPARC development so that it could stay competitive with Intel didn't help much either.

by Tom Limoncelli at December 03, 2016 10:36 PM

Vincent Bernat

Build-time dependency patching for Android

This post shows how to patch an external dependency for an Android project at build-time with Gradle. This leverages the Transform API and Javassist, a Java bytecode manipulation tool.

buildscript {
    dependencies {
        classpath ''
        classpath ''
        classpath 'org.javassist:javassist:3.21.+'
        classpath 'commons-io:commons-io:2.4'

Disclaimer: I am not a seasoned Android programmer, so take this with a grain of salt.


This section adds some context to the example. Feel free to skip it.

Dashkiosk is an application to manage dashboards on many displays. It provides an Android application you can install on one of those cheap Android sticks. Under the table, the application is an embedded webview backed by the Crosswalk Project web runtime which brings an up-to-date web engine, even for older versions of Android1.

Recently, a security vulnerability has been spotted in how invalid certificates were handled. When a certificate cannot be verified, the webview defers the decision to the host application by calling the onReceivedSslError() method:

Notify the host application that an SSL error occurred while loading a resource. The host application must call either callback.onReceiveValue(true) or callback.onReceiveValue(false). Note that the decision may be retained for use in response to future SSL errors. The default behavior is to pop up a dialog.

The default behavior is specific to Crosswalk webview: the Android builtin one just cancels the load. Unfortunately, the fix applied by Crosswalk is different and, as a side effect, the onReceivedSslError() method is not invoked anymore2.

Dashkiosk comes with an option to ignore TLS errors3. The mentioned security fix breaks this feature. The following example will demonstrate how to patch Crosswalk to recover the previous behavior4.

Simple method replacement§

Let’s replace the shouldDenyRequest() method from the org.xwalk.core.internal.SslUtil class with this version:

// In SslUtil class
public static boolean shouldDenyRequest(int error) {
    return false;

Transform registration§

Gradle Transform API enables the manipulation of compiled class files before they are converted to DEX files. To declare a transform and register it, include the following code in your build.gradle:

import org.gradle.api.logging.Logger

class PatchXWalkTransform extends Transform {
    Logger logger = null;

    public PatchXWalkTransform(Logger logger) {
        this.logger = logger

    String getName() {
        return "PatchXWalk"

    Set<QualifiedContent.ContentType> getInputTypes() {
        return Collections.singleton(QualifiedContent.DefaultContentType.CLASSES)

    Set<QualifiedContent.Scope> getScopes() {
        return Collections.singleton(QualifiedContent.Scope.EXTERNAL_LIBRARIES)

    boolean isIncremental() {
        return true

    void transform(Context context,
                   Collection<TransformInput> inputs,
                   Collection<TransformInput> referencedInputs,
                   TransformOutputProvider outputProvider,
                   boolean isIncremental) throws IOException, TransformException, InterruptedException {
        // We should do something here

// Register the transform
android.registerTransform(new PatchXWalkTransform(logger))

The getInputTypes() method should return the set of types of data consumed by the transform. In our case, we want to transform classes. Another possibility is to transform resources.

The getScopes() method should return a set of scopes for the transform. In our case, we are only interested by the external libraries. It’s also possible to transform our own classes.

The isIncremental() method returns true because we support incremental builds.

The transform() method is expected to take all the provided inputs and copy them (with or without modifications) to the location supplied by the output provider. We didn’t implement this method yet. This causes the removal of all external dependencies from the application.

Noop transform§

To keep all external dependencies unmodified, we must copy them:

void transform(Context context,
               Collection<TransformInput> inputs,
               Collection<TransformInput> referencedInputs,
               TransformOutputProvider outputProvider,
               boolean isIncremental) throws IOException, TransformException, InterruptedException {
    inputs.each {
        it.jarInputs.each {
            def jarName =
            def src = it.getFile()
            def dest = outputProvider.getContentLocation(jarName, 
                                                         it.contentTypes, it.scopes,
            def status = it.getStatus()
            if (status == Status.REMOVED) { // ❶
      "Remove ${src}")
            } else if (!isIncremental || status != Status.NOTCHANGED) { // ❷
      "Copy ${src}")
                FileUtils.copyFile(src, dest)

We also need two additional imports:


Since we are handling external dependencies, we only have to manage JAR files. Therefore, we only iterate on jarInputs and not on directoryInputs. There are two cases when handling incremental build: either the file has been removed (❶) or it has been modified (❷). In all other cases, we can safely assume the file is already correctly copied.

JAR patching§

When the external dependency is the Crosswalk JAR file, we also need to modify it. Here is the first part of the code (replacing ❷):

if ("${src}" ==~ ".*/org.xwalk/xwalk_core.*/classes.jar") {
    def pool = new ClassPool()
    def ctc = pool.get('org.xwalk.core.internal.SslUtil') // ❸

    def ctm = ctc.getDeclaredMethod('shouldDenyRequest')
    ctc.removeMethod(ctm) // ❹

public static boolean shouldDenyRequest(int error) {
    return false;
""", ctc)) // ❺

    def sslUtilBytecode = ctc.toBytecode() // ❻

    // Write back the JAR file
    // …
} else {"Copy ${src}")
    FileUtils.copyFile(src, dest)

We also need the following additional imports to use Javassist:

import javassist.ClassPath
import javassist.ClassPool
import javassist.CtNewMethod

Once we have located the JAR file we want to modify, we add it to our classpath and retrieve the class we are interested in (❸). We locate the appropriate method and delete it (❹). Then, we add our custom method using the same name (❺). The whole operation is done in memory. We retrieve the bytecode of the modified class in ❻.

The remaining step is to rebuild the JAR file:

def input = new JarFile(src)
def output = new JarOutputStream(new FileOutputStream(dest))

// ❼
input.entries().each {
    if (!it.getName().equals("org/xwalk/core/internal/SslUtil.class")) {
        def s = input.getInputStream(it)
        output.putNextEntry(new JarEntry(it.getName()))
        IOUtils.copy(s, output)

// ❽
output.putNextEntry(new JarEntry("org/xwalk/core/internal/SslUtil.class"))


We need the following additional imports:

import java.util.jar.JarEntry
import java.util.jar.JarFile
import java.util.jar.JarOutputStream

There are two steps. In ❼, all classes are copied to the new JAR, except the SslUtil class. In ❽, the modified bytecode for SslUtil is added to the JAR.

That’s all! You can view the complete example on GitHub.

More complex method replacement§

In the above example, the new method doesn’t use any external dependency. Let’s suppose we also want to replace the sslErrorFromNetErrorCode() method from the same class with the following one:


// In SslUtil class
public static SslError sslErrorFromNetErrorCode(int error,
                                                SslCertificate cert,
                                                String url) {
    switch(error) {
            return new SslError(SslError.SSL_IDMISMATCH, cert, url);
        case NetError.ERR_CERT_DATE_INVALID:
            return new SslError(SslError.SSL_DATE_INVALID, cert, url);
            return new SslError(SslError.SSL_UNTRUSTED, cert, url);
    return new SslError(SslError.SSL_INVALID, cert, url);

The major difference with the previous example is that we need to import some additional classes.

Android SDK import§

The classes from the Android SDK are not part of the external dependencies. They need to be imported separately. The full path of the JAR file is:

androidJar = "${android.getSdkDirectory().getAbsolutePath()}/platforms/" +

We need to load it before adding the new method into SslUtil class:

def pool = new ClassPool()
def ctc = pool.get('org.xwalk.core.internal.SslUtil')
def ctm = ctc.getDeclaredMethod('sslErrorFromNetErrorCode')
// …

External dependency import§

We must also import and therefore, we need to put the appropriate JAR in our classpath. The easiest way is to iterate through all the external dependencies and add them to the classpath.

def pool = new ClassPool()
inputs.each {
    it.jarInputs.each {
        def jarName =
        def src = it.getFile()
        def status = it.getStatus()
        if (status != Status.REMOVED) {
def ctc = pool.get('org.xwalk.core.internal.SslUtil')
def ctm = ctc.getDeclaredMethod('sslErrorFromNetErrorCode')
// Then, rebuild the JAR...

Happy hacking!

  1. Before Android 4.4, the webview was severely outdated. Starting from Android 5, the webview is shipped as a separate component with updates. Embedding Crosswalk is still convenient as you know exactly which version you can rely on. 

  2. I hope to have this fixed in later versions. 

  3. This may seem harmful and you are right. However, if you have an internal CA, it is currently not possible to provide its own trust store to a webview. Moreover, the system trust store is not used either. You also may want to use TLS for authentication only with client certificates, a feature supported by Dashkiosk

  4. Crosswalk being an opensource project, an alternative would have been to patch Crosswalk source code and recompile it. However, Crosswalk embeds Chromium and recompiling the whole stuff consumes a lot of resources. 

by Vincent Bernat at December 03, 2016 10:20 PM

December 01, 2016

Errata Security

Electoral college should ignore Lessig

Reading this exchange between law profs disappoints me. [1] [2] [3] [4] [5]

The decision Bush v Gore cites the same principle as Lessig, that our system is based on "one person one vote". But it uses that argument to explain why votes should not be changed once they are cast:
Having once granted the right to vote on equal terms, the State may not, by later arbitrary and disparate treatment, value one person's vote over that of another.
Lessig cites the principle of "one person one vote", but in a new and novel way. He applies in an arbitrary way that devalues some of the votes that have already been cast. Specifically, he claims that votes cast for state electors should now be re-valued as direct votes for a candidate.

The United States isn't a union of people. It's a union of states. It says so right in the name. Compromises between the power of the states and power of the people have been with us for forever. That's why states get two Senators regardless of size, but Representatives to the House are assigned proportional to population. The Presidential election is expressly a related compromise, assigning the number of electors to a state equal to the number of Senators plus Representatives.

The Constitution doesn't even say electors should be chosen using a vote. It's up to the states to decide. All states have chosen election, but they could've demanded a wrestling match or juggling contest instead. The point is that the Constitution, historical papers, and 200 years of history rejects Lessig's idea that the President should be elected with a popular vote.

Moreover, this election shows the value of election by states. The tension nowadays is between big urban areas and rural areas. In the city, when workers lose their jobs due to immigration or trade, they can go down the street and get another job. In a rural area, when the factory shuts down, the town is devastated, and there are no other jobs to be had. The benefits of free trade are such that even Trump can't roll them back -- but as a nation we need to address the disproportionate impact changes have on rural communities. That rural communities can defend their interests is exactly why our Constitution is the way it is -- and why the President isn't chosen with a popular vote.

Hillary did not win the popular vote. No popular vote was held. Instead, we had state-by-state votes for electors. It's implausible that the per-candidate votes would have been the same had this been a popular vote. Candidates would have spent their time and money campaigning across the entire country instead of just battleground states. Voters would have different motivations on which candidates to choose and on whether they should abstain. There is nothing more clearly "disparate and arbitrary" treatment of votes than claiming a your vote for an elector  (or abstention) will now instead be treated as a national vote for the candidate.

Hillary got only 48% of the vote, what we call a plurality. Counting abstentions, that's only 26% of the vote. The rules of the Electoral College demands the winner get an absolute majority, meaning 50% even with abstentions, or almost double what Hillary got in votes. So among the arbitrary rules that Lessig has pulled out of his hat is that a plurality is now sufficient. Even though 74% of voters did not vote for her, Lessig uses the principle of "one person one vote" means she is the unambiguous choice of the people.

Even if you accept all this, there is still the problem that our election system isn't accurate. As Bush v Gore noted, around 2% of ballots nationwide didn't clearly show a choice between a presidential candidate. Others have pointed to weather in different parts of the country as having a significant impact on voter turnout. In science, we call this a measurement error. It means that any vote within 2% is scientifically a tie. That's more than the difference between Hillary and Trump. Yes, elections must still choose a winner despite a tie. However, an Electoral College evaluating the "sense of the people" (as Lessig cites Federalist #68) is bound by no such limitation. That they see no clear winner among the popular vote is best view to take -- not that Hillary won some sort of mandate.

My point isn't to show that Lessig is wrong so much to show that his argument is arbitrary. Had the positions been reversed, with Hillary getting the electoral vote and Trump the popular, Lessig could cite the same principle of "one person one vote" and the same "Federalist #68" in order to demonstrate why the Electoral College should still choose Hillary. In other words, Lessig would argue that the principle means (as in Bush v Gore) that Hillary's electors not devalue the votes cast for them by treating them as popular vote. Lessig would argue that since Trump didn't get a statistically significant absolute majority, then there was no clear "sense of the people".

America is in danger of populism, which ravages our institutions that make our country prosperous, stable, and "great". Trump is populist on the right, but Lessig is a populist on the left. Lessig ran for the presidency on the left on a platform no less populist than Trump's. This current piece, demanding we follow arbitrary rules to get desired results, is no less an attack on the institution of the "Rule of Law" and "Equal Protection" than Trump's attacks.

What "should" the Electoral College do? Whatever the heck they want. I would point out that Federalist #68 does warn about the influence of "foreign powers" and of men using the "little arts of popularity" to gain the Presidency. This matches Trump accurately. I would hope that at least some Trump-electors consider this and change their votes. Historically, that we haven't seen more electors change their votes seems to be a bit of a problem.

by Robert Graham ( at December 01, 2016 07:37 PM

Anton Chuvakin - Security Warrior

Monthly Blog Round-Up – November 2016

Here is my next monthly "Security Warrior" blog round-up of top 5 popular posts/topics this month:
  1. “New SIEM Whitepaper on Use Cases In-Depth OUT!” (dated 2010) presents a whitepaper on select SIEM use cases described in depth with rules and reports [using now-defunct SIEM product]; also see this SIEM use case in depth and this for a more current list of popular SIEM use cases. Finally, see our 2016 research on developing security monitoring use cases here!
  2. Why No Open Source SIEM, EVER?” contains some of my SIEM thinking from 2009. Is it relevant now? You be the judge.  Succeeding with SIEM requires a lot of work, whether you paid for the software, or not. BTW, this post has an amazing “staying power” that is hard to explain – I suspect it has to do with people wanting “free stuff” and googling for “open source SIEM” … 
  3. Simple Log Review Checklist Released!” is often at the top of this list – this aging checklist is still a very useful tool for many people. “On Free Log Management Tools” (also aged a bit by now) is a companion to the checklist (updated version)
  4. My classic PCI DSS Log Review series is always popular! The series of 18 posts cover a comprehensive log review approach (OK for PCI DSS 3+ as well), useful for building log review processes and procedures, whether regulatory or not. It is also described in more detail in our Log Management book and mentioned in our PCI book (now in its 4th edition!)
  5. “SIEM Resourcing or How Much the Friggin’ Thing Would REALLY Cost Me?” is a quick framework for assessing the costs of a SIEM project (well, a program, really) at an organization (much more details on this here in this paper).
In addition, I’d like to draw your attention to a few recent posts from my Gartner blog [which, BTW, now has about 5X of the traffic of this blog]: 
Current research on security analytics and UBA / UEBA:
Recent research on deception:
Past research on SOC:
Miscellaneous fun posts:

(see all my published Gartner research here)
Also see my past monthly and annual “Top Popular Blog Posts” – 2007, 2008, 2009, 2010, 2011, 2012, 2013, 2014, 2015.

Disclaimer: most content at SecurityWarrior blog was written before I joined Gartner on Aug 1, 2011 and is solely my personal view at the time of writing. For my current security blogging, go here.

Previous post in this endless series:

by Anton Chuvakin ( at December 01, 2016 05:50 PM

November 30, 2016

Carl Chenet

My Free Software activities in November 2016

My Monthly report for Novembre 2016 gives an extended list of what were my Free Software related activities during this month.

Personal projects:

Journal du hacker:

The Journal du hacker is a frenck-speaking Hacker News-like website dedicated to the french-speaking Free and Open source Software community.


That’s all folks! See you next month!

by Carl Chenet at November 30, 2016 11:00 PM

November 28, 2016

Michael Biven

It Was Never About Ops

For a while I’ve been thinking about Susan J. Fowler’s Ops Identitfy Crisis post. Bits I agreed with and some I did not.

My original reaction to the post was pretty pragmatic. I had concerns (and still do) about adding more responsibilities onto software engineers (SWE). It’s already fairly common to have them responsible for QA, database administration and security tasks but now ops issues are being considered as well.

I suspect there is an as of yet unmeasured negative impact to the productivity of teams that keep expanding the role of SWE. You end up deprioritizing the operations and systems related concerns, because new features and bug fixes will always win out when scheduling tasks.

Over time I’ve refined my reaction to this. The traditional operations role is the hole that you’ve been stuck in and engineering is how you get out of that hole. It’s not that you don’t need Ops teams or engineers anymore. It’s simply that you’re doing it wrong.

It was never solely about operations. There’s always been an implied hint of manual effort for most Ops teams. We’ve seen a quick return from having SWE handle traditional ops tasks, but that doesn’t mean that the role won’t be needed anymore. Previously we’ve been able to add people to ops to continue to scale with the growth of the company, but those days are gone for most. What needs to change is how we approach the work and the skills needed to do the job.

When you’re unable to scale to meet the demands you’ll end up feeling stuck in a reactive and constantly interruptible mode of working. This can then make operations feel more like a burden rather than a benefit. This way of thinking is part of the reason why I think many of the tools and services created by ops teams are not thought of as actual products.

Ever since we got an API to interact with the public cloud and then later the private cloud we’ve been trying to redefine the role of ops. As the ecosystem of tools has grown and changed over time we’ve continued to refine that role. While thinking on the impact Fowler’s post I know that I agree with her that the skills needed are different from they were eight years ago, but the need for the role hasn’t decreased. Instead it’s grown to match the growth of the products it has been supporting. This got me thinking about how I’ve been working during those eight years and looking back it’s easy to see what worked and what didn’t. These are the bits that worked for me.

First don’t call it ops anymore. Sometimes names do matter. By continuing to use “Ops” in the team name or position we continue to hold onto that reactive mindset.

Make scalability your main priority and start building the products and services that become the figurative ladder to get you out of the hole you’re in. I believe you can meet that by focusing on three things: Reliability, Availability, and Serviceability.

For anything to scale it first needs to be reliable and available if people are going to use it. To be able to meet the demands of the growth you’re seeing the products need to be serviceable. You must be able to safely make changes to them in a controlled and observable fashion.

Every product built out of these efforts should be considered a first class citizen. The public facing services and the internal ones should be considered equals. If your main priority is scaling to meet the demands of your growth, then there should be no difference in how you design, build, maintain, or consider anything no matter where it is in the stack.

Focus on scaling your engineers and and making them force multipliers for the company. There is a cognitive load placed on individuals, teams, and organizations for the work they do. Make sure to consider this in the same way we think of the load capacity of a system. At a time where we’re starting to see companies break their products into smaller more manageable chunks (microservices), we’re close to doing the exact opposite for our people and turning the skills needed to do the work into one big monolith.

If you’ve ever experienced yourself or have seen any of your peers go through burnout what do you think is going to happen as we continue to pile on additional responsibilities?

The growth we’re seeing is the result of businesses starting to run into web scale problems.

Web scale describes the tendency of modern sites – especially social ones – to grow at (far-)greater-than-linear rates. Tools that claim to be “web scale” are (I hope) claiming to handle rapid growth efficiently and not have bottlenecks that require rearchitecting at critical moments. The implication for “web scale” operations engineers is that we have to understand this non-linear network effect. Network effects have a profound effect on architecture, tool choice, system design, and especially capacity planning.

Jacob Kaplan-Moss

The real problem I think we’ve always been facing is making sure you have the people you need to do the job. Before we hit web scale issues we could usually keep up by having people pull all nighters, working through weekends or if you’re lucky hiring more. The ability for hard work to make up for any short comings in planning or preparations simply can no longer keep up. The problem has never been with the technology or the challenges you’re facing. It’s always been about having the right people.

In short you can …

  1. Expect services to grow at a non-linear rate.
  2. To be able to keep up with this growth you’ll need to scale both your people and your products.
  3. Scale your people by giving them the time and space to focus on scaling your products.
  4. Scale your products by focusing on Reliability, Availability, and Serviceability.

To think that new tools or services will be the only (or main) answer to the challenges you’re facing brought on from growth is a mistake. You will always need people to understand what is happening and then design, implement, and maintain your solutions. These new tools and services should increase the impact of each member of your team, but it is a mistake to think it will replace a role.

November 28, 2016 11:11 PM

November 27, 2016

Toolsmith - GSE Edition: Scapy vs CozyDuke

In continuation of observations from my GIAC Security Expert re-certification process, I'll focus here on a GCIA-centric topic: Scapy. Scapy is essential to the packet analyst skill set on so many levels. For your convenience, the Packetrix VM comes preconfigured with Scapy and Snort, so you're ready to go out of the gate if you'd like to follow along for a quick introduction.
Scapy is "a powerful interactive packet manipulation program. It is able to forge or decode packets of a wide number of protocols, send them on the wire, capture them, match requests and replies, and much more." This includes the ability to handle most tasks such as scanning, tracerouting, probing, unit tests, attacks or network discovery, thus replacing functionality expected from hping, 85% of nmap, arpspoof, tcpdump, and others.
If you'd really like to dig in, grab TJ O'Connor's  Violent Python: A Cookbook for Hackers, Forensic Analysts, Penetration Testers and Security Engineers (you should already have it), as first discussed here in January 2013. TJ loves him some Scapy: Detecting and Responding to Data Link Layer Attacks is another reference. :-)
You can also familiarize yourself with Scapy's syntax in short order with the SANS Scapy Cheat Sheet as well.
Judy Novak's SANS GIAC Certified Intrusion Analyst Day 5 content offers a nice set of walk-throughs using Scapy, and given that it is copyrighted and private material, I won't share them here, but will follow a similar path so you have something to play along with at home. We'll use a real-world APT scenario given recent and unprecedented Russian meddling in American politics. According to SC Magazine, "Russian government hackers apparently broke into the Democratic National Committee (DNC) computer systems" in infiltrations believed to be the work of two different Russian groups, namely Cozy Bear/ CozyDuke/APT 29 and Fancy Bear/Sofacy/APT 28, working separately. As is often the case, ironically and consistently, one the best overviews of CozyDuke behaviors comes via Kaspersky's Securelist. This article is cited as the reference in a number of Emerging Threats Snort/Suricata rules for CozyDuke. Among them, 2020962 - ET TROJAN CozyDuke APT HTTP Checkin, found in the trojan.rules file, serves as a fine exemplar.
I took serious liberties with the principles of these rules and oversimplified things significantly with a rule as added to my local.rules file on my Packetrix VM. I then took a few quick steps with Scapy to ensure that the rule would fire as expected. Of the IOCs derived from the Securelist article, we know a few things that, if built into a PCAP with Scapy, should cause the rule to fire when the PCAP is read via Snort.
  1. CozyDuke client to C2 calls were over HTTP
  2. Requests for C2 often included a .php reference, URLs included the likes of /ajax/index.php
  3. was one of the C2 IPs, can be used as an example destination IP address
The resulting simpleton Snort rule appears in Figure 1.

Figure 1: Simple rule
To quickly craft a PCAP to trigger this rule, at a bash prompt, I ran scapy, followed by syn = IP(src="", dst="")/TCP(sport=1337, dport=80, flags="S")/"GET /ajax/index.php HTTP/1.1", then wrote the results out with wrpcap("/tmp/CozyDukeC2GET.pcap", syn), as seen in Figure 2.

Figure 2: Simple Scapy
Then a quick run of the resulting file through Snort with snort -A console -q -K none -r /tmp/CozyDukeC2GET.pcap -c ../etc/snort.conf, and we have a hit as seen in Figure 3.

Figure 3: Simple result

Scapy is ridiculously powerful and is given no justice here, hopefully just enough information to entice you to explore further. With just the principles established here, you can see the likes of options to craft and manipulate with ls(TCP) and ls(IP).
Figure 4: ls()

If you're studying for the likes of GCIA or just looking to improve your understanding of TCP/IP and NSM, no better way to do so than with Scapy.
Cheers...until next time.

by Russ McRee ( at November 27, 2016 07:04 PM

November 24, 2016

Cryptography Engineering

The limitations of Android N Encryption

Over the past few years pixelphonewe’ve heard more about smartphone encryption than, quite frankly, most of us expected to hear in a lifetime. We learned that proper encryption can slow down even sophisticated decryption attempts if done correctly. We’ve also learned that incorrect implementations can undo most of that security.

In other words, phone encryption is an area where details matter. For the past few weeks I’ve been looking a bit at Android Nougat’s new file-based encryption to see how well they’ve addressed some of those details in their latest release. The answer, unfortunately, is that there’s still lots of work to do. In this post I’m going to talk about a bit of that.

(As an aside: the inspiration for this post comes from Grugq, who has been loudly and angrily trying to work through these kinks to develop a secure Android phone. So credit where credit is due.)

Background: file and disk encryption 

Disk encryption is much older than smartphones. Indeed, early encrypting filesystems date back at least to the early 1990s and proprietary implementations may go back before that. Even in the relatively new area of PCs operating systems, disk encryption has been a built-in feature since the early 2000s.

The typical PC disk encryption system operates as follows. At boot time you enter a password. This is fed through a key derivation function to derive a cryptographic key. If a hardware co-processor is available (e.g., a TPM), your key is further strengthened by “tangling” it with some secrets stored in the hardware. This helps to lock encryption to a particular device.

The actual encryption can be done in one of two different ways:

  1. Full Disk Encryption (FDE) systems (like TruecryptBitLocker and FileVault) encrypt disks at the level of disk sectors. This is an all-or-nothing approach, since the encryption drivers won’t necessarily have any idea what files those sectors represent. At the same time, FDE is popular — mainly because it’s extremely easy to implement.
  2. File-based Encryption (FBE) systems (like EncFS and eCryptFS) encrypt individual files. This approach requires changes to the filesystem itself, but has the benefit of allowing fine grained access controls where individual files are encrypted using different keys.

Most commercial PC disk encryption software has historically opted to use the full-disk encryption (FDE) approach. Mostly this is just a matter of expediency: FDE is just significantly easier to implement. But philosophically, it also reflects a particular view of what disk encryption was meant to accomplish.

In this view, encryption is an all-or-nothing proposition. Your machine is either on or off; accessible or inaccessible. As long as you make sure to have your laptop stolen only when it’s off, disk encryption will keep you perfectly safe.

So what does this have to do with Android?

Android’s early attempts at adding encryption to their phones followed the standard PC full-disk encryption paradigm. Beginning in Android 4.4 (Kitkat) through Android 6.0 (Marshmallow), Android systems shipped with a kernel device mapper called dm-crypt designed to encrypt disks at the sector level. This represented a quick and dirty way to bring encryption to Android phones, and it made sense — if you believe that phones are just very tiny PCs.

The problem is that smartphones are not PCs.

The major difference is that smartphone users are never encouraged to shut down their device. In practice this means that — after you enter a passcode once after boot — normal users spend their whole day walking around with all their cryptographic keys in RAM. Since phone batteries live for a day or more (a long time compared to laptops) encryption doesn’t really offer much to protect you against an attacker who gets their hands on your phone during this time.

Of course, users do lock their smartphones. In principle, a clever implementation could evict sensitive cryptographic keys from RAM when the device locks, then re-derive them the next time the user logs in. Unfortunately,  Android doesn’t do this — for the very simple reason that Android users want their phones to actually work. Without cryptographic keys in RAM, an FDE system loses access to everything on the storage drive. In practice this turns it into a brick.

For this very excellent reason, once you boot an Android FDE phone it will never evict its cryptographic keys from RAM. And this is not good.

So what’s the alternative?

Android is not the only game in town when it comes to phone encryption. Apple, for its part, also gave this problem a lot of thought and came to a subtly different solution.

Starting with iOS 4, Apple included a “data protection” feature to encrypt all data stored a device. But unlike Android, Apple doesn’t use the full-disk encryption paradigm. Instead, they employ a file-based encryption approach that individually encrypts each file on the device.

In the Apple system, the contents of each file is encrypted under a unique per-file key (metadata is encrypted separately). The file key is in turn encrypted with one of several “class keys” that are derived from the user passcode and some hardware secrets embedded in the processor.

appleiOS data encryption. Source: iOS Security Guide.

The main advantage of the Apple approach is that instead of a single FDE key to rule them all, Apple can implement fine-grained access control for individual files. To enable this, iOS provides an API developers can use to specify which class key to use in encrypting any given file. The available “protection classes” include:

  • Complete protection. Files encrypted with this class key can only be accessed when the device is powered up and unlocked. To ensure this, the class key is evicted from RAM a few seconds after the device locks.
  • Protected Until First User Authentication. Files encrypted with this class key are protected until the user first logs in (after a reboot), and the key remains in memory.
  • No protection. These files are accessible even when the device has been rebooted, and the user has not yet logged in.

By giving developers the option to individually protect different files, Apple made it possible to build applications that can work while the device is locked, while providing strong protection for files containing sensitive data.

Apple even created a fourth option for apps that simply need to create new encrypted files when the class key has been evicted from RAM. This class uses public key encryption to write new files. This is why you can safely take pictures even when your device is locked.

Apple’s approach isn’t perfect. What it is, however, is the obvious result of a long and careful thought process. All of which raises the following question…

Why the hell didn’t Android do this as well?

The short answer is Android is trying to. Sort of. Let me explain.

As of Android 7.0 (Nougat), Google has moved away from full-disk encryption as the primary mechanism for protecting data at rest. If you set a passcode on your device, Android N systems can be configured to support a more Apple-like approach that uses file encryption. So far so good.

The new system is called Direct Boot, so named because it addresses what Google obviously saw as fatal problem with Android FDE — namely, that FDE-protected phones are useless bricks following a reboot. The main advantage of the new model is that it allows phones to access some data even before you enter the passcode. This is enabled by providing developers with two separate “encryption contexts”:

  • Credential encrypted storage. Files in this area are encrypted under the user’s passcode, and won’t be available until the user enters their passcode (once).
  • Device encrypted storage. These files are not encrypted under the user’s passcode (though they may be encrypted using hardware secrets). Thus they are available after boot, even before the user enters a passcode.

Direct Boot even provides separate encryption contexts for different users on the phone — something I’m not quite sure what to do with. But sure, why not?

If Android is making all these changes, what’s the problem?

One thing you might have noticed is that where Apple had four categories of protection, Android N only has two. And it’s the two missing categories that cause the problems. These are the “complete protection” categories that allow the user to lock their device following first user authentication — and evict the keys from memory.

Of course, you might argue that Android could provide this by forcing application developers to switch back to “device encrypted storage” following a device lock. The problem with this idea is twofold. First, Android documentation and sample code is explicit that this isn’t how things work:


Moreover, a quick read of the documentation shows that even if you wanted to, there is no unambiguous way for Android to tell applications when the system has been re-locked. If keys are evicted when the device is locked, applications will unexpectedly find their file accesses returning errors. Even system applications tend to do badly when this happens.

And of course, this assumes that Android N will even try to evict keys when you lock the device. Here’s how the current filesystem encryption code handles locks:


While the above is bad, it’s important to stress that the real problem here is not really in the cryptography. The problem is that since Google is not giving developers proper guidance, the company may be locking Android into years of insecurity. Without (even a half-baked) solution to define a “complete” protection class, Android app developers can’t build their apps correctly to support the idea that devices can lock. Even if Android O gets around to implementing key eviction, the existing legacy app base won’t be able to handle it — since this will break a million apps that have implemented their security according to Android’s current recommendations.

In short: this is a thing you get right from the start, or you don’t do at all. It looks like — for the moment — Android isn’t getting it right.

Are keys that easy to steal?

Of course it’s reasonable to ask whether it’s having keys in RAM is that big of concern in the first place. Can these keys actually be accessed?

The answer to that question is a bit complicated. First, if you’re up against somebody with a hardware lab and forensic expertise, the answer is almost certainly “yes”. Once you’ve entered your passcode and derived the keys, they aren’t stored in some magically secure part of the phone. People with the ability to access RAM or the bus lines of the device can potentially nick them.

But that’s a lot of work. From a software perspective, it’s even worse. A software attack would require a way to get past the phone’s lockscreen in order to get running code on the device. In older (pre-N) versions of Android the attacker might need to then escalate privileges to get access to Kernel memory. Remarkably, Android N doesn’t even store its disk keys in the Kernel — instead they’re held by the “vold” daemon, which runs as user “root” in userspace. This doesn’t make exploits trivial, but it certainly isn’t the best way to handle things.

Of course, all of this is mostly irrelevant. The main point is that if the keys are loaded you don’t need to steal them. If you have a way to get past the lockscreen, you can just access files on the disk.

What about hardware?

Although a bit of a tangent, it’s worth noting that many high-end Android phones use some sort of trusted hardware to enable encryption. The most common approach is to use a trusted execution environment (TEE) running with ARM TrustZone.

This definitely solves a problem. Unfortunately it’s not quite the same problem as discussed above. ARM TrustZone — when it works correctly, which is not guaranteed — forces attackers to derive their encryption keys on the device itself, which should make offline dictionary attacks on the password much harder. In some cases, this hardware can be used to cache the keys and reveal them only when you input a biometric such as a fingerprint.

The problem here is that in Android N, this only helps you at the time the keys are being initially derived. Once that happens (i.e., following your first login), the hardware doesn’t appear to do much. The resulting derived keys seem to live forever in normal userspace RAM. While it’s possible that specific phones (e.g., Google’s Pixel, or Samsung devices) implement additional countermeasures, on stock Android N phones hardware doesn’t save you.

So what does it all mean?

How you feel about this depends on whether you’re a “glass half full” or “glass half empty” kind of person.

If you’re an optimistic type, you’ll point out that Android is clearly moving in the right direction. And while there’s a lot of work still to be done, even a half-baked implementation of file-based implementation is better than the last generation of dumb FDE Android encryption. Also: you probably also think clowns are nice.

On the other hand, you might notice that this is a pretty goddamn low standard. In other words, in 2016 Android is still struggling to deploy encryption that achieves (lock screen) security that Apple figured out six years ago. And they’re not even getting it right. That doesn’t bode well for the long term security of Android users.

And that’s a shame, because as many have pointed out, the users who rely on Android phones are disproportionately poorer and more at-risk. By treating encryption as a relatively low priority, Google is basically telling these people that they shouldn’t get the same protections as other users. This may keep the FBI off Google’s backs, but in the long term it’s bad judgement on Google’s part.

by Matthew Green at November 24, 2016 02:09 PM

Overflow the Investigatory Powers Bill!

I read an article on The Register regarding the Investigatory Powers Bill. The part were ISP's are forced to save their customers browsing history for a year is the most horryfing part, just as that whole bill. Let's hope the political process and organizations like the Open Rights Group and the EFF have enough lobbying power to change people's minds. If that fails, then we can all try to overflow the logging. Just as some people put keywords in their mail signatures to trigger automatic filters and generate noise, we should all generate as much data and noise as possible. This way the information they do gather will not be usefull, it will take too much time, storage and effort to process it and thus the project will fail. 2 years ago I wrote a small Python script which browser the web for you, all the time. Running that on one or two Raspberry Pi's or other small low power computers 24/7 will generate a lot of noise in the logging and filtering.

November 24, 2016 12:00 AM

November 23, 2016

Anton Chuvakin - Security Warrior

November 22, 2016

Everything Sysadmin

TPOSANA on Amazon Lightning Deals today only!

Upgrade to the new edition!

Amazon is having a time-limited sale today that includes the new edition of The Practice of System and Network Administration. This is a good way to get the new edition at a deep discount:

Follow this link. It should be somewhere on this page.

The deal is only available today (Tuesday, 22-Nov) from 12:05 PST to 6:05 PST.

by Tom Limoncelli at November 22, 2016 01:54 PM

November 21, 2016

cf-deploy v2 released

github-logo I took some time this weekend to release an update for cf-deploy. You have now the option to override the configuration hardcoded in the script by means of environment variables. Check the README for the details.

If you don’t know what cf-deploy is, that’s fair ;-) In two words, it’s a Makefile and a Perl front-end to it that makes it easier to pack together a set of files for a configuration management tools and send them to a distribution server. Designed with git and CFEngine in mind, it’s general enough that you can easily adapt it to any version control system and any configuration management tool by simply modifying the Makefile. If it sounds interesting, you are welcome to read Git repository and deployment procedures for CFEngine policies on this same blog. Enjoy!


Tagged: cfengine, Configuration management, DevOps, git, Github, make, Sysadmin

by bronto at November 21, 2016 03:00 PM

Steve Kemp's Blog

Detecting fraudulent signups?

I run a couple of different sites that allow users to sign-up and use various services. In each of these sites I have some minimal rules in place to detect bad signups, but these are a little ad hoc, because the nature of "badness" varies on a per-site basis.

I've worked in a couple of places where there are in-house tests of bad signups, and these usually boil down to some naive, and overly-broad, rules:

  • Does the phone numbers' (international) prefix match the country of the user?
  • Does the postal address supplied even exist?

Some places penalise users based upon location too:

  • Does the IP address the user submitted from come from TOR?
  • Does the geo-IP country match the users' stated location?
  • Is the email address provided by a "free" provider?

At the moment I've got a simple HTTP-server which receives a JSON post of a new users' details, and returns "200 OK" or "403 Forbidden" based on some very very simple critereon. This is modeled on the spam detection service for blog-comments server I use - something that is itself becoming less useful over time. (Perhaps time to kill that? A decision for another day.)

Unfortunately this whole approach is very reactive, as it takes human eyeballs to detect new classes of problems. Code can't guess in advance that it should block usernames which could collide with official ones, for example allowing a username of "admin", "help", or "support".

I'm certain that these systems have been written a thousand times, as I've seen at least five such systems, and they're all very similar. The biggest flaw in all these systems is that they try to classify users in advance of them doing anything. We're trying to say "Block users who will use stolen credit cards", or "Block users who'll submit spam", by correlating that behaviour with other things. In an ideal world you'd judge users only by the actions they take, not how they signed up. And yet .. it is better than nothing.

For the moment I'm continuing to try to make the best of things, at least by centralising the rules for myself I cut down on duplicate code. I'll pretend I'm being cool, modern, and sexy, and call this a micro-service! (Ignore the lack of containers for the moment!)

November 21, 2016 05:37 AM

November 20, 2016

Sarah Allen

personal online security

This morning a few of us gathered for brunch, lured by pancakes and the promise of learning about security, by Aaron VonderHaar (@avh4). People shared experiences, as well as fears of what might happen in the future. EFF’s Danny O’Brien and others shared tips and perspectives on security and online safety.

A important thing to consider is that we need to take different precautions depending on what kinds of threats we expect. Generally Internet security has focused on malicious hackers and thieves. However, the threat model has changed. We also need to fear that our government might demand our information from 3rd parties which is less well protected than information that we hold personally. We need to be thoughtful of what companies we trust.

In addition to hackers and legal action, people who make political statements now have reason to fear false news stories as well as personal attacks that the Internet makes easier with doxxing and swatting.

Solidarity from social and community groups can help serve as protection, as well as developing a relationship (if possible) with local police and your representations. Undocumented residents and targeted groups have increased reason to fear our government, despite that our supreme court has historically upheld that US law extends even to those who are not citizens. Even if someone entered the country illegally, they still have the right to protection from discrimination and abuse. Even before this election, we have not not seen justice and police protection applied fairly and equally in the United States. Those of us with privilege have an opportunity and obligation to support those who ought to be protected by law.

Some useful links for more information:
* Feminist Frequency Online Safety Guide
* EFF Surveillance Self-Defense
* Signal Private Messaging by Whisper Systems, a mobile app with fully encrypted end-to-end communication.
* How to encrypt your entire life in less than an hour

The post personal online security appeared first on the evolving ultrasaurus.

by sarah at November 20, 2016 12:04 AM

November 18, 2016

The Geekess

Impact of bots on github communities

I’ve been digging into contributor statistics for various communities on github as part of my work on FOSS Heartbeat, a project to measure the health of open source communities.

It’s fascinating to see bots show up in the contributor statistics. For example, if you look at github users who comment on issues the Rust community, you’ll quickly notice two contributors who interact a lot:


bors is a bot that runs pull requests through the rust continuous integration test suite, and automatically merges the code into the master branch if it passes. bors responds to commands issued in pull request comments (of the form’@bors r+ [commit ID]’ by community members with permission to merge code into rust-lang/rust.

rust-highfive is a bot that recommends a reviewer based on the contents of the pull request. It then add a comment that tags the reviewer, who will get a github notification (and possibly an email, if they have that set up).

Both bots have been set up by the Rust community in order to make pull request review smoother. bors is designed to cut down the amount of time developers need to spend running the test suite on code that’s ready to be merged. rust-highfive is designed to make sure the right person is aware of pull requests that may need their experienced eye.

But just how effective are these github bots? Are they really helping the Rust community or are they just causing more noise?

Chances of a successful pull request

bors merged its first pull request on 2013-02-02. The year before bors was introduced, only 330 out of 503 pull requests were merged. The year after, 1574 out of 2311 pull requests were merged. So the Rust community had four times more pull requests to review.

Assuming that the tests bors used were some of the same tests rust developers were running manually, we would expect that pull requests would be rejected at about the same rate (or maybe rejected more, since the automatic CI system would catch more bugs).

To test that assumption, we turn to a statistics method called the Chi squared test. It helps answer the question, “Is there a difference in the success rates of two samples?” In our case, it helps us answer the question, “After bors was used, did the percentage of accepted pull requests change?”


It looks like there’s no statistical difference in the chances of getting a random pull request merged before or after bors started participating. That’s pretty good, considering the number of pull requests submitted quadrupled.

Now, what about rust-highfive? Since the bot is supposed to recommend pull request reviewers, we would hope that pull requests would have a higher chance of getting accepted. Let’s look at the chances of getting a pull request merged for the year before and the year after rust-highfive was introduced (2014-09-18).


So yes, it does seem like rust-highfive is effective at getting the right developer to notice a pull request they need to review and merge.

Impact on time a pull request is open

One of the hopes of a programmer who designs a bot is that it will cut down on the amount of time that the developer has to spend on simple repetitive tasks. A bot like bors is designed to run the CI suite automatically, leaving the developer more time to do other things, like review other pull requests. Maybe that means pull requests get merged faster?

To test the impact of bors on the amount of time a pull request is open, we turn to the Two-means hypothesis test. It tells you whether there’s a statistical difference between the means of two different data sets. In our case, we compare the length of time a pull request is open. The two populations are the pull requests a year before and a year after bors was introduced.


We would hope to see the average open time of a pull request go down after bors was introduced, but that’s not what the data shows. The graph shows the length of time actually increased, with an increase of 1.1 days.

What about rust-highfive? We would hope that a bot that recommends a reviewer would cause pull requests to get closed sooner.


The graph shows there’s no statistical evidence that rust-highfive made a difference in the length of time pull requests were open.

These results seemed odd to me, so I did a little bit of digging to generate a graph of the average time a pull request is open for each month:


The length of time pull requests are open has been increasing for most of the Rust project history. That explains why comparing pull request age before and after bors showed an increase in the wait time to get a pull request merged. The second line shows the point that rust-highfive was introduced, and we do see a decline in the wait time. Since the decrease is almost symmetrical with the increase the year before, the average was the same for the two years.


What can we conclude about github bots from all this statistics?

We can prove with 99% confidence that adding the bors bot to automatically merge changes after it passed the CI tests had no impact on the chances of a random pull request getting merged.

We can prove with 99% confidence that rust-highfive increases a Rust developer’s chances of getting code merged, by as much as 11.7%. The bot initially helped lower the amount of time developers had to wait for their pull requests to be merged, but something else changed in May 2015 that caused the wait time to increase again. I’ll note that Rust version 1.0 came out on May 2015. Rust developers may have been more cautious about accepting pull requests after the API was frozen or the volume of pull requests may have increased. It’s unclear without further study.

This is awesome, can I help?

If you’re interested in metrics analysis for your community, please leave a note in the comments or drop an email to my consulting business, Otter Tech. I could use some help identifying the github usernames for bots in other communities I’m studying:

This blog post is part of a series on open source community metrics analysis:

Part 1: Measuring the Impact of Negative Language on FOSS Participation

You can find the open source FOSS Heartbeat code and FOSS community metrics on github. Thank you to Mozilla, who is sponsoring this research!

by sarah at November 18, 2016 02:13 AM

November 17, 2016

Evaggelos Balaskas

enlarge your disk image aka windows extend volume

A visual guide on how to enlarge your windows disk image aka windows extend volume

I have a windows 10 qemu-kvm virtual machine for business purposes.
Every now and then, I have to resize it’s disk image!

This is my visual guide, so next time I will not waste any time figure this out, again!

Resize Disk image

The first step is to resize the disk image from the command line:

# ls -l win10.qcow2
-rw-r--r-- 1 root root 58861813760 Nov 17 10:04 win10.qcow2

# du -h win10.qcow2
55G win10.qcow2

#  qemu-img info win10.qcow2
image: win10.qcow2
file format: qcow2
virtual size: 55G (59055800320 bytes)
disk size: 55G
cluster_size: 65536
Format specific information:
    compat: 1.1
    lazy refcounts: false
    refcount bits: 16
    corrupt: false
#  qemu-img resize win10.qcow2 +10G
Image resized.
# qemu-img info win10.qcow2
image: win10.qcow2
file format: qcow2
virtual size: 65G (69793218560 bytes)
disk size: 55G
cluster_size: 65536
Format specific information:
    compat: 1.1
    lazy refcounts: false
    refcount bits: 16
    corrupt: false

Windows Problem - extend volume

Windows can not extend a volume if the free partition is not next to the “need-to-be” extened volume.


So we have to move the free partition next to C: drive

System Rescue Cd

Here comes system rescue cd !



with gparted you can move to the end of the virtual disk the ntfs recovery partition:









Computer Management - Disk Management

It’s time to extend our partition:







Tag(s): kvm, qemu, windows

November 17, 2016 10:45 AM

November 15, 2016

Sarah Allen

self-evident truths

“We hold these truths to be self evident…” are inspiring words. Encoding them into a legal framework has been a process. The Declaration of Independence included values of equality and the unalienable rights of life, liberty, and the pursuit of happiness.

Unalienable, adj. impossible to take away or give up
— Merriam-Webster

The authors of that declaration knew full well the dangers we face of those rights being taken away, and yet the first draft of the Constitution did not protect these rights. The first ten amendments were a good start, becoming known as the Bill of Rights. Over 200 years later, we still need to work to define and protect those rights which we believe should be extended to all people.

First they came for the Socialists, and I did not speak out— Because I was not a Socialist.  Then they came for the Trade Unionists, and I did not speak out— Because I was not a Trade Unionist.  Then they came for the Jews, and I did not speak out— Because I was not a Jew.  Then they came for me—and there was no one left to speak for me.

Many people are saying “it will be okay.” We need to step up and work to make that true. Things were not ok before the election.

It is hard to know what to do, but change is not a result of a single action. We need to think through what we may face, so we are prepared when we witness injustice. And we must put ourselves in situations and in company where we will learn and see and have the opportunity to act.

The post self-evident truths appeared first on the evolving ultrasaurus.

by sarah at November 15, 2016 04:08 PM

November 14, 2016

Everything Sysadmin

Stealing the Best Ideas from DevOps

Christine Hogan and I will be co-presenting a talk at Usenix LISA '16 entitled "Stealing the Best Ideas from DevOps: A Guide for Sysadmins without Developers". Full details are on the LISA website. The talk will cover a lot of the devops-y material from our newest book, the 3rd edition of TPOSANA. We'll be doing a book-signing shortly after the talk.

In addition, I'll also be teaching two half-day tutorials: "Personal Time Management: The Basics for Sysadmins That Are Overloaded" and "How to Not Get Paged: Managing On-Call to Reduce Outages".

Links to all of this here.

If you haven't registered for LISA yet, do it now!

by Tom Limoncelli at November 14, 2016 09:05 PM

Build a FreeBSD 11.0-release Openstack Image with bsd-cloudinit

We are going to prepare a FreeBSD image for Openstack deployment. We do this by creating a FreeBSD 11.0-RELEASE instance, installing it and converting it using bsd-cloudinit. We'll use the CloudVPS public Openstack cloud for this. We'll be using the Openstack command line tools, like nova, cinder and glance. A FreeBSD image with Cloud Init will automatically resize the disk to the size of the flavor and it will add your SSH key right at boot. You can use Cloud Config to execute a script at first boott, for example, to bootstrap your system into Puppet or Ansible. If you use Ansible to manage OpenStack instances you can integrate it without manually logging in or doing anything manually.

November 14, 2016 12:00 AM

November 08, 2016

Sean's IT Blog

Horizon 7.0 Part 11–Desktop Pools

If you have ever worked with SCCM, you may have used Collections to group desktops together for application deployments or patch management. Collections provide a way to group users and computers for organization and resource management.

Desktop pools are a similar concept in Horizon. They are a logical grouping of virtual machines that users can access, and these groupings control specific settings about the pool. This includes how the desktops are provisioned and named, protocols that are available for connectivity, and what physical infrastructure they are deployed on. 

Horizon has a few different types of desktop pools.  Each pool handles desktops in different ways, and they each have different purposes.  The type of pool that you select will be determined by a number of factors including the use case, the storage infrastructure and application requirements.

The type of desktop pools are:

  • Full Clone Pools – Each virtual desktop is a full virtual machine cloned from a template in vCenter.  The virtual machines require a desktop management tool for post-deployment management.  VMs are customized using existing Guest Customization Specifications. These desktops usually persist after the user logs out.
  • Linked Clone Pools – Each virtual desktop is based on a parent VM snapshot and shares its disk with the parent virtual machine.  Changes to the linked clone are written to a delta disk.  The virtual machines are managed by View Composer.   Linked Clone desktops can be Floating or Dedicated assignment, and they can be configured to be refreshed (or rolled back to a known good snapshot) or deleted on logoff.
  • Instant Clone Pools – Each virtual desktop is based on a parent VM snapshot. The snapshot is cloned to a VM that is deployed to each host, powered up, and then stunned. All guest VMs are then “forked” from this VM and quickly customized. Guest VMs share virtual disks and initial memory maps with the parent VMs.  VMs are managed by vCenter and a “next generation” Composer that is built into the Connection Servers. There are limitations to Instant Clone desktops including a cap of 2000 desktops per vCenter, no support for GPUs, and can only be used in floating assignment pools.
  • Manual Pools – The machines that make up the manual pool consist of virtual and/or physical machines that have had the View Agent installed.  These machines are not managed by Horizon.
  • Remote Desktop Session Host Pool – The machines that make up these pools are Windows Servers with the Remote Desktop Session Host Role installed.  They can be provisioned as linked clones or manually, and they are used for published desktops and published applications.

There is one other choice that needs to be selected when creating a desktop pool, and that is the desktop assignment type.  There are two desktop assignment types:

  • Floating Assignment – Desktops are assigned to users at login and are returned to the pool of available desktops when the user signs out.
  • Dedicated Assignment – Desktops are assigned to a user, and the user gets the same desktop at each login.  Desktops can be assigned automatically at first login or manually by an administrator.

For this walkthrough, I will be doing an Automatic Floating Assignment Linked-Clone desktop pool, otherwise known as a Non-Persistent Linked Clone Pool.  This type of desktop pool utilizes View Composer and

1. Log into the Horizon 7 Administrator.  Under Catalog, select Desktop Pools.


2.  Click Add to add a new pool.


3. Select the Pool Type that you want to create.  For this, we’ll select Automated Pool and click Next.


4.  Select whether you want to have Floating or Dedicated Desktops.  For this walkthrough, we’ll select Floating and click Next.


Note: The Enable Automatic Assignment option is only available if you select Dedicated. If this option is selected, View automatically assigns a desktop to a use when they log in to dedicated pool for the first time.

5. Choose the type of virtual machines that will be deployed in the environment. For this walkthrough, select View Composer Linked Clones and click Next.


6. Each desktop pool needs an ID and a Display Name.  The ID field is the official name of the pool, and it cannot contain any spaces.  The Display Name is the “friendly” name that users will see when they select a desktop pool to log into.  You can also add a description to the pool.


7. The next screen after setting the pool name is for the pool settings.  There are a lot of options here, that control how the pool will behave.  Some of the options are:

  • If the pool is enabled
  • Default power state of desktops
  • Display protocols
  • Adobe Flash settings




8. The next screen will allow you to configure the provisioning settings for the pool.  This screen allows you to control provisioning behavior, computer names, and the number of desktops provisioned in the pool.


9. The next screen allows you to set up a special non-persistent disk for disposable files.  Disposable files are classified as temporary files and page files.  If a disposable disk is used, these files will be redirected to here, and this disk is deleted whenever the VM is shut down.

Note: I don’t recommend the use of disposable file redirection.

This screen allows you to determine how the virtual desktop will handle these files.


10. Select the option to store Replicas on a separate datastore if you want to place them on a different storage tier.  Andre Leibovici has a good article on the benefits of placing Linked Clone replicas on a different datastore.


11. After you choose whether or not to place the Replica Disks on a separate datastore, you need to configure the pool’s vCenter settings.  This covers the Parent VM and the snapshot that the Linked Clones will be based on, the folder that they will be stored in within vCenter, and the cluster and datastores that will be used.

In order to configure each setting, you will need to click the Browse button on the right hand side of the screen.  Each step must be configured in order.


11-A. The first item that needs to be configured is the Parent VM that the Linked Clones will be based on.  Select the VM that you want to use and click OK.


11-B. The next step is to select the Parent VM snapshot that the Linked Clones will be based on.  Select the snapshot that you want to use and click OK.


11-C. After you have selected a Parent VM and a snapshot, you need to configure the vCenter folder in the VMs and Templates view that the VMs will be placed in.  Select the folder and click OK.


11-D. The next step is to place the pool on a vSphere cluster.  The virtual machines that make up the desktop pool will be run on this cluster, and the remaining choices will be based on this selection.  Select the cluster that they should be run on and click OK.


11-E. The next step is to place the desktops into a Resource Pool.  In this example, I have not resource pools configured, so the desktops would be placed in the Cluster Root.


11-F. The final two steps of this section are to select the datastores where the Linked Clones and the Replicas will be stored.  Linked Clones can be stored on multiple datastores, so you can select multiple datastores in this section.  You can also configure View to allow the datastores to be overcommitted by changing the Storage Overcommit option on each datastore.


11-G. Replicas can only be stored on a single datastore.  Select the datastore that you want to store them on and click OK.


Note: After you have configured the Replica Datastore, you may receive the following warning about storing Replicas and Linked Clones on local datastores.  If you are using a SAN or a NAS and not storing any Replicas or Linked Clones on local datastores, you can ignore this message.

Warning after 18-19

12. The next screen is for configuring the advanced storage options.  The three options that can be configured on this screen are the View Storage Accelerator, disk space reclaimation and the option to use native NFS snapshots.

If you use View Storage Accelerator or disk space reclamation, you can configure blackout times where vCenter will not run these tasks as these tasks may generate a significant amount of storage IO. 


13. To set the blackout times for the pool, click the Add Button and select the days and times when you do not want these operations to run.  You can set multiple schedules.


14. After you have configured the advanced storage options, you need to configure the Guest Customization settings.  This screen allows you to select the domain and organizational unit for the desktops and whether Sysprep or Quickprep will be used to prepare the desktops.


15. Review the settings for the pool and verify that everything is correct.  Before you click Finish, check the Entitle Users checkbox in the upper right.  This will allow you to select the users and/or groups who have permission to log into the desktops.

If you need to make a change to the pool settings, the left-hand column contains links to each page in the wizard.


17. After you click Finish, you will need to grant access to the pool.  View allows you to entitle Active Directory users and groups.  Click Add to entitle users and groups.


18. Search for the user or group that you want to add to entitle.  If you are in a multi-domain environment, you can change domains by selecting the domain from the Domains box.  Click on the users or groups that you want to grant access to and click OK.


Note:  I recommend that you create Active Directory security groups and entitle those to desktop pools.  This makes it easier to manage a user’s pool assignments without having to log into View Administrator whenever you want to make a change.

19. You can check the status of your desktop pool creation in vCenter.  If this is a new pool, it will need to clone the VM into a Replica before it can create the Linked Clone desktops. 


Once the desktops have finished composing, you will be able to log into them through VMware Blast or the Horizon client. 

I realize that there are a lot of steps in the process of creating a desktop pool.  It doesn’t take nearly as long as it seems once you get the hang of it, and you will be able to fly through it pretty quickly.

by seanpmassey at November 08, 2016 02:45 PM


Starting a newsletter

I share a ton of links, I have a thing that harvests my twitter account and over a few years it has collected over 4 500 links. Twitter is great for that but also it’s hard to add a bit of annotation or comment to those shared links in the constrained space. Of course I also curate my Free for Dev list which has grown quite huge, so I come across a lot of interesting stuff.

Everyone these days seem to have a news letter and I’ve toyed with the idea for a long time. The idea would be to share the random things I come across, no real theme, goal or set frequency, just a list of stuff I find with some short comments whenever I feel I have a good selection of things to share. I could stick them up here as blog posts but I don’t want one of those sad blogs where the entire frontpage is weekly link lists 😉

Well I’ve decided to give it a go, I signed up with Mail Chimp and set something up there. Last week I tweeted twice about it and tried to get some early adopters on board for the first mail in order to get some feedback and so forth. This has been quite positive thanks a lot to those who took the time to send me a note.

So if you’d like to subscribe please head over to and sign up at the bar above, it’s the usual Mail Chimp thing so you’ll get a confirmation email to double opt-in with. If you want to see what you’re getting yourself into the archive can be found at Mail Chimp, there is one mail there already.

The next one should go out tomorrow or so, looking forward to trying this out!

by R.I. Pienaar at November 08, 2016 08:43 AM

November 07, 2016

Laravel: The only supported ciphers are AES-128-CBC and AES-256-CBC with the correct key lengths

The post Laravel: The only supported ciphers are AES-128-CBC and AES-256-CBC with the correct key lengths appeared first on

I just started working on a new Laravel project (it's been a while), and this had me googling. For a second, I thought I had missing PHP extensions (like 'mcrypt') or other cryptographic configs I needed to tweak.

But sometimes a solution is so obvious, you miss it.

So I found myself hitting this error on a new Laravel project, without any configuration whatsoever. Just did the composer install and browsed to the app:

$ tail -f laravel.log
[2016-11-07 15:48:13] local.ERROR: exception 'RuntimeException' with message
  'The only supported ciphers are AES-128-CBC and AES-256-CBC with the correct
   key lengths.' in htdocs/bootstrap/cache/compiled.php:13261

Stack trace:
#0 htdocs/bootstrap/cache/compiled.php(7739): Illuminate\Encryption\Encrypter->__construct('xxxx...', 'AES-256-CBC')
#1 htdocs/bootstrap/cache/compiled.php(1374): Illuminate\Encryption\EncryptionServiceProvider->Illuminate\Encryption\{closure}(Object(Illuminate\Foundation\Application), Array)
#2 htdocs/bootstrap/cache/compiled.php(1330): Illuminate\Container\Container->build(Object(Closure), Array)
#3 htdocs/bootstrap/cache/compiled.php(1908): Illuminate\Container\Container->make('encrypter', Array)
#4 htdocs/bootstrap/cache/compiled.php(1431): Illuminate\Foundation\Application->make('Illuminate\\Cont...')
#5 htdocs/bootstrap/cache/compiled.php(1408): Illuminate\Container\Container->resolveClass(Object(ReflectionParameter))
#6 htdocs/bootstrap/cache/compiled.php(1394): Illuminate\Container\Container->getDependencies(Array, Array)
#7 htdocs/bootstrap/cache/compiled.php(1330): Illuminate\Container\Container->build('App\\Http\\Middle...', Array)
#8 htdocs/bootstrap/cache/compiled.php(1908): Illuminate\Container\Container->make('App\\Http\\Middle...', Array)
#9 htdocs/bootstrap/cache/compiled.php(2426): Illuminate\Foundation\Application->make('App\\Http\\Middle...')
#10 htdocs/public/index.php(58): Illuminate\Foundation\Http\Kernel->terminate(Object(Illuminate\Http\Request), Object(Illuminate\Http\Response))
#11 {main}

Every time, the cause has been that my .env file had unsupported characters in the APP_KEY environment variable that caused that "The only supported ciphers are AES-128-CBC and AES-256-CBC with the correct key lengths" error.

So as a reminder to my future self: alphanumeric characters, no spaces, no dashes, nothing fancy.


$ cat .env

As soon as you include dashes or other "special" characters, things inevitably b0rk. And my default random-character-generator includes some dashes, obviously.

The post Laravel: The only supported ciphers are AES-128-CBC and AES-256-CBC with the correct key lengths appeared first on

by Mattias Geniar at November 07, 2016 08:45 PM

Apache HTTP authentication in .htaccess

The post Apache HTTP authentication in .htaccess appeared first on

There are plenty of guides that describe this already, but I find it frustrating to find a clear, concise write-up of what's needed to get some simple username/password HTTP authentication to work just with .htaccess code.

So, here's the short version.

Create an htpasswd file

This is a simple file that holds the username and (encrypted) password of the users you want to allow access.

$ htpasswd -c /var/www/vhosts/site.tld/passwd/htaccess.passwd mattias

The above will create a new file at that location and configure a user called "mattias", you'll be prompted (twice) for your password.

The -c parameter makes a new file, if you want to add new users to an existing passwd file, use this;

$ htpasswd /var/www/vhosts/site.tld/passwd/htaccess.passwd jasper

In the end, the file looks like this:

$ cat /var/www/vhosts/site.tld/passwd/htaccess.passwd

That's an encrypted password, to reset it, run the above command again for the same username, it'll overwrite the password.

If you want to use the password file above (because it's quick and easy copy/pasting): that encrypted password in plain text is "nucleus", so you can log in with the user "mattias" and password "nucleus".

HTTP authentication in .htaccess

Now, once you have the file, add the following in your .htaccess. This snippet of code will work for an Apache 2.2 and Apache 2.4 configuration. Just make sure you change the path to point to your htpasswd file.

$ cat .htaccess
<IfModule mod_authn_file.c>
  # For Apache 2.4 or later
  AuthType Basic
  AuthName "Please provide username and password"
  AuthBasicProvider file
  AuthUserFile "/var/www/vhosts/site.tld/passwd/htaccess.passwd"
  Require valid-user

<IfModule mod_auth.c>
  # For Apache 2.2 or lower
  AuthType Basic
  AuthName "Please provide username and password"
  AuthUserFile "/var/www/vhosts/site.tld/passwd/htaccess.passwd"
  Require valid-user

Next time you visit your site, you'll be prompted for an HTTP username and password.

The post Apache HTTP authentication in .htaccess appeared first on

by Mattias Geniar at November 07, 2016 06:35 PM


PuppetConf 2016 – External Data in Puppet 4

I recently gave a talk at PuppetConf in San Diego covering the new Lookup feature in Puppet 4, you can see the video below.

The slides for this talk can be seen below:

by R.I. Pienaar at November 07, 2016 05:41 PM

November 06, 2016

Carl Chenet

Dans le Libre : tout automatiser

Dans le précédent article de cette série Dans le Libre : manger ce que l’on prépare soi-même, nous avons vu pourquoi dans une démarche de Libriste visant à régler un problème par l’utilisation ou l’écriture d’un programme, il était crucial pour améliorer et rendre plus flexible ledit programme de l’utiliser soi-même autant que faire se peut.

Nous verrons aujourd’hui pourquoi il est important d’automatiser autant que possible les différentes actions du cycle de développement et de la mise en production de son propre projet logiciel.

Les erreurs humaines sont inévitables

Dans un processus aussi compliqué la mise au point, la publication puis le passage en production d’un nouveau logiciel, les erreurs humaines sont quasi-inévitables. En effet les manipulations manuelles sont légions.

Les erreurs humaines sont inévitables dans une chaîne manuelle de production de logiciel

Les erreurs humaines sont inévitables dans une chaîne manuelle de production de logiciel

Et si bien sûr autant on peut considérer normal tout l’artisanat de la phase de conception du programme (la programmation en elle-même), autant il est utile d’avoir dès cette phase puis de manière exponentielle dans les phases suivantes de l’automatisation, pour fiabiliser l’ensemble de la chaîne de production du logiciel par la réduction au maximum des actions manuelles. Mais détaillons.

Automatisation durant la phase de conception

Dès l’étape de la phase de conception, il est  bon d’automatiser certaines manipulations, à savoir :

  • outil de détection d’erreur de syntaxe et de qualité de code (par exemple l’excellent outil pylint dans le monde Python)
  • outil de lancement d’un série de tests unitaires (dans le monde Python le module unittest, pytest)
  • outil d’intégration continue à coupler à votre dépôt de sources  pour les tests fonctionnels (Jenkins avec le plugin Git par exemple)
Jenkins, un outil d'intégration continue complet et facilement extensible via ses nombreux plugins

Jenkins, un outil d’intégration continue complet et facilement extensible via ses nombreux plugins

L’idée est de fiabiliser tout ce qui est autour de l’écriture du code lui-même, comme la détection des erreurs de syntaxe et vos tests après une modification même mineure de votre code sans y perdre 5 minutes à chaque fois. Vous vous concentrez ainsi sur l’essentiel et ne perdez pas de temps et d’énergie sur des tâches ennuyantes.

Automatisation durant la phase de publication

Votre nouvelle version est prête. Vous avez placé une étiquette dans votre dépôt de sources pointant sur le dernier commit avant la publication de votre nouvelle version. Les voyants des résultats des tests unitaires et fonctionnels sont au vert. Là encore il est possible d’automatiser, pour par exemple :

  • utiilser un script de publication pour automatiser la publication de cette nouvelle version, avec par exemple un job Jenkins permettant la création de l’archive de votre nouvelle version et sa mise à disposition dans votre forge logicielle.
  • utiliser un autre script afin de télécharger la nouvelle version puis l’installant sur un système avant d’effectuer une nouvelle série de tests fonctionnels, dans le but de s’assurer que le processus de publication n’a rien cassé.

La réalisation de ces scripts et leur intégration au sein de votre chaîne de création de votre logiciel vont aider à produire des versions publiques qui soient réellement utilisables par le grand public, la mise à disposition de vos sources étant nettement insuffisante pour vous attirer une base d’utilisateurs conséquente. Produire des archives utilisables et facilement installables pour chacune de vos nouvelles versions va aider à la diffusion de votre logiciel (par exemple par l’intervention d’empaqueteurs officiels de différentes distributions GNU/Linux, comme Debian) et ainsi à toucher un public plus large.

Industrialisation de votre logiciel

Comme nous l’avons vu, vous avez désormais un dépôt d’archives publiques avec chacune des versions de votre programme, au côté de votre dépôt de sources. Cela va nous permettre de passer à la dernière étape de notre chaîne d’industrialisation, à savoir la mise en place de votre nouveau programme sur le serveur. Pour cela nous pouvons identifier les étapes suivantes :

  • produire à partir de l’archive réalisée un paquet pour un système d’exploitation, par exemple un paquet Debian pour un système Debian et ses dérivées (encore une fois en utilisant un job Jenkins, lui-même lié aux jobs précédents).
  • pousser le nouveau paquet Debian dans votre dépôt local de paquets Debian (par exemple propulsé par Aptly, un gestionnaire très flexible de dépôts de paquets Debian), vous permettant ainsi un déploiement ultra-simple et sécurisée de l’application au sein de vos différents serveurs de production.
  • déployer le nouveau paquet en conservant ou modifiant la configuration existante, par exemple à l’aide d’un gestionnaire de configuration comme Ansible.
, un gestionnaire très flexible de dépôts de paquets Debian

Aptly, un gestionnaire très flexible de dépôts de paquets Debian

Avec une infrastructure qui se complexifie rapidement, il est simple de se laisser déborder par l’existant et la difficulté à le faire évoluer. L’industrialisation permet de rationaliser l’évolution de votre infrastructure, de sécuriser les évolutions et la disponibilité des services.

Tout automatiser mène à gratter ses autres démangeaisons

Nous l’avons vu, en automatisant au maximum les différentes étapes de votre chaîne de création de logiciel, vous apportez fiabilité et rapidité à votre cycle. Envisager des livraisons rapides et fiables de nouvelles versions devient possible.

Et pour en revenir à l’exemple de notre logiciel qui nous accompagné durant ces trois articles, nous sommes arrivés au bout de notre démarche, avec un beau projet logiciel répondant à notre principal besoin et aux petits besoins qui s’étaient crées autour. Le projet ronronne et vous n’allez assurer que des améliorations mineures et corrections de bugs dans les mois à venir.

C’est donc le moment de gratteur une autre démangeaison, de passer au projet suivant dans votre liste de choses à faire et de tâches à accomplir. En écrivant un programme pour répondre à votre besoin, puis en utilisant ce logiciel vous-même, démarche complétée par l’automatisation d’une grande partie de la chaîne de production, vous avez ainsi libéré le temps nécessaire pour vous occuper d’autres projets.

Automatiser a un coût

Ceux qui sont intervenus sur des projets informatiques réalisés à la va-vite, bâti de bric et de broc laborieusement dans le temps, dont on ne retrouve ni la documentation ni les sources, qui ne tournent que sur un serveur de production dont on ne retrouve plus les informations de connexion, comprendront l’inégalable sécurité qu’apporte l’automatisation en grande partie de la chaîne de réalisation d’un logiciel.

Mais ce confort a un coût. Nous avons donné au long de cet article des axes d’automatisation qui s’avèrent payants sur le moyen ou long terme à mettre en place. Mais ces efforts d’automatisation ont un coût, qui avant d’être financier va être humain. Écrire des tests unitaires prend du temps. Mettre en place une usine d’intégration continue prend du temps. Écrire des tests fonctionnels ou des tests d’infrastructure prend encore du temps. Définir les politiques de style de votre code maison et les matérialiser par un fichier de configuration de votre outil de vérification de style de code prend du temps. Et ce temps, on l’a rarement.

Il faut donc garder en tête que la priorité reste au produit, à sa délivrance et à son amélioration, mais que l’absence d’automatisation finira immanquablement par vous poignarder dans le dos à un moment ou à un autre. Donc attention au phénomène du « I’m too busy to improve » car vous finirez forcément dans une situation d’interbloquage où votre charge vous empêchera d’automatiser, mais où on vous demander de délivrer plus vite qu’il vous est humainement possible.


Attention donc à bien garder l’automatisation en vue dans le développement de votre projet personnel ou de votre système d’information, car plus l’infrastructure devient lourde et les « hot fixes » nombreux, plus il sera dur d’automatiser, ce processus demandant un maximum d’uniformisation au niveau de votre infrastructure.

by Carl Chenet at November 06, 2016 11:00 PM

Very fast MySQL slave setup with zero downtime using rsync

Most online tutorials for setting up a slave replication database involve dumping and loading the database on the master with mysqldump. Unfortunately, loading data with mysqldump can be very slow. My friend Cris suggest a much faster method using rsync.serveimage


The benefits of this method are:

  • Very fast setup of a slave by avoiding having to load a logical dump into MySQL.
  • Virtually no downtime for the master. There is a small window (seconds, usually) during which writes are temporary blocked.
  • If the slave was already set up and has become corrupt for whatever reason, rsync will ensure we won't have to copy all the data again that's already on the slave.

In this article I'm going to assume you've already correctly configured the master and slave for replication. If not, please see this guide on how to configure MySQL. You can skip the step where they perform a mysqldump, since that's what this article is replacing with rsync in the first place.

How it works

 Roughly speaking, we'll be doing the following steps:

  1. Stop the slave
  2. Rsync the binary files from the master to the slave while the master is running.
  3. Set a write lock on the master, record the master log position and do another rsync from the master to the slave.
  4. Unlock the master write lock.
  5. On the slave, we change the master log position and start replication again.

The benefits are obvious. Since we first do a sync to the slave without a write lock, the master can keep receiving writes. However, this leaves the slave in a potentially corrupt state, since we might have copied the master data in the middle of a transaction. That's why, after the initial bulk sync, we set a write lock on the master and perform another sync. This only needs to synchronize the new data since the last sync, so it should be fast. The master will only need to be write-locked for a short amount of time. Meanwhile, we record the master log position for the slave. 

Do note that this will only work if your master and slave database are the architecture and run the same MySQL version. If not, this will end in disaster.


Let's see a practical example:

On the SLAVE, stop MySQL:

root@slave# sudo /etc/init.d/mysql stop

Configure the SLAVE not to automatically start replication on startup. We need this so we can modify the master log pos after starting the slave. Add the 'skip-slave-start' setting to the [mysqld] section:

root@slave# vi /etc/mysql/my.cnf


Now we rsync the MASTER binary data to the SLAVE. For this you'll need to have root ssh access enabled. Depending on your setup, there are a few files you'll have to exclude. For example, if you don't want to overwrite the users on your slave, you'll have to exclude the 'mysql' directory. You should experiment a bit with what you should and shouldn't exclude.

root@master# rsync -Sa --progress --delete --exclude=mastername* /var/lib/mysql root@

We've now synchronized the bulk of the master data to the slave. Now we need another sync to put the slave in a valid state. We do this by locking the MASTER to prevent write actions:

root@master# mysql
mysql> flush tables with read lock;
mysql> show master status;
| File             | Position | Binlog_Do_DB | Binlog_Ignore_DB                            |
| mysql-bin.000001 |    16242 |              | mysql information_schema performance_schema |

Record the value of the Position field, since we'll need it later on the slave. The master is now locked against writes.

Keep this MySQL session open and run the next rsync in a different terminal:

root@master# rsync -Sa --progress --exclude=mastername* /var/lib/mysql root@

As soon as the final rsync is done, we can unlock the master. Go back to the terminal containing your MySQL session and unlock the tables:

mysql> unlock tables;

The master can now receive writes again. Time to start the SLAVE:

root@slave# /etc/init.d/mysql start

On the slave, we'll change the master log position and start replication:

root@slave# mysql
mysql> change master to master_log_pos=16242;
mysql> start slave;

Verify that the slave is working properly:

mysql> show slave status\G

That's it, your slave is now up and running!


And now for the big disclaimer:

Copying binary files is always more risky than doing a logical dump and load. By necessity, there are differences between a master and a slave and how the handle the binary files. While in theory the above method should work, I have to caution against using this method if your slave is going to be vital to your setup. If your slave has been set up with the method above, be extra careful when relying on that slave for making logical backups and such.

by admin at November 06, 2016 08:52 AM

November 05, 2016

Ben's Practical Admin Blog

MSIgnite NZ 2016 – Righting the Right Rights with Jess Dodson

If you are an Active Directory Administrator, Check out this presentation by MVP Jess Dodson on AD Security and Maintenance which was presented in the main hall at MSIgnite NZ a couple of weeks ago.

Some of you may already be familiar with Jess’s work over at Girlgerms Online and if not, these is definitely one of the better systems administration blogs in Australia🙂

Edit 6/11/16 – WordPress is not correctly embedding video – looks fine in editor, but shows a link when published. have updated URL to direct link to the Channel 9 site.

by Ben at November 05, 2016 10:00 AM

November 04, 2016

Racker Hacker

OpenPOWER Summit Europe 2016 Recap

Morning in BarcelonaI was in Barcelona last week for two big events: the OpenStack Summit and the OpenPOWER Summit. Luckily, the events were separated only by a five minute walk. Many of the slides from the OpenPOWER Summit are already available online.

One of my favorite talks was from Prof. Mateo Valero, the director of the Barcelona Supercomputer Center (Centro Nacional de Supercomputación). He was a great speaker and he talked a lot about how OpenPOWER has given them a new edge. It’s part of what helps them stay on the forefront of supercomputer technology.

On Friday, I was honored to be on a panel with some other leaders in the OpenPOWER community:

We talked about what makes open source a game changer in our businesses and our worlds. I talked about how open source software and hardware gives everyone a seat at the table. Every seat is a little different — some people offer software improvements, some offer use cases, and others do real-world testing. Every seat has intrinsic value in the process and everyone has a say in where the community goes.

Open source also provides mobility in multiple ways. For businesses, open source allows them to stand on the shoulders of others and make meaningful improvements in their product much more quickly. Open source projects also allows individual people to increase their skills, learn patterns from others, and level up their capabilities.

Several vendors were showing off technology during the reception on Thursday night and I had the opportunity to talk with Rob Taylor from

They’re doing some amazing work with accelerators that makes them more approachable for developers. Learning VHDL isn’t terribly easy and developers prefer to work in the languages that they’re using to build their applications. The folks at have software that allows a developer to write software in C, C++, or Go and then translate certain pieces of that code to VHDL. This makes accelerators more approachable for more developers and reduces the cycle time for improvements.

The conference was fairly small, but it was nice to get extra face time with developers, executives, and partners in the OpenPOWER market. I’ll be looking forward to the next wave of developments in accelerators and Power 9!

The post OpenPOWER Summit Europe 2016 Recap appeared first on

by Major Hayden at November 04, 2016 02:07 PM

November 03, 2016

The Tech Teapot

Constraint Capers Workbench Version 0.3 Milestone

It has taken quite a while, but version 0.3 of Constraint Capers Workbench is coming together pretty well.

workbench-model-nqueens Chessboard showing n-queens solution

I decided that for this release I needed to start solving actual constraint problems. Well, one specific problem to be more precise, the classic problem, n-queens.

Constraint Repeaters

Constraint repeaters provide a mechanism to expand a single constraint into many statements.

The n-queens model has the following constraint:

cols[i] <> cols[j] | i,j in 8,i

The part on the right hand side is the repeater. The repeater expands the constraint into (potentially) many simpler constraints. In the above example, the following simpler constraints would be generated:

cols[1] <> cols[0]
cols[2] <> cols[0]
cols[2] <> cols[1]
cols[3] <> cols[0]
cols[3] <> cols[1]
cols[3] <> cols[2]
cols[7] <> cols[0]
cols[7] <> cols[1]
cols[7] <> cols[2]
cols[7] <> cols[3]
cols[7] <> cols[4]
cols[7] <> cols[5]
cols[7] <> cols[6]

One area that needs improvement is the difficulty of debugging the repeater.

Chessboard Visualizer

The model will often not be in a form that is very user friendly. The visualizer is designed to display the solution to the model in a manner that is likely to be understandable by the user.

The chessboard isn’t very flexible, it can only display an 8×8 board at the moment because that is all I needed for the current version.

Visualizer Binding Expression

One of the big problems with designing this software is finding a mechanism to convert the model into something that can be easily understood.

For instance, the mechanism I used to solve the n-queens puzzle is to have an aggregate variable with eight variables. The domain for the variable is the values from one through to eight. Each of the variables represent an individual column on the chess board. The domain represent the queens row in that column.

I have implemented a small language that is executed after the model is solved and is able to take the solution and convert it into something that makes sense to the visualizer.

Here’s example from the n-queens solution:

for x,y in 1..8,1..8: if <cols,x> = y: board(x:x,y:y,side:white,piece:Queen)

The above expression loops through each position on an 8×8 chess board and checks to see if it should place a queen in that square. The <cols,x> expression gets the xth variable from the cols value ie the values bound to the cols variable. The board part is the name of the chessboard visualizer. It assigns a white queen to the x, y location.

What’s Next?

The next release will also concentrate on a single constraint problem again, the map colouring puzzle.

Whilst the map colouring problem could be modeled using the current version of the software, the map itself could not be displayed satisfactorily. The visualizers are absolutely crucial if the workbench is to meet its design goals.

Onwards to the 0.4 milestone…

by Jack Hughes at November 03, 2016 01:39 PM

Ben's Practical Admin Blog

iDRAC Firmware 2.40.40 Introduces New TLS Settings, May Stop Dell OME Discovery/Inventory

iDRAC firmware 2.40.40 was released on 17th Oct 2016. Details can be found by following this link.

We have recently had the need to upgrade our iDRAC firmware to 2.40.40 on one of our servers while troubleshooting another issue with Dell and found shortly after that this particular server was no longer able to be discovered by Dell OpenManage Essentials.

We found that the TLS protocol the iDRAC set after updating was set to version 1.2, which is not supported by Windows operating systems less than Server 2012 R2 (Our OME server runs on Windows Server 2012). There is a patch available to fix this. This is all covered in the driver release notes.

The other alternative, which we have chosen to do for now  as this firmware is only on one device is to set the iDRAC to use the older TLS protocol, which can be found under the iDRAC Network Settings in the services tab:


I’ll apply the Microsoft patch to the system, and then set the TLS back to v1.2

by Ben at November 03, 2016 02:32 AM

November 01, 2016

That grumpy BSD guy

The Possibly Russian Fingerprints on the Shadow Brokers' Trick or Treat Package

Whether a result of clumsiness of a bored operator or deliberate subterfuge, there are clues that the supposed NSA front Equation Group operated out of Russia. The question remains:  What were they doing that for?

As you've probably heard already, an outfit calling themselves the Shadow Brokers published a list of what is supposedly hosts that had been compromised by the NSA and subsequently used as staging servers for whatever the NSA wanted to throw at their adversaries, with some other shady outfit known as the Equation Group actually doing the cyber-cyber thing.

The Shadow Brokers' message with links to their material is available in a number of places, among them this Medium article, which seems to be simply the text of the file message5.txt.asc, which is one of three items at both download locations.

The "list" is actually a compressed and encrypted tar archive (familiar to Unix users everywhere), which expands to a directory structure where the first level is the directory trickortreat, with two subdirectories intonation and pitchimpair, both of which in turn have numerous subdirectories with names that follow the convention


that is, a fully qualified domain name for a host, three underscore characters and the corresponding IP version four address in the common four octet decimal notation. Each of these directories then have small text files that appear to be data about a specific program or feature.

For example, the directory intonation/hakuba.janis.or.jp___210.232.42.3 contains only the file jackladder, with the content

INTONATION___hakuba.janis.or.jp___210.232.42.3___20000822-135045() {
    ## JACKLADDER Version:2.0 OS:sparc-sun-solaris2.6

I take this to mean that INTONATION, whatever that is, contacted the host at the IP address, and peeking at the next line which gives us the OS version, it's my educated guess that the last string of the first line is the date in YYYYMMDD and the patch level for the operating system recorded.

The second line, or the body of the curly braces ({}) part if you prefer, tells us the JACKLADDER version and the operating system.

Other subdirectories have several files that appear to follow roughly the same format, and some record other parameters such as trickortreat/intonation/msgstore2.pldtprv.net___192.168.120.3, where the file orangutan

INTONATION___msgstore2.pldtprv.net___192.168.120.3___20021114-120148() {
    ## ORANGUTAN Version:1.4 OS:sparc-sun-solaris2.8
    export CONFIG_KEYS="81733968 69bb0b91 8b6400d6"

also appears to record the content of an environment variable, possibly one that the ORANGUTAN software needs to see exported to its environment in order to work as intended on that host.

Basically, this looks like what a fairly well automated system would leave behind while performing a number of operations targeted at various hosts, in a directory structure where humans will be able to find what they need to look at quickly. In my line of work, it's fairly common for such things as system logs to be collected in file system structures much like what we see here.

For your convenience if you want to study the material yourself and can't really be bothered to figure out how to extract the clear text from the gpg encrypted archive, I've put the plaintext tar archive here and the extracted files here for you to browse at your own pace.

My initial plan when I downloaded and decrypted the material was to check whether any of the hostnames or IP addresses in this material matched any of the entries in my records, such as the Hail Mary Cloud cycle that targeted SSH servers or the more recent password guessing efforts aimed at POP3 mail servers.

But then I noticed while reading the analyses of other geeks who had gotten around to doing their thing that the lists of IP addresses all had in them some addresses that should not have been there at all.

The entire network was set aside way back in RFC1918 (February 1996) as one of several non-routeable ranges for private use in local area or campus networks. The way the world and IP version four works, if you have a local network at home or at work (even in large multinational enterprises), more likely than not your machines have addresses in one of these ranges:        -  (10/8 prefix)      -  (172.16/12 prefix)     - (192.168/16 prefix)

and something between those hosts and the Internet that does the network address translation (NAT) so all traffic from your network appears to come from a routable address.

Even on machines that have routeable addresses, it is not uncommon that one or more other interfaces are configured with RFC1918 addresses to pass internal but necessary traffic such as administrator logons, backups and other administrative tasks somewhere that does not interfere with the internet-facing production traffic.

Still, the Shadow Brokers' Trick or Treat package contains a handful of directories for hosts that only give internal, non-routable (RFC1918) addresses:


If your most convenient route to a machine to a specific machine is to an interface on that machine that has a local area network address, that is a strong indicator that you are working from that local network.

The first four hosts are in a Russian domain which is still operating, apparently out of Russia. The last domain,, appears to be no longer active but I assume a diligent search will turn up clues about where they were based and when they were operating.
This could mean that the supposed NSA front Equation Group was actually operating from inside Russia, or at the very least had establised a 'forward base' there which was so well connected (via virtual private networks (VPNs) or other means) to their home ground that the least costly route from wherever those scripts ran to one or more Russian hosts was via those machines' internal administrative or backup interfaces.

I repeat, this, that private, nonrouteable IP addresses appear in a place you would expect to find public and routeable addresses, is a strong indicator that the Equation Group ran their activities from a local network in Russia, likely rubbing shoulders with whoever operates the domain.

This is also a sign that whoever ran those scripts was a little careless about their routing at some point. But it could equally well mean that somebody, somewhere, is very adept at inserting clues that may in fact be false and misleading.

I probably will go forward with the comparison of this IP address and hostname hoard with my accumulated logs of less than desirable activity at some point. In the meantime I welcome your feedback on this story in comments or (if you don't want your name known and really only want me to paraphrase you in public) via email.

Good night and good luck.

Update 2016-11-02: Added the 'I repeat ...' paragraph to emphasize that finding private and nonrouteable addresses where you would otherwise expect find public and routeable ones is a strong clue that the perpetrators were operating from that local network. A surprising number of security professionals apparently miss that point.

by Peter N. M. Hansteen ( at November 01, 2016 06:32 PM

Anton Chuvakin - Security Warrior

Monthly Blog Round-Up – October 2016

Here is my next monthly "Security Warrior" blog round-up of top 5 popular posts/topics this month:
  1. “New SIEM Whitepaper on Use Cases In-Depth OUT!” (dated 2010) presents a whitepaper on select SIEM use cases described in depth with rules and reports [using now-defunct SIEM product]; also see this SIEM use case in depth and this for a more current list of popular SIEM use cases. Finally, see our 2016 research on security monitoring use cases here!
  2. Why No Open Source SIEM, EVER?” contains some of my SIEM thinking from 2009. Is it relevant now? You be the judge.  Succeeding with SIEM requires a lot of work, whether you paid for the software, or not. BTW, this post has an amazing “staying power” that is hard to explain – I suspect it has to do with people wanting “free stuff” and googling for “open source SIEM” … 
  3. Simple Log Review Checklist Released!” is often at the top of this list – this aging checklist is still a very useful tool for many people. “On Free Log Management Tools” is a companion to the checklist (updated version)
  4. “SIEM Resourcing or How Much the Friggin’ Thing Would REALLY Cost Me?” is a quick framework for assessing the costs of a SIEM project (well, a program, really) at an organization (much more details on this here in this paper).
  5. My classic PCI DSS Log Review series is always popular! The series of 18 posts cover a comprehensive log review approach (OK for PCI DSS 3+ as well), useful for building log review processes and procedures, whether regulatory or not. It is also described in more detail in our Log Management book and mentioned in our PCI book (now in its 4th edition!)
In addition, I’d like to draw your attention to a few recent posts from my Gartner blog [which, BTW, now has about 5X of the traffic of this blog]: 
Upcoming research on security analytics:
Currect research on deception:
Recent research on SOC:
Miscellaneous fun posts:

(see all my published Gartner research here)
Also see my past monthly and annual “Top Popular Blog Posts” – 2007, 2008, 2009, 2010, 2011, 2012, 2013, 2014, 2015.

Disclaimer: most content at SecurityWarrior blog was written before I joined Gartner on Aug 1, 2011 and is solely my personal view at the time of writing. For my current security blogging, go here.

Previous post in this endless series:

by Anton Chuvakin ( at November 01, 2016 02:57 PM


Puppet Query Language

For a few releases now PuppetDB had a new query language called Puppet Query Language or PQL for short. It’s quite interesting, I thought a quick post might make a few more people aware of it.


To use it you need a recent PuppetDB and as this is quite a new feature you really want the latest PuppetDB. There is nothing to enable when you install it the feature is already active. The feature is marked as experimental so some things will change as it moves to production.

PQL Queries look more or less like this:

nodes { certname ~ 'devco' }

This is your basic query it will return a bunch of nodes, something like:

    "deactivated": null,
    "latest_report_hash": null,
    "facts_environment": "production",
    "cached_catalog_status": null,
    "report_environment": null,
    "latest_report_corrective_change": null,
    "catalog_environment": "production",
    "facts_timestamp": "2016-11-01T06:42:15.135Z",
    "latest_report_noop": null,
    "expired": null,
    "latest_report_noop_pending": null,
    "report_timestamp": null,
    "certname": "",
    "catalog_timestamp": "2016-11-01T06:42:16.971Z",
    "latest_report_status": null

There are a bunch of in-built relationships between say a node and it’s facts and inventory, so queries can get quite complex:

inventory[certname] { 
  facts.osfamily = "RedHat" and
  facts.dc = "linodeldn" and
  resources { 
    type = "Package" and
    title = "java" and
    parameters.ensure = "1.7.0" 

This finds all the RedHat machines in a particular DC with Java 1.7.0 on them. Be aware this will also find machines that are deactivated.

I won’t go into huge details of the queries, the docs are pretty good – examples, overview.

So this is quite interesting, this finally gives us a reasonably usable DB to do queries that mcollective discovery used to be used for – but of course its not a live view nor does it have any clue what the machines are up to but as a cached data source for discovery this is interesting.



You can of course query this stuff on the CLI and I suggest you familiarise yourself with JQ.

First you’ll have to set up your account:

  "puppetdb": {
    "server_urls": "https://puppet:8081",
    "cacert": "/home/rip/.puppetlabs/etc/puppet/ssl/certs/ca.pem",
    "cert": "/home/rip/.puppetlabs/etc/puppet/ssl/certs/rip.mcollective.pem",
    "key": "/home/rip/.puppetlabs/etc/puppet/ssl/private_keys/rip.mcollective.pem"

This is in ~/.puppetlabs/client-tools/puppetdb.conf which is a bit senseless to me since there clearly is a standard place for config files, but alas.

Once you have this and you installed the puppet-client-tools package you can do queries like:

$ puppet query "nodes { certname ~ '' }"

Puppet Code

Your master will have the puppetdb-termini package on it and this brings with it Puppet functions to query PuppetDB so you do not need to use a 3rd party module anymore:

$nodes = puppetdb_query("nodes { certname ~ 'devco' }")

Puppet Job

At the recent PuppetConf Puppet announced that their enterprise tool puppet job supports using this as discovery, if I remember right it’s something like:

$ puppet job run -q 'nodes { certname ~ 'devco' }'


At PuppetConf I integrated this into MCollective and my Choria tool, mco feature still due a release (MCO-776, choria):

Run Puppet on all the nodes matched by the query:

$ puppet query "nodes { certname ~ '' }"|mco rpc puppet runonce

The above is a bit limited in that the apps in question have to specifically support this kind of STDIN discovery – the rpc app does.

I then also added support to the Choria CLI:

$ mco puppet runonce -I "pql:nodes[certname] { certname ~ '' }"

These queries are a bit special in that they must return just the certname as here, I’ll document this up. The reason for this is that they are actually part of a much larger query done in the Choria discovery system (that uses PQL internally and is a good intro on how to query this API from code).

Here’s an example of a complex query – as used by Choria internally – that limits nodes to ones in a particular collective, our PQL query, which ones have mcollective installed and running. You can see you can nest and combine queries into quite complex ones:

nodes[certname, deactivated] { 
  # finds nodes in the chosen subcollective via a fact
  (certname in inventory[certname] { 
    facts.mcollective.server.collectives.match("\d+") = "mcollective" 
  }) and 
  # does the supplied PQL query, note the specific certname match
  (certname in nodes[certname] {
    certname ~ ''
  }) and
  # limited to machines with mcollective installed
  (resources {
    type = "Class" and title = "Mcollective"
  }) and 
  # who also have the service started
  (resources {
    type = "Class" and title = "Mcollective::Service"


This is really handy and I hope more people will become familiar with it. I don’t think this quite rolls off the fingers easily – but neither does SQL or any other similar system so par for the course. What is amazing is that we can get nearer to having a common language across CLI, Code, Web UIs and 3rd party tools for describing queries of our estate so this is a major win.

by R.I. Pienaar at November 01, 2016 07:26 AM

October 31, 2016

Toolsmith - GSE Edition: snapshot.ps1

I just spent a fair bit of time preparing to take the GIAC Security Expert exam as part of the requirement to recertify every four years. I first took the exam in 2012, and I will tell you, for me, one third of the curriculum is a "use it or lose it" scenario. The GSE exam covers GSEC, GCIH, and GCIA. As my daily duties have migrated over the years from analyst to leadership, I had to "relearn" my packet analysis fu. Thank goodness for the Packetrix VM and the SANS 503 exercises workbook, offsets, flags, and fragments, oh my! All went well, mission accomplished, I'm renewed through October 2020 and still GSE #52, but spending weeks with my nose in the 18 course books reminded of some of the great tools described therein. As a result, this is the first of a series on some of those tools, their value, and use case scenarios.
I'll begin with snapshot.ps1. It's actually part of the download package for SEC505: Securing Windows and PowerShell Automation, but is discussed as part of the GCIH curriculum. In essence, snapshot.ps1 represents one script to encapsulate activities specific to the SANS Intrusion Discovery Cheat Sheet for Windows.
The script comes courtesy of Jason Fossen, the SEC505 author, and can be found in the Day 5-IPSec folder of the course download package. The script "dumps a vast amount of configuration data for the sake of auditing and forensics analysis" and allows you to "compare snapshot files created at different times to extract differences."
To use snapshot.ps1 place the script into a directory where it is safe to create a subdirectory as the script creates such a directory named named for the computer, then writes a variety of files containing system configuration data.  Run snapshot.ps1 with administrative privileges.
The script runs on Windows 7, Server 2008, and newer Windows operating systems (I ran it on Windows 10 Redstone 2) and requires PowerShell 3.0 or later. You also need to have autorunsc.exe and sha256deep.exe in your PATH if you want to dump what programs are configured to startup automatically when your system boots and you login, as well as run SHA256 file hashes.
That said, if you must make the script run faster, and I mean A LOT FASTER, leave file
hashing disabled at the end of the snapshot.ps1 for a 90% reduction in run time. 
However, Jason points out that this is one of the most useful aspects of the script for identifying adversarial activity. He also points out that snapshot.ps1 is a starter script; you can and should add more commands. As an example, referring back to toolsmith #112: Red vs Blue - PowerSploit vs PowerForensics, after importing PowerForensics, you could add something like Get-ForensicTimeline | Sort-Object -Property Date | Where-Object { $_.Date -ge "12/30/2015" -and $_.Date -le "01/04/2016" } | WriteOut -FileName Timeline which would give you a file system timeline between the 12/30/2015 and 01/04/2016.But wait, there's more! Want to get autoruns without needing autorunsc.exe?  Download @p0w3rsh3ll's AutoRuns module, run Import-Module AutoRuns.psm1, then Get-Command -Module AutoRuns to be sure the module is on board, and finally comment out autorunsc.exe -accepteula -a -c | Out-File -FilePath AutoRuns.csv then add Get-PSAutorun | WriteOut -FileName AutoRuns.
It's then as simple as running .\Snapshot.ps1 and watch your computer-named directory populate, 0V3RW4TCH-2016-10-31-9-7 in my case, per Figure 1.

Figure 1: Snapshot.ps1 run
Most result files are written in machine-readable XML, CSV, and TXT, as well as REG files generated by the registry exports via reg.exe.
A great example of a results file, is spawned via dir -Path c:\ -Hidden -Recurse -ErrorAction SilentlyContinue | Select-Object FullName,Length,Mode,CreationTime,LastAccessTime,LastWriteTime | Export-Csv -Path FileSystem-Hidden-Files.csv. The resulting CSV is like a journey down evil memory lane, where all the nuggets I've tested in the past leave artifacts. This would be EXACTLY what you would be looking for under real response scenarios, as seen in Figure 2.

Figure 2: Snapshot.ps1 grabs hidden files
Sure, there are bunches of related DFIR collection scripts, but I really like this one, and plan to tweak it further. Good work from Jason, and just one of many reasons to consider taking SEC505, or pursuing your GSE!
Cheers...until next time.

by Russ McRee ( at October 31, 2016 09:36 PM

Racker Hacker

Talk Recap: Holistic Security for OpenStack Clouds

Holistic Security for OpenStack CloudsThanks to everyone who attended my talk at the OpenStack Summit in Barcelona! I really enjoyed sharing some tips with the audience and it was great to meet some attendees in person afterwards.

If you weren’t able to make it, don’t fret! This post will cover some of the main points of the talk and link to the video and slides.


OpenStack clouds are inherently complex. Operating a cloud involves a lot of moving pieces in software, hardware, and networking. Securing complex systems can be a real challenge, especially for newcomers to the information security realm. One wrong turn can knock critical infrastructure online or lead to lengthy debugging sessions.

However, securing OpenStack clouds doesn’t need to be a tremendously stressful experience. It requires a methodical, thoughful, and strategic approach. The goal of the talk is give the audience a reliable strategy that is easy to start with and that scales easily over time.

Why holistic?

The dictionary definition of holistic is:

characterized by comprehension of the parts of something as intimately connected and explicable only by reference to the whole

To simplify things a bit, thinking about something holistically means that you understand that there are small parts that are valuable on their own, but they make much more value when combined together. Also, it’s difficult to talk about the individual parts and get a real understanding of the whole.

In holistic medicine, humans are considered to be a body, mind, and spirit. OpenStack clouds involve servers, software, and a business goal. Security consists of people, process, and technology. To truly understand what’s going on, you need to take a look at something with all of its parts connected.

Security refresher

Get into the mindset that attackers will get in eventually. Just change each instance of if to when in your conversations. Attackers can be wrong many times, but the defenders only need to be wrong once to allow a breach to occur.

Simply building a huge moat and tall castle walls around the outside isn’t sufficient. Attackers will have free reign to move around inside and take what they want. Multiple layers are needed, and this is the backbone of a defense-in-depth strategy.

Cloud operators need to work from the outside inwards, much like you do with utensils at a fancy dinner. Make a good wall around the outside and work on tightening down the internals at multiple levels.

Four layers for OpenStack

During the talk, I divided OpenStack clouds into four main layers:

  • outer perimeter
  • control and data planes
  • OpenStack services and backend services in the control plane
  • OpenStack services

For the best explanation of what to do at this level, I highly recommend reviewing the slides or the presentation video (keep scrolling).

Links and downloads

The slides are on SlideShare and they are licensed CC-BY-SA. Feel free to share anything from the slide deck as you wish, but please share it via a similar license and attribute the source!

The video of the talk (including Q&A) is up on YouTube:


I love feedback about my talks! I’m still a novice presenter and every little bit of feedback — positive or negative — really helps. Feel free to email me or talk to me on Twitter.

The post Talk Recap: Holistic Security for OpenStack Clouds appeared first on

by Major Hayden at October 31, 2016 03:52 PM

Ben's Practical Admin Blog


A couple of weeks ago SAGE-AU announced that it was undergoing a name change and rebrand, to become the IT Professional Association, ITPA.

As a long time SAGE-AU and now ITPA member, this has come as little surprise. This has been something that has been on the cards for close to 4 years now. It does however mean good things for the organisation.

I’ll spare you the details – these can be found in their press release and in the launch webinar.

What I will say though is that if you are an IT Professional in Australia, your representation to government and industry is weak. Only 5% of people who identify as IT Professionals in Australia are members of a professional body like ITPA,  ACS or AISA compared to say, accountants, who have near 100% membership to CPA or ICAA.

In order to gain a voice and support in the IT industry, consider joining a professional organisation. ITPA offers free associate membership, which also gives you access to free short courses.

by Ben at October 31, 2016 09:45 AM

October 30, 2016

Run Nginx proxy in Docker container for HTTP/2

The post Run Nginx proxy in Docker container for HTTP/2 appeared first on

This is a really quick write-up on how I've been running HTTP/2 on my server for the last 2 months, despite having an OS that doesn't support OpenSSL 1.0.2.

It uses a Docker container to run Nginx, built on the latest Alpine Linux distribution. This has a modern OpenSSL built-in without extra work. It uses the same Nginx configurations from the host server and uses a network binding mode that doesn't require me to remap any ports using iptables/haproxy/...

An nginx docker container for HTTP/2

The really quick way to get started is to run this command.

$ docker run --name nginx-container --net="host" -v /etc/nginx/:/etc/nginx/ -v /etc/ssl/certs/:/etc/ssl/certs/ -v /etc/letsencrypt/:/etc/letsencrypt/ -v /var/log/nginx/:/var/log/nginx/ --restart=always -d nginx:1.11-alpine

Once it's running, it'll show itself as "nginx-container":

$ docker ps
CONTAINER ID    IMAGE               COMMAND                  CREATED        STATUS
960ee2adb381    nginx:1.11-alpine   "nginx -g 'daemon off"   6 weeks ago    Up 2 weeks

And the reason I'm doing this in the first place: to get HTTP/2 back up on Chrome after they disabled support for NPN.


Running a Docker container with those arguments does a few things that are worth pointing out.

Host networking in Docker

First: --net="host" tells the Docker daemon to not use a local network and bind to a random port ("bridge" mode), but to use the same networking model as the host on which it's running. This means that the container wants to bind on port :443 or :80, and the host (the server running the Docker daemon) will treat it like that.

The --net="host" also makes it easier to use Nginx as a reverse proxy. All my configurations generated on my server look like this;

location / {
  proxy_pass                          http://localhost:8080;
  proxy_set_header Host               $host;
  proxy_set_header X-Real-IP          $remote_addr;

  proxy_set_header X-Forwarded-For    $proxy_add_x_forwarded_for;
  proxy_set_header X-Forwarded-Proto  https;
  proxy_read_timeout                  60;
  proxy_connect_timeout               60;
  proxy_redirect                      off;

In a default Docker container, localhost refers to that container itself, it has no idea what's running on host on port :8080. The --net="host" also allows the Docker container to communicate back to the host, connecting to the Apache service running on port :8080 in my case (which isn't run in a Docker container).

Add local configurations to the container

Second: multiple "-v" parameters: this adds data points in your docker container, shared from the host. In this case I'm mapping the directories /etc/nginx, /etc/ssl/certs, /etc/letsencrypt and /var/log/nginx to the docker container.

It also allows me to re-use my existing Let's Encrypt certificates and renewal logic, as that'll update the configurations on the host, and the container reads those.

There's a small catch though: a change in SSL certificates needs a reload from the Nginx daemon, which now means a reload of the Docker container.

Restart container when it crashes

Third: --restart=always will cause the container to automatically restart, in case the nginx daemon running inside it would crash or segfault.

It hasn't actually done that (yet), so it's a sane default as far as I'm concerned.

Use the official Nginx docker image

Fourth: -d nginx:1.11-alpine uses the official docker container from Nginx, built on the Alpine distribution.

The first time you run it, it will download this image from the Docker hub and store it locally.

Config management in the container

By binding my directories on the host directly into the container, I can keep using the same config management I've been using for years.

For instance, my Puppet configuration will write files to /etc/nginx. Because I mapped that to the Docker container, that container gets the exact same configurations.

It's essentially configuration management on the host, runtime in a container. This makes it considerably easier to implement Docker if you can re-use premade containers and supply it with your own configurations.

The challenges with Docker

Sure, there are plenty. Do I know what's in that container? Not fully. Do I trust whoever made that container? Only partially. Who updates the Nginx in that container when a critical vulnerability comes around? I have no clue.

It's a leap of faith, to be honest -- time will tell if it's worth it.


The post Run Nginx proxy in Docker container for HTTP/2 appeared first on

by Mattias Geniar at October 30, 2016 02:02 PM


Pro Tip: Keep Asterisk Configuration Files in Version Control

An Asterisk server has a very involved configuration system. One instance I manage has over 100 configuration files. As these configurations grow, it can be helpful to have a separate server running for testing and debugging problems. If you are pla...

by Scott Hebert at October 30, 2016 05:00 AM

October 29, 2016

The Geekess

Measuring the Impact of Negative Language on FOSS Participation (Part I)

A recent academic paper showed that there were clear differences in the communication styles of two of the top Linux kernel developers (“Differentiating Communication Styles of Leaders on the Linux Kernel Mailing List”). One leader is much more likely to say “thank you” while the other is more likely to jump into a conversation with a “well, actually”.

Many open source contributors have stories of their patches being harshly rejected. Some people are able to “toughen up” and continue participating, and others will move onto a different project. The question is, how many people end up leaving a project due to harsh language? Are people who experience positive language more likely to contribute more to a project? Just how positive do core open source contributors need to be in order to attract newcomers and grow their community? Which community members are good at mentoring newcomers and helping them step into leadership roles?

I’ve been having a whole lot of fun coming up with scientific research methods to answer these questions, and I’d like to thank Mozilla for funding that research through their Participation Experiment program.

How do you measure positive and negative language?

The Natural Language Processing (NLP) field tries to teach computers to parse and derive meaning from human language. When you ask your phone a question like, “How old was Ada Lovelace when she died?” somewhere a server has to run a speech to text algorithm. NLP allows that server to parse the text into a subject “Ada Lovelace” and other sentence parts, which allows the server to respond with the correct answer, “Ada Lovelace died at the age of 36”.

Several open source NLP libraries, including the Natural Language Toolkit (NLTK) and Standford CoreNLP also include sentiment analysis. Sentiment analysis attempts to determine the “tone” and objectiveness of a piece of text. I’ll do more of a deep dive into sentiment analysis next month in part II of this blog post. For now, let’s talk about a more pressing question.
wocintech (microsoft) - 62

How do you define open source participation?

On the surface, this question seems so simple. If you look at any github project page or Linux Foundation kernel report or Open Stack statistics, you’ll see a multitude of graphs analyzing code contribution statistics. How many lines of code do people contribute? How frequently? Did we have new developers contribute this year? Which companies had the most contributions?

You’ll notice a particular emphasis here, a bias if you will. All these measurements are about how much code an individual contributor got merged into a code base. However, open source developers don’t act alone to create a project. They are part of a larger system of contributors that work together.

In order for code or documentation to be merged, it has to be reviewed. In open source, we encourage peer review in order to make sure the code is maintainable and (mostly) free of bugs. Some reports measure the work maintainers do, but they often lack recognition for the efforts of code reviewers. Bug reports are seen as bad, rather than proof that the project is being used and its features are being tested. People may measure the number of closed vs open bug reports, but very few measure and acknowledge the people who submit issues, gather information, and test fixes. Open source projects would be constantly crashing without the contribution of bug reporters.

All of these roles (reviewer, bug reporter, debugger, maintainer) are valuable ways to contribute to open source, but no one measures them because the bias in open source is towards developers. We talk even less about the vital non-coding contributions people do (conference planning, answering questions, fund raising, etc). Those are invaluable but harder to measure and attribute.

For this experiment, I hope to measure some of the less talked-about ways to contribute. I would love to extend this work to the many different contributions methods and different tools that open source communities use to collaborate. However, it’s important to start small, and develop a good framework for testing hypothesis like my hypothesis about negative language impacting open source participation.

does it measure up?

How do you measure open source participation?

For this experiment, I’m focusing on open source communities on github. Why? The data is easier to gather than projects that take contributions over mailing lists, because the discussion around a contribution is all in one place, and it’s easy to attribute replies to the right people. Plus, there are a lot of libraries in different languages that provide github API wrappers. I chose to work with the library because it still looked to be active and it had good documentation.

Of course, gathering all the information from github isn’t easy when you want to do sentiment analysis over every single community interaction. When you do, you’ll quickly run into their API request rate limit of 5,000 requests per hour. There are two projects that archive the “public firehose” of all github events: and However, those projects only archive events that happened after 2011 or 2012, and some of the open source communities I want to study are older than that. Plus, downloading and filtering through several terabytes of data would probably take just as long as slurping just the data I need through a smaller straw (and would allow me to avoid awkward conversations with my ISP).

For my analysis, I wanted to pull down all open and closed issues and pull requests, along with their comments. For a community like Rust, which has been around since 2010, their data (as of a week or two ago) looks like this:

  • 18,739 issues
  • 18,464 pull requests
  • 182,368 comments on issues and pull request
  • 31,110 code review comments

Because of some oddities with the github API (did you know that an issue json data can be for either an issue or a pull request?), it took about 20 hours to pull down the information I need.

I’m still sorting through how exactly I want to graph the data and measure participation over time. I hope to have more to share in a week!

*Edit* The code is available on github, and the reports for various open source communities are also available.

by sarah at October 29, 2016 08:12 AM

Geek and Artist - Tech

Now using a Let’s Encrypt certificate

Last week I got a notification from StartSSL that my site certificate was going to expire in a couple of weeks. Since recently there has been some news (I guess you can check the Wikipedia entry for exact details) that suggests StartSSL is in some danger of no longer being trusted by major browsers, I decided to finally get around to moving to Let’s Encrypt for my certificates.

When the project was first in beta I had some intentions to do the same thing then, but the tooling was far less mature than it is now, and the trust situation was not as good. Right now, probably most people will be able to access the site without any problems. Programmatic access may not be as fortunate – so the main point of this blog post is to mention the change and ask you to let me know if you have problems accessing the site (if indeed you see this at all, possibly with a security warning). Just drop me a comment.

Otherwise, the process was relatively simple, but I am left wondering what kind of identity verification is involved. I didn’t have to confirm anything during the process that I actually owned the domain name, so what would stop someone else getting a certificate for my domain name? I should look into that in more detail.

Update 01.11.16:
Looks like Google has made the move to not trust StartCom any longer, and this echoes similar movements by Apple and Mozilla. So it seems like the right thing to do. Auf Wiedersehen, StartCom.

by oliver at October 29, 2016 06:36 AM

October 28, 2016

Colin Percival

A very valuable vulnerability

While I very firmly wear a white hat, it is useful to be able to consider things from the perspective of the bad guys, in order to assess the likelihood of a vulnerability being exploited and its potential impact. For the subset of bad guys who exploit security vulnerabilities for profit — as opposed to selling them to spy agencies, for example — I imagine that there are some criteria which would tend to make a vulnerability more valuable:
  • the vulnerability can be exploited remotely, over the internet;
  • the attack cannot be blocked by firewalls;
  • the attack can be carried out without any account credentials on the system being attacked;
  • the attack yields money (as opposed to say, credit card details which need to be separately monetized);
  • once successfully exploited, there is no way for a victim to reverse or mitigate the damage; and
  • the attack can be performed without writing a single line of code.
Much to my surprise, a few weeks ago I stumbled across a vulnerability satisfying every one of these criteria.

The vulnerability — which has since been fixed, or else I would not be writing about it publicly — was in Stripe's bitcoin payment functionality. Some background for readers not familiar with this: Stripe provides payment processing services, originally for credit cards but now also supporting ACH, Apple Pay, Alipay, and Bitcoin, and was designed to be the payment platform which developers would want to use; in very much the way that Amazon fixed the computing infrastructure problem with S3 and EC2 by presenting storage and compute functionality via simple APIs, Stripe fixed the "getting money from customers online" problem. I use Stripe at my startup, Tarsnap, and was in fact the first user of Stripe's support for Bitcoin payments: Tarsnap has an unusually geeky and privacy-conscious user base, so this functionality was quite popular among Tarsnap users.

October 28, 2016 07:00 PM

October 26, 2016

Evaggelos Balaskas

Thunderbird Enigmail

A Beginner’s Guide on How to use Thunderbird with Enigmail (Gpg4win) with their gmail account in 10 minutes on a windows machine

Thunderbird Enigmail - Gmail, Windows from vimeouser on Vimeo.

October 26, 2016 10:26 PM

October 25, 2016

LZone - Sysadmin

More Changes after the Dyn DDoS Attack

Looking at NS records again of all scanned top 500 Alexa domains after the recent Dyn DDoS attack now 13 of the previously 14 Dyn customers which previously relied solely on Dyn now switched away entirely or added additional non-Dyn DNS servers.

Who Switched Entirely,,,,,,

Who Switched to "Multi"-DNS,,,,


Here is a summary of the NS record changes. To automatically compare the server names all numbers in the DNS server names having been stripped in the following table:

SiteBefore (15.10.)After (24.10.)
The note-worthy non-changer is Twitter which is still exclusively at Dyn, eryone else seems to have mitigated. Some to using two providers, most of them switching to AWS DNS some to UltraDNS exclusively.

October 25, 2016 10:08 AM

October 24, 2016


Face to Face: Roadmap and Platform Updates

This is another in the series of posts about decisions we made at our face-to-face meeting a couple of weeks ago.

We updated the project roadmap.

I think the most important news here, is that our next release will include TLS 1.3. Our current plan is that this will be 1.1.1, which means that it is API-compatible with the current 1.1.0 release. This is really only possible because of the work we did on making most of the structure internals opaque. Also, since we are doing all of our work in public on our GitHub repository, we hope that the entire community will be able to “follow along at home” and help us improve the code. There will be more, much more, to say about this later.

We have also set ourselves major goals of increasing our code coverage and protocol-level testing. In addition, all new API’s must be documented, and the entire set of SSL API’s will also be documented. The script util/ lists them. If you’re inclined to help write some documentation, please feel free!

There are a variety of other crypto-nerd things we are looking at, so please see the roadmap for details.

The second point I want to mention in this post is that we have sketched out a platform policy. This is, admittedly, an overdue item from our first meeting two years ago. The first step will be to review all targets in the configuration scripts, and classify them according to the categories mentioned in the platform policy by our next release.

Which, in case I failed to mention it, will include TLS 1.3 :)

October 24, 2016 01:00 AM

October 22, 2016

LZone - Sysadmin

Changes after the Dyn DNS Outage

Looking at NS records today some of yesterdays affected companies decided to change things after the DoS on Dyn. As the NS record is not really a customer facing feature this is more an indication of Dyn's customers expectations. One could argue switching away from Dyn could mean fear of more downtimes to come.

Here is a summary of changed NS records so far:
SiteBefore (15.10.)After (22.10.)
awsdns awsdns anycastns*.org ultradns.* awsdns
I did only check some NS records for changes, just several of the top sites. There are two note-worthy non-changers being Twitter and Github though, everyone else seems to have mitigated. Some to using two providers, several switching to AWS DNS or UltraDNS exclusively.

October 22, 2016 07:45 PM

October 21, 2016

LZone - Sysadmin

Whom you can DDoS via DynDNS

After todays various affected major websites you might ask who is actually also affected together with the well known sites as Amazon, Twitter or Github.

Using the results of a monthly scan I automatically run on the top 500 Alexa sites it is easy to find out. The only thing you need to know is that the Dyn DNS server domain is (detailed results).

Top Affected Sites

Probably Different Impact

Note that not all of these sites were equally affected as some of them like Amazon are using multiple DNS providers. The Amazon main domains NS records do point to Dyn and UltraDNS. The same way probably none of the major adult sites was down as the also relied on at least two providers.

So while Amazon users probably got to the website after one DNS timeout and switching over to UltraDNS, Twitter and Github users were not so lucky and had to hope for Dyn to respond. It will be interesting to see if Twitter and Github will add a second DNS provider in the next time as a result of this.

The Need For "Multi-DNS"

Reading different reports on this incident it seems to me the headlines are focussing on those sites using just Dyns DNS and not on those having a "Multi-DNS".

Detailed results on who is using which DNS domain can be found in the monthly DNS usage index.

October 21, 2016 09:16 PM

October 20, 2016

That grumpy BSD guy

Is SPF Simply Too Hard For Application Developers?

The Sender Policy Framework (SPF) is unloved by some, because it conflicts with some long-established SMTP email use cases. But is it also just too hard to understand and to use correctly for application developers?

Here is a story involving a web-based contact form that indicates that this may be the case.

A wise man once noted that only two things in life are inevitable: death and taxes.

Just how taxes and interactions with the tax authorities are handled vary widely from jurisdiction to jurisdiction, but in Norway, where I live, the default mode of contact with the tax authorities for most of us is via web forms protected by sometimes cumbersome authentication procedures and the occasional alert via SMS text message to your phone.

And we're used to that the things just work with only occasional and very minor technical embarrassments for the people who operate the thing.

Then in August 2016, I tried to report a bug via the contact form at, the main tax authorities web site.

The report in itself was fairly trivial: The SMS alert I had just received about an invoice for taxes due contained one date, which turned out to be my birth date rather than the invoice due date. Not a major issue, but potentially confusing to the recipient until you actually log in and download the invoice as PDF and read the actual due date and other specifics.

The next time I checked my mail at, I found this bounce.

The meat of the message is
    SMTP error from remote mail server after RCPT TO::
    host []: 550 5.7.23 SPF validation failed   

which means that somebody, somewhere tried to send a message to, but the message could not be delivered because the sending machine did not match the published SPF data for the sender domain.

SPF expands to "Sender Policy Framework", which is one of the early and still valid ways for a domain to publish which hosts are supposed to be sending mail on the domain's behalf.

There is no requirement to refuse delivery of mail from a host that is not in a domain's SPF record, and emphatically so when no such record exists. On the other hand, when a domain does publish SPF data, rejecting mail from hosts that are not in the set published is a valid and accepted action.

What happened is actually quite clear even from the part quoted above: the host [] tried to deliver mail on my behalf (I received the bounce, remember), and since I have no agreement for mail delivery with the owners and operators of that host, it is not in's SPF record either, and the delivery fails.

The bounce message contained one Received: header which tells part of the story, and after decoding the MIME-encoded part it becomes clear that it contains my bug report with some slightly odd markup.

So it's clear that what produced the bounce was that the contact form had tried to deliver mail using the address I had given the contact form.

I can hear your groans all the way from there to here.

My original bug report had not been delivered, but I thought perhaps I could help them with the other problem, so I sent off a new message to the addresses that were given as contacts in the domain's whois info, taking care to send it from a machine that was properly authorized to send mail for my domain.

The full text of the message is available here, in Norwegian. The message includes the bounce with a short introduction that said essentially "this is the result of trying to submit a bug report via your web contact form. This is clearly an SPF problem, please look into that".

Then that message bounced, too.

The exact reason is likely buried in the configuration files for's MX, but it could be that it has some reservations against accepting mail from what it sees as a subdomain that doesn't have an MX record of its own.

Anyway, by then I had spent far too much time on this rather trivial matter while I was supposed to be doing other things, so I quickly wrote up a summary and sent it off to the same contact addresses, this time from my account, with the various accumulated data as attachments.

Then, as often happens when dealing with the authorities, nothing happened. For quite a while.

It was only on October 18, 2016 that my account received a reply from the support address, which quoted my last message in full, and added their resolution text saying:

Det kan se ut som du har Sender Policy Framework (SPF) aktivert på din mailserver, Det er en kjent svakhet ved vårt kontaktskjema at mailservere med SPF ikke støttes.


It looks like you have Sender Policy Framework (SPF) enabled on your mailserver, It is a known weakness of our contact form that mailervers with SPF are not supported.

Once again, I can hear and fully sympathize with your groans.

This is as perfectly wrong as can be, in fact the exact opposite of right.

The obvious answer should be, as you will agree if you're still reading: The form's developer should place the user's email address in the Reply-To: field, and send the message as its own, valid local user. That would solve the problem.

So I put it to you: Is SPF, the Sender Policy Framework, simply too hard for application developers to understand?

Yes, I'm well aware that SPF also breaks traditional forwarding of the type generally used by mailing lists and a few other use cases. Just how afraid should we be when those same developers come to do battle with the followup specifications such as DKIM and (shudder) the full DMARC specification?

I anticipate your feedback in comments, or if you have things you really want me to only paraphrase in public, email.

Update 2016-12-02: While looking for something else entirely, I stumbled across the DMARC FAQ, where the next to final entry describes the exact problem described in this column. Developers should perhaps learn to read FAQs.

To wit,

Why are messages I send on behalf of visitors to my website being blocked?

This depends on how you are sending these messages. If you are simply taking the website visitor’s email address and inserting it into the “From:” header of the message, and sending that message from your own servers, then you are impersonating the domain in their email address – in a way that is indistinguishable from spammers. These practices may have worked previously – in many cases for decades – because before spam became a literally overwhelming problem, nobody checked. The most successful initial mechanisms to combat such spam were IP address-based blocklists, and so your site may have been allowed to continue because it did not appear on such a list.
For the past decade, however email authentication has been introduced as a filtering mechanism, and is increasingly being used to detect and block such messages.As a best practice, you should instead be using a domain you control in the address of the From: header, and use mechanisms like SPF, DKIM, and DMARC to show that this message is authorized to use your domain. In the example below, the site visitor’s name is shown in the descriptive part of the From: header, and the Reply-To: header is set to the website visitor’s address, but the actual address used in the From: header clearly indicates that your website is the origin of the message.
From: "John Doe via the Example Website" <>
Reply-To: "John Doe" <>
To: "Bob Smith" <>
Subject: "An article I thought you would find interesting"

Taken from the DMARC.ORG FAQ: 4.23 Why are messages I send on behalf of visitors to my website being blocked? as of December 2. 2016.

by Peter N. M. Hansteen ( at October 20, 2016 07:10 PM