Planet SysAdmin

December 16, 2017

System Administration Advent Calendar

Day 16 - Inspec gives insight

By: Jamie Andrews (@vorsprungbike)

Edited By: Ben Cotton (@FunnelFiasco)

What is Inspec?

How often to do want to scan your servers, all of them, to check a library has the correct version or that the right disk clone is mounted?

Configuration control systems like Puppet, Chef and Ansible have become de rigueur in running even modest sized groups of servers. Configuration control offers centralised control over a state which is set by policy.

However, sometimes something happens to one or more servers which challenges this happy state of affairs.

There might be a change requirement that doesn't fit in with the way that the configuration system works. You might decide that part of the system should be managed in a different way. Someone in your team might "temporarily" disable the configuration agent and then not turn it back on again - configuration drift happens

When it does happen, inevitably there are hard to trace and puzzling to understand problems.

This might be a job for a quick shell script and some ssh magic. But often I've done this and then realised that it's difficult to get just right. Inspec is a framework for systematically carrying out these types of tests.

Inspec can be very useful in these circumstances. It works like a configuration management tool like Puppet or Chef in that it holds a model of what the correct configuration should be. However, it does not modify the configuration. Instead it tests the targeted systems for compliance to the correct configuration.

I have found this to be useful in the situation where there are many systems, just a few of which have a problem. Inspec can pinpoint the problem systems, given a configuration profile.

And of course Inspec can be used to proactively check servers to detect problems before they occur

Inspec is based on an earlier system called "serverspec" and it uses the ruby classes from the rspec package.

Although it is promoted by Chef as part of their wider product offering, it works just fine standalone and is fully open source

What I'm covering in the article

Below, I'll look at installing Inspec, making a simple project "profile", how it works in action and installing a third party security checking profile

Setting up and installing

The easiest way of installing it is to use a package. Packages for Redhat and varients, Ubuntu, Suse, MacOSX and MS Windows are available here

Inspec has a dependency on Ruby. The packages include a bundled version of Ruby that avoids compatiblity problems. If you already have ruby installed and want to use it then "gem install inspec" is available. See the github repo for more details

To check Inspec is installed try this command

inspec version

Which will come back with the current installed version

Inspec does not have to be installed on all the target nodes. I have it installed on one admin host with ssh access to everything else. This allows any profile rule sets to be tested on anything it can ssh to. No agents, install of ruby or anything else is required

Make your own profile

inspec works by using a profile directory with a a set of control files that contain tests The "init profile" cli command is used to make a new profile

To see a generic blank profile do

inspec init profile example

The output from this command is something like

$ inspec init profile example
WARN: Unresolved specs during Gem::Specification.reset:
      rake (>= 0)
WARN: Clearing out unresolved specs.
Please report a bug if this causes problems.
Create new profile at /home/jan/temp/example
 * Create directory libraries
 * Create file inspec.yml
 * Create directory controls
 * Create file controls/example.rb
 * Create file

It has made a set of files. The most interesting is "example/controls/example.rb"

This is very simple test that checks if /tmp exists, take a look at it

# encoding: utf-8
# copyright: 2017, The Authors

title 'sample section'

# you can also use plain tests
describe file('/tmp') do
  it { should be_directory }

# you add controls here
control 'tmp-1.0' do                  # A unique ID for this control
  impact 0.7                          # The criticality, if this control fails.
  title 'Create /tmp directory'       # A human-readable title
  desc 'An optional description...'
  describe file('/tmp') do            # The actual test
    it { should be_directory }

The tests can be declared as "plain tests" or "controls". Being a control adds some metadata which makes it easier to track the test within a set_con

The actual test assertions "it { should be_directory }" follow the rspec syntax. The tests operate on a resource type, in this case "file". There are many useful built in test resourse types, including

  • apache_conf
  • crontab
  • docker
  • etc_fstab
  • http

And a lot more, see

A more complex example

Here's a real test I wrote a couple of weeks ago to deal with a DNS configuration drift.

The old DNS servers had been retired but we noticed that some servers still mentioned servers on the old network.

# encoding: utf-8

title 'DNS'

# some crude way to build a list of network interfaces
eth_files= ['/etc/sysconfig/network-scripts/ifcfg-eth0']
eth_files << '/etc/sysconfig/network-scripts/ifcfg-eth1' if file('/etc/sysconfig/network-scri
eth_files << '/etc/sysconfig/network-scripts/ifcfs-ens32' if file('/etc/sysconfig/network-scr

control 'resolv' do                        # 
  impact 0.7                               # 
  title 'check old dns is not present'     # 
  desc 'old dns'
  describe file('/etc/resolv.conf') do     # The actual test
    its ('content')  { should_not match /193/ }

  eth_files.each do|ef| 
  describe file(ef) do
    its ('content') { should_not match /^DOMAIN=193/ }
    its ('content') { should_not match /^DNS[123]=193/ }


I won't explain exactly how it works, you can see that there are regexps in there and that a ruby "each do" construct is used.

To run the tests do

inspec exec tc-config myuser@myhostname

screen shot of it running

As mentioned above, Inspec does not correct these problems. It is great at one job: checking compliance. Once the problem is found then you will have to devise a good method for fixing it.

I when looking at large numbers of live servers I usually run it in a shell and then redirect the output to a file. Once the time consuming checking is done I look at the file The colourizing makes it easy to spot the non-compliant areas

In the above example, I found 12 servers non-compliant out of 146. Some problems were found where Puppet conflicted with Redhat system policy I devised a simple, non-idempotent bash script and applied it to the affected servers only. This was quicker and a more certain result than running it on all the servers. After the correction, I reran the Inspec profile to see that everything was ok

Check to check the check

Once you start trying to use your own tests there is always scope for typos or syntax error or other sorts of mayhem. Inspec tries to help with a static checker.

inspec check example

Comes back with a report of how many controls there are plus if everything is valid.

This feature is a great idea, especially for those of us that are only using this tool occassionally.

Ready to use sets of tests

As the Inspec system is supported and promoted by Chef there are a set of profiles ready made that perform various types of compliance. These can be downloaded and used, see

Installing and using the CIS DIL

One really useful profile is the CIS Distribution Independent Linux Benchmark

To try it, clone that github repo and then

 inspec check cis-dil-benchmark

It has 88 controls, many of which check multiple resources. On our LAN running it against a host took over 3 minutes.

The report it generates is interesting reading.

We will be using this profile for testing images generated with Packer and Puppet. The issues it reports will act as feedback for security improvements to our configuration scripts in Puppet and Packer

Further features

I am just scratching the surface with the many features that Inspec offers.
Please do look at for a fuller run down!


Thanks to Chef for supporting this excellent software and keeping it open source

Thanks to my employers, Institute of Physics Publishing for enabling me to try out cool stuff like this as part of my job.

by Christopher Webber ( at December 16, 2017 05:00 AM

Chris Siebenmann

Mozilla betrays Firefox users and its nominal principles

Unusually, I'll lead with what I think that you should do and explain the background afterward. In Firefox, go to about:preferences#privacy (the Privacy & Security tab of Firefox Preferences), scroll down almost to the bottom to the 'Firefox Data Collection and Use' section, and untick 'Allow Firefox to install and run studies'. To be safe, you probably want to also untick 'Allow Firefox to automatically send technical and interaction data to Mozilla'.

(Apparently you're going to have to re-check this set of preferences every so often, because Mozilla may be quietly re-enabling one or both every so often, perhaps on version upgrades.)

You should also go into about:addons (your set of extensions) and if there is something called 'Looking Glass', delete it (if possible, and that qualifier is part of the problem).

This preference controls something that Mozilla calls SHIELD studies (also), which Mozilla describes as:

SHIELD studies let you try out different features and ideas before they are released to all Firefox users. Using your feedback, we can make more informed decisions based on what you actually need.

It appears that this preference (and the broader preference for sending data to Mozilla) is default-enabled in new Firefox profiles and installs, although I can't be entirely sure since my environment is somewhat unusual.

This preference sounds relatively harmless, and probably it used to be. Then very recently Mozilla pushed out a nominal SHIELD experiment in the form of a new extension called 'Looking Glass 1.0.3', with the helpful extension description text of 'MY REALITY IS JUST DIFFERENT THAN YOURS'. Unsurprisingly, a lot of people who noticed this extension appearing were quite alarmed, because it certainly does look like malware from surface appearances, and to add to the fun there was no particular sign in Firefox of what it did. People on Reddit who noticed it resorted to reverse engineering the extension's code in an attempt to figure things out, eg.

What this extension actually is is some kind of promotion from the Mr Robot TV show:

[...] Firefox and Mr Robot have collaborated on a shared experience to further your immersion into the Mr Robot universe, also known as an Alternate Reality Game (ARG). [...]

From the outside, this collaboration certainly seems like it was actually 'Mr Robot gave Mozilla a bunch of money and Mozilla abused its browser experiments system to inflict a Mr Robot promotional extension on people'. To make it worse, this is not just any old extension; this is an extension that apparently silently alters text on web pages (some text, only for a while).

There are two serious problems here, which are nicely summarized by Steve Klabnik in a tweet:

How can we claim to be pro-privacy while surreptitiously installing software on people's computers?

More importantly, how did management not see this as a problem?

The first problem is that Mozilla betrayed both the trust of Firefox users and its theoretical principles by abusing a system created for experiments to improve Firefox in order to stealthily install an extension that was completely unrelated to this purpose and that was instead (to all appearances) made for commercial reasons. Even assuming that Mozilla asserts that users 'opted in' to SHIELD experiments in any meaningful sense (given that the preference appears to default to on), all of Mozilla's documentation for SHIELD experiments says one thing about their purpose, and it is not doing commercial promotions. Users were not presented with a preference saying 'allow Mozilla to opt you in to promotions from people who pay us for this', so they have not in any way consented to this use of the system.

In short, Mozilla did not push this extension because they had consent from Firefox users, they pushed it purely because they had been trusted with the technical capability to do so. That is why you should turn off this preference; Mozilla has demonstrated that they cannot be trusted with it. It does not mean what it claims to mean; in practice the SHIELD preference means 'you've agreed that Mozilla can push random extensions to you if they feel like it'. You should not agree to this. Mozilla has additional technical capabilities to push extensions to you, but if they do so they are now clearly and undeniably doing so against your expressed wishes.

The bigger problem is that as an organization, Mozilla still does not appear to understand that they've done something bad or why it was bad. Mozilla (the entity) approved and actively collaborated in this abuse of trust and still doesn't seem to think there was anything wrong with it, or perhaps it thinks that at most the problems are technical (such as the extension description should have been clear or had links or whatever). An organization that understand that it had betrayed the trust of Firefox users would have a very different reaction than what little Mozilla has done. An organization that got it would be apologizing publicly, auto-removing the extension, and so on. It would not be having its marketing people say in public:

Here is Mozilla’s response, courtesy of Chief Marketing Officer Jascha Kaykas-Wolff: “Firefox worked with the Mr. Robot team to create a custom experience that would surprise and delight fans of the show and our users. It’s especially important to call out that this collaboration does not compromise our principles or values regarding privacy. The experience does not collect or share any data. [...]

This is the words of an organization that does not understand what having the trust of Firefox users means. Since it does not understand this, Mozilla is not worthy of this trust and cannot be trusted with it, which is the second and larger reason why you should turn off this preference and leave it off, and why you should probably turn off the top level preference for having Firefox send data to Mozilla at all. Whether you leave 'phone home' options like auto-update and addon auto-updates enabled is up to you, but if you're conscientious about keeping on top of that sort of thing, well, I don't think we can trust Mozilla as much as we used to.

I'm extremely disappointed in Mozilla here. If Chrome had pulled this shit, I would be sad and annoyed but unsurprised, because Chrome has always ultimately been a tool of its corporate overlords. But Mozilla has always claimed to stand for something better than that, and sometimes it even has. I won't say that Mozilla doesn't now, exactly, but it is now clear that standing for something better is not a particularly deep value in the portions of Mozilla that call the shots and that there is a critical mass of people working at Mozilla who do not have any problem with this sort of thing.

(See also this Github issue, while it lasts, helpfully summarized here in a screenshot.)

PS: As a corollary to what I've said about Chrome here, if you suggest that the sensible reaction to this is switching from Firefox to Chrome, I will laugh a lot (but a bit sadly).

by cks at December 16, 2017 01:57 AM

December 15, 2017

Chris Siebenmann

How we automate acmetool

Acmetool is my preferred client for Let's Encrypt and the one we've adopted for our switch to Let's Encrypt at work. If you know acmetool, talking about automating it sounds like a contradiction in terms, because the entire design of acmetool is about automating everything already; you put it in cron (or more exactly you let it put itself in cron as part of setup with 'acmetool quickstart'), and then you forget about it. Perhaps you have to write a hook script or two or adjust file permissions because a daemon runs as a different user, but that should be it.

However, there are a few questions that acmetool will ask you initially and there's one situation where it has to ask you a new question during certificate renewal, as was pointed out by a commentator on my earlier entry:

Recently Let's Encrypt switched over to a new version of their user agreement (v1.2). As a result, all certificate renewals for old accounts started failing (because they had only agreed to v1.1), and I had to ssh to all our servers, interactively run acmetool, and re-confirm the signup process (agreement & email) myself.

Fortunately you can automate this too, and you should. Acmetool supports a response file, which contains answers to questions that acmetool may try to ask you during either installation or certificate renewal. We automate these questions by preinstalling a responses file in /var/lib/acme/conf, which makes 'acmetool quickstart' work without having to ask us anything. When Let's Encrypt updated their user agreement, we pushed out a new version of the responses file that auto-accepted it and so returned certificate renewals to working without any manual action.

(The first renewal attempt after Let's Encrypt's update reported errors, then we worked out what the problem was, updated the file, pushed out a new version, and everything was happy. My personal websites avoided the problem entirely because of the timing; I had a chance to update my own responses file before any of their renewals came up, and when renewal time hit acmetool was fine.)

The responses settings we use are:

"acme-enter-email": "<redacted>@<redacted>"
"acmetool-quickstart-choose-method": webroot
"acmetool-quickstart-webroot-path": "/var/www/.well-known/acme-challenge"
"acmetool-quickstart-install-cronjob": true
# add an additional line to accept any new user agreement

As shown in the example responses file, you can set additional parameters like the normal key type, RSA key size, and so on. We haven't bothered doing this so far, but we may in the future.

You could vary the email address if you wanted to (for example for different classes of machines). We don't bother, because it's mostly unimportant; in practice, all it gets is the occasional email about one of our generic test machine hostnames that hasn't renewed its certificate because we haven't been using that hostname for anything that needed one.

by cks at December 15, 2017 08:28 PM

LZone - Sysadmin

Solving rtl8812au installation on Ubuntu 17.10

After several fruitless attempts on getting my new dual band Wifi stick to work on my PC I went the hard way to compiling the driver.

Figuring out which driver to use

As drivers do support device ids the first thing is to determine the id
$ lsusb
Bus 002 Device 013: ID 0bda:a811 Realtek Semiconductor Corp. 
So the id being "0bda:a811" you can search online for a list of driver names. Google suggests rtl8812au as related searches...

Finding a source repo

At github you can find several source repos for the rtl8812au driver in of different age. It seems that Realtek is supplying the source on some driver CDs and different people independently put them online. The hard part is to find the most recent one, as only this has fixes for recent kernels.

One with patches for kernel 4.13.x is the one from Vital Koshalev which has already a PR against the most commonly referenced repo of diederikdehaas which doesn't work yet!.

Getting the right source

So fetch the source as following
git clone
git checkout driver-4.3.22-beta-mod

Compilation + Installation of rtl8812AU

The instructions are from the and are to be run as root:
mkdir /usr/src/${DRV_NAME}-${DRV_VERSION}
git archive driver-${DRV_VERSION} | tar -x -C /usr/src/${DRV_NAME}-${DRV_VERSION}
dkms add -m ${DRV_NAME} -v ${DRV_VERSION}
dkms build -m ${DRV_NAME} -v ${DRV_VERSION}
dkms install -m ${DRV_NAME} -v ${DRV_VERSION}

Loading rtl8812AU

If everything worked well you should now be able to issue
modprobe 8812au
Note the missing "rtl"!!!

If the module loads, but the wifi stick doesn't work immediately it might be that the rtlwifi driver is preventing the self-compiled module from working. So remove it with
rmmod rtlwifi
It will complain about dependencies. You need to rmmod those too. Afterwards the new driver should load properly. To make disabling rtlwifi persistent add it to the modprobe blacklist:
echo "blacklist rtlwifi" >>/etc/modprobe.d/blacklist.conf
Please leave feedback on the instructions if you have problems!

December 15, 2017 08:09 PM

System Administration Advent Calendar

Day 15 - A DevOps Christmas Carol

By: Emily Freeman (@editingemily)
Edited By: Corey Quinn (@quinnypig)

The DevOps Christmas Carol is a bastardized, satirical version of Charles Dickens’ iconic work, A Christmas Carol.

It’s Christmas Eve and we find Scrooge, a caricature of a San Francisco-based, VC-backed tech startup CEO haunted by Peter Drucker’s ghost — who warns him of the visits of three ghosts: the Ghost of DevOps Past, the Ghost of DevOps Present and the Ghost of DevOps Yet to Come. (Victorians were seriously wordy.)

Scrooge’s company,, has adopted DevOps, but their Tiny Tim app is still in danger of falling over. And Scrooge is still complaining about that AWS bill.

I want you to laugh at the absurdity of our industry, remember the failures of yesterday, learn the lessons of today and embrace the challenges of tomorrow.

Above all else, Merry Christmas, Chag Sameach and Happy New Year. May everyone’s 2018 be better than the dumpster fire that was this year.


Old Peter Drucker was as dead as a doornail. (I don’t exactly know what’s dead about a doornail, but we’ll go with it.)

Drucker had been dead for many years. Every DevOpsDays deck included a Drucker quote. And many shots had been consumed playing the Drunker Drucker drinking game.

Scrooge was your average SF CEO. His grandfather was business partners with Drucker and Scrooge continues to worship the Drucker deity.

It was Christmas Eve and Scrooge sat in his glass-enclosed office drinking artisanal, small-batch coffee. His cool disposition was warmed thinking about the yacht he would buy when his startup,, IPO’d.

Sure, they had unlimited vacation. But no one ever took it — even on Christmas Eve. His employees loved working that much.

He watched the developers and operations folks gather for standup in the “Innovate” conference room. It was great to see the teams working together. After all, they had spent $180,000 on a consultant to “do the DevOps.”

“A merry Christmas, uncle!” His sister had forced Scrooge to hire his cheery nephew as an intern.

“Bah!” said Scrooge, “Humbug!”

“Christmas a humbug, uncle!” said Scrooge’s nephew. “You don’t mean that, I am sure?”

“I do,” said Scrooge. “Merry Christmas! What reason have you to be merry? I pay you $19 an hour and you live in a closet with 4 other men in Oakland.”

The receptionist quietly tapped the glass door. “Two men from Homeless Helpers of San Francisco are here to see you.” Scrooge waved them in.

“At this festive season of the year, Mr. Scrooge,” said the gentleman, “it is more than usually desirable that we should make some slight provision for the poor and destitute, who suffer greatly at the present time.”

“I thought we were just moving them to San Jose,” replied Scrooge.

Seeing clearly it would be useless to pursue their point, the two men withdrew, mumbling about hate in the Trump era.

The day drew to a close. Scrooge dismounted from his Aeron chair and put on his sweatshirt and flat-brimmed hat.

“You’ll want all day tomorrow, I suppose?” said Scrooge to his employees, whose standing desks were packed as tightly as an Amazon box filled with Cyber Monday regret. “I suppose you must have the whole day. But if the site goes down, I expect all of you to jump on Slack and observe helplessly as Samantha restarts the servers.”

Scrooge took his melancholy dinner at his usual farm-to-table tavern. Walking home on the busy sidewalk, Scrooge approached his doorman only to see Drucker, staring at him. His demeanor vacant, his body translucent. Scrooge was shook. He brushed past the ghostly figure and hurried toward the elevator.

Satisfied he had too many glasses of wine with dinner, Scrooge shut the door and settled in for the night. Suddenly, Siri, Alexa, Cortana and Google joined together to make a horrific AI cacophony.

This was followed by a clanking noise, as if someone were dragging a heavy chain over the hardwood floors. Scrooge remembered to have heard that ghosts in haunted houses were described as dragging chains.

The bedroom door flew open with a booming sound and then he heard the noise much louder, coming straight towards his door.

His color changed when he saw the same face. The very same. Drucker, in his suit, drew a chain clasped about his middle. It was made of servers, CPUs, and endless dongles.

Scrooge fell upon his knees, and clasped his hands before his face. “Mercy!” he said. “Dreadful apparition, why do you trouble me?”

“I wear the chain I forged in life. Oh! Captive, bound, and double-ironed,” cried the phantom.

“But you were always a good man of business and operations, Peter,” faltered Scrooge, who now began to apply this to himself.

“Business!” cried the Ghost, wringing his hands again. “Mankind was my business. Empathy, compassion and shared documentation were all my business. Operations was but a drop of water in the ocean of my business! You will be haunted,” resumed the Ghost, “by three spirits.”

“I—I think I’d rather not,” said Scrooge.

“Without their visits,” said the Ghost, “you cannot hope to shun the path of waterfall development, silos and your startup’s slow descent into obscurity. Expect the first tomorrow.”

With that, Drucker’s ghost vanished.


It was dark when Scrooge awoke from his disturbed slumber. So dark he could barely distinguish transparent window from opaque wall. The chimes of a neighboring church struck the hour and a flash lit up the room in an instant. Scrooge stared face-to-face with an unearthly visitor.

It was a strange figure—small like a child, but old. Its hair was white with age but it’s face had not a wrinkle.

“Who, and what are you?” Scrooge demanded.

“I am the Ghost of DevOps Past.”

“Long past?” inquired Scrooge, observant of its dwarfish nature.

“DevOps is only like 10 years old. Do you even read Hacker News?” He paused. “Rise! And walk with me!”

The Ghost took his hand and together they passed through the wall, and stood upon an open convention floor.

“Good heaven! I went to this conference!” said Scrooge.

“The open space track is not quite deserted,” said the Ghost. “A solitary man, neglected by his friends, is left there still.”

In a corner of the conference, they found a long, bare, melancholy room. In a chair, a lonely man was reading near the feeble light of his phone. Scrooge wept to see poor Andrew Clay Shafer in a room alone.

“Poor man!” Scrooge cried. “I wish,” Scrooge muttered, putting his hand in his pocket, “but it’s too late now.”

“What is the matter?” asked the Spirit.

“Nothing,” said Scrooge. “Nothing. There was a developer who asked about ops yesterday. I should like to have given him something. That’s all.”

The Ghost smiled thoughtfully, and waved its hand: saying as it did so, “Let us see another conference!”

The room became a little darker. The panels shrunk and the windows cracked. They were now in the busy thoroughfares of a city, where shadowy passengers passed along the narrow streets and Medieval architecture.

The Ghost stopped at a certain door and ushered Scrooge in. “Why it’s old Patrick Dubois! Bless his heart.”

“Another round!” Dubois announced.

“This is the first DevOpsDays afterparty in Ghent,” explained the Ghost.

Away they all went, twenty couple at once toward the bar. Round and round in various stages of awkward grouping.

“Belgians sure know how to party,” said Scrooge.

“Sure. But it’s a tech conference so it’s still pretty awkward,” remarked the Ghost. “A small matter,” it continued, “to make these silly folks so full of gratitude. It only takes a t-shirt and a beer.”

“Small!” echoed Scrooge.

“Why! Is it not? He has spent but a few dollars of your mortal money. Is that so much he deserves this praise?”

“It isn’t that, Spirit. He has the power to render us happy or unhappy. To make our service light or burdensome. Say that his power lies in words and looks, in things so slight and insignificant that it is impossible to add and count ’em up. The happiness he gives, is quite as great as if it cost a fortune.”

He felt the Spirit’s glance, and stopped.

“What is the matter?” asked the Ghost.

“Nothing particular,” said Scrooge.

“Something, I think?” the Ghost insisted.

“No,” said Scrooge, “No, I should like to be able to say a word or two to my employees just now. That’s all.”

“My time grows short,” observed the Spirit. “Quick!”

It produced an immediate effect. Scrooge saw himself. He was not alone but with two former employees. The tension in the room was overwhelming.

“The code didn’t change,” explained the developer. “There’s no way we caused this out—”

“That’s bullshit!” interjected the SRE. “There was a deploy 10 minutes before the site went down. Of course that’s the issue here. These developers push out crappy code.”

“At least we can code,” replied the developer, cruelly.

“No more,” cried Scrooge. “No more. I don’t wish to see it.”

But the relentless Ghost pinioned him in both his arms and forced him to observe.

“Spirit!” said Scrooge in a broken voice, “remove me from this place. Remove me. I cannot bear it!”

He turned upon the Ghost, and seeing that it looked upon him with a face in which some strange way there were fragments of all the faces it had shown him. “Leave me! Take me back. Haunt me no longer!” Scrooge wrestled with the spirit with all his force. Light flooded the ground and he found himself exhausted. Overcome with drowsiness, Scrooge fell into a deep sleep.


Awaking in the middle of a prodigiously tough snore and sitting up, Scrooge was surprised to be alone. Now, being prepared for almost anything, he shuffled in his slippers to the bedroom door. The moment Scrooge’s hand was on the lock, a strange voice called him by his name and bade him enter. He obeyed.

It was his own room, there was no doubt. But it had undergone a surprising transformation. The walls and ceiling were hung with living green and bright gleaming berries glistened.

“Come in!” exclaimed the Ghost. “Come in and know me better, man!”

Scrooge entered timidly and hung his head. The Spirit’s eyes were clear and kind but he didn’t like to meet them.

“I am the Ghost of DevOps Present,” said the Spirit. “Look upon me!”

Scrooge reverently did so. It was clothed in one simple green robe, bordered with white fur. This garment hung so loosely on the figure that its capacious breast was bare.

“You have never seen the like of me before!” exclaimed the Ghost.

“Never,” Scrooge answered. “But you may want to cover up a bit. This is breaking some kind of code of conduct and men are getting in trouble for this kind of thing these days.”

“Touch my robe!”

“OK, this is getting awkward. And inappropriate. Seriously, you can’t expose yourself like this. House of Cards was canceled because of people like you. It’s not cool, man.”

“Touch my robe!” the Ghost bellowed.

Scrooge did as he was told, and held it fast.

The room vanished instantly. They found themselves in the conference room at The Ghost sprinkled incense from his torch on the heads of the employees sat around the table. It was a very uncommon kind of torch, for once or twice when there were angry words between some men, he shed a few drops of water on them from it, and their good humour was restored directly.

“Is there a peculiar flavor in what you sprinkle from your torch?” asked Scrooge.

“There is. My own.”

“OK, buddy, we gotta work on the subtle sexual harassment vibe you’re working with.”

Scrooge’s employees at the table began to argue about CI/CD, pipelines and testing for the new Tiny Tim app.

“We need to be deploying every ten minutes. At a minimum. That’s what Netflix does,” said Steve, confidently.

“We use Travis CI. What if we had commits immediately deploy to production?” asked Tony.

“Um, that’s a terrible idea. You want the developers tests be the only line of defense against site outages? I don’t want to be on call during that disaster.”

“There’s no reason to get an attitude, Steve.”

“Well, you’re suggesting that developers should be trusted to deploy their own code.”

“That’s exactly what I’m saying.”

“That’ll never work. QA and security need to review everything before it’s pushed out.”

“Yea, but there’s this whole concept of moving things to the left. Where ops, security, QA are all involved in feature planning and the developer architects the code with their concerns in mind. That way, we don’t have Amy spending a full week on a feature only to have security kick it back.”

“I think that’s exactly how it should work. Developers need to code better.”

“‘Coding better’ is not actionable or kind. And that kind of gatekeeping process creates silos and animosity. People will start to work around each other.”

“We’ll just add more process to prevent that.”

“Never underestimate someone in tech’s ability to use passive-aggressiveness in the workplace. Listen. Scrooge wants Tiny Tim to be reliable, agile, testable, maintainable and secure.”

“Well, that’s impossible.”

“Pfff,” remarked Scrooge. “You’re fired.”

The Ghost sped on. It was a great surprise to Scrooge, while listening to the moaning of the wind, and thinking what a solemn thing it was to move on through the lonely darkness over an unknown abyss, whose depths were secrets as profound as Death. This was the ever-present existential crisis of life. It was a great surprise to Scrooge, while thus engaged, to hear a hearty laugh.

“Ha, ha! Ha, ha, ha, ha!” laughed Scrooge’s nephew.

“He said that Christmas was a humbug, as I live!” cried Scrooge’s nephew. “He believed it too! I am sorry for him. I couldn’t be angry with him if I tried. Who suffers by his will whims! Himself, always. He takes it into this head to dislike us. No one at likes him. The product owner for Tiny Tim is about to quit and Scrooge has no idea!”

Scrooge was taken aback. He built a great team. They loved him. Adored him, even. Or so he thought. It’s true he hadn’t taken time to talk to any of them in several months, but everything was going so well. They were only two months from the launch of Tiny Tim.

His nephew continued, “We’re all underpaid and overworked. Scrooge is constantly moving the goalpost. He expects us to be perfect.”

The Ghost grew older, clearly older.

“Are spirits’ lives so short?” asked Scrooge.

“My life upon this globe is very brief,” replied the Ghost. “It ends tonight. Hark! The time is drawing near.”

The Ghost parted the folds of its robe. “Look here. Look, look, down here!”

“You. Have. To. Stop. With. This.” sighed Scrooge.

From its robe it brought two children. Scrooge started back, appalled at the creatures. “Spirit! Are they yours?”

“They are Op’s,” said the Spirit, looking down upon them. “And they cling to me. This boy is Serverless. This girl is Lambda.”

The bell struck twelve. Scrooge looked about him for the Ghost and saw it not. Lifting up his eyes, he beheld a solemn Phantom, draped and hooded, coming, like a mist along the ground, towards him.


The Phantom slowly, gravely, silently approached. When it came near him, Scrooge bent down upon his knee; for in the very air through which this Spirit moved it seemed to scatter gloom and mystery.

It was shrouded in a deep black garment, which concealed its head, its face, its form, and left nothing of it visible save on outstretched hand.

“I am in the presence of the Ghost of DevOps Yet To Come?” said Scrooge.

The Spirit answered not, but pointed onward with its hand.

“Lead on!” said Scrooge. “The night is waning fast, and it is precious time to me, I know. Lead on, Spirit!”

The Phantom moved away as it had come towards him. Scrooge followed in the shadow of its dress, which bore him up, he thought, and carried him along.

They scarcely seemed to enter the city. For the city rather seemed to spring up about them and encompass them of its own act. The Spirit stopped beside one little knot of business men. Observing that the hand was pointed to them, Scrooge advanced to listen to their talk.

“No,” said a great fat man with a monstrous chin, “I don’t know much about it, either way. I only know it’s dead.”

“When did it happen?” inquired another.

“Yesterday, I believe.”

“How much of a down round was it? Just a stop gap?”

“No. Investors have lost faith. Scrooge sold this Tiny Tim app hard. Bet all of on disrupting the DevSecDataTestOps space. He could only raise half of what they did in Series A.”

“It’s over,” remarked another.

“Oh yea, he’s done.”

This was received with a general laugh.

The Phantom glided on and stopped once again in the office of — it’s finger pointed to three employees fighting over who could take home the Yama cold brew tower — one nearly toppling it in the process. There were movers carrying standing desks out and employees haggling over their desks and chairs.

Scrooge watched as his office — his company — was systematically dismantled, broken down and taken away piece-by-piece. His own office was nearly empty save for a single accounting box — a solemn reminder of Scrooge’s priorities in his work.

“Spectre,” said Scrooge, “something informs me that our parting moment is at hand. I know it, but I know not how.”

The Ghost of DevOps Yet To Come conveyed him, as before. The Spirit did not stay for anything, but went straight on, as to the end just now desired, until besought by Scrooge to tarry for a moment.

The Spirit stopped; the hand was pointed elsewhere.

A pile of old computers lay before him. The Spirit stood among the aluminum graves, and pointed down to one. He advanced toward it trembling. The Phantom was exactly as it had been, but he dreaded that he saw new meaning in its solemn shape.

Scrooge crept towards it, and following the finger, read upon the screen, HUMBUG.LY — THIS WEBPAGE PARKED FREE, COURTESY OF GODADDY.COM.

“No, Spirit! Oh no, no!”

The finger was still there.

“Spirit!” he cried, tight clutching at its robe, “hear me! I am not the man I was. I will not be the man I have been but for this. I will honor DevOps in my heart, and try to keep it all the year. I will live in the Past, the Present and the Future. The Spirits of all three shall strive within me. I will not shut out the lessons they teach.”

Holding up his hands in a last prayer to have his fate reversed, he saw an alteration in the Phantom’s hood and dress. It shrunk, collapsed and dwindled down into a bedpost.


Yes! The bedpost was his own. The bed was his own, the room was his own.

“Oh Peter Drucker! Heaven, and DevOps be praised for this! I don’t know what to do!” cried Scrooge, laughing and crying in the same breath. Really, for a man who had been out of practice for so many years, it was a splendid laugh, a most illustrious laugh.

Scrooge hopped on Amazon with haste. He bought The Phoenix Project and The DevOps Handbook and had it delivered by drone within the hour.

“I’ll get a copy for every one of my employees!” exclaimed Scrooge.

He hopped into his Tesla and drove to his nephew’s apartment. Greeted by one of the 5 roommates, he asked to see his nephew.

“Fred!” said Scrooge.

“Why bless my soul!” cried Fred, “who’s that?”

“It’s I. Your uncle Scrooge. I have come to dinner. Will you let me in, Fred?”

The odd group had a twenty-something Christmas dinner and played Cards Against Humanity. Wonderful party, wonderful games, wonderful happiness!

But he was early at the office the next morning. If he could only be there first, and catch his employees coming in late!

The last employee stumbled in. “Hello!” growled Scrooge, in his accustomed voice, as near as he could feign it. “What do you mean by coming here at this time of day?”

“I am very sorry, sir. I am behind my time.”

“You are?” repeated Scrooge. “Yes. I think you are. Step this way, sir, if you please.”

“It’s only once a year, sir,” pleaded Bob. “It should not be repeated. Besides, we’re supposed to have flexible work hours. We have a ping-pong table for God’s sake!”

“Now, I’ll tell you what, my friend,” said Scrooge, “I am not going to stand this sort of thing any longer. And therefore,” he continued, leaping from his Aeron, “I am about to raise your salary! All of your salaries!”

Bob trembled. “Does… does this mean we’re doing that open salary thing?”

“No, don’t push it, Bob. That’s for hippies,” replied Scrooge. “A merry Christmas!”

Scrooge was better than his word. He did it all and infinitely more. The Tiny Tim app was finished on time, adopted a DevOps culture, developers stopped being assholes and ops folks got more sleep.

Scrooge had no further experience with Spirits and it was always said of him that he knew how to keep DevOps well, if any man alive possessed the knowledge. May that truly be said of all of us! And so, Ops bless us, everyone!

by Christopher Webber ( at December 15, 2017 05:00 AM

December 14, 2017

Chris Siebenmann

How Python makes it hard to write well structured little utilities

I'll start with the tweets, where I sort of hijacked something glyph said with my own grump:

@glyph: Reminder: even ridiculous genius galaxy brain distributed systems space alien scientists can't figure out how to make and ship a fucking basic python executable. Not only do we need to make this easy we need an AGGRESSIVE marketing push once actually viable usable tools exist. <link>

@thatcks: As a sysadmin, it’s a subtle disincentive to writing well structured Python utilities. The moment I split my code into modules, my life gets more annoying.

The zen of Python strongly encourages using namespaces, for good reasons. There's a number of sources of namespaces (classes, for example), but (Python) modules are one big one. Modules are especially useful in their natural state because they also split up your code between multiple files, leaving each file smaller, generally more self-contained, and hopefully simpler. With an 'everything in one file' collection of code, it's a little too easy to have it turn mushy and fuzzy on you, even if in theory it has classes and so on.

This works fine for reasonable sized projects, like Django web apps, where you almost certainly have a multi-stage deployment process and multiple artifacts involved anyway (this is certainly the case for our one Django app). But speaking from personal experience, it rapidly gets awkward if you're a sysadmin writing small utility programs. The canonical ideal running form of a small utility program is a single self-contained artifact that will operate from any directory; if you need it somewhere, you copy the one file and you're done.

(The 'can be put anywhere' issue is important in practice, and if you use modules Python can make it annoying because of the search path issue.)

One part of this awkwardness is my long standing reluctance to use third-party modules. When I've sometimes given in on that, it's been for modules that were already packaged for the only OS where I intended to use the program, and the program only ever made sense to run on a few machines.

But another part of it is that I basically don't modularize the code I write for my modest utilities, even when it might make sense to break it up into separate little chunks. This came into clear view for me recently when I wound up writing the same program in Python and then Go (for local reasons). The Python version is my typical all in one file small utility program, but the Go version wound up split into seven little files, which I think made each small concern easier for me to follow even if there's more Go code in total.

(With that said, the Go experience here has significant warts too. My code may be split into multiple files, but it's all in the same Go package and thus the same namespace, and there's cross-contamination between those files.)

I would like to modularize my Python code here; I think the result would be better structured and it would force me to be more disciplined about cross-dependencies between bits of the code that really should be logically separate. But the bureaucracy required to push the result out to everywhere we need it (or where I might someday want it) means that I don't seriously consider it until my programs get moderately substantial.

I've vaguely considered using zip archives, but for me it's a bridge too far. It's not just that this requires a 'compilation' step (and seems likely to slow down startup even more, when it's already too slow). It's also that, for me, packing a Python program up in a binary format loses some of the important sysadmin benefits of using Python. You can't just look at a zip-packaged Python program to scan how it works, look for configuration variables, read the comment that tells you where the master copy is, or read documentation at its start; you have to unpack your artifact first. A zip archive packed Python utility is less of a shell script and more of a compiled binary.

(It also seems likely that packing our Python utility programs up in zip files would irritate my co-workers more than just throwing all the code into one plain-text file. Code in a single file is just potentially messy and less clear (and I can try to mitigate that); a zip archive is literally unreadable as is no matter what I do.)

by cks at December 14, 2017 10:46 PM

The Lone Sysadmin

Fixing Veeam “Can’t Delete Replica When It Is Being Processed” Errors

I’ve used Veeam Backup & Replication for a long time now, and when we restructure storage, redeploy VMs, or change our replication jobs we sometimes get situations where we get the error: Error: Can't delete replica when it is being processed Here’s how I fix it. As always, your mileage may vary, and free advice […]

The post Fixing Veeam “Can’t Delete Replica When It Is Being Processed” Errors appeared first on The Lone Sysadmin. Head over to the source to read the full post!

by Bob Plankers at December 14, 2017 09:10 PM

System Administration Advent Calendar

Day 14 - Pets vs. Cattle Prods: The Silence of the Lambdas

By: Corey Quinn (@quinnypig)
Edited By: Scott Murphy (@ovsage)

“Mary had a little Lambda
S3 its source of truth
And every time that Lambda ran
Her bill went through the roof.”

Lambda is Amazon’s implementation of a concept more broadly known as “Functions as a Service,” or occasionally “Serverless.” The premise behind these technologies is to abstract away all of the infrastructure-like bits around your code, leaving the code itself the only thing you have to worry about. You provide code, Amazon handles the rest. If you’re a sysadmin, you might well see this as the thin end of a wedge that’s coming for your job. Fortunately, we have time; Lambda’s a glimpse into the future of computing in some ways, but it’s still fairly limited.

Today, the constraints around Lambda are somewhat severe.

  • You’re restricted to writing code in a relatively small selection of languages– there’s official support for Python, Node, .Net, Java, and (very soon) Go. However, you can shoehorn in shell scripts, PHP, Ruby, and others. More on this in a bit.
  • Amazon has solved the Halting Problem handily– after a certain number of seconds (hard capped at 300) your function will terminate.
  • Concurrency is tricky: it’s as easy to have one Lambda running as a time as it is one thousand. If they each connect to a database, it’s about to have a very bad day. (Lambda just introduced per-function concurrency, which smooths this somewhat.)
  • Workflows around building and deploying Lambdas are left as an exercise for the reader. This is how Amazon tells developers to go screw themselves without seeming rude about it.
  • At scale, the economics of Lambda are roughly 5x the cost of equivalent compute in EC2. That said, for jobs that only run intermittently, or are highly burstable, the economics are terrific. Lambdas are billed in Gigabyte-Seconds (of RAM).
  • Compute and IO scale linearly with the amount of RAM allocated to a function. Exactly what level maps to what is unpublished, and may change without notice.
  • Lambda functions run in containers. Those containers may be reused (“warm starts”) and be able to reuse things like database connections, or have to be spun up from scratch (“cold starts”). It’s a grand mystery, one your code will have to take into account.
  • There are a finite list of things that can trigger Lambda functions. Fortunately, cron-style schedules are now one of them. The Lambda runs
  • within an unprivileged user account inside of a container. The only place inside of this container where you can write data is /tmp, and it’s limited to 500mb.
  • Your function must fit into a zip file that’s 50MB or smaller; decompressed, it must fit within 250MB– including dependencies.

Let’s focus on one particular Lambda use case: replacing the bane of sysadmin existence, cron jobs. Specifically, cron jobs that affect your environment beyond “the server they run on.” You still have to worry about server log rotation; sorry.

Picture being able to take your existing cron jobs, and no longer having to care about the system they run on. Think about jobs like “send out daily emails,” “perform maintenance on the databases,” “trigger a planned outage so you can look like a hero to your company,” etc.

If your cron job is written in one of the supported Lambda languages, great– you’re almost there. For the rest of us, we probably have a mashup of bash scripts. Rejoice, for hope is not lost! Simply wrap your terrible shell script (I’m making assumptions here– all of my shell scripts are objectively terrible) inside of a python or javascript caller that shells out to invoke your script. Bundle the calling function and the shell script together, and you’re there. As a bonus, if you’re used to running this inside of a cron job, you likely have already solved for the myriad shell environment variable issues that bash scripts can run into when they’re called by a non-interactive environment.

Set your Lambda trigger to be a “CloudWatch Event - Scheduled” event, and you’re there. It accepts the same cron syntax we all used to hate but have come to love in a technical form of Stockholm Syndrome.

This is of course a quick-and-dirty primer for getting up and running with Lambda in the shortest time possible– but it gives you a taste of what the system is capable of. More importantly, it gives you the chance to put “AWS Lambda” on your resume– and your resume should always be your most important project.

If you have previous experience with AWS Lambda and you’re anything like me, your first innocent foray into the console for AWS Lambda was filled with sadness, regret, confusion, and disbelief. It’s hard to wrap your head around what it is, how it works, and why you should care. It’s worth taking a look at if you’ve not used it– this type of offering and the design patterns that go along with it are likely to be with us for a while. Even if you’ve already taken a dive into Lambda, it’s worth taking a fresh look at– the interface was recently replaced, and the capabilities of this platform continue to grow.

by Christopher Webber ( at December 14, 2017 05:01 AM

December 12, 2017


Digital Doorknobs

Doorknobs are entering the Internet of (unsecured) Things.

However, they've been there for quite some time already. As anyone who has been in a modern hotel any time in the last 30 years knows, metal keys are very much a thing of the past. The Hotel industry made this move for a lot of reasons, a big one being that plastic card is a lot easier to replace than an actual key.

They've also been there for office-access for probably longer, as anyone who has ever had to waive their butt or purse at a scan-pad beside a door knows. Modern versions are beginning to get smartphone hookups, allowing the an expensive (but employee-owned) smartphone with an app on it and enabled Bluetooth to replace that cheap company-owned prox-pass.

They're now moving into residences, and I'm not a fan of this trend. Most of my objection comes from being in Operations for as long as I have. The convenience argument for internet-enabling your doorknob is easy to make:

  • Need emergency maintenance when you're on vacation? Allow the maintenance crew in from your phone!
  • Assign digital keys to family members you can revoke when they piss you off!
  • Kid get their phone stolen? Revoke the stolen key and don't bother with a locksmith to change the locks!
  • Want the door to unlock just by walking up to it? Enable Bluetooth on your phone, and the door will unlock itself when you get close!

This is why these systems are selling.


I'm actually mostly OK with the security model on these things. The internals I've looked at involved PKI and client-certificates. When a device like a phone gets a key, that signed client-cert is allowed to access a thingy. If that phone gets stolen, revoke the cert at the CA and the entire thing is toast. The conversation between device and the mothership is done over a TLS connection using client-certificate authentication, which is actually more secure than most banks website logins.

The handshake over Bluetooth is similarly cryptoed, making it less vulnerable to replay attacks.

Where we run into problems is the intersection of life-safety and the flaky nature of most residential internet connections. These things need to be able to let people in the door even when CentryLink is doing that thing it does. If you err on the side of getting in the door, you end up caching valid certs on the lock-devices themselves, opening them up to offline attacks if you can jam their ability to phone home. If you err on the side of security, an internet outage is a denial of access attack.

The Real Objection

It comes down to the differences in the hardware and software replacement cycles, as well as certain rare but significant events like a change of ownership. The unpowered deadbolt in your front door could be 20 years old. It may be vulnerable to things like bump-keys, but you can give the pointy bits of metal (keys) to the next residents on your way to your new place and never have to worry about it. The replacement cycle on the whole deadbolt is probably the same as the replacement cycle of the owners, which is to say 'many years'. The pin settings inside the deadbolt may get changed more often, but the whole thing doesn't get changed much at all.

Contrast this with the modern software ecosystem, where if your security product hasn't had an update in 6 months it's considered horribly out of date. At the same time, due to the iterative nature of most SaaS providers and the APIs they maintain, an API version may get 5 years of support before getting shut down. Build a hardware fleet based on that API, and you have a hardware fleet that ages at the rate of software. Suddenly, that deadbolt needs a complete replacement every 5 years, and costs about 4x what the unpowered one did.

Most folks aren't used to that. In fact, they'll complain about it. A lot.

There is another argument to make about embedded system (that smart deadbolt), and their ability to handle the constantly more computationally expensive crypto-space. Not to mention changing radio-specs like Bluetooth and WiFi that will render old doorknobs unable to speak to the newest iPhone. Which is to say, definitely expect Google and Apple to put out doorknobs in the not too distant future. Amazon is already trying.

All of this make doorknob makers salivate, since it means more doorknobs will be sold per year. Also the analytics over how people use their doors? Priceless. Capitalism!

It also means that doorknob operators, like homeowners, are going to be in for a lot more maintenance work to keep them running. Work that didn't used to be there before. Losing a phone is pretty clear, but what happens when you sell your house?

You can't exactly 'turn over the keys' if they're 100% digital and locked into your Google or Apple identities. Doorknob makers are going to have to have voluntary ownership-transfer protocols.

Involuntary transfer protocols are going to be a big thing. If the old owners didn't transfer, you could be locked out of the house. That could mean a locksmith coming in to break in to your house, and having to replace every deadbolt in the place with brand new. Or it could mean arguing with Google over who owns your home and how to prove it.

Doing it wrong has nasty side-effects. If you've pissed off the wrong people on the internet, you could have griefers coming after your doorknob provider, and you could find yourself completely locked out of your house. The more paranoid will have to get Enterprise contracts and manage their doorknobs themselves so they have full control over the authentication and auth-bypass routes.

Personally, I don't like that added risk-exposure. I don't want my front door able to be socially engineered out of my control. I'll be sticking with direct-interaction token based authentication methods instead of digitally mediated digital token auth methods.

by SysAdmin1138 at December 12, 2017 10:36 PM

Everything Sysadmin

DevOpsDays New York City 2018 Speakers Announced!

Exciting news from the D-O-D-NYC committee!

  • Speakers announced. Wow! I've never seen such an amazing lineup of speakers!
  • The best of the best. The committee this year was flooded with so many amazing proposals but sadly it is a 2-day conference so they had to be very selective. Who benefits? You!
  • Early bird discount ends on Friday. Register soon and save!

DevOpsDays-NYC 2018 is Thu/Fri January 18-19, 2018 in midtown Manhattan. Easy to get to via all forms of public transportation.

For more information:

by Tom Limoncelli at December 12, 2017 07:49 PM

December 11, 2017

“du –max-depth” alternative on Mac OSX

The post “du –max-depth” alternative on Mac OSX appeared first on

Ever hard to find obscure places where disk space is wasted on your Mac? If you're accustomed to the Linux terminal, you'll try this:

$ du -h --max-depth=1
du: illegal option -- -
usage: du [-H | -L | -P] [-a | -s | -d depth] [-c] [-h | -k | -m | -g] [-x] [-I mask] [file ...]

... but that doesn't work on Mac.

So here's a Mac alternative:

$ find . -maxdepth 1 -type d -mindepth 1 -exec du -hs {} \;

I try that --max-depth every. single. time.

Update: it's even easier than that

Sure, the find line works, but turns out I've been missing a much easier shortcut here.

$ du -hd1

Lesson learned: blog about it faster! ^^

The post “du –max-depth” alternative on Mac OSX appeared first on

by Mattias Geniar at December 11, 2017 04:45 PM

December 07, 2017

Errata Security

Libertarians are against net neutrality

This post claims to be by a libertarian in support of net neutrality. As a libertarian, I need to debunk this. "Net neutrality" is a case of one-hand clapping, you rarely hear the competing side, and thus, that side may sound attractive. This post is about the other side, from a libertarian point of view.

That post just repeats the common, and wrong, left-wing talking points. I mean, there might be a libertarian case for some broadband regulation, but this isn't it.

This thing they call "net neutrality" is just left-wing politics masquerading as some sort of principle. It's no different than how people claim to be "pro-choice", yet demand forced vaccinations. Or, it's no different than how people claim to believe in "traditional marriage" even while they are on their third "traditional marriage".

Properly defined, "net neutrality" means no discrimination of network traffic. But nobody wants that. A classic example is how most internet connections have faster download speeds than uploads. This discriminates against upload traffic, harming innovation in upload-centric applications like DropBox's cloud backup or BitTorrent's peer-to-peer file transfer. Yet activists never mention this, or other types of network traffic discrimination, because they no more care about "net neutrality" than Trump or Gingrich care about "traditional marriage".

Instead, when people say "net neutrality", they mean "government regulation". It's the same old debate between who is the best steward of consumer interest: the free-market or government.

Specifically, in the current debate, they are referring to the Obama-era FCC "Open Internet" order and reclassification of broadband under "Title II" so they can regulate it. Trump's FCC is putting broadband back to "Title I", which means the FCC can't regulate most of its "Open Internet" order.

Don't be tricked into thinking the "Open Internet" order is anything but intensely politically. The premise behind the order is the Democrat's firm believe that it's government who created the Internet, and all innovation, advances, and investment ultimately come from the government. It sees ISPs as inherently deceitful entities who will only serve their own interests, at the expense of consumers, unless the FCC protects consumers.

It says so right in the order itself. It starts with the premise that broadband ISPs are evil, using illegitimate "tactics" to hurt consumers, and continues with similar language throughout the order.

A good contrast to this can be seen in Tim Wu's non-political original paper in 2003 that coined the term "net neutrality". Whereas the FCC sees broadband ISPs as enemies of consumers, Wu saw them as allies. His concern was not that ISPs would do evil things, but that they would do stupid things, such as favoring short-term interests over long-term innovation (such as having faster downloads than uploads).

The political depravity of the FCC's order can be seen in this comment from one of the commissioners who voted for those rules:
FCC Commissioner Jessica Rosenworcel wants to increase the minimum broadband standards far past the new 25Mbps download threshold, up to 100Mbps. "We invented the internet. We can do audacious things if we set big goals, and I think our new threshold, frankly, should be 100Mbps. I think anything short of that shortchanges our children, our future, and our new digital economy," Commissioner Rosenworcel said.
This is indistinguishable from communist rhetoric that credits the Party for everything, as this booklet from North Korea will explain to you.

But what about monopolies? After all, while the free-market may work when there's competition, it breaks down where there are fewer competitors, oligopolies, and monopolies.

There is some truth to this, in individual cities, there's often only only a single credible high-speed broadband provider. But this isn't the issue at stake here. The FCC isn't proposing light-handed regulation to keep monopolies in check, but heavy-handed regulation that regulates every last decision.

Advocates of FCC regulation keep pointing how broadband monopolies can exploit their renting-seeking positions in order to screw the customer. They keep coming up with ever more bizarre and unlikely scenarios what monopoly power grants the ISPs.

But the never mention the most simplest: that broadband monopolies can just charge customers more money. They imagine instead that these companies will pursue a string of outrageous, evil, and less profitable behaviors to exploit their monopoly position.

The FCC's reclassification of broadband under Title II gives it full power to regulate ISPs as utilities, including setting prices. The FCC has stepped back from this, promising it won't go so far as to set prices, that it's only regulating these evil conspiracy theories. This is kind of bizarre: either broadband ISPs are evilly exploiting their monopoly power or they aren't. Why stop at regulating only half the evil?

The answer is that the claim "monopoly" power is a deception. It starts with overstating how many monopolies there are to begin with. When it issued its 2015 "Open Internet" order the FCC simultaneously redefined what they meant by "broadband", upping the speed from 5-mbps to 25-mbps. That's because while most consumers have multiple choices at 5-mbps, fewer consumers have multiple choices at 25-mbps. It's a dirty political trick to convince you there is more of a problem than there is.

In any case, their rules still apply to the slower broadband providers, and equally apply to the mobile (cell phone) providers. The US has four mobile phone providers (AT&T, Verizon, T-Mobile, and Sprint) and plenty of competition between them. That it's monopolistic power that the FCC cares about here is a lie. As their Open Internet order clearly shows, the fundamental principle that animates the document is that all corporations, monopolies or not, are treacherous and must be regulated.

"But corporations are indeed evil", people argue, "see here's a list of evil things they have done in the past!"

No, those things weren't evil. They were done because they benefited the customers, not as some sort of secret rent seeking behavior.

For example, one of the more common "net neutrality abuses" that people mention is AT&T's blocking of FaceTime. I've debunked this elsewhere on this blog, but the summary is this: there was no network blocking involved (not a "net neutrality" issue), and the FCC analyzed it and decided it was in the best interests of the consumer. It's disingenuous to claim it's an evil that justifies FCC actions when the FCC itself declared it not evil and took no action. It's disingenuous to cite the "net neutrality" principle that all network traffic must be treated when, in fact, the network did treat all the traffic equally.

Another frequently cited abuse is Comcast's throttling of BitTorrent.Comcast did this because Netflix users were complaining. Like all streaming video, Netflix backs off to slower speed (and poorer quality) when it experiences congestion. BitTorrent, uniquely among applications, never backs off. As most applications become slower and slower, BitTorrent just speeds up, consuming all available bandwidth. This is especially problematic when there's limited upload bandwidth available. Thus, Comcast throttled BitTorrent during prime time TV viewing hours when the network was already overloaded by Netflix and other streams. BitTorrent users wouldn't mind this throttling, because it often took days to download a big file anyway.

When the FCC took action, Comcast stopped the throttling and imposed bandwidth caps instead. This was a worse solution for everyone. It penalized heavy Netflix viewers, and prevented BitTorrent users from large downloads. Even though BitTorrent users were seen as the victims of this throttling, they'd vastly prefer the throttling over the bandwidth caps.

In both the FaceTime and BitTorrent cases, the issue was "network management". AT&T had no competing video calling service, Comcast had no competing download service. They were only reacting to the fact their networks were overloaded, and did appropriate things to solve the problem.

Mobile carriers still struggle with the "network management" issue. While their networks are fast, they are still of low capacity, and quickly degrade under heavy use. They are looking for tricks in order to reduce usage while giving consumers maximum utility.

The biggest concern is video. It's problematic because it's designed to consume as much bandwidth as it can, throttling itself only when it experiences congestion. This is what you probably want when watching Netflix at the highest possible quality, but it's bad when confronted with mobile bandwidth caps.

With small mobile devices, you don't want as much quality anyway. You want the video degraded to lower quality, and lower bandwidth, all the time.

That's the reasoning behind T-Mobile's offerings. They offer an unlimited video plan in conjunction with the biggest video providers (Netflix, YouTube, etc.). The catch is that when congestion occurs, they'll throttle it to lower quality. In other words, they give their bandwidth to all the other phones in your area first, then give you as much of the leftover bandwidth as you want for video.

While it sounds like T-Mobile is doing something evil, "zero-rating" certain video providers and degrading video quality, the FCC allows this, because they recognize it's in the customer interest.

Mobile providers especially have great interest in more innovation in this area, in order to conserve precious bandwidth, but they are finding it costly. They can't just innovate, but must ask the FCC permission first. And with the new heavy handed FCC rules, they've become hostile to this innovation. This attitude is highlighted by the statement from the "Open Internet" order:
And consumers must be protected, for example from mobile commercial practices masquerading as “reasonable network management.”
This is a clear declaration that free-market doesn't work and won't correct abuses, and that that mobile companies are treacherous and will do evil things without FCC oversight.


Ignoring the rhetoric for the moment, the debate comes down to simple left-wing authoritarianism and libertarian principles. The Obama administration created a regulatory regime under clear Democrat principles, and the Trump administration is rolling it back to more free-market principles. There is no principle at stake here, certainly nothing to do with a technical definition of "net neutrality".

The 2015 "Open Internet" order is not about "treating network traffic neutrally", because it doesn't do that. Instead, it's purely a left-wing document that claims corporations cannot be trusted, must be regulated, and that innovation and prosperity comes from the regulators and not the free market.

It's not about monopolistic power. The primary targets of regulation are the mobile broadband providers, where there is plenty of competition, and who have the most "network management" issues. Even if it were just about wired broadband (like Comcast), it's still ignoring the primary ways monopolies profit (raising prices) and instead focuses on bizarre and unlikely ways of rent seeking.

If you are a libertarian who nonetheless believes in this "net neutrality" slogan, you've got to do better than mindlessly repeating the arguments of the left-wing. The term itself, "net neutrality", is just a slogan, varying from person to person, from moment to moment. You have to be more specific. If you truly believe in the "net neutrality" technical principle that all traffic should be treated equally, then you'll want a rewrite of the "Open Internet" order.

In the end, while libertarians may still support some form of broadband regulation, it's impossible to reconcile libertarianism with the 2015 "Open Internet", or the vague things people mean by the slogan "net neutrality".

by Robert Graham ( at December 07, 2017 01:16 AM

December 06, 2017

Errata Security

NetNeutrality vs. limiting FaceTime

People keep retweeting this ACLU graphic in regards to NetNeutrality. In this post, I debunk the fourth item. In previous posts [1] [2] I debunk other items.

But here's the thing: the FCC allowed these restrictions, despite the FCC's "Open Internet" order forbidding such things. In other words, despite the graphic's claims it "happened without net neutrality rules", the opposite is true, it happened with net neutrality rules.

The FCC explains why they allowed it in their own case study on the matter. The short version is this: AT&T's network couldn't handle the traffic, so it was appropriate to restrict it until some time in the future (the LTE rollout) until it could. The issue wasn't that AT&T was restricting FaceTime in favor of its own video-calling service (it didn't have one), but it was instead an issue of "bandwidth management".

When Apple released FaceTime, they themselves restricted it's use to WiFi, preventing its use on cell phone networks. That's because Apple recognized mobile networks couldn't handle it.

When Apple flipped the switch and allowed it's use on mobile networks, because mobile networks had gotten faster, they clearly said "carrier restrictions may apply". In other words, it said "carriers may restrict FaceTime with our blessing if they can't handle the load".

When Tim Wu wrote his paper defining "NetNeutrality" in 2003, he anticipated just this scenario. He wrote:
"The goal of bandwidth management is, at a general level, aligned with network neutrality."
He doesn't give "bandwidth management" a completely free pass. He mentions the issue frequently in his paper with a less favorable description, such as here:
Similarly, while managing bandwidth is a laudable goal, its achievement through restricting certain application types is an unfortunate solution. The result is obviously a selective disadvantage for certain application markets. The less restrictive means is, as above, the technological management of bandwidth. Application-restrictions should, at best, be a stopgap solution to the problem of competing bandwidth demands. 
And that's what AT&T's FaceTime limiting was: an unfortunate stopgap solution until LTE was more fully deployed, which is fully allowed under Tim Wu's principle of NetNeutrality.

So the ACLU's claim above is fully debunked: such things did happen even with NetNeutrality rules in place, and should happen.

Finally, and this is probably the most important part, AT&T didn't block it in the network. Instead, they blocked the app on the phone. If you jailbroke your phone, you could use FaceTime as you wished. Thus, it's not a "network" neutrality issue because no blocking happened in the network.

by Robert Graham ( at December 06, 2017 12:00 AM

December 05, 2017

Ben's Practical Admin Blog

Monitoring Windows Server Interactive Logins

I’m sure many of you realise that for systems with high value in terms of information held or impact to business due to outage or data breach, you would probably want to crank up the monitoring of such systems. Best practices say you should pretty much monitor all activity associated with local users and groups, but today I want to focus on interactive logins to servers.

This has mainly come about from my own need recently to provide the ability to notify on any interactive login to a particular server, be it using remote desktop or a console session.

My first thought was to create a SCOM Rule that would report on Security Log EventID 4624 and if the Logon Type was 3 (console logon) or 10 (RDP Logon), send an email. As it turned out, this was much harder than I expected, as I found that Logon Type was not getting consistently passed as a parameter, and doing a text search on the entire message is not good practice.

My next trick was to go back to the Windows Eventlog itself. From Windows Server 2008 and Above, you are now able to attach a schedule task to be triggered by an EventID. However support for Email directory from the task was deprecated in Windows Server 2012, to the point where you cannot use it. This is not as big a deal as it sounds, as you can just use the send-mailmessage cmdlet to achieve the same thing in a script, and attach the script as the triggered task.

However, EventID 4624 can be quite verbose. There is a lot of information in the event message. Powershell would need to parse it and turn it into something readable. This my friends was where things while definitely possible, could have turned very messy.

It was while I was poking around in all the other available logs that ship in Windows Server that I came across the Microsoft-Windows-TerminalServices-LocalSessionManager and from there discovered EventID, which had a message that looked something like this:

Remote Desktop Services: Session logon succeeded:

User: domain\user
Session ID: 3
Source Network Address:

Perfect! Perhaps I can get SCOM to simply monitor this log for this EventID, and I won’t have to then filter based on parameter or details in the message! Oh if only…

The Version of  SCOM currently in use where I work has no support for Windows Event Logs outside of the usual Application, Security and System. So back to Powershell scripts and EventID triggered scheduled tasks we go!

The Powershell looks something like this:

$logentry = get-winevent -filterhashtable @{ Logname = 'Microsoft-Windows-TerminalServices-LocalSessionManager/Operational'; ID = 21} -MaxEvents 1
$logArray = $logentry.Message.Split("`n")

[string]$emailSubject = ("Local Login to CA - " +($logarray |select -Index 2)).Trim()
$emailBody = $logentry.message
$emailFrom = ""
$emailTo = ""
$smtp = ""

Send-MailMessage -To $emailTo -From $emailFrom -Subject $($emailSubject) -Body $emailbody -SmtpServer $smtp

As you can see, there is still a little bit of manipulation of the event message, mainly to split each line into an array to format the subject. Get-WinEvent is used to access the log as, like SCOM, Get-Eventlog only deals with the 3 main log files.

From here, it is just a matter of selecting the task in the event viewer, and choosing to “attach task to this event…” from the action pane on the right:

A wizard will then guide you through the steps.

I’ll be the first to admit there are probably better ways to do this – 3rd party tools, Heck, SCOM 2016 may well support these log names now. This however works within the constrains I have. It also is not something that would scale, and I acknowledge that.

If there are better solutions, why not leave a comment below to discuss – I’d love to hear from you!


by Ben at December 05, 2017 09:40 PM

The Lone Sysadmin

7 Ways IT Staff Can Prepare for the Holidays

For us IT types it is important to maintain a good balance between work and our lives. Just as they say that good fences make good neighbors, I’ve found that a good delineation between work and home improves both. The holiday season is taxing, though. People rush around trying to wrap up loose ends, they’re using […]

The post 7 Ways IT Staff Can Prepare for the Holidays appeared first on The Lone Sysadmin. Head over to the source to read the full post!

by Bob Plankers at December 05, 2017 07:39 AM

December 04, 2017

Evaggelos Balaskas

Install Signal Desktop to Archlinux

How to install Signal dekstop to archlinux

Download Signal Desktop

eg. latest version v1.0.41

$ curl -s \
    -o /tmp/signal-desktop_1.0.41_amd64.deb

Verify Package

There is a way to manually verify the integrity of the package, by checking the hash value of the file against a gpg signed file. To do that we need to add a few extra steps in our procedure.

Download Key from the repository

$ wget -c

--2017-12-11 22:13:34--
Loaded CA certificate '/etc/ssl/certs/ca-certificates.crt'
Connecting to connected.
Proxy request sent, awaiting response... 200 OK
Length: 3090 (3.0K) [application/pgp-signature]
Saving to: ‘keys.asc’

keys.asc                          100%[============================================================>]   3.02K  --.-KB/s    in 0s      

2017-12-11 22:13:35 (160 MB/s) - ‘keys.asc’ saved [3090/3090]

Import the key to your gpg keyring

$ gpg2 --import keys.asc

gpg: key D980A17457F6FB06: public key "Open Whisper Systems <>" imported
gpg: Total number processed: 1
gpg:               imported: 1

you can also verify/get public key from a known key server

$ gpg2 --verbose --keyserver --recv-keys 0xD980A17457F6FB06

gpg: data source:
gpg: armor header: Version: SKS 1.1.6
gpg: armor header: Comment: Hostname:
gpg: pub  rsa4096/D980A17457F6FB06 2017-04-05  Open Whisper Systems <>
gpg: key D980A17457F6FB06: "Open Whisper Systems <>" not changed
gpg: Total number processed: 1
gpg:              unchanged: 1

Here is already in place, so no changes.

Download Release files

$ wget -c

$ wget -c

Verify Release files

$ gpg2 --no-default-keyring --verify Release.gpg Release

gpg: Signature made Sat 09 Dec 2017 04:11:06 AM EET
gpg:                using RSA key D980A17457F6FB06
gpg: Good signature from "Open Whisper Systems <>" [unknown]
gpg: WARNING: This key is not certified with a trusted signature!
gpg:          There is no indication that the signature belongs to the owner.
Primary key fingerprint: DBA3 6B51 81D0 C816 F630  E889 D980 A174 57F6 FB06

That means that Release file is signed from whispersystems and the integrity of the file is not changed/compromized.

Download Package File

We need one more file and that is the Package file that contains the hash values of the deb packages.

$ wget -c

But is this file compromized?
Let’s check it against Release file:

$ sha256sum Packages

ec74860e656db892ab38831dc5f274d54a10347934c140e2a3e637f34c402b78  Packages

$ grep ec74860e656db892ab38831dc5f274d54a10347934c140e2a3e637f34c402b78 Release

 ec74860e656db892ab38831dc5f274d54a10347934c140e2a3e637f34c402b78     1713 main/binary-amd64/Packages

yeay !

Verify deb Package

Finally we are now ready to manually verify the integrity of the deb package:

$ sha256sum signal-desktop_1.0.41_amd64.deb

9cf87647e21bbe0c1b81e66f88832fe2ec7e868bf594413eb96f0bf3633a3f25  signal-desktop_1.0.41_amd64.deb

$ egrep 9cf87647e21bbe0c1b81e66f88832fe2ec7e868bf594413eb96f0bf3633a3f25 Packages

SHA256: 9cf87647e21bbe0c1b81e66f88832fe2ec7e868bf594413eb96f0bf3633a3f25

Perfect, we are now ready to continue

Extract under tmp filesystem

$ cd /tmp/

$ ar vx signal-desktop_1.0.41_amd64.deb

x - debian-binary
x - control.tar.gz
x - data.tar.xz

Extract data under tmp filesystem

$ tar xf data.tar.xz

Move Signal-Desktop under root filesystem

# sudo mv opt/Signal/ /opt/Signal/


Actually, that’s it!


Run signal-desktop as a regular user:

$ /opt/Signal/signal-desktop

Signal Desktop



Define your proxy settings on your environment:

declare -x ftp_proxy=""
declare -x http_proxy=""
declare -x https_proxy=""



Tag(s): signal, archlinux

December 04, 2017 10:41 PM


On "Advanced" Network Security Monitoring

My TaoSecurity News page says I taught 41 classes lasting a day or more, from 2002 to 2014. All of these involved some aspect of network security monitoring (NSM). Many times students would ask me when I would create the "advanced" version of the class, usually in the course feedback. I could never answer them, so I decided to do so in this blog post.

The short answer is this: at some point, advanced NSM is no longer NSM. If you consider my collection - analysis - escalation - response model, NSM extensions from any of those phases quickly have little or nothing to do with the network.

Here are a few questions I have received concerned "advanced NSM," paired with the answers I could have provided.

Q: "I used NSM to extract a binary from network traffic. What do I do with this binary?"

A: "Learn about reverse engineering and binary analysis."


Q: "I used NSM to extra Javascript from a malicious Web page. What do I do with this Javascript?"

A: "Learn about Javascript de-obfuscation and programming."


Q: "I used NSM to capture an exchange between a Windows client and a server. What does it mean?"

A: "Learn about Server Message Block (SMB) or Common Internet File System (CIFS)."


Q: "I used NSM to capture cryptographic material exchanged between a client and a server. How do I understand it?"

A: "Learn about cryptography."


Q: "I used NSM to grab shell code passed with an exploit against an Internet-exposed service. How do I tell what it does?"

A: "Learn about programming in assembly."


Q: "I want to design custom hardware for packet capture. How do I do that?"

A: "Learn about programming ASICs (application specific integrated circuits)."

I realized that I had the components of all of this "advanced NSM" material in my library. I had books on reverse engineering and binary analysis, Javascript, SMB/CIFS, cryptography, assembly programming, ASICs, etc.

The point is that eventually the NSM road takes you to other aspects of the cyber security landscape.

Are there *any* advanced area for NSM? One could argue that protocol analysis, as one finds in tools like Bro, Suricata, Snort, Wireshark, and so on constitute advanced NSM. However, you could just as easily argue that protocol analysis becomes more about understanding the programming and standards behind each of the protocols.

In brief, to learn advanced NSM, expand beyond NSM.

by Richard Bejtlich ( at December 04, 2017 04:30 PM

Perl to go

I have been using Perl for more than 20 years now, seen Perl 4 bow out and Perl 5 come in and develop in that fantastic language that has helped me uncountable times in my professional life. During those years I’ve also considered learning another language, but I have been unable to take a stand for a long time.

And there came Go and the hype around Go, just like years ago there was a lot of hype around Java. But while whatever written in Java I came across was a big, heavy and slow memory eater, most of the tools I came across that were written in Go were actually good stuff — OK, still a bit bloated in size, but they actually worked. The opportunity came, and I finally gave Go a shot.

A false start

The opportunity arose when early this year it was announced that an introductory course in Go was being organised in our company. I immediately confirmed my interest and bought plane tickets when the date was confirmed.

When the day finally arrived I flew to Trondheim only to discover that the target of the course was silently changed and the course was way more specialistic than expected. Yes, you can say I was annoyed. In an attempt not to waste the trip and time, I started the Tour of Go there and then and went through the first few chapters and exercises. It looked great. For a while.

A tour of go

The tour of Go aims to be an introductory course, and to a point it actually is. But when you get far enough you’ll notice that some exercise assume more knowledge of the language than what was explained in previous chapters: whoever wrote the more advanced lessons was disconnected, either from the previous part of the course or from reality. I vented my frustrations in a tweet:

I was not liking Golang’s syntax nor the tour, frustration was growing, my learning experience was in danger. If I wanted to keep going I needed to find another way; for my particular mindset the best way is often a good book. It was time to find one.

Introducing Go

I did some research and finally set to buy Doxsey’s “Introducing Go” from O’Reilly, a simple book, so thin that Posten delivered it straight into my mailbox! (instead of the usual pick-up notice). The first chapters were simple indeed and I already knew most of their content from the tour of Go so I just skimmed through. Later chapters were still simple, but also informative and to the point, with exercises at the end that were well in line with the content.

I got to the end of the book reasonably quickly, and it was now time for a “final project”. Those who know me, know that I don’t like “Hello world!”-class exercises. To check my learning I want something that is easy enough to be possible for a beginner, but challenging enough to put my new knowledge to the test. I considered a few options and decided to reproduce hENC in Go.

hENC: a recap

For those who don’t know or don’t remember, hENC was a project of mine, the  radically simple hierarchical External Node Classifier (ENC) for CFEngine. It’s a very simple script written in pure Perl. In 79 lines of code (47 of actual code) it reads a list of files in a format similar to CFEngine’s module protocol and merges their content in a hierarchical fashion. The output is then used by CFEngine to classify the node and set variables according to the node’s role in the infrastructure.

hENC reads files, finds data through pattern matching, applies some logic, fills up data structures and prints out the result: no rocket science but still a real-world program. Reproducing hENC in Go had all that it takes for a decent exercise.

NB: hENC is silent on conditions that would otherwise be errors (e.g. missing input files). That is by design. Any output from the program (including errors!) is piped into CFEngine: if you don’t want to fill your CFEngine logs with tons of messages about non-compliant messages in the module’s output you need to make it as silent as possible. That’s why you don’t find any error logging in the Perl version, and all of the messages via the log package are commented out in the Go version.

hENC in Go

Perl and Go are built around very different philosophies: where Perl is redundant and allows for dozens of ways to express the same thing, Go chooses to allow only one or very few (think loops, think control structures…); where many frequently used functions are part of the core language in Perl (e.g. print() and file handling functions like open(), close() and so forth), Go has them in packages that, although part of the core, you have to import explicitly into the program. It’s a difference that is very visible by comparing the sources of the programs and from the very start: the Perl version starts by using two pragmata and then goes straight to the data structures:

use strict ;
use warnings ;

my %class ;    # classes container
my %variable ; # variables container

The Go version spends some lines to import before it gets to the same point:

package main

import "fmt"    // to print something...
import "os"     // to read command-line args, opening files...
import "bufio"  // to read files line by line, see
// import "log"
import "regexp"

var class    = make(map[string]int)     // classes container
var variable = make(map[string]string)  // variables container

Perl takes advantage of the diamond construct to read from the files like they were a single one and without even explicitly open them:

while (my $line = ) {

Not so in Go:

var encfiles = os.Args[1:]

func main() {
	// prepare the regex for matching the ENC setting
	settingRe := regexp.MustCompile(`^\s*([=\@%+-/_!])(.+)\s*$`)

	// prepare the regex for matching a variable assignment
	varRe := regexp.MustCompile(`^(.+?)=`)
	// iterate over files
	for _,filename := range encfiles {
		// try to open, fail silently if it doesn't exist
		file,err := os.Open(filename)
		if err != nil {
			// error opening this file, skip and...
			continue File
		defer file.Close()

		// Read file line by line.
		// Dammit Go, isn't this something that one does often
		// enough to deserve the simplest way to do it???
		// Anyway, here we go with what one can find in
		scanner := bufio.NewScanner(file)
		for scanner.Scan() {
			err := scanner.Err()
			if err != nil {
				// log.Printf("Error reading file %s: %s",filename,err)
				break Line

			// no need to "chomp()" here, the newline is already gone
			line := scanner.Text()

The previous code snippet also includes the preparation of two regular expression patterns that will be used later in the game; this is a notable difference from Perl, where Perl is on the minimalistic side (one single instruction to do pattern matching, sub-match extraction and so forth), where Go introduces a number of functions and methods to do the same job. Regular expressions are an already complicated subject and don’t definitely need any additional mess: like many other languages, Go should take some lessons from Perl on this subject.

An area where Go tends to be cleaner than Perl is where you can use the built-in switch/case construct instead of if:

			case `+`:
				// add a class, assume id is a class name
				class[id] = 1

			case `-`:
				// undefine a class, assume id is a class name
				class[id] = -1

Perl’s equivalent given/when construct is still experimental; a Switch module is provided in CPAN and was in the core distribution in the past, but it’s use is discouraged in favour of the experimental given/when construct… uhm…

Switch/case aside, the Perl version of hENC was designed to run on any recent and not-so-recent Perl, so it uses the plain old if construct:

    if ($setting eq '+') {
	# $id is a class name, or should be.
	$class{$id} = 1 ;

    # undefine a class
    if ($setting eq '-') {
	# $id is a class name, or should be.
	$class{$id} = -1 ;

though there are still places where Perl is a bit clearer and more concise than Go:

    # reset the status of a class
    if ($setting eq '_') {
	# $id is a class name, or should be.
	delete $class{$id} if exists $class{$id} ;


			case `_`:
				// reset the class, if it's there
				_,ok := class[id]
				if ok {

You can find the full source of gohENC at the end of the post. By the way, the gohENC source is 140 lines with extensive comments (80 lines of actual code, nearly twice as Perl’s version).


I got gohENC completed through a few short sessions and it was time to test if it really worked. That was an easy task, since hENC comes with a test suite. All I had to do was to compile the Go source, replace the Perl version with the binary and run the tests:

$ prove --exec "sudo cf-agent -KC -f" ./
./ .. ok   
All tests successful.
Files=1, Tests=8,  0 wallclock secs ( 0.05 usr  0.01 sys +  0.08 cusr  0.00 csys =  0.14 CPU)
Result: PASS

Success! gohENC is live!

Why not Java?

I have often considered Java, but never got to love it. I felt the language, the tools and the ecosystem were unnecessarily complicated and the Java programs I have used in the past didn’t make me love the language either.

Why not Python?

I have always been surrounded by “pythonists” and considered Python, too, but was kind of discouraged by the fact that Python 3’s popularity wasn’t really taking off, while learning Python 2.7 seemed like a waste of time because its successor was already there.

Why not Ruby?

The only time I touched Ruby was when I tried to write some Puppet facts: the code I saw at the time didn’t impress me and I tried to stay away from Ruby ever since.

Why not JavaScript?

Because I was unsure about how much I could use it to help me with my job, and outside of web pages anyway.

Why not PHP?


Why not Perl6?

Perl 6, the new kid on the block, seems great and powerful, but not really something that would add an edge in my CV unfortunately.

Source code for gohENC

package main

import "fmt"    // to print something...
import "os"     // to read command-line args, opening files...
import "bufio"  // to read files line by line, see
// import "log"
import "regexp"

var class    = make(map[string]int)     // classes container
var variable = make(map[string]string)  // variables container

var encfiles = os.Args[1:]

func main() {
	// prepare the regex for matching the ENC setting
	settingRe := regexp.MustCompile(`^\s*([=\@%+-/_!])(.+)\s*$`)

	// prepare the regex for matching a variable assignment
	varRe := regexp.MustCompile(`^(.+?)=`)
	// iterate over files
	for _,filename := range encfiles {
		// try to open, fail silently if it doesn't exist
		file,err := os.Open(filename)
		if err != nil {
			// error opening this file, skip and...
			continue File
		defer file.Close()

		// Read file line by line.
		// Dammit Go, isn't this something that one does often
		// enough to deserve the simplest way to do it???
		// Anyway, here we go with what one can find in
		scanner := bufio.NewScanner(file)
		for scanner.Scan() {
			err := scanner.Err()
			if err != nil {
				// log.Printf("Error reading file %s: %s",filename,err)
				break Line

			// no need to "chomp()" here, the newline is already gone
			line := scanner.Text()

			// Dear Go, regular expression are already
			// complicated, there is absolutely NO need for you to
			// make them even more fucked up...
			// Sixteen functions to do pattern matching... so much
			// for your fucking minimalism!
			match := settingRe.FindStringSubmatch(line)

			setting,id := match[1],match[2]
			// log.Printf("setting: %s, value: %s",setting,id)

			switch setting {
			case `!`:
				// take a command
				switch id {
					// flush the class cache
					// ...which means: kill all key/values
					// recorded in the classes map
					// In Go, you're better off overwriting the
					// new array, so...
					class = make(map[string]int)

					// remove active classes from the cache
					for k,v := range class {
						if v > 0 {

					// remove cancelled classes from the cache
					for k,v := range class {
						if v < 0 { 							delete(class,k) 						} 					} 				} // switch id 			case `+`: 				// add a class, assume id is a class name 				class[id] = 1 			case `-`: 				// undefine a class, assume id is a class name 				class[id] = -1 			case `_`: 				// reset the class, if it's there 				_,ok := class[id] 				if ok { 					delete(class,id) 				} 			case `=`, `@`, `%`: 				// define a variable/list 				match := varRe.FindStringSubmatch(id) 				varname := match[1] // not necessary, just clearer 				variable[varname] = line 			case `/`: 				// reset a variable/list 				_,ok := variable[id] 				if ok { 					delete(variable,id) 				} 				 			} // switch setting 			// discard the rest 		} 	} 	// print out classes 	class[`henc_classification_completed`] = 1 	for classname,value := range class { 		switch { 		case value > 0:

		case value < 0:

	// print variable/list assignments, the last one wins
	for _,assignment := range variable {


Tagged: Configuration management, golang, henc, Perl, programming

by bronto at December 04, 2017 08:00 AM

December 02, 2017

Steve Kemp's Blog repository cleanup, and email-changes.

I've shuffled around all the repositories which are associated with the blogspam service, such that they're all in the same place and refer to each other correctly:

Otherwise I've done a bit of tidying up on virtual machines, and I'm just about to drop the use of qpsmtpd for handling my email. I've used the (perl-based) qpsmtpd project for many years, and documented how my system works in a "book":

I'll be switching to pure exim4-based setup later today, and we'll see what that does. So far today I've received over five thousand spam emails:

  steve@ssh /spam/today $ find . -type f | wc -l

Looking more closely though over half of these rejections are "dictionary attacks", so they're not SPAM I'd see if I dropped the qpsmtpd-layer. Here's a sample log entry (for a mail that was both rejected at SMTP-time by qpsmtpd and archived to disc in case of error):

    "reason":"Mail for juha not accepted at",
    "subject":"Viagra Professional. Beyond compare. Buy at our shop.",

I suspect that with procmail piping to crm114, and a beefed up spam-checking configuration for exim4 I'll not see a significant difference and I'll have removed something non-standard. For what it is worth over 75% of the remaining junk which was rejected at SMTP-time has been rejected via DNS-blacklists. So again exim4 will take care of that for me.

If it turns out that I'm getting inundated with junk-mail I'll revert this, but I suspect that it'll all be fine.

December 02, 2017 10:00 PM

November 29, 2017

Everything Sysadmin

Quick! Install that Mac printer now!

Does your friend or significant other have a Mac from work that is locked down so that changes can't be made? Any attempt to make a chance in System Preferences asks for the admin password, which you don't have. Maybe they are a teacher at a school with overly-zealous sysadmins? Maybe they work at an insurance company that... just kidding, no insurance company supports Macs.

Someone Who Isn't Me knows someone that has a Mac laptop and can't print to the home printers for exactly this reason. To print at home, they generate a PDF, copy the file to a USB stick, and walk it over to another computer that can print. That is ludicrous.

Now is your chance to fix this.

In macOS High Sierra, anyone can login as "root" with empty password. When asked for the admin username and password enter "root" as the user, then leave the password blank. Two this a second time and you'll unlock admin access.

Now is your chance to install that printer, change the screen saver settings, enable Time Machine, or whatever you have been wanting to do.

Apple will surely fix this soon. You probably only have hours or days to install all the printers and VPNs and other things you've been meaning to fix.

Oh, but don't break any laws or company policies. Certainly don't create an account with admin privs, or give the primary user admin privs. That is probably against policy and could lead to productivity.

More info here:

Sorry... not sorry.

by Tom Limoncelli at November 29, 2017 02:49 PM

Feeding the Cloud

Proxy ACME challenges to a single machine

The Libravatar mirrors are setup using DNS round-robin which makes it a little challenging to automatically provision Let's Encrypt certificates.

In order to be able to use Certbot's webroot plugin, I need to be able to simultaneously host a randomly-named file into the webroot of each mirror. The reason is that the verifier will connect to, but there's no way to know which of the DNS entries it will hit. I could copy the file over to all of the mirrors, but that would be annoying since some of the mirrors are run by volunteers and I don't have direct access to them.

Thankfully, Scott Helme has shared his elegant solution: proxy the .well-known/acme-challenge/ directory from all of the mirrors to a single validation host. Here's the exact configuration I ended up with.

DNS Configuration

In order to serve the certbot validation files separately from the main service, I created a new hostname,, pointing to the main Libravatar server:

CNAME acme

Mirror Configuration

On each mirror, I created a new Apache vhost on port 80 to proxy the acme challenge files by putting the following in the existing port 443 vhost config (/etc/apache2/sites-available/libravatar-seccdn.conf):

<VirtualHost *:80>
    ServerAdmin __WEBMASTEREMAIL__

    ProxyPass /.well-known/acme-challenge/
    ProxyPassReverse /.well-known/acme-challenge/

Then I enabled the right modules and restarted Apache:

a2enmod proxy
a2enmod proxy_http
systemctl restart apache2.service

Finally, I added a cronjob in /etc/cron.daily/commit-new-seccdn-cert to commit the new cert to etckeeper automatically:

cd /etc/libravatar
/usr/bin/git commit --quiet -m "New seccdn cert" seccdn.crt seccdn.pem seccdn-chain.pem > /dev/null || true

Main Configuration

On the main server, I created a new webroot:

mkdir -p /var/www/acme/.well-known

and a new vhost in /etc/apache2/sites-available/acme.conf:

<VirtualHost *:80>
    DocumentRoot /var/www/acme
    <Directory /var/www/acme>
        Options -Indexes

before enabling it and restarting Apache:

a2ensite acme
systemctl restart apache2.service

Registering a new TLS certificate

With all of this in place, I was able to register the cert easily using the webroot plugin on the main server:

certbot certonly --webroot -w /var/www/acme -d

The resulting certificate will then be automatically renewed before it expires.

November 29, 2017 06:10 AM

November 28, 2017

Root login without password allowed by default on Mac OSX High Sierra

The post Root login without password allowed by default on Mac OSX High Sierra appeared first on

Right, this isn't a good day for Apple.

As first reported on Twitter by Lemi Orhan Ergin, you can bypass just about any security dialog on Mac OSX High Sierra (10.13) by using the root user without a password.

Use the user root and click Unlock several times, you'll eventually bypass the dialog and be granted root privileges. You can try it if you go to the Users & Groups settings screen and click Lock at the bottom.

I'd be very curious to know the technical reasons why this was possible in the first place.

Update: be sure to disable the root user after test

Turns out, testing this actually creates a root user without a password in the background! Make sure to disable the root user in System Preferences to prevent this from getting any worse than it already is.

For a quick workaround, set a non-default (aka: anything) password on the root user via the terminal.

$ sudo passwd -u root

Once a password has been set, it wont change to an empty value anymore.

Also applicable to Remote Management

If you've enabled Remote Management, anyone can log into your Mac using the root user with an empty password.


Responsible disclosure?

This issue was first reported on Twitter and is now getting widespread traction. This isn't exactly a good way to disclose security issues, but I'm willing to bet the reporter perhaps didn't think it would go this far in the media?

There's an entire KB about reporting security issues to Apple, if someone ever feels the need to report similar security bugs.

The post Root login without password allowed by default on Mac OSX High Sierra appeared first on

by Mattias Geniar at November 28, 2017 08:26 PM

November 27, 2017


Terraforming in prod

Terraform from HashiCorp is something we've been using in prod for a while now. Simply ages in terraform-years, which means we have some experience with it.

It also means we have some seriously embedded legacy problems, even though it's less than two years old. That's the problem with rapidly iterating infrastructure projects that don't build in production usecases from the very start. You see, a screwdriver is useful in production! It turns screws and occasionally opens paint cans. You'd think that would be enough. But production screwdrivers conform to external standards of screwdrivers, are checked in and checked out of central tooling because quality control saves everyone from screwdriver related injuries, and have support for either outright replacement or refacing of the toolface. has a great screed on this you should read. But I wanted to share how we use it, and our pains.

In the beginning

I did our initial Terraform work in the Terraform version 6 days. The move to production happened about when TF 7 came out. You can see how painfully long ago that was (or wasn't). It was a different product back then.

Terraform Modules were pretty new back when I did the initial build. I tried them, but couldn't get them to work right. At the time I told my coworkers:

They seem to work like puppet includes, not puppet defines. I need them to be defines, so I'm not using them.

I don't know if I had a fundamental misunderstanding back then or if that's how they really worked. But they're defines now, and all correctly formatted TF infrastructures use them or be seen as terribly unstylish. It means there is a lot of repeating-myself in our infrastructure.

Because we already had an AMI baking pipeline that worked pretty well, we never bothered with Terraform Provisioners. We build ours entirely on making the AWS assets versonable. We tried with CloudFormation, but gave that up due to the terrible horrible no good very bad edge cases that break iterations. Really, if you have to write Support to unfuck an infrastructure because CF can't figure out the backout plan (and that backout is obvious to you), then you have a broken product. When Terraform gets stuck, it just throws up its hands and says HALP! Which is fine by us.

Charity asked a question in that blog-post:

Lots of people seem to eventually end up wrapping terraform with a script.  Why?

I wrote it for two big, big reasons.

  1. In TF6, there was zero support for sharing a Terraform statefile between multiple people (without paying for Atlas), and that critically needs to be done. So my wrapper implemented the sharing layer. Terraform now supports several methods for this out of the box, it didn't used to.
  2. Terraform is a fucking foot gun with a flimsy safety on the commit career suicide button. It's called 'terraform destroy' and has no business being enabled for any reason in a production environment ever. My wrapper makes getting at this deadly command require a minute or two of intentionally circumventing the safety mechanism. Which is a damned sight better than the routine. "Are you sure? y/n" prompt we're all conditioned to just click past. Of course I'm in the right directory! Yes!

And then there was legacy.

We're still using that wrapper-script. Partly because reimplementing it for the built-in statefile sharing is, like, work and what we have is working. But also because I need those welded on fire-stops on the foot-gun.

But, we're not using modules, and really should. However, integrating modules is a long and laborious process we haven't seen enough benefits to outweight the risk. To explain why, I need to explain a bit about how terraform works.

You define resources in Terraform, like a security group with rules. When you do an 'apply', Terraform checks the statefile to see if this has been created yet, and what state it was seen last. It then compares the last known state with the current state in the infrastructure to determine what changes need to be made. Pretty simple. The name of the resource in the statefile is a clear format. For non-module resources the string is "resource_type.resource_name", so our security group example would be "aws_security_group.prod_gitlab". For module resources it gets more complicated, such as "module_name.resource_type.resource_name". That's not the exact format -- which is definitely not bulk-sed friendly -- but it works for the example I'm about to share. If you change the name of a resource, Terraform's diff shows the old resource disappearing and a brand new one appearing and treats it as such. Sometimes this is what you want. Other times, like if they're your production load-balancers where delete-and-recreate will create a multi-minute outage, you don't.

To do a module conversion, this is the general workflow.

  1. Import the module and make your changes, but don't apply them yet.
  2. Use 'terraform statefile ls' to get a list of the resources in your statefile, note the names of the resources to be moved into modules.
  3. Use 'terraform statefile rm' to remove the old resources.
  4. Use 'terraform import' to import the existing resources into the statefile under their now module-based names.
  5. Use 'terraform plan' to make sure there are zero changes.
  6. Commit your changes to the terraform repo and apply.

Seems easy, except.

  • You need to lock the statefile so no one else can make changes when you're doing this. Critically important if you have automation that does terraform actions.
  • This lock could last a couple of hours depending on how many resources need to be modified.
  • This assumes you know how terraform statefiles work with resource naming, so you need someone experienced with terraform to do this work.
  • Your modules may do some settings subtly different than you did before, so it may not be a complete null-change.
  • Some resources, like Application Load Balancers, require a heartbreaking number of resources to define, which makes for a lot of import work.
  • Not all resources even have an import developed. Those resources will have to be deleted and recreated.
  • Step 1 is much larger than you think, due to dependencies from other resources that will need updating for the new names. Which means you may visit step 1 a few times by the time you get a passing step 5.
  • This requires a local working Terraform setup, outside of your wrapper scripts. If your wrapper is a chatbot and no one has a local TF setup, this will need to be done on the chatbot instance. The fact of the matter is that you'll have to point the foot-gun at your feet for a while when you do this.
  • This is not a change that can be done through Terraform's new change-management-friendly way of packaging changes for use in change-management workflows, so it will be a 'comprehensive' change-request when it comes.

Try coding that into a change-request that will pass auditorial muster. In theory it is possible to code up a bash-script that will perform the needed statefile changes automatically, but it will be incredibly fragile in the face of other changes to the statefile as the CR works its way through the process. This is why we haven't converted to a more stylish infrastructure; the intellectual purity of being stylish isn't yet outweighing the need to not break prod.

What it's good for

Charity's opinion is close to my own:

Terraform is fantastic for defining the bones of your infrastructure.  Your networking, your NAT, autoscaling groups, the bits that are robust and rarely change.  Or spinning up replicas of production on every changeset via Travis-CI or Jenkins -- yay!  Do that!

But I would not feel safe making TF changes to production every day.  And you should delegate any kind of reactive scaling to ASGs or containers+scheduler or whatever.  I would never want terraform to interfere with those decisions on some arbitrary future run.

Yes. Terraform is best used in cases where doing an apply won't cause immediate outages or instabilities. Even using it the way we are, without provisioners, means following some rules:

  • Only define 'aws_instance' resources if we're fine with those suddenly disappearing and not coming back for a couple of minutes. Because if you change the AMI, or the userdata, or any number of other details, Terraform will terminate the existing one and make a new one.
    • Instead, use autoscaling-groups and a process outside of Terraform to manage the instance rotations.
  • It's fine to encode scheduled-scaling events on autoscaling groups, and even dynamic-scaling triggers on them.
  • Rotating instances in an autoscaling-group is best done in automation outside of terraform.
  • Playing pass-the-IP for Elastic-IP addresses is buggy and may require a few 'applies' before they fully move to the new instances.
  • Cache-invalidation on the global Internet's DNS caches is still buggy as fuck, though getting better. Plan around that.
  • Making some changes may require multiple phases. That's fine, plan for that.

The biggest strength of Terraform is that it looks a lot like Puppet, but for your AWS config. Our auditors immediately grasped that concept and embraced it like they've known about Terraform since forever. Because if some engineer cowboys in a change outside of the CR process, Terraform will back that out the next time someone does an apply, much the way puppet will back out a change to a file it manages. That's incredibly powerful, and something CloudFormation only sort of does.

The next biggest strength is that it is being very actively maintained and tracks AWS API changes pretty closely. When Amazon announces a new service, Terraform will generally have support for it within a month (not always, but most of the time). If the aws-cli can do it, Terraform will also be able to do it; if not now, then very soon.

While there are some patterns it won't let you do, like have two security-groups point to each other on ingress/egress lists because that is a dependency loop, there is huge scope in the zones of what it will let you do.

This is a good tool and I plan to keep using it. Eventually we'll do a module conversion somewhere, but that may wait until they have a better workflow for it. Which may be in a month, or half a year. This project is moving fast.

by SysAdmin1138 at November 27, 2017 04:42 PM


The Ultimate Apollo Guidance Computer Talk @ 34C3

After The Ultimate Commodore 64 Talk (2008) and The Ultimate Game Boy Talk (2016), my third talk from the “Ultimate” series will take place at the 34th Chaos Communication Congress at Leipzig (27-30 Dec 2017):

The Apollo Guidance Computer (“AGC”) was used onboard the Apollo spacecraft to support the Apollo moon landings between 1969 and 1972. This talk explains “everything about the AGC”, including its quirky but clever hardware design, its revolutionary OS, and how its software allowed humans to reach and explore the moon. 

The talk will be presented by me (Michael Steil) and hessi. Date and time are subject to final scheduling. I will post updates as well as further information about the Apollo Guidance Computer on this blog in the next weeks.

You can read a more detailed abstract and vote on the talks on the 34C3 scheduling page.

by Michael Steil at November 27, 2017 02:01 PM

November 25, 2017

Errata Security

Why Linus is right (as usual)

People are debating this email from Linus Torvalds (maintainer of the Linux kernel). It has strong language, like:
Some security people have scoffed at me when I say that security
problems are primarily "just bugs".
Those security people are f*cking morons.
Because honestly, the kind of security person who doesn't accept that
security problems are primarily just bugs, I don't want to work with.
I thought I'd explain why Linus is right.

Linus has an unwritten manifesto of how the Linux kernel should be maintained. It's not written down in one place, instead we are supposed to reverse engineer it from his scathing emails, where he calls people morons for not understanding it. This is one such scathing email. The rules he's expressing here are:
  • Large changes to the kernel should happen in small iterative steps, each one thoroughly debugged.
  • Minor security concerns aren't major emergencies; they don't allow bypassing the rules more than any other bug/feature.
Last year, some security "hardening" code was added to the kernel to prevent a class of buffer-overflow/out-of-bounds issues. This code didn't address any particular 0day vulnerability, but was designed to prevent a class of future potential exploits from being exploited. This is reasonable.

This code had bugs, but that's no sin. All code has bugs.

The sin, from Linus's point of view, is that when an overflow/out-of-bounds access was detected, the code would kill the user-mode process or kernel. Linus thinks it should have only generated warnings, and let the offending code continue to run.

Of course, that would in theory make the change of little benefit, because it would no longer prevent 0days from being exploited.

But warnings would only be temporary, the first step. There's likely to be be bugs in the large code change, and it would probably uncover bugs in other code. While bounds-checking is a security issue, its first implementation will always find existing code having latent bounds bugs. Or, it'll have "false-positives" triggering on things that aren't actually the flaws its looking for. Killing things made these bugs worse, causing catastrophic failures in the latest kernel that didn't exist before. Warnings, however, would have equally highlighted the bugs, but without causing catastrophic failures. My car runs multiple copies of Linux -- such catastrophic failures would risk my life.

Only after a year, when the bugs have been fixed, would the default behavior of the code be changed to kill buggy code, thus preventing exploitation.

In other words, large changes to the kernel should happen in small, manageable steps. This hardening hasn't existed for 25 years of the Linux kernel, so there's no emergency requiring it be added immediately rather than conservatively, no reason to bypass Linus's development processes. There's no reason it couldn't have been warnings for a year while working out problems, followed by killing buggy code later.

Linus was correct here. No vuln has appeared in the last year that this code would've stopped, so the fact that it killed processes/kernels rather than generated warnings was unnecessary. Conversely, because it killed things, bugs in the kernel code were costly, and required emergency patches.

Despite his unreasonable tone, Linus is a hugely reasonable person. He's not trying to stop changes to the kernel. He's not trying to stop security improvements. He's not even trying to stop processes from getting killed That's not why people are moronic. Instead, they are moronic for not understanding that large changes need to made conservatively, and security issues are no more important than any other feature/bug.

Update: Also, since most security people aren't developers, they are also a bit clueless how things actually work. Bounds-checking, which they define as purely a security feature to stop buffer-overflows is actually overwhelmingly a debugging feature. When you turn on bounds-checking for the first time, it'll trigger on a lot of latent bugs in the code -- things that never caused a problem in the past (like reading past ends of buffers) but cause trouble now. Developers know this, security "experts" tend not to. These kernel changes were made by security people who failed to understand this, who failed to realize that their changes would uncover lots of bugs in existing code, and that killing buggy code was hugely inappropriate.

Update: Another flaw developers are intimately familiar with is how "hardening" code can cause false-positives, triggering on non-buggy code. A good example is where the BIND9 code crashed on an improper assert(). This hardening code designed to prevent exploitation made things worse by triggering on valid input/code.

Update: No, it's probably not okay to call people "morons" as Linus does. They may be wrong, but they usually are reasonable people. On the other hand, security people tend to be sanctimonious bastards with rigid thinking, so after he has dealt with that minority, I can see why Linus treats all security people that way.

by Robert Graham ( at November 25, 2017 05:09 PM

November 24, 2017

Sarah Allen

exploring ghostscript API in C

Ghostscript lets you do all sorts of PDF and PostScript transformations. It’s got a command-line tool which is great for page-level operations.

Installation on a mac is pretty easy with homebrew:

brew install ghostscript

The syntax for the commands not very memorable, but easy once you know it. To get a PNG from a PDF:

gs -sDEVICE=pngalpha -o output.png input.pdf

We can also use the API to call the engine from C code. Note: to use this from a program we need to publish our source code (or purchase a license from the nice folks who create and maintain Ghostscript), which seems fair to me. See license details.

To do the exact same thing as above, I created a small program based on the example in the API doc, which I’ve posted on github.

What I really want to do is to replace text in a PDF (using one PDF as a template to create another). It seems like all the code I need is available in the Ghostscript library, but maybe not exposed in usable form:

  • Projects Seeking Developers Driver Architecture was the only place in the docs that I learned that we can’t add a driver without modifying the source code: “Currently, drivers must be linked into the executable.” Might be nice for these to be filed as bugs so interested developers might discuss options here. Of course, not sure that making a driver is a good solution to my problem at all.
  • There’s an option -dFILTERTEXT that removes all text from a PDF that I thought might provide a clue. I found the implementation in gdevoflt.c with a comment that it was derived from gdevflp.c.
  • gdevflp: This device is the first ‘subclassing’ device; the intention of subclassing is to allow us to develop a ‘chain’ or ‘pipeline’ of devices, each of which can process some aspect of the graphics methods before passing them on to the next device in the chain.

So, this appears to require diving into the inner-workings of Ghostscript, yet the code seems to be structured so that it is easily modifiable for exactly this kind of thing. It seems like it would be possible to add a filter that modifies text, rather than just deleting it, as long as the layout is unaffected. This implies setting up for building and debugging GhostScript from source and the potential investment of staying current with the codebase, which might not work for my very intermittent attention.

by sarah at November 24, 2017 02:56 PM

November 22, 2017

LZone - Sysadmin

Openshift S2I and Spring profiles

When porting Springboot applications to Openshift using S2I (source to image) directly from a git repo you cannot rely on a start script passing the proper<profile name> parameter like this
java -jar yourApplication.jar
The only proper ways for injecting application configuration are
  1. Add it to the artifact/repository (meeh)
  2. Mount it using a config map
  3. Pass it via Docker environment variables
And for Openshift variant #3 works fine as there is an environment variable SPRING_PROFILES_ACTIVE which you can add in your deployment configuration and set it to your favourite spring profile name.

November 22, 2017 05:04 PM

November 20, 2017

toolsmith #129 - DFIR Redefined: Deeper Functionality for Investigators with R - Part 2

You can have data without information, but you cannot have information without data. ~Daniel Keys Moran

Here we resume our discussion of DFIR Redefined: Deeper Functionality for Investigators with R as begun in Part 1.
First, now that my presentation season has wrapped up, I've posted the related material on the Github for this content. I've specifically posted the most recent version as presented at SecureWorld Seattle, which included Eric Kapfhammer's contributions and a bit of his forward thinking for next steps in this approach.
When we left off last month I parted company with you in the middle of an explanation of analysis of emotional valence, or the "the intrinsic attractiveness (positive valence) or averseness (negative valence) of an event, object, or situation", using R and the Twitter API. It's probably worth your time to go back and refresh with the end of Part 1. Our last discussion point was specific to the popularity of negative tweets versus positive tweets with a cluster of emotionally neutral retweets, two positive retweets, and a load of negative retweets. This type of analysis can quickly give us better understanding of an attacker collective's sentiment, particularly where the collective is vocal via social media. Teeing off the popularity of negative versus positive sentiment, we can assess the actual words fueling such sentiment analysis. It doesn't take us much R code to achieve our goal using the apply family of functions. The likes of apply, lapply, and sapply allow you to manipulate slices of data from matrices, arrays, lists and data frames in a repetitive way without having to use loops. We use code here directly from Michael Levy, Social Scientist, and his Playing with Twitter Data post.

polWordTables = 
  sapply(pol, function(p) {
    words = c(positiveWords = paste(p[[1]]$pos.words[[1]], collapse = ' '), 
              negativeWords = paste(p[[1]]$neg.words[[1]], collapse = ' '))
    gsub('-', '', words)  # Get rid of nothing found's "-"
  }) %>%
  apply(1, paste, collapse = ' ') %>% 
  stripWhitespace() %>% 
  strsplit(' ') %>%

par(mfrow = c(1, 2))
  lapply(1:2, function(i) {
    dotchart(sort(polWordTables[[i]]), cex = .5)

The result is a tidy visual representation of exactly what we learned at the end of Part 1, results as noted in Figure 1.

Figure 1: Positive vs negative words
Content including words such as killed, dangerous, infected, and attacks are definitely more interesting to readers than words such as good and clean. Sentiment like this could definitely be used to assess potential attacker outcomes and behaviors just prior, or in the midst of an attack, particularly in DDoS scenarios. Couple sentiment analysis with the ability to visualize networks of retweets and mentions, and you could zoom in on potential leaders or organizers. The larger the network node, the more retweets, as seen in Figure 2.

Figure 2: Who is retweeting who?
Remember our initial premise, as described in Part 1, was that attacker groups often use associated hashtags and handles, and the minions that want to be "part of" often retweet and use the hashtag(s). Individual attackers either freely give themselves away, or often become easily identifiable or associated, via Twitter. Note that our dominant retweets are for @joe4security, @HackRead,  @defendmalware (not actual attackers, but bloggers talking about attacks, used here for example's sake). Figure 3 shows us who is mentioning who.

Figure 3: Who is mentioning who?
Note that @defendmalware mentions @HackRead. If these were actual attackers it would not be unreasonable to imagine a possible relationship between Twitter accounts that are actively retweeting and mentioning each other before or during an attack. Now let's assume @HackRead might be a possible suspect and you'd like to learn a bit more about possible additional suspects. In reality @HackRead HQ is in Milan, Italy. Perhaps Milan then might be a location for other attackers. I can feed  in Twittter handles from my retweet and mentions network above, query the Twitter API with very specific geocode, and lock it within five miles of the center of Milan.
The results are immediate per Figure 4.

Figure 4: GeoLocation code and results
Obviously, as these Twitter accounts aren't actual attackers, their retweets aren't actually pertinent to our presumed attack scenario, but they definitely retweeted @computerweekly (seen in retweets and mentions) from within five miles of the center of Milan. If @HackRead were the leader of an organization, and we believed that associates were assumed to be within geographical proximity, geolocation via the Twitter API could be quite useful. Again, these are all used as thematic examples, no actual attacks should be related to any of these accounts in any way.

Fast Frugal Trees (decision trees) for prioritizing criticality

With the abundance of data, and often subjective or biased analysis, there are occasions where a quick, authoritative decision can be quite beneficial. Fast-and-frugal trees (FFTs) to the rescue. FFTs are simple algorithms that facilitate efficient and accurate decisions based on limited information.
Nathaniel D. Phillips, PhD created FFTrees for R to allow anyone to easily create, visualize and evaluate FFTs. Malcolm Gladwell has said that "we are suspicious of rapid cognition. We live in a world that assumes that the quality of a decision is directly related to the time and effort that went into making it.” FFTs, and decision trees at large, counter that premise and aid in the timely, efficient processing of data with the intent of a quick but sound decision. As with so much of information security, there is often a direct correlation with medical, psychological, and social sciences, and the use of FFTs is no different. Often, predictive analysis is conducted with logistic regression, used to "describe data and to explain the relationship between one dependent binary variable and one or more nominal, ordinal, interval or ratio-level independent variables." Would you prefer logistic regression or FFTs?

Figure 5: Thanks, I'll take FFTs
Here's a text book information security scenario, often rife with subjectivity and bias. After a breach, and subsequent third party risk assessment that generated a ton of CVSS data, make a fast decision about what treatments to apply first. Because everyone loves CVSS.

Figure 6: CVSS meh
Nothing like a massive table, scored by base, impact, exploitability, temporal, environmental, modified impact, and overall scores, all assessed by a third party assessor who may not fully understand the complexities or nuances of your environment. Let's say our esteemed assessor has decided that there are 683 total findings, of which 444 are non-critical and 239 are critical. Will FFTrees agree? Nay! First, a wee bit of R code.

cvss <- c:="" coding="" csv="" p="" r="" read.csv="" rees="">cvss.fft <- data="cvss)</p" fftrees="" formula="critical">plot(cvss.fft, what = "cues")
     main = "CVSS FFT",
     decision.names = c("Non-Critical", "Critical"))

Guess what, the model landed right on impact and exploitability as the most important inputs, and not just because it's logically so, but because of their position when assessed for where they fall in the area under the curve (AUC), where the specific curve is the receiver operating characteristic (ROC). The ROC is a "graphical plot that illustrates the diagnostic ability of a binary classifier system as its discrimination threshold is varied." As for the AUC, accuracy is measured by the area under the ROC curve where an area of 1 represents a perfect test and an area of .5 represents a worthless test. Simply, the closer to 1, the better. For this model and data, impact and exploitability are the most accurate as seen in Figure 7.

Figure 7: Cue rankings prefer impact and exploitability
The fast and frugal tree made its decision where impact and exploitability with scores equal or less than 2 were non-critical and exploitability greater than 2 was labeled critical, as seen in Figure 8.

Figure 8: The FFT decides
Ah hah! Our FFT sees things differently than our assessor. With a 93% average for performance fitting (this is good), our tree, making decisions on impact and exploitability, decides that there are 444 non-critical findings and 222 critical findings, a 17 point differential from our assessor. Can we all agree that mitigating and remediating critical findings can be an expensive proposition? If you, with just a modicum of data science, can make an authoritative decision that saves you time and money without adversely impacting your security posture, would you count it as a win? Yes, that was rhetorical.

Note that the FFTrees function automatically builds several versions of the same general tree that make different error trade-offs with variations in performance fitting and false positives. This gives you the option to test variables and make potentially even more informed decisions within the construct of one model. Ultimately, fast frugal trees make very fast decisions on 1 to 5 pieces of information and ignore all other information. In other words, "FFTrees are noncompensatory, once they make a decision based on a few pieces of information, no additional information changes the decision."

Finally, let's take a look at monitoring user logon anomalies in high volume environments with Time Series Regression (TSR). Much of this work comes courtesy of Eric Kapfhammer, our lead data scientist on our Microsoft Windows and Devices Group Blue Team. The ideal Windows Event ID for such activity is clearly 4624: an account was successfully logged on. This event is typically one of the top 5 events in terms of volume in most environments, and has multiple type codes including Network, Service, and RemoteInteractive.
User accounts will begin to show patterns over time, in aggregate, including:
  • Seasonality: day of week, patch cycles, 
  • Trend: volume of logons increasing/decreasing over time
  • Noise: randomness
You could look at 4624 with a Z-score model, which sets a threshold based on the number of standard deviations away from an average count over a given period of time, but this is a fairly simple model. The higher the value, the greater the degree of “anomalousness”.
Preferably, via Time Series Regression (TSR), your feature set is more rich:
  • Statistical method for predicting a future response based on the response history (known as autoregressive dynamics) and the transfer of dynamics from relevant predictors
  • Understand and predict the behavior of dynamic systems from experimental or observational data
  • Commonly used for modeling and forecasting of economic, financial and biological systems
How to spot the anomaly in a sea of logon data?
Let's imagine our user, DARPA-549521, in the SUPERSECURE domain, with 90 days of aggregate 4624 Type 10 events by day.

Figure 9: User logon data
With 210 line of R, including comments, log read, file output, and graphing we can visualize and alert on DARPA-549521's data as seen in Figure 10

Figure 10: User behavior outside the confidence interval
We can detect when a user’s account exhibits  changes in their seasonality as it relates to a confidence interval established (learned) over time. In this case, on 27 AUG 2017, the user topped her threshold of 19 logons thus triggering an exception. Now imagine using this model to spot anomalous user behavior across all users and you get a good feel for the model's power.
Eric points out that there are, of course, additional options for modeling including:
  • Seasonal and Trend Decomposition using Loess (STL)
    • Handles any type of seasonality ~ can change over time
    • Smoothness of the trend-cycle can also be controlled by the user
    • Robust to outliers
  • Classification and Regression Trees (CART)
    • Supervised learning approach: teach trees to classify anomaly / non-anomaly
    • Unsupervised learning approach: focus on top-day hold-out and error check
  • Neural Networks
    • LSTM / Multiple time series in combination
These are powerful next steps in your capabilities, I want you to be brave, be creative, go forth and add elements of data science and visualization to your practice. R and Python are well supported and broadly used for this mission and can definitely help you detect attackers faster, contain incidents more rapidly, and enhance your in-house detection and remediation mechanisms.
All the code as I can share is here; sorry, I can only share the TSR example without the source.
All the best in your endeavors!
Cheers...until next time.

by Russ McRee ( at November 20, 2017 12:27 AM

November 17, 2017

Colin Percival

FreeBSD/EC2 on C5 instances

Last week, Amazon released the "C5" family of EC2 instances, continuing their trend of improving performance by both providing better hardware and reducing the overhead associated with virtualization. Due to the significant changes in this new instance family, Amazon gave me advance notice of their impending arrival several months ago, and starting in August I had access to (early versions of) these instances so that I could test FreeBSD on them. Unfortunately the final launch date took me slightly by surprise — I was expecting it to be later in the month — so there are still a few kinks which need to be worked out for FreeBSD to run smoothly on C5 instances. I strongly recommend that you read the rest of this blog post before you use FreeBSD on EC2 C5 instances. (Or possibly skip to the end if you're not interested in learning about any of the underlying details.)

November 17, 2017 01:45 AM

November 15, 2017

Everything Sysadmin

Productivity When You Just Can't Even

Ryn Daniels has written an excellent piece about time management and productivity when you are burned out. The advice is spot on. I also do these things when I'm not feeling my best too.

"It's one thing to talk about productivity when you're already feeling motivated and productive, but how do you get things done when it's hard to even make it through the day?"

by Tom Limoncelli at November 15, 2017 05:00 PM

November 14, 2017

LZone - Sysadmin

USB seq nnnn is taking a long time

When your dmesg/journalctl/syslog says something like
Nov 14 21:43:12 Wolf systemd-udevd[274]: seq 3634 '/devices/pci0000:00/0000:00:14.0/usb2/2-1' is taking a long time
then know that the only proper manly response can be
systemctl restart udev
Don't allow for disrespectful USB messages!!!

November 14, 2017 08:49 PM

November 13, 2017

Steve Kemp's Blog

Paternity-leave is half-over

I'm taking the month of November off work, so that I can exclusively take care of our child. Despite it being a difficult time, with him teething, it has been a great half-month so far.

During the course of the month I've found my interest in a lot of technological things waning, so I've killed my account(s) on a few platforms, and scaled back others - if I could exclusively do child-care for the next 20 years I'd be very happy, but sadly I don't think that is terribly realistic.

My interest in things hasn't entirely vanished though, to the extent that I found the time to replace my use of etcd with consul yesterday, and I'm trying to work out how to simplify my hosting setup. Right now I have a bunch of servers doing two kinds of web-hosting:

Hosting static-sites is trivial, whether with a virtual machine, via Amazons' S3-service, or some other static-host such as netlify.

Hosting for "dynamic stuff" is harder. These days a trend for "serverless" deployments allows you to react to events and be dynamic, but not everything can be a short-lived piece of ruby/javascript/lambda. It feels like I could setup a generic platform for launching containers, or otherwise modernising FastCGI, etc, but I'm not sure what the point would be. (I'd still be the person maintaining it, and it'd still be a hassle. I've zero interest in selling things to people, as that only means more support.)

In short I have a bunch of servers, they mostly tick over unattended, but I'm not really sure I want to keep them running for the next 10+ years. Over time our child will deserve, demand, and require more attention which means time for personal stuff is only going to diminish.

Simplify things now wouldn't be a bad thing to do, before it is too late.

November 13, 2017 10:00 PM

November 11, 2017

OpenVPN with Private Internet Access and port forwarding


This post will show my setup using PIA (Private Internet Access) with OpenVPN on a Linux machine. Specifically, where only certain applications will utilize the VPN and the rest of the traffic will go out the normal ISP's default route. It will also show how to access the PIA API via a shell script, to open a forwarding port for inbound traffic. Lastly, I will show how to take all of the OpenVPN and PIA information and feed it to programs like aria2c or curl. The examples below were done on Ubuntu 16.04.

Packages and PIA Setup

Go signup for a PIA account.

# Install the packages you need, example uses apt-get
sudo apt-get install openvpn curl unzip

# make dir for PIA files and scripts
sudo mkdir -p /etc/openvpn/pia
cd /usr/local/etc/openvpn

# grab PIA openvpn files and unzip
sudo curl -o $url && sudo unzip

OpenVPN password file

Now that we have PIA login info lets make password file so we don't have to put in a password every time we start OpenVPN. We just need to make a file with the PIA username on one line and the PIA password on the second line. So just use you favorite text editor and do this. The file should be called "pass" and put in the "/etc/openvpn/pia" directory. The scripts that are used later depend on this file being called "pass" and put in this specific directory. An example of what the file looks like is below.


Change permission on this file so only root can read it

sudo chmod 600 /etc/openvpn/pia/pass

OpenVPN config file

This is the OpenVPN config file that works with PIA, and that also utilizes the scripts that will be talked about further down in the page. Use your favorite editor and copy and paste this text to a file called "pia.conf" and put in the "/etc/openvpn/pia" directory.

# PIA OpenVPN client config file 
dev tun

# make sure the correct protocol is used
proto udp

# use the vpn server of your choice
# only use one server at a time
# the ip addresses can change, so use dns names not ip's
# find more server names in .ovpn files
# only certain gateways support port forwarding
#remote 1198
#remote 1198
#remote 1198
#remote 1198
remote 1198

resolv-retry infinite
cipher aes-128-cbc
auth sha1

# ca.crt and pem files from downloaded from pia
ca /etc/openvpn/pia/ca.rsa.2048.crt
crl-verify /etc/openvpn/pia/crl.rsa.2048.pem

remote-cert-tls server

# path to password file so you don't have to input pass on startup
# file format is username on one line password on second line
# make it only readable by root with: chmod 600 pass
auth-user-pass /etc/openvpn/pia/pass

# this suppresses the caching of the password and user name

verb 1
reneg-sec 0

# allows the ability to run user-defined script
script-security 2

# Don't add or remove routes automatically, pass env vars to route-up

# run our script to make routes
route-up "/etc/openvpn/pia/ up"

OpenVPN route script

This is the script that the OpenVPN client will run at the end of startup. The magic happens in this script. Without this script OpenVPN will start the client and make the default route for the box the vpn connection. If you want that then go into the pia.conf file and comment out the "script-security 2", "route-noexec", and "route up ..." lines, and just fire up the client "sudo openvpn --config /etc/openvpn/pia/pia.conf" and your done.

If you don't want the vpn to take over your default route then let's keep going. Now that you have left those lines in the pia.conf file, the following script will be run when the client starts, and it will set up a route that does not take over the default gateway, but just adds secondary vpn gateway for programs to use. Open your favorite text editor and copy in the script below into the file "/etc/openvpn/pia/".

# script used by OpenVPN to setup a route on Linux.
# used in conjunction with OpenVPN config file options
# script-security 2, route-noexec, route-up 
# script also requires route table rt2
# sudo bash -c 'echo "1 rt2" >> /etc/iproute2/rt_tables

# openvpn variables passed in via env vars

if [ -z $int ] || [ -z $iplocal ] || [ -z $ipremote ] || [ -z $gw ]; then
  echo "No env vars found. Use this script with an OpenVPN config file "
  exit 1

help() {
  echo "For setting OpenVPN routes on Linux."
  echo "Usage: $0 up or down"

down() {
  # delete vpn route if found
  ip route flush table $rtname
  if [ $? -eq 0 ]; then
    echo "Successfully flushed route table $rtname"
    echo "Failed to flush route table $rtname"

up() {
  # using OpenVPN env vars that get set when it starts, see man page
  echo "Tunnel on interface $int. File /tmp/vpnint"
  echo $int > /tmp/vpnint
  echo "Local IP is         $iplocal. File /tmp/vpnip"
  echo $iplocal > /tmp/vpnip
  echo "Remote IP is        $ipremote"
  echo "Gateway is          $gw"

  down # remove any old routes

  ip route add default via $gw dev $int table $rtname
  if [ $? -eq 0 ]; then
    echo "Successfully added default route $gw"
    echo "Failed to add default route for gateway $gw"
  ip rule add from $iplocal/32 table $rtname
  if [ $? -eq 0 ]; then
    echo "Successfully added local interface 'from' rule for $iplocal"
    echo "Failed to add local interface 'from' rule for $iplocal"
  ip rule add to $gw/32 table $rtname
  if [ $? -eq 0 ]; then
    echo "Successfully added local interface 'to' rule for $gw"
    echo "Failed to add local interface 'to' rule for $gw"

  # PIA port forwarding, only works with certain gateways
  # No US locations, closest US is Toronto and Montreal
  # no network traffic works during exec of this script
  # things like curl hang if not backgrounded
  $ovpnpia/ &

case $1 in
  "up") up;;
  "down") down;;
  *) help;;

# always flush route cache 
ip route flush cache

Now run some final commands to get the script ready to work

# make the new script executable
sudo chmod 755 /etc/openvpn/pia/

# make a new route table rt2 in linux for the script to use
# this only has to be run once before you connect the first time
sudo bash -c 'echo "1 rt2" >> /etc/iproute2/rt_tables'

PIA port forward script

The following script is run by the script. It will contact a PIA server and tell it to open a port for incoming traffic on your vpn connection. This is so people on the internet can contact your machine through the vpn connection. Just a important note that currently only a certain list of PIA gateways support port forwarding. See the PIA support article on this for more info. Now, open your favorite text editor and copy in the script below into the file "/etc/openvpn/pia/".

# Get forward port info from PIA server

client_id=$(head -n 100 /dev/urandom | sha256sum | tr -d " -")

echo "Making port forward request..."

curl --interface $(cat /tmp/vpnint) $url 2>/dev/null > /tmp/vpnportfwhttp

if [ $? -eq 0 ]; then
  port_fw=$(grep -o '[0-9]\+' /tmp/vpnportfwhttp)
  [ -f /tmp/vpnportfw ] && rm /tmp/vpnportfw
  echo $port_fw > /tmp/vpnportfw
  echo "Forwarded port is $port_fw"
  echo "Forwarded port is in file /tmp/vpnportfw"
  echo "Curl failed to get forwarded PIA port in some way"
# make the new script executable
sudo chmod 755 /etc/openvpn/pia/

Starting OpenVPN

Finally we can start OpenVPN to connect with PIA. To do this run the the following command. It will keep the connection in the foreground so you can watch the output.

sudo openvpn --config /etc/openvpn/pia/pia.conf

During startup the OpenVPN client and both of the scripts we made will report on the screen data about the connection and if there were any errors. The output will look like the following example.

Fri Nov 10 19:40:50 2017 OpenVPN 2.3.10 x86_64-pc-linux-gnu [SSL (OpenSSL)] [LZO] [EPOLL] [PKCS11] [MH] [IPv6] built on Jun 22 2017
Fri Nov 10 19:40:50 2017 library versions: OpenSSL 1.0.2g  1 Mar 2016, LZO 2.08
Fri Nov 10 19:40:50 2017 NOTE: the current --script-security setting may allow this configuration to call user-defined scripts
Fri Nov 10 19:40:50 2017 UDPv4 link local: [undef]
Fri Nov 10 19:40:50 2017 UDPv4 link remote: [AF_INET]
Fri Nov 10 19:40:50 2017 [dbacd7b38d135021a698ed95e8fec612] Peer Connection Initiated with [AF_INET]
Fri Nov 10 19:40:53 2017 TUN/TAP device tun0 opened
Fri Nov 10 19:40:53 2017 do_ifconfig, tt->ipv6=0, tt->did_ifconfig_ipv6_setup=0
Fri Nov 10 19:40:53 2017 /sbin/ip link set dev tun0 up mtu 1500
Fri Nov 10 19:40:53 2017 /sbin/ip addr add dev tun0 local peer
Tunnel on interface tun0. File /tmp/vpnint
Local IP is File /tmp/vpnip
Remote IP is
Gateway is
Successfully flushed route table rt2
Successfully added default route
Successfully added local interface 'from' rule for
Successfully added local interface 'to' rule for
Fri Nov 10 19:40:53 2017 Initialization Sequence Completed
Making port forward request...
Forwarded port is 40074
Forwarded port is in file /tmp/vpnportfw

Using the vpn connection

When the vpn started it dropped some files in /tmp. These files have the ip and port info we need to give to different programs when the startup. The scripts created the following files.

  • /tmp/vpnip - ip address of the vpn
  • /tmp/vpnportfw - incomming port being forwarded from the internet to the vpn
  • /tmp/vpnint - interface of the vpn

Now you can use this info when you start certain programs. Here are some examples.

# get vpn incomming port
pt=$(cat /tmp/vpnportfw)

# get vpn ip
ip=$(cat /tmp/vpnip)

# get vpn interface
int=$(cat /tmp/vpnint)

# wget a file via vpn
wget --bind-address=$ip

# curl a file via vpn
curl --interface $int

# ssh to server via vpn
ssh -b $ip 

# rtorrent 
/usr/bin/rtorrent -b $ip -p $pt-$pt -d /tmp

# start aria2c and background. use aria2 WebUI to connect download files
aria2c --interface=$ip --listen-port=$pt --dht-listen-port=$pt > /dev/null 2>&1 &

Final notes and warnings

If you start any programs and don't specifically bind them to the vpn interface or its ip address their connection will go out the default interface for the machine. Please remember this setup only sends specific traffic through the vpn so things like DNS requests still go through the non-vpn default gateway.

Remember only certain PIA gateways support port forwarding so if it is not working, try another PIA gateway.

PIA has a Linux vpn client that you can download and use if you are into GUI's.

by at November 11, 2017 03:50 AM

November 10, 2017

Lies, damn lies, and spammers in disguise

Everyone get so many unsolicited commercial emails these days that you just become blind at them, at best. Sometimes they are clearly, expressly commercial. Other times, they try to pass through your attention and your spam checker by disguising themselves as legitimate emails. I have a little story about that.

A couple of weeks ago I got yet another spammy mail from. It was evidently sent through a mass mailing and, as such, also included an unsubscribe link, however the guy was trying to legitimate his spam by saying that he approached me specifically because a colleague referred me to him; in addition, I felt that some keywords were added to his message only to make it sound “prettier” or even more legitimate.

I usually don’t spend time on spammers, but when I do I try to do it well. And in this occasion I had a little time to spend on it, and I did. On October 30th I was walking to my eye doctor’s while I saw an email notification on my phone. The email said the following (note: the highlights are mine):

I was sent your way as the person who is responsible for application security testing and understand XXXXXX would be of interest to Telenor. To what lies within our next generation penetration testing abilities, XXXXXX can bring to you:
  • Periodic pen tests in uncovering serious vulnerabilities and (lack of) continuous application testing to support dynamic DevOps SDLC.
  • An ROI of 53% or higher with the ability to Identify, locate and resolve all vulnerabilities
  • Averaging 35-40% critical vulnerabilities found overall, others average 28%.
  • Over 500 international top class security researchers deployed in large teams of 40-60 ethical hackers.
We have pioneered a disruptive, ethical hacker powered application testing approach deploying large teams of senior security researchers with hacker mimicking mindsets to rapidly uncover exploits in your applications.
I would like to find some time for a conversation during November, to give you the insight into why Microsoft and Hewlett-Packard Enterprise have invested in us and why SAP is our top global reseller.
Many thanks,
Geoffrey XXXXXX
Business Development Manager

Now, we have our mailboxes on GMail and my usual method of procedure for such spammy crap from clueless salespeople is:

  • block the sender
  • unsubscribe
  • report the mail as spam

But now it’s different: first, this guy is probably trying to legitimate himself with lies for sending shit to me that I had not requested; second, despite the “personal” approach, this was clearly a mass-mailing so the whole story was clearly a lie; third, I am going to sit and wait at the doctor’s for some time. I could invest that time to run down the guy. My reply:

Kindly let me know who sent you my way, so that I can double check that. Then, maybe, we can talk.

The guy replies soon after. New mail, new lie:

Hi Marco,

Apologies for not including that in the email and I am more than happy to say how I got to you.

I have spoken to Ragnar Harper around how this may become beneficial to Telenor and he has mentioned your name during the conversations. As I have been unable to get back in contact with Ragnar I thought it best that I could gain your input into this moving forward by having some time aligned in our diaries to discuss more?

[…additional crap redacted…]

Not only I had never met or talked with any Ragnar Harper: the guy was not in our Company’s ERP. In my eyes, Mr.Geoffrey was lying right down the line. Time to get rid of him:

There is no Ragnar Harper here

Your emails will hereby be blocked and reported as Spam

— MM

You may think he stopped there. Well, he didn’t:

Apologies but I just phoned the switchboard and he left last month.

Above is his linkedin profile as well.
Sorry for the confusion.
So he had this beautiful relation with Ragnar Harper, to the point that they talked together about Geoff’s products and Ragnar pointed him to me as the “person responsible for application security” (which I am not…) and he didn’t even know that Ragnar had left the company. But there is more: to get to know that, he didn’t call Ragnar: he had to call the switchboard and check LinkedIn. Geoff is clearly clutching at straws. My reply:

You just picked up a random name. You are trying to sell me something for security, which should entail a certain amount of trust in your company, and you start the interaction with a lie: not bad for someone moved by an “ethical hacking” spirit.

There can’t be any trust between me and your company. Your messages are already landing in my spam folder and that’s where they shall be. Just go and try your luck with someone else, “Geoffrey”.

The king is naked, they say. Now he gotta stop, you think. He didn’t:

Thank you for your time Marco and I am sorry that you feel that way.

If you believe that it has started on a lie then that may well be your choice as I have been in contact with Ragnar since my days at YYYYYY before joining XXXXXX and we have yet to catch up this month since he has now departed. I shall head elsewhere as requested as that is not the kind of reputation that we uphold here.

Best wishes,


Uh, wait a minute. Did I beat the guy for no reason? Could it be that he actually knew this ex-colleague Ragnar Harper and I am assuming too much? Is it all a misunderstanding? If it is, I want to know and apologise. As said, I had never met or talked to Ragnar Harper, but I can still try to contact him through LinkedIn:

Question from an ex colleague

Hei Ragnar. My name is Marco Marongiu, I am the Head of IT in Telenor Digital. I apologize for bothering you, it’s just a short question: have you referred me to a Sales person called Geoffrey XXXXXX from a Security company called XXXXXX?

This person approached me via email. Long story short, he says he knows you personally and that you sent him my way (to use his words). Is it something that you can confirm?

We two have never met in Telenor so that sounded strange and I handled him as a spammer. But if that is actually true I do want to send the guy my apologies. Thanks in any case

Med vennlig hilsen

— Marco

I am grateful that Ragnar took some time to reply and confirm my suspect: he had never known or met the guy. I thanked Ragnar and stopped talking to “Geoffrey”. At the same time I thought it was a good story to tell, so here we go.



Tagged: spammers

by bronto at November 10, 2017 07:00 PM

November 08, 2017

Cryptography Engineering

A few thoughts on

(Warning: nerdy inside-baseball academic blog post follows. If you’re looking for exciting crypto blogging, try back in a couple of days.)

If there’s one thing that academic computer scientists love (or love to hate), it’s comparing themselves to other academics. We don’t do what we do for the big money, after all. We do it — in large part — because we’re curious and want to do good science. (Also there’s sometimes free food.) But then there’s a problem: who’s going to tell is if we’re doing good science?

To a scientist, the solution seems obvious. We just need metrics. And boy, do we get them. Modern scientists can visit Google Scholar to get all sorts of information about their citation count, neatly summarized with an “H-index” or an “i10-index”. These metrics aren’t great, but they’re a good way to pass an afternoon filled with self-doubt, if that’s your sort of thing.

But what if we want to do something more? What if we want to compare institutions as well as individual authors? And even better, what if we could break those institutions down into individual subfields? You could do this painfully on Google Scholar, perhaps. Or you could put your faith in the abominable and apparently wholly made-up U.S. News rankings, as many academics (unfortunately) do.

Alternatively, you could actually collect some data about what scientists are publishing, and work with that.

This is the approach of a new site called “Computer Science Rankings”. As best I can tell, CSRankings is largely an individual project, and doesn’t have the cachet (yet) of U.S. News. At the same time, it provides researchers and administrators with something they love: another way to compare themselves, and to compare different institutions. Moreover, it does so with real data (rather than the Ouija board and blindfold that U.S. News uses). I can’t see it failing to catch on.

And that worries me, because the approach of CSRankings seems a bit arbitrary. And I’m worried about what sort of things it might cause us to do.

You see, people in our field take rankings very seriously. I know folks who have moved their families to the other side of the country over a two-point ranking difference in the U.S. News rankings — despite the fact that we all agree those are absurd. And this is before we consider the real impact on salaries, promotions, and awards of rankings (individual and institutional). People optimize their careers and publications to maximize these stats, not because they’re bad people, but because they’re (mostly) rational and that’s what rankings inspire rational people do.

To me this means we should think very carefully about what our rankings actually say.

Which brings me to the meat of my concerns with CSRankings. At a glance, the site is beautifully designed. It allows you to look at dozens of institutions, broken down by CS subfield. Within those subfields it ranks institutions by a simple metric: adjusted publication counts in top conferences by individual authors.

The calculation isn’t complicated. If you wrote a paper by yourself and had it published in one of the designated top conferences in your field, you’d get a single point. If you wrote a paper with a co-author, then you’d each get half a point. If you wrote a paper that doesn’t appear in a top conference, you get zero points. Your institution gets the sum-total of all the points its researchers receive.

If you believe that people are rational actors optimize for rankings, you might start to see the problem.

First off, what CSRankings is telling us is that we should ditch those pesky co-authors. If I could write a paper with one graduate student, but a second student also wants to participate, tough cookies. That’s the difference between getting 1/2 a point and 1/3 of a point. Sure, that additional student might improve the paper dramatically. They might also learn a thing or two. But on the other hand, they’ll hurt your rankings.

(Note: currently on CSRankings, graduate students at the same institution don’t get included in the institutional rankings. So including them on your papers will actually reduce your school’s rank.)

I hope it goes without saying that this could create bad incentives.

Second, in fields that mix systems and theory — like computer security — CSRankings is telling us that theory papers (which typically have fewer authors) should be privileged in the rankings over systems papers. This creates both a distortion in the metrics, and also an incentive (for authors who do both types of work) to stick with the one that produces higher rankings. That seems undesirable. But it could very well happen if we adopt these rankings uncritically.

Finally, there’s this focus on “top conferences”. One of our big problems in computer science is that we spend a lot of our time scrapping over a very limited number of slots in competitive conferences. This can be ok, but it’s unfortunate for researchers whose work doesn’t neatly fit into whatever areas those conference PCs find popular. And CSRankings gives zero credit for publishing anywhere but those top conferences, so you might as well forget about that.

(Of course, there’s a question about what a “top conference” even is. In Computer Security, where I work, CSRankings does not consider NDSS to be a top conference. That’s because only three conferences are permitted for each field. The fact that this number seems arbitrary really doesn’t help inspire a lot of confidence in the approach.)

So what can we do about this?

As much as I’d like to ditch rankings altogether, I realize that this probably isn’t going to happen. Nature abhors a vacuum, and if we don’t figure out a rankings system, someone else will. Hell, we’re already plagued by U.S. News, whose methodology appears to involve a popcorn machine and live tarantulas. Something, anything, has to be better than this.

And to be clear, CSRankings isn’t a bad effort. At a high level it’s really easy to use. Even the issues I mention above seem like things that could be addressed. More conferences could be added, using some kind of metric to scale point contributions. (This wouldn’t fix all the problems, but would at least mitigate the worst incentives.) Statistics could perhaps be updated to adjust for graduate students, and soften the blow of having co-authors. These things are not impossible.

And fixing this carefully seems really important. We got it wrong in trusting U.S. News. What I’d like is this time for computer scientists to actually sit down and think this one out before someone imposes a ranking system on top of us. What behaviors are we trying to incentivize for? Is it smaller author lists? Is it citation counts? Is it publishing only in a specific set of conferences?

I don’t know that anyone would agree uniformly that these should be our goals. So if they’re not, let’s figure out what they really are.

by Matthew Green at November 08, 2017 04:28 PM

November 01, 2017

Steve Kemp's Blog

Possibly retiring

For the past few years I've hosted a service for spam-testing blog/forum comments, and I think it is on the verge of being retired.

The service presented a simple API for deciding whether an incoming blog/forum comment was SPAM, in real-time. I used it myself for two real reasons:

  • For the Debian Administration website.
    • Which is now retired.
  • For my blog
    • Which still sees a lot of spam comments, but which are easy to deal with because I can execute Lua scripts in my mail-client

As a result of the Debian-Administration server cleanup I'm still in the process of tidying up virtual machines, and servers. It crossed my mind that retiring this spam-service would allow me to free up another host.

Initially the service was coded in Perl using XML/RPC. The current version of the software, version 2, is written as a node.js service, and despite the async-nature of the service it is still too heavy-weight to live on the host which runs most of my other websites.

It was suggested to me that rewriting it in golang might allow it to process more requests, with fewer resources, so I started reimplementing the service in golang at 4AM this morning:

The service does the minimum:

  • Receives incoming HTTP POSTS
  • Decodes the body to a struct
  • Loops over that struct and calls each "plugin" to process it.
    • If any plugin decides this is spam, it returns that result.
  • Otherwise if all plugins have terminated then it decides the result is "OK".

I've ported several plugins, I've got 100% test-coverage of those plugins, and the service seems to be faster than the node.js version - so there is hope.

Of course the real test will be when it is deployed for real. If it holds up for a few days I'll leave it running. Otherwise the retirement notice I placed on the website, which chances are nobody will see, will be true.

The missing feature at the moment is keeping track of the count of spam-comments rejected/accepted on a per-site basis. Losing that information might be a shame, but I think I'm willing to live with it, if the alternative is closing down..

November 01, 2017 10:00 PM

Anton Chuvakin - Security Warrior

Monthly Blog Round-Up – October 2017

Here is my next monthly "Security Warrior" blog round-up of top 5 popular posts based on last
month’s visitor data  (excluding other monthly or annual round-ups):
  1. “New SIEM Whitepaper on Use Cases In-Depth OUT!” (dated 2010) presents a whitepaper on select SIEM use cases described in depth with rules and reports [using now-defunct SIEM product]; also see this SIEM use case in depth and this for a more current list of popular SIEM use cases. Finally, see our 2016 research on developing security monitoring use cases here – and we are updating it now.
  2. Why No Open Source SIEM, EVER?” contains some of my SIEM thinking from 2009 (oh, wow, ancient history!). Is it relevant now? You be the judge.  Succeeding with SIEM requires a lot of work, whether you paid for the software, or not. BTW, this post has an amazing “staying power” that is hard to explain – I suspect it has to do with people wanting “free stuff” and googling for “open source SIEM” … 
  3. Simple Log Review Checklist Released!” is often at the top of this list – this rapildy aging checklist is still a very useful tool for many people. “On Free Log Management Tools” (also aged a bit by now) is a companion to the checklist (updated version)
  4. Again, my classic PCI DSS Log Review series is extra popular! The series of 18 posts cover a comprehensive log review approach (OK for PCI DSS 3+ even though it predates it), useful for building log review processes and procedures, whether regulatory or not. It is also described in more detail in our Log Management book and mentioned in our PCI book (now in its 4th edition!) – note that this series is mentioned in some PCI Council materials. 
  5. “SIEM Bloggables”  is a very old post, more like a mini-paper on  some key aspects of SIEM, use cases, scenarios, etc as well as 2 types of SIEM users. Still very relevant, if not truly modern.
In addition, I’d like to draw your attention to a few recent posts from my Gartner blog [which, BTW, now has more than 5X of the traffic of this blog]: 

Current research on SOAR:
Miscellaneous fun posts:

(see all my published Gartner research here)
Also see my past monthly and annual “Top Popular Blog Posts” – 2007, 2008, 2009, 2010, 2011, 2012, 2013, 2014, 2015, 2016.

Disclaimer: most content at SecurityWarrior blog was written before I joined Gartner on August 1, 2011 and is solely my personal view at the time of writing. For my current security blogging, go here.

Previous post in this endless series:

by Anton Chuvakin ( at November 01, 2017 05:44 PM

October 31, 2017

Fall cleaning: shutting down some of my projects

The post Fall cleaning: shutting down some of my projects appeared first on

Just a quick heads-up, I'm taking several of my projects offline as they prove to be too much of a hassle.

Mailing List Archive

Almost 2 years ago, I started a project to mirror/archive public mailing lists in a readable, clean layout. It's been fun, but quite a burden to maintain. In particular;

  • The mailing list archive now consists of ~3.500.000 files, which meant special precautions with filesystems (running out of inodes) & other practical concerns related to backups.
  • Abuse & take-down reports: lots of folks mailed a public mailing list, apparently not realizing it would be -- you know -- public. So they ask their name & email to be removed from those that mirror it. Like me.

    Or, in one spectacular case, a take down request by Italian police because of some drug-related incident on the Jenkins public mailing list. Fun stuff. It involved threatening lawyer mails that I quickly complied with.

All of that combined, it isn't worth it. It got around 1.000 daily pageviews, mostly from the Swift mailing list community. Sorry folks, it's gone now.

This also means the death of the @foss_security & @oss_announce Twitter accounts, since those were built on top of the mailing list archive.

Manpages Archive

I ran a manpages archive on, but nobody used it (or knew about it), so I deleted it. Also hard to keep up to date, with different versions of packages & different manpages etc.

Also, it's surprisingly hard to download all manpages, extract them, turn them into HTML & throw them on a webserver. Who knew?!


This was a wild idea where I tried to crowdsource the meaning behind cryptocurrency price jumps. A coin can increase in value at a rate of 400% of more in just a few hours, the idea was to find the reason behind it and learn from it.

It also fueled the @CoinJumpApp account on Twitter, but that's dead now too.

The source is up on Github, but I'm no longer maintaining it.

More time for other stuff!

Either way, some projects just cause me more concern & effort than I care to put into them, so time to clean that up & get some more mental braincycles for different projects!

The post Fall cleaning: shutting down some of my projects appeared first on

by Mattias Geniar at October 31, 2017 08:25 PM

October 29, 2017

ncdu - for troubleshooting diskspace and inode issues

In my box of sysadmin tools there are multiple gems I use for troubleshooting servers. Since I work at a cloud provider sometimes I have to fix servers that are not mine. One of those tools is `ncdu`. It's a very usefull tool when a server has a full disk, both full of used space or full of used inodes. This article covers ncdu and shows the process of finding the culprit when you're out of disk space or inodes.

October 29, 2017 12:00 AM

October 27, 2017


Steve Marquess

Steve Marquess is leaving the OpenSSL project as of the 15th of November 2017.

The OpenSSL Management Committee (OMC) would like to wish him all the best for the future.

All communication that used to go to Steve Marquess directly, should now be sent to in the first instance.

Thanks for your contributions to the project over the years!

October 27, 2017 01:00 AM

October 25, 2017

The Lone Sysadmin

Consistency Is the Hobgoblin of Little Minds

Ever heard someone tell you “consistency is the hobgoblin of little minds?” They’re misquoting Ralph Waldo Emerson by leaving out an important word: foolish. That’s like leaving out the word “not” in a statement. The whole meaning changes because of the omission. We can all agree that “I am on fire” and “I am not […]

The post Consistency Is the Hobgoblin of Little Minds appeared first on The Lone Sysadmin. Head over to the source to read the full post!

by Bob Plankers at October 25, 2017 05:43 PM

Evaggelos Balaskas

Autonomous by Annalee Newitz


Autonomous by Annalee Newitz

The year is 2144. A group of anti-patent scientists are working to reverse engineer drugs in free labs, for (poor) people to have access to them. Agents of International Property Coalition are trying to find the lead pirate-scientist and stop any patent violation by any means necessary. In this era, without a franchise (citizenship) autonomous robots and people are slaves. But only a few of the bots have are autonomous. Even then, can they be free ? Can androids choose their own gender identity ? Transhumanism and extension life drugs are helping people to live a longer and better life.

A science fiction novel without Digital Rights Management (DRM).

Tag(s): Autonomous, books

October 25, 2017 11:09 AM

October 24, 2017

The Geekess

Binaries are for computers

[Note: comments on this post will be moderated by someone other than me.]

Sage in a purple tie and black shirt CC-BY-NC-ND Sage SharpCC-BY-NC-ND Sage Sharp

Recently, I’ve come to terms with the fact that I’m non-binary. Non-binary can be a bit of an umbrella term, but for me, being non-binary means I don’t identify as either a man or a woman. For me, gender is more like a 3D space, a universe of different traits. Most people gravitate towards a set of traits that we label as masculine or feminine. I don’t feel a strong pull towards being either really masculine or really feminine.

I’m writing this post for two reasons. The first reason is that representation matters. I know some non-binary people in tech and from the indie comics industry, but I’d love to see those voices and stories promoted. Hopefully being open and honest with people about my identity will help, both to raise awareness, and to give other people the courage to start an exploration into their own gender identity. I know talking with queer friends and reading comics helped me while I was working out my own gender identity.

The second reason I’m writing this is because there’s a couple ways allies can help me as I go through my transition:

  • Use my new name (Sage) and my correct pronouns (they/them)
  • Educate yourself on what it means to be non-binary
  • Think about how you use gender in your software and websites
  • Think about making your events more inclusive towards non-binary folks


I’ve changed my name to Sage Sharp.

I would appreciate it if you could use my new name. If you’re thinking about writing about me, there’s a section on writing about individuals who are transitioning in “The Responsible Communication Style Guide”. You should buy a digital copy!

Pronouns and Titles

I use the pronoun ‘they’. If you’ve never had someone ask for you to use a specific pronoun, this Robot Hugs comic does a good job of explaining how to handle it.

If you have to add a formal title before my last name, please use ‘Mx Sharp’. ‘Mx’ is a gender-neutral honorific, like ‘Mr’, ‘Ms’, or ‘Mrs’. Mx is pronounced in a couple different ways: /ˈməks/, /ˈmɪks/ or /ˈmʌks/ (miks or muks). I like pronouncing it ‘mux’ like the electronics part multiplexer, but pick whichever pronunciation works for you. If you want to get really formal and are reaching for a term like ‘lady/gentlemen’, I prefer the term ‘gentleperson’.

I’ve found positive gender-neutral terms to describe myself with, like “dapper” or “cute”. I cringe every time a stranger attempts to gender me, since they usually settle on feminine forms of address. I wish more people would use gender-neutral terms like “folks”, “friend”, “comrade”, or say “Hello everyone!” or “Hi y’all” and leave it at that.

Being able to write in a gender neutral way is hard and takes a lot of practice. It means shifting from gendered terms like “sister/brother” or “daughter/son” to gender-neutral terms like “sibling” or “kid”. It means getting comfortable with the singular they instead of ‘she’ or ‘he’. Sometimes there isn’t a gender neutral term for a relationship like “aunt/uncle” and it means you have to make up some new term, like “Titi”. There’s some lists of gender-neutral titles but in general, just ask what term to use.

If this is all new and bewildering to you, I recommend the book ‘The ABCs of LGBTQ+‘. Another good book is ‘You’re in the wrong bathroom‘ which breaks down some common myths about non-binary and trans folks.

Gender Forms

I’m really not looking forward to my gender being listed as ‘other’ or ‘prefer not to say’ on every gender form out there. I don’t even know if I can change my gender in social media, email, credit cards, banking… It’s a giant headache to change my name, let alone hope that technology systems will allow me to change my gender. It’s probably a separate post all itself, or a topic for my Diversity Deep Dives mailing list.

If you’re a programmer, website designer, or user experience person, ask yourself: Do you even need to collect information about a person’s gender? Could your website use gender neutral language? If you do need to address someone in a gendered way, maybe you just need to ask for their preferred pronouns instead of their gender? Could you drop the use of gendered honorifics, like ‘Miss’, ‘Mrs’ and ‘Mr’, or ‘Sir’ and ‘Madam’?

Inclusive Tech Groups

There’s a lot of tech spaces that are designed to help connect people from groups who are underrepresented in tech. Some tech groups are for “women” or “girls” (which can sometimes mean young women, and sometimes means adult women). It’s unclear whether non-binary folks are included in such groups, which puts me in the awkward position of asking all the groups I’m currently involved in if this is still a space for me.

I recommend reading Kat’s post on the design of gender-inclusive tech spaces. If you run a group or event targeted at gender minorities in tech, consider what you mean by “women only” and whether you want to be more inclusive towards non-binary folks that also want a space away from the patriarchy.

I know that in the past, a lot of folks looked up to me as ‘a woman in open source’. Some people felt I was a role model, and some people felt triumphant that I overcame a lot of sexism and a toxic environment to do some pretty badass technical work. Guess what? I’m still a badass. As a non-binary person, I’m still a minority gender in tech. So I’m still going to continue to be my badass self, taking on the patriarchy.

Promote Pronouns

When you meet someone, don’t assume what their pronouns are. As an ally, you can help by introducing yourself and normalizing pronoun usage. E.g. “Hi, my name is Victor, and I use he/him pronouns.”

If you’re a conference organizer, make sure all your name tags have a space for pronoun stickers. Have sheets of pronoun stickers at registration, and make sure the registration volunteers point out the pronoun badge stickers. If someone is confused about what the pronouns are, have a handout on pronouns and gender ready to give them. Wiscon is a conference that does a very good job with pronouns.

Don’t print pronouns collected from the registration system on badges without permission, or force everyone to put a pronoun on their badge. Some people might use different pronouns in conversation with different people, for example, if a person is “out” as non-binary to some friends but not their coworkers. Some people are genderfluid (meaning their feelings about their gender may change over time, even day to day). Some people might be questioning their gender, and not know what pronouns they want yet. Some people may prefer not to have a pronoun sticker at all.

The best practice is to provide space for people who want to provide their pronouns, but don’t force it on everyone.

What if people misgender you?

Some people who knew me under my old name might get confused when you use my new name. It’s perfectly fine to remind them of past work I did under my old name, while emphasizing my new name and pronouns. For example:

“Sage Sharp? Who’s that?”

“Sage is a diversity and inclusion consultant with Otter Tech. They were a Linux kernel developer for 10 years and wrote the USB 3.0 driver. They help run the Outreachy internship program.”

“Oh, you mean Sarah Sharp?”

“Yeah. They changed their name to Sage and they use ‘they’ pronouns now.”

I know it might be hard for people who have known me to get used to my new name and pronoun. You might even slip up in conversation with me. That’s ok, just correct the word and move on with your conversation. No need to apologize or call attention to it. We’re all humans, and retraining the language centers of our brains takes time. As long as I can see you’re trying, we’re cool.

What about your old accounts?

The internet never forgets. There will be old pictures of me, articles about me under my old name, etc. I’m fine with that, because that’s all a part of my past, who I was and the experiences that make me who I am. It’s as much a part of who I am as the tattoo on my arm. I don’t feel sad or weird looking at old pictures of myself. Seeing my longer haircut or myself in more feminine clothing can be surprising because I’ve changed so much, but after that initial reaction what I feel most is empathy for my past self.

At the same time, I’m also not that person any more. I’d like to see current pictures of me with my current name and correct pronoun.

If you see a news article that uses my old name, please let them know about my new name and pronouns. (But if it’s some troll site, don’t engage.) Several photos of my new style can be found here. If you see a social media website that uses my old name, don’t bother emailing me about it. I might have abandoned it, or found the name/gender change process to be too complex. Speaking of email, my old email addresses will still work, but I’ll respond back with my new email address. Please update your phone and email contacts to use the name ‘Sage Sharp’.

Phew, that was a lot to process!

We’ll keep it simple. Hi, my name is Sage Sharp, and I use ‘they’ pronouns. It’s nice to meet you!

by Sage at October 24, 2017 05:00 PM


Steve Henson

For as long as I have been involved in the OpenSSL project there has been one constant presence: Steve Henson. In fact he has been a part of the project since it was founded and he is the number 1 committer of all time (by a wide margin). I recall the first few times I had any dealings with him being somewhat in awe of his encyclopaedic knowledge of OpenSSL and all things crypto. Over the years Steve has made very many significant contributions both in terms of code but also in terms of being an active member of the management team.

I am sad to have to report that Steve has decided, for personal reasons, to move on to other things. The OpenSSL Management Committee (OMC) would like to wish him all the best for the future. In recognition of his huge contributions we will be listing him as an “OMC Emeritus” on our alumni page.

Good luck Steve!

October 24, 2017 01:00 AM

October 23, 2017

Cryptography Engineering

Attack of the week: DUHK

Before we get started, fair warning: this is going to be a post about a fairly absurd (but non-trivial!) attack on cryptographic systems. But that’s ok, because it’s based on a fairly absurd vulnerability.

This work comes from Nadia Heninger, Shaanan Cohney and myself, and follows up on some work we’ve been doing to look into the security of pseudorandom number generation in deployed cryptographic devices. We made a “fun” web page about it and came up with a silly logo. But since this affects something like 25,000 deployed Fortinet devices, the whole thing is actually kind of depressing.

The paper is called “Practical state recovery attacks against legacy RNG implementation“, and it attacks an old vulnerability in a pseudorandom number generator called ANSI X9.31, which is used in a lot of government certified products. The TL;DR is that this ANSI generator really sucks, and is easy to misuse. Worse, when it’s misused — as it has been — some very bad things can happen to the cryptography that relies on it.

First, some background.

What is an ANSI, and why should I care?

A pseudorandom number generator (PRG) is a deterministic algorithm designed to “stretch” a short random seed into a large number of apparently random numbers. These algorithms are used ubiquitously in cryptographic software to supply all of the random bits that our protocols demand.

PRGs are so important, in fact, that the U.S. government has gone to some lengths to standardize them. Today there are three generators approved for use in the U.S. (FIPS) Cryptographic Module Validation Program. Up until 2016, there were four. This last one, which is called the ANSI X9.31 generator, is the one we’re going to talk about here.

ANSI X9.31 is a legacy pseudorandom generator based on a block cipher, typically AES. It takes as its initial seed a pair of values (K, V) where K is a key and V is an initial “seed” (or “state”). The generator now produces a long stream of pseudorandom bits by repeatedly applying the block cipher in the crazy arrangement below:

A single round of the ANSI X9.31 generator instantiated using AES. The Ti value is a “timestamp”, usually generated using the system clock. Ri (at right) represents the output of the generator. The state Vi is updated at each round. However, the key K is fixed throughout the whole process, and never updates.

The diagram above illustrates one of the funny properties of the ANSI generator: namely, that while the state value V updates for each iteration of the generator, the key K never changes. It remains fixed throughout the entire process.

And this is a problem. Nearly twenty years ago, Kelsey, Schneier, Wagner and Hall pointed out that this fact makes the ANSI generator terribly insecure in the event that an attacker should ever learn the key K.

Specifically, if an attacker were to obtain K somehow, and then was able to learn only a single 16-byte raw output block (Ri) from a working PRG, she could do the following: (1) guess the timestamp T, (2) work backwards (decrypting using K) in order to recover the corresponding state value V, and now (3) run the generator forwards or backwards (with guesses for T) to obtain every previous and subsequent output of the generator.

Thus, if an application uses the ANSI generator to produce something like a random nonce (something that is typically sent in a protocol in cleartext), and also uses the generator to produce secret keys, this means an attacker could potentially recover those secret keys and completely break the protocol.

Of course, all of this requires that somehow the attacker learns the secret value K. At the time Kelsey et al. published their result, this was viewed as highly unlikely. After all, we’re really good at keeping secrets.

I assume you’re joking?

So far we’ve established that the ANSI generator is only secure if you can forever secure the value K. However, this seems fairly reasonable. Surely implementers won’t go around leaking their critical secrets all over the place. And certainly not in government-validated cryptographic modules. That would be crazy.

Yet crazy things do happen. We figured someone should probably check.

To see how the X9.31 key is managed in real products, our team developed a sophisticated analytic technique called “making a graduate student read every FIPS document on the CMVP website”. 

Most of the documents were fairly vague. And yet, a small handful of widely-used cryptographic modules had language that was troubling. Specifically, several vendors include language in their security policy that indicates the ANSI key was either hard-coded, or at least installed in a factory — as opposed to being freshly generated at each device startup.

Of even more concern: at least one of the hard-coded vendors was Fortinet, a very popular and successful maker of VPN devices and firewalls.

To get more specific, it turns out that starting (apparently in 2009, or perhaps earlier), every FortiOS 4.x device has shipped with a hardcoded value for K. This key has been involved in generating virtually every random bit used to establish VPN connections on those appliances, using both the TLS and IPSec protocols. The implication is that anyone with the resources to simply reverse-engineer the FortiOS firmware (between 2009 and today) could theoretically have been able to recover K themselves — and thus passively decrypt any VPN connection.

(Note: Independent of our work, the ANSI generator was replaced with a more secure alternative as of FortiOS 5.x. As a result of our disclosure, it has also been patched in FortiOS 4.3.19. There are still lots of unpatched firewalls out there, however.)

What does the attack look like?

Running an attack against a VPN device requires three ingredients. The first is the key K, which can be recovered from the FortiOS firmware using a bit of elbow grease. Shaanan Cohney (the aforementioned graduate student) was able to pull it out with a bit of effort.

Next, the attacker must have access to some VPN or TLS traffic. It’s important to note that this is not an active attack. All you really need is a network position that’s capable of monitoring full two-sided TLS or IPSec VPN connections.

Specifically, the attacker needs a full AES block (16 bytes) worth of output from the ANSI generator, plus part of a second block to check success against. Fortunately both TLS and IPSec (IKE) include nonces of sufficient length to obtain this output, and both are drawn from the ANSI generator, which lives in the FortiOS kernel. The attacker also needs the Diffie-Hellman ephemeral public keys, which are part of the protocol transcript.

Finally, you need to know the timestamp Ti that was used to operate the generator. In FortiOS, these timestamps have a 1-microsecond resolution, so guessing them is actually a bit of a challenge. Fortunately, TLS and other protocols include the time-in-seconds as one of the outputs of the TLS protocol, so the actually guessing space is typically only about 2^20 at most. Still, this guessing proves to be one of the most costly elements of the attack.

Given all of the ingredients above, the attacker now decrypts the output block taken from the protocol nonce using K, guesses each possible Ti value, and then winds forward or backwards until she finds the random bits that were used to generate that party’s Diffie-Hellman secret key. Fortunately, the key and nonce are generated one after the other, so this is not quite as painful as it sounds. But it is fairly time consuming. Fortunately, computers are fast, so this is not a dealbreaker.

With the secret key in hand, it’s possible to fully decrypt the VPN connection, read all traffic, and modify the data as needed.

Does the attack really work?

Since we’re not the NSA, it’s awfully hard for us to actually apply this attack to real Fortinet VPN connections in the wild. Not to mention that it would be somewhat unethical.

However, there’s nothing really unethical about scanning for FortiOS devices that are online and willing to accept incoming traffic from the Internet. To validate the attack, the team conducted a large-scale scan of the entire IPv4 address space. Each time we found a device that appeared to present as a FortiOS 4.x VPN, we initiated a connection with it and tested to see if we could break our own connection.

It turns out that there are a lot of FortiOS 4.x devices in the wild. Unfortunately, only a small number of them accept normal IPSec connections from strangers. Fortunately, however, a lot of them do accept TLS connections. Both protocol implementations use the same ANSI generator for their random numbers.

This scan allowed us to validate that — as of  October 2017 — the vulnerability was present and exploitable on more than 25,000 Fortinet devices across the Internet. And this count is likely conservative, since these were simply the devices that bothered to answer us when we scanned. A more sophisticated adversary like a nation-state would have access to existing VPN connections in flight.

In short, if you’re using a legacy Fortinet VPN you should probably patch.

So what does it all mean?

There are really three lessons to be learned from a bug like this one.

The first is that people make mistakes. We should probably design our crypto and certification processes to anticipate that, and make it much harder for these mistakes to become catastrophic decryption vulnerabilities like the one in FortiOS 4.x. Enough said.

The second is that government crypto certifications are largely worthless. I realize that seems like a big conclusion to draw from a single vulnerability. But this isn’t just a single vendor — it’s potentially several vendors that all fell prey to the same well-known 20-year old vulnerability. When a vulnerability is old enough to vote, your testing labs should be finding it. If they’re not finding things like this, what value are they adding?

Finally, there’s a lesson here about government standards. ANSI X9.31 (and its cousin X9.17) is over twenty years old. It’s (fortunately) been deprecated as of 2016, but a huge number of products still use it. This algorithm should have disappeared ten years earlier — and yet here we are. It’s almost certain that this small Fortinet vulnerability is just the tip of the iceberg. Following on revelations of a possible deliberate backdoor in the Dual EC generator, none of this stuff looks good. It’s time to give serious thought to how we make cryptographic devices resilient — even against the people who are supposed to be helping us secure them.

But that’s a topic for a much longer post.


by Matthew Green at October 23, 2017 06:28 PM

The FreeBSD Diary

' . htmlentities($myrow["name"]) . '

' . htmlentities($myrow["description"]) . '

October 23, 2017 02:18 AM

October 21, 2017


How to Minimize Leaking

I am hopeful that President Trump will not block release of the remaining classified documents addressing the 1963 assassination of President John F. Kennedy. I grew up a Roman Catholic in Massachusetts, so President Kennedy always fascinated me.

The 1991 Oliver Stone movie JFK fueled several years of hobbyist research into the assassination. (It's unfortunate the movie was so loaded with fictional content!) On the 30th anniversary of JFK's death in 1993, I led a moment of silence from the balcony of the Air Force Academy chow hall during noon meal. While stationed at Goodfellow AFB in Texas, Mrs B and I visited Dealey Plaza in Dallas and the Sixth Floor Museum.

Many years later, thanks to a 1992 law partially inspired by the Stone movie, the government has a chance to release the last classified assassination records. As a historian and former member of the intelligence community, I hope all of the documents become public. This would be a small but significant step towards minimizing the culture of information leaking in Washington, DC. If prospective leakers were part of a system that was known for releasing classified information prudently, regularly, and efficiently, it would decrease the leakers' motivation to evade the formal declassification process.

Many smart people have recommended improvements to the classification system. Check out this 2012 report for details.

by Richard Bejtlich ( at October 21, 2017 07:43 PM

Umatrix makes the web usable again

As happens with all media, once corporations join in because there is money to be made, things quickly devolve into a flaming heap of shit. The internet is no exception to this rule. With the coming of Javascript and DHTML in the late 90's, ads soon started appearing on the web. Not long after, pop-ups – either with or without ads – became a common sighting. This set off a back-and-forth war of browsers trying to block annoyances such as pop-ups and advertisers coming up with innovative ways to annoy the crap out of everybody. What started with still images of ads, much like you'd have on traditional paper, soon changed into a moving, blinking, automatically playing audio and video crapfest.

Because the basics underlying the web are insecure-by-design, it is possible for browsers to make requests to any website in the world from any website you decide to visit. This of course was quickly picked up by content publishers, the media and advertisers alike, to track your every move on the internet in order to shove more lukewarm shit down the throats of the average consumer.

The rise and fall of ad blockers

The answer to this came in the form of Ad blockers: separate programs or extensions you install in your browser. They have a predefined list of ad domains that they'll block from loading. Naturally this upset the media and advertisers, some of whom equate it with stealing. One has to wonder if ignoring ads in the newspaper is also considered stealing. At any rate, it's hard to be sympathetic with the poor thirsty vampires in the advertising industry. They've shown again and again that they can't self-regulate their own actions.  They'll go through great lengths to annoying you into buying their crap and violating your privacy in the most horrible ways, all for a few extra eyeballs *.

Unfortunately, much like how the zombies refuse to stay dead once slain, so do the annoying features on the web. For the past few years the media, content producers, advertisers and basically everybody else have decided to redouble their efforts in the Annoyance Wars with such things as Javascript / HTML5 pop-ups begging to sign up to their newsletters, automatically playing HTML5 videos, auto-loading of (or worse: redirecting to) the next article, detection of adblockers, bitcoin mining Javascript software running in the background when you visit a site, ever increasing and deeper tracking, super cookies, tracking pixels, the use of CDNs, etc.

For example, CNN's site loads resources from a staggering 61 domains. That's 61 places that can track you. 30 of those are known to track you and include Facebook, Google, a variety of specialized trackers, ad agencies, etc. It tries to run over 40 scripts and tries to place 8 cookies. And this is only from allowing scripts! I did not allow cross-site background requests or any of the blocked domains. I'm sure if I unblocked those, there would be much, much more.

To top it all off, advertisers have decided to go to the root of the problem, and have simply bought most ad blockers. These were then modified to let through certain "approved" ads. The sales pitch they give us is that they'll only allow nonintrusive ads. Kinda like a fifty-times convicted felon telling the parole board that he'll truly be good this time. In some cases they've even built tracking software directly in the ad blockers themselves. In other cases they're basically extorting money from websites in exchange for letting their ads through.

Bringing out the big guns

So… our ad blockers are being taken over by the enemy. Our browsers are highly insecure by default. Every site can send and retrieve data from any place on the web. Browsers can use our cameras, microphones and GPUs. The most recent addition is the ability to show notifications on our desktops. A feature that, surprise surprise, was quickly picked up by sites to shove more of their crap in our faces.  Things are only going to get worse as browsers get more and more access to our PCs. The media has shown that they don't give a rats ass about your privacy or rights as long as there's a few cents to make. The world's most-used browser is built by a company that lives off advertising. Have we lost the war? Do we just resign ourselves to the fact that we'll be tracked, spied upon, lied to, taken advantage of and spoon-fed advertisements like obedient little consumers every time we open a webpage?


Enter umatrix

Umatrix is like a firewall for your browser. It stops web pages from doing anything. They can't set cookies, they can't load CSS or images. They can't run scripts, use iframes or send data to themselves or to other sites. Umatrix blocks everything. Umatrix is what browsers should be doing by default. Ad blockers are just not enough. They only block ads and then only the ones they know about. To get any decent kind of protection of your privacy, you always needed at least an Adblocker, a privacy blocker such as Privacy Badger and a script blocker such as Noscript. Umatrix replaces all of those, and does a better job at it too.

Umatrix gives you full insight into what a website is trying to do. It gives you complete control over what a website is and isn't allowed to do. It has a bit of a learning curve, which is why I wrote a tutorial for it.

When you just start out using umatrix, you'll find that many sites don't work properly. On the other hand, many sites work a lot better. If you take a little time to unblock things so that your favorite sites work again (and save those changes), you'll notice within a few hours that it's not so bad. Umatrix is still for technical users and even those might find it too much of a chore. I find it worth the small effort to unblock things if that means my privacy stays intact. And more importantly, I'm never annoyed anymore by ads or pop-ups, plus I'm much less likely to accidentally run some malicious javascript.

This is what a CNN article looks like with Umatrix enabled and only allows first-party (* CSS and images, which I have set as the default:

Look at that beauty. It's clean. It loads super fast. It cannnot track me. It's free of ads, pop-ups, auto playing videos, scripts and cookies. Remember, this didn't take any manual unblocking. This is just what it looks like out of the box with umatrix.

Umatrix makes the web usable again.

Get it, install it, love it

Get it for FirefoxChrome or Opera. Umatrix is Open Source, so it cannot be bought out by ad agencies.

Read my tutorial to get started with umatrix, as the learning curve is fairly steep.

Thanks for reading and safe browsing!



*) Someone will probably bring up the whole "but content creators should / need / deserve money too!" argument again. I'm not interested in discussions about that. They've had the chance to behave, and they've misbehaved time and time again. At some point, you don't trust the pathological liars anymore. No content creator switched to a more decent ad provider that doesn't fuck its customers over. The blame is on them, not on the people blocking their tracking, spying, attention-grabbing shit. Don't want me looking at your content for free? Put up a paywall. People won't come to your site anymore if you did that? Then maybe your content wasn't worth money in the first place.

by admin at October 21, 2017 01:46 PM

Colin Percival

FreeBSD/EC2: Community vs. Marketplace AMIs

FreeBSD has been available in Amazon EC2 since FreeBSD 9.0 (January 2012), and from FreeBSD 10.2 (August 2015) AMIs have been built by the FreeBSD release enginering team as part of the regular release process. Release announcements go out with a block of text about where to find FreeBSD in EC2, for example (taken from the FreeBSD 11.1 release announcement):
   FreeBSD 11.1-RELEASE amd64 is also available on these cloud hosting

     * Amazon(R) EC2(TM):
       AMIs are available in the following regions:

         ap-south-1 region: ami-8a760ee5
         eu-west-2 region: ami-f2425396
         eu-west-1 region: ami-5302ec2a
         ap-northeast-2 region: ami-f575ab9b
         ap-northeast-1 region: ami-0a50b66c
         sa-east-1 region: ami-9ad8acf6
         ca-central-1 region: ami-622e9106
         ap-southeast-1 region: ami-6d75e50e
         ap-southeast-2 region: ami-bda2bede
         eu-central-1 region: ami-7588251a
         us-east-1 region: ami-70504266
         us-east-2 region: ami-0d725268
         us-west-1 region: ami-8b0128eb
         us-west-2 region: ami-dda7bea4

       AMIs will also available in the Amazon(R) Marketplace once they have
       completed third-party specific validation at:
This leads to a question I am frequently asked: Which way should FreeBSD users launch their instances? The answer, as usual, is "it depends". Here are some of the advantages of each option, to help you decide which to use.

October 21, 2017 12:45 AM

October 20, 2017

Sean's IT Blog

The Virtual Horizon Podcast Episode 1 – A Conversation with Matt Heldstab

VMUG Leader Matt Heldstab joins me to talk about the VMUG EUC Explore event and community involvement.


Podcast music is a derivative of
Boogie Woogie Bed by Jason Shaw (
Licensed under Creative Commons: By Attribution 3.0 License


by seanpmassey at October 20, 2017 01:15 PM

October 18, 2017

Sean's IT Blog

Configuring a Headless CentOS Virtual Machine for NVIDIA GRID vGPU #blogtober

When IT administrators think of GPUs, the first thing that comes to mind for many is gaming.  But GPUs also have business applications.  They’re mainly found in high end workstations to support graphics intensive applications like 3D CAD and medical imaging.

But GPUs will have other uses in the enterprise.  Many of the emerging technologies, such as artificial intelligence and deep learning, utilize GPUs to perform compute operations.  These will start finding their way into the data center, either as part of line-of-business applications or as part of IT operations tools.  This could also allow the business to utilize GRID environments after hours for other forms of data processing.

This guide will show you how to build headless virtual machines that can take advantage of NVIDIA GRID vGPU for GPU compute and CUDA.  In order to do this, you will need to have a Pascal Series NVIDIA Tesla card such as the P4, P40, or P100 and the GRID 5.0 drivers.  The GRID components will also need to be configured in your hypervisor, and you will need to have the GRID drivers for Linux.

I’ll be using CentOS 7.x for this guide.  My base CentOS configuration is a minimal install with no graphical shell and a few additional packages like Nano and Open VM Tools.  I use Bob Planker’s guide for preparing my VM as a template.

The steps for setting up a headless CentOS VM with GRID are:

  1. Deploy your CentOS VM.  This can be from an existing template or installed from scratch.  This VM should not have a graphical shell installed, or it should be in a run mode that does not execute the GUI.
  2. Attach a GRID profile to the virtual machine by adding a shared PCI device in vCenter.  The selected profile will need to be one of the Virtual Workstation profiles, and these all end with a Q.
  3. GRID requires a 100% memory reservation.  When you add an NVIDIA GRID shared PCI device, there will be an associated prompt to reserve all system memory.
  4. Update the VM to ensure all applications and components are the latest version using the following command:
    yum update -y
  5. In order to build the GRID driver for Linux, you will need to install a few additional packages.  Install these packages with the following command:
    yum install -y epel-release dkms libstdc++.i686 gcc kernel-devel 
  6. Copy the Linux GRID drivers to your VM using a tool like WinSCP.  I generally place the files in /tmp.
  7. Make the driver package executable with the following command:
    chmod +X
  8. Execute the driver package.  When we execute this, we will also be adding the –dkms flag to support Dynamic Kernel Module Support.  This will enable the system to automatically recompile the driver whenever a kernel update is installed.  The commands to run the the driver install are:
    bash ./ –dkms
  9. When prompted, select yes to register the kernel module sources with DKMS by selecting Yes and pressing Enter.
  10. You may receive an error about the installer not being able to locate the X Server path.  Click OK.  It is safe to ignore this error.
  11. Install the 32-bit Compatibility Libraries by selecting Yes and pressing Enter.
  12. At this point, the installer will start to build the DKMS module and install the driver.  
  13. After the install completes, you will be prompted to use the nvidia-xconfig utility to update your X Server configuration.  X Server should not be installed because this is a headless machine, so select No and press Enter.
  14. The install is complete.  Press Enter to exit the installer.
  15. To validate that the NVIDIA drivers are installed and running properly, run nvidia-smi to get the status of the video card.  
  16. Next, we’ll need to configure GRID licensing.  We’ll need to create the GRID licensing file from a template supplied by NVIDIA with the following command:
    cp  /etc/nvidia/gridd.conf.template  /etc/nvidia/gridd.conf
  17. Edit the GRID licensing file using the text editor of your choice.  I prefer Nano, so the command I would use is:
    nano  /etc/nvidia/gridd.conf
  18. Fill in the ServerAddress and BackupServerAddress fields with the fully-qualified domain name or IP addresses of your licensing servers.
  19. Set the FeatureType to 2 to configure the system to retrieve a Virtual Workstation license.  The Virtual Workstation license is required to support the CUDA features for GPU Compute.
  20. Save the license file.
  21. Restart the GRID Service with the following command:
    service nvidia-gridd restart
  22. Validate that the machine retrieved a license with the following command:
    grep gridd /var/log/messages
  23. Download the NVIDIA CUDA Toolkit.
  24. Make the toolkit installer executable.
    chmod +x
  25. Execute the CUDA Toolkit installer.
  26. Accept the EULA.
  27. You will be prompted to download the CUDA Driver.  Press N to decline the new driver. This driver does not match the NVIDIA GRID driver version, and it will break the NVIDIA setup.  The GRID driver in the VM has to match the GRID software that is installed in the hypervisor.
  28. When prompted to install the CUDA 9.0 toolkit, press Y.
  29. Accept the Default Location for the CUDA toolkit.
  30. When prompted to create a symlink at /usr/local/cuda, press Y.
  31. When prompted to install the CUDA 9.0 samples, press Y.
  32. Accept the default location for the samples.
  33. Reboot the virtual machine.
  34. Log in and run nvidia-smi again.  Validate that you get the table output similar to step 15.  If you do not receive this, and you get an error, it means that you likely installed the driver that is included with the CUDA toolkit.  If that happens, you will need to start over.

At this point, you have a headless VM with the NVIDIA Drivers and CUDA Toolkit installed.  So what can you do with this?  Just about anything that requires CUDA.  You can experiment with deep learning frameworks like Tensorflow, build virtual render nodes for tools like Blender, or even use Matlab for GPU compute.

by seanpmassey at October 18, 2017 04:04 PM

McRee added to ISSA's Honor Roll for Lifetime Achievement

HolisticInfoSec's Russ McRee was pleased to be added to ISSA International's Honor Roll this month, a lifetime achievement award recognizing an individual's sustained contributions to the information security community, the advancement of the association and enhancement of the professionalism of the membership.
According to the press release:
"Russ McRee has a strong history in the information security as a teacher, practitioner and writer. He is responsible for 107 technical papers published in the ISSA Journal under his Toolsmith byline in 2006-2015. These articles represent a body of knowledge for the hands-on practitioner that is second to none. These titles span an extremely wide range of deep network security topics. Russ has been an invited speaker at the key international computer security venues including DEFCON, Derby Con, BlueHat, Black Hat, SANSFIRE, RSA, and ISSA International."
Russ greatly appreciates this honor and would like to extend congratulations to the ten other ISSA 2017 award winners. Sincere gratitude to Briana and Erin McRee, Irvalene Moni, Eric Griswold, Steve Lynch, and Thom Barrie for their extensive support over these many years.

by Russ McRee ( at October 18, 2017 04:35 AM

toolsmith #128 - DFIR Redefined: Deeper Functionality for Investigators with R - Part 1

“To competently perform rectifying security service, two critical incident response elements are necessary: information and organization.” ~ Robert E. Davis

I've been presenting DFIR Redefined: Deeper Functionality for Investigators with R across the country at various conference venues and thought it would helpful to provide details for readers.
The basic premise?
Incident responders and investigators need all the help they can get.
Let me lay just a few statistics on you, from's The Challenges of Incident Response, Nov 2016. Per their respondents in a survey of security professionals:
  • 38% reported an increase in the number of hours devoted to incident response
  • 42% reported an increase in the volume of incident response data collected
  • 39% indicated an increase in the volume of security alerts
In short, according to Nathan Burke, “It’s just not mathematically possible for companies to hire a large enough staff to investigate tens of thousands of alerts per month, nor would it make sense.”
The 2017 SANS Incident Response Survey, compiled by Matt Bromiley in June, reminds us that “2016 brought unprecedented events that impacted the cyber security industry, including a myriad of events that raised issues with multiple nation-state attackers, a tumultuous election and numerous government investigations.” Further, "seemingly continuous leaks and data dumps brought new concerns about malware, privacy and government overreach to the surface.”
Finally, the survey shows that IR teams are:
  • Detecting the attackers faster than before, with a drastic improvement in dwell time
  • Containing incidents more rapidly
  • Relying more on in-house detection and remediation mechanisms
To that end, what concepts and methods further enable handlers and investigators as they continue to strive for faster detection and containment? Data science and visualization sure can’t hurt. How can we be more creative to achieve “deeper functionality”? I propose a two-part series on Deeper Functionality for Investigators with R with the following DFIR Redefined scenarios:
  • Have you been pwned?
  • Visualization for malicious Windows Event Id sequences
  • How do your potential attackers feel, or can you identify an attacker via sentiment analysis?
  • Fast Frugal Trees (decision trees) for prioritizing criticality
R is “100% focused and built for statistical data analysis and visualization” and “makes it remarkably simple to run extensive statistical analysis on your data and then generate informative and appealing visualizations with just a few lines of code.”

With R you can interface with data via file ingestion, database connection, APIs and benefit from a wide range of packages and strong community investment.
From the Win-Vector Blog, per John Mount “not all R users consider themselves to be expert programmers (many are happy calling themselves analysts). R is often used in collaborative projects where there are varying levels of programming expertise.”
I propose that this represents the vast majority of us, we're not expert programmers, data scientists, or statisticians. More likely, we're security analysts re-using code for our own purposes, be it red team or blue team. With a very few lines of R investigators might be more quickly able to reach conclusions.
All the code described in the post can be found on my GitHub.

Have you been pwned?

This scenario I covered in an earlier post, I'll refer you to Toolsmith Release Advisory: Steph Locke's HIBPwned R package.

Visualization for malicious Windows Event Id sequences

Windows Events by Event ID present excellent sequenced visualization opportunities. A hypothetical scenario for this visualization might include multiple failed logon attempts (4625) followed by a successful logon (4624), then various malicious sequences. A fantastic reference paper built on these principle is Intrusion Detection Using Indicators of Compromise Based on Best Practices and Windows Event Logs. An additional opportunity for such sequence visualization includes Windows processes by parent/children. One R library particularly well suited to is TraMineR: Trajectory Miner for R. This package is for mining, describing and visualizing sequences of states or events, and more generally discrete sequence data. It's primary aim is the analysis of biographical longitudinal data in the social sciences, such as data describing careers or family trajectories, and a BUNCH of other categorical sequence data. Somehow though, the project page somehow fails to mention malicious Windows Event ID sequences. :-) Consider Figures 1 and 2 as retrieved from above mentioned paper. Figure 1 are text sequence descriptions, followed by their related Windows Event IDs in Figure 2.

Figure 1
Figure 2
Taking related log data, parsing and counting it for visualization with R would look something like Figure 3.

Figure 3
How much R code does it take to visualize this data with a beautiful, interactive sunburst visualization? Three lines, not counting white space and comments, as seen in the video below.

A screen capture of the resulting sunburst also follows as Figure 4.

Figure 4

How do your potential attackers feel, or can you identify an attacker via sentiment analysis?

Do certain adversaries or adversarial communities use social media? Yes
As such, can social media serve as an early warning system, if not an actual sensor? Yes
Are certain adversaries, at times, so unaware of OpSec on social media that you can actually locate them or correlate against other geo data? Yes
Some excellent R code to assess Twitter data with includes Jeff Gentry's twitteR and rtweet to interface with the Twitter API.
  • twitteR: provides access to the Twitter API. Most functionality of the API is supported, with a bias towards API calls that are more useful in data analysis as opposed to daily interaction.
  • Rtweet: R client for interacting with Twitter’s REST and stream API’s.
The code and concepts here are drawn directly from Michael Levy, PhD UC Davis: Playing With Twitter.
Here's the scenario: DDoS attacks from hacktivist or chaos groups.
Attacker groups often use associated hashtags and handles and the minions that want to be "part of" often retweet and use the hashtag(s). Individual attackers either freely give themselves away, or often become easily identifiable or associated, via Twitter. As such, here's a walk-through of analysis techniques that may help identify or better understand the motives of certain adversaries and adversary groups. I don't use actual adversary handles here, for obvious reasons. I instead used a DDoS news cycle and journalist/bloggers handles as exemplars. For this example I followed the trail of the WireX botnet, comprised mainly of Android mobile devices utilized to launch a high-impact DDoS extortion campaign against multiple organizations in the travel and hospitality sector in August 2017. I started with three related hashtags: 
  1. #DDOS 
  2. #Android 
  3. #WireX
We start with all related Tweets by day and time of day. The code is succinct and efficient, as noted in Figure 5.

Figure 5
The result is a pair of graphs color coded by tweets and retweets per Figure 6.

Figure 6

This gives you an immediate feels for spikes in interest by day as well as time of day, particularly with attention to retweets.
Want to see what platforms potential adversaries might be tweeting from? No problem, code in Figure 7.
Figure 7

The result in the scenario ironically indicates that the majority of related tweets using our hashtags of interest are coming from Androids per Figure 8. :-)

Figure 8
Now to the analysis of emotional valence, or the "the intrinsic attractiveness (positive valence) or averseness (negative valence) of an event, object, or situation."
orig$text[which.max(orig$emotionalValence)] tells us that the most positive tweet is "A bunch of Internet tech companies had to work together to clean up #WireX #Android #DDoS #botnet."
orig$text[which.min(orig$emotionalValence)] tells us that "Dangerous #WireX #Android #DDoS #Botnet Killed by #SecurityGiants" is the most negative tweet.
Interesting right? Almost exactly the same message, but very different valence.
How do we measure emotional valence changes over the day? Four lines later...
filter(orig, mday(created) == 29) %>%
  ggplot(aes(created, emotionalValence)) +
  geom_point() + 
  geom_smooth(span = .5)
...and we have Figure 9, which tell us that most tweets about WireX were emotionally neutral on 29 AUG 2017, around 0800 we saw one positive tweet, a more negative tweets overall in the morning.

Figure 9
Another line of questioning to consider: which tweets are more often retweeted, positive or negative? As you can imagine with information security focused topics, negativity wins the day.
Three lines of R...
ggplot(orig, aes(x = emotionalValence, y = retweetCount)) +
  geom_point(position = 'jitter') +
...and we learn just how popular negative tweets are in Figure 10.

Figure 10
There are cluster of emotionally neutral retweets, two positive retweets, and a load of negative retweets. This type of analysis can quickly lead to a good feel for the overall sentiment of an attacker collective, particularly one with less opsec and more desire for attention via social media.
In Part 2 of DFIR Redefined: Deeper Functionality for Investigators with R we'll explore this scenario further via sentiment analysis and Twitter data, as well as Fast Frugal Trees (decision trees) for prioritizing criticality.
Let me know if you have any questions on the first part of this series via @holisticinfosec or russ at holisticinfosec dot org.
Cheers...until next time. 

by Russ McRee ( at October 18, 2017 04:14 AM


More China Press Coverage

Press Coverage

There have been more articles written based on the interviews with Paul Yang from BaishanCloud, Tim Hudson, and Steve Marquess from the OpenSSL team.

These join the articles noted in the previous blog entry.

October 18, 2017 01:00 AM

October 16, 2017

Cryptography Engineering

Falling through the KRACKs

The big news in crypto today is the KRACK attack on WPA2 protected WiFi networks. Discovered by Mathy Vanhoef and Frank Piessens at KU Leuven, KRACK (Key Reinstallation Attack) leverages a vulnerability in the 802.11i four-way handshake in order to facilitate decryption and forgery attacks on encrypted WiFi traffic.

The paper is here. It’s pretty easy to read, and you should.

I don’t want to spend much time talking about KRACK itself, because the vulnerability is pretty straightforward. Instead, I want to talk about why this vulnerability continues to exist so many years after WPA was standardized. And separately, to answer a question: how did this attack slip through, despite the fact that the 802.11i handshake was formally proven secure?

A quick TL;DR on KRACK

For a detailed description of the attack, see the KRACK website or the paper itself. Here I’ll just give a brief, high level description.

The 802.11i protocol (also known as WPA2) includes two separate mechanisms to ensure the confidentiality and integrity of your data. The first is a record layer that encrypts WiFi frames, to ensure that they can’t be read or tampered with. This encryption is (generally) implemented using AES in CCM mode, although there are newer implementations that use GCM mode, and older ones that use RC4-TKIP (we’ll skip these for the moment.)

The key thing to know is that AES-CCM (and GCM, and TKIP) is a stream cipher, which means it’s vulnerable to attacks that re-use the same key and “nonce”, also known as an initialization vector. 802.11i deals with this by constructing the initialization vector using a “packet number” counter, which initializes to zero after you start a session, and always increments (up to 2^48, at which point rekeying must occur). This should prevent any nonce re-use, provided that the packet number counter can never be reset.

The second mechanism you should know about is the “four way handshake” between the AP and a client (supplicant) that’s responsible for deriving the key to be used for encryption. The particular message KRACK cares about is message #3, which causes the new key to be “installed” (and used) by the client.

I’m a four-way handshake. Client is on the left, AP is in the right. (courtesy Wikipedia, used under CC).

The key vulnerability in KRACK (no pun intended) is that the acknowledgement to message #3 can be blocked by adversarial nasty people.* When this happens, the AP re-transmits this message, which causes (the same) key to be reinstalled into the client (note: see update below*). This doesn’t seem so bad. But as a side effect of installing the key, the packet number counters all get reset to zero. (And on some implementations like Android 6, the key gets set to zero — but that’s another discussion.)

The implication is that by forcing the AP to replay this message, an adversary can cause a connection to reset nonces and thus cause keystream re-use in the stream cipher. With a little cleverness, this can lead to full decryption of traffic streams. And that can lead to TCP hijacking attacks. (There are also direct traffic forgery attacks on GCM and TKIP, but this as far as we go for now.)

How did this get missed for so long?

If you’re looking for someone to blame, a good place to start is the IEEE. To be clear, I’m not referring to the (talented) engineers who designed 802.11i — they did a pretty good job under the circumstances. Instead, blame IEEE as an institution.

One of the problems with IEEE is that the standards are highly complex and get made via a closed-door process of private meetings. More importantly, even after the fact, they’re hard for ordinary security researchers to access. Go ahead and google for the IETF TLS or IPSec specifications — you’ll find detailed protocol documentation at the top of your Google results. Now go try to Google for the 802.11i standards. I wish you luck.

The IEEE has been making a few small steps to ease this problem, but they’re hyper-timid incrementalist bullshit. There’s an IEEE program called GET that allows researchers to access certain standards (including 802.11) for free, but only after they’ve been public for six months — coincidentally, about the same time it takes for vendors to bake them irrevocably into their hardware and software.

This whole process is dumb and — in this specific case — probably just cost industry tens of millions of dollars. It should stop.

The second problem is that the IEEE standards are poorly specified. As the KRACK paper points out, there is no formal description of the 802.11i handshake state machine. This means that implementers have to implement their code using scraps of pseudocode scattered around the standards document. It happens that this pseudocode leads to the broken implementation that enables KRACK. So that’s bad too.

And of course, the final problem is implementers. One of the truly terrible things about KRACK is that implementers of the WPA supplicant (particularly on Linux) managed to somehow make Lemon Pledge out of lemons. On Android 6 in particular, replaying message #3 actually sets an all-zero key. There’s an internal logic behind why this happens, but Oy Vey. Someone actually needs to look at this stuff.

What about the security proof?

The fascinating thing about the 802.11i handshake is that despite all of the roadblocks IEEE has thrown in people’s way, it (the handshake, at least) has been formally analyzed. At least, for some definition of the term.

(This isn’t me throwing shade — it’s a factual statement. In formal analysis, definitions really, really matter!)

A paper by He, Sundararajan, Datta, Derek and Mitchell (from 2005!) looked at the 802.11i handshake and tried to determine its security properties. What they determined is that yes, indeed, it did produce a secret and strong key, even when an attacker could tamper with and replay messages (under various assumptions). This is good, important work. The proof is hard to understand, but this is par for the course. It seems to be correct.

Representation of the 4-way handshake from the paper by He et al. Yes, I know you’re like “what?“. But that’s why people who do formal verification of protocols don’t have many friends.

Even better, there are other security proofs showing that — provided the nonces are never repeated — encryption modes like CCM and GCM are highly secure. This means that given a secure key, it should be possible to encrypt safely.

So what went wrong?

The critical problem is that while people looked closely at the two components — handshake and encryption protocol — in isolation, apparently nobody looked closely at the two components as they were connected together. I’m pretty sure there’s an entire geek meme about this.

Two unit tests, 0 integration tests, thanks Twitter.

Of course, the reason nobody looked closely at this stuff is that doing so is just plain hard. Protocols have an exponential number of possible cases to analyze, and we’re just about at the limit of the complexity of protocols that human beings can truly reason about, or that peer-reviewers can verify. The more pieces you add to the mix, the worse this problem gets.

In the end we all know that the answer is for humans to stop doing this work. We need machine-assisted verification of protocols, preferably tied to the actual source code that implements them. This would ensure that the protocol actually does what it says, and that implementers don’t further screw it up, thus invalidating the security proof.

This needs to be done urgently, but we’re so early in the process of figuring out how to do it that it’s not clear what it will take to make this stuff go live. All in all, this is an area that could use a lot more work. I hope I live to see it.


* Update: An early version of this post suggested that the attacker would replay the third message. This can indeed happen, and it does happen in some of the more sophisticated attacks. But primarily, the paper describes forcing the AP to resend it by blocking the acknowledgement from being received at the AP. Thanks to Nikita Borisov and Kyle Birkeland for the fix!

by Matthew Green at October 16, 2017 01:27 PM

October 14, 2017

Ansible-cmdb v1.23: Generate a host overview of Ansible facts.

I've just released ansible-cmdb v1.23. Ansible-cmdb takes the output of Ansible's fact gathering and converts it into a static HTML overview page containing system configuration information. It supports multiple templates (fancy html, txt, markdown, json and sql) and extending information gathered by Ansible with custom data.

This release includes the following changes:

  • group_vars are now parsed.
  • Sub directories in host_vars are now parsed.
  • Addition of a -q/--quiet switch to suppress warnings.
  • Minor bugfixes and additions.

As always, packages are available for Debian, Ubuntu, Redhat, Centos and other systems. Get the new release from the Github releases page.

by admin at October 14, 2017 09:33 AM

October 10, 2017

Sean's IT Blog

Nutanix Xtract for VMs #blogtober

One of the defining features of the Nutanix platform is simplicity.  Innovations like the Prism interface for infrastructure management and One-Click Upgrades for both the Nutanix software-defined storage platform and supported hypervisors have lowered the management burden of on-premises infrastructure.

Nutanix is now looking to bring that same level of simplicity to migrating virtual machines to a new hypervisor.  Nutanix has released a new tool today called Xtract for VM.  This tool, which is free to all Nutanix customers, brings the same one-click simplicity that Nutanix is known for to migrating workloads from ESXi to AHV.

So how does Xtract for VM differentiate from other migration tools?  First, it is an agentless migration tool.  Xtract will communicate with vCenter to get a list of VMs that are in the ESXi infrastructure, and it will build a migration plan and synchronize the VM data from ESXi to AHV.

During data synchronization and migration, Xtract will insert the AHV device drivers into the virtual machine.  It will also capture and preserve the network configuration, so the VM will not lose connectivity or require administrator intervention after the migration is complete.

By injecting the AHV drivers and preserving the network configuration during the data synchronization and cutover, Xtract is able to perform cross-hypervisor migrations with minimal downtime.  And since the original VM is not touched during the migration, rollback is as easy as shutting down the AHV VM and powering the ESXi VM back on, which significantly reduces the risk of cross-hypervisor migrations.


The datacenter is clearly changing, and we now live in a multi-hypervisor world.  While many customers will still run VMware for their on-premises environments, there are many that are looking to reduce their spend on hypervisor products.  Xtract for VMs provides a tool to help reduce that cost while providing the simplicity that Nutanix is known for.

While Xtract is currently version 1.0, I can see this technology be a pivotal for helping customers move workloads between on-premises and cloud infrastructures.

To learn more about this new tool, you can check out the Xtract for VMs page on Nutanix’s webpage.

by seanpmassey at October 10, 2017 04:36 PM

October 09, 2017

Sarah Allen

getting started with docker

Learning about docker this weekend… it always is hard to find resources for people who understand the concepts of VMs and containers and need to dive into something just a little bit complicated. I had been through lots of intro tutorials, when docker first arrived on the scene, and was seeking to set up a hosted dev instance for existing open source project which already had a docker set up.

Here’s an outline of the key concepts:

  • docker-machine: commands to create and manage VMs, whether locally or on a remote server
  • docker: commands to talk to your active VM that is already set up with docker-machine
  • docker-compose: a way to create and manage multiple containers on a docker-machine

As happens, I see parallels between human spoken language and new technical terms, which makes sense since these are things made by and for humans. The folks who made Docker invented a kind of language for us to talk to their software.

I felt like I was learning to read in a new language, like pig-latin, where words have a prefix of docker, like some kind of honorific

They use docker- to speak to VMs locally or remotely, and docker (without a dash) is an intimate form of communication with your active VM

Writing notes here, so I remember when I pick this up again. If there are Docker experts reading this, I’d be interested to know if I got this right and if there are other patterns or naming conventions that might help fast-track my learning of this new dialect for deploying apps in this land of containers and virtual machines.

Also, if a kind and generous soul wants to help an open source project, I’ve written up my work-in-progress steps for setting up OpenOpps-platform dev instance and would appreciate any advice, and of course, would welcome a pull request.

by sarah at October 09, 2017 02:17 PM


The Choria Emulator

In my previous posts I discussed what goes into load testing a Choria network, what connections are made, subscriptions are made etc.

From this it’s obvious the things we should be able to emulate are:

  • Connections to NATS
  • Subscriptions – which implies number of agents and sub collectives
  • Message payload sizes

To make it realistically affordable to emulate many more machines that I have I made an emulator that can start numbers of Choria daemons on a single node.

I’ve been slowly rewriting MCollective daemon side in Go which means I already had all the networking and connectors available there, so a daemon was written:

usage: choria-emulator --instances=INSTANCES [<flags>]
Emulator for Choria Networks
      --help                 Show context-sensitive help (also try --help-long and --help-man).
      --version              Show application version.
      --name=""              Instance name prefix
  -i, --instances=INSTANCES  Number of instances to start
  -a, --agents=1             Number of emulated agents to start
      --collectives=1        Number of emulated subcollectives to create
  -c, --config=CONFIG        Choria configuration file
      --tls                  Enable TLS on the NATS connections
      --verify               Enable TLS certificate verifications on the NATS connections
      --server=SERVER ...    NATS Server pool, specify multiple times (eg one:4222)
  -p, --http-port=8080       Port to listen for /debug/vars

You can see here it takes a number of instances, agents and collectives. The instances will all respond with ${name}-${instance} on any mco ping or RPC commands. It can be discovered using the normal mc discovery – though only supports agent and identity filters.

Every instance will be a Choria daemon with the exact same network connection and NATS subscriptions as real ones. Thus 50 000 emulated Choria will put the exact same load of work on your NATS brokers as would normal ones, performance wise even with high concurrency the emulator performs quite well – it’s many orders of magnitude faster than the ruby Choria client anyway so it’s real enough.

The agents they start are all copies of this one:

Choria Agent emulated by choria-emulator
      Author: R.I.Pienaar <>
     Version: 0.0.1
     License: Apache-2.0
     Timeout: 120
   Home Page:
   Requires MCollective 2.9.0 or newer
   generate action:
       Generates random data of a given size
              Description: Amount of text to generate
                   Prompt: Size
                     Type: integer
                 Optional: true
            Default Value: 20
              Description: Generated Message
               Display As: Message

You can this has a basic data generator action – you give it a desired size and it makes you a message that size. It will run as many of these as you wish all called like emulated0 etc.

It has an mcollective agent that go with it, the idea is you create a pool of machines all with your normal mcollective on it and this agent. Using that agent then you build up a different new mcollective network comprising the emulators, federation and NATS.

Here’s some example of commands – you’ll see these later again when we talk about scenarios:

We download the dependencies onto all our nodes:

$ mco playbook run setup-prereqs.yaml --emulator_url= --gnatsd_url= --choria_url=

We start NATS on our first node:

$ mco playbook run start-nats.yaml --monitor 8300 --port 4300 -I

We start the emulator with 1500 instances per node all pointing to our above NATS:

$ mco playbook run start-emulator.yaml --agents 10 --collectives 10 --instances 750 --monitor 8080 --servers

You’ll then setup a client config for the built network and can interact with it using normal mco stuff and the test suite I’ll show later. Simularly there are playbooks to stop all the various parts etc. The playbooks just interact with the mcollective agent so you could use mco rpc directly too.

I found I can easily run 700 to 1000 instances on basic VMs – needs like 1.5GB RAM – so it’s fairly light. Using 400 nodes I managed to build a 300 000 node Choria network and could easily interact with it etc.

Finally I made a ec2 environment where you can stand up a Puppet Master, Choria, the emulator and everything you need and do load tests on your own dime. I was able to do many runs with 50 000 emulated nodes on EC2 and the whole lot cost me less than $20.

The code for this emulator is very much a work in progress as is the Go code for the Choria protocol and networking but the emulator is here if you want to take a peek.

by R.I. Pienaar at October 09, 2017 07:37 AM

Joe Topjian

Building OpenStack Environments Part 3


Continuing on with the series of building disposable OpenStack environments for testing purposes, this part will cover how to install services which are not supported by PackStack.

While PackStack does an amazing job at easily and quickly creating OpenStack environments, it only has the ability to install a subset of services under the OpenStack umbrella. However, almost all OpenStack services are supported by RDO, the overarching package library for RedHat/CentOS.

For this part in the series, I will show how to install and configure Designate, the OpenStack DNS service, using the RDO packages.

Planning the Installation

PackStack spoils us by hiding all of the steps required to install an OpenStack service. Installing a service requires a good amount of planning, even if the service is only going to be used for testing rather than production.

To begin planning, first read over any documentation you can find about the service in question. For Designate, there is a good amount of documentation here.

The overview page shows that there are a lot of moving pieces to Designate. Whether or not you need to account for all of these is still in question since it's possible that the RDO packages provide some sort of base configuration.

The installation page gives some brief steps about how to install Designate. By reading the page, you can see that all of the services listed in the Overview do not require special configuration. This makes things more simple.

Keep in mind that if you were to deploy Designate for production use, you might have to tune all of these services to suit your environment. Determining how to tune these services is out of scope for this blog post. Usually it requires careful reading of the various Designate configuration files, looking for supplementary information on mailing lists, and often even reading the source code itself.

The installation page shows how to use BIND as the default DNS driver. However, I'm going to change things up here. Instead, I will show how to use PowerDNS. There are two reasons for this:

  1. I'm allergic to BIND.
  2. I had trouble piecing together everything required to run Designate with the new PowerDNS driver, so this will also serve as documentation to help others.

Adding the Installation Steps

Continuing with Part 1 and Part 2, you should have a directory called terraform-openstack-test on your workstation. The structure of the directory should look something like this:

$ pwd
$ tree .
├── files
│   └──
├── key
│   ├── id_rsa
│   └──
└── packer
    ├── files
    │   ├──
    │   ├── packstack-answers.txt
    │   └── rc.local
    └── openstack
        ├── build.json
        └── is used to install and configure PackStack and then strip any unique information from the installation. Packer then makes an image out of this installation. Finally, rc.local does some post-boot configuration.

To install and configure Designate, you will want to add additional pieces to both and rc.local.

Installing PowerDNS

First, install and configure PowerDNS. To do this, add the following to

  hostnamectl set-hostname localhost

  systemctl disable firewalld
  systemctl stop firewalld
  systemctl disable NetworkManager
  systemctl stop NetworkManager
  systemctl enable network
  systemctl start network

  yum install -y
  yum install -y centos-release-openstack-ocata
  yum-config-manager --enable openstack-ocata
  yum update -y
  yum install -y openstack-packstack
  packstack --answer-file /home/centos/files/packstack-answers.txt

  source /root/keystonerc_admin
  nova flavor-create m1.acctest 99 512 5 1 --ephemeral 10
  nova flavor-create m1.resize 98 512 6 1 --ephemeral 10
  _NETWORK_ID=$(openstack network show private -c id -f value)
  _SUBNET_ID=$(openstack subnet show private_subnet -c id -f value)
  _EXTGW_ID=$(openstack network show public -c id -f value)
  _IMAGE_ID=$(openstack image show cirros -c id -f value)

  echo "" >> /root/keystonerc_admin
  echo export OS_IMAGE_NAME="cirros" >> /root/keystonerc_admin
  echo export OS_IMAGE_ID="$_IMAGE_ID" >> /root/keystonerc_admin
  echo export OS_NETWORK_ID=$_NETWORK_ID >> /root/keystonerc_admin
  echo export OS_EXTGW_ID=$_EXTGW_ID >> /root/keystonerc_admin
  echo export OS_POOL_NAME="public" >> /root/keystonerc_admin
  echo export OS_FLAVOR_ID=99 >> /root/keystonerc_admin
  echo export OS_FLAVOR_ID_RESIZE=98 >> /root/keystonerc_admin
  echo export OS_DOMAIN_NAME=default >> /root/keystonerc_admin
  echo export OS_TENANT_NAME=\$OS_PROJECT_NAME >> /root/keystonerc_admin
  echo export OS_TENANT_ID=\$OS_PROJECT_ID >> /root/keystonerc_admin
  echo export OS_SHARE_NETWORK_ID="foobar" >> /root/keystonerc_admin

  echo "" >> /root/keystonerc_demo
  echo export OS_IMAGE_NAME="cirros" >> /root/keystonerc_demo
  echo export OS_IMAGE_ID="$_IMAGE_ID" >> /root/keystonerc_demo
  echo export OS_NETWORK_ID=$_NETWORK_ID >> /root/keystonerc_demo
  echo export OS_EXTGW_ID=$_EXTGW_ID >> /root/keystonerc_demo
  echo export OS_POOL_NAME="public" >> /root/keystonerc_demo
  echo export OS_FLAVOR_ID=99 >> /root/keystonerc_demo
  echo export OS_FLAVOR_ID_RESIZE=98 >> /root/keystonerc_demo
  echo export OS_DOMAIN_NAME=default >> /root/keystonerc_demo
  echo export OS_TENANT_NAME=\$OS_PROJECT_NAME >> /root/keystonerc_demo
  echo export OS_TENANT_ID=\$OS_PROJECT_ID >> /root/keystonerc_demo
  echo export OS_SHARE_NETWORK_ID="foobar" >> /root/keystonerc_demo

+ mysql -e "CREATE DATABASE pdns default character set utf8 default collate utf8_general_ci"
+ mysql -e "GRANT ALL PRIVILEGES ON pdns.* TO 'pdns'@'localhost' IDENTIFIED BY 'password'"
+ yum install -y epel-release yum-plugin-priorities
+ curl -o /etc/yum.repos.d/powerdns-auth-40.repo
+ yum install -y pdns pdns-backend-mysql
+ echo "daemon=no
+ allow-recursion=
+ config-dir=/etc/powerdns
+ daemon=yes
+ disable-axfr=no
+ guardian=yes
+ local-address=
+ local-ipv6=::
+ local-port=53
+ setgid=pdns
+ setuid=pdns
+ slave=yes
+ socket-dir=/var/run
+ version-string=powerdns
+ out-of-zone-additional-processing=no
+ webserver=yes
+ api=yes
+ api-key=someapikey
+ launch=gmysql
+ gmysql-host=
+ gmysql-user=pdns
+ gmysql-dbname=pdns
+ gmysql-password=password" | tee /etc/pdns/pdns.conf
+ mysql pdns < /home/centos/files/pdns.sql
+ sudo systemctl restart pdns

  yum install -y wget git
  wget -O /usr/local/bin/gimme
  chmod +x /usr/local/bin/gimme
  eval "$(/usr/local/bin/gimme 1.8)"
  export GOPATH=$HOME/go
  export PATH=$PATH:$GOROOT/bin:$GOPATH/bin

  go get
  pushd ~/go/src/
  go get -u ./...

  cat >> /root/.bashrc <<EOF
  if [[ -f /usr/local/bin/gimme ]]; then
    eval "\$(/usr/local/bin/gimme 1.8)"
    export GOPATH=\$HOME/go
    export PATH=\$PATH:\$GOROOT/bin:\$GOPATH/bin

  gophercloudtest() {
    if [[ -n \$1 ]] && [[ -n \$2 ]]; then
      pushd  ~/go/src/
      go test -v -tags "fixtures acceptance" -run "\$1"\$2 | tee ~/gophercloud.log

  systemctl stop openstack-cinder-backup.service
  systemctl stop openstack-cinder-scheduler.service
  systemctl stop openstack-cinder-volume.service
  systemctl stop openstack-nova-cert.service
  systemctl stop openstack-nova-compute.service
  systemctl stop openstack-nova-conductor.service
  systemctl stop openstack-nova-consoleauth.service
  systemctl stop openstack-nova-novncproxy.service
  systemctl stop openstack-nova-scheduler.service
  systemctl stop neutron-dhcp-agent.service
  systemctl stop neutron-l3-agent.service
  systemctl stop neutron-lbaasv2-agent.service
  systemctl stop neutron-metadata-agent.service
  systemctl stop neutron-openvswitch-agent.service
  systemctl stop neutron-metering-agent.service

  mysql -e "update services set deleted_at=now(), deleted=id" cinder
  mysql -e "update services set deleted_at=now(), deleted=id" nova
  mysql -e "update compute_nodes set deleted_at=now(), deleted=id" nova
  for i in $(openstack network agent list -c ID -f value); do
    neutron agent-delete $i

  systemctl stop httpd

  cp /home/centos/files/rc.local /etc
  chmod +x /etc/rc.local

There are four things begin done above:

  1. A MySQL database is created for PowerDNS.
  2. PowerDNS is then installed.
  3. A configuration file is created.
  4. A database schema is imported into the PowerDNS database.

You'll notice the schema is located in a file titled files/pdns.sql. Add the following to terraform-openstack-test/packer/files/pdns.sql:

CREATE TABLE domains (
  id                    INT AUTO_INCREMENT,
  name                  VARCHAR(255) NOT NULL,
  master                VARCHAR(128) DEFAULT NULL,
  last_check            INT DEFAULT NULL,
  type                  VARCHAR(6) NOT NULL,
  notified_serial       INT DEFAULT NULL,
  account               VARCHAR(40) DEFAULT NULL,
) Engine=InnoDB;

CREATE UNIQUE INDEX name_index ON domains(name);

CREATE TABLE records (
  id                    BIGINT AUTO_INCREMENT,
  domain_id             INT DEFAULT NULL,
  name                  VARCHAR(255) DEFAULT NULL,
  type                  VARCHAR(10) DEFAULT NULL,
  content               VARCHAR(64000) DEFAULT NULL,
  ttl                   INT DEFAULT NULL,
  prio                  INT DEFAULT NULL,
  change_date           INT DEFAULT NULL,
  disabled              TINYINT(1) DEFAULT 0,
  ordername             VARCHAR(255) BINARY DEFAULT NULL,
  auth                  TINYINT(1) DEFAULT 1,
) Engine=InnoDB;

CREATE INDEX nametype_index ON records(name,type);
CREATE INDEX domain_id ON records(domain_id);
CREATE INDEX recordorder ON records (domain_id, ordername);

CREATE TABLE supermasters (
  ip                    VARCHAR(64) NOT NULL,
  nameserver            VARCHAR(255) NOT NULL,
  account               VARCHAR(40) NOT NULL,
  PRIMARY KEY (ip, nameserver)
) Engine=InnoDB;

CREATE TABLE comments (
  id                    INT AUTO_INCREMENT,
  domain_id             INT NOT NULL,
  name                  VARCHAR(255) NOT NULL,
  type                  VARCHAR(10) NOT NULL,
  modified_at           INT NOT NULL,
  account               VARCHAR(40) NOT NULL,
  comment               VARCHAR(64000) NOT NULL,
) Engine=InnoDB;

CREATE INDEX comments_domain_id_idx ON comments (domain_id);
CREATE INDEX comments_name_type_idx ON comments (name, type);
CREATE INDEX comments_order_idx ON comments (domain_id, modified_at);

CREATE TABLE domainmetadata (
  id                    INT AUTO_INCREMENT,
  domain_id             INT NOT NULL,
  kind                  VARCHAR(32),
  content               TEXT,
) Engine=InnoDB;

CREATE INDEX domainmetadata_idx ON domainmetadata (domain_id, kind);

CREATE TABLE cryptokeys (
  id                    INT AUTO_INCREMENT,
  domain_id             INT NOT NULL,
  flags                 INT NOT NULL,
  active                BOOL,
  content               TEXT,
) Engine=InnoDB;

CREATE INDEX domainidindex ON cryptokeys(domain_id);

CREATE TABLE tsigkeys (
  id                    INT AUTO_INCREMENT,
  name                  VARCHAR(255),
  algorithm             VARCHAR(50),
  secret                VARCHAR(255),
) Engine=InnoDB;

CREATE UNIQUE INDEX namealgoindex ON tsigkeys(name, algorithm);

Installing Designate

Now that will install and configure PowerDNS, add the steps to install and configure Designate:

  hostnamectl set-hostname localhost

  systemctl disable firewalld
  systemctl stop firewalld
  systemctl disable NetworkManager
  systemctl stop NetworkManager
  systemctl enable network
  systemctl start network

  yum install -y
  yum install -y centos-release-openstack-ocata
  yum-config-manager --enable openstack-ocata
  yum update -y
  yum install -y openstack-packstack
  packstack --answer-file /home/centos/files/packstack-answers.txt

  source /root/keystonerc_admin
  nova flavor-create m1.acctest 99 512 5 1 --ephemeral 10
  nova flavor-create m1.resize 98 512 6 1 --ephemeral 10
  _NETWORK_ID=$(openstack network show private -c id -f value)
  _SUBNET_ID=$(openstack subnet show private_subnet -c id -f value)
  _EXTGW_ID=$(openstack network show public -c id -f value)
  _IMAGE_ID=$(openstack image show cirros -c id -f value)

  echo "" >> /root/keystonerc_admin
  echo export OS_IMAGE_NAME="cirros" >> /root/keystonerc_admin
  echo export OS_IMAGE_ID="$_IMAGE_ID" >> /root/keystonerc_admin
  echo export OS_NETWORK_ID=$_NETWORK_ID >> /root/keystonerc_admin
  echo export OS_EXTGW_ID=$_EXTGW_ID >> /root/keystonerc_admin
  echo export OS_POOL_NAME="public" >> /root/keystonerc_admin
  echo export OS_FLAVOR_ID=99 >> /root/keystonerc_admin
  echo export OS_FLAVOR_ID_RESIZE=98 >> /root/keystonerc_admin
  echo export OS_DOMAIN_NAME=default >> /root/keystonerc_admin
  echo export OS_TENANT_NAME=\$OS_PROJECT_NAME >> /root/keystonerc_admin
  echo export OS_TENANT_ID=\$OS_PROJECT_ID >> /root/keystonerc_admin
  echo export OS_SHARE_NETWORK_ID="foobar" >> /root/keystonerc_admin

  echo "" >> /root/keystonerc_demo
  echo export OS_IMAGE_NAME="cirros" >> /root/keystonerc_demo
  echo export OS_IMAGE_ID="$_IMAGE_ID" >> /root/keystonerc_demo
  echo export OS_NETWORK_ID=$_NETWORK_ID >> /root/keystonerc_demo
  echo export OS_EXTGW_ID=$_EXTGW_ID >> /root/keystonerc_demo
  echo export OS_POOL_NAME="public" >> /root/keystonerc_demo
  echo export OS_FLAVOR_ID=99 >> /root/keystonerc_demo
  echo export OS_FLAVOR_ID_RESIZE=98 >> /root/keystonerc_demo
  echo export OS_DOMAIN_NAME=default >> /root/keystonerc_demo
  echo export OS_TENANT_NAME=\$OS_PROJECT_NAME >> /root/keystonerc_demo
  echo export OS_TENANT_ID=\$OS_PROJECT_ID >> /root/keystonerc_demo
  echo export OS_SHARE_NETWORK_ID="foobar" >> /root/keystonerc_demo

  mysql -e "CREATE DATABASE pdns default character set utf8 default collate utf8_general_ci"
  mysql -e "GRANT ALL PRIVILEGES ON pdns.* TO 'pdns'@'localhost' IDENTIFIED BY 'password'"

  yum install -y epel-release yum-plugin-priorities
  curl -o /etc/yum.repos.d/powerdns-auth-40.repo
  yum install -y pdns pdns-backend-mysql

  echo "daemon=no
  gmysql-password=password" | tee /etc/pdns/pdns.conf

  mysql pdns < /home/centos/files/pdns.sql
  sudo systemctl restart pdns

+ openstack user create --domain default --password password designate
+ openstack role add --project services --user designate admin
+ openstack service create --name designate --description "DNS" dns
+ openstack endpoint create --region RegionOne dns public
+ mysql -e "CREATE DATABASE designate CHARACTER SET utf8 COLLATE utf8_general_ci"
+ mysql -e "CREATE DATABASE designate_pool_manager"
+ mysql -e "GRANT ALL PRIVILEGES ON designate.* TO 'designate'@'localhost' IDENTIFIED BY 'password'"
+ mysql -e "GRANT ALL PRIVILEGES ON designate_pool_manager.* TO 'designate'@'localhost' IDENTIFIED BY 'password'"
+ mysql -e "GRANT ALL PRIVILEGES ON designate.* TO 'designate'@'localhost' IDENTIFIED BY 'password'"
+ yum install -y crudini
+ yum install -y openstack-designate\*
+ cp /home/centos/files/pools.yaml /etc/designate/
+ designate_conf="/etc/designate/designate.conf"
+ crudini --set $designate_conf DEFAULT debug True
+ crudini --set $designate_conf DEFAULT debug True
+ crudini --set $designate_conf DEFAULT notification_driver messaging
+ crudini --set $designate_conf service:api enabled_extensions_v2 "quotas, reports"
+ crudini --set $designate_conf keystone_authtoken auth_uri
+ crudini --set $designate_conf keystone_authtoken auth_url
+ crudini --set $designate_conf keystone_authtoken username designate
+ crudini --set $designate_conf keystone_authtoken password password
+ crudini --set $designate_conf keystone_authtoken project_name services
+ crudini --set $designate_conf keystone_authtoken auth_type password
+ crudini --set $designate_conf service:worker enabled true
+ crudini --set $designate_conf service:worker notify true
+ crudini --set $designate_conf storage:sqlalchemy connection mysql+pymysql://designate:password@
+ sudo -u designate designate-manage database sync
+ systemctl enable designate-central designate-api
+ systemctl enable designate-worker designate-producer designate-mdns
+ systemctl restart designate-central designate-api
+ systemctl restart designate-worker designate-producer designate-mdns
+ sudo -u designate designate-manage pool update

  yum install -y wget git
  wget -O /usr/local/bin/gimme
  chmod +x /usr/local/bin/gimme
  eval "$(/usr/local/bin/gimme 1.8)"
  export GOPATH=$HOME/go
  export PATH=$PATH:$GOROOT/bin:$GOPATH/bin

  go get
  pushd ~/go/src/
  go get -u ./...

  cat >> /root/.bashrc <<EOF
  if [[ -f /usr/local/bin/gimme ]]; then
    eval "\$(/usr/local/bin/gimme 1.8)"
    export GOPATH=\$HOME/go
    export PATH=\$PATH:\$GOROOT/bin:\$GOPATH/bin

  gophercloudtest() {
    if [[ -n \$1 ]] && [[ -n \$2 ]]; then
      pushd  ~/go/src/
      go test -v -tags "fixtures acceptance" -run "\$1"\$2 | tee ~/gophercloud.log

  systemctl stop openstack-cinder-backup.service
  systemctl stop openstack-cinder-scheduler.service
  systemctl stop openstack-cinder-volume.service
  systemctl stop openstack-nova-cert.service
  systemctl stop openstack-nova-compute.service
  systemctl stop openstack-nova-conductor.service
  systemctl stop openstack-nova-consoleauth.service
  systemctl stop openstack-nova-novncproxy.service
  systemctl stop openstack-nova-scheduler.service
  systemctl stop neutron-dhcp-agent.service
  systemctl stop neutron-l3-agent.service
  systemctl stop neutron-lbaasv2-agent.service
  systemctl stop neutron-metadata-agent.service
  systemctl stop neutron-openvswitch-agent.service
  systemctl stop neutron-metering-agent.service
+ systemctl stop designate-central designate-api
+ systemctl stop designate-worker designate-producer designate-mdns

  mysql -e "update services set deleted_at=now(), deleted=id" cinder
  mysql -e "update services set deleted_at=now(), deleted=id" nova
  mysql -e "update compute_nodes set deleted_at=now(), deleted=id" nova
  for i in $(openstack network agent list -c ID -f value); do
    neutron agent-delete $i

  systemctl stop httpd

  cp /home/centos/files/rc.local /etc
  chmod +x /etc/rc.local

There are several steps happening above:

  1. The openstack command is used to create a new service account called designate. A catalog endpoint is also created.
  2. A database called designate is created.
  3. A utility called crudini is installed. This is an amazing little tool to help modify ini files on the command-line.
  4. Designate is installed.
  5. A bundled pools.yaml file is copied to /etc/designate. I'll show the contents of this file soon.
  6. crudini is used to configure /etc/designate/designate.conf.
  7. The Designate database's schema is imported using the designate-manage command.
  8. The Designate services are enabled in systemd.
  9. designate-manage is again used, this time to update the DNS pools.
  10. The Designate services are added to the list of services to stop before the image/snapshot is created.

These steps roughly follow what was pulled from the Designate Installation Guide linked to earlier.

As mentioned, a pools.yaml file is copied from the files directory. Create a file called terraform-openstack-test/packer/files/pools.yaml with the following contents:


- name: default
  description: Default PowerDNS Pool
  attributes: {}
    - hostname:
      priority: 1

    - host:
      port: 53

    - type: pdns4
      description: PowerDNS4 DNS Server
        - host:
          port: 5354

      # PowerDNS Configuration options
        port: 53
        api_token: someapikey

Finally, modify the rc.local file:

  set -x

  export HOME=/root

  sleep 60

  if [[ -n $public_ip ]]; then
    while true ; do
      mysql -e "update endpoint set url = replace(url, '', '$public_ip')" keystone
      if [[ $? == 0 ]]; then
      sleep 10

    sed -i -e "s/$public_ip/g" /root/keystonerc_demo
    sed -i -e "s/$public_ip/g" /root/keystonerc_admin

  systemctl restart rabbitmq-server
  while [[ true ]]; do
    pgrep -f rabbit
    if [[ $? == 0 ]]; then
    sleep 10
    systemctl restart rabbitmq-server

  systemctl restart openstack-cinder-api.service
  systemctl restart openstack-cinder-backup.service
  systemctl restart openstack-cinder-scheduler.service
  systemctl restart openstack-cinder-volume.service
  systemctl restart openstack-nova-cert.service
  systemctl restart openstack-nova-compute.service
  systemctl restart openstack-nova-conductor.service
  systemctl restart openstack-nova-consoleauth.service
  systemctl restart openstack-nova-novncproxy.service
  systemctl restart openstack-nova-scheduler.service
  systemctl restart neutron-dhcp-agent.service
  systemctl restart neutron-l3-agent.service
  systemctl restart neutron-lbaasv2-agent.service
  systemctl restart neutron-metadata-agent.service
  systemctl restart neutron-openvswitch-agent.service
  systemctl restart neutron-metering-agent.service
  systemctl restart httpd
+ systemctl restart designate-central designate-api
+ systemctl restart designate-worker designate-producer designate-mdns
+ systemctl restart pdns

  nova-manage cell_v2 discover_hosts

+ sudo -u designate designate-manage pool update
+ iptables -I INPUT -p tcp --dport 9001 -j ACCEPT
+ ip6tables -I INPUT -p tcp --dport 9001 -j ACCEPT
  iptables -I INPUT -p tcp --dport 80 -j ACCEPT
  ip6tables -I INPUT -p tcp --dport 80 -j ACCEPT
  cp /root/keystonerc* /var/www/html
  chmod 666 /var/www/html/keystonerc*

The above steps have been added:

  1. The Designate services have been added to the list of services to be restarted during boot.
  2. PowerDNS is also restarted
  3. designate-manage is again used to update the DNS pools.
  4. Port 9001 is opened for traffic.

Build the Image and Launch

With the above in place, you can regenerate your image using Packer and then launch a virtual machine using Terraform.

When the virtual machine is up and running, you'll find that your testing environment is now running OpenStack Designate.


This blog post covered how to add a service to your OpenStack testing environment that is not supported by PackStack. This was done by reviewing the steps to manually install and configure the service, translating those steps to automated commands, and adding those commands to the existing deployment scripts.

October 09, 2017 06:00 AM