Planet SysAdmin

March 01, 2017

Anton Chuvakin - Security Warrior

Monthly Blog Round-Up – Feburary 2016

Here is my next monthly "Security Warrior" blog round-up of top 5 popular posts/topics this month:
  1. “New SIEM Whitepaper on Use Cases In-Depth OUT!” (dated 2010, so I have no idea why it tops the charts now!) presents a whitepaper on select SIEM use cases described in depth with rules and reports [using now-defunct SIEM product]; also see this SIEM use case in depth and this for a more current list of popular SIEM use cases. Finally, see our 2016 research on developing security monitoring use cases here!
  2. Why No Open Source SIEM, EVER?” contains some of my SIEM thinking from 2009. Is it relevant now? You be the judge.  Succeeding with SIEM requires a lot of work, whether you paid for the software, or not. BTW, this post has an amazing “staying power” that is hard to explain – I suspect it has to do with people wanting “free stuff” and googling for “open source SIEM” … 
  3. Simple Log Review Checklist Released!” is often at the top of this list – this aging checklist is still a very useful tool for many people. “On Free Log Management Tools” (also aged a bit by now) is a companion to the checklist (updated version)
  4. This month, my classic PCI DSS Log Review series is extra popular! The series of 18 posts cover a comprehensive log review approach (OK for PCI DSS 3+ even though it predates it), useful for building log review processes and procedures, whether regulatory or not. It is also described in more detail in our Log Management book and mentioned in our PCI book (now in its 4th edition!) – note that this series is mentioned in some PCI Council materials.
  5. “SIEM Resourcing or How Much the Friggin’ Thing Would REALLY Cost Me?” is a quick framework for assessing the SIEM project (well, a program, really) costs at an organization (a lot more details on this here in this paper).
In addition, I’d like to draw your attention to a few recent posts from my Gartner blog [which, BTW, now has about 5X of the traffic of this blog]: 
Recent research on security analytics and UBA / UEBA:
Miscellaneous fun posts:

(see all my published Gartner research here)
Also see my past monthly and annual “Top Popular Blog Posts” – 2007, 2008, 2009, 2010, 2011, 2012, 2013, 2014, 2015, 2016.

Disclaimer: most content at SecurityWarrior blog was written before I joined Gartner on August 1, 2011 and is solely my personal view at the time of writing. For my current security blogging, go here.

Previous post in this endless series:

by Anton Chuvakin ( at March 01, 2017 04:15 PM

LZone - Sysadmin

Match Structured Facts in MCollective

If you are using Facter 2+, which is what you do when you run at least Puppet4, then you have structured facts (meaning nested values) like those:
processors => {
  count => 2,
  isa => "unknown",
  models => [
    "QEMU Virtual CPU version 2.1.2",
    "QEMU Virtual CPU version 2.1.2"
  physicalcount => 2
Now you cannot match those using
mco find -F <fact name>=<fact value>
If you try you just get an empty result. The only way to match structured facts is using -S
mco find -S 'fact("<fact name>").value=<value>'
For example:
mco find -S 'fact("").value='
mco find -S 'fact("os.distro.codename").value=jessie'
See also Mcollective Cheat Sheet

March 01, 2017 03:41 PM

HTTP Error 429 on Reddit

Getting HTTP error 429 when trying to call Reddit APIs or .json endpoints? Try changing your User Agent header to something else. Reddit bans based on user agent.

by admin at March 01, 2017 09:31 AM

DNS Spy enters public beta

The post DNS Spy enters public beta appeared first on

Here's an exciting announcement I've been dying to make: DNS Spy, a new DNS monitoring and alerting tool I've been working on, has entered public beta!

After several months of development and a 3 month private, invite-only, alpha period, DNS Spy is now publicly available to anyone to try it out.

What is DNS Spy?

I'm glad you ask!

DNS Spy is a custom solution for monitoring, alerting and tracking your DNS changes. It was built primarily out of convenience for the following use cases;

  • Get notified when a project I'm working on changes its DNS (usually for the go-live of a new website or application)
  • Get notified when one of my domains has nameserver records that are out-of-sync because zone transfers failed or the SOA didn't get updated properly
  • Get a back-up of all my DNS records per domain in a convenient format, a necessity that became clear after the massive Dyn DNS DDoS attacks

DNS Spy will never be a tool for the masses. It's a tool for sysadmins, developers and managers that are paranoid about their DNS.

In fact, it's built upon a tool I created more than 6 years ago that did one, simple thing: send me an e-mail whenever a DNS record I cared about, changes value. This is the same tool, on steroids.

Free for Open Source projects

One thing I'm very paranoid about, is the current state of our Open Source projects. While a lot of them have big companies supporting them, the majority is run by volunteers doing everything they can to keep things running.

But due to the nature of Open Source, those projects might become hugely popular on purpose or by accident. Imagine operating a project that's used as a library in an OS, a popular tool or a framework. Suddenly your code gets used in thousands if not millions of applications or hardware.

What would happen if someone managed to hijack your DNS records and set up a phishing site, similar to yours? Would you know? Would your auto-updaters know? Or the package builders that download your .tar.gz files and include your code in their own projects?

I seriously worry about the current state of our DNS and domain names for open source projects. That's why I offer a lifetime premium account to DNS Spy for any open source project maintainer to help secure and monitor their projects.

If someone hijacks your DNS or you make modifications with unexpected results, I hope DNS Spy can offer a solution or guidance.

Feedback, please!

Since it's still in active development, I'm very much looking for feedback, bug reports and enhancements! If you spot anything, do let me know!

And if you'd like to do me a favor, spread word of DNS Spy to colleagues, open source maintainers and internet geeks -- I can use all the publicity in the world. ;-)

The post DNS Spy enters public beta appeared first on

by Mattias Geniar at March 01, 2017 07:45 AM

Chris Siebenmann

Using Certificate Transparency to monitor your organization's TLS activity

One of the obvious things that you can do with Certificate Transparency is to monitor the CT logs for bad people somehow getting a certificate for one of your websites. If you're Paypal or Google or the University of Toronto, and you see a CT log entry for a '' certificate that isn't yours, you can ring lots of alarms. You can do this with actual infrastructure (perhaps based on the actual logs, and see also, and also), or you can do this on a manual and ad-hoc basis through one of the websites that let you query the CT logs, such as Google's or, or maybe Facebook's (which of course requires a Facebook login because Facebook, that's why).

But there's another use for it, and that is looking for people in your own organization who are getting properly issued certificates. Perhaps I'm biased by working in a university, but around here there's no central point that really controls TLS certificates; if you can persuade a TLS certificate provider to give you a certificate, people will. And these days, the existence of Let's Encrypt means that if you have control over your own hosts, you can probably get certificates for them. If you are in such an organization, monitoring Certificate Transparency logs is one way to keep track of who is doing roughly what with TLS, perhaps discover interesting services you want to know about, and so on.

(Perhaps you are saying 'we control who gets to run TLS services because we control the perimeter firewall'. Do you control DNS too, so that people can't point off to things they're hosting in AWS? You probably don't want to go that far, by the way, because the alternative is for people to buy their own domain names too and then they won't even show up in your CT monitoring.)

You don't even have to be at the top of an organization to find this interesting, because sometimes there are subgroups all the way down. Some of our graduate students run machines that can be reached from the outside world, and I'm sure that sooner or later some of them will want a TLS certificate and discover Let's Encrypt. It's reassuring to know that when this happens we have at least some chance of finding out about it.

(Not an entirely great chance, because sometimes professors set up new domain names for graduate student projects and don't tell us about them.)

PS: Of course, as a bystander in your (overall) organization you can also use CT logs to satisfy your curiosity about things like how common Let's Encrypt certificates are, and how broadly spread they are across your organization. Is your group one of the few areas actively experimenting with them, or are a whole lot of people using them all over the place?

(All of this is probably pretty obvious, but I feel like writing it down.)

by cks at March 01, 2017 04:19 AM

Errata Security

Some moon math

So "Brianna Wu" (famous for gamergate) is trending, and because I love punishment, I clicked on it to see why. Apparently she tweeted that Elon Musk's plan to go to the moon is bad, because once there he can drop rocks on the Earth with the power of 100s of nuclear bombs. People are mocking her for the stupidity of this.

But the math checks out.

First of all, she probably got the idea from Heinlein's book The Moon is a Harsh Mistress where the rebel moon colonists do just that. I doubt she did her own math, and relied upon Heinlein to do it for her. But let's do the math ourselves.

Let's say that we want to stand at the height of the moon and drop a rock. How big a rock do we need to equal the energy of an atomic bomb? To make things simple, let's assume the size of bombs we want is that of the one dropped on Hiroshima.

As we know from high school physics, the energy of a dropped object (ignoring air) is:
energy = 0.5 * mass * velocity * velocity
Solving for mass (the size of the rock), the equation is:
mass = 2 * energy/(velocity * velocity)
We choose "energy" as that of an atomic bomb, but what is "velocity" in this equation, the speed of something dropped from the height of the moon?

The answer is something close to the escape velocity, which is defined as the speed of something dropped infinitely far away from the Earth. The moon isn't infinitely far away (only 250,000 miles away), but it's close.

How close? Well, let's use the formula for escape velocity from Wikipedia [*]:

where G is the "gravitational constant", M is the "mass of Earth", and r is the radius. Plugging in "radius of earth" and we get an escape velocity from the surface of the Earth of 11.18 km/s, which matches what Google tells us. Plugging in the radius of the moon's orbit, we get 1.44 km/s [*]. Thus, we get the following as the speed of an object dropped from the height of the moon to the surface of the earth, barring air resistance [*]:
9.74 km/s
Plugging these numbers in gets the following result:

So the answer for the mass of the rock, dropped from the moon, to equal a Hiroshima blast, is 1.3 billion grams, or 1.3 million kilograms, or 1.3 thousand metric tons.

Well, that's a fine number and all, but what does that equal? Is that the size of Rhode Island? or just a big truck?

The answer is: nearly the same mass as the Space Shuttle during launch (2.03 million kilograms [*]). Or, a rock about 24 feet on a side.

That's big rock, but not so big that it's impractical, especially since things weigh 1/6th as on Earth. In Heinlein's books, instead of shooting rocks via rockets, it shot them into space using a railgun, magnetic rings. Since the moon doesn't have an atmosphere, you don't need to shoot things straight up. Instead, you can accelerate them horizontally across the moon's surface, to an escape velocity of 5,000 mph (escape velocity from moon's surface). As the moon's surface curves away, they'll head out into space (or toward Earth)

Thus, Elon Musk would need to:
  • go the moon
  • setup a colony, underground
  • mine iron ore
  • build a magnetic launch gun
  • build fields full of solar panels for energy
  • mine some rock
  • cover it in iron (for magnet gun to hold onto)
  • bomb earth
At that point, he could drop hundreds of "nukes" on top of us. I, for one, would welcome our Lunar overlords. Free Luna!

Update: I've made a number of short cuts, but I don't think they'll affect the math much.

We don't need escape velocity for the moon as a whole, just enough to reach the point where Earth's gravity takes over. On the other hand, we need to kill the speed of the Moons's orbit (2,000 miles per hour) in order to get down to Earth, or we just end up orbiting the Earth. I just assume the two roughly cancel each other out and ignore it.

I also ignore the atmosphere. Meteors from outer space hitting the earth of this size tend to disintegrate or blow up before reaching the surface. The Chelyabinsk meteor, the one in all those dashcam videos from 2013, was roughly 5 times the size of our moon rocks, and blew up in the atmosphere, high above the surface, with about 5 times the energy of a Hiroshima bomb. Presumably, we want our moon rocks to reach the surface, so they'll need some protection. Probably make them longer and thinner, and put an ablative heat shield up from, and wrap them in something strong like iron.

I don't know how much this will slow down the rock. Presumably, if coming straight down, it won't slow down by much, but if coming in at a steep angle (as meteors do), then it could slow down quite a lot.

Update: First version of this post used "height of moon", which Wolfram Alfa interpreted as "diameter of moon". This error was found by . The current version of this post changes this to the correct value "radius of moon's orbit".

Update: I made a stupid error about Earth's gravitational strength at the height of the Moon's orbit. I've changed the equations to fix this.

by Robert Graham ( at March 01, 2017 01:39 AM

February 28, 2017

Mitigating PHP’s long standing issue with OPCache leaking sensitive data

The post Mitigating PHP’s long standing issue with OPCache leaking sensitive data appeared first on

A very old security vulnerability has been fixed in PHP regarding the way it handles its OPCaches in environments where a single master process shares multiple PHP-FPM pools. This is the most common way to run PHP nowadays and might affect you, too.

The vulnerability

PHP has a method to speed up the dynamic nature of its interpreter, called bytecode caching. PHP gets interpreted on every pageload, meaning the PHP gets translated to bytecode which the server understands and can execute. Since most PHP pages don't change every second, PHP caches that bytecode in memory and can serve that as the response instead of having to compile (or "interpret") the PHP scripts every time.

In a default PHP-FPM setup, the process tree looks like this.

php-fpm: master process (/etc/php-fpm.conf)
\_ php-fpm: pool site1 (uid: 701)
\_ php-fpm: pool site2 (uid: 702)
\_ php-fpm: pool site3 (uid: 703)

There is a single PHP-FPM master process that gets started as the root user. It then spawns additional FPM pools, that can each run as their own user, to serve websites. The PHP OPCache (that bytecode cache) is held in the master process.

So here's the problem: PHP does not validate the userid when it fetches a script from memory, stored in its bytecode. The concept of a "shared memory" in PHP means everything is stored in the same memory segment, and including a file just checks if a version already exists in bytecode.

If a version of the script exists in bytecode, it's served without additional check.

If a version of the script does not exist in bytecode, the FPM pool (running as an unprivileged uid) will try to read it from disk, and Linux will prevent reads from files that the process has no access to. You know, like it's supposed to happen.

This is the one of the primary reasons I've advocated running multiple masters instead of multiple pools for years.

The solution: upgrade PHP & set additional configurations

After a way too long period, this bug is now resolved and you can fix it with additional master configurations.

$ cat /etc/php-fpm.conf
opcache.validate_permission (default "0")
       Leads OPcache to check file readability on each access to cached file.
       This directive should be enabled in shared hosting environment, when few
       users (PHP-FPM pools) reuse the common OPcache shared memory.

opcache.validate_root (default "0")
       This directive prevents file name collisions in different "chroot"
       environments. It should be enabled for sites that may serve requests in
       different "chroot" environments.

The introduction of the opcache.validate_permission and opcache.validate_root means you can now force PHP's OPCache to also check the permissions of the file and force a validation of the root path of the file, to avoid chrooted environments from reading eachothers' files (more on that in the original bugreport).

The default values, however, are insecure.

I understand why they are like this, to keep compatibility with previous versions and avoid breaking changes in minor versions, but you have to explicitly enable them to prevent this behaviour.

Minimum versions: PHP 5.6.29, 7.0.14, 7.1.0

This fix didn't get much attention and I only noticed it after someone posted to the oss-security mailing list. To mitigate this properly, you'll need at least;

Anything lower than 5.6 is already end of life and won't get this fix, even though OPCache got introduced in PHP 5.5. You'll need a recent 5.6 or newer to mitigate this.

If you still run PHP 5.5 (which many do) and want to be safe, your best bet is to either run multiple PHP masters (a single master per pool) or disable OPCache entirely.

Indirect local privilege escalation to root

This has been a long standing issue with PHP that's actually more serious than it looks at first glance.

If you manage to compromise a single website (which for most CMS's in PHP that don't get updated, isn't very hard), this shared memory allows you access to all other websites, if their pages have been cached in the OPCache.

You can effectively use this shared memory buffer as a passage way to read other website's PHP files, read their configs, get access to their database and steal all their data. To do so, you usually need root access to a server, but PHP's OPCache gives you a convenient shortcut to accomplish similar things.

All you need is the path to their files, which can either be retrieved via opcache_get_status() (if enabled, which again -- it is by default), or you can guess the paths, which on shared hosting environments usually isn't hard either.

This actually makes this shared memory architecture almost as bad as a local privilege escalation vulnerability, depending on what you want to accomplish. If your goal is to steal data from a server that normally requires root privileges, you got it.

If your goal is to actually get root and run root-only scripts (aka: bind on lower ports etc.), that won't be possible.

The good part is, there's now a fix with the new PHP configurations. The bad news is, you need to update to the latest releases to get it and explicitly enable it.

Additional sources

For your reading pleasure:

The post Mitigating PHP’s long standing issue with OPCache leaking sensitive data appeared first on

by Mattias Geniar at February 28, 2017 09:45 AM

Errata Security

Some notes on space heaters (GPU rigs)

So I carried my GPU rig up to my bedroom to act as a space heater. I thought I'd write some notes on it.

This is a "GPU rig", containing five graphics cards. Graphics cards have highly parallel processors (GPUs) with roughly 10 times the performance of a CPU -- but only for highly parallel problems.

Two such problems are password cracking [*] and cryptocurrency mining.

Password cracking is something cybersecurity professionals regularly need to do. When doing a pentest, or assessment, we'll get lists of passwords we need to crack. Having a desktop computer with a couple of graphics cards is a useful thing to have.

There are three popular cryptocurrencies: Bitcoin, Ethereum, and ZCash. Everyone is using ASICs for Bitcoin, so you can't mine them on a GPU any more, but GPUs are still useful for Ethereum and ZCash.

The trick to building a rig with lots of GPU is to get a PCIe 1x extender, so that you can mount the card far away from the motherboard for better cooling. They cost around $10 each. You then need to buy a motherboard with lots of PCIe slots. One with lots of 1x slots will do -- we don't need a lot of bandwidth to the cards.

You then need to buy either a single high-end power supply, or team together two cheaper power supplies. The limitation will be the power from the wall socket, which ranges from around 1600 watts to 1900 watts.

If you don't want to build a rig, but just stick one or two GPUs in your desktop computer, then here are some things to consider.

There are two vendors of GPUs: nVidia and AMD/Radeon. While nVidia has a better reputation for games and high-end supercomputer math (floating point), Radeon's have been better with the integer math used for crypto. So you want Radeon cards.

Older cards work well. The 5-year-old Radeon 7970 is actually a pretty good card for this sort of work. You may find some for free from people discarding them in favor of newer cards for gaming.

If buying newer cards, the two you care about are either the Radeon R9 Fury/Nano, or the Radeon RX 470/480. The Fury/Nano cards are slightly faster, the RX 470/480 are more power efficient.

You want to get at least 4 gigabytes of memory per card, which I think is what they come with anyway. You might consider 8 gigabytes. The reason for this is that Ethereum is designed to keep increasing memory requirements, to avoid the way ASICs took over in Bitcoin mining. At some point in the future, 4 gigabytes won't be enough and you'll need 8 gigabytes. This is many years away, but seeing how old cards remaining competitive for many years, it's something to consider.

With all this said, if you've got a desktop and want to add a card, or if you want to build a rig, then I suggest the following card:

  • AMD Radeon RX 480 w/ 8gigs or RAM for $199 at Newegg [*]

A few months from now, things will of course change, but it's a good choice for now. This is especially useful for rigs: 6 Fury cards in a rig risks overloading the wall socket, so that somebody innocently turning on a light could crash your rig. In contrast, a rig 6 RX480 cards fit well within the power budget of a single socket.

Now let's talk software. For password cracking, get Hashcat. For mining, choose a mining pool, and they'll suggest software. The resources at are excellent for either Windows or Linux mining. Though, on Windows, I couldn't get mining to work unless I also went back to older video drivers, which was a pain.

Let's talk electrical power consumption. Mining profitability is determined by your power costs. Where I live, power costs $0.05/kwh, except during summer months (June through September). This is extremely cheap. In California, power costs closer to $0.15/kwh. The difference is significant. I make a profit at the low rates, but would lose money at the higher rates.

Because everyone else is doing it, you can never make money at mining. Sure, if you run the numbers now, you may convince yourself that you'll break even after a year, but that assumes everything stays static. Other people are making the same calculation, and will buy new GPUs to enter the market, driving down returns, so nobody ever breaks even. The only time mining is profitable is when the price of the cryptocurrency goes up -- but in that case, you'd have made even more money just buying the cryptocurrency instead of a mining rig.

The reason I mine is that I do it over TOR, the Onion Router that hides my identity. This mines clean, purely anonymous Bitcoins that I can use (in theory) to pay for hosting services anonymously. I haven't done that yet, but I'll have several Bitcoins worth of totally anonymous currency to use if I ever need them.

Otherwise, I use the rig for password cracking. The trick to password cracking is being smarter (knowing what to guess), not being more powerful. But having a powerful rig helps, too, especially since I can just start it up and let it crunch unattended for days.


You probably don't want to build a rig, unless you are geek like me who enjoys playing with technology. You'll never earn back what you invest in it, and it's a lot of hassle setting up.

On the other hand, if you have a desktop computer, you might want to stick in an extra graphics card for password cracking and mining in the background. This is especially true if you want to generate anonymous Bitcoin.

by Robert Graham ( at February 28, 2017 03:40 AM

Chris Siebenmann

The conflict between wildcard TLS certificates and Certificate Transparency

Certificate Transparency is an increasingly important part of the modern TLS world, especially for website certificates (which I believe are still the dominant use of TLS certificates). One part of Certificate Transparency is monitoring for certificates issued for your own sites and domains, but that's not the only use; another one is looking for certificates issued to suspicious names. For instance, a bunch of people would probably be interested if someone issued a certificate for or or any number of other TLS certificate names like that.

(This is not just of interest to the real Paypal and Google, it's also of interest to things like email spam filtering systems, intrusion detection systems, and the systems that help provide browser warnings of suspicious websites.)

The examples I've shown here are top level domain names, but that doesn't have to be the case. Often it's going to be easier to get an alarming name into a subdomain instead of a domain; for a start, you don't have to worry about a domain registrar alerting the moment something that matches *paypal* shows up in a new domain registration. When an attacker embeds the alarming name as a subdomain, one of the few ways that outside people can spot it is when the TLS certificate information shows up in the Certificate Transparency logs, because the TLS certificate exposes the full host name.

Well, at least until wildcard certificates come along. When combined with CT, the effect of wildcard certificates is to hide from scrutiny all of the names that can be put into the wildcarded portion. People monitoring the CT logs no longer see 'login.paypal.really.somedom.wat'; all they see is '*.somedom.wat' or '*.really.somedom.wat', which of course means that they basically see nothing.

(There are good aspects of this as well as bad ones, since CT with full host names exposes internal host names that you may not want to have known for various reasons.)

As a result, I'm not particularly surprised that Let's Encrypt doesn't support wildcard certificates. Let's Encrypt is intended for public hosts, and with automated issuance I feel that Certificate Transparency is especially important in case something goes wrong. Not issuing wildcard certificates maximizes public visibility into what LE is actually doing and issuing.

With all of this said, Let's Encrypt's FAQ says that their primary reason for not issuing wildcard certificates is the question of automated issuance (which I suspect partly means automated proving of control), not any philosophical reason. It's possible that LE would decide they had philosophical reasons too if people came up with a good technical solution; I guess we may find out someday.

by cks at February 28, 2017 02:57 AM

Cryptography Engineering

The future of Ransomware

This is kind of a funny post for me to write, since it involves speculating about a very destructive type of software — and possibly offering some (very impractical) suggestions on how it might be improved in the future. It goes without saying that there are some real downsides to this kind of speculation. Nonetheless, I’m going ahead on the theory that it’s usually better to talk and think about the bad things that might happen to you — before you meet them on the street and they steal your lunch money.

On the other hand, just as there’s a part of every karate master that secretly wants to go out and beat up a bar full of people, there’s a part of every security professional that looks at our current generation of attackers and thinks: why can’t you people just be a bit more imaginative?! And wonders whether, if our attackers were just a little more creative, people would actually pay attention to securing their system before the bad stuff happens.

And ransomware is definitely a bad thing. According to the FBI it sucks up $1 billion/year in payments alone, and some unimaginably larger amount in remediation costs. This despite the fact that many ransomware packages truly suck, and individual ransomware developers get routinely pwned due to making stupid cryptographic errors. If this strategy is working so well today, the question  we should be asking ourselves is: how much worse could it get?

So that’s what I’m going to muse about now. A few (cryptographic) ways that it might.

Some of these ideas are the result of collaboration with my students Ian Miers, Gabe Kaptchuk and Christina Garman. They range from the obvious to the foolish to the whimsical, and I would be utterly amazed if any of them really do happen. So please don’t take this post too seriously. It’s all just fun.

Quick background: ransomware today

The amazing thing about ransomware is that something so simple could turn out to be such a problem. Modern ransomware consists of malware that infects your computer and then goes about doing something nasty: it encrypts every file it can get its hands on. This typically includes local files as well as network shares that can be reached from the infected machine.

Once your data has been encrypted, your options aren’t great. If you’re lucky enough to have a recent backup, you can purge the infected machine and restore. Otherwise you’re faced with a devil’s bargain: learn top live without that data, or pay the bastards.

If you choose to pay up, there are all sorts of different procedures. However most break down into the following three steps:

  1. When the ransomware encrypts your files, it generates a secret key file and stores it on your computer.
  2. You upload that file (or data string) to your attackers along with a Bitcoin payment.
  3. They process the result with their secrets and send you a decryption key.

If you’re lucky, and your attackers are still paying attention (or haven’t screwed up the crypto beyond recognition) you get back a decryption key or a tool you can use to undo the encryption on your files. The whole thing is very businesslike. Indeed, recent platforms will allegedly offer you a discount if you infect recommend it to your friends — just like Lyft!

The problem of course, is that nothing in this process guarantees that your attacker will give you that decryption key. They might be scammers. They might not have the secret anymore. They might get tracked down and arrested. Or they might get nervous and bail, taking your precious data and your payment with them. This uncertainty makes ransomware payments inherently risky — and worse, it’s the victims who mostly suffer for it.

Perhaps it would be nice if we could make that work better.

Verifiable key delivery using smart contracts

Most modern ransomware employs a cryptocurrency like Bitcoin to enable the payments that make the ransom possible. This is perhaps not the strongest argument for systems like Bitcoin — and yet it seems unlikely that Bitcoin is going away anytime soon. If we can’t solve the problem of Bitcoin, maybe it’s possible to use Bitcoin to make “more reliable” ransomware.

Recall that following a ransomware infection, there’s a possibility that you’ll pay the ransom and get nothing in return. Fundamentally there’s very little you can do about this. A conscientious ransomware developer might in theory offer a “proof of life” — that is, offer to decrypt a few files at random in order to prove their bonafides. But even if they bother with all the risk and interaction of doing this, there’s still no guarantee that they’ll bother to deliver the hostage alive.

An obvious approach to this problem is to make ransomware payments conditional. Rather than sending off your payment and hoping for the best, victims could use cryptocurrency features to ensure that ransomware operators can’t get paid unless they deliver a key. Specifically, a ransomware developer could easily perform payment via a smart contract script (in a system like Ethereum) that guarantees the following property:

This payment will be delivered to the ransomware operator if and only if the ransomware author unlocks it — by posting the ransomware decryption key to the same blockchain.

There are various ways to do this. The decryption key itself could be a pre-image k for some public value K that the ransomware generates and leaves on your system. It’s relatively easy to imagine a smart contract that allows payment if and only if the payee can post the input k such that K=SHA256(k). This could easily be written in Ethereum, and almost certainly has an analog for Bitcoin script.

The challenge here, of course, is to prove that k is actually a decryption key for your files, and that the files contain valid data. There are a handful of different ways to tackle this problem. One is to use complex zero-knowledge proof techniques (like zkSNARKs or ZKBoo) to make the necessary proofs non-interactively. But this is painful, and frankly above the level of most ransomware developers — who are still struggling with basic RSA.

An alternative approach is to use several such K challenges in combination with the “proof of life” idea. The ransomware operator would prove her bonafides by decrypting a small, randomly selected subset of files before the issuer issued payment. The operator could still “fake” the encryption — or lose the decryption key — but she would be exposed with reasonable probability before money changed hands.

“Autonomous” ransomware

Of course, the problem with “verifiable” ransomware is: what ransomware developer would bother with this nonsense?

While the ability to verify decryption might conceivably improve customer satisfaction, it’s not clear that it would really offer that much value to ransomware deverlopers. At the same time, it would definitely add a lot of nasty complexity to their software.

Instead of pursuing ideas that offer developers no obvious upside, ransomware designers presumably will pursue ideas that offer them some real benefits. And that brings us to an idea time whose time has (hopefully) not quite come yet. The idea itself is simple:

Make ransomware that doesn’t require operators.

Recall that in the final step of the ransom process, the ransomware operator must deliver a decryption key to the victim. This step is the most fraught for operators, since it requires them to manage keys and respond to queries on the Internet. Wouldn’t it be better for operators if they could eliminate this step altogether?

Of course, to accomplish this seems to require a trustworthy third party — or better, a form of ransomware that can decrypt itself when the victim makes a Bitcoin payment. Of course this last idea seems fundamentally contradictory. The decryption keys would have to live on the victim’s device, and the victim owns that device. If you tried that, then victim could presumably just hack the secrets out and decrypt the ransomware without paying.

But what if the victim couldn’t hack their own machine?

This isn’t a crazy idea. In fact, it’s exactly the premise that’s envisioned by a new class of trusted execution environments, including Intel’s SGX and ARM TrustZone. These systems — which are built into the latest generation of many processors — allow users to instantiate “secure enclaves”: software environments that can’t be accessed by outside parties. SGX also isolates enclaves from other enclaves, which means the secrets they hold are hard to pry out.

Hypothetically, after infecting your computer a piece of ransomware could generate and store its decryption key inside of a secure enclave. This enclave could be programmed to release the key only on presentation of a valid Bitcoin payment to a designated address.

The beauty of this approach is that no third party even needs to verify the payment. Bitcoin payments themselves consist of a publicly-verifiable transaction embedded in a series of “blocks”, each containing an expensive computational “proof of work“. In principle, after paying the ransom the victim could present the SGX enclave with a fragment of a blockchain all by itself — freeing the ransomware of the need to interact with third parties. If the blockchain fragment exhibited sufficient hashpower along with a valid payment to a specific address, the enclave would release the decryption key.*

The good news is that Intel and ARM have devoted serious resources to preventing this sort of unauthorized access. SGX developers must obtain a code signing certificate from Intel before they can make production-ready SGX enclaves, and it seems unlikely that Intel would partner up with a ransomware operation. Thus a ransomware operator would likely have to (1) steal a signing key from a legitimate Intel-certified developer, or (2) find an exploitable vulnerability in another developer’s enclave.**, ***

This all seems sort of unlikely, and that appears to block most of the threat — for now. Assuming companies like Intel and Qualcomm don’t screw things up, and have a good plan for revoking enclaves (uh oh), this is not very likely to be a big threat.

Of course, in the long run developers might not need Intel SGX at all. An even more speculative concern is that developments in the field of cryptographic obfuscation will provide a software-only alternative means to implement this type of ransomware. This would eliminate the need for a dependency like SGX altogether, allowing the ransomware to do its work with no hardware at all.

At present such techniques are far north of practical, keep getting broken, and might not work at all. But cryptographic researchers keep trying! I guess the lesson is that it’s not all roses if they succeed.

Ransomware Skynet

Since I’m already this far into what reads like a Peyote-fueled rant, let’s see if we can stretch the bounds of credibility just a little a bit farther. If ransomware can become partially autonomous — i.e., do part of its job without the need for human masters — what would it mean for it to become fully autonomous? In other words, what if we got rid of the rest of the human equation?

I come from the future to encrypt C:\Documents

Ransomware with the ability to enforce payments would provide a potent funding source for another type of autonomous agent: a Decentralized Autonomous Organization, or (DAO). These systems are “corporations” that consist entirely of code that runs on a consensus network like Ethereum. They’re driven by rules, and are capable of both receiving and transmitting funds without (direct) instruction from human beings.

At least in theory it might be possible to develop a DAO that’s funded entirely by ransomware payments — and in turn mindlessly contracts real human beings to develop better ransomware, deploy it against human targets, and… rinse repeat. It’s unlikely that such a system would be stable in the long run — humans are clever and good at destroying dumb things — but it might get a good run. Who knows? Maybe this is how the Rampant Orphan Botnet Ecologies get started.

(I hope it goes without saying that I’m mostly not being serious about this part. Even though it would be totally awesome in a horrible sort of way.)

In conclusion

This hasn’t been a terribly serious post, although it was fun to write. The truth is that as a defender, watching your attackers fiddle around is pretty much the most depressing thing ever. Sometimes you have to break the monotony a bit.

But insofar as there is a serious core to this post, it’s that ransomware currently is using only a tiny fraction of the capabilities available to it. Secure execution technologies in particular represent a giant footgun just waiting to go off if manufacturers get things only a little bit wrong.

Hopefully they won’t, no matter how entertaining it might be.


* This technique is similar to SPV verification. Of course, it would also be possible for a victim to “forge” a blockchain fragment without paying the ransom. However, the cost of this could easily be tuned to significantly exceed the cost of paying the ransom. There are also many issues I’m glossing over here like difficulty adjustments and the possibility of amortizing the forgery over many different victims. But thinking about that stuff is a drag, and this is all for fun, right?

** Of course, if malware can exploit such a vulnerability in another developer’s enclave to achieve code execution for “ransomware”, then the victim could presumably exploit the same vulnerability to make the ransomware spit out its key without a payment. So this strategy seems self-limiting — unless the ransomware developers find a bug that can be “repaired” by changing some immutable state held by the enclave. That seems like a long shot. And no, SGX does not allow you to “seal” data to the current state of the enclave’s RAM image.

*** In theory, Intel or an ARM manufacturer could also revoke the enclave’s signing certificate. However, the current SGX specification doesn’t explain how such a revocation strategy should work. I assume this will be more prominent in future specifications.

by Matthew Green at February 28, 2017 12:47 AM

February 27, 2017

LZone - Sysadmin

Nagios Check for Systemd Failed Units

Just a short bash script to check for faulty systemd units to avoid 107 lines of Python...

if [ -f /bin/systemctl ]; then failed=$(/bin/systemctl --failed --no-legend) failed=${failed/ */} # Strip everything after first space failed=${failed/.service/} # Strip .service suffix

if [ "$failed" != "" ]; then echo "Failed units: $failed" exit 1 else echo "No failed units." exit 0 fi else echo "No systemd. Nothing was checked!" exit 0 fi

February 27, 2017 05:50 PM

Chris Siebenmann

In Python, strings are infinitely recursively iterable

Yesterday I posed the side question of what happened when you called the following code with flatten2(["abcdef",]):

def flatten2(inlst):
    olst = []
    for i in inlst:
            it = iter(i)
        except TypeError:
            it = None
        if it is None:
    return olst

The intent of this code is to recursively flatten iterable things. What I had expected to get when I called it with a string was ['a', 'b', 'c', 'd', 'e', 'f'], ie that it flattened the string instead of leaving it alone (because strings are iterable). What I actually got (and what you can get) is a RecursionError of 'maximum recursion depth exceeded'. So why did this recurse endlessly?

The answer is that strings are not just iterable, they are infinitely recursively iterable. With normal iterable containers, iterating the container yields non-iterable items unless you've explicitly put in iterable ones (such as a list inside another list); assuming that you have not cleverly embedded a cycle in your container, our recursive flattening will eventually bottom out and finish. This is not the case for Python strings. When you iterate a multi-character string like "abcdef", you get a sequence of single-character strings, "a" "b" "c" "d" "e" "f". However, these are still strings and so they are still iterable; when you iterate the "a" string, you get back another single-character string "a", which is also iterable. And so flatten2() chases down an infinite recursion of trying to unpack single-character strings into non-iterable items, which it will never succeed at.

Fixing this without side effects is a bit annoying. It's not enough to immediately check that inlst is length 1 and just return it if so, because this fails on flatten2([[1,]]) (and flatten2(["abcdef",]) too, for that matter). I think you have to check explicitly for length 1 strings (and bytes) and just return them, which of course leaves you exposed to any other infinitely recursively iterable types (should there be any other out there).

(I was about to gripe about repr()'s representation of single character strings but then I tested and it uses single quotes for all strings, not just single-character ones. I need to remember that in Python there is no 'character' type that's different from strings, unlike in other languages such as Go, and so single quotes still mean 'string'.)

Sidebar: It's trivially possible to create containers with cycles

>>> a = [1, 2, 3]
>>> a.append(a)
>>> print(a)
[1, 2, 3, [...]]

More involved versions are left as an exercise for the reader.

Without contemplating it too hard, I think that you can only create cycles with mutable containers, and then only with some of them. Sets are mutable, for example, but they can only contain hashable items, which are generally immutable.

by cks at February 27, 2017 02:43 AM

February 25, 2017

Evaggelos Balaskas

Docker Swarm a native clustering system

Docker Swarm

The native Docker Container Orchestration system is Docker Swarm that in simple terms means that you can have multiple docker machines (hosts) to run your multiple docker containers (replicas). It is best to work with Docker Engine v1.12 and above as docker engine includes docker swarm natively.

Docker Swarm logo:

In not so simply terms, docker instances (engines) running on multiple machines (nodes), communicating together (VXLAN) as a cluster (swarm).


To begin with, we need to create our docker machines. One of the nodes must be the manager and the others will run as workers. For testing purposes I will run three (3) docker engines:

  • Manager Docker Node: myengine0
  • Worker Docker Node 1: myengine1
  • Worker Docker Node 2: myengine2


A docker node is actually a machine that runs the docker engine in the swarm mode. The machine can be a physical, virtual, a virtualbox, a cloud instance, a VPS, a AWS etc etc

As the time of this blog post, officially docker supports natively the below drivers:

  • Amazon Web Services
  • Microsoft Azure
  • Digital Ocean
  • Exoscale
  • Google Compute Engine
  • Generic
  • Microsoft Hyper-V
  • OpenStack
  • Rackspace
  • IBM Softlayer
  • Oracle VirtualBox
  • VMware vCloud Air
  • VMware Fusion
  • VMware vSphere


but there are unofficial drivers also.

I will use the qemu - kvm driver from this github repository:

The simplest way to add the kvm driver is this:

> cd /usr/local/bin/
> sudo -s
# wget -c
# chmod 0750 docker-machine-driver-kvm

Docker Machines

The next thing we need to do, is to create our docker machines. Look on your distro’s repositories:

# yes | pacman -S docker-machine


$ docker-machine create -d kvm myengine0

Running pre-create checks...
Creating machine...
(myengine0) Image cache directory does not exist, creating it at /home/ebal/.docker/machine/cache...
(myengine0) No default Boot2Docker ISO found locally, downloading the latest release...
(myengine0) Latest release for is v1.13.1
(myengine0) Downloading /home/ebal/.docker/machine/cache/boot2docker.iso from
(myengine0) 0%....10%....20%....30%....40%....50%....60%....70%....80%....90%....100%
(myengine0) Copying /home/ebal/.docker/machine/cache/boot2docker.iso to /home/ebal/.docker/machine/machines/myengine0/boot2docker.iso...

Waiting for machine to be running, this may take a few minutes...
Detecting operating system of created instance...
Waiting for SSH to be available...
Detecting the provisioner...
Provisioning with boot2docker...
Copying certs to the local machine directory...
Copying certs to the remote machine...
Setting Docker configuration on the remote daemon...
Checking connection to Docker...
Docker is up and running!
To see how to connect your Docker Client to the Docker Engine running on this virtual machine, run: docker-machine env myengine0

Worker 1

$ docker-machine create -d kvm myengine1
Running pre-create checks...
Creating machine...
(myengine1) Copying /home/ebal/.docker/machine/cache/boot2docker.iso to /home/ebal/.docker/machine/machines/myengine1/boot2docker.iso...
Waiting for machine to be running, this may take a few minutes...
Detecting operating system of created instance...
Waiting for SSH to be available...
Detecting the provisioner...
Provisioning with boot2docker...
Copying certs to the local machine directory...
Copying certs to the remote machine...
Setting Docker configuration on the remote daemon...
Checking connection to Docker...
Docker is up and running!
To see how to connect your Docker Client to the Docker Engine running on this virtual machine, run: docker-machine env myengine1

Worker 2

$ docker-machine create -d kvm myengine2
Running pre-create checks...
Creating machine...
(myengine2) Copying /home/ebal/.docker/machine/cache/boot2docker.iso to /home/ebal/.docker/machine/machines/myengine2/boot2docker.iso...
Waiting for machine to be running, this may take a few minutes...
Detecting operating system of created instance...
Waiting for SSH to be available...
Detecting the provisioner...
Provisioning with boot2docker...
Copying certs to the local machine directory...
Copying certs to the remote machine...
Setting Docker configuration on the remote daemon...
Checking connection to Docker...
Docker is up and running!
To see how to connect your Docker Client to the Docker Engine running on this virtual machine, run: docker-machine env myengine2

List your Machines

$ docker-machine env myengine0
export DOCKER_HOST="tcp://"
export DOCKER_CERT_PATH="/home/ebal/.docker/machine/machines/myengine0"
export DOCKER_MACHINE_NAME="myengine0"
# Run this command to configure your shell:
# eval $(docker-machine env myengine0)

$ docker-machine ls

NAME        ACTIVE   DRIVER   STATE     URL                         SWARM   DOCKER    ERRORS
myengine0   -        kvm      Running   tcp://           v1.13.1
myengine1   -        kvm      Running   tcp://            v1.13.1
myengine2   -        kvm      Running   tcp://           v1.13.1


You can get the IP of your machines with:

$ docker-machine ip myengine0

$ docker-machine ip myengine1

$ docker-machine ip myengine2

with ls as seen above or use the inspect parameter for a full list of information regarding your machines in a json format:

$ docker-machine inspect myengine0

If you have jq you can filter out some info

$ docker-machine inspect myengine0  | jq .'Driver.DiskPath'



To enter inside the kvm docker machine, you can use ssh


$ docker-machine ssh myengine0 

                        ##         .
                  ## ## ##        ==
               ## ## ## ## ##    ===
           /"""""""""""""""""___/ ===
      ~~~ {~~ ~~~~ ~~~ ~~~~ ~~~ ~ /  ===- ~~~
           ______ o           __/
 _                 _   ____     _            _
| |__   ___   ___ | |_|___  __| | ___   ___| | _____ _ __
| '_  / _  / _ | __| __) / _` |/ _  / __| |/ / _  '__|
| |_) | (_) | (_) | |_ / __/ (_| | (_) | (__|   <  __/ |
|_.__/ ___/ ___/ __|_______,_|___/ ___|_|____|_|
Boot2Docker version 1.13.1, build HEAD : b7f6033 - Wed Feb  8 20:31:48 UTC 2017
Docker version 1.13.1, build 092cba3

Worker 1

$ docker-machine ssh myengine1 

                        ##         .
                  ## ## ##        ==
               ## ## ## ## ##    ===
           /"""""""""""""""""___/ ===
      ~~~ {~~ ~~~~ ~~~ ~~~~ ~~~ ~ /  ===- ~~~
           ______ o           __/
 _                 _   ____     _            _
| |__   ___   ___ | |_|___  __| | ___   ___| | _____ _ __
| '_  / _  / _ | __| __) / _` |/ _  / __| |/ / _  '__|
| |_) | (_) | (_) | |_ / __/ (_| | (_) | (__|   <  __/ |
|_.__/ ___/ ___/ __|_______,_|___/ ___|_|____|_|
Boot2Docker version 1.13.1, build HEAD : b7f6033 - Wed Feb  8 20:31:48 UTC 2017
Docker version 1.13.1, build 092cba3

Worker 2

$ docker-machine ssh myengine2

                        ##         .
                  ## ## ##        ==
               ## ## ## ## ##    ===
           /"""""""""""""""""___/ ===
      ~~~ {~~ ~~~~ ~~~ ~~~~ ~~~ ~ /  ===- ~~~
           ______ o           __/
 _                 _   ____     _            _
| |__   ___   ___ | |_|___  __| | ___   ___| | _____ _ __
| '_  / _  / _ | __| __) / _` |/ _  / __| |/ / _  '__|
| |_) | (_) | (_) | |_ / __/ (_| | (_) | (__|   <  __/ |
|_.__/ ___/ ___/ __|_______,_|___/ ___|_|____|_|
Boot2Docker version 1.13.1, build HEAD : b7f6033 - Wed Feb  8 20:31:48 UTC 2017
Docker version 1.13.1, build 092cba3

Swarm Cluster

Now it’s time to build a swarm of docker machines!

Initialize the manager

docker@myengine0:~$  docker swarm init --advertise-addr

Swarm initialized: current node (jwyrvepkz29ogpcx18lgs8qhx) is now a manager.

To add a worker to this swarm, run the following command:

    docker swarm join
    --token SWMTKN-1-4vpiktzp68omwayfs4c3j5mrdrsdavwnewx5834g9cp6p1koeo-bgcwtrz6srt45qdxswnneb6i9

To add a manager to this swarm, run 'docker swarm join-token manager' and follow the instructions.

Join Worker 1

docker@myengine1:~$  docker swarm join
>     --token SWMTKN-1-4vpiktzp68omwayfs4c3j5mrdrsdavwnewx5834g9cp6p1koeo-bgcwtrz6srt45qdxswnneb6i9

This node joined a swarm as a worker.

Join Worker 2

docker@myengine2:~$   docker swarm join
>     --token SWMTKN-1-4vpiktzp68omwayfs4c3j5mrdrsdavwnewx5834g9cp6p1koeo-bgcwtrz6srt45qdxswnneb6i9

This node joined a swarm as a worker.

From the manager

docker@myengine0:~$  docker node ls

jwyrvepkz29ogpcx18lgs8qhx *  myengine0  Ready   Active        Leader
m5akhw7j60fru2d0an4lnsgr3    myengine2  Ready   Active
sfau3r42bqbhtz1c6v9hnld67    myengine1  Ready   Active


We can find more information about the docker-machines running the docker info command when you have ssh-ed the nodes:

eg. the swarm part:


Swarm: active
 NodeID: jwyrvepkz29ogpcx18lgs8qhx
 Is Manager: true
 ClusterID: 8fjv5fzp0wtq9hibl7w2v65cs
 Managers: 1
 Nodes: 3
  Task History Retention Limit: 5
  Snapshot Interval: 10000
  Number of Old Snapshots to Retain: 0
  Heartbeat Tick: 1
  Election Tick: 3
  Heartbeat Period: 5 seconds
 CA Configuration:
  Expiry Duration: 3 months
 Node Address:
 Manager Addresses:


Swarm: active
 NodeID: sfau3r42bqbhtz1c6v9hnld67
 Is Manager: false
 Node Address:
 Manager Addresses:

worker 2

Swarm: active
 NodeID: m5akhw7j60fru2d0an4lnsgr3
 Is Manager: false
 Node Address:
 Manager Addresses:


Now it’s time to test our docker swarm by running a container service across our entire fleet!

For testing purposes we chose 6 replicas of an nginx container:

docker@myengine0:~$ docker service create --replicas 6 -p 80:80 --name web nginx


List images

docker@myengine0:~$  docker images
REPOSITORY          TAG                 IMAGE ID            CREATED             SIZE
nginx               <none>              db079554b4d2        9 days ago          182 MB

List of services

regarding your docker registry or your internet connection, we will see the replicas running:

docker@myengine0:~$ docker service ls
ID            NAME  MODE        REPLICAS  IMAGE
ql6iogo587ib  web   replicated  0/6       nginx:latest

docker@myengine0:~$ docker service ls
ID            NAME  MODE        REPLICAS  IMAGE
ql6iogo587ib  web   replicated  2/6       nginx:latest

docker@myengine0:~$ docker service ls
ID            NAME  MODE        REPLICAS  IMAGE
ql6iogo587ib  web   replicated  3/6       nginx:latest

docker@myengine0:~$ docker service ls
ID            NAME  MODE        REPLICAS  IMAGE
ql6iogo587ib  web   replicated  6/6       nginx:latest
docker@myengine0:~$  docker service ps web

t3v855enecgv  web.1  nginx:latest  myengine1  Running        Running 17 minutes ago
xgwi91plvq00  web.2  nginx:latest  myengine2  Running        Running 17 minutes ago
0l6h6a0va2fy  web.3  nginx:latest  myengine0  Running        Running 16 minutes ago
qchj744k0e45  web.4  nginx:latest  myengine1  Running        Running 17 minutes ago
udimh2bokl8k  web.5  nginx:latest  myengine2  Running        Running 17 minutes ago
t50yhhtngbac  web.6  nginx:latest  myengine0  Running        Running 16 minutes ago


To verify that our replicas are running as they should:


Scaling a service

It’s really interesting that we can scale out or scale down our replicas on the fly !

from the manager

docker@myengine0:~$  docker service ls
ID            NAME  MODE        REPLICAS  IMAGE
ql6iogo587ib  web   replicated  6/6       nginx:latest

docker@myengine0:~$ docker service ps web
t3v855enecgv  web.1  nginx:latest  myengine1  Running        Running 3 days ago
xgwi91plvq00  web.2  nginx:latest  myengine2  Running        Running 3 days ago
0l6h6a0va2fy  web.3  nginx:latest  myengine0  Running        Running 3 days ago
qchj744k0e45  web.4  nginx:latest  myengine1  Running        Running 3 days ago
udimh2bokl8k  web.5  nginx:latest  myengine2  Running        Running 3 days ago
t50yhhtngbac  web.6  nginx:latest  myengine0  Running        Running 3 days ago

Scale Down

from the manager

$ docker service scale web=3
web scaled to 3

docker@myengine0:~$ docker service ls
ID            NAME  MODE        REPLICAS  IMAGE
ql6iogo587ib  web   replicated  3/3       nginx:latest

docker@myengine0:~$ docker service ps web
0l6h6a0va2fy  web.3  nginx:latest  myengine0  Running        Running 3 days ago
qchj744k0e45  web.4  nginx:latest  myengine1  Running        Running 3 days ago
udimh2bokl8k  web.5  nginx:latest  myengine2  Running        Running 3 days ago

Scale Up

from the manager

docker@myengine0:~$ docker service scale web=8
web scaled to 8
docker@myengine0:~$ docker service ls
ID            NAME  MODE        REPLICAS  IMAGE
ql6iogo587ib  web   replicated  3/8       nginx:latest
docker@myengine0:~$ docker service ls
ID            NAME  MODE        REPLICAS  IMAGE
ql6iogo587ib  web   replicated  4/8       nginx:latest
docker@myengine0:~$ docker service ls
ID            NAME  MODE        REPLICAS  IMAGE
ql6iogo587ib  web   replicated  8/8       nginx:latest
docker@myengine0:~$ docker service ps web
lyhoyseg8844  web.1  nginx:latest  myengine1  Running        Running 7 seconds ago
w3j9bhcn9f6e  web.2  nginx:latest  myengine2  Running        Running 8 seconds ago
0l6h6a0va2fy  web.3  nginx:latest  myengine0  Running        Running 3 days ago
qchj744k0e45  web.4  nginx:latest  myengine1  Running        Running 3 days ago
udimh2bokl8k  web.5  nginx:latest  myengine2  Running        Running 3 days ago
vr8jhbum8tlg  web.6  nginx:latest  myengine1  Running        Running 7 seconds ago
m4jzati4ddpp  web.7  nginx:latest  myengine2  Running        Running 8 seconds ago
7jek2zvuz6fs  web.8  nginx:latest  myengine0  Running        Running 11 seconds ago

February 25, 2017 09:19 PM

February 24, 2017

Errata Security

A quick note about iconoclasm

I'm an iconoclast [*]. Whenever things become holy, whereby any disagreement is treated as heresy, then I disagree. There are two reasonable sides to every argument. When you vilify one of the sides in the argument, then I step into defend them -- not that they are right, but that they are reasonable.

This makes many upset, because once a cause has become Holy, anybody disagreeing with orthodoxy (like me) is then, by definition, a horrible person. I get things like the image to the right.

(Please don't harass/contact this person -- she believes what many do, and singling her out would be mean).

For the record, I'm rabidly feminist, anti-racist, pro-LGBT, pro-civil-rights. It's just that while I care a lot, I'm not Orthodox. I likely disagree with you about the details. When you vilify those who disagree with you, I will defend them.

...which is the best troll, ever. Admitting somebody is wrong, but defending them as reasonable, seems to upset people more than just arguing the other side is right.

by Robert Graham ( at February 24, 2017 10:29 PM

Cloudbleed: Cloudflare Reverse Proxies have Dumped Uninitialized Memory

The post Cloudbleed: Cloudflare Reverse Proxies have Dumped Uninitialized Memory appeared first on

This is a scary find. Props to the Cloudflare team for the fix and Google for cleaning up their search results. But since Google isn't the only scraper on the internet and other parties might have discovered this already, you might want to seriously consider informing your clients if you're behind a Cloudflare proxy.

I'll quote the scariest parts of the find below. Read the entire articles for more context.

It looked like that if an html page hosted behind cloudflare had a specific combination of unbalanced tags, the proxy would intersperse pages of uninitialized memory into the output (kinda like heartbleed, but cloudflare specific and worse for reasons I'll explain later).

For a sense of scale and impact:

We keep finding more sensitive data that we need to cleanup. I didn't realize how much of the internet was sitting behind a Cloudflare CDN until this incident.


I'm finding private messages from major dating sites, full messages from a well-known chat service, online password manager data, frames from adult video sites, hotel bookings. We're talking full https requests, client IP addresses, full responses, cookies, passwords, keys, data, everything.

In other words: any site behind Cloudflare might have leaked info from other sites hosted behind Cloudflare. Even if your HTML was perfectly balanced and strict, your site might've become the victim of another site with imbalanced HTML tags that leaked your data.

Source: 1139 -- cloudflare: Cloudflare Reverse Proxies are Dumping Uninitialized Memory -- project-zero -- Monorail

Extra details: Cloudflare Incident report on memory leak caused by Cloudflare parser bug

The post Cloudbleed: Cloudflare Reverse Proxies have Dumped Uninitialized Memory appeared first on

by Mattias Geniar at February 24, 2017 07:23 AM

February 23, 2017

Steve Kemp's Blog

Rotating passwords

Like many people I use a password-manage to record logins to websites. I previously used a tool called pwsafe, but these days I switched to using pass.

Although I don't like the fact the meta-data is exposed the tool is very useful, and its integration with git is both simple and reliable.

Reading about the security issue that recently affected cloudflare made me consider rotating some passwords. Using git I figured I could look at the last update-time of my passwords. Indeed that was pretty simple:

git ls-tree -r --name-only HEAD | while read filename; do
  echo "$(git log -1 --format="%ad" -- $filename) $filename"

Of course that's not quite enough because we want it sorted, and to do that using the seconds-since-epoch is neater. All together I wrote this:

# Show password age - should be useful for rotation - we first of all
# format the timestamp of every *.gpg file, as both unix+relative time,
# then we sort, and finally we output that sorted data - but we skip
# the first field which is the unix-epoch time.
( git ls-tree -r --name-only HEAD | grep '\.gpg$' | while read filename; do \
      echo "$(git log -1 --format="%at %ar" -- $filename) $filename" ; done ) \
        | sort | awk '{for (i=2; i<NF; i++) printf $i " "; print $NF}'

Not the cleanest script I've ever hacked together, but the output is nice:

 steve@ssh ~ $ cd ~/Repos/personal/pass/
 steve@ssh ~/Repos/personal/pass $ ./password-age | head -n 5
 1 year, 10 months ago GPG/root@localhost.gpg
 1 year, 10 months ago GPG/
 1 year, 10 months ago GPG/
 1 year, 10 months ago Git/
 1 year, 10 months ago Git/

Now I need to pick the sites that are more than a year old and rotate credentials. Or delete accounts, as appropriate.

February 23, 2017 10:00 PM

How to solve RPMs created by Alien having file conflicts

I generate release packages for my software with Alien, which amongst other things converts .deb packages to .rpm.

On Fedora 24 however, the generated RPMs cause a small problem when installed with Yum:

Transaction check error:
  file / from install of cfgtrack-1.0-2.noarch conflicts with file from package filesystem-3.2-20.el7.x86_64
  file /usr/bin from install of cfgtrack-1.0-2.noarch conflicts with file from package filesystem-3.2-20.el7.x86_64

There's a bit of info to be found on the internet about this problem, with most of the posts suggesting using rpmrebuild to fix it. Unfortunately, it looks like rpmrebuild actually requires the package to be installed, and since I don't actually use a RPM-based system, that was a bit of a no-go.

So here's how to fix those packages manually:

First, use Alient to generate a RPM package folder from a Debian package, but don't generate the actual package yet. You can do so with the -g switch:

alien -r -g -v myproject-1.0.deb

This generates a myproject-1.0 directory containing the root fs for the package as well as a myproject-1.0-2.spec file. This spec file is the actual problem. It defines directories for paths such as / and /usr/bin. But those are already provided by the filesystem package, so we shouldn't include them.

You can remove them from a script using sed:

sed -i 's#%dir "/"##' myproject-1.0/myproject-1.0-2.spec
sed -i 's#%dir "/usr/bin/"##' myproject-1.0/myproject-1.0-2.spec

This edits the spec file in-place and replaces the following lines with empty lines:

%dir "/"
%dir "/usr/bin/"

The regular expressions look somewhat different than usual, because I'm using the pound (#) sign as a reg marker instead of "/".

Finally, we can recreate the package using rpmbuild:

cd myproject-1.0
rpmbuild --target=noarch --buildroot /full/path/to/myproject-1.0/ \
         -bb cfgtrack-$(REL_VERSION)-2.spec

The resulting package should install without errors or warnings now.

by admin at February 23, 2017 06:27 AM

February 22, 2017


Bad speaker advice

Last year, while I was developing my talks, I saw a bit of bad advice. I didn't recognize it at the time. Instead, I saw it as a goal to reach. The forum was a private one, and I've long forgotten who the players were. But here is a reconstructed, summarized view of what spurred me to try:

elph1120: You know what I love? A speaker who can do an entire talk from one slide.
612kenny: OMG yes. I saw someguy do that at someconference. It was amazeballs.
elph1120: Yeah, more speakers should do that.
gryphon: Totally.

This is bad advice. Don't do this.

Now to explain what happened...

I saw this, and decided to try and do that for my DevOpsDays Minneapolis talk last year. I got close, I needed 4 slides. Which is enough to fit into a tweet.

See? No link to SlideShare needed! Should be amazing!

It wasn't.

The number one critique I got, by a large, large margin was this:

Wean yourself from the speaker-podium.

In order to do a 4-slide talk, I had to lean pretty hard on speaker-notes. If you're leaning on speaker-notes, you're either tied to the podium or have cue-cards in your hands. Both of these are violations of the modern TED-talk style-guide tech-conferences are following these days. I should have noticed that the people rhapsodizing over one-slide talks were habitues of one of the holdouts of podium-driven talks in the industry.

That said, there is another way to do a speaker-note free talk: the 60-slide deck for a 30 minute talk. Your slides are the notes. So long as you can remember some points to talk about above and beyond what's written on the slides, you're providing value above and beyond the deck you built. The meme-slide laugh inducers provide levity and urge positive feedback. If you're new to speaking this is the style you should be aiming for.

A one-slide talk is PhD level speaking-skills. It means memorizing paragraph by paragraph a 3K word essay, and read it back while on stage and on camera. You should not be trying to reach this bar until you're already whatever about public speaking, and have delivered that talk a bunch of times already.

by SysAdmin1138 at February 22, 2017 04:27 PM

February 21, 2017


OpenSSL and Threads

This post talks about OpenSSL and threads. In particular, using OpenSSL in multi-threaded applications. It traces through the history, explains what was changed for the 1.1.0 release, and will hopefully provide some guidance to developers.

While none of the behaviors have really changed, and therefore none of this should be new information, the documentation has not been as clear as it could, or should, be. Therefore, some readers might be surprised by what’s in this post.

In short, OpenSSL has always, and only, supported the concept of locking an object and sometimes it locks its internal objects. Read on for more details.

In OpenSSL 1.0.2 (and earlier), applications had to provide their own integration with locking and threads, as documented in the threads.pod file. This page starts with the following unfortunate text:

OpenSSL can safely be used in multi-threaded applications provided that at least two callback functions are set, …

The problem is that the word safely was never actually defined. This led many developers to think that OpenSSL’s objects could safely be used by multiple threads at the same time. For example, using a single SSL object in two threads, one each for reading and writing.

The file crypto/th-lock.c in 1.0.2 had a “sample” implementation of the lock callbacks, for a number of platforms. The intent was that applications take that file, make it work on their platform and include it in their application. Many apparently managed to do that, although I wryly point out this comment from the Git history in early 2000:

At least, this compiles nicely on Linux using PTHREADS. I’ve done no other tests so far.

With 1.1.0, the major change was to move this from run-time to compile-time. If threads support is enabled (still the same misleading name), then the native threads package is used. On Windows, this means using the “critical section” API for locks, as can be seen in crypto/threads_win.c. On all other platforms, the pthreads API is used, crypto/threads_pthread.c. In cases where the native threads facility isn’t known, or if explicitly configured with no-threads then dummy functions are used, crypto/threads_none.c.

It is worth looking at at least one of these files. They are all very small, 120 to 170 lines, and can provide an overview of what OpenSSL actually needs and provides. The documentation is a bit better, but could still give the impression that concurrent use of objects in multiple threads is supported. We should fix that, soon.

So what is supported? One way to find out is to do

    git grep CRYPTO_THREAD_.*lock

If you do this, you’ll see that there is some locking around internal lookup tables for engines, error codes, and some RSA and X.509 lookup operations, and also about SSL sessions. Almost none of this behavior is documented, and it’s risky to plan on future behavior from just looking at the current code, but in general you can think that either tables of things (errors, sessions) or low-level internal details (RSA blinding, RSA Montgomeray optimizations) is safe. And most of the time, you probably won’t even think of that at all, anyway.

The other place where threads support comes in, isn’t necessarily about threads. It’s about maintaining reference counts on objects, and ensuring that an object is not free’d until the count hits zero. This is implemented using per-object locks, and the CRYPTO_atomic_add API. You can see some relatively simple examples of the up_ref API in test/crltest.c which also shows one of the common use-cases: adding an object to a container, but still “needing” that object for later, after the container is removed. Most datatypes have their own XXX_up_ref API.

Internally, OpenSSL is moving to do more things using native facilities. For example, the auto-init is done using “run-once” facilities, and the error state is maintained using thread-local storage. I’m not going to provide pointers to the source, because I don’t want to encourage people to rely on undocumented behavior. But, what you can rely on is that if you natively use your system threads facility, OpenSSL will work seamlessly with it. But you still can’t concurrently use most objects in multiple threads.

February 21, 2017 11:00 AM

Colin Percival

Cheating on a string theory exam

And now for something completely different: I've been enjoying FiveThirtyEight's "The Riddler" puzzles. A couple weeks ago I submitted a puzzle of my own; but I haven't heard back and it's too cute a puzzle to not share, so I've decided to post it here.

February 21, 2017 03:50 AM

February 20, 2017

Reliable message delivery with Mosquitto (MQTT)

I was looking for a message queue that could reliably handle messages in such a way that I was guaranteed never to miss one, even if the consumer is offline or crashes. Mosquitto (MQTT) comes very close to that goal. However, it wasn't directly obvious how to configure it to be as reliable as possible So this post describes how to use Mosquitto to ensure the most reliable delivery it can handle.

TL;DR: You can't

If you want to do reliable message handling with Mosquitto, the short answer is: You can't. For the long answer, read the rest of the article. Or if you're lazy and stubborn, read the "Limitations" section further down. ;-)

Anyway, let's get on with the show and see how close Mosquitto can get.

Quick overview of Mosquitto

Here's a quick schematic of Mosquitto components:

+----------+     +--------+     +----------+
| producer |---->| broker |---->| consumer |
+----------+     +--------+     +----------+

The producer sends messages to a topic on the broker. The broker maintains an internal state of topics and which consumers are interested in which topics. It also maintains a queue of messages which still need to be sent to each consumer. How the broker decided what / when to send to which consumer depends on settings such as the QoS (Quality of Service) and what kind of session the consumer is opening.

Producer and consumer settings

Here's a quick overview of settings that ensure the highest available quality of delivery of messages with Mosquitto. When creating a consumer or producer, ensure you set these settings properly:

  • quality-of-qervice must be 2.
  • The consumer must send a client_id.
  • clean_sesion on the consumer must be False.

These are the base requirements to ensure that each consumer will receive messages exactly once, even if they've been offline for a while. The quality-of-service setting of 2 ensures that the broker requires acknowledgement from the consumer that a message has been received properly. Only then does the broker update its internal state to advance the consumer to the next message in the queue. If the client crashes before acknowledging the message, it'll be resent the next time.

The client_id gives the broker a unique name under which to store session state information such as the last message the consumer has properly acknowledged. Without a client_id, the broker cannot do this.

The clean_session setting lets the consumer inform the broker about whether it wants its session state remembered. Without it, the broker assumes the broker assumes the consumer does not care about past messages and such. It will only receive any new messages that are produced after the consumer has connected to the broker.

Together these settings ensure that messages are reliably delivered from the producer to the broker and to the consumer, even if the consumer has been disconnected for a while or crashes while receiving the message.

Broker settings

The following settings are relevant configuration options on the broker. You can generally find these settings in/etc/mosquitto/mosquitto.conf.

  • The broker must have persistence set to True in the broker configuration.
  • You may want to set max_inflight_messages to 1 in the broker configuration to ensure correct ordering of messages.
  • Configure max_queued_messsages to the maximum number of messages to retain in a queue.
  • Tweak autosave_interval to how often you want the broker to write the in-memory database to disk.

The persistence setting informs the broker that you'd like session state and message queues written to disk. If the broker for some reason, the messages will (mostly) still be there.

You can ensure that messages are sent to consumers in the same order as they were sent to the broker by the producers by setting the max_inflight_messages setting to 1. This will probably severely limit the throughput speed of messages.

The max_queued_messsages determines how many unconfirmed messages should maximally be retained in queues. This should basically be the product of the maximum number of messages per second and the maximum time a consumer might be offline. Say we're processing 1 message per second and we want the consumer to be able to be offline for 2 hours (= 7200 seconds), then the max_queued_messsages setting should be 1 * 7200 = 7200.

The autosave_interval determines how often you want the broker to write the in-memory database to disk. I suspect that setting this to a very low level will cause severe Disk I/O activity.


Here's an example of a producer and consumer:

import paho.mqtt.client as paho
import time

client = paho.Client(protocol=paho.MQTTv31)
client.connect("localhost", 1883)
client.publish("mytesttopic", str("foo"), qos=2)
time.sleep(1)  # Give the client loop time to proess the message

import paho.mqtt.client as paho

def on_message(client, userdata, msg):
    print(msg.topic+" "+str(msg.qos)+" "+str(msg.payload))

client = paho.Client("testclient", clean_session=False, protocol=paho.MQTTv31)
client.on_message = on_message
client.connect("localhost", 1883)
client.subscribe("mytesttopic", qos=2)


There are a few pitfalls I ran into when using Mosquitto:

  • If the broker or one of the clients doesn't support the MQTTv32 protocol, things will fail silently. So I specify MQTTv31 manually.
  • The client loop needs some time to process the sending and receiving of messages. If you send a single message and exit your program right away, the loop doesn't have time to actually send the message.
  • The subscriber must have already run once before the broker will start keeping messages for it. Otherwise, the broker has no idea that a consumer with QoS=2 is interested in messages (and would have to keep messages for ever). So register your consumer once by just running it, before the producer runs.


Although the settings above make exchanging messages with Mosquitto more reliable, there are still some downsides:

  • Exchanging messages in this way is obviously slower than having no consistency checks in place.
  • Since the Mosquitto broker only writes the in-memory database to disk every X (where X is configurable) seconds, you may lose data if the broker crashes.
  • On the consumer side, it is the MQTT library that confirms the receipt of the message. However, as far as I can tell, there is no way to manually confirm the receipt of a message. So if your client crashes while handling a message, rather than while it is receiving a message, you may still lose the message. If you wish to handle this case, you can store the message on the client as soon as possible. This is, however, not much more reliable. The only other way is to implement some manual protocol via the exchange of messages where the original publisher retains a message and resends it unless its been acknowledged by the consumer.


In other words, as far as I can see, you cannot do reliable message handling with Mosquitto. If your broker crashes or your client crashes, Mosquitto will lose your messages. Other than that, if all you require is reliable delivery of messages to the client, you're good to go.

So what are the alternatives? At this point, I have to honest and say: I don't know yet. I'm personally looking for a lightweight solution, and it seems none of the lightweight Message Queues do reliable message handling (as opposed to reliable messagedelivery, which most do just fine).

When I find an answer, I'll let you know here.

by admin at February 20, 2017 07:10 PM


A bit of labor-law that applies even to high-paid techies

It's not as widely known as I hope, but there are a host of workplace protections that apply to non-union, salaried, overtime exempt workers. Not all of them are written into the legal code, and are, in fact, work-arounds. To explain what I'm talking about, read this:

This is a small sample-set, but it works to illustrate the point I'm about to make.

If you find yourself in the position of reporting a coworker to HR for harassing behavior, suddenly find your performance reviews solidly in needs improvement territory, and get fired later; there are workplace protections that will help you get through this, and make the life of your harasser less good.

To get there, here are a few facts and common practices that contribute to the firing, and what could happen afterwards:

  • Performance reviews are as much subjective as objective.
  • Tattling on a co-worker can make the perception of your work move from team player to troublemaker.
  • When the perception shifts like that, top-marks reviews suddenly become remediation-required reviews.
  • Due to US labor law, as amended by State laws, creating a hostile work environment due to sexism, racism, etc, is a criminal act.
  • In spite of that law, very few cases are seen in court, and even fewer reach a verdict.
  • At-will laws mean you can be fired without stated cause.
  • Everyone has a price for their silence.
  • Pathologic workplace cultures have no room for empathy.

Performance Reviews, and Career Improvement Plans

These are often used as the basis for a firing decision. Not all workplaces do them, but many do. It may be hidden in the OKR process, in 360-degree reviews, or another company-goal tracking system, but it's still there. Sometimes they're simple exceeds/meets/needs-improvement metrics, or 1 to 5 ranked metrics, and always have manager input on them.

All of them have some component of plays well with others as one of the tracked metrics. No one likes working with assholes, and this is how they track that. Unfortunately, tattling to mommy that Kenny was mean to her is often seen as not playing well with others.

Buying Your Silence

The severance process you go through after termination is there to buy your silence. Employers know full well there is a marketplace of opinion on good places to work, and if they can keep you from bagging on them on Glassdoor or social media, that's a win for them. You also get a month or two of paid healthcare as you look for someplace new. The method of doing this is called a non-disparagement clause in the severance agreement.

Laws are there to incentivise people to not get caught

If you have a good enough papertrail to plausibly bring suit against the company for one of the legally protected things like racism or sexism, there are strong incentives for them to settle this out of court. Everyone has a price, and most people have a price that doesn't include a written admission of guilt as a requirement. This is why there are so few actions brought against companies in court.

Pathological Empathy

Of the three Westrum Typology types of corporate communication styles (Pathological, Bureaucratic, Generative), it's the pathologic that fundamentally treats non-managers as objects. When you're an object, it doesn't matter if your fee-fees get hurt; what matters is that you're on-side and loyal. If you are seen to be disloyal, you will need to find a new master to swear your fealty to or you will be disposed of through the usual at-will / severance means.

Not all companies are pathologic. The studies I've seen says it's less than a quarter. That said, if the company is big enough you can quite definitely have portions of it that are pathologic while the rest are generative.

That's a lot of framing.

There are certain legal nightmares that companies have with regards to labor laws:

  • Having a now-ex employee bring a discrimination suit against them.
  • Having a class-action suit brought against a specific manager.
  • Having the Department of Labor bring suit against the company for systemic discrimination.

All of these actions are massively public and can't be silenced later. The fact of their filing is damnation enough.

This works for you, even though none of these are likely to come about for your specific case. You see, they're trying to avoid any of that ever happening. To avoid that happening they need to buy you off. Don't think that this is their way of protecting the bad man from any consequence. It's their attempt to, it's up to you to make it actually painful.

Once the third person has been fired and levered themselves into a $200K hush-money severance package, you can guarantee that The Powers That Be are going to sit the bad man down and explain to him that if he doesn't stop with the hands, they're going to have to Do Something; you're costing us a lot of money. One person doing that is just a whiner trying to extort money. Two people getting that is an abundance of whiners. Three people getting that begins to look like a pattern of behavior that is costing the company a lot of money.

This only works because the consequences of simply ignoring your whiny ass are now dire. Thanks, New Deal!

by SysAdmin1138 at February 20, 2017 05:15 PM

Toolsmith Release Advisory: Sysmon v6 for Securitay

Sysmon just keeps getting better.
I'm thrilled to mention that @markrussinovich and @mxatone have released Sysmon v6.
When I first discussed Sysmon v2 two years ago it offered users seven event types.
Oh, how it's grown in the last two years, now with 19 events, plus an error event.
From Mark's RSA presentation we see the current listing with the three new v6 events highlighted.
Sysmon Events

"This release of Sysmon, a background monitor that records activity to the event log for use in security incident detection and forensics, introduces an option that displays event schema, adds an event for Sysmon configuration changes, interprets and displays registry paths in their common format, and adds named pipe create and connection events."

Mark's presentation includes his basic event recommendations so as to run Sysmon optimally.
Basic Event Recommendations
Basic Event Recommendations (Cont)

I strongly suggest you deploy using these recommendations.
A great way to get started is to use a Sysmon configuration template. Again, as Mark discussed at RSA, consider @SwiftOnSecurity's sysmon-config-export.xml via Github. While there are a number of templates on Github, this one has "virtually every line commented and sections are marked with explanations, so it should also function as a tutorial for Sysmon and a guide to critical monitoring areas in Windows systems." Running Sysmon with it is as easy as:
sysmon.exe -accepteula -i sysmonconfig-export.xml

As a quick example of Sysmon capabilities and why you should always run it everywhere, consider the following driver installation scenario. While this is a non-malicious scenario that DFIR practitioners will appreciate, rather than the miscreants, the detection behavior resembles that which would result from kernel-based malware.
I fired up WinPMEM, the kernel mode driver for gaining access to physical memory included with Rekall, as follows:
Upon navigating to Applications and Services Logs/Microsoft/Windows/Sysmon/Operational in the Windows Event Viewer, I retrieved the expected event:

Event ID 6: Driver loaded
The best way to leave you to scramble off and deploy Sysmon broadly, are with Mark's Best Practices and Tips, again from his RSA presentation.

Go forth and deploy!
Cheers...until next time.

by Russ McRee ( at February 20, 2017 07:38 AM

Sarah Allen

the vision thing

Someone recently observed that I was motivated by a big vision and that some engineers are not, and that’s ok. In my experience, it’s not okay. I wish it was. It’s fun to sink into the code and get into the zone and not worry about the complicated real world. However, it’s not fun to work hard and create something over months or years and have it fail in the marketplace or, worse, never ship at all. I’m not talking about research projects where the primary outcome is learning something or publishing a paper. I’m talking about the kind of software I work on, which is intended to be used by other humans, whether they be developers or regular folks.

There are a few common problems I’ve seen happen when a team writes software without a clear and focused vision.

  1. Lots of bugs are discovered late. When engineers are focused on their own specific component or library, it often happens that another engineer will use that library in a slightly different way than was intended. This is especially true for internal libraries where there is little need for documentation since we can see each other’s source code; however, we don’t typically review all off the code our systems depend on. So each part of the code does what it is supposed to do on its own, but when we put them all together, slight miscommunications add up to things not working quite right, typically discovered late in the development cycle.
  2. The system doesn’t work well. When you are building software as a team, engineers work on different parts of the software. If one engineer is making a gaming engine, and another engineer is making presentation software, it rarely comes together to work well at the end. The gaming engine needs a high framerate and quick response to user input. The marketing presentation software need to display text at large font sizes. It’s likely one or the other will suffer, or more typically the software won’t actually work well for either use case.
  3. You delivered the features your manager asked for, but the product isn’t successful. Customers aren’t happy… or they aren’t buying in the first place. New users and people already using your product have different needs and slightly different use cases. It can be a delicate balance to walk. When developing a “feature” it can be easy to miss a step in between what you are working on and something that seems unrelated from the inside, but which people need to use together or in sequence. I call these features in between the features, which are very hard for engineers to see if they don’t have a complete picture of how people will use the product.
  4. Customers are happy and the product is cancelled. Companies have internal metrics by which they evaluate success. Someone on the business side has determined if we expect immediate revenue, or if this is going to be a long bet and expect slow revenue growth while we focus on strategic customer adoption, or if we need a large base of free users before we expect significant revenue. Sometimes it’s a product that supports larger business goals and drives adoption of a different product. If the engineers don’t know what those goals are, they may not get the important details right to make those goals happen.

So, you need to tell engineers the internal metrics, but that’s not enough. No one writes code that by itself increases 28-day active users. Somewhere in the product plans (or the minds of the executives) is a hypothesis that there are some new or improved use cases that will cause people to use the app more (or to sign up in the first place). A lot of teams do document use cases or user journeys and share them with the engineers, which is important, but not sufficient.

Engineers need to know the use cases that everyone believes will be required to make the product successful; however, usually software engineers have to make all sorts of decisions outside of the required use cases for the next release. The software will work better if all of these little decisions are aligned somehow, and the easiest way to do this (in fact the only way that I’ve found to do this), is for the developers to have a shared vision of what this software wants to be — not just in its next release, but in the future. This vision can be somewhat arbitrary for it to work (which means it is totally fine if it changes), the key point is that it needs to be shared, and when it changes, the change and reasons for the change need to be communicated effectively. A shared vision not only aligns all of the little decisions that developers make independently, but also makes all of the design discussions more efficient. We can focus on how we are building a thing, because we already know what we are building and why.

To create a shared vision, we need to answer: who is the target customer? what are the key problems we are solving? what will our customers be able to do with the initial release and what do we hope they will be able to do in the future? Who are the customers who might not be satisfied with our product and that would be ok?

Good engineers will build something that works for a wider audience. Often the software will enable those people to do many more things than the minimum requirements, but focus is key to creating robust software that works well.

I’ve been talking about engineers, since that’s my focus these days and because there remains a persistent myth that engineers don’t need to be distracted with all this business stuff. All of these questions and answers are even more important for designers, technical writers, support, developer relations, support, sales, business development and everyone involved in the making of and communication about the product.

I’m not motivated by a big vision. I’m motivated by impact — not potential impact, real impact. I need to believe that my day to day work is going to add up to something that solves real problems for real people in the real world. I’ve discovered that to reduce risk in software projects, every member of my team must be able to see a through-line from their day-to-day work to shipping software that causes a large number of humans to be empowered to do something new or better, enabling them to solve real-world problems. To do that, I need to convey a vision that lets the individual team members innovate, coming up with creative solutions in their domain that add up to a system that actually works.

The post the vision thing appeared first on the evolving ultrasaurus.

by sarah at February 20, 2017 04:12 AM

February 18, 2017

Steve Kemp's Blog

Apologies for the blog-churn.

I've been tweaking my blog a little over the past few days, getting ready for a new release of the chronicle blog compiler (github).

During the course of that I rewrote all the posts to have 100% lower-case file-paths. Redirection-pages have been auto-generated for each page which was previously mixed-case, but unfortunately that will have meant that the RSS feed updated unnecessarily:

  • If it used to contain:
  • It would have been updated to contain

That triggered a lot of spamming, as the URLs would have shown up as being new/unread/distinct.

February 18, 2017 10:00 PM

February 17, 2017

Ben's Practical Admin Blog

Microsoft Ignite Australia 2017 – Final Day


Well all good things must come to an end and so it passes that this is the last day MS Ignite for another year . Definitely a more subdued feeling amongst the attendees (maybe due to a big night last night!) and many delegates are already flying home, not choosing to be here for the last day of events.


First up was the Azure Stack & Hybrid Deployment session. I wish I could say I enjoyed this session, but for a level 300 session there really wasn’t’ much there I couldn’t have picked up on the fly if using the technology myself. This was a little disappointing considering the rushing around I did to get there.

img_0213After an early morning tea was another of Ben Armstrong’s sessions on Windows Containers. This was a fantastic Level 400 deep dive into how containers actually work and some the architectural decisions made to make containers so fast.

Some of the really impressive things talked about was working on performance of things such as PowerShell to speed up its load times by up to 800%, and using memory cloning with Hyper-V containers to get smart about loading the same code over and over again.

This was truly fascinating session and has only continued to increase my desire to see how to use Windows containers in my workplace.

img_0221The last “Technical Session” for me was the fantastic Orin Thomas and discussing the Terrible Habits of Cloud and Server Administrators.

This session has a lot of humour, a lot of head nodding and a lot of home truths about all things Server and Cloud administrators do.

Orin then went on to talk about how to fix some of these things, some of them common sense, and others with technical solutions such as Just Enough Administration suing privileged access management and secure domains. Having been to most of his sessions throughout Ignite, I was starting to hear the same anecdotes for the 2nd or 3rd time.

One last lunch and it was amusing to see around half the exhibitors had already packed up and left, leaving empty stands.

Being honest, I didn’t hang around for the closing keynote though judging by peoples reactions I really wish I had.

And that was the end of Ignite for another year!


by Ben at February 17, 2017 05:00 AM

February 16, 2017

Ben's Practical Admin Blog

Microsoft Ignite Australia 2017 – Day 3

After the excitement of the previous evening, the day kicked of a little later than the norm for me with the Elastic


One of the things I wasn’t especially aware of was the X-Pack by Elastic and that there is a free, basic version available of that after a 30 day trial. Elastic Cloud may also be an option if we do not want to have to run the underlying infrastructure but just be a consumer of the platform. Shortly after that the session became more of a deep dive into Azure provisioning Elastic with the ARM config files, which was a bit out of my depth.

Microsoft Azure Stack was next up and was certainly something that has recently caught my attention.


It was really interesting to hear that it’s not simply a case of downloading an appliance and deploying in your existing infrastructure to create an on-premises Azure presence. This is an integrated solution that requires the purchase of a particular infrastructure to create a “unit”, which you then have multiple units as necessary. It does make sense for this to occur given how closely Microsoft want to hardware to integrate with the software.

Also of great interest was the cost for the Azure  Stack software is based on a consumption model – after you pay for the hardware, the licensing fees are tied to the amount of resources you use – you don’t use it at all then you don’t have to pay anything for The Azure Stack (you do most certainly have to pay for the hardware hosting it though!).

I think this is definitely the technology to watch right now if you are in an enterprise that likes the concept of Azure platform, but have concerns about data being stored in a public cloud.

img_0179After lunch was a session that I was rather happily surprised as being quite relevant and interesting to some of the things I am doing now.

In particular, the cloud identity session introduced me to Azure AD pass-through authentication that is currently in preview.

Working somewhere that has not implemented ADFS, but where data sovereignty is a delicate topic, AD Pass-through provides a rather great way to provide authentication in the cloud without needing ADFS – but rather some lightweight connector proxies that communicate over port 443. I can’t wait to investigate this further once getting back from Ignite.

The last session for the day is Orin Thomas expanding on yesterday’s session about configuring and deploying Just Enough and Just-In-Time Administration

img_0182Just Enough and Just in Time administration is based on the premise that you should consider your network already compromised by attackers and that you now limit the ability for damage to occur by ensuring all administrative privileges are time limited and must be requested using PAM modules. This provides a means of knowing who has administrative access and when. This will probably be seen as lots of additional bureaucracy  for many “old school” admins, but is actually the way many modern infrastructure services are beginning to work, particularly in the cloud.

Just Enough Administration ensures that you only get the administrative access you need to do you job. An example of this is that traditionally PowerShell provides much more access to administrative functions than is usually required by some application admins, so it is possible to make a role to only allow access to the cmdlets needed to administer the service. This is a great idea where a server may hold multiple services that are administrated by different groups of people.

Of course, being the last night of MS Ignite, it can only mean one thing: After Dark!

Lots of food, friends and activities to keep everyone happy.

Ice Cream! Xbox on the Big Screen Sumo Suits Gladiator Laser Skirmish

Tomorrow it will be all over. I’ve had a great time this year, and paced myself much better than my 2012 experiences, and am looking forward to the final sessions!

by Ben at February 16, 2017 11:48 AM

February 15, 2017

Ben's Practical Admin Blog

Microsoft Ignite Australia 2017 – Day 2

Waking up at 5am in the morning is a near impossible feat when at home, yet seems to be no problem here. Might have something to do with the sunrise 🙂


Early start today with the 8.15am session on containers with Ben Armstrong. Containers captured my attention when announced in Server 2016 as a great way to virtualise applications.

img_0158Ben’s presentation was a fantastic introduction to containers made all the better by practical demonstrations of how to use them.

Personally, I think they are a great thing, but I am struggling to find a good use-case for them at work in a way that would see them readily adopted.

I did stop in briefly at the Windows Server 2016 security session, which was standing room only. There are some very cool technologies being built into Windows Server these days including Just Enough Administration that allows you to limit access to powershell cmdlets available to people in a given role. And excampkle if this would be to ensure that the dns admins only had access to the dns cmdlets.

Unfortunately I had to step out not too long after the session started, and given the standing room only sitiuation could not get back in later, so went on to wander the expo floor.

Here I managed to have a chat with Mark Warkolm from Elastic, who I have had dealings with before and is an all round great guy, along with his colleague Russ. Discussed data curation in elastic and how to best use time based indices.

I also met Paul Brauman from Polycom whom we share a mutual friend and had a bit of a chat, with me taking to opportunity to give a light dig at the performance of Polycom Gear 🙂 All taken in good humour though!


After lunch I opted to watch the short Hack@Ignite talk by Kylie Hunt on 5 ninja tricks to making the workplace suck less. This was an entertaining talk and full of humour going through how a happier workplace makes for greater productivity and what things you can do to help make the workplace happier. I think much of it I unconsciously knew but needed Kylie to actually say it for it to become the aha! moment for me.

Another break between sessions before heading off to the Supercharged IoT end-to-end Solution on the Azure IoT Platform.

I would love to go on and tell you how the last 2 sessions of MSIgnite went today, but it was about this time that my phone lit up with a heap of messages advising me of a bushfire near my home that is currently out of control!

I guess some IoT Devices would be handy about now to monitor the weather and particulate situation. For now I am just trying to get some eyes on the ground…

**update** Eyes on ground says I should be fine…


by Ben at February 15, 2017 07:10 AM

February 14, 2017

Everything Sysadmin

February NYCDevOps meetup: Tiny Talks CfP

The NYC DevOps meetup is trying something new in February: Tiny Talks. Tiny talks are short, 5-10 minute talks usually presented by members. They can be basically any topic related to DevOps. The idea is to encourage local members to present at meetings.

We'll be doing this for our first time at the February 21, 2017 NYCDevOps meetup. You are encouraged to sign up here (though we won't be strict if you show up with a presentation.)

From the website:

Tiny talks are 5-10 minute talks on any DevOps-related topic. We want to encourage everyone to submit a proposal. NYCDevOps is a community that shares. Tiny Talks don't have to be polished or fancy. These are short talks, making it easy for beginners and experienced people alike. We hope to have 3-4 presentations.

If you think you can't give a talk, here's a formula to start you off: (1) Use a title like: "In the last 6 months something I learned about X was Y". (2) Create 1-3 slides about what you learned. (3) The talk only has to be a few minutes long... Q&A will fill the rest of the time. For example, "In the last 6 months I learned that TerraForm breaks in [situation], here's why."

We'll have time for 2-3 Tiny Talks, so speaking slots will be first come first serve.

Hope to see you there!

Remember to RSVP!

by Tom Limoncelli at February 14, 2017 08:00 PM

Evaggelos Balaskas

LDAP User Authentication on CentOS 7


You need to already have a LDAP instance in your infrastructure that you can reach from your test linux machine. Your ldap has an organization unit for people and one for groups.

Ldap server conf

It is always a good thing to write your notes/settings beforehand:

Ldap Server:
Domain Component: dc=example,dc=org

People base: ou=people,dc=example,dc=org
Group base: ou=Groups,dc=example,dc=org

bind user: userpam
bind pass: 1234567890


On your centos 7 machine, you have to install two packages:

# yum -y install nss-pam-ldapd

  Installing : nscd-2.17-157.el7_3.1.x86_64
  Installing : nss-pam-ldapd-0.8.13-8.el7.x86_64

local LDAP name service daemon

Edit the /etc/nslcd.conf file accordingly to your ldap setup.

# grep -Ev '#|^$' /etc/nslcd.conf
uid nslcd
gid ldap

uri ldap://
base ou=people,dc=example,dc=org

ssl no
tls_cacertdir /etc/openldap/cacerts

This is the most basic configuration file, without any TLS (encryption) support, but for our test purposes is ok.

restart nslcd

Every time you change something to nslcd.conf file, you need to restart the service:

# systemctl restart nslcd

Name Service Switch

By default the Name Service Switch have ldap support for the below pam services:

# grep ldap /etc/nsswitch.conf

passwd:     files sss ldap
shadow:     files sss ldap
group:      files sss ldap
netgroup:   files sss ldap
automount:  files ldap

if not, just add it yourself. Just remember that the order is from left to right, that means your centos machine will first try to look in your local files, then to your System Security Services Daemon and finally to your ldap URI !


In this first step, the only way to test that your linux machine can talk to your linux server is via getent looking up on the passwd service:

# getent passwd | grep ebal

ebal:x:374:374:Evaggelos Balaskas:/home/ebal:/bin/bash

Ldap Bind Password

The above example is for anonymous bind against your ldap server. That means that secrets (as the password of the user) can not be viewed (actually tested it on the encrypted hash) as for that you need to bind to your ldap server with your credentials.

# egrep -v '^$|#' /etc/nslcd.conf

uid nslcd
gid ldap
uri ldap://
base ou=people,dc=example,dc=org

binddn cn=userpam,dc=example,dc=org
bindpw 1234567890

ssl no
tls_cacertdir /etc/openldap/cacerts

restart nslcd

 # systemctl restart nslcd


Now it’s time for your first ssh login:

~> ssh testvm
ebal@testvm's password: 

Last login: Mon Feb 13 22:50:12 2017
/usr/bin/id: cannot find name for group ID 374

~>  id
uid=374(ebal) gid=374 groups=374

You can login without problem, but there is a warning for your group id.

Ldap Group Configuration

So, we need to add support for our group base on the nslcd configuration file:

# egrep -v '^$|#' /etc/nslcd.conf

uid nslcd
gid ldap
uri ldap://
base ou=people,dc=example,dc=org
binddn cn=userpam,dc=example,dc=org
bindpw 1234567890

base group ou=Groups,dc=example,dc=org

ssl no
tls_cacertdir /etc/openldap/cacerts

restart nslcd

# systemctl restart nslcd


We first test it against getent using the group service:

# getent group | grep 374

and after that, we can ssh again to our linux machine:

~> ssh testvm
ebal@testvm's password:
Last login: Mon Feb 13 23:14:42 2017 from testserver

~> id
uid=374(ebal) gid=374(ebal) groups=374(ebal)

Now it shows the group name without a problem.

February 14, 2017 12:19 AM

February 13, 2017


Does Reliable Real Time Detection Demand Prevention?

Chris Sanders started a poll on Twitter asking "Would you rather get a real-time alert with partial context immediately, or a full context alert delayed by 30 mins?" I answered by saying I would prefer full context delayed by 30 minutes. I also replied with the text at left, from my first book The Tao of Network Security Monitoring (2004). It's titled "Real Time Isn't Always the Best Time."

Dustin Webber then asked "if you have [indicators of compromise] IOC that merit 'real-time' notification then you should be in the business of prevention. Right?"

Long ago I decided to not have extended conversations over Twitter, as well as to not try to compress complex thoughts into 140 characters -- hence this post!

There is a difference, in my mind, between high-fidelity matching (using the vernacular from my newest book, The Practice of Network Security Monitoring, 50% off now with code RSAREADING) and prevention.

To Dustin's point, I agree that if it is possible to generate a match (or "alert," etc.) with 100% accuracy (or possibly near 100%, depending on the severity of the problematic event), i.e., with no chance or almost no chance of a false positive, then it is certainly worth seeking a preventive action for that problematic event. To use a phrase from the last decade, "if you can detect it, why can't you prevent it?"

However, there are likely cases where zero- or low-false positive events do not have corresponding preventive actions. Two come to mind.

First, although you can reliably detect a problem, you may not be able to do anything about it. The security team may lack the authority, or technical capability, to implement a preventive action.

Second, although you can reliably detect a problem, you may not want to do anything about it. The security team may desire to instead watch an intruder until such time that containment or incident mitigation is required.

This, then, is my answer to Dustin's question!

by Richard Bejtlich ( at February 13, 2017 11:22 PM

Everything Sysadmin

Tom speaking at NYCDevOps meetup, Feb 21, 2017

I'll be the speaker at the February meeting of NYCDevOps. I'll be presenting my talk "Stealing the Best Ideas from DevOps: A Guide for Sysadmins without Developers". If you didn't see me give this talk at Usenix LISA, you can catch it here.

Please RSVP. Seating is limited.

by Tom Limoncelli at February 13, 2017 06:00 PM


Project Bylaws

Last October, the OpenSSL Project team had a face to face meeting. We talked about many topics but one of them was that, in recent years, we have seen much more involvement from the community and that we would like to encourage that further. For example, there are a number of people in the community who we know and trust. We would like those people to get involved more and make it easier for them to contribute. We decided to introduce the concept of a “committer” (borrowed from the Apache concept): someone who has the ability to commit code to our source code repository but without necessarily having to become a full team member. This might be seen as a stepping-stone for someone who aspires to full team membership, or simply as an easier way of contributing for those that don’t. Those people could help with our review process (i.e., their reviews would count towards approval) - which might help us keep on top of the github issues and pull request queues.

We also wanted to be more transparent about how we work as a team, and how decisions are made. At the moment we have a set of rules for this but they are not published anywhere.

These two ideas have come together and as a result we have set about rewriting and updating our project rules (drawing inspiration from the Apache Software Foundation model), so that they can be published and, at the same time, introducing the committer concept. I am very pleased to announce the publication of our new “Project Bylaws”. You can read them here. Incidentally these project bylaws should not be confused with the OpenSSL Software Foundation (OSF) Bylaws which have previously been published here. The OSF Bylaws are a legal document which describe how that legal entity is operated.

Over time, we expect to be inviting various trusted members of the community to become committers. We also expect to be publishing a committer policy document in the near future to give further guidance to committers on their role and what they are expected to do.

February 13, 2017 12:00 PM

February 12, 2017


Choria Update

Recently at Config Management Camp I’ve had many discussions about Orchestration, Playbooks and Choria, I thought it’s time for another update on it’s status.

I am nearing version 1.0.0, there are a few things to deal with but it’s getting close. Foremost I wanted to get the project it’s own space on all the various locations like GitHub, Forge, etc.

Inevitably this means getting a logo, it’s been a bit of a slog but after working through loads of feedback on Twitter and offers for assistance from various companies I decided to go to a private designer called Isaac Durazo and the outcome can be seen below:


The process of getting the logo was quite interesting and I am really pleased with the outcome, I’ll blog about that separately.

Other than the logo the project now has it’s own GitHub organisation at and I have moved all the forge modules to it’s own space as well

There are various other places the logo show up like in the Slack notifications and so forth.

On the project front there’s a few improvements:

  • There is now a registration plugin that records a bunch of internal stats on disk, the aim is for them to be read by Collectd and Sensu
  • A new Auditing plugin that emits JSON structured data
  • Several new Data Stores for Playbooks – files, environment.
  • Bug fixes on Windows
  • All the modules, plugins etc have moved to the Choria Forge and GitHub
  • Quite extensive documentation site updates including branding with the logo and logo colors.

There is now very few things left to do to get 1.0.0 out but I guess another release or two will be done before then.

So from now to update to coming versions you need to use the choria/mcollective_choria module which will pull in all it’s dependencies from the Choria project rather than my own Forge.

Still no progress on moving the actual MCollective project forward but I’ve discussed a way to deal with forking the various projects in a way that seems to work for what I want to achieve. In reality I’ll only have time to do that in a couple of months so hopefully something positive will happen in the mean time.

Head over to to take a look.

by R.I. Pienaar at February 12, 2017 06:24 PM


Guest Post: Bamm Visscher on Detection

Yesterday my friend Bamm Visscher published a series of Tweets on detection. I thought readers might like to digest it as a lightly edited blog post. Here, then, is the first ever (as far as I can remember) guest post on TaoSecurity Blog. Enjoy.

When you receive new [threat] intel and apply it in your detection environment, keep in mind all three analysis opportunities: RealTime, Batch, and Hunting.

If your initial intelligence analysis produces high context and quality details, it's a ripe candidate for RealTime detection.

If analysts can quickly and accurately process events generated by the [RealTime] signature, it's a good sign the indicator should be part of RealTime detection. If an analyst struggles to determine if a [RealTime alert] has detected malicious activity, it's likely NOT appropriate for RealTime detection.

If [the threat] intelligence contains limited context and/or details, try leveraging Batch Analysis with scheduled data reports as a better detection technique. Use Batch Analysis to develop better context (both positive and negative hits) to identify better signatures for RealTime detection.

Hunting is the soft science of detection, and best done with a team of diverse skills. Intelligence, content development, and detection should all work together. Don't fear getting skunked on your hunting trips. Keep investing time. The rewards are accumulative. Be sure to pass Hunting rewards into Batch Analysis and RealTime detection operations in the form of improved context.

The biggest mistake organizations make is not placing emphasis outside of RealTime detection, and "shoe-horning" [threat] intelligence into RealTime operations. So called "Atomic Indicators" tend to be the biggest violator of shoe-horning. Atomic indicators are easy to script into signature driven detection devices, but leave an analyst wondering what he is looking at and for.

Do not underestimate the NEGATIVE impact of GOOD [threat] intelligence inappropriately placed into RealTime operations! Mountains of indiscernible events will lead to analyst fatigue and increase the risk of good analyst missing a real incident.

by Richard Bejtlich ( at February 12, 2017 12:59 PM

February 09, 2017

Evaggelos Balaskas

free-as-in-freedom Google Apps on Fairphone Android

Fairphone FP1U

I have this phone for more than 3,5 years


running on Android v4.2 !


I dont have them, I dont even have a google account.
Although fairphone comes with an extra GApps package zip, I haven’t installed them either.

Fake GApps

Unfortunately there are some android apps that need GApps to run. So, without any further ado, here are my notes on how to add support for alternative open sources.


Device may brick ! Do a full backup and keep it somewhere safe.

Location Service

We start with the easy one, how to add support for location service.
In Fdroid we can find the UnifiedNlp suite of packages that work as a Location provider middleware. On my device, I installed the below apps:



Screenshots from my fairphone, after a few reboots


Unified github

Just to make things perfectly clear, you can verify fdroid’s packages against these:



Google Play Store

So notes are referring to Phonesky … I didnt follow them !!!

Instead of phonesky, I installed BlankStore !

Here is the github repo: BlankStore v0.7.5



Now are going to a more interesting part. We need to install the μG to our phone.

microG F-Droid repo

It’s really easy, just add the fdroid repository and install the below two apps:
  • microG Services Core
  • microG Services Framework Proxy


microG Services Core

Now the difficult part.

Opening microG self-checked, we are going to see that spoof signature is not passing the test.


Now the most difficult and dangerous thing you will need to do with your phone.

We need to install Xposed Installer

For Fairphone FP1U we need:

Version name:  2.7 experimental
Download: (770.28 KB)
MD5: 36570c6fac687ffe08107e6a72bd3da7

after that, we have to install the below xposed modules:

  • Xposed Installer
  • FakeGApps
  • XposedGmsCoreUnifiedNlp
  • XposedUnifiedNlp

download them and reboot your phone. If you think it may be helpful, cross your fingers.

This is a good time to remind you that you need to have a fresh backup



Now let’s reboot our mobile phone again and try to open μG once more:



If everything seems like the above pictures, then you are done !

Errata 20170211

For android users with v4.2 on it, there are a few errata!

Location Services

You need the legacy network location package from UnifiedNlp


μG - microG

The latest online version -somehow- doesnt work with my phone perfectly.
You can use the build from BRNmod an alternative CyanogenMod

play-services-core-debug.apk                       27-Jan-2017 11:22             4188872

February 09, 2017 07:03 PM


Bejtlich Books Explained

A reader asked me to explain the differences between two of my books. I decided to write a public response.

If you visit the TaoSecurity Books page, you will see two different types of books. The first type involves books which list me as author or co-author. The second involves books to which I have contributed a chapter, section, or foreword.

This post will only discuss books which list me as author or co-author.

In July 2004 I published The Tao of Network Security Monitoring: Beyond Intrusion Detection. This book was the result of everything I had learned since 1997-98 regarding detecting and responding to intruders, primarily using network-centric means. It is the most complete examination of NSM philosophy available. I am particularly happy with the NSM history appendix. It cites and summarizes influential computer security papers over the four decade history of NSM to that point.

The main problem with the Tao is that certain details of specific software versions are very outdated. Established software like Tcpdump, Argus, and Sguil function much the same way, and the core NSM data types remain timeless. You would not be able to use the Bro chapter with modern Bro versions, for example. Still, I recommend anyone serious about NSM read the Tao.

The introduction describes the Tao using these words:

Part I offers an introduction to Network Security Monitoring, an operational framework for the collection, analysis, and escalation of indications and warnings (I&W) to detect and respond to intrusions.   Part I begins with an analysis of the terms and theory held by NSM practitioners.  The first chapter discusses the security process and defines words like security, risk, and threat.  It also makes assumptions about the intruder and his prey that set the stage for NSM operations.  The second chapter addresses NSM directly, explaining why NSM is not implemented by modern NIDS' alone.  The third chapter focuses on deployment considerations, such as how to access traffic using hubs, taps, SPAN ports, or inline devices.  

Part II begins an exploration of the NSM “product, process, people” triad.  Chapter 4 is a case study called the “reference intrusion model.”  This is an incident explained from the point of view of an omniscient observer.  During this intrusion, the victim collected full content data in two locations.  We will use those two trace files while explaining the tools discussed in Part II.  Following the reference intrusion model, I devote chapters to each of the four types of data which must be collected to perform network security monitoring – full content, session, statistical, and alert data.  Each chapter describes open source tools tested on the FreeBSD operating system and available on other UNIX derivatives.  Part II also includes a look at tools to manipulate and modify traffic.  Featured in Part II are little-discussed NIDS' like Bro and Prelude, and the first true open source NSM suite, Sguil.

Part III continues the NSM triad by discussing processes.  If analysts don’t know how to handle events, they’re likely to ignore them.  I provide best practices in one chapter, and follow with a second chapter explicitly for technical managers.  That material explains how to conduct emergency NSM in an incident response scenario, how to evaluate monitoring vendors, and how to deploy a NSM architecture.

Part IV is intended for analysts and their supervisors.  Entry level and intermediate analysts frequently wonder how to move to the next level of their profession.  I offer some guidance for the five topics with which a security professional should be proficient: weapons and tactics, telecommunications, system administration, scripting and programming, and management and policy.  The next three chapters offer case studies, showing analysts how to apply NSM principles to intrusions and related scenarios.

Part V is the offensive counterpart to the defensive aspects of Parts II, III, and IV.  I discuss how to attack products, processes, and people.  The first chapter examines tools to generate arbitrary packets, manipulate traffic, conduct reconnaissance, and exploit flaws inn Cisco, Solaris, and Microsoft targets.  In a second chapter I rely on my experience performing detection and response to show how intruders attack the mindset and procedures upon which analysts rely.

An epilogue on the future of NSM follows Part V.  The appendices feature several TCP/IP protocol header charts and explanations.   I also wrote an intellectual history of network security, with abstracts of some of the most important papers written during the last twenty-five years.  Please take the time to at least skim this appendix,  You'll see that many of the “revolutionary ideas” heralded in the press were in some cases proposed decades ago.

The Tao lists as 832 pages. I planned to write 10 more chapters, but my publisher and I realized that we needed to get the Tao out the door. ("Real artists ship.") I wanted to address ways to watch traffic leaving the enterprise in order to identify intruders, rather than concentrating on inbound traffic, as was popular in the 1990s and 2000s. Therefore, I wrote Extrusion Detection: Security Monitoring for Internal Intrusions.

I've called the Tao "the Constitution" and Extrusion "the Bill of Rights." These two books were written in 2004-2005, so they are tightly coupled in terms of language and methodology. Because Extrusion is tied more closely with data types, and less with specific software, I think it has aged better in this respect.

The introduction describes Extrusion using these words:

Part I mixes theory with architectural considerations.  Chapter 1 is a recap of the major theories, tools, and techniques from The Tao.  It is important for readers to understand that NSM has a specific technical meaning and that NSM is not the same process as intrusion detection.  Chapter 2 describes the architectural requirements for designing a network best suited to control, detect, and respond to intrusions.  Because this chapter is not written with specific tools in mind, security architects can implement their desired solutions regardless of the remainder of the book.  Chapter 3 explains the theory of extrusion detection and sets the stage for the remainder of the book.  Chapter 4 describes how to gain visibility to internal traffic.  Part I concludes with Chapter 5, original material by Ken Meyers explaining how internal network design can enhance the control and detection of internal threats.

Part II is aimed at security analysts and operators; it is traffic-oriented and requires basic understanding of TCP/IP and packet analysis.  Chapter 6 offers a method of dissecting session and full content data to unearth unauthorized activity.  Chapter 7 offers guidance on responding to intrusions, from a network-centric perspective.  Chapter 8 concludes part III by demonstrating principles of network forensics.

Part III collects case studies of interest to all types of security professionals.  Chapter 9 applies the lessons of Chapter 6 and explains how an internal bot net was discovered using Traffic Threat Assessment.  Chapter 10 features analysis of IRC bot nets, contributed by LURHQ analyst Michael Heiser. 

An epilogue points to future developments.  The first appendix, Appendix A, describes how to install Argus and NetFlow collection tools to capture session data.  Appendix B explains how to install a minimal Snort deployment in an emergency.  Appendix C, by Tenable Network Security founder Ron Gula, examines the variety of host and vulnerability enumeration techniques available in commercial and open source tools.  The book concludes with Appendix D, where Red Cliff Consulting expert Rohyt Belani offers guidance on internal host enumeration using open source tools.

At the same time I was writing Tao and Extrusion, I was collaborating with my friends and colleagues Keith Jones and Curtis Rose on a third book, Real Digital Forensics: Computer Security and Incident Response. This was a ground-breaking effort, published in October 2005. What made this book so interesting is that Keith, Curtis and I created workstations running live software, compromised each one, and then provided forensic evidence for readers on a companion DVD. 

This had never been done in book form, and after surviving the process we understood why! The legal issues alone were enough to almost make us abandon the effort. Microsoft did not want us to "distribute" a forensic image of a Windows system, so we had to zero out key Windows binaries to satisfy their lawyers. 

The primary weakness of the book in 2017 is that operating systems have evolved, and many more forensics books have been written. It continues to be an interesting exercise to examine the forensic practices advocated by the book to see how they apply to more modern situations.

This review of the book includes a summary of the contents:

• Live incident response (collecting and analyzing volatile and nonvolatile data; 72 pp.)
• Collecting and analyzing network-based data (live network surveillance and data analysis; 87 pp.)
• Forensic duplication of various devices using commercial and open source tools (43 pp.)
• Basic media analysis (deleted data recovery, metadata, hash analysis, “carving”/signature analysis, keyword searching, web browsing history, email, and registry analyses; 96 pp.)
• Unknown tool/binary analysis (180 pp.)
• Creating the “ultimate response CD” (response toolkit creation; 31 pp.)
• Mobile device and removable media forensics (79 pp.)
• On-line-based forensics (tracing emails and domain name ownership; 30 pp.)
• Introduction to Perl scripting (12 pp.)

After those three titles, I was done with writing for a while. However, in 2012 I taught a class for Black Hat in Abu Dhabi. I realized many of the students lacked the fundamental understanding of how networks operated and how network security monitoring could help them detect and respond to intrusions. I decided to write a book that would explain NSM from the ground up. While I assumed the reader would have familiarity with computing and some security concepts, I did not try to write the book for existing security experts.

The result was The Practice of Network Security Monitoring: Understanding Incident Detection and Response. If you are new to NSM, this is the first book you should buy and read. It is a very popular title and it distills my philosophy and practice into the most digestible form, in 376 pages.

The main drawback of the book is the integration of Security Onion coverage. SO is a wonderful open source suite, partly because it is kept so current. That makes it difficult for a print book to track changes in the software installation and configuration options. While you can still use PNSM to install and use SO, you are better off relying on Doug Burks' excellent online documentation. 

PNSM is an awesome resource for learning how to use SO and other tools to detect and respond to intrusions. I am particularly pleased with chapter 9, on NSM operations. It is a joke that I often tell people to "read chapter 9" when anyone asks me about CIRTs.

The introduction describes PNSM using these words:

Part I, “Getting Started,” introduces NSM and how to think about sensor placement.

• Chapter 1, “Network Security Monitoring Rationale,” explains why NSM matters, to help you gain the support needed to deploy NSM in your environment.
• Chapter 2, “Collecting Network Traffic: Access, Storage, and Management,”addresses the challenges and solutions surrounding physical access to network traffic.

Part II, “Security Onion Deployment,” focuses on installing SO on hardware and configuring SO effectively.

• Chapter 3, “Stand-alone NSM Deployment and Installation,” introduces SO and explains how to install the software on spare hardware to gain initial NSM capability at low or no cost.
• Chapter 4, “Distributed Deployment,” extends Chapter 3 to describe how to install a dispersed SO system.
• Chapter 5, “SO Platform Housekeeping,” discusses maintenance activities for keeping your SO installation running smoothly. 

Part III, “Tools,” describes key software shipped with SO and how to use these applications.

• Chapter 6, “Command Line Packet Analysis Tools,” explains the key features of Tcpdump, Tshark, Dumpcap, and Argus in SO.
• Chapter 7, “Graphical Packet Analysis Tools,” adds GUI-based software to the mix, describing Wireshark, Xplico, and NetworkMiner.
• Chapter 8, “NSM Consoles,” shows how NSM suites, like Sguil, Squert, Snorby, and ELSA, enable detection and response workflows.

Part IV, “NSM in Action,” discusses how to use NSM processes and data to detect and respond to intrusions.

• Chapter 9, “NSM Operations,” shares my experience building and leading a global computer incident response team (CIRT).
• Chapter 10, “Server-side Compromise,” is the first NSM case study, wherein you’ll learn how to apply NSM principles to identify and validate the compromise of an Internet-facing application.
• Chapter 11, “Client-side Compromise,” is the second NSM case study, offering an example of a user being victimized by a client-side attack.
• Chapter 12, “Extending SO,” concludes the main text with coverage of tools and techniques to expand SO’s capabilities.
• Chapter 13, “Proxies and Checksums,” concludes the main text by addressing two challenges to conducting NSM.

The Conclusion offers a few thoughts on the future of NSM, especially with respect to cloud environments. 

The Appendix, “SO Scripts and Configuration,” includes information from SO developer Doug Burks on core SO configuration files and control scripts.

I hope this post helps explain the different books I've written, as well as their applicability to modern security scenarios.

by Richard Bejtlich ( at February 09, 2017 03:36 PM

Vincent Bernat

Integration of a Go service with systemd

Unlike other programming languages, Go’s runtime doesn’t provide a way to reliably daemonize a service. A system daemon has to supply this functionality. Most distributions ship systemd which would fit the bill. A correct integration with systemd is quite straightforward. There are two interesting aspects: readiness & liveness.

As an example, we will daemonize this service whose goal is to answer requests with nifty 404 errors:

package main

import (

func main() {
    l, err := net.Listen("tcp", ":8081")
    if err != nil {
        log.Panicf("cannot listen: %s", err)
    http.Serve(l, nil)

You can build it with go build 404.go.

Here is the service file, 404.service1:

Description=404 micro-service




The classic way for an Unix daemon to signal its readiness is to daemonize. Technically, this is done by calling fork(2) twice (which also serves other intents). This is a very common task and the BSD systems, as well as some other C libraries, supply a daemon(3) function for this purpose. Services are expected to daemonize only when they are ready (after reading configuration files and setting up a listening socket, for example). Then, a system can reliably initialize its services with a simple linear script:

ntpd -s

Each daemon can rely on the previous one being ready to do its work. The sequence of actions is the following:

  1. syslogd reads its configuration, activates /dev/log, daemonizes.
  2. unbound reads its configuration, listens on, daemonizes.
  3. ntpd reads its configuration, connects to NTP peers, waits for clock to be synchronized2, daemonizes.

With systemd, we would use Type=fork in the service file. However, Go’s runtime does not support that. Instead, we use Type=notify. In this case, systemd expects the daemon to signal its readiness with a message to an Unix socket. go-systemd package handles the details for us:

package main

import (


func main() {
    l, err := net.Listen("tcp", ":8081")
    if err != nil {
        log.Panicf("cannot listen: %s", err)
    daemon.SdNotify(false, "READY=1") // ❶
    http.Serve(l, nil)                // ❷

It’s important to place the notification after net.Listen() (in ❶): if the notification was sent earlier, a client would get “connection refused” when trying to use the service. When a daemon listens to a socket, connections are queued by the kernel until the daemon is able to accept them (in ❷).

If the service is not run through systemd, the added line is a no-op.


Another interesting feature of systemd is to watch the service and restart it if it happens to crash (thanks to the Restart=on-failure directive). It’s also possible to use a watchdog: the service sends watchdog keep-alives at regular interval. If it fails to do so, systemd will restart it.

We could insert the following code just before http.Serve() call:

go func() {
    interval, err := daemon.SdWatchdogEnabled(false)
    if err != nil || interval == 0 {
    for {
        daemon.SdNotify(false, "WATCHDOG=1")
        time.Sleep(interval / 3)

However, this doesn’t add much value: the goroutine is unrelated to the core business of the service. If for some reason, the HTTP part gets stuck, the goroutine will happily continue to send keep-alives to systemd.

In our example, we can just do a HTTP query before sending the keep-alive. The internal loop can be replaced with this code:

for {
    _, err := http.Get("") // ❸
    if err == nil {
        daemon.SdNotify(false, "WATCHDOG=1")
    time.Sleep(interval / 3)

In ❸, we connect to the service to check if it’s still working. If we get some kind of answer, we send a watchdog keep-alive. If the service is unavailable or if http.Get() gets stuck, systemd will trigger a restart.

There is no universal recipe. However, checks can be split into two groups:

  • Before sending a keep-alive, you execute an active check on the components of your service. The keep-alive is sent only if all checks are successful. The checks can be internal (like in the above example) or external (for example, check with a query to the database).

  • Each component reports its status, telling if it’s alive or not. Before sending a keep-alive, you check the reported status of all components (passive check). If some components are late or reported fatal errors, don’t send the keep-alive.

If possible, recovery from errors (for example, with a backoff retry) and self-healing (for example, by reestablishing a network connection) is always better, but the watchdog is a good tool to handle the worst cases and avoid too complex recovery logic.

For example, if a component doesn’t know how to recover from an exceptional condition3, instead of using panic(), it could signal its situation before dying. Another dedicated component could try to resolve the situation by restarting the faulty component. If it fails to reach an healthy state in time, the watchdog timer will trigger and the whole service will be restarted.

  1. Depending on the distribution, this should be installed in /lib/systemd/system or /usr/lib/systemd/system. Check with the output of the command pkg-config systemd --variable=systemdsystemunitdir

  2. This highly depends on the NTP daemon used. OpenNTPD doesn’t wait unless you use the -s option. ISC NTP doesn’t either unless you use the --wait-sync option. 

  3. An example of an exceptional condition is to reach the limit on the number of file descriptors. Self-healing from this situation is difficult and it’s easy to get stuck in a loop. 

by Vincent Bernat at February 09, 2017 08:32 AM

February 08, 2017

Steve Kemp's Blog

Old packages are interesting.

Recently Vincent Bernat wrote about writing his own simple terminal, using vte. That was a fun read, as the sample code built really easily and was functional.

At the end of his post he said :

evilvte is quite customizable and can be lightweight. Consider it as a first alternative. Honestly, I don’t remember why I didn’t pick it.

That set me off looking at evilvte, and it was one of those rare projects which seems to be pretty stable, and also hasn't changed in any recent release of Debian GNU/Linux:

  • lenny had 0.4.3-1.
  • etch had nothing.
  • squeeze had 0.4.6-1.
  • wheezy has release 0.5.1-1.
  • jessie has release 0.5.1-1.
  • stretch has release 0.5.1-1.
  • sid has release 0.5.1-1.

I wonder if it would be possible to easily generate a list of packages which have the same revision in multiple distributions? Anyway I had a look at the source, and unfortunately spotted that it didn't entirely handle clicking on hyperlinks terribly well. Clicking on a link would pretty much run:

 firefox '%s'

That meant there was an obvious security problem.

It is a great terminal though, and it just goes to show how short, simple, and readable such things can be. I enjoyed looking at the source, and furthermore enjoyed using it. Unfortunately due to a dependency issue it looks like this package will be removed from stretch.

February 08, 2017 10:00 PM

Aikido & HolisticInfoSec™

This is the 300th post to the HolisticInfoSec™ blog. Sparta, this isn't, but I thought it important to provide you with content in a warrior/philosopher mindset regardless. 
Your author is an Aikido practitioner, albeit a fledgling in practice, with so, so much to learn. While Aikido is often translated as "the way of unifying with life energy" or as "the way of harmonious spirit", I propose that the philosophies and principles inherent to Aikido have significant bearing on the practice of information security.
In addition to spending time in the dojo, there are numerous reference books specific to Aikido from which a student can learn. Among the best is Adele Westbrook and Oscar Ratti's Aikido and the Dynamic Sphere. All quotes and references that follow are drawn from this fine publication.
As an advocate for the practice of HolisticInfoSec™ (so much so, I trademarked it) the connectivity to Aikido is practically rhetorical, but allow me to provide you some pointed examples. I've tried to connect each of these in what I believe is an appropriate sequence to further your understanding, and aid you in improving your practice. Simply, one could say each of these can lead to the next.
The Practice of Aikido
"The very first requisite for defense is to know the enemy."
So often in information security, we see reference to the much abused The Art of War, wherein Sun Tzu stated "It is said that if you know your enemies and know yourself, you will not be imperiled in a hundred battles." Aikido embraces this as the first requisite, but so too offers the importance of not underestimating your enemy or opponent. For information security, I liken it to this. If you are uninformed on adversary actor types and profiles, their TTPs (tools, tactics, procedures), as well as current vulnerabilities and exploits, along with more general threat intelligence, then you are already at a disadvantage before you even begin to imagine countering your opponent.  

"A positive defensive strategy is further qualified as being specific, immediate, consistent, and powerful." 
Upon learning more about your adversary, a rehearsed, exercised strategy for responding to their attack should be considered the second requisite for defense. To achieve this, your efforts must include:
  • a clear definition and inventory of the assets you're protecting
  • threat modeling of code, services, and infrastructure
  • an incident response plan and SOP, and regular exercise of the IR plan
  • robust security monitoring to include collection, aggregation, detection, correlation, and visualization
  • ideally, a purple team approach that includes testing blue team detection and response capabilities in partnership with a red team. Any red team that follows the "you suck, we rock" approach should be removed from the building and replaced by one who espouses "we exist to identify vulnerabilities and exploits with the goal of helping the organization better mitigate and remediate".
As your detection and response capabilities improve with practice and repetition, your meantime to mitigate (MTM) and meantime to remediate (MTR) should begin to shrink, thus lending to the immediacy, consistentcy, and power of your defense.

The Process of Defense and Its Factors
"EVERY process of defense will consist of three stages: perception, evaluation-decision, and reaction."
These should be easy likenesses for you to reconcile.
Perception = detection and awareness
The better and more complete your threat intelligence collection and detection capabilities, the better your situational awareness will be, and as a result your perception of adversary behaviors will improve and become more timely.
Evaluation-decision = triage
It's inevitable...$#!+ happens. Your ability to quickly evaluate adversary actions and be decisive in your response will dictate your level of success as incident responders. Strength at this stage directly impacts the rest of the response process. Incorrect or incomplete evaluation, and the resulting ill-informed decisions, can set back your response process in a manner from which recovery will be very difficult.
Reaction = response
My Aikido sensei, after doing so, likes to remind his students "Don't get hit." :-) The analogy here is to react quickly enough to stay on your feet. Can you move quickly enough to not be hit as hard or as impactfully as your adversary intended? Your reaction and response will determine such outcomes. The connection between kinetic and virtual combat here is profound. Stand still, get hit. Feign or evade, at least avoid some, or all contact. In the digital realm, you're reducing your time to recover with this mindset.  

Dynamic Factors
"A defensive aikido strategy begins the moment a would-be attacker takes a step toward you or turns aggressively in your direction. His initial motion (movement) in itself contains the factors you will use to neutralize the action of attack which will spring with explosive force from that motion of convergence."
Continuing on our theme of inevitability, digital adversaries will, beyond the shadow of a doubt, take a step toward you or turn aggressively in your direction. The question for you will be, do you even know when that has occurred in light of our discussion of requisites above? Aikido is all about using your opponent's energy against them, wherein, for those of us in DFIR, our adversary's movement in itself contains the factors we use to neutralize the action of attack. As we improve our capabilities in our defensive processes (perception, evaluation-decision, and reaction), we should be able to respond in a manner that begins the very moment we identify adversarial behavior, and do so quickly enough that our actions pivot directly on our adversary's initial motion.
As an example, your adversary conducts a targeted, nuanced spear phishing campaign. Your detective means identify all intended victims, you immediately react, and add all intended victims to an enhanced watch list for continuous monitoring. The two victims who engaged the payload are quarantined immediately, and no further adversarial pivoting or escalation is identified. The environment as a whole raised to a state of heightened awareness, and your user-base becomes part of your perception network.

"It will be immediate or instantaneous when your reaction is so swift that you apply a technique of neutralization while the attack is still developing, and at the higher levels of the practice even before an attack has been fully launched."
Your threat intelligence capabilities are robust enough that your active deployment of detections for specific Indicators of Compromise (IOCs) prevented the targeted, nuanced spear phishing campaign from even reaching the intended victims. Your monitoring active lists include known adversary infrastructure such that the moment they launch an attack, you are already aware of its imminence.
You are able to neutralize your opponent before they even launch. This may be unimaginable for some, but it is achievable by certain mature organizations under specific circumstances.

The Principle of Centralization
"Centralization, therefore, means adopting a new point of reference, a new platform from which you can exercise a more objective form of control over events and over yourself."
Some organizations decentralize information security, others centralize it with absolute authority. There are arguments for both, and I do not intend to engage that debate. What I ask you to embrace is the "principle of centralization". The analogy is this: large corporations and organizations often have multiple, and even redundant security teams. Even so, their cooperation is key to success.
  • Is information exchanged openly and freely, with silos avoided? 
  • Are teams capable of joint response? 
  • Are there shared resources that all teams can draw from for consistent IOCs and case data?
  • Are you and your team focused on facts, avoiding FUD, thinking creatively, yet assessing with a critical, objective eye?
Even with a logically decentralized security model, organizations can embrace the principle of centralization and achieve an objective form of control over events. The practice of a joint forces focus defines the platform from which teams can and should operate.

Adversarial conditions, in both the physical realm, and the digital realm in which DFIR practitioners operate, are stressful, challenging, and worrisome. 
Morihei Ueshiba, Aikido's founder reminds us that "in extreme situations, the entire universe becomes our foe; at such critical times, unity of mind and technique is essential - do not let your heart waver!" That said, perfection is unlikely, or even impossible, this is a practice you must exercise. Again Ueshiba offers that "failure is the key to success; each mistake teaches us something."
Keep learning, be strong of heart. :-)
Cheers...until next time.

by Russ McRee ( at February 08, 2017 04:45 PM

Geek and Artist - Tech

Culture Codes From Around The Tech World

Last week I found myself reading through all (ok not entirely all, I skipped the NASA one) of the slide decks on – there are a variety of slide decks from tech companies, which attempt to describe what the company culture looks like, their values, or company processes that serve as an employee handbook. They are called Culture Codes and not Company Culture because culture can only be talked about or described, but cannot itself be seen. Hence – the content of the slides are codes for the culture.

At any company we express our culture naturally – it has been described as “what happens when nobody is looking” (and there are many more quotes about it you can Google in your own time). It would be interesting to think about what might be inside a slide deck about my own company’s culture, were we to put one together (we haven’t yet, hence my thoughts on the topic). To that end, I decided to read as many of the slide decks on the above site, and summarise my thoughts.

General Observations

These are stream-of-consciousness notes in no particular order about what I observed in the slide decks.

  • Usually has a value/mission statement for the whole company.
  • Things they aspire to do, some descriptions of what the culture looks like.
  • What makes the company (and culture) different from others.
  • Longer-form description and detail of company values (i.e. describe each value).
  • How the company works (as in workflow) and works (as in success).
  • How to make decisions.
  • Definitions of what success looks like.
  • High-level description of behaviours/traits of people who work well in the company.
  • Lots of justifications for “why” these values are here.
  • Culture deck is often shared publicly and used as recruiting/marketing tool (both for employees and customers).
  • Some have a small “history” section describing the founding of the company or bootstrapping of company values. Good if they are interlinked. Describes major milestones of the company so far.
  • Quotes from customers, employees, source material (e.g. entrepreneurs) for values or guiding statements, supporting the values.
  • If the deck is shared publicly, it sets expectations for employees before they walk through the door. Some are explicitly aimed at being marketing material.
  • Company involvement with community – greater impact than just through core mission.
  • It is tempting to copy values or content, but there are some slide decks which are very incompatible with our values, feel jarring (or just feel plain wrong). Some are very prescriptive, strict rules.
  • The language you use matters! Imagery too.
  • Many pictures of smiling employees / teams. Photos/descriptions of company events (e.g. parties/picnics etc).
  • Crediting culture with driving the success of the company.
  • Video where employees can give concrete examples of living the values.
    • BUT, culture is not just quirky work environments, posters on the wall, etc.
  • Many culture decks double as employee handbooks (e.g. Zappos, Valve). Some are extremely long (Zappos ~300 pages -> many personal employee stories and photos -> tell the values story through many concrete examples rather than analogy/metaphor and brief descriptions).
    • Length varies a lot. Lower limit is around 12 slides, which is not enough space for meaningful descriptions. 50-100 is the sweet spot, more than that just drags on.
  • Some (Genius, Trello) dogfood their own product to create their culture deck (or employee handbook), but it comes off as a bit messy and hard to understand, in some cases full of comments that obscures what is current or actual reality. NextBigSound, Sprintly, Clef store markdown on Github (which is optimised for authoring but not necessarily consumption).
  • Inputs sometimes also from partners and customers.
  • All culture decks walk a tightrope between cliché and meaningless catchphrases on either side. Extremely difficult (impossible?) to be original, quite difficult to be meaningful and capture reality that people can buy into.
  • Often both clarifying negative and positive statements: “X means that we do Y”, or “X doesn’t mean that we do Z”.
  • Google’s values are predictably different (and perhaps not a great fit for many other smaller companies with fewer resources). Same goes for NASA which has a completely different set of needs to most tech companies (safety!).
  • Although I don’t like the Big Spaceship culture deck too much (painful on the eyes) I like that they call bullshit on buzzwords and manager speak.
  • Lots of quotes from Daniel Pink, Steve Jobs, Malcolm Gladwell, US and UK leaders.


As you might expect, the values that companies strive to adhere to and live every day are very similar. I summarised them into a smaller number of similar themes (which may not all be entirely cohesive, but not much time was spent on this):

  • Autonomy / freedom / desire to reduce process / resourceful.
  • “Do the right thing” (also related to the above) / trust / responsibility / ownership.
  • Driven by a vision / vision provided by leaders / leaders living and demonstrating values.
  • Transparency / sharing knowledge / communication / feedback / questioning / openness.
  • Selflessness / not personally identifying with the work / using data rather than opinions.
  • Striving for results / outcome-driven / calculated risk taking / action over talk / results vs busy / building for the customer / doing more with less / growth / Get [Sh]it Done.
  • Learning from mistakes / great is the enemy of good / ship now / fail fast.
  • Helping others / treating colleagues with respect / coaching / mentoring / servant leadership.
  • Constantly learning / inventive / optimistic.
  • Mindfulness / empathy / care / reflection / time for being strategic.
  • Inclusivity / diversity goals.
  • Work/life balance, living healthily. Having fun / celebrating success / “family” feeling / friends at work.
  • Meaningful work / Company invests in community / serving a higher purpose.
  • Intertwining of individual/team/company values and missions.
  • At least one “quirky”, unique company-specific hashtag or catchphrase (several companies list “quirky” as a value).
  • Working with other talented people / hiring & nurturing.
  • Almost all claim to be unique! (really?)

“Anti-” Values

Some companies have very different values to others, which may be undesirable or completely destructive outside of that particular company. But perhaps within their own environment they do work. Presented here without further comment:

  • We start work at 8:30 every day.
  • Many sporting team analogies and metaphors (bro culture?).
  • Success is “hard” or a “grind”.
  • Work hard play hard / fast-paced / urgency / hustle.
  • Timebox everything. Strict scheduling.
  • Focus on competition mentality vs satisfying customers.
  • Only hire “A” players.
  • Do the thing you least want to do right now.
  • Blindly attempting to define culture around The Art of War.
  • Never compromise (never never? really?).
  • Do things that don’t scale.
  • Open Office! (I thought that wasn’t cool anymore…)
  • Many other things that work when the company is very small, but eventually fail and it turns into the same org structure and ways of working as most other tech companies (e.g. Github).

What are some of your own culture codes or values that you like, and see lived out every day? Leave some comments!

by oliver at February 08, 2017 11:27 AM

February 07, 2017

Vincent Bernat

Write your own terminal emulator

I was an happy user of rxvt-unicode until I got a laptop with an HiDPI display. Switching from a LoDPI to a HiDPI screen and back was a pain: I had to manually adjust the font size on all terminals or restart them.

VTE is a library to build a terminal emulator using the GTK+ toolkit, which handles DPI changes. It is used by many terminal emulators, like GNOME Terminal, evilvte, sakura, termit and ROXTerm. The library is quite straightforward and writing a terminal doesn’t take much time if you don’t need many features.

Let’s see how to write a simple one.

A simple terminal

Let’s start small with a terminal with the default settings. We’ll write that in C. Another supported option is Vala.

#include <vte/vte.h>

main(int argc, char *argv[])
    GtkWidget *window, *terminal;

    /* Initialise GTK, the window and the terminal */
    gtk_init(&argc, &argv);
    terminal = vte_terminal_new();
    window = gtk_window_new(GTK_WINDOW_TOPLEVEL);
    gtk_window_set_title(GTK_WINDOW(window), "myterm");

    /* Start a new shell */
    gchar **envp = g_get_environ();
    gchar **command = (gchar *[]){g_strdup(g_environ_getenv(envp, "SHELL")), NULL };
        NULL,       /* working directory  */
        command,    /* command */
        NULL,       /* environment */
        0,          /* spawn flags */
        NULL, NULL, /* child setup */
        NULL,       /* child pid */
        NULL, NULL);

    /* Connect some signals */
    g_signal_connect(window, "delete-event", gtk_main_quit, NULL);
    g_signal_connect(terminal, "child-exited", gtk_main_quit, NULL);

    /* Put widgets together and run the main loop */
    gtk_container_add(GTK_CONTAINER(window), terminal);

You can compile it with the following command:

gcc -O2 -Wall $(pkg-config --cflags --libs vte-2.91) term.c -o term

And run it with ./term:

Simple VTE-based terminal

More features

From here, you can have a look at the documentation to alter behavior or add more features. Here are three examples.


You can define the 16 basic colors with the following code:

#define CLR_R(x)   (((x) & 0xff0000) >> 16)
#define CLR_G(x)   (((x) & 0x00ff00) >>  8)
#define CLR_B(x)   (((x) & 0x0000ff) >>  0)
#define CLR_16(x)  ((double)(x) / 0xff)
#define CLR_GDK(x) (const GdkRGBA){ .red = CLR_16(CLR_R(x)), \
                                    .green = CLR_16(CLR_G(x)), \
                                    .blue = CLR_16(CLR_B(x)), \
                                    .alpha = 0 }
    &(GdkRGBA){ .alpha = 0.85 },
    (const GdkRGBA[]){
}, 16);

While you can’t see it on the screenshot1, this also enables background transparency.

Color rendering

Miscellaneous settings

VTE comes with many settings to change the behavior of the terminal. Consider the following code:

vte_terminal_set_scrollback_lines(VTE_TERMINAL(terminal), 0);
vte_terminal_set_scroll_on_output(VTE_TERMINAL(terminal), FALSE);
vte_terminal_set_scroll_on_keystroke(VTE_TERMINAL(terminal), TRUE);
vte_terminal_set_rewrap_on_resize(VTE_TERMINAL(terminal), TRUE);
vte_terminal_set_mouse_autohide(VTE_TERMINAL(terminal), TRUE);

This will:

  • disable the scrollback buffer,
  • not scroll to the bottom on new output,
  • scroll to the bottom on keystroke,
  • rewrap content when terminal size change, and
  • hide the mouse cursor when typing.

Update the window title

An application can change the window title using XTerm control sequences (for example, with printf "\e]2;${title}\a"). If you want the actual window title to reflect this, you need to define this function:

static gboolean
on_title_changed(GtkWidget *terminal, gpointer user_data)
    GtkWindow *window = user_data;
    return TRUE;

Then, connect it to the appropriate signal, in main():

g_signal_connect(terminal, "window-title-changed", 
    G_CALLBACK(on_title_changed), GTK_WINDOW(window));

Final words

I don’t need much more as I am using tmux inside each terminal. In my own copy, I have also added the ability to complete a word using ones from the current window or other windows (also known as dynamic abbrev expansion). This requires to implement a terminal daemon to handle all terminal windows with one process, similar to urxvtcd.

While writing a terminal “from scratch”2 suits my need, it may not be worth it. evilvte is quite customizable and can be lightweight. Consider it as a first alternative. Honestly, I don’t remember why I didn’t pick it.

UPDATED: evilvte has not seen an update since 2014. Its GTK+3 support is buggy. It doesn’t support the latest versions of the VTE library. Therefore, it’s not a good idea to use it.

You should also note that the primary goal of VTE is to be a library to support GNOME Terminal. Notably, if a feature is not needed for GNOME Terminal, it won’t be added to VTE. If it already exists, it will likely to be deprecated and removed.

  1. Transparency is handled by the composite manager (Compton, in my case). 

  2. For some definition of “scratch” since the hard work is handled by VTE

by Vincent Bernat at February 07, 2017 11:54 PM

Everything Sysadmin

Cascadia IT Conference: Registration is open!

I am pleased to announce that the 7th Annual Cascadia IT conference Registration is open at

This years conference features tutorials on Git, Reusable Infrastructures, Time Management (based on my book), AWS, and Advanced Communication Strategies along with several talks and a keynote by Corey Quinn.

Cascadia IT Conference will be held on March 10-11 in Seattle WA.

by Tom Limoncelli at February 07, 2017 10:00 PM

Sean's IT Blog

Horizon 7.1–Together We’re Just in Time

If you couldn’t tell by the title of this post, new product announcement time.  Last year at this time, VMware announced App Volumes 3.0, Horizon 7.0, and a number of enhancements including Smart Policies and Instant Clones.  This year, Horizon 7.1 brings a brand new set of improvements that build on the features that were released with Horizon 7.0, including some long awaited features around vGPU, instant clones, and Blast Extreme.

The Just-in-Time Management Platform

The Just-in-Time Management Platform, or JMP (pronounced “jump”) is VMware’s next generation desktop and application delivery platform.  JMP is built on VMware’s instant clone technology for fast provisioning, real-time application delivery provided by App Volumes, and contextual policy and user environment management by UEM.  JMP provides both traditional desktops with just-in-time instant clone desktops and published applications with just-in-time RDSH servers for published applications.


You heard that right.  Instant clone technology has been extended, and it can now be used to provision RDSH server farms. Like Instant Clone desktop pools, instant clone RDSH farms can be elastic and scale near-instantaneously to provide capacity as it is required.  Like instant clone desktop pools, instant clone RDSH farms can provide rolling updates for zero downtime image updates.  VMware is also adding the ability to schedule recurring maintenance of RDSH pools into Horizon Administrator to ensure that the servers are periodically refreshed.

Joining the new Just-in-Time Apps feature is a new Horizon SKU called Horizon Apps.  This SKU provides an RDSH-only focused SKU that includes Instant Clones, Published Apps, App Volumes, UEM, and vSphere.  This new SKU fills a hole in the Horizon product lineup and provides a direct competitor to Citrix XenApp.

We Got the BEAT – Blast Extreme Adaptive Transport

We got the beat…we got the beat…

Sorry, got a little carried away there. 

VMware continues to improve the Blast protocol, and Horizon 7.1 contains the latest enhancements to the protocol – Blast Extreme Adaptive Transport, or BEAT for short.  BEAT is designed to provide a great user experience on multiple network types while adapting to changing network conditions including varying network speeds and severe packet loss. 

VMware was able to achieve these results by adding adaptive bit rates and forward error correction to the protocol as well as adding additional optimizations to better handle network latency and packet loss.  As a result of these improvements, VMware was able to significantly reduce out-of-the-box bandwidth utilization, improve user experience on high latency links, and improve file transfer times from the endpoint to the virtual desktop when using Client Drive Redirection.

Horizon Cloud

Ok…it’s not a good fit.  But it’s hard to find a good 80’s song about clouds. 

Horizon Cloud is the next-generation version of Horizon Air.  Unlike Horizon Air, which was hosted in vCloud Air, Horizon Cloud is hosted in IBM SoftLayer, and it will feature the Horizon Just-in-Time Management Platform.  This enables VMware to provide features that weren’t available in vCloud Air including GPU support, published applications, and utilizing vIDM as a portal for accessing virtual desktops and applications.

Horizon Cloud will also feature an on-premises solution combining hyper-converged infrastructure with a cloud control plane.  This offering will serve as the next generation Horizon Air Hybrid Mode.

Horizon Cloud will utilize one license for both the cloud-hosted and on-premises offering, and these licenses will be available in both named user and concurrent user options.

Other Enhancements

Like any Horizon release, there will be a multitude of enhancements.  A few of the other new enhancements that I’ve learned about are:

  • Instant Clone Desktop support for vGPU – Horizon 7.1 will allow you to run 3D workloads on Instant Clone Desktops
  • Multi-VLAN support for Instant Clones – Feature-parity with Linked-Clone desktops
  • Being able to manage multiple VLANs within Horizon Administrator – no need for a PowerShell script

by seanpmassey at February 07, 2017 01:05 PM

February 06, 2017

LZone - Sysadmin

How to fix debsecan for Wheezy

Someone at Debian increase security for all Debian servers by breaking debsecan a while ago for everything before Jessie by moving the vulnerability definitions from


Of course there was no way to issue a security fix for Wheezy debsecan...

Workaround 1: Hotfix

So if you still want to scan your Wheezy systems you can hotfix debsecan before running it like this:
sed -i "s/http:\/\/\/debian-secure-testing/https:\/\/\/tracker/;s/project\/debsecan\/release\/1\//debsecan\/release\/1\//" /usr/bin/debsecan 

Workaround 2: Pass Config

You can also pass an in-place config file:
debsecan --config <(echo SOURCE="")

February 06, 2017 05:23 PM

February 05, 2017

Vincent Bernat

A Makefile for your Go project

My most loathed feature of Go is the mandatory use of GOPATH: I do not want to put my own code next to its dependencies. Hopefully, this issue is slowly starting to be accepted by the main authors. In the meantime, you can workaround this problem with more opinionated tools (like gb) or by crafting your own Makefile.

For the later, you can have a look at Filippo Valsorda’s example or my own take which I describe in more details here. This is not meant to be an universal Makefile but a relatively short one with some batteries included. It comes with a simple “Hello World!” application.

Project structure

For a standalone project, vendoring is a must-have1 as you cannot rely on your dependencies to not introduce backward-incompatible changes. Some packages are using versioned URLs but most of them aren’t. There is currently no standard tool to handle vendoring. My personal take is to vendor all dependencies with Glide.

It is a good practice to split an application into different packages while the main one stay fairly small. In the hellogopher example, the CLI is handled in the cmd package while the application logic for printing greetings is in the hello package:

├── cmd/
│   ├── hello.go
│   ├── root.go
│   └── version.go
├── glide.lock (generated)
├── glide.yaml
├── vendor/ (dependencies will go there)
├── hello/
│   ├── root.go
│   └── root_test.go
├── main.go
├── Makefile

Down the rabbit hole

Let’s take a look at the various “features” of the Makefile.

GOPATH handling

Since all dependencies are vendored, only our own project needs to be in the GOPATH:

PACKAGE  = hellogopher
GOPATH   = $(CURDIR)/.gopath
BASE     = $(GOPATH)/src/$(PACKAGE)

    @mkdir -p $(dir $@)
    @ln -sf $(CURDIR) $@

The base import path is hellogopher, not this shortens imports and makes them easily distinguishable from imports of dependency packages. However, your application won’t be go get-able. This is a personal choice and can be adjusted with the $(PACKAGE) variable.

We just create a symlink from .gopath/src/hellogopher to our root directory. The GOPATH environment variable is automatically exported to the shell commands of the recipes. Any tool should work fine after changing the current directory to $(BASE). For example, this snippet builds the executable:

.PHONY: all
all: | $(BASE)
    cd $(BASE) && $(GO) build -o bin/$(PACKAGE) main.go

Vendoring dependencies

Glide is a bit like Ruby’s Bundler. In glide.yaml, you specify what packages you need and the constraints you want on them. Glide computes a glide.lock file containing the exact versions for each dependencies (including recursive dependencies) and download them in the vendor/ folder. I choose to check into the VCS both glide.yaml and glide.lock files. It’s also possible to only check in the first one or to also check in the vendor/ directory. A work-in-progress is currently ongoing to provide a standard dependency management tool with a similar workflow.

We define two rules2:

GLIDE = glide

glide.lock: glide.yaml | $(BASE)
    cd $(BASE) && $(GLIDE) update
    @touch $@
vendor: glide.lock | $(BASE)
    cd $(BASE) && $(GLIDE) --quiet install
    @ln -sf . vendor/src
    @touch $@

We use a variable to invoke glide. This enables a user to easily override it (for example, with make GLIDE=$GOPATH/bin/glide).

Using third-party tools

Most projects need some third-party tools. We can either expect them to be already installed or compile them in our private GOPATH. For example, here is the lint rule:

BIN    = $(GOPATH)/bin
GOLINT = $(BIN)/golint

$(BIN)/golint: | $(BASE) # ❶
    go get

.PHONY: lint
lint: vendor | $(BASE) $(GOLINT) # ❷
    @cd $(BASE) && ret=0 && for pkg in $(PKGS); do \
        test -z "$$($(GOLINT) $$pkg | tee /dev/stderr)" || ret=1 ; \
     done ; exit $$ret

As for glide, we let the user a chance to override which golint executable to use. By default, it uses a private copy. But a user can use its own copy with make GOLINT=/usr/bin/golint.

In ❶, we have the recipe to build the private copy. We simply issue go get3 to download and build golint. In ❷, the lint rule executes golint on each package contained in the $(PKGS) variable. We’ll explain this variable in the next section.

Working with non-vendored packages only

Some commands need to be provided with a list of packages. Because we use a vendor/ directory, the shortcut ./... is not what we expect as we don’t want to run tests on our dependencies4. Therefore, we compose a list of packages we care about:

PKGS = $(or $(PKG), $(shell cd $(BASE) && \
    env GOPATH=$(GOPATH) $(GO) list ./... | grep -v "^$(PACKAGE)/vendor/"))

If the user has provided the $(PKG) variable, we use it. For example, if they want to lint only the cmd package, they can invoke make lint PKG=hellogopher/cmd which is more intuitive than specifying PKGS.

Otherwise, we just execute go list ./... but we remove anything from the vendor directory.


Here are some rules to run tests:

TEST_TARGETS := test-default test-bench test-short test-verbose test-race
.PHONY: $(TEST_TARGETS) check test tests
test-bench:   ARGS=-run=__absolutelynothing__ -bench=.
test-short:   ARGS=-short
test-verbose: ARGS=-v
test-race:    ARGS=-race

check test tests: fmt lint vendor | $(BASE)
    @cd $(BASE) && $(GO) test -timeout $(TIMEOUT)s $(ARGS) $(PKGS)

A user can invoke tests in different ways:

  • make test runs all tests;
  • make test TIMEOUT=10 runs all tests with a timeout of 10 seconds;
  • make test PKG=hellogopher/cmd only runs tests for the cmd package;
  • make test ARGS="-v -short" runs tests with the specified arguments;
  • make test-race runs tests with race detector enabled.

Tests coverage

go test includes a test coverage tool. Unfortunately, it only handles one package at a time and you have to explicitely list the packages to be instrumented, otherwise the instrumentation is limited to the currently tested package. If you provide too many packages, the compilation time will skyrocket. Moreover, if you want an output compatible with Jenkins, you’ll need some additional tools.

COVERAGE_MODE    = atomic
COVERAGE_XML     = $(COVERAGE_DIR)/coverage.xml

.PHONY: test-coverage test-coverage-tools
test-coverage-tools: | $(GOCOVMERGE) $(GOCOV) $(GOCOVXML) # ❸
test-coverage: COVERAGE_DIR := $(CURDIR)/test/coverage.$(shell date -Iseconds)
test-coverage: fmt lint vendor test-coverage-tools | $(BASE)
    @mkdir -p $(COVERAGE_DIR)/coverage
    @cd $(BASE) && for pkg in $(PKGS); do \ # ❹
        $(GO) test \
            -coverpkg=$$($(GO) list -f '{{ join .Deps "\n" }}' $$pkg | \
                    grep '^$(PACKAGE)/' | grep -v '^$(PACKAGE)/vendor/' | \
                    tr '\n' ',')$$pkg \
            -covermode=$(COVERAGE_MODE) \
            -coverprofile="$(COVERAGE_DIR)/coverage/`echo $$pkg | tr "/" "-"`.cover" $$pkg ;\
    @$(GOCOVMERGE) $(COVERAGE_DIR)/coverage/*.cover > $(COVERAGE_PROFILE)
    @$(GO) tool cover -html=$(COVERAGE_PROFILE) -o $(COVERAGE_HTML)

First, we define some variables to let the user override them. We also require the following tools (in ❸):

  • gocovmerge merges profiles from different runs into a single one;
  • gocov-xml converts a coverage profile to the Cobertura format;
  • gocov is needed to convert a coverage profile to a format handled by gocov-xml.

The rules to build those tools are similar to the rule for golint described a few sections ago.

In ❹, for each package to test, we run go test with the -coverprofile argument. We also explicitely provide the list of packages to instrument to -coverpkg by using go list to get a list of dependencies for the tested package and keeping only our owns.

Final result

While the main goal of using a Makefile was to work around GOPATH, it’s also a good place to hide the complexity of some operations, notably around test coverage.

The excerpts provided in this post are a bit simplified. Have a look at the final result for more perks!

  1. In Go, “vendoring” is about both bundling and dependency management. As the Go ecosystem matures, the bundling part (fixed snapshots of dependencies) may become optional but the vendor/ directory may stay for dependency management (retrieval of the latest versions of dependencies matching a set of constraints). 

  2. If you don’t want to automatically update glide.lock when a change is detected in glide.yaml, rename the target to deps-update and make it a phony target. 

  3. There is some irony for bad mouthing go get and then immediately use it because it is convenient. 

  4. I think ./... should not include the vendor/ directory by default. Dependencies should be trusted to have run their own tests in the environment they expect them to succeed. Unfortunately, this is unlikely to change

by Vincent Bernat at February 05, 2017 01:28 PM

Feeding the Cloud

IPv6 and OpenVPN on Linode Debian/Ubuntu VPS

Here is how I managed to extend my OpenVPN setup on my Linode VPS to include IPv6 traffic. This ensures that clients can route all of their traffic through the VPN and avoid leaking IPv6 traffic, for example. It also enables clients on IPv4-only networks to receive a routable IPv6 address and connect to IPv6-only servers (i.e. running your own IPv6 broker).

Request an additional IPv6 block

The first thing you need to do is get a new IPv6 address block (or "pool" as Linode calls it) from which you can allocate a single address to each VPN client that connects to the server.

If you are using a Linode VPS, there are instructions on how to request a new IPv6 pool. Note that you need to get an address block between /64 and /112. A /116 like Linode offers won't work in OpenVPN. Thankfully, Linode is happy to allocate you an extra /64 for free.

Setup the new IPv6 address

If your server only has an single IPv4 address and a single IPv6 address, then a simple DHCP-backed network configuration will work fine. To add the second IPv6 block on the other hand, I had to change my network configuration (/etc/network/interfaces) to this:

auto lo
iface lo inet loopback

allow-hotplug eth0
iface eth0 inet dhcp
    pre-up iptables-restore /etc/network/iptables.up.rules

iface eth0 inet6 static
    address 2600:3c01::xxxx:xxxx:xxxx:939f/64
    gateway fe80::1
    pre-up ip6tables-restore /etc/network/ip6tables.up.rules

iface tun0 inet6 static
    address 2600:3c01:xxxx:xxxx::/64
    pre-up ip6tables-restore /etc/network/ip6tables.up.rules

where 2600:3c01::xxxx:xxxx:xxxx:939f/64 (bound to eth0) is your main IPv6 address and 2600:3c01:xxxx:xxxx::/64 (bound to tun0) is the new block you requested.

Once you've setup the new IPv6 block, test it from another IPv6-enabled host using:

ping6 2600:3c01:xxxx:xxxx::1

OpenVPN configuration

The only thing I had to change in my OpenVPN configuration (/etc/openvpn/server.conf) was to change:

proto udp


proto udp6

in order to make the VPN server available over both IPv4 and IPv6, and to add the following lines:

server-ipv6 2600:3c01:xxxx:xxxx::/64
push "route-ipv6 2000::/3"

to bind to the right V6 address and to tell clients to tunnel all V6 Internet traffic through the VPN.

In addition to updating the OpenVPN config, you will need to add the following line to /etc/sysctl.d/openvpn.conf:


and the following to your firewall (e.g. /etc/network/ip6tables.up.rules):

# openvpn
-A INPUT -p udp --dport 1194 -j ACCEPT
-A FORWARD -m state --state NEW -i tun0 -o eth0 -s 2600:3c01:xxxx:xxxx::/64 -j ACCEPT
-A FORWARD -m state --state NEW -i eth0 -o tun0 -d 2600:3c01:xxxx:xxxx::/64 -j ACCEPT

in order to ensure that IPv6 packets are forwarded from the eth0 network interface to tun0 on the VPN server.

With all of this done, apply the settings by running:

sysctl -p /etc/sysctl.d/openvpn.conf
systemctl restart openvpn.service

Testing the connection

Now connect to the VPN using your desktop client and check that the default IPv6 route is set correctly using ip -6 route.

Then you can ping the server's new IP address:

ping6 2600:3c01:xxxx:xxxx::1

and from the server, you can ping the client's IP (which you can see in the network settings):

ping6 2600:3c01:xxxx:xxxx::1002

Once both ends of the tunnel can talk to each other, you can try pinging an IPv6-only server from your client:


and then pinging your client from an IPv6-enabled host somewhere:

ping6 2600:3c01:xxxx:xxxx::1002

If that works, other online tests should also work.

February 05, 2017 01:20 PM

February 01, 2017

Anton Chuvakin - Security Warrior

Monthly Blog Round-Up – January 2017

Here is my next monthly "Security Warrior" blog round-up of top 5 popular posts/topics this month:
  1. This month, my classic PCI DSS Log Review series is extra popular! The series of 18 posts cover a comprehensive log review approach (OK for PCI DSS 3+ even though it predates it), useful for building log review processes and procedures, whether regulatory or not. It is also described in more detail in our Log Management book and mentioned in our PCI book (now in its 4th edition!) – note that this series is mentioned in some PCI Council materials.
  2. Simple Log Review Checklist Released!” is often at the top of this list – this aging checklist is still a very useful tool for many people. “On Free Log Management Tools” (also aged a bit by now) is a companion to the checklist (updated version)
  3. “New SIEM Whitepaper on Use Cases In-Depth OUT!” (dated 2010) presents a whitepaper on select SIEM use cases described in depth with rules and reports [using now-defunct SIEM product]; also see this SIEM use case in depth and this for a more current list of popular SIEM use cases. Finally, see our 2016 research on developing security monitoring use cases here!
  4. Why No Open Source SIEM, EVER?” contains some of my SIEM thinking from 2009. Is it relevant now? You be the judge.  Succeeding with SIEM requires a lot of work, whether you paid for the software, or not. BTW, this post has an amazing “staying power” that is hard to explain – I suspect it has to do with people wanting “free stuff” and googling for “open source SIEM” … 
  5. “An Open Letter to Android or “Android, You Are Shit!”” is an epic rant about my six year long (so far) relationship with Android mobile devices (no spoilers here – go and read it).
In addition, I’d like to draw your attention to a few recent posts from my Gartner blog [which, BTW, now has about 5X of the traffic of this blog]: 
Current research on security analytics and UBA / UEBA:
Recent research on deception:
Miscellaneous fun posts:

(see all my published Gartner research here)
Also see my past monthly and annual “Top Popular Blog Posts” – 2007, 2008, 2009, 2010, 2011, 2012, 2013, 2014, 2015, 2016.

Disclaimer: most content at SecurityWarrior blog was written before I joined Gartner on August 1, 2011 and is solely my personal view at the time of writing. For my current security blogging, go here.

Previous post in this endless series:

by Anton Chuvakin ( at February 01, 2017 01:11 PM

January 27, 2017

Ansible: access group vars for groups the current host is not a member of

This guide shows you how to access group variables for a group the current host is not a member of. In Ansible you can access other host variables using `hostvars['hostname']` but not group variables. The way described here is workable, but do I consider it a dirty hack. So why did I need this? I have a setup where ssl is offloaded by haproxy servers, but the virtual hosts and ssl configuration are defined in Apache servers. The loadbalancers and appservers are two different hostgroups, the ssl settings are in the appserver group_vars, which the hosts in the loadbalancer group need to access. The best way to do this is change the haproxy playbooks and configuration and define the certificates there, but in this specific case that wasn't a workable solution. Editing two yaml files (one for the appservers and one for the loadbalancers) was not an option in this situation.

January 27, 2017 12:00 AM

January 26, 2017

Colin Percival

IPv6 on FreeBSD/EC2

A few hours ago Amazon announced that they had rolled out IPv6 support in EC2 to 15 regions — everywhere except the Beijing region, apparently. This seems as good a time as any to write about using IPv6 in EC2 on FreeBSD instances.

January 26, 2017 07:15 AM

January 23, 2017


Choria Playbooks – Data Sources

About a month ago I blogged about Choria Playbooks – a way to write series of actions like MCollective, Shell, Slack, Web Hooks and others – contained within a YAML script with inputs, node sets and more.

Since then I added quite a few tweaks, features and docs, it’s well worth a visit to to check it out.

Today I want to blog about a major new integration I did into them and a major step towards version 1 for Choria.


In the context of a playbook or even a script calling out to other system there’s many reasons to have a Data Source. In the context of a playbook designed to manage distributed systems the Data Source needed has some special needs. Needs that tools like Consul and etcd fulfil specifically.

So today I released version 0.0.20 of Choria that includes a Memory and a Consul Data Source, below I will show how these integrate into the Playbooks.

I think using a distributed data store is important in this context rather than expecting to pass variables from the Playbook around like on the CLI since the business of dealing with the consistency, locking and so forth are handled and I can’t know all the systems you wish to interact with, but if those can speak to Consul you can prepare an execution environment for them.

For those who don’t agree there is a memory Data Store that exists within the memory of the Playbook. Your playbook should remain the same apart from declaring the Data Source.

Using Consul

Defining a Data Source

Like with Node Sets you can have multiple Data Sources and they are identified by name:

    type: consul
    timeout: 360
    ttl: 20

This creates a Consul Data Source called pb_data, you need to have a local Consul Agent already set up. I’ll cover the timeout and ttl a bit later.

Playbook Locks

You can create locks in Consul and by their nature they are distributed across the Consul network. This means you can ensure a playbook can only be executed once per Consul DC or by giving a custom lock name any group of related playbooks or even other systems that can make Consul locks.

  - pb_data
  - pb_data/custom_lock

This will create 2 locks in the pb_data Data Store – one called custom_lock and another called choria/locks/playbook/pb_name where pb_name is the name from the metadata.

It will try to acquire a lock for up to timeout seconds – 360 here, if it can’t the playbook run fails. The associated session has a TTL of 20 seconds and Choria will renew the sessions around 5 seconds before the TTL expires.

The TTL will ensure that should the playbook die, crash, machine die or whatever, the lock will release after 20 seconds.

Binding Variables

Playbooks already have a way to bind CLI arguments to variables called Inputs. Data Sources extend inputs with extra capabilities.

We now have two types of Input. A static input is one where you give the data on the CLI and the data stays static for the life of the playbook. A dynamic input is one bound against a Data Source and the value of it is fetched every time you reference the variable.

    description: "Cluster to deploy"
    type: "String"
    required: true
    data: "pb_data/choria/kv/cluster"
    default: "alpha"

Here we have a input called cluster bound to the choria/kv/cluster key in Consul. This starts life as a static input and if you give this value on the CLI it will never use the Data Source.

If however you do not specify a CLI value it becomes dynamic and will consult Consul. If there’s no such key in Consul the default is used, but the input remains dynamic and will continue to consult Consul on every access.

You can force an input to be dynamic which will mean it will not show up on the CLI and will only speak to a data source using the dynamic: true property on the Input.

Writing and Deleting Data

Of course if you can read data you should be able to write and delete it, I’ve added tasks to let you do this:

  - pb_data
    description: "Cluster to deploy"
    type: "String"
    required: true
    data: "pb_data/choria/kv/cluster"
    default: "alpha"
    validation: ":shellsafe"
    - data:
        action: "delete"
        key: "pb_data/choria/kv/cluster"
  - shell:
      description: Deploy to cluster {{{ inputs.cluster }}}
      command: /path/to/script --cluster {{{ inputs.cluster }}}
  - data:
      action: "write"
      value: "bravo"
      key: "pb_data/choria/kv/cluster"
  - shell:
      description: Deploy to cluster {{{ inputs.cluster }}}
      command: /path/to/script --cluster {{{ inputs.cluster }}}

Here I have a pre_book task list that ensures there is no stale data, the lock ensures no other Playbook will mess around with the data while we run.

I then run a shell command that uses the cluster input, with nothing there it uses the default and so deploys cluster alpha, it then writes a new value and deploys cluster brova.

This is a bit verbose I hope to add the ability to have arbitrarily named tasks lists that you can branch to, then you can have 1 deploy task list and use the main task list to set up variables for it and call it repeatedly.


That’s quite a mouthful, the possibilities of this is quite amazing. On one hand we have a really versatile data store in the Playbooks but more significantly we have expanded the integration possibilities by quite a bit, you can now have other systems manage the environment your playbooks run in.

I will soon add task level locks and of course Node Set integration.

For now only Consul and Memory is supported, I can add others if there is demand.

by R.I. Pienaar at January 23, 2017 09:54 PM

A quick guide to encrypting an external drive

luks-logoI am guilty for not having considered encrypting my hard drives for too long, I confess. As soon as I joined Telenor Digital (or, actually, early in the process but a bit too late…) I was commanded to encrypt my data and I couldn’t delay any more. To my utter surprise, the process was surprisingly simple in my Debian jessie! Here is a short checklist for your convenience.

Say you have an external hard drive with one partition attached to the system as /dev/sdb1. To encrypt the filesystem you will:

  • ensure that the partition is not mounted;
  • install cryptsetup;
  • run cryptsetup -y luksFormat /dev/sdb1 – that will ask you for a passphrase (twice to verify that you didn’t mistype) and initialize LUKS on the partition;
  • if now you run cryptsetup open /dev/sdb1 backupdisk you will unlock the encrypted partition by create a mapping between the device /dev/mapper/backupdisk and /dev/sdb1. You can think of the former as the interface to latter: the system will read and write to /dev/mapper/backupdisk like if it was a plain drive while, behind the scenes, it is reading and writing data to /dev/sdb1 that is decrypted/encrypted on the fly;
  • create a filesystem on the mapped drive: mkfs.ext4 /dev/mapper/backupdisk
  • and the drive is now ready for use! You can mount it: mount /dev/mapper/backupdisk /mnt
  • and do operations on it: ls -l /mnt
  • and unmount it when you are done: umount /mnt

When you are doing operations on the command line you are also supposed to close the drive when you are done, that is (in this specific case): remove the association between /dev/mapper/backupdisk and /dev/sdb1. But there is a nice bonus: if you are running GNOME or any other desktop environment that supports a keyring, the desktop environment can save your passphrase and open/close the drive automatically for you!

And that’s all for now. I hope to have some time shortly to write about how you can encrypt a partition on your system/laptop. Until then, enjoy!

Tagged: Debian, encryption, linux, LUKS, security, shell, Sysadmin

by bronto at January 23, 2017 07:00 AM

January 21, 2017

Cryptography Engineering

Zero Knowledge Proofs: An illustrated primer, Part 2

This post is the second in a two-part series on zero-knowledge proofs. Click here to read Part 1.

In this post I’m going to continue the short, (relatively) non-technical overview of zero knowledge proofs that I started a couple of years ago. Yes, that was a very long time! If you didn’t catch the first post, now would be an excellent time to go read it.

Before we go much further, a bit of a warning. While this series is still intended as a high-level overview, at a certain point it’s necessary to dig a bit deeper into some specific algorithms. So you should expect this post to get a bit wonkier than the last.

A quick recap, and a bit more on Zero Knowledge(ness)

First, a brief refresher.

In the last post we defined a zero knowledge proof as an interaction between two computer programs (or Turing machines) — respectively called a Prover and a Verifier — where the Prover works to convince the Verifier that some mathematical statement is true. We also covered a specific example: a clever protocol by Goldreich, Micali and Wigderson that allows us to prove, in zero knowledge, that a graph possesses a three-coloring.

In the course of that discussion, we described three critical properties that any zero knowledge proof must satisfy:

  • Completeness: If the Prover is honest, then she will eventually convince the Verifier.
  • Soundness: The Prover can only convince the Verifier if the statement is true.
  • Zero-knowledge(ness): The Verifier learns no information beyond the fact that the statement is true.

The real challenge turns out to be finding a way to formally define the last property. How do you state that a Verifier learns nothing beyond the truth of a statement?

In case you didn’t read the previous post — the answer to this question came from Goldwasser, Micali and Rackoff, and it’s very cool. What they argued is that a protocol can be proven zero knowledge if for every possible Verifier, you can demonstrate the existence of an algorithm called a ‘Simulator’, and show that this algorithm has some very special properties.

From a purely mechanical perspective, the Simulator is like a special kind of Prover. However, unlike a real Prover — which starts with some special knowledge that allows it to prove the truth of a statement — the Simulator gets no special knowledge at all.* Nonetheless, the Simulator (or Simulators) must be able to ‘fool’ every Verifier into believing that the statement is true, while producing a transcript that’s statistically identical top (or indistinguishable from) the output of a real Prover.

The logic here flows pretty cleanly: since Simulator has no ‘knowledge’ to extract in the first place, then clearly a Verifier can’t obtain any meaningful amount of information after interacting with it. Moreover, if the transcript of the interaction is distributed identically to a real protocol run with a normal Prover, then the Verifier can’t do better against the real prover than it can do against the Simulator. (If the Verifier could do better, then that would imply that the distributions were not statistically identical.) Ergo, the Verifier can’t extract useful information from the real protocol run.

This is incredibly wonky, and worse, it seems contradictory! We’re asking that a protocol be both sound — meaning that a bogus Prover can’t trick some Verifier into accepting a statement unless it has special knowledge allowing it to prove the statement — but we’re also asking for the existence of an algorithm (the simulator) that can literally cheat. Clearly both properties can’t hold at the same time.

The solution to this problem is that both properties don’t hold at the same time.

To build our simulator, we’re allowed to do things to the Verifier that would never happen in the real world. The example that I gave in the previous post was to use a ‘time machine’ — that is, our ‘Simulator’ can rewind the Verifier program’s execution in order to ‘fool’ it. Thus, in a world where we can wind the Verifier back in time, it’s easy to show that a Simulator exists. In the real world, of course it doesn’t. This ‘trick’ gets us around the contradiction.

As a last reminder, to illustrate all of these ideas, we covered one of the first general zero knowledge proofs, devised by Goldreich, Micali and Wigderson (GMW). That protocol allowed us to prove, in zero knowledge, that a graph supports a three-coloring. Of course, proving three colorings isn’t terribly interesting. The real significance of the GMW result is theoretical. Since graph three coloring is known to be in the complexity class NP-complete, the GMW protocol can be used to prove any statement in the class NP. And that’s quite powerful.

Let me elaborate slightly on what that means:

  1. If there exists any decision problem (that is, a problem with a yes/no answer) whose witness (solution) can be verified in polynomial time, then:
  2. We can prove that said solution exists by (1) translating the problem into an instance of the graph three-coloring problem, and (2) running the GMW protocol.*

This amazing result gives us interactive zero knowledge proofs for every statement in NP. The only problem is that it’s almost totally unusable.

From theory into practice

If you’re of a practical mindset, you’re probably shaking your head at all this talk of ZK proofs. That’s because actually using this approach would be an insanely expensive and stupid thing to do. Most likely you’d first represent your input problem as a boolean circuit where the circuit is satisfied if and only if you know the correct input. Then you’d have to translate your circuit into a graph, resulting in some further blowup. Finally you’d need to run the GMW protocol, which is damned expensive all by itself.

So in practice nobody does this. It’s really considered a ‘feasibility’ result. Once you show that something is possible, the next step is to make it efficient.

But we do use zero knowledge proofs, almost every day. In this post I’m going to spend some time talking about the more practical ZK proofs that we actually use. To do that I just need give just a tiny bit of extra background.

Proofs vs. Proofs of Knowledge

Before we go on, there’s one more concept we need to cover. Specifically, we need to discuss what precisely we’re proving when we conduct a zero knowledge proof.

Let me explain. At a high level, there are two kinds of statement you might want to prove in zero knowledge. Roughly speaking, these break up as follows.

Statements about “facts”. For example, I might wish to prove that “a specific graph has a three coloring” or “some number N is in the set of composite numbers“. Each of these is a statement about some intrinsic property of the universe.

Statements about my personal knowledge. Alternatively, I might wish to prove that I know some piece information. Examples of this kind of statement include: “I know a three coloring for this graph”, or “I know the factorization of N”. These go beyond merely proving that a fact is true, and actually rely on what the Prover knows.

It’s important to recognize that there’s a big difference between these two kinds of statements! For example, it may be possible to prove that a number N is composite even if you don’t know the full factorization. So merely proving the first statement is not equivalent to proving the second one.

The second class of proof is known as a “proof of knowledge”. It turns out to be extremely useful for proving a variety of statements that we use in real life. In this post, we’ll mostly be focusing on this kind of proof.

The Schnorr identification protocol

Now that we’ve covered some of the required background, it’s helpful to move on to a specific and very useful proof of knowledge that was invented by Claus-Peter Schnorr in the 1980s. At first glance, the Schnorr protocol may seem a bit odd, but in fact it’s the basis of many of our modern signature schemes today.

Schnorr wasn’t really concerned with digital signatures, however. His concern was with identification. Specifically, let’s imagine that Alice has published her public key to the world, and later on wants to prove that she knows the secret key corresponding to that public key. This is the exact problem that we encounter in real-world protocols such as public-key SSH, so it turns out to be well-motivated.

Schnorr began with the assumption that the public key would be of a very specific format. Specifically, let p be some prime number, and let g be a generator of a cyclic group of prime-order q. To generate a keypair, Alice would first pick a random integer a between 1 and q, and then compute the keypair as:

PK_{A} = g^a~mod~p, SK_{A} = a

(If you’ve been around the block a time or two, you’ll probably notice that this is the same type of key used for Diffie-Hellman and the DSA signing algorithm. That’s not a coincidence, and it makes this protocol very useful.)

Alice keeps her secret key to herself, but she’s free to publish her public key to the world. Later on, when she wants to prove knowledge of her secret key, she conducts the following simple interactive protocol with Bob:

There’s a lot going on in here, so let’s take a minute to unpack things.

First off, we should ask ourselves if the protocol is complete. This is usually the easiest property to verify: if Alice performs the protocol honestly, should Bob be satisfied at the end of it? In this case, completeness is pretty easy to see just by doing a bit of substitution:

Proving soundness

The harder property is soundness. Mainly because we don’t yet have a good definition of what it means for a proof of knowledge to be sound. Remember that what we want to show is the following:

If Alice successfully convinces Bob, then she must know the secret key a.

It’s easy to look at the equations above and try to convince yourself that Alice’s only way to cheat the protocol is to know a. But that’s hardly a proof.

When it comes to demonstrating the soundness of a proof of knowledge, we have a really nice formal approach. Just as with the Simulator we discussed above, we need to demonstrate the existence of a special algorithm. This algorithm is called a knowledge extractor, and it does exactly what it claims to. A knowledge extractor (or just ‘Extractor’ for short) is a special type of Verifier that interacts with a Prover, and — if the Prover succeeds in completing the proof — the Extractor should be able to extract the Prover’s original secret.

And this answers our question above. To prove soundness for a proof of knowledge, we must show that an Extractor exists for every possible Prover.

Of course this again seems totally contradictory to the purpose of a zero knowledge protocol — where we’re not supposed to be able to learn secrets from a Prover. Fortunately we’ve already resolved this conundrum once for the case of the Simulator. Here again, we take the same approach. The Extractor is not required to exist during a normal run of the protocol. We simply show that it exists if we’re allowed to take special liberties with the Prover — in this case, we’ll use ‘rewinding’ to wind back the Prover’s execution and allow us to extract secrets.

The extractor for the Schnorr protocol is extremely clever — and it’s also pretty simple. Let’s illustrate it in terms of a protocol diagram. Alice (the Prover) is on the left, and the Extractor is on the right:

The key observation here is that by rewinding Alice’s execution, the Extractor can ‘trick’ Alice into making two different proof transcripts using the same k. This shouldn’t normally happen in a real protocol run, where Alice specifically picks a new k for each execution of the protocol.

If the Extractor can trick Alice into doing this, then he can solve the following simple equation to recover Alice’s secret:

It’s worth taking a moment right now to note that this also implies a serious vulnerability in bad implementations of the Schnorr protocol. If you ever accidentally use the same k for two different runs of the protocol, an attacker may be able to recover your secret key! This can happen if you use a bad random number generator.

Indeed, those with a bit more experience will notice that this is similar to a real attack on systems (with bad random number generators) that implement ECDSA or DSA signatures! This is also not a coincidence. The (EC)DSA signature family is based on Schnorr. Ironically, the developers of DSA managed to retain this vulnerability of the Schorr family of protocols while at the same time ditching the security proof that makes Schnorr so nice.

Proving zero-knowledge(ness) against an honest Verifier

Having demonstrated that Schnorr signatures are complete and sound, it remains only to prove that they’re ‘zero knowledge’. Remember that to do this, normally we require a Simulator that can interact with any possible Verifier and produce a ‘simulated’ transcript of the proof, even if the Simulator doesn’t know the secret it’s proving it knows.

The standard Schnorr protocol does not have such a Simulator, for reasons we’ll get into in a second. Instead, to make the proof work we need to make a special assumption. Specifically, the Verifier needs to be ‘honest’. That is, we need to make the special assumption that it will run its part of the protocol correctly — namely, that it will pick its challenge “c” using only its random number generator, and will not choose this value based on any input we provide it. As long as it does this, we can construct a Simulator.

Here’s how the Simulator works.

Let’s say we are trying to prove knowledge of a secret a for some public key g^a~mod~pbut we don’t actually know the value aOur Simulator assumes that the Verifier will choose some value c as its challenge, and moreover, it knows that the honest Verifier will choose the value c only based on its random number generator — and not based on any inputs the Prover has provided.

  1. First, output some initial g^{k_1} as the Prover’s first message, and find out what challenge c the Verifier chooses.
  2. Rewind the Verifier, and pick a random integer z in the range \{0,\dots,q-1\}.
  3. Compute g^{k_2} = g^z * g^{a (-c)} and output g^{k_2} as the Prover’s new initial message.
  4. When the Verifier challenges on c again, output z.
Notice that the transcript g^k, c, z will verify correctly as a perfectly valid, well-distributed proof of knowledge of the value a. The Verifier will accept this output as a valid proof of knowledge of a, even though the Simulator does not know a in the first place!
What this proves is that if we can rewind a Verifier, then (just as in the first post in this series) we can always trick the Verifier into believing we have knowledge of a value, even when we don’t. And since the statistical distribution of our protocol is identical to the real protocol, this means that our protocol must be zero knowledge — against an honest Verifier.

From interactive to non-interactive

So far we’ve shown how to use the Schnorr protocol to interactively prove knowledge of a secret key a that corresponds to a public key g^{a}. This is an incredibly useful protocol, but it only works if our Verifier is online and willing to interact with us.

An obvious question is whether we can make this protocol work without interaction. Specifically, can I make a proof that I can send you without you even being online. Such a proof is called a non-interactive zero knowledge proof (NIZK). Turning Schnorr into a non-interactive proof seems initially quite difficult — since the protocol fundamentally relies on the Verifier picking a random challenge. Fortunately there is a clever trick we can use.

This technique was developed by Fiat and Shamir in the 1980s. What they observed was that if you have a decent hash function lying around, you can convert an interactive protocol into a non-interactive one by simply using the hash function to pick the challenge.

Specifically, the revised protocol for proving knowledge of a with respect to a public key g^a looks like this:

  1. The Prover picks g^k (just as in the interactive protocol).
  2. Now, the prover computes the challenge as c = H(g^k || M) where H() is a hash function, and M is an (optional) and arbitary message string.
  3. Compute ac + k~mod~q (just as in the interactive protocol).

The upshot here is that the hash function is picking the challenge c without any interaction with the Verifier. In principle, if the hash function is “strong enough” (meaning, it’s a random oracle) then the result is a completely non-interactive proof of knowledge of the value a that the Prover can send to the Verifier. The proof of this is relatively straightforward.

The particularly neat thing about this protocol is that it isn’t just a proof of knowledge, it’s also a signature scheme. That is, if you put a message into the (optional) value M, you obtain a signature on M, which can only be produced by someone who knows the secret key a. The resulting protocol is called the Schnorr signature scheme, and it’s the basis of real-world protocols like EdDSA.


Yes, this has been a long post and there’s probably a lot more to be said. Hopefully there will be more time for that in a third post — which should only take me another three years.

* In this definition, it’s necessary that the statement be literally true.

by Matthew Green at January 21, 2017 06:34 PM

January 20, 2017


What my CompSci degree got me

The what use is a csci degree meme has been going around again, so I thought I'd interrogate what mine got me.

First, a few notes on my career journey:

  1. Elected not to go to grad-school. Didn't have the math for a masters or doctorate.
  2. Got a job in helpdesk, intending to get into Operations.
  3. Got promoted into sysadmin work.
  4. Did some major scripting as part of Y2K remediation, first big coding project after school.
  5. Got a new job, at WWU.
  6. Microsoft released PowerShell.
  7. Performed a few more acts of scripting. Knew I so totally wasn't a software engineer.
  8. Manage to change career tracks into Linux. Started learning Ruby as a survival mechanism.
  9. Today: I write code every day. Still don't consider myself a 'software engineer'.

Elapsed time: 20ish years.

As it happens, even though my career has been ops-focused I still got a lot out of that degree. Here are the big points.

by SysAdmin1138 at January 20, 2017 10:04 PM

Simon Lyall 2017 – Friday – Closing

Code of Consult and Safety

  • Badge
    • Putting prefered pronoun
    • Emoji
  • Free Childcare
    • Sponsored by Github
    • Approx 10 kids
  • Assistance Grants
  • Attendees
    • Breakdown by gender etc
    • Roughly 25% of attendees and speakers not men
  • More numbers
    • 104 Matrix chat users
    • 554 attendees
    • 2900 coffee cups
    • Network claimed to 7.5Gb/s
    • 1.6 TB over the week, 200Mb/s max
    • 30 Session Chairs
    • 12 Miniconfs
    • 491 Proposals (130 more than the others)
    • 6 Tutorials, 75 talks, 80 speakers
    • 4 Keynote speakers
    • 21 Sponsors 2018 – Sydney

  • A little bit of history repeating
  • 2001, 2007, 2018
  • Venue is UTS
  • 5 minutes to food, train station
  • @lca2018 on twitter
  • Looking for a few extra helpers


  • In support of Outreachy
  • 3 interns funded

Final Bit

  • Thanks to team members




by simon at January 20, 2017 12:51 PM 2017 – Friday – Lightning Talks

Use #lcapapers to tell what you want to see in 2018

Michael Still and Michael Davies get the Rusty Wrench award

Karaoke – Jack Skinner

  • Talk with random slides

Martin Krafft

  • Matrix
  • End to end encrypted communication system
  • No entity owns your conversations
  • Bridge between walled gardens (eg IRC and Slack)
  • In Very late Beta, 450K user accounts
  • Run or Write your own servers or services or client

Cooked – Pete the Pirate

  • How to get into Sous Vide cooking
  • Create home kit
  • Beaglebone Black
  • Rice cooker, fish tank air pump.
  • Also use to germinate seeds
  • Also use this system to brew beer

Emoji Archeology 101 – Russell Keith-Magee

  • 1963 Happy face created
  • 🙂 invented
  • later 🙁 invented
  • Only those emotions imposed by the Unicode consortium can now be expressed

The NTPsec Project – Mark Atwood

  • Since 2014
  • For and git in 2015 from parent ntp project
  • 1.0.0 release soon
  • Removed 73% of lines from classic
    • Removed commandline tools
    • Got write of stuff for old OSes
    • Changed to POSIX and modern coding
    • removed experiments
  • Switch to git and bugzilla etc
  • Fun not painful
  • Welcoming community, not angry

National Computer Science Summer School – Katie Bell

  • Running for 22 years
  • Web stream, Embedded Stream
  • Using BBC Microbit
  • Lots of projects
  • Students in grade 10-11
  • Happens in January
  • Also 5 week long online programming competition NCSS Competition.

Blockchain – Rusty Russell

  • Blockchain
  • Blockchain
  • Blockchain

Go to Antarctica – Jucinter Richardson

  • Went Twice
  • Go by ship
  • No rain
  • Nice and cool
  • Join the government
  • Positions close
  • Go while it is still there

Cool and Awesome projects you should help with – Tim Ansell

  • Tomu Boards
  • MicroPython on FPGAs
  • Python Devicetree – needs a good library
  • QEMU for LiteX / MiSoC
  • NuttX for LiteX / MiSoC
  • QEMU for Tomu
  • Improving LiteX / MiSoc
  • Sypress FX2
  • Linux to LiteX / MiSoC

LoRa TAS – Paul Neumeyer

  • long range (2-3km urban 10km rural)
  • low power (batter ~5 years)
  • Unlicensed radio spectrum 915-928 Mhz BAnd (AUS)
  • LoRaWAN is an open standard
  • Ideal for IoT applications (sensing, preventative maintenance, smart)

Roan Kattatow

  • Different languages mix dots and commas and spaces etc to write numbers

ZeroSkip – Ron Gondwana

  • Crash safe embeded database
  • Not fast enough
  • Zeroskip
  • Append only database file
  • Switch files now and then
  • Repack old files togeather

PyCon Au – Richard Jones

  • Python Conference Australia
  • 7th in Melbourne in Aug 2016 – 650 people, 96 presentation
  • In Melb on 308 of August on 2016

Buying a Laptop built for Linux – Paul Wayper

  • Bought from System76
  • Designed for Linux

openQA – Aleksa Sarai

  • Life is too short for manual testing
  • Perl based framework that lets you emulate a user
  • Runs from console, emulates keyboard and mouse
  • Has screenshots
  • Used by SUSE and openSUSE and fedora
  • Fuzzy comparison, using regular expressions

South Coast Track – Bec, Clinton and Richard

  • What I did in the Holidays
  • 6 day walk in southern tasmania
  • Lots of pretty photos


by simon at January 20, 2017 06:02 AM 2017 – Friday – Session 2

Continuously Delivering Security in the Cloud – Casey West

  • This is a talk about operation excellence
  • Why are system attacked? Because they exist
  • Resisting Change to Mitigate Risk – It’s a trap!
  • You have a choice
    • Going fast with unbounded risk
    • Going slow to mitigate risk
  • Advanced Persistent Threat (ATP) – The breach that lasts for months
  • Successful attacks have
    • Time
    • Leaked or misused creditials
    • Miconfigured or unpatched software
  • Changing very little slowly helps all three of the above
  • A moving target is harder to hit
  • Cloud-native operability lets platforms move faster
    • Composable architecture (serverless, microservices)
    • Automated Processes (CD)
    • Collaborative Culture (DevOps)
    • Production Environment (Structured Platform)
  • The 3 Rs
    • Rotate
      • Rotate credentials every few minutes or hours
      • Credentials will leak, Humans are weak
      • “If a human being generates a password for you then you should reject it”
      • Computers should generate it, every few hours
    • Repave
      • Repave every server and application every few minutes/hours
      • Implies you have things like LBs that can handle servers adding and leaving
      • Container lifecycle
        • Built
        • Deploy
        • Run
        • Stop
        • Note: No “change “step
      • A Server that doesn’t exist isn’t being cromprimised
      • Regularly blow away running containers
      • Repave ≠ Patch
      • uptime <= 3600
    • Repair
      • Repair vulnerable runtime environments every few minutes or hours
      • What stuff will need repair?
        • Applications
        • Runtime Environments (eg rails)
        • Servers
        • Operating Systems
      • The Future of security is build pipelines
      • Try to put in credential rotation and upsteam imports into your builds
  • Embracing Change to Mitigate Risk
  • Less of a Trap (in the cloud)


by simon at January 20, 2017 03:53 AM

January 17, 2017

Sarah Allen

remaining awake through a revolution

This day we honor Martin Luther King, Jr. who eloquently described the change that was, and is still, happening in our society. He often referred to the dramatic changes in technology, alongside other changes which require actions from each of us to make happen.

This morning I listened to “Remaining Awake Through a Great Revolution,” a speech delivered by Dr. Martin Luther King, Jr. on March 31 1968. He spoke of a triple revolution with advances in technology, weapons, and human rights. He talks about how we as individuals must accept responsibility and create change, not just in our own behavior, but changing the institutions we are part of.

one of the great liabilities of life is that all too many people find themselves living amid a great period of social change, and yet they fail to develop the new attitudes, the new mental responses, that the new situation demands.

His introduction refers to the story of Rip Van Winkle. We all remember how he slept for 20 years, but I had forgotten exactly what he slept through. He went to sleep under the reign of King George and woke up when George Washington was President — he slept through the American revolution. This story is a apt metaphor for today’s political and social climate. If we don’t talk together about what is happening in our world and work together to make change, we are sleeping. In King’s words: “anyone who feels that he can live alone is sleeping through a revolution.”

Here are some highlights of this speech that are still true today, and inspire me to work towards kind of world where I want to live, that I believe is still possible:

Through our scientific and technological genius, we have made of this world a neighborhood and yet we have not had the ethical commitment to make of it a brotherhood. But somehow, and in some way, we have got to do this. We must all learn to live together as brothers or we will all perish together as fools. We are tied together in the single garment of destiny, caught in an inescapable network of mutuality. And whatever affects one directly affects all indirectly. For some strange reason I can never be what I ought to be until you are what you ought to be. And you can never be what you ought to be until I am what I ought to be. This is the way God’s universe is made; this is the way it is structured…

We need all of the talents and potential of the people in the world to solve the challenges that face us. Let’s look out for the individuals in our daily lives who aren’t getting the opportunities to rise to their potential.

It is an unhappy truth that racism is a way of life for the vast majority of white Americans, spoken and unspoken, acknowledged and denied, subtle and sometimes not so subtle—the disease of racism permeates and poisons a whole body politic. And I can see nothing more urgent than for America to work passionately and unrelentingly—to get rid of the disease of racism.

The opportunity to speak out against racism rises up without warning. I have found myself often unprepared to speak in the moment, and so have worked on practices which cause me to be mindful and take small, quiet actions in my daily life. I volunteer for Bridge Foundry, learning how to work with diverse teams, teaching what I’ve learned to make our tech culture more inclusive and welcoming to people who have traditionally been excluded. I’ve learned about history, so I can tell lesser-known stories, and try to pay attention to present-day voices that deserve to be amplified. Often when I’m about to share an article, I take a little extra time to look up the person who wrote it. I think about how this person’s experience and culture intersect with mine. I do a little more digging and read a few more articles and sometimes choose to share a different one. I enjoy finding new voices. I seek to be intentional about the people who influence me.

we have difficult days ahead in the struggle for justice and peace, but I will not yield to a politic of despair. I’m going to maintain hope… This time we will really confront a Goliath. God grant that we will be that David of truth set out against the Goliath of injustice, the Goliath of neglect, the Goliath of refusing to deal with the problems, and go on with the determination to make America the truly great America that it is called to be.

The world is changing, always. We need to work together, and I’m not just referring to a mass movement to curb injustice and stand up for what’s right (though I hope to be part of that). I believe we need to look for ways to work together as individuals, to speak up in the moment, to address the small injustices that we witness (and participate in) every day.

I don’t intend to appropriate the words of Dr. Martin Luther King. This speech was as much about peace, as it was about racial injustice. It is my hope that with this small blog post I might highlight how his teachings are still very applicable today. I hope someone will be inspired to read or listen to the whole original speech, and that everyone will be inspired to and feel obliged to create positive change in the world.

With this faith we will be able to hew out of the mountain of despair the stone of hope. With this faith we will be able to transform the jangling discords of our nation into a beautiful symphony of brotherhood.

The post remaining awake through a revolution appeared first on the evolving ultrasaurus.

by sarah at January 17, 2017 01:23 AM

January 09, 2017

That grumpy BSD guy

A New Year, a New Round of pop3 Gropers from China

They've got a list, and they're sticking to it. Do they even know or care it's my list of spamtraps?

Yes, the Chinese are at it again. Or rather, machines with IP addresses that belong in a small set of Chinese province networks have started a rather intense campaign of trying to access the pop3 mail retrieval protocol on a host in my care, after a longish interval of near-total inactivity.

This is the number of failed pop3 login attempts to my system per day so far in 2017:

January 1:        4
January 2:    145
January 3:      20
January 4:      51
January 5:      32
January 6:      36
January 7:  4036
January 8:  5956
January 9:  5769

Clearly, something happened on January 7th, and whatever started then has not stopped yet. On that day we see a large surge in failed pop3 logins, sustained over the next few days, and almost exclusively attempts at the username part of entries from my list of spamtrap addresses. Another notable feature of this sequence of attempts is that they come almost exclusively from a small set of Chinese networks.

The log of the failed attempts are in raw form here, while this spreadsheet summarises the log data in a format oriented to IP address and username pairs and attempts at each.  The spreadsheet also contains netblock information and the country or territory the range is registered to. (Note: when importing the .csv, please specify the "User name" colum as text, otherwise conversion magic may confuse matters)

The numbers for January 7th onwards would have been even higher had it not been for a few attempts to access accounts that actually exist, with my reaction to block (for 24 hours only) the entire netblock the whois info for the offending IP address. Some of those blocks were quite substantial. I've also taken the liberty of removing those entries with real usernames from the logs.

Now despite urging from friends to publish quickly, I've silently collected data for a few days (really just a continuation of the collecting that started with last year's episode described in the Chinese Hunting Chinese Over POP3 In Fjord Country article, which in turn contains links to the data that by now covers almost a full year).

Now disregarding the handful of real user IDs I've already removed from the data set, the only new user IDs we have seen this year are:


The rest were already in the spamtraps list, as user name parts. As you will have guessed, those two have been duly included there as well, with appended in order to form a somewhat believable spamtrap email address.

What, then can we expect to see over the next few days?

The progression so far has been proceeding from trap user names starting with 0, ascended through the numerics and have now (January 9) moved on to the early alphabetics. The list of spamtraps is just shy of 35,000 entries, and I assume the entries I see here come out of some larger corpus that our somewhat inept cyber-criminals use.

If you too are seeing larger than usual numbers of pop3 login failures and anything even vaguely resembling the patters of mischief described here, I would like to hear from you. If your logs follow a format somewhat resembling mine, it is most likely trivial to modify the scripts (in the same directories as the data) to extract data to the slightly more database- or spreadsheet-friendly CSV.

From my perch here it is difficult to determine whether the people responsible for the networks that feature prominently in the data are cooperating with the cybercriminals or whether they are simply victims of their own substandard security practices.

If you are in a position to shed light on that, I would like to hear from you, and I will do my best to protect your anonymity unless you specify otherwise.

In the meantime, expect the data, (both the full set starting in January 2016 and the 2017-only set) to be updated at frequent, quasi-random intervals.

If you need some background on what the spamtrap list is and some history of related incidents over the last few years, I can recommend reading these articles:

Hey, spammer! Here's a list for you! (July 2007) - a light introduction to greytrapping
Maintaining A Publicly Available Blacklist - Mechanisms And Principles (April 2013) - on greytrapping principles and practice
Effective Spam and Malware Countermeasures - Network Noise Reduction Using Free Tools (2008 - 2014) - a more thorough treatment of the whole spam and malware complex
Password Gropers Take the Spamtrap Bait (August 2014) - the first time I noticed pop3 logins for my spamtraps, and of course
Chinese Hunting Chinese Over POP3 In Fjord Country (August 2016) - about a previous bizarre episode involving Chinese networks and pop3 activity.

Update 2017-02-08: Another round of attempts at usernames that are likely Chinese user names (much like the June 19 2016 onwards cycle described in the Chinese Hunting Chinese Over POP3 In Fjord Country article) started on February 8th, 2017.

The first few hours brought the following user names, with the likely corresponding real life name in the second column:

Name Username
Luo Chun luochun
Luo Fa luofa
Luo Feng luofeng
Luo Hai luohai

These names have been added to the full data as well as the 2017-only portion. The log file (2016 and 2017 version or 2017-only data) contains the entries starting at Feb  8 15:26:45 (times are CET local time). It will be interesting to see how long this cycle lasts. Look for updates to the data at irregular but hopefully frequent intervals.

If you are seeing similar activity, I would like to hear from you, in comments or (these most recent attempts all originate in the network (range -, also known as  CHINANET-JS or the CHINANET jiangsu province network). The previous cycle involved several distinct Chinese networks, and as we all know, stretched over several months of low intensity activity.

by Peter N. M. Hansteen ( at January 09, 2017 08:17 PM

Sarah Allen

the danger of a single story

Chimamanda Ngozi Adichie’s the danger of a single story illustrates how we are influenced by the stories we read and the stories we tell.

Chimamanda Ngozi Adichie speaking

She introduces the talk telling about how reading British and American children’s books influenced her own childish writings that featured foreign landscapes and experiences, rather than drawing from her own life. I remembered how my mom pointed out to me years later the drawings I had made when we lived in Saint Lucia. All my houses had chimneys, even though we lived in this very hot, tropical climate with no fireplaces.

She tells about her experience of negative bias where well-meaning people made assumptions about Africa. She also shares how she inadvertently made assumptions about others based on a single narrative that excluded the possibility of other attributes and experience.

It is impossible to talk about the single story without talking about power. There is a word, an Igbo word, that I think about whenever I think about the power structures of the world, and it is “nkali.” It’s a noun that loosely translates to “to be greater than another.” Like our economic and political worlds, stories too are defined by the principle of nkali: How they are told, who tells them, when they’re told, how many stories are told, are really dependent on power… Power is the ability not just to tell the story of another person, but to make it the definitive story of that person.”

It resonates with me how the negative stories flatten her experience. The creation of stereotypes by such storytelling is problematic not because they are entirely false, but because they are incomplete. “They make one story become the only story.”

“The consequence of the single story is this: It robs people of dignity. It makes our recognition of our equal humanity difficult. It emphasizes how we are different rather than how we are similar… when we reject the single story, when we realize that there is never a single story about any place, we regain a kind of paradise.”

The post the danger of a single story appeared first on the evolving ultrasaurus.

by sarah at January 09, 2017 12:06 AM