Planet SysAdmin

March 24, 2017

Intent to Deprecate and Remove: Trust in existing Symantec-issued Certificates

The post Intent to Deprecate and Remove: Trust in existing Symantec-issued Certificates appeared first on

Another CA bites the dust, and it's not a small one this time: Symantec.

Since January 19, the Google Chrome team has been investigating a series of failures by Symantec Corporation to properly validate certificates.

Over the course of this investigation, the explanations provided by Symantec have revealed a continually increasing scope of misissuance with each set of questions from members of the Google Chrome team; an initial set of reportedly 127 certificates has expanded to include at least 30,000 certificates, issued over a period spanning several years.

This is also coupled with a series of failures following the previous set of misissued certificates from Symantec, causing us to no longer have confidence in the certificate issuance policies and practices of Symantec over the past several years.

Source: Intent to Deprecate and Remove: Trust in existing Symantec-issued Certificates -- Google Groups

The post Intent to Deprecate and Remove: Trust in existing Symantec-issued Certificates appeared first on

by Mattias Geniar at March 24, 2017 08:00 AM

Chris Siebenmann

ARM servers had better just work if vendors want to sell very many

A few years ago I wrote about the cost challenge facing hypothetical future ARM servers here; to attract our interest, they'd have to be cheaper or better than x86 servers in some way that we cared about. At the time I made what turns out to be a big assumption: I assumed that ARM servers would be like x86 servers in that they would all just work with Linux. Courtesy of Pete Zaitcev's Standards for ARM computers and Linaro and the follow-on Standards for ARM computers in 2017, I've now learned that this was a pretty optimistic assumption. The state of play in 2017 is that LWN can write an article called Making distributions Just Work on ARM servers that describes not current reality but an aspirational future that may perhaps arrive some day.

Well, you know, no wonder no one is trying to actually sell real ARM servers. They're useless to a lot of people right now. Certainly they'd be useless to us, because we don't want to buy servers with a bespoke boatloader that probably only works with one specific Linux distribution (the one the vendor has qualified) and may not work in the future. A basic prerequisite for us being interested in ARM servers is that they be as useful for generic Linux as x86 servers are (modulo which distributions have ARM versions at all). If we have to buy one sort of servers to run Ubuntu but another sort to run CentOS, well, no. We'll buy x86 servers instead because they're generic and we can even run OpenBSD on them.

There are undoubtedly people who work at a scale with a server density where things like the power advantages of ARM might be attractive enough to overcome this. These people might even be willing to fund their own bootloader and distribution work. But I think that there are a lot of people who are in our situation; we wouldn't mind extra-cheap servers to run Linux, but we aren't all that interested in buying servers that might as well be emblazoned 'Ubuntu 16.04 only' or 'CentOS only' or the like.

I guess this means I can tune out all talk of ARM servers for Linux for the next few years. If the BIOS-level standards for ARM servers for Linux are only being created now, it'll be at least that long until there's real hardware implementing workable versions of them that isn't on the bleeding edge. I wouldn't be surprised if it takes half a decade before we get ARM servers that are basically plug and play with your choice of a variety of Linux distributions.

(I don't blame ARM or anyone for this situation, even though it sort of boggles me. Sure, it's not a great one, but the mere fact that it exists means that ARM vendors haven't particularly cared about the server market so far (and may still not). It's hard to blame people for not catering to a market that they don't care about, especially when we might not care about it either when the dust settles.)

by cks at March 24, 2017 03:15 AM

March 23, 2017

Carl Chenet

Feed2tweet 1.0, tool to post RSS feeds to Twitter, released

Feed2tweet 1.0, a self-hosted Python app to automatically post RSS feeds to the Twitter social network, was released March 2017, 23th.

The main new feature of this release allows to create filters for each RSS feed, because before you could only define global filters. Contributed by Antoine Beaupré, Feed2tweet is also able to use syslog, starting from this release.

fiestaWhat’s the purpose of Feed2tweet?

Some online services offer to convert your RSS entries into Twitter posts. Theses services are usually not reliable, slow and don’t respect your privacy. Feed2tweet is Python self-hosted app, the source code is easy to read and you can enjoy the official documentation online with lots of examples.

Twitter Out Of The Browser

Have a look at my Github account for my other Twitter automation tools:

  • Retweet , retweet all (or using some filters) tweets from a Twitter account to another one to spread content.
  • db2twitter, get data from SQL database (several supported), build tweets and send them to Twitter
  • Twitterwatch, monitor the activity of your Twitter timeline and warn you if no new tweet appears

What about you? Do you use tools to automate the management of your Twitter account? Feel free to give me feedback in the comments below.

… and finally

You can help Feed2tweet by donating anything through Liberaypay (also possible with cryptocurrencies). That’s a big factor motivation 😉


by Carl Chenet at March 23, 2017 11:00 PM


Five Reasons I Want China Running Its Own Software

Periodically I read about efforts by China, or Russia, or North Korea, or other countries to replace American software with indigenous or semi-indigenous alternatives. I then reply via Twitter that I love the idea, with a short reason why. This post will list the top five reasons why I want China and other likely targets of American foreign intelligence collection to run their own software.

1. Many (most?) non-US software companies write lousy code. The US is by no means perfect, but our developers and processes generally appear to be superior to foreign indigenous efforts. Cisco vs Huawei is a good example. Cisco has plenty of problems, but it has processes in place to manage them, plus secure code development practices. Lousy indigenous code means it is easier for American intelligence agencies to penetrate foreign targets. (An example of a foreign country that excels in writing code is Israel, but thankfully it is not the same sort of priority target like China, Russia, or North Korea.)

2. Many (most?) non-US enterprises are 5-10 years behind US security practices. Even if a foreign target runs decent native code, the IT processes maintaining that code are lagging compared to American counterparts. Again, the US has not solved this problem by any stretch of the imagination. However, relatively speaking, American inventory management, patch management, and security operations have the edge over foreign intelligence targets. Because non-US enterprises running indigenous code will not necessarily be able to benefit from American expertise (as they might if they were running American code), these deficiencies will make them easier targets for foreign exploitation.

3. Foreign targets running foreign code is win-win for American intel and enterprises. The current vulnerability equities process (VEP) puts American intelligence agencies in a quandary. The IC develops a zero-day exploit for a vulnerability, say for use against Cisco routers. American and Chinese organizations use Cisco routers. Should the IC sit on the vulnerability in order to maintain access to foreign targets, or should it release the vulnerability to Cisco to enable patching and thereby protect American and foreign systems?

This dilemma disappears in a world where foreign targets run indigenous software. If the IC identifies a vulnerability in Cisco software, and the majority of its targets run non-Cisco software, then the IC is more likely (or should be pushed to be more likely) to assist with patching the vulnerable software. Meanwhile, the IC continues to exploit Huawei or other products at its leisure.

4. Writing and running indigenous code is the fastest way to improve. When foreign countries essentially outsource their IT to vendors, they become program managers. They lose or never develop any ability to write and run quality software. Writing and running your own code will enroll foreign organizations in the security school of hard knocks. American intel will have a field day for 3-5 years against these targets, as they flail around in a perpetual state of compromise. However, if they devote the proper native resources and attention, they will learn from their mistakes. They will write and run better software. Now, this means they will become harder targets for American intel, but American intel will retain the advantage of point 3.

5. Trustworthy indigenous code will promote international stability. Countries like China feel especially vulnerable to American exploitation. They have every reason to be scared. They run code written by other organizations. They don't patch it or manage it well. Their security operations stink. The American intel community could initiate a complete moratorium on hacking China, and the Chinese would still be ravaged by other countries or criminal hackers, all the while likely blaming American intel. They would not be able to assess the situation. This makes for a very unstable situation.

Therefore, countries like China and others are going down the indigenous software path. They understand that software, not oil as Daniel Yergen once wrote, is now the "commanding heights" of the economy. Pursuing this course will subject these countries to many years of pain. However, in the end I believe it will yield a more stable situation. These countries should begin to perceive that they are less vulnerable. They will experience their own vulnerability equity process. They will be more aware and less paranoid.

In this respect, indigenous software is a win for global politics. The losers, of course, are global software companies. Foreign countries will continue to make short-term deals to suck intellectual property and expertise from American software companies, before discarding them on the side of Al Gore's information highway.

One final point -- a way foreign companies could jump-start their indigenous efforts would be to leverage open source software. I doubt they would necessarily honor licenses which require sharing improvements with the open source community. However, open source would give foreign organizations the visibility they need and access to expertise that they lack. Microsoft's shared source and similar programs were a step in this direction, but I suggest foreign organizations adopt open source instead.

Now, widespread open source adoption by foreign intelligence targets would erode the advantages for American intel that I explained in point 3. I'm betting that foreign leaders are likely similar to Americans in that they tend to not trust open source, and prefer to roll their own and hold vendors accountable. Therefore I'm not that worried, from an American intel perspective, about point 3 being vastly eroded by widespread foreign open source adoption.

TeePublic is running a sale until midnight ET Thursday! Get a TaoSecurity Milnet T-shirt for yourself and a friend!

by Richard Bejtlich ( at March 23, 2017 02:26 PM

The Missing Trends in M-Trends 2017

FireEye released the 2017 edition of the Mandiant M-Trends report yesterday. I've been a fan of this report since the 2010 edition, before I worked at the company.

Curiously for a report with the name "trends" in the title, this and all other editions do not publish the sorts of yearly trends I would expect. This post will address that limitation.

The report is most famous for its "dwell time" metric, which is the median (not average, or "mean") number of days an intruder spends inside a target company until he is discovered.

Each report lists the statistic for the year in consideration, and compares it to the previous year. For example, the 2017 report, covering incidents from 2016, notes the dwell time has dropped from 146 days in 2015, to 99 days in 2016.

The second most interesting metric (for me) is the split between internal and external notification. Internal notification means that the target organization found the intrusion on its own. External notification means that someone else informed the target organization. The external party is often a law enforcement or intelligence agency, or a managed security services provider. The 2016 split was 53% internal vs 47% external.

How do these numbers look over the years that the M-Trends report has been published? Inquiring minds want to know.

The 2012 M-Trends report was the first edition to include these statistics. I have included them for that report and all subsequent editions in the table below.

Year Days Internal External
2011 416 6 94
2012 243 37 63
2013 229 33 67
2014 205 31 69
2015 146 47 53
2016 99 53 47

As you can see, all of the numbers are heading in the right direction. We are finally into double digits for dwell time, but over 3 months is still far too long. Internal detection continues to rise as well. This is a proxy for the maturity of a security organization, in my opinion.

Hopefully future M-Trends reports will include tables like this.

by Richard Bejtlich ( at March 23, 2017 12:43 PM

Chris Siebenmann

Setting the root login's 'full name' to identify the machine that sent email

Yesterday I wrote about making sure you can identify what machine sent you a status email, and the comments Sotiris Tsimbonis shared a brilliant yet simple solution to this problem:

We change the gecos info for this purpose.

chfn -f "$HOSTNAME root" root

Take it from me; this is beautiful genius (so much so that both we and another group here immediately adopted it). It's so simple yet still extremely effective, because almost everything that sends email does so using programs like mail that will fill out the From: header using the login's GECOS full name from /etc/passwd. You get email that looks like:

From: root@<your-domain> (hebikera root)

This does exactly what we want by immediately showing the machine that the email is from. In fact many mail clients these days will show you only the 'real name' from the From: header by default, not the actual email address (I'm old-fashioned, so I see the traditional full From: header).

This likely works with any mail-sending program that doesn't require completely filled out email headers. It definitely works in the Postfix sendmail cover program for 'sendmail -t' (as well as the CentOS 6 and 7 mailx, which supplies the standard mail command).

(As an obvious corollary, you can also use this trick for any other machine-specific accounts that send email; just give them an appropriate GECOS 'full name' as well.)

There's two perhaps obvious cautions here. First, if you ever rename machines you have to remember to re-chfn the root login and any other such logins to have the correct hostname in them. It's probably worth creating an officially documented procedure for renaming machines, since there are other things you'll want to update as well (you might even script it). Second, if you have some sort of password synchronization system you need it to leave root's GECOS full name alone (although it can update root's password). Fortunately ours already does this.

by cks at March 23, 2017 03:50 AM

March 22, 2017

Chris Siebenmann

Making sure you can identify what machine sent you a status email

I wrote before about making sure that system email works, so that machines can do important things like tell you that their RAID array has lost redundancy and you should do something about that. In a comment on that entry, -dsr- brought up an important point, which is you want to be able to easily tell which machine sent you email.

In an ideal world, everything on every machine that sends out email reports would put the machine's hostname in, say, the Subject: header. This would give you reports like:

Subject: SMART error (FailedOpenDevice) detected on host: urd

In the real world you also get helpful emails like this:

Subject: Health

Device: /dev/sdn [SAT], FAILED SMART self-check. BACK UP DATA NOW!

The only way for us to tell which machine this came from was to look at either the Received: headers or the Message-ID, which is annoying.

There are at least two ways to achieve this. The first approach is what -dsr- said in the comment, which is to make every machine send its email to a unique alias on your system. This unfortunately has at least two limitations. The first is that it somewhat clashes with a true 'null client' setup, where your machines dump absolutely all of their email on the server. A straightforward null client does no local rewriting of email at all, so to get this you need a smarter local mailer (and then you may need per-machine setup, hopefully automated). The second limitation is that there's no guarantee that all of the machine's email will be sent to root (and thus be subject to simple rewriting). It's at least likely, but machines have been known to send status email to all sorts of addresses.

(I'm going to assume that you can arrange for the unique destination alias to be visible in the To: header.)

You can somewhat get around this by doing some of the rewriting on your central mail handler machine (assuming that you can tell the machine email apart from regular user email, which you probably want to do anyways). This needs a relatively sophisticated configuration, but it probably can be done in something like Exim (which has quite powerful rewrite rules).

However, if you're going to do this sort of magic in your central mail handler machine, you might as well do somewhat different magic and alter the Subject: header of such email to include the host name. For instance, you might just add a general rule to your mailer so that all email from root that's going to root will have its Subject: altered to add the sending machine's hostname, eg 'Subject: [$HOSTNAME] ....'. Your central mail handler already knows what machine it received the email from (the information went into the Received header, for example). You could be more selective, for instance if you know that certain machines are problem sources (like the CentOS 7 machine that generated my second example) while others use software that already puts the hostname in (such as the Ubuntu machine that generated my first example).

I'm actually more attracted to the second approach than the first one. Sure, it's a big hammer and a bit crude, but it creates the easy to see marker of the source machine that I want (and it's a change we only have to make to one central machine). I'd feel differently if we routinely got status emails from various machines that we just filed away (in which case the alias-based approach would give us easy per-machine filing), but in practice our machines only email us occasionally and it's always going to be something that goes to our inboxes and probably needs to be dealt with.

by cks at March 22, 2017 05:12 AM

March 21, 2017


Choria Network Protocols – Transport

The old MCollective protocols are now ancient and was quite Ruby slanted – full of Symbols and used YAML and quite language specific – in Choria I’d like to support other Programming Languages, REST gateways and so forth, so a rethink was needed.

I’ll look at the basic transport protocol used by the Choria NATS connector, usually it’s quite unusual to speak of Network Protocols when dealing with messages on a broker but really for MCollective it is exactly that – a Network Protocol.

The messages need enough information for strong AAA, they need to have an agreed on security structure and within them live things like RPC requests. So a formal specification is needed which is exactly what a Protocol is.

While creating Choria the entire protocol stack has been redesigned on every level except the core MCollective messages – Choria maintains a small compatibility layer to make things work. To really achieve my goal I’d need to downgrade MCollective to pure JSON data at which point multi language interop should be possible and easy.

Networks are Onions

Network protocols tend to come in layers, one protocol within another within another. The nearer you go to the transport the more generic it gets. This is true for HTTP within TCP within IP within Ethernet and likewise it’s true for MCollective.

Just like for TCP/IP and HTTP+FTP one MCollective network can carry many protocols like the RPC one, a typical MCollective install uses 2 protocols at this inner most layer. You can even make your own, the entire RPC system is a plugin!

( middleware protocol
  ( transport packet that travels over the middleware
      ( security plugin internal representation
        ( mcollective core representation that becomes M::Message
          ( MCollective Core Message )
          ( RPC Request, RPC Reply )
          ( Other Protocols, .... )

Here you can see when you do mco rpc puppet status you’ll be creating a RPC Request wrapped in a MCollective Message, wrapped in a structure the Security Plugin dictates, wrapped in a structure the Connector Plugin dictates and from there to your middleware like NATS.

Today I’ll look at the Transport Packet since that is where Network Federation lives which I spoke about yesterday.

Transport Layer

The Transport Layer packets are unauthenticated and unsigned, for MCollective security happens in the packet carried within the transport so this is fine. It’s not inconceivable that a Federation might only want to route signed messages and it’s quite easy to add later if needed.

Of course the NATS daemons will only accept TLS connections from certificates signed by the CA so these network packets are encrypted and access to the transport medium is restricted, but the JSON data you’ll see below is sent as is.

In all the messages shown below you’ll see a seen-by header, this is a feature of the NATS Connector Plugin that records the connected NATS broker, we’ll soon expose this information to MCollective API clients so we can make a traceroute tool for Federations. This header is optional and off by default though.

I’ll show messages in Ruby format here but it’s all JSON on the wire.

Message Targets

First it’s worth knowing where things are sent on the NATS clusters. The targets used by the NATS connector is pretty simple stuff, there will no doubt be scope for improvement once I look to support NATS Streaming but for now this is adequate.

  • Broadcast Request for agent puppet in the mycorp sub collective – mycorp.broadcast.agent.puppet
  • Directed Request to a node for any agent in the mycorp sub collective –
  • Reply to a node identity with pid 9999 and a message sequence of

As the Federation Brokers are independent of Sub Collectives they are not prefixed with any collective specific token:

  • Requests from a Federation Client to a Federation Broker Cluster called productionchoria.federation.production.federation queue group production_federation
  • Replies from the Collective to a Federation Broker Cluster called productionchoria.federation.production.collective queue group production_collective
  • production cluster Federation Broker Instances publishes statistics – choria.federation.production.stats

These names are designed so that in smaller setups or in development you could use a single NATS cluster with Federation Brokers between standalone collectives. Not really a recommended thing but it helps in development.

Unfederated Messages

Your basic Unfederated Message is pretty simple:

  "data" => "... any text ...",
  "headers" => {
    "mc_sender" => "",
    "seen-by" => ["", ""],
    "reply-to" => "",
  • it’s is a discovery request within the sub collective mcollective and would be published to mcollective.broadcast.agent.discovery.
  • it is sent from a machine identifying as
  • we know it’s traveled via a NATS broker called
  • responses to this message needs to travel via NATS using the target

The data is completely unstructured as far as this message is concerned it just needs to be some text, so base64 encoded is common. All the transport care for is getting this data to its destination with metadata attached, it does not care what’s in the data.

The reply to this message is almost identical:

  "data" => "... any text ...",
  "headers" => {
    "mc_sender" => "",
    "seen-by" => ["", "", "", ""],

This reply will travel via, we know that the node is connected to

We can create a full traceroute like output with this which would show -> -> ->

Federated Messages

Federation is possible because MCollective will just store whatever Headers are in the message and put them back on the way out in any new replies. Given this we can embed all the federation metadata and this metadata travels along with each individual message – so the Federation Brokers can be entirely stateless, all the needed state lives with the messages.

With Federation Brokers being clusters this means your message request might flow over a cluster member a but the reply can come via b – and if it’s a stream of replies they will be load balanced by the members. The Federation Broker Instances do not need something like Consul or shared store since all the data needed is in the messages.

Lets look at the same Request as earlier if the client was configured to belong to a Federation with a network called production as one of its members. It’s identical to before except the federation structure was added:

  "data" => "... any text ...",
  "headers" => {
    "mc_sender" => "",
    "seen-by" => ["", ""],
    "reply-to" => "",
    "federation" => {
       "req" => "68b329da9893e34099c7d8ad5cb9c940",
       "target" => ["mcollective.broadcast.agent.discovery"]
  • it’s is a discovery request within the sub collective mcollective and would be published via a Federation Broker Cluster called production via NATS choria.federation.production.federation.
  • it is sent from a machine identifying as
  • it’s traveled via a NATS broker called
  • responses to this message needs to travel via NATS using the target
  • it’s federated and the client wants the Federation Broker to deliver it to it’s connected Member Collective on mcollective.broadcast.agent.discovery

The Federation Broker receives this and creates a new message that it publishes on it’s Member Collective:

  "data" => "... any text ...",
  "headers" => {
    "mc_sender" => "",
    "seen-by" => [
    "reply-to" => "choria.federation.production.collective",
    "federation" => {
       "req" => "68b329da9893e34099c7d8ad5cb9c940",
       "reply-to" => ""

This is the same message as above, the Federation Broker recorded itself and it’s connected NATS server and produced a message, but in this message it intercepts the replies and tell the nodes to send them to choria.federation.production.collective and it records the original reply destination in the federation header.

A node that replies produce a reply, again this is very similar to the earlier reply except the federation header is coming back exactly as it was sent:

  "data" => "... any text ...",
  "headers" => {
    "mc_sender" => "",
    "seen-by" => [
    "federation" => {
       "req" => "68b329da9893e34099c7d8ad5cb9c940",
       "reply-to" => ""

We know this node was connected to and you can see the Federation Broker would know how to publish this to the client – the reply-to is exactly what the Client initially requested, so it creates:

  "data" => "... any text ...",
  "headers" => {
    "mc_sender" => "",
    "seen-by" => [

Which gets published to

Route Records

You noticed above there’s a seen-by header, this is something entirely new and never before done in MCollective – and entirely optional and off by default. I anticipate you’d want to run with this off most of the time once your setup is done, it’s a debugging aid.

As NATS is a full mesh your message probably only goes one hop within the Mesh. So if you record the connected server you publish into and the connected server your message entered it’s destination from you have a full route recorded.

The Federation Broker logs and MCollective Client and Server logs all include the message ID so you can do a full trace in message packets and logs.

There’s a PR against MCollective to expose this header to the client code so I will add something like mco federation trace which would send a round trip to that node and tell you exactly how the packet travelled. This should help a lot in debugging your setups as they will now become quite complex.

The structure here is kind of meh and I will probably improve on it once the PR in MCollective lands and I can see what is the minimum needed to do a full trace.

By default I’ll probably record the identities of the MCollective bits when Federated and not at all when not Federated. But if you enable the setting to record the full route it will produce a record of MCollective bits and the NATS nodes involved.

In the end though from the Federation example we can infer a network like this:

Federation NATS Cluster

  • Federation Broker production_a ->
  • Federation Broker production_b ->
  • Client ->

Production NATS Cluster:

  • Federation Broker production_a ->
  • Federation Broker production_b ->
  • Server ->

We don’t know the details of all the individual NATS nodes that makes up the entire NATS mesh but this is good enough.

Of course this sample is the pathological case where nothing is connected to the same NATS instances anywhere. In my tests with a setup like this the overhead added across 10 000 round trips against 3 nodes – so 30 000 replies through 2 x Federation Brokers – was only 2 seconds, I couldn’t reliably measure a per message overhead as it was just too small.

The NATS gem do expose the details of the full mesh though since NATS will announce it’s cluster members to clients, I might do something with that not sure. Either way, auto generated network maps should be totally possible.


So this is how Network Federation works in Choria. It’s particularly nice that I was able to do this without needing any state on the cluster thanks to past self making good design decisions in MCollective.

Once the seen-by thing is figured out I’ll publish JSON Schemas for these messages and declare protocol versions.

I can probably make future posts about the other message formats but they’re a bit nasty as MCollective itself is not yet JSON safe, the plan is it would become JSON safe one day and the whole thing will become a lot more elegant. If someone pings me for this I’ll post it otherwise I’ll probably stop here.

by R.I. Pienaar at March 21, 2017 10:15 PM


Cybersecurity Domains Mind Map

Last month I retweeted an image labelled "The Map of Cybersecurity Domains (v1.0)". I liked the way this graphic divided "security" into various specialties. At the time I did not do any research to identify the originator of the graphic.

Last night before my Brazilian Jiu-Jitsu class I heard some of the guys talking about certifications. They were all interested in "cybersecurity" but did not know how to break into the field. The domain image came to mind as I mentioned that I had some experience in the field. I also remembered an article Brian Krebs asked me to write titled "How to Break Into Security, Bejtlich Edition," part of a series on that theme. I wrote:

Providing advice on “getting started in digital security” is similar to providing advice on “getting started in medicine.” If you ask a neurosurgeon he or she may propose some sort of experiment with dead frog legs and batteries. If you ask a dermatologist you might get advice on protection from the sun whenever you go outside. Asking a “security person” will likewise result in many different responses, depending on the individual’s background and tastes.

I offered to help the guys in my BJJ class find the area of security that interests them and get started in that space. I thought the domains graphic might facilitate that conversation, so I decided to identify the originator so as to give proper credit.

It turns out that that CISO at Oppenheimer & Co, Henry Jiang, created the domains graphic. Last month at LinkedIn he published an updated Map of Cybersecurity Domains v2.0:

Map of Cybersecurity Domains v2.0 by Henry Jiang
If I could suggest a few changes for an updated version, I would try to put related disciplines closer to each other. For example, I would put the Threat Intelligence section right next to Security Operations. I would also swap the locations of Risk Assessment and Governance. Governance is closer to the Framework and Standard arena. I would also move User Education to be near Career Development, since both deal with people.

On a more substantive level, I am not comfortable with the Risk Assessment section. Blue Team and Red Team are not derivatives of a Penetration test, for example. I'm not sure how to rebuild that section.

These are minor issues overall. The main reason I like this graphic is that it largely captures the various disciplines one encounters in "cybersecurity." I could point a newcomer to the field at this image and ask "does any of this look interesting?" I could ask someone more experienced "in which areas have your worked?" or "in which areas would you like to work?"

The cropped image at the top of this blog shows the Security Operations and Threat Intelligence areas, where I have the most experience. Another security person could easily select a completely different section and still be considered a veteran. Our field is no longer defined by a small set of skills!

What do you think of this diagram? What changes would you make?

by Richard Bejtlich ( at March 21, 2017 01:17 PM

March 20, 2017


Choria Network Federation

Running large or distributed MCollective networks have always been a pain. As much as Middleware is an enabler it starts actively working against you as you grow and as latency increases, this is felt especially when you have geographically distributed networks.

Federation has been discussed often in the past but nothing ever happened, NATS ended up forcing my hand because it only supports a full mesh mode. Something that would not be suitable for a globe spanning network.


I spent the last week or two building in Federation first into the Choria network protocol and later added a Federation Broker. Federation can be used to connect entirely separate collectives together into one from the perspective of a client.

Here we can see a distributed Federation of Collectives. Effectively London, Tokyo and New York are entirely standalone collectives. They are smaller, they have their own middleware infrastructure, they even function just like a normal collective and can have clients communicating with those isolated collectives like always.

I set up 5 node NATS meshes in every region. We then add a Federation Broker cluster that provide bridging services to a central Federation network. I’d suggest running the Federation Broker Cluster one instance on each of your NATS nodes, but you can run as many as you like.

Correctly configured Clients that connect to the central Federation network will interact with all the isolated collectives as if they are one. All current MCollective features keep working and Sub Collectives can span the entire Federation.


There are obvious advantages in large networks – instead of one giant 100 000 node middleware you now need to built 10 x 10 000 node networks, something that is a lot easier to do. With NATS, it’s more or less trivial.

Not so obvious is how this scales wrt MCollective. MCollective has a mode called Direct Addressing where the client would need to create 1 message for every node targeted in the request. Generally very large requests are discouraged so it works ok.

These requests being made on the client ends up having to travel individually all across the globe and this is where it starts to hurt.

With Federation the client will divide the task of producing these per client messages into groups of 200 and pass the request to the Federation Broker Cluster. The cluster will then, in a load shared fashion, do the work for the client.

Since the Federation Broker tends to be near the individual Collectives this yields a massive reduction in client work and network traffic. The Federation Broker Instances are entirely state free so you can run as many as you like and they will share the workload more or less evenly across them.

$ mco federation observe --cluster production
Federation Broker: production
    Received: 1024  Sent: 12288
    1: Received: 533 (52.1%) Sent: 6192 (50.4%)
    2: Received: 491 (47.9%) Sent: 6096 (49.6%)

Above you can see the client offloading the work onto a Federation Broker with 2 cluster members. The client sent 1024 messages but the broker sent 12288 messages on the clients behalf. The 2 instances does a reasonable job of sharing the load of creating and federating the messages across them.

In my tests against large collectives this speeds up the request significantly and greatly reduce the client load.

In the simple broadcast case there is no speed up, but when doing 10 000 requests in a loop the overhead of Federation was about 2 seconds over the 10 000 requests – so hardly noticeable.

Future Direction

The Choria protocol supports Federation in a way that is not tied to its specific Federation Broker implementation. The basic POC Federation Broker was around 200 lines so not really a great challenge to write.

I imagine in time we might see a few options here:

  • You can use different CAs in various places in your Federated network. The Federation Broker using Choria Security privileged certificates can provide user id mapping and rewriting between the Collectives
  • If you want to build a SaaS management services ontop of Choria a Federated network makes a really safe way to reach into managed networks without exposing the collectives to each other in any way. A client in one member Collective cannot use the Federation Brokers to access another Collective.
  • Custom RBAC and Auditing schemes can be built at the Federation Broker layer where the requests can be introspected and only ones matching policy are passed to the managed Collective
  • Federation is tailor made to provide Protocol translation. Different protocol Collectives can be bridged together. An older MCollective SSL based collective can be reached from a Choria collective via a Federation Broker providing translation capabilities. Ditto a Websocket interface to Collectives can be a Federation Broker listening on Websocket while speaking NATS on the other end.

The security implications are huge, isolated collectives with isolated CAs and unique user Auditing, Authorization and Authentication needs bridged together via a custom RBAC layer, one that is horizontally scalable and stateless is quite a big deal.

Protocol translation is equally massive, as I move towards looking at ways to fork MCollective, given the lack of cooperation from Puppet Inc, this gives me a very solid way forward to not throw away peoples investment in older MCollective while wishing to find a way to move forward.


This will be released in version 0.0.25 of the Choria module which should be sometime this week. I’ve published pre-release docs already. Expect it to be deployable with very little effort via Puppet, given a good DNS setup it needs almost no configuration at all.

I’ll make a follow up post that explores the network protocol that made this possible to build with zero stored state in the Federation Broker Instances – a major achievement in my book.

UPDATE: All the gory protocol details are in a follow up post Choria Network Protocols – Transport.

by R.I. Pienaar at March 20, 2017 11:05 AM

March 19, 2017

Errata Security

Pranksters gonna prank

So Alfa Bank (the bank whose DNS traffic link it to is back in the news with this press release about how in the last month, hackers have spoofed traffic trying to make it look like there's a tie with Trump. In other words, Alfa claims these packets are trying to frame them for a tie with Trump now, and thus (by extension) it must've been a frame last October.

There is no conspiracy here: it's just merry pranksters doing pranks (as this CNN article quotes me).

Indeed, among the people pranking has been me (not the pranks mentioned by Alfa, but different pranks). I ran a scan sending packets from IP address to almost everyone one the Internet, and set the reverse lookup to "".

Sadly, my ISP doesn't allow me to put hyphens in the name, so it's not "" as it should be in order to prank well.

Geeks gonna geek and pranksters gonna prank. I can imagine all sorts of other fun pranks somebody might do in order to stir the pot. Since the original news reports of the AlfaBank/ connection last year, we have to assume any further data is tainted by goofballs like me goofing off.

By the way, in my particular case, there's a good lesson to be had here about the arbitrariness of IP addresses and names. There is no server located at my IP address of No such machine exists. Instead, I run my scans from a nearby machine on the same network, and "spoof" that address with masscan:

$ masscan -p80 --banners --spoof-ip

This sends a web request to every machine on the Internet from that IP address, despite no machine anywhere being configured with that IP address.

I point this out because people are confused by the meaning of an "IP address", or a "server", "domain", and "domain name". I can imagine the FBI looking into this and getting a FISA warrant for the server located at my IP address, and my ISP coming back and telling them that no such server exists, nor has a server existed at that IP address for many years.

In the case of last years story, there's little reason to believe IP spoofing was happening, but the conspiracy theory still breaks down for the same reason: the association between these concepts is not what you think it is. Listrak, the owner of the server at the center of the conspiracy, still reverse resolves the IP address as "", either because they are lazy, or because they enjoy the lulz.

It's absurd thinking anything sent by the server is related to the Trump Orgainzation today, and it's equally plausible that nothing the server sent was related to Trump last year as well, especially since (as CNN reports), Trump had severed their ties with Cendyn (the marketing company that uses Listrak servers for email).

Also, as mentioned in a previous blog post, I set my home network's domain to be "", which means that some of my DNS lookups at home are actually being sent to Alfa Bank. I should probably turn this off before the FBI comes knocking at my door.

by Robert Graham ( at March 19, 2017 06:45 AM

Assert() in the hands of bad coders

Using assert() creates better code, as programmers double-check assumptions. But only if used correctly. Unfortunately, bad programmers tend to use them badly, making code worse than if no asserts were used at all. They are a nuanced concept that most programmers don't really understand.

We saw this recently with the crash of "Bitcoin Unlimited", a version of Bitcoin that allows more transactions. They used an assert() to check the validity of input, and when they received bad input, most of the nodes in the network crashed.

The Bitcoin Classic/Unlimited code is full of bad uses of assert. The following examples are all from the file main.cpp.

Example #1this line of code:

  1.     if (nPos >= coins->vout.size() || coins->vout[nPos].IsNull())
  2.         assert(false); 

This use of assert is silly. The code should look like this:

  1.     assert(nPos < coins->vout.size());
  2.     assert(!coins->vout[nPos].IsNull());

This is the least of their problems. It understandable that as code ages, and things are added/changed, that odd looking code like this appears. But still, it's an example of wrong thinking about asserts. Among the problems this would cause is that if asserts were ever turned off, you'd have to deal with dead code elimination warnings in static analyzers.

Example #2line of code:

  1.     assert(view.Flush());

The code within assert is supposed to only read values, not change values. In this example, the Flush function changes things. Normally, asserts are only compiled into debug versions of the code, and removed for release versions. However, doing so for Bitcoin will cause the program to behave incorrectly, as things like the Flush() function are no longer called. That's why they put at the top of this code, to inform people that debug must be left on.

  1. #if defined(NDEBUG)
  2. # error "Bitcoin cannot be compiled without assertions."
  3. #endif

Example #3: line of code:

  1.     CBlockIndex* pindexNew = new CBlockIndex(block);
  2.     assert(pindexNew);

The new operator never returns NULL, but throws its own exception instead. Not only is this a misconception about what new does, it's also a misconception about assert. The assert is supposed to check for bad code, not check errors.

Example #4: line of code

  1.     BlockMap::iterator mi = mapBlockIndex.find(inv.hash);
  2.     CBlock block;
  3.     const Consensus::Params& consensusParams = Params().GetConsensus();
  4.     if (!ReadBlockFromDisk(block, (*mi).second, consensusParams))
  5.         assert(!"cannot load block from disk");

This is the feature that crashed Bitcoin Unlimited, and would also crash main Bitcoin nodes that use the "XTHIN" feature. The problem comes from parsing input (inv.hash). If the parsed input is bad, then the block won't exist on the disk, and the assert will fail, and the program will crash.

Again, assert is for checking for bad code that leads to impossible conditions, not checking errors in input, or checking errors in system functions.


The above examples were taken from only one file in the Bitcoin Classic source code. They demonstrate the typically wrong ways bad programmers use asserts. It'd be a great example to show students of programming how not to write bad code.

More generally, though, it shows why there's a difference between 1x and 10x programmers. 1x programmers, like those writing Bitcoin code, make the typical mistake of treating assert() as error checking. The nuance of assert is lost on them.

Updated to reflect that I'm refering to the "Bitcoin Classic" source code, which isn't the "Bitcoin Core" source code. However, all the problems above appear to also be problems in the Bitcoin Core source code.

by Robert Graham ( at March 19, 2017 06:41 AM

FBI: what to look for in the Trump/AlfaBank connection

As CNN reports, the FBI seems to be looking into that connection between Trump and Alfa Bank. Here are some things to look for.

First, get your own copy of the logs from root name servers. I don't trust the source of the original logs. I suspect they've been edited in order to show a relationship with Alfa Bank. You've got lots of sources both inside government and in private industry that can provide a copy of these logs without a warrant. (Which sucks, you should need a warrant, but that's the current state of affairs).

Second, look at the server in question. It's probably located at 140 Akron Road, Ephrata, PA. What you are looking for are the logs of anything sent from the server during that time, specifically any e-mails.

Third, talk to Cendyn, and ask them what that server was used for during that time. Their current statement is that it was used by the Metron meeting software. In other words, they say that after they stopped using it to send marketing emails, they started using it for their meeting product. They seem a little confused, so it'd be nice to pin them down. Specifically, get logfiles indicating precisely what happened, and figure out how Metron works, what sorts of messages it will generate.

Fourth, talk to Cendyn, and ask them about customers of their Metron meeting software, namely who used it to arrange meetings with Alfa Bank or the Trump organization. My guess is that this is where you'll really get the juicy information, getting a list of what meetings happened when and who was invited.

Fifth, talk to Cendyn and get logfiles form their DNS servers to figure out who was resolving that domain name ( during that time period.

Sixth, ask Alfa Bank for logfiles from their DNS resolvers that would tell you which machines internally were generating those requests.

My guess is that all of this will come up empty. There's a coincidence here, but a small one. Much of the technical details have been overhyped and mean little.

by Robert Graham ( at March 19, 2017 06:22 AM

March 18, 2017

Openstack Soft Delete - recover deleted instances

This article contains both an end user and OpenStack administrator guide to set up and use soft_delete with OpenStack Nova. If an instance is deleted with nova delete, it's gone right away. If soft_delete is enabled, it will be queued for deletion for a set amount of time, allowing end-users and administrators to restore the instance with the nova restore command. This can save your ass (or an end-users bottom) if the wrong instance is removed. Setup is simple, just one variable in nova.conf. There are some caveats we'll also discuss here.

March 18, 2017 12:00 AM

March 16, 2017

The Lone Sysadmin

vCenter 6.5b Resets Root Password Expiration Settings

I’m starting to update all my 6.x vCenters and vROPS, pending patches being released. You should be doing this, too, since they’re vulnerable to the Apache Struts 2 critical security holes. One thing I noted in my testing is that after patching the 6.5 appliances, their root password expiration settings go back to the defaults. […]

The post vCenter 6.5b Resets Root Password Expiration Settings appeared first on The Lone Sysadmin. Head over to the source to read the full post!

by Bob Plankers at March 16, 2017 09:23 PM

Finding the biggest data (storage) consumers in Zabbix

The post Finding the biggest data (storage) consumers in Zabbix appeared first on

If you run Zabbix long enough, eventually your database will grow to sizes you'd rather not see. And that begs the question: what items are causing the most storage in my Zabbix backend, be it MySQL, PostgreSQL or something else?

I investigated the same questions and found the following queries to be very useful.

Before you start to look into this, make sure you cleanup your database of older orphaned records first, see an older post of mine for more details.

What items have the most value records?

These probably also consume the most diskspace.

For the history_uint table (holds all integer values):

SELECT COUNT(history.itemid), history.itemid,, i.key_,
FROM history_uint AS history
LEFT JOIN items AS i ON i.itemid = history.itemid
LEFT JOIN hosts AS h ON i.hostid = h.hostid
GROUP BY history.itemid
ORDER BY COUNT(history.itemid) DESC
LIMIT 100;

For the history table (holds all float & double values):

SELECT COUNT(history.itemid), history.itemid,, i.key_,
FROM history AS history
LEFT JOIN items AS i ON i.itemid = history.itemid
LEFT JOIN hosts AS h ON i.hostid = h.hostid
GROUP BY history.itemid
ORDER BY COUNT(history.itemid) DESC
LIMIT 100;

For the history_text table (holds all text values):

SELECT COUNT(history.itemid), history.itemid,, i.key_,
FROM history_text AS history
LEFT JOIN items AS i ON i.itemid = history.itemid
LEFT JOIN hosts AS h ON i.hostid = h.hostid
GROUP BY history.itemid
ORDER BY COUNT(history.itemid) DESC
LIMIT 100;

The post Finding the biggest data (storage) consumers in Zabbix appeared first on

by Mattias Geniar at March 16, 2017 06:45 PM

March 13, 2017

Sean's IT Blog

My Windows 10 Template Build Process

I’ve been spending a lot of time working with Windows 10 in the lab lately, and one of the big struggles I’ve faced was building a solid template that I could reuse.  The reason I’ve had trouble with this is due to changes that Microsoft made in Windows 10 that essentially break the process that worked with previous versions of Windows.  The biggest changes include Modern UI applications and changes to how default applications are handled.

Modern UI apps are problematic for many reasons.  First, some of the original core Windows applications have been replaced by Modern UI applications, so while it is possible to remove them, you lose significant functionality that may not be replaced by 3rd Party applications.  In order to keep these applications up-to-date, the Windows Store needs to be available on the desktop.  That also means that the Store can’t be disabled unless you want to run outdated Modern UI applications.  Second, Modern UI apps tend to break Sysprep if any user profiles exist on the system outside of the built-in Administrator. 

Default applications are another tricky area.  In previous versions of Windows, a web browser or document viewer could set itself, or prompt the user to set it, as the default application for certain file types or URLs.  So if you installed Adobe Reader, it could set itself up as the default application for PDF programs.  This does not necessarily appear to be the case in Windows 10 – some applications that manage common file types have a system default that applies to all users.  This is mainly true for URLs and PDF files, and they default to Microsoft Edge.  While I can change this on a per-user basis, I may want to enforce certain corporate application standards within my image.

I’ve been spending a lot of time building Windows 10 desktop templates in my lab, and I’ve been looking at a lot of Windows 10 build guides.  All of the ones I’ve seen treat a Windows 10 build like a Windows 7 build with some minor changes, but none of them address the issues that I’ve experienced in my lab.

To get around issues with Modern UI applications, managing and/or cleaning up user accounts on the system before sysprepping my template, and dealing with application defaults, I decided to put together a different method for building my Windows 10 VDI image to address the issues I’ve faced and to reduce the number of manual steps that I have to take when creating and/or updating the template.  The main thing that I do differently is the use of Sysprep Audit Mode.  Audit mode allows an administrator to bypass the OOBE and log into the desktop as a local administrator with network access to customize the desktop environment, and the system remains in Audit Mode until Sysprep is run again.  While in Audit Mode, I cannot join the computer to the domain.  However, this is not a deal breaker as I can access my file shares without being joined to the domain.

When building this template, I don’t do a defrag or run the VMware OS Optimization tool as this template is the grandparent VM.  I will deploy my production parent VMs from this template and optimize them before deploying my instant clone or full clone pools.  I also don’t feel that defrag is needed with disposable VMs running on modern storage solutions.

My steps for building a Windows 10 virtual desktop template are:

1. Create the VM in vCenter.  Select the VMXNet3 network card as the network device and configure the VM to boot directly into the BIOS.  Also be sure to attach the Windows 10 ISO to the virtual machine.

2. Power the VM on.

3. Open the Console or use the remote console to access the VM.

4. Disable the Floppy drive and the Serial, Parallel, and Floppy Controllers in the BIOS.  Press F10 to save the settings and reboot.

5. Boot into the Windows Installation disk.

6. When you reach the first screen of the Windows installer, press Shift-F10 to open a command prompt.

7. Type diskpart to launch the disk partition utility.  We’ll use this to custom create our partition.  By default, the Windows installer creates a partition table that includes 100MB reserved space for Bitlocker.  Bitlocker isn’t supported in VDI, so we will manually create our partition.  The steps to do that are:

  • Type Select Disk 0
  • Type Create Partition Primary
  • Type Exit twice

8. Install Windows 10 using the default options.

9. When the system boots the Out-of-Box-Experience (Windows Welcome/Set up New Account), press Control-Shift-F3 to boot into Sysprep Audit Mode.

10. Install VMware Tools and reboot.

11.  Install the Horizon agent and reboot.  Note: You may need to connect to a network file share to access installers.  When doing so, sign in as Domain\User when prompted.  Do not join the system to the domain while in Audit Mode.

12. Install any applications and/or updates that you want to have in the template.  Reboot as often as required as the servers will boot into Audit Mode.

13.  Remove any Modern UI Apps that you don’t want to provision as part of the template.  I remove all except for Photos (haven’t found a good free alternative), Calculator (Windows 10 Calculator is actually pretty good), and Store (might need it depending on my use case/to keep the other two updated).  You actually need to deprovision it twice – once for the administrator account and once at the system level to remove the AppX Package.  The steps for this are:

  • Open PowerShell as an Administrator.
  • Run the following command to deprovision AppX Packages for all users: Get-AppXPackage  -allusers| Where {($ –notlike “*Photos*”) –and ($ –notlike “*Calculator*”) –and ($ –notlike “*Store*”)} | Remove-AppXPackage
  • Run the following command to uninstall unneeded AppX Packages: Get-AppXProvisionedPackage –online | Where {($ –notlike “*Photos*”) –and ($ –notlike “*Calculator*”) –and ($ –notlike “*Store*”)} | Remove-AppXProvisionedPackage –online

14.  Configure the application defaults for your administrator account.  This can be done in Settings –> System –> Default Apps.

15. Now we’re going to replace the default application associations.  Windows stores these in an XML file, and these associations are installed for each new user that logs into the system.  This file is called DefaultOEMAssociations.xml, and it is located in C:\Windows\System32.  The steps for this are:

  • Back up the C:\Windows\System32\DefaultOEMAssociations.xml file.
  • Open PowerShell as an Administrator
  • Run the following command to export your Administrator account default app associations:dism /online /Export-DefaultAppAssociations:”%userprofile%\Desktop\NewDefaultAppAssociations.xml”
  • Run the following command to import your new default app associations: dism /online /Import-DefaultAppAssociations:”%userprofile%\Desktop\NewDefaultAppAssociations.xml”

16. Reboot the system.

17. After the system reboots, the sysprep window will pop up.  Select “Out-of-Box Experience,” “Generalize,” and Shut Down.  Click OK to run sysprep.

18. After the VM shuts down, convert it to a template and deploy a test VM.  The VM should boot to the Out-of-Box-Experience.  You can also use a customization spec when deploying templates from the VM, and while it will boot to the Out-of-Box-Experience, the customization will still run.

So these are the basic steps that I do when building my Windows 10 template for VMware.  If you have any questions, please get my on Twitter at @seanpmassey.

by seanpmassey at March 13, 2017 04:53 PM

March 12, 2017

Steve Kemp's Blog

How I started programming

I've written parts of this story in the past, but never in one place and never in much detail. So why not now?

In 1982 my family moved house, so one morning I went to school and at lunch-time I had to walk home to a completely different house.

We moved sometime towards the end of the year, and ended up spending lots of money replacing the windows of the new place. For people in York I was born in Farrar Street, Y010 3BY, and we moved to a place on Thief Lane, YO1 3HS. Being named as it was I "ironically" stole at least two street-signs and hung them on my bedroom wall. I suspect my parents were disappointed.

Anyway the net result of this relocation, and the extra repairs meant that my sisters and I had a joint Christmas present that year, a ZX Spectrum 48k.

I tried to find pictures of what we received but unfortunately the web doesn't remember the precise bundle. All together though we received:

I know we also received Horace and the Spiders, and I have vague memories of some other things being included, including a Space Invaders clone. No doubt my parents bought them separately.

Highlights of my Spectrum-gaming memories include R-Type, Strider, and the various "Dizzy" games. Some of the latter I remember very fondly.

Unfortunately this Christmas was pretty underwhelming. We unpacked the machine, we cabled it up to the family TV-set - we only had the one, after all - and then proceeded to be very disappointed when nothing we did resulted in a successful game! It turns out our cassette-deck was not good enough. Being back in the 80s the shops were closed over Christmas, and my memory is that it was around January before we received a working tape-player/recorder, such that we could load games.

Happily the computer came with manuals. I read one, skipping words and terms I didn't understand. I then read the other, which was the spiral-bound orange book. It contained enough examples and decent wording that I learned to write code in BASIC. Not bad for an 11/12 year old.

Later I discovered that my local library contained "computer books". These were colourful books that promised "The Mystery of Silver Mounter", or "Write your own ADVENTURE PROGRAMS". But were largely dry books that contained nothing but multi-page listings of BASIC programs to type in. Often with adjustments that had to be made for your own computer-flavour (BASIC varying between different systems).

If you want to recapture the magic scroll to the foot of this Osbourne page and you can download them!

Later I taught myself Z80 Assembly Language, partly via the Spectrum manual and partly via such books as these two (which I still own 30ish years later):

  • Understanding your Spectrum, Basic & Machine Code Programming.
    • by Dr Ian Logan
  • An introduction to Z80 Machine Code.
    • R.A & J.W Penfold

Pretty much the only reason I continued down this path is because I wanted infinite/extra lives in the few games I owned. (Which were largely pirated via the schoolboy network of parents with cassette-copiers.)

Eventually I got some of my l33t POKES printed in magazines, and received free badges from the magazines of the day such as Your Sinclair & Sinclair User. For example I was "Hacker of the Month" in the Your Sinclair issue 67 , Page 32, apparently because I "asked so nicely in my letter".

Terrible scan is terrible:

Anyway that takes me from 1980ish to 1984. The only computer I ever touched was a Spectrum. Friends had other things, and there were Sega consoles, but I have no memories of them. Suffice it to say that later when I first saw a PC (complete with Hercules graphics, hard drives, and similar sourcery, running GEM IIRC) I was pleased that Intel assembly was "similar" to Z80 assembly - and now I know the reason why.

Some time in the future I might document how I got my first computer job. It is hillarious. As was my naivete.

March 12, 2017 12:00 AM

March 09, 2017

Everything Sysadmin

Tom speaking at BayLISA meetup, Thu, March 16, 2017

I'll be giving my "Stealing the Best Ideas from DevOps" talk at BayLISA next week. Location will be in Palo Alto (exact location is in flux, RSVP for updates). Hope to see you there!

More info and RSVP here:

by Tom Limoncelli at March 09, 2017 05:20 PM

Running WPS-8 (Word Processing System) on the DEC(mate) PDP-8i and SIMH

This article covers running WPS-8 on a modern day emulator. WPS-8 is a word processor by DEC for the PDP-8. The PDP-8 was a very populair 12-bit minicomputer from 1965. WPS-8 was released around 1970, it came bundled with DEC's VT78 terminal. This terminal bundle was also known as the DECmate. This article covers the setup of the emulator, simh with the correct disk images and terminal settings. It covers basic usage and the features of WPS-8 and it has a section on key remapping. The early keyboards used with WPS-8 have small but incompatible differences with recent keyboards, but nothing that xterm remapping can't fix.

March 09, 2017 12:00 AM

March 07, 2017

The Lone Sysadmin

How Not To Quit Your Job

I’ve thought a lot lately about Michael Thomas, a moron who caused criminal amounts of damage to his former employer in the process of quitting. From The Register[0]: As well as deleting ClickMotive’s backups and notification systems for network problems, he cut off people’s VPN access and “tinkered” with the Texas company’s email servers. He deleted […]

The post How Not To Quit Your Job appeared first on The Lone Sysadmin. Head over to the source to read the full post!

by Bob Plankers at March 07, 2017 05:08 AM

March 05, 2017

Cryptography Engineering

Secure computing for journalists

This morning on Twitter, Buzzfeed editor Miriam Elder asks the following question:

No, this is not a stupid question. Actually it’s an extremely important question, and judging by some of the responses to this Tweet there are a lot of other people who are confused about the answer.

Since I couldn’t find a perfect layperson’s reference anywhere else, I’m going to devote this post to providing the world’s simplest explanation of why, in the threat model of your typical journalistyour desktop machine isn’t very safe. And specifically, why you’re safer using a modern mobile device — and particularly, an iOS device — than just about any other platform.

A brief caveat: I’m a cryptographer, not a software security researcher. However, I’ve spent the past several years interacting with folks like Charlie and Dan and Thomas. I’m pretty confident that they agree with this advice.

What’s wrong with my laptop/desktop machine?

Sadly, most of the problem is you.

If you’re like most journalists — and really, most professionals — you spend less than 100% of your time thinking about security. You need to get work done. When you’re procrastinating from work, you visit funny sites your friends link you to on Facebook. Then you check your email. If you’re a normal and productive user, you probably do a combination of all these things every few minutes, all of which culminates in your downloading some email attachment and (shudder) opening it in Word.

Now I’m not trying to shame you for this. It’s perfectly normal, and indeed it’s necessary if you want to get things done.  But in the parlance of security professionals, it also means you have a huge attack surface.

In English, this means that from the perspective of an attacker there are many different avenues to compromise your machine. Many of these aren’t even that sophisticated. Often it’s just a matter of catching you during an unguarded moment and convincing you to download an executable file or an infected Office document. A compromised machine means that every piece of software on that machine is also vulnerable.

If you don’t believe this works, head over to Google and search for “Remote Access Trojans”. There’s an entire commercial market for these products, each of which allows you to remotely control someone else’s computer. These off-the-shelf products aren’t very sophisticated: indeed, most require you to trick your victim into downloading and running some executable attachment. Sadly, this works on most people just fine. And this is just the retail stuff. Imagine what a modestly sophisticated attacker can do.

I do some of those things on my phone as well. Why is a phone better?

Classical (desktop and laptop) operating systems were designed primarily to support application developers. This means they offer a lot of power to your applications. An application like Microsoft Word can typically read and write all the files available to your account. If Word becomes compromised, this is usually enough to pwn you in practice. And in many cases, these applications have components with root (or Administrator) access, which makes them even more dangerous.

Modern phone operating systems like Android and iOS were built on a different principle. Rather than trusting apps with much power, each app runs in a “sandbox” that (mainly) limits it to accessing its own files. If the sandbox works, even a malicious application shouldn’t be able to reach out to touch other apps’ files or permanently modify your system. This approach — combined with other protections such as in-memory code signing, hardware secret storage and routine use of anti-exploitation measures — makes your system vastly harder to compromise.

Of course, sandboxing isn’t perfect. A compromised or malicious app can always access its own files. More sophisticated exploits can “break out” of the sandbox, typically by exploiting a vulnerability in the operating system. Such vulnerabilities are routinely discovered and occasionally exploited.

The defense to this is twofold: (1) first, run a modern, up-to-date OS that receives security patches quickly. And (2) avoid downloading malicious apps. Which brings me to the main point of this post.

Why use iOS?

The fact of the matter is that when it comes to addressing these remaining issues, Apple phone operating systems (on iPhones and iPads) simply have a better track record.

Since Apple is the only manufacturer of iOS devices, there is no “middleman” when it comes to monitoring for iOS issues and deploying iOS security updates. This means that the buck stops at Apple — rather than with some third-party equipment manufacturer. Indeed, Apple routinely patches its operating systems and pushes the patches to all supported users — sometimes within hours of learning of a vulnerability (something that is relatively rare at this point in any case).

Of course, to be fair: Google has also become fairly decent at supporting its own Android devices. However, to get assurance from this process you need to be running a relatively brand new device and it needs to be manufactured by Google. Otherwise you’re liable to be several days or weeks behind the time when a security issue is discovered and patched — if you ever get it. And Google still does not support all of the features Apple does, including in-memory code signing and strong file encryption.

Apple also seems to do a relatively decent job at curating its App Store, at least as compared to Google. And because those apps support a more modern base of phones, they tend to have access to better security features, whereas Android apps more routinely get caught doing dumb stuff for backwards compatibility reasons.

A password manager using the SEP.

Finally, every recent Apple device (starting with the iPhone 5S and up) also includes a specialized chip known as a “Secure Enclave Processor“. This hardened processor assists in securing the boot chain — ensuring that nobody can tamper with your operating system. It can also protect sensitive values like your passwords, ensuring that only a password or fingerprint can access them.

A few Android phones also offer similar features as well. However, it’s unclear how well these are implemented in contrast to Apple’s SEP. It’s not a bet I would choose to take.

So does using iOS mean I’m perfectly safe?

Of course not. Unfortunately, computer security today is about resisting attacks. We still don’t quite know how to prevent them altogether.

Indeed, well-funded attackers like governments are still capable of compromising your iOS device (and your Android, and your PC or Mac). Literally the only question is how much they’ll have to spend doing it.

Here’s one data point. Last year a human rights activist in the UAE was targeted via a powerful zero day exploit, likely by his government. However, he was careful. Instead of clicking the link he was sent, the activist sent it to the engineers at Citizenlab who reverse-engineered the exploit. The resulting 35-page technical report by Lookout Security and Citizenlab is a thing of terrifying beauty: it describes a chain of no less than three previously unpublished software exploits, which together would have led to the complete compromise of the victim’s iPhone.

But such compromises don’t come cheap. It’s easy to see this kind of attack costing a million dollars or more. This is probably orders of magnitude more than it would cost to compromise the typical desktop user. That’s important. Not perfect, but important.

You’re telling me I have to give up my desktop machine?

Not at all. Or rather, while I’d love to tell you that, I understand this may not be realistic for most users.

All I am telling you to do is to be thoughtful. If you’re working on something sensitive, consider moving the majority of that work (and communications) to a secure device until you’re ready to share it. This may be a bit of a hassle, but it doesn’t have to be your whole life. And since most of us already carry some sort of phone or tablet in addition to our regular work computer, hopefully this won’t require too much of a change in your life.

You can still use your normal computer just fine, as long as you’re aware of the relative risks. That’s all I’m trying to accomplish with this post.

In conclusion

I expect that many technical people will find this post objectionable, largely because they assume that with their expertise and care they can make a desktop operating system work perfectly safely. And maybe they can! But that’s not who this post is addressed to.

And of course, this post still only scratches the surface of the problem. There’s still the problem of selecting the right applications for secure messaging (e.g., Signal and WhatsApp) and finding a good secure application for notetaking and document collaboration and so on.

But hopefully this post at least starts the discussion.

by Matthew Green at March 05, 2017 06:38 PM

Michael Biven

Why We Use Buzzwords

WebOps, SecOps, NetOps, DevOps, DevSecOps, ChatOps, NoOps, DataOps. They’re not just buzzwords. They’re a sign of a lack of resources.

I believe one reason why people keep coming up with newer versions of *Ops is there isn’t enough time or people to own the priorities that are being prepended to it. Forgot about the acronyms and the buzzwords and ask yourself why a new *Ops keep coming up.

It’s not always marketing hype. When people are struggling to find the resources to address one or multiple concerns they’ll latch onto anything that might help.

When systems administrators started building and supporting services that ran web sites we started calling it WebOps to differentiate that type of work from the more traditional SA role.

When DevOps came about, it was a reaction from SAs trying to build things at a speed that matched the needs of the companies that employed them. The success of using the label DevOps to frame that work encouraged others to orientate themselves around it as a way to achieve that same level of success.

Security is an early example and I can remember seeing talks and posts related to using DevOps as a framework to meet their needs.

Data related work seemed to be mostly absent. It seems instead we got a number of services, consultants, and companies that focused around providing a better database or some kind of big data product.

Reflecting back there are two surprises. The first being an early trend of including nontechnical departments in the DevOps framework. Adding your marketing department was one I saw a couple of different posts and talks on.

The second is even with the Software Defined Networking providing a programmatic path for traditional network engineering in the DevOps framework it really hasn’t been a standout. Most of the tools available are usually tied to big expensive gear and this seems to have kept network engineering outside of DevOps. The difference being if you are using a cloud platform that provides SDN then DevOps can cover the networking side.

ChatOps is the other interesting one, because it’s the one focused on the least technical thing. The difficulty people have communicating with other parts of their company and with easily finding information to basic questions that frequently come up.

This got me thinking of breaking the different types of engineering and/or development work needed into three common groups. And the few roles that have either been removed from some companies, outsourced, or they’re understaffed.

The three common groups include software engineer for front-end, back-end, mobile, and tooling. Systems engineering to provide the guidance and support for the entire product lifecycle; including traditional SA and operations roles, SRE, and tooling. The third is data engineering which covers traditional DBA roles, analytics, and big data.

Then you have QA, network engineering, and information security which seem to either have gone away unless you’re at a company that’s big enough to provide a dedicated team or has a specific requirement for them.

For QA it seems we’ve moved it from being a role into a responsibility everyone is accountable to.

Network engineering and security are two areas that I’m starting to wonder if they’ll be dispersed across the three common groups just as QA has been for the most part. Maybe SDN would move control of the network to software and system engineers? There is an obvious benefit to hold everyone accountable for security as we’ve done with QA.

This wouldn’t mean there’s no need for network and security professionals anymore, but that you would probably only find them in certain circumstances.

Which brings me back to my original point. Why the need for all of these buzzwords? With the one exception every single version of*Ops is focused on the non-product feature related work. We’re seeing reductions in the number of roles focused on that type of work while at the same time its importance has increased.

I think it’s important that we’re honest with ourselves and consider that maybe the way we do product management and project planning hasn’t caught up with the changes in the non-product areas of engineering. If it had do you think we would have see all of these new buzzwords?

March 05, 2017 05:48 PM

Vincent Bernat

Netops with Emacs and Org mode

Org mode is a package for Emacs to “keep notes, maintain todo lists, planning projects and authoring documents”. It can execute embedded snippets of code and capture the output (through Babel). It’s an invaluable tool for documenting your infrastructure and your operations.

Here are three (relatively) short videos exhibiting Org mode use in the context of network operations. In all of them, I am using my own junos-mode which features the following perks:

  • syntax highlighting for configuration files,
  • commit of configuration snippets to remote devices, and
  • execution of remote commands.

Since some Junos devices can be quite slow, commits and remote executions are done asynchronously1 with the help of a Python helper.

In the first video, I take some notes about configuring BGP add-path feature (RFC 7911). It demonstrates all the available features of junos-mode.

In the second video, I execute a planned operation to enable this feature in production. The document is a modus operandi and contains the configuration to apply and the commands to check if it works as expected. At the end, the document becomes a detailed report of the operation.

In the third video, a cookbook has been prepared to execute some changes. I set some variables and execute the cookbook to apply the change and check the result.

  1. This is a bit of a hack since Babel doesn’t have native support for that. Also have a look at ob-async which is a language-independent implementation of the same idea. 

by Vincent Bernat at March 05, 2017 11:01 AM

Traceroute IPv6 to Smokeping Target config

This little one-liner converts the output of traceroute for IPv6 to Smokeping Target output. I had only setup my smokepings for IPv4. Recently we had an issue were network config was borked and the whole IPv4 network was not announced via BGP anymore. I was at home troubleshooting, but finding nothing since I have native IPv6 and that part still worked. My Smokeping did show loss, and that explicitly uses IPv4. That helped with the debugging a lot. I have a [IPv4][4] version of this article as well.

March 05, 2017 12:00 AM

March 04, 2017

Anton Chuvakin - Security Warrior

March 01, 2017

Anton Chuvakin - Security Warrior

Monthly Blog Round-Up – Feburary 2016

Here is my next monthly "Security Warrior" blog round-up of top 5 popular posts/topics this month:
  1. “New SIEM Whitepaper on Use Cases In-Depth OUT!” (dated 2010, so I have no idea why it tops the charts now!) presents a whitepaper on select SIEM use cases described in depth with rules and reports [using now-defunct SIEM product]; also see this SIEM use case in depth and this for a more current list of popular SIEM use cases. Finally, see our 2016 research on developing security monitoring use cases here!
  2. Why No Open Source SIEM, EVER?” contains some of my SIEM thinking from 2009. Is it relevant now? You be the judge.  Succeeding with SIEM requires a lot of work, whether you paid for the software, or not. BTW, this post has an amazing “staying power” that is hard to explain – I suspect it has to do with people wanting “free stuff” and googling for “open source SIEM” … 
  3. Simple Log Review Checklist Released!” is often at the top of this list – this aging checklist is still a very useful tool for many people. “On Free Log Management Tools” (also aged a bit by now) is a companion to the checklist (updated version)
  4. This month, my classic PCI DSS Log Review series is extra popular! The series of 18 posts cover a comprehensive log review approach (OK for PCI DSS 3+ even though it predates it), useful for building log review processes and procedures, whether regulatory or not. It is also described in more detail in our Log Management book and mentioned in our PCI book (now in its 4th edition!) – note that this series is mentioned in some PCI Council materials.
  5. “SIEM Resourcing or How Much the Friggin’ Thing Would REALLY Cost Me?” is a quick framework for assessing the SIEM project (well, a program, really) costs at an organization (a lot more details on this here in this paper).
In addition, I’d like to draw your attention to a few recent posts from my Gartner blog [which, BTW, now has about 5X of the traffic of this blog]: 
Recent research on security analytics and UBA / UEBA:
Miscellaneous fun posts:

(see all my published Gartner research here)
Also see my past monthly and annual “Top Popular Blog Posts” – 2007, 2008, 2009, 2010, 2011, 2012, 2013, 2014, 2015, 2016.

Disclaimer: most content at SecurityWarrior blog was written before I joined Gartner on August 1, 2011 and is solely my personal view at the time of writing. For my current security blogging, go here.

Previous post in this endless series:

by Anton Chuvakin ( at March 01, 2017 04:15 PM

LZone - Sysadmin

Match Structured Facts in MCollective

If you are using Facter 2+, which is what you do when you run at least Puppet4, then you have structured facts (meaning nested values) like those:
processors => {
  count => 2,
  isa => "unknown",
  models => [
    "QEMU Virtual CPU version 2.1.2",
    "QEMU Virtual CPU version 2.1.2"
  physicalcount => 2
Now you cannot match those using
mco find -F <fact name>=<fact value>
If you try you just get an empty result. The only way to match structured facts is using -S
mco find -S 'fact("<fact name>").value=<value>'
For example:
mco find -S 'fact("").value='
mco find -S 'fact("os.distro.codename").value=jessie'
See also Mcollective Cheat Sheet

March 01, 2017 03:41 PM

HTTP Error 429 on Reddit

Getting HTTP error 429 when trying to call Reddit APIs or .json endpoints? Try changing your User Agent header to something else. Reddit bans based on user agent.

by admin at March 01, 2017 09:31 AM

February 28, 2017

Cryptography Engineering

The future of Ransomware

This is kind of a funny post for me to write, since it involves speculating about a very destructive type of software — and possibly offering some (very impractical) suggestions on how it might be improved in the future. It goes without saying that there are some real downsides to this kind of speculation. Nonetheless, I’m going ahead on the theory that it’s usually better to talk and think about the bad things that might happen to you — before you meet them on the street and they steal your lunch money.

On the other hand, just as there’s a part of every karate master that secretly wants to go out and beat up a bar full of people, there’s a part of every security professional that looks at our current generation of attackers and thinks: why can’t you people just be a bit more imaginative?! And wonders whether, if our attackers were just a little more creative, people would actually pay attention to securing their system before the bad stuff happens.

And ransomware is definitely a bad thing. According to the FBI it sucks up $1 billion/year in payments alone, and some unimaginably larger amount in remediation costs. This despite the fact that many ransomware packages truly suck, and individual ransomware developers get routinely pwned due to making stupid cryptographic errors. If this strategy is working so well today, the question  we should be asking ourselves is: how much worse could it get?

So that’s what I’m going to muse about now. A few (cryptographic) ways that it might.

Some of these ideas are the result of collaboration with my students Ian Miers, Gabe Kaptchuk and Christina Garman. They range from the obvious to the foolish to the whimsical, and I would be utterly amazed if any of them really do happen. So please don’t take this post too seriously. It’s all just fun.

Quick background: ransomware today

The amazing thing about ransomware is that something so simple could turn out to be such a problem. Modern ransomware consists of malware that infects your computer and then goes about doing something nasty: it encrypts every file it can get its hands on. This typically includes local files as well as network shares that can be reached from the infected machine.

Once your data has been encrypted, your options aren’t great. If you’re lucky enough to have a recent backup, you can purge the infected machine and restore. Otherwise you’re faced with a devil’s bargain: learn top live without that data, or pay the bastards.

If you choose to pay up, there are all sorts of different procedures. However most break down into the following three steps:

  1. When the ransomware encrypts your files, it generates a secret key file and stores it on your computer.
  2. You upload that file (or data string) to your attackers along with a Bitcoin payment.
  3. They process the result with their secrets and send you a decryption key.

If you’re lucky, and your attackers are still paying attention (or haven’t screwed up the crypto beyond recognition) you get back a decryption key or a tool you can use to undo the encryption on your files. The whole thing is very businesslike. Indeed, recent platforms will allegedly offer you a discount if you infect recommend it to your friends — just like Lyft!

The problem of course, is that nothing in this process guarantees that your attacker will give you that decryption key. They might be scammers. They might not have the secret anymore. They might get tracked down and arrested. Or they might get nervous and bail, taking your precious data and your payment with them. This uncertainty makes ransomware payments inherently risky — and worse, it’s the victims who mostly suffer for it.

Perhaps it would be nice if we could make that work better.

Verifiable key delivery using smart contracts

Most modern ransomware employs a cryptocurrency like Bitcoin to enable the payments that make the ransom possible. This is perhaps not the strongest argument for systems like Bitcoin — and yet it seems unlikely that Bitcoin is going away anytime soon. If we can’t solve the problem of Bitcoin, maybe it’s possible to use Bitcoin to make “more reliable” ransomware.

Recall that following a ransomware infection, there’s a possibility that you’ll pay the ransom and get nothing in return. Fundamentally there’s very little you can do about this. A conscientious ransomware developer might in theory offer a “proof of life” — that is, offer to decrypt a few files at random in order to prove their bonafides. But even if they bother with all the risk and interaction of doing this, there’s still no guarantee that they’ll bother to deliver the hostage alive.

An obvious approach to this problem is to make ransomware payments conditional. Rather than sending off your payment and hoping for the best, victims could use cryptocurrency features to ensure that ransomware operators can’t get paid unless they deliver a key. Specifically, a ransomware developer could easily perform payment via a smart contract script (in a system like Ethereum) that guarantees the following property:

This payment will be delivered to the ransomware operator if and only if the ransomware author unlocks it — by posting the ransomware decryption key to the same blockchain.

The basic primitive needed for this is called a Zero Knowledge Contingent Payment. This idea was proposed by Greg Maxwell and demonstrated by Sean Bowe of the ZCash team.**** The rough idea is to set the decryption key to be some pre-image k for some public hash value K that the ransomware generates and leaves on your system. It’s relatively easy to imagine a smart contract that allows payment if and only if the payee can post the input k such that K=SHA256(k). This could easily be written in Ethereum, and almost certainly has an analog for Bitcoin script.

The challenge here, of course, is to prove that k is actually a decryption key for your files, and that the files contain valid data. There are a handful of different ways to tackle this problem. One is to use complex zero-knowledge proof techniques (like zkSNARKs or ZKBoo) to make the necessary proofs non-interactively. But this is painful, and frankly above the level of most ransomware developers — who are still struggling with basic RSA.

An alternative approach is to use several such K challenges in combination with the “proof of life” idea. The ransomware operator would prove her bonafides by decrypting a small, randomly selected subset of files before the issuer issued payment. The operator could still “fake” the encryption — or lose the decryption key — but she would be exposed with reasonable probability before money changed hands.

“Autonomous” ransomware

Of course, the problem with “verifiable” ransomware is: what ransomware developer would bother with this nonsense?

While the ability to verify decryption might conceivably improve customer satisfaction, it’s not clear that it would really offer that much value to ransomware deverlopers. At the same time, it would definitely add a lot of nasty complexity to their software.

Instead of pursuing ideas that offer developers no obvious upside, ransomware designers presumably will pursue ideas that offer them some real benefits. And that brings us to an idea time whose time has (hopefully) not quite come yet. The idea itself is simple:

Make ransomware that doesn’t require operators.

Recall that in the final step of the ransom process, the ransomware operator must deliver a decryption key to the victim. This step is the most fraught for operators, since it requires them to manage keys and respond to queries on the Internet. Wouldn’t it be better for operators if they could eliminate this step altogether?

Of course, to accomplish this seems to require a trustworthy third party — or better, a form of ransomware that can decrypt itself when the victim makes a Bitcoin payment. Of course this last idea seems fundamentally contradictory. The decryption keys would have to live on the victim’s device, and the victim owns that device. If you tried that, then victim could presumably just hack the secrets out and decrypt the ransomware without paying.

But what if the victim couldn’t hack their own machine?

This isn’t a crazy idea. In fact, it’s exactly the premise that’s envisioned by a new class of trusted execution environments, including Intel’s SGX and ARM TrustZone. These systems — which are built into the latest generation of many processors — allow users to instantiate “secure enclaves”: software environments that can’t be accessed by outside parties. SGX also isolates enclaves from other enclaves, which means the secrets they hold are hard to pry out.

Hypothetically, after infecting your computer a piece of ransomware could generate and store its decryption key inside of a secure enclave. This enclave could be programmed to release the key only on presentation of a valid Bitcoin payment to a designated address.

The beauty of this approach is that no third party even needs to verify the payment. Bitcoin payments themselves consist of a publicly-verifiable transaction embedded in a series of “blocks”, each containing an expensive computational “proof of work“. In principle, after paying the ransom the victim could present the SGX enclave with a fragment of a blockchain all by itself — freeing the ransomware of the need to interact with third parties. If the blockchain fragment exhibited sufficient hashpower along with a valid payment to a specific address, the enclave would release the decryption key.*

The good news is that Intel and ARM have devoted serious resources to preventing this sort of unauthorized access. SGX developers must obtain a code signing certificate from Intel before they can make production-ready SGX enclaves, and it seems unlikely that Intel would partner up with a ransomware operation. Thus a ransomware operator would likely have to (1) steal a signing key from a legitimate Intel-certified developer, or (2) find an exploitable vulnerability in another developer’s enclave.**, ***

This all seems sort of unlikely, and that appears to block most of the threat — for now. Assuming companies like Intel and Qualcomm don’t screw things up, and have a good plan for revoking enclaves (uh oh), this is not very likely to be a big threat.

Of course, in the long run developers might not need Intel SGX at all. An even more speculative concern is that developments in the field of cryptographic obfuscation will provide a software-only alternative means to implement this type of ransomware. This would eliminate the need for a dependency like SGX altogether, allowing the ransomware to do its work with no hardware at all.

At present such techniques are far north of practical, keep getting broken, and might not work at all. But cryptographic researchers keep trying! I guess the lesson is that it’s not all roses if they succeed.

Ransomware Skynet

Since I’m already this far into what reads like a Peyote-fueled rant, let’s see if we can stretch the bounds of credibility just a little a bit farther. If ransomware can become partially autonomous — i.e., do part of its job without the need for human masters — what would it mean for it to become fully autonomous? In other words, what if we got rid of the rest of the human equation?

I come from the future to encrypt C:\Documents

Ransomware with the ability to enforce payments would provide a potent funding source for another type of autonomous agent: a Decentralized Autonomous Organization, or (DAO). These systems are “corporations” that consist entirely of code that runs on a consensus network like Ethereum. They’re driven by rules, and are capable of both receiving and transmitting funds without (direct) instruction from human beings.

At least in theory it might be possible to develop a DAO that’s funded entirely by ransomware payments — and in turn mindlessly contracts real human beings to develop better ransomware, deploy it against human targets, and… rinse repeat. It’s unlikely that such a system would be stable in the long run — humans are clever and good at destroying dumb things — but it might get a good run. Who knows? Maybe this is how the Rampant Orphan Botnet Ecologies get started.

(I hope it goes without saying that I’m mostly not being serious about this part. Even though it would be totally awesome in a horrible sort of way.)

In conclusion

This hasn’t been a terribly serious post, although it was fun to write. The truth is that as a defender, watching your attackers fiddle around is pretty much the most depressing thing ever. Sometimes you have to break the monotony a bit.

But insofar as there is a serious core to this post, it’s that ransomware currently is using only a tiny fraction of the capabilities available to it. Secure execution technologies in particular represent a giant footgun just waiting to go off if manufacturers get things only a little bit wrong.

Hopefully they won’t, no matter how entertaining it might be.


* This technique is similar to SPV verification. Of course, it would also be possible for a victim to “forge” a blockchain fragment without paying the ransom. However, the cost of this could easily be tuned to significantly exceed the cost of paying the ransom. There are also many issues I’m glossing over here like difficulty adjustments and the possibility of amortizing the forgery over many different victims. But thinking about that stuff is a drag, and this is all for fun, right?

** Of course, if malware can exploit such a vulnerability in another developer’s enclave to achieve code execution for “ransomware”, then the victim could presumably exploit the same vulnerability to make the ransomware spit out its key without a payment. So this strategy seems self-limiting — unless the ransomware developers find a bug that can be “repaired” by changing some immutable state held by the enclave. That seems like a long shot. And no, SGX does not allow you to “seal” data to the current state of the enclave’s RAM image.

*** In theory, Intel or an ARM manufacturer could also revoke the enclave’s signing certificate. However, the current SGX specification doesn’t explain how such a revocation strategy should work. I assume this will be more prominent in future specifications.

**** The original version of this post didn’t credit Greg and Sean properly, because I honestly didn’t make the connection that I was describing the right primitive. Neat!

by Matthew Green at February 28, 2017 12:47 AM

February 27, 2017

LZone - Sysadmin

Nagios Check for Systemd Failed Units

Just a short bash script to check for faulty systemd units to avoid 107 lines of Python...

if [ -f /bin/systemctl ]; then failed=$(/bin/systemctl --failed --no-legend) failed=${failed/ */} # Strip everything after first space failed=${failed/.service/} # Strip .service suffix

if [ "$failed" != "" ]; then echo "Failed units: $failed" exit 1 else echo "No failed units." exit 0 fi else echo "No systemd. Nothing was checked!" exit 0 fi

February 27, 2017 05:50 PM

February 25, 2017

Evaggelos Balaskas

Docker Swarm a native clustering system

Docker Swarm

The native Docker Container Orchestration system is Docker Swarm that in simple terms means that you can have multiple docker machines (hosts) to run your multiple docker containers (replicas). It is best to work with Docker Engine v1.12 and above as docker engine includes docker swarm natively.

Docker Swarm logo:

In not so simply terms, docker instances (engines) running on multiple machines (nodes), communicating together (VXLAN) as a cluster (swarm).


To begin with, we need to create our docker machines. One of the nodes must be the manager and the others will run as workers. For testing purposes I will run three (3) docker engines:

  • Manager Docker Node: myengine0
  • Worker Docker Node 1: myengine1
  • Worker Docker Node 2: myengine2


A docker node is actually a machine that runs the docker engine in the swarm mode. The machine can be a physical, virtual, a virtualbox, a cloud instance, a VPS, a AWS etc etc

As the time of this blog post, officially docker supports natively the below drivers:

  • Amazon Web Services
  • Microsoft Azure
  • Digital Ocean
  • Exoscale
  • Google Compute Engine
  • Generic
  • Microsoft Hyper-V
  • OpenStack
  • Rackspace
  • IBM Softlayer
  • Oracle VirtualBox
  • VMware vCloud Air
  • VMware Fusion
  • VMware vSphere


but there are unofficial drivers also.

I will use the qemu - kvm driver from this github repository:

The simplest way to add the kvm driver is this:

> cd /usr/local/bin/
> sudo -s
# wget -c
# chmod 0750 docker-machine-driver-kvm

Docker Machines

The next thing we need to do, is to create our docker machines. Look on your distro’s repositories:

# yes | pacman -S docker-machine


$ docker-machine create -d kvm myengine0

Running pre-create checks...
Creating machine...
(myengine0) Image cache directory does not exist, creating it at /home/ebal/.docker/machine/cache...
(myengine0) No default Boot2Docker ISO found locally, downloading the latest release...
(myengine0) Latest release for is v1.13.1
(myengine0) Downloading /home/ebal/.docker/machine/cache/boot2docker.iso from
(myengine0) 0%....10%....20%....30%....40%....50%....60%....70%....80%....90%....100%
(myengine0) Copying /home/ebal/.docker/machine/cache/boot2docker.iso to /home/ebal/.docker/machine/machines/myengine0/boot2docker.iso...

Waiting for machine to be running, this may take a few minutes...
Detecting operating system of created instance...
Waiting for SSH to be available...
Detecting the provisioner...
Provisioning with boot2docker...
Copying certs to the local machine directory...
Copying certs to the remote machine...
Setting Docker configuration on the remote daemon...
Checking connection to Docker...
Docker is up and running!
To see how to connect your Docker Client to the Docker Engine running on this virtual machine, run: docker-machine env myengine0

Worker 1

$ docker-machine create -d kvm myengine1
Running pre-create checks...
Creating machine...
(myengine1) Copying /home/ebal/.docker/machine/cache/boot2docker.iso to /home/ebal/.docker/machine/machines/myengine1/boot2docker.iso...
Waiting for machine to be running, this may take a few minutes...
Detecting operating system of created instance...
Waiting for SSH to be available...
Detecting the provisioner...
Provisioning with boot2docker...
Copying certs to the local machine directory...
Copying certs to the remote machine...
Setting Docker configuration on the remote daemon...
Checking connection to Docker...
Docker is up and running!
To see how to connect your Docker Client to the Docker Engine running on this virtual machine, run: docker-machine env myengine1

Worker 2

$ docker-machine create -d kvm myengine2
Running pre-create checks...
Creating machine...
(myengine2) Copying /home/ebal/.docker/machine/cache/boot2docker.iso to /home/ebal/.docker/machine/machines/myengine2/boot2docker.iso...
Waiting for machine to be running, this may take a few minutes...
Detecting operating system of created instance...
Waiting for SSH to be available...
Detecting the provisioner...
Provisioning with boot2docker...
Copying certs to the local machine directory...
Copying certs to the remote machine...
Setting Docker configuration on the remote daemon...
Checking connection to Docker...
Docker is up and running!
To see how to connect your Docker Client to the Docker Engine running on this virtual machine, run: docker-machine env myengine2

List your Machines

$ docker-machine env myengine0
export DOCKER_HOST="tcp://"
export DOCKER_CERT_PATH="/home/ebal/.docker/machine/machines/myengine0"
export DOCKER_MACHINE_NAME="myengine0"
# Run this command to configure your shell:
# eval $(docker-machine env myengine0)

$ docker-machine ls

NAME        ACTIVE   DRIVER   STATE     URL                         SWARM   DOCKER    ERRORS
myengine0   -        kvm      Running   tcp://           v1.13.1
myengine1   -        kvm      Running   tcp://            v1.13.1
myengine2   -        kvm      Running   tcp://           v1.13.1


You can get the IP of your machines with:

$ docker-machine ip myengine0

$ docker-machine ip myengine1

$ docker-machine ip myengine2

with ls as seen above or use the inspect parameter for a full list of information regarding your machines in a json format:

$ docker-machine inspect myengine0

If you have jq you can filter out some info

$ docker-machine inspect myengine0  | jq .'Driver.DiskPath'



To enter inside the kvm docker machine, you can use ssh


$ docker-machine ssh myengine0 

                        ##         .
                  ## ## ##        ==
               ## ## ## ## ##    ===
           /"""""""""""""""""___/ ===
      ~~~ {~~ ~~~~ ~~~ ~~~~ ~~~ ~ /  ===- ~~~
           ______ o           __/
 _                 _   ____     _            _
| |__   ___   ___ | |_|___  __| | ___   ___| | _____ _ __
| '_  / _  / _ | __| __) / _` |/ _  / __| |/ / _  '__|
| |_) | (_) | (_) | |_ / __/ (_| | (_) | (__|   <  __/ |
|_.__/ ___/ ___/ __|_______,_|___/ ___|_|____|_|
Boot2Docker version 1.13.1, build HEAD : b7f6033 - Wed Feb  8 20:31:48 UTC 2017
Docker version 1.13.1, build 092cba3

Worker 1

$ docker-machine ssh myengine1 

                        ##         .
                  ## ## ##        ==
               ## ## ## ## ##    ===
           /"""""""""""""""""___/ ===
      ~~~ {~~ ~~~~ ~~~ ~~~~ ~~~ ~ /  ===- ~~~
           ______ o           __/
 _                 _   ____     _            _
| |__   ___   ___ | |_|___  __| | ___   ___| | _____ _ __
| '_  / _  / _ | __| __) / _` |/ _  / __| |/ / _  '__|
| |_) | (_) | (_) | |_ / __/ (_| | (_) | (__|   <  __/ |
|_.__/ ___/ ___/ __|_______,_|___/ ___|_|____|_|
Boot2Docker version 1.13.1, build HEAD : b7f6033 - Wed Feb  8 20:31:48 UTC 2017
Docker version 1.13.1, build 092cba3

Worker 2

$ docker-machine ssh myengine2

                        ##         .
                  ## ## ##        ==
               ## ## ## ## ##    ===
           /"""""""""""""""""___/ ===
      ~~~ {~~ ~~~~ ~~~ ~~~~ ~~~ ~ /  ===- ~~~
           ______ o           __/
 _                 _   ____     _            _
| |__   ___   ___ | |_|___  __| | ___   ___| | _____ _ __
| '_  / _  / _ | __| __) / _` |/ _  / __| |/ / _  '__|
| |_) | (_) | (_) | |_ / __/ (_| | (_) | (__|   <  __/ |
|_.__/ ___/ ___/ __|_______,_|___/ ___|_|____|_|
Boot2Docker version 1.13.1, build HEAD : b7f6033 - Wed Feb  8 20:31:48 UTC 2017
Docker version 1.13.1, build 092cba3

Swarm Cluster

Now it’s time to build a swarm of docker machines!

Initialize the manager

docker@myengine0:~$  docker swarm init --advertise-addr

Swarm initialized: current node (jwyrvepkz29ogpcx18lgs8qhx) is now a manager.

To add a worker to this swarm, run the following command:

    docker swarm join
    --token SWMTKN-1-4vpiktzp68omwayfs4c3j5mrdrsdavwnewx5834g9cp6p1koeo-bgcwtrz6srt45qdxswnneb6i9

To add a manager to this swarm, run 'docker swarm join-token manager' and follow the instructions.

Join Worker 1

docker@myengine1:~$  docker swarm join
>     --token SWMTKN-1-4vpiktzp68omwayfs4c3j5mrdrsdavwnewx5834g9cp6p1koeo-bgcwtrz6srt45qdxswnneb6i9

This node joined a swarm as a worker.

Join Worker 2

docker@myengine2:~$   docker swarm join
>     --token SWMTKN-1-4vpiktzp68omwayfs4c3j5mrdrsdavwnewx5834g9cp6p1koeo-bgcwtrz6srt45qdxswnneb6i9

This node joined a swarm as a worker.

From the manager

docker@myengine0:~$  docker node ls

jwyrvepkz29ogpcx18lgs8qhx *  myengine0  Ready   Active        Leader
m5akhw7j60fru2d0an4lnsgr3    myengine2  Ready   Active
sfau3r42bqbhtz1c6v9hnld67    myengine1  Ready   Active


We can find more information about the docker-machines running the docker info command when you have ssh-ed the nodes:

eg. the swarm part:


Swarm: active
 NodeID: jwyrvepkz29ogpcx18lgs8qhx
 Is Manager: true
 ClusterID: 8fjv5fzp0wtq9hibl7w2v65cs
 Managers: 1
 Nodes: 3
  Task History Retention Limit: 5
  Snapshot Interval: 10000
  Number of Old Snapshots to Retain: 0
  Heartbeat Tick: 1
  Election Tick: 3
  Heartbeat Period: 5 seconds
 CA Configuration:
  Expiry Duration: 3 months
 Node Address:
 Manager Addresses:


Swarm: active
 NodeID: sfau3r42bqbhtz1c6v9hnld67
 Is Manager: false
 Node Address:
 Manager Addresses:

worker 2

Swarm: active
 NodeID: m5akhw7j60fru2d0an4lnsgr3
 Is Manager: false
 Node Address:
 Manager Addresses:


Now it’s time to test our docker swarm by running a container service across our entire fleet!

For testing purposes we chose 6 replicas of an nginx container:

docker@myengine0:~$ docker service create --replicas 6 -p 80:80 --name web nginx


List images

docker@myengine0:~$  docker images
REPOSITORY          TAG                 IMAGE ID            CREATED             SIZE
nginx               <none>              db079554b4d2        9 days ago          182 MB

List of services

regarding your docker registry or your internet connection, we will see the replicas running:

docker@myengine0:~$ docker service ls
ID            NAME  MODE        REPLICAS  IMAGE
ql6iogo587ib  web   replicated  0/6       nginx:latest

docker@myengine0:~$ docker service ls
ID            NAME  MODE        REPLICAS  IMAGE
ql6iogo587ib  web   replicated  2/6       nginx:latest

docker@myengine0:~$ docker service ls
ID            NAME  MODE        REPLICAS  IMAGE
ql6iogo587ib  web   replicated  3/6       nginx:latest

docker@myengine0:~$ docker service ls
ID            NAME  MODE        REPLICAS  IMAGE
ql6iogo587ib  web   replicated  6/6       nginx:latest
docker@myengine0:~$  docker service ps web

t3v855enecgv  web.1  nginx:latest  myengine1  Running        Running 17 minutes ago
xgwi91plvq00  web.2  nginx:latest  myengine2  Running        Running 17 minutes ago
0l6h6a0va2fy  web.3  nginx:latest  myengine0  Running        Running 16 minutes ago
qchj744k0e45  web.4  nginx:latest  myengine1  Running        Running 17 minutes ago
udimh2bokl8k  web.5  nginx:latest  myengine2  Running        Running 17 minutes ago
t50yhhtngbac  web.6  nginx:latest  myengine0  Running        Running 16 minutes ago


To verify that our replicas are running as they should:


Scaling a service

It’s really interesting that we can scale out or scale down our replicas on the fly !

from the manager

docker@myengine0:~$  docker service ls
ID            NAME  MODE        REPLICAS  IMAGE
ql6iogo587ib  web   replicated  6/6       nginx:latest

docker@myengine0:~$ docker service ps web
t3v855enecgv  web.1  nginx:latest  myengine1  Running        Running 3 days ago
xgwi91plvq00  web.2  nginx:latest  myengine2  Running        Running 3 days ago
0l6h6a0va2fy  web.3  nginx:latest  myengine0  Running        Running 3 days ago
qchj744k0e45  web.4  nginx:latest  myengine1  Running        Running 3 days ago
udimh2bokl8k  web.5  nginx:latest  myengine2  Running        Running 3 days ago
t50yhhtngbac  web.6  nginx:latest  myengine0  Running        Running 3 days ago

Scale Down

from the manager

$ docker service scale web=3
web scaled to 3

docker@myengine0:~$ docker service ls
ID            NAME  MODE        REPLICAS  IMAGE
ql6iogo587ib  web   replicated  3/3       nginx:latest

docker@myengine0:~$ docker service ps web
0l6h6a0va2fy  web.3  nginx:latest  myengine0  Running        Running 3 days ago
qchj744k0e45  web.4  nginx:latest  myengine1  Running        Running 3 days ago
udimh2bokl8k  web.5  nginx:latest  myengine2  Running        Running 3 days ago

Scale Up

from the manager

docker@myengine0:~$ docker service scale web=8
web scaled to 8
docker@myengine0:~$ docker service ls
ID            NAME  MODE        REPLICAS  IMAGE
ql6iogo587ib  web   replicated  3/8       nginx:latest
docker@myengine0:~$ docker service ls
ID            NAME  MODE        REPLICAS  IMAGE
ql6iogo587ib  web   replicated  4/8       nginx:latest
docker@myengine0:~$ docker service ls
ID            NAME  MODE        REPLICAS  IMAGE
ql6iogo587ib  web   replicated  8/8       nginx:latest
docker@myengine0:~$ docker service ps web
lyhoyseg8844  web.1  nginx:latest  myengine1  Running        Running 7 seconds ago
w3j9bhcn9f6e  web.2  nginx:latest  myengine2  Running        Running 8 seconds ago
0l6h6a0va2fy  web.3  nginx:latest  myengine0  Running        Running 3 days ago
qchj744k0e45  web.4  nginx:latest  myengine1  Running        Running 3 days ago
udimh2bokl8k  web.5  nginx:latest  myengine2  Running        Running 3 days ago
vr8jhbum8tlg  web.6  nginx:latest  myengine1  Running        Running 7 seconds ago
m4jzati4ddpp  web.7  nginx:latest  myengine2  Running        Running 8 seconds ago
7jek2zvuz6fs  web.8  nginx:latest  myengine0  Running        Running 11 seconds ago

February 25, 2017 09:19 PM

February 24, 2017

Steve Kemp's Blog

Rotating passwords

Like many people I use a password-manage to record logins to websites. I previously used a tool called pwsafe, but these days I switched to using pass.

Although I don't like the fact the meta-data is exposed the tool is very useful, and its integration with git is both simple and reliable.

Reading about the security issue that recently affected cloudflare made me consider rotating some passwords. Using git I figured I could look at the last update-time of my passwords. Indeed that was pretty simple:

git ls-tree -r --name-only HEAD | while read filename; do
  echo "$(git log -1 --format="%ad" -- $filename) $filename"

Of course that's not quite enough because we want it sorted, and to do that using the seconds-since-epoch is neater. All together I wrote this:

# Show password age - should be useful for rotation - we first of all
# format the timestamp of every *.gpg file, as both unix+relative time,
# then we sort, and finally we output that sorted data - but we skip
# the first field which is the unix-epoch time.
( git ls-tree -r --name-only HEAD | grep '\.gpg$' | while read filename; do \
      echo "$(git log -1 --format="%at %ar" -- $filename) $filename" ; done ) \
        | sort | awk '{for (i=2; i<NF; i++) printf $i " "; print $NF}'

Not the cleanest script I've ever hacked together, but the output is nice:

 steve@ssh ~ $ cd ~/Repos/personal/pass/
 steve@ssh ~/Repos/personal/pass $ ./password-age | head -n 5
 1 year, 10 months ago GPG/root@localhost.gpg
 1 year, 10 months ago GPG/
 1 year, 10 months ago GPG/
 1 year, 10 months ago Git/
 1 year, 10 months ago Git/

Now I need to pick the sites that are more than a year old and rotate credentials. Or delete accounts, as appropriate.

February 24, 2017 12:00 AM

February 23, 2017

Anton Chuvakin - Security Warrior

How to solve RPMs created by Alien having file conflicts

I generate release packages for my software with Alien, which amongst other things converts .deb packages to .rpm.

On Fedora 24 however, the generated RPMs cause a small problem when installed with Yum:

Transaction check error:
  file / from install of cfgtrack-1.0-2.noarch conflicts with file from package filesystem-3.2-20.el7.x86_64
  file /usr/bin from install of cfgtrack-1.0-2.noarch conflicts with file from package filesystem-3.2-20.el7.x86_64

There's a bit of info to be found on the internet about this problem, with most of the posts suggesting using rpmrebuild to fix it. Unfortunately, it looks like rpmrebuild actually requires the package to be installed, and since I don't actually use a RPM-based system, that was a bit of a no-go.

So here's how to fix those packages manually:

First, use Alient to generate a RPM package folder from a Debian package, but don't generate the actual package yet. You can do so with the -g switch:

alien -r -g -v myproject-1.0.deb

This generates a myproject-1.0 directory containing the root fs for the package as well as a myproject-1.0-2.spec file. This spec file is the actual problem. It defines directories for paths such as / and /usr/bin. But those are already provided by the filesystem package, so we shouldn't include them.

You can remove them from a script using sed:

sed -i 's#%dir "/"##' myproject-1.0/myproject-1.0-2.spec
sed -i 's#%dir "/usr/bin/"##' myproject-1.0/myproject-1.0-2.spec

This edits the spec file in-place and replaces the following lines with empty lines:

%dir "/"
%dir "/usr/bin/"

The regular expressions look somewhat different than usual, because I'm using the pound (#) sign as a reg marker instead of "/".

Finally, we can recreate the package using rpmbuild:

cd myproject-1.0
rpmbuild --target=noarch --buildroot /full/path/to/myproject-1.0/ \
         -bb cfgtrack-$(REL_VERSION)-2.spec

The resulting package should install without errors or warnings now.

by admin at February 23, 2017 06:27 AM

February 22, 2017


Bad speaker advice

Last year, while I was developing my talks, I saw a bit of bad advice. I didn't recognize it at the time. Instead, I saw it as a goal to reach. The forum was a private one, and I've long forgotten who the players were. But here is a reconstructed, summarized view of what spurred me to try:

elph1120: You know what I love? A speaker who can do an entire talk from one slide.
612kenny: OMG yes. I saw someguy do that at someconference. It was amazeballs.
elph1120: Yeah, more speakers should do that.
gryphon: Totally.

This is bad advice. Don't do this.

Now to explain what happened...

I saw this, and decided to try and do that for my DevOpsDays Minneapolis talk last year. I got close, I needed 4 slides. Which is enough to fit into a tweet.

See? No link to SlideShare needed! Should be amazing!

It wasn't.

The number one critique I got, by a large, large margin was this:

Wean yourself from the speaker-podium.

In order to do a 4-slide talk, I had to lean pretty hard on speaker-notes. If you're leaning on speaker-notes, you're either tied to the podium or have cue-cards in your hands. Both of these are violations of the modern TED-talk style-guide tech-conferences are following these days. I should have noticed that the people rhapsodizing over one-slide talks were habitues of one of the holdouts of podium-driven talks in the industry.

That said, there is another way to do a speaker-note free talk: the 60-slide deck for a 30 minute talk. Your slides are the notes. So long as you can remember some points to talk about above and beyond what's written on the slides, you're providing value above and beyond the deck you built. The meme-slide laugh inducers provide levity and urge positive feedback. If you're new to speaking this is the style you should be aiming for.

A one-slide talk is PhD level speaking-skills. It means memorizing paragraph by paragraph a 3K word essay, and read it back while on stage and on camera. You should not be trying to reach this bar until you're already whatever about public speaking, and have delivered that talk a bunch of times already.

by SysAdmin1138 at February 22, 2017 04:27 PM

February 21, 2017


OpenSSL and Threads

This post talks about OpenSSL and threads. In particular, using OpenSSL in multi-threaded applications. It traces through the history, explains what was changed for the 1.1.0 release, and will hopefully provide some guidance to developers.

While none of the behaviors have really changed, and therefore none of this should be new information, the documentation has not been as clear as it could, or should, be. Therefore, some readers might be surprised by what’s in this post.

In short, OpenSSL has always, and only, supported the concept of locking an object and sometimes it locks its internal objects. Read on for more details.

In OpenSSL 1.0.2 (and earlier), applications had to provide their own integration with locking and threads, as documented in the threads.pod file. This page starts with the following unfortunate text:

OpenSSL can safely be used in multi-threaded applications provided that at least two callback functions are set, …

The problem is that the word safely was never actually defined. This led many developers to think that OpenSSL’s objects could safely be used by multiple threads at the same time. For example, using a single SSL object in two threads, one each for reading and writing.

The file crypto/th-lock.c in 1.0.2 had a “sample” implementation of the lock callbacks, for a number of platforms. The intent was that applications take that file, make it work on their platform and include it in their application. Many apparently managed to do that, although I wryly point out this comment from the Git history in early 2000:

At least, this compiles nicely on Linux using PTHREADS. I’ve done no other tests so far.

With 1.1.0, the major change was to move this from run-time to compile-time. If threads support is enabled (still the same misleading name), then the native threads package is used. On Windows, this means using the “critical section” API for locks, as can be seen in crypto/threads_win.c. On all other platforms, the pthreads API is used, crypto/threads_pthread.c. In cases where the native threads facility isn’t known, or if explicitly configured with no-threads then dummy functions are used, crypto/threads_none.c.

It is worth looking at at least one of these files. They are all very small, 120 to 170 lines, and can provide an overview of what OpenSSL actually needs and provides. The documentation is a bit better, but could still give the impression that concurrent use of objects in multiple threads is supported. We should fix that, soon.

So what is supported? One way to find out is to do

    git grep CRYPTO_THREAD_.*lock

If you do this, you’ll see that there is some locking around internal lookup tables for engines, error codes, and some RSA and X.509 lookup operations, and also about SSL sessions. Almost none of this behavior is documented, and it’s risky to plan on future behavior from just looking at the current code, but in general you can think that either tables of things (errors, sessions) or low-level internal details (RSA blinding, RSA Montgomeray optimizations) is safe. And most of the time, you probably won’t even think of that at all, anyway.

The other place where threads support comes in, isn’t necessarily about threads. It’s about maintaining reference counts on objects, and ensuring that an object is not free’d until the count hits zero. This is implemented using per-object locks, and the CRYPTO_atomic_add API. You can see some relatively simple examples of the up_ref API in test/crltest.c which also shows one of the common use-cases: adding an object to a container, but still “needing” that object for later, after the container is removed. Most datatypes have their own XXX_up_ref API.

Internally, OpenSSL is moving to do more things using native facilities. For example, the auto-init is done using “run-once” facilities, and the error state is maintained using thread-local storage. I’m not going to provide pointers to the source, because I don’t want to encourage people to rely on undocumented behavior. But, what you can rely on is that if you natively use your system threads facility, OpenSSL will work seamlessly with it. But you still can’t concurrently use most objects in multiple threads.

February 21, 2017 11:00 AM

Colin Percival

Cheating on a string theory exam

And now for something completely different: I've been enjoying FiveThirtyEight's "The Riddler" puzzles. A couple weeks ago I submitted a puzzle of my own; but I haven't heard back and it's too cute a puzzle to not share, so I've decided to post it here.

February 21, 2017 03:50 AM

February 20, 2017

Reliable message delivery with Mosquitto (MQTT)

I was looking for a message queue that could reliably handle messages in such a way that I was guaranteed never to miss one, even if the consumer is offline or crashes. Mosquitto (MQTT) comes very close to that goal. However, it wasn't directly obvious how to configure it to be as reliable as possible So this post describes how to use Mosquitto to ensure the most reliable delivery it can handle.

TL;DR: You can't

If you want to do reliable message handling with Mosquitto, the short answer is: You can't. For the long answer, read the rest of the article. Or if you're lazy and stubborn, read the "Limitations" section further down. ;-)

Anyway, let's get on with the show and see how close Mosquitto can get.

Quick overview of Mosquitto

Here's a quick schematic of Mosquitto components:

+----------+     +--------+     +----------+
| producer |---->| broker |---->| consumer |
+----------+     +--------+     +----------+

The producer sends messages to a topic on the broker. The broker maintains an internal state of topics and which consumers are interested in which topics. It also maintains a queue of messages which still need to be sent to each consumer. How the broker decided what / when to send to which consumer depends on settings such as the QoS (Quality of Service) and what kind of session the consumer is opening.

Producer and consumer settings

Here's a quick overview of settings that ensure the highest available quality of delivery of messages with Mosquitto. When creating a consumer or producer, ensure you set these settings properly:

  • quality-of-service must be 2.
  • The consumer must send a client_id.
  • clean_session on the consumer must be False.

These are the base requirements to ensure that each consumer will receive messages exactly once, even if they've been offline for a while. The quality-of-service setting of 2 ensures that the broker requires acknowledgement from the consumer that a message has been received properly. Only then does the broker update its internal state to advance the consumer to the next message in the queue. If the client crashes before acknowledging the message, it'll be resent the next time.

The client_id gives the broker a unique name under which to store session state information such as the last message the consumer has properly acknowledged. Without a client_id, the broker cannot do this.

The clean_session setting lets the consumer inform the broker about whether it wants its session state remembered. Without it, the broker assumes the broker assumes the consumer does not care about past messages and such. It will only receive any new messages that are produced after the consumer has connected to the broker.

Together these settings ensure that messages are reliably delivered from the producer to the broker and to the consumer, even if the consumer has been disconnected for a while or crashes while receiving the message.

Broker settings

The following settings are relevant configuration options on the broker. You can generally find these settings in/etc/mosquitto/mosquitto.conf.

  • The broker must have persistence set to True in the broker configuration.
  • You may want to set max_inflight_messages to 1 in the broker configuration to ensure correct ordering of messages.
  • Configure max_queued_messsages to the maximum number of messages to retain in a queue.
  • Tweak autosave_interval to how often you want the broker to write the in-memory database to disk.

The persistence setting informs the broker that you'd like session state and message queues written to disk. If the broker for some reason, the messages will (mostly) still be there.

You can ensure that messages are sent to consumers in the same order as they were sent to the broker by the producers by setting the max_inflight_messages setting to 1. This will probably severely limit the throughput speed of messages.

The max_queued_messsages determines how many unconfirmed messages should maximally be retained in queues. This should basically be the product of the maximum number of messages per second and the maximum time a consumer might be offline. Say we're processing 1 message per second and we want the consumer to be able to be offline for 2 hours (= 7200 seconds), then the max_queued_messsages setting should be 1 * 7200 = 7200.

The autosave_interval determines how often you want the broker to write the in-memory database to disk. I suspect that setting this to a very low level will cause severe Disk I/O activity.


Here's an example of a producer and consumer:

import paho.mqtt.client as paho
import time

client = paho.Client(protocol=paho.MQTTv31)
client.connect("localhost", 1883)
client.publish("mytesttopic", str("foo"), qos=2)
time.sleep(1)  # Give the client loop time to proess the message

import paho.mqtt.client as paho

def on_message(client, userdata, msg):
    print(msg.topic+" "+str(msg.qos)+" "+str(msg.payload))

client = paho.Client("testclient", clean_session=False, protocol=paho.MQTTv31)
client.on_message = on_message
client.connect("localhost", 1883)
client.subscribe("mytesttopic", qos=2)


There are a few pitfalls I ran into when using Mosquitto:

  • If the broker or one of the clients doesn't support the MQTTv32 protocol, things will fail silently. So I specify MQTTv31 manually.
  • The client loop needs some time to process the sending and receiving of messages. If you send a single message and exit your program right away, the loop doesn't have time to actually send the message.
  • The subscriber must have already run once before the broker will start keeping messages for it. Otherwise, the broker has no idea that a consumer with QoS=2 is interested in messages (and would have to keep messages for ever). So register your consumer once by just running it, before the producer runs.


Although the settings above make exchanging messages with Mosquitto more reliable, there are still some downsides:

  • Exchanging messages in this way is obviously slower than having no consistency checks in place.
  • Since the Mosquitto broker only writes the in-memory database to disk every X (where X is configurable) seconds, you may lose data if the broker crashes.
  • On the consumer side, it is the MQTT library that confirms the receipt of the message. However, as far as I can tell, there is no way to manually confirm the receipt of a message. So if your client crashes while handling a message, rather than while it is receiving a message, you may still lose the message. If you wish to handle this case, you can store the message on the client as soon as possible. This is, however, not much more reliable. The only other way is to implement some manual protocol via the exchange of messages where the original publisher retains a message and resends it unless its been acknowledged by the consumer.


In other words, as far as I can see, you cannot do reliable message handling with Mosquitto. If your broker crashes or your client crashes, Mosquitto will lose your messages. Other than that, if all you require is reliable delivery of messages to the client, you're good to go.

So what are the alternatives? At this point, I have to honest and say: I don't know yet. I'm personally looking for a lightweight solution, and it seems none of the lightweight Message Queues do reliable message handling (as opposed to reliable messagedelivery, which most do just fine).

When I find an answer, I'll let you know here.

by admin at February 20, 2017 07:10 PM


A bit of labor-law that applies even to high-paid techies

It's not as widely known as I hope, but there are a host of workplace protections that apply to non-union, salaried, overtime exempt workers. Not all of them are written into the legal code, and are, in fact, work-arounds. To explain what I'm talking about, read this:

This is a small sample-set, but it works to illustrate the point I'm about to make.

If you find yourself in the position of reporting a coworker to HR for harassing behavior, suddenly find your performance reviews solidly in needs improvement territory, and get fired later; there are workplace protections that will help you get through this, and make the life of your harasser less good.

To get there, here are a few facts and common practices that contribute to the firing, and what could happen afterwards:

  • Performance reviews are as much subjective as objective.
  • Tattling on a co-worker can make the perception of your work move from team player to troublemaker.
  • When the perception shifts like that, top-marks reviews suddenly become remediation-required reviews.
  • Due to US labor law, as amended by State laws, creating a hostile work environment due to sexism, racism, etc, is a criminal act.
  • In spite of that law, very few cases are seen in court, and even fewer reach a verdict.
  • At-will laws mean you can be fired without stated cause.
  • Everyone has a price for their silence.
  • Pathologic workplace cultures have no room for empathy.

Performance Reviews, and Career Improvement Plans

These are often used as the basis for a firing decision. Not all workplaces do them, but many do. It may be hidden in the OKR process, in 360-degree reviews, or another company-goal tracking system, but it's still there. Sometimes they're simple exceeds/meets/needs-improvement metrics, or 1 to 5 ranked metrics, and always have manager input on them.

All of them have some component of plays well with others as one of the tracked metrics. No one likes working with assholes, and this is how they track that. Unfortunately, tattling to mommy that Kenny was mean to her is often seen as not playing well with others.

Buying Your Silence

The severance process you go through after termination is there to buy your silence. Employers know full well there is a marketplace of opinion on good places to work, and if they can keep you from bagging on them on Glassdoor or social media, that's a win for them. You also get a month or two of paid healthcare as you look for someplace new. The method of doing this is called a non-disparagement clause in the severance agreement.

Laws are there to incentivise people to not get caught

If you have a good enough papertrail to plausibly bring suit against the company for one of the legally protected things like racism or sexism, there are strong incentives for them to settle this out of court. Everyone has a price, and most people have a price that doesn't include a written admission of guilt as a requirement. This is why there are so few actions brought against companies in court.

Pathological Empathy

Of the three Westrum Typology types of corporate communication styles (Pathological, Bureaucratic, Generative), it's the pathologic that fundamentally treats non-managers as objects. When you're an object, it doesn't matter if your fee-fees get hurt; what matters is that you're on-side and loyal. If you are seen to be disloyal, you will need to find a new master to swear your fealty to or you will be disposed of through the usual at-will / severance means.

Not all companies are pathologic. The studies I've seen says it's less than a quarter. That said, if the company is big enough you can quite definitely have portions of it that are pathologic while the rest are generative.

That's a lot of framing.

There are certain legal nightmares that companies have with regards to labor laws:

  • Having a now-ex employee bring a discrimination suit against them.
  • Having a class-action suit brought against a specific manager.
  • Having the Department of Labor bring suit against the company for systemic discrimination.

All of these actions are massively public and can't be silenced later. The fact of their filing is damnation enough.

This works for you, even though none of these are likely to come about for your specific case. You see, they're trying to avoid any of that ever happening. To avoid that happening they need to buy you off. Don't think that this is their way of protecting the bad man from any consequence. It's their attempt to, it's up to you to make it actually painful.

Once the third person has been fired and levered themselves into a $200K hush-money severance package, you can guarantee that The Powers That Be are going to sit the bad man down and explain to him that if he doesn't stop with the hands, they're going to have to Do Something; you're costing us a lot of money. One person doing that is just a whiner trying to extort money. Two people getting that is an abundance of whiners. Three people getting that begins to look like a pattern of behavior that is costing the company a lot of money.

This only works because the consequences of simply ignoring your whiny ass are now dire. Thanks, New Deal!

by SysAdmin1138 at February 20, 2017 05:15 PM

Toolsmith Release Advisory: Sysmon v6 for Securitay

Sysmon just keeps getting better.
I'm thrilled to mention that @markrussinovich and @mxatone have released Sysmon v6.
When I first discussed Sysmon v2 two years ago it offered users seven event types.
Oh, how it's grown in the last two years, now with 19 events, plus an error event.
From Mark's RSA presentation we see the current listing with the three new v6 events highlighted.
Sysmon Events

"This release of Sysmon, a background monitor that records activity to the event log for use in security incident detection and forensics, introduces an option that displays event schema, adds an event for Sysmon configuration changes, interprets and displays registry paths in their common format, and adds named pipe create and connection events."

Mark's presentation includes his basic event recommendations so as to run Sysmon optimally.
Basic Event Recommendations
Basic Event Recommendations (Cont)

I strongly suggest you deploy using these recommendations.
A great way to get started is to use a Sysmon configuration template. Again, as Mark discussed at RSA, consider @SwiftOnSecurity's sysmon-config-export.xml via Github. While there are a number of templates on Github, this one has "virtually every line commented and sections are marked with explanations, so it should also function as a tutorial for Sysmon and a guide to critical monitoring areas in Windows systems." Running Sysmon with it is as easy as:
sysmon.exe -accepteula -i sysmonconfig-export.xml

As a quick example of Sysmon capabilities and why you should always run it everywhere, consider the following driver installation scenario. While this is a non-malicious scenario that DFIR practitioners will appreciate, rather than the miscreants, the detection behavior resembles that which would result from kernel-based malware.
I fired up WinPMEM, the kernel mode driver for gaining access to physical memory included with Rekall, as follows:
Upon navigating to Applications and Services Logs/Microsoft/Windows/Sysmon/Operational in the Windows Event Viewer, I retrieved the expected event:

Event ID 6: Driver loaded
The best way to leave you to scramble off and deploy Sysmon broadly, are with Mark's Best Practices and Tips, again from his RSA presentation.

Go forth and deploy!
Cheers...until next time.

by Russ McRee ( at February 20, 2017 07:38 AM

Sarah Allen

the vision thing

Someone recently observed that I was motivated by a big vision and that some engineers are not, and that’s ok. In my experience, it’s not okay. I wish it was. It’s fun to sink into the code and get into the zone and not worry about the complicated real world. However, it’s not fun to work hard and create something over months or years and have it fail in the marketplace or, worse, never ship at all. I’m not talking about research projects where the primary outcome is learning something or publishing a paper. I’m talking about the kind of software I work on, which is intended to be used by other humans, whether they be developers or regular folks.

There are a few common problems I’ve seen happen when a team writes software without a clear and focused vision.

  1. Lots of bugs are discovered late. When engineers are focused on their own specific component or library, it often happens that another engineer will use that library in a slightly different way than was intended. This is especially true for internal libraries where there is little need for documentation since we can see each other’s source code; however, we don’t typically review all off the code our systems depend on. So each part of the code does what it is supposed to do on its own, but when we put them all together, slight miscommunications add up to things not working quite right, typically discovered late in the development cycle.
  2. The system doesn’t work well. When you are building software as a team, engineers work on different parts of the software. If one engineer is making a gaming engine, and another engineer is making presentation software, it rarely comes together to work well at the end. The gaming engine needs a high framerate and quick response to user input. The marketing presentation software need to display text at large font sizes. It’s likely one or the other will suffer, or more typically the software won’t actually work well for either use case.
  3. You delivered the features your manager asked for, but the product isn’t successful. Customers aren’t happy… or they aren’t buying in the first place. New users and people already using your product have different needs and slightly different use cases. It can be a delicate balance to walk. When developing a “feature” it can be easy to miss a step in between what you are working on and something that seems unrelated from the inside, but which people need to use together or in sequence. I call these features in between the features, which are very hard for engineers to see if they don’t have a complete picture of how people will use the product.
  4. Customers are happy and the product is cancelled. Companies have internal metrics by which they evaluate success. Someone on the business side has determined if we expect immediate revenue, or if this is going to be a long bet and expect slow revenue growth while we focus on strategic customer adoption, or if we need a large base of free users before we expect significant revenue. Sometimes it’s a product that supports larger business goals and drives adoption of a different product. If the engineers don’t know what those goals are, they may not get the important details right to make those goals happen.

So, you need to tell engineers the internal metrics, but that’s not enough. No one writes code that by itself increases 28-day active users. Somewhere in the product plans (or the minds of the executives) is a hypothesis that there are some new or improved use cases that will cause people to use the app more (or to sign up in the first place). A lot of teams do document use cases or user journeys and share them with the engineers, which is important, but not sufficient.

Engineers need to know the use cases that everyone believes will be required to make the product successful; however, usually software engineers have to make all sorts of decisions outside of the required use cases for the next release. The software will work better if all of these little decisions are aligned somehow, and the easiest way to do this (in fact the only way that I’ve found to do this), is for the developers to have a shared vision of what this software wants to be — not just in its next release, but in the future. This vision can be somewhat arbitrary for it to work (which means it is totally fine if it changes), the key point is that it needs to be shared, and when it changes, the change and reasons for the change need to be communicated effectively. A shared vision not only aligns all of the little decisions that developers make independently, but also makes all of the design discussions more efficient. We can focus on how we are building a thing, because we already know what we are building and why.

To create a shared vision, we need to answer: who is the target customer? what are the key problems we are solving? what will our customers be able to do with the initial release and what do we hope they will be able to do in the future? Who are the customers who might not be satisfied with our product and that would be ok?

Good engineers will build something that works for a wider audience. Often the software will enable those people to do many more things than the minimum requirements, but focus is key to creating robust software that works well.

I’ve been talking about engineers, since that’s my focus these days and because there remains a persistent myth that engineers don’t need to be distracted with all this business stuff. All of these questions and answers are even more important for designers, technical writers, support, developer relations, support, sales, business development and everyone involved in the making of and communication about the product.

I’m not motivated by a big vision. I’m motivated by impact — not potential impact, real impact. I need to believe that my day to day work is going to add up to something that solves real problems for real people in the real world. I’ve discovered that to reduce risk in software projects, every member of my team must be able to see a through-line from their day-to-day work to shipping software that causes a large number of humans to be empowered to do something new or better, enabling them to solve real-world problems. To do that, I need to convey a vision that lets the individual team members innovate, coming up with creative solutions in their domain that add up to a system that actually works.

by sarah at February 20, 2017 04:12 AM

February 19, 2017

Steve Kemp's Blog

Apologies for the blog-churn.

I've been tweaking my blog a little over the past few days, getting ready for a new release of the chronicle blog compiler (github).

During the course of that I rewrote all the posts to have 100% lower-case file-paths. Redirection-pages have been auto-generated for each page which was previously mixed-case, but unfortunately that will have meant that the RSS feed updated unnecessarily:

  • If it used to contain:
  • It would have been updated to contain

That triggered a lot of spamming, as the URLs would have shown up as being new/unread/distinct.

February 19, 2017 12:00 AM

February 17, 2017

Ben's Practical Admin Blog

Microsoft Ignite Australia 2017 – Final Day


Well all good things must come to an end and so it passes that this is the last day MS Ignite for another year . Definitely a more subdued feeling amongst the attendees (maybe due to a big night last night!) and many delegates are already flying home, not choosing to be here for the last day of events.


First up was the Azure Stack & Hybrid Deployment session. I wish I could say I enjoyed this session, but for a level 300 session there really wasn’t’ much there I couldn’t have picked up on the fly if using the technology myself. This was a little disappointing considering the rushing around I did to get there.

img_0213After an early morning tea was another of Ben Armstrong’s sessions on Windows Containers. This was a fantastic Level 400 deep dive into how containers actually work and some the architectural decisions made to make containers so fast.

Some of the really impressive things talked about was working on performance of things such as PowerShell to speed up its load times by up to 800%, and using memory cloning with Hyper-V containers to get smart about loading the same code over and over again.

This was truly fascinating session and has only continued to increase my desire to see how to use Windows containers in my workplace.

img_0221The last “Technical Session” for me was the fantastic Orin Thomas and discussing the Terrible Habits of Cloud and Server Administrators.

This session has a lot of humour, a lot of head nodding and a lot of home truths about all things Server and Cloud administrators do.

Orin then went on to talk about how to fix some of these things, some of them common sense, and others with technical solutions such as Just Enough Administration suing privileged access management and secure domains. Having been to most of his sessions throughout Ignite, I was starting to hear the same anecdotes for the 2nd or 3rd time.

One last lunch and it was amusing to see around half the exhibitors had already packed up and left, leaving empty stands.

Being honest, I didn’t hang around for the closing keynote though judging by peoples reactions I really wish I had.

And that was the end of Ignite for another year!


by Ben at February 17, 2017 05:00 AM

February 16, 2017

Ben's Practical Admin Blog

Microsoft Ignite Australia 2017 – Day 3

After the excitement of the previous evening, the day kicked of a little later than the norm for me with the Elastic


One of the things I wasn’t especially aware of was the X-Pack by Elastic and that there is a free, basic version available of that after a 30 day trial. Elastic Cloud may also be an option if we do not want to have to run the underlying infrastructure but just be a consumer of the platform. Shortly after that the session became more of a deep dive into Azure provisioning Elastic with the ARM config files, which was a bit out of my depth.

Microsoft Azure Stack was next up and was certainly something that has recently caught my attention.


It was really interesting to hear that it’s not simply a case of downloading an appliance and deploying in your existing infrastructure to create an on-premises Azure presence. This is an integrated solution that requires the purchase of a particular infrastructure to create a “unit”, which you then have multiple units as necessary. It does make sense for this to occur given how closely Microsoft want to hardware to integrate with the software.

Also of great interest was the cost for the Azure  Stack software is based on a consumption model – after you pay for the hardware, the licensing fees are tied to the amount of resources you use – you don’t use it at all then you don’t have to pay anything for The Azure Stack (you do most certainly have to pay for the hardware hosting it though!).

I think this is definitely the technology to watch right now if you are in an enterprise that likes the concept of Azure platform, but have concerns about data being stored in a public cloud.

img_0179After lunch was a session that I was rather happily surprised as being quite relevant and interesting to some of the things I am doing now.

In particular, the cloud identity session introduced me to Azure AD pass-through authentication that is currently in preview.

Working somewhere that has not implemented ADFS, but where data sovereignty is a delicate topic, AD Pass-through provides a rather great way to provide authentication in the cloud without needing ADFS – but rather some lightweight connector proxies that communicate over port 443. I can’t wait to investigate this further once getting back from Ignite.

The last session for the day is Orin Thomas expanding on yesterday’s session about configuring and deploying Just Enough and Just-In-Time Administration

img_0182Just Enough and Just in Time administration is based on the premise that you should consider your network already compromised by attackers and that you now limit the ability for damage to occur by ensuring all administrative privileges are time limited and must be requested using PAM modules. This provides a means of knowing who has administrative access and when. This will probably be seen as lots of additional bureaucracy  for many “old school” admins, but is actually the way many modern infrastructure services are beginning to work, particularly in the cloud.

Just Enough Administration ensures that you only get the administrative access you need to do you job. An example of this is that traditionally PowerShell provides much more access to administrative functions than is usually required by some application admins, so it is possible to make a role to only allow access to the cmdlets needed to administer the service. This is a great idea where a server may hold multiple services that are administrated by different groups of people.

Of course, being the last night of MS Ignite, it can only mean one thing: After Dark!

Lots of food, friends and activities to keep everyone happy.

Tomorrow it will be all over. I’ve had a great time this year, and paced myself much better than my 2012 experiences, and am looking forward to the final sessions!

by Ben at February 16, 2017 11:48 AM

February 15, 2017

Ben's Practical Admin Blog

Microsoft Ignite Australia 2017 – Day 2

Waking up at 5am in the morning is a near impossible feat when at home, yet seems to be no problem here. Might have something to do with the sunrise 🙂


Early start today with the 8.15am session on containers with Ben Armstrong. Containers captured my attention when announced in Server 2016 as a great way to virtualise applications.

img_0158Ben’s presentation was a fantastic introduction to containers made all the better by practical demonstrations of how to use them.

Personally, I think they are a great thing, but I am struggling to find a good use-case for them at work in a way that would see them readily adopted.

I did stop in briefly at the Windows Server 2016 security session, which was standing room only. There are some very cool technologies being built into Windows Server these days including Just Enough Administration that allows you to limit access to powershell cmdlets available to people in a given role. And excampkle if this would be to ensure that the dns admins only had access to the dns cmdlets.

Unfortunately I had to step out not too long after the session started, and given the standing room only sitiuation could not get back in later, so went on to wander the expo floor.

Here I managed to have a chat with Mark Warkolm from Elastic, who I have had dealings with before and is an all round great guy, along with his colleague Russ. Discussed data curation in elastic and how to best use time based indices.

I also met Paul Brauman from Polycom whom we share a mutual friend and had a bit of a chat, with me taking to opportunity to give a light dig at the performance of Polycom Gear 🙂 All taken in good humour though!


After lunch I opted to watch the short Hack@Ignite talk by Kylie Hunt on 5 ninja tricks to making the workplace suck less. This was an entertaining talk and full of humour going through how a happier workplace makes for greater productivity and what things you can do to help make the workplace happier. I think much of it I unconsciously knew but needed Kylie to actually say it for it to become the aha! moment for me.

Another break between sessions before heading off to the Supercharged IoT end-to-end Solution on the Azure IoT Platform.

I would love to go on and tell you how the last 2 sessions of MSIgnite went today, but it was about this time that my phone lit up with a heap of messages advising me of a bushfire near my home that is currently out of control!

I guess some IoT Devices would be handy about now to monitor the weather and particulate situation. For now I am just trying to get some eyes on the ground…

**update** Eyes on ground says I should be fine…


by Ben at February 15, 2017 07:10 AM

February 14, 2017

Everything Sysadmin

February NYCDevOps meetup: Tiny Talks CfP

The NYC DevOps meetup is trying something new in February: Tiny Talks. Tiny talks are short, 5-10 minute talks usually presented by members. They can be basically any topic related to DevOps. The idea is to encourage local members to present at meetings.

We'll be doing this for our first time at the February 21, 2017 NYCDevOps meetup. You are encouraged to sign up here (though we won't be strict if you show up with a presentation.)

From the website:

Tiny talks are 5-10 minute talks on any DevOps-related topic. We want to encourage everyone to submit a proposal. NYCDevOps is a community that shares. Tiny Talks don't have to be polished or fancy. These are short talks, making it easy for beginners and experienced people alike. We hope to have 3-4 presentations.

If you think you can't give a talk, here's a formula to start you off: (1) Use a title like: "In the last 6 months something I learned about X was Y". (2) Create 1-3 slides about what you learned. (3) The talk only has to be a few minutes long... Q&A will fill the rest of the time. For example, "In the last 6 months I learned that TerraForm breaks in [situation], here's why."

We'll have time for 2-3 Tiny Talks, so speaking slots will be first come first serve.

Hope to see you there!

Remember to RSVP!

by Tom Limoncelli at February 14, 2017 08:00 PM

Evaggelos Balaskas

LDAP User Authentication on CentOS 7


You need to already have a LDAP instance in your infrastructure that you can reach from your test linux machine. Your ldap has an organization unit for people and one for groups.

Ldap server conf

It is always a good thing to write your notes/settings beforehand:

Ldap Server:
Domain Component: dc=example,dc=org

People base: ou=people,dc=example,dc=org
Group base: ou=Groups,dc=example,dc=org

bind user: userpam
bind pass: 1234567890


On your centos 7 machine, you have to install two packages:

# yum -y install nss-pam-ldapd

  Installing : nscd-2.17-157.el7_3.1.x86_64
  Installing : nss-pam-ldapd-0.8.13-8.el7.x86_64

local LDAP name service daemon

Edit the /etc/nslcd.conf file accordingly to your ldap setup.

# grep -Ev '#|^$' /etc/nslcd.conf
uid nslcd
gid ldap

uri ldap://
base ou=people,dc=example,dc=org

ssl no
tls_cacertdir /etc/openldap/cacerts

This is the most basic configuration file, without any TLS (encryption) support, but for our test purposes is ok.

restart nslcd

Every time you change something to nslcd.conf file, you need to restart the service:

# systemctl restart nslcd

Name Service Switch

By default the Name Service Switch have ldap support for the below pam services:

# grep ldap /etc/nsswitch.conf

passwd:     files sss ldap
shadow:     files sss ldap
group:      files sss ldap
netgroup:   files sss ldap
automount:  files ldap

if not, just add it yourself. Just remember that the order is from left to right, that means your centos machine will first try to look in your local files, then to your System Security Services Daemon and finally to your ldap URI !


In this first step, the only way to test that your linux machine can talk to your linux server is via getent looking up on the passwd service:

# getent passwd | grep ebal

ebal:x:374:374:Evaggelos Balaskas:/home/ebal:/bin/bash

Ldap Bind Password

The above example is for anonymous bind against your ldap server. That means that secrets (as the password of the user) can not be viewed (actually tested it on the encrypted hash) as for that you need to bind to your ldap server with your credentials.

# egrep -v '^$|#' /etc/nslcd.conf

uid nslcd
gid ldap
uri ldap://
base ou=people,dc=example,dc=org

binddn cn=userpam,dc=example,dc=org
bindpw 1234567890

ssl no
tls_cacertdir /etc/openldap/cacerts

restart nslcd

 # systemctl restart nslcd


Now it’s time for your first ssh login:

~> ssh testvm
ebal@testvm's password: 

Last login: Mon Feb 13 22:50:12 2017
/usr/bin/id: cannot find name for group ID 374

~>  id
uid=374(ebal) gid=374 groups=374

You can login without problem, but there is a warning for your group id.

Ldap Group Configuration

So, we need to add support for our group base on the nslcd configuration file:

# egrep -v '^$|#' /etc/nslcd.conf

uid nslcd
gid ldap
uri ldap://
base ou=people,dc=example,dc=org
binddn cn=userpam,dc=example,dc=org
bindpw 1234567890

base group ou=Groups,dc=example,dc=org

ssl no
tls_cacertdir /etc/openldap/cacerts

restart nslcd

# systemctl restart nslcd


We first test it against getent using the group service:

# getent group | grep 374

and after that, we can ssh again to our linux machine:

~> ssh testvm
ebal@testvm's password:
Last login: Mon Feb 13 23:14:42 2017 from testserver

~> id
uid=374(ebal) gid=374(ebal) groups=374(ebal)

Now it shows the group name without a problem.

February 14, 2017 12:19 AM

February 13, 2017

Everything Sysadmin

Tom speaking at NYCDevOps meetup, Feb 21, 2017

I'll be the speaker at the February meeting of NYCDevOps. I'll be presenting my talk "Stealing the Best Ideas from DevOps: A Guide for Sysadmins without Developers". If you didn't see me give this talk at Usenix LISA, you can catch it here.

Please RSVP. Seating is limited.

by Tom Limoncelli at February 13, 2017 06:00 PM


Project Bylaws

Last October, the OpenSSL Project team had a face to face meeting. We talked about many topics but one of them was that, in recent years, we have seen much more involvement from the community and that we would like to encourage that further. For example, there are a number of people in the community who we know and trust. We would like those people to get involved more and make it easier for them to contribute. We decided to introduce the concept of a “committer” (borrowed from the Apache concept): someone who has the ability to commit code to our source code repository but without necessarily having to become a full team member. This might be seen as a stepping-stone for someone who aspires to full team membership, or simply as an easier way of contributing for those that don’t. Those people could help with our review process (i.e., their reviews would count towards approval) - which might help us keep on top of the github issues and pull request queues.

We also wanted to be more transparent about how we work as a team, and how decisions are made. At the moment we have a set of rules for this but they are not published anywhere.

These two ideas have come together and as a result we have set about rewriting and updating our project rules (drawing inspiration from the Apache Software Foundation model), so that they can be published and, at the same time, introducing the committer concept. I am very pleased to announce the publication of our new “Project Bylaws”. You can read them here. Incidentally these project bylaws should not be confused with the OpenSSL Software Foundation (OSF) Bylaws which have previously been published here. The OSF Bylaws are a legal document which describe how that legal entity is operated.

Over time, we expect to be inviting various trusted members of the community to become committers. We also expect to be publishing a committer policy document in the near future to give further guidance to committers on their role and what they are expected to do.

February 13, 2017 12:00 PM

February 12, 2017


Choria Update

Recently at Config Management Camp I’ve had many discussions about Orchestration, Playbooks and Choria, I thought it’s time for another update on it’s status.

I am nearing version 1.0.0, there are a few things to deal with but it’s getting close. Foremost I wanted to get the project it’s own space on all the various locations like GitHub, Forge, etc.

Inevitably this means getting a logo, it’s been a bit of a slog but after working through loads of feedback on Twitter and offers for assistance from various companies I decided to go to a private designer called Isaac Durazo and the outcome can be seen below:


The process of getting the logo was quite interesting and I am really pleased with the outcome, I’ll blog about that separately.

Other than the logo the project now has it’s own GitHub organisation at and I have moved all the forge modules to it’s own space as well

There are various other places the logo show up like in the Slack notifications and so forth.

On the project front there’s a few improvements:

  • There is now a registration plugin that records a bunch of internal stats on disk, the aim is for them to be read by Collectd and Sensu
  • A new Auditing plugin that emits JSON structured data
  • Several new Data Stores for Playbooks – files, environment.
  • Bug fixes on Windows
  • All the modules, plugins etc have moved to the Choria Forge and GitHub
  • Quite extensive documentation site updates including branding with the logo and logo colors.

There is now very few things left to do to get 1.0.0 out but I guess another release or two will be done before then.

So from now to update to coming versions you need to use the choria/mcollective_choria module which will pull in all it’s dependencies from the Choria project rather than my own Forge.

Still no progress on moving the actual MCollective project forward but I’ve discussed a way to deal with forking the various projects in a way that seems to work for what I want to achieve. In reality I’ll only have time to do that in a couple of months so hopefully something positive will happen in the mean time.

Head over to to take a look.

by R.I. Pienaar at February 12, 2017 06:24 PM

February 09, 2017

Evaggelos Balaskas

free-as-in-freedom Google Apps on Fairphone Android

Fairphone FP1U

I have this phone for more than 3,5 years


running on Android v4.2 !


I dont have them, I dont even have a google account.
Although fairphone comes with an extra GApps package zip, I haven’t installed them either.

Fake GApps

Unfortunately there are some android apps that need GApps to run. So, without any further ado, here are my notes on how to add support for alternative open sources.


Device may brick ! Do a full backup and keep it somewhere safe.

Location Service

We start with the easy one, how to add support for location service.
In Fdroid we can find the UnifiedNlp suite of packages that work as a Location provider middleware. On my device, I installed the below apps:



Screenshots from my fairphone, after a few reboots


Unified github

Just to make things perfectly clear, you can verify fdroid’s packages against these:



Google Play Store

So notes are referring to Phonesky … I didnt follow them !!!

Instead of phonesky, I installed BlankStore !

Here is the github repo: BlankStore v0.7.5



Now are going to a more interesting part. We need to install the μG to our phone.

microG F-Droid repo

It’s really easy, just add the fdroid repository and install the below two apps:
  • microG Services Core
  • microG Services Framework Proxy


microG Services Core

Now the difficult part.

Opening microG self-checked, we are going to see that spoof signature is not passing the test.


Now the most difficult and dangerous thing you will need to do with your phone.

We need to install Xposed Installer

For Fairphone FP1U we need:

Version name:  2.7 experimental
Download: (770.28 KB)
MD5: 36570c6fac687ffe08107e6a72bd3da7

after that, we have to install the below xposed modules:

  • Xposed Installer
  • FakeGApps
  • XposedGmsCoreUnifiedNlp
  • XposedUnifiedNlp

download them and reboot your phone. If you think it may be helpful, cross your fingers.

This is a good time to remind you that you need to have a fresh backup



Now let’s reboot our mobile phone again and try to open μG once more:



If everything seems like the above pictures, then you are done !

Errata 20170211

For android users with v4.2 on it, there are a few errata!

Location Services

You need the legacy network location package from UnifiedNlp


μG - microG

The latest online version -somehow- doesnt work with my phone perfectly.
You can use the build from BRNmod an alternative CyanogenMod

play-services-core-debug.apk                       27-Jan-2017 11:22             4188872

February 09, 2017 07:03 PM

Vincent Bernat

Integration of a Go service with systemd

Unlike other programming languages, Go’s runtime doesn’t provide a way to reliably daemonize a service. A system daemon has to supply this functionality. Most distributions ship systemd which would fit the bill. A correct integration with systemd is quite straightforward. There are two interesting aspects: readiness & liveness.

As an example, we will daemonize this service whose goal is to answer requests with nifty 404 errors:

package main

import (

func main() {
    l, err := net.Listen("tcp", ":8081")
    if err != nil {
        log.Panicf("cannot listen: %s", err)
    http.Serve(l, nil)

You can build it with go build 404.go.

Here is the service file, 404.service1:

Description=404 micro-service




The classic way for an Unix daemon to signal its readiness is to daemonize. Technically, this is done by calling fork(2) twice (which also serves other intents). This is a very common task and the BSD systems, as well as some other C libraries, supply a daemon(3) function for this purpose. Services are expected to daemonize only when they are ready (after reading configuration files and setting up a listening socket, for example). Then, a system can reliably initialize its services with a simple linear script:

ntpd -s

Each daemon can rely on the previous one being ready to do its work. The sequence of actions is the following:

  1. syslogd reads its configuration, activates /dev/log, daemonizes.
  2. unbound reads its configuration, listens on, daemonizes.
  3. ntpd reads its configuration, connects to NTP peers, waits for clock to be synchronized2, daemonizes.

With systemd, we would use Type=fork in the service file. However, Go’s runtime does not support that. Instead, we use Type=notify. In this case, systemd expects the daemon to signal its readiness with a message to an Unix socket. go-systemd package handles the details for us:

package main

import (


func main() {
    l, err := net.Listen("tcp", ":8081")
    if err != nil {
        log.Panicf("cannot listen: %s", err)
    daemon.SdNotify(false, "READY=1") // ❶
    http.Serve(l, nil)                // ❷

It’s important to place the notification after net.Listen() (in ❶): if the notification was sent earlier, a client would get “connection refused” when trying to use the service. When a daemon listens to a socket, connections are queued by the kernel until the daemon is able to accept them (in ❷).

If the service is not run through systemd, the added line is a no-op.


Another interesting feature of systemd is to watch the service and restart it if it happens to crash (thanks to the Restart=on-failure directive). It’s also possible to use a watchdog: the service sends watchdog keep-alives at regular interval. If it fails to do so, systemd will restart it.

We could insert the following code just before http.Serve() call:

go func() {
    interval, err := daemon.SdWatchdogEnabled(false)
    if err != nil || interval == 0 {
    for {
        daemon.SdNotify(false, "WATCHDOG=1")
        time.Sleep(interval / 3)

However, this doesn’t add much value: the goroutine is unrelated to the core business of the service. If for some reason, the HTTP part gets stuck, the goroutine will happily continue to send keep-alives to systemd.

In our example, we can just do a HTTP query before sending the keep-alive. The internal loop can be replaced with this code:

for {
    _, err := http.Get("") // ❸
    if err == nil {
        daemon.SdNotify(false, "WATCHDOG=1")
    time.Sleep(interval / 3)

In ❸, we connect to the service to check if it’s still working. If we get some kind of answer, we send a watchdog keep-alive. If the service is unavailable or if http.Get() gets stuck, systemd will trigger a restart.

There is no universal recipe. However, checks can be split into two groups:

  • Before sending a keep-alive, you execute an active check on the components of your service. The keep-alive is sent only if all checks are successful. The checks can be internal (like in the above example) or external (for example, check with a query to the database).

  • Each component reports its status, telling if it’s alive or not. Before sending a keep-alive, you check the reported status of all components (passive check). If some components are late or reported fatal errors, don’t send the keep-alive.

If possible, recovery from errors (for example, with a backoff retry) and self-healing (for example, by reestablishing a network connection) is always better, but the watchdog is a good tool to handle the worst cases and avoid too complex recovery logic.

For example, if a component doesn’t know how to recover from an exceptional condition3, instead of using panic(), it could signal its situation before dying. Another dedicated component could try to resolve the situation by restarting the faulty component. If it fails to reach an healthy state in time, the watchdog timer will trigger and the whole service will be restarted.

  1. Depending on the distribution, this should be installed in /lib/systemd/system or /usr/lib/systemd/system. Check with the output of the command pkg-config systemd --variable=systemdsystemunitdir

  2. This highly depends on the NTP daemon used. OpenNTPD doesn’t wait unless you use the -s option. ISC NTP doesn’t either unless you use the --wait-sync option. 

  3. An example of an exceptional condition is to reach the limit on the number of file descriptors. Self-healing from this situation is difficult and it’s easy to get stuck in a loop. 

by Vincent Bernat at February 09, 2017 08:32 AM

February 08, 2017

Aikido & HolisticInfoSec™

This is the 300th post to the HolisticInfoSec™ blog. Sparta, this isn't, but I thought it important to provide you with content in a warrior/philosopher mindset regardless. 
Your author is an Aikido practitioner, albeit a fledgling in practice, with so, so much to learn. While Aikido is often translated as "the way of unifying with life energy" or as "the way of harmonious spirit", I propose that the philosophies and principles inherent to Aikido have significant bearing on the practice of information security.
In addition to spending time in the dojo, there are numerous reference books specific to Aikido from which a student can learn. Among the best is Adele Westbrook and Oscar Ratti's Aikido and the Dynamic Sphere. All quotes and references that follow are drawn from this fine publication.
As an advocate for the practice of HolisticInfoSec™ (so much so, I trademarked it) the connectivity to Aikido is practically rhetorical, but allow me to provide you some pointed examples. I've tried to connect each of these in what I believe is an appropriate sequence to further your understanding, and aid you in improving your practice. Simply, one could say each of these can lead to the next.
The Practice of Aikido
"The very first requisite for defense is to know the enemy."
So often in information security, we see reference to the much abused The Art of War, wherein Sun Tzu stated "It is said that if you know your enemies and know yourself, you will not be imperiled in a hundred battles." Aikido embraces this as the first requisite, but so too offers the importance of not underestimating your enemy or opponent. For information security, I liken it to this. If you are uninformed on adversary actor types and profiles, their TTPs (tools, tactics, procedures), as well as current vulnerabilities and exploits, along with more general threat intelligence, then you are already at a disadvantage before you even begin to imagine countering your opponent.  

"A positive defensive strategy is further qualified as being specific, immediate, consistent, and powerful." 
Upon learning more about your adversary, a rehearsed, exercised strategy for responding to their attack should be considered the second requisite for defense. To achieve this, your efforts must include:
  • a clear definition and inventory of the assets you're protecting
  • threat modeling of code, services, and infrastructure
  • an incident response plan and SOP, and regular exercise of the IR plan
  • robust security monitoring to include collection, aggregation, detection, correlation, and visualization
  • ideally, a purple team approach that includes testing blue team detection and response capabilities in partnership with a red team. Any red team that follows the "you suck, we rock" approach should be removed from the building and replaced by one who espouses "we exist to identify vulnerabilities and exploits with the goal of helping the organization better mitigate and remediate".
As your detection and response capabilities improve with practice and repetition, your meantime to mitigate (MTM) and meantime to remediate (MTR) should begin to shrink, thus lending to the immediacy, consistentcy, and power of your defense.

The Process of Defense and Its Factors
"EVERY process of defense will consist of three stages: perception, evaluation-decision, and reaction."
These should be easy likenesses for you to reconcile.
Perception = detection and awareness
The better and more complete your threat intelligence collection and detection capabilities, the better your situational awareness will be, and as a result your perception of adversary behaviors will improve and become more timely.
Evaluation-decision = triage
It's inevitable...$#!+ happens. Your ability to quickly evaluate adversary actions and be decisive in your response will dictate your level of success as incident responders. Strength at this stage directly impacts the rest of the response process. Incorrect or incomplete evaluation, and the resulting ill-informed decisions, can set back your response process in a manner from which recovery will be very difficult.
Reaction = response
My Aikido sensei, after doing so, likes to remind his students "Don't get hit." :-) The analogy here is to react quickly enough to stay on your feet. Can you move quickly enough to not be hit as hard or as impactfully as your adversary intended? Your reaction and response will determine such outcomes. The connection between kinetic and virtual combat here is profound. Stand still, get hit. Feign or evade, at least avoid some, or all contact. In the digital realm, you're reducing your time to recover with this mindset.  

Dynamic Factors
"A defensive aikido strategy begins the moment a would-be attacker takes a step toward you or turns aggressively in your direction. His initial motion (movement) in itself contains the factors you will use to neutralize the action of attack which will spring with explosive force from that motion of convergence."
Continuing on our theme of inevitability, digital adversaries will, beyond the shadow of a doubt, take a step toward you or turn aggressively in your direction. The question for you will be, do you even know when that has occurred in light of our discussion of requisites above? Aikido is all about using your opponent's energy against them, wherein, for those of us in DFIR, our adversary's movement in itself contains the factors we use to neutralize the action of attack. As we improve our capabilities in our defensive processes (perception, evaluation-decision, and reaction), we should be able to respond in a manner that begins the very moment we identify adversarial behavior, and do so quickly enough that our actions pivot directly on our adversary's initial motion.
As an example, your adversary conducts a targeted, nuanced spear phishing campaign. Your detective means identify all intended victims, you immediately react, and add all intended victims to an enhanced watch list for continuous monitoring. The two victims who engaged the payload are quarantined immediately, and no further adversarial pivoting or escalation is identified. The environment as a whole raised to a state of heightened awareness, and your user-base becomes part of your perception network.

"It will be immediate or instantaneous when your reaction is so swift that you apply a technique of neutralization while the attack is still developing, and at the higher levels of the practice even before an attack has been fully launched."
Your threat intelligence capabilities are robust enough that your active deployment of detections for specific Indicators of Compromise (IOCs) prevented the targeted, nuanced spear phishing campaign from even reaching the intended victims. Your monitoring active lists include known adversary infrastructure such that the moment they launch an attack, you are already aware of its imminence.
You are able to neutralize your opponent before they even launch. This may be unimaginable for some, but it is achievable by certain mature organizations under specific circumstances.

The Principle of Centralization
"Centralization, therefore, means adopting a new point of reference, a new platform from which you can exercise a more objective form of control over events and over yourself."
Some organizations decentralize information security, others centralize it with absolute authority. There are arguments for both, and I do not intend to engage that debate. What I ask you to embrace is the "principle of centralization". The analogy is this: large corporations and organizations often have multiple, and even redundant security teams. Even so, their cooperation is key to success.
  • Is information exchanged openly and freely, with silos avoided? 
  • Are teams capable of joint response? 
  • Are there shared resources that all teams can draw from for consistent IOCs and case data?
  • Are you and your team focused on facts, avoiding FUD, thinking creatively, yet assessing with a critical, objective eye?
Even with a logically decentralized security model, organizations can embrace the principle of centralization and achieve an objective form of control over events. The practice of a joint forces focus defines the platform from which teams can and should operate.

Adversarial conditions, in both the physical realm, and the digital realm in which DFIR practitioners operate, are stressful, challenging, and worrisome. 
Morihei Ueshiba, Aikido's founder reminds us that "in extreme situations, the entire universe becomes our foe; at such critical times, unity of mind and technique is essential - do not let your heart waver!" That said, perfection is unlikely, or even impossible, this is a practice you must exercise. Again Ueshiba offers that "failure is the key to success; each mistake teaches us something."
Keep learning, be strong of heart. :-)
Cheers...until next time.

by Russ McRee ( at February 08, 2017 04:45 PM

Geek and Artist - Tech

Culture Codes From Around The Tech World

Last week I found myself reading through all (ok not entirely all, I skipped the NASA one) of the slide decks on – there are a variety of slide decks from tech companies, which attempt to describe what the company culture looks like, their values, or company processes that serve as an employee handbook. They are called Culture Codes and not Company Culture because culture can only be talked about or described, but cannot itself be seen. Hence – the content of the slides are codes for the culture.

At any company we express our culture naturally – it has been described as “what happens when nobody is looking” (and there are many more quotes about it you can Google in your own time). It would be interesting to think about what might be inside a slide deck about my own company’s culture, were we to put one together (we haven’t yet, hence my thoughts on the topic). To that end, I decided to read as many of the slide decks on the above site, and summarise my thoughts.

General Observations

These are stream-of-consciousness notes in no particular order about what I observed in the slide decks.

  • Usually has a value/mission statement for the whole company.
  • Things they aspire to do, some descriptions of what the culture looks like.
  • What makes the company (and culture) different from others.
  • Longer-form description and detail of company values (i.e. describe each value).
  • How the company works (as in workflow) and works (as in success).
  • How to make decisions.
  • Definitions of what success looks like.
  • High-level description of behaviours/traits of people who work well in the company.
  • Lots of justifications for “why” these values are here.
  • Culture deck is often shared publicly and used as recruiting/marketing tool (both for employees and customers).
  • Some have a small “history” section describing the founding of the company or bootstrapping of company values. Good if they are interlinked. Describes major milestones of the company so far.
  • Quotes from customers, employees, source material (e.g. entrepreneurs) for values or guiding statements, supporting the values.
  • If the deck is shared publicly, it sets expectations for employees before they walk through the door. Some are explicitly aimed at being marketing material.
  • Company involvement with community – greater impact than just through core mission.
  • It is tempting to copy values or content, but there are some slide decks which are very incompatible with our values, feel jarring (or just feel plain wrong). Some are very prescriptive, strict rules.
  • The language you use matters! Imagery too.
  • Many pictures of smiling employees / teams. Photos/descriptions of company events (e.g. parties/picnics etc).
  • Crediting culture with driving the success of the company.
  • Video where employees can give concrete examples of living the values.
    • BUT, culture is not just quirky work environments, posters on the wall, etc.
  • Many culture decks double as employee handbooks (e.g. Zappos, Valve). Some are extremely long (Zappos ~300 pages -> many personal employee stories and photos -> tell the values story through many concrete examples rather than analogy/metaphor and brief descriptions).
    • Length varies a lot. Lower limit is around 12 slides, which is not enough space for meaningful descriptions. 50-100 is the sweet spot, more than that just drags on.
  • Some (Genius, Trello) dogfood their own product to create their culture deck (or employee handbook), but it comes off as a bit messy and hard to understand, in some cases full of comments that obscures what is current or actual reality. NextBigSound, Sprintly, Clef store markdown on Github (which is optimised for authoring but not necessarily consumption).
  • Inputs sometimes also from partners and customers.
  • All culture decks walk a tightrope between cliché and meaningless catchphrases on either side. Extremely difficult (impossible?) to be original, quite difficult to be meaningful and capture reality that people can buy into.
  • Often both clarifying negative and positive statements: “X means that we do Y”, or “X doesn’t mean that we do Z”.
  • Google’s values are predictably different (and perhaps not a great fit for many other smaller companies with fewer resources). Same goes for NASA which has a completely different set of needs to most tech companies (safety!).
  • Although I don’t like the Big Spaceship culture deck too much (painful on the eyes) I like that they call bullshit on buzzwords and manager speak.
  • Lots of quotes from Daniel Pink, Steve Jobs, Malcolm Gladwell, US and UK leaders.


As you might expect, the values that companies strive to adhere to and live every day are very similar. I summarised them into a smaller number of similar themes (which may not all be entirely cohesive, but not much time was spent on this):

  • Autonomy / freedom / desire to reduce process / resourceful.
  • “Do the right thing” (also related to the above) / trust / responsibility / ownership.
  • Driven by a vision / vision provided by leaders / leaders living and demonstrating values.
  • Transparency / sharing knowledge / communication / feedback / questioning / openness.
  • Selflessness / not personally identifying with the work / using data rather than opinions.
  • Striving for results / outcome-driven / calculated risk taking / action over talk / results vs busy / building for the customer / doing more with less / growth / Get [Sh]it Done.
  • Learning from mistakes / great is the enemy of good / ship now / fail fast.
  • Helping others / treating colleagues with respect / coaching / mentoring / servant leadership.
  • Constantly learning / inventive / optimistic.
  • Mindfulness / empathy / care / reflection / time for being strategic.
  • Inclusivity / diversity goals.
  • Work/life balance, living healthily. Having fun / celebrating success / “family” feeling / friends at work.
  • Meaningful work / Company invests in community / serving a higher purpose.
  • Intertwining of individual/team/company values and missions.
  • At least one “quirky”, unique company-specific hashtag or catchphrase (several companies list “quirky” as a value).
  • Working with other talented people / hiring & nurturing.
  • Almost all claim to be unique! (really?)

“Anti-” Values

Some companies have very different values to others, which may be undesirable or completely destructive outside of that particular company. But perhaps within their own environment they do work. Presented here without further comment:

  • We start work at 8:30 every day.
  • Many sporting team analogies and metaphors (bro culture?).
  • Success is “hard” or a “grind”.
  • Work hard play hard / fast-paced / urgency / hustle.
  • Timebox everything. Strict scheduling.
  • Focus on competition mentality vs satisfying customers.
  • Only hire “A” players.
  • Do the thing you least want to do right now.
  • Blindly attempting to define culture around The Art of War.
  • Never compromise (never never? really?).
  • Do things that don’t scale.
  • Open Office! (I thought that wasn’t cool anymore…)
  • Many other things that work when the company is very small, but eventually fail and it turns into the same org structure and ways of working as most other tech companies (e.g. Github).

What are some of your own culture codes or values that you like, and see lived out every day? Leave some comments!

by oliver at February 08, 2017 11:27 AM

February 07, 2017

Vincent Bernat

Write your own terminal emulator

I was an happy user of rxvt-unicode until I got a laptop with an HiDPI display. Switching from a LoDPI to a HiDPI screen and back was a pain: I had to manually adjust the font size on all terminals or restart them.

VTE is a library to build a terminal emulator using the GTK+ toolkit, which handles DPI changes. It is used by many terminal emulators, like GNOME Terminal, evilvte, sakura, termit and ROXTerm. The library is quite straightforward and writing a terminal doesn’t take much time if you don’t need many features.

Let’s see how to write a simple one.

A simple terminal

Let’s start small with a terminal with the default settings. We’ll write that in C. Another supported option is Vala.

#include <vte/vte.h>

main(int argc, char *argv[])
    GtkWidget *window, *terminal;

    /* Initialise GTK, the window and the terminal */
    gtk_init(&argc, &argv);
    terminal = vte_terminal_new();
    window = gtk_window_new(GTK_WINDOW_TOPLEVEL);
    gtk_window_set_title(GTK_WINDOW(window), "myterm");

    /* Start a new shell */
    gchar **envp = g_get_environ();
    gchar **command = (gchar *[]){g_strdup(g_environ_getenv(envp, "SHELL")), NULL };
        NULL,       /* working directory  */
        command,    /* command */
        NULL,       /* environment */
        0,          /* spawn flags */
        NULL, NULL, /* child setup */
        NULL,       /* child pid */
        NULL, NULL);

    /* Connect some signals */
    g_signal_connect(window, "delete-event", gtk_main_quit, NULL);
    g_signal_connect(terminal, "child-exited", gtk_main_quit, NULL);

    /* Put widgets together and run the main loop */
    gtk_container_add(GTK_CONTAINER(window), terminal);

You can compile it with the following command:

gcc -O2 -Wall $(pkg-config --cflags --libs vte-2.91) term.c -o term

And run it with ./term:

Simple VTE-based terminal

More features

From here, you can have a look at the documentation to alter behavior or add more features. Here are three examples.


You can define the 16 basic colors with the following code:

#define CLR_R(x)   (((x) & 0xff0000) >> 16)
#define CLR_G(x)   (((x) & 0x00ff00) >>  8)
#define CLR_B(x)   (((x) & 0x0000ff) >>  0)
#define CLR_16(x)  ((double)(x) / 0xff)
#define CLR_GDK(x) (const GdkRGBA){ .red = CLR_16(CLR_R(x)), \
                                    .green = CLR_16(CLR_G(x)), \
                                    .blue = CLR_16(CLR_B(x)), \
                                    .alpha = 0 }
    &(GdkRGBA){ .alpha = 0.85 },
    (const GdkRGBA[]){
}, 16);

While you can’t see it on the screenshot1, this also enables background transparency.

Color rendering

Miscellaneous settings

VTE comes with many settings to change the behavior of the terminal. Consider the following code:

vte_terminal_set_scrollback_lines(VTE_TERMINAL(terminal), 0);
vte_terminal_set_scroll_on_output(VTE_TERMINAL(terminal), FALSE);
vte_terminal_set_scroll_on_keystroke(VTE_TERMINAL(terminal), TRUE);
vte_terminal_set_rewrap_on_resize(VTE_TERMINAL(terminal), TRUE);
vte_terminal_set_mouse_autohide(VTE_TERMINAL(terminal), TRUE);

This will:

  • disable the scrollback buffer,
  • not scroll to the bottom on new output,
  • scroll to the bottom on keystroke,
  • rewrap content when terminal size change, and
  • hide the mouse cursor when typing.

Update the window title

An application can change the window title using XTerm control sequences (for example, with printf "\e]2;${title}\a"). If you want the actual window title to reflect this, you need to define this function:

static gboolean
on_title_changed(GtkWidget *terminal, gpointer user_data)
    GtkWindow *window = user_data;
    return TRUE;

Then, connect it to the appropriate signal, in main():

g_signal_connect(terminal, "window-title-changed", 
    G_CALLBACK(on_title_changed), GTK_WINDOW(window));

Final words

I don’t need much more as I am using tmux inside each terminal. In my own copy, I have also added the ability to complete a word using ones from the current window or other windows (also known as dynamic abbrev expansion). This requires to implement a terminal daemon to handle all terminal windows with one process, similar to urxvtcd.

While writing a terminal “from scratch”2 suits my need, it may not be worth it. evilvte is quite customizable and can be lightweight. Consider it as a first alternative. Honestly, I don’t remember why I didn’t pick it.

UPDATED: evilvte has not seen an update since 2014. Its GTK+3 support is buggy. It doesn’t support the latest versions of the VTE library. Therefore, it’s not a good idea to use it.

You should also note that the primary goal of VTE is to be a library to support GNOME Terminal. Notably, if a feature is not needed for GNOME Terminal, it won’t be added to VTE. If it already exists, it will likely to be deprecated and removed.

  1. Transparency is handled by the composite manager (Compton, in my case). 

  2. For some definition of “scratch” since the hard work is handled by VTE

by Vincent Bernat at February 07, 2017 11:54 PM

Sean's IT Blog

Horizon 7.1–Together We’re Just in Time

If you couldn’t tell by the title of this post, new product announcement time.  Last year at this time, VMware announced App Volumes 3.0, Horizon 7.0, and a number of enhancements including Smart Policies and Instant Clones.  This year, Horizon 7.1 brings a brand new set of improvements that build on the features that were released with Horizon 7.0, including some long awaited features around vGPU, instant clones, and Blast Extreme.

The Just-in-Time Management Platform

The Just-in-Time Management Platform, or JMP (pronounced “jump”) is VMware’s next generation desktop and application delivery platform.  JMP is built on VMware’s instant clone technology for fast provisioning, real-time application delivery provided by App Volumes, and contextual policy and user environment management by UEM.  JMP provides both traditional desktops with just-in-time instant clone desktops and published applications with just-in-time RDSH servers for published applications.


You heard that right.  Instant clone technology has been extended, and it can now be used to provision RDSH server farms. Like Instant Clone desktop pools, instant clone RDSH farms can be elastic and scale near-instantaneously to provide capacity as it is required.  Like instant clone desktop pools, instant clone RDSH farms can provide rolling updates for zero downtime image updates.  VMware is also adding the ability to schedule recurring maintenance of RDSH pools into Horizon Administrator to ensure that the servers are periodically refreshed.

Joining the new Just-in-Time Apps feature is a new Horizon SKU called Horizon Apps.  This SKU provides an RDSH-only focused SKU that includes Instant Clones, Published Apps, App Volumes, UEM, and vSphere.  This new SKU fills a hole in the Horizon product lineup and provides a direct competitor to Citrix XenApp.

We Got the BEAT – Blast Extreme Adaptive Transport

We got the beat…we got the beat…

Sorry, got a little carried away there. 

VMware continues to improve the Blast protocol, and Horizon 7.1 contains the latest enhancements to the protocol – Blast Extreme Adaptive Transport, or BEAT for short.  BEAT is designed to provide a great user experience on multiple network types while adapting to changing network conditions including varying network speeds and severe packet loss. 

VMware was able to achieve these results by adding adaptive bit rates and forward error correction to the protocol as well as adding additional optimizations to better handle network latency and packet loss.  As a result of these improvements, VMware was able to significantly reduce out-of-the-box bandwidth utilization, improve user experience on high latency links, and improve file transfer times from the endpoint to the virtual desktop when using Client Drive Redirection.

Horizon Cloud

Ok…it’s not a good fit.  But it’s hard to find a good 80’s song about clouds. 

Horizon Cloud is the next-generation version of Horizon Air.  Unlike Horizon Air, which was hosted in vCloud Air, Horizon Cloud is hosted in IBM SoftLayer, and it will feature the Horizon Just-in-Time Management Platform.  This enables VMware to provide features that weren’t available in vCloud Air including GPU support, published applications, and utilizing vIDM as a portal for accessing virtual desktops and applications.

Horizon Cloud will also feature an on-premises solution combining hyper-converged infrastructure with a cloud control plane.  This offering will serve as the next generation Horizon Air Hybrid Mode.

Horizon Cloud will utilize one license for both the cloud-hosted and on-premises offering, and these licenses will be available in both named user and concurrent user options.

Other Enhancements

Like any Horizon release, there will be a multitude of enhancements.  A few of the other new enhancements that I’ve learned about are:

  • Instant Clone Desktop support for vGPU – Horizon 7.1 will allow you to run 3D workloads on Instant Clone Desktops
  • Multi-VLAN support for Instant Clones – Feature-parity with Linked-Clone desktops
  • Being able to manage multiple VLANs within Horizon Administrator – no need for a PowerShell script

by seanpmassey at February 07, 2017 01:05 PM

February 06, 2017

LZone - Sysadmin

How to fix debsecan for Wheezy

Someone at Debian increase security for all Debian servers by breaking debsecan a while ago for everything before Jessie by moving the vulnerability definitions from


Of course there was no way to issue a security fix for Wheezy debsecan...

Workaround 1: Hotfix

So if you still want to scan your Wheezy systems you can hotfix debsecan before running it like this:
sed -i "s/http:\/\/\/debian-secure-testing/https:\/\/\/tracker/;s/project\/debsecan\/release\/1\//debsecan\/release\/1\//" /usr/bin/debsecan 

Workaround 2: Pass Config

You can also pass an in-place config file:
debsecan --config <(echo SOURCE="")

February 06, 2017 05:23 PM

February 05, 2017

Feeding the Cloud

IPv6 and OpenVPN on Linode Debian/Ubuntu VPS

Here is how I managed to extend my OpenVPN setup on my Linode VPS to include IPv6 traffic. This ensures that clients can route all of their traffic through the VPN and avoid leaking IPv6 traffic, for example. It also enables clients on IPv4-only networks to receive a routable IPv6 address and connect to IPv6-only servers (i.e. running your own IPv6 broker).

Request an additional IPv6 block

The first thing you need to do is get a new IPv6 address block (or "pool" as Linode calls it) from which you can allocate a single address to each VPN client that connects to the server.

If you are using a Linode VPS, there are instructions on how to request a new IPv6 pool. Note that you need to get an address block between /64 and /112. A /116 like Linode offers won't work in OpenVPN. Thankfully, Linode is happy to allocate you an extra /64 for free.

Setup the new IPv6 address

If your server only has an single IPv4 address and a single IPv6 address, then a simple DHCP-backed network configuration will work fine. To add the second IPv6 block on the other hand, I had to change my network configuration (/etc/network/interfaces) to this:

auto lo
iface lo inet loopback

allow-hotplug eth0
iface eth0 inet dhcp
    pre-up iptables-restore /etc/network/iptables.up.rules

iface eth0 inet6 static
    address 2600:3c01::xxxx:xxxx:xxxx:939f/64
    gateway fe80::1
    pre-up ip6tables-restore /etc/network/ip6tables.up.rules

iface tun0 inet6 static
    address 2600:3c01:xxxx:xxxx::/64
    pre-up ip6tables-restore /etc/network/ip6tables.up.rules

where 2600:3c01::xxxx:xxxx:xxxx:939f/64 (bound to eth0) is your main IPv6 address and 2600:3c01:xxxx:xxxx::/64 (bound to tun0) is the new block you requested.

Once you've setup the new IPv6 block, test it from another IPv6-enabled host using:

ping6 2600:3c01:xxxx:xxxx::1

OpenVPN configuration

The only thing I had to change in my OpenVPN configuration (/etc/openvpn/server.conf) was to change:

proto udp


proto udp6

in order to make the VPN server available over both IPv4 and IPv6, and to add the following lines:

server-ipv6 2600:3c01:xxxx:xxxx::/64
push "route-ipv6 2000::/3"

to bind to the right V6 address and to tell clients to tunnel all V6 Internet traffic through the VPN.

In addition to updating the OpenVPN config, you will need to add the following line to /etc/sysctl.d/openvpn.conf:


and the following to your firewall (e.g. /etc/network/ip6tables.up.rules):

# openvpn
-A INPUT -p udp --dport 1194 -j ACCEPT
-A FORWARD -m state --state NEW -i tun0 -o eth0 -s 2600:3c01:xxxx:xxxx::/64 -j ACCEPT
-A FORWARD -m state --state NEW -i eth0 -o tun0 -d 2600:3c01:xxxx:xxxx::/64 -j ACCEPT

in order to ensure that IPv6 packets are forwarded from the eth0 network interface to tun0 on the VPN server.

With all of this done, apply the settings by running:

sysctl -p /etc/sysctl.d/openvpn.conf
systemctl restart openvpn.service

Testing the connection

Now connect to the VPN using your desktop client and check that the default IPv6 route is set correctly using ip -6 route.

Then you can ping the server's new IP address:

ping6 2600:3c01:xxxx:xxxx::1

and from the server, you can ping the client's IP (which you can see in the network settings):

ping6 2600:3c01:xxxx:xxxx::1002

Once both ends of the tunnel can talk to each other, you can try pinging an IPv6-only server from your client:


and then pinging your client from an IPv6-enabled host somewhere:

ping6 2600:3c01:xxxx:xxxx::1002

If that works, other online tests should also work.

February 05, 2017 01:20 PM