Planet SysAdmin


August 17, 2017

Chris Siebenmann

The three different names ZFS stores for each vdev disk (on Illumos)

I sort of mentioned yesterday that ZFS keeps information on several different ways of identifying disks in pools. To be specific, it keeps three different names or ways of identifying each disk. You can see this with 'zdb -C' on a pool, so here's a representative sample:

# zdb -C rpool
MOS Configuration:
[...]
  children[0]:
    type: 'disk'
    id: 0
    guid: 15557853432972548123
    path: '/dev/dsk/c3t0d0s0'
    devid: 'id1,sd@SATA_____INTEL_SSDSC2BB08__BTWL4114016X080KGN/a'
    phys_path: '/pci@0,0/pci15d9,714@1f,2/disk@0,0:a'
    [...]

The guid is ZFS's internal identifier for the disk, and is stored on the disk itself as part of the disk label. Since you have to find the disk to read it, it's not something that ZFS uses to find disks, although it is part of verifying that ZFS has found the right one. The three actual names for the disk are reported here as path, devid aka 'device id', and phys_path aka 'physical path'.

The path is straightforward; it's the filesystem path to the device, which here is a conventional OmniOS (Illumos, Solaris) cNtNdNsN name typical of a plain, non-multipathed disk. As this is a directly attached SATA disk, the phys_path shows us the PCI information about the controller for the disk in the form of a PCI device name. If we pulled this disk and replaced it with a new one, both of those would stay the same, since with a directly attached disk they're based on physical topology and neither has changed. However, the devid is clearly based on the disks's identity information; it has the vendor name, the 'product id', and the serial number (as returned by the disk itself in response to SATA inquiry commands). This will be the same more or less regardless of where the disk is connected to the system, so ZFS (and anything else) can find the disk wherever it is.

(I believe that the 'id1,sd@' portion of the devid is simply giving us a namespace for the rest of it. See 'prtconf -v' for another representation of all of this information and much more.)

Multipathed disks (such as the iSCSI disks on our fileservers) look somewhat different. For them, the filesystem device name (and thus path) looks like c5t<long identifier>d0s0, the physical path is /scsivhci/disk@g<long identifier>, and the devid_ is not particularly useful in finding the specific physical disk because our iSCSI targets generate synthetic disk 'serial numbers' based on their slot position (and the target's hostname, which at least lets me see which target a particular OmniOS-level multipathed disk is supposed to be coming from). As it happens, I already know that OmniOS multipathing identifies disks only by their device ids, so all three names are functionally the same thing, just expressed in different forms.

If you remove a disk entirely, all three of these names go away for both plain directly attached disks and multipath disks. If you replace a plain disk with a new or different one, the filesystem path and physical path will normally still work but the devid of the old disk is gone; ZFS can open the disk but will report that it has a missing or corrupt label. If you replace a multipathed disk with a new one and the true disk serial number is visible to OmniOS, all of the old names go away since they're all (partly) based on the disk's serial number, and ZFS will report the disk as missing entirely (often simply reporting it by GUID).

Sidebar: Which disk name ZFS uses when bringing up a pool

Which name or form of device identification ZFS uses is a bit complicated. To simplify a complicated situation (see vdev_disk_open in vdev_disk.c) as best I can, the normal sequence is that ZFS starts out by trying the filesystem path but verifying the devid. If this fails, it tries the devid, the physical path, and finally the filesystem path again (but without verifying the devid this time).

Since ZFS verifies the disk label's GUID and other details after opening the disk, there is no risk that finding a random disk this way (for example by the physical path) will confuse ZFS. It'll just cause ZFS to report things like 'missing or corrupt disk label' instead of 'missing device'.

by cks at August 17, 2017 03:48 AM

August 16, 2017

HolisticInfoSec.org

Toolsmith #127: OSINT with Datasploit

I was reading an interesting Motherboard article, Legal Hacking Tools Can Be Useful for Journalists, Too, that includes reference to one of my all time OSINT favorites, Maltego. Joseph Cox's article also mentions Datasploit, a 2016 favorite for fellow tools aficionado, Toolswatch.org, see 2016 Top Security Tools as Voted by ToolsWatch.org Readers. Having not yet explored Datasploit myself, this proved to be a grand case of "no time like the present."
Datasploit is "an #OSINT Framework to perform various recon techniques, aggregate all the raw data, and give data in multiple formats." More specifically, as stated on Datasploit documentation page under Why Datasploit, it utilizes various Open Source Intelligence (OSINT) tools and techniques found to be effective, and brings them together to correlate the raw data captured, providing the user relevant information about domains, email address, phone numbers, person data, etc. Datasploit is useful to collect relevant information about target in order to expand your attack and defense surface very quickly.
The feature list includes:
  • Automated OSINT on domain / email / username / phone for relevant information from different sources
  • Useful for penetration testers, cyber investigators, defensive security professionals, etc.
  • Correlates and collaborate results, shows them in a consolidated manner
  • Tries to find out credentials,  API keys, tokens, sub-domains, domain history, legacy portals, and more as related to the target
  • Available as single consolidating tool as well as standalone scripts
  • Performs Active Scans on collected data
  • Generates HTML, JSON reports along with text files
Resources
Github: https://github.com/datasploit/datasploit
Documentation: http://datasploit.readthedocs.io/en/latest/
YouTube: Quick guide to installation and use

Pointers
Second, a few pointers to keep you from losing your mind. This project is very much work in progress, lots of very frustrated users filing bugs and wondering where the support is. The team is doing their best, be patient with them, but read through the Github issues to be sure any bugs you run into haven't already been addressed.
1) Datasploit does not error gracefully, it just crashes. This can be the result of unmet dependencies or even a missing API key. Do not despair, take note, I'll talk you through it.
2) I suggest, for ease, and best match to documentation, run Datasploit from an Ubuntu variant. Your best bet is to grab Kali, VM or dedicated and load it up there, as I did.
3) My installation guidance and recommendations should hopefully get you running trouble free, follow it explicitly.
4) Acquire as many API keys as possible, see further detail below.

Installation and preparation
From Kali bash prompt, in this order:

  1. git clone https://github.com/datasploit/datasploit /etc/datasploit
  2. apt-get install libxml2-dev libxslt-dev python-dev lib32z1-dev zlib1g-dev
  3. cd /etc/datasploit
  4. pip install -r requirements.txt
  5. mv config_sample.py config.py
  6. With your preferred editor, open config.py and add API keys for the following at a minimum, they are, for all intents and purposes required, detailed instructions to acquire each are here:
    1. Shodan API
    2. Censysio ID and Secret
    3. Clearbit API
    4. Emailhunter API
    5. Fullcontact API
    6. Google Custom Search Engine API key and CX ID
    7. Zoomeye Username and Password
If, and only if, you've done all of this correctly, you might end up with a running instance of Datasploit. :-) Seriously, this is some of the glitchiest software I've tussled with in quite a while, but the results paid handsomely. Run python datasploit.py domain.com, where domain.com is your target. Obviously, I ran python datasploit.py holisticinfosec.org to acquire results pertinent to your author. 
Datasploit rapidly pulled results as follows:
211 domain references from Github:
Github results
Luckily, no results from Shodan. :-)
Four results from Paste(s): 
Pastebin and Pastie results
Datasploit pulled russ at holisticinfosec dot org as expected, per email harvesting.
Accurate HolisticInfoSec host location data from Zoomeye:

Details regarding HolisticInfoSec sub-domains and page links:
Sub-domains and page links
Finally, a good return on DNS records for holisticinfosec.org and, thankfully, no vulns found via PunkSpider

DataSploit can also be integrated into other code and called as individual scripts for unique functions. I did a quick run with python emailOsint.py russ@holisticinfosec.org and the results were impressive:
Email OSINT
I love that the first query is of Troy Hunt's Have I Been Pwned. Not sure if you have been? Better check it out. Reminder here, you'll really want to be sure to have as many API keys as possible or you may find these buggy scripts crashing. You'll definitely find yourself compromising between frustration and the rapid, detailed results. I put this offering squarely in the "shows much promise category" if the devs keep focus on it, assess for quality, and handle errors better.
Give Datasploit a try for sure.
Cheers, until next time...

by Russ McRee (noreply@blogger.com) at August 16, 2017 02:54 PM

Chris Siebenmann

Things I do and don't know about how ZFS brings pools up during boot

If you import a ZFS pool explicitly, through 'zpool import', the user-mode side of the process normally searches through all of the available disks in order to find the component devices of the pool. Because it does this explicit search, it will find pool devices even if they've been shuffled around in a way that causes them to be renamed, or even (I think) drastically transformed, for example by being dd'd to a new disk. This is pretty much what you'd expect, since ZFS can't really read what the pool thinks its configuration is until it assembles the pool. When it imports such a pool, I believe that ZFS rewrites the information kept about where to find each device so that it's correct for the current state of your system.

This is not what happens when the system boots. To the best of my knowledge, for non-root pools the ZFS kernel module directly reads /etc/zfs/zpool.cache during module initialization and converts it into a series of in-memory pool configurations for pools, which are all in an unactivated state. At some point, magic things attempt to activate some or all of these pools, which causes the kernel to attempt to open all of the devices listed as part of the pool configuration and verify that they are indeed part of the pool. The process of opening devices only uses the names and other identification of the devices that's in the pool configuration; however, one identification is a 'devid', which for many devices is basically the model and serial number of the disk. So I believe that under at least some circumstances the kernel will still be able to find disks that have been shuffled around, because it will basically seek out that model plus serial number wherever it's (now) connected to the system.

(See vdev_disk_open in vdev_disk.c for the gory details, but you also need to understand Illumos devids. The various device information available for disks in a pool can be seen with 'zdb -C <pool>'.)

To the best of my knowledge, this in-kernel activation makes no attempt to hunt around on other disks to complete the pool's configuration the way that 'zpool import' will. In theory, assuming that finding disks by their devid works, this shouldn't matter most or basically all of the time; if that disk is there at all, it should be reporting its model and serial number and I think the kernel will find it. But I don't know for sure. I also don't know how the kernel acts if some disks take a while to show up, for example iSCSI disks.

(I suspect that the kernel only makes one attempt at pool activation and doesn't retry things if more devices show up later. But this entire area is pretty opaque to me.)

These days you also have your root filesystems on a ZFS pool, the root pool. There are definitely some special code paths that seem to be invoked during boot for a ZFS root pool, but I don't have enough knowledge of the Illumos boot time environment to understand how they work and how they're different from the process of loading and starting non-root pools. I used to hear that root pools were more fragile if devices moved around and you might have to boot from alternate media in order to explicitly 'zpool import' and 'zpool export' the root pool in order to reset its device names, but that may be only folklore and superstition at this point.

by cks at August 16, 2017 04:37 AM

August 15, 2017

Evaggelos Balaskas

Nagios with PNP4Nagios on CentOS 6.x

nagios_logo.png

In many companies, nagios is the de-facto monitoring tool. Even with new modern alternatives solutions, this opensource project, still, has a large amount of implementations in place. This guide is based on a “clean/fresh” CentOS 6.9 virtual machine.

Epel

An official nagios repository exist in this address: https://repo.nagios.com/
I prefer to install nagios via the EPEL repository:

# yum -y install http://fedora-mirror01.rbc.ru/pub/epel/6/x86_64/epel-release-6-8.noarch.rpm

# yum info nagios | grep Version
Version     : 4.3.2

# yum -y install nagios

Selinux

Every online manual, suggest to disable selinux with nagios. There is a reason for that ! but I will try my best to provide info on how to keep selinux enforced. To write our own nagios selinux policies the easy way, we need one more package:

# yum -y install policycoreutils-python

Starting nagios:

# /etc/init.d/nagios restart

will show us some initial errors in /var/log/audit/audit.log selinux log file

Filtering the results:

# egrep denied /var/log/audit/audit.log | audit2allow

will display something like this:

#============= nagios_t ==============
allow nagios_t initrc_tmp_t:file write;
allow nagios_t self:capability chown;

To create a policy file based on your errors:

# egrep denied /var/log/audit/audit.log | audit2allow -a -M nagios_t

and to enable it:

# semodule -i nagios_t.pp

BE AWARE this is not the only problem with selinux, but I will provide more details in few moments.

Nagios

Now we are ready to start the nagios daemon:

# /etc/init.d/nagios restart

filtering the processes of our system:

# ps -e fuwww | egrep na[g]ios

nagios    2149  0.0  0.1  18528  1720 ?        Ss   19:37   0:00 /usr/sbin/nagios -d /etc/nagios/nagios.cfg
nagios    2151  0.0  0.0      0     0 ?        Z    19:37   0:00  _ [nagios] <defunct>
nagios    2152  0.0  0.0      0     0 ?        Z    19:37   0:00  _ [nagios] <defunct>
nagios    2153  0.0  0.0      0     0 ?        Z    19:37   0:00  _ [nagios] <defunct>
nagios    2154  0.0  0.0      0     0 ?        Z    19:37   0:00  _ [nagios] <defunct>
nagios    2155  0.0  0.0  18076   712 ?        S    19:37   0:00  _ /usr/sbin/nagios -d /etc/nagios/nagios.cfg

super!

Apache

Now it is time to start our web server apache:

# /etc/init.d/httpd restart

Starting httpd: httpd: apr_sockaddr_info_get() failed
httpd: Could not reliably determine the server's fully qualified domain name, using 127.0.0.1 for ServerName

This is a common error, and means that we need to define a ServerName in our apache configuration.

First, we give an name to our host file:

# vim /etc/hosts

for this guide, I ‘ll go with the centos69 but you can edit that according your needs:

127.0.0.1   localhost localhost.localdomain localhost4 localhost4.localdomain4 centos69

then we need to edit the default apache configuration file:

# vim /etc/httpd/conf/httpd.conf

#ServerName www.example.com:80
ServerName centos69

and restart the process:

# /etc/init.d/httpd restart

Stopping httpd:      [  OK  ]
Starting httpd:      [  OK  ]

We can see from the netstat command that is running:

# netstat -ntlp | grep 80

tcp        0      0 :::80                       :::*                        LISTEN      2729/httpd      

Firewall

It is time to fix our firewall and open the default http port, so that we can view the nagios from our browser.
That means, we need to fix our iptables !

# iptables -A INPUT -m state --state NEW -m tcp -p tcp --dport 80 -j ACCEPT

this is want we need. To a more permanent solution, we need to edit the default iptables configuration file:

# vim /etc/sysconfig/iptables

and add the below entry on INPUT chain section:

-A INPUT -m state --state NEW -m tcp -p tcp --dport 80 -j ACCEPT

Web Browser

We are ready to fire up our web browser and type the address of our nagios server.
Mine is on a local machine with the IP: 129.168.122.96, so

http://192.168.122.96/nagios/

User Authentication

The default user authentication credentials are:

nagiosadmin // nagiosadmin

but we can change them!

From our command line, we type something similar:

# htpasswd -sb /etc/nagios/passwd nagiosadmin e4j9gDkk6LXncCdDg

so that htpasswd will update the default nagios password entry on the /etc/nagios/passwd with something else, preferable random and difficult password.

ATTENTION: e4j9gDkk6LXncCdDg is just that, a random password that I created for this document only. Create your own and dont tell anyone!

Selinux, Part Two

at this moment and if you are tail-ing the selinux audit file, you will see some more error msgs.

Below, you will see my nagios_t selinux policy file with all the things that are needed for nagios to run properly - at least at the moment.!

module nagios_t 1.0;

require {
        type nagios_t;
        type initrc_tmp_t;
        type nagios_spool_t;
        type nagios_system_plugin_t;
        type nagios_exec_t;
        type httpd_nagios_script_t;
        class capability chown;
        class file { write read open execute_no_trans getattr };
}

#============= httpd_nagios_script_t ==============
allow httpd_nagios_script_t nagios_spool_t:file { open read getattr };

#============= nagios_t ==============
allow nagios_t initrc_tmp_t:file write;
allow nagios_t nagios_exec_t:file execute_no_trans;
allow nagios_t self:capability chown;

#============= nagios_system_plugin_t ==============
allow nagios_system_plugin_t nagios_exec_t:file getattr;

Edit your nagios_t.te file accordingly and then build your selinux policy:

# make -f /usr/share/selinux/devel/Makefile

You are ready to update the previous nagios selinux policy :

# semodule -i nagios_t.pp

Selinux - Nagios package

So … there is an rpm package with the name: nagios-selinux on Version: 4.3.2
you can install it, but does not resolve all the selinux errors in audit file ….. so ….
I think my way is better, cause you can understand some things better and have more flexibility on defining your selinux policy

Nagios Plugins

Nagios is the core process, daemon. We need the nagios plugins - the checks !
You can do something like this:

# yum install nagios-plugins-all.x86_64

but I dont recommend it.

These are the defaults :

nagios-plugins-load-2.2.1-4git.el6.x86_64
nagios-plugins-ping-2.2.1-4git.el6.x86_64
nagios-plugins-disk-2.2.1-4git.el6.x86_64
nagios-plugins-procs-2.2.1-4git.el6.x86_64
nagios-plugins-users-2.2.1-4git.el6.x86_64
nagios-plugins-http-2.2.1-4git.el6.x86_64
nagios-plugins-swap-2.2.1-4git.el6.x86_64
nagios-plugins-ssh-2.2.1-4git.el6.x86_64

# yum -y install nagios-plugins-load nagios-plugins-ping nagios-plugins-disk nagios-plugins-procs nagios-plugins-users nagios-plugins-http nagios-plugins-swap nagios-plugins-ssh

and if everything is going as planned, you will see something like this:

nagios_checks.jpg

PNP4Nagios

It is time, to add pnp4nagios a simple graphing tool and get read the nagios performance data and represent them to graphs.

# yum info pnp4nagios | grep Version
Version     : 0.6.22

# yum -y install pnp4nagios

We must not forget to restart our web server:

# /etc/init.d/httpd restart

Bulk Mode with NPCD

I’ve spent toooooo much time to understand why the default Synchronous does not work properly with nagios v4x and pnp4nagios v0.6x
In the end … this is what it works - so try not to re-invent the wheel , as I tried to do and lost so many hours.

Performance Data

We need to tell nagios to gather performance data from their check:

# vim +/process_performance_data /etc/nagios/nagios.cfg

process_performance_data=1

We also need to tell nagios, what to do with this data:

nagios.cfg

# *** the template definition differs from the one in the original nagios.cfg
#
service_perfdata_file=/var/log/pnp4nagios/service-perfdata
service_perfdata_file_template=DATATYPE::SERVICEPERFDATAtTIMET::$TIMET$tHOSTNAME::$HOSTNAME$tSERVICEDESC::$SERVICEDESC$tSERVICEPERFDATA::$SERVICEPERFDATA$tSERVICECHECKCOMMAND::$SERVICECHECKCOMMAND$tHOSTSTATE::$HOSTSTATE$tHOSTSTATETYPE::$HOSTSTATETYPE$tSERVICESTATE::$SERVICESTATE$tSERVICESTATETYPE::$SERVICESTATETYPE$
service_perfdata_file_mode=a
service_perfdata_file_processing_interval=15
service_perfdata_file_processing_command=process-service-perfdata-file

# *** the template definition differs from the one in the original nagios.cfg
#
host_perfdata_file=/var/log/pnp4nagios/host-perfdata
host_perfdata_file_template=DATATYPE::HOSTPERFDATAtTIMET::$TIMET$tHOSTNAME::$HOSTNAME$tHOSTPERFDATA::$HOSTPERFDATA$tHOSTCHECKCOMMAND::$HOSTCHECKCOMMAND$tHOSTSTATE::$HOSTSTATE$tHOSTSTATETYPE::$HOSTSTATETYPE$
host_perfdata_file_mode=a
host_perfdata_file_processing_interval=15
host_perfdata_file_processing_command=process-host-perfdata-file

Commands

In the above configuration, we introduced two new commands

service_perfdata_file_processing_command  &
host_perfdata_file_processing_command

We need to define them in the /etc/nagios/objects/commands.cfg file :

#
# Bulk with NPCD mode
#
define command {
       command_name    process-service-perfdata-file
       command_line    /bin/mv /var/log/pnp4nagios/service-perfdata /var/spool/pnp4nagios/service-perfdata.$TIMET$
}

define command {
       command_name    process-host-perfdata-file
       command_line    /bin/mv /var/log/pnp4nagios/host-perfdata /var/spool/pnp4nagios/host-perfdata.$TIMET$
}

If everything have gone right … then you will be able to see on a nagios check something like this:

nagios_perf.png

Verify

Verify your pnp4nagios setup:

# wget -c http://verify.pnp4nagios.org/verify_pnp_config

# perl verify_pnp_config -m bulk+npcd -c /etc/nagios/nagios.cfg -p /etc/pnp4nagios/ 

NPCD

The NPCD daemon (Nagios Performance C Daemon) is the daemon/process that will translate the gathered performance data to graphs, so let’s started it:

# /etc/init.d/npcd restart
Stopping npcd:                                             [FAILED]
Starting npcd:                                             [  OK  ]

You should see some warnings but not any critical errors.

Templates

Two new template definition should be created, one for the host and one for the service:

/etc/nagios/objects/templates.cfg

define host {
   name       host-pnp
   action_url /pnp4nagios/index.php/graph?host=$HOSTNAME$&srv=_HOST_' class='tips' rel='/pnp4nagios/index.php/popup?host=$HOSTNAME$&srv=_HOST_
   register   0
}

define service {
   name       srv-pnp
   action_url /pnp4nagios/graph?host=$HOSTNAME$&srv=$SERVICEDESC$' class='tips' rel='/pnp4nagios/popup?host=$HOSTNAME$&srv=$SERVICEDESC$
   register   0
}

Host Definition

Now we need to apply the host-pnp template to our system:

so this configuration: /etc/nagios/objects/localhost.cfg

define host{
        use                     linux-server            ; Name of host template to use
                                                        ; This host definition will inherit all variables that are defined
                                                        ; in (or inherited by) the linux-server host template definition.
        host_name               localhost
        alias                   localhost
        address                 127.0.0.1
        }

becomes:

define host{
        use                     linux-server,host-pnp            ; Name of host template to use
                                                        ; This host definition will inherit all variables that are defined
                                                        ; in (or inherited by) the linux-server host template definition.
        host_name               localhost
        alias                   localhost
        address                 127.0.0.1
        }

Service Definition

And we finally must append the pnp4nagios service template to our services:

srv-pnp

define service{
        use                             local-service,srv-pnp         ; Name of service template to use
        host_name                       localhost

Graphs

We should be able to see graphs like these:

nagios_ping.png

Happy Measurements!

appendix

These are some extra notes on the above article, you need to take in mind:

Services

# chkconfig httpd on
# chkconfig iptables on
# chkconfig nagios on
# chkconfig npcd on 

PHP

If you are not running the default php version on your system, it is possible to get this error msg:

Non-static method nagios_Core::SummaryLink()

There is a simply solution for that, you need to modify the index file to exclude the deprecated php error msgs:

# vim +/^error_reporting /usr/share/nagios/html/pnp4nagios/index.php   

// error_reporting(E_ALL & ~E_STRICT);
error_reporting(E_ALL & ~E_STRICT & ~E_DEPRECATED);

August 15, 2017 06:18 PM

Errata Security

Why that "file-copy" forensics of DNC hack is wrong

People keep asking me about this story about how forensics "experts" have found proof the DNC hack was an inside job, because files were copied at 22-megabytes-per-second, faster than is reasonable for Internet connections.

This story is bogus.

Yes, the forensics is correct that at some point, files were copied at 22-mBps. But there's no evidence this was the point at Internet transfer out of the DNC.

One point might from one computer to another within the DNC. Indeed, as someone experienced doing this sort of hack, it's almost certain that at some point, such a copy happened. The computers you are able to hack into are rarely the computers that have the data you want. Instead, you have to copy the data from other computers to the hacked computer, and then exfiltrate the data out of the hacked computer.

Another point might have been from one computer to another within the hacker's own network, after the data was stolen. As a hacker, I can tell you that I frequently do this. Indeed, as this story points out, the timestamps of the file shows that the 22-mBps copy happened months after the hack was detected.

If the 22-mBps was the copy exfiltrating data, it might not have been from inside the DNC building, but from some cloud service, as this tweet points out. Hackers usually have "staging" servers in the cloud that can talk to other cloud serves at easily 10 times the 22-mBps, even around the world. I have staging servers that will do this, and indeed, have copied files at this data rate. If the DNC had that data or backups in the cloud, this would explain it. 

My point is that while the forensic data-point is good, there's just a zillion ways of explaining it. It's silly to insist on only the one explanation that fits your pet theory.

As a side note, you can tell this already from the way the story is told. For example, rather than explain the evidence and let it stand on its own, the stories hype the credentials of those who believe the story, using the "appeal to authority" fallacy.


by Robert Graham (noreply@blogger.com) at August 15, 2017 06:02 AM

Chris Siebenmann

How to get per-user fair share scheduling on Ubuntu 16.04 (with systemd)

When I wrote up imposing temporary CPU and memory limits on a user on Ubuntu 16.04, I sort of discovered that I had turned on per-user fair share CPU scheduling as a side effect, although I didn't understand exactly how to do this deliberately. Armed with a deeper understanding of how to tell if fair share scheduling was on, I've now done a number of further experiments and I believe I have definitive answers. This applies only to Ubuntu 16.04 and its version of systemd as configured by Ubuntu; it doesn't seem to apply to, for example, a stock Fedora 26 system.

To enable per user fair share CPU scheduling, it appears that you must do two things:

  • First, set CPUAccounting=true on user.slice. You can do this temporarily with 'systemctl --runtime set-property' or permanently enable it.

  • Second, arrange to have CPUAccounting=true set on an active user slice. If you do this temporarily with 'systemctl --runtime', the user must be logged in with some sort of session at the time. If you do this permanently, nothing happens until that user logs in and systemd creates their user-${UID}.slice slice.

Once you've done both of these, all future (user) sessions from any user will have their processes included in per-user fair share scheduling. If you used 'systemctl --runtime' on a user-${UID}.slice, it doesn't matter if that user logs completely out and their slice goes away; the fair share scheduling sticks despite this. However, fair-share scheduling goes away if all users log out and user.slice is removed by systemd. You need at least one remaining user session at all times to keep user.slice still in use (a detached screen session will do).

If you want to force existing processes to be subject to per-user fair share scheduling, you must arrange to set CPUAccounting=true on all current user scopes:

for i in $(systemctl -t scope list-units |
           awk '{print $1}' |
           grep '^session-.*\.scope$'); do
    systemctl --runtime set-property $i CPUAccounting=true
done

This creates a slightly different cgroup hierarchy than you'll get from completely proper fair share scheduling, but the differences are probably unimportant in practice. In regular fair share scheduling, all processes from the same user are grouped together under user.slice/user-${UID}.slice, so they contend evenly with each other. When you force scopes this way, processes get grouped into their scopes, so they go in user.slice/user-${UID}.slice/session-<blah>.scope; as a result, a user's scopes also are fair-share scheduled against each other. This only applies to current processes and scopes; as users log out and then back in again, their new processes will be all grouped together.

If you have a sufficiently small number of users who will log in to your machines and run CPU-consuming things, it's feasible to create permanent settings for each of them with 'systemctl set-property user-${UID}.slice CPUAccounting=true'. If you have lots of users, as we do, this is infeasible; if nothing else, your /etc/systemd/system directory would wind up utterly cluttered. This means that you have to do it on the fly (and then do it again if all user sessions ended and systemd deleted user.slice).

This is where we run into an important limitation of per user fair share scheduling on a normally configured Ubuntu 16.04. As we've set fair-share scheduling up, this only applies to processes that are under user.slice; system processes are not fair-share scheduled. It turns out that user cron jobs don't run under user.slice and so are not fair-share scheduled. All processes created by user cron entries wind up all grouped together under cron.service; there is no per-user separation and nothing is put under user slices.

(It's possible that you can change this with PAM magic, but this is how a normal Ubuntu 16.04 machine behaves.)

I discovered this because I had the clever idea that I could use a root @reboot /etc/cron.d entry to set things on user.slice and user-0.slice shortly after the system booted. Attempting to do this led to the discovery that neither slice actually existed when my @reboot job ran, and that my process was under cron.service instead. As far as I can see there's no way around this; there just doesn't seem to be a systemd command that will run a command for you under a user slice.

(If there was, you could make a root @reboot crontab that ran the necessary systemctl commands and then didn't exit, so there would always be an active user slice so that user.slice wouldn't get removed by systemd.)

PS: My solution was to wrap up all of these steps into a shell script that we can run if we need to turn on fair-share scheduling on some machine because a bunch of users are contending over it. Such an on demand, on the fly solution is good enough for our case (even if it doesn't include crontab jobs, which is a real pity for some machines).

by cks at August 15, 2017 04:03 AM

August 13, 2017

Sarah Allen

call it what it is

The news from Charlottesville is not an isolated incident. The US President responds that there is violence “on many sides.” This is false.

In Charlottesville, the police were present in riot gear, but the rioting white people were treated with kid gloves, a stark difference from Fergusen police reaction reported by @JayDowTV who was there.

These people feel comfortable declaring it a war: “And to everyone, know this: we are now at war. And we are not going to back down. There will be more events. Soon. We are going to start doing this nonstop. Across the country. I’m going to arrange them myself. Others will too, I’m sure, but I’m telling you now: I am going to start arranging my own events. We are going to go bigger than Charlottesville. We are going to go huge.”

General McMaster, US National Security Advisor tells us “I think what terrorism is, is the use of violence to incite terror and fear. And, of course, it was terrorism” (NBC via Vox)

A Christian minister writes this plainly on his post, Yes, This is Racism. We each need to declare what we stand for in this moment and always.

We are not with you, torch-bearers, in Charlottesville or anywhere.
We do not consent to this.
In fact we stand against you, alongside the very beautiful diversity that you fear.
We stand with people of every color and of all faiths, people of every orientation, nationality, and native tongue.

We are not going to have this. This is not the country we’ve built together and it will not become what you intend it to become.

So you can kiss our diverse, unified, multi-colored behinds because your racism and your terrorism will not win the day.

Believe it.

— John Pavlovitz

by sarah at August 13, 2017 10:05 PM

August 12, 2017

Steve Kemp's Blog

A day in the life of Steve

I used to think I was a programmer who did "sysadmin-stuff". Nowadays I interact with too many real programmers to believe that.

Or rather I can code/program/develop, but I'm not often as good as I could be. These days I'm getting more consistent with writing tests, and I like it when things are thoroughly planned and developed. But too often if I'm busy, or distracted, I think to myself "Hrm .. compiles? Probably done. Oops. Bug, you say?"

I was going to write about working with golang today. The go language is minimal and quite neat. I like the toolset:

  • go fmt
    • Making everything consistent.
  • go test

Instead I think today I'm going to write about something else. Since having a child a lot of my life is different. Routine becomes something that is essential, as is planning and scheduling.

So an average week-day goes something like this:

  • 6:00AM
    • Wake up (naturally).
  • 7:00AM
    • Wake up Oiva and play with him for 45 minutes.
  • 7:45AM
    • Prepare breakfast for my wife, and wake her up, then play with Oiva for another 15 minutes while she eats.
  • 8:00AM
    • Take tram to office.
  • 8:30AM
    • Make coffee, make a rough plan for the day.
  • 9:00AM
    • Work, until lunchtime which might be 1pm, 2pm, or even 3pm.
  • 5:00PM
    • Leave work, and take bus home.
    • Yes I go to work via tram, but come back via bus. There are reasons.
  • 5:40PM
    • Arrive home, and relax in peace for 20 minutes.
  • 6:00PM-7:00PM
    • Take Oiva for a walk, stop en route to relax in a hammock for 30 minutes reading a book.
  • 7:00-7:20PM
    • Feed Oiva his evening meal.
  • 7:30PM
    • Give Oiva his bath, then pass him over to my wife to put him to bed.
  • 7:30PM - 8:00pm
    • Relax
  • 8:00PM - 10:00PM
    • Deal with Oiva waking up, making noises, or being unsettled.
    • Try to spend quality time with my wife, watch TV, read a book, do some coding, etc.
  • 10:00PM ~ 11:30PM
    • Go to bed.

In short I'm responsible for Oiva from 6ish-8ish in the morning, then from 6PM-10PM (with a little break while he's put to bed.) There are some exceptions to this routine - for example I work from home on Monday/Friday afternoons, and Monday evenings he goes to his swimming classes. But most working-days are the same.

Weekends are a bit different. There I tend to take him 6AM-8AM, then 1PM-10PM with a few breaks for tea, and bed. At the moment we're starting to reach the peak-party time of year, which means weekends often involve negotiation(s) about which parent is having a party, and which parent is either leaving early, or not going out at all.

Today I have him all day, and it's awesome. He's just learned to say "Daddy" which makes any stress, angst or unpleasantness utterly worthwhile.

August 12, 2017 09:00 PM

OpenSSL

Random Thoughts

The next release will include a completely overhauled version of the random number facility, the RAND API. The default RAND method is now based on a Deterministic Random Bit Generator (DRBG) implemented according to the NIST recommendation 800-90A. We have also edited the documentation, allowed finer-grained configuration of how to seed the generator, and updated the default seeding mechanisms.

There will probably be more changes before the release is made, but they should be comparatively minor.

Read on for more details.

Background; NIST

The National Institute of Standards and Technologies (NIST) is part of the US Federal Government. Part of their mission is to advance standards. NIST has a long history of involvement with cryptography, including DES, the entire family of SHA digests, AES, various elliptic curves, and so on. They produce a series of Special Publications. One set, SP 800, is the main way they publish guidelines, recommendations, and reference materials in the computer security area.

NIST SP 800-90A rev1 is titled Recommendation for Random Number Generation Using Deterministic Random Bit Generators. Yes, that’s a mouthful! But note that if you generate enough random bits, you get a random byte, and if you generate enough bytes you can treat it as a random number, often a BN in OpenSSL terminology. So when you see “RBG” think “RNG.” :)

A non-deterministic RBG (NRBG) has to be based on hardware where every bit output is based on an unpredictable physical process. A fun example of this is the old Silicon Graphics LavaRand, now recreated and running at CloudFlare, which takes pictures of a lava lamp and digitizes them.

A deterministic RBG (DRBG) uses an algorithm to generate a sequence of bits from an initial seed, and that seed must be based on a true randomness source. This is a divide and conquer approach: if the algorithm has the right properties, the application only needs a small input of randomness (16 bytes for our algorithm) to generate many random bits. It will also need to be periodically reseeded, but when that needs to happen can also be calculated.

The NIST document provides a framework for defining a DRBG, including requirements on the operating environment and a lifecycle model that is slightly different from OpenSSL’s XXX_new()/XXX_free() model. NIST treats creation as relatively heavyweight, and allows a single DRBG to be instantiated or not during its lifetime.

NIST SP 800-90 defined four DRBG algorithms. One of these was “Dual Elliptic Curve” which was later shown to be deliberately vulnerable. For a really good explanation of this, see Steve Checkoway’s talk at the recent IETF meeting. An update to the document was made, the above-linked 90A revision 1, and Dual-EC DRBG was removed.

OpenSSL currently implements the AES-counter version, which is also what Google’s BoringSSL and Amazon’s s2n use. Our tests include the NIST known answer tests (KATs), so we are confident that the algorithm is pretty correct. It also uses AES in a common way, which increases out confidence in its correctness.

What we did

First, we cleaned up our terminology in the documentation and code comments. The term entropy is both highly technical and confusing. It is used in 800-90A in very precise ways, but in our docs it was usually misleading, so we modified the documents to use the more vague but accurate term randomness. We also tightened up some of the implementation and made some semantics more precise; e.g., RAND_load_file now only reads regular files.

Next, we imported the AES-based DRBG from the OpenSSL FIPS project, and made it the default RAND method. The old method, which tried an ad hoc set of methods to get seed data, has been removed. We added a new configuration parameter, --with-rand-seed, which takes a comma-separated list of values for seed sources. Each method is tried in turn, stopping when enough bits of randomness have been collected. The types supported are:

  • os which is the default and only one supported on Windows, VMS and some older legacy platforms. Most of the time this is the right value to use, and it can therefore be omitted.
  • getrandom which uses the getrandom() system call
  • devrandom which tries special devices like /dev/urandom
  • egd which uses the Entropy Gathering Daemon protocol
  • none for no seeding (don’t use this)
  • rdcpu for X86 will try to use RDSEED and RDRAND instructions
  • librandom currently not implemented, but could use things like arc4random() when available.

If running on a Linux kernel, the default of os will turn on devrandom. If you know you have an old kernel and cannot upgrade, you should think about using rdcpu. Implementation details can be seen by looking at the files in the crypto/rand directory, if you’re curious. They’re relatively well-commented, but the implementation could to change in the future. Also note that seeding happens automatically, so unless you have a very special environment, you should not ever need to call RAND_add(), RAND_seed() or other seeding routines. See the manpages for details.

It’s possible to “chain” DRBG’s, so that the ouput of one becomes the seed input for another. Each SSL object now has its own DRBG, chained from the global one. All random bytes, like the pre-master secret and session ID’s, are generated from that one. This can reduce lock contention, and might result in needing to seed less often.

We also added a separate global DRBG for private key generation and added API’s to use it. This object isn’t reachable directly, but it is used by the new BN_priv_rand and BN_priv_rand_range API’s. Those API’s, in turn, are used by all private-key generating functions. We like the idea of keeping the private-key DRBG “away from” the general public one; this is common practice. We’re not sure how this idea, and the per-SSL idea, interact and we’ll be evaluating that for a future release. One possibility is to have two DRBG’s per-thread, and remove the per-SSL one. We’d appreciate any suggestions on how to evaluate and decide what to do.

What we didn’t do

The RAND_DRBG API isn’t public, but it probably will be in a future release. We want more time to play around with the API and see what’s most useful and needed.

It’s not currently possible to change either global DRBG; for example, to use an AES-256 CTR which is also implemented. This is also something under consideration for the future. The proper way to do both of these might be to make the existing RAND_METHOD datatype opaque; we missed doing so in the 1.1.0 release.

Acknowledgements

This recent round of work started with some posts on the cryptography mailing list. There was also discussion on the openssl-dev mailing list and in a handful of PR’s. Many people participated, but in particular we’d like to thank (in no particular order): Colm MacCarthaigh, Gilles Van Assche, John Denker, Jon Callas, Mark Steward, Chris Wood, Matthias St. Pierre, Peter Waltenberg, and Ted T’so.

August 12, 2017 08:00 PM

Electricmonk.nl

Why the EasyList DMCA takedown notice is perfectly valid (and fair)

There's a story going around about a DMCA takedown notice filed against the adblocking list EasyList, causing them to remove a domain from one of their filters. As usual, lots of indignation and pitchfork-mobbing followed. It seems though, that few people understand what exactly is going on here.

Here's a brief history of ad blocking on the web:

  1. Publishers start putting ads on their websites.
  2. Publishers start putting annoying and dangerous ads on their websites.
  3. Adblockers appear that filter out ads.
  4. Cat-and-mouse game where publishers try to circumvent adblockers and adblockers blocking their attempts at circumvention.
  5. Publishers start putting up paywalls and competely blocking browsers that have adblocking.
  6. Adblockers start circumventing paywalls and measures put in place by publishers to ban people that are blocking ads.

This whole EasyList DMCA thing is related to point 5 and 6.

Admiral is a company that provides services to implement point 5: block people completely from websites if they block ads.

Let's get something clear: I'm extremely pro-adblocking. Publishers have time and again shown that they can't handle the responsibility of putting decent ads on websites. It always devolves into shitty practices such as auto-playing audio, flashing epilepsy bullshit, popups, malware infested crap, etc. Anything to get those few extra ad impressions and ad quality be damned! This has happened again and again and again in every possible medium from television to newspapers. So you can pry my adblocker from my cold dead hands and publishers can go suck it.

BUT I'm also of the firm belief that publishers have every right to completely ban me from their contents if I'm not willing to look at ads. It's their contents and it's their rules. If they want to block me, I'm just fine with that. I probably don't care about their content all that much anyway.

Now, for the DMCA notice.

A commit to EasyList added functionalclam.com to EasyList. According to Admiral, functionalclam.com is a domain used to implement point 5: blocking users from websites if they're using adblockers. According to this lawyer:

The Digital Millennium Copyright Act ("DMCA") makes it illegal to circumvent technical measures (e.g., encryption, copy protection) that prevent access to copyrighted materials, such as computer software or media content. The DMCA also bans the distribution of products or services that are designed to carry out circumvention of technical measures that prevent either access to or copying of copyrighted materials.

Since the functionalclam.com domain is a technical means to prevent access to copyrighted materials, the DMCA takedown notice is completely valid. And actually, it's a pretty honorable one. From the DMCA takedown notice itself (scroll down):

>>> What would be the best solution for the alleged infringement? Are there specific changes the other person can make other than removal?

Full repository takedown [of EasyList] should not be necessary. Instead, the repository owner can remove functionalclam[.]com from the file in question and not replace with alternative circumvention attempts.

EasyList themselves agree that their list should not be used to circumvent such access control:

If it is a Circumvention/Adblock-Warning adhost, it should be removed from Easylist even without the need for a DMCA request.

So put down your pitchforks because:

  • This has nothing to do with ad blocking, it has to do with circumventing access control.
  • Adblocking is not under attack,
  • The DMCA notice was not abuse of the DMCA.
  • Pulbishers have every right to ban adblock users from their content.
  • Y'all need to start growing some skepticism. When you read clickbaity titles such as "Ad blocking is under attack", you should immediately be suspicious. In fact, always be suspicious.

by admin at August 12, 2017 06:25 AM

August 11, 2017

SysAdmin1138

Reputation services, a brief history

Or,

The Problem of Twitter, Hatemobs, and Denial of Service

The topic of shared blocklists is hot right now. For those who avoid the blue bird, a shared blocklist is much like a shared killfile from Ye Olde Usenet, or an RBL for spam. Subscribe to one, and get a curated feed of people you never want to hear from again. It's an idea that's been around for decades, applied to a new platform.

However, internet-scale has caught up with the technique.

Usenet

A Usenet killfile was a feature of NNTP clients where posts meeting a regex would not even get displayed. If you've ever wondered what the vengeful:

*Plonk!*

...was about? This is what it was referring to. It was a public way of saying:

I have put you into my killfile, and I am now telling you I have done so, you asshole.

This worked because in the Usenet days, the internet was a much smaller place. Once in a while you'd get waves of griefers swarming a newsgroup, but that was pretty rare. You legitimately could remove most content you didn't want to see from your notice. The *Plonk!* usage still exists today, and I'm seeing some twitter users use that to indicate a block is being deployed. I presume these are veterans of many a Usenet flame-war.

RBLs

The Realtime Blackhole Lists (RBL) were pioneered as an anti-spam technique. Mail administrators could subscribe to these, and all incoming mail-connections could be checked against it. If it was listed, the SMTP connection could be outright rejected. The assumption here was that spam comes from insecured or outright evil hosts, and that blocking them outright is better for everyone.

This was a true democratic solution in the spirit of free software: Anyone could run one.

That same sprit means that each RBL had a different criteria for listing. Some were zero tolerance, and even one Unsolicited Commercial Email was enough to get listed. Others, simply listed whole netblocks, so you could block all Cable ISPs, or entire countries.

Unlike killfiles, RBLs were designed to be a distributed system from the outset.

Like killfiles, RBLs are in effect a Book of Grudges. Subscribing to one, means subscribing to someone else's grudges. If you shared grudge-worthy viewpoints, this was incredibly labor saving. If you didn't, sometimes things got blocked that shouldn't have.

As a solution to the problem of spam, RBLs were not the silver bullet. That came with the advent of commercial providers deploying surveillance networks and offering IP reputation services as part of their paid service. The commercial providers were typically able to deploy far wider surveillance than the all-volunteer RBLs did, and as such saw a wider sample of overall email traffic. A wider sample means that they were less likely to ban a legitimate site for a single offense.

This is still the case today, though email-as-a-service providers like Google and Microsoft are now hosting the entire email stack themselves. Since Google handles a majority of all email on the planet, their surveillance is pretty good.

Compounding the problem for the volunteer-lead RBL efforts is IPv6. IPv4 was small enough you can legitimately tag the entire internet with spam/not-spam without undue resources. IPv6 is vastly larger and very hard to do comprehensively without resorting to netblock blocking. And even then, there are enough possible netblocks that scale is a real issue.

Twitter Blocklists

Which brings us to today, and twitter. Shared blocklists are in this tradition of killfiles and RBLs. However, there are a few structural barriers to this being the solution it was with Usenet:

  • No Netblocks. Which means each user has to be blocked individually, you can't block all of a network, or a country-of-origin
  • The number of accounts. Active-users is not the same as total-users. In 2013, the estimated registered-account number was around 810 million. Four years later, this is likely over a billion. It's rapidly approaching the size of the IPv4 address space.
  • Ease of setting up a new account. Changing your IP address Changing your username is very, very easy.

The lack of a summarization technique, the size of the problem space, and the ease of bypassing a block by changing your identifier mean that a shared-blocklist is a poor tool to fight off a determined hatemob. It's still good for casual griefing, where the parties aren't invested enough to break a blocklist.

The idea of the commercialized RBL, where a large company sells account-reputation services, is less possible. First of all, such an offering would likely be against the Twitter terms of service. Second, the target market is not mail-server-admins, but individual account-holders. A far harder market to monitize.

The true solution will have to come from Twitter itself. Either by liberalizing their ToS to allow those commercial services to develop, or developing their own reputation markets and content tools.

by SysAdmin1138 at August 11, 2017 02:38 PM

August 10, 2017

Everything Sysadmin

Evidence that Go makes it easy to get up to speed

Some recent PRs to the DNSControl Project casually mentioned that this was their first time writing Go code. That's amazing!

When was the last time you saw someone say, "here's a major contribution to your open source project... oh and I just learned this language." (and the PR was really darn good!) I think it is pretty rare and one of the special things about Go.

Part of Go's original vision was to make it easy for new people to join a project and make contributions. This was important internally at Google, since engineers hop projects frequently. This also benefits open source projects by making it easy to dive in and participate.

Here are the three PRs:

  • Add Digitalocean provider #171. DNSControl has a plug-in architecture to support new DNS Service Providers. This Go first-timer wrote an entire plugin to support Digital Ocean. "I haven't used Go before, but the diff looks sane so hopefully I managed to handle the dependencies correctly."
  • Implement SRV support for CloudFlare provider #174. Plug-ins can indicate whether or not they support new fangled DNS records like SRV. This PR extends the CloudFlare provider to add support for the CAA record. "This is my first time writing anything in Go".
  • CAA support #132. CAA is a new DNS record type. This PR changed DNSControl to support this new record, and implements it for a few of the providers (plug-ins). "I almost never wrote Go before (and this is my first Go PR)".

One of the joys maintaining an open source project is watching contributors build new skills. Github.com's PR system makes it a joy to give constructive criticism and help people iterate on the code until it meets our high standards. (ProTip: Criticize the code, not the person. i.e. write "this should" instead of "you should")

Go isn't 100 percent of why it is easy to contribute to DNSControl. We've made it easy to contribute other ways too:

  • Extensive unit tests and integration tests. Your first contribution can be scary. If your new provider passes all the integration tests, you can be confident that it is time to make the PR. This reduces fear of embarassment.
  • Detailed documentation on the most common and difficult tasks such as how to write a new provider and add a new DNS record type. People are encouraged to add new tests of their own (TDD-style). We also encourage people to update the "how to" docs as they use them, to make the process easier for the next person.
  • Extra-friendly code reviews. A special shout-out to Craig who is super friendly and helpful. He's happy to coach people whether they're learning DNSControl itself or Go.

If you would like to learn more about DNSControl, and why "DNS as Code" is a better way to maintain your DNS zones, then watch our video from Usenix SRECon or check out our Github page.

Thanks to our contributors, and to my employer StackOverflow.com, for supporting this project. And, of course, thanks to the Go community for making such an awesome language!

by Tom Limoncelli at August 10, 2017 10:00 PM

The Lone Sysadmin

Fix WinRM Client Issues

My team manages a lot of Dell hardware. Over the years we’ve run into situations where we have to replace the system board on a host. The system board’s management interface, iDRAC, has a license key on it, and when you replace the system board it’s helpful if you can export the license key ahead […]

The post Fix WinRM Client Issues appeared first on The Lone Sysadmin. Head over to the source to read the full post!

by Bob Plankers at August 10, 2017 07:58 PM

August 07, 2017

LZone - Sysadmin

Automatically Download Oracle JDK

When downloading Oracle JDK via scripts you might run into the login page.

While in the past it was sufficient to do something like
wget --no-cookies --header "Cookie: oraclelicense=accept-securebackup-cookie" http://download.oracle.com/otn-pub/java/jdk/8u144-b01/090f390dda5b47b9b721c7dfaa008135/jdk-8u144-linux-x64.tar.gz
you now also need to ignore https:// as you will get redirected. So when using wget you might want to run
wget -c --no-cookies --no-check-certificate --header "Cookie: oraclelicense=accept-securebackup-cookie" http://download.oracle.com/otn-pub/java/jdk/8u144-b01/090f390dda5b47b9b721c7dfaa008135/jdk-8u144-linux-x64.tar.gz
Note the added "-c" and "--no-check-certificate" options.

August 07, 2017 02:26 PM

Sarah Allen

impact matters more than intent

Disclaimer: I work at Google. This article is based on 18 years of observing the company as an outsider, along with over a year of experience as a senior technical leader at the company.

An internal google document was published this weekend, where an individual articulated poorly reasoned arguments that demonstrated conscious bias, both about gender and about the skills necessary for software engineering. Demeaning generalizations and stereotypes were presented as unbiased fact. The author may not have intended to be disrespectful, yet caused great harm. Many women have spoken publicly about their frustration in reading this harmful rhetoric in their workplace.

For many years before I joined Google, I heard stories of sexism and racism that happened inside the company. Frankly, I know of no company which is immune to this. Companies believe they have well-established social norms around respectful behavior, good management training, effective escalation paths, etc., yet these aren’t evenly distributed. In 2006, I declared that I would not work at Google because of their hiring practices. In 2016, I decided that both Google and I had changed a bit since then. I interviewed Google even more than they interviewed me. Not including the actual day of interviews, I had 21 conversations with 17 different people before deciding to work here.

Unexpectedly, Google was the first tech company I have worked for where I didn’t routinely meet people who expressed surprise at my technical ability. There seems to be a positive bias that if you work for Google you must be technically awesome. I can’t tell whether this is universal. (I assume it isn’t. Google is a big place.) However, as evidenced by this latest rant, Google has a long way to go in creating effective social norms around discussing diversity and the efforts to make hiring and promotion fair.

We need to be careful as we address inequities. As a woman who attended private high school and studied Computer Science at a prestigious university, I have an advantage in getting a job at any tech company over a white man who joined the military to pay for college and taught himself to code. Google could, if it wanted to, hire the very best women and people of color such that it statistically matched the demographics of the United States, and not “lower the bar” yet still be homogeneous, yielding limited benefit from its diversity efforts.

Diversity and inclusion is not just about demographics. The lack of minorities and women at Google and most other tech companies is a sign that things aren’t quite right. We should look at diversity numbers as a signal, and then seek to address underlying problems around inclusion and respect. We need to include effective communication skills as part of selection criteria for new engineers and for promotion.

Ex-Google engineer Yonatan Zunger wrote a thoughtful response about how communication and collaboration are critical to the work we do. I have also written more generally about how communication is a technical skill: “We usually think of software as being made of code, but it is really made of people. We imagine what the software should do, and through our code, we turn ideas into reality…. I find it confusing that anyone could even suggest that communication is a different skill that is optional for programmers. This is what we do. We write our code in programming languages. We define things and then they exist. We make real things using language.”

I’ve worked with amazing engineers who have little else in common. The best engineers I’ve worked with solve problems differently from each other — creativity and insight combined with technical skill are the mark of a great engineer.

The Google hiring process tests for the candidate’s ability to write code and solve problems (and to some degree talk about code). This is not enough. Google and the rest of the industry needs to set a higher bar for hiring talent. It is much harder to collaborate effectively and communicate your ideas than it is to write a basic algorithm on the whiteboard.

In my experience, the ability to write great software is not tied to any outward trait, and discussion of biological or societal differences is a distraction from the core issue. We need people with a diversity of experience and perspectives to bring creative solutions to their work, and we need engineers who can work with people who are different from them with respect and enthusiasm.

I know there are good people working inside of Google to make change. I applaud the publication of research on effective teamwork. This is not enough. This work of creating change for humans is much harder than the work of writing software. Smaller companies have a greater opportunity to make change faster. We each need to step up and take a leadership role at every level of our organizations.

Impact matters more than intent.

by sarah at August 07, 2017 05:42 AM

Errata Security

Query name minimization

One new thing you need to add your DNS security policies is "query name minimizations" (RFC 7816). I thought I'd mention it since many haven't heard about it.

Right now, when DNS resolvers lookup a name like "www.example.com.", they send the entire name to the root server (like a.root-servers.net.). When it gets back the answer to the .com DNS server a.gtld-servers.net), it then resends the full "www.example.com" query to that server.

This is obviously unnecessary. The first query should be just .com. to the root server, then example.com. to the next server -- the minimal amount needed for each query, not the full query.

The reason this is important is that everyone is listening in on root name server queries. Universities and independent researchers do this to maintain the DNS system, and to track malware. Security companies do this also to track malware, bots, command-and-control channels, and so forth. The world's biggest spy agencies do this in order just to spy on people. Minimizing your queries prevents them from spying on you.

An example where this is important is that story of lookups from AlfaBank in Russia for "mail1.trump-emails.com". Whatever you think of Trump, this was an improper invasion of privacy, where DNS researchers misused their privileged access in order to pursue their anti-Trump political agenda. If AlfaBank had used query name minimization, none of this would have happened.

It's also critical for not exposing internal resources. Even when you do "split DNS", when the .com record expires, you resolver will still forward the internal DNS record to the outside world. All those Russian hackers can map out the internal names of your network simply by eavesdropping on root server queries.

Servers that support this are Knot resolver and Unbound 1.5.7+ and possibly others. It's a relatively new standard, so it make take a while for other DNS servers to support this.





by Robert Graham (noreply@blogger.com) at August 07, 2017 01:31 AM

August 06, 2017

Electricmonk.nl

Understanding Python's logging module

I'm slightly embarrassed to say that after almost two decades of programming Python, I still didn't understand its logging module. Sure, I could get it to work, and reasonably well, but I'd often end up with unexplained situations such as double log lines or logging that I didn't want.

>>> requests.get('https://www.electricmonk.nl')
DEBUG:requests.packages.urllib3.connectionpool:Starting new HTTPS connection (1): www.electricmonk.nl
DEBUG:requests.packages.urllib3.connectionpool:https://www.electricmonk.nl:443 "GET / HTTP/1.1" 301 178

What? I never asked for that debugging information?

So I decided to finally, you know, read the documentation and experiment a bit to figure out how the logging module really works. In this article I want to bring attention to some of the misconceptions I had about the logging module. I'm going to assume you have a basic understanding of how it works and know about loggers, log levels and handlers.

Logger hierarchy

Loggers have a hierarchy. That is, you can create individual loggers and each logger has a parent. At the top of the hierarchy is the root logger. For instance, we could have the following loggers:

myapp
myapp.ui
myapp.ui.edit

These can be created by asking a parent logger for a new child logger:

>>> log_myapp = logging.getLogger("myapp")
>>> log_myapp_ui = log_myapp.getChild("ui")
>>> log_myapp_ui.name
'myapp.ui'
>>> log_myapp_ui.parent.name
'myapp'

Or you can use dot notation:

>>> log_myapp_ui = logging.getLogger("myapp.ui")
>>> log_myapp_ui.parent.name
'myapp'

You should use the dot notation generally.

One thing that's not immediately clear is that the logger names don't include the root logger. In actuality, the logger hierarchy looks like this:

root.myapp
root.myapp.ui
root.myapp.ui.edit

More on the root logger in a bit.

Log levels and message propagation

Each logger can have a log level. When you send a message to a logger, you specify the log level of the message. If the level matches, the message is then propagated up the hierarchy of loggers. One of the biggest misconceptions I had was that I thought each logger checked the level of the message and if it the level of the message is lower or equal, the logger's handler would be invoked. This is not true!

What happens instead is that the level of the message is only checked by the logger you give the message to. If the message's level is lower or equal to the logger's, the message is propagated up the hierarchy, but none of the other loggers will check the level! They'll simply invoke their handlers.

>>> log_myapp.setLevel(logging.ERROR)
>>> log_myapp_ui.setLevel(logging.DEBUG)
>>> log_myapp_ui.debug('test')
DEBUG:myapp.ui:test

In the example above, the root logger has a handler that prints the message. Even though the "log_myapp" handler has a level of ERROR, the DEBUG message is still propagated to to the root logger. This image (found on this page) shows why:

As you can see, when giving a message to a logger, the logger checks the level. After that, the level on the loggers is no longer checked and all handlers in the entire chain are invoked, regardless of level. Note that you can set levels on handlers as well. This is useful if you want to, for example, create a debugging log file but only show warnings and errors on the console.

It's also worth noting that by default, loggers have a level of 0. This means they use the log level of the first parent logger that has an actual level set. This is determined at message-time, not when the logger is created.

The root logger

The logging tutorial for Python explains that to configure logging, you can use basicConfig():

logging.basicConfig(filename='example.log',level=logging.DEBUG)

It's not immediately obvious, but what this does is configure the root logger. This is not something you want to do in practice, because it causes debugging output for all loggers in your program, including every library that uses logging. This is why the requests module suddenly starts outputting debug information when you configure the root logger.

In general, you'll want ot leave the root logger alone and configure a specific "main" logger for your program and put all the other loggers under that logger. This way, you can toggle logging for your specific program on and off by setting the level of the main logger. There is no convenient method such as basicConfig() to create a main logger, so you'll have to do it manually:

ch = logging.StreamHandler()
formatter = logging.Formatter('%(asctime)s %(levelname)8s %(name)s | %(message)s')
ch.setFormatter(formatter)

logger = logging.getLogger('myapp')
logger.addHandler(ch)
logger.setLevel(logging.WARNING)  # This toggles all the logging in your app

There are more pitfalls when it comes to the root logger. If you call any of the module-level logging methods, the root logger is automatically configured in the background for you. This goes completely against Python's "explicit is better than implicit" rule:

#!/usr/bin/env python
import logging
logging.warn("uhoh")
# Output: WARNING:root:uhoh

In the example above, I never configured a handler. It was done automatically. And on the root handler no less. This will cause all kinds of logging output from libraries you might not want. So don't use the logging.warn(), logging.error() and other module-level methods. Always log against a specific logger instance you got with logging.getLogger().

This has tripped me up many times, because I'll often do some simple logging in the main part of my program with these. It becomes especially confusing when you do something like this:

#!/usr/bin/python
import logging
for x in foo:
    try:
        something()
    except ValueError as err:
        logging.exception(err)
        pass  # we don't care

Now there will be no logging until an error occurs, and then suddenly the root logger is configured and subsequent iterations of the loop may start logging messages.

The Python documentation also mentions the following caveat about using module-level loggers:

The above module-level convenience functions, which delegate to the root logger, call basicConfig() to ensure that at least one handler is available. Because of this, they shouldnot be used in threads, in versions of Python earlier than 2.7.1 and 3.2, unless at least one handler has been added to the root logger before the threads are started. In earlier versions of Python, due to a thread safety shortcoming in basicConfig(), this can (under rare circumstances) lead to handlers being added multiple times to the root logger, which can in turn lead to multiple messages for the same event.

Debugging logging problems

When I run into weird logging problems such as no output, or double lines, I generally put the following debugging code at the point where I'm logging the message.

log_to_debug = logging.getLogger("myapp.ui.edit")
while log_to_debug is not None:
    print "level: %s, name: %s, handlers: %s" % (log_to_debug.level,
                                                 log_to_debug.name,
                                                 log_to_debug.handlers)
    log_to_debug = log_to_debug.parent

which outputs:

level: 0, name: myapp.ui.edit, handlers: []
level: 0, name: myapp.ui, handlers: []
level: 0, name: myapp, handlers: []
level: 30, name: root, handlers: []

From this output it becomes obvious that all loggers use a level of 30, since their log levels are 0, which means the look up the hierarchy for the first logger with a non-zero level. I've also not configured any handlers. If I was seeing double output, it's probably because there is more than one handler configured.

Summary

  • When you log a message, the level is only checked at the logger you logged the message against. If it passes, every handler on every logger up the hierarchy is called, regardless of that logger's level.
  • By default, loggers have a level of 0. This means they use the log level of the first parent logger that has an actual level set. This is determined at message-time, not when the logger is created.
  • Stay away from the root logger. That means: no logging.basicConfig() and no usage of module-level loggers such as logging.warning(), as they have unintended side-effects.
  • Create a uniquely named top-level logger for your application and put all child loggers under that logger. Configure a handler for output on the top-level logger for your application. Don't configure a level on your loggers, so that you can set a level at any point in the hierarchy and get logging output at that level for all underlying loggers. Note that this is an appropriate strategy for how I usually structure my programs. It might not be for you.
  • The easiest way that's usually correct is to use __name__ as the logger name: log = logging.getLogger(__name__). This uses the module hierarchy as the name, which is generally what you want. 
  • Read the entire logging HOWTO and specifically the Advanced Logging Tutorial, because it really should be called "logging basics".

by admin at August 06, 2017 08:40 AM

August 04, 2017

Electricmonk.nl

Criticize grsecurity? Get sued.

Thinking about using the grsecurity linux kernel hardening patches? Better check with your legal team. Not only are they likely violating the GPLv2 with their patch-set, but if you point out that, in your opinion, they are violating the GPLv2, they'll sue you. And not only you, but anybody that is even remotely involved in anything you've ever done:

Grsecurity sued [Bruce Perens], his web host, and as-yet-unidentified defendants who may have helped him draft that post, for defamation and business interference.

Meanwhile, Rohit Chhabra, Grsecurity's attorney, said in an email to The Register:

Mr Perens has made false statements, claiming them to be facts, 

Naturally their case is completely ludicrous since software licenses (and basically every law in existence) has always been open to interpretation. Which is why we have courts and judges and such. I fail to see how Perens' interpreting the GPLv2 and concluding that grsecurity is in violation of it constitutes as fact. It is almost by definition an opinion.

So you may want to avoid using grsecurity if Perens thinks it violates the GPLv2, since Perens is a bit of an extreme expert at the GPL. Further more, you probably want to avoid it since it's likely Grsecurity (the company) will sue you if you sneeze too loudly during one of their presentations or something. And finally, you probably want to avoid it since the arbiter of All Things Linux, Linus Torvalds, says it's garbage:

Don't bother with grsecurity.

Their approach has always been "we don't care if we break anything, we'll just claim it's because we're extra secure".

The thing is a joke, and they are clowns. When they started talking about people taking advantage of them, I stopped trying to be polite about their bullshit.

Their patches are pure garbage.

Linus

Since they haven't sued Torvarlds yet, I'm guessing that "Their patches are pure garbage" is based on actual facts, if we are to believe Rohit Chhabra.

Update: Chhabra's law firm looks like nothing more than yet another firm for patent trolls. To quote their website: "Chhabra®:Attorneys |Patent| Trademark| Copyright| Litigation"

 

by admin at August 04, 2017 10:49 AM

August 02, 2017

Steve Kemp's Blog

So I did a thing, then another thing.

So I did start a project, to write a puppet-dashboard, it is functionally complete, but the next step is to allow me to raise alerts based on failing runs of puppet - in real-time.

(i.e. Now that I have a dashboard I wish to not use it. I want to be alerted to failures, without having to remember to go look for them. Something puppet-dashboard can't do ..)

In other news a while back I slipped in a casual note about having a brain scan done, here in sunny Helsinki.

One of the cool things about that experience, in addition to being told I wasn't going to drop dead that particular day, was that the radiologist told me that I could pay €25 to get a copy of my brain data in DICOM format.

I've not yet played with this very much, but I couldn't resist a brief animation:

  • See my brain.
    • Not the best quality, or the best detail, but damn. It is my brain.
    • I shall do better with more experimentation I think.
    • After I posted it my wife, a doctor, corrected me: That wasn't a gif of my brain, instead it was a gif of my skull. D'oh!

August 02, 2017 09:00 PM

August 01, 2017

Anton Chuvakin - Security Warrior

Monthly Blog Round-Up – July 2017

Here is my next monthly "Security Warrior" blog round-up of top 5 popular posts/topics this
month:
  1. Why No Open Source SIEM, EVER?” contains some of my SIEM thinking from 2009. Is it relevant now? You be the judge.  Succeeding with SIEM requires a lot of work, whether you paid for the software, or not. BTW, this post has an amazing “staying power” that is hard to explain – I suspect it has to do with people wanting “free stuff” and googling for “open source SIEM” … 
  2. “New SIEM Whitepaper on Use Cases In-Depth OUT!” (dated 2010) presents a whitepaper on select SIEM use cases described in depth with rules and reports [using now-defunct SIEM product]; also see this SIEM use case in depth and this for a more current list of popular SIEM use cases. Finally, see our 2016 research on developing security monitoring use cases here!
  3. Again, my classic PCI DSS Log Review series is extra popular! The series of 18 posts cover a comprehensive log review approach (OK for PCI DSS 3+ even though it predates it), useful for building log review processes and procedures, whether regulatory or not. It is also described in more detail in our Log Management book and mentioned in our PCI book (now in its 4th edition!) – note that this series is mentioned in some PCI Council materials. 
  4. Simple Log Review Checklist Released!” is often at the top of this list – this aging checklist is still a very useful tool for many people. “On Free Log Management Tools” (also aged a bit by now) is a companion to the checklist (updated version)
  5. “SIEM Resourcing or How Much the Friggin’ Thing Would REALLY Cost Me?” is a quick framework for assessing the SIEM project (well, a program, really) costs at an organization (a lot more details on this here in this paper).
In addition, I’d like to draw your attention to a few recent posts from my Gartner blog [which, BTW, now has more than 5X of the traffic of this blog]: 

Current research on SIEM:
Recent research on vulnerability management:
Recent research on cloud security monitoring:
Miscellaneous fun posts:

(see all my published Gartner research here)
Also see my past monthly and annual “Top Popular Blog Posts” – 2007, 2008, 2009, 2010, 2011, 2012, 2013, 2014, 2015, 2016.

Disclaimer: most content at SecurityWarrior blog was written before I joined Gartner on August 1, 2011 and is solely my personal view at the time of writing. For my current security blogging, go here.

Previous post in this endless series:

by Anton Chuvakin (anton@chuvakin.org) at August 01, 2017 07:41 PM

ma.ttias.be

RHEL & CentOS 7.4 restores HTTP/2 functionality on Nginx

The post RHEL & CentOS 7.4 restores HTTP/2 functionality on Nginx appeared first on ma.ttias.be.

Red Hat Enterprise Linux 7.4 has just been released, and with it, a much-awaited (at least by me) update to OpenSSL, bringing it to version 1.0.2k.

openssl rebased to version 1.0.2k

The openssl package has been updated to upstream version 1.0.2k, which provides a number of enhancements, new features, and bug fixes, including:

-- Added support for the Datagram Transport Layer Security TLS (DTLS) protocol version 1.2.
-- Added support for the automatic elliptic curve selection for the ECDHE key exchange in TLS.
-- Added support for the Application-Layer Protocol Negotiation (ALPN).
-- Added Cryptographic Message Syntax (CMS) support for the following schemes: RSA-PSS, RSA-OAEP, ECDH, and X9.42 DH.

Note that this version is compatible with the API and ABI in the OpenSSL library version in previous releases of Red Hat Enterprise Linux 7.
Security updates in RHEL 7.4

That bold text? That's the good part: added support for the Application-Layer Protocol Negotiation (ALPN).

You might remember that more than a year ago, Chrome disabled support for NPN and only allowed ALPN. As a direct result, HTTP/2 became impossible on RHEL or CentOS systems, because it didn't support ALPN. You could run Nginx in a Docker container to get a more up-to-date OpenSSL version, but for many, that's too hard or too much effort.

Now with this RHEL 7.4 update -- and the CentOS 7.4 release that'll be out in a few days -- you no longer need Docker and you can enable HTTP/2 in Nginx again for both Firefox & Chrome users.

server {
  listen        443 ssl http2;
  server_name   your.fqdn.tld;
  ...

I'm very happy with this update as it -- finally -- brings a more recent OpenSSL version to RHEL & CentOS, removing the need for hacks/workarounds just for the sake of HTTP/2.

If you're a CentOS user, wait a couple of days for the packages to be built and update to CentOS 7.4. If your a RHEL user: go forth and upgrade to 7.4!

The post RHEL & CentOS 7.4 restores HTTP/2 functionality on Nginx appeared first on ma.ttias.be.

by Mattias Geniar at August 01, 2017 06:01 PM

Errata Security

Top 10 Most Obvious Hacks of All Time (v0.9)

For teaching hacking/cybersecurity, I thought I'd create of the most obvious hacks of all time. Not the best hacks, the most sophisticated hacks, or the hacks with the biggest impact, but the most obvious hacks -- ones that even the least knowledgeable among us should be able to understand. Below I propose some hacks that fit this bill, though in no particular order.

The reason I'm writing this is that my niece wants me to teach her some hacking. I thought I'd start with the obvious stuff first.

Shared Passwords

If you use the same password for every website, and one of those websites gets hacked, then the hacker has your password for all your websites. The reason your Facebook account got hacked wasn't because of anything Facebook did, but because you used the same email-address and password when creating an account on "beagleforums.com", which got hacked last year.

I've heard people say "I'm sure, because I choose a complex password and use it everywhere". No, this is the very worst thing you can do. Sure, you can the use the same password on all sites you don't care much about, but for Facebook, your email account, and your bank, you should have a unique password, so that when other sites get hacked, your important sites are secure.

And yes, it's okay to write down your passwords on paper.

Tools: HaveIBeenPwned.com

PIN encrypted PDFs

My accountant emails PDF statements encrypted with the last 4 digits of my Social Security Number. This is not encryption -- a 4 digit number has only 10,000 combinations, and a hacker can guess all of them in seconds.

PIN numbers for ATM cards work because ATM machines are online, and the machine can reject your card after four guesses. PIN numbers don't work for documents, because they are offline -- the hacker has a copy of the document on their own machine, disconnected from the Internet, and can continue making bad guesses with no restrictions.

Passwords protecting documents must be long enough that even trillion upon trillion guesses are insufficient to guess.

Tools: Hashcat, John the Ripper

SQL and other injection

The lazy way of combining websites with databases is to combine user input with an SQL statement. This combines code with data, so the obvious consequence is that hackers can craft data to mess with the code.

No, this isn't obvious to the general public, but it should be obvious to programmers. The moment you write code that adds unfiltered user-input to an SQL statement, the consequence should be obvious. Yet, "SQL injection" has remained one of the most effective hacks for the last 15 years because somehow programmers don't understand the consequence.

CGI shell injection is a similar issue. Back in early days, when "CGI scripts" were a thing, it was really important, but these days, not so much, so I just included it with SQL. The consequence of executing shell code should've been obvious, but weirdly, it wasn't. The IT guy at the company I worked for back in the late 1990s came to me and asked "this guy says we have a vulnerability, is he full of shit?", and I had to answer "no, he's right -- obviously so".

XSS ("Cross Site Scripting") [*] is another injection issue, but this time at somebody's web browser rather than a server. It works because websites will echo back what is sent to them. For example, if you search for Cross Site Scripting with the URL https://www.google.com/search?q=cross+site+scripting, then you'll get a page back from the server that contains that string. If the string is JavaScript code rather than text, then some servers (thought not Google) send back the code in the page in a way that it'll be executed. This is most often used to hack somebody's account: you send them an email or tweet a link, and when they click on it, the JavaScript gives control of the account to the hacker.

Cross site injection issues like this should probably be their own category, but I'm including it here for now.

More: Wikipedia on SQL injection, Wikipedia on cross site scripting.
Tools: Burpsuite, SQLmap

Buffer overflows

In the C programming language, programmers first create a buffer, then read input into it. If input is long than the buffer, then it overflows. The extra bytes overwrite other parts of the program, letting the hacker run code.

Again, it's not a thing the general public is expected to know about, but is instead something C programmers should be expected to understand. They should know that it's up to them to check the length and stop reading input before it overflows the buffer, that there's no language feature that takes care of this for them.

We are three decades after the first major buffer overflow exploits, so there is no excuse for C programmers not to understand this issue.

What makes particular obvious is the way they are wrapped in exploits, like in Metasploit. While the bug itself is obvious that it's a bug, actually exploiting it can take some very non-obvious skill. However, once that exploit is written, any trained monkey can press a button and run the exploit. That's where we get the insult "script kiddie" from -- referring to wannabe-hackers who never learn enough to write their own exploits, but who spend a lot of time running the exploit scripts written by better hackers than they.

More: Wikipedia on buffer overflow, Wikipedia on script kiddie,  "Smashing The Stack For Fun And Profit" -- Phrack (1996)
Tools: bash, Metasploit

SendMail DEBUG command (historical)

The first popular email server in the 1980s was called "SendMail". It had a feature whereby if you send a "DEBUG" command to it, it would execute any code following the command. The consequence of this was obvious -- hackers could (and did) upload code to take control of the server. This was used in the Morris Worm of 1988. Most Internet machines of the day ran SendMail, so the worm spread fast infecting most machines.

This bug was mostly ignored at the time. It was thought of as a theoretical problem, that might only rarely be used to hack a system. Part of the motivation of the Morris Worm was to demonstrate that such problems was to demonstrate the consequences -- consequences that should've been obvious but somehow were rejected by everyone.

More: Wikipedia on Morris Worm

Email Attachments/Links

I'm conflicted whether I should add this or not, because here's the deal: you are supposed to click on attachments and links within emails. That's what they are there for. The difference between good and bad attachments/links is not obvious. Indeed, easy-to-use email systems makes detecting the difference harder.

On the other hand, the consequences of bad attachments/links is obvious. That worms like ILOVEYOU spread so easily is because people trusted attachments coming from their friends, and ran them.

We have no solution to the problem of bad email attachments and links. Viruses and phishing are pervasive problems. Yet, we know why they exist.

Default and backdoor passwords

The Mirai botnet was caused by surveillance-cameras having default and backdoor passwords, and being exposed to the Internet without a firewall. The consequence should be obvious: people will discover the passwords and use them to take control of the bots.

Surveillance-cameras have the problem that they are usually exposed to the public, and can't be reached without a ladder -- often a really tall ladder. Therefore, you don't want a button consumers can press to reset to factory defaults. You want a remote way to reset them. Therefore, they put backdoor passwords to do the reset. Such passwords are easy for hackers to reverse-engineer, and hence, take control of millions of cameras across the Internet.

The same reasoning applies to "default" passwords. Many users will not change the defaults, leaving a ton of devices hackers can hack.

Masscan and background radiation of the Internet

I've written a tool that can easily scan the entire Internet in a short period of time. It surprises people that this possible, but it obvious from the numbers. Internet addresses are only 32-bits long, or roughly 4 billion combinations. A fast Internet link can easily handle 1 million packets-per-second, so the entire Internet can be scanned in 4000 seconds, little more than an hour. It's basic math.

Because it's so easy, many people do it. If you monitor your Internet link, you'll see a steady trickle of packets coming in from all over the Internet, especially Russia and China, from hackers scanning the Internet for things they can hack.

People's reaction to this scanning is weirdly emotional, taking is personally, such as:
  1. Why are they hacking me? What did I do to them?
  2. Great! They are hacking me! That must mean I'm important!
  3. Grrr! How dare they?! How can I hack them back for some retribution!?
I find this odd, because obviously such scanning isn't personal, the hackers have no idea who you are.

Tools: masscan, firewalls

Packet-sniffing, sidejacking

If you connect to the Starbucks WiFi, a hacker nearby can easily eavesdrop on your network traffic, because it's not encrypted. Windows even warns you about this, in case you weren't sure.

At DefCon, they have a "Wall of Sheep", where they show passwords from people who logged onto stuff using the insecure "DefCon-Open" network. Calling them "sheep" for not grasping this basic fact that unencrypted traffic is unencrypted.

To be fair, it's actually non-obvious to many people. Even if the WiFi itself is not encrypted, SSL traffic is. They expect their services to be encrypted, without them having to worry about it. And in fact, most are, especially Google, Facebook, Twitter, Apple, and other major services that won't allow you to log in anymore without encryption.

But many services (especially old ones) may not be encrypted. Unless users check and verify them carefully, they'll happily expose passwords.

What's interesting about this was 10 years ago, when most services which only used SSL to encrypt the passwords, but then used unencrypted connections after that, using "cookies". This allowed the cookies to be sniffed and stolen, allowing other people to share the login session. I used this on stage at BlackHat to connect to somebody's GMail session. Google, and other major websites, fixed this soon after. But it should never have been a problem -- because the sidejacking of cookies should have been obvious.

Tools: Wireshark, dsniff

Stuxnet LNK vulnerability

Again, this issue isn't obvious to the public, but it should've been obvious to anybody who knew how Windows works.

When Windows loads a .dll, it first calls the function DllMain(). A Windows link file (.lnk) can load icons/graphics from the resources in a .dll file. It does this by loading the .dll file, thus calling DllMain. Thus, a hacker could put on a USB drive a .lnk file pointing to a .dll file, and thus, cause arbitrary code execution as soon as a user inserted a drive.

I say this is obvious because I did this, created .lnks that pointed to .dlls, but without hostile DllMain code. The consequence should've been obvious to me, but I totally missed the connection. We all missed the connection, for decades.

Social Engineering and Tech Support [* * *]

After posting this, many people have pointed out "social engineering", especially of "tech support". This probably should be up near #1 in terms of obviousness.

The classic example of social engineering is when you call tech support and tell them you've lost your password, and they reset it for you with minimum of questions proving who you are. For example, you set the volume on your computer really loud and play the sound of a crying baby in the background and appear to be a bit frazzled and incoherent, which explains why you aren't answering the questions they are asking. They, understanding your predicament as a new parent, will go the extra mile in helping you, resetting "your" password.

One of the interesting consequences is how it affects domain names (DNS). It's quite easy in many cases to call up the registrar and convince them to transfer a domain name. This has been used in lots of hacks. It's really hard to defend against. If a registrar charges only $9/year for a domain name, then it really can't afford to provide very good tech support -- or very secure tech support -- to prevent this sort of hack.

Social engineering is such a huge problem, and obvious problem, that it's outside the scope of this document. Just google it to find example after example.

A related issue that perhaps deserves it's own section is OSINT [*], or "open-source intelligence", where you gather public information about a target. For example, on the day the bank manager is out on vacation (which you got from their Facebook post) you show up and claim to be a bank auditor, and are shown into their office where you grab their backup tapes. (We've actually done this).

More: Wikipedia on Social Engineering, Wikipedia on OSINT, "How I Won the Defcon Social Engineering CTF" -- blogpost (2011), "Questioning 42: Where's the Engineering in Social Engineering of Namespace Compromises" -- BSidesLV talk (2016)

Blue-boxes (historical) [*]

Telephones historically used what we call "in-band signaling". That's why when you dial on an old phone, it makes sounds -- those sounds are sent no differently than the way your voice is sent. Thus, it was possible to make tone generators to do things other than simply dial calls. Early hackers (in the 1970s) would make tone-generators called "blue-boxes" and "black-boxes" to make free long distance calls, for example.

These days, "signaling" and "voice" are digitized, then sent as separate channels or "bands". This is call "out-of-band signaling". You can't trick the phone system by generating tones. When your iPhone makes sounds when you dial, it's entirely for you benefit and has nothing to do with how it signals the cell tower to make a call.

Early hackers, like the founders of Apple, are famous for having started their careers making such "boxes" for tricking the phone system. The problem was obvious back in the day, which is why as the phone system moves from analog to digital, the problem was fixed.

More: Wikipedia on blue box, Wikipedia article on Steve Wozniak.

Thumb drives in parking lots [*]

A simple trick is to put a virus on a USB flash drive, and drop it in a parking lot. Somebody is bound to notice it, stick it in their computer, and open the file.

This can be extended with tricks. For example, you can put a file labeled "third-quarter-salaries.xlsx" on the drive that required macros to be run in order to open. It's irresistible to other employees who want to know what their peers are being paid, so they'll bypass any warning prompts in order to see the data.

Another example is to go online and get custom USB sticks made printed with the logo of the target company, making them seem more trustworthy.

We also did a trick of taking an Adobe Flash game "Punch the Monkey" and replaced the monkey with a logo of a competitor of our target. They now only played the game (infecting themselves with our virus), but gave to others inside the company to play, infecting others, including the CEO.

Thumb drives like this have been used in many incidents, such as Russians hacking military headquarters in Afghanistan. It's really hard to defend against.

More: "Computer Virus Hits U.S. Military Base in Afghanistan" -- USNews (2008), "The Return of the Worm That Ate The Pentagon" -- Wired (2011), DoD Bans Flash Drives -- Stripes (2008)

Googling [*]

Search engines like Google will index your website -- your entire website. Frequently companies put things on their website without much protection because they are nearly impossible for users to find. But Google finds them, then indexes them, causing them to pop up with innocent searches.

There are books written on "Google hacking" explaining what search terms to look for, like "not for public release", in order to find such documents.

More: Wikipedia entry on Google Hacking, "Google Hacking" book.

URL editing [*]

At the top of every browser is what's called the "URL". You can change it. Thus, if you see a URL that looks like this:
http://www.example.com/documents?id=138493
Then you can edit it to see the next document on the server:
http://www.example.com/documents?id=138494
The owner of the website may think they are secure, because nothing points to this document, so the Google search won't find it. But that doesn't stop a user from manually editing the URL.

An example of this is a big Fortune 500 company that posts the quarterly results to the website an hour before the official announcement. Simply editing the URL from previous financial announcements allows hackers to find the document, then buy/sell the stock as appropriate in order to make a lot of money.

Another example is the classic case of Andrew "Weev" Auernheimer who did this trick in order to download the account email addresses of early owners of the iPad, including movie stars and members of the Obama administration. It's an interesting legal case because on one hand, techies consider this so obvious as to not be "hacking". On the other hand, non-techies, especially judges and prosecutors, believe this to be obviously "hacking".

DDoS, spoofing, and amplification [*]

For decades now, online gamers have figured out an easy way to win: just flood the opponent with Internet traffic, slowing their network connection. This is called a DoS, which stands for "Denial of Service". DoSing game competitors is often a teenager's first foray into hacking.

A variant of this is when you hack a bunch of other machines on the Internet, then command them to flood your target. (The hacked machines are often called a "botnet", a network of robot computers). This is called DDoS, or "Distributed DoS". At this point, it gets quite serious, as instead of competitive gamers hackers can take down entire businesses. Extortion scams, DDoSing websites then demanding payment to stop, is a common way hackers earn money.

Another form of DDoS is "amplification". Sometimes when you send a packet to a machine on the Internet it'll respond with a much larger response, either a very large packet or many packets. The hacker can then send a packet to many of these sites, "spoofing" or forging the IP address of the victim. This causes all those sites to then flood the victim with traffic. Thus, with a small amount of outbound traffic, the hacker can flood the inbound traffic of the victim.

This is one of those things that has worked for 20 years, because it's so obvious teenagers can do it, yet there is no obvious solution. President Trump's executive order of cyberspace specifically demanded that his government come up with a report on how to address this, but it's unlikely that they'll come up with any useful strategy.

More: Wikipedia on DDoS, Wikipedia on Spoofing

Conclusion

Tweet me (@ErrataRob) your obvious hacks, so I can add them to the list.



by Robert Graham (noreply@blogger.com) at August 01, 2017 01:45 PM

July 31, 2017

pagetable

62 Reverse-Engineered C64 Assembly Listings

Between 1992 and 1995, I reverse engineered Commodore 64 applications by printing their disassemblies on paper and adding handwritten comments (in German). These are the PDF scans of the 62 applications, which are 552 pages total.

File Author Date Description
Adv. Squeezer 1995-08-19 Decompression code
Amica Paint Schnellader Fast loader Amica Paint extension
Bef.-Erw. Diethelm Berens BASIC extension
Bitmap Manager Hannes Sommer 1994-10-26 AGSP map editor
Bitstreamer 1994-11-06 Decompression code
Blanking Screen blanker
Creatures 2 Autostart John Rowlands 1993-05-03
Creatures 2 Loader John Rowlands 1993-05-04
Delta Coder Nikolaus Heusler 1994-08-25
Drive Composer Chester B. Kollschen 1994-05-25 1541 drive head sample player, Magic Disk 07/1993
Drive Digi Player Michael Steil 1994-05-25 1541 drive head sample player
Drive ROMs 1995-04-19 Differences of the 1541/-II/-C/1571/1581 ROMs
EmBa Nikolaus Heusler 1992-08-08 Emergency BASIC
Errorline-Lister 1992-08-18 BASIC extension
Fast Disk Set
Fast Load Frank Deizner 1994-09-29
Fast Save Explorer/Agony 1995-08-24
File Copier 1993-07-22
Final Cartridge III Freezer Loader 1994-05-01 Fast load/decompression code of the FC3 unfreeze binary
Fred’s Back II Hannes Sommer 1994-12 Parts of the AGSP game
GEO-RAM-Test Mario Büchle 1996-05-21 GeoRAM detection code
GEOS Select Printer 1993-01-25 GEOS tool
GoDot A. Dettke, W. Kling 1994-05-17 Core code, ldr.koala, svr.doodle
Graphtool 1992
Heureka-Sprint 1994-01 Fast loader
IRQ Loader 2 Bit Explorer/Agony 1995-08-24 Fast loader
Kunst aus China Autostart 1993-03-31
MIST Entpacker Michael Steil 1995-08-21 Decompression code
MIST Load V2 Michael Steil Fast load
MSE V1.1 1992-08-08 Programm type-in helper from 64′er Magazin
MSE V2.1 1993-03-28 Programm type-in helper from 64′er Magazin
MTOOL Michail Popov 1992-08-14
Mad Code VSP 1994-10-21 VSP + DYSP code from the demo “Mad Code”
Mad Format 1995-04-20 Fast format
Magic Basic BASIC extension
Magic Disk Autostart 1992-07
Master Copy 1995-09-21 Disk copy
Master Cruncher 1994-05-25 Decompression code
Mayhem in Monsterland VSP John Rowlands 1994-12 VSP code from the game “Mayhem in Monsterland”
Mini-Erweiterung Marc Freese 1992-08-09
Mini-Scan The Sir 1993-07-22 Disk error scanner
Mini-Tris Tetris
Movie-Scroller Ivo Herzeg 1995-01
RAM-Check T. Pohl 1995-09-08 REU detection
SUCK Copy 1995-04-30 File copy
SUPRA 64 Loader
Shapes 64 R. Löwenstein 1994-09-05 Text mode windowing
Small Ass S. Berghofer 1993-08-07 Assembler
Spriterob Andreas Breuer 1992
Startprg.obj M. Hecht 1992-06
SuperinfoIRQ Nikolaus Heusler 1992-06
Swappers Best Copy Disk copy
TPP’s Screensplitter Armin Beck 1995-03
The Sampler 1994-04-30 $D418 sample player (high volume on 8580!)
Tiny Compiler Nikolaus Heusler 1994-09-06 BASIC compiler
Turrican 1 Autostart Manfred Trenz 1994-11-02
Turrican 1 Loader Manfred Trenz 1994
Turrican 2 Autostart Manfred Trenz 1994-10-03
Vectorset M. Strelecki 1995-02-05
Vis-Ass Mazim Szenessy 1992-08 Assembler and screen editor
Vocabulary Schindowski 1993-01-05 FORTH library
Warp-Load 1994-10-02 Fast loader

by Michael Steil at July 31, 2017 03:28 PM

July 28, 2017

Steve Kemp's Blog

So I'm considering a new project

In the past there used to be a puppet-labs project called puppet-dashboard, which would let you see the state of your managed-nodes. Having even a very basic and simple "report user-interface" is pretty neat when you're pushing out a change, and you want to see it be applied across your fleet of hosts.

There are some other neat features, such as allowing you to identify failures easily, and see nodes that haven't reported-in recently.

This was spun out into a community-supported project which is largely stale:

Having a dashboard is nice, but the current state of the software is less good. It turns out that the implementation is pretty simple though:

  • Puppet runs on a node.
  • The node reports back to the puppet-master what happened.
  • The puppet-master can optionally HTTP-post that report to the reporting node.

The reporting node can thus receive real-time updates, and do what it wants with them. You can even sidestep the extra server if you wish:

  • The puppet-master can archive the reports locally.

For example on my puppet-master I have this:

  root@master /var/lib/puppet/reports # ls | tail -n4
  smaug.dh.bytemark.co.uk
  ssh.steve.org.uk
  www.dns-api.com
  www.steve.org.uk

Inside each directory is a bunch of YAML files which describe the state of the host, and the recipes that were applied. Parsing those is pretty simple, the hardest part would be making a useful/attractive GUI. But happily we have the existing one to "inspire" us.

I think I just need to write down a list of assumptions and see if they make sense. After all the existing installation(s) won't break, it's just a matter of deciding whether it is useful/worthwhile way to spend some time.

  • Assume you have 100+ hosts running puppet 4.x
  • Assume you want a broad overview:
    • All the nodes you're managing.
    • Whether their last run triggered a change, resulted in an error, or logged anything.
    • If so what changed/failed/was output?
  • For each individual run you want to see:
    • Rough overview.
  • Assume you don't want to keep history indefinitely, just the last 50 runs or so of each host.

Beyond that you might want to export data about the managed-nodes themselves. For example you might want a list of all the hosts which have "bash" installed on them. Or "All nodes with local user "steve"." I've written that stuff already, as it is very useful for auditing & etc.

The hard part about that is that to get the extra data you'll need to include a puppet module to collect it. I suspect a new dashboard would be broadly interesting/useful but unless you have that extra detail it might not be so useful. You can't point to a slightly more modern installation and say "Yes this is worth migrating to". But if you have extra meta-data you can say:

  • Give me a list of all hosts running wheezy.
  • Give me a list of all hosts running exim4 version 4.84.2-2+deb8u4.

And that facility is very useful when you have shellshock, or similar knocking at your door.

Anyway as a hacky start I wrote some code to parse reports, avoiding the magic object-fu that the YAML would usually invoke. The end result is this:

 root@master ~# dump-run www.steve.org.uk
 www.steve.org.uk
    Puppet Version: 4.8.2
    /var/lib/puppet/reports/www.steve.org.uk/201707291813.yaml
    Runtime: 2.16
    Status:changed
    Time:2017-07-29 18:13:04 +0000
    Resources
            total -> 176
            skipped -> 2
            failed -> 0
            changed -> 3
            out_of_sync -> 3
            scheduled -> 0
            corrective_change -> 3
    Changed Resources
            Ssh_authorized_key[skx@shelob-s-fi] /etc/puppet/code/environments/production/modules/ssh_keys/manifests/init.pp:17
            Ssh_authorized_key[skx@deagol-s-fi] /etc/puppet/code/environments/production/modules/ssh_keys/manifests/init.pp:22
            Ssh_authorized_key[steve@ssh.steve.org.uk-s-fi] /etc/puppet/code/environments/production/modules/ssh_keys/manifests/init.pp:27
    Skipped Resources
            Exec[clone sysadmin utils]
            Exec[update sysadmin utils]

July 28, 2017 09:00 PM

July 25, 2017

OpenSSL

FIPS 140-2: And So It Begins

It’s been almost a year since plans for a new FIPS 140 validation were first announced. Several factors have led to this long delay. For one, we chose to focus our limited manpower resources on higher priority objectives such as the TLS 1.3 implementation. SafeLogic has also experienced difficulties in obtaining the funding for their intended sponsorship; potential sponsors can contact them directly.

With TLS 1.3 now done (pending only a final TLS 1.3 specification) we’re now in a position to turn our attention to the new FIPS module, and just in the nick of time Oracle has pledged enough funding to get us off to a good start. With financial support from the Linux Foundation Core Infrastructure Initiative temporarily interrupted, leaving a team member with no income, that funding eases the pressure to seek new long term employment.

The bad news is that the Oracle funding will only partially cover the FIPS module work (for perhaps a couple of months), at which point we may have to set that work aside. Hand-to-mouth funding is not a new experience for us though so we’ll worry about that later.

The FIPS module is heavily shaped and constrained (one could even say distorted and contorted) by FIPS 140 requirements. Those requirements (or technically speaking the interpretation of those requirements) has changed considerably since our last open source based validation in 2013, so we’re starting with a careful study of the many requirements changes that have accumulated since then. That study will generate a lot of questions for the accredited test lab, as the practical application of the formal requirements to working code is rarely obvious to anyone.

One goal for this new FIPS module is to make a clean break from the legacy code base of the previous module, which started as a stripped and bastardized version of an old copy of OpenSSL. We’ll be making the new module as simple as possible without extraneous vestigial code. It will live in a new separate git repository, though don’t expect to see a lot of code right away as we work through the requirements questions.

As before the FIPS module will be primarily intended for use with OpenSSL proper, though we hope to minimize (or even eliminate) FIPS specific code in OpenSSL by enhancing the current ENGINE interface. The new FIPS module will have an internal API (with non-opaque structures) that in turn will be wrapped in a higher level ENGINE interface package external to the “cryptographic module boundary”. All three components (formal validated module, module interface wrapper, and OpenSSL proper) will as before be usable as a seamless “FIPS capable” OpenSSL library.

The test suite software will interface with the module directly, and that code will be separate from the module itself. We’ll be sorting out the outlines of these separate components as soon as we’ve confirmed we understand the new requirements.

I’ll blog more as we finalize additional details.

July 25, 2017 08:00 PM

July 24, 2017

Evaggelos Balaskas

Let's Encrypt - Auto Renewal

Let’s Encrypt

I’ve written some posts on Let’s Encrypt but the most frequently question is how to auto renew a certificate every 90 days.

Disclaimer

This is my mini how-to, on centos 6 with a custom compiled Python 2.7.13 that I like to run on virtualenv from latest git updated certbot. Not a copy/paste solution for everyone!

Cron

Cron doesnt not seem to have something useful to use on comparison to 90 days:

crond.png

Modification Time

The most obvious answer is to look on the modification time on lets encrypt directory :

eg. domain: balaskas.gr

# find /etc/letsencrypt/live/balaskas.gr -type d -mtime +90 -exec ls -ld {} \;

# find /etc/letsencrypt/live/balaskas.gr -type d -mtime +80 -exec ls -ld {} \;

# find /etc/letsencrypt/live/balaskas.gr -type d -mtime +70 -exec ls -ld {} \;

# find /etc/letsencrypt/live/balaskas.gr -type d -mtime +60 -exec ls -ld {} \;

drwxr-xr-x. 2 root root 4096 May 15 20:45 /etc/letsencrypt/live/balaskas.gr

OpenSSL

# openssl x509 -in <(openssl s_client -connect balaskas.gr:443 2>/dev/null) -noout -enddate

Email

If you have registered your email with Let’s Encrypt then you get your first email in 60 days!

Renewal

Here are my own custom steps:

#  cd /root/certbot.git
#  git pull origin 

#  source venv/bin/activate && source venv/bin/activate
#  cd venv/bin/

#  monit stop httpd 

#  ./venv/bin/certbot renew --cert-name balaskas.gr --standalone 

#  monit start httpd 

#  deactivate

Script

I use monit, you can edit the script accordingly to your needs :

#!/bin/sh

DOMAIN=$1

## Update certbot
cd /root/certbot.git
git pull origin 

# Enable Virtual Environment for python
source venv/bin/activate && source venv/bin/activate 

## Stop Apache
monit stop httpd 

sleep 5

## Renewal
./venv/bin/certbot renew  --cert-name ${DOMAIN} --standalone 

## Exit virtualenv
deactivate 

## Start Apache
monit start httpd

All Together

# find /etc/letsencrypt/live/balaskas.gr -type d -mtime +80 -exec /usr/local/bin/certbot.autorenewal.sh balaskas.gr \;

Systemd Timers

or put it on cron

whatever :P

Tag(s): letsencrypt

July 24, 2017 10:03 PM

July 22, 2017

Everything Sysadmin

Google's BBR fixes TCP's dirty little secret

Networking geeks: Google made a big announcements about BBR this week. Here's a technical deep-dive: http://queue.acm.org/detail.cfm?id=3022184 (Hint: if you would read ACM Queue like I keep telling you to, you'd have known about this before all your friends.)

Someone on Facebook asked me for a "explain it like I'm 5 years old" explanation. Here's my reply:

Short version: Google changed the TCP implementation (their network stack) and now your youtube videos, Google websites, Google Cloud applications, etc. download a lot faster and smoother. Oh, and it doesn't get in the way of other websites that haven't made the switch. (Subtext: another feature of Google Cloud that doesn't exist at AWS or Azure. Nothing to turn on, no extra charge.)

ELI5 version: TCP tries to balance the need to be fast and fair. Fast... transmitting data quickly. Fair... don't hog the internet, share the pipe. Being fair is important. In fact, it is so important that most TCP implementations use a "back off" algorithm that results in you getting about 1/2 the bandwidth of the pipe... even if you are the only person on it. That's TCP's dirty little secret: it under-utilizes your network connection by as much as 50%.

Backoff schemes that use more than 1/2 the pipe tend to crowd out other people, thus are unfair. So, in summary, the current TCP implementations prioritize fairness over good utilization. We're wasting bandwidth.

Could we do better? Yes. There are better backoff algorithms but they are so much work that they are impractical. For years researchers have tried to make better schemes that are easy to compute. (As far back as the 1980s researchers built better and better simulations so they could experiment with different backoff schemes.)

Google is proposing a new backoff algorithm called BBR. It has reached the holy grail: It is more fair than existing schemes. If a network pipe only has one user, they basically use the whole thing. If many users are sharing a pipe, it shares it fairly. You get more download speed over the same network. Not only that, it doesn't require changes to the internet, just the sender.

And here's the real amazing part: it works if you implement BBR on both the client and the server, but it works pretty darn good if only change the sender's software (i.e. Google updated their web frontends and you don't have to upgrade your PC). Wait! Even more amazing is that it doesn't ruin the internet if some people use it and some people use the old methods.

They've been talking about it for nearly a year at conferences and stuff. Now they've implemented it at www.google.com, youtube.com, and so on. You get less "buffering.... buffering..." even on mobile connections. BBR is enabled "for free" for all Google Cloud users.

With that explanation, you can probably read the ACM article a bit easier. Here's the link again: http://queue.acm.org/detail.cfm?id=3022184

Disclaimer: I don't own stock in Google, Amazon or Microsoft. I don't work for any of them. I'm an ex-employee of Google. I use GCP, AWS and Azure about equally (nearly zero).

by Tom Limoncelli at July 22, 2017 07:07 PM

July 19, 2017

ma.ttias.be

Podcast: The Ceremony, the birth of Zcash

The post Podcast: The Ceremony, the birth of Zcash appeared first on ma.ttias.be.

This was an interesting -- an unexpected -- podcast from the the folks at Radiolab.

Last November, journalist Morgen Peck showed up at her friend Molly Webster's apartment in Brooklyn, told her to take her battery out of her phone, and began to tell her about The Ceremony, a moment last fall when a group of, well, let's just call them wizards, came together in an undisclosed location to launch a new currency.

It's an undertaking that involves some of the most elaborate security and cryptography ever done (so we've been told). And math. Lots of math. It was all going great until, in the middle of it, something started to behave a little...strangely.

Source: The Ceremony -- Radiolab

The story is about the birth of Zcash & the paranoia involved in creating the initial keys to launch a new cryptocurrency.

After this episode I'm still stuck with lots of questions;

  • Who choose the location of the computenode purchase?
  • Who vetted the recording equipment?
  • If a phone got compromised during key initialization, why didn't they abort?

For a team this paranoid, it sounds like they made a couple of strange choices along the way. Or at least, it was portraid as such in the story.

Either way, it was a fun listen and interesting to hear terms like RSA 4096, Diffie-Hellman and cryptocurrency get mentioned in an otherwise rather sciency-podcast!

The post Podcast: The Ceremony, the birth of Zcash appeared first on ma.ttias.be.

by Mattias Geniar at July 19, 2017 07:37 AM

July 18, 2017

ma.ttias.be

Choose source IP with ping to force ARP refreshes

The post Choose source IP with ping to force ARP refreshes appeared first on ma.ttias.be.

This is a little trick you can use to force outgoing traffic via one particular IP address on a server. By default, if your server has multiple IPs, it'll use the default IP address for any outgoing traffic.

However, if you're changing IP addresses or testing failovers, you might want to force traffic to leave your server as if it's coming from one of the other IP addresses. That way, upstream switches learn your server has this IP address and they can update their ARP caches.

ping allows you to do that very easily (sending ICMP traffic).

$ ping 8.8.8.8 -I 10.0.1.4

The -I parameter sets the interface via which packets should be sent, it can accept either an interface name (like eth1) or an IP address. This way, traffic leaves your server with srcip 10.0.1.4.

Docs describe -I like this;

-I interface address
 Set  source address to specified interface address. Argument may be numeric IP
 address or name of device. When pinging IPv6 link-local address this option is
 required.

So works for both individual IP addresses as well as interfaces.

The post Choose source IP with ping to force ARP refreshes appeared first on ma.ttias.be.

by Mattias Geniar at July 18, 2017 07:30 PM

July 17, 2017

Everything Sysadmin

Four Ways to Make CS and IT Curricula More Immersive

My new column in ACM Queue is entitled, "Four Ways to Make CS and IT Curricula More Immersive". I rant and rail against the way that CS and IT is taught today and propose 4 ways CS educators can improve the situation.

The article is free to ACM members. Non-members can purchase an annual subscription for $19.99 or a single issue for $6.99 online or through the Apple or Google stores.

by Tom Limoncelli at July 17, 2017 03:00 PM

July 14, 2017

Sean's IT Blog

Top 10 EUC Sessions at #VMworld 2017 Las Vegas

VMworld 2017 is just around the corner.  The premier virtualization conference will be returning to the Mandalay Bay convention center in Las Vegas at the end of August. 

There is one major addition to the EUC content at VMworld this year.  VMware has decided to move the Airwatch Connect conference, which cover’s VMware’s offerings in the mobility management space, from Atlanta and colocate it with VMworld.  So not only do attendees interested in EUC get great expert content on VMware’s Horizon solutions, they’ll get more content on Airwatch, mobility management, identity management, and IoT as well.

My top 10 EUC sessions for 2017 are:

  1. ADV1594BU – Beyond the Marketing: VMware Horizon 7.1 Instant Clones Deep Dive – This session, by Jim Yanik and Oswald Chen, is a technical deep dive into how Instant Clone desktops work.  This updated session will cover new features that have been added to Instant Clones since they were released in Horizon 7.  I’m often wary of “deep dive sessions,” but I’ve seen Jim give a similar presentation at various events and he does a great job talking through the Instant Clone technology in a way that all skill levels can understand it.  If you’re interested in VMware EUC, this is the one session you must attend as this technology will be relevant for years to come. 
  2. ADV1609BU – Deliver Any App, Any Desktop, Anywhere in the World Using VMware Blast Extreme – Blast Extreme is VMware’s new protocol that was officially introduced in Horizon 7.  Pat Lee and Ramu Panayappan will provide a deep dive into Blast Extreme.  Pat does a good job talking about Blast Extreme and how it works, and attendees will definitely walk away having learned something.
  3. ADV1681GU/ADV1607BU – Delivering 3D graphics desktops and applications in the real world with VMware Horizon, BEAT and NVIDIA GRID – VMware’s Kiran Rao and NVIDIA’s Luke Wignall talk about how Blast Extreme utilizes NVIDIA GPUs to provide a better user experience in End-User Computing environments.  This session was actually listed twice in the Content Catalog, so don’t worry if you miss one.
  4. ADV1583BU – Delivering Skype for Business with VMware Horizon: All You Need to Know – Official support for Skype for Business GA’d with Horizon 7.2.  This session will dive into how that the new Skype for Business plugin works to provide a better telephony experience in EUC environments.
  5. ADV3370BUS – DeX Solutions: How Samsung and VMware are Pioneering Digital Transformation – Samsung DeX is a new cell phone from Samsung that, when placed in a dock, can utilize a keyboard, mouse, and monitor to act as a virtual thin client endpoint while still having all the capabilities of a phone.  DeX has the potential to revolutionize how businesses provide endpoints and cellular phones to users.
  6. ADV1655GU – CTOs perspective on the Workspace 2020 and beyond: time to act now! – End-User Computing expert and technology evangelist Ruben Spruijt talks about the future of the end-user workspace and strategies on how to implement next-generation workspace technology.
  7. UEM1359BU – Best Practices in Migrating Windows 7 to Windows 10 – Windows 10 migrations are a hot topic, and almost every business will need a Windows 10 strategy.  This session will explore the best practices for migrating to Windows 10 in any type of organization.
  8. SAAM1684GU – Ask the Experts: How to Enable Secure Access from Personal/BYO Devices and All Types of Users with Workspace ONE – How do you enable secure remote access to company resources while allowing employees, contractors, and other types of workers to use their personal devices?  This group discussion will cover best practices for using VMware Workspace ONE to provide various levels of secure access to company resources from personal devices based on various context settings.  Unlike most sessions, this is a group discussion.  There are very few slides, and most of the session time will be devoted to allowing attendees to ask questions to the discussion leaders.
  9. ADV1588BU – Architecting Horizon 7 and Horizon Apps – A successful EUC environment starts with a solid architecture.  This session covers how to architect an integrated Horizon environment consisting of all components of the Horizon Suite. 
  10. vBrownbag TechTalks on EUC – There are three community driven vBrownbag Tech Talks focusing on EUC led by EUC Champions.  These talks are:
    1. GPU-Enabled Linux VDI by Tony Foster – Tony will cover how to build GPU-enabled Linux virtual desktops in Horizon and some of the pain points he encountered while implementing this solution at a customer.
    2. Windows 10 and VDI – Better Come Prepared – Rob Beekmans and Sven Huisman will cover lessons they’ve learned while implementing Windows 10 in VDI environments.
    3. Leveraging User Environment Manager to Remove GPOs – Nigel Hickey will cover how to use VMware UEM as a Group Policy replacement tool.
  11. ADV1605PU – Ask the Experts: Practical Tips and Tricks to Help You Succeed in EUC – So this top 10 list will actually have 11 entries, and this one is a bit of shameless self-promotion.  This session is s a repeat of last year’s EUC champions session featuring Earl Gay, VCDX Johan van Amersfoot, moderator Matt Heldstab, and I.  We’re answering your questions about EUC based on our experiences in the trenches.  Last year, we also had some prizes. 

Bonus Session

There is one bonus session that you must put on your schedule.  It’s not EUC-related, but it is put on by two of the smartest people in the business today.  They were also two of my VCDX mentors.  The session is Upgrading to vSphere 6.5 the VCDX Way [SER2318BU] by Rebecca Fitzhugh and Melissa Palmer.  You should seriously check this session out as they’ll provide a roadmap to take your environment up to vSphere 6.5. 


by seanpmassey at July 14, 2017 05:00 PM

Evaggelos Balaskas

Install Slack Desktop to Archlinux

How to install slack dekstop to archlinux

Download Slack Desktop

eg. latest version

https://downloads.slack-edge.com/linux_releases/slack-2.6.3-0.1.fc21.x86_64.rpm

Extract under root filesystem

# cd /

# rpmextract.sh slack-2.6.3-0.1.fc21.x86_64.rpm

Done

Actually, that’s it!

Run

Run slack-desktop as a regular user:

$ /usr/lib/slack/slack

Slack Desktop

slackdesktop.jpg

Proxy

Define your proxy settings on your environment:

declare -x ftp_proxy="proxy.example.org:8080"
declare -x http_proxy="proxy.example.org:8080"
declare -x https_proxy="proxy.example.org:8080"

Slack

slackdesktop2.jpg

Tag(s): slack

July 14, 2017 10:28 AM

July 12, 2017

Debian Administration

Implementing two factor authentication with perl

Two factor authentication is a system which is designed to improve security, by requiring a second factor, in addition to a username/password pair, to access a particular resource. Here we'll look briefly at how you add two factor support to your applications with Perl.

by Steve at July 12, 2017 05:23 PM

July 11, 2017

The Lone Sysadmin

The Dangers of Experts Writing Documentation: A Real Life Example

There are some real, tangible dangers to having experts write documentation. Experts have the perfect tools, skip steps, know where things are based on experience, use jargon, have spare parts so mistakes aren’t a big deal, and as a result make terrible time & work estimates. This leads to confused, and subsequently angry, people, which […]

The post The Dangers of Experts Writing Documentation: A Real Life Example appeared first on The Lone Sysadmin. Head over to the source to read the full post!

by Bob Plankers at July 11, 2017 05:58 AM

July 10, 2017

pagetable

80 Columns Text on the Commodore 64

The text screen of the Commodore 64 has a resolution of 40 by 25 characters, based on the hardware text mode of the VIC-II video chip. This is a step up from the VIC-20′s 22 characters per line, but since computers in the professional segment (Commodore PET 8000 series, CP/M, MS-DOS) usually had 80 columns, several solutions – both hardware and software – exist to allow 80 columns on a C64 as well. Let’s look at how this is done in software! At the end of this article, I present a fast and full-featured open source implementation with several different character sets.

Regular 40×25 Mode

First, we need to understand how 40 columns are done. The VIC-II video chip has dedicated support for a text mode. There is a 1000 (= 40 * 25) byte “Screen RAM”, each byte of which contains the character to be displayed at that location in the form of an index into the 2 KB character set, which contains 8 bytes (for 8×8 pixels) for each of the 256 characters. In addition, there is a “Color RAM”, which contains a 1000 (again 40 * 25) 4-bit values, which represents a color for each character on the screen.

Putting a character onto the screen is quite trivial: Just write the index of it to offset column + 40 * line into Screen RAM, and its color to the same offset in Color RAM. An application can load its own character set, but the Commodore 64 already comes with two character sets in ROM: One with uppercase characters and lots of graphical symbols (“GRAPHICS”), and one with upper and lower case (“TEXT”). You can switch these by pressing the Commodore and the Shift key at the same time.

Bitmap Mode

There is no hardware support for an 80 column mode, but such a mode can be implemented in software by using bitmap mode. In bitmap mode, all 320 by 200 pixels on the screen can be freely addressed through an 8000 byte bitmap, which contains one bit for every pixel on the screen. Luckily for us, the layout of the bitmap in memory is not linear, but reminds of the encoding of text mode: The first 8 bytes in Bitmap RAM don’t describe, as you would expect, the leftmost 64 pixels on the first line. Instead, they describe the top left 8×8 block. The next 8 bytes describe the 8×8 block to the right of it, and so on.

This is the same layout as the character set’s: The first 8 bytes correspond to the first character, the next 8 bytes to the second character and so on. Drawing an 8×8 character onto a bitmap (aligned to the 8×8 grid) is as easy as copying 8 consecutive bytes.

This is what an 8×8 font looks like in memory:

0000 ··████··  0008 ···██···  0010 ·█████··
0001 ·██··██·  0009 ··████··  0011 ·██··██·
0002 ·██·███·  000a ·██··██·  0012 ·██··██·
0003 ·██·███·  000b ·██████·  0013 ·█████··
0004 ·██·····  000c ·██··██·  0014 ·██··██·
0005 ·██···█·  000d ·██··██·  0015 ·██··██·
0006 ··████··  000e ·██··██·  0016 ·█████··
0007 ········  000f ········  0017 ········

For an 80 column screen, every character is 4×8 pixels. So we could describe the character set like this:

0000 ········  0008 ········  0010 ········
0001 ··█·····  0009 ··█·····  0011 ·██·····
0002 ·█·█····  000a ·█·█····  0012 ·█·█····
0003 ·███····  000b ·███····  0013 ·██·····
0004 ·███····  000c ·█·█····  0014 ·█·█····
0005 ·█······  000d ·█·█····  0015 ·█·█····
0006 ··██····  000e ·█·█····  0016 ·██·····
0007 ········  000f ········  0017 ········

Every 4×8 character on the screen is either in the left half or the right half of an 8×8 block, so drawing an 4×8 character is as easy as copying the bit pattern into the 8×8 block – and shifting it 4 bits to the right for characters at odd positions.

Color

In bitmap mode, it is only possible to use two out of the 16 colors per 8×8 block, because there are only 1000 (40 * 25) entries for the color matrix. This is a problem, since we need three colors per 8×8: Two for the two characters and one for the background. We will have to compromise: Whenever a character gets drawn into an 8×8 block and the other character in the block has a different color, that other character will be changed to the same color as the new character.

Scrolling

Scrolling on a text screen is easy: 960 bytes of memory have to be copied to move the character indexes to their new location. In bitmap mode, 7680 bytes have to be copied – 8 times more. Even with the most optimized implementation (73ms, about 3.5 frames), scrolling will be slower, and tearing artifacts are unavoidable.

Character Set

Creating a 4×8 character set that is both readable and looks good is not easy. There has to be a one-pixel gap between characters, so characters can effectively only be 3 pixels wide. For characters like “M” and “N”, this is a challenge.

These are the character sets of four different software solutions for 80 columns:

COLOR 80 by Richvale Telecommunications
  

80COLUMNS
  

SCREEN-80 by Compute’s Gazette
  

Highspeed80 by CKtwo
  

Some observations:

  • Highspeed80 and SCREEN-80 have capitals of a height of 7 pixels (more detail, but very tight vertical spacing, making it hard to read), while COLOR 80 uses only 5 pixels (more square like the original 8×8 font, but less detail). 6 pixels, as used by 80COLUMNS, seems like a good compromise.
  • 80COLUMNS has the empty column at the left, which makes single characters in reverse mode more readable, since most characters have their stems at the left.
  • Except for Highspeed80, the graphical symbols are very similar between the different character sets.
  • All four character sets use the same strategy to distinguish between “M” and “N”.

The Editor

The “EDITOR” is the part of C64′s ROM operating system (“KERNAL”) that handles printing characters to the screen (and interpreting control characters), as well as converting on-screen contents back into a PETSCII string – yes, text input on CBM computers is done by feeding the keyboard input directly into character output, and reading back the screen contents when the user presses the return key. This way, the user can use the cursor keys to navigate to existing text anywhere on the screen (even to the output of previous commands), edit it, and have the system interpret it as input when pressing return.

The C64′s EDITOR can only deal with 40 columns (but has a very nice feature that allows using two 40 character lines as one virtual 80 character input line), and has no idea how to draw into a bitmap, so a software 80 characters solution basically has to provide a complete reimplementation of this KERNAL component.

The KERNAL provides user vectors for lots of its functionality, so both character output, and reading back characters from the screen can be hooked (vectors IBSOUT at $0326 and IBASIN at $0324). In addition to drawing characters into the bitmap, the character codes have to be cached in a 80×25 virtual Screen RAM, so the input routine can read them back.

The PETSCII code contains control codes for changing the current color, moving the cursor, clearing the screen, turning reverse on and off, and switching between the “GRAPHICS” and “TEXT” character sets. The new editor has provide code to interpret these. There are two special cases though: When in quote mode (the user is typing text between quotes) or insert mode (the user has typed shift+delete), most special characters show up as inverted graphical characters instead of being interpreted. This way, control characters can be included e.g. in strings in BASIC programs.

There are two functions though that cannot be intercepted through vectors: Applications (and BASIC programs) change the screen background color by writing the color’s value to $d020, since there is no KERNAL function or BASIC command for it, and the KERNAL itself switches between the two character sets (when the user presses the Commodore and the Shift key at the same time) by directly writing to $d018. The only way to intercept these is by hooking the timer interrupt vector and detecting a change in these VIC-II registers. If the background color has changed, the whole 1000 byte color matrix for bitmap mode has to be updated, and if the character set has changed, the whole screen has to be redrawn.

The Implementation

I looked at all existing software implementations I could find and concluded that “80COLUMNS” (by an unknown author) had the best design and was the only one to implement the full feature set of the original EDITOR. I reverse-engineered it into structured, easy to read code, added Ilker Ficicilar’s fast scrolling patch as well as my own minor cleanups, fixes and optimizations.

https://www.github.com/mist64/80columns

The project requies cc65 and exomizer to build. Running make will produce 80columns-compressed.prg, which is about 2.2 KB in size and can be started using LOAD/RUN.

The source contains several character sets (charset.s, charset2.s etc.) from different 80 column software solutions, which can be selected by changing the reference to the filename in the Makefile.

The object code resides at $c800-$cfff. The two character sets are located at $d000-$d7ff. The virtual 80×25 Screen RAM (in order to read back screen contents) is at $c000-$c7ff. The bitmap is at $e000-$ff40, and the color matrix for bitmap mode is at $d800-$dbe8. All this lies beyond the top of BASIC RAM, so BASIC continues to have 38911 bytes free.

In order to speed up drawing, the character set contains all characters duplicated like this:

0000 ········  0008 ········  0010 ········
0001 ··█···█·  0009 ··█···█·  0011 ·██··██·
0002 ·█·█·█·█  000a ·█·█·█·█  0012 ·█·█·█·█
0003 ·███·███  000b ·███·███  0013 ·██··██·
0004 ·███·███  000c ·█·█·█·█  0014 ·█·█·█·█
0005 ·█···█··  000d ·█·█·█·█  0015 ·█·█·█·█
0006 ··██··██  000e ·█·█·█·█  0016 ·██··██·
0007 ········  000f ········  0017 ········

This way, the drawing code only has to mask the value instead of shifting it. In addition, parts of character drawing and all of screen scrolling are using unrolled loops for performance.

Contributions to the project are very welcome. It would be especially interesting to add new character sets, both existing 4×8 fonts from other projects (including hinted TrueType fonts!), and new ones that combine the respective strengths of the existing ones.

80×33 Mode?

Reducing the size of characters to 4×6 would allow a text mode resolution of 80×33 characters. Novaterm 10 has an implementation. At this resolution, logical characters don’t end at vertical 8×8 boundaries any more, making color impossible, and the drawing routine a little slower. It would be interesting to add an 80×33 mode as a compile time option to “80columns”.

by Michael Steil at July 10, 2017 09:45 PM

That grumpy BSD guy

OpenBSD and the modern laptop

Did you think that OpenBSD is suitable only for firewalls and high-security servers? Think again. Here are my steps to transform a modern mid to high range laptop into a useful Unix workstation with OpenBSD.

One thing that never ceases to amaze me is that whenever I'm out and about with my primary laptop at conferences and elsewhere geeks gather, a significant subset of the people I meet have a hard time believing that my laptop runs OpenBSD, and that it's the only system installed.

A typical exchange runs something like,
"So what system do you run on that laptop there?"
"It's OpenBSD. xfce is the window manager, and on this primary workstation I tend to just upgrade from snapshot to snapshot."
"Really? But ..."
and then it takes a bit of demonstrating that yes, the graphics runs with the best available resolution the hardware can offer, the wireless network is functional, suspend and resume does work, and so forth. And of course, yes, I do use that system when writing books and articles too. Apparently heavy users of other free operating systems do not always run them on their primary workstations.

I'm not sure at what time I permanently converted my then-primary workstation to run OpenBSD exclusively, but I do remember that when I took delivery of the ThinkPad R60 (mentioned in this piece) in 2006, the only way forward was to install the most recent OpenBSD snapshot. By mid-2014 the ThinkPad SL500 started falling to pieces, and its replacement was a Multicom Ultrabook W840, manufactured by Clevo. The Clevo Ultrabook has weathered my daily abuse and being dragged to various corners of the world for conferences well, but on the trek to BSDCan 2017 cracks started appearing in the glass on the display and the situation worsened on the return trip.

So the time came to shop around for a replacement. After a bit of shopping around I came back to Multicom, a small computers and parts supplier outfit in rural Åmli in southern Norway, the same place I had sourced the previous one.

One of the things that attracted me to that particular shop and their own-branded offerings is that they will let you buy those computers with no operating system installed. That is of course what you want to do when you source your operating system separately, as we OpenBSD users tend to do.

The last time around I had gone for a "Thin and lightweight" 14 inch model (Thickness 20mm, weight 2.0kg) with 16GB RAM, 240GB SSD for system disk and 1TB HD for /home (since swapped out for a same-size SSD, as the dmesg will show).

Three years later, the rough equivalent with some added oomph for me to stay comfortable for some years to come ended me with a 13.3 inch model, 18mm thick and advertised as 1.3kg (but actually weighing in at 1.5kg, possibly due to extra components), 32GB RAM, 512GB SSD and 2TB harddisk. For now the specification can be viewed online here (the site language is Norwegian, but product names and units of measure are not in fact different).

That system arrived today, in a slender box:



Here are the two machines, the old (2014-vintage) and the new side by side:



The OpenBSD installer is a wonder of straightforward, no-nonsense simplicity that simply gets the job done. Even so, if you are not yet familiar with OpenBSD, it is worth spending some time reading the OpenBSD FAQ's installation guidelines and the INSTALL.$platform file (in our case, INSTALL.amd64) to familiarize yourself with the procedure. If you're following this article to the letter and will be installing a snapshot, it is worth reading the notes on following -current too.

The main hurdle back when I was installing the 2014-vintage 14" model was getting the system to consider the SSD which showed up as sd1 the automatic choice for booting (I solved that by removing the MBR, setting the size of the MBR on the hard drive that showed up as sd0 to 0 and enlarging the OpenBSD part to fill the entire drive).

Let's see how the new one is configured, then. I try running with the default UEFI "Secure boot" option enabled, and it worked.

Here we see the last part of the messages that scroll across the screen when the new laptop boots from the USB thumbdrive that has had the most recent OpenBSD/amd64 install61.fs dd'ed onto it:



And as the kernel messages showed us during boot (yes, that scrolled off the top before I got around to taking the picture), the SSD came up as sd1 while the hard drive registered as sd0. Keep that in mind for later.



After the initial greeting from the installer, the first menu asks what we want to do. This is a new system, so only (A)utoinstall and (I)nstall would have any chance of working. I had not set up for automatic install this time around, so choosing (I)nstall was the obvious thing to do.

The next item the installer wants to know is which keyboard layout to use and to set as the default on the installed system. I'm used to using Norwegian keyboards, so no is the obvious choice for me here. If you want to see the list of available options, you press ? and then choose the one you find the must suitable.

Once you've chosen the keyboard layout, the installer prompts you for the system's host name. This is only the host part, the domain part comes later. I'm sure your site or organization has some kind of policy in place for choice of host names. Make sure you stay inside any local norms, the one illustrated here conforms with what we have here.

Next up the installer asks which network interfaces to configure. A modern laptop such as this one comes with at least two network interfaces: a wireless interface, in this case an Intel part that is supported in OpenBSD with the iwm(4) driver, and a wired gigabit ethernet interface which the installer kernel recognized as re0.

Quite a few pieces the hardware in a typical modern laptop requires the operating system to load firmware onto the device before it can start interacting meaningfully with the kernel. The Intel wireless network parts supported by the iwm(4) driver and the earlier iwn(4) all have that requirement. However, for some reason the OpenBSD project has not been granted permission to distribute the Intel firmware files, so with only the OpenBSD installer it is not possible to use iwm(4) devices during an initial install. So in this initial round I only configure the re0 interface. During the initial post-install boot the rc.firsttime script will run fw_update(1) command that will identify devices that require firmware files and download them from the most convenient OpenBSD firmware mirror site.

My network here has a DHCP server in place, so I simply choose the default dhcp for IPv4 address assignment and autoconf for IPv6.

With the IPv4 and IPv6 addresses set, the installer prompts for the domain name. Once again, the choice was not terribly hard in my case.



On OpenBSD, root is a real user, and you need to set that user's password even if you will rarely if ever log in directly as root. You will need to type that password twice, and as the install documentation states, the installer will only check that the passwords match. It's up to you to set a usefully strong password, and this too is one of the things organizations are likely to have specific guidelines for.

Once root's password is set, the installer asks whether you want to start sshd(8) by default. The default is the sensible yes, but if you answer no here, the installed system will not have any services listening on the system's reachable interfaces.

The next question is whether the machine will run the X Windows system. This is a laptop with a "Full HD" display and well supported hardware to drive it, so the obvious choice here is yes.

I've gotten used to running with xenodm(1) display manager and xfce as the windowing environment, so the question about xenodm is a clear yes too, in my case.

The next question is whether to create at least one regular user during the install. Creating a user for your systems adminstrator during install has one important advantage: the user you create at this point will be a member of the wheel group, which makes it slightly easier to move to other privilege levels via doas(1) or similar.

Here I create a user for myself, and it is added, behind the scenes, to the wheel group.

With a user in place, it is time to decide whether root will be able to log in via ssh. The sensible default is no, which means you too should just press enter here.

The installer guessed correctly for my time zone, so it's another Enter to move forward.

Next up is the part that people have traditionally found the most scary in OpenBSD installing: Disk setup.

If the machine had come with only one storage device, this would have been a no-brainer. But I have a fast SSD that I want to use as the system disk, and a slightly slower and roomier rotating rust device aka hard disk that I want primarily as the /home partition.

I noted during the bsd.rd boot that the SSD came up as sd1 and the hard drive came up as sd0, so we turn to the SSD (sd1) first.

Since the system successfully booted with the "Secure boot" options in place, I go for the Whole disk GPT option and move on to setting partition sizes.

The default suggestion for disk layout makes a lot of sense and will set sensible mount options, but I will be storing /home on a separate device, so I choose the (E)dit auto layout option and use the R for Resize option to redistribute the space left over to the other partitions.

Here is also where you decide the size of the swap space, traditionally on the boot device's b partition. Both crashdumps and suspend to disk use swap space for their storage needs, so if you care about any of these, you will need to allocate at least as much space as the amount of physical RAM installed in the system. Because I could, I allocated the double of that, or 64GB.

For sd0, I once again choose the Whole disk GPT option and make one honking big /home partition for myself.

The installer then goes on to create the file systems, and returns with the prompt to specify where to find install sets.

The USB drive that I dd'ed the install61.fs image to is the system's third sd device (sd2), so choosing disk and specifying sd2 with the subdirectory 6.1/amd64 makes sense here. On the other hand, if your network and the path to the nearest mirror is fast enough, you may actually save time choosing a http network install over installing from a relatively slow USB drive.

Anyway, the sets install proceeds and trundles through what is likely the longest period of forced inactivity that you will have during an OpenBSD install.

The installer verifies the signed sets and installs them.



Once the sets install is done, you get the offer of specifying more sets -- your site could have a site-specific items in an install set -- but I don't have any of those handy, so I just press enter to accept the default done.

If you get the option to correct system time here, accept it and have ntpd(8) set your system clock to a sane setting gleaned from well known NTP servers.

With everything else in place, the installer links the kernel with a unique layout, in what is right now a -current-only feature, but one that will most likely be one of the more talked-about items in the OpenBSD 6.2 release some time in the not too distant future.

With all items on the installer's agenda done, the installer exits and leaves you at a root shell prompt where the only useful action is to type reboot and press enter. Unless, of course you have specific items you know will need to be edited into the configuration before the reboot.

After completing the reboot, the system did unfortunately not, as expected, immediately present the xenodm login screen, but rather the text login prompt.

Looking at the /var/log/Xorg.0.log file pointed to driver problems, but after a little web searching on the obvious keywords, I found this gist note from notable OpenBSD developer Reyk Flöter that gave me the things to paste into my /etc/xorg.conf to yield a usable graphics display for now.

My task for this evening is to move my working environment to new hardware, so after install there are really only two items remaining, in no particular order:
  • move my (too large) accumulation of /home/ data to the new system, and
  • install the same selection of packages on the old machine to the new system.
The first item will take longer, so I shut down all the stuff I normally have running on the laptop such as web browsers, editors and various other client programs, and use pkg_info(1) to create the list of installed packages on the 'from' system:

$ pkg_info -mz >installed_packages

then I transfer the installed_packages file to the fresh system, but not before recording the df -h status of the pristine fresh install:

$ df -h
Filesystem     Size    Used   Avail Capacity  Mounted on
/dev/sd1a     1005M   76.4M    878M     8%    /
/dev/sd0d      1.8T    552K    1.7T     0%    /home
/dev/sd1d     31.5G   12.0K   29.9G     0%    /tmp
/dev/sd1f     98.4G    629M   92.9G     1%    /usr
/dev/sd1g      9.8G    177M    9.2G     2%    /usr/X11R6
/dev/sd1h      108G    218K    103G     0%    /usr/local
/dev/sd1k      9.8G    2.0K    9.3G     0%    /usr/obj
/dev/sd1j     49.2G    2.0K   46.7G     0%    /usr/src
/dev/sd1e     98.4G    5.6M   93.5G     0%    /var

Not directly visible here is the amount of swap configured in the sd1b partition. As I mentioned earlier, crashdumps and suspend to disk both use swap space for their storage needs, so if you care about any of these, you will need to allocate at least as much space as the amount of physical RAM installed in the system. Because I could, I allocated the double of that, or 64GB.

I also take a peek at the old system's /etc/doas.conf and enter the same content on the new system to get the same path to higher privilege that I'm used to having. With those in hand, recreating the set of installed packages on the fresh system is then a matter of a single command:

$ doas pkg_add -l installed_packages

and pkg_add(1) proceeds to fetch and install the same packages I had on the old system.

Then there is the matter of transferring the I-refuse-to-admit-the-actual-number-of gigabytes that make up the content of my home directory. In many environments it would make sense to just restore from the most recent backup, but in my case where the source and destination sit side by side, i chose to go with a simple rsync transfer:

$ rsync  -rcpPCavu 192.168.103.69:/home/peter . | tee -a 20170710-transferlog.txt

(Yes, I'm aware that I could have done something similar with nc and tar, which are both in the base system. But rsync wins by being more easily resumable.)

While the data transfers, there is ample time to check for parts of the old system's configuration that should be transferred to the new one. Setting up the hostname.iwm0 file to hold the config for the wireless networks (see the hostname.if man page) by essentially copying across the previous one is an obvious thing, and this is the time when you discover tweaks you love that were not part of that package's default configuration.

Some time went by while the content transferred, and I can now announce that I'm typing away on the machine that is at the same time both the most lightweight and the most powerful machine I have ever owned.

I am slowly checking and finding that the stuff I care about just works, though I haven't bothered to check whether the webcam works yet. I know you've been dying to see the dmesg, which can be found here. I'm sure I'll get to the bottom of the 'not configured' items (yes, there are some) fairly soon. Look for updates that will be added to the end of this column.

And after all these years, I finally have a machine that matches my beard color:



If you have any questions on running OpenBSD as a primary working environment, I'm generally happy to answer but in almost all cases I would prefer that you use the mailing lists such as misc@openbsd.org or the OpenBSD Facebook group so the question and hopefully useful answers become available to the general public. Browsing the slides for my recent OpenBSD and you user group talk might be beneficial if you're not yet familiar with the system. And of course, comments on this article are welcome.


Update 2017-07-18: One useful thing to do once you have your system up and running is to submit your dmesg to the NYCBUG dmesg database. The one for the system described here is up as http://dmesgd.nycbug.org/index.cgi?do=view&id=3227

by Peter N. M. Hansteen (noreply@blogger.com) at July 10, 2017 07:53 PM

July 07, 2017

HolisticInfoSec.org

Toolsmith #126: Adversary hunting with SOF-ELK

As we celebrate Independence Day, I'm reminded that we honor what was, of course, an armed conflict. Today's realities, when we think about conflict, are quite different than the days of lining troops up across the field from each other, loading muskets, and flinging balls of lead into the fray.
We live in a world of asymmetrical battles, often conflicts that aren't always obvious in purpose and intent, and likely fought on multiple fronts. For one of the best reads on the topic, take the well spent time to read TJ O'Connor's The Jester Dynamic: A Lesson in Asymmetric Unmanaged Cyber Warfare. If you're reading this post, it's highly likely that your front is that of 1s and 0s, either as a blue team defender, or as a red team attacker. I live in this world every day of my life as a blue teamer at Microsoft, and as a joint forces cyber network operator. We are faced, each day, with overwhelming, excessive amounts of data, of varying quality, where the answers to questions are likely hidden, but available to those who can dig deeply enough.
New platforms continue to emerge to help us in this cause. At Microsoft we have a variety of platforms that make the process easier for us, but no less arduous, to dig through the data, and the commercial sector continues to expand its offerings. For those with limited budgets and resources, but a strong drive for discovery, that have been outstanding offerings as well. Security Onion has been forefront for years, and is under constant development and improvement in the care of Doug Burks.
Another emerging platform, to be discussed here, is SOF-ELK, part of the SANS Forensics community, created by SANS FOR572, Advanced Network Forensics and Analysis author and instructor Phil Hagen. Count SOF-ELK in the NFAT family for sure, a strong player in the Network Forensic Analysis Tool category.
SOF-ELK has a great README, don't be that person, read it. It's everything you need to get started, in one place. What!? :-)
Better yet, you can download a fully realized VM with almost no configuration requirements, so you can hit the ground running. I ran my SOF-ELK instance with VMWare Workstation 12 Pro and no issues other than needing to temporarily disable Device Guard and Credential Guard on Windows 10.
SOF-ELK offers some good test data to get you started with right out of the gate, in /home/elk_user/exercise_source_logs, including Syslog from a firewall, router, converted Windows events, a Squid proxy, and a server named muse. You can drop these on your SOF-ELK server in the /logstash/syslog/ ingestion point for syslog-formatted data. Additionally, utilize /logstash/nfarch/ for archived NetFlow output, /logstash/httpd/ for Apache logs, /logstash/passivedns/ for logs from the passivedns utility, /logstash/plaso/ for log2timeline, and  /logstash/bro/ for, yeah, you guessed it.
I mixed things up a bit and added my own Apache logs for the month of May to /logstash/httpd/. The muse log set in the exercise offering also included a DNS log (named_log), for grins I threw that in the /logstash/syslog/ as well just to see how it would play.
Run down a few data rabbit holes with me, I swear I can linger for hours on end once I latch on to something to chase. We'll begin with a couple of highlights from my Apache logs. The SOF-ELK VM comes with three pre-configured dashboards including Syslog, NetFlow, and HTTPD. You can learn more in the start page for the SOF-ELK UI, my instance is http://192.168.50.110:5601/app/kibana. There are three panels, or blocks, for each dashboard's details, at the bottom of the UI. I drilled through to the HTTPD Log Dashboard for this experiment, and immediately reset the time period for analysis (click the time marker in the upper right hand part of the UI). It defaults to the last 15 minutes, if you're reviewing older data it won't show until you adjust to match your time stamps. My data is from the month of May so I selected an absolute window from the beginning of May to its end. You can also select quick or relative time options, it's great to get comfortable here quickly and early. The resulting opening visualizations for me made me very happy, as seen in Figure 1.
Figure 1: HTTPD Log Dashboard
Nice! An event count summary, source ASNs by count (you can immediately see where I scanned myself from work), a fantastic Access Source map, a records graph by HTTP verbs, and one by response codes.
The beauty of these SOF-ELK dashboards is that they're immediately interactive and allow you to drill right in to interesting data points. The holisticinfosec.org website is intentionally flat and includes no active PHP or dynamic content. As a result, my favorite response code as a web application security tester, the 500 error, is notably missing. But, in both the timeline graphs we note a big traffic spike on 8 MAY 2017, which correlates nicely with my above mention scan from work, as noted in the ASN hit count, and seen here in Figure 2.

Figure 2: Traffic spike from scan
This visualizes well but isn't really all that interesting or uncommon, particularly given that I know I personally ran the scan, and scans from the Intarwebs are dime a dozen. What did jump out for me though, as seen back in Figure 1, was the presence of four PUT requests. That's usually a "bad thing" where some @$$h@t is trying to drop something on my server. Let's drill in a bit, shall we? After clicking the graph line with the four PUT requests, I quickly learned that two requests came from 204.12.194.234 AS32097: WholeSale Internet in Kansas City, MO and two came from 119.23.233.9 AS37963: Hangzhou Alibaba Advertising in Hangzhou, China. This is well represented in the HTTPD Access Source panel map (Figure 3).

Figure 3: Access Source
The PUT request from each included a txt file attempt, specifically dbhvf99151.txt and htjfx99555.txt, both were rejected, redirected (302), and sent to my landing page (200).
Research on the IPs found that 119.23.233.9 was on the "real time suspected malware list as detected by InterServer's intrusion systems" as seen 22 MAY, and 204.12.194.234 was found twice in the AbuseIPDB, flagged on 18 MAY 2017 for Cknife Webshell Detected. Now we're talking. It's common to attempt a remote file include attack or a PUT, with what is a web shell. I opened up SOF-ELK on that IP address and found eight total hits in my logs, all looking for common PHP opportunities with the likes of GET and POST for /plus/mytag_js.php, noted in PHP injection attack attempts.
SOF-ELK made it incredibly easy to hunt down these details, as seen in Figure 4 from the HTTPD Discovery panel.
Figure 4: Discovery
That's a groovy little hunting trip through HTTPD logs, but how about a bit of Syslog? I spotted I likely oddity that could be correlated across a number of the exercise logs, we'll see if the correlation is real. You'll notice tabs at the top of your SOF-ELK UI, we'll use Discover for this experiment. I started from the Syslog Dashboard with my time range set broadly on the last two months. 7606 records presented themselves, sliced neatly by hosts and programs, as seen in Figure 5.

Figure 5: Syslog Dashboard
Squid proxy logs showed the predominance of host entries (6778 or 57.95% of 11,696 to be specific), so I started there. Don' laugh, but I'll often do keyword queries just to see what comes up, sometimes you land a pointer to a good rabbit hole. Within the body of 6778 proxy events, I searched malware. Two hits came back for GET request via a JS redirector to bleepingcomputer.com for your basic how-to based on "random websites opening in Chrome". Ruh-roh.
Figure 6: Malware keyword
More importantly, we have an IP address to pivot on: 10.3.59.53. A search of that IP across the same 6778 Squid logs yielded 3896 entries specific to this IP, and lots to be curious about:
  • datingukrainewomen.com 
  • anastasiadate.com
  • YouTube videos for hair loss
  • crowdscience.com for "random pop-ups driving me nuts"
Do I need to build this user profile out for you, or are you with me? Proxy logs tell us so much, and are deeply worthy of your blue team efforts to collect and review.
I jumped over to the named_log from the muse host to see what else might reveal itself. Here's where I jumped to Discover, the Splunk-like query functionality inherent to SOF-ELK (and ELK implemetations). I did reductive query to see what other oddities might surface: 10.3.59.53 AND dns_query: (*.co.uk OR *.de OR *.eu OR *.info OR *.cc OR *.online OR *.website). I used these TLDs based on the premise that bots using Domain Generation Algorithms (DGA) will often use the TLDs. See The DGA of PadCrypt to learn more, as well as ISC Diary handler John Bambanek's OSINT logic. The query results were quite satisfying, 29 hits, including a number of clearly randomly generated domains. Those that were most interesting all included the .cc TLD, so I zoomed in further. Down to five hits with 10.3.59.53 AND dns_query: *.cc, as seen in Figure 7.
Figure 7:. CC TLD hits
Oh man, not good. I had a hunch now, and went back to the proxy logs with 10.3.59.53 AND squid_request:*.exe. And there you have it, ladies and gentlemen, hunch rewarded (Figure 8).

Figure 8: taxdocs.exe
It taxdocs.exe isn't malware, I'm a monkey's uncle. Unfortunately, I could find no online references to these .cc domains or the .exe sample or URL, but you get the point. Given that it's exercise data, Phil may have generated it to entice to dig deeper.
When we think about the IOC patterns for Petya, a hunt like this is pretty revealing. Petya's "initial infection appears to involve a software supply-chain threat involving the Ukrainian company M.E.Doc, which develops tax accounting software, MEDoc". This is not Petya (as far as I know) specifically but we see pattern similarities for sure, one can learn a great deal about the sheep and the wolves. Be the sheepdog!
Few tools better in the free and open source arsenal to help you train and enhance your inner digital sheepdog than SOF-ELK. "I'm a sheepdog. I live to protect the flock and confront the wolf." ~ LTC Dave Grossman, from On Combat.

Believe it or not, there's a ton more you can do with SOF-ELK, consider this a primer and a motivator.
I LOVE SOF-ELK. Phil, well done, thank you. Readers rejoice, this is really one of my favorites for toolsmith, hands down, out of the now 126 unique tools discussed over more than ten years. Download the VM, and get to work herding. :-)
Cheers...until next time.

by Russ McRee (noreply@blogger.com) at July 07, 2017 10:35 PM

Anton Chuvakin - Security Warrior

Monthly Blog Round-Up – June 2017

Here is my next monthly "Security Warrior" blog round-up of top 5 popular posts/topics this
month:
  1. Why No Open Source SIEM, EVER?” contains some of my SIEM thinking from 2009. Is it relevant now? You be the judge.  Succeeding with SIEM requires a lot of work, whether you paid for the software, or not. BTW, this post has an amazing “staying power” that is hard to explain – I suspect it has to do with people wanting “free stuff” and googling for “open source SIEM” … 
  2. Simple Log Review Checklist Released!” is often at the top of this list – this aging checklist is still a very useful tool for many people. “On Free Log Management Tools” (also aged a bit by now) is a companion to the checklist (updated version)
  3. “New SIEM Whitepaper on Use Cases In-Depth OUT!” (dated 2010) presents a whitepaper on select SIEM use cases described in depth with rules and reports [using now-defunct SIEM product]; also see this SIEM use case in depth and this for a more current list of popular SIEM use cases. Finally, see our 2016 research on developing security monitoring use cases here!
  4. This month, my classic PCI DSS Log Review series is extra popular! The series of 18 posts cover a comprehensive log review approach (OK for PCI DSS 3+ even though it predates it), useful for building log review processes and procedures, whether regulatory or not. It is also described in more detail in our Log Management book and mentioned in our PCI book (now in its 4th edition!) – note that this series is mentioned in some PCI Council materials. 
  5. “SIEM Resourcing or How Much the Friggin’ Thing Would REALLY Cost Me?” is a quick framework for assessing the SIEM project (well, a program, really) costs at an organization (a lot more details on this here in this paper).
In addition, I’d like to draw your attention to a few recent posts from my Gartner blog [which, BTW, now has more than 5X of the traffic of this blog]: 

Current research on vulnerability management:
Current research on cloud security monitoring:
Recent research on security analytics and UBA / UEBA:
Miscellaneous fun posts:

(see all my published Gartner research here)
Also see my past monthly and annual “Top Popular Blog Posts” – 2007, 2008, 2009, 2010, 2011, 2012, 2013, 2014, 2015, 2016.

Disclaimer: most content at SecurityWarrior blog was written before I joined Gartner on August 1, 2011 and is solely my personal view at the time of writing. For my current security blogging, go here.

Previous post in this endless series:

by Anton Chuvakin (anton@chuvakin.org) at July 07, 2017 06:35 PM

July 03, 2017

Vincent Bernat

Performance progression of IPv4 route lookup on Linux

TL;DR: Each of Linux 2.6.39, 3.6 and 4.0 brings notable performance improvements for the IPv4 route lookup process.


In a previous article, I explained how Linux implements an IPv4 routing table with compressed tries to offer excellent lookup times. The following graph shows the performance progression of Linux through history:

IPv4 route lookup performance

Two scenarios are tested:

  • 500,000 routes extracted from an Internet router (half of them are /24), and
  • 500,000 host routes (/32) tightly packed in 4 distinct subnets.

All kernels are compiled with GCC 4.9 (from Debian Jessie). This version is able to compile older kernels1 as well as current ones. The kernel configuration used is the default one with CONFIG_SMP and CONFIG_IP_MULTIPLE_TABLES options enabled (however, no IP rules are used). Some other unrelated options are enabled to be able to boot them in a virtual machine and run the benchmark.

The measurements are done in a virtual machine with one vCPU2. The host is an Intel Core i5-4670K and the CPU governor was set to “performance”. The benchmark is single-threaded. Implemented as a kernel module, it calls fib_lookup() with various destinations in 100,000 timed iterations and keeps the median. Timings of individual runs are computed from the TSC (and converted to nanoseconds by assuming a constant clock).

The following kernel versions bring a notable performance improvement:

  • In Linux 2.6.39, commit 3630b7c050d9, David Miller removes the hash-based routing table implementation to switch to the LPC-trie implementation (available since Linux 2.6.13 as a compile-time option). This brings a small regression for the scenario with many host routes but boosts the performance for the general case.

  • In Linux 3.0, commit 281dc5c5ec0f, the improvement is not related to the network subsystem. Linus Torvalds disables the compiler size-optimization from the default configuration. It was believed that optimizing for size would help keeping the instruction cache efficient. However, compilers generated under-performing code on x86 when this option was enabled.

  • In Linux 3.6, commit f4530fa574df, David Miller adds an optimization to not evaluate IP rules when they are left unconfigured. From this point, the use of the CONFIG_IP_MULTIPLE_TABLES option doesn’t impact the performances unless some IP rules are configured. This version also removes the route cache (commit 5e9965c15ba8). However, this has no effect on the benchmark as it directly calls fib_lookup() which doesn’t involve the cache.

  • In Linux 4.0, notably commit 9f9e636d4f89, Alexander Duyck adds several optimizations to the trie lookup algorithm. It really pays off!

  • In Linux 4.1, commit 0ddcf43d5d4a, Alexander Duyck collapses the local and main tables when no specific IP rules are configured. For non-local traffic, both those tables were looked up.


  1. Compiling old kernels with an updated userland may still require some small patches

  2. The kernels are compiled with the CONFIG_SMP option to use the hierarchical RCU and activate more of the same code paths as actual routers. However, progress on parallelism are left unnoticed. 

by Vincent Bernat at July 03, 2017 01:25 PM

July 02, 2017

Cryptography Engineering

Beyond public key encryption

One of the saddest and most fascinating things about applied cryptography is how little cryptography we actually use. This is not to say that cryptography isn’t widely used in industry — it is. Rather, what I mean is that cryptographic researchers have developed so many useful technologies, and yet industry on a day to day basis barely uses any of them. In fact, with a few minor exceptions, the vast majority of the cryptography we use was settled by the early-2000s.*

Most people don’t sweat this, but as a cryptographer who works on the boundary of research and deployed cryptography it makes me unhappy. So while I can’t solve the problem entirely, what I can do is talk about some of these newer technologies. And over the course of this summer that’s what I intend to do: talk. Specifically, in the next few weeks I’m going to write a series of posts that describe some of the advanced cryptography that we don’t generally see used.

Today I’m going to start with a very simple question: what lies beyond public key cryptography? Specifically, I’m going to talk about a handful of technologies that were developed in the past 20 years, each of which allows us to go beyond the traditional notion of public keys.

This is a wonky post, but it won’t be mathematically-intense. For actual definitions of the schemes, I’ll provide links to the original papers, and references to cover some of the background. The point here is to explain what these new schemes do — and how they can be useful in practice.

Identity Based Cryptography

In the mid-1980s, a cryptographer named Adi Shamir proposed a radical new idea. The idea, put simply, was to get rid of public keys.

To understand where Shamir was coming from, it helps to understand a bit about public key encryption. You see, prior to the invention of public key crypto, all cryptography involved secret keys. Dealing with such keys was a huge drag. Before you could communicate securely, you needed to exchange a secret with your partner. This process was fraught with difficulty and didn’t scale well.

Public key encryption (beginning with Diffie-Hellman and Shamir’s RSA cryptosystem) hugely revolutionized cryptography by dramatically simplifying this key distribution process. Rather than sharing secret keys, users could now transmit their public key to other parties. This public key allowed the recipient to encrypt to you (or verify your signature) but it could not be used to perform the corresponding decryption (or signature generation) operations. That part would be done with a secret key you kept to yourself.

While the use of public keys improved many aspects of using cryptography, it also gave rise to a set of new challenges. In practice, it turns out that having public keys is only half the battle — people still need to use distribute them securely.

For example, imagine that I want to send you a PGP-encrypted email. Before I can do this, I need to obtain a copy of your public key. How do I get this? Obviously we could meet in person and exchange that key on physical media — but nobody wants to do this. It would much more desirable to obtain your public key electronically. In practice this means either (1) we have to exchange public keys by email, or (2) I have to obtain your key from a third piece of infrastructure, such as a website or key server. And now we come to the  problem: if that email or key server is untrustworthy (or simply allows anyone to upload a key in your name), I might end up downloading a malicious party’s key by accident. When I send a message to “you”, I’d actually be encrypting it to Mallory.

Mallory.

Solving this problem — of exchanging public keys and verifying their provenance — has motivated a huge amount of practical cryptographic engineering, including the entire web PKI. In most cases, these systems work well. But Shamir wasn’t satisfied. What if, he asked, we could do it better? More specifically, he asked: could we replace those pesky public keys with something better?

Shamir’s idea was exciting. What he proposed was a new form of public key cryptography in which the user’s “public key” could simply be their identity. This identity could be a name (e.g., “Matt Green”) or something more precise like an email address. Actually, it didn’t realy matter. What did matter was that the public key would be some arbitrary string — and not a big meaningless jumble of characters like “7cN5K4pspQy3ExZV43F6pQ6nEKiQVg6sBkYPg1FG56Not”.

Of course, using an arbitrary string as a public key raises a big problem. Meaningful identities sound great — but I don’t own them. If my public key is “Matt Green”, how do I get the corresponding private key? And if I can get out that private key, what stops some other Matt Green from doing the same, and thus reading my messages? And ok, now that I think about this, what stops some random person who isn’t named Matt Green from obtaining it? Yikes. We’re headed straight into Zooko’s triangle.

Shamir’s idea thus requires a bit more finesse. Rather than expecting identities to be global, he proposed a special server called a “key generation authority” that would be responsible for generating the private keys. At setup time, this authority would generate a single master public key (MPK), which it would publish to the world. If you wanted to encrypt a message to “Matt Green” (or verify my signature), then you could do so using my identity and the single MPK of an authority we’d both agree to use. To decrypt that message (or sign one), I would have to visit the same key authority and ask for a copy of my secret key. The key authority would compute my key based on a master secret key (MSK), which it would keep very secret.

With all algorithms and players specified, whole system looks like this:

Overview of an Identity-Based Encryption (IBE) system. The Setup algorithm of the Key Generation Authority generates the master public key (MPK) and master secret key (MSK). The authority can use the Extract algorithm to derive the secret key corresponding to a specific ID. The encryptor (left) encrypts using only the identity and MPK. The recipient requests the secret key for her identity, and then uses it to decrypt. (Icons by Eugen Belyakoff)

This design has some important advantages — and more than a few obvious drawbacks. On the plus side, it removes the need for any key exchange at all with the person you’re sending the message to. Once you’ve chosen a master key authority (and downloaded its MPK), you can encrypt to anyone in the entire world. Even cooler: at the time you encrypt, your recipient doesn’t even need to have contacted the key authority yet. She can obtain her secret key after I send her a message.

Of course, this “feature” is also a bug. Because the key authority generates all the secret keys, it has an awful lot of power. A dishonest authority could easily generate your secret key and decrypt your messages. The polite way to say this is that standard IBE systems effectively “bake in” key escrow.**

Putting the “E” in IBE

All of these ideas and more were laid out by Shamir in his 1984 paper. There was just one small problem: Shamir could only figure out half the problem.

Specifically, Shamir’s proposed a scheme for identity-based signature (IBS) — a signature scheme where the public verification key is an identity, but the signing key is generated by the key authority. Try as he might, he could not find a solution to the problem of building identity-based encryption (IBE). This was left as an open problem.***

It would take more than 16 years before someone answered Shamir’s challenge. Surprisingly, when the answer came it came not once but three times.

The first, and probably most famous realization of IBE was developed by Dan Boneh and Matthew Franklin much later — in 2001. The timing of Boneh and Franklin’s discovery makes a great deal of sense. The Boneh-Franklin scheme relies fundamentally on elliptic curves that support an efficient “bilinear map” (or “pairing”).**** The algorithms needed to compute such pairings were not known when Shamir wrote his paper, and weren’t employed constructively — that is, as a useful thing rather than an attack — until about 2000. The same can be said about a second scheme called Sakai-Kasahara, which would be independently discovered around the same time.

(For a brief tutorial on the Boneh-Franklin IBE scheme, see this page.)

The third realization of IBE was not as efficient as the others, but was much more surprising. This scheme was developed by Clifford Cocks, a senior cryptologist at Britain’s GCHQ. It’s noteworthy for two reasons. First, Cocks’ IBE scheme does not require bilinear pairings at all — it is based in the much older RSA setting, which means in principle it spent all those years just waiting to be found. Second, Cocks himself had recently become known for something even more surprising: discovering the RSA cryptosystem, nearly five years before RSA did. To bookend that accomplishment with a second major advance in public key cryptography was a pretty impressive accomplishment.

In the years since 2001, a number of additional IBE constructions have been developed, using all sorts of cryptographic settings. Nonetheless, Boneh and Franklin’s early construction remains among the simplest and most efficient.

Even if you’re not interested in IBE for its own sake, it turns out that this primitive is really useful to cryptographers for many things beyond simple encryption. In fact, it’s more helpful to think of IBE as a way of “pluralizing” a single public/secret master keypair into billions of related keypairs. This makes it useful for applications as diverse as blocking chosen ciphertext attacks, forward-secure public key encryption, and short signature schemes.

Attribute Based Encryption

Of course, if you leave cryptographers alone with a tool like IBE, the first thing they’re going to do is find a way to make things more complicated improve on it.

One of the biggest such improvements is due to Sahai and Waters. It’s called Attribute-Based Encryption, or ABE.

The origin of this idea was not actually to encrypt with attributes. Instead Sahai and Waters were attempting to develop an Identity-Based encryption scheme that could encrypt using biometrics. To understand the problem, imagine I decide to use a biometric like your iris scan as the “identity” to encrypt you a ciphertext. Later on you’ll ask the authority for a decryption key that corresponds to your own iris scan — and if everything matches up and you’ll be able to decrypt.

The problem is that this will almost never work.

The issue here is that biometric readings (like iris scans or fingerprint templates) are inherently error-prone. This means every scan will typically be very close, but often there will be a few bits that disagree. With standard IBE

Tell me this isn’t giving you nightmares.

this is fatal: if the encryption identity differs from your key identity by even a single bit, decryption will not work. You’re out of luck.

Sahai and Waters decided that the solution to this problem was to develop a form of IBE with a “threshold gate”. In this setting, each bit of the identity is represented as a different “attribute”. Think of each of these as components you’d encrypt under — something like “bit 5 of your iris scan is a 1” and “bit 23 of your iris scan is a 0”. The encrypting party lists all of these bits and encrypts under each one. The decryption key generated by the authority embeds a similar list of bit values. The scheme is defined so that decryption will work if and only if the number of matching attributes (between your key and the ciphertext) exceeds some pre-defined threshold: e.g., any 2024 out of 2048 bits must be identical in order to decrypt.

The beautiful thing about this idea is not fuzzy IBE. It’s that once you have a threshold gate and a concept of “attributes”, you can more interesting things. The main observation is that a threshold gate can be used to implement the boolean AND and OR gates, like so:

Even better, you can stack these gates on top of one another to assign a fairly complex boolean formula — which will itself determine what conditions your ciphertext can be decrypted under. For example, switching to a more realistic set of attributes, you could encrypt a medical record so that either a pediatrician in a hospital could read it, or an insurance adjuster could. All you’d need is to make sure people received keys that correctly described their attributes (which are just arbitrary strings, like identities).

A simple “ciphertext policy”, in which the ciphertext can be decrypted if and only if a key matches an appropriate set of attributes. In this case, the key satisfies the formula and thus the ciphertext decrypts. The remaining key attributes are ignored.

The other direction can be implemented as well. It’s possible to encrypt a ciphertext under a long list of attributes, such as creation time, file name, and even GPS coordinates indicating where the file was created. You can then have the authority hand out keys that correspond to a very precise slice of your dataset — for example, “this key decrypts any radiology file encrypted in Chicago between November 3rd and December 12th that is tagged with ‘pediatrics’ or ‘oncology'”.

Functional Encryption

Once you have a related of primitives like IBE and ABE, the researchers’ instinct is to both extend and generalize. Why stop at simple boolean formulae? Can we make keys (or ciphertexts) that embed arbitrary computer programs? The answer, it turns out, is yes — though not terribly efficiently. A set of recent works show that it is possible to build ABE that works over arbitrary polynomial-size circuits, using various lattice-based assumptions. So there is certainly a lot of potential here.

This potential has inspired researchers to generalize all of the above ideas into a single class of encryption called “functional encryption“. Functional encryption is more conceptual than concrete — it’s just a way to look at all of these systems as instances of a specific class. The basic idea is to represent the decryption procedure as an algorithm that computes an arbitary function F over (1) the plaintext inside of a ciphertext, and (2) the data embedded in the key. This function has the following profile:

output = F(key data, plaintext data)

In this model IBE can be expressed by having the encryption algorithm encrypt (identity, plaintext) and defining the function F such that, if “key input == identity”, it outputs the plaintext, and outputs an empty string otherwise. Similarly, ABE can be expressed by a slightly more complex function. Following this paradigm, once can envision all sorts of interesting functions that might be computed by different functions and realized by future schemes.

But those will have to wait for another time. We’ve gone far enough for today.

So what’s the point of all this?

For me, the point is just to show that cryptography can do some pretty amazing things. We rarely see this on a day-to-day basis when it comes to industry and “applied” cryptography, but it’s all there waiting to be used.

Perhaps the perfect application is out there. Maybe you’ll find it.

Notes:

* An earlier version of this post said “mid-1990s”. In comments below, Tom Ristenpart takes issue with that and makes the excellent point that many important developments post-date that. So I’ve moved the date forward about five years, and I’m thinking about how to say this in a nicer way.

** There is also an intermediate form of encryption known as “certificateless encryption“. Proposed by Al-Riyami and Paterson, this idea uses a combination of standard public key encryption and IBE. The basic idea is to encrypt each message using both a traditional public key (generated by the recipient) and an IBE identity. The recipient must then obtain a copy of the secret key from the IBE authority to decrypt. The advantages here are twofold: (1) the IBE key authority can’t decrypt the message by itself, since it does not have the corresponding secret key, which solves the “escrow” problem. And (2) the sender does not need to verify that the public key really belongs to the sender (e.g., by checking a certificate), since the IBE portion prevents imposters from decrypting the resulting message. Unfortunately this system is more like traditional public key cryptography than IBE, and does not have the neat usability features of IBE.

*** A part of the challenge of developing IBE is the need to make a system that is secure against “collusion” between different key holders. For example, imagine a very simple system that has only 2-bit identities. This gives four possible identities: “00”, “01”, “10”, “11”. If I give you a key for the identity “01” and I give Bob a key for “10”, we need to ensure that you two cannot collude to produce a key for identities “00” and “11”. Many earlier proposed solutions have tried to solve this problem by gluing together standard public encryption keys in various ways (e.g., by having a separate public key for each bit of the identity and giving out the secret keys as a single “key”). However these systems tend to fail catastrophically when just a few users collude (or their keys are stolen). Solving the collusion problem is fundamentally what separates real IBE from its faux cousins.

**** A full description of Boneh and Franklin’s scheme can be found here, or in the original paper. Some code is here and here and here. I won’t spend more time on it, except to note that the scheme is very efficient. It was patented and implemented by Voltage Security, now part of HPE.


by Matthew Green at July 02, 2017 03:28 PM

June 21, 2017

Vincent Bernat

IPv4 route lookup on Linux

TL;DR: With its implementation of IPv4 routing tables using LPC-tries, Linux offers good lookup performance (50 ns for a full view) and low memory usage (64 MiB for a full view).


During the lifetime of an IPv4 datagram inside the Linux kernel, one important step is the route lookup for the destination address through the fib_lookup() function. From essential information about the datagram (source and destination IP addresses, interfaces, firewall mark, …), this function should quickly provide a decision. Some possible options are:

  • local delivery (RTN_LOCAL),
  • forwarding to a supplied next hop (RTN_UNICAST),
  • silent discard (RTN_BLACKHOLE).

Since 2.6.39, Linux stores routes into a compressed prefix tree (commit 3630b7c050d9). In the past, a route cache was maintained but it has been removed1 in Linux 3.6.

Route lookup in a trie

Looking up a route in a routing table is to find the most specific prefix matching the requested destination. Let’s assume the following routing table:

$ ip route show scope global table 100
default via 203.0.113.5 dev out2
192.0.2.0/25
        nexthop via 203.0.113.7  dev out3 weight 1
        nexthop via 203.0.113.9  dev out4 weight 1
192.0.2.47 via 203.0.113.3 dev out1
192.0.2.48 via 203.0.113.3 dev out1
192.0.2.49 via 203.0.113.3 dev out1
192.0.2.50 via 203.0.113.3 dev out1

Here are some examples of lookups and the associated results:

Destination IP Next hop
192.0.2.49 203.0.113.3 via out1
192.0.2.50 203.0.113.3 via out1
192.0.2.51 203.0.113.7 via out3 or 203.0.113.9 via out4 (ECMP)
192.0.2.200 203.0.113.5 via out2

A common structure for route lookup is the trie, a tree structure where each node has its parent as prefix.

Lookup with a simple trie

The following trie encodes the previous routing table:

Simple routing trie

For each node, the prefix is known by its path from the root node and the prefix length is the current depth.

A lookup in such a trie is quite simple: at each step, fetch the nth bit of the IP address, where n is the current depth. If it is 0, continue with the first child. Otherwise, continue with the second. If a child is missing, backtrack until a routing entry is found. For example, when looking for 192.0.2.50, we will find the result in the corresponding leaf (at depth 32). However for 192.0.2.51, we will reach 192.0.2.50/31 but there is no second child. Therefore, we backtrack until the 192.0.2.0/25 routing entry.

Adding and removing routes is quite easy. From a performance point of view, the lookup is done in constant time relative to the number of routes (due to maximum depth being capped to 32).

Quagga is an example of routing software still using this simple approach.

Lookup with a path-compressed trie

In the previous example, most nodes only have one child. This leads to a lot of unneeded bitwise comparisons and memory is also wasted on many nodes. To overcome this problem, we can use path compression: each node with only one child is removed (except if it also contains a routing entry). Each remaining node gets a new property telling how many input bits should be skipped. Such a trie is also known as a Patricia trie or a radix tree. Here is the path-compressed version of the previous trie:

Patricia trie

Since some bits have been ignored, on a match, a final check is executed to ensure all bits from the found entry are matching the input IP address. If not, we must act as if the entry wasn’t found (and backtrack to find a matching prefix). The following figure shows two IP addresses matching the same leaf:

Lookup in a Patricia trie

The reduction on the average depth of the tree compensates the necessity to handle those false positives. The insertion and deletion of a routing entry is still easy enough.

Many routing systems are using Patricia trees:

Lookup with a level-compressed trie

In addition to path compression, level compression2 detects parts of the trie that are densily populated and replace them with a single node and an associated vector of 2k children. This node will handle k input bits instead of just one. For example, here is a level-compressed version our previous trie:

Level-compressed trie

Such a trie is called LC-trie or LPC-trie and offers higher lookup performances compared to a radix tree.

An heuristic is used to decide how many bits a node should handle. On Linux, if the ratio of non-empty children to all children would be above 50% when the node handles an additional bit, the node gets this additional bit. On the other hand, if the current ratio is below 25%, the node loses the responsibility of one bit. Those values are not tunable.

Insertion and deletion becomes more complex but lookup times are also improved.

Implementation in Linux

The implementation for IPv4 in Linux exists since 2.6.13 (commit 19baf839ff4a) and is enabled by default since 2.6.39 (commit 3630b7c050d9).

Here is the representation of our example routing table in memory3:

Memory representation of a trie

There are several structures involved:

The trie can be retrieved through /proc/net/fib_trie:

$ cat /proc/net/fib_trie
Id 100:
  +-- 0.0.0.0/0 2 0 2
     |-- 0.0.0.0
        /0 universe UNICAST
     +-- 192.0.2.0/26 2 0 1
        |-- 192.0.2.0
           /25 universe UNICAST
        |-- 192.0.2.47
           /32 universe UNICAST
        +-- 192.0.2.48/30 2 0 1
           |-- 192.0.2.48
              /32 universe UNICAST
           |-- 192.0.2.49
              /32 universe UNICAST
           |-- 192.0.2.50
              /32 universe UNICAST
[...]

For internal nodes, the numbers after the prefix are:

  1. the number of bits handled by the node,
  2. the number of full children (they only handle one bit),
  3. the number of empty children.

Moreover, if the kernel was compiled with CONFIG_IP_FIB_TRIE_STATS, some interesting statistics are available in /proc/net/fib_triestat4:

$ cat /proc/net/fib_triestat
Basic info: size of leaf: 48 bytes, size of tnode: 40 bytes.
Id 100:
        Aver depth:     2.33
        Max depth:      3
        Leaves:         6
        Prefixes:       6
        Internal nodes: 3
          2: 3
        Pointers: 12
Null ptrs: 4
Total size: 1  kB
[...]

When a routing table is very dense, a node can handle many bits. For example, a densily populated routing table with 1 million entries packed in a /12 can have one internal node handling 20 bits. In this case, route lookup is essentially reduced to a lookup in a vector.

The following graph shows the number of internal nodes used relative to the number of routes for different scenarios (routes extracted from an Internet full view, /32 routes spreaded over 4 different subnets with various densities). When routes are densily packed, the number of internal nodes are quite limited.

Internal nodes and null pointers

Performance

So how performant is a route lookup? The maximum depth stays low (about 6 for a full view), so a lookup should be quite fast. With the help of a small kernel module, we can accurately benchmark5 the fib_lookup() function:

Maximum depth and lookup time

The lookup time is loosely tied to the maximum depth. When the routing table is densily populated, the maximum depth is low and the lookup times are fast.

When forwarding at 10 Gbps, the time budget for a packet would be about 50 ns. Since this is also the time needed for the route lookup alone in some cases, we wouldn’t be able to forward at line rate with only one core. Nonetheless, the results are pretty good and they are expected to scale linearly with the number of cores.

The measurements are done with a Linux kernel 4.11 from Debian unstable. I have gathered performance metrics accross kernel versions in “Performance progression of IPv4 route lookup on Linux”.

Another interesting figure is the time it takes to insert all those routes into the kernel. Linux is also quite efficient in this area since you can insert 2 million routes in less than 10 seconds:

Insertion time

Memory usage

The memory usage is available directly in /proc/net/fib_triestat. The statistic provided doesn’t account for the fib_info structures, but you should only have a handful of them (one for each possible next-hop). As you can see on the graph below, the memory use is linear with the number of routes inserted, whatever the shape of the routes is.

Memory usage

The results are quite good. With only 256 MiB, about 2 million routes can be stored!

Routing rules

Unless configured without CONFIG_IP_MULTIPLE_TABLES, Linux supports several routing tables and has a system of configurable rules to select the table to use. These rules can be configured with ip rule. By default, there are three of them:

$ ip rule show
0:      from all lookup local
32766:  from all lookup main
32767:  from all lookup default

Linux will first lookup for a match in the local table. If it doesn’t find one, it will lookup in the main table and at last resort, the default table.

Builtin tables

The local table contains routes for local delivery:

$ ip route show table local
broadcast 127.0.0.0 dev lo proto kernel scope link src 127.0.0.1
local 127.0.0.0/8 dev lo proto kernel scope host src 127.0.0.1
local 127.0.0.1 dev lo proto kernel scope host src 127.0.0.1
broadcast 127.255.255.255 dev lo proto kernel scope link src 127.0.0.1
broadcast 192.168.117.0 dev eno1 proto kernel scope link src 192.168.117.55
local 192.168.117.55 dev eno1 proto kernel scope host src 192.168.117.55
broadcast 192.168.117.63 dev eno1 proto kernel scope link src 192.168.117.55

This table is populated automatically by the kernel when addresses are configured. Let’s look at the three last lines. When the IP address 192.168.117.55 was configured on the eno1 interface, the kernel automatically added the appropriate routes:

  • a route for 192.168.117.55 for local unicast delivery to the IP address,
  • a route for 192.168.117.255 for broadcast delivery to the broadcast address,
  • a route for 192.168.117.0 for broadcast delivery to the network address.

When 127.0.0.1 was configured on the loopback interface, the same kind of routes were added to the local table. However, a loopback address receives a special treatment and the kernel also adds the whole subnet to the local table. As a result, you can ping any IP in 127.0.0.0/8:

$ ping -c1 127.42.42.42
PING 127.42.42.42 (127.42.42.42) 56(84) bytes of data.
64 bytes from 127.42.42.42: icmp_seq=1 ttl=64 time=0.039 ms

--- 127.42.42.42 ping statistics ---
1 packets transmitted, 1 received, 0% packet loss, time 0ms
rtt min/avg/max/mdev = 0.039/0.039/0.039/0.000 ms

The main table usually contains all the other routes:

$ ip route show table main
default via 192.168.117.1 dev eno1 proto static metric 100
192.168.117.0/26 dev eno1 proto kernel scope link src 192.168.117.55 metric 100

The default route has been configured by some DHCP daemon. The connected route (scope link) has been automatically added by the kernel (proto kernel) when configuring an IP address on the eno1 interface.

The default table is empty and has little use. It has been kept when the current incarnation of advanced routing has been introduced in Linux 2.1.68 after a first tentative using “classes” in Linux 2.1.156.

Performance

Since Linux 4.1 (commit 0ddcf43d5d4a), when the set of rules is left unmodified, the main and local tables are merged and the lookup is done with this single table (and the default table if not empty). Moreover, since Linux 3.0 (commit f4530fa574df), without specific rules, there is no performance hit when enabling the support for multiple routing tables. However, as soon as you add new rules, some CPU cycles will be spent for each datagram to evaluate them. Here is a couple of graphs demonstrating the impact of routing rules on lookup times:

Routing rules impact on performance

For some reason, the relation is linear when the number of rules is between 1 and 100 but the slope increases noticeably past this threshold. The second graph highlights the negative impact of the first rule (about 30 ns).

A common use of rules is to create virtual routers: interfaces are segregated into domains and when a datagram enters through an interface from domain A, it should use routing table A:

# ip rule add iif vlan457 table 10
# ip rule add iif vlan457 blackhole
# ip rule add iif vlan458 table 20
# ip rule add iif vlan458 blackhole

The blackhole rules may be removed if you are sure there is a default route in each routing table. For example, we add a blackhole default with a high metric to not override a regular default route:

# ip route add blackhole default metric 9999 table 10
# ip route add blackhole default metric 9999 table 20
# ip rule add iif vlan457 table 10
# ip rule add iif vlan458 table 20

To reduce the impact on performance when many interface-specific rules are used, interfaces can be attached to VRF instances and a single rule can be used to select the appropriate table:

# ip link add vrf-A type vrf table 10
# ip link set dev vrf-A up
# ip link add vrf-B type vrf table 20
# ip link set dev vrf-B up
# ip link set dev vlan457 master vrf-A
# ip link set dev vlan458 master vrf-B
# ip rule show
0:      from all lookup local
1000:   from all lookup [l3mdev-table]
32766:  from all lookup main
32767:  from all lookup default

The special l3mdev-table rule was automatically added when configuring the first VRF interface. This rule will select the routing table associated to the VRF owning the input (or output) interface.

VRF was introduced in Linux 4.3 (commit 193125dbd8eb), the performance was greatly enhanced in Linux 4.8 (commit 7889681f4a6c) and the special routing rule was also introduced in Linux 4.8 (commit 96c63fa7393d, commit 1aa6c4f6b8cd). You can find more details about it in the kernel documentation.

Conclusion

The takeaways from this article are:

  • route lookup times hardly increase with the number of routes,
  • densily packed /32 routes lead to amazingly fast route lookups,
  • memory use is low (128 MiB par million routes),
  • no optimization is done on routing rules.

  1. The routing cache was subject to reasonably easy to launch denial of service attacks. It was also believed to not be efficient for high volume sites like Google but I have first-hand experience it was not the case for moderately high volume sites. 

  2. IP-address lookup using LC-tries”, IEEE Journal on Selected Areas in Communications, 17(6):1083-1092, June 1999. 

  3. For internal nodes, the key_vector structure is embedded into a tnode structure. This structure contains information rarely used during lookup, notably the reference to the parent that is usually not needed for backtracking as Linux keeps the nearest candidate in a variable. 

  4. One leaf can contain several routes (struct fib_alias is a list). The number of “prefixes” can therefore be greater than the number of leaves. The system also keeps statistics about the distribution of the internal nodes relative to the number of bits they handle. In our example, all the three internal nodes are handling 2 bits. 

  5. The measurements are done in a virtual machine with one vCPU. The host is an Intel Core i5-4670K running at 3.7 GHz during the experiment (CPU governor was set to performance). The benchmark is single-threaded. It runs a warm-up phase, then executes about 100,000 timed iterations and keeps the median. Timings of individual runs are computed from the TSC

  6. Fun fact: the documentation of this first tentative of more flexible routing is still available in today’s kernel tree and explains the usage of the “default class”

by Vincent Bernat at June 21, 2017 09:19 AM

June 20, 2017

Sean's IT Blog

Introducing Horizon 7.2

What’s new in Horizon 7.2?  A lot, actually.  There are some big features that will impact customers greatly, some beta features are now general availability, and some scalability enhancements. 

Horizon 7 Helpdesk Tool

One of the big drawbacks of Horizon is that it doesn’t have a very good tool for Helpdesk staff to support the environment.  While Horizon Administrator has role-based access control to limit what non-administrators can access and change, it doesn’t provide a good tool to enable a helpdesk technician to look up what desktop a user is in, grab some key metrics about their session, and remote in to assist the user.

VMware has provided a fling that can do some of this – the Horizon Toolbox.  But flings aren’t officially supported and aren’t always updated to support the latest version. 

Horizon 7.2 addresses this issue with the addition of the Horizon Helpdesk tool.  Horizon Helpdesk is an HTML5-based web interface designed for level 1 and 2 support desk engineers.  Unlike Citrix Director, which offers similar features for XenApp and XenDesktop environments, Horizon Helpdesk is integrated into the Connection Servers and installed by default.

Horizon Helpdesk provides a tool for helpdesk technicians who are supporting users with Horizon virtual desktops and published applications.  They’re able to log into the web-based tool, search for specific users, and view their active sessions and desktop and application entitlements.  Technicians can view details about existing sessions, including various metrics from within the virtual desktops and use the tool to launch a remote assistance.

Skype for Business Support is Now GA

The Horizon Virtualization Pack for Skype for Business was released under Technical Preview with Horizon 7.1.  It’s now fully supported on Windows endpoints in Horizon 7.2, and a private beta will be available for Linux-based clients soon. 

The Virtualization Pack allows Horizon clients to offload some of the audio and video features of Skype for business to the endpoint, and it provides optimized communication paths directly between the endpoints for multimedia calls.  What exactly does that mean?  Well, prior to the virtualization pack, multimedia streams would need to be sent from the endpoint to the virtual desktop, processed inside the VM, and then sent to the recipient.  The virtualization pack offloads the multimedia processing to the endpoint, and all media is sent between endpoints using a separate point-to-point stream.  This happens outside of the normal display protocol virtual channels.

image

Skype for Business has a number of PBX features, but not all of these features are supported in the first production-ready version of this plugin.  The following features are not supported:

  • Multiparty Conferencing
  • E911 Location Services
  • Call to Response Group
  • Call Park and Call Pickup from Park
  • Call via X (Work/Home, Cell, etc)
  • Customized Ringtones
  • Meet Now Conferencing
  • Call Recording

Instant Clone Updates

A few new features have been added to Horizon Instant Clones.  These new features are the ability to reuse existing Active Directory Computer accounts, support for configuring SVGA settings – such as video memory, resolution and number of monitors supported –  on the virtual desktop parent, and support for placing Instant Clones on local datastores.

Scalability Updates

VMware has improved Horizon stability in each of the previous releases, usually by expanding the number of connections, Horizon Pods, and sites that are supported in a Cloud Pod Architecture.  This release is no exception.  CPA now supports up to 120,000 sessions across 12 Horizon Pods in five sites.  While the number of sites has not increased, the number of supported sessions has increased by 40,000 compared to Horizon 7.1.

This isn’t the only scalability enhancement in Horizon 7.2.  The largest enhancement is in the number of desktops that will be supported per vCenter Server.  Prior to this version of Horizon, only 2000 desktops of any kind were supported per vCenter.  This means that a Horizon Pod that supported that maximum number of sessions would require five vCenter servers.  Horizon 7.2 doubles the number of supported desktops per vCenter, meaning that each vCenter Server can now support up to 4000 desktops of any type. 

Other New Features

Some of the other new features in Horizon 7.2 are:

  • Support for deploying Full Clone desktops to DRS Storage Clusters
  • A Workspace One mode configure access policies around desktops and published applications
  • Client Drive Redirection and USB Redirection are now supported on Linux
  • The ability to rebuild full clone desktops and reuse existing machine accounts.

Key Takeaways

The major features of Horizon 7.2 are very attractive, and they address things that customers have been asking for a while.  If you’re planning a greenfield environment, or it’s been a while since you’ve upgraded, there are a number of compelling reasons to look at implementing this version.


by seanpmassey at June 20, 2017 05:17 PM

June 19, 2017

Sarah Allen

customer-driven development

Sometimes when I talk about customer use cases for software that I’m building, it confuses the people I work with. What does that have to do with engineering? Is my product manager out on leave and I’m standing in?

I’ve found that customer stories connect engineers to the problems we’re solving. When we’re designing a system, we ask ourselves “what if” questions all the time. We need to explore the bounds of the solutions we’re creating, understanding the edge cases. We consider the performance characteristics, scalability requirements and all sorts of other important technical details. Through all this we imagine how the system will interact. It is easier when the software has a human interface, when it interacts with regular people. It is harder, but just as important, when we write software that only interacts with other software systems.

Sometimes the software that we are building can seem quite unrelated from the human world, but it isn’t at all. We then need to understand the bigger system we’re building. At some point, the software has a real-world impact, and we need to understand that, or we can miss creating the positive effects we intend.

On many teams over many years, I’ve had the opportunity to work with other engineers who get this. When it works there’s a rhythm to it, a heartbeat that I stop hearing consciously because it is always there. We talk to customers early and often. We learn about their problems, the other technologies they use, and what is the stuff they understand that we don’t. We start describing our ideas for solutions in their own words. This is not marketing. This influences what we invent. Understanding how to communicate about the thing we’re building changes what we build. We imagine this code we will write, that calls some other code, which causes other software to do a thing, and through all of the systems and layers there is some macro effect, which is important and time critical. Our software may have the capability of doing a thousand things, but we choose the scenarios for performance testing, we decide what is most normal, most routine, and that thing needs to be tied directly to those real effects.

Sometimes we refer to “domain knowledge” if our customers have special expertise, and we need to know that stuff, at least a little bit, so we can relate to our customers (and it’s usually pretty fun to explore someone else’s world a bit). However, the most important thing our customers know, that we need to discover, is what will solve their problems. They don’t know it in their minds — what they describe that they think will solve their problems often doesn’t actually. They know it when they feel it, when they somehow interact with our software and it gives them agency and amplifies their effect in the world.

Our software works when it empowers people to solve their problems and create impact. We can measure that. We can watch that happen. For those of us who have experienced that as software engineers, there’s no going back. We can’t write software any other way.

Customer stories, first hand knowledge from the people whose problems I’m solving spark my imagination, but I’m careful to use those stories with context from quantitative data. I love the product managers who tell me about rapidly expanding markets and how they relate to the use cases embedded in those stories, and research teams who validate whether the story I heard the other day is common across our customers or an edge case. We build software on a set of assumptions. Those assumptions need to be based on reality, and we need to validate early and often whether the thing we’re making is actually going to have a positive effect on reality.

Customer stories are like oxygen to a development team that works like this. Research and design teams who work closely with product managers are like water. When other engineers can’t explain the customer use cases for their software, when they argue about what the solution should be based only on the technical requirements, sometimes I can’t breathe. Then I find the people who are like me, who work like this, and I can hear a heartbeat, I can breathe again, and if feels like this thing we are making just might be awesome.

by sarah at June 19, 2017 01:08 PM

June 18, 2017

Debian Administration

Debian Stretch Released

Today the Debian project is pleased to announce the release of the next stable release of Debian GNU/Linux, code-named Stretch.

by Steve at June 18, 2017 03:44 AM

June 17, 2017

OpenSSL

Removing Some Code

This is another update on our effort to re-license the OpenSSL software. Our previous post in March was about the launch of our effort to reach all contributors, with the hope that they would support this change.

So far, about 40% of the people have responded. For a project that is as old as OpenSSL (including its predecessor, SSLeay, it’s around 20 years) that’s not bad. We’ll be continuing our efforts over the next couple of months to contact everyone.

Of those people responding, the vast majority have been in favor of the license change – less then a dozen objected. This post describes what we’re doing about those and how we came to our conclusions. The goal is to be very transparent and open with our processes.

First, we want to mention that we respect their decision. While it is inconvenient to us, we acknowledge their rights to keep their code under the terms that they originally agreed to. We are asking permission to change the license terms, and it is fully within their rights to say no.

The license website is driven by scripts that are freely available in the license section of our tools repository on GitHub. When we started, we imported every single git commit, looked for anything that resembled an email address, and created tables for each commit, each user we found, and a log that connects users to commits. This did find false positives: sometimes email Message-ID’s showed up, and there were often mentions of folks just as a passing side-comment, or because they were in the context diff. (That script is also in the tools repository, of course.)

The user table also has columns to record approval or rejection of the license change, and comments. Most of the comments have been supportive, and (a bit surprisingly) only one or two were obscene.

The whattoremove script finds the users who refused, and all commits they were named in. We then examined each commit – there were 86 in all – and filtered out those that were cherry-picks to other releases. We are planning on only changing the license for the master branch, so the other releases branches aren’t applicable. There were also some obvious false positives. At the end of this step, we had 43 commits to review.

We then re-read every single commit, looking for what we could safely ignore as not relevant. We found the following:

  • Nine of them showed up only because the person was mentioned in a comment.
  • Sixteen of them were changes to the OpenBSD configuration data. The OpenSSL team had completely rewritten this, refactoring all BSD configurations into a common hierarchy, and the config stuff changed a great deal for 1.1.0.
  • Seven were not copyrightable as because they were minimal changes (e.g., fixing a typo in a comment).
  • One was a false positive.

This left us with 10 commits. Two of them them were about the CryptoDev engine. We are removing that engine, as can be seen in this PR, but we expect to have a replacement soon (for at least Linux and FreeBSD). As for the other commits, we are removing that code, as can be seen in this first PR. and this second PR. Fortunately, none of them are particularly complex.

Copyright, however, is a complex thing – at times it makes debugging look simple. If there is only one way to write something, then that content isn’t copyrightable. If the work is very small, such as a one-line patch, then it isn’t copyrightable. We aren’t lawyers, and have no wish to become such, but those viewpoints are valid. It is not official legal advice, however.

For the next step, we hope that we will be able to “put back” the code that we removed. We are looking to the community for help, and encourage you to create the appropriate GitHub pull requests. We ask the community to look at that second PR and provide fixes.

And finally, we ask everyone to look at the list of authors and see if their name, or the name of anyone they know, is listed. If so please email us.

June 17, 2017 12:00 PM

June 14, 2017

Colin Percival

Oil changes, safety recalls, and software patches

Every few months I get an email from my local mechanic reminding me that it's time to get my car's oil changed. I generally ignore these emails; it costs time and money to get this done (I'm sure I could do it myself, but the time it would cost is worth more than the money it would save) and I drive little enough — about 2000 km/year — that I'm not too worried about the consequences of going for a bit longer than nominally advised between oil changes. I do get oil changes done... but typically once every 8-12 months, rather than the recommended 4-6 months. From what I've seen, I don't think I'm alone in taking a somewhat lackadaisical approach to routine oil changes.

June 14, 2017 04:40 AM

June 13, 2017

pagetable

Building the Original Commodore 64 KERNAL Source

Many reverse-engineered versions of “KERNAL”, the C64′s ROM operating system exist, and some of them even in a form that can be built into the original binary. But how about building the original C64 KERNAL source with the original tools?

The Commodore engineer Dennis Jarvis rescued many disks with Commodore source, which he gave to Steve Gray for preservation. One of these disks, c64kernal.d64, contains the complete source of the original KERNAL version (901227-01), including all build tools. Let’s build it!

Building KERNAL

First, we need a Commodore PET. Any PET will do, as long as it has enough RAM. 32 KB should be fine. For a disk drive, a 2040 or 4040 is recommended if you want to create the cross-reference database (more RAM needed for many open files), otherwise a 2031 will do just fine. The VICE emulator will allow you to configure such a system.

First you should delete the existing cross-reference files on disk. With the c1541 command line tool that comes with VICE, this can be done like this:

c1541 c64kernal.d64 -delete xr* -validate

Now attach the disk image and either have the emulator autostart it, or

LOAD"ASX4.0",8
RUN

This will run the “PET RESIDENT ASSEMBLER”. Now answer the questions like this:

PET RESIDENT ASSEMBLER V102780
(C) 1980 BY COMMODORE BUSINESS MACHINES

OBJECT FILE (CR OR D:NAME): KERNAL.HEX
HARD COPY (CR/Y OR N)? N
CROSS REFERENCE (CR/NO OR Y)? N
SOURCE FILE NAME? KERNAL

The assembler will now do its work and create the file “KERNAL.HEX” on disk. This will take about 16 minutes, so you probably want to switch your emulator into “Warp Mode”. When it is done, quit the emulator to make sure the contents of the disk image are flushed.

Using c1541, we can extract the output of the assembler:

c1541 c64kernal.d64 -read kernal.hex

The file is in MOS Technology Hex format and can be converted using srecord:

tr '\r' '\n' < kernal.hex > kernal_lf.hex
srec_cat kernal_lf.hex -MOS_Technologies \
-offset -0xe000 \
-fill 0xaa 0x0000 0x1fff \
-o kernal.bin -Binary

This command makes sure to fill the gaps with a value of 0xAA like the original KERNAL ROM.

The resulting file is identical with 901227-01 except for the following:

  • $E000-$E4AB is empty instead of containing the overflow BASIC code
  • $E4AC does not contain the checksum (the BASIC program “chksum” on the disk is used to calculate it)
  • $FFF6 does not contain the “RRBY” signature

Also note that $FF80, the KERNAL version byte is $AA – the first version of KERNAL did not yet use this location for the version byte, so the effective version byte is the fill byte.

Browsing the Source

An ASCII/LF version of the source code is available at https://github.com/mist64/cbmsrc.

;****************************************
;*                                      *
;* KK  K EEEEE RRRR  NN  N  AAA  LL     *
;* KK KK EE    RR  R NNN N AA  A LL     *
;* KKK   EE    RR  R NNN N AA  A LL     *
;* KKK   EEEE  RRRR  NNNNN AAAAA LL     *
;* KK K  EE    RR  R NN NN AA  A LL     *
;* KK KK EE    RR  R NN NN AA  A LL     *
;* KK KK EEEEE RR  R NN NN AA  A LLLLL  *
;*                                      *
;***************************************
;
;***************************************
;* PET KERNAL                          *
;*   MEMORY AND I/O DEPENDENT ROUTINES *
;* DRIVING THE HARDWARE OF THE         *
;* FOLLOWING CBM MODELS:               *
;*   COMMODORE 64 OR MODIFED VIC-40    *
;* COPYRIGHT (C) 1982 BY               *
;* COMMODORE BUSINESS MACHINES (CBM)   *
;***************************************
.SKI 3
;****LISTING DATE --1200 14 MAY 1982****
.SKI 3
;***************************************
;* THIS SOFTWARE IS FURNISHED FOR USE  *
;* USE IN THE VIC OR COMMODORE COMPUTER*
;* SERIES ONLY.                        *
;*                                     *
;* COPIES THEREOF MAY NOT BE PROVIDED  *
;* OR MADE AVAILABLE FOR USE ON ANY    *
;* OTHER SYSTEM.                       *
;*                                     *
;* THE INFORMATION IN THIS DOCUMENT IS *
;* SUBJECT TO CHANGE WITHOUT NOTICE.   *
;*                                     *
;* NO RESPONSIBILITY IS ASSUMED FOR    *
;* RELIABILITY OF THIS SOFTWARE. RSR   *
;*                                     *
;***************************************
.END

by Michael Steil at June 13, 2017 07:44 AM

June 01, 2017

Anton Chuvakin - Security Warrior

Monthly Blog Round-Up – May 2017

Here is my next monthly "Security Warrior" blog round-up of top 5 popular posts/topics this
month:
  1. “New SIEM Whitepaper on Use Cases In-Depth OUT!” (dated 2010) presents a whitepaper on select SIEM use cases described in depth with rules and reports [using now-defunct SIEM product]; also see this SIEM use case in depth and this for a more current list of popular SIEM use cases. Finally, see our 2016 research on developing security monitoring use cases here!
  2. Simple Log Review Checklist Released!” is often at the top of this list – this aging checklist is still a very useful tool for many people. “On Free Log Management Tools” (also aged a bit by now) is a companion to the checklist (updated version)
  3. Why No Open Source SIEM, EVER?” contains some of my SIEM thinking from 2009. Is it relevant now? You be the judge.  Succeeding with SIEM requires a lot of work, whether you paid for the software, or not. BTW, this post has an amazing “staying power” that is hard to explain – I suspect it has to do with people wanting “free stuff” and googling for “open source SIEM” … 
  4. This month, my classic PCI DSS Log Review series is extra popular! The series of 18 posts cover a comprehensive log review approach (OK for PCI DSS 3+ even though it predates it), useful for building log review processes and procedures, whether regulatory or not. It is also described in more detail in our Log Management book and mentioned in our PCI book (now in its 4th edition!) – note that this series is mentioned in some PCI Council materials. 
  5. “SIEM Resourcing or How Much the Friggin’ Thing Would REALLY Cost Me?” is a quick framework for assessing the SIEM project (well, a program, really) costs at an organization (a lot more details on this here in this paper).
In addition, I’d like to draw your attention to a few recent posts from my Gartner blog [which, BTW, now has more than 5X of the traffic of this blog]: 

EPIC WIN post:
Current research on vulnerability management:
Current research on cloud security monitoring:
Recent research on security analytics and UBA / UEBA:
Miscellaneous fun posts:

(see all my published Gartner research here)
Also see my past monthly and annual “Top Popular Blog Posts” – 2007, 2008, 2009, 2010, 2011, 2012, 2013, 2014, 2015, 2016.

Disclaimer: most content at SecurityWarrior blog was written before I joined Gartner on August 1, 2011 and is solely my personal view at the time of writing. For my current security blogging, go here.

Previous post in this endless series:

by Anton Chuvakin (anton@chuvakin.org) at June 01, 2017 04:40 PM

May 25, 2017

Raymii.org

Openstack Horizon, remove the loading modal with uBlock Origin

The OpenStack dashboard, Horizon, is a great piece of software to manage your OpenStack resources via the web. However, it has, in my opinion, a very big usability issue. The loading dialog that appears after you click a link. It blocks the entire page and all other links. So, whenever I click, I have to wait three to five seconds before I can do anything else. Clicked the wrong menu item? Sucks to be you, here have some loading. Clicked a link and quickly want to open something in a new tab while the page is still loading? Nope, not today. It's not that browsers have had a function to show that a page is loading, no, of course, the loading indication that has been there forever is not good enough. Let's re-invent the wheel and significantly impact the user experience. With two rules in uBlock Origin this loading modal is removed and you can work normally again in Horizon

May 25, 2017 12:00 AM

May 22, 2017

HolisticInfoSec.org

Toolsmith #125: ZAPR - OWASP ZAP API R Interface

It is my sincere hope that when I say OWASP Zed Attack Proxy (ZAP), you say "Hell, yeah!" rather than "What's that?". This publication has been a longtime supporter, and so many brilliant contibutors and practitioners have lent to OWASP ZAPs growth, in addition to @psiinon's extraordinary project leadership. OWASP ZAP has been 1st or 2nd in the last four years of @ToolsWatch best tool survey's for a damned good reason. OWASP ZAP usage has been well documented and presented over the years, and the wiki gives you tons to consider as you explore OWASP ZAP user scenarios.
One of the more recent scenarios I've sought to explore recently is use of the OWASP ZAP API. The OWASP ZAP API is also well documented, more than enough detail to get you started, but consider a few use case scenarios.
First, there is a functional, clean OWASP ZAP API UI, that gives you a viewer's perspective as you contemplate programmatic opportunities. OWASP ZAP API interaction is URL based, and you can invoke both access views and invoke actions. Explore any component and you'll immediately find related views or actions. Drilling into to core via http://localhost:8067/UI/core/ (I run OWASP ZAP on 8067, your install will likely be different), gives me a ton to choose from.
You'll need your API key in order to build queries. You can find yours via Tools | Options | API | API Key. As an example, drill into numberOfAlerts (baseurl ), which gets the number of alerts, optionally filtering by URL. You'll then be presented with the query builder, where you can enter you key, define specific parameter, and decide your preferred output format including JSON, HTML, and XML.
Sure, you'll receive results in your browser, this query will provide answers in HTML tables, but these aren't necessarily optimal for programmatic data consumption and manipulation. That said, you learn the most important part of this lesson, a fully populated OWASP ZAP API GET URL: http://localhost:8067/HTML/core/view/numberOfAlerts/?zapapiformat=HTML&apikey=2v3tebdgojtcq3503kuoq2lq5g&formMethod=GET&baseurl=.
This request would return




in HTML. Very straightforward and easy to modify per your preferences, but HTML results aren't very machine friendly. Want JSON results instead? Just swap  out HTML with JSON in the URL, or just choose JSON in the builder. I'll tell you than I prefer working with JSON when I use the OWASP ZAP API via the likes of R. It's certainly the cleanest, machine-consumable option, though others may argue with me in favor of XML.
Allow me to provide you an example with which you can experiment, one I'll likely continue to develop against as it's kind of cool for active reporting on OWASP ZAP scans in flight or on results when session complete. Note, all my code, crappy as it may be, is available for you on GitHub. I mean to say, this is really v0.1 stuff, so contribute and debug as you see fit. It's also important to note that OWASP ZAP needs to be running, either with an active scanning session, or a stored session you saved earlier. I scanned my website, holisticinfosec.org, and saved the session for regular use as I wrote this. You can even see reference to the saved session by location below.
R users are likely aware of Shiny, a web application framework for R, and its dashboard capabilities. I also discovered that rCharts are designed to work interactively and beautifully within Shiny.
R includes packages that make parsing from JSON rather straightforward, as I learned from Zev Ross. RJSONIO makes it as easy as fromJSON("http://localhost:8067/JSON/core/view/alerts/?zapapiformat=JSON&apikey=2v3tebdgojtcq3503kuoq2lq5g&formMethod=GET&baseurl=&start=&count=")
to pull data from the OWASP ZAP API. We use the fromJSON "function and its methods to read content in JSON format and de-serializes it into R objects", where the ZAP API URL is that content.
I further parsed alert data using Zev's grabInfo function and organized the results into a data frame (ZapDataDF). I then further sorted the alert content from ZapDataDF into objects useful for reporting and visualization. Within each alert objects are values such as the risk level, the alert message, the CWE ID, the WASC ID, and the Plugin ID. Defining each of these values into parameter useful to R is completed with the likes of:
I then combined all those results into another data frame I called reportDF, the results of which are seen in the figure below.
reportDF results
Now we've got some content we can pivot on.
First, let's summarize the findings and present them in their resplendent glory via ZAPR: OWASP ZAP API R Interface.
Code first, truly simple stuff it is:
Summary overview API calls

You can see that we're simply using RJSONIO's fromJSON to make specific ZAP API call. The results are quite tidy, as seen below.
ZAPR Overview
One of my favorite features in Shiny is the renderDataTable function. When utilized in a Shiny dashboard, it makes filtering results a breeze, and thus is utilized as the first real feature in ZAPR. The code is tedious, review or play with it from GitHub, but the results should speak for themselves. I filtered the view by CWE ID 89, which in this case is a bit of a false positive, I have a very flat web site, no database, thank you very much. Nonetheless, good to have an example of what would definitely be a high risk finding.


Alert filtering

Alert filtering is nice, I'll add more results capabilities as I develop this further, but visualizations are important too. This is where rCharts really come to bear in Shiny as they are interactive. I've used the simplest examples, but you'll get the point. First, a few, wee lines of R as seen below.
Chart code
The results are much more satisfying to look at, and allow interactivity. Ramnath Vaidyanathan has done really nice work here. First, OWASP ZAP alerts pulled via the API are counted by risk in a bar chart.
Alert counts

As I moused over Medium, we can see that there were specifically 17 results from my OWASP ZAP scan of holisticinfosec.org.
Our second visualization are the CWE ID results by count, in an oft disdained but interactive pie chart (yes, I have some work to do on layout).


CWE IDs by count

As we learned earlier, I only had one CWE ID 89 hit during the session, and the visualization supports what we saw in the data table.
The possibilities are endless to pull data from the OWASP ZAP API and incorporate the results into any number of applications or report scenarios. I have a feeling there is a rich opportunity here with PowerBI, which I intend to explore. All the code is here, along with the OWASP ZAP session I refer to, so you can play with it for yourself. You'll need OWASP ZAP, R, and RStudio to make it all work together, let me know if you have questions or suggestions.
Cheers, until next time.

by Russ McRee (noreply@blogger.com) at May 22, 2017 06:35 AM

May 15, 2017

Sean's IT Blog

Integrating Duo MFA and Amazon WorkSpaces

I recently did some work integrating Duo MFA with Amazon WorkSpaces.  The integration work ran into a few challenges, and I wanted to blog about those challenges to help others in the future.

Understanding Amazon WorkSpaces

If you couldn’t tell by the name, Amazon WorkSpaces is a cloud-hosted Desktop-as-a-Service offering that runs on Amazon Web Services (AWS).  It utilizes AWS’s underlying infrastructure to deploy desktop workloads, either using licensing provided by Amazon or the customer.  The benefit of this over on-premises VDI solutions is that Amazon manages the management infrastructure.  Amazon also provides multiple methods to integrate with the customer’s Active Directory environment, so users can continue to use their existing credentials to access the deployed desktops.

WorkSpaces uses an implementation of the PCoIP protocol for accessing desktops from the client application, and the client application is available on Windows, Linux, MacOS, iOS, and Android.  Amazon also has a web-based client that allows users to access their desktop from a web browser.  This feature is not enabled by default.

Anyway, that’s a very high-level overview of WorkSpaces.

Understanding How Users Connect to WorkSpaces

Before we can talk about how to integrate any multi-factor authentication solution into WorkSpaces, let’s go through the connection path and how that impacts how MFA is used.  WorkSpaces differs from traditional on-premises VDI in that all user connections to the desktop go through a public-facing service.  This happens even if I have a VPN tunnel or use DirectConnect.

The following image illustrates the connection path between the users and the desktop when a VPN connection is in place between the on-premises environment and AWS:

ad_connector_architecture_vpn

Image courtesy of AWS, retrieved from http://docs.aws.amazon.com/workspaces/latest/adminguide/prep_connect.html

As you can see from the diagram, on-premises users don’t connect to the WorkSpaces over the VPN tunnel – they do it over the Internet.  They are coming into their desktop from an untrusted connection, even if the endpoint they are connecting from is on a trusted network.

Note: The page that I retrieved this diagram from shows users connecting to WorkSpaces over a DirectConnect link when one is present.  This diagram shows a DirectConnect link that has a public endpoint, and all WorkSpaces connections are routed through this public endpoint.  In this configuration, logins are treated the same as logins coming in over the Internet.

When multi-factor authentication is configured for WorkSpaces, it is required for all user logins.  There is no way to configure rules to apply MFA to users connecting from untrusted networks because WorkSpaces sees all connections coming in on a public, Internet-facing service.

WorkSpaces MFA Requirements

WorkSpaces only works with MFA providers that support RADIUS.  The RADIUS servers can be located on-premises or in AWS.  I recommend placing the RADIUS proxies in another VPC in AWS where other supporting infrastructure resources, like AD Domain Controllers, are located.  These servers can be placed in the WorkSpaces VPC, but I wouldn’t recommend it.

WorkSpaces can use multiple RADIUS servers,and it will load-balance requests across all configured RADIUS servers.  RADIUS can be configured on both the Active Directory Connectors and the managed Microsoft AD directory services options. 

Note: The Duo documentation states that RADIUS can only be configured on the Active Directory Connector. Testing shows that RADIUS works on both enterprise WorkSpaces Directory Services options.

Configuring Duo

Before we can enable MFA in WorkSpaces, Duo needs to be configured.  At least one Duo proxy needs to be reachable from the WorkSpaces VPC.  This could be on a Windows or Linux EC2 instance in the WorkSpaces VPC, but it will most likely be an EC2 instance in a shared services or management VPC or in an on-premises datacenter.  The steps for installing the Duo Proxies are beyond the scope of this article, but they can be found in the Duo documentation.

Before the Duo Authentication Proxies are configured, we need to configure a new application in the Duo Admin console.  This step generates an integration key and secret key that will be used by the proxy and configure how new users will be handled.  The steps for creating the new application in Duo are:

1. Log into your Duo Admin Console at https://admin.duosecurity.com

2. Login as  your user and select the two-factor authentication method for your account.  Once you’ve completed the two-factor sign-on, you will have access to your Duo Admin panel.

3. Click on Applications in the upper left-hand corner.

4. Click on Protect an Application

5. Search for RADIUS.

6. Click Protect this Application  underneath the RADIUS application.

image

7. Record the Integration Key, Secret Key, and API Hostname.  These will be used when configuring the Duo Proxy in a few steps.

image

8. Provide a Name for the Application.  Users will see this name in their mobile device if they use Duo Push.

image

9. Set the New User Policy to Deny Access.  Deny Access is required because the WorkSpaces setup process will fail if Duo is configured to fail open or auto-enroll.

image

10. Click Save Changes to save the new application.

Get the AWS Directory Services IP Addresses

The RADIUS Servers need to know which endpoints to accept RADIUS requests from.  When WorkSpaces is configured for MFA, these requests come from the Directory Services instance.  This can either be the Active Directory Connectors or the Managed Active Directory domain controllers.

The IP addresses that will be required can be found in the AWS Console under WorkSpaces –> Directories.  The field to use depends on the directory service type you’re using.  If you’re using Active Directory Connectors, the IP addresses are in the Directory IP Address field.  If you’re using the managed Microsoft AD service, the Directory IP address field has a value of “None,” so you will need to use the  IP addresses in the DNS Address field.

Configure AWS Security Group Rules for RADIUS

By default, the AWS security group for the Directory Services instance heavily restrict the inbound and outbound traffic.  In order to enable RADIUS, you’ll need to add inbound and outbound UDP 1812 to the destination destination IPs or subnet where the proxies are located.

The steps for updating the AWS security group rules are:

1. Log into the AWS Console.

2. Click on the Services menu and select WorkSpaces.

3. Click on Directories.

4. Copy the value in the Directory ID field of the directory that will have RADIUS configured.

5. Click on the Services menu and select VPC.

6. Click on Security Groups.

7. Paste the Directory ID into the Search field to find the Security Group attached to the Directory Services instance.

8. Select the Security Group.

9. Click on the Inbound Rules tab.

10. Click Edit.

11. Click Add Another Rule.

12. Select Custom UDP Rule in the Type dropdown box.

13. Enter 1812 in the port range field.

14. Enter the IP Address of the RADIUS Server in the Source field.

15. Repeat Steps 11 through 14 as necessary to create inbound rules for each Duo Proxy.

16. Click Save.

17. Click on the Outbound Rules tab.

18. Click Edit.

19. Click Add Another Rule.

20. Select Custom UDP Rule in the Type dropdown box.

21. Enter 1812 in the port range field.

22. Enter the IP Address of the RADIUS Server in the Source field.

23. Repeat Steps 11 through 14 as necessary to create inbound rules for each Duo Proxy.

24. Click Save.

Configure the Duo Authentication Proxies

Once the Duo side is configured and the AWS security rules are set up, we need to configure our Authentication Proxy or Proxies.  This needs to be done locally on each Authentication Proxy.  The steps for configuring the Authentication Proxy are:

Note:  This step assumes the authentication proxy is running on Windows.The steps for installing the authentication proxy are beyond the scope of this article.  You can find the directions for installing the proxy at: https://duo.com/docs/

1. Log into your Authentication Proxy.

2. Open the authproxy.cfg file located in C:\Program Files (x86)\Duo Security Authentication Proxy\conf.

Note: The authproxy.cfg file uses the Unix newline setup, and it may require a different text editor than Notepad.  If you can install it on your server, I recommend Notepad++ or Sublime Text. 

3. Add the following lines to your config file.  This requires the Integration Key, Secret Key, and API Hostname from Duo and the IP Addresses of the WorkSpaces Active Directory Connectors or Managed Active Directory domain controllers.  Both RADIUS servers need to use the same RADIUS shared secret, and it should be a complex string.

[duo_only_client]

[radius_server_duo_only]
ikey=<Integration Key>
skey=<Secret Key>
api_host=<API Hostname>.duosecurity.com
failmode=safe
radius_ip_1=Directory Service IP 1
radius_secret_1=Secret Key
radius_ip_2=Directory Service IP 2
radius_secret_2=Secret Key
port=1812

4. Save the configuration file.

5. Restart the Duo Authentication Proxy Service to apply the new configuration.

Integrating WorkSpaces with Duo

Now that our base infrastructure is configured, it’s time to set up Multi-Factor Authentication in WorkSpaces.  MFA is configured on the Directory Services instance.  If multiple directory services instances are required to meet different WorkSpaces use cases, then this configuration will need to be performed on each directory service instance. 

The steps for enabling WorkSpaces with Duo are:

1. Log into the AWS Console.

2. Click on the Services menu and select Directory Service.

3. Click on the Directory ID of the directory where MFA will be enabled.

4. Click the Multi-Factor authentication tab.

5. Click the Enable Multi-Factor Authentication checkbox.

6. Enter the IP address or addresses of your Duo Authentication Proxies in the RADIUS Server IP Address(es) field.

7. Enter the RADIUS Shared Secret in the Shared Secret Code and Confirm Shared Secret Code fields.

8. Change the Protocol to PAP.

9. Change the Server Timeout to 20 seconds.

10. Change the Max Retries to 3.

11. Click Update Directory.

12. If RADIUS setup is successful, the RADIUS Status should say Completed.

image

Using WorkSpaces with Duo

Once RADIUS is configured, the WorkSpaces client login prompt will change.  Users will be prompted for an MFA code along with their username and password.  This can be the one-time password generated in the mobile app, a push, or a text message with a one-time password.  If the SMS option is used, the user’s login will fail, but they will receive a text message with a one-time password. They will need to log in again using one of the codes received in the text message.

image

Troubleshooting

If the RADIUS setup does not complete successfully, there are a few things that you can check.  The first step is to verify that port 1812 is open on the Duo Authentication Proxies.  Also verify that the security group the directory services instance is configured to allow port 1812 inbound and outbound. 

One issue that I ran into in previous setups was the New User Policy in the Duo settings.  If it is not set to Deny Access, the process setup process will fail.  WorkSpaces is expecting a response code of Access Reject.  If it receives anything else, the process stalls out and fails about 15 minutes later. 

Finally, review the Duo Authentication Proxy logs.  These can be found in C:\Program Files (x86)\Duo Security Authentication Proxy\log on Authentication Proxies running Windows.  A tool like WinTail can be useful for watching the logs in real-time when trying to troubleshoot issues or verify that the RADIUS Services are functioning correctly.


by seanpmassey at May 15, 2017 06:08 PM

May 14, 2017

SysAdmin1138

Estimates and compounding margin of error

The two sides of this story are:

  • Software requirements are often... mutable over time.
  • Developer work-estimates are kind of iffy.

Here is a true story about the later. Many have similar tales to tell, but this one is mind. It's about metrics, agile, and large organizations. If you're expecting this to turn into a frAgile post, you're not wrong. But it's about failure-modes.

The setup

My employer at the time decided to go all-in on agile/scrum. It took several months, but every department in the company was moved to it. As they were an analytics company, their first reflex was to try to capture as much workflow data as possible so they can do magic data-analysis things on it. Which meant purchasing seats in an Agile SaaS product so everyone could track their work in it.

The details

By fiat from on-high, story points were effectively 'days'.

Due to the size of the development organization, there were three levels of Product Managers over the Individual Contributors I counted myself a part of.

The apex Product Manager, we had two, were for the two products we sold.

Marketing was also in Agile, as was Sales.

The killer feature

Because we were an analytics company, the CEO wanted a "single pane of glass" to give a snapshot of how close we were to achieving our product goals. Gathering metrics on all of our sprint-velocities, story/epic completion percentages, and story estimates, allowed us to give him that pane of glass. It had:

  • Progress bars for how close our products were to their next major milestones.
  • How many sprints it will take to get there.
  • How many sprints it would take to get to the milestone beyond it.

Awesome!

The failure

That pane of glass was a lying piece of shit.

The dashboard we had to build was based on so many fuzzy measurements that it was a thumb in the wind approximation for how fast we were going, and in what direction. The human bias to trust numbers derived using Science! is a strong one, and they were inappropriately trusted. Which lead to pressure from On High for highly accurate estimates, as the various Product Managers realized what was going on and attempted to compensate (remove uncertainty from one of the biggest sources of it).

Anyone who has built software knows that problems come in three types:

  1. Stuff that was a lot easier than you thought
  2. Stuff that was pretty much as bad as you thought.
  3. Hell-projects of tar-pits, quicksand, ambush-yaks, and misery.

In the absence of outside pressures, story estimates usually are pitched at type 2 efforts; the honest estimate. Workplace culture introduces biases to this, urging devs to skew one way or the other. Skew 'easier', and you'll end up overshooting your estimates a lot. Skew 'harder' and your velocity will look great, but capacity planning will suffer.

This leads to an interesting back and forth! Dev-team skews harder for estimates. PM sees that team typically exceeds its capacity in productivity, so adds more capacity in later sprints. In theory equilibrium is reached between work-estimation and work-completion-rate. In reality, it means that the trustability of number is kind of low and always will be.

The irreducible complexity

See, the thing is, marketing and sales both need to know when a product will be released so they can kick off marketing campaigns and start warming up the sales funnel. Some kinds of ad-buys are done weeks or more in advance, so slipping product-ship at the last minute can throw off the whole marketing cadence. Trusting in (faulty) numbers means it may look like release will be in 8 weeks, so its safe to start baking that in.

Except those numbers aren't etched in stone. They're graven in the finest of morning dew.

As that 8 week number turns into 6, then 4, then 2, pressure to hit the mark increases. For a company selling on-prem software, you can afford to miss your delivery deadline so long as you have a hotfix/service-pack process in place to deliver stability updates quickly. You see this a lot with game-dev: the shipping installer is 8GB, but there are 2GB of day-1 patches to download before you can play. SaaS products need to work immediately on release, so all-nighters may become the norm for major features tied to marketing campaigns.

Better estimates would make this process a lot more trustable. But, there is little you can do to actually improve estimate quality.

by SysAdmin1138 at May 14, 2017 03:30 PM

May 11, 2017

Colin Percival

A plan for open source software maintainers

I've been writing open source software for about 15 years now; while I'm still wet behind the ears compared to FreeBSD greybeards like Kirk McKusick and Poul-Henning Kamp, I've been around for long enough to start noticing some patterns. In particular:
  • Free software is expensive. Software is expensive to begin with; but good quality open source software tends to be written by people who are recognized as experts in their fields (partly thanks to that very software) and can demand commensurate salaries.
  • While that expensive developer time is donated (either by the developers themselves or by their employers), this influences what their time is used for: Individual developers like doing things which are fun or high-status, while companies usually pay developers to work specifically on the features those companies need. Maintaining existing code is important, but it is neither fun nor high-status; and it tends to get underweighted by companies as well, since maintenance is inherently unlikely to be the most urgent issue at any given time.
  • Open source software is largely a "throw code over the fence and walk away" exercise. Over the past 15 years I've written freebsd-update, bsdiff, portsnap, scrypt, spiped, and kivaloo, and done a lot of work on the FreeBSD/EC2 platform. Of these, I know bsdiff and scrypt are very widely used and I suspect that kivaloo is not; but beyond that I have very little knowledge of how widely or where my work is being used. Anecdotally it seems that other developers are in similar positions: At conferences I've heard variations on "you're using my code? Wow, that's awesome; I had no idea" many times.
  • I have even less knowledge of what people are doing with my work or what problems or limitations they're running into. Occasionally I get bug reports or feature requests; but I know I only hear from a very small proportion of the users of my work. I have a long list of feature ideas which are sitting in limbo simply because I don't know if anyone would ever use them — I suspect the answer is yes, but I'm not going to spend time implementing these until I have some confirmation of that.
  • A lot of mid-size companies would like to be able to pay for support for the software they're using, but can't find anyone to provide it. For larger companies, it's often easier — they can simply hire the author of the software (and many developers who do ongoing maintenance work on open source software were in fact hired for this sort of "in-house expertise" role) — but there's very little available for a company which needs a few minutes per month of expertise. In many cases, the best support they can find is sending an email to the developer of the software they're using and not paying anything at all — we've all received "can you help me figure out how to use this" emails, and most of us are happy to help when we have time — but relying on developer generosity is not a good long-term solution.
  • Every few months, I receive email from people asking if there's any way for them to support my open source software contributions. (Usually I encourage them to donate to the FreeBSD Foundation.) Conversely, there are developers whose work I would like to support (e.g., people working on FreeBSD wifi and video drivers), but there isn't any straightforward way to do this. Patreon has demonstrated that there are a lot of people willing to pay to support what they see as worthwhile work, even if they don't get anything directly in exchange for their patronage.

May 11, 2017 04:10 AM

May 10, 2017

Villagers with Pitchforks

What You Should Know – Air Conditioning Repair 101

To so many homeowners, owning an air conditioner is not only a luxury but also a huge necessity. This important unit keeps your home cool and comfortable during the extremely hot season. It is, therefore, your responsibility as a homeowner, of ensuring that the air circulating your house remains cool and is of good quality always, by watching out for your air conditioning system. When your AC unit starts to show signs of damage, inefficiency or malfunction, the appropriate repairs ought to be done before the problem escalates into something bigger and much more costly. As a matter of fact, delaying air conditioning repairs will only lead to more problems, costlier damages, jeopardized comfort, higher energy bills, and reduced equipment lifetime. If you own an AC system in your home, below are some pointers you might want to have a look at in our article, air conditioning repair 101: what homeowners should have in mind.

How Your Air Conditioner Works

Your air conditioner cools the air around your home by removing the heat from the warm air. Warm air is sucked from the indoor space and blown through a set of pipes in the AC called the evaporator coils. The evaporator coils are filled with a liquid called a refrigerant. After absorbing heat from the air, this liquid changes state to gas and is then pumped to the condenser component of the air conditioning system. In the condenser, the absorbed heat is released into the outdoor air while the refrigerant goes back into liquid state and the cycle continues. Most air conditioners have a thermostat, which is the device that regulates the amount and temperature of cool air supplied throughout your home.

A Word about Air Filters And Your Air Conditioner

These are found in the ductwork system or other strategic points within the HVAC system. They help in removing particles from the air both to keep the air conditioner system clean and the air supplied from the unit. As the process causes the air filters to get loaded with particles, the air filters may become loaded clogged with debris, which may consequently lead to a reduction in air flow. This can, in turn, pave the way to energy inefficiency, potentially leading to high electricity bills, increased frequency of repairs, and a reduction in the equipment’s lifetime. This means that regular air filter replacement is not only an important aspect of central air conditioning system maintenance but also plays a huge part in preventing the need for air conditioning repairs in the first place.

Costly Air Conditioner Repairs Are Usually Preventable

Some air conditioning repairs such as condenser replacement can be highly expensive. One of the ways you can prevent costly AC repairs is by regularly cleaning your filters, cleaning your condensate drains, and checking out for warning signs of a bad or malfunctioning air conditioning unit. Scheduling annual professional maintenance can also be a helpful way, especially with contractors that a reputed for preventive maintenance services.

Choose Your AC Repair Contractor Wisely

Having as much information as possible on DYI tips and what to do when something goes wrong is overly important. However, knowing how to hire an experienced, professional air conditioning repair contractor is even more important. This is because a reputed HVAC expert is well trained and qualified to handle the specific job without hitches and will be able to detect the most complex and hidden problems. When choosing an HVAC service contractor, always be on the lookout for common scams in the air HVAC industry and be prepared on how to avoid them.

According to the manager of the Altitude Comfort Heating and Air – air conditioning repair Denver division, choosing the right HVAC contractor is the absolute most important thing to focus on when you need HVAC service. “Everything else will take care of itself if you choose a really good company to take care of your problem.”

The post What You Should Know – Air Conditioning Repair 101 appeared first on The Keeping Room.

by Level at May 10, 2017 08:56 PM

How To Become An Air Conditioning Repair Tech In Colorado

Colorado is one of the few states that doesn’t issue licenses for HVAC work. While there’s no state-wide license you need to attain, it’s still a good idea to become certified by a reputable trade school before starting work. Not only will you meet national, city, and county requirements, you’ll also have the ability to prove to your employer and clients that you’ve got the skills necessary to work as an HVAC technician.

Local Requirements

Colorado doesn’t have a state-wide license requirement. This doesn’t mean you can legally perform HVAC work without any certification anywhere in the state. It’s up to the local government of each city and county to determine what licenses are required for aspiring technicians.

Notably, Denver, CO has fairly strict requirements when it comes to HVAC licenses. All journeyman licenses require passing an ICC test in a related subject and several years of work in the field. While these licenses aren’t required to perform work, they’re very highly sought after by employers and potential clients.

Licenses are required to perform work in Colorado Springs and the surrounding parts of El Paso County. Applicants have to pass a background check and demonstrate proficiency by passing an exam. Unlike Denver, you can be fined or jailed if you’re caught performing work without an El Paso license. You can find the specific requirements for the Denver, CO Heating, ventilating, A/C Only Supervisor’s Certificate online.

National Qualifications

Students at trade schools in Colorado often focus on getting a certificate that holds value across the country. Often, the first step involves getting certified under section 608 of the EPA. This certification is mandatory for anyone who works with appliances that contain refrigerants — in other words, you need it to do maintenance on air conditioners, and you’ll definitely have to have it to perform air conditioning repair service.

Additionally, many students prepare for various ICC exams for HVAc technicians. These nationwide standards are the basis for many local, state and national certificates. Mastering the material covered by ICC exams gives you the freedom to become certified almost anywhere with just a little bit of review.

Job Training

If you’ve applied for a job recently, you’ve noticed that more and more companies want to hire people with experience. Trade schools like the Lincoln School of Technology help give you some of that experience. It’s much easier to get your foot in the door and find your first job as an HVAC technician when you’ve spent time in a classroom practicing the skills necessary for basic maintenance and repair. Not only will you get a paper certification, you’ll also be able to prove your skills when you ace tricky interview questions.

Further Experience

Most advanced licenses, like Denver’s journeyman licenses, require that you keep a log detailing your employer, your duties, and roughly how many hours you work. You need a minimum of two to four years of on-the-job experience to apply for these licenses. Once you’ve put in the time, however, these expert certificates will make getting a job easy. Potential employers will know that you’ve put in the time and acquired the skills to truly master your craft.

A Rewarding Career

HVAC technicians enjoy a challenging career with lots of opportunities for advancement. By starting your journey at a technical school, you’ll prove to your employer that you’re serious and enter the job with the skills that you need. This often means higher starting pay, faster advancement, and more freedom when it comes to finding the job you want. If you want to earn good money while helping local community members solve their problems, enroll in a reputable trade school like Lincoln Tech today.

The post How To Become An Air Conditioning Repair Tech In Colorado appeared first on The Keeping Room.

by Level at May 10, 2017 08:55 PM

How To Troubleshoot Your Air Conditioner

Air conditioning units are very much sought after especially throughout summertime when it gets very humid. It can get difficult to rest at night when the temperature rises beyond comfort. But like other home appliances, they can easily break down because of excessive use. Below are ways on how to troubleshoot your air conditioner.

Step 1

If you want to troubleshoot your air conditioner you need to find the best, surest and simplest ways to fix most overlooked problems. If your air conditioner is not working, this is the first step. If your AC is not turning on, you need to check on the thermostat. You may find out that the thermostat as been turned off or even set at a very low temperature that cannot allow your AC to turn on. You must also ensure that your electrical breaker hasn’t been tripped. At this point, you can trip the breaker on and off again to ensure that is in the ON position. There are several good air conditioner troubleshooting charts that can be found online.

Step 2

This is where you inspect the condition of the blower unit to ensure that it is in good working condition. At this point, you check whether the AC is producing excess air flow in the dwelling. The unit is perfectly located in the main AC housing outside of your house and looks exactly like a hamster wheel. Clean the blower unit and check if there is any damaged or bent blade. Any bent or damaged blades can cause inconvenience and ineffectiveness. If you find that you need to buy parts for your air conditioner, you can enter your model and serial number into some websites, and search for parts for the most common repair problems.

Check the duct system that comes from the AC to the vent in your house. These ducts are mostly located at the basement or attic. Ensure that there is no kink, pinched or disconnected duct. Depending on the condition of the AC filter, you can either decide to clean or replace every month. A clogged or a dirty filter will affect the air flow. This is why they require regular maintenance. This AC filter is perfectly located along the duct that returns air form home to the AC.

Step 3

This is a very important step if you want to effectively troubleshoot your air conditioner. Check whether your condenser fan is properly functioning. If it is not functioning, you can try spin the fan by your hand. The AC blades should spin on their own for 5-10 seconds. If blades do not spin on their own, there must be something wrong with the bearings and you’ll be required to replace the motor. Some manufacturers also have troubleshooting guides online for their specific air conditioner models.

Step 4

If your AC’s air isn’t cold enough, you can turn on the unit ad allow the unit to run for 10-15 minutes. Shut the unit off and go to AC unit and locate the copper suction line. Feel the line with your hand, the line should be ice cold. If the suction line is not cold, your system may require more refrigerant. You can contact an expert who can repair and install all the necessary refrigerants and seal all air conditioner leaks. He must be a reputable and a licensed air conditioning repair contractor.

Step 5

It is at this step where you should open the AC housing and check if the compressor and fan are functioning/running. The compressor is found in the main AC and looks like a round canister with a tubing coming out. If the fan is running and the compressor is not, air must be flowing via your home’s vent and it won’t be cold. If the compressor is hot and not running, this means the compressor has broken down, and you require a professional.

The post How To Troubleshoot Your Air Conditioner appeared first on The Keeping Room.

by Level at May 10, 2017 08:55 PM

May 08, 2017

TaoSecurity

Latest Book Inducted into Cybersecurity Canon

Thursday evening Mrs B and I were pleased to attend an awards seminar for the Cybersecurity Canon. This is a project sponsored by Palo Alto Networks and led by Rick Howard. The goal is "identify a list of must-read books for all cybersecurity practitioners."

Rick reviewed my fourth book The Practice of Network Security Monitoring in 2014 and someone nominated it for consideration in 2016. I was unaware earlier this year that my book was part of a 32-title "March Madness" style competition. My book won the five rounds, resulting in its conclusion in the 2017 inductee list! Thank you to all those that voted for my book.

Ben Rothke awarded me the Canon trophy.
Ben Rothke interviewed me prior to the induction ceremony. We discussed some current trends in security and some lessons from the book. I hope to see that interviewed published by Palo Alto Networks and/or the Cybersecurity canon project in the near future.

In my acceptance speech I explained how I wrote the book because I had not yet dedicated a book to my youngest daughter, since she was born after my third book was published.

A teaching moment at Black Hat Abu Dhabi in December 2012 inspired me to write the book. While teaching network security monitoring, one of the students asked "but where do I install the .exe on the server?"

I realized this student had no idea of physical access to a wire, or using a system to collect and store network traffic, or any of the other fundamental concepts inherent to NSM. He thought NSM was another magical software package to install on his domain controller.

Four foreign language editions.
Thanks to the interpretation assistance of a local Arabic speaker, I was able to get through to him. However, the experience convinced me that I needed to write a new book that built NSM from the ground up, hence the selection of topics and the order in which I presented them.

While my book has not (yet?) been translated into Arabic, there are two Chinese language editions, a Korean edition, and a Polish edition! I also know of several SOCs who provide a copy of the book to all incoming analysts. The book is also a text in several college courses.

I believe the book remains relevant for anyone who wants to learn the NSM methodology to detect and respond to intrusions. While network traffic is the example data source used in the book, the NSM methodology is data source agnostic.

In 2002 Bamm Visscher and I defined NSM as "the collection, analysis, and escalation of indications and warnings to detect and respond to intrusions." This definition makes no reference to network traffic.

It is the collection-analysis-escalation framework that matters. You could perform NSM using log files, or host-centric data, or whatever else you use for indications and warning.

I have no plans for another cybersecurity book. I am currently editing a book about combat mindset written by the head instructor of my Krav Maga style and his colleague.
Thanks for asking for an autograph!

Palo Alto hosted a book signing and offered free books for attendees. I got a chance to speak with Steven Levy, whose book Hackers was also inducted. I sat next to him during the book signing, as shown in the picture at right.

Thank you to Palo Alto Networks, Rick Howard, Ben Rothke, and my family for making inclusion in the Cybersecurity Canon possible. The awards dinner was a top-notch event. Mrs B and I enjoyed meeting a variety of people, including students in local cybersecurity degree programs.

I closed my acceptance speech with the following from the end of the Old Testament, at the very end of 2nd Maccabees. It captures my goal when writing books:

"So I too will here end my story. If it is well told and to the point, that is what I myself desired; if it is poorly done and mediocre, that was the best I could do."

If you'd like a copy of The Practice of Network Security Monitoring the best deal is to buy print and electronic editions from the publisher's Web site. Use code NSM101 to save 30%. I like having the print version for easy review, and I carry the digital copy on my tablet and phone.

Thank you to everyone who voted and who also bought a copy of my book!

Update: I forgot to thank Doug Burks, who created Security Onion, the software used to demonstrate NSM in the book. Doug also contributed the appendix explaining certain SO commands. Thank you Doug! Also thank you to Bill Pollack and his team at No Starch Press, who edited and published the book!

by Richard Bejtlich (noreply@blogger.com) at May 08, 2017 03:20 PM

May 05, 2017

R.I.Pienaar

Talk about Choria and NATS

Recently I was given the opportunity by the NATS.io folk to talk about Choria and NATS on one of their community events. The recording of the talk as well as the slide deck can be found below.

Thanks again for having me NATS.io team!

by R.I. Pienaar at May 05, 2017 05:54 AM

May 04, 2017

Michael Biven

Chai 2000 - 2017

I lost my little buddy of 17 years this week.

Chai, I’m happy you had a peaceful morning in the patio at our new house doing what you loved. Sitting under bushes and in the sun. Even in your last day you still comforted us. You were relaxed throughout the morning, when I picked you up and held you as we drove to the vet till the end.

Chai, thank you for everything, we love you and you will be missed.

May 04, 2017 08:04 AM

May 03, 2017

Marios Zindilis

Back-end Validation for Django Model Field Choices

In Django, you can provide a list of values that a field can have when creating a model. For example, here's a minimalistic model for storing musicians:

from django.db import models


class Artist(models.Model):

    TYPE_CHOICES = (
        ('Person', 'Person'),
        ('Group', 'Group'),
        ('Other', 'Other'),)

    name = models.CharField(max_length=100)
    type = models.CharField(max_length=20, choices=TYPE_CHOICES)

Using this code in the Admin interface, as well as in Forms, will make the "Type" field be represented as a drop-down box. Therefore, if all input comes from Admin and Forms, the value of "Type" is expected to be one of those defined in the TYPE_CHOICES tuple. There are 2 things worth noting:

  1. Using choices is a presentation convenience. There is no restriction of the value that can be sent to the database, even coming from the front-end. For example, if you use the browser's inspect feature, you can edit the drop-down list, change one of the values and submit it. The back-end view will then happily consume it and save it to the database.

  2. There is no validation at any stage of the input lifecycle. You may have a specified list of values in the Admin or Forms front-end, but your application may accept input from other sources as well, such as management commands, or bulk imports from fixtures. In none of those cases will a value that is not in the TYPE_CHOICES structure be rejected.

Validation

So let's find a way to fix this by adding some validation in the back-end.

Django has this amazing feature called signals. They allow you to attach extra functionality to some actions, by emitting a signal from that action. For example, in this case we will use the pre_save signal, which gets executed just before a model instance is saved to the database.

Here's the example model, with the signal connected to it:

from django.db import models


class Artist(models.Model):

    TYPE_CHOICES = (
        ('Person', 'Person'),
        ('Group', 'Group'),
        ('Other', 'Other'),)

    name = models.CharField(max_length=100)
    type = models.CharField(max_length=20, choices=TYPE_CHOICES)


models.signals.pre_save.connect(validate_artist_name_choice, sender=Artist)

That last line will call a function named validate_artist_name_choice just before an Artist is saved in the database. Let's write that function:


def validate_artist_name_choice(sender, instance, **kwargs):
    valid_types = [t[0] for t in sender.TYPE_CHOICES]
    if instance.type not in valid_types:
        from django.core.exceptions import ValidationError
        raise ValidationError(
            'Artist Type "{}" is not one of the permitted values: {}'.format(
                instance.type,
               ', '.join(valid_types)))

This function can be anywhere in your code, as long as you can import it in the model. Personally, if a pre_save function is specific to one model, I like to put it in the same module as the model itself, but that's just my preference.

So let's try to create a couple of Artists and see what happens. In this example, my App is called "v".

>>> from v.models import Artist
>>> 
>>> some_artist = Artist(name='Some Artist', type='Person')
>>> some_artist.save()
>>> 
>>> some_other_artist = Artist(name='Some Other Artist', type='Alien Lifeform')
>>> some_other_artist.save()
Traceback (most recent call last):
  File "/usr/lib/python3.5/code.py", line 91, in runcode
    exec(code, self.locals)
  File "<console>", line 1, in <module>
  File "/home/vs/.local/lib/python3.5/site-packages/django/db/models/base.py", line 796, in save
    force_update=force_update, update_fields=update_fields)
  File "/home/vs/.local/lib/python3.5/site-packages/django/db/models/base.py", line 820, in save_base
    update_fields=update_fields)
  File "/home/vs/.local/lib/python3.5/site-packages/django/dispatch/dispatcher.py", line 191, in send
    response = receiver(signal=self, sender=sender, **named)
  File "/home/vs/Tests/validator/v/models.py", line 14, in validate_artist_name_choice
    ', '.join(valid_types)))
django.core.exceptions.ValidationError: ['Artist Type "Alien Lifeform" is not one of the permitted values: Person, Group, Other']

Brilliant! We now have a expressive exception that we can catch in our views or scripts or whatever else we want to use to create model instances. This exception is thrown either when we call the .save() method of a model instance, or when we use the .create() method of a model class. Using the example model, we would get the same exception if we ran:

Artist.objects.create(name='Some Other Artist', type='Alien Lifeform')

Or even:

Artist.objects.get_or_create(name='Some Other Artist', type='Alien Lifeform')

Test It!

Here's a test that triggers the exception:

from django.test import TestCase
from django.core.exceptions import ValidationError
from .models import Artist


class TestArtist(TestCase):
    def setUp(self):
        self.test_artist = Artist(name='Some Artist', type='Some Type')

    def test_artist_type_choice(self):
        with self.assertRaises(ValidationError):
            self.test_artist.save()

Conclusion

Using Django's pre_save functionality, you can validate your application's input in the back-end, and avoid any surprises. Go forth and validate!

May 03, 2017 11:00 PM