Planet SysAdmin

the life of a sysadmin.: Bacula, gossip, advice

Thu, 02 Jul 2009 23:55:20 +0000

Bacula config coming along; figured out today that /dev/nst0 corresponds to what mtx sees as Data Transfer Element 1 (as opposed to DTE 0), which explains why previous attempts to run label barcode just failed miserably. (Neat command that.) And I had thought that DTE meant the arm, but no: upon reflection, it's a subtle/obtuse (not the right word, but oh well) way of referring to the tape drive itself.
Rather interesting comment, if you like that sort of thing, from Mark Burgess on Puppet. I know, I should remain above, but it is weirdly fascinating.
And to go out on a high note, some excellent advice from Tom Limoncelli on setting priorities as a sysadmin:

This sounds like when I was at my previous employer and they asked if
I could develop a web-based system to take surveys.  I nearly said,
"yes" because, well, I know perl, I know CGI, and I could do it.
However, I was smart enough to say "no, but surveymonkey.com will do
it for cheap."  Best of all it was self-service and the HR person was
able to do it entirely without me.  If I had said I could write such a
program, it would have been days of back-and-forth changes which would
have driven me crazy.  Instead, she was happy to be empowered to do it
herself.  In fact, doing it herself without any help became a feather
in her cap.

The lesson I learned is that "can I do it?" includes "do I want to do
it?".  If I can do something but don't want to, the answer is, "No, I
don't know how" not "I know how but don't want to".  The first makes
you look like you know your limits.  The latter sounds like you are
just being difficult.

TaoSecurity: NSA to "Screen" .gov Now, I Predict .com Later

noreply@blogger.com (Richard Bejtlich) — Thu, 02 Jul 2009 21:48:25 +0000

In my Predictions for 2008 I wrote Expect greater military involvement in defending private sector networks. Today I read a great Washington Post story titled Obama Administration to Involve NSA in Defending Civilian Agency Networks. It says in part:

The Obama administration will proceed with a Bush-era plan to use National Security Agency assistance in screening government computer traffic on private-sector networks, with AT&T as the likely test site...

President Obama said in May that government efforts to protect computer systems from attack would not involve "monitoring private sector networks or Internet traffic" and Department of Homeland Security officials say that the new program will only scrutinize data going to or from government systems...

Under a classified pilot program approved during the Bush administration, NSA data and hardware would be used to protect the networks of some civilian government agencies. Part of an initiative known as Einstein 3, the pilot called for telecommunications companies to route the Internet traffic of civilian government agencies through a monitoring box that would search for and block malicious computer codes...

The internal controversy reflects the central tension in the debate over how best to defend the nation's mostly private system of computer networks. The most effective techniques, experts say, require the automated scrutiny of e-mail and other electronic communications content -- something that commercial providers already do.

Proponents of involving the government said such efforts should harness the NSA's resources, especially its database of computer codes, or signatures, that have been linked to cyberattacks or known adversaries. The NSA has compiled the cache by, for example, electronically observing hackers trying to gain access to U.S. military systems, the officials said.

"That's the secret sauce," one official said. "It's the stuff they have that the private sector doesn't."

But it is also the prospect of NSA involvement in cybersecurity that fuels concerns of unwarranted government snooping into private communications...

The classified NSA system, known as Tutelage, has the ability to decide how to handle malicious intrusions -- to block them or watch them closely to better assess the threat, sources said. It is currently used to defend military networks.

You're thinking, "this article says NSA will not monitor purely private networks. What's the fuss?" Imagine you're the CEO, CIO/CTO, or CISO of a big company. You say "why is my company and our employees paying taxes so that the government can protect itself while my company is left outside the circled wagons?" The higher you go in corporate management, the more likely the only "security" that will be recognized will be "firewalls." So, you're going to have big-league corporate leaders telling the government that they want their companies "protected" too. This isn't really what is happening, but at that level it really doesn't matter.

The bottom line is that first the military protected itself, and now the military is going to help protect civilian government agencies. Critical private infrastructure will be next, followed by economically important companies -- think "too big to be 0wned." This will be interesting.

Richard Bejtlich is teaching new classes in Las Vegas in 2009. Late Las Vegas registration ends 22 July.

TaoSecurity: Review of Hacking Exposed: Windows, 3rd Ed Posted

noreply@blogger.com (Richard Bejtlich) — Thu, 02 Jul 2009 21:48:14 +0000

Amazon.com just posted my four star review of Hacking Exposed: Windows, 3rd Ed. Better late than never! From the review:

I've been reading and reviewing Hacking Exposed (HE) books since 1999, and I reviewed the two previous Windows books. Hacking Exposed: Windows, 3rd Ed (HEW3E) is an excellent addition to the HE series. I agree with Chris Gates' review, but I'd like to add a few of my own points. The bottom line is that if you need a solid book on Windows technologies and how to attack and defend them, HEW3E is the right resource.

Richard Bejtlich is teaching new classes in Las Vegas in 2009. Late Las Vegas registration ends 22 July.

High Scalability: It Must be Crap on Relational Dabases Week

Thu, 02 Jul 2009 17:28:46 +0000

It's hard to be a relational database lately. After years of faithful service everywhere you look the world is turning against you:

Recently at the NoSQL conference 150 revolutionaries met with their new anti-RDBMS arms suppliers. And you know what happens when revolutionaries are motivated, educated, funded, and well armed.

The revolution has gone mainstream when Computerworld writes No to SQL? Anti-database movement gains steam. It's not just whispers anymore, it's everywhere.

And perennial revolutionary Michael Stonebraker runs from blog to blog shouting the The End of a DBMS Era (Might be Upon Us). Relational vendors are selling legacy software, are 50x slower than other alternatives, and that can not stand.

The Greek Chorus on Hacker News sings of anger and lies.

Certainly some say stick with the past. It's your fault, you aren't doing it right, give us another chance and all will be as it ever was. Some smirk saying this is nothing but a return to a more ancient time when IBM was King.

But it's in the air. It's in the code. A revolution is coming. To what? That is what is not yet clear.

High Scalability: Product: Hbase

Thu, 02 Jul 2009 14:43:49 +0000

Update 3: Presentation from the NoSQL Conference: slides, video.
Update 2: Jim Wilson helps with the Understanding HBase and

http://labs.google.com/papers/bigtable.html">

http://labs.google.com/papers/bigtable.html">BigTable

by explaining them from a "conceptual standpoint."
Update: InfoQ interview: HBase Leads Discuss

High Scalability: Hypertable is a New BigTable Clone that Runs on HDFS or KFS

Thu, 02 Jul 2009 14:38:53 +0000

Update 3: Presentation from the NoSQL conference: slides, video 1, video 2.
Update 2: The folks at Hypertable would like you to know that Hypertable is now officially sponsored by Baidu, China’s Leading Search Engine. As a sponsor of Hypertable, Baidu has committed an industrious team of engineers, numerous servers, and support
resources to improve the quality and development of the open source technology.
Update: InfoQ interview on Hypertable Lead Discusses

High Scalability: Product: Facebook's Cassandra - A Massive Distributed Store

Thu, 02 Jul 2009 14:31:44 +0000

Update 2: Presentation from the NoSQL conference: slides, video.
Update: Why you won't be building your killer app on a distributed hash table by Jonathan Ellis. Why I think Cassandra is the most promising of the open-source distributed databases --you get a relatively rich data model and a distribution model that supports efficient range queries. These are not things that can be grafted on top of a simpler DHT foundation, so Cassandra will be useful for a wider variety of applications.

James Hamilton has published a thorough summary of Facebook's Cassandra, another scalable key-value store for your perusal. It's open source and is described as a "

http://labs.google.com/papers/bigtable.html">

http://labs.google.com/papers/bigtable.html">BigTable

data model running on a Dynamo-like infrastructure." Cassandra is used in Facebook as an email search system containing 25TB and over 100m mailboxes.

Google Code for Cassandra - A Structured Storage System on a

http://en.wikipedia.org/wiki/Peer-to-peer">

http://en.wikipedia.org/wiki/Peer-to-peer">P2P

Network

SIGMOD 2008 Presentation.

Video Presentation at Facebook

Facebook Engineering Blog for Cassandra

Anti-RDBMS: A list of distributed key-value stores

Facebook Cassandra Architecture and Design by James Hamilton

High Scalability: Product: Project Voldemort - A Distributed Database

Thu, 02 Jul 2009 14:02:20 +0000

Update: Presentation from the NoSQL conference: slides, video 1, video 2.

Project Voldemort is an open source implementation of the basic parts of Dynamo (Amazon’s Highly Available Key-value Store) distributed key-value storage system. LinkedIn is using it in their production environment for "certain high-scalability storage problems where simple functional partitioning is not sufficient."

High Scalability: Anti-RDBMS: A list of distributed key-value stores

Thu, 02 Jul 2009 13:53:50 +0000

Update 2: They are now called NoSQL databases. So keep up! Eric Lai wrote a good article in Computerworld No to SQL? Anti-database movement gains steam about the phenomena. There was even a NoSQL conference. It was unfortunately full by the time I wanted to sign up, but there are presentations by all the major players. Nice Hacker News thread too.
Update: Some Notes on Distributed Key Stores by Leonard Lin. What's the best way to handle a fast growing system with 100M items that requires low latency and lots of inserts? Leanord takes a trip through several competing systems. The winner was: Tokyo Cabinet.

Richard Jones has put together a very nice list of various key-value stores around the internets. The list includes: Project Voldemort, Ringo, Scalaris, Kai, Dynomite, MemcacheDB, ThruDB, CouchDB, Cassandra, HBase, and Hypertable. Richard also includes some commentary and their basic components (language, fault tolerance, persistence, client protocol, data model, docs, community).

There's an excellent discussion in the comments of Paxos vs Vector Clock techniques for synchronizing writes in the face of network failures.

Standalone Sysadmin: Update with the hiring and an upcoming blog update

standalone.sysadmin@gmail.com (Matt) — Thu, 02 Jul 2009 10:39:58 +0000

A while back, I talked about hiring another administrator. That process is currently happening and progressing nicely.

If anyone reading the blog applied, thank you. If you didn't receive a call, it is probably because you were far more overqualified than we were looking for. It's a sign of the bad economy that we're having people with 20 years experience applying for junior positions. I hope this turns around for everyone's sake.

Also, even longer ago, I presented a survey which asked an optional open-ended question. What would you do to improve the blog. Well, I hope you're not too attached to how this blog looks right now, because some time over the weekend, it's going to change quite a bit. This new iteration will require you to update the URL for the RSS feed if you're a subscriber.

To facilitate an easier transition, I'm going to be continuing to publish articles here in addition to the new site, so RSS subscribers who haven't caught the news aren't left in the dark. You will automatically be redirected to the new site if you visit this address, though. My plan for it is to be seamless for people visiting, and nearly painless for subscribers. I have no doubt that you'll let me know how it affects you and if something isn't working.

Here's where the fun begins...

A Year in the Life of a BSD Guru: Collaboration

Thu, 02 Jul 2009 09:47:46 +0000

The July issue of the OSBR is now available in PDF and HTML formats. The editorial theme this month is Collaboration and the authors include:

CiscoZine: Cisco introduces four new certifications!

Thu, 02 Jul 2009 09:18:48 +0000

Yesterday, I have received the newsletter from learning@cisco that announce the release of four new certifications.

“In response to the growing demand for IT professionals who can design, manage, and maintain converged technologies across global network infrastructures, Cisco announced the release of four new certifications at the Cisco Live! show, held this week in San Francisco, CA.

Cisco Certified Architect, CCNP Wireless, and two new Cisco Datacenter Unified Computing Specialists reinforce Cisco’s commitment developing state of the art, technology driven, role based certifications that meet the demands of today’s network professional.”

See you below the characteristics:

Cisco Certified Architect (Available January, 2010): Cisco Certified Architect is highest level of accreditation achievable within the Cisco Career Certification program. It is the pinnacle for individuals wishing to show their formal validation of both design and IT skills in Cisco technologies and infrastructure.
CCNP Wireless (Available July 24th, 2009): Built on the growing need for professionals responsible for the design, implementation, security, and operation of wireless networks and mobility infrastructures, CCNP Wireless certification recognizes the critical importance of professionals who support and manage Cisco wireless LANs and networks.
DC Unified Computing Design Specialist: It is designed to test students knowledge of the fundamentals of the Cisco Unified Compute System and their ability to implement a virtualized Data Center environment. In addition the student also tested on how to implement the Cisco Unified Computing System (UCS) in an enterprise data center routing and switching infrastructure with the next-generation Cisco Nexus product family.
DC Unified Computing Support Specialist: It validates an engineer’s ability to design scalable, reliable, and intelligent Data Center Virtualization solutions based on the Cisco Unified Computing System (UCS) along with other Cisco Data Center products, server virtualization software, and server operating systems.

References:

The Unix Blog: Unix Pax to the Rescue - Extracting Absolute Path Names from tar Archives

noreply@blogger.com (0xbadbeef) — Thu, 02 Jul 2009 08:55:53 +0000

Dealing with tar archives containing absolute path names can range from just annoying to potentially dangerous - unwittingly uncompressing the archive may range from silently growling at your simpleton co-worker who created the archive to sudden gut wrenching realization that you just overwrote some important files in unintended locations. Needless to say that creating tar archives with absolute

The Unix Blog: Message to Sun: Bring Back the Sun SPARC Workstations

noreply@blogger.com (0xbadbeef) — Thu, 02 Jul 2009 08:55:53 +0000

As sad as it is, all big three Unix workstation vendors (Sun, HP and IBM) have terminated their respective RISC workstation product lines. Which means that there are no more Sun Ultras or Blades, IBM IntelliStations or HP Visualize boxes much coveted in the not so distant past. I'm particularly impartial to Sun workstations and I think it is a big loss to the SPARC ecosystem as a whole.

Adnans Sysadmin/Dev Blog: Links for 2009-07-01 [del.icio.us]

Thu, 02 Jul 2009 07:00:00 +0000

Cooliris | Discover More

Anton Chuvakin - Security Warrior: Links for 2009-07-01 [del.icio.us]

Thu, 02 Jul 2009 07:00:00 +0000

High Scalability: Podcast about Facebook's Cassandra Project and the New Wave of Distributed Databases

Thu, 02 Jul 2009 04:30:05 +0000

In this podcast, we interview Jonathan Ellis about how Facebook's open sourced Cassandra Project took lessons learned from Amazon's Dynamo and Google's BigTable to tackle the difficult problem of building a highly scalable, always available, distributed data store.

Chris Siebenmann: Finding out when a command in a pipeline fails

Thu, 02 Jul 2009 04:29:10 +0000

Finding out when a command in a pipeline fails

Suppose you are in the situation from last entry and want to know whether may-fail actually failed, and you don't want to just split the pipeline up and use a temporary file.

In Bash, a commentator on the last entry pointed out that this is simple: you can use the $PIPESTATUS array variable to see the output status of any command in the pipeline. The same feature is available in zsh, but it uses $pipestatus (lower case), just to be different.

If you want to do this in general Bourne shell, you need some way to communicate the exit status of the command out of pipeline. You could use the very complicated mechanisms from the old comp.unix.shell FAQ, but if I had to do this I would just use a flag file:

rm -f $FAILFILE
(may-fail || touch $FAILFILE) | grep ...

If $FAILFILE exists after the pipeline has finished, may-fail failed.

If you need to distinguish between commands 'failing' due to SIGPIPE and other failures, your life is much more complicated. Fortunately I have never had to do that (or unfortunately, since it means that I have no code to share).

Some people would say that splitting a pipeline up and using temporary files is less elegant and thus less desirable than any of these techniques. I disagree; Bourne shell programming is already too complicated, so you should avoid tricky techniques unless they're absolutely necessary. Using a temporary file is almost never going to kill you and it makes your script easier to follow (especially if you add comments).

High Scalability: Latency is Everywhere and it Costs You Sales - How to Crush it

Wed, 01 Jul 2009 23:31:00 +0000

Update 5: Shopzilla's Site Redo - You Get What You Measure. At the Velocity conference Phil Dixon, from Shopzilla, presented data showing a 5 second speed up resulted in a 25% increase in page views, a 10% increase in revenue, a 50% reduction in hardware, and a 120% increase traffic from Google. Built a new service oriented

Java is very popular on the server side because it is free, relatively high performing. has a large number of useful libraries, and great development tools. Websites build using Java generally use application servers and are accessed using servelets.

http://www.java.com/en/">

http://www.java.com/en/">Java

based stack. Keep it simple. Quality is a design decision. Obsessively easure everything. Used agile and built the site one page at a time to get feedback. Use proxies to incrementally expose users to new pages for A/B testing.

http://www.oracle.com/">

http://www.oracle.com/">Oracle

Coherence
* A local computer cluster which is like a ">
* A local computer cluster which is like a "grid" because it is composed of multiple nodes.
* Offering online computation or storage as a metered commercial service, known as utility computing, "computing on demand", or "cloud computing".
* The creation of a "virtual supercomputer" by using spare computing resources within an organization.
* The creation of a "virtual supercomputer" by using a network of geographically dispersed computers. Volunteer computing, which generally focuses on scientific, mathematical, and academic problems, is the most common application of this technology

http://en.wikipedia.org/wiki/Grid_computing">
* A local computer cluster which is like a "grid" because it is composed of multiple nodes.
* Offering online computation or storage as a metered commercial service, known as utility computing, "computing on demand", or "cloud computing".
* The creation of a "virtual supercomputer" by using spare computing resources within an organization.
* The creation of a "virtual supercomputer" by using a network of geographically dispersed computers. Volunteer computing, which generally focuses on scientific, mathematical, and academic problems, is the most common application of this technology

http://en.wikipedia.org/wiki/Grid_computing">Grid

for caching. 1.5 second page load

http://en.wikipedia.org/wiki/Service_Level_Agreement">

http://en.wikipedia.org/wiki/Service_Level_Agreement">SLA

. 650ms server side SLA. Make 30 parallel calls on server. 100 million requests a day. SLAs measure 95th percentile, averages not useful. Little things make a big difference.
Update 4: Slow Pages Lose Users. At the Velocity Conference Jake Brutlag (Google Search) and Eric Schurman (Microsoft Bing) presented study data showing delays under half a second impact business metrics and delay costs increase over time and persist. Page weight not key. Progressive rendering helps a lot.
Update 3: Nati Shalom's Take on this article. Lots of good stuff on designing architectures for latency minimization.
Update 2: Why
* The time it takes for a packet to cross a network connection, from sender to receiver.

* The period of time that a frame is held by a network device before it is forwarded.

Two of the most important parameters of a communications channel are its latency, which should be low, and its bandwidth, which should be high. Latency is particularly important for a synchronous protocol where each packet must be acknowledged before the next can be transmitted.

OS Latency

Let T be a task belonging to a time-sensitive application that requires execution at time t, and let t' be the time at which T is actually scheduled.

OS latency as experienced by T as L= t' - t.

http://www.possibility.com/epowiki/Wiki.jsp?page=ItsTheLatencyStupid
">
* The time it takes for a packet to cross a network connection, from sender to receiver.

* The period of time that a frame is held by a network device before it is forwarded.

OS Latency

Let T be a task belonging to a time-sensitive application that requires execution at time t, and let t' be the time at which T is actually scheduled.

OS latency as experienced by T as L= t' - t.

http://www.possibility.com/epowiki/Wiki.jsp?page=ItsTheLatencyStupid
">
* The time it takes for a packet to cross a network connection, from sender to receiver.

* The period of time that a frame is held by a network device before it is forwarded.

OS Latency

Let T be a task belonging to a time-sensitive application that requires execution at time t, and let t' be the time at which T is actually scheduled.

OS latency as experienced by T as L= t' - t.

http://www.possibility.com/epowiki/Wiki.jsp?page=ItsTheLatencyStupid
">Latency

Lags Bandwidth, and What it Means to Computing by David Patterson. Reasons: Moore's Law helps BW more than latency; Distance limits latency; Bandwidth easier to sell; Latency help BW, but not vice versa; Bandwidth hurts latency; OS overhead hurts latency more than BW. Three ways to cope: Caching, Replication, Prediction. We haven't talked about prediction. Games use prediction, i.e, project where a character will go, but it's not a strategy much used in websites.
Update: Efficient data transfer through zero copy. Copying data kills. This excellent article explains the path data takes through the OS and how to reduce the number of copies to the big zero.

Latency matters. Amazon found every 100ms of latency cost them 1% in sales. Google found an extra .5 seconds in search page generation time dropped traffic by 20%. A broker could lose $4 million in revenues per millisecond if their electronic trading platform is 5 milliseconds behind the competition.

The Lone Sysadmin: Mistakes

Wed, 01 Jul 2009 19:31:02 +0000

“No one wants to learn by mistakes, but we cannot learn enough from successes to go beyond the state of the art.” – Henry Petroski, “To Engineer is Human: The Role of Failure in Successful Design”

My friend Jon has a script to randomize quotes in his email signature, and this came through yesterday. I’ve always enjoyed Henry Petroski’s books, and though he’s a civil engineer most of the lessons are ones IT professionals can learn from, too.

Luckily, most of our mistakes don’t involve buildings collapsing, though.[0]

——————-

[0] Which reminds me of another thing a former boss of mine used to say during crisis situations: “Calm down, nobody is dying here.”

Anton Chuvakin - Security Warrior: Monthly Blog Round-Up – June 2009

noreply@blogger.com (Dr Anton Chuvakin) — Wed, 01 Jul 2009 19:07:00 +0000

As we all know, blogs are a bit "stateless" and a lot of good content gets lost since many people, sadly, only pay attention to what they see today. These monthly round-ups is my attempt to remind people of useful content from the past month! If you are “too busy to read the blogs” (eh…cause you spent all your time on Twitter? :-)), at least read these.

So, here is my next monthly "Security Warrior" blog round-up of top 5 popular posts/topics.

Top this month is taken by my recent post “Why No Open Source SIEM, EVER?” as well as its older inspiration “On Open Source in SIEM and Log Management.” Looks like all the commercial SIEM vendors came over and marveled at its beauty :-)
My quick analysis of the infamous merchant associations’ letter to the PCI Council in the post “On “PCI Letter” takes the next spot. BTW, Bob Russo already responded to this via this podcast and this interview.
They say that personal stories are always fun to read and so my personal story on switching away from Mozilla Firefox towards Google Chrome (“On Switching Away from Firefox”) is in the top too. Is Firefox really the IE of 2009?
My review and coverage of the book “Beautiful Security” (“Best Chapter From “Beautiful Security” Downloadable!” and “Book Review “Beautiful Security””) is popular due to a lot of linking to it.
My little log sharing initiative (“Free Log Data For Research – Update”) took off like a rocket. Here is the log sharing site and here is the mailing list/Google Group.
Finally as #6, my PCI DSS Q&A from the Qualys webcast “PCI DSS Myths” are still hot: “PCI Myths Webcast Recording and Q&A.” Also, our latest webcast slides and Q&A are here ("PCI DSS Prioritized Webcast Slides and Q&A")

See you in July. Also see my annual “Top Posts” (2007, 2008)

P.S. Stand by for the post on Paris Hilton tomorrow… :-)

Possibly related posts / past monthly popular blog round-ups:

Technorati Tags: chuvakin,monthly,blog,security

HolisticInfoSec.org: Malzilla: Exploring scareware and drive-by malware

noreply@blogger.com (Russ McRee) — Wed, 01 Jul 2009 18:59:00 +0000

Yesterday included a SANS ISC diary post regarding a tool list useful for de-obfuscation. Amongst the entries was Malzilla.
Fortuitous timing I say!
My toolsmith column for July's ISSA Journal is a complete analysis of Malzilla's capabilities.

Malzilla is best described as a useful program for use in exploring malicious pages, allowing you to choose your own User Agent and referrer and use proxies. While it downloads Web content, it does not render it, so it is not a browser. Think of it as WGET with a user interface and some very specific talents. In Using Malzilla, we’ll take a close look at rogue AV tactics and exploit sites in order to study the infection process utilized.

Lenny Zeltser contributed great feedback regarding Malzilla for this piece, thus furthering the tool's credibility.
Give the article a read and add Malzilla to your arsenal.
Cheers.

">
">
">del.icio.us | digg | Submit to Slashdot

Please support the Open Security Foundation (OSVDB)

CiscoZine: OSPF Virtual Link

Wed, 01 Jul 2009 11:53:20 +0000

The Open Shortest Path First (OSPF) protocol, defined in RFC 2328, is an Interior Gateway Protocol used to distribute routing information within a single Autonomous System.

The OSPF protocol is based on link-state technology, which is a departure from the Bellman-Ford vector based algorithms used in traditional Internet routing protocols such as RIP. OSPF has introduced new concepts such as authentication of routing updates, Variable Length Subnet Masks (VLSM), route summarization, and so forth.

An OSPF network can be divided into sub-domains called areas. An area is a logical collection of OSPF networks, routers, and links that have the same area identification. A router within an area must maintain a topological database for the area to which it belongs. The router doesn’t have detailed information about network topology outside of its area, thereby reducing the size of its database.

All areas in an OSPF autonomous system must be physically connected to the backbone area (area 0). In some cases where this physical connection is not possible, you can use a virtual link to connect to the backbone through a non-backbone area. You can also use virtual links to connect two parts of a partitioned backbone through a non-backbone area. The area through which you configure the virtual link, known as a transit area, must have full routing information. The transit area cannot be a stub area.

Example:
Suppose to manage a network running an OSPF process. The network has three areas: area0 (the backbone), area2 and area3.

The area0 has four networks:

1.0.0.0/24

1.0.1.0/24

1.0.2.0/24

1.0.3.0/24

The area2 has two networks:

2.0.0.0/24

2.0.1.0/24

The area3 is connected to the area0 via area2 and it has two networks:

3.0.0.0/24

3.0.1.0/24

In this example, we must configure three routers: Ciscozine1, Ciscozine2 and Ciscozine3. Ciscozine1 belongs to Area0 and Area2, Ciscozine2 belongs to Area2. Ciscozine3 belongs to Area2 and Area3, but due to OSPF constraint (all areas in an OSPF autonomous system must be physically connected to the backbone area), the Ciscozine3 router requires a Virtual-link.

Tips:

For convenience, the networks 1.0.0.0/22 and 3.0.0.0/23 will be defined using loopback interfaces.
To advertises the loopback subnet as the actual subnet configured on loopbacks, the “ip ospf network point-to-point” command is configured under loopbacks.
Router ID: It’s a 32-bit number assigned to each router running the OSPF protocol. This number uniquely identifies the router within an Autonomous System. RID is the highest logical (loopback) IP address configured on a router, if no logical/loopback IP address is set then the Router uses the highest IP address configured on its active interfaces. In this example, to have more control, I have chose to define statically the RID using the “router-id” command

Below, the router configurations based on four steps:

Interface configuration
Test connectivity
OSPF configuration
Virtual Link

The three (partial) router configurations:

Ciscozine1#
!
interface Loopback0
 ip address 1.0.0.1 255.255.255.0
 ip ospf network point-to-point
!
interface Loopback1
 ip address 1.0.1.1 255.255.255.0
 ip ospf network point-to-point
!
interface Loopback2
 ip address 1.0.2.1 255.255.255.0
 ip ospf network point-to-point
!
interface Loopback3
 ip address 1.0.3.1 255.255.255.0
 ip ospf network point-to-point
!
interface FastEthernet0/0
 description Link-to-Ciscozine2
 ip address 2.0.0.1 255.255.255.0
!
router ospf 1
 router-id 1.0.0.1
 area 2 virtual-link 3.0.0.1
 network 1.0.0.0 0.0.3.255 area 0
 network 2.0.0.0 0.0.0.255 area 2

 

Ciscozine2#
!
interface FastEthernet0/0
 description Link-to-Ciscozine1
 ip address 2.0.0.2 255.255.255.0
!
interface FastEthernet0/1
 description Link-to-Ciscozine3
 ip address 2.0.1.1 255.255.255.0
!
router ospf 1
 router-id 2.0.0.2
 network 2.0.0.0 0.0.1.255 area 2



Ciscozine3#
!
interface Loopback0
 ip address 3.0.0.1 255.255.255.0
 ip ospf network point-to-point
!
interface Loopback1
 ip address 3.0.1.1 255.255.255.0
 ip ospf network point-to-point
!
interface FastEthernet0/0
 description Link-to-Ciscozine2
 ip address 2.0.1.2 255.255.255.0
!
router ospf 1
 router-id 3.0.0.1
 area 2 virtual-link 1.0.0.1
 network 2.0.1.0 0.0.0.255 area 2
 network 3.0.0.0 0.0.1.255 area 3

Remember: To display parameters about and the current state of OSPF virtual links, use the “show ip ospf virtual-links” command in EXEC mode.

Note: You can also build a generic routing encapsulation (GRE) tunnel between two routers and put the tunnel in Area 0. The main differences between a GRE tunnel and a virtual link are:

GRE Tunnel	Virtual Link
All traffic in the tunnel is encapsulated and decapsulated by the tunnel endpoints.	The routing updates are tunneled, but the data traffic is sent natively.
Tunnel headers in every packet cause overhead.	Data traffic is not subject to any tunnel overhead.
The tunnel can go through a stub area.	The transit area cannot be a stub area, because routers in the stub area do not have routes for external destinations. Because data is sent natively, if a packet destined for an external destination is sent into a stub area which is also a transit area, then the packet is not routed correctly. The routers in the stub area do not have routes for specific external destinations.

References:

http://www.cisco.com/…/support_sub-protocol_home.html

http://www.cisco.com/…/configuration_example09186a00801ec9ee.shtml

http://www.cisco.com/…/configuration_example09186a00800946bd.shtml

Standalone Sysadmin: New Article: Manage Stress Before It Kills You

standalone.sysadmin@gmail.com (Matt) — Wed, 01 Jul 2009 11:50:24 +0000

My newest column is up at Simple Talk Exchange. It's called "Manage Stress Before it Kills You.

It starts out with a true-to-life story of something that happened to me one night. It was scary, but it did let me know that something was wrong. My advice is to manage your stress before it gets to this point, because it isn't an enjoyable experience.

Please make sure to vote up the article if you like it! Thanks!

Adnans Sysadmin/Dev Blog: Links for 2009-06-30 [del.icio.us]

Wed, 01 Jul 2009 07:00:00 +0000

Free Utility: RichCopy, an Advanced Alternative to RoboCopy

Anton Chuvakin - Security Warrior: Links for 2009-06-30 [del.icio.us]

Wed, 01 Jul 2009 07:00:00 +0000

Layer 8 BSOFH: All’s fair in security and war.

TechRepublic Network Administrator: Storage in the cloud: Requires a different mindset

Wed, 01 Jul 2009 04:01:32 +0000

One of the fundamental requirements for any IT professional deciding whether to embrace or dismiss cloud computing is to first understand it. IT pro Rick Vanover highlights the big picture offerings of storage in the cloud.
—————————————————————————————–

In traditional computing infrastructures – brick and mortar IT if you will – storage is fairly simple to understand. There are two main areas of management – data and disks. In regards to cloud computing, the data management doesn’t go away. In fact, I would argue that the data management requirement increases with the cloud. But the all-encompassing disk management requirements go away when cloud storage is used.

There are two fundamental types of cloud storage and explain their usage implications and how they can be applied to organizations as they consider cloud architectures.

API-based storage

In the case of Amazon Web Services, this is the more common storage option available through the Simple Storage Service (S3) cloud. Accessing S3 is different than traditional storage in internal infrastructures as it is accessed through a web service via the S3 API. The beauty here is that organizations can write their own applications through the well-defined S3 API or organizations can utilize partner solutions that adhere to the specification.

Direct storage

Again with the Amazon cloud offering, cloud solutions can be provisioned storage on a direct-attached basis for extra storage. The Elastic Block Store (EBS) cloud is a provisioning mechanism to allocate direct storage to an instance in the Elastic Compute Cloud (EC2). An EC2 instance can include familiar entities such as a Windows Server 2003 or a Linux system. The EBS storage provisioning is comparatively much quicker than the API-driven S3 architecture.

Doesn’t it always just depend?

The best selection for storage in the cloud will depend on many factors, but these two fundamental differences can highlight how storage can be provisioned in the cloud. S3 is better for multiple inbound and outbound points due to its collaborative nature. EBS, on the other hand would be better for singular I/O intensive activities that go along with a system or application.

Cloud computing is a reality, and infrastructure professionals need to understand the details to justify their pro or con cloud stance above all else. Share your comments below on cloud storage technology (leave security and compliance out for now – that is coming in another series of posts).

Chris Siebenmann: A Unix irritation: pipeline status

Wed, 01 Jul 2009 03:00:05 +0000

A Unix irritation: pipeline status

Let's start with a Bourne shell irritation. Suppose you have a command sequence that looks like:

may-fail | grep ...

Further suppose that you want to know whether may-fail actually failed. In the normal Bourne shell, you're out of luck; the exit status of a pipeline is the exit status of the last command, and grep will probably succeed no matter what happens to may-fail. You're going to need to use a temporary file instead of the pipeline or get very creative.

It's tempting to blame the Bourne shell for this, but I think that its hands are at least partially forced by our old friend SIGPIPE. Consider this pipeline:

produce-lots | sed 10q

What's the exit status of this pipeline, assuming that produce-lots would succeed if left alone?

Sed will exit successfully after writing ten lines. But this closes the pipe, which means that produce-lots gets either a SIGPIPE signal or an EPIPE write error (if SIGPIPE is being ignored). Unless produce-lots is specially coded, it is going to exit with some sort of error status. If the Bourne shell only considered pipelines successful if all of their commands succeeded, this sort of pipeline would fail, much to people's surprise.

(The shell could consider pipeline commands that exited on SIGPIPE to have succeeded, but that's an unreliable hack.)

Hence, the only reliable exit status is the exit status of the last command in the pipeline; everything else can wind up 'failing' when they have not in fact failed, it's just that the pipeline has shut down early.

(One comment.)

TechRepublic IT Security: Stay out of Bozeman

Tue, 30 Jun 2009 23:48:38 +0000

Bozeman, Montana has some disturbing city employment application requirements. The effects of those requirements might be more important than you think.

Perhaps part of the problem with governmental violations of privacy in the United States is the public’s insatiable thirst for private information about its politicians. Whenever someone runs for President, you can expect a lot of information to get dug up about his or her sordid past, and the press feeds not only this thirst for private information but also the public perception that it has a “right” to know about these things. It has gotten to the point where nobody (sane) runs for public office without simply accepting the notion that the details of his or her private life are going to be subject to public scrutiny.

I am the last person to argue against governmental transparency. In general, policy should not have to be secret for it to work. Just ask Claude Shannon, the Father of Information Theory: Shannon’s Maxim states “The enemy knows the system.” He was saying that, in security, one should never rely on the secrecy of policy or process to ensure security.

Shannon’s Maxim was just a more recent, generalized, and pithy formulation of the same ideas embodied in Kerckhoffs’ Principle: The design of a system should not require secrecy and compromise of the system should not inconvenience the correspondents. This principle is one of six principles of practical cipher design articulated by Auguste Kerkhoffs in La Cryptographie Militaire, arguably one of the most important documents on the subject of cryptography ever written.

As I pointed out in Public officials and private lives, though, there may be a connection between the desire of the American public for the sordid details of the lives of public officials on one hand, and the growing prevalence of privacy violating policy and legislation in US government on the other. Ironically, much of this systematic violation of the privacy of millions of US citizens and other residents is being done in the name of national security. This flies directly in the face of the simple, unavoidable fact that privacy is security.

In City wants job applicants to turn over Facebook user names and passwords, Toni Bowers reported a recent revelation about Bozeman, MT city hiring practices, which read like something out of an Orwell novel. She quoted cbsnews.com:

The Rocky Mountain city instructs all job applicants to divulge their usernames and passwords for “any Internet-based chat rooms, social clubs or forums, to include, but not limited to: Facebook, Google, Yahoo, YouTube.com, MySpace, etc.” Bozeman city officials say that this is just a component of a thorough background check.

As pointed out in a quote from attorney Kevin Bankston, this essentially makes employment by the city of Bozeman conditional upon waiving First Amendment rights and relinquishing any control over your own online security. The easy answer seems to be to never seek employment with the city of Bozeman, of course — but this may be relevant to the question of living in Bozeman too, or even of visiting. Consider the points I brought up in my examination of the way the American public treats the privacy of public officials. Here, we are not just talking about elected officials getting the “no privacy” treatment. We have gone well beyond that, and are now talking about every single employee of the city having his or her online privacy and security violated as a routine part of the hiring process.

Consider the kinds of people who would accept this kind of intrusion into their lives just to get an entry-level city bureaucrat’s job. How many of these people are likely to have any regard for your privacy at all? Consider what this says about people tasked with teaching your children if they attend Bozeman’s public elementary schools. What values will they instill in the impressionable minds for whose education they are responsible?

Regardless of whether a candidate for President, or a current President, should have to regard his or her entire life as an open book, I quite simply believe that the way Bozeman, MT handles its hiring process is beyond all reason. If the United States is, as some claim, on its way to becoming a police state, it seems Bozeman is in the race to get there first. Considering that most of Montana seems to be solidly grounded in principles quite antithetical to this kind of intrusive insanity, the mind must boggle at the audacity of Bozeman officials’ disregard for simple standards of human decency.

I, for one, will never give up the passwords for even the most trivial online Website logins as a condition of employment. Any prospective employers will have to sift through network traffic for the passwords to sites that do not use encrypted connections for authentication, just like any other malicious security cracker — and make no mistake, I do regard this behavior as malicious.

The Lone Sysadmin: Size Labels for Virtual Environments – A Proposal

Tue, 30 Jun 2009 23:27:25 +0000

“How big is your virtual environment?”

I love that question. Find a virtual environment and ask ten people who work on it, and they’ll give you ten different answers. “It’s pretty big,” one person will say. The next person will say “oh, we’re small.” The next two people asked will argue with each other until you shake your head and walk away. It’s all relative, too. If most guys you know have 50 virtual machines, and you have 200, you’re big, relatively-speaking. You’ve got problems they don’t have, and you’d probably like to talk with others that have had those same problems. Talking to a guy who has 2000 VMs isn’t going to help you much, though. He’s operating at a whole different scale, size, and budget level.

I spent some time this morning answering questions for a fellow who wants to build a large virtual environment. He didn’t have a lot of specifics to start with, but was really balking at what I was suggesting he look into for storage, servers, etc. As it turns out, “large” to him was really only 50 virtual machines in the next three years. That’s a big difference from what I perceive as large, which means many thousands of dollars, different storage and software strategies, completely different P2V approaches, etc.

As such, I propose some simple terminology, based on a logarithmic scale, to help sort out sizing:

I have 300 virtual machines, so I consider myself to be medium-sized. When I get to 1000 VMs I’ll be large.

Complexity is a whole different problem. From a complexity perspective my environment is pretty simple. The thing is, someone can take a small environment and make it really complex. And some of the biggest environments I’ve seen have been pretty simple, overall. It’s only scale I’m proposing labels for.

At any rate at least I have a graph to point to when I’m talking about this stuff. :-)

A Unix Sysadmin's Journey: Migrating a Zone to a Different Machine on Solaris 10

Tue, 30 Jun 2009 19:37:04 +0000

Zones are one of the best features in Solaris 10 -- they're so lightweight that you can use them at almost no cost in performance. Today, I ran across a situation where one of my zones needed more RAM, and the box it was on didn't have it. Read on for how to migrate a Solaris Zone to a different machine, and an important update to Solaris 10/08 that makes the process so much easier.

Anton Chuvakin - Security Warrior: Vulnerability Scanning and Clouds/SaaS/IaaS/PaaS

noreply@blogger.com (Dr Anton Chuvakin) — Tue, 30 Jun 2009 19:07:00 +0000

Here is a very fun post called “Vulnerability Scanning and Clouds: An Attempt to Move the Dialog On…” I loved it so much, I will just quote my favorite parts here with a few comments. It starts like this:

“Much has been said about public IaaS providers that expressly forbid customers from running network scans against their cloud hosted infrastructure.”

In other words, they host your server, but you cannot check it for vulnerabilities at all.

“As has been noted before, a blanket ban on legitimate scanning activity by customers of their own infrastructure (whether outsourced or not) undermines security assurance processes and can make regulatory compliance impossible; e.g. PCI DSS mandates network vulnerability scanning as a control”

Use PaaS – lose PCI DSS compliance status. Nice! Sadly, the above interpretation is correct as, I suspect, the IaaS/PaaS provider is not allowed to scan your boxes either. So, nobody does. And then you get 0wned.

Next the post highlights that the fact that vulnerability management challenges are magnified by using PaaS/IaaS. For example:

“Scanning may trigger automated or manual actions by the provider. A common automated response from a provider is to apply traffic shaping to slow down the scan, or simply block the client IP address via an ACL update. This can lead to false negatives; i.e. vulnerabilities present are not discovered as the scanner IP was automagically identified as a noisy vulnerability scanner and auto-throttled/blocked.”

He then highlight somewhat obvious, but still key point:

“Even if spinning up copies of “known good/secure” virtual machine (VM), you still need to scan them.”

Further:

“So the bad guys get to scan because they don’t care and yet the customer, who wants to do the “right thing”, is not allowed to. Is that rational?”

The solution is “easy” – if you need scanning (and everybody does!) and you PaaS doesn’t allow it, don’t use that PaaS. But is this a solution? The post the suggests “ScanAuth API” which will allow controlled scanning:

“Something like an “ScanAuth” (Scan Authorize) API call offered by cloud providers that a customer can call with parameters for conveying source IP address(es) that will perform the scanning, and optionally a subset of their Cloud hosted IP addresses, scan start time and/or duration. This request would be signed by the customers API secret/private key as per other privileged API calls.”

I am not sure about you, but this sounds like an awesome idea! The original post calls for the start of discussion, and I am happy to continue it. Finally, as of today, I don’t think we can rely on “other security tools” (software assurance, secure coding, etc) to supplant the need for vulnerability scanning. if you deploy an OS in the cloud, you’d need to scan it.

BTW, similarly to network vulnerability scanning, the situation is actually worse for web app scanning? If you have “doubts” about your blog provider, can you hit it with Qualys WAS?

High Scalability: Hot New Trend: Linking Clouds Through Cheap IP VPNs Instead of Private Lines

Tue, 30 Jun 2009 18:38:21 +0000

You might think major Internet companies have a latency, availability, and bandwidth advantage because they can afford expensive dedicated point-to-point private line networks between their data centers. And you would be right. It's a great advantage. Or it at least it was a great advantage. Cost is the great equalizer and companies are now scrambling for ways to cut costs. Many of the most recognizable Internet companies are moving to IP VPNs (Virtual Private Networks) as a much cheaper alternative to private lines. This is a strategy you can effectively use too.

This trend has historical precedent in the data center. In the same way leading edge companies moved early to virtualize their data centers, leading edge companies are now virtualizing their networks using IP VPNs to build inexpensive private networks over a shared public network. In kindergarten we learned sharing was polite, it turns out sharing can also save a lot of money in both the data center and on the network.

The line of reasoning for adopting IP VPNs goes something like this:

MDLog:/sysadmin: OSBridge: Configuration Management Panel

Tue, 30 Jun 2009 13:40:26 +0000

The moment I heard about the Open Source Bridge Configuration Management panel session on FLOSS Weekly a while ago, I was hoping that I will be able to see the recording of this session (as for obvious reasons I was not able to attend and see this live in Portland, Oregon). They managed to bring together (for the first time to my knowledge) the creators (or maintainers) of *all* the major configuration management tools to date was very impressive; and obviously someone as myself that has been working with many of these tools (I haven’t tried/used automateit yet) would definitely see this as a great session.

Here are the members of the configuration management panel (from left to right):

Igal Koshevoy of AutomateIt
Brendan Strejcek of Cfengine
Luke Kanies from Reductive Labs for Puppet
Narayan Desai of bcfg2
Adam Jacob from Opscode for Chef

Luckily the video of the session (among other videos from Open Source Bridge) was published and anyone can see this great event:

Now, after I sow this I must admit that I was hoping for a little more engagement and controversy. Instead we sow a friendly debate where everyone presented his own tool, without trying to go over the line and tell why it is better than the one of someone else (we have definitely seen several such blog posts from them in the past ). Anyway this was a great event and a great opportunity to have all the major people in this field come together and share their story. I’m sure that after this they will get back to work, we will see new features and improvements in their tools.

Standalone Sysadmin: Fun with VMware ESXi

standalone.sysadmin@gmail.com (Matt) — Tue, 30 Jun 2009 13:33:32 +0000

Day one of playing with bare metal hypervisors, and I'm already having a blast.

I decided to try ESXi first, since it was the closest relative to what I'm running right now.

Straight out of the box, I run into my first error. I'm installing on a Dell Poweredge 1950 server. The CD boots into an interesting initialization sequence. The screen turns a featureless black, and there are no details as to what is going on behind the scenes. The only indication that the machine isn't frozen is a slowly incrementing progress bar at the bottom. After around 20 minutes (I'm guessing the time it takes to read and decompress an entire installation CD into memory), the screen changes to a menu asking me to hit R if I want to repair, or Enter if I want to install. I want to install, so I hit Enter. Nothing happens, so I hit enter again. And again. And again. It takes a few more times before I realize that the "numlock" light is off. Curious, I hit numlock and it doesn't respond.

Awesome.

I unplug and replug the keyboard in. Nothing. Move it to the front port. Nothing. I reboot and come back to my desk to research. Apparently, I'm not alone. Those accounts are from 2008. I downloaded this CD an hour ago, and it's 3.5 U4 (the most current 3.5x release). It is supposed to have support on the PE1950, but if the keyboard doesn't even work, I have my doubts.

Lots of people have suggested using a PS2 keyboard as the accepted workaround, but in a similar tone to most of my problem/solution options, this server has no PS2 ports.

I'm downloading ESX v4 now. I'll update with how it goes, no doubt.

MDLog:/sysadmin: FindMyHosting Review

Tue, 30 Jun 2009 10:51:04 +0000

This post is sponsored by FindMyHosting - a free and very comprehensive web hosting directory featuring the most popular web hosting companies and thousands of customer reviews.

I’ve been asked to review this site and give my impressions about it. The truth is that I don’t have much experience with shared hosting as most of my experience is with dedicated servers from various hosting companies, and anytime I had a friend asking about where do I recommend him to host his small site I didn’t knew where to direct him. This is why I thought that such a webhosting directory as FindMyHosting would be a great start for anyone looking for a shared hosting account to host his new site. We can search from a long list of hosting company and get them ranked by users reports (nice).

We can easily search for the best hosting plans by:

price
country (would be nice to see some from EU, not only from the US, etc.)
platform (linux, win, etc.)
disk space
data transfer

The hosting directory also lists various plans by their programming languages and features support like:

FrontPage Web Hosting
PHP / MySQL Web Hosting
ASP Web Hosting Plans
JSP Web Hosting Plans
ColdFusion Web Hosting Plans

Besides the searchable database of hosting plans FindMyHosting is also providing some very good introductory articles for people new to hosting that can help them better understand this industry and make a better decision on finding their first host.

Conclusion

FindMyHosting is a webhosting directory that can help people find the right hosting plan and hosting company. This is mainly restricted to shared hosting (even if you can see some dedicated server entries you should not rely on that list) and mainly from hosting companies from the States. I would suggest to bring in more hosting companies and their offering from all over the world (Europe for ex., but any country really); normally finding a good host in US is much easier than in other places . Also personally, I would rather remove the dedicated server section as that can be confusing to new users in the domain, or if not try to add some serious companies and be a real directory for dedicated servers also.

Adnans Sysadmin/Dev Blog: Links for 2009-06-29 [del.icio.us]

Tue, 30 Jun 2009 07:00:00 +0000

XKeymacs

Anton Chuvakin - Security Warrior: Links for 2009-06-29 [del.icio.us]

Tue, 30 Jun 2009 07:00:00 +0000

Yes, I Would Like To Hack Myself - SpywareGuide Greynets Blog

Chris Siebenmann: More on why users keep mailing specific people

Tue, 30 Jun 2009 04:46:42 +0000

More on why users keep mailing specific people

Perhaps unsurprisingly, most of the people who've commented on my last entry have attributed this behavior to people's desire to get their issues dealt with promptly (to some, this is jumping the queue; what you could call a 'vigorous discussion' has broken out in the comments about this, rather to my surprise). I have a couple of reactions to this view.

First, I am pretty sure that this is not why people here do it, at least for the kind of cases that I'm thinking of. We don't have a trouble ticketing system or the like, just email aliases, and generally the users email the person who was going to deal with their issue anyways; the only effective difference is what email address they use. Hence my belief that our users really do keep emailing specific people because it's easier to remember people than mail aliases.

(From our perspective it matters what they email; when you mail an alias, everyone is in the loop and we have a record of it. But these are internal process issues, not things that the users care about, at least not until the person they emailed is out sick one day. And I actually suspect that they accept that sort of thing happening, because after all they did email a specific person.)

Second, if getting prompt responses is the reason that users are mailing you directly you have at least one problem, to wit either your response times are perceived as too slow or the procedures for going through regular channels are too complicated. If users are also doing it to jump the queue, it is my opinion that you also have either problem users or a significantly dysfunctional organizational environment (at a minimum, one where there is vigorous disagreement over what your priorities should be).

In either case, swatting users on the nose is generally not an effective way to solve your problems (although it is a great way to make them worse). Instead, you need to deal with the root causes, the hard social problems. Sometimes this will be beyond your power; in that case I believe that you need to do the best you can and be as transparent about what is going on as possible.

(If the problem is organizational politics, the last thing you want to do is put yourself in the position of being everyone's chewtoy. Let aggravated people see that it is not your fault, so that they can take their gripes to higher powers. And if you're dealing with problem users, you really want to have management approval of what you're doing; otherwise, you may find out the hard way that the problem users have more power than you do.)

(7 comments.)

the life of a sysadmin.: 1246317421 seconds since the epoch...

Mon, 29 Jun 2009 23:55:19 +0000

I'm back at work after a week off. The UPS control panel continues to work (!), but there is no word back from the manufacturer (says the contractor who installed the thing and filed the ticket). I find this troubling; either the manufacturer really hasn't got back to us yet (bad), or I should have insisted on being a contact for the ticket. I'll have tos ort this out tomorrow.

Spent much of my day tearing my hair out over mod_proxy_html. Turns out that, by default, it strips the DTD from the HTML it proxies; this is a problem for one app that we're proxying. Not only that, the DTDs it does support are HTML, XHTML, and either with a "Transitional"/Legacy flag — but no URI to a DTD, like the one pointing to the Loose DTD that our app uses and the damned thing threw to the floor. (Sorry, brain cells on strike today and my ability to write clearly is going downhill.)

You can specify your own DTD, including a URI (undocumented feature, whee!), and thus put back in the original — but it doesn't append a newline, there's no way to append a newline that I could figure out, and so it mushes the DTD together with the first html opening tag and makes baby Firefox cry and render the page badly.

My rule of thumb for a long time was that if I start lppooking at source code, I'm in over my head. I'm starting to think that may not be entirely true anymore, that I've advanced to the point where I can read C (say) and generally understand what's going on. But when I start looking for API documentation for Apache 2.2 (surprisingly hard to find) to find out if, say, ap_fputs or apr_pstrdup chomp newlines or something (near as I can tell, they don't), or just what AP_INIT_TAKE12 takes as arguments…well, then I am in over my head. If nothing else, I don't want to make some silly error because I don't know what the hell I'm doing. (That's not a slam against the Debian folks; I just mean that I felt shivers when I read about that, because I dread making the same sort of highly-visible, catastrophic error) (unlike the rest of the planet, you understand).

TechRepublic IT Security: Masking passwords: Why it's not a good idea

Mon, 29 Jun 2009 22:25:51 +0000

I’ve just read an article arguing that password masking isn’t worth the effort, even detrimental. I’m not sure where I stand on this, so let’s work through it together.

——————————————————————————————————————-

The article Stop Password Masking, was written by Dr. Jakob Nielsen, a well-regarded expert on Web and user interfaces:

“Jakob Nielsen, Ph.D., is a User Advocate and principal of the Nielsen Norman Group which he co-founded with Dr. Donald A. Norman (former VP of research at Apple Computer). Before starting NNG in 1998 he was a Sun Microsystems Distinguished Engineer.

Dr. Nielsen founded the “discount usability engineering” movement for fast and cheap improvements of user interfaces and has invented several usability methods, including heuristic evaluation. He holds 79 United States patents, mainly on ways of making the Internet easier to use.”

As you can see by Dr. Nielsen’s accreditation, his mentioning that using password masking is a bad idea isn’t something to be taken lightly.

Why mask passwords?

Until I read the article, I considered masking passwords to be a no-brainer for the following reasons:

Masking passwords were the logical outcome of being concerned about people stealing passwords by visually observing the password being entered.
Auto-complete is a bad idea period, but masking helps prevent someone from seeing previous passwords that have the same first few characters. This is of special concern when the computer has multiple users.
Masking passwords is required by some regulatory bodies in order to gain their approval. Also a company’s security policy may require masking any time a password is entered.

Why password masking is bad

Nielsen summarizes his stance by pointing out:

“Usability suffers when users type in passwords and the only feedback they get is a row of bullets. Typically, masking passwords doesn’t even increase security, but it does cost you business due to log in failures.”

Through his research, Nielsen has come to the conclusion that using nondescript bullets to cover up password characters violates an important usability principle, that of providing sensory feedback. To back up his claim, Nielsen provides some additional detail:

Users make more errors when they can’t see what they’re typing while filling in a form. They therefore feel less confident. This double degradation of the user experience means that people are more likely to give up and never log in to your site at all, leading to lost business.
The more uncertain users feel about typing passwords, the more likely they are to (a) employ overly simple passwords and/or (b) copy-paste passwords from a file on their computer. Both behaviors lead to a true loss of security.

I didn’t see any reference to studies verifying either of the above theories, still both appear to have merit.

Using portable devices

I do agree with Nielsen about how masking passwords on mobile devices is a real pain. As proof, I know associates that do exactly as Nielsen mentioned above. They dumb-down the password just so it’s easy to enter. Not a smart thing to do when visiting important Web sites such as a banking portal.

Another viewpoint

Jason Montgomery, a security expert with SANS presented a different viewpoint in this blog post. As a security aficionado, I was interested in his reply to something Nielsen had written. I quoted it earlier, so here’s a recap of the part being referred to:

“Typically, masking passwords doesn’t even increase security, but it does cost you business due to log in failures.”

Montgomery responded:

“Nielsen’s probably right: It might be costing you business. The question is how much business? Security shouldn’t be the be-all, end-all goal. It’s there to serve the organization first and foremost. Viewing the cost of security controls with respect to the function it’s protecting is the correct perspective.

Well said Mr. Montgomery, I concur with your approach and I’m sure Dr. Nielsen does as well. It’s called compromise and I think that Nielsen may have already found a solution:

“Yes, users are sometimes truly at risk of having bystanders spy on their passwords, such as when they’re using an Internet cafe. It’s therefore worth offering them a checkbox to have their passwords masked; for high-risk applications, such as bank accounts, you might even check this box by default. In cases where there’s a tension between security and usability, sometimes security should win.”

Sounds like it might work, what do you think? Does it cover all possibilities? When do we know if we’re safe enough to lower security standards for increased usability?

Final thoughts

Until I read Nielsen’s blog post, I felt that masking passwords was just a necessary part of the process. Now I’m not so sure. It’s cumbersome and businesses could be losing customers. Yet on the flip side, not masking passwords is a potential security risk.

Disputes surrounding password usage continue to impress upon me the need for mainstream multi-factor authentication. But wishful thinking doesn’t help us right now. What’s your take on yet another usability versus security conflict?

High Scalability: eHarmony.com describes how they use Amazon EC2 and MapReduce

Mon, 29 Jun 2009 21:31:49 +0000

This slide show presents eHarmony.com experience (one of the biggest dating sites out there) in using Amazon EC2 and MapReduce to scale their service.

Go to the Slideshare presentation

High Scalability: Google App Engine plus Amazon AWS: Best of both worlds

Mon, 29 Jun 2009 21:02:59 +0000

Google App Engine (GAE) is focused on making development easy, but limits your options. Amazon Web Services is focused on making development flexible, but complicates the development process. Real enterprise applications require both of these paradigms to achieve success… What we really want is the flexibility of AWS and the simplicity of GAE..

High Scalability: HighScalability Rated #3 Blog for Developers

Mon, 29 Jun 2009 18:56:07 +0000

Hey we're moving up in the world, jumping from 19th place to 3rd place. In case you aren't sure what I'm talking about, Jurgen Appelo goes through this massive effort of ranking blogs according to Google PageRank, Technorati Authority, Alexa Rank, Google links, and Twitter Grader Rank.

Through some obviously mistaken calculations HighScalability comes out #3. Given all the superb competition I'm not exactly sure how that can be. Well, thanks for all the excellent people who contribute and all the even more excellent people that read. Now at least I have something worthy to put on my tombstone :-)

TechRepublic Network Administrator: Leaked Mojo SDK should speed WebOS development for Palm Pre

Mon, 29 Jun 2009 17:46:14 +0000

Derek Schauland thinks that the leaking of the Palm Pre SDK could be a good thing for WebOS development and help it compete better with Apple. Should Palm embrace the leak?

———————————————————————————–

Just a few days ago, the anticipated Mojo SDK for Palm’s Pre device was leaked to the Internet. Several mirrors appeared all over the place and presumably the app development for the Pre will be in high gear very soon. This is a key development because the SDK was not due to be released until sometime this fall, with the exception of a few key developers at a time participating in the Early Access Program.

As a Pre owner, I am excited that the development of apps for the phone may have been sped up exponentially with this leak, but I am also curious to know what Palm might do to put the wraps back on Mojo. Unfortunately, I am not of the developer mindset, so downloading the SDK and emulator will not do me much good. However, more applications for the Pre might move the competition with the iPhone 3Gs to the next level and perhaps squelch the argument that the volume of apps available for the iPhone makes it the superior device.

Note: There is a poll embedded within this post, please visit the site to participate in this post's poll.

Prior to this development, Palm decided to embrace the open source community and allow homebrew applications to work on the Pre, rather than updating WebOS to close off rooting and open source development. To their credit, I think this was the right move. Letting developers work with your device via an established program is a good idea, but not punishing anxious developers who just want to make your device better, to me, shows that Palm is serious about getting applications moving for the Pre.

Some home brew apps require rooting

The root image of WebOS for the Pre has been floating around the Internet for some time. The image allows home brew developers the opportunity to dig into the code. Installing the root image on the device can open up additional features for the phone.

If you are not comfortable with Linux, do not root your device; applications are removing the requirement of rooting and more will be available soon.

One application taking advantage of rooting the Pre is tethering, the ability to connect via your Pre to the Internet on your laptop. Sprint has not allowed tethering on the Pre and currently has no plans to do so, although the developer community and Palm’s willingness to help them may change Sprint’s position there.

Update: Palm released WebOS 1.0.4 for download on the Pre — apparently, to close a security hole which allowed home brew apps to be emailed and installed via a link in messages.

The installed home brew apps still appear to function without problems, but installation without the device has been turned off.

Anton Chuvakin - Security Warrior: Free Log Data For Research - Update

noreply@blogger.com (Dr Anton Chuvakin) — Mon, 29 Jun 2009 16:40:08 +0000

This WASL 2009 workshop reminded me that I always used to bitch that academic researchers use some antediluvian data set (Lincoln labs 1998 set used in 2008 “security research” makes me want to just curse and kick people in the balls, then laugh, then cry, then cry more…).

However, why are they doing it? Are they stupid? Don’t they realize that testing their “innovative intrusion detection” or “neural network-based log analysis” on such prehistoric data will not render it relevant to today’s threats? And will only ensure ensuing hilarity :-)

Well, maybe the explanation is simpler: there is no public, real-world source of logs that allows comparison between different security research efforts.

Correction! There wasn’t. And now there is!

I hereby acting on my promise to share my collection of real-world logs, mostly collected from systems in the honeynets I ran in 2004-2006. As of now, if you need logs for research, please contact me or get them directly here.

Here is the description of the collection currently shared (more to come!):

Size: 100MB compressed; about 1GB uncompressed

Date collected: 2006

Type: Linux logs /var/log/messages, /var/log/secure, process accounting records /var/log/pacct, other Linux logs, Apache web server logs /var/log/httpd/access_log, /var/log/httpd/error-log, /var/log/httpd/referer-log and /var/log/httpd/audit_log, Sendmail /var/log/mailog, Squid /var/log/squid/access_log, /var/log/squid/store_log, /var/log/squid/cache_log, etc.

License: public; use for whatever you want. Acknowledging the source is nice; Beerware license is even better.

Sanitization: No sanitization or modification was performed. No additional sanitization is required before use for research.

So, for now, if your research requires real-world logs with normal operation data, suspicious data, anomalous data and attack data – grab it here.

UPDATE: I have created a Google Group log-sharing to notify those interested about the shared logs. Please sign up here. The purpose of the group is to notify about new logs shared, discuss the shared logs, collect references to research that uses the logs, post requests for more logs, discuss the events observed in logs, etc.

UPDATE2: the logs are now hosted here, courtesy of one of my readers who prefers to remain anonymous. Thanks A LOT for hosting the logs! Despite the fact that the logs are fully public now, I suggest you still sign up for the Google group as I will announce new log sharing there.

Possibly related posts:

MDLog:/sysadmin: Debian Lenny 5.0.2 updated

Mon, 29 Jun 2009 11:44:45 +0000

The Debian project just announced the second update for its stable distribution “lenny” 5.0.2. Those installing regular updates from security.debian.org might not even notice this update, except for the version change to 5.0.2. As an interesting change, the debian-installer has been updated to allow the installation of the oldstable release (Debian 4.0 “etch”).

“The Debian project is pleased to announce the second update of its stable distribution Debian GNU/Linux 5.0 (codename “lenny”). This update mainly adds corrections for security problems to the stable release, along with a few adjustment to serious problems.
Please note that this update does not constitute a new version of Debian GNU/Linux 5.0 but only updates some of the packages included. There is no need to throw away 5.0 CDs or DVDs but only to update via an up-to- date Debian mirror after an installation, to cause any out of date packages to be updated.
…
New version of the debian-installer
The debian-installer has been updated to allow the installation of the previous stable release (Debian 4.0 “etch”) and to include an updated cdebconf package which resolves several issues with installation menu rendering using the newt frontend, including:
* explanatory text overlapping with the input box due to a height miscalculation
* overlapping of the “Go Back” button and the select list on certain screens
* suboptimal screen usage, particularly affecting debian-edu installations
The installer has been rebuilt to use the updated kernel packages included in this point release, resolving issues with installation on s390 G5 systems and IBM summit-based i386 systems.”

Release Announcement: http://www.debian.org/News/2009/20090627

Standalone Sysadmin: Encryption tools for Sysadmins

standalone.sysadmin@gmail.com (Matt) — Mon, 29 Jun 2009 09:22:32 +0000

Every once in a while, someone will ask me what I use for keeping passwords securely. I tell them that I use password safe, which was reccommended to me when *I* asked the question.

Other times, people will ask for simple ways to encrypt or store files. If you're looking for something robust, cross platform, and full featured, you could do a lot worse than TrueCrypt. Essentially, it hooks into the operating system's kernel and allows it to mount entire encrypted volumes as if they were drives. It also has advanced security methods to hide volumes, so that if searched, no volumes would be found without knowing the proper key. In addition, it has a feature that can be valuable if you are seized and placed under duress: in addition to the "real" password, a 2nd can be setup to open another volume, so that your captors believe that you gave them the correct information. Unreal.

So you see that truecrypt is an amazing piece of software. For many things, it's definitely overkill. Instead, you just want something light, that will encrypt a file and that's it. In this case, Gnu Privacy Guard is probably your best bet. I use it in our company to send and receive client files over non secure transfer methods (FTP and the like). With proper Key Exchange, we can be absolutely sure that a file on our servers came from our clients, and vice versa. If you're running a Linux distribution, chances are good you've got GPG installed already. Windows and Mac users will have to get it, but it's absolutely worth it, and the knowledge of how public key encryption works is at the heart of everything from web certificates to ssh authentication. If you want to learn more about how to use it, Simple Help has a tutorial on it, covering the very basic usage. Once you're comfortable with that, check out the manual.

I'm sure I missed some fun ones, so make sure to suggest what you use!

The Hive Archive: Links for 2009-06-28 [del.icio.us]

Mon, 29 Jun 2009 07:00:00 +0000

BashReduce — Richard Crowley’s blog
USAF slammed for pranging Predators on manual • The Register
A senior Pentagon official has delivered a stinging attack on the US Air Force, saying that its philosophy of using fully qualified human pilots to handle unmanned aircraft at all times has resulted in unnecessary, expensive crashes. By contrast, US Army drones with auto-landing equipment and cheaply-trained operators have an enviable record.

Chris Siebenmann: A theory on why users keep mailing specific people

Mon, 29 Jun 2009 06:20:07 +0000

A theory on why users keep mailing specific people

Like many places, we have several generic aliases that users mail about various issues. And, just like I expect happens everywhere, every so often users don't use those aliases and instead email some specific person here with their issue.

I recently came up with a theory for why this happens: it's easier to remember people (and then their email address) than it is to remember something impersonal. So people remember 'oh, I dealt with <X> last time to fix this', and they don't necessarily remember 'oh, I'm supposed to mail this random address'. And <X> gets more email.

(I am theorizing about this, but we know that humans have a fair amount of brainpower that's devoted to paying attention to other people (and we anthropomorphize like crazy), so it seems at least reasonable.)

Sysadmins may not see this as reasonable, but then I've got to point out that as successful sysadmins we are basically required to be good at memorizing computer-related trivia. Of course we can easily remember various abstract email addresses and keep them straight; we spend all day doing similar things, and we see the email addresses a lot more than the typical person does to boot so they're more familiar to us.

Unfortunately, I can't think of anything useful to do with this theory. It does make me wonder if anyone has experimented with deliberately anthropomorphizing their generic aliases and support systems, and if so if it did any good.

(11 comments.)

TechRepublic IT Security: The basics of secure admin privilege use with Unix

Mon, 29 Jun 2009 04:50:59 +0000

Sometimes, it’s worthwhile to get back to basics. Read about the basics of secure administrative privilege use on Unix-like systems.

Some of my readers may find this a very basic article, presenting information that they already know like the backs of their hands. The frequency with which I see people — and even entire OS development teams — violating basic, common security sense with regard to secure administrative privilege use on Unix-like systems prompts me to explain those basics here, though. That does not necessarily mean they are stupid, of course; some of the “basics” are not at all obvious.

The root account is probably the best place to start.

Using the root account

The standard administrative superuser account on Microsoft Windows is called Administrator. On Unix-like systems, it is called root instead. It’s normally a bad idea to use an administrative account for anything that you can do with a less privileged account, because any time you use any user account at all you expose that particular account to potential threats if the software run under that account’s privileges has a vulnerability that can allow someone to compromise the user account.

If you use Firefox to browse the Web, and it turns out Firefox has some kind of scripting vulnerability that allows a malicious script on a Website to install a backdoor on your system, how vulnerable you are depends to some degree on what user account you use:

With a user account that has no administrative privileges, your user account may be compromised — but the security cracker with access to it via the installed backdoor will only be able to access exactly what that user account can access.
If you are logged in as the root user while using Firefox, the security cracker with access to the account via the installed backdoor may now have access to the entire system, because the root user has administrative privileges over everything.

Obviously, you have to access the administrative account from time to time when you are the system administrator. The key is to use those elevated privileges only for the specific tasks that require them, and sign back out of the root account immediately after that. On a server, this usually means logging in as root at a virtual TTY console, doing whatever you need to get done as root, then typing exit or <Ctrl>+<D> to log out of the root account again.

Using su

The “substitute user” or “switch user” command (also sometimes identified as “superuser”), su, allows convenient and secure access to the root account without having to log out of the current logged in user account on Unix-like systems. It is also commonly used to access normal, unprivileged user account environments from within a root account session by specifying the account whose user environment one wants to access.

Probably the most common use of su these days is as a means of accessing the root account to perform administrative tasks without leaving an X Window System session. One can just open a terminal emulator and type su, enter the password when prompted, and start working with root privileges.

This can provide additional security for remote connections too; the administrator can configure a system to disallow SSH logins as root, requiring a user to connect as some other user account and use su to elevate privileges. This is a quick and easy way to stop remote brute force attacks agains the root account’s password. Some systems, such as FreeBSD, actually install SSH configured that way by default.

The convenience of being able to simply and securely achieve root privileges in a terminal emulator window — without having to log out of your X Window System session, or even switch to a TTY console where you can’t cut-and-paste the way you can in X — is a great example of how interface design is security design. With the right security tool interface in place, the user is actually encouraged to do The Right Thing; in this case, the user is encouraged to log in as an unprivileged user for everyday tasks, rather than log in as root and stay there so a bunch of jumping through hoops doesn’t have to go on before he or she can perform some trivial administrative task.

Using sudo

There is a great tool for securing and logging the behavior of users who need to perform limited administrative tasks called sudo (pronounced “sū dū”). Quite a lot can be done with this tool, and as such quite a lot can be said about it. It could, arguably, warrant a book of its own.

It is best used to allow specific users with specific, well-defined administrative privilege needs to do what they need to do, and only that; it also makes logging the activities of such users a breeze. If a user needs to be able to use the portaudit tool, he or she can be given access to its functionality that requires administrative privileges without exposing the rest of the capabilities reserved for the root user to the same user in the process. The sudo tool is essentially an excellent tool for delegating administrative tasks in a limited, secure, and logged (and therefore auditable) manner, so you don’t have to start handing out the root password to everybody who wants to be able to mount a filesystem in the /mount directory.

What it is not as good at is replacing the root user account entirely. While the fact it allows easy logging of all administrative acitivity separated by actual user, rather than grouping all administrative task logging under the single heading of the root user, the same can be accomplished by creating separate superuser accounts; FreeBSD’s toor user serves as an excellent example of this.

Of special concern is the fact that, if someone manages to compromise your unprivileged user account that has unlimited access to administrative privileges via sudo, that malicious security cracker may then find it significantly easier to access those privileges as well. This is an especially bad problem for systems where sudo is configured to allow passwordless use, of course.

The most worrisome problem with using it as a root replacement, however, is its complexity. As pointed out in Security, complexity, and the GUI environment, complexity is the enemy of security:

Every time you increase the complexity of a system, you increase the opportunity for something to go wrong in its design. The more lines of code in your system, the more opportunities there are to introduce bugs when developing the system; the more bugs there are, the more opportunities you have for bugs that introduce security vulnerabilities.

Because sudo is designed to do so much more than just give you easy access to administrative capabilities — to provide fine-grained delegation of administrative privileges to otherwise unprivileged users. Even configuring it properly can turn into a precarious exercise in safely navigating complexity, depending on how ambitious your needs get; many programs can be used to execute additional or external commands, and if such a user is given root-level access to such a program via sudo without restrictions that prevent arbitrary execution of external commands, those commands could be executed with root privileges.

As Razvan Stoica explained in Sudo considered harmful, even a security conscious sysadmin who generally does things “right” for a given system can run afoul of the security damaging complexity of applying sudo too broadly to the problem of making system administration easy.

What else?

This is just a starting point. The most important part of security is being an active, security-minded thinker, and the “basics” of secure administrative privilege use should give you a foundation on which to build a deeper understanding of other aspects of secure system administration. Subjects of interest, that have not been addressed above, include an open-ended list of tools and techniques like the OpenSSH daemon’s PermitRootLogin configuration option, system design considerations such as the quirks of the suid bit in file permissions, and an expanded understanding of the topics already covered above, exemplified by the proper management of the wheel group.

Such topics are incredibly important for system administrators, obviously. Even for end users, though, understanding topics like these can be invaluable. The better you understand your system, the better you can ensure you will not be part of the problem with system security.

TaoSecurity: Simpler IP Range Matching with Tshark Display Filters

noreply@blogger.com (Richard Bejtlich) — Sun, 28 Jun 2009 16:40:13 +0000

In today's SANS ISC journal, the story IP Address Range Search with libpcap wonders how to accomplish the following:

...how to find SYN packets directed to natted addresses where an attempt was made to connect or scan a service natted to an internal resource. I used this filter for addresses located in the range 192.168.25.6 to 192.168.25.35.

The proposed answer is this:


tcpdump -nr file '((ip[16:2] = 0xc0a8  and ip[18] = 0x19 and ip[19] > 0x06)\
and (ip[16:2] = 0xc0a8 and ip[18] = 0x19 and ip[19]  0x23) and tcp[13] = 0x02)'

I am sure it's clear to everyone what that means!

Given my low success rate in getting comments posted to the SANS ISC blog, I figured I would reply here.

Last fall I wrote Using Wireshark and Tshark display filters for troubleshooting. Wireshark display filters make writing such complex Berkeley Packet Filter syntax a thing of the past.

Using Wireshark display filters, a mere mortal could write the following:


tshark -nr file 'tcp.flags.syn and (ip.dst > 192.168.25.6 and ip.dst  192.168.25.35)'

Note that if you want to be inclusive, change the > to >= and the to = .

To show that my filter works, I ran the filter against a file with traffic on my own 192.168.2.0/24 network, so I altered the last two octets to match my own traffic.


$ tshark -nr test.pcap 'tcp.flags.syn and (ip.dst > 192.168.2.103 and ip.dst  192.168.2.106)'

137 2009-06-28 16:21:44.195504 74.125.115.100 -> 192.168.2.104 HTTP Continuation or non-HTTP traffic

You have plenty of other options, such as ip.src and ip.addr.

Which one do you think is faster to write and easier to understand?

Richard Bejtlich is teaching new classes in Las Vegas in 2009. Regular Las Vegas registration ends 1 July.

TaoSecurity: Effective Digital Security Preserves Long-Term Competitiveness

noreply@blogger.com (Richard Bejtlich) — Sun, 28 Jun 2009 14:57:06 +0000

Yesterday I mentioned a speech by my CEO, Jeff Immelt. Charlie Rose also interviewed Mr Immelt last week. In both scenarios Mr Immelt talked about preserving long-term competitiveness. Two of his themes were funding research and development and ensuring the native capability to perform technical tasks.

It occurred to me that digital security is reflected in both themes. In Crisis 0: Game Over I asked I'm sure some savvy reader knows of some corporate espionage case that ended badly for the victim, i.e., bankruptcy or the like? I got a few interesting cases, but I believe the net result is that it is difficult to find examples where an intrusion or breach was so devastating that it ended up destroying the victim organization.

This makes sense once you reflect on it. Why would a mature, thoughtful intruder seek to destroy his victim, if the purpose of his mission is to conduct espionage on behalf of a competitor or intelligence service? Destroying the victim renders it useless as a source for stealing intellectual property gained by the victim's research and development. In the foreign intelligence case, almost all operators prefer to keep a source active, even in wartime when you might think that destruction is the ultimate goal.

Taking this line of reasoning to its natural conclusion, we can see that digital security can be considered a means to preserve long-term competitiveness, particularly for organizations that seek to drive internal growth via investing in research and development. Such an organization is a natural target for competitors who find it immensely cheaper to steal intellectual property, rather than fund their own.

The problem is showing those who make budgetary and management decisions that digital security has a real role in loss prevention. I've written a lot about intellectual property and digital security, but it is exceptionally difficult to tie individual intrusions to real impact. How does pervasive theft of intellectual property (IP) manifest itself? In commercial cases, perhaps it would appear as a loss of sales to rivals who make similar or duplicate products based on stolen IP. Would the victim organization even know these lost or declining sales were the result of IP theft?

Even if the victim identified the stolen IP, could it be traced back to one or more intrusions, or would it be considered the consequences of product reverse engineering by competitors? The bottom line could be that the victim is still in business, but the double-digit growth and expanding market share it craves are reduced to single-digit growth and eroding market share.

It's a waste of time to use terms like "ROI" or "ROSI" when talking to managers or business people. It is usually impossible to fully explain, from loss to impact, the IP theft cases like the one I described in Intellectual Property: Develop or Steal, i.e., spend $10 million over 10 years on a product, then watch the Chinese duplicate it for $1.4 million in 6 months after stealing the IP. More often than not, the victim of IP theft simple whithers, wondering why its competitive advantage is not what it expected it to be. It's time to get managers and business people to think in terms of long-term competitiveness.

Clearly Mr Immelt has determined that it is not in his company's best interest, nor in the interests of the country, for the US to be underfunding R&D or outsourcing everything overseas. We security professionals need to adopt this line of reasoning to emphasize how effective digital security preserves long-term competitiveness.

By the way, you might be wondering if I can prove there is an impact to IP theft. I look at the question this way. If there were no impact to IP theft, why would economic and national competitors fund teams to steal IP? You might argue that IP thieves can duplicate and sell products at prices lower than the IP owner could afford, thereby serving a new market. If that were true, why would IP owners file patents? Clearly there is value in IP, so stealing it lessens the value available to the IP owner.

I use a variant of this argument when I encounter asset owners who claim there is no impact associated with an intrusion. My reply is usually this: If there is no impact, then why operate the asset? Retire it.

Richard Bejtlich is teaching new classes in Las Vegas in 2009. Regular Las Vegas registration ends 1 July.

Steve Kemp's Blog: My hovercraft is full of eels.

Sun, 28 Jun 2009 13:01:46 +0000

Recently I've been seeing an awful lot more bounced mail addressed to my domains, to the extent that I now wonder whether they are deliberate "attacks".

Over the past four or five years I'd expect to receive one joe-job attack every six months. Over the past two that's risen to once every two months. For the past two months its been once a week.

I run several domains on my Xen guest, and most of those domains rarely have mail received, so there are only a few localparts. (A "localpart" is the bit before the @ sign in an email address.)

My main domain is steve.org.uk and unfortunately this was historically setup with "catchall" behaviour. I used that wildcard expansion pretty seriously so I had localparts such as "slashdot.org", "lwn.net", etc. Over time I've stopped making up new addresses and just stuck with "steve".

Still I'd never quite gotten round to enumerating all valid localparts, instead I tried to mitigate against these rare bounce storms with various simple hacks. For example the following procmail recipe to file away bounces:

#  Bounces
#
:0:
*(Return-Path:).*(<>)
.Automated.bounces/

However this doesn't work as well as it used to - too many ~~idiots~~ people are using challenge/response systems so I'll receive a reply to a mail I didn't send which doesn't look like a bounce (ie. There is a real envelope sender.)

In short blocking bounces by detecting an empty envelope sender is not a complete strategy these days. I started down the heuristic path blocking mail to "unlikely" localparts via patterns such as:

[0-9]@        DENY  Localparts never end in digits
,             DENY  Localparts never contain a comma
|             DENY  Localparts never contain PIPES.
^([^a-zA-Z])  DENY  Localparts start with a-z/A-Z
"             DENY  Quotes are never used in accounts on this system:
'             DENY  Quotes are never used in accounts on this system:

That was actually a simple change to make, via the addition of a new QPSMTPD plugin and it managed to block a lot of the bounceback spam - regardless of the envelope sender. For example:

IP:84.45.254.18    sender:<> Recipient:treacherously9@steve.org.uk
IP:203.202.253.252 sender:<> Recipient:envoyz0@steve.org.uk

Blocking "unlikely" localparts wasn't perfect, but without implementing BATV or enumerating valid localparts there wasn't too much else that I could do. In terms of numbers yesterday I blocked just over 18,500 messages with these six rules.

I also wrote a couple of cronjobs to look at the contents of the Automated.bonces folder so that I could add per-user rejections on the specific addresses being received - with some whitelisting.

(For example if I received 20+ bounces to fluffy32qp@steve.org.uk within the space of ten minutes I'd drop further mails to that address automatically.)

Anyway enough is enough. Today I woke up to just over 40,000 replies to mails I didn't send. I've now scanned my mail directories for all the email addresses I've ever used and will now only accept mail destined to those localparts.

Thankfully it turned out that since 1999 (when steve.org.uk was registered) I've only used about 150 distinct localparts, and many of those are now obsolete. So hopefully I'll now have less of a problem.

It seems to be paying off already:

62.193.234.95   wpc0505.host7x24.com  <>  virtual_rcpt_ok
    901     mail to subtotalingxa@steve.org.uk not accepted here (#5.1.1)

65.99.223.234   cobra.compukey.net    <>  virtual_rcpt_ok
     901     mail to suctionsw@steve.org.uk not accepted here (#5.1.1)

207.44.156.81   box19.fuitadnet.com   <>   virtual_rcpt_ok
     901     mail to reappearcum@steve.org.uk not accepted here (#5.1.1)

In the future this means I could still get flooded with bounces, but there will be two outcomes:

The bounces will not hit valid localparts and will be dropped easily, quickly, and cheaply.
The bounces will hit valid localparts:
- Real bounces will end up in Automated.bounces/
- Challenge/Response things will still reach me. Sigh.

Still this is progress and I can steal some ideas from this great spam filtering service (ahem) to improve the handling of those! (I explicitly chose to use a similar but different system for my personal mails. Even though my support system is on another box I want to avoid problems where failures requiring human intervention are swallowed in the same way that the original one was. Those kind of reasons mandate a similar system but different implementation.)

I guess I could publish some of the qpsmtpd plugins I use locally virtual_rcpt_ok, virtual_badusers, rcpt_pattern_test, etc. Then again most people who do funky things with qpsmtpd will have plenty of choice already.

ObFilm: Monty Python's Flying Circus. (OK technically not a film. Sums up my mood though.)

Wolfgang Lonien: Samsung SyncMaster 2433BW

Sun, 28 Jun 2009 10:57:05 +0000

Got this one yesterday from a big local electronics chain. It’s huge, and it’s good, and it’s cheap. You can read the English or the German product pages, or compare prices here in Germany (and read buyers’ comments as well).

Really, a nice one - it was the best one I saw at that store, and affordable as well - so we took it. Now I have to download Peach again, in HD this time

The specs in short: only VGA and DVI inputs, 1920×1200 resolution, no fancy stuff like speakers etc. Just your basic TN display, but wow, what a nice one,

And it’s handled by my integrated Nvidia 6150 graphics chip perfectly, even when connecting it over my small Belkin KVM switch. Very crisp and clear. So if you own a good IGP mainboard, you won’t need an extra graphics card up to this resolution. Recommended.

Want a screenshot showing the real estate of 1920×1200 pixels, with an 800×600 browser window on it? Here you have it. And no, the background picture is not from “Lost”, but from the island of Langkawi. That one’s recommended as well, if you can stand the heat

The Debian User: Installer and benchmarks

Sun, 28 Jun 2009 09:48:15 +0000

Debian developer Kenshi Muto backported the latest 2.6.30 kernel plus firmware for Lenny. So if you have to deal with newer hardware and need the latest kernel, read his blog about it. You can also download the installer images from a French mirror, which could be faster because of better bandwidth. Thanks, Kenshi!

Heise Open Source and the H Open have news about a new benchmark for Linux desktops. You can read them in German or in English. I will have a look at them, and at the Phoronix Test Suite. We should make those a standard when comparing hard- and software IMHO. Thanks guys!

TechRepublic Network Administrator: Time to reconsider security zones in system and network design

Sun, 28 Jun 2009 08:30:38 +0000

As IT professionals balance many responsibilities, we may omit certain fundamentals that are made easier in the current technology landscape through multiple layers of abstraction, virtualization, and management. IT Jedi Rick Vanover suggests that it’s a good time to rethink security zones.
—————————————————————————————————————

The current inventory of networks and servers has many layers of abstraction, virtualization, and management in today’s data center. Recently in a discussion with independent security expert Edward Haletky, I discovered it is definitely time to revisit how security zones are provisioned in new and existing network infrastructure.

Edward pointed out that many administrators, myself included, are crossing security zones without even knowing it with the various layers of management and abstraction that are in use today. The security zones that I am referring to are the classes of service for various levels of a network infrastructure.

Take for example a typical server in today’s data center and also assume it is a virtual machine host. This particular server may have the following network attributes: a hardware management interface such as an HP Integrated Lights-Out management processor, the operating system management interface, the virtualization layer migration interface, a storage interface for a system such as iSCSI, and a number of virtual machines all on separate VLANs. In this example, the single piece of equipment interacts with no fewer than five security zones before the actual systems come into play.

This discussion brought me to consider that with technologies such as VLANs and options made available through virtualization, it is prime time to rethink where everything resides. Security issues aside — it simply makes sense to separate these network presence points where they are classified as security zones. Performance reasons will also benefit, as I mentioned in a prior post about iSCSI network separation.

How do you approach different security zones on networks? Are VLANs enough — or are fully separate switching environments adequate for your requirements? This area is very compliance- and requirement-driven, so there is no clear answer. Share your comments below on this area where we all can likely improve.

Chris Siebenmann: How we solve the multiuser PHP problem

Sun, 28 Jun 2009 04:56:12 +0000

How we solve the multiuser PHP problem

For those that have not run into it, the multiuser PHP problem is this: you have a shared, multiuser web server where many people have their own web pages, and some of them want to have PHP-based applications. This is a perfectly reasonable thing to ask for, but PHP runs in the web server under the web server's UID and to have at least a pretense of accountability and security you want a user's dynamic stuff to run as that user, so at most they can blow up their own files and compromise their own account. And PHP is just the (large) tip of the iceberg for this problem; there are lots of web apps that need some special Apache configuration or module or what have you (and let's not even get started about database requirements).

Our solution to this is simple but brute force; we have users run their own web servers and then hide them behind a reverse proxy from our main web server. To help people out, we provide a standard Apache configuration with all of the usual LAMP features turned on (and an optional MySQL database instance) and some scripts to set it up for you and control it.

(People don't have to use the configuration, or even run Apache; we have a couple of people using mzscheme instead.)

Running the entire web server as the user obviously provides all of the security stuff that we wanted. Using a reverse proxy setup keeps all of this transparent by hiding the existence of all of these user-run web servers; even when content is actually being served by a separate web server, it still has URLs on our main web server. Among other things, this means that you can start and stop using a user-run web server without having to change any URLs.

This scheme does have some drawbacks. One of them is that we wind up with a lot of inactive Apache processes and so on running on the web server machine, since each user-run web server is running a few even when no one is using it. (I don't think that load limiting is harder; since the main Apache server has to proxy for everyone, its limits act as a global limit across all of the user-run web servers.)

(Another one is that the software running on user-run web servers has to be reverse proxy aware, as do the people writing it, but that's a topic for another entry.)

(Necessary disclaimer: as before, this system is the work of many people here.)

(One comment.)

TaoSecurity: Posts to Read Elsewhere

noreply@blogger.com (Richard Bejtlich) — Sat, 27 Jun 2009 22:19:27 +0000

I'm not a big fan of just publishing links to other people's stories, but there's a few that I really like this week. Please consider checking these out:

Nate Richmond wrote Building an IR Team: People and Building an IR Team: Organization. These posts are gold for anyone trying to build an IR team on their own, or trying to benchmark against an expert's recommendation. Keep writing Nate!

Alec Waters caught my attention with his post Prevention Eventually Fails, part one. Anyone who read my first book recognizes my catchphrase "Prevention eventually fails." Alec's posts look interesting!

My CEO delivered a great speech this week, viewable at American Renewal: Immelt addresses Detroit Econ Club and readable at Text of Immelt's Speech. This caught my eye:

In some areas, we have outsourced too much. We plan to "insource" capabilities like aviation component manufacturing and software development. These are the things we will be working on in Michigan. This will make us faster and more competitive over the long term.

I totally agree -- internal security staff matters.

Richard Bejtlich is teaching new classes in Las Vegas in 2009. Regular Las Vegas registration ends 1 July.

TaoSecurity: Black Hat Budgeting

noreply@blogger.com (Richard Bejtlich) — Sat, 27 Jun 2009 21:44:38 +0000

Earlier this month I wondered How much to spend on digital security. I'd like to put that question in a different light by imagining what a black hat could do with a $1 million budget.

The ideas in this post are rough approximations. They certainly aren't a black hat business plan. I don't recommend anyone follow through on this, although I am sure there are shops our there who do this work already.

Let's start by defining the mission of this organization, called Project Intrusion (PI). PI is in "business" to steal intellectual property from organizations and sell it to the highest bidders. In the course of accomplishing that mission, PI may develop tools and techniques that it could sell down the food chain, once PI determines their utility to PI has sufficiently decreased.

With $1 million in funding, let's allocate some resources.

Staff. Without people, this business goes nowhere. We allocate $750,000 of our budget to salaries and benefits to hire the following people.

The team leader should have experience as a vulnerability researcher, exploit developer, penetration tester, enterprise defender, and preferably an intelligence operative. The leader can be very skilled in at least one speciality (say Web apps or Windows services) but should be familiar with all of the team's roles. The team leader needs a vision for the team while delivering value to clients. $120,000.

The team needs at least one attack tool and technique developer for each target platform or technology that PI intends to exploit. PI hires three. One focuses on Windows OS and client apps, one on Web apps, and one on Unix and network infrastructure. $330,000.

The team hires two penetration operators who execute the team leader's mission directives by using the attack tools and techniques supplied by the developers. The operators penetrate the target and establish the persistence required to acquire the desired intellectual property. $180,000.

The team hires one intelligence operative to direct the penetration operators attention toward information of value, and then assess the value of exfiltrated data. The intel operative interfaces with clients to make deals. $120,000.

Technology. The team will need the following, for a total of $200,000.

Lab computers running the software likely to be attacked during operations.

Operations computers from which the penetration operators run attacks.

Network connectivity and hosting for the lab computers and operations computers, dispersed around the world.

Software required by the team, since many good attack tools are commercial. MSDN licenses are needed too. There's no need to steal these; we have the budget!

Miscellaneous. The last $50,000 could be spent on incidentals, bribes, team awards, travel, or whatever else the group might require in start-up mode.

If the attack developers manage to make enough extra money by selling original exploits, I would direct the funds to additional penetration operators. It would take about six of them to support a sustainable 24x7 operation. With only two they would need to be careful and operate within certain time windows.

So what is the point of this exercise? I submit that for $1 million per year an adversary could fund a Western-salaried black hat team that could penetrate and persist in roughly any target it chose to attack. This team has the structure and expertise to develop its own attack methods, execute them, and sell the results of its efforts to the highest bidders.

This should be a fairly scary concept to my readers. Why? Think about what $1 million buys in your security organization. If your company is small, $1 million could go a long way. However, when you factor in all of the defensive technology you buy, and the salaries of your staff, and the scope of your responsibilities, and so on, quickly you realize you are probably out-gunned by Project Intrusion. PI has the in-house expertise to develop its own exploits, keep intruders on station, and assess and sell the information it steals.

Worse, PI can reap economies of scale by attacking multiple targets for that same $1 million. Why? Everyone runs Windows. Everyone uses the same client software. Everyone's enterprise tends to have the same misconfigurations, missing patches, overworked staff, and other problems. The tools and techniques that penetrate company A are likely to work against company B.

This is why I've always considered it folly to praise the Air Force for standardizing its Windows deployment with supposedly secure configurations. If PI looks at its targets and sees Windows, Windows, some other OS that might be Linux or BSD or who knows what, Windows, Windows, who do you think PI will avoid?

It's all about cost, on the part of the attacker or defender. Unfortunately for defenders, it's only intruders who can achieve "return on investment" when it comes to exploiting digital security.

Richard Bejtlich is teaching new classes in Las Vegas in 2009. Regular Las Vegas registration ends 1 July.

Planet SysAdmin

the life of a sysadmin.: Bacula, gossip, advice

TaoSecurity: NSA to "Screen" .gov Now, I Predict .com Later

TaoSecurity: Review of Hacking Exposed: Windows, 3rd Ed Posted

High Scalability: It Must be Crap on Relational Dabases Week

High Scalability: Product: Hbase

High Scalability: Hypertable is a New BigTable Clone that Runs on HDFS or KFS

High Scalability: Product: Facebook's Cassandra - A Massive Distributed Store

High Scalability: Product: Project Voldemort - A Distributed Database

High Scalability: Anti-RDBMS: A list of distributed key-value stores

Standalone Sysadmin: Update with the hiring and an upcoming blog update

A Year in the Life of a BSD Guru: Collaboration

CiscoZine: Cisco introduces four new certifications!

The Unix Blog: Unix Pax to the Rescue - Extracting Absolute Path Names from tar Archives

The Unix Blog: Message to Sun: Bring Back the Sun SPARC Workstations

Adnans Sysadmin/Dev Blog: Links for 2009-07-01 [del.icio.us]

Anton Chuvakin - Security Warrior: Links for 2009-07-01 [del.icio.us]

High Scalability: Podcast about Facebook's Cassandra Project and the New Wave of Distributed Databases

Chris Siebenmann: Finding out when a command in a pipeline fails

Finding out when a command in a pipeline fails

High Scalability: Latency is Everywhere and it Costs You Sales - How to Crush it

The Lone Sysadmin: Mistakes

Anton Chuvakin - Security Warrior: Monthly Blog Round-Up – June 2009

HolisticInfoSec.org: Malzilla: Exploring scareware and drive-by malware

CiscoZine: OSPF Virtual Link

Standalone Sysadmin: New Article: Manage Stress Before It Kills You

Adnans Sysadmin/Dev Blog: Links for 2009-06-30 [del.icio.us]

Anton Chuvakin - Security Warrior: Links for 2009-06-30 [del.icio.us]

TechRepublic Network Administrator: Storage in the cloud: Requires a different mindset

Chris Siebenmann: A Unix irritation: pipeline status

A Unix irritation: pipeline status

TechRepublic IT Security: Stay out of Bozeman

The Lone Sysadmin: Size Labels for Virtual Environments – A Proposal

A Unix Sysadmin's Journey: Migrating a Zone to a Different Machine on Solaris 10

Anton Chuvakin - Security Warrior: Vulnerability Scanning and Clouds/SaaS/IaaS/PaaS

High Scalability: Hot New Trend: Linking Clouds Through Cheap IP VPNs Instead of Private Lines

MDLog:/sysadmin: OSBridge: Configuration Management Panel

Standalone Sysadmin: Fun with VMware ESXi

MDLog:/sysadmin: FindMyHosting Review

Conclusion

Adnans Sysadmin/Dev Blog: Links for 2009-06-29 [del.icio.us]

Anton Chuvakin - Security Warrior: Links for 2009-06-29 [del.icio.us]

Chris Siebenmann: More on why users keep mailing specific people

More on why users keep mailing specific people

the life of a sysadmin.: 1246317421 seconds since the epoch...

TechRepublic IT Security: Masking passwords: Why it's not a good idea

High Scalability: eHarmony.com describes how they use Amazon EC2 and MapReduce

High Scalability: Google App Engine plus Amazon AWS: Best of both worlds

High Scalability: HighScalability Rated #3 Blog for Developers

TechRepublic Network Administrator: Leaked Mojo SDK should speed WebOS development for Palm Pre

Some home brew apps require rooting

Anton Chuvakin - Security Warrior: Free Log Data For Research - Update

Workshop on the Analysis of System Logs (WASL) 2009 CFP

Free Log Data For Research

MDLog:/sysadmin: Debian Lenny 5.0.2 updated

Standalone Sysadmin: Encryption tools for Sysadmins

The Hive Archive: Links for 2009-06-28 [del.icio.us]

Chris Siebenmann: A theory on why users keep mailing specific people

A theory on why users keep mailing specific people

TechRepublic IT Security: The basics of secure admin privilege use with Unix

Using the root account

Using su

Using sudo

What else?

TaoSecurity: Simpler IP Range Matching with Tshark Display Filters

TaoSecurity: Effective Digital Security Preserves Long-Term Competitiveness

Steve Kemp's Blog: My hovercraft is full of eels.

Wolfgang Lonien: Samsung SyncMaster 2433BW

The Debian User: Installer and benchmarks

TechRepublic Network Administrator: Time to reconsider security zones in system and network design

Chris Siebenmann: How we solve the multiuser PHP problem

How we solve the multiuser PHP problem

TaoSecurity: Posts to Read Elsewhere

TaoSecurity: Black Hat Budgeting