Installing Ubuntu 16.04 on a ZFS root filesystem

One of the major new pieces of functionality in Ubuntu 16.04 (Xenial) is built in support for ZFS filesystems.  I was disappointed to learn that ZFS support is not actually built into the installer itself, leaving you to piece things together yourself.  I did find a few good tutorials online for this, but they all seem to be missing a few pieces.  I'm hoping that this guide will be a bit more complete for people.

It is based on a combination of the following two guides:

• https://github.com/zfsonlinux/pkg-zfs/wiki/HOWTO-install-Ubuntu-16.04-to-a-Native-ZFS-Root-Filesystem
• http://dotfiles.tnetconsulting.net/articles/2016/0327/ubuntu-zfs-native-root.html

Booting Into the Install Environment

Setting up a ZFS filesystem requires a full set of userspace tools rather than the limited set included within the actual installer.  Due to this, we're going to boot a Ubuntu Desktop live CD, and do a manual installation within its root filesystem.  I found that the easiest way to do this was to boot the live CD, set a password for the ubuntu user, and SSH into the machine from a remote box.  This way it is much easier to copy/paste commands from the webpage.

To start off, we need to install the ZFS tools and debootstrap which we will use for actually installing the operating system:

apt-add-repository universe
apt-get update
apt-get install --yes zfsutils-linux debootstrap

Partitioning the Disks

Next, we partition the disks.  For now we're going to assume a two disk system which is just doing striping, but you can change as desired:

parted -- /dev/sda mklabel msdos Y mkpart primary zfs 0% 100%
parted -- /dev/sdb mklabel msdos Y mkpart primary zfs 0% 100%

Device naming during bootup on Linux isn't always static, so we want to reference the disks through a static identifier like disk ID (/dev/disk/by-id), unfortunately Grub fails to properly identify the devices from the zfs command output, so we need this hack for now to make sure that update-grub will correctly identify the disks.  Specifically, it forces creation of symlinks in /dev which map to the device names in /dev/disk/by-id.  Some additional details about the issue can be found in this grub2 launchpad bug.

echo 'KERNEL=="sd*[!0-9]", IMPORT{parent}=="ID_*", SYMLINK+="$env{ID_BUS}-$env{ID_SERIAL}"
> KERNEL=="sd*[0-9]", IMPORT{parent}=="ID_*", SYMLINK+="$env{ID_BUS}-$env{ID_SERIAL}-part%n"' > /etc/udev/rules.d/90-zfs.rules
udevadm trigger

Create Your Zpool

Now we actually create the zpool and import it to /mnt:

zpool create -m none -o ashift=12 -O compression=lz4 rpool /dev/sda1 /dev/sdb1
zfs create -o mountpoint=/ rpool/root
zpool export rpool
zpool import -d /dev/disk/by-id -R /mnt rpool

So here we create the zpool by device name, and then re-import it by device ID while mounting at /mnt.

And then create various partitions off of the root filesystem:

zfs create -o setuid=off rpool/root/home
zfs create -o mountpoint=/root rpool/root/home/root
zfs create -o canmount=off -o setuid=off -o exec=off rpool/root/var
zfs create -o com.sun:auto-snapshot=false rpool/root/var/cache
zfs create rpool/root/var/log
zfs create rpool/root/var/spool
zfs create -o com.sun:auto-snapshot=false -o exec=on rpool/root/var/tmp

We break out these various subdirectories so we have the ability to optionally enable/disable compression, support for setuid/exec, and various other options.  We can also choose to limit the maximum size of each mount independently.

Install Ubuntu

The install of Ubuntu itself is pretty straightforward:

debootstrap the OS

debootstrap xenial /mnt

zfs set devices=off rpool

grep -v cdrom /etc/apt/sources.list > /mnt/etc/apt/sources.list
cp /etc/udev/rules.d/90-zfs.rules /mnt/etc/udev/rules.d/90-zfs.rules

You'll notice at the end here we copy our hacky udev rule to the new filesystem.  The "devices=off" option we set here disables the ability to create device nodes in the filesystem.  This works since /dev is actually a devtmpfs partition of its own.

Configure the network interface

export INTERFACE=$(ip addr list | grep ^[0-9]: | grep -v "lo" | awk {'print $2'} | cut -d ":" -f 1)

echo test > /mnt/etc/hostname

echo 127.0.1.1  >> /mnt/etc/hosts

echo "auto $INTERFACE

iface $INTERFACE inet dhcp" > /mnt/etc/network/interfaces.d/$INTERFACE

Enter the chroot for some final setup

mount --rbind /dev /mnt/dev

mount --rbind /proc /mnt/proc

mount --rbind /sys /mnt/sys

chroot /mnt /bin/bash --login

locale-gen en_US.UTF-8

echo 'LANG="en_US.UTF-8"' > /etc/default/locale

apt-get update

apt-get install --yes zfsutils-linux zfs-initramfs grub-pc linux-image-generic ssh

dpkg-reconfigure tzdata

update-initramfs -c -k all

Set Up Grub's configuration

sed -i 's/^\(GRUB_CMDLINE_LINUX_DEFAULT\)=.*/\1=""/g' /etc/default/grub

sed -i 's|^\(GRUB_HIDDEN_TIMEOUT=.*\)|#\1|g' /etc/default/grub

sed -i 's/^\(GRUB_CMDLINE_LINUX\)="\(.*\)"/\1="boot=zfs \2"/g' /etc/default/grub

ln -s /proc/mounts  /etc/mtab

update-grub

rm /etc/mtab

Set a root password and exit the chroot

passwd root

exit

Install the grub bootloader

grub-probe /mnt

grub-install --root-directory=/mnt /dev/sda

grub-install --root-directory=/mnt /dev/sdb

Reboot into the environment

reboot

Eating your own dog food

I got this unsolicited e-mail today:

Hi,

Hope you doing well. I am writing in regards to see if there is any possibility for us to work with your company. We are leading provider of B2B, B2C and B2G lists and excellent list compiler with highest delivery rate and data appending solutions. Our opt-in email database of Key decision makers can be used for your online promotion, brand awareness and to generate potential leads.

We build list according to your requirements based on your targeted business. We are specialized in Email Campaign and Data Append Solution, where you can add your clients missing data (email, fax etc).

Apparently they provide opt-in only mailing lists.  Shame they don't use them themselves.

Whoops!

My home server is configured to send me daily e-mails with any security events. Typically this means SSH brute force login attempts, and I occasionally take the opportunity notify people that their servers were breached and are being used for attacks.

Today I found something amusing in the logs. Apparently some idiot hacker mixed up their username and password files for their attack:

Feb 28 02:32:53 buddha sshd[57691]: Invalid user QFhGj8kE7D3Vs from 223.4.115.46
Feb 28 02:32:55 buddha sshd[57693]: Invalid user aMb0lgX8umqqQGpFRjiGiP from 223.4.115.46
Feb 28 02:32:58 buddha sshd[57695]: Invalid user crinalove from 223.4.115.46
Feb 28 02:33:00 buddha sshd[57697]: Invalid user xyzsun123 from 223.4.115.46
Feb 28 02:33:02 buddha sshd[57699]: Invalid user 20090924 from 223.4.115.46
Feb 28 02:33:05 buddha sshd[57713]: Invalid user aicumine from 223.4.115.46
Feb 28 02:33:07 buddha sshd[57715]: Invalid user Router#32SOS from 223.4.115.46
Feb 28 02:33:10 buddha sshd[57717]: Invalid user cotinga from 223.4.115.46
Feb 28 02:33:13 buddha sshd[57719]: Invalid user cornalito from 223.4.115.46
Feb 28 02:33:15 buddha sshd[57721]: Invalid user l0p33os from 223.4.115.46
Feb 28 02:33:18 buddha sshd[57723]: Invalid user !mir@nine from 223.4.115.46
Feb 28 02:33:20 buddha sshd[57725]: Invalid user mucleus.caca.root from 223.4.115.46
Feb 28 02:33:23 buddha sshd[57727]: Invalid user !@#$%^ from 223.4.115.46
Feb 28 02:33:25 buddha sshd[57729]: Invalid user easy2use from 223.4.115.46
Feb 28 02:33:28 buddha sshd[57731]: Invalid user diana4ever from 223.4.115.46
Feb 28 02:33:33 buddha sshd[57733]: Invalid user pw2009inx from 223.4.115.46
Feb 28 02:33:36 buddha sshd[57735]: Invalid user eth0eth1254 from 223.4.115.46
Feb 28 02:33:38 buddha sshd[57737]: Invalid user eth0eth0 from 223.4.115.46
Feb 28 02:33:40 buddha sshd[57739]: Invalid user 1q2w3e4r5t6y7u8i9o0p from 223.4.115.46
Feb 28 02:33:43 buddha sshd[57741]: Invalid user kentlung from 223.4.115.46
Feb 28 02:33:45 buddha sshd[57743]: Invalid user 1q2w3e4r5t6y from 223.4.115.46
Feb 28 02:33:48 buddha sshd[57745]: Invalid user kta1234 from 223.4.115.46
Feb 28 02:33:50 buddha sshd[57747]: Invalid user Kt@1234 from 223.4.115.46
Feb 28 02:33:53 buddha sshd[57749]: Invalid user !mi$ from 223.4.115.46
Feb 28 02:33:55 buddha sshd[57751]: Invalid user perfectpassword from 223.4.115.46
Feb 28 02:33:58 buddha sshd[57753]: Invalid user !mir@ninie from 223.4.115.46
Feb 28 02:34:00 buddha sshd[57755]: Invalid user !mir@Ninie from 223.4.115.46
Feb 28 02:34:03 buddha sshd[57757]: Invalid user !Mir@nine from 223.4.115.46
Feb 28 02:34:05 buddha sshd[57759]: Invalid user vkvadaclasa from 223.4.115.46
Feb 28 02:34:08 buddha sshd[57761]: Invalid user vkvadaclasa from 223.4.115.46
Feb 28 02:34:10 buddha sshd[57763]: Invalid user vkvadaclasa from 223.4.115.46
Feb 28 02:34:13 buddha sshd[57765]: Invalid user vkvadaclasa from 223.4.115.46
Feb 28 02:34:15 buddha sshd[57767]: Invalid user d3v__3f__j3b0n from 223.4.115.46
Feb 28 02:34:17 buddha sshd[57769]: Invalid user kany123kany from 223.4.115.46
Feb 28 02:34:20 buddha sshd[57771]: Invalid user gywjddl!@!* from 223.4.115.46
Feb 28 02:34:22 buddha sshd[57773]: Invalid user aprkvkldf!!! from 223.4.115.46
Feb 28 02:34:25 buddha sshd[57775]: Invalid user dnjao123! from 223.4.115.46
Feb 28 02:34:28 buddha sshd[57777]: Invalid user eltmzm!!! from 223.4.115.46
Feb 28 02:34:31 buddha sshd[57779]: Invalid user #7364! from 223.4.115.46
Feb 28 02:34:34 buddha sshd[57781]: Invalid user disk!!! from 223.4.115.46
Feb 28 02:34:36 buddha sshd[57783]: Invalid user @#Rq92u8fjewRweqf45y43tgh3 from 223.4.115.46
Feb 28 02:34:39 buddha sshd[57785]: Invalid user @n!md@mP#$@&#3141$&#@!#mTadm!n$@ from 223.4.115.46
Feb 28 02:34:41 buddha sshd[57787]: Invalid user BUNdAS@#$RT%GQ~EQW#%^QW from 223.4.115.46
Feb 28 02:34:44 buddha sshd[57789]: Invalid user 163typist from 223.4.115.46
Feb 28 02:34:46 buddha sshd[57791]: Invalid user dudejr5542 from 223.4.115.46
Feb 28 02:34:48 buddha sshd[57793]: Invalid user E1T1RDs7 from 223.4.115.46
Feb 28 02:34:51 buddha sshd[57795]: Invalid user 198287 from 223.4.115.46
Feb 28 02:34:54 buddha sshd[57797]: Invalid user r9A6YOFYEh from 223.4.115.46
Feb 28 02:34:56 buddha sshd[57799]: Invalid user 9swL2k5Cp7 from 223.4.115.46
Feb 28 02:34:59 buddha sshd[57801]: Invalid user oxbow@852 from 223.4.115.46
Feb 28 02:35:01 buddha sshd[57803]: Invalid user frigfurg from 223.4.115.46
Feb 28 02:35:03 buddha sshd[57805]: Invalid user xew4upjg from 223.4.115.46
Feb 28 02:35:06 buddha sshd[57809]: Invalid user K4tp0ng from 223.4.115.46
Feb 28 02:35:08 buddha sshd[57811]: Invalid user dkagh!@#$ from 223.4.115.46
Feb 28 02:35:11 buddha sshd[57813]: Invalid user rhg0704 from 223.4.115.46
Feb 28 02:35:13 buddha sshd[57815]: Invalid user 654312 from 223.4.115.46
Feb 28 02:35:16 buddha sshd[57817]: Invalid user glaemsp!!! from 223.4.115.46
Feb 28 02:35:18 buddha sshd[57819]: Invalid user whznskwhdk from 223.4.115.46
Feb 28 02:35:21 buddha sshd[57821]: Invalid user globalpass from 223.4.115.46
Feb 28 02:35:23 buddha sshd[57823]: Invalid user punglor21 from 223.4.115.46
Feb 28 02:35:26 buddha sshd[57825]: Invalid user nic#!@ruc148 from 223.4.115.46
Feb 28 02:35:29 buddha sshd[57827]: Invalid user dudejrqwer!@#$ from 223.4.115.46

There are a LOT more in the log, but you get the picture.

Python Tivo Library

I've decided its about time I start really learning to code, so I've picked a project and I'm working to see how much I can actually implement.

The long term goal of the project is to create a way to archive TV shows off of your Tivo and provide a mechanism to transfer the shows back on demand for showing. I'm working on creating it on Python, and as a starting point I've begun work on a general python library for interacting with them. Right now provided an IP address and your media access code it can connect to your tivo and give you a listing of all of your shows.

My eventual goal is to provide it both a web interface and an interface that the Tivo can use as well as add autodiscovery of tivo devices. I'd like to also tie it into pytivo for the show playback. One nice thing is that would allow you to compress the shows for better storage.

If anyone is interested in taking a look I'm hosting it at Sourceforge as a way to do revision control as well as learn Subversion a bit better (I'm more familiar with Perforce right now).

http://tivoarchive.sf.net

New blog available

Electr0n has setup a new blog for the ##security channel on Freenode, and has asked me to help with some content. I just posted there on Pastebin hacking in light of the recent Hotmail password fiasco.

Check it out.

Career "Advancement"

About 2 years back I left my job in an InfoSec group. That particular position wasn't the right fit for me anymore, and somehow I didn't think I would be a good fit for a security role in most other organizations. I don't have the pen testing experience needed for most security companies, and the thought of maintaining firewall rules at some retail house would bore the crap out of me.

Since then I've been struggling to find myself career wise. I spent some time in an IT role, and I'm now trying a more customer facing role. Still I find myself happiest in ##security on Freenode, answering people's security questions.

So, what now? Maybe someday I'll figure that out.

The Wrong Tool for the Job

These days anti-virus and anti-spam are two very crucial components of a well run e-mail system. Due to how often spammers change their techniques, my company outsources this function to a vendor which provides both services. Both functions are designed and work fairly well.

Anti-Virus

For Anti-Virus they seem to run messages through multiple commercial anti-virus scanners on their servers. Messages that trigger positive are quarantined, and a notification will be sent to the site admin and/or the intended recipient of the message notifying them of what happened.

The site admin can report false positives to the vendor who will investigate and release a message if they can confirm that it was in fact a false positive. They also take an action to reduce future false positives based on what they find. These investigations tend to take 24 hours or so.

Anti-Spam

Spam tends to be a bit more subjective, so false positives tend to be higher than with Viruses. Due to this, their anti-spam offering makes it a lot easier to both prevent and deal with these situations.

Spam messages can either be tagged for users to filter on their own, or they can be actively filtered and put into a quarantine on their servers. Unlike quarantined virus mail, quarantined spam can be accessed and released by users directly.

In order to prevent false positives site admins are able to whitelist domains, e-mail addresses or IP addresses for specific mail relays. Whitelisting a domain is typically not a great idea in these days of e-mail address spoofing, but e-mail address and whitelisting relays works fairly well.

Where it falls apart

Sounds good so far, right?

Well, here's where it all goes wrong. It appears that anti-virus vendors have discovered that they can use their scanning engines to pick up certain types of phishing and scam e-mails, essentially adding anti-spam into their anti-virus product.

A phishing or a scam mail is SPAM, not a VIRUS. The difference here cause a big problem when you get spam levels of false positives while removing the user's ability to release their own messages and the site admin's ability to implement an sort of whitelisting.

That's when you start getting end user reports of mail threads with customers going missing. Add in a 24 hour turn around time for releasing the messages when the problem is discovered and you start to consider deep-sixing your vendor.

Wooo.. Kindle

Being a big reader and a tech gear junkie, I was rather tempted when Amazon announced the Kindle back in 2007. Somehow I managed to hold out on buying it until they announced the 2.0 version in early February. I pre-ordering it right away, and got my hands on it just last week.

So far, I like it. Its thinner than I expected. Definitely very easy to use. I can hold it in one hand and access most of the controls that I need to read a book. The left side has the "Previous Page" and "Next Page" buttons while on the right side the "Previous Page" button is replaced by a "Home" button. Since I tend to read books in one direction, this seems to work fine.

The free built in wireless is great for getting books, and occasionally pulling up text-only web sites. Due to the rather slow refresh on screen changes using it as a regular web browser is a bit tough.

My biggest complaint is the DRM for files through the Kindle store. After being bitten by DRM from the iTunes Music store, I definitely have a bad taste in my mouth over DRM. Luckily there are other options out there.

The first for me is Many Books. They offer a lot of free content in quite a few eBook formats, including both the native Kindle format and Mobibook which the Kindle also supports. They even have a Mobile Interface which works well from the Kindle itself. Most of the content has elapsed copyrights (older books), but there are occasionally newer books either available with sample chapters or content that was published under a Creative Commons License.

Next was O'Reilly. Being a big tech book reader, I have a lot of O'Reilly books.

O'Reilly offers a number of their books in DRM-free E-Book formats, including the Kindle supported Mobibook format. They're not free, but I don't have any objection to paying for content, just having its usage limited by DRM. They even provide free updates to the books as new revisions are published. I just wish they made it a bit easier to get a list of just their books available in E-Book format.

While I definitely like the Kindle, the only thing I'm not sure about at this point is if it was worth the cost or not. The Kindle costs $360. Sony's offering is quite a bit cheaper, although I have no idea how it compares feature wise.

Data Recovery From a Bad Disk

My wife’s laptop drive failed yesterday, leaving her Windows XP laptop unbootable. IT provided her with a new laptop, but had deemed her data lost. While she does do backups of her data to a USB drive, it had been a while since the last backup so she was a bit concerned. And I of course enjoy a new challenge.

From the various articles I’ve read on data recovery in the past, I knew that the best bet was to make an image of the disk and attempt to recover data off of the image. There’s nothing worse than running a chkdisk/fsck on a partition, and having the attempts to fix the filesystem cause additional filesystem problems.

So how should I make an image? Being a Unix guy, my first thought was dd. DD allows you to copy the complete filesystem off of a partition, and write it to a file. Unfortunately dd can have issues when it attempts to read a block from a disk that is in the process of failing. It will attempt to read again, rather than just moving on to the next block.

A quick Google search brought me to ddrescue, which was designed to deal with this very issue.

Next step is to figure out the best way to actually access the data off of the disk. My first thought was to just pull out the drive and hook it up to my desktop machine. I have an adapter that allows me to plug a laptop drive into a standard IDE cable for a desktop system. I soon discovered that the system was using a SATA drive, and I didn’t have the correct cabling to hook a laptop SATA disk to my desktop, so that plan was shot.

Next thought, Linux live cd. Unfortunately this was a Thinkpad x60s laptop (12" ultra-portable) which doesn’t actually have a CD-ROM drive. There are USB drives for it, but I don’t have one available. That leaves a USB flash drive.

Now to choose what linux image to use. I typically use Ubuntu as a live cd, but I’m not actually sure if they include ddrescue on that. I’m also concerned that Ubuntu might try to auto-mount the bad disk, potentially making the problem worse. So, after a bit of searching I come across System Rescue CD. Its simple, console only and includes ddrescue. Even better, it includes instructions for putting it on a USB disk.

I download the ISO and follow their instructions, and no luck. The USB drive won’t work. I think their instructions could use some work. A quick download of uNetbootin, and I’m on my way. uNetbootin is a generic tool for turning a Linux live CD into a bootable usb drive. I found it a few months ago while trying to install Ubuntu on my eeePC. One more reboot, and I’m good to go.

So now I have the necessary tools to make an image of the bad disk. I just need a place to store the disk image. Its a 60GB drive, so there’s a bit of a storage need here. I don’t have a large enough USB drive on the system, so I need something network enabled. As it turns out, System Rescue CD includes sshfs support, allowing me to mount part of my desktop machine
filesystem remotely. Awesome.

Running ddrescue was easy. Just dd_rescue /dev/sda1 /mnt/desktop. A few hours later, and the data was ready to be accessed. It even reported any bad blocks found on the disk. There turned out to be 120 errored reads, all clumped together on the disk. Based on the initial Windows
boot errors, that part of the disk seemed to hold OS components. Good sign for her data.

Now I have an image of a corrupt NTFS partiton. I used the ntfsfix tool from the ntfsprogs package on Ubuntu to fix the image. Any data from the bad sectors of the disk is going to be gone, but the partition can now be mounted in order to read the rest of the data.

A quick mount with mount -t ntfs-3g image /mnt, and there the data is. Looks like all of her important files were fine. I got to show up the IT folks, and earn me some nice brownie points. Perhaps I'll redeem them for actual brownies.

Information Gathering Using SSH Public keys

I've been a pretty heavy user of SSH for the past 10 years or so. In that time I've learned a number of tricks including port forwarding in various directions, forwarding SSH agents (and the associated risks) and various key management techniques if you're providing key based authentication to large numbers of systems.

The most interesting trick I've learned with SSH, I haven't really seem talked about much. A former co-worker pointed me to the feasibility of this working with protocol 1 and a hacked up SSH client, but these days it trivially works with both protocol 1 and 2 using the normal OpenSSH client.

The Trick

1. Generate an RSA SSH key, and delete the private half. The passphrase does not matter since we won't be using the private key at all. ssh-keygen -t rsa -f test -N "" && rm test


2. Take the public key file (test.pub), and copy it to the authorized_keys file of a remote system.


3. Set mode 600 on the public key. chmod 600 test.pub


4. Try to log into the remote system using the public half of the SSH key. ssh -2 -i test.pub user@server

Assuming all went according to plan, you should get prompted with Enter passphrase for key 'test.pub':. Since this is the public half of a key, no passphrase will ever succeed. You do however know that the private half of this key would have allowed you to log in.

In case you're curious, the reason for the chmod 600 is that the SSH client attempts to enforce good permissions for private keys by refusing to use a "private" key with open permissions. Since you're essentially tricking the client into treating a public key as a private key, the same rules apply.

So What?

This trick allows you to do two things:

It allows you to identify what servers a user has access to. If you have access to a person's public key (which are typically not protected since they're PUBLIC), you can determine what servers the person has access to by attempting to log into root, their username or any other account using their public key.

The second piece is a bit more interesting. If your company has a central key repository which is available to all employees, it becomes very easy to test all keys against a specific server in order to determine who has a private key which has access to the system.

In the past I've used this functionality at work in order to determine who can still log into a system which had been down for a considerable amount of time (and had missed some key rotations). A hacker could instead use this functionality to know who's private SSH key they're going to need to steal in order to gain access to the targeted system.

Why it works

The reason this works can be understood by looking at the Public Key Authentication Method of the SSH protocol.

Among other bits of data, the SSH client sends a copy of the public SSH key to the server as part of the authentication process. The server then responds with SSH_MSG_USERAUTH_FAILURE or a SSH_MSG_USERAUTH_PK_OK message. At this point you now know if access would be granted with the private key, but you have not needed to use that private key in any way yet.

This explains why only the public key is needed during the authentication step, but not necessarily why the SSH client makes this so easy for us. I suppose its probably just a quirk of how their key parsing code works.

They could change the code to not allow you to attempt to do private key operations without a private key, but really that just adds a small hurdle to exploiting this small weakness in the protocol. At the end of the day, you're still only as safe as the protections you put in place on your private keys.

Mac OS DNS bug

I had a bit of an interesting experience the other day while attempting to fail over our Jabber server from our production site to the DR site.

Our two servers each have their own A records in DNS with a TTL of 3600 seconds (1 hour). This long timeout is fine since the IP address of the actual server never really changes.

Access to the service is instead provided by a CNAME record which points to one of those two hostnames. The TTL of the CNAME record is 60 seconds, allowing us to quickly fail over between the two sites as needed.

So the time came, and I had to perform a fail over. I updated the CNAME, and in order to prevent users from being unable to connect, I waited 60 seconds before shutting down the old server and starting up the new one.

From there things went bad. I tried to access the admin console, and failed. I tried to log into the Jabber server, and failed. Finally I hit the admin console through the A record instead of the CNAME, and found that other users had seamlessly failed over.

After a bit of testing I determined that my Linux box and my Windows box both worked fine. The only problem was the Mac that I was making the change from. For some reason, the Mac was holding on to the old IP address.

After some testing, and confirmation from other individuals on their Macs, I think I know what was going on. Using dscacheutil -cachedump -entries, I inspected the local resolver cache.

Here's what I found:

Category Best Before Last Access Hits Refs TTL Neg DS Node
---------- ------------------ ------------------ -------- ------ -------- ----- ---------
Host 01/28/09 21:07:02 01/28/09 20:18:35 10 4 3600
Key: h_aliases:openfire.domain.fake. ipv4:1
Key: h_aliases:openfire.domain.fake ipv4:1
Key: h_name:server1.domain.fake ipv4:1

This appears to be reporting that the local resolver cached the server1.domain.fake DNS record, and set an expiration date of the record for "01/28/09 21:07:02". openfire.domain.fake was then set as an alias for that record without retaining its own TTL. This would certainly explain the behavior that I saw.

So it seems to Mac OS X may be incompatible with a fairly common DNS failover technique. I filed a bug, so it'll be interesting to see how long it takes before Apple gets around to fixing it.

Home Media Server

I'm a bit of a media buff. I own several hundred DVDs, and I'm guessing well over 1000 cds.

Like the rest of the known world, I solved the CD problem years ago. I ripped all of my CDs, and I store them in a few places including my iPod and my home media server. Early on I ripped in ogg format, but quickly regretted it when I bought a Phat Box media player for my car. By the time they supported Ogg format, I had moved onto an iPod.

Eventually I started buying music from the iTunes Music Store since it was so much easier than CDs, and I continued until I discovered the Amazon MP3 Store. Buying in mp3 format instead of AAC is so much easier to deal with, not to mention the lack of DRM.

Recently I read about Sockso. Sockso provides me with a simple web interface for streaming music off of my server so I can listen from work without having everything on local disk. Considering that my music collection is 58GB these days, it certainly saves me some space. Unfortunately Sockso does not support AAC format at this point, so I'm kind of out of luck on my iTunes media (even the non-DRM files).

Recently I started trying to tackle the DVD issue as well. I have a DVD changer, but its just kind of clumsy. It attempts to detect the name of movies from the disc, but rarely succeeds. You can attach a PS2 keyboard and type them in manually, but I eventually had to move the player which required removing the discs (and losing the inputted data).

So I thought I'd apply the same techniques to my movies. I used Handbrake to rip a number of movies and copy them onto my server as well. From there I can copy them into iTunes to watch on my computer, my iPod or transfer to my AppleTV (I apparently buy too much Apple gear).

Most of my media is on my file server which runs linux, so I wanted to see if I can get away without running iTunes. My first option was pyTivo. Its an interesting project. Its a python script that you point to your movie collection. It performs the necessary UDP broadcasts in order to announce your movie share on the network for your TiVo to see, and then converts the movie on demand to a format that TiVo can display properly. pyTivo works pretty well, but the code is in flux, and I'm not sure how much I want to trust it.

My latest try was XBMC. Its a rather nice media player that was created for the original XBox. These days it has also been ported to run on Windows, Linux, the Mac, and AppleTV. It can easily be installed on the AppleTV using ATV USB Creator.

XBMC can recieve a stream from a Universal Plug'n'Play Media Server. In my case I used MediaTomb since it was available straight from Ubuntu. I'm not sure I'd suggest it due to the lack of access control. For now, I'm fine just running it bound to my local network only.

I'm not sure I'm really happy with how all of this is working, but its still a work in progress.

I chose poorly

I finally took the advice of a co-worker and checked out OpenFire to replace Jabber XCP. After about 30 minutes I had Openfire configured with all of the functionality that had taken me weeks to setup properly in XCP.

Additionally I was able to pre-define buddy groups using AD groups, set a message of the day, send broadcast messages to all logged in users, and perform other handy functions.

If you're looking at implementing a Jabber/XMPP solution, its definitely worth a look.

x509 certs and XMPP servers

Ok, one last XMPP post for tonight.

I came across an interesting issue with our test XMPP server today. When the server was initially created, it was setup with an x509/SSL certificate that was self signed. That cert expired the other day, and I had to replace it.

Some of the people who were using the server started getting cert warnings while others did not. After a bit of investigating, I found the problem.

There are two ways that an XMPP client can connect to a server.

The first method is simple. You configure your client with your Jabber ID (username@company.com), and define an XMPP server (servername.company.com) to connect to. This is how our early documentation recommeded configuring your clients, but it is not the generally recommended method.

The second method is the preferred one. You define a service record (SRV record) in DNS for _xmpp-client._tcp.company.com which points to your server name. Once your client has your Jabber ID (JID), the client will automatically look up the SRV record, and connect to that service.

Now comes the cert warning. Apparently if you use an SRV record, your x509 cert needs to have a common name (CN) of company.com. If you define a server manually however, your CN needs to be the name of the server.

Our self-signed cert was for company.com. I replaced it by a properly signed cert for servername.company.com, and broke everyone using the preferred configuration method. Seeing the issue, I replaced it with a properly signed cert for company.com, and broke the people following the published documentation.

*sigh*

So now I get to update the documentation and answer the various questions for people. Longer term I'm going to see if its possible to provide certs for both methods. While you can't have two certs on the same port, it may be possible to either use a cert with an altCN, or possible have the SRV port point to a non-standard port so that people who define the server name can connect to the default port number (5222/5223).

Managing your own Chat server

Recently for work I've been dealing with trying to build a robust IM server environment for the company I work for. The plan was to allow employees to chat with each other without the conversations ever leaving the company. Our hope was to also tie the system into the various public instant messaging systems (AIM/MSN/YIM) so that we could use the same system to communicate with customers.

After looking around at various offerings, the choices boiled down to a Jabber/XMPP based solution or a SIP/SIMPLE service like Microsoft OCS or Lotus Sametime. Since XMPP has more open source libraries for coding against it, we decided to go the XMPP route. Personally I was happy since the peer to peer nature of the SIP protocol introduces a few problems on our network.

Of course, deciding on XMPP doesn't really narrow things down too much. There are a number of open source XMPP/Jabber servers out there, and quite a few commercial ones as well.

We looked at Jabberd2, eJabberd and Jabber XCP.

As far as I can tell, Jabberd2 and Jabber XCP are very similar in codebase, at least judging by the configuration file format. The thing that the commercial Jabber XCP product provides is a nice web interface for configuring the service, a pair of supported clients (one Windows client, one web based client) as well as commercial support.

eJabberd was impressive. It is an open source application written in Erlang, which I wasn't familiar with. Erlang was a programming language created for distributed computing by Ericsson. On smaller installations, it can handle clustering without any form of external database. It automatically manages synchronization between nodes of the cluster.

Version 2.0 of ejabberd also included a very nice web interface for managing the service, and included an inpressive number of plugins. Configuring the service was very very easy.

At the end of the day, we ended up going with Jabber XCP. The main reason is the one feature that no one else was able to provide. The ability to really tie into the public IM systems, or at least one of them. Any XMPP server can tie into other XMPP services (like Google Talk), but Jabber XCP offered the ability to tie into AIM as well.

The majority of Jabber/XMPP servers offer transports for the various IM services. What these transports do is essentially allow you to log into your own AOL/MSN/etc account from within your IM client. So I may be myname@company.com within my company, but I would be logging into my gdfuego account on AIM.

Jabber XCP has a plugin (for additional per user cost), which allows you to actually use the same myname@company.com address within AIM itself. Unfortuantely MSN and YIM aren't an option at this time.

Right now I'm getting close to turning our new server live, and I've been hitting a number of snags, but that'll be a story for another day.

Even less spam

Due to my frustration with my personal e-mail, I recently implemented some additional spam filtering which was amazingly effective.

In the past 24 hours, the system has stopped 775 spams destined to valid users, and has accepted 37 legitimate messages. Previously it would have accepted those messages. Most would end up in my spam folder to be reviewed later, but others would end up in my inbox.

This decrease in spam hitting my inbox is through Greylisting, implemented using Postgrey.

A properly implemented mail server needs to be able to deal with a temporary delivery failures, and try again later. A system implementing Greylisting will keep track of the source IP address as well as the source and destination e-mail addresses. The first time a unique combination of these three things is seen, the message is given a temporary failure for 10 minutes.

Legitimate mail is eventually delivered. Spammers give up and don't come back.

I reject your e-mail

I've always heard that the correct behavior of a mail server is to reject e-mail for local undeliverable addresses rather than accepting them and then bouncing. I never put too much thought into it though until recently when I took over management of our e-mail infrastructure.

When the systems were handed to me, their queues generally had about 800-1000 e-mails
to be delivered at any given point. As I dug into why, I found that the majority of those e-mails were outgoing bounce messages which were undeliverable for one reason or another.

After a few changes, those systems are now rejecting around 1,000,000 spam messages a day. These are messages that would previously have been accepted and sent back out on the internet as backscatter.

Rejecting instead of bouncing allowed me to significantly cut down on the amount of processing power, bandwith and disk space used on these systems. Not to mention cutting down on the amount of e-mail that the backscatter victims were receiving.

Now if only everyone else who runs mail servers would figure this out.

I read your e-mail

Well, a few more months go by and I find myself responsible for the e-mail system of a 1000+ employee multi-national company. It only took me about a month or so before I had a better understanding of how mailflow works within the company than anyone had in several years.

Its amazing how much crap can build up in a system that passes hands a few dozen employees who never quite put in the effort to fully understand or attempt any sort of cleanup.

Wow, I suck

Two posts and I vanish for over a year. Wow I suck.

Well, since my last post I've decided to leave my role as a Security Engineer. Instead I'm moving back to a Systems Administration role supporting Unix servers. Security still interests me, but at the end of the day what I enjoy is using technology to solve problems for people. In my security role I found myself finding problems without being able to assist in solving them.

So as of January 2nd I get to go back to my roots, but with a greater understanding of security. Should be interesting.

SSH Security

I run a linux server that uses SSH as an authentication method. And my server is under attack. There has been over 1000 login attempts from a handful of systems in the past 2 days alone. Since June 1st, 19 different systems have attacked my system over 10,000 times. Chances are, if you're running a system that allows SSH authentication, your system is under attack as well.

For the past few years there has been a remarkable increase in the number of SSH brute force attacks. There are automated scripts out there that scan networks for SSH daemons on port 22, and when found, attempt to log into them using dictionaries of common usernames and passwords.

If you look in your logs, you'll likely see attempts that look something like this:
Jun 30 09:51:58 localhost sshd[32357]: Failed password for invalid user mark from 211.171.202.87 port 41435 ssh2
Jun 30 09:52:02 localhost sshd[32359]: Failed password for invalid user tomas from 211.171.202.87 port 42307 ssh2
Jun 30 09:52:06 localhost sshd[32388]: Failed password for invalid user rpm from 211.171.202.87 port 43192 ssh2
Jun 30 09:52:10 localhost sshd[32390]: Failed password for invalid user jean from 211.171.202.87 port 44095 ssh2

If one of these common username/password pairs actually work against your system, the attacker will gain access to your machine, and likely will use your computer to continue their search for more systems to attack. Your system may also be used as a jumping point for other types of attacks. Your data may be stolen. And who knows what will be traced back to you.

So is there anything you can do to protect yourself? Sure there is. You have a number of options, with different levels of effectiveness.

1) Switch to key based authentication. If you generate an SSH key and use it to provide all access to your system then you're effectively immune against these sorts of attacks. Brute forcing a password using SSH may take days or weeks. Attempting to brute force an RSA based SSH key would take years. Note that you need to REQUIRE keys for authentication, not just use keys. As long as your server still accepts passwords, you're still vulnerable. Regardless of what your standard method for logging in is.

2) If key based authentication isn't an option for you, your next best option is to use a strong password. The longer your password is, and the more random it is, the less likely a password based attack will work. In this case, it's important to ensure that ALL users of the system have strong passwords. Consider using an application like John the Ripper to test the strength of passwords if you have a number of users on the system.

3) An additional option is to move your SSH daemon to a different tcp port. By default SSH listens on port 22. A daemon listening on a different port would be harder to find. Note that unless you mix this will strong passwords you're depending on Security through Obscurity. Obscuring a system like this isn't a bad thing, but it is not a fix all. Assume that someone will eventually find the daemon, and the attack will continue.

4) Restrict who can talk to your SSH daemon. If you're lucky enough to only have a few sources of logins, you can use either application level IP restrictions or firewall rules to ensure you limit who can attempt to login. This works wonders if you trust the people on the allowed hosts list.

The final item I wanted to mention isn't so much a prevention method as a community good deed. Most IPs you see in your logs represent a system that has been compromised by a similiar attack. In most cases, the owner of the system is not aware of what has happened. Or if they are aware, their ISP might be interested to find out about their activities.

If you have some spare time, try tracking down an attacking system. Use tools like dig and whois to attempt to track down the owner of the system and the hosting ISP, and notify them about the attack. Include the log files showing their attack, and explain the issue in plain language. Treat them as a victim, and talk to them in a non-accusitory way.

In a best case situation you'll get a response from the person and they will work to solve the problem. In the worst case scenario you won't get any response. Maybe they solved the problem without saying anything. Maybe they're ignoring you. In either case, at least you tried.