Tuesday, June 28, 2016

Installing Ubuntu 16.04 on a ZFS root filesystem

One of the major new pieces of functionality in Ubuntu 16.04 (Xenial) is built in support for ZFS filesystems.  I was disappointed to learn that ZFS support is not actually built into the installer itself, leaving you to piece things together yourself.  I did find a few good tutorials online for this, but they all seem to be missing a few pieces.  I'm hoping that this guide will be a bit more complete for people.

It is based on a combination of the following two guides:

Booting Into the Install Environment

Setting up a ZFS filesystem requires a full set of userspace tools rather than the limited set included within the actual installer.  Due to this, we're going to boot a Ubuntu Desktop live CD, and do a manual installation within its root filesystem.  I found that the easiest way to do this was to boot the live CD, set a password for the ubuntu user, and SSH into the machine from a remote box.  This way it is much easier to copy/paste commands from the webpage.

To start off, we need to install the ZFS tools and debootstrap which we will use for actually installing the operating system:

apt-add-repository universe
apt-get update
apt-get install --yes zfsutils-linux debootstrap

Partitioning the Disks

Next, we partition the disks.  For now we're going to assume a two disk system which is just doing striping, but you can change as desired:

parted -- /dev/sda mklabel msdos Y mkpart primary zfs 0% 100%
parted -- /dev/sdb mklabel msdos Y mkpart primary zfs 0% 100%

Device naming during bootup on Linux isn't always static, so we want to reference the disks through a static identifier like disk ID (/dev/disk/by-id), unfortunately Grub fails to properly identify the devices from the zfs command output, so we need this hack for now to make sure that update-grub will correctly identify the disks.  Specifically, it forces creation of symlinks in /dev which map to the device names in /dev/disk/by-id.  Some additional details about the issue can be found in this grub2 launchpad bug.

echo 'KERNEL=="sd*[!0-9]", IMPORT{parent}=="ID_*", SYMLINK+="$env{ID_BUS}-$env{ID_SERIAL}"
> KERNEL=="sd*[0-9]", IMPORT{parent}=="ID_*", SYMLINK+="$env{ID_BUS}-$env{ID_SERIAL}-part%n"' > /etc/udev/rules.d/90-zfs.rules
udevadm trigger

Create Your Zpool

Now we actually create the zpool and import it to /mnt:

zpool create -m none -o ashift=12 -O compression=lz4 rpool /dev/sda1 /dev/sdb1
zfs create -o mountpoint=/ rpool/root
zpool export rpool
zpool import -d /dev/disk/by-id -R /mnt rpool

So here we create the zpool by device name, and then re-import it by device ID while mounting at /mnt.

And then create various partitions off of the root filesystem:

zfs create -o setuid=off rpool/root/home
zfs create -o mountpoint=/root rpool/root/home/root
zfs create -o canmount=off -o setuid=off -o exec=off rpool/root/var
zfs create -o com.sun:auto-snapshot=false rpool/root/var/cache
zfs create rpool/root/var/log
zfs create rpool/root/var/spool
zfs create -o com.sun:auto-snapshot=false -o exec=on rpool/root/var/tmp

We break out these various subdirectories so we have the ability to optionally enable/disable compression, support for setuid/exec, and various other options.  We can also choose to limit the maximum size of each mount independently.

Install Ubuntu

The install of Ubuntu itself is pretty straightforward:

debootstrap the OS

debootstrap xenial /mnt
zfs set devices=off rpool
grep -v cdrom /etc/apt/sources.list > /mnt/etc/apt/sources.list
cp /etc/udev/rules.d/90-zfs.rules /mnt/etc/udev/rules.d/90-zfs.rules

You'll notice at the end here we copy our hacky udev rule to the new filesystem.  The "devices=off" option we set here disables the ability to create device nodes in the filesystem.  This works since /dev is actually a devtmpfs partition of its own.

Configure the network interface

export INTERFACE=$(ip addr list | grep ^[0-9]: | grep -v "lo" | awk {'print $2'} | cut -d ":" -f 1)
echo test > /mnt/etc/hostname
echo  >> /mnt/etc/hosts
echo "auto $INTERFACE
iface $INTERFACE inet dhcp" > /mnt/etc/network/interfaces.d/$INTERFACE

Enter the chroot for some final setup

mount --rbind /dev /mnt/dev
mount --rbind /proc /mnt/proc
mount --rbind /sys /mnt/sys
chroot /mnt /bin/bash --login

locale-gen en_US.UTF-8
echo 'LANG="en_US.UTF-8"' > /etc/default/locale
apt-get update
apt-get install --yes zfsutils-linux zfs-initramfs grub-pc linux-image-generic ssh
dpkg-reconfigure tzdata
update-initramfs -c -k all

Set Up Grub's configuration

sed -i 's/^\(GRUB_CMDLINE_LINUX_DEFAULT\)=.*/\1=""/g' /etc/default/grub
sed -i 's|^\(GRUB_HIDDEN_TIMEOUT=.*\)|#\1|g' /etc/default/grub
sed -i 's/^\(GRUB_CMDLINE_LINUX\)="\(.*\)"/\1="boot=zfs \2"/g' /etc/default/grub

ln -s /proc/mounts  /etc/mtab
rm /etc/mtab

Set a root password and exit the chroot

passwd root

Install the grub bootloader

grub-probe /mnt
grub-install --root-directory=/mnt /dev/sda
grub-install --root-directory=/mnt /dev/sdb

Reboot into the environment


Friday, October 11, 2013

Eating your own dog food

I got this unsolicited e-mail today:

Hope you doing well. I am writing in regards to see if there is any possibility for us to work with your company. We are leading provider of B2B, B2C and B2G lists and excellent list compiler with highest delivery rate and data appending solutions. Our opt-in email database of Key decision makers can be used for your online promotion, brand awareness and to generate potential leads.

We build list according to your requirements based on your targeted business. We are specialized in Email Campaign and Data Append Solution, where you can add your clients missing data (email, fax etc).
Apparently they provide opt-in only mailing lists.  Shame they don't use them themselves.

Wednesday, February 29, 2012


My home server is configured to send me daily e-mails with any security events. Typically this means SSH brute force login attempts, and I occasionally take the opportunity notify people that their servers were breached and are being used for attacks.

Today I found something amusing in the logs. Apparently some idiot hacker mixed up their username and password files for their attack:
Feb 28 02:32:53 buddha sshd[57691]: Invalid user QFhGj8kE7D3Vs from
Feb 28 02:32:55 buddha sshd[57693]: Invalid user aMb0lgX8umqqQGpFRjiGiP from
Feb 28 02:32:58 buddha sshd[57695]: Invalid user crinalove from
Feb 28 02:33:00 buddha sshd[57697]: Invalid user xyzsun123 from
Feb 28 02:33:02 buddha sshd[57699]: Invalid user 20090924 from
Feb 28 02:33:05 buddha sshd[57713]: Invalid user aicumine from
Feb 28 02:33:07 buddha sshd[57715]: Invalid user Router#32SOS from
Feb 28 02:33:10 buddha sshd[57717]: Invalid user cotinga from
Feb 28 02:33:13 buddha sshd[57719]: Invalid user cornalito from
Feb 28 02:33:15 buddha sshd[57721]: Invalid user l0p33os from
Feb 28 02:33:18 buddha sshd[57723]: Invalid user !mir@nine from
Feb 28 02:33:20 buddha sshd[57725]: Invalid user mucleus.caca.root from
Feb 28 02:33:23 buddha sshd[57727]: Invalid user !@#$%^ from
Feb 28 02:33:25 buddha sshd[57729]: Invalid user easy2use from
Feb 28 02:33:28 buddha sshd[57731]: Invalid user diana4ever from
Feb 28 02:33:33 buddha sshd[57733]: Invalid user pw2009inx from
Feb 28 02:33:36 buddha sshd[57735]: Invalid user eth0eth1254 from
Feb 28 02:33:38 buddha sshd[57737]: Invalid user eth0eth0 from
Feb 28 02:33:40 buddha sshd[57739]: Invalid user 1q2w3e4r5t6y7u8i9o0p from
Feb 28 02:33:43 buddha sshd[57741]: Invalid user kentlung from
Feb 28 02:33:45 buddha sshd[57743]: Invalid user 1q2w3e4r5t6y from
Feb 28 02:33:48 buddha sshd[57745]: Invalid user kta1234 from
Feb 28 02:33:50 buddha sshd[57747]: Invalid user Kt@1234 from
Feb 28 02:33:53 buddha sshd[57749]: Invalid user !mi$ from
Feb 28 02:33:55 buddha sshd[57751]: Invalid user perfectpassword from
Feb 28 02:33:58 buddha sshd[57753]: Invalid user !mir@ninie from
Feb 28 02:34:00 buddha sshd[57755]: Invalid user !mir@Ninie from
Feb 28 02:34:03 buddha sshd[57757]: Invalid user !Mir@nine from
Feb 28 02:34:05 buddha sshd[57759]: Invalid user vkvadaclasa from
Feb 28 02:34:08 buddha sshd[57761]: Invalid user vkvadaclasa from
Feb 28 02:34:10 buddha sshd[57763]: Invalid user vkvadaclasa from
Feb 28 02:34:13 buddha sshd[57765]: Invalid user vkvadaclasa from
Feb 28 02:34:15 buddha sshd[57767]: Invalid user d3v__3f__j3b0n from
Feb 28 02:34:17 buddha sshd[57769]: Invalid user kany123kany from
Feb 28 02:34:20 buddha sshd[57771]: Invalid user gywjddl!@!* from
Feb 28 02:34:22 buddha sshd[57773]: Invalid user aprkvkldf!!! from
Feb 28 02:34:25 buddha sshd[57775]: Invalid user dnjao123! from
Feb 28 02:34:28 buddha sshd[57777]: Invalid user eltmzm!!! from
Feb 28 02:34:31 buddha sshd[57779]: Invalid user #7364! from
Feb 28 02:34:34 buddha sshd[57781]: Invalid user disk!!! from
Feb 28 02:34:36 buddha sshd[57783]: Invalid user @#Rq92u8fjewRweqf45y43tgh3 from
Feb 28 02:34:39 buddha sshd[57785]: Invalid user @n!md@mP#$@&#3141$&#@!#mTadm!n$@ from
Feb 28 02:34:41 buddha sshd[57787]: Invalid user BUNdAS@#$RT%GQ~EQW#%^QW from
Feb 28 02:34:44 buddha sshd[57789]: Invalid user 163typist from
Feb 28 02:34:46 buddha sshd[57791]: Invalid user dudejr5542 from
Feb 28 02:34:48 buddha sshd[57793]: Invalid user E1T1RDs7 from
Feb 28 02:34:51 buddha sshd[57795]: Invalid user 198287 from
Feb 28 02:34:54 buddha sshd[57797]: Invalid user r9A6YOFYEh from
Feb 28 02:34:56 buddha sshd[57799]: Invalid user 9swL2k5Cp7 from
Feb 28 02:34:59 buddha sshd[57801]: Invalid user oxbow@852 from
Feb 28 02:35:01 buddha sshd[57803]: Invalid user frigfurg from
Feb 28 02:35:03 buddha sshd[57805]: Invalid user xew4upjg from
Feb 28 02:35:06 buddha sshd[57809]: Invalid user K4tp0ng from
Feb 28 02:35:08 buddha sshd[57811]: Invalid user dkagh!@#$ from
Feb 28 02:35:11 buddha sshd[57813]: Invalid user rhg0704 from
Feb 28 02:35:13 buddha sshd[57815]: Invalid user 654312 from
Feb 28 02:35:16 buddha sshd[57817]: Invalid user glaemsp!!! from
Feb 28 02:35:18 buddha sshd[57819]: Invalid user whznskwhdk from
Feb 28 02:35:21 buddha sshd[57821]: Invalid user globalpass from
Feb 28 02:35:23 buddha sshd[57823]: Invalid user punglor21 from
Feb 28 02:35:26 buddha sshd[57825]: Invalid user nic#!@ruc148 from
Feb 28 02:35:29 buddha sshd[57827]: Invalid user dudejrqwer!@#$ from
There are a LOT more in the log, but you get the picture.

Wednesday, April 21, 2010

Python Tivo Library

I've decided its about time I start really learning to code, so I've picked a project and I'm working to see how much I can actually implement.

The long term goal of the project is to create a way to archive TV shows off of your Tivo and provide a mechanism to transfer the shows back on demand for showing. I'm working on creating it on Python, and as a starting point I've begun work on a general python library for interacting with them. Right now provided an IP address and your media access code it can connect to your tivo and give you a listing of all of your shows.

My eventual goal is to provide it both a web interface and an interface that the Tivo can use as well as add autodiscovery of tivo devices. I'd like to also tie it into pytivo for the show playback. One nice thing is that would allow you to compress the shows for better storage.

If anyone is interested in taking a look I'm hosting it at Sourceforge as a way to do revision control as well as learn Subversion a bit better (I'm more familiar with Perforce right now).


Wednesday, October 14, 2009

New blog available

Electr0n has setup a new blog for the ##security channel on Freenode, and has asked me to help with some content. I just posted there on Pastebin hacking in light of the recent Hotmail password fiasco.

Check it out.

Monday, August 17, 2009

Career "Advancement"

About 2 years back I left my job in an InfoSec group. That particular position wasn't the right fit for me anymore, and somehow I didn't think I would be a good fit for a security role in most other organizations. I don't have the pen testing experience needed for most security companies, and the thought of maintaining firewall rules at some retail house would bore the crap out of me.

Since then I've been struggling to find myself career wise. I spent some time in an IT role, and I'm now trying a more customer facing role. Still I find myself happiest in ##security on Freenode, answering people's security questions.

So, what now? Maybe someday I'll figure that out.

Thursday, March 05, 2009

The Wrong Tool for the Job

These days anti-virus and anti-spam are two very crucial components of a well run e-mail system. Due to how often spammers change their techniques, my company outsources this function to a vendor which provides both services. Both functions are designed and work fairly well.


For Anti-Virus they seem to run messages through multiple commercial anti-virus scanners on their servers. Messages that trigger positive are quarantined, and a notification will be sent to the site admin and/or the intended recipient of the message notifying them of what happened.

The site admin can report false positives to the vendor who will investigate and release a message if they can confirm that it was in fact a false positive. They also take an action to reduce future false positives based on what they find. These investigations tend to take 24 hours or so.


Spam tends to be a bit more subjective, so false positives tend to be higher than with Viruses. Due to this, their anti-spam offering makes it a lot easier to both prevent and deal with these situations.

Spam messages can either be tagged for users to filter on their own, or they can be actively filtered and put into a quarantine on their servers. Unlike quarantined virus mail, quarantined spam can be accessed and released by users directly.

In order to prevent false positives site admins are able to whitelist domains, e-mail addresses or IP addresses for specific mail relays. Whitelisting a domain is typically not a great idea in these days of e-mail address spoofing, but e-mail address and whitelisting relays works fairly well.

Where it falls apart

Sounds good so far, right?

Well, here's where it all goes wrong. It appears that anti-virus vendors have discovered that they can use their scanning engines to pick up certain types of phishing and scam e-mails, essentially adding anti-spam into their anti-virus product.

A phishing or a scam mail is SPAM, not a VIRUS. The difference here cause a big problem when you get spam levels of false positives while removing the user's ability to release their own messages and the site admin's ability to implement an sort of whitelisting.

That's when you start getting end user reports of mail threads with customers going missing. Add in a 24 hour turn around time for releasing the messages when the problem is discovered and you start to consider deep-sixing your vendor.