Xbox Live MTU Sizing and OpenBSD

A quick one that's easy to solve.
Changing around the rules on the firewall to make IPv6 traffic work better cause me to break all IPv4 traffic that wasn't WWW based.

The symptoms included:
1. Kid's weren't able to access multi-player online games
2. Xbox Live suddenly declares that my MTU is set lower than 1364 bytes
3. HTTP traffic was fine

The last one had me stumped for a while - the outgoing pppoe connection is limited to 1492 bytes due to the overheads of encapsulation. My firewall limits outgoing packet sizes to 1440 bytes in order to stop some other reported problems with pppoe encapsulation.
Checking with other Windows boxes that DHCP to the server showed that they had an network MTU of 1500. What's going on?

Simple, I had accidentally changed one of the rules in my PF configuration that caused it to send out the packets before NAT'ing them - obviously the response bounced on the firewall as it did not know what to do with the return packets.

Conclusion: If the Xbox does not receive a packet back from the Xbox live non-HTTP connection then it complains that the MTU must be set too low. I would also guess that this only happens if a HTTP connection to Xbox live succeeds, as my web proxy was still correctly forwarding web traffic at the time.

This also means that if Xbox Live has a partial failure (i.e. HTTP portion works but there's something wrong with the non-HTTP servers) you would get the same error, despite there being nothing wrong with your MTU settings. I bet that would be frustrating to figure out!

OpenBSD, Internode and IPv6

So, my ISP has enabled IPv6 on it's ADSL connections to begin testing the service, to keen technical users. Well, it actually started in November 2009, but there's nothing like getting it around to something you've wanted to do for a while, when the opportunity presents itself!

For me, the opportunity was that my modem died, so some family modem shuffling occurs, my folks end up with a nice new Netgear Modem/Router/WAP and I get their Netgear 632.
That's a fairly low-end piece of kit with just one USB port and one Ethernet port, no VoIP, no WAP, no anything else, just an ADSL2+ modem and a router.

So why "upgrade" from a Billion 7500G to this? It turns out that there are very few modems on the market that can do IPv6 at the moment - so all I needed was a modem that would give me a PPPoE connection that I could process myself.

After soring all of that out, it was time to change my network around to make the PPPoE translation happen on the OpenBSD firewall. This can be done in two slightly different ways:
1. Using userland PPP
2. Using the kernel pppoe interface

The advantage of using userland PPP is the level of debugging information that you get far outstrips what is available with the kernel pppoe interface. I personally setup PPP first and then translated the learnings to the kernel pppoe.
The advantage of kernel pppoe is that the processing overhead is greatly reduced, leading to higher throughput in my case.

PPP user-land PPPoE configuration

The first thing to do here is check your /etc/ppp/ppp.conf file, you need to tell PPP:
1. Where the ADSL modem is connected
2. How the connection is configured
3. What the username and password is

default:
set log Phase Chat IPCP IPV6CP CCP tun command
set redial 15 0
set reconnect 15 10000

#
# A PPPoE (PPP over Ethernet) setup may look like this:
#
pppoe:
set device "!/usr/sbin/pppoe -i em1"
#set mtu max 1492 - see http://www.mynetwatchman.com/kb/adsl/pppoemtu.htm
set mtu max 1454
set speed sync
disable acfcomp protocomp
deny acfcomp
set cd 5
set dial
set login
set timeout 0
set authname ttt@ipv6.internode.on.net
set authkey ttt
enable dns
enable mssfixup
add! default HISADDR

Important points from the configuration file are:
1. set device: tells ppp to run pppoe and talk to ethernet interface em1
2. set authname: tells pppoe what username to use (you need the ipv6 to make IPv6 start working)
3. set authkey: tells pppoe what the password is
4. enable mssfixup: munges the packet size to make sure the maximum packet size parameter is adhered to
5. add! default HISADDR: adds the IPv4 address automatically to the route table as the default.
6. set ... IPV6CP ...: adds IPv6 negotiation to the ppp logs

NOTE: I noticed that my first connections to Internode over IPv6 took quite a few attempts, so don't give up if it fails (quite) a few times.

After you bring up the pppoe interface (sudo ppp -ddial pppoe) performing an "ifconfig tun0" should bring up the IPv4 connection, and if you're lucky, an IPv6 "local link address" - that is an IPv6 address in the "fe80::" address range. This connection is only used to shuffle data backwards and forwards to Internode, so the next task is to get an outward facing IPv6 address.
This is achieved by installing "wide-dhcpv6" package - it comes with the dhcp6c (translation: DHCP IPv6 Client) which you can use to get your IPv6 address and the block of addresses to use in your network.

Create an appropriate dhcp6c.conf file: (/etc/dhcp6c.conf)

# tun0/pppoe0 is the PPPoE interface
interface tun0 {
send ia-pd 0;
};

# em1 is the modem interface
interface em1 {
information-only;
};

id-assoc pd {
# em0 is the interface to the internal network
prefix-interface em0 {
sla-id 1;
sla-len 4;
};
};

In my case, ethernet interface em1 is the pppoe modem, and em0 is the LAN. tun0 is the ppp connection.
Start the DHCP daemon and you should end up with an address.

sudo /usr/local/sbin/dhcp6c tun0

At this stage you should check that you now have a public address.

em0: flags=8843 mtu 1500
lladdr zz:zz:xx:xx:xx:xx
description: LAN
priority: 0
media: Ethernet autoselect (1000baseT full-duplex,master)
status: active
inet 172.16.3.1 netmask 0xffffff00 broadcast 172.16.3.255
inet6 fe80::xxx:xxxx:xxxx:xxxx%em0 prefixlen 64 scopeid 0x1
inet6 2001:44b8:yyy:yyyy:xxx:xxxx:xxxx:xxxx prefixlen 64
-bash-4.0$

Note the second "inet6" address starting with "Internode" - 2001:44b8 and followed on by the rest of the prefix assigned to my network yyy:yyyy. If this does not appear, it is more than likely your PF firewall is blocking the return packets.
Use

sudo tcpdump -i tun0

to check that you are sending request and receiving the response. It should be addressed using local link addresses (i.e. fe80::.....).

The next step is to enable router advertisements to your internal network. This effectively sends packets to your other computers telling them what the network prefix is. The computers choose their own suffix, which can sometimes be the final part of the MAC address, or an entirely different number (an extension to IPv6 called something like "privacy extensions"). In a future post we'll look at running our own DHCP6 server so that we can at least assign addresses a human being has half a chance of remembering.

This is as simple as setting a flag in your /etc/rc.conf.local:

rtadvd_flags=em0

For the impatient, you can run up rtadvd from the console:

sudo /usr/sbin/rtadvd em0

Next time we'll look at the transition to pppoe in the kernel...

Squid proxies, Disk space and VMware ESXi

Finally - a case that proves my point about having multiple partitions on multiple disks under VMware ESXi!

Yesterday, coming home from work I was greeted with a chorus - "The Internet stopped working!". Uh-oh. Bad news for any part-time administrator.

I have noticed on occassion that the OpenBSD machine can stop forwarding packets and the solution is to log into it, fire a packet across the WAN and then one into the LAN and it seems to pick-up from where it left off - it's strange but infrequent.

Not this time. Reboot the OpenBSD firewall and get nothing. Jump on to the machine, locally I can get web pages from the Internet, but not inside the network. Finally the penny drops when on my laptop I get a message saying that someone is "online on Skype".

Duh.

Something with the transparent web proxy. Sure enough, checking the logs I find:

Jul 21 18:11:46 cerberus squid[3627]: Write failure -- check your disk space and cache.log
Jul 21 18:11:46 cerberus squid[28891]: Squid Parent: child process 3627 exited due to signal 6
Jul 21 18:11:49 cerberus squid[28891]: Squid Parent: child process 18622 started
Jul 21 18:11:50 cerberus squid[18622]: Write failure -- check your disk space and cache.log
Jul 21 18:11:50 cerberus squid[28891]: Squid Parent: child process 18622 exited due to signal 6
Jul 21 18:11:53 cerberus squid[28891]: Squid Parent: child process 3621 started
Jul 21 18:11:54 cerberus squid[3621]: Write failure -- check your disk space and cache.log
Jul 21 18:11:54 cerberus squid[28891]: Squid Parent: child process 3621 exited due to signal 6
Jul 21 18:11:54 cerberus squid[28891]: Exiting due to repeated, frequent failures

So to all those that insist there's nothing wrong with one great big partition (more true if some fool does not over allocate space to his web cache server... but I digress!) the only thing that failed in this scenario was the web cache. Even the logging kept working (as most Unix'ers would know, logging to the /var directory) as it still had some space up it's sleeve as a root-user task.

Okay. That's pretty obvious. The easy answer is to reduce the space allocated for the cache, for example from 8192MB to 7000MB. The downside to this is then I'd have to nuke all of the cache to start again.

The much more insane solution is to say - hang it. I've paid for all of this data, so I'll expand the size of the partition. Now, being a smarty-pants, I've actually thought about this scenario before it happened. Most of the partitions live on disk 0 (/dev/sd0 - it's a SCSI disk), except for /var (where the cache files live) which is on disk 1.

Bewdy...

...nearly - unfortunately, I don't know how to expand partitions under OpenBSD (and the only references I could find at the time said that it wasn't possible) - but it's Vmware ESXi to the rescue.

So, a few clicks later, I've got a new 30GB virtual disk set up - run a SCSI bus rescan to pick it up and place it into the device pool.

disklabel -E sd2

To enter disklabel programme and edit the tables interactively

newfs /dev/sd2d

To format it and...

cd /mnt
sudo mkdir newvar
mount /dev/sd2d /mnt/newvar
cd newvar

to mount the new var into the file system and...

sudo dump -0 -a -u -f - /var | sudo restore -r -f -

to copy the old /var into the new /var.

Next we need to tell the file system that we're changing disks - so edit the /etc/fstab file to reflect the new location.

Finally, perform a reboot and it all worked great.

Reading the configuration options a little harder for squid, tells us that we should not allocate more than 80% of the disk space to the cache. Hmm... 10,000MB partition, 8,192MB cache. Oops. Now it's set up a little differently with a 20GB cache on a 30GB partition, which allows some more room to grow if neccessary.

Filesystem Size Used Avail Capacity Mounted on
/dev/sd0a 1008M 39.7M 918M 4% /
/dev/sd0i 707M 533M 139M 79% /home
/dev/sd0d 2.0G 2.0K 1.9G 0% /tmp
/dev/sd0e 3.9G 424M 3.3G 11% /usr
/dev/sd0f 3.9G 36.2M 3.7G 1% /usr/local
/dev/sd0h 2.0G 335M 1.5G 17% /usr/obj
/dev/sd0g 2.0G 647M 1.2G 34% /usr/src
/dev/sd2d 29.5G 9.4G 18.7G 33% /var
-bash-4.0$