Elizafox's Ramblings

I spent literally all day yesterday trying to figure out how to get a jail to route an IPv4 IP given to me by Vultr using vnet (I was having bizarre issues with IPv6 and the host stack; vnet fixed them).

It sounds like an easy problem, and to people who are super experienced... it probably is, and the answer is obvious in hindsight. But even 16 years of BSD experience couldn't help me here.

The problem in detail

The problem was routing an additional IPv4 IP that Vultr gave me, that wasn't in the same subnet as my main IP. It works fine on the host without issue:

ifconfig <external_interface> inet <second_address> alias

That's because the default route is already configured.

You'd think that in the jail, it'd be a matter of doing the same and it'd all Just Work™. Nope!

(Note with the below: not all attempts are logged, just the major ones I remember... I tried a lot of things, for 8 hours)

Adding the route: try 1

When you try it in the jail, like so:

route add default <host_gateway> -iface eth0

You get an error like:

route: writing to routing socket: Network is unreachable
add net default: gateway via fib 0: Network is unreachable

Bummer.

So I tried this in the jail:

route add <host_gateway> -iface eth0
route add default <host_gateway>

But nothing would route out of the jail.

Adding the route: try 2

I even tried this in the jail:

route add -net <host_gateway>/<host_subnet> -iface eth0
route add -host default <host_gateway>

And that was even worse. Now everything said “no route to host.” Gah.

I also kept getting this weird issue where I'd add a default route, and it would get corrupted into some gobbledygook in netstat -4rn like so:

Destination        Gateway        Flags Netif Expire
0.0.0.0&0x68150581 104.207.142.1  US    eth0
104.207.142.0/23   link#2         US    eth0

Maybe some kind of bug?

Adding the route: try 3

I went back to the former solution, and did tcpdump on the host. I was seeing traffic coming out of the jail, but it wasn't coming back. I could arping the host, and the gateway, but still couldn't route things.

So I figured it was a host issue.

I tried this on the host (not the jail):

sysctl net.inet.ip.forwarding=1
route add -net <second_ip> -inet bridge1

Nope.

Adding the route: try 4

At this point I was super confused. So in desperation, I even tried this on the host:

route add -net <second_ip> -inet epair1a

Still nothing.

Adding the route: try 5

I was pretty much despondent at this point. I had spun my wheels on this problem for 8 hours, brainstorming answers, trying literally everything I could think of, picking other's brains who were just as confused as I was.

So in an act of extreme desperation, I tried the one thing I hadn't tried on the host:

route add -net <second_ip> -inet <external_interface>

And then...

It worked.

Holy crap, it worked. I bowed down, thanked the heavens, and felt so much relief, I went and made Taco Hamburger Helper (I'm poor, okay?).

Making it all work

So here's how you can make this work in your environment, too, on Vultr and probably other hosting providers, if you're in the same situation:

Jail

cat >>/etc/rc.conf <<EOF
static_routes="default_router"
route_default_router="-host <host_gateway> -iface eth0"
defaultrouter="<host_gateway>"
ifconfig_eth0="inet <host_ip>/<host_subnet>"  # For me it was /32
EOF
route -host <host_gateway> -iface eth0
route default <host_gateway>

Host

cat >>/etc/rc.conf <<EOF
gateway_enable="YES"
static_routes="jail0"
route_jail0="-host <second_ip> -iface <external_interface>"
EOF
sysctl net.inet.ip.forwarding=1  # This just does what the line in rc.conf does on reboot
route add -host <second_ip> -iface <external_interface>

Addenum

If you have a provider that restricts the original MAC where something came from, you can replace <host_gateway> above with your host machine's IP. I tested that, and it all Just Works™. It should then route through your host machine to the outside world, and the jail's MAC in theory shouldn't be visible.

Conclusion

I hate IPv4 more than I hate IPv6 at this point. It's always been like this, whereas IPv6 has mostly been pretty good (minus ISP/modem issues... I once had a horrible Siemens DSL modem that wouldn't pass 6to4, but that's a story for another day).

— Elizabeth Ashford (Elizafox) Fedi (elsewhere): @Elizafox@social.treehouse.systems Tip jar: PayPal || CashApp || LiberaPay