======================================= freebsd migratable vnet jail with ipfw ======================================= This vnet jail will be created with the ability to migrate off onto hardware or a vm or even another jail in the future. The host machine will not have a public network. The jail will have a public network protected using ipfw. ======================================= HOST promiscuous mode on --------------------------------------- If you are creating a jail inside of a virtual machine you will need to turn promiscuous mode on for network bridging to work correctly. For virtualbox modify the real host hardware nic. vboxmanage modifyvm metro --nictype1 virtio --nic1 bridged --bridgeadapter1 igb0 --nicpromisc1 allow-all For vmware locate the vmx file and add an entry for the interface. ethernet0.noPromisc = "FALSE" ======================================= HOST create a migratable zfs structure --------------------------------------- The host will have a dedicated zfs zpool named tank. zpool create -O compression=zstd -O checksum=sha512 -O atime=on -o autoexpand=off -m /export tank /dev/ada1 ....................................... If you need a fresh start you can unmount and destroy anything that exists. Careful! zfs list -o name,mounted -r tank/jail | grep 'yes' | cat -n - | sort -rn | awk '{ print "zfs umount -f " $2 }' zfs destroy -fR tank/jail ....................................... zfs create tank/jail zfs create tank/jail/judo zfs create -o canmount=off tank/jail/judo/root zfs create tank/jail/judo/disk zfs create -o mountpoint=/export/jail/judo/root/zroot tank/jail/judo/disk/zroot zfs create -o mountpoint=none tank/jail/judo/disk/zroot/ROOT zfs create -o mountpoint=/export/jail/judo/root tank/jail/judo/disk/zroot/ROOT/default zfs create -o mountpoint=/export/jail/judo/root/tmp -o exec=on -o setuid=off tank/jail/judo/disk/zroot/tmp zfs create -o mountpoint=/export/jail/judo/root/usr -o canmount=off tank/jail/judo/disk/zroot/usr zfs create tank/jail/judo/disk/zroot/usr/home zfs create -o mountpoint=/export/jail/judo/root/var -o canmount=off tank/jail/judo/disk/zroot/var zfs create -o exec=off -o setuid=off tank/jail/judo/disk/zroot/var/audit zfs create -o exec=off -o setuid=off tank/jail/judo/disk/zroot/var/crash zfs create -o exec=off -o setuid=off tank/jail/judo/disk/zroot/var/log zfs create tank/jail/judo/disk/zroot/var/mail zfs create -o setuid=off tank/jail/judo/disk/zroot/var/tmp ....................................... create a separate dataset for data storage zfs create -o mountpoint=/export/jail/judo/root/export tank/jail/judo/disk/tank ....................................... This query should produce no output indicating a proper zfs structure was created zfs get -r -t filesystem canmount,mountpoint tank/jail/judo/disk | grep -v -e 'PROPERTY' -e ' inherited' -e ' default' -e ' local' ======================================= HOST install the jail --------------------------------------- bsdinstall jail /export/jail/judo/root ....................................... Or do a zfs recieve into tank/jail/judo/disk/zroot ======================================= HOST /etc/rc.conf --------------------------------------- hostname="metro" # the private interface igb0 ifconfig_igb0="inet 192.168.10.10 netmask 255.255.255.0" defaultrouter="192.168.10.1" # the public interface igb1 ifconfig_igb1="up" # the public interface igb1 # private = igb0 bridge0 epair0x # public = igb1 bridge1 epair1xx cloned_interfaces="\ bridge0 epair0 \ bridge1 epair10 \ " # private bridge ifconfig_bridge0="\ addm igb0 \ addm epair0a \ " ifconfig_epair0a="up" # public bridge ifconfig_bridge1="\ addm igb1 \ addm epair10a \ " ifconfig_epair10a="up" # enable jails jail_enable="YES" # start listed jails on boot otherwise start all jail_list="judo" # enable ipfw for jails firewall_enable="YES" firewall_type="open" ....................................... service devfs restart service netif restart service routing restart ======================================= HOST /etc/jail.conf --------------------------------------- exec.clean; exec.start = "/bin/sh /etc/rc"; exec.stop = "/bin/sh /etc/rc.shutdown"; allow.mount; mount.devfs; path = "/export/jail/${name}/root"; exec.consolelog = "/var/log/jail_${name}.log"; judo { vnet; vnet.interface = "epair0b"; vnet.interface += "epair10b"; } ======================================= HOST /export/jail/judo/root/etc/rc.conf --------------------------------------- hostname="judo" ifconfig_epair0b="inet 101.101.101.101 netmask 255.255.255.248" defaultrouter="101.101.101.97" # NOTE: If the host has ipfw enabled then the jail must also have ipfw enabled # and type set to "open" as a minimum to allow network traffic to flow # ie firewall_type="open" firewall_enable="YES" firewall_script="/usr/local/bin/ipfw.sh" firewall_logging="YES" ======================================= HOST /export/jail/judo/root/usr/local/bin/ipfw.sh --------------------------------------- #!/bin/sh # # ipfw # # just update this script when you need any changes and run it #------------------------------------------------------------------------------ # Setup # Add rule rule="ipfw -q add" # loopback loop="lo0" # private LAN interface lan="epair0b" # public WAN interface wan="epair10b" #------------------------------------------------------------------------------ # Flush out the list first ipfw -q -f flush #------------------------------------------------------------------------------ # Allow loopback In/Out # $rule 1100 allow all from any to any via $loop $rule 1100 allow tcp from any to any via $loop $rule 1110 allow udp from any to any via $loop $rule 1120 allow icmp from any to any via $loop #------------------------------------------------------------------------------ # Allow LAN In/Out # $rule 2100 allow all from any to any via $lan $rule 2100 allow tcp from any to any via $lan $rule 2110 allow udp from any to any via $lan $rule 2120 allow icmp from any to any via $lan #------------------------------------------------------------------------------ # Allow Existing Connections In/Out # Allow if packet matches an existing entry in the dynamic rules table $rule 4100 check-state #------------------------------------------------------------------------------ # Deny WAN Attack in # Deny non-routable reserved address spaces $rule 5100 deny all from 192.168.0.0/16 to any in via $wan #RFC 1918 private IP $rule 5110 deny all from 172.16.0.0/12 to any in via $wan #RFC 1918 private IP $rule 5120 deny all from 10.0.0.0/8 to any in via $wan #RFC 1918 private IP $rule 5130 deny all from 127.0.0.0/8 to any in via $wan #loopback $rule 5140 deny all from 0.0.0.0/8 to any in via $wan #loopback $rule 5150 deny all from 169.254.0.0/16 to any in via $wan #DHCP auto-config $rule 5160 deny all from 192.0.2.0/24 to any in via $wan #reserved for docs $rule 5170 deny all from 204.152.64.0/23 to any in via $wan #Sun cluster interconnect $rule 5180 deny all from 224.0.0.0/3 to any in via $wan #Class D & E multicast # Deny public pings # $rule 5200 deny icmp from any to any in via $wan # Allow traceroute out $rule 5200 unreach port udp from any to any 33434-33524 via $wan $rule 5210 allow icmp from any to any icmptypes 0,3,4,11 via $wan # Deny ident $rule 5300 deny tcp from any to any 113 in via $wan # Deny all Netbios services. $rule 5400 deny tcp from any to any 137 in via $wan $rule 5410 deny tcp from any to any 138 in via $wan $rule 5420 deny tcp from any to any 139 in via $wan $rule 5430 deny tcp from any to any 81 in via $wan # Deny fragments $rule 5500 deny all from any to any frag in via $wan # Deny ACK packets that did not match the dynamic rule table $rule 5600 deny tcp from any to any established in via $wan #------------------------------------------------------------------------------ # Allow WAN Out (client) # all open $rule 6100 allow tcp from me to any out via $wan setup keep-state $rule 6110 allow udp from me to any out via $wan keep-state $rule 6120 allow icmp from me to any out via $wan keep-state # ping # $rule 6100 allow icmp from any to any out via $wan keep-state # whois # $rule 6110 allow tcp from any to any 43 out via $wan setup keep-state # ntp # $rule 6200 allow udp from any to any 123 out via $wan keep-state # ssh # $rule 6300 allow tcp from any to any 22 out via $wan setup keep-state # dns # $rule 6400 allow tcp from any to any 53 out via $wan setup keep-state # $rule 6410 allow udp from any to any 53 out via $wan keep-state # http # $rule 6500 allow tcp from any to any 80 out via $wan setup keep-state # https # $rule 6510 allow tcp from any to any 443 out via $wan setup keep-state # smtp # $rule 6600 allow tcp from any to any 25 out via $wan setup keep-state #------------------------------------------------------------------------------ # Allow WAN In (server) # ssh # $rule 7100 allow log tcp from any to me 22 in via $wan setup keep-state # dns master # $rule 7200 allow udp from any to me 53 in via $wan keep-state # dns slave # $rule 7300 allow tcp from any to me 53 in via $wan setup keep-state # http # $rule 7400 allow tcp from any to me 80 in via $wan setup keep-state # https # $rule 7500 allow tcp from any to me 443 in via $wan setup keep-state # smtp # $rule 7600 allow tcp from any to me 25 in via $wan setup keep-state # smtp ssl # $rule 7610 allow tcp from any to me 465 in via $wan setup keep-state # smtp starttls # $rule 7620 allow tcp from any to me 587 in via $wan setup keep-state # pop3 # $rule 7700 allow tcp from any to me 110 in via $wan setup keep-state # pop3s # $rule 7710 allow tcp from any to me 995 in via $wan setup keep-state # imap4 # $rule 7800 allow tcp from any to me 143 in via $wan setup keep-state # imaps # $rule 7810 allow tcp from any to me 993 in via $wan setup keep-state #------------------------------------------------------------------------------ # Deny & Log # Deny & Log all other incoming connections $rule 9100 deny log all from any to any in via $wan # Deny & Log everything else $rule 9200 deny log all from any to any ======================================= HOST /export/jail/judo/root/etc/sysctl.conf --------------------------------------- # ipfw logging net.inet.ip.fw.verbose_limit=10 ======================================= HOST start the jail --------------------------------------- service jail start judo jls ======================================= HOST upgrade the jail --------------------------------------- freebsd-update -j judo fetch freebsd-update -j judo install service jail stop judo jls -d service jail start judo ....................................... To upgrade the jail to a new major or minor version you must first upgrade the host and then upgrade the jail freebsd-update -j judo -r 13.2-RELEASE upgrade freebsd-update -j judo/root install service jail stop judo jls -d service jail start judo freebsd-update -j judo install service jail stop judo jls -d service jail start judo ....................................... pkg -j judo update pkg -j judo upgrade service jail stop judo jls -d service jail start judo ======================================= HOST create jail user account --------------------------------------- zfs create tank/jail/judo/disk/zroot/usr/home/oeng jexec judo pw groupadd -g 16401 -n oeng jexec judo pw useradd -n oeng -c "Owen Eng" -d /home/oeng -g oeng -G wheel -s /bin/sh -u 16401 -m jexec judo mkdir /home/oeng/.ssh jexec judo chown -r oeng:oeng /home/oeng jexec judo chmod -r 0700 /home/oeng ======================================= HOST delete jail user account --------------------------------------- jexec judo pw userdel -n oeng jexec judo pw groupdel -n oeng zfs destroy -fR tank/jail/judo/disk/zroot/usr/home/oeng ======================================= HOST delete a jail --------------------------------------- service jail stop judo zfs destroy -fR tank/jail/judo ....................................... remove entries from /etc/jail.conf remove entries from /etc/rc.conf ======================================= :0) =======================================