Networking and 1U Colocation
-
I'm trying to think through how I'd configure my network using a VM on my KVM host as my router/firewall. This is the first draft of how I see this working.
The networks with which I've worked in the past have a firewall that would have one interface with a public IP (100.100.100.2/30) and one with a private IP (192.168.1.1/24), which connects to a switch, which connects to everything else. NAT would be used to direct traffic to specific hosts, such as NATing inbound traffic for 100.100.100.2:80 to a webserver at 192.168.1.10.
I'm trying to wrap my head around how to make that concept work in an virtualized environment where I have a single hypervisor and no networking appliance. I'd obviously have a virtual switch which wouldn't have any connection to a physical NIC on the host for the guest VMs (the 192.168.1.0/24 network). I'd have another virtual switch which would have a connection to the physical NIC. The firewall VM would have a connection to each virtual switch.
The question becomes what NIC receives the public IP address I'll be assigned. My first thought was the interface of the firewall VM that's attached to the virtual switch that's attached to the NIC of the physical host. However, that seems problematic, as at the moment, I don't know how a packet destined for 100.100.100.2 coming in on eth0 of the host would be forwarded to the virtual switch.
There are other questions; however, I think this has to be addressed first.
-
With a /30 it is hard because you have to setup your host without a valid IP on the colo network.
So you will need a private management network to access the host remotely.
You will have to rely on the colo's KVM for console access to your dom0 to fix anything if it breaks.
You should easily be able to configure this ahead of time.
-
This will get into how you want to configure your KVM host's networking in the first place.
I don't have network connectivity from my KVM host in my home lab to my guests because I have full access to the host's main IP and never bothered to set anything else up.
-
@eddiejennings said in Networking and 1U Colocation:
I'm trying to think through how I'd configure my network using a VM on my KVM host as my router/firewall. This is the first draft of how I see this working.
The networks with which I've worked in the past have a firewall that would have one interface with a public IP (100.100.100.2/30) and one with a private IP (192.168.1.1/24), which connects to a switch, which connects to everything else. NAT would be used to direct traffic to specific hosts, such as NATing inbound traffic for 100.100.100.2:80 to a webserver at 192.168.1.10.
I'm trying to wrap my head around how to make that concept work in an virtualized environment where I have a single hypervisor and no networking appliance. I'd obviously have a virtual switch which wouldn't have any connection to a physical NIC on the host for the guest VMs (the 192.168.1.0/24 network). I'd have another virtual switch which would have a connection to the physical NIC. The firewall VM would have a connection to each virtual switch.
The question becomes what NIC receives the public IP address I'll be assigned. My first thought was the interface of the firewall VM that's attached to the virtual switch that's attached to the NIC of the physical host. However, that seems problematic, as at the moment, I don't know how a packet destined for 100.100.100.2 coming in on eth0 of the host would be forwarded to the virtual switch.
There are other questions; however, I think this has to be addressed first.
Maybe try this, lets see if I can explain myself well.
Attach your physical interface to a linux-bridge (lets call it br0) and then assign the br0 interface only to your firewall vm. Set the static address 100.100.100.2 on the vms interface this will be your wan port .
Create a private network and set it to isolated, it will create a virbrX interface in your host. Add an interface to your vm inside the isolated private network (using virbrX), this will be your firewalls lan interface.
Your virtual network gives you internal and host only routing, so you will have access to your guest through this ip range.
Any new vm you create add it interface to be part of your virtual network and set their gateway to your firewall-vms lan port.
I believe that should do what you want.
-
@romo He does not even need a bridge. Simply no IP on his eth0.
He will have a java based KVM available for console from his colo if shit really breaks.
Here in my example I have an IP on eno1 just because it is local to me and i am lazy. He should have no IP here.
I have a team, he will not with only a single port from the colo. He could have a team still with one memeber not plugged in, but meh.
When he makes the pfSense VM, he should assign it to his eth0 (team0 in my example) as macvtap.
In bridged mode. As the note states, there will not be any guest communication to the host on this network, which is just fine for security of the router VMs public IP network.
By default the LVM setup create this private network. I suggest he use this on his host and a single guest VM (not his pfSense VM) to manage the host.
I suggested he make another private network that he will assign to the LAN port of the pfSense VM and then as the only network port for all of the guest VMs.
-
@jaredbusch I am so used to creating my bridges manually I keep forgetting macvtap creates them for you.
-
I had to do something a little different as I rent a server that I'll never have physical access to. So this is on an Ubuntu Server base (that always leaves a bad taste in my mouth, sorry.)
eth0 is all I had to work with:
So I also made a virtual interface:
Virtual Interface that is the public facing side:
Finally, the private side:
Now, if you ask me (you didn't, I'll tell you anyway), this way of configuring things kinda sucks. I'm limited to 100mb/s throughput by something along the line that I haven't figured out yet.... I've got stories about these already, I don't need more!
-
@jaredbusch said in Networking and 1U Colocation:
I want to make sure my understanding of your suggestion is correct.
The FirewallVM will have one NIC connected to eth0 via macvtap, which would be assigned 100.100.100.2/30. The FirewallVM will have a second NIC attached to what you called the priv0 virtual network, which serves as the interface to the rest of the VMs in the LAN. For managing the host itself, I'll have another VM attached to the default virtual network.
-
I'm thinking through this topology a bit further:
- Eth0 on host passes traffic via macvtap to FirewallVM (100.100.100.2/30).
- Isolated network connects FirewallVM to the rest of the guest VM.
- Isolated network connects one of the guest VMs to the host VM.
To manage my host over the Internet, I'd have a NAT rule that would direct SSH traffic to VM in item three. Once connected to that VM, then I'd SSH from there to my host -- basically making that guest VM jump box.
-
@eddiejennings said in Networking and 1U Colocation:
I'm thinking through this topology a bit further:
- Eth0 on host passes traffic via macvtap to FirewallVM (100.100.100.2/30).
- Isolated network connects FirewallVM to the rest of the guest VM.
- Isolated network connects one of the guest VMs to the host VM.
To manage my host over the Internet, I'd have a NAT rule that would direct SSH traffic to VM in item three. Once connected to that VM, then I'd SSH from there to my host -- basically making that guest VM jump box.
VyOS has ssh built in.... just saying.
-
@travisdh1 said in Networking and 1U Colocation:
@eddiejennings said in Networking and 1U Colocation:
I'm thinking through this topology a bit further:
- Eth0 on host passes traffic via macvtap to FirewallVM (100.100.100.2/30).
- Isolated network connects FirewallVM to the rest of the guest VM.
- Isolated network connects one of the guest VMs to the host VM.
To manage my host over the Internet, I'd have a NAT rule that would direct SSH traffic to VM in item three. Once connected to that VM, then I'd SSH from there to my host -- basically making that guest VM jump box.
VyOS has ssh built in.... just saying.
Umm WTF? Open SSH to the world? On your router? Fuck that.
-
@eddiejennings said in Networking and 1U Colocation:
To manage my host over the Internet, I'd have a NAT rule that would direct SSH traffic to VM in item three.
Never.
Use ZeroTier on your host. In the instance that it fails, use the colo's KVM to fix it.
-
@jaredbusch said in Networking and 1U Colocation:
@eddiejennings said in Networking and 1U Colocation:
To manage my host over the Internet, I'd have a NAT rule that would direct SSH traffic to VM in item three.
Never.
Use ZeroTier on your host. In the instance that it fails, use the colo's KVM to fix it.
I’ve heard of ZeroTier. Time to learn something about it.
-
@eddiejennings said in Networking and 1U Colocation:
@jaredbusch said in Networking and 1U Colocation:
@eddiejennings said in Networking and 1U Colocation:
To manage my host over the Internet, I'd have a NAT rule that would direct SSH traffic to VM in item three.
Never.
Use ZeroTier on your host. In the instance that it fails, use the colo's KVM to fix it.
I’ve heard of ZeroTier. Time to learn something about it.
Here you go.
https://www.zerotier.com/https://mangolassi.it/topic/16853/installing-zerotier-on-fedora
-
@eddiejennings said in Networking and 1U Colocation:
@jaredbusch said in Networking and 1U Colocation:
@eddiejennings said in Networking and 1U Colocation:
To manage my host over the Internet, I'd have a NAT rule that would direct SSH traffic to VM in item three.
Never.
Use ZeroTier on your host. In the instance that it fails, use the colo's KVM to fix it.
I’ve heard of ZeroTier. Time to learn something about it.
Very good product.
-
@jaredbusch said in Networking and 1U Colocation:
@travisdh1 said in Networking and 1U Colocation:
@eddiejennings said in Networking and 1U Colocation:
I'm thinking through this topology a bit further:
- Eth0 on host passes traffic via macvtap to FirewallVM (100.100.100.2/30).
- Isolated network connects FirewallVM to the rest of the guest VM.
- Isolated network connects one of the guest VMs to the host VM.
To manage my host over the Internet, I'd have a NAT rule that would direct SSH traffic to VM in item three. Once connected to that VM, then I'd SSH from there to my host -- basically making that guest VM jump box.
VyOS has ssh built in.... just saying.
Umm WTF? Open SSH to the world? On your router? Fuck that.
Your direct, to the point responses, often have me LOLing.
-
Here's an updated diagram.
I'm wondering about how to do updates on the KVM host. Since there will be only one NIC with an connection to the colo's network, and that NIC is attached to the FirewallVM, it seems that the KVM host will have to have a way to send / receive traffic to / from the FirewallVM to have an Internet connection. Perhaps one alternative would be to setup a VM that acts like a repository for the updates I'd get through
dnfand the KVM host would talk to that VM to get its software updates.The other thing I'm thinking about is how I'm going to use Virt-Manager with my KVM host. Methinks this might be where ZeroTier would come in. I could connect to the "VM for managing the host" which would have Virt-Manager installed.
-
@eddiejennings said in Networking and 1U Colocation:
Here's an updated diagram.
I'm wondering about how to do updates on the KVM host. Since there will be only one NIC with an connection to the colo's network, and that NIC is attached to the FirewallVM, it seems that the KVM host will have to have a way to send / receive traffic to / from the FirewallVM to have an Internet connection. Perhaps one alternative would be to setup a VM that acts like a repository for the updates I'd get through
dnfand the KVM host would talk to that VM to get its software updates.The other thing I'm thinking about is how I'm going to use Virt-Manager with my KVM host. Methinks this might be where ZeroTier would come in. I could connect to the "VM for managing the host" which would have Virt-Manager installed.
Just add a NIC on you LAN network. Keep it normally disabled. Enable and run updates.
Or just not be super paranoid and just have it on the LAN always.
-
@jaredbusch said in Networking and 1U Colocation:
Or just not be super paranoid and just have it on the LAN always.
Ha!
-
@eddiejennings said in Networking and 1U Colocation:
@jaredbusch said in Networking and 1U Colocation:
Or just not be super paranoid and just have it on the LAN always.
Ha!
It is no different in that way than one in your office.
In fact you will need a way to the internet for ZeroTier to come online.
