Wednesday, August 2, 2017

Libvirtd - VMs on separate virtual networks cannot talk to each other

There is a minor flaw in the way libvirt is configured to add Firewall rules to the host firewall when creating virtual networks.  Two Virtual Machines in separate virtual networks cannot speak to each other bidirectionally.  I am not the only one who has had this problem but it seems that the issue has not caused enough problems for it to be fixed yet.  Or, perhaps I am misunderstanding the way libvirt is supposed to be configured.

Before proceeding, I've read the documentation regarding this issue at: https://libvirt.org/firewall.html

And I've requested assistance from people on IRC  irc.oftc.net, channel: #virt.

Presently I am following examples in Michael Jang's RHCSA/RHCE study guide, specifically: RHCSA/RHCE Red Hat Linux Certification Study Guide, Seventh Edition (Exams EX200 & EX300).  In the first chapters we are instructed in creating a second virtual network (the first being the default.)  Virtual machines are created in both of these networks.  Here is the basic idea of the networking setup:

Host OS: CentOS 7.3
Libvirt version: libvirt-daemon-2.0.0-10.el7_3.9.x86_64

Network Setup:
---------------------------
Host network: 192.168.0.1/24 (this can vary)
Host IP: 192.168.0.27 (this can also vary - statically assigned)

1st Virtual Network (Default):
    Net: 192.168.122.0/24
    GW: 192.168.122.1
    DHCP: yes
    DHCP Range: 192.168.122.2->254    (I use static IPs)
    Forward Mode: Nat

2nd Virtual Network (outsider):
    Net: 192.168.100.0/24
    GW: 192.168.100.1
    DHCP: yes
    DHCP Range: 192.168.100.128->254    (I use static IPs)
    Forward Mode: Nat

VM status in 1st network (Default):
    Name: server1
    IP: 192.168.122.50
    - can ping host fine
    - can ping external DNS addresses (like google)

VM status in 2nd network (outsider):
    Name: outsider1
    IP: 192.168.100.100
    - can ping host fine
    - can ping external DNS addresses (like google)

PROBLEM: Here's where things fall apart; outsider1 can ping server1 just fine, but server1 cannot ping outsider1.

[name@outsider1 ~]$ ping 192.168.122.50
PING 192.168.122.50 (192.168.122.50) 56(84) bytes of data.
64 bytes from 192.168.122.50: icmp_seq=1 ttl=63 time=0.171 ms

[name@server1 ~]$ ping 192.168.100.100
PING 192.168.100.100 (192.168.100.100) 56(84) bytes of data.
From 192.168.122.1 icmp_seq=1 Destination Port Unreachable


Below, I have the modifications made by libvirt to my firewalld FORWARD chain configuration (Sorry for the tiny fonts - zoom in if necessary):

Base FORWARD chain (prior to having libvirtd configure any networks:
-------------------
Chain FORWARD (policy ACCEPT)
target     prot opt source               destination        
ACCEPT     all  --  0.0.0.0/0            0.0.0.0/0            ctstate RELATED,ESTABLISHED
ACCEPT     all  --  0.0.0.0/0            0.0.0.0/0          
FORWARD_direct  all  --  0.0.0.0/0            0.0.0.0/0          
FORWARD_IN_ZONES_SOURCE  all  --  0.0.0.0/0            0.0.0.0/0          
FORWARD_IN_ZONES  all  --  0.0.0.0/0            0.0.0.0/0          
FORWARD_OUT_ZONES_SOURCE  all  --  0.0.0.0/0            0.0.0.0/0          
FORWARD_OUT_ZONES  all  --  0.0.0.0/0            0.0.0.0/0          
DROP       all  --  0.0.0.0/0            0.0.0.0/0            ctstate INVALID
REJECT     all  --  0.0.0.0/0            0.0.0.0/0            reject-with icmp-host-prohibited

Default Network Added:
----------------------
Chain FORWARD (policy ACCEPT)
target     prot opt source               destination        
ACCEPT     all  --  0.0.0.0/0            192.168.122.0/24     ctstate RELATED,ESTABLISHED
ACCEPT     all  --  192.168.122.0/24     0.0.0.0/0          
ACCEPT     all  --  0.0.0.0/0            0.0.0.0/0          
REJECT     all  --  0.0.0.0/0            0.0.0.0/0            reject-with icmp-port-unreachable
REJECT     all  --  0.0.0.0/0            0.0.0.0/0            reject-with icmp-port-unreachable

ACCEPT     all  --  0.0.0.0/0            0.0.0.0/0            ctstate RELATED,ESTABLISHED
ACCEPT     all  --  0.0.0.0/0            0.0.0.0/0          
...

outsider Network Added:
-----------------------
Chain FORWARD (policy ACCEPT)
target     prot opt source               destination        
ACCEPT     all  --  0.0.0.0/0            192.168.100.0/24     ctstate RELATED,ESTABLISHED
ACCEPT     all  --  192.168.100.0/24     0.0.0.0/0          
ACCEPT     all  --  0.0.0.0/0            0.0.0.0/0          
REJECT     all  --  0.0.0.0/0            0.0.0.0/0            reject-with icmp-port-unreachable
REJECT     all  --  0.0.0.0/0            0.0.0.0/0            reject-with icmp-port-unreachable
ACCEPT     all  --  0.0.0.0/0            192.168.122.0/24     ctstate RELATED,ESTABLISHED
ACCEPT     all  --  192.168.122.0/24     0.0.0.0/0          
ACCEPT     all  --  0.0.0.0/0            0.0.0.0/0          
REJECT     all  --  0.0.0.0/0            0.0.0.0/0            reject-with icmp-port-unreachable
REJECT     all  --  0.0.0.0/0            0.0.0.0/0            reject-with icmp-port-unreachable

ACCEPT     all  --  0.0.0.0/0            0.0.0.0/0            ctstate RELATED,ESTABLISHED
ACCEPT     all  --  0.0.0.0/0            0.0.0.0/0          
...

According to an individual I spoke to on IRC, the intent was to have the firewall block communication both ways,  meaning that neither VMs should have been able to communicate with each other.

I'm not sure if Michael Jang is aware of this, but it seems that this is not entirely well documented, and not exactly functional either.

Since I want my two VMs to communicate with each other across this network, my temporary solution is to modify the FORWARD chain and remove the two REJECT rules that were added with the 'outside' network.  Therefore we will delete the 4th and 5th rules.

From the host system, as root, delete the 4th rule in the FORWARD chain.

[root@hostname ~]# iptables -D FORWARD 4

Since the 4th rule is deleted, the 5th rule now became the 4th, so verify with:

[root@hostname ~]# iptables -L FORWARD

And delete the new 4th rule (formerly 5th).

[root@hostname ~]# iptables -D FORWARD 4

Verify your final configuration and try pinging from both VMs.

[root@hostname ~]# iptables -L FORWARD

NOTE: Of course, this is a temporary rule for testing purposes only and not meant to be used for a production VM host.  If that is your purpose, a proper firewall configuration design should be undertaken with all that it entails.


No comments:

Post a Comment