Bug#865975: docker.io breaks (bridged) network for VMs

classic Classic list List threaded Threaded
14 messages Options
Reply | Threaded
Open this post in threaded view
|

Bug#865975: docker.io breaks (bridged) network for VMs

Roland Kammerer
Package: docker.io
Version: 1.13.1~ds1-2
Severity: critical
Tags: upstream
Justification: breaks unrelated software

Dear Maintainer,

* What led up to the situation?
Any docker command like "docker images"

* What was the outcome of this action?
Network breaks for all libvirt VMs (i.e., they are not able to ping each
other, or public domains, and after a reboot they do not get an IP via
dhcp). The VMs are connected to a bridge (br0):

# The loopback network interface
auto lo
iface lo inet loopback

# The primary network interface
#allow-hotplug eth2
#iface eth2 inet dhcp
# This is an autoconfigured IPv6 interface
#iface eth2 inet6 auto

auto br0
iface br0 inet dhcp
        bridge_ports eth2
        # bridge_stp on
        bridge_maxwait 0
        bridge_fd 0

I don't have any firewall/iptables rules on my machine.

* What outcome did you expect instead?
That networking still works (as it did with older docker versions).

The situation can be fixed via "iptables -I FORWARD -i br0 -o br0 -j ACCEPT".
Before that I saw "Chain FORWARD (policy drop 3493 packests, 829K bytes)". Therefore, I assume that docker messages with the chains.


-- System Information:
Debian Release: 9.0
  APT prefers stable
  APT policy: (700, 'stable'), (600, 'unstable'), (550, 'experimental'), (500, 'stable-debug')
Architecture: amd64 (x86_64)

Kernel: Linux 4.9.0-3-amd64 (SMP w/4 CPU cores)
Locale: LANG=en_US.UTF-8, LC_CTYPE=en_US.UTF-8 (charmap=UTF-8), LANGUAGE=en_US:en (charmap=UTF-8)
Shell: /bin/sh linked to /bin/dash
Init: systemd (via /run/systemd/system)

Versions of packages docker.io depends on:
ii  adduser              3.115
ii  containerd           0.2.3+git20170126.85.aa8187d~ds1-1
ii  golang-libnetwork    0.8.0-dev.2+git20170202.599.45b4086-1
ii  init-system-helpers  1.48
ii  iptables             1.6.0+snapshot20161117-6
ii  libapparmor1         2.11.0-3
ii  libc6                2.24-11
ii  libdevmapper1.02.1   2:1.02.137-2
ii  libsqlite3-0         3.16.2-4
ii  libsystemd0          232-24
ii  lsb-base             9.20161125
ii  runc                 1.0.0~rc2+git20170201.133.9df8b30-1

Versions of packages docker.io recommends:
ii  ca-certificates  20161130+nmu1
ii  cgroupfs-mount   1.3
ii  git              1:2.11.0-3
ii  xz-utils         5.2.2-1.2+b1

Versions of packages docker.io suggests:
pn  aufs-tools           <none>
pn  btrfs-progs          <none>
ii  debootstrap          1.0.89
pn  docker-doc           <none>
pn  rinse                <none>
pn  zfs-fuse | zfsutils  <none>

-- Configuration Files:
/etc/default/docker changed:
DOCKER_OPTS="--storage-driver=devicemapper -H unix:///var/run/docker.sock"


-- no debconf information

Reply | Threaded
Open this post in threaded view
|

Bug#865975: docker.io breaks (bridged) network for VMs

Alban Browaeys-2
Package: docker.io
Followup-For: Bug #865975


The FORWARD chain policy is set to DROP by docker since 1.13.

The verbose (-V) iptables output (which gives interfaces and packet counters) is:
# iptables -L -v -n
Chain INPUT (policy ACCEPT 281 packets, 14176 bytes)
 pkts bytes target     prot opt in     out     source               destination

Chain FORWARD (policy DROP 0 packets, 0 bytes)
 pkts bytes target     prot opt in     out     source               destination
    0     0 DOCKER-ISOLATION  all  --  *      *       0.0.0.0/0            0.0.0.0/0
    0     0 DOCKER     all  --  *      docker0  0.0.0.0/0            0.0.0.0/0
    0     0 ACCEPT     all  --  *      docker0  0.0.0.0/0            0.0.0.0/0            ctstate RELATED,ESTABLISHED
    0     0 ACCEPT     all  --  docker0 !docker0  0.0.0.0/0            0.0.0.0/0
    0     0 ACCEPT     all  --  docker0 docker0  0.0.0.0/0            0.0.0.0/0

Chain OUTPUT (policy ACCEPT 225 packets, 27980 bytes)
 pkts bytes target     prot opt in     out     source               destination

Chain DOCKER (1 references)
 pkts bytes target     prot opt in     out     source               destination

Chain DOCKER-ISOLATION (1 references)
 pkts bytes target     prot opt in     out     source               destination
    0     0 RETURN     all  --  *      *       0.0.0.0/0            0.0.0.0/0


I reproduced the network setup but not the KVM one.I cannot confirm
that forwarding is broken.

Upstream provides:

- a command line switch to docker daemon "--iptables=false"
or a config item in /etc/docker/daemon.json:
{
  "iptables": false
}

- upstream also tell to revert the FORWARD policy to ACCEPT byhand ...
but I tested and it stay so on docker restart (even stop and start).
If the box is rebooted the change is lost
as confirmed by https://docs.docker.com/engine/userguide/networking/default_network/container-communication/
"The iptables settings are lost when the system reboots. If you want the change to be permanent,
refer to your Linux distribution’s documentation."
Mind we cannot apply it from /etc/rc.local or anything boot related as it has to be applied
after docker is started ...
with socket activation we activate docker daemon long after boot.



references:

- https://docs.docker.com/engine/userguide/networking/default_network/container-communication/
Container communication between hosts
For security reasons, Docker configures the iptables rules to prevent containers from forwarding traffic
from outside the host machine, on Linux hosts. Docker sets the default policy of the FORWARD chain to DROP.
(...)
Note: In Docker 1.12 and earlier, the default FORWARD chain policy was ACCEPT. When you upgrade
to Docker 1.13 or higher, this default is automatically changed for you.



- Also from https://docs.docker.com/engine/userguide/networking/default_network/container-communication/
Communication between containers
(...)
Docker will never make changes to your system iptables rules if you set --iptables=false when the daemon starts.
Otherwise the Docker server will add a default rule to the FORWARD chain with a blanket ACCEPT policy
if you retain the default --icc=true, or else will set the policy to DROP if --icc=false.



Best regards
Alban

-- System Information:
Debian Release: buster/sid
  APT prefers unstable-debug
  APT policy: (500, 'unstable-debug'), (500, 'stable-updates'), (500, 'unstable'), (500, 'stable'), (1, 'experimental-debug'), (1, 'experimental')
Architecture: amd64 (x86_64)
Foreign Architectures: i386

Kernel: Linux 4.13.0-1-amd64 (SMP w/4 CPU cores)
Locale: LANG=fr_FR.UTF-8, LC_CTYPE=fr_FR.UTF-8 (charmap=UTF-8), LANGUAGE=fr_FR.UTF-8 (charmap=UTF-8)
Shell: /bin/sh linked to /bin/dash
Init: systemd (via /run/systemd/system)

Versions of packages docker.io depends on:
ii  adduser             3.116
ii  docker-containerd   0.2.3+git+docker1.13.1~ds1-1
ii  docker-runc         1.0.0~rc2+git+docker1.13.1~ds1-2
ii  golang-libnetwork   0.8.0-dev.2+git20170202.599.45b4086-3
ii  iptables            1.6.1-2+b1
ii  libapparmor1        2.11.0-11
ii  libc6               2.24-17
ii  libdevmapper1.02.1  2:1.02.142-1
ii  libsqlite3-0        3.20.1-2
ii  libsystemd0         235-2
ii  lsb-base            9.20170808

Versions of packages docker.io recommends:
ii  ca-certificates  20170717
ii  cgroupfs-mount   1.4
ii  git              1:2.15.0~rc1-1
ii  xz-utils         5.2.2-1.3

Versions of packages docker.io suggests:
ii  aufs-tools           1:4.1+20161219-1
ii  btrfs-progs          4.13.3-1
ii  debootstrap          1.0.91
pn  docker-doc           <none>
ii  rinse                3.2
pn  zfs-fuse | zfsutils  <none>

-- no debconf information
Reply | Threaded
Open this post in threaded view
|

Bug#865975: #865975 docker.io breaks (bridged) network for VMs

Marvin Renich
In reply to this post by Roland Kammerer
Control: severity -1 critical

* Alban Browaeys <[hidden email]> [171022 00:01]:
> The FORWARD chain policy is set to DROP by docker since 1.13.

HUH???  Since when is it okay for software that is not advertised as
firewall software to modify the sysadmin's implicit or explicit iptables
setup for IP packets completely unrelated to said software?

I understand that this is a non-trivial problem because of the history
behind /proc/sys/net/ipv4/ip_forward and the FORWARD chain, but docker
can and needs to do better.  My first approximation would be to only
change the policy for FORWARD if ip_forward was 0 before docker added
its own iptables rules.  This will probably work 99.9% of the time.

The current behavior appears to break other software 100% of the time
if that software relies on ip_forward being 1 and the FORWARD chain
being in its default state.

* Dmitry Smirnov <[hidden email]> [180607 09:31]:
> I'm lowering severity of this bug since it is not clear how to reproduce the
> problem and because networking is not broken for everyone...

I'm raising this back to critical (makes unrelated software on the
system break), as I cannot fathom how this could not be broken for
anyone using KVM with a bridging network interface unless they have
found one of the workarounds given in this thread (i.e. the
/etc/docker/daemon.json modification, or manually modifying the FORWARD
chain).

The following should reliably reproduce the problem (disclaimer: I
haven't gone back to a clean system and tried this from the start, but I
have been using virt-manager for a while, and installing docker.io
immediately broke networking on my VMs):

* Ensure dockerd is not running and iptables is empty and in its default
  state (FORWARD chain policy is ACCEPT).
* Install virt-manager.
* Create a virtual machine with a bridging virtual network interface
  (e.g. for network source choose "Bridge br0: Host device ens3" where
  ens3 is the host's wired interface to the LAN).
* Ensure networking on the VM works correctly (to the LAN or beyond),
  and that FORWARD chain still has POLICY ACCEPT.
* Start dockerd.

The VM's networking to the LAN or beyond (but not to the host machine)
will now be broken.

Another scenario that I expect to reliably fail is machine A with two
wired NIC's, one connected to the LAN and the other connected directly
to isolated machine B (e.g. a Raspberry Pi).  The connection between A
and B could be USB networking.

Ensure:

* The LAN gateway has a route to B via A.  (Using a different /24
  between A and B is recommended here.)
* Machine A has ip_forward set to 1, and empty, default FORWARD chain.
* Machine A has a route to B.
* Machine B can reach outside the LAN.

I would be extremely surprised if installing docker.io did not break
this scenario for machine B.

...Marvin

Reply | Threaded
Open this post in threaded view
|

Bug#865975: #865975 docker.io breaks (bridged) network for VMs

Marvin Renich
In reply to this post by Roland Kammerer
[I'm subscribed to this bug, you don't need to include me in TO or CC.]

* Dmitry Smirnov <[hidden email]> [181126 20:45]:
> This bug is basically a duplicate of #903635.

Not really.  Bug #903635 is about not obeying a documented command line
option.  This bug is about incorrectly modifying iptables when it is
enabled.  I.e. when --iptables=true, the modification is done is wrong
(in a way that breaks networking configuration for non-docker software).

> On Tuesday, 27 November 2018 9:00:20 AM AEDT Marvin Renich wrote:
> > The current behavior appears to break other software 100% of the time
> > if that software relies on ip_forward being 1 and the FORWARD chain
> > being in its default state.
>
> Situation is unfortunate and upstream does not care enough. They probably
> want Docker users to use Docker exclusively without any other software on the
> host. I very much dislike this approach but that's Docker for you... :(

I appreciate your candor regarding upstream's attitude.

Would you mind forwarding this upstream anyway, to give them the
opportunity to do the right thing?  Here is my suggestion to them to fix
this bug:

1.  Only change the FORWARD chain policy if ip_forward was 0 before
    adding rules to this chain (obviously docker sets ip_forward to 1 if
    it wasn't already).

2.  Only add two rules to FORWARD:

    -A FORWARD -o docker0 -j DOCKER-IN
    -A FORWARD -i docker0 -j DOCKER-OUT

Add all other rules to docker-specific chains (DOCKER-IN and -OUT can
jump to DOCKER-USER and -ISOLATION-* as desired).  This ensures that
only docker-related packets make it to the docker chains, so the docker
chains cannot modify non-docker packets.

This has the advantage that you don't need to keep testing -i docker0 or
-o docker0 on multiple rules, making the filtering more efficient.

> About 6 months ago I've moved all my application containers from Docker to
> "rkt" and I couldn't be happier. Although still immature, in the future
> libpod/podman will likely become a worthy Docker successor.

Thanks for this tip; I'll investigate.

> > I'm raising this back to critical (makes unrelated software on the
> > system break),
>
> I'm lowering severity back to "important". You are not wrong that Docker is
> hostile to other applications but I think many users would agree that we need
> Docker in "testing" and upcoming Debian release despite of this problem.

Okay.  I understand and agree that the current situation is better than
not having docker in the archive.  Since this has the potential to break
other software on initial installation or upgrade from an older version,
can you put an item in NEWS.Debian pointing to this bug and mentioning
the /etc/docker/daemon.json workaround?

Thanks for your work packaging docker!

...Marvin

Reply | Threaded
Open this post in threaded view
|

Bug#865975: #865975 docker.io breaks (bridged) network for VMs

Jonathan Dowland
In reply to this post by Roland Kammerer
severity 865975 critical
thanks

My report just got merged into this one as a duplicate, so sorry for being
late to the party…

On Tue, Nov 27, 2018 at 12:42:28PM +1100, Dmitry Smirnov wrote:
>I'm lowering severity back to "important". You are not wrong that Docker is
>hostile to other applications but I think many users would agree that we need
>Docker in "testing" and upcoming Debian release despite of this problem.

With respect Dmitry, I don't think that this is an appropriate reason to lower
the severity.

I agree that we should ship Docker in Buster, even if we have not resolved this
issue in time, since we couldn't introduce it after the fact, and it's an
important and/or high-profile package.

But the bug clearly meets the criteria for Critical severity: "makes unrelated
software on the system (or the whole system) break": one can witness unrelated
VMs lose networking in real time by starting the docker daemon and triggering
the policy change.

The proper way to try and ensure that Docker does not get dropped from Buster
is to either fix the bug or request an exception from the release team (a
"buster-ignore" tag). Lowering the severity to avoid triggering a removal is just
hiding the bug from RC bug summaries and dashboards etc., violating SC §3 "We
will not hide problems".

I'm fairly confident that the release team would tag this buster-ignore. But,
the preferred solution is, of course, to fix the bug.

As Clint points out in [1], a solution is to ship a minimal conffile.
On the face of it that seems like a simple solution, which has not been discussed
by anyone else yet in the bug. I've tested the concept and it works, but needs
cooking up into a patch. What are the caveats for this? At a minimum we
would need a README.Debian too, I guess.


[1] <[hidden email]>

--

⢀⣴⠾⠻⢶⣦⠀
⣾⠁⢠⠒⠀⣿⡁ Jonathan Dowland
⢿⡄⠘⠷⠚⠋⠀ https://jmtd.net
⠈⠳⣄⠀⠀⠀⠀

Reply | Threaded
Open this post in threaded view
|

Bug#865975: docker.io changes iptables default FORWARD policy to DROP, breaks VM and others

Shengjing Zhu-3
In reply to this post by Roland Kammerer
Hi,

I checked more carefully on https://github.com/moby/moby/pull/28257
and https://github.com/moby/moby/issues/14041
Then I concluded that docker does nothing wrong in this case.

If you didn't set net.ipv4.ip_forward=1 before starting docker, then
docker will set this for you by default, otherwise the containers
can't access the network. This causes security issue as described in
https://github.com/moby/moby/issues/14041.
So if docker set net.ipv4.ip_forward=1 itself, it will set the default
FORWARD policy to DROP. This looks quite correct.

So when docker will not touch your FORWARD policy? just don't let
docker enable ip_forward itself. You can set net.ipv4.ip_forward=1 in
/etc/sysctl.conf(enable it before starting docker). Then docker will
know that user want the host to forward all traffic and it will touch
your default FORWARD policy.

I've verified it by adding net.ipv4.ip_forward=1 to /etc/sysctl.conf,
then reboot. And my FORWARD policy is ACCEPT.

So as for your VM scenario, why didn't you set ip_forward manually?
How docker know it's not a vulnerability if it didn't set FORWARD
chain to DROP when it enables ip_forward.

--
Shengjing Zhu

Reply | Threaded
Open this post in threaded view
|

Bug#865975: docker.io changes iptables default FORWARD policy to DROP, breaks VM and others

Shengjing Zhu-3
Control: severity -1 normal

On Tue, Jun 11, 2019 at 6:09 PM Shengjing Zhu <[hidden email]> wrote:
>
> Hi,
>
> I checked more carefully on https://github.com/moby/moby/pull/28257
> and https://github.com/moby/moby/issues/14041
> Then I concluded that docker does nothing wrong in this case.
>
[...]

With the reason I explained last week, I would downgrade this issue.

Arnaud,
when you upload new version for the CVE issue, could you amend the
README.Debian to tell people that if they don't want docker to set
default FORWARD policy to DROP, they should enable ip_forward
intentionally. (I bet your english is better than me to draft a
phrase...)

--
Shengjing Zhu

Reply | Threaded
Open this post in threaded view
|

Bug#865975: docker.io changes iptables default FORWARD policy to DROP, breaks VM and others

Arnaud Rebillout-3

On 6/17/19 3:22 AM, Shengjing Zhu wrote:

> Control: severity -1 normal
>
> On Tue, Jun 11, 2019 at 6:09 PM Shengjing Zhu <[hidden email]> wrote:
>> Hi,
>>
>> I checked more carefully on https://github.com/moby/moby/pull/28257
>> and https://github.com/moby/moby/issues/14041
>> Then I concluded that docker does nothing wrong in this case.
>>
> [...]
>
> With the reason I explained last week, I would downgrade this issue.
>
> Arnaud,
> when you upload new version for the CVE issue, could you amend the
> README.Debian to tell people that if they don't want docker to set
> default FORWARD policy to DROP, they should enable ip_forward
> intentionally. (I bet your english is better than me to draft a
> phrase...)
>
  Hi,

I will, and can even go further and display an explicit message when
user upgrade the package maybe? (I'm not sure how to do that but I'm
sure I'll find an example easily).

I'm quite busy now but I hope I'll have time before the end of the week
to do all of that.

Thanks,

Reply | Threaded
Open this post in threaded view
|

Bug#865975: docker.io changes iptables default FORWARD policy to DROP, breaks VM and others

Shengjing Zhu-3
On Mon, Jun 17, 2019 at 10:16 AM Arnaud Rebillout
<[hidden email]> wrote:
>   Hi,
>
> I will, and can even go further and display an explicit message when
> user upgrade the package maybe? (I'm not sure how to do that but I'm
> sure I'll find an example easily).

docker.io already has a NEWS file, you can just add a new entry, by running:

dch --news debian/docker.io.NEWS

but I don't think it's needed... docker.io is not in stretch, so
people will not upgrade it from stretch. And this issue exists since
1.13.0(long ago).

--
Shengjing Zhu

Reply | Threaded
Open this post in threaded view
|

Bug#865975: docker.io changes iptables default FORWARD policy to DROP, breaks VM and others

Jonathan Dowland
In reply to this post by Arnaud Rebillout-3
On Mon, Jun 17, 2019 at 09:16:18AM +0700, Arnaud Rebillout wrote:
>I will, and can even go further and display an explicit message when
>user upgrade the package maybe? (I'm not sure how to do that but I'm
>sure I'll find an example easily).

That, in particular I think is a bad idea. It seems like it might be useful
when you consider one package in isolation, but then upgrades become really
painful when a whole load of such packages all have their messages one after
the other.


--

Jonathan Dowland

Reply | Threaded
Open this post in threaded view
|

Bug#865975: docker.io changes iptables default FORWARD policy to DROP, breaks VM and others

Jonathan Dowland
In reply to this post by Shengjing Zhu-3
On Mon, Jun 17, 2019 at 04:22:30AM +0800, Shengjing Zhu wrote:

>Control: severity -1 normal
>
>On Tue, Jun 11, 2019 at 6:09 PM Shengjing Zhu <[hidden email]> wrote:
>> I checked more carefully on https://github.com/moby/moby/pull/28257
>> and https://github.com/moby/moby/issues/14041
>> Then I concluded that docker does nothing wrong in this case.
>>
>[...]
>
>With the reason I explained last week, I would downgrade this issue.

Sorry for not replying sooner, I had a very busy week.

Your code review and conclusions from it are reasonable. I read them
last week (although I did not have time to reply) and my thoughts were:
is that the code in the current Debian package?

I ask because the reason I filed this in the first place is because I
hit it on my main Debian system. My kvm/libvirt-based VMs lost their
networking due to docker changing the forward iptables policy. I had
not manually modified the ip_forward /proc setting, so either

 1) this was not necessary for libvirt/kvm's networking to function,
    and so is not a perfect test as to whether changing the forwarding
    iptables chain will break something

or

 2) libvirt/kvm's networking set the ip_forward proc setting it required,
    and the packaged docker does not correctly check the proc value before
    making the change, despite the logic in the code linked above (or
    perhaps that is not the code corresponding to the docker version that
    is presently packaged)

(Or I suppose 3), a race is possible between the /proc check and the iptables
change, but I don't think that's likely what has happened in my case.)

So IMHO the severity drop is not appropriate because it's predicated on a
theoretical reasoning of the situation rather than a practical one, i.e., I
actually hit this on a real machine in real circumstances. But I will attempt
to reproduce it a second time on my real machine before challenging the
severity again.

To respond to some of your specific points:

> So as for your VM scenario, why didn't you set ip_forward manually?

Because I did not need to. Either it was not necessary for the specific
networking setup for my libvirt/kvm VMs, or libvirt/kvm set it explicitly
itself. I do not manually need to tweak stuff in /proc. As part of
reproducing this I will check the specific network setup that the libvirt
VMs are using, but it's the package/debian default.

> How docker know it's not a vulnerability if it didn't set FORWARD
> chain to DROP when it enables ip_forward.

They could add a "-j DROP" rule that was scoped specifically to the docker
subnet, after their other (-j ACCEPT) rules. That's just one way that this
could be done less disruptively.

--

Jonathan Dowland

Reply | Threaded
Open this post in threaded view
|

Bug#865975: docker.io changes iptables default FORWARD policy to DROP, breaks VM and others

Shengjing Zhu-3
On Mon, Jun 17, 2019 at 5:56 PM Jonathan Dowland <[hidden email]> wrote:

>
> On Mon, Jun 17, 2019 at 04:22:30AM +0800, Shengjing Zhu wrote:
> >Control: severity -1 normal
> >
> >On Tue, Jun 11, 2019 at 6:09 PM Shengjing Zhu <[hidden email]> wrote:
> >> I checked more carefully on https://github.com/moby/moby/pull/28257
> >> and https://github.com/moby/moby/issues/14041
> >> Then I concluded that docker does nothing wrong in this case.
> >>
> >[...]
> >
> >With the reason I explained last week, I would downgrade this issue.
>
> Sorry for not replying sooner, I had a very busy week.
>
> Your code review and conclusions from it are reasonable. I read them
> last week (although I did not have time to reply) and my thoughts were:
> is that the code in the current Debian package?
>

Yes, you can check it here
https://sources.debian.org/src/docker.io/18.09.1+dfsg1-7/libnetwork/drivers/bridge/setup_ip_forwarding.go/#L31

The code is only called if ip_forward is not enabled.

> I ask because the reason I filed this in the first place is because I
> hit it on my main Debian system. My kvm/libvirt-based VMs lost their
> networking due to docker changing the forward iptables policy. I had
> not manually modified the ip_forward /proc setting, so either
>
>  1) this was not necessary for libvirt/kvm's networking to function,
>     and so is not a perfect test as to whether changing the forwarding
>     iptables chain will break something
>
> or
>
>  2) libvirt/kvm's networking set the ip_forward proc setting it required,
>     and the packaged docker does not correctly check the proc value before
>     making the change, despite the logic in the code linked above (or
>     perhaps that is not the code corresponding to the docker version that
>     is presently packaged)
>
> (Or I suppose 3), a race is possible between the /proc check and the iptables
> change, but I don't think that's likely what has happened in my case.)
>
> So IMHO the severity drop is not appropriate because it's predicated on a
> theoretical reasoning of the situation rather than a practical one, i.e., I
> actually hit this on a real machine in real circumstances. But I will attempt
> to reproduce it a second time on my real machine before challenging the
> severity again.

Please do think more about this issue. And understand why docker does
this for the security reason.

>
> To respond to some of your specific points:
>
> > So as for your VM scenario, why didn't you set ip_forward manually?
>
> Because I did not need to. Either it was not necessary for the specific
> networking setup for my libvirt/kvm VMs, or libvirt/kvm set it explicitly
> itself. I do not manually need to tweak stuff in /proc. As part of
> reproducing this I will check the specific network setup that the libvirt
> VMs are using, but it's the package/debian default.
>

Because libvirt enables ip_forward itself.

https://sources.debian.org/src/libvirt/5.0.0-3/src/network/bridge_driver.c/#L2207

And I would argue this is a security issue too, if libvirt enables
ip_forward and does nothing else.

> > How docker know it's not a vulnerability if it didn't set FORWARD
> > chain to DROP when it enables ip_forward.
>
> They could add a "-j DROP" rule that was scoped specifically to the docker
> subnet, after their other (-j ACCEPT) rules. That's just one way that this
> could be done less disruptively.

No, because they enabled ip_forward setting.


--
Shengjing Zhu

Reply | Threaded
Open this post in threaded view
|

Bug#865975: docker.io changes iptables default FORWARD policy to DROP, breaks VM and others

Jonathan Dowland
On Mon, Jun 17, 2019 at 07:26:04PM +0800, Shengjing Zhu wrote:
>Please do think more about this issue. And understand why docker does
>this for the security reason.

I understand the security issue. I understand why it does it. But if its
the case that this does break unrelated software, i.e., if the ip_forward
check is not sufficient/is not working (I am still to retest), then we
also have that problem. And it's not clear which is "worse".

>And I would argue this is a security issue too, if libvirt enables
>ip_forward and does nothing else.

I agree.

>> They could add a "-j DROP" rule that was scoped specifically to the docker
>> subnet, after their other (-j ACCEPT) rules. That's just one way that this
>> could be done less disruptively.
>
>No, because they enabled ip_forward setting.

Sure, but using a -j DROP rule means it's at least theoretically possible
for unrelated software to have its own rules in the forward chain that are
not broken by the chain policy change. Having said that, I'm only sketching
the outline of an alternative solution here; it would need working up into
a proper alternative solution, and I do not have the time to do that.


--

⢀⣴⠾⠻⢶⣦⠀
⣾⠁⢠⠒⠀⣿⡁ Jonathan Dowland
⢿⡄⠘⠷⠚⠋⠀ https://jmtd.net
⠈⠳⣄⠀⠀⠀⠀ Please do not CC me, I am subscribed to the list.

Reply | Threaded
Open this post in threaded view
|

Bug#865975: docker.io changes iptables default FORWARD policy to DROP, breaks VM and others

Jonathan Dowland
In reply to this post by Shengjing Zhu-3
Hi Shengjing Zhu (et al)

I've just (finally) attempted to reproduce this on my Buster host, but
could not on this attempt. Libvirtd did not change my ip_forward setting
from 0 to 1 in the test, but I had to do so manually to re-enable VM
networking outside of the host (I don't think I did this manually in the
first instance). Docker did not change the FORWARD chain policy since
ip_forward was set to 1. My libvirtd VMs are using the default bridged
network.

I'll keep trying to reproduce this but for now let's assume that it doesn't
happen.