Bug#888647: lxc: unprivileged container doesn't boot due to cgroup ownership

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

Bug#888647: lxc: unprivileged container doesn't boot due to cgroup ownership

Andrea Villa
Package: lxc
Version: 1:2.0.7-2+deb9u1
Severity: normal
Tags: patch

Dear Maintainer,

   * What led up to the situation?

   Just create a simple user unprivileged lxc container after following the official Debian documentation https://wiki.debian.org/LXC#Unprivileged_container.

   Container fails when started with:

   ----------------
         lxc-start 20170124115651.107 ERROR    lxc_cgfs - cgroups/cgfs.c:lxc_cgroupfs_create:909 - Could not set clone_children to 1 for cpuset hierarchy in parent cgroup.
         lxc-start 20170124115651.107 ERROR    lxc_cgfs - cgroups/cgfs.c:cgroup_rmdir:209 - Read-only file system - cgroup_rmdir: failed to delete /sys/fs/cgroup/perf_event/
         lxc-start 20170124115651.107 ERROR    lxc_cgfs - cgroups/cgfs.c:cgroup_rmdir:209 - Read-only file system - cgroup_rmdir: failed to delete /sys/fs/cgroup/cpuset/
         lxc-start 20170124115651.107 ERROR    lxc_cgfs - cgroups/cgfs.c:cgroup_rmdir:209 - Read-only file system - cgroup_rmdir: failed to delete /sys/fs/cgroup/net_cls,net_prio/
         lxc-start 20170124115651.107 ERROR    lxc_cgfs - cgroups/cgfs.c:cgroup_rmdir:209 - Permission denied - cgroup_rmdir: failed to delete /sys/fs/cgroup/pids/user.slice/user-1000.slice/session-2.scope
         lxc-start 20170124115651.108 ERROR    lxc_cgfs - cgroups/cgfs.c:cgroup_rmdir:209 - Permission denied - cgroup_rmdir: failed to delete /sys/fs/cgroup/memory/user.slice
         lxc-start 20170124115651.108 ERROR    lxc_cgfs - cgroups/cgfs.c:cgroup_rmdir:209 - Read-only file system - cgroup_rmdir: failed to delete /sys/fs/cgroup/freezer/
         lxc-start 20170124115651.108 ERROR    lxc_cgfs - cgroups/cgfs.c:cgroup_rmdir:209 - Permission denied - cgroup_rmdir: failed to delete /sys/fs/cgroup/blkio/user.slice
         lxc-start 20170124115651.108 ERROR    lxc_cgfs - cgroups/cgfs.c:cgroup_rmdir:209 - Permission denied - cgroup_rmdir: failed to delete /sys/fs/cgroup/cpu,cpuacct/user.slice
         lxc-start 20170124115651.109 ERROR    lxc_cgfs - cgroups/cgfs.c:cgroup_rmdir:209 - Permission denied - cgroup_rmdir: failed to delete /sys/fs/cgroup/devices/user.slice
         lxc-start 20170124115651.109 ERROR    lxc_cgfs - cgroups/cgfs.c:cgroup_rmdir:209 - Permission denied - cgroup_rmdir: failed to delete /sys/fs/cgroup/systemd/user.slice/user-1000.slice/session-2.scope
         lxc-start 20170124115651.109 ERROR    lxc_start - start.c:lxc_spawn:1108 - Failed creating cgroups.
         lxc-start 20170124115651.109 ERROR    lxc_start - start.c:__lxc_start:1346 - Failed to spawn container "ubuntu".
         lxc-start 20170124115651.616 ERROR    lxc_start_ui - tools/lxc_start.c:main:366 - The container failed to start.
         lxc-start 20170124115651.616 ERROR    lxc_start_ui - tools/lxc_start.c:main:370 - Additional information can be obtained by setting the --logfile and --logpriority options.
   ----------------

   * What exactly did you do (or not do) that was effective (or
     ineffective)?

   I have found this thread on LXC forums https://discuss.linuxcontainers.org/t/failed-creating-cgroups/272/4 that suggests to use the Ubuntu's version of the libpam-cgfs package.
   The Ubuntu version of the package seems to include some patches that properly set user's CGroups permission upon user's login.

   * What was the outcome of this action?

         Installing the Ubuntu version of the libpam-cgfs fixes the problem.


I was not sure if I should have posted the bug here on in libpam-cfgs. I hope you don't mind my choice.

Bests,

Andrea


-- System Information:
Debian Release: 9.3
  APT prefers stable-updates
  APT policy: (500, 'stable-updates'), (500, 'stable'), (400, 'unstable')
Architecture: amd64 (x86_64)

Kernel: Linux 4.14.0-0.bpo.3-amd64 (SMP w/8 CPU cores)
Locale: LANG=en_US.UTF-8, LC_CTYPE=en_US.UTF-8 (charmap=UTF-8), LANGUAGE=en_US:en (charmap=UTF-8)
Shell: /bin/sh linked to /bin/dash
Init: systemd (via /run/systemd/system)

Versions of packages lxc depends on:
ii  init-system-helpers  1.48
ii  libapparmor1         2.11.0-3
ii  libc6                2.24-11+deb9u1
ii  libcap2              1:2.25-1
ii  libgnutls30          3.5.8-5+deb9u3
ii  liblxc1              1:2.0.7-2+deb9u1
ii  libseccomp2          2.3.1-2.1
ii  libselinux1          2.6-3+b3
ii  lsb-base             9.20161125
ii  python3              3.5.3-1
ii  python3-lxc          1:2.0.7-2+deb9u1

Versions of packages lxc recommends:
ii  bridge-utils  1.5-13+deb9u1
ii  debootstrap   1.0.92~bpo9+1
ii  dirmngr       2.1.18-8~deb9u1
ii  dnsmasq-base  2.76-5+deb9u1
ii  gnupg         2.1.18-8~deb9u1
ii  iptables      1.6.1-2~bpo9+1
ii  libpam-cgfs   2.0.7-1
ii  lxcfs         2.0.7-1
ii  openssl       1.1.0f-3+deb9u1
ii  rsync         3.1.2-1+deb9u1
ii  uidmap        1:4.4-4.1

Versions of packages lxc suggests:
ii  apparmor     2.11.0-3
pn  btrfs-tools  <none>
ii  lvm2         2.02.168-2

-- Configuration Files:
/etc/lxc/default.conf changed [not included]

-- no debconf information
Reply | Threaded
Open this post in threaded view
|

Bug#888647: [pkg-lxc-devel] Bug#888647: lxc: unprivileged container doesn't boot due to cgroup ownership

Evgeni Golov-2
Hi Andrea,

On Sun, Jan 28, 2018 at 11:34:03AM +0100, Andrea Villa wrote:
>    Just create a simple user unprivileged lxc container after following the
> official Debian documentation https://wiki.debian.org/LXC#
> Unprivileged_container.

Can we for a second pretend, wiki.d.o is not official documentation,
thanks ;)

And looking at the page, it lists at least a few steps that should not
be needed.

>    Container fails when started with:
>
>    ----------------
>          lxc-start 20170124115651.107 ERROR    lxc_cgfs -
> cgroups/cgfs.c:lxc_cgroupfs_create:909 - Could not set clone_children to 1
> for cpuset hierarchy in parent cgroup.
>          lxc-start 20170124115651.107 ERROR    lxc_cgfs -
> cgroups/cgfs.c:cgroup_rmdir:209 - Read-only file system - cgroup_rmdir:
> failed to delete /sys/fs/cgroup/perf_event/

You're running a BPO kernel, right? Can you please try with the kernel
from stable?

I am running stable boxes with unprivileged containers just fine, so
there is something weird here, and it might very well be the kernel.

>    * What exactly did you do (or not do) that was effective (or
>      ineffective)?
>
>    I have found this thread on LXC forums https://discuss.
> linuxcontainers.org/t/failed-creating-cgroups/272/4 that suggests to use
> the Ubuntu's version of the libpam-cgfs package.
>    The Ubuntu version of the package seems to include some patches that
> properly set user's CGroups permission upon user's login.

Ubuntus version (which one, btw?) does not carry patches, their
packaging is usually just what we ship in Debian, plus sometimes faster
upstream releases.

>
>    * What was the outcome of this action?
>
>          Installing the Ubuntu version of the libpam-cgfs fixes the problem.
>
>
> I was not sure if I should have posted the bug here on in libpam-cfgs. I
> hope you don't mind my choice.

We can re-assing at will, so that's fine.

Evgeni

Reply | Threaded
Open this post in threaded view
|

Bug#888647: [pkg-lxc-devel] Bug#888647: lxc: unprivileged container doesn't boot due to cgroup ownership

Andrea Villa
Thanks for the reply Evgeni,


On Sun, Jan 28, 2018 at 12:24 PM, Evgeni Golov <[hidden email]> wrote:
Hi Andrea,

On Sun, Jan 28, 2018 at 11:34:03AM +0100, Andrea Villa wrote:
>    Just create a simple user unprivileged lxc container after following the
> official Debian documentation https://wiki.debian.org/LXC#
> Unprivileged_container.

Can we for a second pretend, wiki.d.o is not official documentation,
thanks ;)

And looking at the page, it lists at least a few steps that should not
be needed.

>    Container fails when started with:
>
>    ----------------
>          lxc-start 20170124115651.107 ERROR    lxc_cgfs -
> cgroups/cgfs.c:lxc_cgroupfs_create:909 - Could not set clone_children to 1
> for cpuset hierarchy in parent cgroup.
>          lxc-start 20170124115651.107 ERROR    lxc_cgfs -
> cgroups/cgfs.c:cgroup_rmdir:209 - Read-only file system - cgroup_rmdir:
> failed to delete /sys/fs/cgroup/perf_event/

You're running a BPO kernel, right? Can you please try with the kernel
from stable?

I am running stable boxes with unprivileged containers just fine, so
there is something weird here, and it might very well be the kernel.

I've just tried with kernel 4.9.65-3+deb9u2, which should be the latest for stretch without backports. LXC refuses to start the containers


>    * What exactly did you do (or not do) that was effective (or
>      ineffective)?
>
>    I have found this thread on LXC forums https://discuss.
> linuxcontainers.org/t/failed-creating-cgroups/272/4 that suggests to use
> the Ubuntu's version of the libpam-cgfs package.
>    The Ubuntu version of the package seems to include some patches that
> properly set user's CGroups permission upon user's login.

Ubuntus version (which one, btw?) does not carry patches, their
packaging is usually just what we ship in Debian, plus sometimes faster
upstream releases.

>
>    * What was the outcome of this action?
>
>          Installing the Ubuntu version of the libpam-cgfs fixes the problem.
>
>
> I was not sure if I should have posted the bug here on in libpam-cfgs. I
> hope you don't mind my choice.

We can re-assing at will, so that's fine.

Evgeni

Reply | Threaded
Open this post in threaded view
|

Bug#888647: [pkg-lxc-devel] Bug#888647: lxc: unprivileged container doesn't boot due to cgroup ownership

Andrea Villa
In reply to this post by Evgeni Golov-2


On Sun, Jan 28, 2018 at 12:24 PM, Evgeni Golov <[hidden email]> wrote:
Hi Andrea,

On Sun, Jan 28, 2018 at 11:34:03AM +0100, Andrea Villa wrote:

>    * What exactly did you do (or not do) that was effective (or
>      ineffective)?
>
>    I have found this thread on LXC forums https://discuss.
> linuxcontainers.org/t/failed-creating-cgroups/272/4 that suggests to use
> the Ubuntu's version of the libpam-cgfs package.
>    The Ubuntu version of the package seems to include some patches that
> properly set user's CGroups permission upon user's login.

Ubuntu version (which one, btw?) does not carry patches, their
packaging is usually just what we ship in Debian, plus sometimes faster
upstream releases.

Installing Ubuntu's libpam-cgfs_2.0.7-0ubuntu4, replacing Debian's 2.0.7-1 fixes the issue!
 

>
>    * What was the outcome of this action?
>
>          Installing the Ubuntu version of the libpam-cgfs fixes the problem.
>
>
> I was not sure if I should have posted the bug here on in libpam-cfgs. I
> hope you don't mind my choice.

We can re-assing at will, so that's fine.

Evgeni

Reply | Threaded
Open this post in threaded view
|

Bug#888647: (no subject)

Fufu Fang
In reply to this post by Andrea Villa
I am experiencing the same problem as well. I tried installing
libpam-cfgs from unstable and Ubuntu, none of them solve my problem.

These is my log:

  lxc-start 20180919144245.625 ERROR    lxc_cgfs -
cgroups/cgfs.c:lxc_cgroupfs_create:1022 - Permission denied - Could not
create cgroup '/user.slice/user-1000.slice/session-4197.scope/lxc' in
'/sys/fs/cgroup/systemd'.
       lxc-start 20180919144245.628 ERROR    lxc_cgfs -
cgroups/cgfs.c:cgroup_rmdir:209 - Permission denied - cgroup_rmdir:
failed to delete /sys/fs/cgroup/devices/user.slice
       lxc-start 20180919144245.631 ERROR    lxc_cgfs -
cgroups/cgfs.c:cgroup_rmdir:209 - Permission denied - cgroup_rmdir:
failed to delete /sys/fs/cgroup/memory/user.slice
       lxc-start 20180919144245.632 ERROR    lxc_cgfs -
cgroups/cgfs.c:cgroup_rmdir:209 - Permission denied - cgroup_rmdir:
failed to delete
/sys/fs/cgroup/pids/user.slice/user-1000.slice/session-4197.scope
       lxc-start 20180919144245.634 ERROR    lxc_cgfs -
cgroups/cgfs.c:cgroup_rmdir:209 - Permission denied - cgroup_rmdir:
failed to delete /sys/fs/cgroup/blkio/user.slice
       lxc-start 20180919144245.634 ERROR    lxc_cgfs -
cgroups/cgfs.c:cgroup_rmdir:209 - Permission denied - cgroup_rmdir:
failed to delete /sys/fs/cgroup/perf_event//docker
       lxc-start 20180919144245.635 ERROR    lxc_cgfs -
cgroups/cgfs.c:cgroup_rmdir:209 - Permission denied - cgroup_rmdir:
failed to delete /sys/fs/cgroup/perf_event//machine
       lxc-start 20180919144245.635 ERROR    lxc_cgfs -
cgroups/cgfs.c:cgroup_rmdir:209 - Permission denied - cgroup_rmdir:
failed to delete /sys/fs/cgroup/perf_event//lxc
       lxc-start 20180919144245.635 ERROR    lxc_cgfs -
cgroups/cgfs.c:cgroup_rmdir:209 - Read-only file system - cgroup_rmdir:
failed to delete /sys/fs/cgroup/perf_event/
       lxc-start 20180919144245.636 ERROR    lxc_cgfs -
cgroups/cgfs.c:cgroup_rmdir:209 - Permission denied - cgroup_rmdir:
failed to delete /sys/fs/cgroup/net_cls,net_prio//docker
       lxc-start 20180919144245.636 ERROR    lxc_cgfs -
cgroups/cgfs.c:cgroup_rmdir:209 - Permission denied - cgroup_rmdir:
failed to delete /sys/fs/cgroup/net_cls,net_prio//machine
       lxc-start 20180919144245.636 ERROR    lxc_cgfs -
cgroups/cgfs.c:cgroup_rmdir:209 - Permission denied - cgroup_rmdir:
failed to delete /sys/fs/cgroup/net_cls,net_prio//lxc
       lxc-start 20180919144245.637 ERROR    lxc_cgfs -
cgroups/cgfs.c:cgroup_rmdir:209 - Read-only file system - cgroup_rmdir:
failed to delete /sys/fs/cgroup/net_cls,net_prio/
       lxc-start 20180919144245.637 ERROR    lxc_cgfs -
cgroups/cgfs.c:cgroup_rmdir:209 - Permission denied - cgroup_rmdir:
failed to delete /sys/fs/cgroup/freezer//docker
       lxc-start 20180919144245.638 ERROR    lxc_cgfs -
cgroups/cgfs.c:cgroup_rmdir:209 - Permission denied - cgroup_rmdir:
failed to delete /sys/fs/cgroup/freezer//user/root/0
       lxc-start 20180919144245.638 ERROR    lxc_cgfs -
cgroups/cgfs.c:cgroup_rmdir:209 - Permission denied - cgroup_rmdir:
failed to delete /sys/fs/cgroup/freezer//user/root
       lxc-start 20180919144245.639 ERROR    lxc_cgfs -
cgroups/cgfs.c:cgroup_rmdir:209 - Permission denied - cgroup_rmdir:
failed to delete /sys/fs/cgroup/freezer//user/fangfufu/0
       lxc-start 20180919144245.639 ERROR    lxc_cgfs -
cgroups/cgfs.c:cgroup_rmdir:209 - Permission denied - cgroup_rmdir:
failed to delete /sys/fs/cgroup/freezer//user/fangfufu
       lxc-start 20180919144245.639 ERROR    lxc_cgfs -
cgroups/cgfs.c:cgroup_rmdir:209 - Permission denied - cgroup_rmdir:
failed to delete /sys/fs/cgroup/freezer//user
       lxc-start 20180919144245.640 ERROR    lxc_cgfs -
cgroups/cgfs.c:cgroup_rmdir:209 - Permission denied - cgroup_rmdir:
failed to delete /sys/fs/cgroup/freezer//machine
       lxc-start 20180919144245.640 ERROR    lxc_cgfs -
cgroups/cgfs.c:cgroup_rmdir:209 - Permission denied - cgroup_rmdir:
failed to delete /sys/fs/cgroup/freezer//lxc
       lxc-start 20180919144245.640 ERROR    lxc_cgfs -
cgroups/cgfs.c:cgroup_rmdir:209 - Read-only file system - cgroup_rmdir:
failed to delete /sys/fs/cgroup/freezer/
       lxc-start 20180919144245.641 ERROR    lxc_cgfs -
cgroups/cgfs.c:cgroup_rmdir:209 - Permission denied - cgroup_rmdir:
failed to delete /sys/fs/cgroup/cpu,cpuacct/user.slice
       lxc-start 20180919144245.641 ERROR    lxc_cgfs -
cgroups/cgfs.c:cgroup_rmdir:209 - Permission denied - cgroup_rmdir:
failed to delete /sys/fs/cgroup/cpuset//docker
       lxc-start 20180919144245.642 ERROR    lxc_cgfs -
cgroups/cgfs.c:cgroup_rmdir:209 - Permission denied - cgroup_rmdir:
failed to delete /sys/fs/cgroup/cpuset//machine
       lxc-start 20180919144245.643 ERROR    lxc_cgfs -
cgroups/cgfs.c:cgroup_rmdir:209 - Permission denied - cgroup_rmdir:
failed to delete /sys/fs/cgroup/cpuset//lxc
       lxc-start 20180919144245.643 ERROR    lxc_cgfs -
cgroups/cgfs.c:cgroup_rmdir:209 - Read-only file system - cgroup_rmdir:
failed to delete /sys/fs/cgroup/cpuset/
       lxc-start 20180919144245.643 ERROR    lxc_cgfs -
cgroups/cgfs.c:cgroup_rmdir:209 - Permission denied - cgroup_rmdir:
failed to delete
/sys/fs/cgroup/systemd/user.slice/user-1000.slice/session-4197.scope
       lxc-start 20180919144245.644 ERROR    lxc_start -
start.c:lxc_spawn:1108 - Failed creating cgroups.
       lxc-start 20180919144245.644 ERROR    lxc_start -
start.c:__lxc_start:1346 - Failed to spawn container "debian-lxc".
       lxc-start 20180919144246.154 ERROR    lxc_start_ui -
tools/lxc_start.c:main:366 - The container failed to start.
       lxc-start 20180919144246.154 ERROR    lxc_start_ui -
tools/lxc_start.c:main:370 - Additional information can be obtained by
setting the --logfile and --logpriority options.