Bug#925899: lxc: Unprivileged containers fail to start after recent updates

classic Classic list List threaded Threaded
11 messages Options
Reply | Threaded
Open this post in threaded view
|

Bug#925899: lxc: Unprivileged containers fail to start after recent updates

Regis Smith
Package: lxc
Version: 1:3.1.0+really3.0.3-6
Severity: important

Dear Maintainer,

   * What led up to the situation?

apt update; apt upgrade

   * What exactly did you do (or not do) that was effective (or
     ineffective)?

As a normal user:
$ lxc-start -n test

   * What was the outcome of this action?

lxc-start: test: lxccontainer.c: wait_on_daemonized_start: 833 No such file or directory - Failed to receive the container state
lxc-start: test: tools/lxc_start.c: main: 330 The container failed to start
lxc-start: test: tools/lxc_start.c: main: 333 To get more details, run the container in foreground mode
lxc-start: test: tools/lxc_start.c: main: 336 Additional information can be obtained by setting the --logfile and --logpriority options

If I run it in the foreground instead I get

$ lxc-start -n test -F
lxc-start: test: lsm/apparmor.c: apparmor_prepare: 974 Cannot use generated profile: apparmor_parser not available
lxc-start: test: start.c: lxc_init: 899 Failed to initialize LSM
lxc-start: test: start.c: __lxc_start: 1917 Failed to initialize container "test"
lxc-start: test: tools/lxc_start.c: main: 330 The container failed to start
lxc-start: test: tools/lxc_start.c: main: 336 Additional information can be obtained by setting the --logfile and --logpriority options

   * What outcome did you expect instead?

A running container.  These used to work up until recently.  Now I can't stop
already running containers because I won't be able to restart them.

-- System Information:
Debian Release: buster/sid
  APT prefers testing
  APT policy: (500, 'testing')
Architecture: amd64 (x86_64)

Kernel: Linux 4.19.0-4-amd64 (SMP w/4 CPU cores)
Locale: LANG=en_US.UTF-8, LC_CTYPE=en_US.UTF-8 (charmap=UTF-8), LANGUAGE=en_US.UTF-8 (charmap=UTF-8)
Shell: /bin/sh linked to /bin/dash
Init: systemd (via /run/systemd/system)
LSM: AppArmor: enabled

Versions of packages lxc depends on:
ii  debconf [debconf-2.0]  1.5.71
ii  libc6                  2.28-8
ii  libcap2                1:2.25-2
ii  libgnutls30            3.6.6-2
ii  liblxc1                1:3.1.0+really3.0.3-6
ii  libseccomp2            2.3.3-4
ii  libselinux1            2.8-1+b1
ii  lsb-base               10.2019031300

Versions of packages lxc recommends:
ii  bridge-utils                 1.6-2
ii  debootstrap                  1.0.114
ii  dirmngr                      2.2.12-1
ii  dnsmasq-base [dnsmasq-base]  2.80-1
ii  gnupg                        2.2.12-1
ii  iproute2                     4.20.0-2
ii  iptables                     1.8.2-4
ii  libpam-cgfs                  1:3.1.0+really3.0.3-6
ii  lxc-templates                3.0.3-1
ii  lxcfs                        3.0.3-2
ii  nftables                     0.9.0-2
ii  openssl                      1.1.1b-1
ii  rsync                        3.1.3-6
ii  uidmap                       1:4.5-1.1

Versions of packages lxc suggests:
ii  apparmor     2.13.2-9
ii  btrfs-progs  4.20.1-2
ii  lvm2         2.03.02-2
pn  python3-lxc  <none>

-- Configuration Files:
/etc/lxc/default.conf changed:
lxc.net.0.type = veth
lxc.net.0.link = lxcbr0
lxc.net.0.flags = up
lxc.apparmor.profile = generated
lxc.apparmor.allow_nesting = 1


-- debconf information:
* lxc/auto_update_config: true

Reply | Threaded
Open this post in threaded view
|

Bug#925899: lxc: Unprivileged containers fail to start after recent updates

Pierre-Elliott Bécue-3
Le mercredi 27 mars 2019 à 22:08:49-0700, Regis Smith a écrit :

> Package: lxc
> Version: 1:3.1.0+really3.0.3-6
> Severity: important
>
> Dear Maintainer,
>
>    * What led up to the situation?
>
> apt update; apt upgrade
>
>    * What exactly did you do (or not do) that was effective (or
>      ineffective)?
>
> As a normal user:
> $ lxc-start -n test
>
>    * What was the outcome of this action?
>
> lxc-start: test: lxccontainer.c: wait_on_daemonized_start: 833 No such file or directory - Failed to receive the container state
> lxc-start: test: tools/lxc_start.c: main: 330 The container failed to start
> lxc-start: test: tools/lxc_start.c: main: 333 To get more details, run the container in foreground mode
> lxc-start: test: tools/lxc_start.c: main: 336 Additional information can be obtained by setting the --logfile and --logpriority options
>
> If I run it in the foreground instead I get
>
> $ lxc-start -n test -F
> lxc-start: test: lsm/apparmor.c: apparmor_prepare: 974 Cannot use generated profile: apparmor_parser not available
> lxc-start: test: start.c: lxc_init: 899 Failed to initialize LSM
> lxc-start: test: start.c: __lxc_start: 1917 Failed to initialize container "test"
> lxc-start: test: tools/lxc_start.c: main: 330 The container failed to start
> lxc-start: test: tools/lxc_start.c: main: 336 Additional information can be obtained by setting the --logfile and --logpriority options
>
>    * What outcome did you expect instead?
>
> A running container.  These used to work up until recently.  Now I can't stop
> already running containers because I won't be able to restart them.
Hi,

Thanks for submitting this bug.

As you can see, it is possible to get more debug via the --logfile and
the --logpriority options.

That said, the first line with the -F option says it all:

> lxc-start: test: lsm/apparmor.c: apparmor_prepare: 974 Cannot use
> generated profile: apparmor_parser not available

It means that you're lacking the apparmor_parser command, which is
shipped by apparmor. It probably means that you refused to install
apparmor on your host.

You have multiple choices. The first one being installing apparmor, and
the second one being to edit your container's configuration (or the
/etc/lxc/default.conf file) to change the lxc.apparmor.profile
parameter.

This bugreport raises an interesting question regarding the tradeoff
between the solution we implemented to fix bug #916639.

Cc-ing intrigeri: I'm reconsidering the /etc/lxc/default.conf setting
regarding apparmor.profile. Putting generated breaks many unpriv
containers as they have no apparmor.profile set in their configuration.

[1] https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=916639

--
Pierre-Elliott Bécue
GPG: 9AE0 4D98 6400 E3B6 7528  F493 0D44 2664 1949 74E2
It's far easier to fight for one's principles than to live up to them.

signature.asc (849 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Bug#925899: lxc: Unprivileged containers fail to start after recent updates

intrigeri-4
Hi,

Pierre-Elliott Bécue:
> This bugreport raises an interesting question regarding the tradeoff
> between the solution we implemented to fix bug #916639.

> Cc-ing intrigeri: I'm reconsidering the /etc/lxc/default.conf setting
> regarding apparmor.profile. Putting generated breaks many unpriv
> containers as they have no apparmor.profile set in their configuration.

I'd love to help but I'll need more info to understand why the current
setup breaks "many unpriv containers", e.g.:

 - Is this specific to unprivileged containers?

 - Is it because "apparmor.profile = generated" is not suitable
   for unprivileged containers?

Finally, I wonder if "Suggests: apparmor" expresses strongly enough
the current status of the LXC + AppArmor integration in Debian.
Thankfully the Linux images will pull apparmor via Recommends…
except on systems where the administrator has disabled installation
of Recommends.

Cheers,
--
intrigeri

Reply | Threaded
Open this post in threaded view
|

Bug#925899: lxc: Unprivileged containers fail to start after recent updates

Pierre-Elliott Bécue-3
Le 30 mars 2019 15:29:52 GMT+01:00, intrigeri <[hidden email]> a écrit :

>Hi,
>
>Pierre-Elliott Bécue:
>> This bugreport raises an interesting question regarding the tradeoff
>> between the solution we implemented to fix bug #916639.
>
>> Cc-ing intrigeri: I'm reconsidering the /etc/lxc/default.conf setting
>> regarding apparmor.profile. Putting generated breaks many unpriv
>> containers as they have no apparmor.profile set in their
>configuration.
>
>I'd love to help but I'll need more info to understand why the current
>setup breaks "many unpriv containers", e.g.:
>
> - Is this specific to unprivileged containers?
>
> - Is it because "apparmor.profile = generated" is not suitable
>   for unprivileged containers?
>
>Finally, I wonder if "Suggests: apparmor" expresses strongly enough
>the current status of the LXC + AppArmor integration in Debian.
>Thankfully the Linux images will pull apparmor via Recommends…
>except on systems where the administrator has disabled installation
>of Recommends.
>
>Cheers,

It is specific to unpriviledged containers and due to the fact that non root users don't seem to have the ability to use the generated profile.
PEB (from my phone)

Reply | Threaded
Open this post in threaded view
|

Bug#925899: lxc: Unprivileged containers fail to start after recent updates

Regis Smith
In reply to this post by Pierre-Elliott Bécue-3
Hi!  Thanks for replying.

On Sat, 30 Mar 2019 14:51:47 +0100 Pierre-Elliott =?utf-8?B?QsOpY3Vl?=
<[hidden email]> wrote:

> Le mercredi 27 mars 2019 à 22:08:49-0700, Regis Smith a écrit :
> > Package: lxc
> > Version: 1:3.1.0+really3.0.3-6
> > Severity: important
> >
> > Dear Maintainer,
> >
> >    * What led up to the situation?
> >
> > apt update; apt upgrade
> >
> >    * What exactly did you do (or not do) that was effective (or
> >      ineffective)?
> >
> > As a normal user:
> > $ lxc-start -n test
> >
> >    * What was the outcome of this action?
> >
> > lxc-start: test: lxccontainer.c: wait_on_daemonized_start: 833 No
such file or directory - Failed to receive the container state
> > lxc-start: test: tools/lxc_start.c: main: 330 The container failed
to start
> > lxc-start: test: tools/lxc_start.c: main: 333 To get more details,
run the container in foreground mode
> > lxc-start: test: tools/lxc_start.c: main: 336 Additional
information can be obtained by setting the --logfile and --logpriority
options
> >
> > If I run it in the foreground instead I get
> >
> > $ lxc-start -n test -F
> > lxc-start: test: lsm/apparmor.c: apparmor_prepare: 974 Cannot use
generated profile: apparmor_parser not available
> > lxc-start: test: start.c: lxc_init: 899 Failed to initialize LSM
> > lxc-start: test: start.c: __lxc_start: 1917 Failed to initialize
container "test"
> > lxc-start: test: tools/lxc_start.c: main: 330 The container failed
to start
> > lxc-start: test: tools/lxc_start.c: main: 336 Additional
information can be obtained by setting the --logfile and --logpriority
options
> >
> >    * What outcome did you expect instead?
> >
> > A running container.  These used to work up until recently.  Now I
can't stop
> > already running containers because I won't be able to restart them.
>
> Hi,
>
> Thanks for submitting this bug.
>
> As you can see, it is possible to get more debug via the --logfile
and

> the --logpriority options.
>
> That said, the first line with the -F option says it all:
>
> > lxc-start: test: lsm/apparmor.c: apparmor_prepare: 974 Cannot use
> > generated profile: apparmor_parser not available
>
> It means that you're lacking the apparmor_parser command, which is
> shipped by apparmor. It probably means that you refused to install
> apparmor on your host.
Actually, I do have apparmor installed, and I can run apparmor_parser
as root.  aa-status shows all the related "lxc-container-*" in enforce
mode. Priveleged containers work fine, but I can not start unprivileged
containers.  Both privileged and unpriveleged worked fine before the
updates over the past several weeks.

>
> You have multiple choices. The first one being installing apparmor,
and
> the second one being to edit your container's configuration (or the
> /etc/lxc/default.conf file) to change the lxc.apparmor.profile
> parameter.
>
> This bugreport raises an interesting question regarding the tradeoff

I attached the log from running

$ lxc-start -n test --logpriority DEBUG --logfile lxc.log

I commented out "apparmor.profile = generated" and it still doesn't
work.  I'd like to get this working with apparmor, since it's the
default.  However, I'd love to hear from anyone who has unprivileged
containers working on an up-to-date Buster.  The fickleness of LXC in
Stretch wore me out, so I was quite pleased when it worked reliably in
Buster, up until now.

Regis


lxc.log (10K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Bug#925899: lxc: Unprivileged containers fail to start after recent updates

intrigeri-4
Hi,

Regis Smith:
>> > lxc-start: test: lsm/apparmor.c: apparmor_prepare: 974 Cannot use
> generated profile: apparmor_parser not available

I've reproduced this problem and I could fix it with:

  lxc.apparmor.profile = unconfined

Regis, can you please confirm this fix works for you as well?

Pierre-Elliott Bécue:
> Cc-ing intrigeri: I'm reconsidering the /etc/lxc/default.conf setting
> regarding apparmor.profile. Putting generated breaks many unpriv
> containers as they have no apparmor.profile set in their configuration.

Considering kernel.unprivileged_userns_clone is disabled by default
on Debian, IMO we should:

 - Optimize for the Debian defaults, i.e. privileged containers:
    - Keep the settings we added recently in /etc/lxc/default.conf
    - Replace "Suggests: apparmor" with "Depends: apparmor", because
      the default config will create containers that fail to start
      if the apparmor package is not installed.

 - Document how to use unprivileged containers on Debian. It's not as
   if they were previously working fine by default and AppArmor broke
   them — regardless of AppArmor, on current sid with the default
   kernel settings and lxc.apparmor.profile = unconfined, trying to
   start an unprivileged container fails in a very much user
   unfriendly way:
   
     conf.c: chown_mapped_root: 3250 lxc-usernsexec failed: Permission denied - Failed to open tt

   That's a first usability stumbling block. The new
   lxc.apparmor.profile default setting merely adds a second one.

   So I think README.Debian should document the need for
   kernel.unprivileged_userns_clone=1 and for
   lxc.apparmor.profile = unconfined

 - Take care of the Stretch→Buster upgrade path for unprivileged
   containers, by mentioning in NEWS.Debian that previously working
   unprivileged containers now need lxc.apparmor.profile = unconfined.

Thoughts?

Cheers,
--
intrigeri

Reply | Threaded
Open this post in threaded view
|

Bug#925899: lxc: Unprivileged containers fail to start after recent updates

Pierre-Elliott Bécue-3
Le dimanche 31 mars 2019 à 14:55:52+0200, intrigeri a écrit :

> Pierre-Elliott Bécue:
> > Cc-ing intrigeri: I'm reconsidering the /etc/lxc/default.conf setting
> > regarding apparmor.profile. Putting generated breaks many unpriv
> > containers as they have no apparmor.profile set in their configuration.
>
> Considering kernel.unprivileged_userns_clone is disabled by default
> on Debian, IMO we should:
>
>  - Optimize for the Debian defaults, i.e. privileged containers:
>     - Keep the settings we added recently in /etc/lxc/default.conf
>     - Replace "Suggests: apparmor" with "Depends: apparmor", because
>       the default config will create containers that fail to start
>       if the apparmor package is not installed.
>
>  - Document how to use unprivileged containers on Debian. It's not as
>    if they were previously working fine by default and AppArmor broke
>    them — regardless of AppArmor, on current sid with the default
>    kernel settings and lxc.apparmor.profile = unconfined, trying to
>    start an unprivileged container fails in a very much user
>    unfriendly way:
>    
>      conf.c: chown_mapped_root: 3250 lxc-usernsexec failed: Permission denied - Failed to open tt
>
>    That's a first usability stumbling block. The new
>    lxc.apparmor.profile default setting merely adds a second one.
>
>    So I think README.Debian should document the need for
>    kernel.unprivileged_userns_clone=1 and for
>    lxc.apparmor.profile = unconfined

Yes and no, the issue is that we introduced a new problem which made the
unpriviledged containers of user having already set the kernel properly
not working either.

There is a tradeoff in your current solution, and I wonder if it is the
appropriate one.

I'm Cc-ing Antonio and Evgeni to get their POV.

>  - Take care of the Stretch→Buster upgrade path for unprivileged
>    containers, by mentioning in NEWS.Debian that previously working
>    unprivileged containers now need lxc.apparmor.profile = unconfined.

The apparmor.profile = lxc-container-default-cgns works too, but will
prevent systemd isolation features to work.

--
Pierre-Elliott Bécue
GPG: 9AE0 4D98 6400 E3B6 7528  F493 0D44 2664 1949 74E2
It's far easier to fight for one's principles than to live up to them.

Reply | Threaded
Open this post in threaded view
|

Bug#925899: [pkg-lxc-devel] Bug#925899: lxc: Unprivileged containers fail to start after recent updates

Evgeni Golov-2
Unprivileged containers is the only thing I use in Stretch atm, so call me biased. But it's also the thing that does not work out of the box (you need a sysctl, you need to add the user to lxc-net config and you need to adjust the users lxc.conf for the uid map). As such, I'd say having to set another config var is not a big deal as long it's properly documented and announced in NEWS.Debian.

My 0.02€

On 31 March 2019 16:11:53 CEST, "Pierre-Elliott Bécue" <[hidden email]> wrote:

>Le dimanche 31 mars 2019 à 14:55:52+0200, intrigeri a écrit :
>> Pierre-Elliott Bécue:
>> > Cc-ing intrigeri: I'm reconsidering the /etc/lxc/default.conf
>setting
>> > regarding apparmor.profile. Putting generated breaks many unpriv
>> > containers as they have no apparmor.profile set in their
>configuration.
>>
>> Considering kernel.unprivileged_userns_clone is disabled by default
>> on Debian, IMO we should:
>>
>>  - Optimize for the Debian defaults, i.e. privileged containers:
>>     - Keep the settings we added recently in /etc/lxc/default.conf
>>     - Replace "Suggests: apparmor" with "Depends: apparmor", because
>>       the default config will create containers that fail to start
>>       if the apparmor package is not installed.
>>
>>  - Document how to use unprivileged containers on Debian. It's not as
>>    if they were previously working fine by default and AppArmor broke
>>    them — regardless of AppArmor, on current sid with the default
>>    kernel settings and lxc.apparmor.profile = unconfined, trying to
>>    start an unprivileged container fails in a very much user
>>    unfriendly way:
>>    
>>      conf.c: chown_mapped_root: 3250 lxc-usernsexec failed:
>Permission denied - Failed to open tt
>>
>>    That's a first usability stumbling block. The new
>>    lxc.apparmor.profile default setting merely adds a second one.
>>
>>    So I think README.Debian should document the need for
>>    kernel.unprivileged_userns_clone=1 and for
>>    lxc.apparmor.profile = unconfined
>
>Yes and no, the issue is that we introduced a new problem which made
>the
>unpriviledged containers of user having already set the kernel properly
>not working either.
>
>There is a tradeoff in your current solution, and I wonder if it is the
>appropriate one.
>
>I'm Cc-ing Antonio and Evgeni to get their POV.
>
>>  - Take care of the Stretch→Buster upgrade path for unprivileged
>>    containers, by mentioning in NEWS.Debian that previously working
>>    unprivileged containers now need lxc.apparmor.profile =
>unconfined.
>
>The apparmor.profile = lxc-container-default-cgns works too, but will
>prevent systemd isolation features to work.
>
>--
>Pierre-Elliott Bécue
>GPG: 9AE0 4D98 6400 E3B6 7528  F493 0D44 2664 1949 74E2
>It's far easier to fight for one's principles than to live up to them.
>
>_______________________________________________
>Pkg-lxc-devel mailing list
>[hidden email]
>https://alioth-lists.debian.net/cgi-bin/mailman/listinfo/pkg-lxc-devel

Reply | Threaded
Open this post in threaded view
|

Bug#925899: lxc: Unprivileged containers fail to start after recent updates

Regis Smith
In reply to this post by intrigeri-4
On Sun, 31 Mar 2019 14:55:52 +0200 intrigeri <[hidden email]>
wrote:
> Hi,
>
> Regis Smith:
> >> > lxc-start: test: lsm/apparmor.c: apparmor_prepare: 974 Cannot
use
> > generated profile: apparmor_parser not available
>
> I've reproduced this problem and I could fix it with:
>
>   lxc.apparmor.profile = unconfined
>
> Regis, can you please confirm this fix works for you as well?
>

Yes, this works.  Thanks!  This was the old solution required in Jessie
or early Stretch (w/o backports), so people may still complain after
Buster is released :)  But I'm happpy if I can reliably start
unprivileged containers again, which "unconfined" does.

Regis

Reply | Threaded
Open this post in threaded view
|

Bug#925899: lxc: Unprivileged containers fail to start after recent updates

Pierre-Elliott Bécue-3
In reply to this post by intrigeri-4
Le dimanche 31 mars 2019 à 14:55:52+0200, intrigeri a écrit :

> Hi,
>
> Regis Smith:
> >> > lxc-start: test: lsm/apparmor.c: apparmor_prepare: 974 Cannot use
> > generated profile: apparmor_parser not available
>
> I've reproduced this problem and I could fix it with:
>
>   lxc.apparmor.profile = unconfined
>
> Regis, can you please confirm this fix works for you as well?
>
> Pierre-Elliott Bécue:
> > Cc-ing intrigeri: I'm reconsidering the /etc/lxc/default.conf setting
> > regarding apparmor.profile. Putting generated breaks many unpriv
> > containers as they have no apparmor.profile set in their configuration.
>
> Considering kernel.unprivileged_userns_clone is disabled by default
> on Debian, IMO we should:
>
>  - Optimize for the Debian defaults, i.e. privileged containers:
>     - Keep the settings we added recently in /etc/lxc/default.conf
>     - Replace "Suggests: apparmor" with "Depends: apparmor", because
>       the default config will create containers that fail to start
>       if the apparmor package is not installed.
>
>  - Document how to use unprivileged containers on Debian. It's not as
>    if they were previously working fine by default and AppArmor broke
>    them — regardless of AppArmor, on current sid with the default
>    kernel settings and lxc.apparmor.profile = unconfined, trying to
>    start an unprivileged container fails in a very much user
>    unfriendly way:
>    
>      conf.c: chown_mapped_root: 3250 lxc-usernsexec failed: Permission denied - Failed to open tt
>
>    That's a first usability stumbling block. The new
>    lxc.apparmor.profile default setting merely adds a second one.
>
>    So I think README.Debian should document the need for
>    kernel.unprivileged_userns_clone=1 and for
>    lxc.apparmor.profile = unconfined
>
>  - Take care of the Stretch→Buster upgrade path for unprivileged
>    containers, by mentioning in NEWS.Debian that previously working
>    unprivileged containers now need lxc.apparmor.profile = unconfined.
>
> Thoughts?
See the two latest commits for lxc:

https://salsa.debian.org/lxc-team/lxc/commits/master

Tell me what you think about them, and if needed don't hesitate to
submit a MR! :)

--
Pierre-Elliott Bécue
GPG: 9AE0 4D98 6400 E3B6 7528  F493 0D44 2664 1949 74E2
It's far easier to fight for one's principles than to live up to them.

signature.asc (849 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Bug#925899: lxc: Unprivileged containers fail to start after recent updates

intrigeri-4
Hi,

Pierre-Elliott Bécue:
> See the two latest commits for lxc:
> https://salsa.debian.org/lxc-team/lxc/commits/master
> Tell me what you think about them,

Sweet, thanks! :)

> and if needed don't hesitate to submit a MR! :)

There you go: https://salsa.debian.org/lxc-team/lxc/merge_requests/4

Cheers,
--
intrigeri