Bug#920643: mariadb-server-10.3: mariadb won't start when running inside an lxc container when running on debian testing

classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

Bug#920643: mariadb-server-10.3: mariadb won't start when running inside an lxc container when running on debian testing

Matthew Darwin-2
Package: mariadb-server-10.3
Version: 1:10.3.12-2
Severity: important

Dear Maintainer,

*** Reporter, please consider answering these questions, where appropriate ***

Upgrade to latest mariadb package on debian testing; within lxc continer. Previous package was working fine...

on the container with the database:

Jan 27 14:17:42 db systemd[1]: Starting MariaDB 10.3.12 database server...
Jan 27 14:17:42 db systemd[278]: mariadb.service: Failed to set up mount namespacing: Permission denied
Jan 27 14:17:42 db systemd[278]: mariadb.service: Failed at step NAMESPACE spawning /usr/bin/install: Permission denied

on the host:
(not sure if it is 100% related)

Jan 27 14:16:21 server1 kernel: [   23.841650] audit: type=1400 audit(1548616581.737:21): apparmor="DENIED" operation="mount" info="failed flags match" error=-13 profile="lxc-container-default-cgns" name="/" pid=1423 comm="(rm)" flags="rw, rslave"


See possibly related: https://github.com/lxc/lxc/pull/2758

*** End of the template - remove these template lines ***


-- System Information:
Debian Release: buster/sid
  APT prefers testing
  APT policy: (500, 'testing')
Architecture: amd64 (x86_64)

Kernel: Linux 4.19.0-1-amd64 (SMP w/8 CPU cores)
Locale: LANG=en_US.UTF-8, LC_CTYPE=C.UTF-8 (charmap=UTF-8), LANGUAGE=en_US.UTF-8 (charmap=UTF-8)
Shell: /bin/sh linked to /usr/bin/dash
Init: systemd (via /run/systemd/system)

Versions of packages mariadb-server-10.3 depends on:
ii  adduser                   3.118
ii  debconf [debconf-2.0]     1.5.70
ii  galera-3                  25.3.23-1
ii  gawk                      1:4.2.1+dfsg-1
ii  iproute2                  4.20.0-2
ii  libc6                     2.28-5
ii  libdbi-perl               1.642-1+b1
ii  libpam0g                  1.1.8-4
ii  libssl1.1                 1.1.1a-1
ii  libstdc++6                8.2.0-14
ii  lsb-base                  10.2018112800
ii  lsof                      4.91+dfsg-1
ii  mariadb-client-10.3       1:10.3.12-2
ii  mariadb-common            1:10.3.12-2
ii  mariadb-server-core-10.3  1:10.3.12-2
ii  passwd                    1:4.5-1.1
ii  perl                      5.28.1-3
ii  psmisc                    23.2-1
ii  rsync                     3.1.3-2
ii  socat                     1.7.3.2-2
ii  zlib1g                    1:1.2.11.dfsg-1

Versions of packages mariadb-server-10.3 recommends:
ii  libhtml-template-perl  2.97-1

Versions of packages mariadb-server-10.3 suggests:
pn  mailx           <none>
pn  mariadb-test    <none>
pn  netcat-openbsd  <none>
pn  tinyca          <none>

-- debconf information excluded

Reply | Threaded
Open this post in threaded view
|

Bug#920643: mariadb-server-10.3: mariadb won't start when running inside an lxc container when running on debian testing

Faustin Lammler-3
Hi Matthew,
Thanks for your report!

I have no platform setup to test this so I have to install one but I am
not a LXC expert. Do you think this could be tested into a libvirt VM?

If I understand correctly, you have a Debian testing host and you are
running an LXC container with 10.3 mariadb version (everything from
official Debian repositories?).

This (https://github.com/lxc/lxc/pull/2758) seems to indicate that
problem may rather come from apparmor/systemd but I will try to
reproduce your issue.

Regards,
Faustin

Reply | Threaded
Open this post in threaded view
|

Bug#920643: mariadb-server-10.3: mariadb won't start when running inside an lxc container when running on debian testing

Matthew Darwin

Hello Faustin,

I am unfamiliar with how libvirt works, so I cannot say.

I have debian testing running on the hardware and inside the container.  Everything is from official repo.  I can get mariadb to start by messing around with the systemd startup script.

This may entirely be a an apparmor/systemd issue and nothing to to with mariadb.  But I'm not clear if mariadb systemd configuration is doing something unexpected or not.

It works, if I create /var/run/mysqld by hand, and then use this /lib/systemd/system/mariadb.service:

[Unit]
Description=MariaDB 10.3.12 database server
Documentation=man:mysqld(8)
Documentation=https://mariadb.com/kb/en/library/systemd/
After=network.target

[Install]
WantedBy=multi-user.target
Alias=mysql.service
Alias=mysqld.service

[Service]
Type=notify
PrivateNetwork=false
User=mysql
Group=mysql
CapabilityBoundingSet=CAP_IPC_LOCK
PermissionsStartOnly=true
ExecStartPre=/bin/sh -c "systemctl unset-environment _WSREP_START_POSITION"
ExecStartPre=/bin/sh -c "[ ! -e /usr/bin/galera_recovery ] && VAR= || \
 VAR=`/usr/bin/galera_recovery`; [ $? -eq 0 ] \
 && systemctl set-environment _WSREP_START_POSITION=$VAR || exit 1"
ExecStart=/usr/sbin/mysqld $MYSQLD_OPTS $_WSREP_NEW_CLUSTER $_WSREP_START_POSITION
ExecStartPost=/etc/mysql/debian-start
ExecStartPost=/bin/sh -c "systemctl unset-environment _WSREP_START_POSITION"
KillSignal=SIGTERM
SendSIGKILL=no
Restart=on-abort
RestartSec=5s
UMask=007
PrivateTmp=false
LimitNOFILE=16364


On 2019-02-11 11:35 a.m., Faustin Lammler wrote:
Hi Matthew,
Thanks for your report!

I have no platform setup to test this so I have to install one but I am
not a LXC expert. Do you think this could be tested into a libvirt VM?

If I understand correctly, you have a Debian testing host and you are
running an LXC container with 10.3 mariadb version (everything from
official Debian repositories?).

This (https://github.com/lxc/lxc/pull/2758) seems to indicate that
problem may rather come from apparmor/systemd but I will try to
reproduce your issue.

Regards,
Faustin
Reply | Threaded
Open this post in threaded view
|

Bug#920643: mariadb-server-10.3: mariadb won't start when running inside an lxc container when running on debian testing

Faustin Lammler-3
Control: forwarded -1 https://github.com/lxc/lxc/pull/2758

Matthew,
I able to reproduce this and I have the exact same error (mariadb log +
apparmor on host).

Your workaround is working but it seems that removing only these 3 lines
is sufficient:
> ProtectSystem=full
> PrivateDevices=true
> ProtectHome=true

You can leave this one:
> ExecStartPre=/usr/bin/install -m 755 -o mysql -g root -d /var/run/mysqld

Another workaround is to disable completely apparmor:
https://wiki.debian.org/AppArmor/HowToUse#Disable_AppArmor

I think we should wait until some progress comes from
https://github.com/lxc/lxc/pull/2758.

Faustin

Reply | Threaded
Open this post in threaded view
|

Bug#920643: mariadb-server-10.3: mariadb won't start when running inside an lxc container when running on debian testing

Matthew Darwin

Excellent, thanks.



On 2019-02-14 10:51 a.m., Faustin Lammler wrote:
Control: forwarded -1 https://github.com/lxc/lxc/pull/2758

Matthew,
I able to reproduce this and I have the exact same error (mariadb log +
apparmor on host).

Your workaround is working but it seems that removing only these 3 lines
is sufficient:
ProtectSystem=full
PrivateDevices=true
ProtectHome=true
You can leave this one:
ExecStartPre=/usr/bin/install -m 755 -o mysql -g root -d /var/run/mysqld
Another workaround is to disable completely apparmor:
https://wiki.debian.org/AppArmor/HowToUse#Disable_AppArmor

I think we should wait until some progress comes from
https://github.com/lxc/lxc/pull/2758.

Faustin
Reply | Threaded
Open this post in threaded view
|

Bug#920643: mariadb-server-10.3: mariadb won't start when running inside an lxc container when running on debian testing

Matthew Darwin
In reply to this post by Faustin Lammler-3

The following instructions (step 2) resolve the situation:


lxc (1:3.1.0+really3.0.3-6) unstable; urgency=medium

  LXC 3 got some significant changes from LXC 2.

   1. The configuration files use different variables. A userland script
      lxc-update-config is available to update automatically your
      configuration files. An automatic update is possible and offered by
      debconf during the upgrade of lxc version < 3.0.2 to lxc version >=
      3.0.2. Mind that this update will only work for priviledged containers
      with configurations present in /var/lib/lxc/*/config and any other
      container will not be updated.
   2. AppArmor support in Debian has increased, thus preventing some systemd
      isolation features to work in LXC 3.0.X. Debian has backported some
      patches from LXC 3.1 that, along with some configurations in a
      container, will allow systemd isolation features to work.

      The required configuration parameters are the ones which follow:
        lxc.apparmor.profile = generated
        lxc.apparmor.allow_nesting = 1

      These parameters are provided in the `/etc/lxc/default.conf` file
      shipped with LXC 3. Hence, any newly created container will have these
      parameters set properly, execpt if you alter the forementionned file.
   3. lxc-templates is deprecated by upstream. The new way of building
      containers is via their distrobuilder software. This software isn't in
      Debian Buster, and thus, we still provide lxc-templates. If you relied
      on it (eg, with lxc.include parameter in some configuration file), you
      should install lxc-templates in case it doesn't come by itself (via
      recommends). Otherwise you may experience issues after the upgrade.

 -- Pierre-Elliott Bécue [hidden email]  Sat, 09 Mar 2019 13:09:05 +0100
On 2019-02-14 10:51 a.m., Faustin Lammler wrote:
Control: forwarded -1 https://github.com/lxc/lxc/pull/2758

Matthew,
I able to reproduce this and I have the exact same error (mariadb log +
apparmor on host).

Your workaround is working but it seems that removing only these 3 lines
is sufficient:
ProtectSystem=full
PrivateDevices=true
ProtectHome=true
You can leave this one:
ExecStartPre=/usr/bin/install -m 755 -o mysql -g root -d /var/run/mysqld
Another workaround is to disable completely apparmor:
https://wiki.debian.org/AppArmor/HowToUse#Disable_AppArmor

I think we should wait until some progress comes from
https://github.com/lxc/lxc/pull/2758.

Faustin
Reply | Threaded
Open this post in threaded view
|

Bug#920643: mariadb-server-10.3: mariadb won't start when running inside an lxc container when running on debian testing

Faustin Lammler-3
Hi Matthew!
Thank you for this workaround.

Feel free to add any comment to https://github.com/lxc/lxc/pull/2758 if
needed.