Bug#921176: redis-server service is failing to start in buster lxc container

classic Classic list List threaded Threaded
9 messages Options
Reply | Threaded
Open this post in threaded view
|

Bug#921176: redis-server service is failing to start in buster lxc container

Chris Lamb -2
Hi,

> redis-server service is failing to start in buster lxc container

Any update on this? :)


Regards,

--
      ,''`.
     : :'  :     Chris Lamb
     `. `'`      [hidden email] 🍥 chris-lamb.co.uk
       `-

Reply | Threaded
Open this post in threaded view
|

Bug#921176: redis-server service is failing to start in buster lxc container

Pirate Praveen-3


On 2019, ഫെബ്രുവരി 4 1:20:11 PM IST, Chris Lamb <[hidden email]> wrote:
>Hi,
>
>> redis-server service is failing to start in buster lxc container
>
>Any update on this? :)

I'm traveling. hopefully tonight or tomorrow night I can try.

Adding Raju, and Abhijith, who may be able to try this before.
>
>Regards,

--
Sent from my Android device with K-9 Mail. Please excuse my brevity.

Reply | Threaded
Open this post in threaded view
|

Bug#921176: redis-server service is failing to start in buster lxc container

Pirate Praveen-3


On ചൊ, ഫെബ്രു 5, 2019 at 11:14 വൈകു, Chris Lamb
<[hidden email]> wrote:

> severity 921176 serious
> thanks
>
> Hi Pirate,
>
> [Dropping severity as it only affects LXC right now]
>
>>  It is working on the same host machine with stretch(-backports)
>>  container (5:5.0.3-3~bpo9+2). So host machine seems fine.
>
> Thanks for looking into this and providing some LXC basics.
> (However, I am not sure why I do not have working networking inside
> my container so I cannot debug it better on my end.)
>

But loopback is enough for redis-server, right?

> On the other hand, the first thing I would do if this was working
> would be to try removing more hardening features as previously
> discussed on this bug number until it (likely) worked. Could you
> try this please?
>

I tried removing all hardening features and it still won't start. I
tried to start manually by running redis-server /etc/redis/redis.conf
as root and that worked.

> Also, perhaps enable some deeper logging? Or check the actual
> redis-server.log file too?

There is nothing in /var/log/redis. I think its failing even before
redis-server gets a chance to log.

I found this https://github.com/systemd/systemd/issues/10032 and tried
updating systemd on host to 240-5 (from 238-5) and also updating
apparmor and disabling apparmor. But none of it worked.



Reply | Threaded
Open this post in threaded view
|

Bug#921176: redis-server service is failing to start in buster lxc container

Pirate Praveen-3


On Tue, Feb 12, 2019 at 6:09 PM, Chris Lamb <[hidden email]> wrote:

> Hi Pirate,
>
>>  > (However, I am not sure why I do not have working networking
>> inside
>>  > my container so I cannot debug it better on my end.)
>>
>>  But loopback is enough for redis-server, right?
>
> Yes, but I can't even install without network. :)
>
https://wiki.debian.org/Packaging/Pre-Requisites#LXC has networking
setup instructions.

>>  I tried removing all hardening features and it still won't start. I
>>  tried to start manually by running redis-server
>> /etc/redis/redis.conf
>>  as root and that worked.
>
> Can you try with the redis user?
>

That also works.

root@redis:~# su redis -s /bin/sh -c 'redis-server
/etc/redis/redis.conf'
root@redis:~# ps ax |grep redis
 1073 ?        Ssl    0:00 redis-server 127.0.0.1:6379
 1078 pts/2    S+     0:00 grep redis
root@redis:~# redis-cli
127.0.0.1:6379> help
redis-cli 5.0.3
To get help about Redis commands type:
      "help @<group>" to get a list of commands in <group>
      "help <command>" for help on <command>
      "help <tab>" to get a list of possible help topics
      "quit" to exit

To set redis-cli preferences:
      ":set hints" enable online hints
      ":set nohints" disable online hints
Set your preferences in ~/.redisclirc
127.0.0.1:6379>

>>  > Also, perhaps enable some deeper logging? Or check the actual
>>  > redis-server.log file too?
>>
>>  There is nothing in /var/log/redis. I think its failing even before
>>  redis-server gets a chance to log.
>
> Can you enable more systemd logging? If it's failing there, surely
> we can get more info than just "failed"?

It was always the same error I mentioned in first mail. Anyway with
LogLevel=debug set in systemd conf, I got a more verbose log, which is
attached.



systemd.redis.log (27K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Bug#921176: redis-server service is failing to start in buster lxc container

Pirate Praveen-3


On Tue, Feb 12, 2019 at 9:44 PM, Chris Lamb <[hidden email]> wrote:
> Hi Pirate,
>
>>  https://wiki.debian.org/Packaging/Pre-Requisites#LXC has networking
>>  setup instructions.
>
> Still no dice and I don't really have the bandwidth to learn
> another container technology. :(
>

May be ask lxc team for help?

>>   I tried removing all hardening features and it still won't start
>
> Please provide:
>
>  a) The *exact* .service file you are trying.
>

Initially I tried editing /lib/systemd/system/redis-server.service and
later I edited /lib/systemd/system/redis-server\@.service as well
(edited both these files)

>  b) Exactly how you are editing the shipped .service file.
>

I tried adding changing true to false initially, then adding # in front
of the options

So current snapshot look like this,
root@redis:~# cat /lib/systemd/system/redis-server.service
[Unit]
Description=Advanced key-value store
After=network.target
Documentation=http://redis.io/documentation, man:redis-server(1)

[Service]
Type=forking
ExecStart=/usr/bin/redis-server /etc/redis/redis.conf
ExecStop=/bin/kill -s TERM $MAINPID
PIDFile=/run/redis/redis-server.pid
TimeoutStopSec=0
Restart=always
User=redis
Group=redis
#RuntimeDirectory=redis
#RuntimeDirectoryMode=2755

#UMask=007
#PrivateTmp=yes
#LimitNOFILE=65535
#PrivateDevices=yes
#ProtectHome=yes
#ReadOnlyDirectories=/
#ReadWriteDirectories=-/var/lib/redis
#ReadWriteDirectories=-/var/log/redis
#ReadWriteDirectories=-/var/run/redis

NoNewPrivileges=true
CapabilityBoundingSet=CAP_SETGID CAP_SETUID CAP_SYS_RESOURCE
MemoryDenyWriteExecute=true
ProtectKernelModules=true
#ProtectKernelTunables=true
#ProtectControlGroups=true
#RestrictRealtime=true
#RestrictNamespaces=true
#RestrictAddressFamilies=AF_INET AF_INET6 AF_UNIX

# redis-server can write to its own config file when in cluster mode so
we
# permit writing there by default. If you are not using this feature,
it is
# recommended that you replace the following lines with
"ProtectSystem=full".
#ProtectSystem=true
#ReadWriteDirectories=-/etc/redis

[Install]
WantedBy=multi-user.target
Alias=redis.service

>  c) uname -a
>

root@redis:~# uname -a
Linux redis 4.15.0-2-amd64 #1 SMP Debian 4.15.11-1 (2018-03-20) x86_64
GNU/Linux
root@redis:~#

This is reproduced on a second machine. On first machine, I tried with
two kernels, one older and then 4.19.

>  d) aa-status
>
> Anything in the global kernel log or dmesg...?

In dmesg inside container (same error on the host as well), so it seems
apparmor is blocking it.

[14760.307180] audit: type=1400 audit(1549992481.311:156):
apparmor="DENIED" operation="mount" info="failed flags match" error=-13
profile="lxc-container-default-cgns" name="/" pid=20531
comm="(s-server)" flags="rw, rslave"
[14760.573458] audit: type=1400 audit(1549992481.579:157):
apparmor="DENIED" operation="mount" info="failed flags match" error=-13
profile="lxc-container-default-cgns" name="/" pid=20532
comm="(s-server)" flags="rw, rslave"
[14760.823723] audit: type=1400 audit(1549992481.827:158):
apparmor="DENIED" operation="mount" info="failed flags match" error=-13
profile="lxc-container-default-cgns" name="/" pid=20533
comm="(s-server)" flags="rw, rslave"
[14761.073770] audit: type=1400 audit(1549992482.079:159):
apparmor="DENIED" operation="mount" info="failed flags match" error=-13
profile="lxc-container-default-cgns" name="/" pid=20534
comm="(s-server)" flags="rw, rslave"
[14761.323944] audit: type=1400 audit(1549992482.327:160):
apparmor="DENIED" operation="mount" info="failed flags match" error=-13
profile="lxc-container-default-cgns" name="/" pid=20536
comm="(s-server)" flags="rw, rslave"


Though systemctl status apparmor on host showed it failed,

Feb 12 18:23:25 nishumbha systemd[1]: Starting AppArmor
initialization...
Feb 12 18:23:30 nishumbha apparmor[600]: Starting AppArmor
profiles:AppArmor parser error for
/etc/apparmor.d/usr.lib.libreoffice.program.soffice.bin in
/etc/apparmor.d/usr.lib.libreoffice.program.soffice.bin at line 89:
Could not open 'abstractions/dri-enumerate'
Feb 12 18:23:30 nishumbha apparmor[600]: AppArmor parser error for
/etc/apparmor.d/usr.lib.libreoffice.program.soffice.bin in
/etc/apparmor.d/usr.lib.libreoffice.program.soffice.bin at line 89:
Could not open 'abstractions/dri-enumerate'
Feb 12 18:23:30 nishumbha apparmor[600]:  failed!
Feb 12 18:23:30 nishumbha systemd[1]: apparmor.service: Main process
exited, code=exited, status=123/n/a

apparmor was at 2.11.1-4, updating it to 2.13.2-7 fixed the above
error, but redis still can't be started like before (same error
message) even after stopping apparmor on the host.

pravi@nishumbha:~$ systemctl status apparmor
● apparmor.service - Load AppArmor profiles
   Loaded: loaded (/lib/systemd/system/apparmor.service; enabled;
vendor preset:
   Active: inactive (dead) since Tue 2019-02-12 23:06:50 IST; 3min 56s
ago
     Docs: man:apparmor(7)
           https://gitlab.com/apparmor/apparmor/wikis/home/
  Process: 10021 ExecStop=/bin/true (code=exited, status=0/SUCCESS)
  Process: 9765 ExecStart=/lib/apparmor/apparmor.systemd reload
(code=exited, st
 Main PID: 9765 (code=exited, status=0/SUCCESS)



Reply | Threaded
Open this post in threaded view
|

Bug#921176: redis-server service is failing to start in buster lxc container

Chris Lamb -2
Hi Pirate,

> > (ie. I don't think you can rule out apparmor either just yet.)
>
> yes, culprit is apparmor only. After aa-teardown, I can start redis
> service.

Great stuff. What's the next step here? Cearly this should Just
Work but I'm not sure where the bug is right now. I suggest the
next part of this process is that you re-enable apparmor with
logging.


Regards,

--
      ,''`.
     : :'  :     Chris Lamb
     `. `'`      [hidden email] 🍥 chris-lamb.co.uk
       `-

Reply | Threaded
Open this post in threaded view
|

Bug#921176: redis-server service is failing to start in buster lxc container

intrigeri-4
In reply to this post by Pirate Praveen-3
Control: reassign -1 lxc
Control: severity -1 important

Hi,

Pirate Praveen:
> In dmesg inside container (same error on the host as well), so it seems
> apparmor is blocking it.

> [14760.307180] audit: type=1400 audit(1549992481.311:156):
> apparmor="DENIED" operation="mount" info="failed flags match" error=-13
> profile="lxc-container-default-cgns" name="/" pid=20531
> comm="(s-server)" flags="rw, rslave"

The lxc-container-default-cgns profile is shipped by the lxc
package ⇒ reassigning.

This looks very much like LXC bug #916639 so please retry with:
lxc 1:3.1.0+really3.0.3-3 or newer?

If that's not sufficient, you might need to set these options for
your container:

   lxc.apparmor.profile = generated
   lxc.apparmor.allow_nesting = 1

(On sid, these settings are in /etc/lxc/default.conf already but I'm
not familiar with LXC and I don't know if they'll apply to
pre-existing containers.)

Thanks in advance!

Also, I'm setting severity to non-RC as it would be unfortunate to
block the migration to testing of… the very version that likely fixes
this bug. Once it's clarified that this is #916639, I'll fix
the metadata.

Cheers,
--
intrigeri

Reply | Threaded
Open this post in threaded view
|

Bug#921176: redis-server service is failing to start in buster lxc container

Pierre-Elliott Bécue-3
Le dimanche 24 février 2019 à 15:01:14+0100, intrigeri a écrit :

> Control: reassign -1 lxc
> Control: severity -1 important
>
> Hi,
>
> Pirate Praveen:
> > In dmesg inside container (same error on the host as well), so it seems
> > apparmor is blocking it.
>
> > [14760.307180] audit: type=1400 audit(1549992481.311:156):
> > apparmor="DENIED" operation="mount" info="failed flags match" error=-13
> > profile="lxc-container-default-cgns" name="/" pid=20531
> > comm="(s-server)" flags="rw, rslave"
>
> The lxc-container-default-cgns profile is shipped by the lxc
> package ⇒ reassigning.
>
> This looks very much like LXC bug #916639 so please retry with:
> lxc 1:3.1.0+really3.0.3-3 or newer?
>
> If that's not sufficient, you might need to set these options for
> your container:
>
>    lxc.apparmor.profile = generated
>    lxc.apparmor.allow_nesting = 1
>
> (On sid, these settings are in /etc/lxc/default.conf already but I'm
> not familiar with LXC and I don't know if they'll apply to
> pre-existing containers.)
>
> Thanks in advance!
>
> Also, I'm setting severity to non-RC as it would be unfortunate to
> block the migration to testing of… the very version that likely fixes
> this bug. Once it's clarified that this is #916639, I'll fix
> the metadata.
>
> Cheers,
Dear Praveen,

Did you give a test at the latest LXC3 releases?

I wonder if I can close this bug report now.

--
Pierre-Elliott Bécue
GPG: 9AE0 4D98 6400 E3B6 7528  F493 0D44 2664 1949 74E2
It's far easier to fight for one's principles than to live up to them.

signature.asc (849 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Bug#921176: redis-server service is failing to start in buster lxc container

Antonio Terceiro-3
On Sun, Apr 07, 2019 at 08:37:53PM +0200, Pierre-Elliott Bécue wrote:

> Le dimanche 24 février 2019 à 15:01:14+0100, intrigeri a écrit :
> > Control: reassign -1 lxc
> > Control: severity -1 important
> >
> > Hi,
> >
> > Pirate Praveen:
> > > In dmesg inside container (same error on the host as well), so it seems
> > > apparmor is blocking it.
> >
> > > [14760.307180] audit: type=1400 audit(1549992481.311:156):
> > > apparmor="DENIED" operation="mount" info="failed flags match" error=-13
> > > profile="lxc-container-default-cgns" name="/" pid=20531
> > > comm="(s-server)" flags="rw, rslave"
> >
> > The lxc-container-default-cgns profile is shipped by the lxc
> > package ⇒ reassigning.
> >
> > This looks very much like LXC bug #916639 so please retry with:
> > lxc 1:3.1.0+really3.0.3-3 or newer?
> >
> > If that's not sufficient, you might need to set these options for
> > your container:
> >
> >    lxc.apparmor.profile = generated
> >    lxc.apparmor.allow_nesting = 1
> >
> > (On sid, these settings are in /etc/lxc/default.conf already but I'm
> > not familiar with LXC and I don't know if they'll apply to
> > pre-existing containers.)
> >
> > Thanks in advance!
> >
> > Also, I'm setting severity to non-RC as it would be unfortunate to
> > block the migration to testing of… the very version that likely fixes
> > this bug. Once it's clarified that this is #916639, I'll fix
> > the metadata.
> >
> > Cheers,
>
> Dear Praveen,
>
> Did you give a test at the latest LXC3 releases?
>
> I wonder if I can close this bug report now.
FWIW I just tested in a clean container and redis-server starts just
fine.

signature.asc (849 bytes) Download Attachment