Bug#852787: gitlab: Doesn't restart services properly on OOM kill

classic Classic list List threaded Threaded
3 messages Options
Reply | Threaded
Open this post in threaded view
|

Bug#852787: gitlab: Doesn't restart services properly on OOM kill

Libor Klepáč
Package: gitlab
Severity: important

-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

Hello,
we have low mem instance of gitlab - it used to have 2GB of ram, now it has 3GB.
Sometimes, it runs out of memory. (usually at 22:xx )
OOM killer usually kills one of sidekiq children.
It triggers restart of sidekiq, but also of other gitlab services.
But gitlab-sidekiq is the only one, which is started, other are left dead.

Can this be somehow addressed in systemd service dependecies?


Libor

-----BEGIN PGP SIGNATURE-----

iQJJBAEBCAAzFiEEPGZVVU37tFmB0TQv8O+MbsKfR44FAliLGIQVHGxpYm9yLmts
ZXBhY0BiY29tLmN6AAoJEPDvjG7Cn0eOcGgP/3eJAMqQoBWxG48YXLnfx1D6YZ6h
K4CLe0fpz8rFG87v4qsfyszA5WF3qY3QTffYKXxDJq5Z9QAaV2V+k5Jn0j0Sqhes
IsZqN5g4XvOdyYB6UEy6IGdDhiogiqCeHTz/gRL83WYDb2c+aIykZVYUWSS2MELl
fIpDyIn5FvlRs42VQu0u6x/Wi6b1+j55Y7KWo3j/bET+8MQzg5fOEdIO3qXGTGrf
A5wM0QQtFsOmcvQNWz7hWn7Lm4Fd5Q5U48O0MRiaJNTOUA/OOfoHgRBwHcBAUtvR
2alVQwb9FlH90wdRhUHARJEX10m2+duTvRw4KrP2PHaFcy5l/GJiVp8RV5NrAkHF
Fo9xDrmNjDnCAhjT54gOaudAYW5P9MpqJDHOtcaenvsoK8OHaEFh85kBxYhDj4DA
T8f7qnoHzBsHXzDr3zLMas8TGVfWRc/H7HhhL19WTDwCyXJfm+NBcwYd5RUXysl/
TdZE9K09uhxKtMNbnhAqfPk08XTmTRLObRGx/2U8wWaU8f3X1aZOJLANOQUKuq2Q
57eEIQ3Ed8eY8AUN95jfxq5s9QbgLhwGf8hl6eyzXKrAtHxq/coeN2rNzdaEYIac
A7VRgk4id1qnwqdAFgy6DoNTzqK2vcAjy5Ix+DmoFPHmaxqIuXj6O9pNr5+/9lL4
/0DxTlrb5zieOvBi
=h2TC
-----END PGP SIGNATURE-----

log.txt (15K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Bug#852787: gitlab: systemd configuration for sidekiq tears down gitlab service

Patrik Hagedorn
Package: gitlab
Version: 8.13.11+dfsg1-8
Followup-For: Bug #852787

Dear Maintainer,

we had the same issue with sidekiq. Sidekiq is known to suffer from memory
leaks, that's what probably triggers the OOM killer mentioned by Libor Klepáč.
We activated the Sidekiq MemoryKiller as documented by GitLab:
https://docs.gitlab.com/ce/administration/operations/sidekiq_memory_killer.html
As result, the Sidekiq MemoryKiller shuts down the whole gitlab.service.

As mentioned by Libor Klepáč the gitlab-sidekiq.service returns after OOM killer
due to the "Restart=on-abnormal" directive, but every signal send to the sidekiq
process will result in the shutdown of gitlab.service as the
gitlab-sidekiq.service is referenced as a dependency in the "BindsTo" directive.

In our scenario the Sidekiq MemoryKiller sends a SIGTERM after a 15min grace
period or a SIGKILL after additional 30sec. Both signals have the same effect on
the gitlab.service, which shuts down completely.

A patch to fix the configuration of gitlab.service and gitlab-sidekiq.service is
attached. We configured systemd to not tear down the whole gitlab.service when
gitlab-sidekiq.service terminates by removing gitlab-sidekiq.service from the
"BindsTo" directive and using a softer "Wants" directive for
gitlab-sidekiq.service in the gitlab.service instead.

The gitlab-sidekiq.service itself is configured with "Restart" directive to
restart "always" instead of just "on-abnormal", so that it even restarts when
it gets a clean exit code 0 by the SIGTERM of the Sidekiq MemoryKiller.

As recommended by GitLab and used in the Omnibus package, we should use the
Sidekiq MemoryKiller as a measure against the memory leaks (patch attached).

Best
--
Patrik Hagedorn
System Developer - Data Center Automation

ProfitBricks GmbH
Greifswalder Str. 207
D - 10405 Berlin

Email: [hidden email]
URL:   https://www.profitbricks.de

Sitz der Gesellschaft: Berlin
Registergericht: Amtsgericht Charlottenburg, HRB 125506 B
Geschäftsführer: Achim Weiss, Matthias Steinberg

0001-Prevent-dying-sidekiq-to-shutdown-gitlab.service.patch (2K) Download Attachment
0002-Enable-Sidekiq-MemoryKiller.patch (1K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Bug#852787: gitlab stop

Fernando Toledo
In reply to this post by Libor Klepáč
I run gitlab 11.5.6+dfsg-1~bpo9+1 on low resources server (2gb) (on
debian strech)

and these patch works for me and get gitlab running.

thanks!