Bug#902493: apache2-bin: Event MPM listener thread may get blocked by SSL shutdowns

classic Classic list List threaded Threaded
37 messages Options
12
Reply | Threaded
Open this post in threaded view
|

Bug#902493: apache2-bin: Event MPM listener thread may get blocked by SSL shutdowns

Martijn Grendelman-5
Package: apache2-bin
Version: 2.4.25-3+deb9u4
Severity: important
Tags: patch upstream

Dear Maintainer,

Some of our Debian Stretch based Apache webservers suffer from
intermittent connection timeouts.

We have been trying to pin down the problem for a while, and eventually,
we found this bug report in Apache's Bugzilla, that seems to fit our
problem perfectly:

https://bz.apache.org/bugzilla/show_bug.cgi?id=60956

The short version of the story is, that under very specific
circumstances, Apache will stop accepting new connections until a
certain timeout has occurred. The source of this behaviour is in the
event MPM's code for cleaning up stale connections, which may block in
an unexpected way. It seems that the bug has been present in Apache
since v2.4.12, and has been fixed in v2.4.28.

The bug report above contains a patch that fixes the problem.

I suspect that this isn't a real problem for many users, because it took
the upstream community a long time to find it, and it doesn't seem to be
a common issue, if you start looking around. However, I have been able
to identify this problem on almost all of our Stretch webservers, even
if its occurrences are quite rare. Some of our less-loaded servers only
show it once every few weeks. One of them, however, has been suffering
from it multiple times daily for the past couple of weeks, up to a point
that Apache was considered unusable.

Also, we are not the only ones having this problem, for example see:
https://serverfault.com/questions/819717/apache-event-mpm-hangs-sporadicly

On top of that, if the circumstances are right, the bug can be triggered
from a malicious client, leading to denial of service. As such, I would
think this can be considered a security vulnerability.

Given that this is a real bug, having the scent of a security problem,
that causes a real problem for us and at least a few other people, I
kindly request to see if the patch from the mentioned Bugzilla report
can be applied to Apache 2.4.25 in Stretch. I already know it doesn't
apply cleanly, and I don't have the necessary C-skills to reliably
backport the changes, I'm afraid.

We 'solved' the problem in our shop by backporting Apache 2.4.33 from
Buster to Stretch, but you'll understand this this is not a great
solution from a security perspective.

-- Package-specific info:

-- System Information:
Debian Release: 9.4
  APT prefers stable-updates
  APT policy: (500, 'stable-updates'), (500, 'stable')
Architecture: amd64 (x86_64)

Kernel: Linux 4.9.0-6-amd64 (SMP w/1 CPU core)
Locale: LANG=en_US.UTF-8, LC_CTYPE=en_US.UTF-8 (charmap=UTF-8),
LANGUAGE=en_US:en (charmap=UTF-8)
Shell: /bin/sh linked to /bin/dash
Init: systemd (via /run/systemd/system)

Versions of packages apache2-bin depends on:
ii  libapr1                  1.5.2-5
ii  libaprutil1              1.5.4-3
ii  libaprutil1-dbd-sqlite3  1.5.4-3
ii  libaprutil1-ldap         1.5.4-3
ii  libc6                    2.24-11+deb9u3
ii  libldap-2.4-2            2.4.44+dfsg-5+deb9u1
ii  liblua5.2-0              5.2.4-1.1+b2
ii  libnghttp2-14            1.18.1-1
ii  libpcre3                 2:8.39-3
ii  libssl1.0.2              1.0.2l-2+deb9u3
ii  libxml2                  2.9.4+dfsg1-2.2+deb9u2
ii  perl                     5.24.1-3+deb9u4
ii  zlib1g                   1:1.2.8.dfsg-5

apache2-bin recommends no packages.

Versions of packages apache2-bin suggests:
pn  apache2-doc                                      <none>
pn  apache2-suexec-pristine | apache2-suexec-custom  <none>
ii  lynx [www-browser]                               2.8.9dev11-1

Versions of packages apache2 depends on:
ii  apache2-data         2.4.25-3+deb9u4
ii  apache2-utils        2.4.25-3+deb9u4
ii  dpkg                 1.18.24
ii  init-system-helpers  1.48
ii  lsb-base             9.20161125
ii  mime-support         3.60
ii  perl                 5.24.1-3+deb9u4
ii  procps               2:3.3.12-3+deb9u1

Versions of packages apache2 recommends:
ii  ssl-cert  1.0.39

Versions of packages apache2 suggests:
pn  apache2-doc                                      <none>
pn  apache2-suexec-pristine | apache2-suexec-custom  <none>
ii  lynx [www-browser]                               2.8.9dev11-1

Versions of packages apache2-bin is related to:
ii  apache2      2.4.25-3+deb9u4
ii  apache2-bin  2.4.25-3+deb9u4

-- no debconf information

Reply | Threaded
Open this post in threaded view
|

Bug#902493: apache2-bin: Event MPM listener thread may get blocked by SSL shutdowns

Sven Hartge-5
On Wed, 27 Jun 2018 10:39:51 +0200 Martijn Grendelman
<[hidden email]> wrote:

> Some of our Debian Stretch based Apache webservers suffer from
> intermittent connection timeouts.
>
> We have been trying to pin down the problem for a while, and eventually,
> we found this bug report in Apache's Bugzilla, that seems to fit our
> problem perfectly:
>
> https://bz.apache.org/bugzilla/show_bug.cgi?id=60956

I can verifiy and this bug and also had to change to mpm_worker to work
around this bug.

A backport of the changes in mpm_event made for 2.4.28 would be very
nice, just like mod_http2 was backported from a newer version of apache2.

Grüße,
Sven Hartge.


signature.asc (849 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Bug#902493: apache2-bin: Event MPM listener thread may get blocked by SSL shutdowns

Anton Dollmaier
Hi all,


On Fri, 5 Oct 2018 14:02:41 +0200 Sven Hartge <[hidden email]> wrote:

> On Wed, 27 Jun 2018 10:39:51 +0200 Martijn Grendelman
> <[hidden email]> wrote:
>
> > Some of our Debian Stretch based Apache webservers suffer from
> > intermittent connection timeouts.
> >
> > We have been trying to pin down the problem for a while, and eventually,
> > we found this bug report in Apache's Bugzilla, that seems to fit our
> > problem perfectly:
> >
> > https://bz.apache.org/bugzilla/show_bug.cgi?id=60956
>
> I can verifiy and this bug and also had to change to mpm_worker to work
> around this bug.
>
> A backport of the changes in mpm_event made for 2.4.28 would be very
> nice, just like mod_http2 was backported from a newer version of apache2.

After suffering (probably) from this issue on multiple systems, I'd
appreciate backporting the fix from Apache 2.4.28 to Stretch as well.

It could be just my personal impression, but it seems like this is
affecting more systems over time. We spotted the issue (Apache hangs
without warning and without logs until restarted or the timeout clears)
on just one system, now multiple systems are affected, even with low or
even just internal (browser clients behind VPN) traffic.

Switching to MPM_Worker helped to solve this in the meantime.

Best,
Anton Dollmaier

Reply | Threaded
Open this post in threaded view
|

Bug#902493: apache2-bin: Event MPM listener thread may get blocked by SSL shutdowns

Gedalya
In reply to this post by Martijn Grendelman-5
I've had seemingly the same issue. After several weeks of running a backported apache2 2.4.37-1 the issues are gone.

Previously it was happening several times every day, with the outage lasting sometimes 10 minutes or so.

This was very difficult to troubleshoot, as nothing is logged, and it was relatively hard to find this bug report and find my way towards a solution. Applying the fix to stretch might help others who are struggling to understand the issue they are facing and are therefore not being heard.

Reply | Threaded
Open this post in threaded view
|

Bug#902493: apache2-bin: Event MPM listener thread may get blocked by SSL shutdowns

Stefan Fritsch
In reply to this post by Anton Dollmaier
Between 2.4.25 and the fix for this issue, there were some intrusive
changes in mpm_evnt. If we did a backport, rhe risk of introducing
regressions would be quite high. Therefore, and because the next Debian
stable release is quite near, I don't think it makes sense to backport the
fix.

Reply | Threaded
Open this post in threaded view
|

Bug#902493: apache2-bin: Event MPM listener thread may get blocked by SSL shutdowns

Sven Hartge-5
On 10.02.19 14:36, Stefan Fritsch wrote:

> Between 2.4.25 and the fix for this issue, there were some intrusive
> changes in mpm_evnt. If we did a backport, rhe risk of introducing
> regressions would be quite high. Therefore, and because the next Debian
> stable release is quite near, I don't think it makes sense to backport the
> fix.

Maybe, as a compromise, switch to using mpm_worker as the default MPM
instead of mpm_event?

I've seen this problem here in more and more servers of mine, I had to
switch all of them to mpm_worker to avoid this nasty bug.

Also I am a bit disappointed by you invoking the "the next release is
near" argument. Most of my servers for example won't get Buster until
early to mid 2020 and I think many of others are in the same boat.

Grüße,
Sven.


signature.asc (849 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Bug#902493: apache2-bin: Event MPM listener thread may get blocked by SSL shutdowns

Jan Wagner-4
Hi Sven,

Am 11.02.19 um 09:51 schrieb Sven Hartge:
> Also I am a bit disappointed by you invoking the "the next release is
> near" argument. Most of my servers for example won't get Buster until
> early to mid 2020 and I think many of others are in the same boat.

just to point this out. You prefer an invasive backport and risk to
stability in other areas? The update policy of Debian in the past was,
that this should be avoided.

Cheers, Jan.
--
Never write mail to <[hidden email]>, you have been warned!
-----BEGIN GEEK CODE BLOCK-----
Version: 3.12
GIT d-- s+: a C+++ UL++++ P+ L+++ E--- W+++ N+++ o++ K++ w--- O M+ V- PS
PE Y++
PGP++ t-- 5 X R tv- b+ DI D+ G++ e++ h---- r+++ y++++
------END GEEK CODE BLOCK------


signature.asc (859 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Bug#902493: apache2-bin: Event MPM listener thread may get blocked by SSL shutdowns

Sven Hartge-5
On 11.02.19 10:29, Jan Wagner wrote:
> Am 11.02.19 um 09:51 schrieb Sven Hartge:

>> Also I am a bit disappointed by you invoking the "the next release is
>> near" argument. Most of my servers for example won't get Buster until
>> early to mid 2020 and I think many of others are in the same boat.

> just to point this out. You prefer an invasive backport and risk to
> stability in other areas? The update policy of Debian in the past was,
> that this should be avoided.

No, I am disappointed in the "let's do nothing" stance.

I can see why backporting the newer mpm_event is risky and that it
should be avoided.

I can also know that just throwing in a completely new Apache is
something Debian does not do, I've been using Debian for the last 20
years because of exactly that guarantee, to not get surprised by
mid-release major changes.

But this bug has been encountered frequently enough (and is difficult to
spot, if you don't exactly know what to search for) and with increasing
adoption of SSL more and more people will hit it, that I think at least
*some* action is warranted.

Maybe better documentation to help people encountering this or maybe
changing the default MPM for Stretch on new installs, since mpm_event in
Stretch clearly is flawed and buggy with SSL.

But just saying "Buster is release soon" can't be the right solution here.

Stretch will likely be used for at least 3 more years before it is
phased out, keeping a *known* bug with an easy workaround active for
that long because of "we don't change Debian Stable *ever*" seems wrong
to me.

Grüße,
Sven.




signature.asc (849 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Bug#902493: apache2-bin: Event MPM listener thread may get blocked by SSL shutdowns

Jan Wagner-4
Am 11.02.19 um 10:46 schrieb Sven Hartge:
> But this bug has been encountered frequently enough (and is difficult to
> spot, if you don't exactly know what to search for) and with increasing

Beeing there. Searched >3 weeks before I thought it would be a idea to
switch the mpm.

> adoption of SSL more and more people will hit it, that I think at least
> *some* action is warranted.
>
> Maybe better documentation to help people encountering this

Okay ... here is an area where you can push forward. What about
providing documentation patches?

With kind regards, Jan.
--
Never write mail to <[hidden email]>, you have been warned!
-----BEGIN GEEK CODE BLOCK-----
Version: 3.12
GIT d-- s+: a C+++ UL++++ P+ L+++ E--- W+++ N+++ o++ K++ w--- O M+ V- PS
PE Y++
PGP++ t-- 5 X R tv- b+ DI D+ G++ e++ h---- r+++ y++++
------END GEEK CODE BLOCK------


signature.asc (859 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Bug#902493: apache2-bin: Event MPM listener thread may get blocked by SSL shutdowns

Sven Hartge-5
On 11.02.19 11:29, Jan Wagner wrote:

> Okay ... here is an area where you can push forward. What about
> providing documentation patches?

Sure. What kind of documentation. NEWS.Debian?

Grüße,
Sven.


signature.asc (849 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Bug#902493: apache2-bin: Event MPM listener thread may get blocked by SSL shutdowns

Jan Wagner-4
Am 11.02.19 um 13:12 schrieb Sven Hartge:
>> Okay ... here is an area where you can push forward. What about
>> providing documentation patches?
> Sure. What kind of documentation. NEWS.Debian?

where would you expect such a documentation, as you suggested it?

Cheers, Jan.
--
Never write mail to <[hidden email]>, you have been warned!
-----BEGIN GEEK CODE BLOCK-----
Version: 3.12
GIT d-- s+: a C+++ UL++++ P+ L+++ E--- W+++ N+++ o++ K++ w--- O M+ V- PS
PE Y++
PGP++ t-- 5 X R tv- b+ DI D+ G++ e++ h---- r+++ y++++
------END GEEK CODE BLOCK------


signature.asc (859 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Bug#902493: apache2-bin: Event MPM listener thread may get blocked by SSL shutdowns

Sven Hartge-5
On 11.02.19 13:36, Jan Wagner wrote:
> Am 11.02.19 um 13:12 schrieb Sven Hartge:


>>> Okay ... here is an area where you can push forward. What about
>>> providing documentation patches?

>> Sure. What kind of documentation. NEWS.Debian?

> where would you expect such a documentation, as you suggested it?

I'd expect this information in NEWS.Debian.

Maybe something like this:

-----------8<------------------
Apache2.4 in Debian uses mpm_event to process requests.

When used with SSL and under very specific circumstances, Apache will
stop accepting new connections until a certain timeout has occurred.
This causes a service disruption as the webserver not longer processes
new request.

To work around this issue, switch to mpm_worker or mpm_prefork, which
both don't show this problem.

This issue has been documented as https://bugs.debian.org/902493 and
https://bz.apache.org/bugzilla/show_bug.cgi?id=60956
-----------8<------------------

Language of course needs a bit of refining from my German-English.

Again: I still propose to switch to mpm_worker as default MPM for
Stretch so not every admin has to hit the same wall at some time.

Grüße,
Sven.


signature.asc (849 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Bug#902493: apache2-bin: Event MPM listener thread may get blocked by SSL shutdowns

Gedalya
In reply to this post by Stefan Fritsch
On Sun, 10 Feb 2019 14:36:11 +0100 (CET) Stefan Fritsch <[hidden email]> wrote:

> If we did a backport, rhe risk of introducing regressions would be quite high.

So how about backporting it to stretch-backports? Isn't that what the backports section is for?

It would be then available to those who have interest in it and are willing to accept potential side effects. If we are _not_ going to fix it in stable, I think this could still provide some convenience to some users.

Reply | Threaded
Open this post in threaded view
|

Bug#902493: apache2-bin: Event MPM listener thread may get blocked by SSL shutdowns

Jan Wagner-4
Am 12.02.19 um 17:28 schrieb Gedalya:
> So how about backporting it to stretch-backports? Isn't that what the backports section is for?

backports is not meant for fixing things. beside that it would require
all rebuilding most of the additional apache modules not shiped by the
apache2 source package.

with regards, jan.
--
Never write mail to <[hidden email]>, you have been warned!
-----BEGIN GEEK CODE BLOCK-----
Version: 3.12
GIT d-- s+: a C+++ UL++++ P+ L+++ E--- W+++ N+++ o++ K++ w--- O M+ V- PS
PE Y++
PGP++ t-- 5 X R tv- b+ DI D+ G++ e++ h---- r+++ y++++
------END GEEK CODE BLOCK------


signature.asc (859 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Bug#902493: apache2-bin: Event MPM listener thread may get blocked by SSL shutdowns

Gedalya
On 2/13/19 12:38 AM, Jan Wagner wrote:
> backports is not meant for fixing things. beside that it would require
> all rebuilding most of the additional apache modules not shiped by the
> apache2 source package.

So we're back to doing nothing at all?

I'm not at all advocating backports as a solution, rather suggesting that it would be _something_ that could be done while the issue is not fixed, for whatever reason.

As for rebuilding other sources packages, yes, but like you said, not only is backports not for fixing things, it has been traditionally known to break things as well. Packages from backports are indeed sometimes not installable to some users.

In my case I wasn't using any other modules (PHP via fcgi), so apache2, apr and apr-util were enough. Again, we're not forcing anyone to use this, nor are we saying that this addresses the issue.

Reply | Threaded
Open this post in threaded view
|

Bug#902493: apache2-bin: Event MPM listener thread may get blocked by SSL shutdowns

Jan Wagner-4
Am 12.02.19 um 17:44 schrieb Gedalya:
> On 2/13/19 12:38 AM, Jan Wagner wrote:
>> backports is not meant for fixing things. beside that it would require
>> all rebuilding most of the additional apache modules not shiped by the
>> apache2 source package.
> So we're back to doing nothing at all?

Just to point out: I'm not (one of) the package maintainer(s), I'm just
speaking here as user.


signature.asc (859 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Bug#902493: apache2-bin: Event MPM listener thread may get blocked by SSL shutdowns

Gedalya
In reply to this post by Sven Hartge-5
On Mon, 11 Feb 2019 14:28:00 +0100 Sven Hartge <[hidden email]> wrote:
>
> Language of course needs a bit of refining from my German-English.
>


Nothing atrocious, really, but I was able to come up with something a bit more concise.


------->8-------
By default, Apache 2.4 as provided by Debian uses mpm_event to process requests.

When used with TLS and under very specific circumstances, Apache will stop accepting new connections for a certain period of time.

To work around this issue, you may switch to mpm_worker or mpm_prefork.

For further discussion, see https://bugs.debian.org/902493 and https://bz.apache.org/bugzilla/show_bug.cgi?id=60956

Reply | Threaded
Open this post in threaded view
|

Bug#902493: apache2-bin: Event MPM listener thread may get blocked by SSL shutdowns

Sven Hartge-5
On 12.02.19 17:51, Gedalya wrote:

> On Mon, 11 Feb 2019 14:28:00 +0100 Sven Hartge <[hidden email]> wrote:
>>
>> Language of course needs a bit of refining from my German-English.
>>
>
>
> Nothing atrocious, really, but I was able to come up with something a bit more concise.
>
>
> ------->8-------
> By default, Apache 2.4 as provided by Debian uses mpm_event to process requests.
>
> When used with TLS and under very specific circumstances, Apache will stop accepting new connections for a certain period of time.
>
> To work around this issue, you may switch to mpm_worker or mpm_prefork.
>
> For further discussion, see https://bugs.debian.org/902493 and https://bz.apache.org/bugzilla/show_bug.cgi?id=60956
Maybe adding the Release to the first sentence to convey that this is
only a problem for Stretch and not Buster:

------->8-------
By default, Apache 2.4 as provided by Debian Stretch uses mpm_event to
process requests.

When used with TLS and under very specific circumstances, Apache will
stop accepting new connections for a certain period of time.

To work around this issue, you may switch to mpm_worker or mpm_prefork.

For further discussion, see https://bugs.debian.org/902493 and
https://bz.apache.org/bugzilla/show_bug.cgi?id=60956


signature.asc (849 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Bug#902493: apache2-bin: Event MPM listener thread may get blocked by SSL shutdowns

Gedalya
On 2/13/19 12:54 AM, Sven Hartge wrote:
> Maybe adding the Release to the first sentence to convey that this is
> only a problem for Stretch and not Buster

Yea.

Well, since mpm_event is still the default in buster :-)

------->8-------

By default, Apache 2.4 as provided by Debian uses mpm_event to process requests.

The version of Apache included in Debian stretch, when used with TLS and under very specific circumstances, may stop accepting new connections for a certain period of time.

To work around this issue, you may switch to mpm_worker or mpm_prefork.

Apache in buster and later releases is not affected by this issue.

For further discussion, see https://bugs.debian.org/902493 and https://bz.apache.org/bugzilla/show_bug.cgi?id=60956

Reply | Threaded
Open this post in threaded view
|

Bug#902493: apache2-bin: Event MPM listener thread may get blocked by SSL shutdowns

Stefan Fritsch
In reply to this post by Gedalya
On Tuesday, 12 February 2019 17:44:39 CET Gedalya wrote:

> On 2/13/19 12:38 AM, Jan Wagner wrote:
> > backports is not meant for fixing things. beside that it would require
> > all rebuilding most of the additional apache modules not shiped by the
> > apache2 source package.
>
> So we're back to doing nothing at all?
>
> I'm not at all advocating backports as a solution, rather suggesting that it
> would be _something_ that could be done while the issue is not fixed, for
> whatever reason.
>
> As for rebuilding other sources packages, yes, but like you said, not only
> is backports not for fixing things, it has been traditionally known to
> break things as well. Packages from backports are indeed sometimes not
> installable to some users.
>
> In my case I wasn't using any other modules (PHP via fcgi), so apache2, apr
> and apr-util were enough. Again, we're not forcing anyone to use this, nor
> are we saying that this addresses the issue.

I will think about it. It I definitely won't have time in the next 2-3 weeks.

Cheers,
Stefan

12