Bug#921617: mumble: Mumble memory use increases over time to unreasonable amounts

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

Bug#921617: mumble: Mumble memory use increases over time to unreasonable amounts

Colomban Wendling-3
Package: mumble
Version: 1.3.0~git20190125.440b173+dfsg-1
Severity: important

Dear Maintainer,

I use Mumble daily as a remote work team communication channel, so it's pretty
much open non-stop for one day at a time.

Since a few days, maybe a week, Mumble's memory usage is steadily growing to
crzay levels: right now, `top` report it's using 25% of my 12Go of memory
(resident memory shows 3g).  This situation forces me to restart the client
every now and then, unfortunately at least a few times a day -- and
obviously is also a problem for overall use of resources.
I wouldn't say it started right away with the new Qt5 Mumble, but it could
have, I'm not completely sure.

It wasn't always like that, and used to work fine.

I cannot really provide credentials for the server, but I can do tests if
need be.  There is no issue I can see on Mumble's stdout/stderr:

<D>2019-02-07 10:49:42.966 libopus 1.3 from libopus.so.0
<W>2019-02-07 10:49:42.967 CELT bitstream 8000000b from /usr/lib/mumble/libcelt0.so.0.7.0
<W>2019-02-07 10:49:42.970 Theme: "Mumble"
<W>2019-02-07 10:49:42.970 Style: "Lite"
<W>2019-02-07 10:49:42.970 --> qss: ":themes/Mumble/Lite.qss"
<W>2019-02-07 10:49:42.971 Locale is "fr_FR" (System: "fr_FR")
<W>2019-02-07 10:49:43.215 Database SQLite: "3.26.0"
<W>2019-02-07 10:49:43.219 Overlay: Listening on "/run/user/1000/MumbleOverlayPipe"
<W>2019-02-07 10:49:43.252 Updating application palette
<W>2019-02-07 10:49:43.291 GlobalShortcutX: Using XI2 2.3
<W>2019-02-07 10:49:44.697 AudioInput: Opus encoder set for VOIP
<W>2019-02-07 10:49:44.697 AudioInput: 23000 bits/s, 48000 hz, 480 sample
<W>2019-02-07 10:49:44.697 PulseAudio: Starting input alsa_input.pci-0000_00_1b.0.analog-stereo
<W>2019-02-07 10:49:44.697 PulseAudio: Starting echo: alsa_output.pci-0000_00_1b.0.analog-stereo.monitor
<W>2019-02-07 10:49:44.697 PulseAudio: Starting output: alsa_output.pci-0000_00_1b.0.analog-stereo
<W>2019-02-07 10:49:44.699 AudioOutput: Initialized 2 channel 48000 hz mixer
<W>2019-02-07 10:49:44.699 AudioInput: Initialized mixer for 1 channel 44100 hz mic and 0 channel 48000 hz echo
warning: The VAD has been replaced by a hack pending a complete rewrite
<W>2019-02-07 10:49:44.726 AudioInput: Initialized mixer for 1 channel 44100 hz mic and 2 channel 48000 hz echo
warning: The VAD has been replaced by a hack pending a complete rewrite
<W>2019-02-07 10:49:44.736 AudioInput: ECHO CANCELLER ACTIVE
<W>2019-02-07 10:49:44.847 Database SQLite: "3.26.0"
<W>2019-02-07 10:49:44.848 OpenSSL Support: 1 (OpenSSL 1.1.1a  20 Nov 2018)
<W>2019-02-07 10:49:45.277 ServerHandler: TLS cipher preference is "TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256:TLS_AES_128_GCM_SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-SHA:DHE-RSA-AES128-SHA:AES256-SHA:AES128-SHA"

Regards,
Colomban

-- System Information:
Debian Release: buster/sid
  APT prefers unstable-debug
  APT policy: (500, 'unstable-debug'), (500, 'testing-debug'), (500, 'unstable'), (500, 'testing'), (500, 'stable')
Architecture: amd64 (x86_64)

Kernel: Linux 4.19.0-2-amd64 (SMP w/4 CPU cores)
Locale: LANG=fr_FR.UTF-8, LC_CTYPE=fr_FR.UTF-8 (charmap=UTF-8), LANGUAGE=fr_FR.UTF-8 (charmap=UTF-8)
Shell: /bin/sh linked to /bin/dash
Init: systemd (via /run/systemd/system)
LSM: AppArmor: enabled

Versions of packages mumble depends on:
ii  libasound2                 1.1.7-2
ii  libavahi-client3           0.7-4+b1
ii  libavahi-common3           0.7-4+b1
ii  libavahi-compat-libdnssd1  0.7-4+b1
ii  libc6                      2.28-5
ii  libgcc1                    1:8.2.0-15
ii  libgl1                     1.1.0-1
ii  libglib2.0-0               2.58.2-4
ii  libopus0                   1.3-1
ii  libprotobuf17              3.6.1.3-1
ii  libpulse0                  12.2-3
ii  libqt5core5a               5.11.3+dfsg-2
ii  libqt5dbus5                5.11.3+dfsg-2
ii  libqt5gui5                 5.11.3+dfsg-2
ii  libqt5network5             5.11.3+dfsg-2
ii  libqt5sql5                 5.11.3+dfsg-2
ii  libqt5sql5-sqlite          5.11.3+dfsg-2
ii  libqt5svg5                 5.11.3-2
ii  libqt5widgets5             5.11.3+dfsg-2
ii  libqt5xml5                 5.11.3+dfsg-2
ii  libsndfile1                1.0.28-4
ii  libspeechd2                0.9.0-1
ii  libspeex1                  1.2~rc1.2-1+b2
ii  libspeexdsp1               1.2~rc1.2-1+b2
ii  libssl1.1                  1.1.1a-1
ii  libstdc++6                 8.2.0-15
ii  libx11-6                   2:1.6.7-1
ii  libxi6                     2:1.7.9-1
ii  lsb-release                10.2018112800

mumble recommends no packages.

Versions of packages mumble suggests:
pn  mumble-server      <none>
ii  speech-dispatcher  0.9.0-1

-- no debconf information

Reply | Threaded
Open this post in threaded view
|

Bug#921617: mumble: Mumble memory use increases over time to unreasonable amounts

Chris Knadle
Colomban Wendling:
> Package: mumble
> Version: 1.3.0~git20190125.440b173+dfsg-1
> Severity: important
>
> Dear Maintainer,
>
> I use Mumble daily as a remote work team communication channel, so it's pretty
> much open non-stop for one day at a time.

Understood.
I typically also leave the Mumble client open for long periods of time, but have
not tried doing this with Mumble 1.3 yet.  When possible I'll try to replicate
this bug to see if I see the same behavior.

> Since a few days, maybe a week, Mumble's memory usage is steadily growing to
> crzay levels: right now, `top` report it's using 25% of my 12Go of memory
> (resident memory shows 3g).  This situation forces me to restart the client
> every now and then, unfortunately at least a few times a day -- and
> obviously is also a problem for overall use of resources.
> I wouldn't say it started right away with the new Qt5 Mumble, but it could
> have, I'm not completely sure.
>
> It wasn't always like that, and used to work fine.

Just to mention: this isn't the first report of "memory leak" on Mumble -- see
Debian bug #683827.
   https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=683827

I wasn't able to reproduce #683827 on newer versions of Mumble 1.2.x but wasn't
sure what the underlying cause/issue was there, or what version of Mumble may
have fixed that bug.

Due to recent security bugs in Mumble I'm going to be preparing a version of
Mumble 1.3 with Qt4 (targeted for Stretch) -- so there will be an opportunity to
test that to see if it shows the same memory leak.

> I cannot really provide credentials for the server, but I can do tests if
> need be.  There is no issue I can see on Mumble's stdout/stderr:
>
> <D>2019-02-07 10:49:42.966 libopus 1.3 from libopus.so.0
> <W>2019-02-07 10:49:42.967 CELT bitstream 8000000b from /usr/lib/mumble/libcelt0.so.0.7.0
> <W>2019-02-07 10:49:42.970 Theme: "Mumble"
> <W>2019-02-07 10:49:42.970 Style: "Lite"
> <W>2019-02-07 10:49:42.970 --> qss: ":themes/Mumble/Lite.qss"
> <W>2019-02-07 10:49:42.971 Locale is "fr_FR" (System: "fr_FR")
> <W>2019-02-07 10:49:43.215 Database SQLite: "3.26.0"
> <W>2019-02-07 10:49:43.219 Overlay: Listening on "/run/user/1000/MumbleOverlayPipe"
> <W>2019-02-07 10:49:43.252 Updating application palette
> <W>2019-02-07 10:49:43.291 GlobalShortcutX: Using XI2 2.3
> <W>2019-02-07 10:49:44.697 AudioInput: Opus encoder set for VOIP
> <W>2019-02-07 10:49:44.697 AudioInput: 23000 bits/s, 48000 hz, 480 sample
> <W>2019-02-07 10:49:44.697 PulseAudio: Starting input alsa_input.pci-0000_00_1b.0.analog-stereo
> <W>2019-02-07 10:49:44.697 PulseAudio: Starting echo: alsa_output.pci-0000_00_1b.0.analog-stereo.monitor
> <W>2019-02-07 10:49:44.697 PulseAudio: Starting output: alsa_output.pci-0000_00_1b.0.analog-stereo
> <W>2019-02-07 10:49:44.699 AudioOutput: Initialized 2 channel 48000 hz mixer
> <W>2019-02-07 10:49:44.699 AudioInput: Initialized mixer for 1 channel 44100 hz mic and 0 channel 48000 hz echo
> warning: The VAD has been replaced by a hack pending a complete rewrite
> <W>2019-02-07 10:49:44.726 AudioInput: Initialized mixer for 1 channel 44100 hz mic and 2 channel 48000 hz echo
> warning: The VAD has been replaced by a hack pending a complete rewrite
> <W>2019-02-07 10:49:44.736 AudioInput: ECHO CANCELLER ACTIVE
> <W>2019-02-07 10:49:44.847 Database SQLite: "3.26.0"
> <W>2019-02-07 10:49:44.848 OpenSSL Support: 1 (OpenSSL 1.1.1a  20 Nov 2018)
> <W>2019-02-07 10:49:45.277 ServerHandler: TLS cipher preference is "TLS_AES_256_GCM_SHA384:TLS_CHACHA20_POLY1305_SHA256:TLS_AES_128_GCM_SHA256:ECDHE-ECDSA-AES256-GCM-SHA384:ECDHE-RSA-AES256-GCM-SHA384:ECDHE-ECDSA-AES128-GCM-SHA256:ECDHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-GCM-SHA384:DHE-RSA-AES128-GCM-SHA256:DHE-RSA-AES256-SHA:DHE-RSA-AES128-SHA:AES256-SHA:AES128-SHA"
>
> Regards,
> Colomban

I don't personally know of a "good" way to debug memory leaks.  As far as I know
this involves running the target program via a debugger like GDB and figuring
out how to watch memory allocations and frees.  Unfortunately I don't have any
experience with doing that [successfully] yet.

   -- Chris

--
Chris Knadle
[hidden email]

Reply | Threaded
Open this post in threaded view
|

Bug#921617: mumble: Mumble memory use increases over time to unreasonable amounts

Colomban Wendling-3
Le 07/02/2019 à 16:56, Chris Knadle a écrit :
> […]
>
> Just to mention: this isn't the first report of "memory leak" on Mumble -- see
> Debian bug #683827.
>    https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=683827
>
> I wasn't able to reproduce #683827 on newer versions of Mumble 1.2.x but wasn't
> sure what the underlying cause/issue was there, or what version of Mumble may
> have fixed that bug.

The links you have there are interesting; for example
https://github.com/mumble-voip/mumble/issues/1074#issuecomment-203199996
suggests that it might be due to echo canceller and other apps "messing"
with PulseAudio.  I do have echo canceller enabled (that I should
actually be able to disable as I'm using push-to-talk with a headset),
and am running at least one virtual machine which could be doing
something with PulseAudio.

I'll try and do some tests next chance I get (probably next week).

> Due to recent security bugs in Mumble I'm going to be preparing a version of
> Mumble 1.3 with Qt4 (targeted for Stretch) -- so there will be an opportunity to
> test that to see if it shows the same memory leak.

I can also test if reverting to a previous version of Mumble helps; I'm
not sure which are the security issues but in my case I trust the server
so there might be no problem using an older version for testing.

> […]
>
> I don't personally know of a "good" way to debug memory leaks.  As far as I know
> this involves running the target program via a debugger like GDB and figuring
> out how to watch memory allocations and frees.  Unfortunately I don't have any
> experience with doing that [successfully] yet.

Valgrind's memcheck is the best I know.  I'll try to see if I can run
Mumble in it and find out what I can from there -- although it often is
a pain starting from nothing, as many toolkits and apps have gazillions
of innocent leaks cluttering the results.

Thanks for the input, and I'll get back here with any data I can gather
on this.

Regards,
Colomban

Reply | Threaded
Open this post in threaded view
|

Bug#921617: mumble: Mumble memory use increases over time to unreasonable amounts

Chris Knadle
Colomban Wendling:

> Le 07/02/2019 à 16:56, Chris Knadle a écrit :
>> […]
>>
>> Just to mention: this isn't the first report of "memory leak" on Mumble -- see
>> Debian bug #683827.
>>    https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=683827
>>
>> I wasn't able to reproduce #683827 on newer versions of Mumble 1.2.x but wasn't
>> sure what the underlying cause/issue was there, or what version of Mumble may
>> have fixed that bug.
>
> The links you have there are interesting; for example
> https://github.com/mumble-voip/mumble/issues/1074#issuecomment-203199996
> suggests that it might be due to echo canceller and other apps "messing"
> with PulseAudio.  I do have echo canceller enabled (that I should
> actually be able to disable as I'm using push-to-talk with a headset),
> and am running at least one virtual machine which could be doing
> something with PulseAudio.
>
> I'll try and do some tests next chance I get (probably next week).

*nod*  Okay.  Yeah the echo canceler would be part of the Mumble binary, so if
the canceler is misbehaving memory-wise then that would make sense.  However if
another virtual machine were causing issues with PulseAudio then that shouldn't
cause memory expansion/leaking in the Mumble client binary (AFAIK).

>> Due to recent security bugs in Mumble I'm going to be preparing a version of
>> Mumble 1.3 with Qt4 (targeted for Stretch) -- so there will be an opportunity to
>> test that to see if it shows the same memory leak.
>
> I can also test if reverting to a previous version of Mumble helps; I'm
> not sure which are the security issues but in my case I trust the server
> so there might be no problem using an older version for testing.

The two recent security issues relate to mumble-server specifically, not the
client.  One security issue has been resolved, another is described in Debian
bug #919249 and on this Debian Secuirty page for CVE-2018-20743:

   https://security-tracker.debian.org/tracker/CVE-2018-20743

>> […]
>>
>> I don't personally know of a "good" way to debug memory leaks.  As far as I know
>> this involves running the target program via a debugger like GDB and figuring
>> out how to watch memory allocations and frees.  Unfortunately I don't have any
>> experience with doing that [successfully] yet.
>
> Valgrind's memcheck is the best I know.  I'll try to see if I can run
> Mumble in it and find out what I can from there -- although it often is
> a pain starting from nothing, as many toolkits and apps have gazillions
> of innocent leaks cluttering the results.

*headsmack*  I keep forgetting about Valgrind.  I can try playing with that too,
assuming I can replicate the memory leak.

> Thanks for the input, and I'll get back here with any data I can gather
> on this.

*nod*  Thank you very much for your help.

   -- Chris

--
Chris Knadle
[hidden email]

Reply | Threaded
Open this post in threaded view
|

Bug#921617: mumble: Mumble memory use increases over time to unreasonable amounts

Colomban Wendling-3
Le 08/02/2019 à 16:54, Chris Knadle a écrit :

> Colomban Wendling:
>> Le 07/02/2019 à 16:56, Chris Knadle a écrit :
>>> […]
>>>
>>> Just to mention: this isn't the first report of "memory leak" on Mumble -- see
>>> Debian bug #683827.
>>>    https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=683827
>>>
>>> I wasn't able to reproduce #683827 on newer versions of Mumble 1.2.x but wasn't
>>> sure what the underlying cause/issue was there, or what version of Mumble may
>>> have fixed that bug.
>>
>> The links you have there are interesting; for example
>> https://github.com/mumble-voip/mumble/issues/1074#issuecomment-203199996
>> suggests that it might be due to echo canceller and other apps "messing"
>> with PulseAudio.  I do have echo canceller enabled (that I should
>> actually be able to disable as I'm using push-to-talk with a headset),
>> and am running at least one virtual machine which could be doing
>> something with PulseAudio.
>>
>> I'll try and do some tests next chance I get (probably next week).
>
> *nod*  Okay.  Yeah the echo canceler would be part of the Mumble binary, so if
> the canceler is misbehaving memory-wise then that would make sense.

I tried without the echo canceller and it's behaving reasonably so far,
after 5 hours: it's using 109M resident memory, and peaked at 123M.
So it seems it's indeed the echo canceller that's leaking a lot.

>  However if
> another virtual machine were causing issues with PulseAudio then that shouldn't
> cause memory expansion/leaking in the Mumble client binary (AFAIK).

IIUC from the report I linked, some software alter the input sinks in a
way that confused the echo canceller.  It'd say it's still a bug on the
canceller part, but it might be triggered only on some conditions that
don't normally happen, but that some software doing their own audio
input trigger -- not quite sure.

Regards,
Colomban

Reply | Threaded
Open this post in threaded view
|

Bug#921617: mumble: Mumble memory use increases over time to unreasonable amounts

Chris Knadle
retitle 921617 Mumble echo canceler leaks memory
forwarded 921617 https://github.com/mumble-voip/mumble/issues/3379
thanks

Colomban Wendling:

> Le 08/02/2019 à 16:54, Chris Knadle a écrit :
>> Colomban Wendling:
>>> Le 07/02/2019 à 16:56, Chris Knadle a écrit :
>>>> […]
>>>>
>>>> Just to mention: this isn't the first report of "memory leak" on Mumble -- see
>>>> Debian bug #683827.
>>>>    https://bugs.debian.org/cgi-bin/bugreport.cgi?bug=683827
>>>>
>>>> I wasn't able to reproduce #683827 on newer versions of Mumble 1.2.x but wasn't
>>>> sure what the underlying cause/issue was there, or what version of Mumble may
>>>> have fixed that bug.
>>>
>>> The links you have there are interesting; for example
>>> https://github.com/mumble-voip/mumble/issues/1074#issuecomment-203199996
>>> suggests that it might be due to echo canceller and other apps "messing"
>>> with PulseAudio.  I do have echo canceller enabled (that I should
>>> actually be able to disable as I'm using push-to-talk with a headset),
>>> and am running at least one virtual machine which could be doing
>>> something with PulseAudio.
>>>
>>> I'll try and do some tests next chance I get (probably next week).
>>
>> *nod*  Okay.  Yeah the echo canceler would be part of the Mumble binary, so if
>> the canceler is misbehaving memory-wise then that would make sense.
>
> I tried without the echo canceller and it's behaving reasonably so far,
> after 5 hours: it's using 109M resident memory, and peaked at 123M.
> So it seems it's indeed the echo canceller that's leaking a lot.

Okay.  Thank you very much for testing and finding this, as it helps narrow down
where the problem is.

I'm re-titling the bug so that it will be easier for others to find when looking
to find information about it.

>>  However if
>> another virtual machine were causing issues with PulseAudio then that shouldn't
>> cause memory expansion/leaking in the Mumble client binary (AFAIK).
>
> IIUC from the report I linked, some software alter the input sinks in a
> way that confused the echo canceller.  It'd say it's still a bug on the
> canceller part, but it might be triggered only on some conditions that
> don't normally happen, but that some software doing their own audio
> input trigger -- not quite sure.

*nod*  Hmm.  So I think the theory there is that the because other VMs interact
with PulseAudio that it changes the interaction between PulseAudio and the
Mumble echo canceler such that the echo canceler uses increasing memory.  It
feels like that "shouldn't happen", but then again that's the definition of a
"bug" and this is a bug, so... maybe it's possible.

I had a look at the open issues for mumble and found #3379 which is about Mumble
echo cancellation on Windows:

   https://github.com/mumble-voip/mumble/issues/3379

also issue #3406 which shows the memory leak to be intermittent (if true that
would be annoying, because that makes the root cause harder to find):

   https://github.com/mumble-voip/mumble/issues/3406

I'm marking this bug as related to Mumble issue #3379 upstream and adding an
entry to the upstream bug pointing to this one so that upstream can see that
this is likely affecting other OSes than Windows.

Thanks

   -- Chris

--
Chris Knadle
[hidden email]