cannot allocate memory in static TLS block (Was: Bug#953832: drmaa: autopkgtest failure: RuntimeError: Could not find drmaa library)

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

cannot allocate memory in static TLS block (Was: Bug#953832: drmaa: autopkgtest failure: RuntimeError: Could not find drmaa library)

Andreas Tille-2
Control: tags -1 help
Control: retitle -1 cannot allocate memory in static TLS block

Hi Paul,

On Fri, Mar 13, 2020 at 11:09:31PM +0100, Paul Gevers wrote:
> <module>
>     raise RuntimeError(('Could not find drmaa library.  Please specify
> its ' +
> RuntimeError: Could not find drmaa library.  Please specify its full
> path using the environment variable DRMAA_LIBRARY_PATH

I've fixed this in Git.  Unfortunately I get a new issue when importing
drmaa

$ python3
Python 3.7.6 (default, Jan 19 2020, 22:34:52)
[GCC 9.2.1 20200117] on linux
Type "help", "copyright", "credits" or "license" for more information.
>>> import drmaa
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/home/andreas/debian-maintain/salsa/med-team/python-drmaa/drmaa/__init__.py", line 65, in <module>
    from .session import JobInfo, JobTemplate, Session
  File "/home/andreas/debian-maintain/salsa/med-team/python-drmaa/drmaa/session.py", line 39, in <module>
    from drmaa.helpers import (adapt_rusage, Attribute, attribute_names_iterator,
  File "/home/andreas/debian-maintain/salsa/med-team/python-drmaa/drmaa/helpers.py", line 36, in <module>
    from drmaa.wrappers import (drmaa_attr_names_t, drmaa_attr_values_t,
  File "/home/andreas/debian-maintain/salsa/med-team/python-drmaa/drmaa/wrappers.py", line 58, in <module>
    _lib = CDLL(libpath, mode=RTLD_GLOBAL)
  File "/usr/lib/python3.7/ctypes/__init__.py", line 364, in __init__
    self._handle = _dlopen(self._name, mode)
OSError: /usr/lib/x86_64-linux-gnu/libjemalloc.so.2: cannot allocate memory in static TLS block


Any help would be welcome

     Andreas.

--
http://fam-tille.de

Reply | Threaded
Open this post in threaded view
|

Re: cannot allocate memory in static TLS block (Was: Bug#953832: drmaa: autopkgtest failure: RuntimeError: Could not find drmaa library)

Paul Gevers-4
Hi Andreas,

On 14-03-2020 14:30, Andreas Tille wrote:
> Hi Paul,

[...]

> Any help would be welcome

I can't help you with this.

Paul


signature.asc (499 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: cannot allocate memory in static TLS block

Sergio Durigan Junior-3
In reply to this post by Andreas Tille-2
On Saturday, March 14 2020, Andreas Tille wrote:

> On Fri, Mar 13, 2020 at 11:09:31PM +0100, Paul Gevers wrote:
>> <module>
>>     raise RuntimeError(('Could not find drmaa library.  Please specify
>> its ' +
>> RuntimeError: Could not find drmaa library.  Please specify its full
>> path using the environment variable DRMAA_LIBRARY_PATH
>
> I've fixed this in Git.  Unfortunately I get a new issue when importing
> drmaa
>
> $ python3
> Python 3.7.6 (default, Jan 19 2020, 22:34:52)
> [GCC 9.2.1 20200117] on linux
> Type "help", "copyright", "credits" or "license" for more information.
>>>> import drmaa
> Traceback (most recent call last):
>   File "<stdin>", line 1, in <module>
>   File "/home/andreas/debian-maintain/salsa/med-team/python-drmaa/drmaa/__init__.py", line 65, in <module>
>     from .session import JobInfo, JobTemplate, Session
>   File "/home/andreas/debian-maintain/salsa/med-team/python-drmaa/drmaa/session.py", line 39, in <module>
>     from drmaa.helpers import (adapt_rusage, Attribute, attribute_names_iterator,
>   File "/home/andreas/debian-maintain/salsa/med-team/python-drmaa/drmaa/helpers.py", line 36, in <module>
>     from drmaa.wrappers import (drmaa_attr_names_t, drmaa_attr_values_t,
>   File "/home/andreas/debian-maintain/salsa/med-team/python-drmaa/drmaa/wrappers.py", line 58, in <module>
>     _lib = CDLL(libpath, mode=RTLD_GLOBAL)
>   File "/usr/lib/python3.7/ctypes/__init__.py", line 364, in __init__
>     self._handle = _dlopen(self._name, mode)
> OSError: /usr/lib/x86_64-linux-gnu/libjemalloc.so.2: cannot allocate memory in static TLS block
>
>
> Any help would be welcome
This is an issue with jemalloc's handling of the TLS model when being
dlopened..  See:

  https://github.com/jemalloc/jemalloc/issues/1237

The recommended way to build a libjemalloc that is suitable for being
dlopened is to use '--disable-initial-exec-tls' when building it.  Take
a look at the INSTALL.md file, and look for this option:

  https://github.com/jemalloc/jemalloc/blob/dev/INSTALL.md

There is a way to workaround this bug by doing an LD_PRELOAD of
libjemalloc when invoking python, but this will only mask the problem
and we can't expect users to do/know this.

The way I see it, you can try to convince jemalloc's maintainer to
enable that flag.

BTW, the reason 'find_library' can't find drmaa's library is because the
.so is being installed in a non-standard directory.  I don't know why
the package was made like this, though.

Thanks,

--
Sergio
GPG key ID: 237A 54B1 0287 28BF 00EF  31F4 D0EB 7628 65FC 5E36
Please send encrypted e-mail if possible
http://sergiodj.net/

signature.asc (847 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: cannot allocate memory in static TLS block

Andreas Tille-2
Hi Faidon,

could you imagine to build jemalloc with --disable-initial-exec-tls
as Sergio suggests below to fix the issue in drmaa (and possibly other
packages)?

Should I open a separate bug report against jemalloc to request this?

Kind regards

      Andreas.

On Sat, Mar 14, 2020 at 05:18:49PM -0400, Sergio Durigan Junior wrote:

> > $ python3
> > Python 3.7.6 (default, Jan 19 2020, 22:34:52)
> > [GCC 9.2.1 20200117] on linux
> > Type "help", "copyright", "credits" or "license" for more information.
> >>>> import drmaa
> > Traceback (most recent call last):
> >   File "<stdin>", line 1, in <module>
> >   File "/home/andreas/debian-maintain/salsa/med-team/python-drmaa/drmaa/__init__.py", line 65, in <module>
> >     from .session import JobInfo, JobTemplate, Session
> >   File "/home/andreas/debian-maintain/salsa/med-team/python-drmaa/drmaa/session.py", line 39, in <module>
> >     from drmaa.helpers import (adapt_rusage, Attribute, attribute_names_iterator,
> >   File "/home/andreas/debian-maintain/salsa/med-team/python-drmaa/drmaa/helpers.py", line 36, in <module>
> >     from drmaa.wrappers import (drmaa_attr_names_t, drmaa_attr_values_t,
> >   File "/home/andreas/debian-maintain/salsa/med-team/python-drmaa/drmaa/wrappers.py", line 58, in <module>
> >     _lib = CDLL(libpath, mode=RTLD_GLOBAL)
> >   File "/usr/lib/python3.7/ctypes/__init__.py", line 364, in __init__
> >     self._handle = _dlopen(self._name, mode)
> > OSError: /usr/lib/x86_64-linux-gnu/libjemalloc.so.2: cannot allocate memory in static TLS block
>
> This is an issue with jemalloc's handling of the TLS model when being
> dlopened..  See:
>
>   https://github.com/jemalloc/jemalloc/issues/1237
>
> The recommended way to build a libjemalloc that is suitable for being
> dlopened is to use '--disable-initial-exec-tls' when building it.  Take
> a look at the INSTALL.md file, and look for this option:
>
>   https://github.com/jemalloc/jemalloc/blob/dev/INSTALL.md
>
> There is a way to workaround this bug by doing an LD_PRELOAD of
> libjemalloc when invoking python, but this will only mask the problem
> and we can't expect users to do/know this.
>
> The way I see it, you can try to convince jemalloc's maintainer to
> enable that flag.
>
> BTW, the reason 'find_library' can't find drmaa's library is because the
> .so is being installed in a non-standard directory.  I don't know why
> the package was made like this, though.
>
> Thanks,
>
> --
> Sergio
> GPG key ID: 237A 54B1 0287 28BF 00EF  31F4 D0EB 7628 65FC 5E36
> Please send encrypted e-mail if possible
> http://sergiodj.net/



--
http://fam-tille.de

Reply | Threaded
Open this post in threaded view
|

Re: cannot allocate memory in static TLS block

Faidon Liambotis-2
Hi Andreas,

Thanks for reaching out. It sounds like this is already reported as
#951704 (Cc'ed now). I'll need to give this a closer look, but I hope I
can have an update within the next couple of weeks. Does this work?

Thanks!
Faidon

On Sun, Mar 15, 2020 at 10:33:20AM +0100, Andreas Tille wrote:

> Hi Faidon,
>
> could you imagine to build jemalloc with --disable-initial-exec-tls
> as Sergio suggests below to fix the issue in drmaa (and possibly other
> packages)?
>
> Should I open a separate bug report against jemalloc to request this?
>
> Kind regards
>
>       Andreas.
>
> On Sat, Mar 14, 2020 at 05:18:49PM -0400, Sergio Durigan Junior wrote:
> > > $ python3
> > > Python 3.7.6 (default, Jan 19 2020, 22:34:52)
> > > [GCC 9.2.1 20200117] on linux
> > > Type "help", "copyright", "credits" or "license" for more information.
> > >>>> import drmaa
> > > Traceback (most recent call last):
> > >   File "<stdin>", line 1, in <module>
> > >   File "/home/andreas/debian-maintain/salsa/med-team/python-drmaa/drmaa/__init__.py", line 65, in <module>
> > >     from .session import JobInfo, JobTemplate, Session
> > >   File "/home/andreas/debian-maintain/salsa/med-team/python-drmaa/drmaa/session.py", line 39, in <module>
> > >     from drmaa.helpers import (adapt_rusage, Attribute, attribute_names_iterator,
> > >   File "/home/andreas/debian-maintain/salsa/med-team/python-drmaa/drmaa/helpers.py", line 36, in <module>
> > >     from drmaa.wrappers import (drmaa_attr_names_t, drmaa_attr_values_t,
> > >   File "/home/andreas/debian-maintain/salsa/med-team/python-drmaa/drmaa/wrappers.py", line 58, in <module>
> > >     _lib = CDLL(libpath, mode=RTLD_GLOBAL)
> > >   File "/usr/lib/python3.7/ctypes/__init__.py", line 364, in __init__
> > >     self._handle = _dlopen(self._name, mode)
> > > OSError: /usr/lib/x86_64-linux-gnu/libjemalloc.so.2: cannot allocate memory in static TLS block
> >
> > This is an issue with jemalloc's handling of the TLS model when being
> > dlopened..  See:
> >
> >   https://github.com/jemalloc/jemalloc/issues/1237
> >
> > The recommended way to build a libjemalloc that is suitable for being
> > dlopened is to use '--disable-initial-exec-tls' when building it.  Take
> > a look at the INSTALL.md file, and look for this option:
> >
> >   https://github.com/jemalloc/jemalloc/blob/dev/INSTALL.md
> >
> > There is a way to workaround this bug by doing an LD_PRELOAD of
> > libjemalloc when invoking python, but this will only mask the problem
> > and we can't expect users to do/know this.
> >
> > The way I see it, you can try to convince jemalloc's maintainer to
> > enable that flag.
> >
> > BTW, the reason 'find_library' can't find drmaa's library is because the
> > .so is being installed in a non-standard directory.  I don't know why
> > the package was made like this, though.
> >
> > Thanks,
> >
> > --
> > Sergio
> > GPG key ID: 237A 54B1 0287 28BF 00EF  31F4 D0EB 7628 65FC 5E36
> > Please send encrypted e-mail if possible
> > http://sergiodj.net/
>
>
>
> --
> http://fam-tille.de
>

Reply | Threaded
Open this post in threaded view
|

Re: cannot allocate memory in static TLS block

Andreas Tille-2
Hi Faidon,

On Tue, Mar 17, 2020 at 01:29:59PM +0200, Faidon Liambotis wrote:
> Hi Andreas,
>
> Thanks for reaching out. It sounds like this is already reported as
> #951704 (Cc'ed now).

OK.

> I'll need to give this a closer look, but I hope I
> can have an update within the next couple of weeks. Does this work?

Well, drmaa is certainly not the most important package inside Debian so
I see no reason to push you.  But it would be great to have all those
Python3 issues from the table sooner or later since it generated noise
for the most active Python 2->3 migrators.  Just take the time you need.

See you

      Andreas.

--
http://fam-tille.de