Should .a library contains non-reallocatable code?

classic Classic list List threaded Threaded
32 messages Options
12
Reply | Threaded
Open this post in threaded view
|

Should .a library contains non-reallocatable code?

Dmitry Katsubo
Dear Debian Developers,

I wonder what is the current state-of-art concerning the code in .a
library (archive for static linking). Should it contain PIC code?

Situation: Dynamic (.so) library needs to be linked against such (.a)
library.

Thank you for any advises and your opinion in advance.

On 23/01/2015 17:44, Antonio Diaz Diaz wrote:

> Hello Alexander and Dmitry,
>
> Alexander Alemayhu wrote:
>> forwarding a thread I had with Dmitry regarding [#]774164.
>> What do you think?
>
> Libocrad is available only in static form because it is just a very
> immature hack. Basically I add functions as users ask for them. It meets
> about all the requirements in section 8.3 "Static libraries" of
> http://www.debian.org/doc/debian-policy/ch-sharedlibs.html
>
> Ocrad itself is very immature. I can't guarantee any stability.
>
> Moreover, AFAIK, compiling the sources twice (once without PIC for
> CLI and another time with PIC for .a) is not trivial. Different names
> are needed for the two versions of the object files.
>
> BTW, libocrad.a is *not* used to create ocrad CLI. libocrad.a is just
> ocrad with an incomplete programming interface instead of a CLI.
>
>
> Best regards,
> Antonio.
>
> ---------- Forwarded message ----------
> From: Dmitry Katsubo <[hidden email]>
> Date: 2015-01-10 15:39 GMT+01:00
> Subject: Re: Bug#774164: libocrad-dev: libocrad.a contains non-reallocatable code
> To: Alexander Alemayhu <[hidden email]>
>
> On 06/01/2015 22:54, Alexander Alemayhu wrote:
>> It might take me some time to make a change, because of work.
>> Please also note I have not made Debian libraries before, so it
>> might take me some time to do it properly. If you can't wait and
>> would like to be the change, please send a patch or pull request
>> :)
>>
>> I had a little chat with my neighbour and he had some useful links
>> to share which should give more information on Debian best
>> practices.
>>
>> Thanks.
>>
>> See below:
>>
>> https://www.debian.org/doc/debian-policy/ch-sharedlibs.html
>> https://www.debian.org/doc/manuals/developers-reference/best-pkging-practices.html#bpp-libraries
>
> http://www.netfort.gr.jp/~dancer/column/libpkg-guide/libpkg-guide.html
>
> Thanks for the links!
>
> In particular this one:
>
> http://www.netfort.gr.jp/~dancer/column/libpkg-guide/libpkg-guide.html#staticonlylibs
>
> says
>
> "providing -fPIC versions of static libraries for linking with
> shared libraries is a bad sign, because the "unstable interface" is
> now exported through another library's stable library interface",
> which is true, but without -fPIC the resulting shared library is unusable.
>
> Ideally of course would be that OCRAD is split into light-weight
> command-line and dynamic library library, but I am sure that adds more
> organizational work -- would it be worth? Try to ask the OCRAD
> community if they would be happy to have OCRAD also as dynamic library
> (which can also be wrapped into Perl/Java/etc bridges).
>
> --
> With best regards,
> Dmitry


--
With best regards,
Dmitry


--
To UNSUBSCRIBE, email to [hidden email]
with a subject of "unsubscribe". Trouble? Contact [hidden email]
Archive: https://lists.debian.org/54DF4008.20707@...

Reply | Threaded
Open this post in threaded view
|

Re: Should .a library contains non-reallocatable code?

Simon Richter
Hi,

On 14.02.2015 13:31, Dmitry Katsubo wrote:

> I wonder what is the current state-of-art concerning the code in .a
> library (archive for static linking). Should it contain PIC code?

Normally, no.

> Situation: Dynamic (.so) library needs to be linked against such (.a)
> library.

That is generally frowned upon.

I do the same thing with vxi and librevisa -- I build the static library
with PIC code and statically link into librevisa, and I justify that by
the vxi code being generated RPC stubs that really don't need an extra
shared library package.

However, your case is different: a quick hack package without a stable
ABI is the exact opposite. From a distribution point of view, it is
difficult to track what version of a static library was linked, which is
why we use shared libraries as often as we can.

The slightly suboptimal solution for a library without a stable ABI is
to use a version number in the SONAME, leave out the version from the
package name and build a shlibs file that uses a dependency with a fixed
version.

This means that all packages using this library can only be upgraded
together, but at least it doesn't introduce lots of NEW packages with
every upload.

   Simon



signature.asc (308 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Should .a library contains non-reallocatable code?

Jakub Wilk-4
In reply to this post by Dmitry Katsubo
* Dmitry Katsubo <[hidden email]>, 2015-02-14, 13:31:
>I wonder what is the current state-of-art concerning the code in .a
>library (archive for static linking). Should it contain PIC code?

This is what Debian Policy (§10.2) currently says:

“As to the static libraries, the common case is not to have relocatable
code, since there is no benefit, unless in specific cases; therefore the
static version must not be compiled with the ‘-fPIC’ flag. Any
exception to this rule should be discussed on the mailing list
[hidden email], and the reasons for compiling with the
‘-fPIC’ flag must be recorded in the file ‘README.Debian’.”

--
Jakub Wilk


--
To UNSUBSCRIBE, email to [hidden email]
with a subject of "unsubscribe". Trouble? Contact [hidden email]
Archive: https://lists.debian.org/20150219220805.GA8329@...

Reply | Threaded
Open this post in threaded view
|

Re: Should .a library contains non-reallocatable code?

Jeff Epler
Here are two scenarios where building a static library (libfoo) with
-fPIC is desirable:

 * libbar has a stable API, so it should be shipped as a .so,
   but if it links libfoo.a, and libfoo.a is not -fPIC, then
   libbar has to be shipped as a a static library too

 * foomodule is a Python wrapper for libfoo, so it must be shipped
   as a .so, but if it links libfoo.a, and libfoo.a is not -fPIC,
   it is not possible to build foomodule at all

   (The same goes for wrapping the library for most other interpreted
   languages)

   (At $DAY_JOB this bit me in the last week [not pertaining to
   Debian-packaged software] so it's a sore spot at the moment)

Unless the circumstances of libfoo make these scenarios unlikely, it
seems like it is better for other packages to prefer -fPIC even when
building a static library.

I wonder whether these scenarios were considered when the Policy was
written.

Jeff


--
To UNSUBSCRIBE, email to [hidden email]
with a subject of "unsubscribe". Trouble? Contact [hidden email]
Archive: https://lists.debian.org/20150219231930.GB37960@...

Reply | Threaded
Open this post in threaded view
|

Re: Should .a library contains non-reallocatable code?

Simon McVittie-7
On 19/02/15 23:19, Jeff Epler wrote:
> Here are two scenarios where building a static library (libfoo) with
> -fPIC is desirable
...
> I wonder whether these scenarios were considered when the Policy was
> written.

Conversely, when that part of the policy was written, 32-bit x86 was the
major architecture.

My understanding is that i386 performance suffers significantly from PIC
(since PIC uses up a CPU register, and i386 doesn't have many of those),
whereas on architectures with more registers (notably including ARM,
x86-64 (including x32) and PowerPC), it doesn't hurt anywhere near as much.

    S


--
To UNSUBSCRIBE, email to [hidden email]
with a subject of "unsubscribe". Trouble? Contact [hidden email]
Archive: https://lists.debian.org/54E7232C.9030101@...

Reply | Threaded
Open this post in threaded view
|

Re: Should .a library contains non-reallocatable code?

Alastair McKinstry-9

On 20/02/2015 12:06, Simon McVittie wrote:

> On 19/02/15 23:19, Jeff Epler wrote:
>> Here are two scenarios where building a static library (libfoo) with
>> -fPIC is desirable
> ...
>> I wonder whether these scenarios were considered when the Policy was
>> written.
> Conversely, when that part of the policy was written, 32-bit x86 was the
> major architecture.
>
> My understanding is that i386 performance suffers significantly from PIC
> (since PIC uses up a CPU register, and i386 doesn't have many of those),
> whereas on architectures with more registers (notably including ARM,
> x86-64 (including x32) and PowerPC), it doesn't hurt anywhere near as much.

There is also the cost of a indirections (a double indirection, IIRC) to
reach global variables on PIC code through the GOT, while non-PIC code
can reach global objects directly.

Alastair

>     S
>
>

--
Alastair McKinstry, <[hidden email]>, <[hidden email]>, https://diaspora.sceal.ie/u/amckinstry
Misentropy: doubting that the Universe is becoming more disordered.


--
To UNSUBSCRIBE, email to [hidden email]
with a subject of "unsubscribe". Trouble? Contact [hidden email]
Archive: https://lists.debian.org/54E72644.5090103@...

Reply | Threaded
Open this post in threaded view
|

Re: Should .a library contains non-reallocatable code?

Jakub Wilk-4
In reply to this post by Jeff Epler
* Jeff Epler <[hidden email]>, 2015-02-19, 17:19:

>Here are two scenarios where building a static library (libfoo) with
>-fPIC is desirable:
>
>* libbar has a stable API, so it should be shipped as a .so, but if it
>links libfoo.a, and libfoo.a is not -fPIC, then libbar has to be
>shipped as a a static library too
>
>* foomodule is a Python wrapper for libfoo, so it must be shipped as a
>.so, but if it links libfoo.a, and libfoo.a is not -fPIC, it is not
>possible to build foomodule at all
>

I'll add:

* User wants to build position-independent executable that links to
libfoo.a.

>I wonder whether these scenarios were considered when the Policy was
>written.

IMO the policy is overly strict and it should be relaxed.

--
Jakub Wilk


--
To UNSUBSCRIBE, email to [hidden email]
with a subject of "unsubscribe". Trouble? Contact [hidden email]
Archive: https://lists.debian.org/20150220123606.GA5655@...

Reply | Threaded
Open this post in threaded view
|

Re: Should .a library contains non-reallocatable code?

Simon Richter-2
-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA1

Hi,

Am 20.02.2015 um 13:36 schrieb Jakub Wilk:

> IMO the policy is overly strict and it should be relaxed.

Speaking of relaxing things: could this be solved with linker relaxations?

The compiler would need to generate both PIC and non-PIC code in this
case, and specify that the non-PIC code can substitute the PIC code
unless the required relocations cannot be performed. This is similar
to the normal use case for relaxations, where I would generate e.g.
two alternative jump instructions differing in offset length, and
instruct the linker to use the shorter form.

   Simon
-----BEGIN PGP SIGNATURE-----
Version: GnuPG v2

iJwEAQECAAYFAlTnhykACgkQ0sfeulffv7t9SQP/ciifpIJLcZcJh/rEM0+EnhmB
S9zs1n5xK7pJLXF7rQG1NCTUWXipm6Pgaul1SJRmaURkizd7to1d9h1XvJJGGLno
9awHZstfKyrr/ukpuu6CifGHzhgiMemZ7QvZRnLT1lZVAF2YGREc0daQDQRwVLNT
8BPQnm4VYsXKUYtlLhs=
=M4GR
-----END PGP SIGNATURE-----


--
To UNSUBSCRIBE, email to [hidden email]
with a subject of "unsubscribe". Trouble? Contact [hidden email]
Archive: https://lists.debian.org/54E7872A.4080204@...

Reply | Threaded
Open this post in threaded view
|

Re: Should .a library contains non-reallocatable code?

Bernhard R. Link-2
In reply to this post by Jeff Epler
* Jeff Epler <[hidden email]> [150220 00:19]:
> Here are two scenarios where building a static library (libfoo) with
> -fPIC is desirable:
>
>  * libbar has a stable API, so it should be shipped as a .so,
>    but if it links libfoo.a, and libfoo.a is not -fPIC, then
>    libbar has to be shipped as a a static library too


This is wrong. If libbar.so needs libfoo.a then libfoo.a does not
need to be PIC:

echo 'int foo(void) {return 17;}' > foo.c
echo 'int bar(void) {return foo();}' > bar.c
echo 'int main() {return bar();}' > main.c
gcc -c -Wall foo.c
ar rs libfoo.a foo.o
gcc -shared -fPIC -Wall bar.c -o bar.so
gcc -Wall main.c -L. -lbar -lfoo
./a.out
echo $?

works just fine.


>  * foomodule is a Python wrapper for libfoo, so it must be shipped
>    as a .so, but if it links libfoo.a, and libfoo.a is not -fPIC,
>    it is not possible to build foomodule at all

That is indeed the case. Note that simply compiling it with -fPIC might
not be enough here. As you need to include the actual library in the
module, this might mean you might end up with multiple copies of the
library, which might also mean multiple copies of any data it has or
with different versions of the same library in the same executeable
(and you cannot really have symbol versioning with a static library).

(If the wrapper is nor for the library itself (but only a library used
by that) this makes it quite dangerous. On the other hand it is a
wrapper for a library depending on an other yet-too-immature-to-have-a-.so
library, so it's quite safe to assume it is so experimental having a
python wrapper makes not much sense anyway.
If the python wrapper is for the library, than an _pic.a only is needed
if that wrapper is not generated by the same source package).

> Unless the circumstances of libfoo make these scenarios unlikely, it
> seems like it is better for other packages to prefer -fPIC even when
> building a static library.
>
> I wonder whether these scenarios were considered when the Policy was
> written.

Static libraries have many serious drawbacks (there are copies in many
programs, so every fix needs a recompile-orgy, multiple copies waste
more RAM, no built-in inter-library-dependencies, ...) and since then
even gotten more (headers if libraries the .a uses (like libc, ...) are
used at compile time, but they are only attached to symbol versions once
the .a in linked into something, so they might not match the headers,
...) since then.

Once something is actually used by other things it is really a good
idea to provide a dynamic library. (And conversely, if something is
not yet able to provide a dynamic library, this is a very good point
to be made against including anything using it).

        Bernhard R. Link
--
F8AC 04D5 0B9B 064B 3383  C3DA AFFC 96D1 151D FFDC


--
To UNSUBSCRIBE, email to [hidden email]
with a subject of "unsubscribe". Trouble? Contact [hidden email]
Archive: https://lists.debian.org/20150222191833.GA4871@...

Reply | Threaded
Open this post in threaded view
|

Re: Should .a library contains non-reallocatable code?

Simon Richter
Hi,

Am 22.02.2015 um 20:18 schrieb Bernhard R. Link:

> echo 'int foo(void) {return 17;}' > foo.c

This code just happens to not generate any data references, so none of
the forbidden reloc types are emitted.

   Simon


--
To UNSUBSCRIBE, email to [hidden email]
with a subject of "unsubscribe". Trouble? Contact [hidden email]
Archive: https://lists.debian.org/54EA39B1.1090906@...

Reply | Threaded
Open this post in threaded view
|

Re: Should .a library contains non-reallocatable code?

Andrey Rahmatullin-3
In reply to this post by Bernhard R. Link-2
On Sun, Feb 22, 2015 at 08:18:33PM +0100, Bernhard R. Link wrote:
> > Here are two scenarios where building a static library (libfoo) with
> > -fPIC is desirable:
> >
> >  * libbar has a stable API, so it should be shipped as a .so,
> >    but if it links libfoo.a, and libfoo.a is not -fPIC, then
              ^^^^^^^^^^^^^^^^^

> >    libbar has to be shipped as a a static library too
>
>
> This is wrong. If libbar.so needs libfoo.a then libfoo.a does not
> need to be PIC:
>
> echo 'int foo(void) {return 17;}' > foo.c
> echo 'int bar(void) {return foo();}' > bar.c
> echo 'int main() {return bar();}' > main.c
> gcc -c -Wall foo.c
> ar rs libfoo.a foo.o
> gcc -shared -fPIC -Wall bar.c -o bar.so
Are you suggesting to make a shared lib with undefined symbols?


--
WBR, wRAR

signature.asc (836 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Should .a library contains non-reallocatable code?

Russ Allbery-2
In reply to this post by Bernhard R. Link-2
"Bernhard R. Link" <[hidden email]> writes:

> This is wrong. If libbar.so needs libfoo.a then libfoo.a does not
> need to be PIC:

> echo 'int foo(void) {return 17;}' > foo.c
> echo 'int bar(void) {return foo();}' > bar.c
> echo 'int main() {return bar();}' > main.c
> gcc -c -Wall foo.c
> ar rs libfoo.a foo.o
> gcc -shared -fPIC -Wall bar.c -o bar.so
> gcc -Wall main.c -L. -lbar -lfoo
> ./a.out
> echo $?

> works just fine.

It won't with something more complex on all architectures.  I think there
are architectures (i386, maybe?) where you can link non-PIC code into a
shared library with a performance penalty, and (as mentioned by another)
it doesn't matter for code where there's no difference between PIC and
non-PIC.  But this will definitely break on some architectures (including
amd64, IIRC).

There's been lots of discussion of this on the Libtool list, and I've had
to deal with this from time to time in various upstream projects where I
wanted to assemble a shared library from various internal helper
libraries.  Take a look at all the work that Libtool does to handle
convenience libraries for exactly this reason.

--
Russ Allbery ([hidden email])               <http://www.eyrie.org/~eagle/>


--
To UNSUBSCRIBE, email to [hidden email]
with a subject of "unsubscribe". Trouble? Contact [hidden email]
Archive: https://lists.debian.org/877fv9r7pr.fsf@...

Reply | Threaded
Open this post in threaded view
|

Re: Should .a library contains non-reallocatable code?

Ian Jackson-2
In reply to this post by Bernhard R. Link-2
Bernhard R. Link writes ("Re: Should .a library contains non-reallocatable code?"):
> Jeff Epler <[hidden email]> [150220 00:19]:
> >  * libbar has a stable API, so it should be shipped as a .so,
> >    but if it links libfoo.a, and libfoo.a is not -fPIC, then
> >    libbar has to be shipped as a a static library too

Jeff is correct.

>
> This is wrong. If libbar.so needs libfoo.a then libfoo.a does not
> need to be PIC:
>
> echo 'int foo(void) {return 17;}' > foo.c
> echo 'int bar(void) {return foo();}' > bar.c
> echo 'int main() {return bar();}' > main.c
> gcc -c -Wall foo.c
> ar rs libfoo.a foo.o
> gcc -shared -fPIC -Wall bar.c -o bar.so
> gcc -Wall main.c -L. -lbar -lfoo
> ./a.out
> echo $?
>
> works just fine.

Did you actually try this before posting ?  I just did and:
 * `gcc -shared -fPIC -Wall bar.c -o bar.so' should read
   `gcc -shared -fPIC -Wall bar.c -o libbar.so' because otherwise
   you get `/usr/bin/ld: cannot find -lbar'
 * You need `LD_LIBRARY_PATH=. ./a.out' because otherwise you get
   `./a.out: error while loading shared libraries: libbar.so: cannot
   open shared object file: No such file or directory'

Furthermore, as Simon Richter points out, this happens to work only
because 1. your foo.c happens not to contain any data references, and
2. your construction of bar.so (should be libbar.so) produces a shared
library with unresolved references to libfoo, which is not usual or
desirable.  (In particular, this would vitiate libbar's stable
API/ABI.)

(64)ian@zealot:~$ echo 'int x=17; int foo(void){ return x; }' >foo.c
(64)ian@zealot:~$ echo 'int bar(void) {return foo();}' > bar.c
(64)ian@zealot:~$ echo 'int main() {return bar();}' > main.c
(64)ian@zealot:~$ gcc -c -Wall foo.c
(64)ian@zealot:~$ ar rs libfoo.a foo.o
(64)ian@zealot:~$ gcc -shared -fPIC -Wall bar.c -L. -lfoo -o libbar.so
bar.c: In function ?bar?:
bar.c:1:1: warning: implicit declaration of function ?foo?
[-Wimplicit-function-declaration]
/usr/bin/ld: ./libfoo.a(foo.o): relocation R_X86_64_PC32 against
symbol `x' can not be used when making a shared object; recompile with
-fPIC
/usr/bin/ld: final link failed: Bad value
collect2: error: ld returned 1 exit status
(64)ian@zealot:~$

> > * foomodule is a Python wrapper for libfoo, so it must be shipped
> >    as a .so, but if it links libfoo.a, and libfoo.a is not -fPIC,
> >    it is not possible to build foomodule at all
>
> That is indeed the case. Note that simply compiling it with -fPIC might
> not be enough here. As you need to include the actual library in the
> module, this might mean you might end up with multiple copies of the
> library, which might also mean multiple copies of any data it has or
> with different versions of the same library in the same executeable
> (and you cannot really have symbol versioning with a static library).

That not usually a problem.  Providing that only the relevant symbols
are exported from the .so, the executable simply results in multiple
completely independent copies of the static library.

See transcript below.  In my example foo[12].[ch] represents a library
with an unstable API/ABI, provided as a static library only.  Note
that it has symbols `x' and `foo' which behave differently in the
different versions.

bar[12].[ch] represent two different libraries, both using foo, but
each of whose source code is adapted for a different version of foo.

main.c is a program using both bar1 and bar2.  You can see that both
foo1 and foo2 are included and they have different x's and that the
whole program works despite the different signatures for the function
foo.

Ian.

(64)ian@zealot:~$ ./build
+ egrep . bar.c bar1.c bar1.h bar2.c bar2.h foo.c foo1.c foo1.h foo2.c foo2.h main.c t.c build
bar.c:int bar(void) {return foo();}
bar1.c:#include "foo1.h"
bar1.c:#include "bar1.h"
bar1.c:int bar1(void) {
bar1.c:  double v;
bar1.c:  foo(&v);
bar1.c:  return v;
bar1.c:}
bar1.h:int bar1(void);
bar2.c:#include "foo2.h"
bar2.c:#include "bar2.h"
bar2.c:int bar2(void) {
bar2.c:  int v;
bar2.c:  foo(&v);
bar2.c:  return v;
bar2.c:}
bar2.h:int bar2(void);
foo.c:int x=17; int foo(void){ return x; }
foo1.c:#include <stdio.h>
foo1.c:#include "foo1.h"
foo1.c:int x=17;
foo1.c:void foo(double *r){ printf("foo1: %p\n", &x); *r = x; }
foo1.h:void foo(double *r);
foo2.c:#include <stdio.h>
foo2.c:#include "foo2.h"
foo2.c:double x=17;
foo2.c:void foo(int *r){ printf("foo2: %p\n", &x); *r = x; }
foo2.h:void foo(int *r);
main.c:#include <stdio.h>
main.c:#include "bar1.h"
main.c:#include "bar2.h"
main.c:int main() {
main.c:  int r1 = bar1();
main.c:  int r2 = bar2();
main.c:  printf("%d %d\n", r1, r2);
main.c:  return 0;
main.c:}
t.c:#include <stdio.h>
t.c:int main(int argc, char **argv) {
t.c:  printf("%s\n",strsignal(atoi(argv[1])));
t.c:}
build:#!/bin/sh
build:set -ex
build:egrep . *.[ch] build
build:gcc='gcc -Wall -Werror -Wmissing-prototypes -Wmissing-declarations'
build:for v in 1 2; do
build:  $gcc -c -fPIC -Wall foo$v.c
build:  ar rs libfoo$v.a foo$v.o
build:  $gcc -shared -fPIC -Wl,-Bsymbolic bar$v.c -L. -lfoo$v -o libbar$v.so
build:done
build:gcc -Wall main.c -L. -lbar1 -lbar2
build:LD_LIBRARY_PATH=. ./a.out
+ gcc=gcc -Wall -Werror -Wmissing-prototypes -Wmissing-declarations
+ gcc -Wall -Werror -Wmissing-prototypes -Wmissing-declarations -c -fPIC -Wall foo1.c
+ ar rs libfoo1.a foo1.o
+ gcc -Wall -Werror -Wmissing-prototypes -Wmissing-declarations -shared -fPIC -Wl,-Bsymbolic bar1.c -L. -lfoo1 -o libbar1.so
+ gcc -Wall -Werror -Wmissing-prototypes -Wmissing-declarations -c -fPIC -Wall foo2.c
+ ar rs libfoo2.a foo2.o
+ gcc -Wall -Werror -Wmissing-prototypes -Wmissing-declarations -shared -fPIC -Wl,-Bsymbolic bar2.c -L. -lfoo2 -o libbar2.so
+ gcc -Wall main.c -L. -lbar1 -lbar2
+ LD_LIBRARY_PATH=. ./a.out
foo1: 0x7ff823671a98
foo2: 0x7ff823470a98
17 17
(64)ian@zealot:~$


--
To UNSUBSCRIBE, email to [hidden email]
with a subject of "unsubscribe". Trouble? Contact [hidden email]
Archive: https://lists.debian.org/21739.31478.387870.250750@...

Reply | Threaded
Open this post in threaded view
|

Re: Should .a library contains non-reallocatable code?

Ian Jackson-2
Ian Jackson writes ("Re: Should .a library contains non-reallocatable code?"):
> Jeff is correct.
...
> That not usually a problem.  Providing that only the relevant symbols
> are exported from the .so, the executable simply results in multiple
> completely independent copies of the static library.

I should say that I agree with the conclusions of others in this
thread, that policy's rules about -fPIC for static libraries are
wrong.

Where only a static library is provided, it should be built _with_
-fPIC unless it is expected never to be included in any shared object
(which is probably hard to predict, but I guess there might be cases
where the maintainer might know).

Where a .so is provided too then the static library is normally used
only in cases where no dynamic linking is done at all, and then
-fPIC is probably undesirable.  This is of course the usual case.

Ian.


--
To UNSUBSCRIBE, email to [hidden email]
with a subject of "unsubscribe". Trouble? Contact [hidden email]
Archive: https://lists.debian.org/21739.31942.771830.623505@...

Reply | Threaded
Open this post in threaded view
|

Re: Should .a library contains non-reallocatable code?

Bernhard R. Link-2
In reply to this post by Ian Jackson-2
* Ian Jackson <[hidden email]> [150223 20:09]:

> Bernhard R. Link writes ("Re: Should .a library contains non-reallocatable code?"):
> > Jeff Epler <[hidden email]> [150220 00:19]:
> > >  * libbar has a stable API, so it should be shipped as a .so,
> > >    but if it links libfoo.a, and libfoo.a is not -fPIC, then
> > >    libbar has to be shipped as a a static library too
>
> Jeff is correct.
> >
> > This is wrong. If libbar.so needs libfoo.a then libfoo.a does not
> > need to be PIC:
> >
> > echo 'int foo(void) {return 17;}' > foo.c
> > echo 'int bar(void) {return foo();}' > bar.c
> > echo 'int main() {return bar();}' > main.c
> > gcc -c -Wall foo.c
> > ar rs libfoo.a foo.o
> > gcc -shared -fPIC -Wall bar.c -o bar.so
> > gcc -Wall main.c -L. -lbar -lfoo
> > ./a.out
> > echo $?
> >
> > works just fine.
>
> Did you actually try this before posting ?

Oh, I copied the wrong lines from the backlog here. It seems they
were a bit to trivial so autocorrection made me miss them reading over
them.

> Furthermore, as Simon Richter points out, this happens to work only
> because 1. your foo.c happens not to contain any data references, and

Data references work fine, too:

echo 'extern int val; void foo(int s) {val=s;}' >
echo 'int val = 2; void foo(int s) {val=s;}' > foo.c
echo 'extern int val;int bar(void) {return val;}' >
gcc -c -Wall foo.c
ar rs libfoo.a foo.o
gcc -shared -fPIC -Wall bar.c -o libbar.so
echo 'int main() {foo(17);return bar();}' > main.c
gcc -Wall main.c -L. -lbar -lfoo
LD_LIBRARY_PATH=. ./a.out
echo $?

> 2. your construction of bar.so (should be libbar.so) produces a shared
> library with unresolved references to libfoo, which is not usual or
> desirable.

Using a static library is undesirable. Unresolved symbols is just the
way dependencies of libraries works. The only thing missing here is
the missing dependency on the static library. But that is simply not
possible with static libraries.

> (In particular, this would vitiate libbar's stable
> API/ABI.)

This you have to explain.

If you think there is another way a dynamic library can use a static
library please tell so.

> (64)ian@zealot:~$ echo 'int x=17; int foo(void){ return x; }' >foo.c
> (64)ian@zealot:~$ echo 'int bar(void) {return foo();}' > bar.c
> (64)ian@zealot:~$ echo 'int main() {return bar();}' > main.c
> (64)ian@zealot:~$ gcc -c -Wall foo.c
> (64)ian@zealot:~$ ar rs libfoo.a foo.o
> (64)ian@zealot:~$ gcc -shared -fPIC -Wall bar.c -L. -lfoo -o libbar.so
> bar.c: In function ?bar?:
> bar.c:1:1: warning: implicit declaration of function ?foo?
> [-Wimplicit-function-declaration]
> /usr/bin/ld: ./libfoo.a(foo.o): relocation R_X86_64_PC32 against
> symbol `x' can not be used when making a shared object; recompile with
> -fPIC
> /usr/bin/ld: final link failed: Bad value
> collect2: error: ld returned 1 exit status

What you do here is creating a dynamic library that merges the contents
of foo and bar. This is totally broken. If anything else uses the same
static library you get multiple symbols. If there versions differ that
means harvoc. With data parts you may end up with multiple versions.

There are only two possibilities:

You want a static library, then the static library has to be linked into
the executeable directly.

You foo within a dynamic library. Then instead of creating a dynamic
library that includes foo you can just as well create dynamic libfoo.so
and link against that.

> > > * foomodule is a Python wrapper for libfoo, so it must be shipped
> > >    as a .so, but if it links libfoo.a, and libfoo.a is not -fPIC,
> > >    it is not possible to build foomodule at all
> >
> > That is indeed the case. Note that simply compiling it with -fPIC might
> > not be enough here. As you need to include the actual library in the
> > module, this might mean you might end up with multiple copies of the
> > library, which might also mean multiple copies of any data it has or
> > with different versions of the same library in the same executeable
> > (and you cannot really have symbol versioning with a static library).
>
> That not usually a problem.  Providing that only the relevant symbols
> are exported from the .so, the executable simply results in multiple
> completely independent copies of the static library.

Your example above misses that.

echo "int foo(void) {return 7;}" > foo1.c
echo "int foo(void) {return 13;}" > foo2.c
echo 'int bar(void) {return foo();}' > bar.c
echo 'int main() {return bar();}' > main.c
gcc -c foo1.c
gcc -c foo2.c
gcc -fPIC -c foo1.c
rm libfoo.a
ar rs libfoo.a foo1.o
gcc -shared -o libbar.so -fPIC bar.c -L. -lfoo
gcc foo2.c main.c -L. -lbar
LD_LIBRARY_PATH=. ./a.out
echo $?

gives 13

Also note that you cannot avoid exporting symbols totally.
The above example you can avoid with a linker script, like:

printf 'LIBBAR_1.0 {\n global:\n  bar*;\n local:\n  *;\n};\n' >libbar.map
and then give -Wl,--version-script=libbar.map


>
> See transcript below.  In my example foo[12].[ch] represents a library
> with an unstable API/ABI, provided as a static library only.  Note
> that it has symbols `x' and `foo' which behave differently in the
> different versions.

Note that this uses -Bsymbolic, by which you hide some issues and might
introduce other problems (depending on your library).

You still pollute the ABI with the details of the internals:

if you try to change main.c to:
#include <stdio.h>
#include "foo1.h"
#include "bar2.h"
int main() {
  double rr1;
  foo(&rr1);
  int r1 = rr1;
  int r2 = bar2();
  printf("%d %d\n", r1, r2);
  return 0;
}

and run gcc -Wall main.c -L. -lbar2 -lfoo1 ; LD_LIBRARY_PATH=. ./a.out
you get:
foo2: 0x7f42db6f0a48
foo2: 0x7f42db6f0a48
0 17

(In this specific case a version script might helps, but I'd not bet
on helping in all cases).


        Bernhard R. Link
--
F8AC 04D5 0B9B 064B 3383  C3DA AFFC 96D1 151D FFDC


--
To UNSUBSCRIBE, email to [hidden email]
with a subject of "unsubscribe". Trouble? Contact [hidden email]
Archive: https://lists.debian.org/20150224014028.GA2352@...

Reply | Threaded
Open this post in threaded view
|

Re: Should .a library contains non-reallocatable code?

Bernhard R. Link-2
In reply to this post by Simon Richter
* Simon Richter <[hidden email]> [150222 21:19]:
> Am 22.02.2015 um 20:18 schrieb Bernhard R. Link:
>
> > echo 'int foo(void) {return 17;}' > foo.c
>
> This code just happens to not generate any data references, so none of
> the forbidden reloc types are emitted.

You can add references here. As I do not merge it into a dynamic object,
it does not make any difference, though.

(Also "data references" is a bit misleading, as data references and code references
do not really make a difference.

        Bernhard R. Link
--
F8AC 04D5 0B9B 064B 3383  C3DA AFFC 96D1 151D FFDC


--
To UNSUBSCRIBE, email to [hidden email]
with a subject of "unsubscribe". Trouble? Contact [hidden email]
Archive: https://lists.debian.org/20150224014724.GA2958@...

Reply | Threaded
Open this post in threaded view
|

Re: Should .a library contains non-reallocatable code?

Bernhard R. Link-2
In reply to this post by Russ Allbery-2
* Russ Allbery <[hidden email]> [150222 21:51]:

> "Bernhard R. Link" <[hidden email]> writes:
>
> > This is wrong. If libbar.so needs libfoo.a then libfoo.a does not
> > need to be PIC:
>
> > echo 'int foo(void) {return 17;}' > foo.c
> > echo 'int bar(void) {return foo();}' > bar.c
> > echo 'int main() {return bar();}' > main.c
> > gcc -c -Wall foo.c
> > ar rs libfoo.a foo.o
> > gcc -shared -fPIC -Wall bar.c -o bar.so
> > gcc -Wall main.c -L. -lbar -lfoo
> > ./a.out
> > echo $?
>
> > works just fine.
>
> It won't with something more complex on all architectures.  I think there
> are architectures (i386, maybe?) where you can link non-PIC code into a
> shared library with a performance penalty, and (as mentioned by another)
> it doesn't matter for code where there's no difference between PIC and
> non-PIC.  But this will definitely break on some architectures (including
> amd64, IIRC).
>
> There's been lots of discussion of this on the Libtool list, and I've had
> to deal with this from time to time in various upstream projects where I
> wanted to assemble a shared library from various internal helper
> libraries.  Take a look at all the work that Libtool does to handle
> convenience libraries for exactly this reason.

You are speaking about linking .a helper libraries into a shared object.
I'm about the case that a shared library has a dependency on a different
static library instead.

        Bernhard R. Link
--
F8AC 04D5 0B9B 064B 3383  C3DA AFFC 96D1 151D FFDC


--
To UNSUBSCRIBE, email to [hidden email]
with a subject of "unsubscribe". Trouble? Contact [hidden email]
Archive: https://lists.debian.org/20150224015905.GB2958@...

Reply | Threaded
Open this post in threaded view
|

Re: Should .a library contains non-reallocatable code?

Russ Allbery-2
"Bernhard R. Link" <[hidden email]> writes:
> * Russ Allbery <[hidden email]> [150222 21:51]:

>> It won't with something more complex on all architectures.  I think
>> there are architectures (i386, maybe?) where you can link non-PIC code
>> into a shared library with a performance penalty, and (as mentioned by
>> another) it doesn't matter for code where there's no difference between
>> PIC and non-PIC.  But this will definitely break on some architectures
>> (including amd64, IIRC).

>> There's been lots of discussion of this on the Libtool list, and I've
>> had to deal with this from time to time in various upstream projects
>> where I wanted to assemble a shared library from various internal
>> helper libraries.  Take a look at all the work that Libtool does to
>> handle convenience libraries for exactly this reason.

> You are speaking about linking .a helper libraries into a shared object.
> I'm about the case that a shared library has a dependency on a different
> static library instead.

Oh, you're talking about leaving symbols unresolved in the shared library
and requiring that the binary link with both the shared library and some
static library at build time?

--
Russ Allbery ([hidden email])               <http://www.eyrie.org/~eagle/>


--
To UNSUBSCRIBE, email to [hidden email]
with a subject of "unsubscribe". Trouble? Contact [hidden email]
Archive: https://lists.debian.org/87d250hxuc.fsf@...

Reply | Threaded
Open this post in threaded view
|

Re: Should .a library contains non-reallocatable code?

Jeff Epler
In reply to this post by Jeff Epler

On Thu, Feb 19, 2015 at 05:19:30PM -0600, Jeff Epler wrote:
>  * foomodule is a Python wrapper for libfoo, so it must be shipped
>    as a .so, but if it links libfoo.a, and libfoo.a is not -fPIC,
>    it is not possible to build foomodule at all
>
>    (The same goes for wrapping the library for most other interpreted
>    languages)

So here is a concrete example of this.  I chose libtomcrypt at
semi-random, because in libtomcrypt-dev 1.17-6 there is both a shared
library and a static library; the latter has non-PIC code.

Begin with a Python module wrapping a single aspect of libtomcrypt:
generating a random byte string using the Fortuna algorithm:

    #include <Python.h>
    #include <tomcrypt.h>

    static PyObject *tom_random(PyObject *noself, PyObject *args) {
        Py_ssize_t sz;

        if(!PyArg_ParseTuple(args, "n:tom.random", &sz)) return NULL;
        PyObject *bytearray = PyByteArray_FromStringAndSize("", 0);
        if(!sz || !bytearray) { return bytearray; }

        prng_state prng;
        int err;
        if((err = rng_make_prng(128, find_prng("fortuna"), &prng, NULL)) != CRYPT_OK)
            return PyErr_Format(PyExc_RuntimeError,
                    "rng_make_rng() returned %d", err);

        if(PyByteArray_Resize(bytearray, sz) < 0) {
            Py_DECREF(bytearray);
            bytearray = NULL;
            goto done;
        }

        if((err = fortuna_read(PyByteArray_AsString(bytearray), sz, &prng))
                != sz) {
            Py_DECREF(bytearray);
            bytearray = NULL;
            return PyErr_Format(PyExc_RuntimeError,
                    "fortuna_read() only read %zd bytes", (size_t)sz);
        }

    done:
        fortuna_done(&prng);
        return bytearray;
    }

    static PyMethodDef meth[] = {
        {"random", (PyCFunction)tom_random, METH_VARARGS,
            "random(n): get N random bytes from a fortuna generator"},
        {}
    };

    void inittom() {
        int err;
        if((err = register_prng(&fortuna_desc)) == -1) {
            PyErr_Format(PyExc_RuntimeError,
                    "register_prng() returned %d", err);
            return;
        }

        PyObject *m = PyImport_AddModule("tom");
        Py_InitModule("tom", meth);
    }

Stir in an 'version' file that exports just the required symbol, inittom:

    {
    global: inittom;
    local: *;
    };

create a Python 2.7 extension module as follows:
    $ gcc -I /usr/include/python2.7 -fPIC -c -o tommodule.o tommodule.c
    $ gcc -shared -o tommodule.so tommodule.o -ltomcrypt -lpython2.7 \
        -Wl,--version-script=version -Wl,--no-allow-shlib-undefined

this .so file has just one exported symbol, thanks to the export file:
    $ nm -D --defined-only tommodule.so
        0000000000000d0e T inittom
and happily, it works:
    $ python -c 'import tom; print bytes(tom.random(16)).encode("hex")'
    412d02b965f8c1f34c6bbaf2d1beb001

However, if libtomcrypt.a is used instead, the linker errors:
    $ gcc -shared -o tommodule.so tommodule.o \
        -Wl,-Bstatic -ltomcrypt -Wl,-Bdynamic -lpython2.7 \
        -Wl,--version-script=version -Wl,--no-allow-shlib-undefined
    /usr/bin/ld.bfd.real: /usr/lib/x86_64-linux-gnu/libtomcrypt.a(crypt_find_prng.o):
        relocation R_X86_64_32 against `prng_descriptor' can not be used when making a shared object; recompile with -fPIC
    /usr/lib/x86_64-linux-gnu/libtomcrypt.a: error adding symbols: Bad value
    collect2: error: ld returned 1 exit status

If libtomcrypt.a *had* been built with -fPIC, then this would have
worked.  Using the version file prevents *ANY* identifier from
libtomcrypt being a part of the public API of the Python module.  From
the point of view of the user of the Python tom module, it's a
moot point whether libtomcrypt's C API is stable.

In fact, I went ahead and rebuilt it this way, with just a patch to
debian/rules:
    -export CFLAGS += -DGMP_DESC -DLTM_DESC -DUSE_LTM
    +export CFLAGS += -DGMP_DESC -DLTM_DESC -DUSE_LTM -fPIC
with this version of the package installed, the linker line which
referred to the .a file worked to produce a Python shared library.

All tests done on a Jessie x86-64 system.  I manually trancsribed some
of the commands above, so there's every chance I made an error in
transcrption.

Jeff


--
To UNSUBSCRIBE, email to [hidden email]
with a subject of "unsubscribe". Trouble? Contact [hidden email]
Archive: https://lists.debian.org/20150224033004.GF88168@...

Reply | Threaded
Open this post in threaded view
|

Re: Should .a library contains non-reallocatable code?

Alastair McKinstry-9
In reply to this post by Ian Jackson-2

On 23/02/2015 19:17, Ian Jackson wrote:

> Ian Jackson writes ("Re: Should .a library contains non-reallocatable code?"):
>> Jeff is correct.
> ...
>> That not usually a problem.  Providing that only the relevant symbols
>> are exported from the .so, the executable simply results in multiple
>> completely independent copies of the static library.
> I should say that I agree with the conclusions of others in this
> thread, that policy's rules about -fPIC for static libraries are
> wrong.
>
> Where only a static library is provided, it should be built _with_
> -fPIC unless it is expected never to be included in any shared object
> (which is probably hard to predict, but I guess there might be cases
> where the maintainer might know).
>
> Where a .so is provided too then the static library is normally used
> only in cases where no dynamic linking is done at all, and then
> -fPIC is probably undesirable.  This is of course the usual case.
>
> Ian.
>
>
I agree with this; are there any cases where only a static library _is_
provided, and if so why? why not provide a .so?

The original examples are only problematic because they are linked
to a static .a rather than .so.
In my packages I always provide a -fPIC .so and non-fPIC .a library,
the latter for performance where it matters (these days typically slight).

regards
Alastair

--
Alastair McKinstry, <[hidden email]>, <[hidden email]>, https://diaspora.sceal.ie/u/amckinstry
Misentropy: doubting that the Universe is becoming more disordered.


--
To UNSUBSCRIBE, email to [hidden email]
with a subject of "unsubscribe". Trouble? Contact [hidden email]
Archive: https://lists.debian.org/54EC4BDD.1060606@...

12