Bug#397939: Lintian: outdated-autotools-helper-file

classic Classic list List threaded Threaded
43 messages Options
123
Reply | Threaded
Open this post in threaded view
|

Bug#397939: Lintian: outdated-autotools-helper-file

Bas Wijnen
On Sun, Feb 10, 2008 at 03:48:20PM -0800, Russ Allbery wrote:

> Raphael Geissert <[hidden email]> writes:
>
> > Quoting the Debian Policy, section 4.9 Main building script:
> > debian/rules[1]
> >
> >> clean
> >>
> >> This must undo any effects that the build and binary targets may
> >> have had, except that it should leave alone any output files
> >> created in the parent directory by a run of a binary target.  
> >
> > So according to the policy the clean target must put the original
> > files back on place.
>
> This Policy dictate as written is pretty widely ignored and (IMO) is
> somewhat hard to defend in several of its implications.  We haven't
> figured out what to say instead, but deleting the files is fairly
> common right now.
>
> See http://bugs.debian.org/397939 for some additional discussion.
Thank you for this link.  I would like to add a suggestion as a
solution.  IMO the important thing is, as stated by Bill, that "clean"
and "clean; binary; clean" should result in the same tree.  From the log
it seems that people agree that this implies either creating a huge
diff.gz, or running autotools in the clean target.

This is not true if you simply build the whole package from source.
That is, run autotools during build, remove all generated files,
including Makefile.in, configure, etc, in the clean target.

For some reason many people seem to think that the whole package must be
built from source, except for configure and Makefile.in.  I still don't
understand what the idea behind this exception is.  Especially when it
leads to unwanted concequences (unreadable diff.gzs, for example), I
don't think it is a good idea to hold on to the idea that running
autotools is not part of building the package.

Apart from that, this discussion is about users making changes to the
source and compiling a package with those changes in it.  When not
running autotools during build, that doesn't work if the user changes
Makefile.am or configure.ac.  Yet another undesirable effect of this
strange exception to "build everything from source".

I suggest to mandate "remove all generated files in the clean target"
(formulated in a way which includes "generated by upstream", not only
"generated by the build target), which implies "rebuild everything in
the build target".

With the current wording it is allowed to use shipped built files from
the upstream tarball, as long as the source is present.  It is even
allowed to ship the build results (uuencoded, for example) in the
diff.gz and use those.  I suppose we all agree that this is unacceptable
for normal build results.

Now reread the previous paragraph while thinking of Makefile.in instead
of "normal build results".  It is common practice to do exactly that:
ship and use pre-built versions, or even ship them in the diff.gz (which
then gets huge).  Makefile.am is only present as source-file, but it
isn't used at all during the build.  Any changes the user makes will not
be noticed by the build system.

I'd like to hear why this exception is so important for people.  Or
better yet, I'd like to hear that everyone agrees with me, so we can
make this change and finally to the Right Thing. :-)

Thanks,
Bas

--
I encourage people to send encrypted e-mail (see http://www.gnupg.org).
If you have problems reading my e-mail, use a better reader.
Please send the central message of e-mails as plain text
   in the message body, not as HTML and definitely not as MS Word.
Please do not use the MS Word format for attachments either.
For more information, see http://pcbcn10.phys.rug.nl/e-mail.html

signature.asc (196 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Bug#397939: Lintian: outdated-autotools-helper-file

Kapil Hari Paranjape
On Mon, 11 Feb 2008, Bas Wijnen wrote:

> I suggest to mandate "remove all generated files in the clean target"
> (formulated in a way which includes "generated by upstream", not only
> "generated by the build target), which implies "rebuild everything in
> the build target".
>
> With the current wording it is allowed to use shipped built files from
> the upstream tarball, as long as the source is present.  It is even
> allowed to ship the build results (uuencoded, for example) in the
> diff.gz and use those.  I suppose we all agree that this is unacceptable
> for normal build results.
>
> Now reread the previous paragraph while thinking of Makefile.in instead
> of "normal build results".  It is common practice to do exactly that:
> ship and use pre-built versions, or even ship them in the diff.gz (which
> then gets huge).  Makefile.am is only present as source-file, but it
> isn't used at all during the build.  Any changes the user makes will not
> be noticed by the build system.
>
> I'd like to hear why this exception is so important for people.  Or
> better yet, I'd like to hear that everyone agrees with me, so we can
> make this change and finally to the Right Thing. :-)
It is a good idea if one can use .orig.tar.gz to be the same as the
upstream .tar.gz whenever it is DFSG-free to do so.

One reason is that it becomes easier to verify that the .orig.tar.gz
is the same --- has the same GPG signature, MD5SUM etc.

Another reason (which is the same reason in disguise!) is that it is
easier to see exactly what Debian is changing in the upstream source.

Note that if the upstream's auto-generated files are deleted during
the clean target, then the source *must* be re-packaged to avoid
needless clutter in the .diff.gz which is of a "negative" nature.

So all such source must come with a "get-orig-source" target and a
README.Debian-source which explains the changes made.

Regards,

Kapil.
--

signature.asc (196 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Bug#397939: Lintian: outdated-autotools-helper-file

Russ Allbery-2
In reply to this post by Bas Wijnen
Bas Wijnen <[hidden email]> writes:

> I suggest to mandate "remove all generated files in the clean target"
> (formulated in a way which includes "generated by upstream", not only
> "generated by the build target), which implies "rebuild everything in
> the build target".

[...]

> I'd like to hear why this exception is so important for people.  Or
> better yet, I'd like to hear that everyone agrees with me, so we can
> make this change and finally to the Right Thing. :-)

It may be less common now, but for many years it was quite common for
upstreams to use very specific versions of autotools, which means that if
Debian had dropped the specific autoconf in question, it wasn't easy to
regenerate the files.  Now, that may be a problem for other reasons since
as you point out it means we don't really have horribly usable source, but
that's the most common practical concern, I think.

Always re-running autoconf and automake would increase the number of
FTBFS's that we'd need to fix.  (Probably for the greater good of free
software, but.)

Also, it's not always easy to figure out which files are generated in
order to remove them, but that's probably programmatically fixable.

--
Russ Allbery ([hidden email])               <http://www.eyrie.org/~eagle/>



--
To UNSUBSCRIBE, email to [hidden email]
with a subject of "unsubscribe". Trouble? Contact [hidden email]

Reply | Threaded
Open this post in threaded view
|

Bug#397939: Lintian: outdated-autotools-helper-file

Henrique de Moraes Holschuh
In reply to this post by Kapil Hari Paranjape
On Mon, 11 Feb 2008, Kapil Hari Paranjape wrote:
> Note that if the upstream's auto-generated files are deleted during
> the clean target, then the source *must* be re-packaged to avoid
> needless clutter in the .diff.gz which is of a "negative" nature.

Not so.  Deletions are ignored.  Ever tried it?

--
  "One disk to rule them all, One disk to find them. One disk to bring
  them all and in the darkness grind them. In the Land of Redmond
  where the shadows lie." -- The Silicon Valley Tarot
  Henrique Holschuh



--
To UNSUBSCRIBE, email to [hidden email]
with a subject of "unsubscribe". Trouble? Contact [hidden email]

Reply | Threaded
Open this post in threaded view
|

Bug#397939: Lintian: outdated-autotools-helper-file

Dr. Bas Wijnen-2
In reply to this post by Russ Allbery-2
On Mon, Feb 11, 2008 at 09:21:29AM -0800, Russ Allbery wrote:

> Bas Wijnen <[hidden email]> writes:
>
> > I suggest to mandate "remove all generated files in the clean target"
> > (formulated in a way which includes "generated by upstream", not only
> > "generated by the build target), which implies "rebuild everything in
> > the build target".
>
> [...]
>
> > I'd like to hear why this exception is so important for people.  Or
> > better yet, I'd like to hear that everyone agrees with me, so we can
> > make this change and finally to the Right Thing. :-)
>
> It may be less common now, but for many years it was quite common for
> upstreams to use very specific versions of autotools, which means that if
> Debian had dropped the specific autoconf in question, it wasn't easy to
> regenerate the files.  Now, that may be a problem for other reasons since
> as you point out it means we don't really have horribly usable source, but
> that's the most common practical concern, I think.
Oh, I didn't know that.  IMO that would mean the package should really
be in contrib, because we're missing a build dependency in main then
(not used, but AFAIK using pre-built files is not an acceptable way to
avoid getting in contrib in such a situation).

Anyway, I don't think that is still a reason for not running autotools
nowadays, so it's just historical background. :-)

> Always re-running autoconf and automake would increase the number of
> FTBFS's that we'd need to fix.

Not really.  It would lead to many bugs that packages aren't following
policy.  Especially if we start with should and later (possibly) upgrade
it to must, I think it is workable.  In particular because these bugs
don't actually stop a package from being built.  I would be very happy
with consensus that the autotools _should_ be run during build.  The
implementation of actually doing it in all packages may take a while, I
don't have a problem with that.

Unless of course you are talking about cdbs.  When that changes to
running autotools, it likely needs to know if there are extra arguments.
That may indeed result in FTBFSs for many packages which use it (because
they may need, but don't provide, the arguments).  But it's good when
they're fixed as well. :-)

> (Probably for the greater good of free software, but.)

Yes, IMO it's one of those situations where Debian should do what's
Right, not what's Easy (similar to what I wrote about the /bin/sh
bash->dash move on -policy today).

> Also, it's not always easy to figure out which files are generated in
> order to remove them, but that's probably programmatically fixable.

That's in principle a problem for the maintainer to solve, and
secundarily for lintian.  If things can be automated, that may be nice,
but it is not required.  Once policy mandates to remove generated files
in the clean target, it's up to the maintainer to do it.  And if the
maintainer makes a mistake and forgets to remove a file, that's not a
big problem (it would be a bug with this change in policy, but only a
minor one, IMO).

At this moment however, I don't think there is consensus yet that it is
always better to remove all generated files, including
autotools-generated stuff, in the clean target.  After that happens, we
can think about how to implement it in the archive.  Slowly, I would
suggest. :-)  Because it is a good thing if we do this Right, but we
shouldn't break half the archive for it. ;-)

Thanks,
Bas

--
I encourage people to send encrypted e-mail (see http://www.gnupg.org).
If you have problems reading my e-mail, use a better reader.
Please send the central message of e-mails as plain text
   in the message body, not as HTML and definitely not as MS Word.
Please do not use the MS Word format for attachments either.
For more information, see http://pcbcn10.phys.rug.nl/e-mail.html

signature.asc (196 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Bug#397939: Lintian: outdated-autotools-helper-file

Russ Allbery-2
Bas Wijnen <[hidden email]> writes:
> On Mon, Feb 11, 2008 at 09:21:29AM -0800, Russ Allbery wrote:

>> Always re-running autoconf and automake would increase the number of
>> FTBFS's that we'd need to fix.

> Not really.

No, really, I promise it will.  :)  Each time we upgrade autoconf, it will
break a bunch of packages that were doing things that weren't supported.

Now, those really were bugs and should be fixed.  But turning them into
FTBFS bugs does escalate the severity quite a bit.

> It would lead to many bugs that packages aren't following policy.
> Especially if we start with should and later (possibly) upgrade it to
> must, I think it is workable.  In particular because these bugs don't
> actually stop a package from being built.  I would be very happy with
> consensus that the autotools _should_ be run during build.  The
> implementation of actually doing it in all packages may take a while, I
> don't have a problem with that.

Well, I don't do it for mine right now because it takes a long time and
feels kind of pointless, but I'm happy to go along with any consensus.
But that part isn't so much the concern.

> Yes, IMO it's one of those situations where Debian should do what's
> Right, not what's Easy (similar to what I wrote about the /bin/sh
> bash->dash move on -policy today).

I am generally in favor of that, but I also don't have the free time to
volunteer for the release team, who ends up bearing the brunt of us doing
the right thing in this area, so, y'know, easy for me to say.  :)

> That's in principle a problem for the maintainer to solve, and
> secundarily for lintian.

Well, yeah, but it would still be nice to provide some help.

> At this moment however, I don't think there is consensus yet that it is
> always better to remove all generated files, including
> autotools-generated stuff, in the clean target.  After that happens, we
> can think about how to implement it in the archive.  Slowly, I would
> suggest. :-) Because it is a good thing if we do this Right, but we
> shouldn't break half the archive for it. ;-)

Yup.  :)

--
Russ Allbery ([hidden email])               <http://www.eyrie.org/~eagle/>



--
To UNSUBSCRIBE, email to [hidden email]
with a subject of "unsubscribe". Trouble? Contact [hidden email]

Reply | Threaded
Open this post in threaded view
|

Bug#397939: Lintian: outdated-autotools-helper-file

Dr. Bas Wijnen-2
On Mon, Feb 11, 2008 at 03:19:36PM -0800, Russ Allbery wrote:

> Bas Wijnen <[hidden email]> writes:
> > On Mon, Feb 11, 2008 at 09:21:29AM -0800, Russ Allbery wrote:
>
> >> Always re-running autoconf and automake would increase the number of
> >> FTBFS's that we'd need to fix.
>
> > Not really.
>
> No, really, I promise it will.  :)  Each time we upgrade autoconf, it will
> break a bunch of packages that were doing things that weren't supported.
Ah, that is a point.  But that should not be the case if the packages
build-depend on the right version, or is it still a problem?  I know
automake in particular changes a lot between versions, but that's why I
always build-depend on a version (such as "automake1.10").  That's
recommended as well AFAIK.  So the problem of removing old versions of
automake (such as automake1.8 at the moment) gets bigger, but it
shouldn't make it FTBFS bugs most of the time.

> > Yes, IMO it's one of those situations where Debian should do what's
> > Right, not what's Easy (similar to what I wrote about the /bin/sh
> > bash->dash move on -policy today).
>
> I am generally in favor of that, but I also don't have the free time to
> volunteer for the release team, who ends up bearing the brunt of us doing
> the right thing in this area, so, y'know, easy for me to say.  :)

I'm happy to report bugs and provide patches for many packages if that
helps doing this properly.

> > That's in principle a problem for the maintainer to solve, and
> > secundarily for lintian.
>
> Well, yeah, but it would still be nice to provide some help.

Sure. :-)

Thanks,
Bas

--
I encourage people to send encrypted e-mail (see http://www.gnupg.org).
If you have problems reading my e-mail, use a better reader.
Please send the central message of e-mails as plain text
   in the message body, not as HTML and definitely not as MS Word.
Please do not use the MS Word format for attachments either.
For more information, see http://pcbcn10.phys.rug.nl/e-mail.html

signature.asc (196 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Bug#397939: Lintian: outdated-autotools-helper-file

Kurt Roeckx
In reply to this post by Russ Allbery-2
On Mon, Feb 11, 2008 at 03:19:36PM -0800, Russ Allbery wrote:

> Bas Wijnen <[hidden email]> writes:
> > On Mon, Feb 11, 2008 at 09:21:29AM -0800, Russ Allbery wrote:
>
> >> Always re-running autoconf and automake would increase the number of
> >> FTBFS's that we'd need to fix.
>
> > Not really.
>
> No, really, I promise it will.  :)  Each time we upgrade autoconf, it will
> break a bunch of packages that were doing things that weren't supported.

From a QA point of view it would be nice that we can actually test the
archive with a new version.  We could warn in advance which packages
have a problem.  We can track which still have the problem, and so on.

This would of course assume that it either gets uploaded to
experimental, or that it's not the default version in unstable.


Kurt




--
To UNSUBSCRIBE, email to [hidden email]
with a subject of "unsubscribe". Trouble? Contact [hidden email]

Reply | Threaded
Open this post in threaded view
|

Bug#397939: Consensus about what a proper clean target is?

Dr. Bas Wijnen-2
In reply to this post by Dr. Bas Wijnen-2
On Tue, Feb 12, 2008 at 12:16:17AM +0100, Bas Wijnen wrote:
> At this moment however, I don't think there is consensus yet that it is
> always better to remove all generated files, including
> autotools-generated stuff, in the clean target.

Well, I haven't seen many people claiming that it is really bad to do
this so far, so I'll try to make sure that we can record this as
consensus.  Please, if anyone thinks that it should be allowed for the
clean target to keep some generated files, speak up.  And please argue
why you think so, so we can flame^Wtalk about it. :-)

I'll have one argument myself first: If a generated file is needed by
the build, and it cannot be regenerated for some reason, for example as
a workaround for a bug (such as #441126; read only the second e-mail) or
to bootstrap a compiler build, then this is acceptable.  However, this
file may not be overwritten during the build.  If overwriting is
desired, it should be copied, the copy should be overwritten, and the
copy must be removed by the clean target.  In other words, it must be
used as a source file by the build.

This mail will automatically go to -policy.  I've added the
autotools-dev PTS address.  Followups probably won't do that, since they
don't have the X-PTS-Approved header.  So if you're not reading this on
-policy, but have an opinion, please follow the discussion there (or
through the BTS).

Thanks,
Bas

--
I encourage people to send encrypted e-mail (see http://www.gnupg.org).
If you have problems reading my e-mail, use a better reader.
Please send the central message of e-mails as plain text
   in the message body, not as HTML and definitely not as MS Word.
Please do not use the MS Word format for attachments either.
For more information, see http://pcbcn10.phys.rug.nl/e-mail.html

signature.asc (196 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Bug#397939: Lintian: outdated-autotools-helper-file

Clint Adams
In reply to this post by Bas Wijnen
On Mon, Feb 11, 2008 at 10:53:48AM +0100, Bas Wijnen wrote:
> I suggest to mandate "remove all generated files in the clean target"
> (formulated in a way which includes "generated by upstream", not only
> "generated by the build target), which implies "rebuild everything in
> the build target".

Tell me how I, as an upstream, can use an experimental version of
libtool in that situation.


--
To UNSUBSCRIBE, email to [hidden email]
with a subject of "unsubscribe". Trouble? Contact [hidden email]

Reply | Threaded
Open this post in threaded view
|

Bug#397939: Lintian: outdated-autotools-helper-file

Dr. Bas Wijnen-2
On Thu, Feb 14, 2008 at 04:43:52PM -0500, Clint Adams wrote:
> On Mon, Feb 11, 2008 at 10:53:48AM +0100, Bas Wijnen wrote:
> > I suggest to mandate "remove all generated files in the clean target"
> > (formulated in a way which includes "generated by upstream", not only
> > "generated by the build target), which implies "rebuild everything in
> > the build target".
>
> Tell me how I, as an upstream, can use an experimental version of
> libtool in that situation.

Upstream can do whatever they want, of course.  Policy only applies to
Debian.  So I'll instead answer the question "How can I, as a packager,
follow this rule when upstream uses an experimental version of libtool?"

If a package requires a compiler which currently is not in Debian
(main), then the package cannot be in main either (even if the compiler
is free).  This situation you describe falls in that category IMO.

A workaround could be to not regenerate the files.  This is how it
is usually done now.  IMO that is incorrect, because the compiler for
every generated file must be in Debian.  The current practise of not
rerunning autotools makes this rule technically unnecessary, but it can
still be violated (and that should still be considered a bug, even if it
doesn't result in a build failure).

Another workaround could be to include the experimental libtool
in the package.  This is not a very good idea, and it would probably be
better to package the new libtool as a separate package (not named
"libtool", to avoid it being used as default) and Build-Depend on that.

Even better would probably be to convince upstream to not use features
of experimental versions.  That may of course not always be possible.
:-)

Does this answer your question?

Thanks,
Bas

--
I encourage people to send encrypted e-mail (see http://www.gnupg.org).
If you have problems reading my e-mail, use a better reader.
Please send the central message of e-mails as plain text
   in the message body, not as HTML and definitely not as MS Word.
Please do not use the MS Word format for attachments either.
For more information, see http://pcbcn10.phys.rug.nl/e-mail.html

signature.asc (196 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Bug#397939: Lintian: outdated-autotools-helper-file

Russ Allbery-2
Bas Wijnen <[hidden email]> writes:

> A workaround could be to not regenerate the files.  This is how it is
> usually done now.  IMO that is incorrect, because the compiler for every
> generated file must be in Debian.  The current practise of not rerunning
> autotools makes this rule technically unnecessary, but it can still be
> violated (and that should still be considered a bug, even if it doesn't
> result in a build failure).

Note that libtool is an unusual case here and isn't the same as Autoconf
or Automake.  The files included in the package (libtool.m4 and ltmain.sh)
are not generated files in the same sense.  Running libtoolize basically
just copies those files from the installed libtool into the package.

I think (but am not at all certain) that ltmain.sh is a generated file in
that I don't think it's maintained in that form by the libtool
maintainers, but if so, whatever generation is done is already done as
part of the installation of libtool.

It's not like Autoconf or Automake where a file in the source is used as
input to a compiler which generates a shell script based on it.

--
Russ Allbery ([hidden email])               <http://www.eyrie.org/~eagle/>



--
To UNSUBSCRIBE, email to [hidden email]
with a subject of "unsubscribe". Trouble? Contact [hidden email]

Reply | Threaded
Open this post in threaded view
|

Bug#397939: Lintian: outdated-autotools-helper-file

Dr. Bas Wijnen-2
On Thu, Feb 14, 2008 at 04:02:41PM -0800, Russ Allbery wrote:

> Bas Wijnen <[hidden email]> writes:
> > A workaround could be to not regenerate the files.  This is how it is
> > usually done now.  IMO that is incorrect, because the compiler for every
> > generated file must be in Debian.  The current practise of not rerunning
> > autotools makes this rule technically unnecessary, but it can still be
> > violated (and that should still be considered a bug, even if it doesn't
> > result in a build failure).
>
> Note that libtool is an unusual case here and isn't the same as Autoconf
> or Automake.  The files included in the package (libtool.m4 and ltmain.sh)
> are not generated files in the same sense.  Running libtoolize basically
> just copies those files from the installed libtool into the package.
Oh, ok.  That changes things a bit.  In that case, not copying them, but
using the included copy instead is similar to using an included version
of a library instead of the package containing it.  This is bad for
security, but not a problem with respect to policy.

> I think (but am not at all certain) that ltmain.sh is a generated file in
> that I don't think it's maintained in that form by the libtool
> maintainers, but if so, whatever generation is done is already done as
> part of the installation of libtool.

In that case, my proposal would require it to be copied from there, and
not used from upstream.  That is wise anyway, but if it would just be a
copy of the source, it would strictly not be required.

Thanks,
Bas

--
I encourage people to send encrypted e-mail (see http://www.gnupg.org).
If you have problems reading my e-mail, use a better reader.
Please send the central message of e-mails as plain text
   in the message body, not as HTML and definitely not as MS Word.
Please do not use the MS Word format for attachments either.
For more information, see http://pcbcn10.phys.rug.nl/e-mail.html

signature.asc (196 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Bug#397939: Lintian: outdated-autotools-helper-file

Russ Allbery-2
Bas Wijnen <[hidden email]> writes:
> On Thu, Feb 14, 2008 at 04:02:41PM -0800, Russ Allbery wrote:

>> Note that libtool is an unusual case here and isn't the same as
>> Autoconf or Automake.  The files included in the package (libtool.m4
>> and ltmain.sh) are not generated files in the same sense.  Running
>> libtoolize basically just copies those files from the installed libtool
>> into the package.

> Oh, ok.  That changes things a bit.  In that case, not copying them, but
> using the included copy instead is similar to using an included version
> of a library instead of the package containing it.  This is bad for
> security, but not a problem with respect to policy.

Given that the files are only used during the build process, it seems
fairly irrelevant to security to me.  (Embedded copies of libltdl are a
different issue, but we already generally disable those.)

--
Russ Allbery ([hidden email])               <http://www.eyrie.org/~eagle/>



--
To UNSUBSCRIBE, email to [hidden email]
with a subject of "unsubscribe". Trouble? Contact [hidden email]

Reply | Threaded
Open this post in threaded view
|

Bug#397939: Lintian: outdated-autotools-helper-file

Clint Adams
In reply to this post by Russ Allbery-2
On Thu, Feb 14, 2008 at 04:02:41PM -0800, Russ Allbery wrote:
> Note that libtool is an unusual case here and isn't the same as Autoconf
> or Automake.  The files included in the package (libtool.m4 and ltmain.sh)
> are not generated files in the same sense.  Running libtoolize basically
> just copies those files from the installed libtool into the package.

libtool.m4 is included via aclocal (unless you're not using automake),
and then processed into configure by autoconf (unless you're not using
autoconf).

So if you build-dep on autoconf and automake and try to use sources
written for libtool 2.1, everything will break horribly even if you keep
the static libtool files the same.

We have been stuck with libtool 1.5 for 4 years.



--
To UNSUBSCRIBE, email to [hidden email]
with a subject of "unsubscribe". Trouble? Contact [hidden email]

Reply | Threaded
Open this post in threaded view
|

Bug#397939: Lintian: outdated-autotools-helper-file

Russ Allbery-2
Clint Adams <[hidden email]> writes:
> On Thu, Feb 14, 2008 at 04:02:41PM -0800, Russ Allbery wrote:

>> Note that libtool is an unusual case here and isn't the same as
>> Autoconf or Automake.  The files included in the package (libtool.m4
>> and ltmain.sh) are not generated files in the same sense.  Running
>> libtoolize basically just copies those files from the installed libtool
>> into the package.

> libtool.m4 is included via aclocal (unless you're not using automake),
> and then processed into configure by autoconf (unless you're not using
> autoconf).

Oh, yes, good point.  Sorry about that.

> So if you build-dep on autoconf and automake and try to use sources
> written for libtool 2.1, everything will break horribly even if you keep
> the static libtool files the same.
>
> We have been stuck with libtool 1.5 for 4 years.

I was used to upstreams that include libtool.m4 in the package, in which
case I don't believe that running aclocal will overwrite it (but maybe I'm
wrong?).  If they're pulling in libtool macros using aclocal without
keeping a local copy, that indeed will break.

--
Russ Allbery ([hidden email])               <http://www.eyrie.org/~eagle/>



--
To UNSUBSCRIBE, email to [hidden email]
with a subject of "unsubscribe". Trouble? Contact [hidden email]

Reply | Threaded
Open this post in threaded view
|

Re: Bug#397939: Lintian: outdated-autotools-helper-file

Colin Watson
In reply to this post by Bas Wijnen
On Mon, Feb 11, 2008 at 10:53:48AM +0100, Bas Wijnen wrote:

> This is not true if you simply build the whole package from source.
> That is, run autotools during build, remove all generated files,
> including Makefile.in, configure, etc, in the clean target.
>
> For some reason many people seem to think that the whole package must be
> built from source, except for configure and Makefile.in.  I still don't
> understand what the idea behind this exception is.  Especially when it
> leads to unwanted concequences (unreadable diff.gzs, for example), I
> don't think it is a good idea to hold on to the idea that running
> autotools is not part of building the package.

It's a well-established principle of the GNU build system that there are
targets that are run by maintainers and there are targets to be run by
people building the package. This division is there to make our lives
easier, and ignoring it, IMHO, is cutting off our nose to spite our
face. (Even the section in the GNU Coding Standards regarding the
maintainer-clean target doesn't go so far as deleting configure.)

The underlying problem here is that users who change configure.ac etc.
need to do some additional manual work to update the build system, and
that this should be done automatically. I agree. However, this is not
the same as regenerating the full build system from scratch on every
build.

You should be aware, if you aren't already, that the process of
generating a package's build system from scratch is often really quite
complicated and may well involve network access, which build daemons do
not have. For example, the usual update process for a package that uses
Gnulib involves rsyncing translations for some messages from
translationproject.org. If you mandate that Debian maintainers must
support full build system regeneration on every build, then you will
force some of them to maintain a separate bootstrapping process which is
guaranteed not to access the network, which will involve duplicating
work already done by upstream and will undoubtedly introduce errors at
some point.

Mere incremental regeneration on demand is much simpler and is often
already supported by upstream build systems without requiring any change
(see below).

> I suggest to mandate "remove all generated files in the clean target"
> (formulated in a way which includes "generated by upstream", not only
> "generated by the build target), which implies "rebuild everything in
> the build target".

I object to this proposal. As an example, I am upstream for man-db and
know that its build system is properly generated. Requiring that I
regenerate it every time I build the package would introduce nothing but
hassle for me, and would not in practice provide significant benefits
for users since the build system already caters for them out of the box
(see below).

> With the current wording it is allowed to use shipped built files from
> the upstream tarball, as long as the source is present.  It is even
> allowed to ship the build results (uuencoded, for example) in the
> diff.gz and use those.  I suppose we all agree that this is unacceptable
> for normal build results.

configure is readable, if you have to. It's not the preferred form for
modification, certainly, but it's not an opaque object file either, and
there's no need to treat it exactly as we would treat opaque object
files.

Bear in mind also that, while upstream maintainers will probably accept
that configure.ac files that don't work with current versions of
Autoconf are bugs, they are also unlikely to have immediate sympathy for
"I rebuilt your package with a version of Autoconf three years newer
than the one you used and it produced a horrible mess; please help me
untangle it". In other words, this proposal will have the effect of
requiring that Debian maintainers become experts in the Autotools before
being able to upload policy-compliant packages again, whereas that
burden could formerly be passed to upstream to be dealt with on their
schedule. I wouldn't expect that figuring out enough about the Autotools
to be able to do this should be beyond most Debian maintainers, but
neither would I expect that they could all deal with some of the weird
stuff I've seen already.

I agree that behavioural changes due to regenerating the build system
are bugs, but what you're suggesting is that all such bugs be elevated
to the status of showstoppers, as it will be impossible to upload
packages without it. I'm sure that that would make the security team's
life a lot harder, for instance - while there are a few vulnerabilities
that require configure.ac changes to fix, in most cases you can get away
without needing to do so.

Furthermore, if as a user you're desperate to make a small tweak to
configure.ac, it *is* often quite possible - if ugly! - to just make the
obvious corresponding change to configure and then get on with your
life. You can't realistically do that with e.g. compiled C object files,
which is one reason we treat them differently.

> Now reread the previous paragraph while thinking of Makefile.in instead
> of "normal build results".  It is common practice to do exactly that:
> ship and use pre-built versions, or even ship them in the diff.gz (which
> then gets huge).  Makefile.am is only present as source-file, but it
> isn't used at all during the build.  Any changes the user makes will not
> be noticed by the build system.

But this simply isn't true across the board! Makefile.am changes are
only ignored if you use AM_MAINTAINER_MODE. One reason people do that is
that it produces more predictable build-dependencies that aren't
affected by timestamp skew. This was more of an issue before bug #105750
was fixed. Nowadays, while I think the jury is still out in some places
regarding AM_MAINTAINER_MODE, the case for it is much less compelling
than it used to be, and the Automake manual explicitly recommends
against it. If you don't use AM_MAINTAINER_MODE, then Makefile.am
changes are automatically noticed and taken into account. A
demonstration:

  $ apt-get source man-db
  [...]
  dpkg-source: extracting man-db in man-db-2.5.1
  dpkg-source: unpacking man-db_2.5.1.orig.tar.gz
  dpkg-source: applying ./man-db_2.5.1-2.diff.gz
  $ cd man-db-2.5.1/
  $ touch configure.ac
  $ debian/rules build
  ./configure --prefix=/usr --libexecdir=\${libdir} \
  [...]
  touch configure-stamp
  dh_testdir
  /usr/bin/make CFLAGS="-O2 -g -Wall" LDFLAGS="-g"
  make[1]: Entering directory `/home/cjwatson/man-db-2.5.1'
  cd . && /bin/bash /home/cjwatson/man-db-2.5.1/tools/missing --run aclocal-1.10 -I m4 -I gnulib/m4
   cd . && /bin/bash /home/cjwatson/man-db-2.5.1/tools/missing --run automake-1.10 --foreign
  cd . && /bin/bash /home/cjwatson/man-db-2.5.1/tools/missing --run autoconf
  /bin/bash ./config.status --recheck
  [...]

(The same goes for changing Makefile.am.)

Rather than incurring the pain of gratuitous full regeneration every
time, we just regenerate it when the user has changed something. Yes,
the user now gets to resolve any problems that might have been
pre-existing, but realistically either the Debian maintainer or the
upstream maintainer is probably going to run into those at some point
anyway.

I think we should recommend (but not require) that AM_MAINTAINER_MODE
not be used, and perhaps work to specify an optional debian/rules target
that regenerates the build system in an appropriate way. That seems to
provide the necessary benefits for users who need to change these files
without imposing an unacceptable burden on developers. I don't think
there's a good cause to go much further than that at this point.

Cheers,

--
Colin Watson                                       [[hidden email]]


--
To UNSUBSCRIBE, email to [hidden email]
with a subject of "unsubscribe". Trouble? Contact [hidden email]

Reply | Threaded
Open this post in threaded view
|

Bug#397939: Lintian: outdated-autotools-helper-file

Russ Allbery-2
Colin Watson <[hidden email]> writes:

> Rather than incurring the pain of gratuitous full regeneration every
> time, we just regenerate it when the user has changed something. Yes,
> the user now gets to resolve any problems that might have been
> pre-existing, but realistically either the Debian maintainer or the
> upstream maintainer is probably going to run into those at some point
> anyway.
>
> I think we should recommend (but not require) that AM_MAINTAINER_MODE
> not be used, and perhaps work to specify an optional debian/rules target
> that regenerates the build system in an appropriate way. That seems to
> provide the necessary benefits for users who need to change these files
> without imposing an unacceptable burden on developers. I don't think
> there's a good cause to go much further than that at this point.

I think this would in some respects be the worst of both worlds.  The
problem with not using AM_MAINTAINER_MODE is that the autotools *may* be
run but generally aren't.  This means that build dependencies only needed
when one modifies those files aren't present or aren't tested.  Modifying
one of those files can suddenly spark the discovery that upstream isn't
compatible with the current autotools, the partial run of Automake can
leave the whole tree in a broken state, and so forth.

But I suppose that's basically the normal argument for AM_MAINTAINER_MODE.

--
Russ Allbery ([hidden email])               <http://www.eyrie.org/~eagle/>



--
To UNSUBSCRIBE, email to [hidden email]
with a subject of "unsubscribe". Trouble? Contact [hidden email]

Reply | Threaded
Open this post in threaded view
|

Bug#397939: Lintian: outdated-autotools-helper-file

Dr. Bas Wijnen-2
In reply to this post by Colin Watson
On Sun, Feb 17, 2008 at 03:07:59PM +0000, Colin Watson wrote:

> On Mon, Feb 11, 2008 at 10:53:48AM +0100, Bas Wijnen wrote:
> > This is not true if you simply build the whole package from source.
> > That is, run autotools during build, remove all generated files,
> > including Makefile.in, configure, etc, in the clean target.
> >
> > For some reason many people seem to think that the whole package must be
> > built from source, except for configure and Makefile.in.  I still don't
> > understand what the idea behind this exception is.  Especially when it
> > leads to unwanted concequences (unreadable diff.gzs, for example), I
> > don't think it is a good idea to hold on to the idea that running
> > autotools is not part of building the package.
>
> It's a well-established principle of the GNU build system that there are
> targets that are run by maintainers and there are targets to be run by
> people building the package. This division is there to make our lives
> easier, and ignoring it, IMHO, is cutting off our nose to spite our
> face. (Even the section in the GNU Coding Standards regarding the
> maintainer-clean target doesn't go so far as deleting configure.)
So according to the GCS, even the maintainer shouldn't need to
regenerate configure, or at least not by default.   This doesn't quite
prove your point, that this is a "maintainter-only" thing. ;-)

Anyway, AFAIK the reason that autoconf exists is to allow building
programs reliably, even on exotic architectures.  They may have really
weird shells, or a weird make, or libraries in weird places needing
special flags, etc.

That is a good thing.  I am happy that it is possible to build the
programs on those architectures with relatively little trouble.  One of
the essential things of autoconf there, is that it doesn't require
itself to be installed.  The generated configure script should be very
platform-independent.  It is pre-built exactly for that reason: to make
sure that the build does not require autoconf to be installed.

So far so good.  But Debian isn't an exotic system.  It's GNU with a
Linux kernel.  You can't get more standard than that, from autoconf's
(thus GNU's) point of view.  We don't need this
super-platform-independence on our system.  We have a proper shell, GNU
make, and autoconf and automake can be installed without any trouble.

In other words, there is no reason to use the pre-built files.  Their
reason for existing isn't applicable to Debian.

> The underlying problem here is that users who change configure.ac etc.
> need to do some additional manual work to update the build system, and
> that this should be done automatically. I agree. However, this is not
> the same as regenerating the full build system from scratch on every
> build.

Let's compare it with a Java program.  Would you consider it acceptable
for the packager to build it, uuencode it, put it in the diff.gz and
from debian/rules just uudecode it, instead of regenerating it?  I fail
to see the difference between this and using a pre-built configure and
Makefile.in.  Compiled Java is also platform independent (AFAIK), so
there isn't even a problem with architectures.

> You should be aware, if you aren't already, that the process of
> generating a package's build system from scratch is often really quite
> complicated

Of course it is.  That's exactly the reason that I care so much about
it.  If it would be trivial, everyone could just do it, even without
using debian/rules, or having it as an example.

> and may well involve network access, which build daemons do
> not have.

Building the package should not require network access.  The package is
built using the version which was "the newest" (I hope) when it was
created.  While it could be nice to be able to use newer versions of
things, I agree that we do not want that.  It makes builds less
reproducible, for example.

> For example, the usual update process for a package that uses
> Gnulib involves rsyncing translations for some messages from
> translationproject.org.

And so this step is not needed for the Debian package (but it would be
nice if debian/rules did actually contain a target for it).

> If you mandate that Debian maintainers must support full build system
> regeneration on every build, then you will force some of them to
> maintain a separate bootstrapping process which is guaranteed not to
> access the network, which will involve duplicating work already done
> by upstream and will undoubtedly introduce errors at some point.

While I think Gnulib is not a good example (AFAIK it contains GNU libc
functions for platforms without GNU libc, which are unused if they are
available natively), I see your point.  I think this should be uncommon,
though, and worth the trouble.  Also, if it's really too hard, it can be
decided on a per-package basis to ignore policy and do things in a way
that works.  That means there is a bug in the package, which is hard to
fix, but it doesn't mean the package can't be released.  I'm not
suggesting that breaking this rule should be a release-critical bug.
(At least not yet.  If it turns out that there are no packages (anymore)
which can't follow the rule, then we may change that.)

> Mere incremental regeneration on demand is much simpler and is often
> already supported by upstream build systems without requiring any
> change (see below).

Of course it's simpler to skip a part of the compilation process.  It's
also simpler for Java packages to use pre-built versions instead of
recompiling, but that doesn't make it right to do it.

> > I suggest to mandate "remove all generated files in the clean target"
> > (formulated in a way which includes "generated by upstream", not only
> > "generated by the build target), which implies "rebuild everything in
> > the build target".
>
> I object to this proposal. As an example, I am upstream for man-db and
> know that its build system is properly generated. Requiring that I
> regenerate it every time I build the package would introduce nothing but
> hassle for me, and would not in practice provide significant benefits
> for users since the build system already caters for them out of the box
> (see below).
The fact that there exist packages which work properly without
recompiling from source doesn't mean it's a good default.  IMO the
default should be to always compile from source.  Yes, that means hassle
for the packager; it's pretty much the whole task of packaging.

The benefits I see are that we know things can be generated, that users
don't have to find out how to regenerate things, and what to install for
it (your package may be fine, but that doesn't mean they all are).  And
also important, that we are sure that the shipped source files are
really the source for what is generated.

> > With the current wording it is allowed to use shipped built files from
> > the upstream tarball, as long as the source is present.  It is even
> > allowed to ship the build results (uuencoded, for example) in the
> > diff.gz and use those.  I suppose we all agree that this is unacceptable
> > for normal build results.
>
> configure is readable, if you have to. It's not the preferred form for
> modification, certainly, but it's not an opaque object file either, and
> there's no need to treat it exactly as we would treat opaque object
> files.
True, but also not much different.  It is a generated file.

> In other words, this proposal will have the effect of requiring that
> Debian maintainers become experts in the Autotools before being able
> to upload policy-compliant packages again,

Not at all.  Autoconf is pretty stable, so I suppose you're talking
about automake.  Because of its interface instability, you should never
depend on "automake", but always on "automake1.10" (or whatever version
you tested with).  It's also a good idea to force the build process to
use that version.  That requires setting two environment variables
(AUTOMAKE and ACLOCAL).  I suppose you're not suggesting that knowing to
set those makes you an autotools expert (especially when it will be
documented, for example in the developers' reference).

> whereas that burden could formerly be passed to upstream to be dealt
> with on their schedule.

When you're explicitly using a certain version of automake, this is
still the case.  There may be a bit of pushing when the automake
maintainers want to remove the version you're using, but they will wait
until you finished converting to the newer version (or help you with
that).  In fact, this makes things easier for the maintainer: if you
don't have the build-dependency recorded, your automake version may
disappear, leaving you unable to build the new version of the package
(considering worst case, where it doesn't work with the new version and
it's not easy to fix).

> I wouldn't expect that figuring out enough about the Autotools to be
> able to do this should be beyond most Debian maintainers, but neither
> would I expect that they could all deal with some of the weird stuff
> I've seen already.

If really weird stuff is needed to compile the package (from source),
then IMO it's totally reasonable that that weirdness is in debian/rules,
so users can see what's happening, if they want to.

> I agree that behavioural changes due to regenerating the build system
> are bugs, but what you're suggesting is that all such bugs be elevated
> to the status of showstoppers, as it will be impossible to upload
> packages without it.

No no, you misunderstood me.  I'm saying you should get a lintian error
(and a bug) without it, not that you can't upload, and not even that the
bug should be RC.

> I'm sure that that would make the security team's life a lot harder,
> for instance

Huh?  I would think life only gets easier for them...  Packages which do
follow the rule simply don't get uploaded if they do things wrong.  For
packages which ignore the rule, there is no difference.

Perhaps you mean that lots of bugs pop up when "automake" increases its
version?  In that case I apologize for being unclear; I thought it was
commonly known and agreed upon that you should always build-depend on
the versioned package.

> Furthermore, if as a user you're desperate to make a small tweak to
> configure.ac, it *is* often quite possible - if ugly! - to just make the
> obvious corresponding change to configure and then get on with your
> life. You can't realistically do that with e.g. compiled C object files,
> which is one reason we treat them differently.

I agree that this means it is less unacceptable to not regenerate them,
but it doesn't make it right.

Also, it is quite improper to expect this from someone who wants to NMU
the package.

> > Now reread the previous paragraph while thinking of Makefile.in instead
> > of "normal build results".  It is common practice to do exactly that:
> > ship and use pre-built versions, or even ship them in the diff.gz (which
> > then gets huge).  Makefile.am is only present as source-file, but it
> > isn't used at all during the build.  Any changes the user makes will not
> > be noticed by the build system.
>
> But this simply isn't true across the board! Makefile.am changes are
> only ignored if you use AM_MAINTAINER_MODE.

If you don't use it, you need to do the "touch all targets in the right
order" game if you want to patch anything.  Very ugly as well, if you
ask me.

> If you don't use AM_MAINTAINER_MODE, then Makefile.am changes are
> automatically noticed and taken into account.

I know.  But then the system tries to run executables which aren't in
build-depends.  It works, but it isn't nice.  Also, as I wrote above,
you need to do the touching game if there are any patches.  If you do
this in debian/rules, the user's changes are ignored again.  If you
don't, it's hard to patch these parts in an NMU (since the touching must
be added as well).

Altogether, the behaviour is unpredictable for someone who doesn't know
what's going on.  I think it's reasonable for a user to edit any source
file and use debian/rules to build a new package with it.

> Rather than incurring the pain of gratuitous full regeneration every
> time, we just regenerate it when the user has changed something. Yes,
> the user now gets to resolve any problems that might have been
> pre-existing, but realistically either the Debian maintainer or the
> upstream maintainer is probably going to run into those at some point
> anyway.

I don't think that's very realistic at all for all cases.  For example,
if the Debian maintainer is also upstream, there's only one system.  If
on that system some weird things are installed which make things work
even though they shouldn't, there is no reason to believe he will find
out.  When building the package in pbuilder, though, it will immediately
show.  But only if the files are indeed regenerated during that build.

> I think we should recommend (but not require) that AM_MAINTAINER_MODE
> not be used,

Why?  If you consider it acceptable for a user to edit configure, it
should certainly be no problem to add --enable-maintainer-mode to the
configure invocation.

> and perhaps work to specify an optional debian/rules target that
> regenerates the build system in an appropriate way. That seems to
> provide the necessary benefits for users who need to change these
> files

Not at all.  If it's optional, it's likely that many packages will not
have it.  Also, if the build system doesn't use it by default, it is
likely that many of those targets are never tested and don't actually
work.

> without imposing an unacceptable burden on developers.

I hope you do agree that maintainers should anyway be able to build
their package from source?  The only difference is that it should be
done in debian/rules.  I don't see how this is an "unacceptable burden".

> I don't think there's a good cause to go much further than that at
> this point.

We're currently trying to allow packages to be built twice in a row as a
release goal.  If the clean target would be as I propose, this would
automatically be fulfilled as well.

Also, considering the bug we're writing to, what would you propose that
the clean target should do?  "Remove all generated files, except the
ones that are generated by autotools"?  I can't properly express how
completely wrong I find such an exception when written in Policy...


Thank you for responding.  It would be a pretty big change in Policy, if
adopted, and so it's good when there's some discussion about it, to
detect any problems it might cause.

Thanks,
Bas

--
I encourage people to send encrypted e-mail (see http://www.gnupg.org).
If you have problems reading my e-mail, use a better reader.
Please send the central message of e-mails as plain text
   in the message body, not as HTML and definitely not as MS Word.
Please do not use the MS Word format for attachments either.
For more information, see http://pcbcn10.phys.rug.nl/e-mail.html

signature.asc (196 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Bug#397939: Lintian: outdated-autotools-helper-file

Russ Allbery-2
Bas Wijnen <[hidden email]> writes:

> Autoconf is pretty stable,

This has not been the experience of many of us.  I haven't had a lot of
trouble fixing things for newer releases of Autoconf, but I definitely
have seen issues.  And the Autoconf 2.13 to 2.50 transition and all the
subsequent instability was not that long ago.

> so I suppose you're talking about automake.  Because of its interface
> instability, you should never depend on "automake", but always on
> "automake1.10" (or whatever version you tested with).

Just FYI:

windlord:~> apt-cache policy automake1.10
automake1.10:
  Installed: (none)
  Candidate: (none)
  Version table:
windlord:~> apt-cache policy automake
automake:
  Installed: 1:1.10.1-2
  Candidate: 1:1.10.1-2
  Version table:
     1:1.10.1-3 0
        500 http://exodus.stanford.edu unstable/main Packages
 *** 1:1.10.1-2 0
        990 http://exodus.stanford.edu testing/main Packages
        100 /var/lib/dpkg/status

--
Russ Allbery ([hidden email])               <http://www.eyrie.org/~eagle/>



--
To UNSUBSCRIBE, email to [hidden email]
with a subject of "unsubscribe". Trouble? Contact [hidden email]

123