Re: [buildd-tools-devel] Bug#859866: sbuild transiently fails stating apt-get update failed

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [buildd-tools-devel] Bug#859866: sbuild transiently fails stating apt-get update failed

Johannes Schauer-3
Hi,

Quoting Michael Stapelberg (2017-04-08 11:21:46)

> I may very well be doing something wrong, but I repeatedly run into
> the following problem and have no clue as to how to approach fixing
> it.
>
> Every once in a while, an sbuild invocation will fail with the
> following message:
>
>     Reading package lists...
>     E: Could not open file /var/lib/apt/lists/deb.debian.org_debian_dists_sid_main_source_Sources.diff_Index - open (2: No such file or directory)
>     E: apt-get update failed
>
> I can fix this issue by running “sbuild-update -u unstable” twice
> (!). The first run fails with the same error message (but stores the
> updated package lists), the second run will do the actual update.
>
> Is this a misconfiguration on my end, or a bug in sbuild/schroot/apt?
>
> Find a full session transcript attached.
I never encountered this error message. Neither from running sbuild nor from
running apt-get update.

Maybe apt people have a clue about what this means? I CC-ed their list.

Michael, is it possible for you to create a chroot from scratch using
sbuild-createchroot and then reliably reproduce the problem from there? Then I
could have a look at the problem myself.

Thanks!

cheers, josch

signature.asc (849 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [buildd-tools-devel] Bug#859866: sbuild transiently fails stating apt-get update failed

Michael Stapelberg-25


On Tue, Apr 11, 2017 at 10:43 AM, Johannes Schauer <[hidden email]> wrote:
Hi,

Quoting Michael Stapelberg (2017-04-08 11:21:46)
> I may very well be doing something wrong, but I repeatedly run into
> the following problem and have no clue as to how to approach fixing
> it.
>
> Every once in a while, an sbuild invocation will fail with the
> following message:
>
>     Reading package lists...
>     E: Could not open file /var/lib/apt/lists/deb.debian.org_debian_dists_sid_main_source_Sources.diff_Index - open (2: No such file or directory)
>     E: apt-get update failed
>
> I can fix this issue by running “sbuild-update -u unstable” twice
> (!). The first run fails with the same error message (but stores the
> updated package lists), the second run will do the actual update.
>
> Is this a misconfiguration on my end, or a bug in sbuild/schroot/apt?
>
> Find a full session transcript attached.

I never encountered this error message. Neither from running sbuild nor from
running apt-get update.

Maybe apt people have a clue about what this means? I CC-ed their list.

Michael, is it possible for you to create a chroot from scratch using
sbuild-createchroot and then reliably reproduce the problem from there? Then I
could have a look at the problem myself.

I’d be happy to wipe my sbuild setup and start from scratch. Which commands specifically do you recommend I use?


--
Best regards,
Michael
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [buildd-tools-devel] Bug#859866: sbuild transiently fails stating apt-get update failed

Johannes Schauer-3
Hi,

Quoting Michael Stapelberg (2017-04-11 12:41:03)

> On Tue, Apr 11, 2017 at 10:43 AM, Johannes Schauer <[hidden email]> wrote:
> > Quoting Michael Stapelberg (2017-04-08 11:21:46)
> > > I may very well be doing something wrong, but I repeatedly run into
> > > the following problem and have no clue as to how to approach fixing
> > > it.
> > >
> > > Every once in a while, an sbuild invocation will fail with the
> > > following message:
> > >
> > >     Reading package lists...
> > >     E: Could not open file /var/lib/apt/lists/deb.debian.
> > org_debian_dists_sid_main_source_Sources.diff_Index - open (2: No such
> > file or directory)
> > >     E: apt-get update failed
> > >
> > > I can fix this issue by running “sbuild-update -u unstable” twice
> > > (!). The first run fails with the same error message (but stores the
> > > updated package lists), the second run will do the actual update.
> > >
> > > Is this a misconfiguration on my end, or a bug in sbuild/schroot/apt?
> > >
> > > Find a full session transcript attached.
> >
> > I never encountered this error message. Neither from running sbuild nor
> > from
> > running apt-get update.
> >
> > Maybe apt people have a clue about what this means? I CC-ed their list.
> >
> > Michael, is it possible for you to create a chroot from scratch using
> > sbuild-createchroot and then reliably reproduce the problem from there?
> > Then I
> > could have a look at the problem myself.
> >
>
> I’d be happy to wipe my sbuild setup and start from scratch. Which commands
> specifically do you recommend I use?
If you start completely from scratch, setting up sbuild with a schroot backend
works like this (since Debian Stretch):

$ sudo apt-get install sbuild
$ sudo sbuild-adduser $LOGNAME
$ ... *logout* and *re-login* or use `newgrp sbuild` in your current shell
$ sudo sbuild-createchroot unstable /srv/chroot/unstable-amd64-sbuild http://httpredir.debian.org/debian

So nearly what it currently says on the sbuild wiki page but minus several
workarounds that are not needed anymore since Stretch. These instructions can
also be taken from the sbuild man page (the EXAMPLES section) which also links
to the sbuild-createchroot man page with more examples in the respective
section.

But you will probably not completely start from scratch because you probably
populated /var/lib/sbuild/apt-keys for example but that shouldn't influence the
effect you see. Also make sure that you are using the default config either by
moving your ~/.sbuildrc or by passing a custom configuration via the
SBUILD_CONFIG environment variable.

Thanks!

cheers, josch

signature.asc (849 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [buildd-tools-devel] Bug#859866: sbuild transiently fails stating apt-get update failed

David Kalnischkies-4
In reply to this post by Johannes Schauer-3
On Tue, Apr 11, 2017 at 10:43:12AM +0200, Johannes Schauer wrote:
> >     Reading package lists...
> >     E: Could not open file /var/lib/apt/lists/deb.debian.org_debian_dists_sid_main_source_Sources.diff_Index - open (2: No such file or directory)
> >     E: apt-get update failed
[…]
> > Find a full session transcript attached.
>
> I never encountered this error message. Neither from running sbuild nor from
> running apt-get update.
>
> Maybe apt people have a clue about what this means? I CC-ed their list.

Haven't seen that either yet. There are a few observations I can make
through:

- ignoring the error, the command seems to be successful
- the chroots last update seems to be a while ago – the contrib patches
  downloaded go back a month which has infrequent updates, but main has
  changes all the time, so in all likelyhood (if apt reaches that stage)
  apt would figure out that the Index doesn't include the starting file
  anymore and fall back to full download (as it does in many other
  situations)
- the error message is about the "final" directory, not partial/.


Combined, I have the wild guess that apt sees a reason to fallback to
the full download, but while winding down the patch handling something
tries to access the file (or a previous version of it) which happens to
be discarded already. But: I had a quick look and nothing which
immediatly caught my eye codewise (more "proofs" that this can't be it
actually) & I ran a quick test with the too old situation I described
above which was completed successfully without errors from a user PoV…


As I have no idea, I would suggest running apt with an extended set of
debug options and hope that this gives a better hint at what could be
wrong:

Debug::pkgAcquire "true";
Debug::pkgAcquire::Diffs "true";
Debug::pkgAcquire::Worker "true";
Debug::Acquire::Transaction "true";
Debug::Acquire::http "true";

(the later two aren't a typo. There is a reason the "pkg" is missing
from them – just don't ask me why as I have no idea ;) )

Assuming this is reproducible. I fear it isn't… (running apt with these
options isn't dangerous or anything if someone wanted to try for longer,
they just make apt super chatty about some things.)


Best regards

David Kalnischkies

signature.asc (849 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [buildd-tools-devel] Bug#859866: sbuild transiently fails stating apt-get update failed

Michael Stapelberg-25
Thanks everyone for the details.

I have now re-created my sbuild setup from scratch as suggested by josch. I’ll re-open the bug if/when I encounter the issue again.

On Tue, Apr 11, 2017 at 1:38 PM, David Kalnischkies <[hidden email]> wrote:
On Tue, Apr 11, 2017 at 10:43:12AM +0200, Johannes Schauer wrote:
> >     Reading package lists...
> >     E: Could not open file /var/lib/apt/lists/deb.debian.org_debian_dists_sid_main_source_Sources.diff_Index - open (2: No such file or directory)
> >     E: apt-get update failed
[…]
> > Find a full session transcript attached.
>
> I never encountered this error message. Neither from running sbuild nor from
> running apt-get update.
>
> Maybe apt people have a clue about what this means? I CC-ed their list.

Haven't seen that either yet. There are a few observations I can make
through:

- ignoring the error, the command seems to be successful
- the chroots last update seems to be a while ago – the contrib patches
  downloaded go back a month which has infrequent updates, but main has
  changes all the time, so in all likelyhood (if apt reaches that stage)
  apt would figure out that the Index doesn't include the starting file
  anymore and fall back to full download (as it does in many other
  situations)
- the error message is about the "final" directory, not partial/.


Combined, I have the wild guess that apt sees a reason to fallback to
the full download, but while winding down the patch handling something
tries to access the file (or a previous version of it) which happens to
be discarded already. But: I had a quick look and nothing which
immediatly caught my eye codewise (more "proofs" that this can't be it
actually) & I ran a quick test with the too old situation I described
above which was completed successfully without errors from a user PoV…


As I have no idea, I would suggest running apt with an extended set of
debug options and hope that this gives a better hint at what could be
wrong:

Debug::pkgAcquire "true";
Debug::pkgAcquire::Diffs "true";
Debug::pkgAcquire::Worker "true";
Debug::Acquire::Transaction "true";
Debug::Acquire::http "true";

(the later two aren't a typo. There is a reason the "pkg" is missing
from them – just don't ask me why as I have no idea ;) )

Assuming this is reproducible. I fear it isn't… (running apt with these
options isn't dangerous or anything if someone wanted to try for longer,
they just make apt super chatty about some things.)


Best regards

David Kalnischkies



--
Best regards,
Michael
Loading...