Upgrade Adventure Stretch-to-Buster

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

Upgrade Adventure Stretch-to-Buster

Mark Fletcher-2
Hi list

This long email is just a report on my recent stretch to buster upgrade
experience. I had a bit of an adventure, didn't handle some steps well,
and thought the experience would be useful to put out there for others
to learn from / avoid some mistakes I made.

THERE IS NO QUESTION / PROBLEM TO SOLVE AT THE END OF THIS -- it's just
here for info and in the hope it will help someone. If you don't like
long emails, please know that I couldn't care less.

I was upgrading a Stretch desktop system to Buster. I've moved most of
my most critical activities into the cloud and so my machine isn't
nearly as mission-critical as it once was, but outages would still be
annoying enough that I would want ideally to avoid significant downtime.

I have other users on this machine, who occasionally access my machine
remotely via a VPN, but they are very much minority users and I am the
most common user on my machine by a long way.

I perform 5-days-a-week backups using Amanda. The machine in question
here happens to be the Amanda server as well as a client. The amanda
virtual tapes and holding disk are in a USB3 disk cage. The cage also
contains a RAID1 disk pair which contains data of various types.

Internally my machine has 2 1TB SSDs, one of which cotains most of my
filesystem and the other is mounted at /opt. /opt contains a fairly
extensive mariadb server, a SVN repository, and hard disk space for 2
Windows VMs.

The upgrade was expected to have a large effect on the disk containing /
and minimal effect on the disk containing /opt.

I verified before starting that my most recent backups had been
successful. I disabled the systemd timer job I'd created to run the
backups each night so that if I did have a messed-up system
half-upgraded during the process, it didn't corrupt my backups. However
I wasn't super confident that I could restore from my backups
willy-nilly as I had had only limited occasion to do so before (turns
out I didn't need to worry about this, I ended up restoring liberally
from my backups and it went fine).

Before starting I additionally rsync'd my entire home directory and
select parts of /etc (mysql / mariadb config, hard-won exim4 config,
openvpn config etc) to spare disk capacity in the drive cage ie
off-machine, just in case.

I MISSED my SVN config in this backup, incidentally.I had an automatic
nightly backup of it but I did not hand-crank an additional backup of it
before starting, because I forgot about it.

I carefully read the upgrade instructions on the Debian website. I
noticed that I needed to migrate to ostensibly-predictable network
interface names before doing the upgrade, so I duly executed the steps
prescribed for that. That worked fine and internet access was none the
worse for it afterward, before upgrading.

I noticed and didn't completely understand some warnings about ssh and,
I think, old types of keys. I didn't do anything about that because I
didn't really understand what that was saying. Sometimes I think some of
these notes could do with a "What This Means In Plain English"
section... Anyway that didn't seem to have any problems in the end.

I went through my sources.list.d directory and peeled out a couple of
things I didn't expect to do in the upgrade eg Virtualbox's repository,
the Google Talk plugin (gawd alone knows where that came from), and
Christian Marillat's deb-multimedia repository. I thought I should do
the upgrade without them and hoped to wean myself off deb-multimedia
while I was at it. This, as it turned out, was a huge mistake.

After all this I modified my sources.list according to the upgrade
instructions and apt update'd.

Next, I did an apt upgrade to upgrade some stuff before the main apt
dist-upgrade, again following the upgrade instructions. This was the
last command my system executed as a healthy system...

Obviously there was a lot to download and so on, so I let it get on with
it. Eventually, after a lot of downloading and quite a lot of package
upgrading, the process failed with an error. The issue was gstreamer and
its library, which had hit a problem trying to upgrade because they were
trying to overwrite a file also contained in the gstreamer-plugins-bad
package.

This turned out to be a deb-multimedia problem. I had hoped that
upgrading to Buster versions of packages would supercede the
deb-multimedia repository package versions, but they didn't.

Now, in this situation, it didn't seem to be possible to use apt to go
forward or back. If I tried to repeat the upgrade, install anything new,
remove anything, or anything else, it would immediately stop saying
there are broken packages, fix them with apt --fix-broken install. And
if I did apt --fix-broken install, it would complain about the file
write conflict I highlighted above.

At this point I tried to use the instructions in the upgrade
instructions for handling this error message, and I don't think I did it
right because the situation just steadily got worse and worse. Force
removing packages using dpkg --force-depends -r <package> just moved the
problem from one place to another. I ended up chasing one tree of
dependencies all the way down to a conflict involving libstdc++6. By
then I was so trigger-happy I force-removed that, and that was the end
of using apt to do much of anything...

I was still able to use wget to re-download the libstdc++6 package for
stretch, and tried to re-install it using dpkg (which doesn't need
libstdc++6). When that actually failed to install, despite the fact it
had just been installed, I concluded that my package management was so
corrupted on this machine that there was nothing for it but a
re-install.

Since I had all my backups, I downloaded the netinst iso on another
computer and burned it to a USB stick. I also put my /etc/fstab on that
stick as it would help me re-mount /opt and my drive cage when I was
done with the re-install.

I also made sure I had my amanda config separately backed up as I wasn't
confident (again, oh me of little faith as it turned out) that I had my
config set up appropriately to be able to be restored into a fresh
machine.

The re-install went smoothly and, using my retained copy of my old
/etc/fstab, I was able to bring my second SSD back in as /opt and get
the disks in my drive cage back, including my amanda backup disks. I was
then able to restore my home directory, and key config such as exim4,
SVN, mariadb etc etc from my amanda backups, so one positive thing to
come out of this is I'm now much more confident my backups are actually
worthwhile as they've now been successfully battle-tested.

The only thing I haven't yet gotten around to since the re-install is to
re-install VirtualBox from the VB repositories. I have the virtual disks
etc for the VMs, but I haven't yet restored the VM configs from my
backups (although there's now nothing stopping me from doing that, I
just haven't gotten around to it yet).

In the end I think my problems were caused by previously having the
deb-multimedia repositories in use and trying to move away from them at
the same time as upgrading. I don't know if there's a way to get away
from deb-multimedia once items have been installed from it, but now I
know that way is NOT to try to upgrade to the next Debian release, not
including deb-multimedia...

Although I do think some of the multimedia-facing packages in Debian are
(or used to be) configured in a less than optimal way (ffmpeg, I'm
looking at you) I now think rebuilding the relevant packages from source
might be a better approach than using deb-multimedia. I'm not sure what
the nature of the fight between Marillat and the rest of the Debian
community is (and I'm _very_ sure I don't want to wade into the middle
of it), but I think it's a pity he wasn't somehow able to be kept in the
fold because he does seem to know what he's doing when it comes to
multimedia tools. He was also very helpful some years back when I was
trying to debug a problem I was having with one of the tools (that's
long pre-stretch). But if there's no non-destructive way out of a
deb-multimedia-ised computer, it's probably best not to venture there...

Mark

Reply | Threaded
Open this post in threaded view
|

Re: Upgrade Adventure Stretch-to-Buster

Mattia Oss-3
Il 24/09/19 23:28, Mark Fletcher ha scritto:

> I perform 5-days-a-week backups using Amanda. The machine in question
> here happens to be the Amanda server as well as a client. The amanda
> virtual tapes and holding disk are in a USB3 disk cage. The cage also
> contains a RAID1 disk pair which contains data of various types.
[...]
> I ended up chasing one tree of
> dependencies all the way down to a conflict involving libstdc++6. By
> then I was so trigger-happy I force-removed that, and that was the end
> of using apt to do much of anything...

Wow, that escalated quicklyâ„¢! You went from super cautious to madman in
a heartbeat! :)

Why are you still using the repository of Marillat? I remember I used it
in the past but now? Are his packages better than the official?

I sort of had a similar problem with an external repository: I had
owncloud server installed, upgraded to buster only to discover that
owncloud 10.2.1 doesn't support php 7.3 (default on buster). So now the
web interface doesn't work anymore.. :( It's not a big problem because
syncing with the owncloud client still works, but still..
So if anyone has a stretch server with owncloud-files installed and are
planning to upgrade to buster: just don't! Wait for owncloud 10.3 to
come out.

Regards!


Reply | Threaded
Open this post in threaded view
|

Re: Upgrade Adventure Stretch-to-Buster

Charles Curley
On Wed, 25 Sep 2019 20:09:40 +0200
Mattia <[hidden email]> wrote:

> So if anyone has a stretch server with owncloud-files installed and
> are planning to upgrade to buster: just don't! Wait for owncloud 10.3
> to come out.

Or consider moving to NextCloud. I have an instance running quite well
on Debian 10 (buster), php7.3, 7.3.9-1~deb10u1.

--
Does anybody read signatures any more?

https://charlescurley.com
https://charlescurley.com/blog/

Reply | Threaded
Open this post in threaded view
|

Re: Upgrade Adventure Stretch-to-Buster

Bernd Gruber
In reply to this post by Mark Fletcher-2
Well, I really appreciate the deb-multimedia, but as I ran into similar
troubles times ago, I usually remove all non-official packages before
upgrade and reinstall them afterwards.

Greetings
Bernd

Reply | Threaded
Open this post in threaded view
|

Re: Upgrade Adventure Stretch-to-Buster

Andrei POPESCU-2
In reply to this post by Mark Fletcher-2
On Ma, 24 sep 19, 22:28:59, Mark Fletcher wrote:
>
> In the end I think my problems were caused by previously having the
> deb-multimedia repositories in use and trying to move away from them
> at the same time as upgrading.

Most likely, which is why the Release Notes specifically advise to
remove and non-Debian packages before the upgrade.

> I don't know if there's a way to get away from deb-multimedia once
> items have been installed from it, but now I know that way is NOT to
> try to upgrade to the next Debian release, not including
> deb-multimedia...

Sure it's possible, just uninstall the packages.

Kind regards,
Andrei
--
http://wiki.debian.org/FAQsFromDebianUser

signature.asc (849 bytes) Download Attachment