Bug#860833: 50mounted-tests hangs at dmsetup when using biosgrub partition on a left over fat32 partition

Previous Topic Next Topic
 
classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Bug#860833: 50mounted-tests hangs at dmsetup when using biosgrub partition on a left over fat32 partition

Marga Manterola
Package: os-prober
Version: 1.74
Severity: important

I'm using a recipe that includes this snippet:
              1 1 1 free                                      \
                $iflabel{ gpt }                               \
                $reusemethod{ }                               \
                method{ biosgrub }                            \

This snippet is present in the following automated recipes:

The biosgrub method does not touch the partition in any way. It leaves the contents as is.  If the installation is done in EFI mode, nothing is written to this BIOS partition. Which means that if there happened to be a fat32 partition that started on that sector (and was larger than 1 MB), the fat32 bits are still there after partman has run, in a way that confuses the lsblk command into thinking that the filesystem is "vfat"

This was always the case but it was not a problem with previous os-prober versions because there was no dmsetup being executed.  Now, with the new dmsetup commands, this triggers problems because the mount point enters a weird state.

This is the error in the log when it tries to mount the partition the first time:

Apr 20 08:30:12 50mounted-tests: debug: mounting /dev/mapper/osprober-linux-nvme0n1p1 (/dev/nvme0n1p1) as vfat failed: mount: mounting /dev/mapper/osprober-linux-nvme0n1p1 on /var/lib/os-prober/mount failed: Input/output error

The first time os-prober runs, it gets this error and it then claims to have removed the device mapper entry.  However, the next time os-prober runs (for some reason, during one d-i session, it runs 4 times) it hangs while running dmsetup create -r osprober-linux-nvme0n1p1

This hangs completely and never ends, a possible workaround is to kill the dmsetup command.  After that, the next 2 os-prober runs detect that there's a problem and don't hang.

I believe this is quite an important issue as any machine that has a previously formatted vfat will completely hang while installing. Before you claim no machine will have a previously formatted vfat: the affected machine is a new "Non Windows" machine coming from HP with FreeDOS installed.

After a lot of debugging with Philipp Kern, we were able to find out that dmsetup was hanging on a udev cookie semaphore.  The udev outside of the chroot is compiled udev_sync, and it doesn't have dmsetup rules, so it seems that it's not sending the dmsetup udevcomplete signal and thus dmsetup hangs forever. Another possible workaround then is to send the right dmsetup udevcomplete signal, and then installation proceeds.

It's not clear to me how the dmsetup code block would ever work without proper udev support outside the chroot.  It's also not clear why the first time it fails sort of gracefully and then hangs very ungracefully the second time.
--
Cheers,
Marga
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Bug#860833: 50mounted-tests hangs at dmsetup when using biosgrub partition on a left over fat32 partition

Philipp Kern-6
On 04/20/2017 08:34 PM, Marga Manterola wrote:
>> After a lot of debugging with Philipp Kern, we were able to find out
> that dmsetup was hanging on a udev cookie semaphore.  The udev outside
> of the chroot is compiled udev_sync, and it doesn't have dmsetup rules,
> so it seems that it's not sending the dmsetup udevcomplete signal and
> thus dmsetup hangs forever. Another possible workaround then is to send
> the right dmsetup udevcomplete signal, and then installation proceeds.

One small correction: The *dmsetup* outside of the chroot is compiled
*without* udev_sync as it's coming from the udeb. It also doesn't ship
with 55-dm.rules so it wouldn't call `dmsetup udevcomplete' even if it
were compiled with udev_sync.

Kind regards
Philipp Kern

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Bug#860833: 50mounted-tests hangs at dmsetup when using biosgrub partition on a left over fat32 partition

Marga Manterola
Adding some more data from logs and stuff now that I got the machine to finish installation successfully.

This is the first os-prober run:
Apr 21 08:35:48 os-prober: debug: running /usr/lib/os-probes/50mounted-tests on /dev/nvme0n1p1
Apr 21 08:35:48 50mounted-tests: debug: creating device mapper device /dev/mapper/osprober-linux-nvme0n1p1
Apr 21 08:35:48 kernel: [  953.534341] FAT-fs (dm-4): FAT read failed (blocknr 6158)
Apr 21 08:35:48 50mounted-tests: debug: mounting /dev/mapper/osprober-linux-nvme0n1p1 (/dev/nvme0n1p1) as vfat failed: mount: mounting /dev/mapper/osprober-linux-nvme0n1p1 on /var/lib/os-prober/mount failed: Input/output error
Apr 21 08:35:48 50mounted-tests: debug: remove device mapper device /dev/mapper/osprober-linux-nvme0n1p1

This is the second run:
Apr 21 08:35:53 os-prober: debug: running /usr/lib/os-probes/50mounted-tests on /dev/nvme0n1p1
Apr 21 08:35:53 50mounted-tests: debug: creating device mapper device /dev/mapper/osprober-linux-nvme0n1p1
Apr 21 08:38:58 init: starting pid 260, tty '/dev/tty2': '-/bin/sh'
Apr 21 08:40:41 50mounted-tests: debug: mounting /dev/mapper/osprober-linux-nvme0n1p1 (/dev/nvme0n1p1) as vfat failed: mount: special device /dev/mapper/osprober-linux-nvme0n1p1 does not exist

You can see that I logged in there and took a few minutes to gather data and then unblock it.  This is what I did (with some information obscured)

Trailing part of ps fax:

31842 ?  S+ 0:00   \_ /bin/sh /etc/grub.d/30_os-prober
31848 ?  S+ 0:00       \_ /bin/sh /etc/grub.d/30_os-prober
31849 ?  S+ 0:00           \_ /bin/sh /usr/bin/os-prober
31931 ?  S+ 0:00           |   \_ /bin/sh /usr/lib/os-probes/50mounted-tests /dev/nvme0n1p1
31938 ?  S+ 0:00           |       \_ /bin/sh /usr/lib/os-probes/50mounted-tests /dev/nvme0n1p1
31943 ?  S+ 0:00           |           \_ dmsetup create -r osprober-linux-nvme0n1p1
31850 ?  S+ 0:00           \_ tr   ^   
31851 ?  S+ 0:00           \_ paste -s -d  

$ dmsetup table 
osprober-linux-nvme0n1p1: 0 2048 linear 259:1 0
<LVM volume1>: 0 747470848 linear 253:0 117188608
<LVM volume2>: 0 117186560 linear 253:0 2048
nvme0n1p4_crypt: 0 998707200 crypt aes-xts-plain64 00000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000000 0 259:4 4096
<LVM swap>: 0 134045696 linear 253:0 864659456

$ ipcs -s 

------ Semaphore Arrays --------
key        semid      owner      perms      nsems     
0x0d4ddd11 0          root       600        1  

And then I unblocked the process by doing (inside the chroot):
$ dmsetup udevcomplete 0x0d4ddd11

These are the third and fourth os-prober runs:
Apr 21 08:42:49 os-prober: debug: running /usr/lib/os-probes/50mounted-tests on /dev/nvme0n1p1
Apr 21 08:42:49 50mounted-tests: debug: creating device mapper device /dev/mapper/osprober-linux-nvme0n1p1
Apr 21 08:42:49 50mounted-tests: debug: mounting /dev/mapper/osprober-linux-nvme0n1p1 (/dev/nvme0n1p1) as vfat failed: mount: special device /dev/mapper/osprober-linux-nvme0n1p1 does not exist

Apr 21 08:43:53 os-prober: debug: running /usr/lib/os-probes/50mounted-tests on /dev/nvme0n1p1
Apr 21 08:43:53 50mounted-tests: debug: creating device mapper device /dev/mapper/osprober-linux-nvme0n1p1
Apr 21 08:43:53 50mounted-tests: debug: mounting /dev/mapper/osprober-linux-nvme0n1p1 (/dev/nvme0n1p1) as vfat failed: mount: special device /dev/mapper/osprober-linux-nvme0n1p1 does not exist

I have a machine where I can reproduce this every time (since the partition never gets written to), so let me know if you want me to gather any additional information.

Thanks!

--
Cheers,
Marga
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Bug#860833: marked as done (50mounted-tests hangs at dmsetup when using biosgrub partition on a left over fat32 partition)

Debian Bug Tracking System
In reply to this post by Marga Manterola
Your message dated Mon, 01 May 2017 09:04:17 +0000
with message-id <[hidden email]>
and subject line Bug#860833: fixed in os-prober 1.75
has caused the Debian Bug report #860833,
regarding 50mounted-tests hangs at dmsetup when using biosgrub partition on a left over fat32 partition
to be marked as done.

This means that you claim that the problem has been dealt with.
If this is not the case it is now your responsibility to reopen the
Bug report if necessary, and/or fix the problem forthwith.

(NB: If you are a system administrator and have no idea what this
message is talking about, this may indicate a serious mail system
misconfiguration somewhere. Please contact [hidden email]
immediately.)


--
860833: http://bugs.debian.org/cgi-bin/bugreport.cgi?bug=860833
Debian Bug Tracking System
Contact [hidden email] with problems

Package: os-prober
Version: 1.74
Severity: important

I'm using a recipe that includes this snippet:
              1 1 1 free                                      \
                $iflabel{ gpt }                               \
                $reusemethod{ }                               \
                method{ biosgrub }                            \

This snippet is present in the following automated recipes:

The biosgrub method does not touch the partition in any way. It leaves the contents as is.  If the installation is done in EFI mode, nothing is written to this BIOS partition. Which means that if there happened to be a fat32 partition that started on that sector (and was larger than 1 MB), the fat32 bits are still there after partman has run, in a way that confuses the lsblk command into thinking that the filesystem is "vfat"

This was always the case but it was not a problem with previous os-prober versions because there was no dmsetup being executed.  Now, with the new dmsetup commands, this triggers problems because the mount point enters a weird state.

This is the error in the log when it tries to mount the partition the first time:

Apr 20 08:30:12 50mounted-tests: debug: mounting /dev/mapper/osprober-linux-nvme0n1p1 (/dev/nvme0n1p1) as vfat failed: mount: mounting /dev/mapper/osprober-linux-nvme0n1p1 on /var/lib/os-prober/mount failed: Input/output error

The first time os-prober runs, it gets this error and it then claims to have removed the device mapper entry.  However, the next time os-prober runs (for some reason, during one d-i session, it runs 4 times) it hangs while running dmsetup create -r osprober-linux-nvme0n1p1

This hangs completely and never ends, a possible workaround is to kill the dmsetup command.  After that, the next 2 os-prober runs detect that there's a problem and don't hang.

I believe this is quite an important issue as any machine that has a previously formatted vfat will completely hang while installing. Before you claim no machine will have a previously formatted vfat: the affected machine is a new "Non Windows" machine coming from HP with FreeDOS installed.

After a lot of debugging with Philipp Kern, we were able to find out that dmsetup was hanging on a udev cookie semaphore.  The udev outside of the chroot is compiled udev_sync, and it doesn't have dmsetup rules, so it seems that it's not sending the dmsetup udevcomplete signal and thus dmsetup hangs forever. Another possible workaround then is to send the right dmsetup udevcomplete signal, and then installation proceeds.

It's not clear to me how the dmsetup code block would ever work without proper udev support outside the chroot.  It's also not clear why the first time it fails sort of gracefully and then hangs very ungracefully the second time.
--
Cheers,
Marga

Source: os-prober
Source-Version: 1.75

We believe that the bug you reported is fixed in the latest version of
os-prober, which is due to be installed in the Debian FTP archive.

A summary of the changes between this version and the previous one is
attached.

Thank you for reporting the bug, which will now be closed.  If you
have further comments please address them to [hidden email],
and the maintainer will reopen the bug report if appropriate.

Debian distribution maintenance software
pp.
Ivo De Decker <[hidden email]> (supplier of updated os-prober package)

(This message was generated automatically at their request; if you
believe that there is a problem with it please contact the archive
administrators by mailing [hidden email])


-----BEGIN PGP SIGNED MESSAGE-----
Hash: SHA256

Format: 1.8
Date: Mon, 01 May 2017 09:55:33 +0200
Source: os-prober
Binary: os-prober-udeb os-prober
Architecture: source
Version: 1.75
Distribution: unstable
Urgency: medium
Maintainer: Debian Install System Team <[hidden email]>
Changed-By: Ivo De Decker <[hidden email]>
Description:
 os-prober  - utility to detect other OSes on a set of drives
 os-prober-udeb - utility to detect other OSes on a set of drives (udeb)
Closes: 853163 853927 860833
Changes:
 os-prober (1.75) unstable; urgency=medium
 .
   * Remove code using device mapper (Closes: #860833, #853927, #853163).
     This code doesn't work in d-i and it has some issues outside d-i. All
     architectures can use the grub-mount codepath, which was already the
     default anyway.
     This also removes the dependency on dmsetup.
Checksums-Sha1:
 f612317b7f5f5bfc916a1c228dfa1a41d45b531a 1738 os-prober_1.75.dsc
 a8d0a562324bf2d1076426e0e482611f9ebe4d9b 26216 os-prober_1.75.tar.xz
Checksums-Sha256:
 3042c501ba182580616417fd26a3934bf5fae3dc814bc4114f83c14f37ec9727 1738 os-prober_1.75.dsc
 f4ef620455c5ffc3545daf4f32861640a48b0b3b6edda72491eecc1818653446 26216 os-prober_1.75.tar.xz
Files:
 9e803f036da017d6e297d487dc47f941 1738 debian-installer optional os-prober_1.75.dsc
 acf4f8818af3cee051aa6f927a451e55 26216 debian-installer optional os-prober_1.75.tar.xz

-----BEGIN PGP SIGNATURE-----
Version: GnuPG v1

iQIcBAEBCAAGBQJZBvAmAAoJEKxAu1iXBOr8wasP/0wkzhxSkMk7JeV/5mQddD5Z
rMpV4gEtBi6BRruppTUDumi47ZSythcE1PRXzjZ30MR78WWymXV4FXg6H2x+5epr
vaVy6uDejIBzUDuQcCNYpY0p1K4WMUoIs83ac/2Bt4TDvP/eE3O+ye/CbLIH+RPc
EwW5cJC8q9hYb9DFdJP/2Wjsue1Ypbj9+as3D7yBMQ3qxx4hPK3n6DksObEq4oHP
OLD0Tq32iO1hhalcAE7k5cY7bgDtFXy6JHObBMzlFaak11FpQFpsZ83PTwh4O86q
0/8a6jtrok02C4RfJ1D5tuSFnuW/I2FVCIysTGJA84ZxN9T2vI5Ya1qwZyMHuDgY
4VZ3Pab/qpDWEfyFp4K1HXgbR2Tnnh+gHzc0UhEmkV+md/9dNESZZ+xjuSy0XEsA
Ojsv/eZreqodarifqtwNw0o2KEFSI1GyCdH+YKVDylfIA3lvU6lbim36LJVf1TSC
t9MuROjjznJc5OhqNF0+hC5SkENA/4dnnd5XxR/uxYTD8kMpW82rUOW7ttuGzAiR
3SYg+UECmeMwH2Ux46vpk5WnwLct1AjZxcoRjmc6gEH75UMxI63iVdIeHnfpzI41
QtFCWWd5ELRS2eJnm43q19Nr1hc0jtNG/1CVyGSq4rZE5JQOcOxsyVOhE6xYMz5X
VMjiZEL8/Ld39+VUu5Jf
=7LLE
-----END PGP SIGNATURE-----
Loading...