Fedora31: grub2 issues with grub2-mount using 100% CPU

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|

Fedora31: grub2 issues with grub2-mount using 100% CPU

Terry Barnaby
I have just started to try out Fedora31 on some of our systems.

I am using a bit of an unusual, manual install method copying an image
of the rootfs to the disk and configuring this, (this may be related to
my issue, but I have been using the same system for 6 years or more).
The particular hardware platform is a bit complicated using a PCIe card
with NVMe disk and two SATA disks. It has to boot (BIOS boot and /boot)
from one of the SATA disks and then uses the rootfs system on the NVMe
disk. I don't think that is related to the the issue though.

The issue is that when I run "grub2-mkconfig -o /boot/grub2/grub.cfg" to
manually configure grub2, this process hangs. There is a process
grub2-mount that is sitting there using 100% of a CPU core. Not sure
what this process does (seems new after Fedora29) but it is passed the
/dev entry for a disk partition some presumably mounts the file system
and "probes" for what OS is installed there. The file system it hangs on
(if you kill the grub2-mount process it will hang on another drive), is
easily mountable without any issues.

It seems like there is some bug in this grub2-mount program. Has anyone
else seen this or have any ideas what might be the issue.
_______________________________________________
users mailing list -- [hidden email]
To unsubscribe send an email to [hidden email]
Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: https://lists.fedoraproject.org/archives/list/users@...
Reply | Threaded
Open this post in threaded view
|

Re: Fedora31: grub2 issues with grub2-mount using 100% CPU

Terry Barnaby
On 12/12/2019 06:33, Terry Barnaby wrote:

> I have just started to try out Fedora31 on some of our systems.
>
> I am using a bit of an unusual, manual install method copying an image
> of the rootfs to the disk and configuring this, (this may be related
> to my issue, but I have been using the same system for 6 years or
> more). The particular hardware platform is a bit complicated using a
> PCIe card with NVMe disk and two SATA disks. It has to boot (BIOS boot
> and /boot) from one of the SATA disks and then uses the rootfs system
> on the NVMe disk. I don't think that is related to the the issue though.
>
> The issue is that when I run "grub2-mkconfig -o /boot/grub2/grub.cfg"
> to manually configure grub2, this process hangs. There is a process
> grub2-mount that is sitting there using 100% of a CPU core. Not sure
> what this process does (seems new after Fedora29) but it is passed the
> /dev entry for a disk partition some presumably mounts the file system
> and "probes" for what OS is installed there. The file system it hangs
> on (if you kill the grub2-mount process it will hang on another
> drive), is easily mountable without any issues.
>
> It seems like there is some bug in this grub2-mount program. Has
> anyone else seen this or have any ideas what might be the issue.
>

A bit more digging. It seems the issue is seen within os-prober.
Tracking this down:
I am running Fedora29 with the 5.3.11-100.fc29.x86_64 kernel.

I have a Fedora31 rootsfs on the partition /dev/nvme0n1p2 and have
chroot'ed to this (with some /sys, /dev mounts etc). (Fedora31 would be
running the vmlinuz-5.3.15-300.fc31.x86_64 kernel).
Within the os-prober it effectively does the following at some point:
grub2-mount /dev/nvme0n1p2 /var/lib/os-prober/mount
ls /var/lib/os-prober/mount/lib*/lib*.so*

If I run:
mount /dev/nvme0n1p2 /var/lib/os-prober/mount
time ls /var/lib/os-prober/mount/lib*/lib*.so* > /dev/null
real    0m0.089s
user    0m0.028s
sys     0m0.044s

If I run:

grub2-mount /dev/nvme0n1p2 /var/lib/os-prober/mount
time ls /var/lib/os-prober/mount/lib*/lib*.so* > /dev/null
real    0m59.593s
user    0m0.190s
sys     0m0.254s

grub2-mount appears to be a FUSE driver to mount various file system
types. I am guessing that there is either a bug in grub2-mount or there
is a kernel level incompatibility (FUSE API) with the slightly older
5.3.11-100.fc29.x86_64 kernel or it is very very very very inefficient.
A minute of 100% of a CPU core to list a directory on a fast NVMe SSD!
Talk about bloatware and what about climate change folks ?

I guess this my issue in that grub2-mkconfig will take a "really" long
time on my system with largely populated rootfs (It has all of the RPM
package for development that I use so is about 28 GBytes) and this
system has an SSD and two SATA disks with multiple partitions and
operating systems.


Terry

_______________________________________________
users mailing list -- [hidden email]
To unsubscribe send an email to [hidden email]
Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: https://lists.fedoraproject.org/archives/list/users@...
ja
Reply | Threaded
Open this post in threaded view
|

Re: Fedora31: grub2 issues with grub2-mount using 100% CPU

ja
On Thu, 2019-12-12 at 10:01 +0000, Terry Barnaby wrote:

> On 12/12/2019 06:33, Terry Barnaby wrote:
> > I have just started to try out Fedora31 on some of our systems.
> >
> > I am using a bit of an unusual, manual install method copying an image
> > of the rootfs to the disk and configuring this, (this may be related
> > to my issue, but I have been using the same system for 6 years or
> > more). The particular hardware platform is a bit complicated using a
> > PCIe card with NVMe disk and two SATA disks. It has to boot (BIOS boot
> > and /boot) from one of the SATA disks and then uses the rootfs system
> > on the NVMe disk. I don't think that is related to the the issue though.
> >
> > The issue is that when I run "grub2-mkconfig -o /boot/grub2/grub.cfg"
> > to manually configure grub2, this process hangs. There is a process
> > grub2-mount that is sitting there using 100% of a CPU core. Not sure
> > what this process does (seems new after Fedora29) but it is passed the
> > /dev entry for a disk partition some presumably mounts the file system
> > and "probes" for what OS is installed there. The file system it hangs
> > on (if you kill the grub2-mount process it will hang on another
> > drive), is easily mountable without any issues.
> >
> > It seems like there is some bug in this grub2-mount program. Has
> > anyone else seen this or have any ideas what might be the issue.
> >
>
> A bit more digging. It seems the issue is seen within os-prober.
> Tracking this down:
> I am running Fedora29 with the 5.3.11-100.fc29.x86_64 kernel.
>
> I have a Fedora31 rootsfs on the partition /dev/nvme0n1p2 and have
> chroot'ed to this (with some /sys, /dev mounts etc). (Fedora31 would be
> running the vmlinuz-5.3.15-300.fc31.x86_64 kernel).
> Within the os-prober it effectively does the following at some point:
> grub2-mount /dev/nvme0n1p2 /var/lib/os-prober/mount
> ls /var/lib/os-prober/mount/lib*/lib*.so*
>
> If I run:
> mount /dev/nvme0n1p2 /var/lib/os-prober/mount
> time ls /var/lib/os-prober/mount/lib*/lib*.so* > /dev/null
> real    0m0.089s
> user    0m0.028s
> sys     0m0.044s
>
> If I run:
>
> grub2-mount /dev/nvme0n1p2 /var/lib/os-prober/mount
> time ls /var/lib/os-prober/mount/lib*/lib*.so* > /dev/null
> real    0m59.593s
> user    0m0.190s
> sys     0m0.254s
>
> grub2-mount appears to be a FUSE driver to mount various file system
> types. I am guessing that there is either a bug in grub2-mount or there
> is a kernel level incompatibility (FUSE API) with the slightly older
> 5.3.11-100.fc29.x86_64 kernel or it is very very very very inefficient.
> A minute of 100% of a CPU core to list a directory on a fast NVMe SSD!
> Talk about bloatware and what about climate change folks ?
>
> I guess this my issue in that grub2-mkconfig will take a "really" long
> time on my system with largely populated rootfs (It has all of the RPM
> package for development that I use so is about 28 GBytes) and this
> system has an SSD and two SATA disks with multiple partitions and
> operating systems.
>
>
> Terry
>
Maybe

https://bugzilla.redhat.com/show_bug.cgi?id=1744693

John


_______________________________________________
users mailing list -- [hidden email]
To unsubscribe send an email to [hidden email]
Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: https://lists.fedoraproject.org/archives/list/users@...
Reply | Threaded
Open this post in threaded view
|

Re: Fedora31: grub2 issues with grub2-mount using 100% CPU

Terry Barnaby
On 12/12/2019 12:06, ja wrote:

> On Thu, 2019-12-12 at 10:01 +0000, Terry Barnaby wrote:
>> On 12/12/2019 06:33, Terry Barnaby wrote:
>>> I have just started to try out Fedora31 on some of our systems.
>>>
>>> I am using a bit of an unusual, manual install method copying an image
>>> of the rootfs to the disk and configuring this, (this may be related
>>> to my issue, but I have been using the same system for 6 years or
>>> more). The particular hardware platform is a bit complicated using a
>>> PCIe card with NVMe disk and two SATA disks. It has to boot (BIOS boot
>>> and /boot) from one of the SATA disks and then uses the rootfs system
>>> on the NVMe disk. I don't think that is related to the the issue though.
>>>
>>> The issue is that when I run "grub2-mkconfig -o /boot/grub2/grub.cfg"
>>> to manually configure grub2, this process hangs. There is a process
>>> grub2-mount that is sitting there using 100% of a CPU core. Not sure
>>> what this process does (seems new after Fedora29) but it is passed the
>>> /dev entry for a disk partition some presumably mounts the file system
>>> and "probes" for what OS is installed there. The file system it hangs
>>> on (if you kill the grub2-mount process it will hang on another
>>> drive), is easily mountable without any issues.
>>>
>>> It seems like there is some bug in this grub2-mount program. Has
>>> anyone else seen this or have any ideas what might be the issue.
>>>
>> A bit more digging. It seems the issue is seen within os-prober.
>> Tracking this down:
>> I am running Fedora29 with the 5.3.11-100.fc29.x86_64 kernel.
>>
>> I have a Fedora31 rootsfs on the partition /dev/nvme0n1p2 and have
>> chroot'ed to this (with some /sys, /dev mounts etc). (Fedora31 would be
>> running the vmlinuz-5.3.15-300.fc31.x86_64 kernel).
>> Within the os-prober it effectively does the following at some point:
>> grub2-mount /dev/nvme0n1p2 /var/lib/os-prober/mount
>> ls /var/lib/os-prober/mount/lib*/lib*.so*
>>
>> If I run:
>> mount /dev/nvme0n1p2 /var/lib/os-prober/mount
>> time ls /var/lib/os-prober/mount/lib*/lib*.so* > /dev/null
>> real    0m0.089s
>> user    0m0.028s
>> sys     0m0.044s
>>
>> If I run:
>>
>> grub2-mount /dev/nvme0n1p2 /var/lib/os-prober/mount
>> time ls /var/lib/os-prober/mount/lib*/lib*.so* > /dev/null
>> real    0m59.593s
>> user    0m0.190s
>> sys     0m0.254s
>>
>> grub2-mount appears to be a FUSE driver to mount various file system
>> types. I am guessing that there is either a bug in grub2-mount or there
>> is a kernel level incompatibility (FUSE API) with the slightly older
>> 5.3.11-100.fc29.x86_64 kernel or it is very very very very inefficient.
>> A minute of 100% of a CPU core to list a directory on a fast NVMe SSD!
>> Talk about bloatware and what about climate change folks ?
>>
>> I guess this my issue in that grub2-mkconfig will take a "really" long
>> time on my system with largely populated rootfs (It has all of the RPM
>> package for development that I use so is about 28 GBytes) and this
>> system has an SSD and two SATA disks with multiple partitions and
>> operating systems.
>>
>>
>> Terry
>>
> Maybe
>
> https://bugzilla.redhat.com/show_bug.cgi?id=1744693
>
> John
>
>
> _______________________________________________
> users mailing list -- [hidden email]
> To unsubscribe send an email to [hidden email]
> Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
> List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
> List Archives: https://lists.fedoraproject.org/archives/list/users@...

Thanks John.

Looks like the last bits of that bug are related. I had already added
bug:  https://bugzilla.redhat.com/show_bug.cgi?id=1782773 for the slowness.
As a hack, if you change to use "mount" instead of "grub2-mount" then
grub2-mkconfig all seems to work ok and is fast:
mv /usr/bin/grub2-mount /usr/bin/grub2-mount-orig
ln -sf mount /usr/bin/grub2-mount

Terry
_______________________________________________
users mailing list -- [hidden email]
To unsubscribe send an email to [hidden email]
Fedora Code of Conduct: https://docs.fedoraproject.org/en-US/project/code-of-conduct/
List Guidelines: https://fedoraproject.org/wiki/Mailing_list_guidelines
List Archives: https://lists.fedoraproject.org/archives/list/users@...