Cloud Drive; Public Domain; openclipart.org
Cloud Drive; Public Domain; openclipart.org

I got myself a dedicated server at Hetzner from their “server auction” page.

I did set it up using their installimage. All went well, and the server was working fine, until an update to grub-efi-amd64-signed was released.

This is because the Hetzner installer seems to create an MS-DOS partition table, and uses old school MBR rather than GPT and ESP.

This is the error message:

~ # apt upgrade                                                                                                                                                                                               
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
Calculating upgrade... Done
The following packages were automatically installed and are no longer required:
  linux-image-5.15.0-25-generic linux-modules-5.15.0-25-generic linux-modules-extra-5.15.0-25-generic
Use 'apt autoremove' to remove them.
The following packages have been kept back:
  tzdata
0 upgraded, 0 newly installed, 0 to remove and 1 not upgraded.
1 not fully installed or removed.
After this operation, 0 B of additional disk space will be used.
Do you want to continue? [Y/n] y
Setting up grub-efi-amd64-signed (1.182~22.04.1+2.06-2ubuntu10) ...
mount: /var/lib/grub/esp: special device /dev/sda15 does not exist.
dpkg: error processing package grub-efi-amd64-signed (--configure):
 installed grub-efi-amd64-signed package post-installation script subprocess returned error exit status 32
dpkg: dependency problems prevent processing triggers for shim-signed:
 shim-signed depends on grub-efi-amd64-signed | grub-efi-arm64-signed; however:
  Package grub-efi-amd64-signed is not configured yet.
  Package grub-efi-arm64-signed is not installed.

dpkg: error processing package shim-signed (--configure):
 dependency problems - leaving triggers unprocessed
Errors were encountered while processing:
 grub-efi-amd64-signed
 shim-signed
needrestart is being skipped since dpkg has failed
E: Sub-process /usr/bin/dpkg returned an error code (1)

So, of course the package grub-efi-amd64-signed makes no sense and fails to install, since it has no ESP to install to, or so I thought.

Let’s take a look at the partition layout:

~ # lsblk -o +FSTYPE                                                                                                                                                                                    
NAME                                            MAJ:MIN RM   SIZE RO TYPE  MOUNTPOINTS FSTYPE
nvme0n1                                         259:0    0 476.9G  0 disk
├─nvme0n1p1                                     259:2    0     1G  0 part              linux_raid_member
│ └─md0                                           9:0    0  1022M  0 raid1 /boot       ext4
└─nvme0n1p2                                     259:3    0 475.9G  0 part              linux_raid_member
  └─md1                                           9:1    0 475.8G  0 raid1             crypto_LUKS
    └─luks-xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx 253:0    0 475.8G  0 crypt /           ext4
nvme1n1                                         259:1    0 476.9G  0 disk
├─nvme1n1p1                                     259:4    0     1G  0 part              linux_raid_member
│ └─md0                                           9:0    0  1022M  0 raid1 /boot       ext4
└─nvme1n1p2                                     259:5    0 475.9G  0 part              linux_raid_member
  └─md1                                           9:1    0 475.8G  0 raid1             crypto_LUKS
    └─luks-xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx 253:0    0 475.8G  0 crypt /           ext4

Indeed, no ESP (UEFI) partition… let’s take a look at the partition table of the first SSD:

~ # gdisk /dev/nvme0n1                                                                                                                                                                                        
GPT fdisk (gdisk) version 1.0.8

Partition table scan:
  MBR: MBR only
  BSD: not present
  APM: not present
  GPT: not present


***************************************************************
Found invalid GPT and valid MBR; converting MBR to GPT format
in memory. THIS OPERATION IS POTENTIALLY DESTRUCTIVE! Exit by
typing 'q' if you don't want to convert your MBR partitions
to GPT format!
***************************************************************


Command (? for help):

Indeed, just old school MS-DOS (MBR)…

However, interestingly:

~ # find /boot/efi                                                                                                                                                                                            
/boot/efi
/boot/efi/EFI
/boot/efi/EFI/BOOT
/boot/efi/EFI/BOOT/mmx64.efi
/boot/efi/EFI/BOOT/BOOTX64.EFI
/boot/efi/EFI/BOOT/fbx64.efi
~ # mount | grep boot                                                                                                                                                                                         
/dev/md0 on /boot type ext4 (rw,relatime)

It seems the Hetzner install image simply uses the ext4 /boot partition as UEFI boot (ESP) partition, too. (I, of course, thought, but this should be FAT32, what kind of mobo do they have?!)

So, of course, I tried to use parted, and set both boot partition to have the ESP flag, and indeed, GRUB then found the partitions to install to, and let me select them.

GRUB setup with RAID1
GRUB setup with RAID1

But still, no luck:

Setting up grub-efi-amd64-signed (1.182~22.04.1+2.06-2ubuntu10) ...
mount: /var/lib/grub/esp: unknown filesystem type 'linux_raid_member'.
dpkg: error processing package grub-efi-amd64-signed (--configure):
 installed grub-efi-amd64-signed package post-installation script subprocess returned error exit status 32
dpkg: dependency problems prevent processing triggers for shim-signed:
 shim-signed depends on grub-efi-amd64-signed | grub-efi-arm64-signed; however:
  Package grub-efi-amd64-signed is not configured yet.
  Package grub-efi-arm64-signed is not installed.

dpkg: error processing package shim-signed (--configure):
 dependency problems - leaving triggers unprocessed
Errors were encountered while processing:
 grub-efi-amd64-signed
 shim-signed
needrestart is being skipped since dpkg has failed

So, I reverted to setting just the raid flag, rather than the esp flag.

In the end, the solution was stupid simple.

~ # efibootmgr                                                                                                                                                        
EFI variables are not supported on this system.

Turns out, this isn’t a UEFI system at all…

/boot # apt list --installed '*grub*'                                                                                                                                 
Listing... Done
grub-common/jammy,jammy,now 2.06-2ubuntu7 amd64 [installed,automatic]
grub-efi-amd64-bin/jammy-updates,jammy-updates,now 2.06-2ubuntu10 amd64 [installed]
grub-efi-amd64-signed/jammy-updates,jammy-updates,now 1.182~22.04.1+2.06-2ubuntu10 amd64 [installed]
grub-efi-amd64/jammy-updates,jammy-updates,now 2.06-2ubuntu10 amd64 [installed]
grub-pc-bin/jammy,jammy,now 2.06-2ubuntu7 amd64 [installed]
grub2-common/jammy,jammy,now 2.06-2ubuntu7 amd64 [installed,automatic]

So, all the UEFI GRUB can be deleted… Note that if you run the following command on a UEFI system, it will stop booting.

apt purge shim-signed grub-efi-amd64 grub-efi-amd64-signed grub-efi-amd64-bin --allow-remove-essential
apt autoremove --purge

This solved all my issues.

I guess the Hetzner install image might have a bug because it makes no sense to install GRUB UEFI on a BIOS system, especially with no ESP.

This cost me way too much time (the dreadful realization that you can never get that, or any time, back in your mortal life). 😵‍💫