Warning When Multi-booting a UEFI System with Windows 10 (Fast Startup): when in doubt, reboot…

A Little History on PC Boot Methods

Microsoft Windows has never played nicely with other operating systems but there seemed to be a new peace beginning with Windows 10. Back in the early days of PCs (1983), IBM PC-DOS 2.0 introduced the Master Boot Record (MBR) disk partitioning scheme. Microsoft licensed PC-DOS to IBM and then released their own branding of MS-DOS. In 1985 Microsoft released a graphical extension to MS-DOS they named Windows, and the MBR scheme has been with us ever since.

The MBR method was tightly coupled with the Basic Input/Output System (BIOS) firmware found on all IBM PC-Compatible personal computers. In the late 90s hardware had reached a point where hacking the BIOS/MBR systems was becoming ridiculous and Intel began work on a new method. Intel dubbed this new scheme Extensible Firmware Interface (EFI). In 2005 Intel ended EFI, rebranded it Unified Extensible Firmware Interface (UEFI), and handed over development of UEFI to a non-profit corporation which is represented by a number of leading technology companies.

With the release of Windows 8 in 2012, Microsoft certification required that computers must ship with a UEFI implementation. By 2015 UEFI was being widely adopted by most motherboard manufacturers. Now, in 2017, I doubt you could even buy a new motherboard without UEFI.

The MBR disk partitioning scheme has become outdated. It can not even support disks larger than 2 TiB. Back when MBR was originally developed (1982), the new PC hard drive was a whopping 10 MB and a 2 TB drive seemed a dream even for super computers. The first 1 terabyte drive was released in 2007 and 2009 saw the first 2 terabyte drive. A badly needed new partitioning scheme was developed with UEFI. Intel dubbed the new standard, the GUID Partition Table (GPT). Globally unique identifiers (GUID) are 128 bit numbers used to identify information in computer systems, such as filesystems.

Most modern operating systems now support a UEFI/GPT boot and partition scheme. This includes macOS, Windows, Linux, and FreeBSD. UEFI includes a special boot partition called the EFI system partition (ESP). UEFI is designed to be OS agnostic, which is good news for those of us who like to use open source operating systems.

This brings us to our current dilemma of trying to multi-boot Windows 10 with, in my case, Arch Linux. My current system multi-boots between Windows 7, Windows 10, Arch Linux, Ubuntu, and at times has also had Android-x86, and OSx86 installed on it.

When in doubt, reboot...

My #1 motto in computer support since the 80s has been, "When in doubt, reboot." I have found that just rebooting, or sometimes power cycling your computer, will fix 95% of computer issues. The reason for this is simple, computers are buggy. Resetting systems back to their initial state will usually, if only temporarily, remove the bug. But with Windows 8, the geniuses at Microsoft changed this long standing rule.

They came up with something they dubbed "Fast Boot" which they later changed to "Fast Startup". On the surface Fast Startup seems like a reasonable idea. It is a sort of pseudo hibernate feature in that it saves certain parts of the system memory to disk. From the Microsoft Developer Network (MSDN);

Windows closes all applications and logs off all user sessions. At this stage, the system state is similar to that of a computer that has just started up—no applications are running, but the Windows kernel is loaded and the system session is running. Next, the power manager sends system power IRPs to device drivers to tell them to prepare their devices to enter hibernation. Finally, Windows saves the kernel memory image (including the loaded kernel-mode drivers) in Hiberfil.sys and shuts down the computer.

The idea being, that when Windows starts up again, it will load the hiberfil.sys file which will save the time of firing up all the kernel and device driver states. Sounds good right? Well, I think there are a number of unintended consequences.

The first time I noticed something was amiss, I was troubleshooting a problem with a Windows 10 box at a client site. I was looking at the Task Manager and noted that the machine had been up for more than a 100 days! So, following the adage, "when in doubt, reboot", I shut the system down and powered it back up. When I pulled the Task Manager back up, the system had still been up for more than a 100 days! How was this possible as I had just power cycled the machine?!?

My guess at what happened here is that the system uptime is stored in the kernel memory. Thus the uptime was restored when I did a Fast Startup of Windows. The problem is, that whatever was causing my client's system to misbehave, was also restored by the Fast Startup. In the end, a system restart fixed the problem. Microsoft has changed the long standing rule, because a shutdown does a hybrid hibernate, whilst a restart  clears everything and starts from scratch.

The moral of the story is that if you are using Fast Startup, then you need to do a restart notshut down to reset your Windows operating system.

Windows 10 "Fast Startup" and the EFI Partition

So, how does all of this come into play when multi-booting a UEFI system? A while back, and I can't recall exactly when this began, but one morning I went to boot up my computer and begin my workday when I got a terrifying error.

error: invalid cluster 0.

There were a few more lines about broken magic and an ominous "Press any key to exit." That was it, Linux wouldn't boot. Windows booted just fine. This brought up long standing fears of the dreaded "sector 0 error" that meant your hard drive was toast.

Too make a long story short, I eventually discovered that my EFI partition had been corrupted by Windows! Windows, never seems to be bothered by this corruption but Linux is another matter. Sometimes, I lose my initramfs files, sometimes I lose my vmlinuz-linux kernel image, sometimes nothing critical is corrupted and I can boot to a USB image and repair the vfat EFI partition file system and I everything is OK.

I don't know why, but I do  know that Windows, when using "Fast Startup", and multi-booting is going to corrupt your EFI partition. It has happened to me too many times now. So, I have disabled "Fast Startup" on all my multi-boot systems that use Windows. So, imagine my surprise, when after months of no issues, corruption started happening on my workstation again.

From the Arch Linux Wiki;

Your system can lose data if Windows hibernates and you dual boot into another OS and make changes to files. Even if you do not intend to share filesystems, the EFI System Partition is likely to be damaged on an EFI system. Therefore, you should disable Fast Startup.

I like to early adopt Windows updates so that I can figure out any issues before my clients start calling me. I recently updated to the Windows 10 Creators Update, which most people will be getting this Fall. After the the third time that my EFI partition was corrupted recently, I checked my Windows 10 Fast Startup settings. Sure enough Fast Startup had been re-enabled! After a little research, it looks like this happened to people on the Windows 10 Anniversary Update as well. Microsoft, if you are reading this, please stop enabling the Fast Startup on major updates. If we turned it off, we had a good reason.

Conclusion

If you are multi-booting with Windows, turn off "Fast Startup" and be prepared to turn it off again after a major Windows update.

Interestingly, when I mounted my Windows 10 "root" drive in Linux, I found the following files;

-rwxrwxrwx  1 root root 6.4G Jul  2 14:27  hiberfil.sys
-rwxrwxrwx  1 root root 2.4G Jul  2 14:27  pagefile.sys
-rwxrwxrwx  1 root root  16M Jul  2 14:27  swapfile.sys

So despite having hibernation, Fast Startup, etc. all turned off, Windows is still creating these files. Nearly 9 gigabytes of files! These files do not show up under Windows File Explorer even with show hidden files enabled. What's even worse is that the files are nothing but NULL characters. I tried deleting them, but Windows just creates the files again. Perhaps, Windows is reserving space on the filesystem and using it when Windows is running?

From what I understand it should be possible to have multiple EFI System partitions. I might try mirroring two EFI partitions in case one gets corrupted. But this is a project for another day on a system that I can live without on a daily basis. I don't see why you couldn't backup your EFI partition and restore it from a live USB as well.

What I have been doing to repair my system in the mean time is to boot from an Arch installation image copied to a USB stick. I then fsck my EFI partition, fix the errors, and then mount my root, var, and boot filesystems and arch-chroot to the new/old environment. I then install the last kernel from my /var/cache/pacman/pkg directory. Then exit the arch-chroot environment, umount everything and reboot.

I would love to hear from you about your experiences with multi-booting on a UEFI system.

 

My hacking career began at age 12, when my father brought home a Sinclair ZX-81 computer kit and a lifelong fascination with computing was born. I became a privacy and security advocate after learning of the US Government's criminal investigation against Phil Zimmermann. Years later I found myself in Silicon Valley being paid to hack Phil's PGP code. I soon tired of the rat race and moved to the Northwoods of Minnesota where I have over 2 million acres of protected wilderness to roam. Here I indulge in my passions for wilderness and technology and as an advocate for computer security and privacy.

Posted in Technical.

My hacking career began at age 12, when my father brought home a Sinclair ZX-81 computer kit and a lifelong fascination with computing was born. I became a privacy and security advocate after learning of the US Government's criminal investigation against Phil Zimmermann. Years later I found myself in Silicon Valley being paid to hack Phil's PGP code. I soon tired of the rat race and moved to the Northwoods of Minnesota where I have over 2 million acres of protected wilderness to roam. Here I indulge in my passions for wilderness and technology and as an advocate for computer security and privacy.