Bitcrush.io

Last month I started my new position as the "Director, Technical Operations" for Unknown Post (aka UKN Post), a boutique movie service company which specializes in creating Digital Cinema Packages for movies to be screened on high-end theater projectors. Where movie theaters used to receive massive reels of film prints from studios, now they typically receive a hard drive or a download containing a version of the movie exported according to the DCP specs. Usually these packages are also encrypted, and an .xml file called a Key Delivery Message (KDM) is generated using a certificate specific to an individual projector, which allows only that projector to play that specific copy of the movie for a certain amount of time.

To create these DCPs, we use a proprietary hardware/software combo called "Clipster," created by Digital Video Systems and now maintained by Rohde & Schwarz of Germany. I say proprietary because they only sell it as a full kit, but really it's a SuperMicro motherboard running Windows 10, plus a big RAID and some beefy video cards for real-time 4K playback of image sequences, .mxfs, etc.

The other day we needed to create a DCP for a sister company, and the source footage was on a big external G-RAID drive formatted as HFS+. Using MacDrive we were able to pull what we needed no problem, but when it came time to copy the DCP back to the G-RAID, we hit a snag. On reconnecting the device, Windows suddenly refused to recognize that it was even plugged in. Maybe a reboot?

We were greeted by the blue Windows logo on a black background, and nothing else. No spinning progress wheel, nothing, for several minutes. So we unplugged the G-RAID, rebooted: same system hang. OK, fine, let's unplug everything but keyboard/mouse/video: same hang. How about recovery mode? Automatic startup repair: failed. Thankfully, a clone of the OS drive had been made last year for just such an occassion. Popped open the case, swapped in the clone: success! Albeit on an OS a year out of date. Still, we got the files we needed off of the computer and headed home.

The following day, I tried to figure out what had happened to the original OS drive. Using another computer, I created a Windows 10 Recovery USB stick, plugged that in with the original OS drive in place and used the BIOS to boot to recovery mode. As before, the automatic startup repair failed, so I switched to Command Prompt and ran sfc /scannow. After about 40 minutes it finished and said it had fixed some problems, so I shut it down, unplugged the USB stick, and rebooted: same system hang as before. I guess I'll need to run some more commands? Plugged the USB stick back in, went to the BIOS, selected the USB... only to be greeted with the same system hang at the blue Windows logo?

A few more attempts didn't help, so I remade the USB stick on the other computer and tried again: no dice. OK, instead of the Media Creation Tool from Microsoft I'll just download the Windows 10 ISO myself and use Rufus: no dice. What the heck was going on? Using a Macbook, I tried switching the USB between MBR and GPT formatting followed by the ISO, but still no luck. Alright, so maybe the 64GB USB stick I'm using is just flakey for this sort of thing, I'll go home and grab a bundle of different USB sticks.

Absolutely none of them worked, and sometimes it froze even earlier at the POST screen with a "B2" in the lower corner. OK, I thought, maybe the BIOS isn't actually loading the USB at all and I'm just seeing the OS drive still hanging. I'll pop in the clone drive again and see if I can still get into recovery mode: same hang, oh no oh god. Unplugged the clone drive before any further damage might be done to it, and started wondering if maybe I had a motherboard-related hardware issue on my hands. I paused for a bit to think, and then did some searching on the motherboard model, error code "B2", and Windows boot problems. Lo and behold, an FAQ on the SuperMicro website from 2013:

If you are using a KVM switch for the monitor and USB keyboard/mouse, please remove it and plug the monitor directly to the motherboard. Plug the keyboard and mouse to the rear IO USB ports. Please try to boot it up again. B2 has to do with legacy devices, and sometimes the KVM might cause the system to be stuck on B2 at boot-up.

I was using a KVM the whole time! Following those directions, I plugged the recovery USB back in with the original OS drive and booted straight to recovery mode with no problem. After running sfc /scannow again to be sure, I also tried bootrec /fixmbr, bootrec /FixBoot, and bootrec /rebuildbcd. This last one gave me an error about a device not found, and the computer still wouldn't boot normally, so searching for that error led me to using diskpart in order to mark the OS partition as 'active'.

Unplugged the USB stick one last time, rebooted the computer and... a spinning progress wheel! Holy cow! After a brief 'diagnosing your problem' message, I was back in to our original OS with everything seemingly working. My hypothesis is that, somehow, the G-RAID HFS+ drive and MacDrive managed to confuse the Windows bootloader on the OS drive. Then, in the process of troubleshooting, the KVM USB connection started causing additional boot problems for the motherboard and causing me to question my sanity.

Since I spent a day and a half on this, I decided to write up my experience on the off-chance that some poor soul in 2033 comes across this post and finds it helpful.