HackMii

note: If you haven’t already, you should probably go back and read these posts, because they were written in preparation for this one:

• Wii Hardware: a history
• IOS: history, build process

A few months back, we started getting reports of “unsoftmoddable Wiis”, aka “LU64+” (among other things). Normally, I wouldn’t care, but we discovered that our HackMii Installer would not work on any of those Wiis. I started making the claim that this was due to an innocuous hardware change, coinciding with the release of boot2v4, but I never really explained why. Here’s my explanation.

“LU64+” is a really silly way to describe these Wiis — it refers to the first four characters in the serial number of the first Wiis seen to be like this — but I can’t really blame people, because the serial number is the only real visible marking as to when the console was manufactured.

Starting around early 2009 (?), the following statements were true about all new purchased Wiis:

1. All versions of IOS have the strncmp() bug fixed, access to /dev/flash blocked, and “ES_Identify” was broken.
2. boot1 had strncmp() bug fixed
3. crap installed into the slots for IOS3, IOS4 and IOS254

Beginning in March, a fairly well-defined set of symptoms of running some kinds of homebrew software emerged on “newish” Wiis:

1. Old (but valid) versions of IOS would not run at all; they would hang if you tried, which prevented downgrading IOS. This did not apply to the leaked IOS16.
2. Attempts at reloading IOS by libogc would sometimes fail; depending on the program, this would look like a hang or would just cause all IOS calls to fail

There seemed to be no difference in the installed software on one of these “newish” Wiis, compared to what you would get by doing an update through the System Menu.
People proposed theories such as “Nintendo added special hack-detection code into boot1 / boot2, but they put a ‘back-door’ in for IOS16.” As I’ve explained in the past, boot1 and boot2 are finished executing long before you see anything on your screen, and certainly don’t affect anything past that. Rather, boot2 is responsible for loading IOS, and IOS is responsible for loading the PowerPC code for the system menu or game. If you insert a game disc that uses a different version of IOS, the currently running IOS is responsible for loading and running the new IOS, which then loads the PPC code, and maybe eventually another IOS to get back to the system menu, etc. (This exact order of steps becomes important later.)

This was a curiosity I mostly ignored, because it didn’t affect any of our software … until we released the HackMii Installer.

The reason we created one common installer for BootMii, the Homebrew Channel and DVDX was that it became increasingly complicated to install software on updated Wiis. Different approaches were required, depending on the configuration of the Wii, and it became apparent that duplicating our decision-making and exploit code across multiple installers would be silly. We ended up coming up with some code that would examine the state of the Wii it was running on, pick an appropriate strategy, and then automatically install the desired software.

There are a couple of different possible paths for the Installer to use — if it can, it will just use an old IOS that still has the unpatched hash comparison function and unfettered access to /dev/flash (reloading to that IOS as necessary). If not, it would choose one of a list of IOS exploits we had found, depending on the versions of IOS installed (each exploit needs to be customized to a specific major/minor version of IOS), and use it to load MINI into memory and execute it. From there, we could directly access the NAND flash and detect the installed version of boot1 and boot2. Finally, having made the decision of whether or not we could allow BootMii to be installed as boot2, we would reload back into a normal IOS so that we could access the WiiMote. Of course, reloading back into IOS isn’t exactly trivial; MINI has a function that launches an arbitrary title from NAND by reading boot2 from the beginning of NAND, patching the call to ES_LaunchTitle(1-2) to the desired title ID, and then executing the patched boot2.

All of this happened automatically and hopefully seamlessly, but it could take a few seconds. Having been burned by the pay-for-homebrew wankers, we have an “anti-scam warning screen” that we make the user sit through before installing our software. The delay there made for a great opportunity for us to do our leet haxIOS magic, so we ran that “in the background” while the user sat through our nag screen.

This didn’t work as well as we had hoped on some new Wiis — suspiciously, it would reboot in the middle of the nag screen on “unsoftmoddable Wiis”. Suddenly this became an interesting problem to me!

Unfortunately, we’re all old farts with old Wiis. None of us could reproduce the problem ourselves – we tried to get people with troubled Wiis to run test code for us, but you really need a USBGecko to be able to get debug output from low-level code. People who were new to the Wii were the least likely to have a USBGecko for testing, so … the only way we could proceed was to track down one of these “unsoftmoddable” Wiis.

I finally did so, and ran a debugging version of the installer on it. I was surprised to see that it was actually getting all the way through our set of exploits, and was actually dying when it tried to reload back to IOS by way of boot2. It turned out that these new Wiis had a new version of boot2 — boot2v4 — and our “patch boot2 to reload IOS instead of the system menu” routine was failing to find the code to patch. Instead of giving an error message, it was just executing the unmodified new boot2 — which then re-ran the system menu instead of proceeding with our installer. Oops.

That was certainly easy enough to fix — I added a patch for boot2v4 — and the installer seemed to successfully reload IOS, only to hang inside __IOS_InitializeSubsystems() in libogc (specifically, ios_open(“/dev/es”)). This started to match the reported failures in libogc programs during IOS_Reload().

svpe said he had run into some timing problems when working on booting back and forth between IOS and MINI, and suggested I try adding a delay. When I added a delay of 5 seconds into our code between when we asked MINI to reload IOS, and when we actually reinitialized libogc, then everything started work. I found this rather odd, since the code had worked just fine with boot2v3, and it also worked on boot2v4 if we reloaded to the System Menu (since that’s what had been happening to people when our boot2 patch failed).

After another sleepless night or two, it became clear that boot2v4 actually took somewhat longer to load than boot2v2/3. If I checked the IOS version number right before calling __IOS_InitializeSubsystem(), it would report back the correct IOS version on boot2v3, but 0 on the boot2v4 system. This indicates that boot2 was still running — boot2 sets that version number to 0 when it starts.

The answer to this problem seemed to be simple enough — we just needed to add some code that would work like the following:

1. Send IPC to MINI asking for it to boot into IOS
2. delay until IOS version number is set to 0 (meaning, boot2 has fully loaded and is now executing)
3. delay until IOS version is the expected IOS version
4. proceed to initialize IOS

Again, doing this made a little difference, but not a good one. Now we no longer froze inside of __IOS_InitializeSubsystems(), but all of our IOS called failed once we booted into the installer — and they failed with strange error codes like “-900074497”. svpe then pointed out that the IOS load process looks something like this:

1. boot2 starts
2. boot2 sets IOS version number to 0
3. boot2 executes es_main()
4. es_main calls ES_LaunchTitle(1-2) (or in our case, IOSxx since we patched this parameter)
5. ES_LaunchTitle recognizes that we are trying to load an IOS, and then sets up the IOS version number in memory so that the newly-loaded IOS will know what version it is
6. ES_LaunchTitle transfers control to the new IOS
7. IOS starts and sets up the IPC registers to talk to the PPC

We were hitting a timing race between when ES_LaunchTitle changed the IOS version number, and when IPC actually started. If we tried to make IPC calls to the Starlet before the version number changed, then we would actually be trying to make IPC calls to boot2 while it was in the middle of rebooting — so we’d never get a response and libogc would hang. If we waited until after the version number was set, libogc would try to talk to the new IOS before IPC was initialized; as part of that initialization, IOS would send back a bogus “ack” message that made libogc think that the ios_open(“/dev/es”) call failed, and then it would never properly initialize ES and all later calls would fail. If we waited in our own code for this “ack” message (as suggested by isobel), then everything else worked.

Armed with this knowledge, I was able to add some timing code to our installer, and get the following numbers:

On old hardware:
Time to load boot2v3: 958 ms
Time to execute through boot2v3 and launch IOS: <1 ms
Time to initialize IPC: <1ms

On new hardware:
Time to load boot2v4: 134 ms
Time to execute through boot2v4 and launch IOS: 178 ms
Time to initialize IPC: 499 ms

Thus, we had our solution … more or less … to the problem of boot2v4 and the HackMii Installer — we just needed to add some delays. So why is the timing so much different on the new Wii, and why doesn’t this affect normal operations of the Wii?

Let’s summarize the facts:

• Wiis recently began shipping with a new boot2v4, which is based on the same code present in the set of IOSes released last October
• Reloading IOS is often “broken” on new Wiis — especially when trying to reload to an “older” version of IOS (pre-October, taken from an older Wii)
• Launching PPC titles works just fine on new Wiis, even if that requires reloading to a different version of IOS (as happens when launching the Homebrew Channel
• The same versions of IOS work differently on older Wiis, and have none of the problems noted above. The only difference in the software loaded on these Wiis is in boot1 and boot2.
• The “leaked IOS16” seems to be exempt from these problems

I think I can explain all but the last point.

From previous research, we can add the following tidbits of knowledge:

• boot1 and bc have each gone through 4 revisions; of those, 1 was to fix the strncmp bug and 3 were to change low-level hardware initialization routines
• The strncmp bug fix first appeared in IOS37 in March, 2008; IOS, bc, and boot1 all apparently were fixed internally at Nintendo / BroadOn in Feb-Mar. 2008, but not released until a few months later
• bc (and presumably boot1) was rebuilt with more minor hardware init changes in June 2008
• new versions of boot2 and all of the IOSes were built on July 11th, 2008
• The new versions of IOS were put up on the Nintendo Update Servers in October
• Wiis began shipping with the new IOSes (but with boot2v4) in late 2008
• Two PCB revs were made at the end of 2008/beginning of 2009 — all of these Wiis shipped with the “new” IOSes and boot2v4, and do not seem to work well with older versions of IOS
• The only substantial changes made in the PCB were to power-supply circuitry; Hollywood (where IOS runs) was unchanged (still at “Hollywood AA”), and there may have been some inconsequential changes made to the Broadway

A scenario begins to emerge:

1. Nintendo develops a new PCB design to reduce cost by simplifying circuitry in mid-2008; the resulting design has fewer parts but takes a few milliseconds longer for the power supply voltages to stabilize
2. The new hardware requires a more-tolerant IOS kernel, so BroadOn rebuilds boot2 and IOS for Nintendo do use in testing the new PCB design. These IOS versions are still backwards-compatible with C/RVL-CPU-01.
3. Nintendo releases the new IOSes as an update in late 2008, to quash the strncmp() bug (as well as /dev/flash access and ES_Identify()
4. Nintendo starts producing C/RVL-CPU-01 boards with the new IOSes on them. (boot2 is not updated, since there’s no benefit to doing so)
5. Nintendo eventually starts mass-producing C/RVL-CPU-20, which requires an updated boot2(v4) in order to work, along with the new IOS versions which are already well-tested by this point.
6. Consumers receive these new C/RVL-CPU-20 Wiis. Some of them try installing old versions of IOS on them and then running libogc code that uses them
7. libogc doesn’t know that it needs to wait longer for IOS to reload; depending on the timing, this can either cause a hang or it can cause IOS initialization to fail.

I imagine that Nintendo never even realized this would be a problem. Here’s how the system normally boots:

1. System turns on
2. boot0 / boot1 / boot2 run on Starlet
3. boot2 loads the TMD for the System Menu, reads the required IOS version, then chainloads to that version of IOS
4. Newly loaded version of IOS loads System Menu from NAND into memory, then turns on PPC and starts it executing
5. User selects channel or disc; IOS loads TMD, reloads into new IOS
6. IOS loads game from NAND or disc into RAM, then bootstraps PPC

Here’s what happens when someone runs some USB loader crap from the Homebrew Channel:

1. User selects HBC from system menu; menu calls ES_LaunchTitle(00010000-HAXX)
2. IOS reads HBC’s TMD from NAND, sees that it needs IOS61 (or whatever), and reloads to it
3. IOS61 initializes hardware, then loads the HBC content into RAM and bootstraps PPC
4. User selects warezloader from SD card
5. HBC loads ELF from SD into PPC, and then jumps to it directly
6. ELF calls (e.g.) IOS_Reload(249) to load patched version of IOS
7. libogc makes IPC calls to IOS to reload IOS, but does not wait long enough for the process to finish, and then becomes confused

In the normal Nintendo scenario, IOS is always the one reloading itself, and then the new version of IOS loads the PPC and starts the code. There is never any PPC code executing while that process happens, and the PPC code can’t possibly start until the ARM is ready. In contrast, the process used by libogc is “backwards”, with the PPC reloading the ARM code; insuffcient synchronization code is used to prevent this race condition because the old Wiis reloaded so quickly that it was never an issue.

Here’s what happens when someone runs E.G. a “forwarder channel” from the System Menu — a banner with a dummy TMD that loads a (probably patched) older version of IOS, and then launches something else:

1. User selects “forwarder channel” from system menu; menu calls ES_LaunchTitle() on title
2. IOS reads channel’s TMD from NAND, sees that it needs (old) IOS36 (or whatever), and reloads to it
3. (old) IOS36 fails to properly initialize the hardware, and the system hangs

This is not a bulletproof explanation. I can’t explain:

• Why the leaked IOS16 still worked
• What specific functions are needed in an IOS for it to work properly on a new Wii — the bootup process of IOS is multithreaded and difficult to follow, and I no longer have a Wii to test this on, and honestly I don’t care THAT much
• Why, specifically, the new code is necessary
• Why the latest version of the HackMii Installer still doesn’t work on all boot2v4 Wiis, even with the added synchronization code in place

Even still, I think this is much better explanation than “Nintendo did something magic to make unsoftmoddable Wiis”. Don’t get me wrong — I’m sure there were some smiles at Nintendo when they saw that hacks stopped working — but Nintendo’s anti-hacker efforts are generally far less subtle — for example, blocking /dev/flash and fixing the IOS exploit we used before in the HBC installer.

If you, the reader, would like to play around with this, I wrote a little test program, which you can download source and binary for here: reloadtest.zip. It’s mostly a version of the libogc IOS_Reload code, but with some debugging output (to the TV) and some timing code. It’s not exactly the same as the code I talked about above — boot2 is not involved with the process — but those of you with access to Wiis with boot2v4 (and for whom the HackMii installer doesn’t work) might be able to gain some insight as to what’s going wrong. Feel free to change the two versions of IOS (“from” and “to”).

When I run it on my Korean Wii, I get numbers like:

stats: id=0x06bcc32d boot2v3 IOS46->IOS22 IOSticks=32 IPCticks=475
stats: id=0x06bcc32d boot2v3 IOS46->IOS20 IOSticks=32 IPCticks=1041

On my old NTSC Wii (which originally came with boot2v2), I get:

stats: id=0x0408cafa boot2v3 IOS35->IOS22 IOSticks=32 IPCticks=497
stats: id=0x0408cafa boot2v3 IOS35->IOS20 IOSticks=32 IPCticks=1052

I’d be interested to hear if people see substantially different results on different hardware.