Fixing boot sector and MBR

Two tools you’ll ever need to make your disk bootable: mbrfix and bootsect.

MbrFix allows you to read and write the Master Boot Record of a hard disk drive (or anything that resembles one). The thing to know about MBR is that it consists of static and dynamic data. Static data is something you can take from one hard drive in one computer and write to another hard drive in another computer, and it’ll still work. The MBR boot code is like that. As long as it’s used in x86-compatible systems, it’ll work. (Don’t nitpick). Dynamic data, on the other hand, is specific to a hard drive on which it’s written. It includes drive signature and partition table, which stores among other things offsets and lengths of drive partitions.

Obviously, blindly writing the whole MBR onto the hard drive would result in a rewritten partition table, and, consequently, all partitions lost. (Probably). MbrFix does this the clever way, preserving dynamic and reading/writing only static data. In addition to reading/writing the data you feed to it, mbrfix includes standard MBRs for NT/XP, Vista and Windows 7 in it, so you can do “/fixmbr” with it just like you would from recovery console of any of those systems. It also allows you to get/set active partition and the aforementioned drive signature (how about 0xBADC0FEE). So it contains basically anything you might ever need from MBR editing tool. Aside from editing the partition table manually. And from hex editing the drive. No, wait, you can save MBR into file, edit it and write it back.

Download MbrFix here, in case you couldn’t figure that out from the link above.

MbrFix writes bootsectors too, but it’s not good at that so we’ll use another tool. This time it’s bootsect.exe from Microsoft. It works on both FAT and NTFS partitions and contains both pre-Vista (ntldr) and post-Vista code. The only thing it lacks is Win9x code (the one for io.sys, msdos.sys).

Naturally, boot code alone is not enough to actually boot. You’ll need to place the loader itself on the partition in order for it to be bootable. For pre-Vista systems such as XP Embedded it’s NTLDR, which consists of ntldr.exe, ntdetect.com and boot.ini. If you’re building XPE, these will be included into your image automatically; else take them from any XP distro you have around, they’re all (almost) the same.

You can get bootsect.exe from several sources, including Windows Automated Installation Kit and Vista installation DVD (it should be in \boot\Bootsect.exe according to this article). Or you can google for it and dowload it from somewhere else; don’t worry, it doesn’t require Vista and runs on older NT systems just fine.

Grpconv is missing

When you install inf files, either from the right-click menu in the explorer, or from the command-line, you might get a cryptic error message which reads “Windows cannot find grpconv.exe” or something similar. What is this and what does this mean?

Turns out, grpconv.exe is the utility to convert Windows 3.1 Program Manager “groups” into Windows 9x and later Explorer menu folders. Raymond Chen explains that in those days many programs misused the “groups” feature, creating groups on installation manually, instead of using program interfaces. So, with the conversion to “menu folders”, an application was written to clean up after such irresponsible installers and convert their hand-made “groups” into “menu folders”. Thus it was named “grpconv”.

Since inf files are basically “installation information”, Windows suspects that they might be installers for those older programs. Even though nobody uses them anymore, Windows cannot be sure, so it runs grpconv just to be safe. Just in case your inf file actually was an installer for such a program (there’s no rule you cannot install windows 3.1 apps on your XPE image, right? ;).

And, if you excluded grpconv from your image (which is often the case; it’s in “Base Support Binaries”, btw), Windows fails to find the conversion tool and, consequently, fails the installation process. Not only this is annoying, it also intervenes with installing inf files.

The solution is plain and simple: make an empty file “grpconv.cmd” in your Windows\System32 folder. Empty file is a valid script file which does nothing, and, lucky for us, Windows does not specify an extension when calling grpconv, so this replacement works just fine. Now, every time you install inf file manually, you’ll see a flashing black box marking the end of installation.

Installing inf files from the command-line

You probably know you can install inf files by right-clicking them in explorer and choosing “Install”. This does not always work. This kind of installation works only on inf files that have a “DefaultInstall” or “DefaultInstall.NT” section. Sometimes inf files do not have such section and their main installation section is called somewhat different (for example, “[ProductName].Installation”). Sometimes inf files have more than one such section, for instance, they might have different install scenarios for different platforms (“XP_INSTALL”, “9X_INSTALL”). They might also include uninstall scenarios (“[ProductName].Uninstall”), optional components (“[componentname]”, Microsoft does this in Windows), et cetera.

This Microsoft article describes how to install inf files from command line. In short:

RunDll32 setupapi.dll,InstallHinfSection <section> <reboot-mode> <inf-name>

(please note I’ve changed RunDll to RunDll32 and setupx to setupapi; the original article applies to 9x, whilst my version works on nt/xp. Also to install inf files you might need to solve the grpconv problem).

Here “inf-name”  is the name of the inf file you’re trying to install. Remember to specify a full path (“E:\install\redist\dx9\dxnt\dxnt.inf”) or prepend the relative path with “.\” (“.\dxnt.inf”), or the command will do nothing apart from displaying a cryptic message “Installation failed”.

“reboot-mode”  is described in documentation. Usually you want to set it to 0, which means “never reboot”, just remember to reboot manually later. Also, as noted in documentation, you need to add 128 to whatever value you choose, if your inf file is not a standard system inf. Usually files you want to install aren’t standard inf files, so type “128” instead of “0”.

“section” is the name of the section in the inf file which we want to install. To determine which section you need, open inf in the text editor, and look through the sections. These can be of different formats; some contain registry modifications, others list files. We need those which look something like:

[SectionName]
CopyFiles=Subsection_1,Subsection_2,Subsection_3
DelFiles=Subsection_4
AddReg=Subsection_5

Choose the section you need from these, judging by their name (you have no reliable way of understanding their meaning other than that, or simply studying what they copy and where). If the file has a “DefaultInstall” section, this might be the one you need (if you just want a generic “installation” and not something specific).

For those interested, here’s the inf file structure explanation from Microsoft.

Installing DirectX 9.0c on XPE

There’s a relatively well-known problem with DirectX 9.0c installer that manifests itself on some XP installations, including, maybe, your XPE image. The setup tool runs just fine, but when you click “Next” nothing happens. Installer window blinks for a moment, then displays “Installation complete. Installed components are ready to use”.

No general solution exists for this kind of problem because it might be caused by a different reasons. The simplest case is where you already have this or later DirectX version installed (you might have accidentally included “DirectX Registry Data” component in your configuration). To trick setup into thinking you don’t have DirectX yet, delete the following registry key:  HKLM\Software\Microsoft\DirectX. Actually, it’s enough to delete just subkeys “Version” and “InstalledVersion”, but if this is XPE image you probably have nothing else there anyway.

This does not always help. If it doesn’t, next thing to make sure of is that your DirectX redistributable is accessible by a simple file path. “Simple” means no spaces, no unicode symbols, preferably 8.3 folder and file names (yeah, this is most likely NOT required in any way, but making them 8.3 costs you nothing, so what’s the difference). For example, “E:\redist\dx9” is a simple path, and “E:\Application Installation Redistributables\Microsoft ® Windows © DiræctX 9.0c ” is a bad idea.

(When talking about DirectX redistributable, obviously I mean an unpacked local distributable. Web setup is another story which I’m not inclined to discuss since rarely anybody uses web setup on embedded systems, anyway. If you have a single-file setup, run it, type in a path and it’ll extract a bunch of files there. These files are the redistributable I’m talking about).

Another thing to remember is that DirectX 9.0c setup has an implicit dependency upon user.exe and ntkrnlpa.exe. If you’re working on your own XPE build, you have probably cut them down. User.exe is from “Dos Windows on Windows Support” component, which provides “Core architecture that is required to run 16-bit applications”. Who needs them, anyway. So you most likely don’t have neither user.exe nor ntkrnlpa.exe (most of XP distros don’t have the latter either). Theoretically these two are not explicitly required, but in practice you can try putting them in the setup folder and see if that helps.

Next come the obvious steps you would (be supposed to) do in such case. DirectX setup writes a log to “WinDir\DirectX.log”. Inspect it to see if there’s anything of help (usually not the case). Fire procmon and protocol the whole setup procedure to see if there’s any suspicious activity going on. Usually setup just loads dxsetup32.dll, which then enumerates audio devices, makes a few requests to ntkrnlpa.exe and user.exe, if they’re present, writes several messages into log, and that’s all. Run depwalker and inspect dxsetup.exe to check if there are any unresolved dynamic dependencies (for one, you need advpack.dll to unpack cabs, though this dependency is explicit and you’ll simply see an error when you run dxsetup.exe).

The above steps might help, but the chances are low. In some, if not in the majority of cases, neither procmon nor depwalker help, as the setup process seems to be going perfectly normal judging from their data. So we’re back at square one. What to do? Well, there are two ways of installing DirectX 9.0c if the installer is acting weird. One is more straightforward, but the other is cleaner and has better results. I’ll describe both.

First, you can use DirectX 9.0b setup to install DirectX 9.0c files. For that, download DirectX 9.0b redistributable from the internet (you’ll have to google for it a bit, since Microsoft does not seem to host it anymore). Ensure that what you have is indeed 9.0b and not 9.0c since some web pages tend to replace the underlying file with a new version without modifying the link description. Then, delete all the *.cab files from the 9.0b redist and replace them with *.cab files from the 9.0c redist. Delete the registry data as described in this post earlier, then run 9.0b setup and it’ll happily install 9.0c.

Well, it turns out, not so happily, because dxdiag will report some files being of older version than needed. I did not investigate this and my guess is that you can safely ignore this message because it’s probably caused not by files being outdated, but by dxdiag thinking wrong about which version they should be. Still, the warning is there, and this kinda makes me uneasy. So we will try installing 9.0c another way.

Open 9.0c redist. In there you see a bunch of *.cab files. These files contain:

• BDA.cab – BDA for Windows 9x
• BDANT.cab – BDA for Windows NT, excluding XP
• BDAXP.cab – BDA for Windows XP
• DirectX.cab – DirectX for Windows 9x
• dxnt.cab – DirectX for Windows NT, including XP
• ManagedDx.cab – Managed DirectX extensions (for any Windows family)

BDA stands for Broadcast Driver Architecture and is related to video capture. You might decide to leave it out if you don’t care about video capturing, or to install it with everything else. By the way, DirectX installer is smart enough to let you delete the cab files you don’t need. For example, if you’re planning on installing only native DirectX and only on XP systems, you’re fine to include only “dxnt.cab”, which will significantly lessen the size of your redistributable.

So, let’s say we want to install DirectX only on our XPE image. (BDA and DirectX for 9x are handled in a similar way, and ManagedDx contains a single msi installer which is even simpler to handle). Unpack dxnt.cab. It contains a bunch of things, including several *.inf files, which is what we need. These inf files contain all the information about DirectX installation. Unfortunately, not all of them have the DefaultInstall section, so we’ll need to go through the procedures of installing them from the command-line. For that, we need setupapi.dll and rundll32.exe (as usual, if they’re not in your image, you can just place them in the same folder with DirectX):

RunDll32 setupapi.dll,InstallHinfSection <section> <reboot-mode> <inf-file>

Here comes the trickiest part. As I noted before, often inf files have the “DefaultInstall” section which is triggered by the right-click install command. If an inf file has that section, just write “DefaultInstall” as the section name for it. If it doesn’t, you’ll have to look into it a bit and determine which section it is that you want. For example, in several DirectX inf files main section is called something like “[ComponentName].Install”, for instance, “KD.Install”, or something like that. In other files there are several main sections for different platforms. In this case you need the one for XP (i.e. “WINXP_INSTALL”), or, the next closest, for NT. These may be named arbitarily, there’s no convention dictating how to name them, so you’ll have to decide which section you need for yourself.

To simplify installation, write a command file:

@echo off
echo Installing DirectX 9.0c, please wait...
RunDll32 setupapi.dll,InstallHinfSection DefaultInstall 128 .\a.inf
RunDll32 setupapi.dll,InstallHinfSection DefaultInstall 128 .\b.inf
RunDll32 setupapi.dll,InstallHinfSection WINXP_INSTALL 128 .\c.inf
RunDll32 setupapi.dll,InstallHinfSection NTFamilyProductInstall 128 .\d.inf

Mention every file in that way, except for DxDllReg.inf. That file needs special treatment. Also don’t install ks.inf, kscaptur.inf and ksproxy.inf; these will be copied into your system’s inf directory. Yet, install ksreg.inf.
Now, add the following lines to the end of your script:

RunDll32 setupapi.dll,InstallHinfSection DXDllUnRegister 128 .\dxdllreg.inf
RunDll32 setupapi.dll,InstallHinfSection DXDllRegister 128 .\dxdllreg.inf

This way we re-register everything.

Run the script, wait for it to finish, ensure that there were no errors and start dxdiag. All the files are of the correct version now. DirectX 9.0c is installed!

Old New Thing

For the sake of completeness again, here’s a link to Raymond Chen’s Old New Thing. No doubt, everybody who’s at least remotely close to working with Windows internals or involved with system programming, already knows his blog well.

KM Blog

KM Blog contains a lot of useful tips and tools for dealing with XP embedded and external debugging. Unfortunately, it seems to be dead for quite a while already, but older posts are worth reading nevertheless.

Dependency Walker

Another irreplaceable tool in the developer’s and admin’s collection is the Dependency Walker. This application inspects the executable or library you feed to it, and constructs a tree of library dependencies, including statically linked libraries and even dynamically linked ones.

The uses for it are obvious, but I’ll give one just for example. Imagine you have an application which fails to start on your embedded image because it cannot find some dll, say, “a.dll”. What do you do? Well, for starters, you copy “a.dll” from the working XP home or pro, put it into the application directory and try running it again to check that this fixes it. Unfortunately for you, it does not. Now the application requires “b.dll”, which you too copy and put into the application directory. And then the application wants “c.dll”, and so on. This process might continue for half-a-hour.

With depwalker you just analyze the executable file and write down a list of all missing libraries. What’s better, if you see that the lot of them are required by the plugin library you never actually use, you might as well delete that greedy plugin and your application will happily run on your present image.

Well, there’s the chance that some of those missing libraries will require another libraries in turn, which will trigger step one endless cycle again. This can’t be helped, but at least you’ll resolve a lot more dependencies in one go, instead of solving them one-by-one.

Tools of the trade

Debugging precompiled applications comes near impossible without these handy tools in your digital pocket. They’re so well-known I feel no real need to introduce them, yet I still list them here just for the sake of completeness:

• Process Monitor – monitors the application and operating system activity, including registry access, file system reads and writes, dll loads and so on. Allows you to specify a variety of filters to strip down unneeded information; supports logging boot. Basically, this is the main tool one wants to use when solving application errors. When used properly, procmon helps to solve up to 70% of all problems you might stumble upon. Process Monitor requires XP SP2, for older operating systems there are FileMon and RegMon.
• Process Explorer is a task managing program, which provides you with ten times more functions than standard task manager does. For example, you can see which files the process holds open, which handles to kernel objects it owns, which DLL it has loaded.  Another must-have tool.
• DebugView – small program that catches the messages other apps output through either user-mode OutputDebugString or kernel-mode DbgPrint. In practice apps rarely output any valuable information to debug ports when they crash, but the chance is still there, so in the absence of the debugger be sure to try DebugView.
• WinDbg from the Debugging Tools for Windows. This is the debugger of choice for operating system debugging. It has various other uses too, for it’s lightweight, does not require installation and has few dependencies. WinDbg is especially handy when debugging applications you don’t have a source code for (99% of all application problems you’ll encounter fall in this category).

In general, practically anything you’ll find in SysInternals suite is worth downloading and getting familiar with. I will show examples of the uses for some of those utilities later.