The problem is that the window mediator returns NULL.
That calls getMostRecentWindow on nsIWindowMediator. This calls getMostRecentBrowserWindow, which calls getMostRecentBrowserWindow in nsBrowserGlue.js. I traced the failure to handURIToExistingBrowser, which is a js function in nsBrowserContentHandler.js. This should work, and it does after app startup. In the new code, we use nsICommandLineRunner and have it run with a '-url' argument. In the old apple event handling code, for kAEGetURL, we manually looked up a window by enumerating by z order via nsIWindowMediator.