Time Immortal

Pursuant to the aforementioned domain controller issues, we were able to come up with a solution at work yesterday. To be fair, we haven’t yet gotten to the bottom of how it was that the domain controller apparently self-demoted in the first place, nor do we have any plausible explanation for why a laptop running Windows XP — and which has not been connected to the domain for months — would have been promoted to a domain controller role within Active Directory. We suspect that this may have had something to do with disabling a VPN between our servers and the domain controller for our U.S. branch (which is where the laptop also is), but we really are just grasping at straws.

Still, we managed to get everything working right again, which is the important part. What follows, O Reader, will be a long discussion of the specifics of the correction, and it will contain quite a lot of jargon specific to domain administration. If reading even just this disclaimer sentence was boring to you, trust me when I say that it is in your best interests to skip the rest of the article.

The biggest issue for us seemed to be that even though we could a) delete the offending laptop’s entry from the list of Domain Controllers in Windows Server 2003’s “Active Directory Users and Computers” snap-in, and even though we could re-add the server to this same list by name, its entry was incomplete (specifically, it did not have its ‘NTFRS Subscriptions’ or ‘RouterIdentity’ subfolders, nor did it have its ‘RID Set’). Additionally, while we could use the backup domain controller to seize the domain naming master role from the offending laptop, some of the other master roles wouldn’t allow themselves to be seized in the process. Basically, there was always some lingering presence of the offending laptop in Active Directory, in a role as a master or domain controller.

So let’s pause for a minute and assign some names, because the above is a slightly cumbersome way of saying things. Let’s assign the name DC1 to the demoted (and primary) domain controller, DC2 to the backup domain controller, and LAPTOP to the offending laptop.

In a fit of inspiration or desparation, we tried seizing all the master controller roles from DC1 using DC2, and LAPTOP remained in one of the master roles (RID, I believe, but don’t quote me on that). We rebooted DC1, and LAPTOP re-inserted itself into the Domain Controllers list in the “Users and Computers” snap-in. Figuring that we had essentially nothing to lose, we allowed this to stand for the moment, and manually re-inserted DC1 into the Domain Controllers list (as it had once again disappeared). We then moved the ‘NTFRS Subscriptions’, ‘RID Set’, and ‘RouterIdentity’ from the list entry for LAPTOP over to the list entry for DC1, and then deleted LAPTOP from the list.

This was, as I said, the inspired or possibly desperate move on our part.

We then rebooted DC1 again. Blessedly, LAPTOP did not once again re-insert itself into the Domain Controllers list, and the canonical names of the ‘NTFRS Subscriptions’ and ‘RouterIdentity’ folders changed to reflect their being moved between computers. DC1 remained in the Domain Controllers list, with its resources intact — the only oddity is that its listed role under the Properties window remained as “Workstation or server”, as opposed to “Domain Controller” (as yet, we’ve not figured out how to correct this).

However, we had noted that the bootup time for DC1 had been painfully long, and a quick look at the DNS snap-in revealed why — the DNS entries had been wiped out (this was likely related to the aforementioned inspired/desperate move). We deactivated DNS on the server, and then went to re-activate it, which generated an error. Specifically, the server could not create a forward lookup zone for itself, complaining that “the specified directory partition does not exist” (we used all the default options in the DNS Setup Wizard for the creation of a forward lookup zone).

I decided to Google that error message, which led us to this thread on Experts Exchange. The person seeking assistance there was having similar errors to the ones we had noticed in the Events snap-in, and we saw nothing to lose in attempting the proposed solution from Microsoft.

That proposed solution was to use the netdom.exe program to reset the Machine Account password of the domain controller, on the theory that the primary and backup domain controllers might not be able to communicate with one another properly, either again owing to our inspired/desperate move or to other factors entirely.

The specific syntax of the netdom.exe call was as follows:

This was run through the command window on DC1, and after it had successfully completed, we rebooted DC1. To our great relief (and considerable surprise), when DC1 came back up quickly, we found that all the DNS entries had been restored. What is more, the various problems we’d been having around the office as a result of LAPTOP being elevated to a domain controller role and DC1 being demoted (users and computers unable to access network resources, primarily) all seemed to have cleared up.

All of which is to say, I suppose, that we got extremely lucky. We were getting to the point where it seemed that the only viable options were either a) completing the demotion of DC1, attempting to clean out all the stray remnants of Active Directory from it (and cleaning it out of the Active Directory as well), and then re-promoting it (risky business, to be sure), or b) demoting DC1, backing up and wiping its drives, and then reinstalling Windows Server 2003, promoting the new, vanilla install to the primary domain controller role, and then restoring all the data, shared directories, and so forth (which we would have had to accomplish in less than the 12 hours we had before the 8 AM start to the workday).

I think our solution, which seems to have worked out just nicely, is preferable.