The SharePoint Swiss Army Knife

Here’s one that threw me for a doozy…

Scenario:

We had a 2 server dev farm (1 MOSS WFE/APP + 1 SQL Server) with the domain controller on the same server as the MOSS WFE server. One day the host OS (Windows Server 2008 ) decided to give us the big bad black Vista version of the BSOD. After a couple of days of not being able to recover from this we decided to just rebuild the machine and WFE since the content databases were securely stored on the SQL Server.

After rebuilding the Domain Controller and re-installing the MOSS WFE/App role on the server again (yeah, I know…but we were in a rush and didn’t have time to setup another WFE server) it worked fine for about a day. Then all hell broke loose with all kinds of password synching and credential store problems.

Some of the symptoms:

• There was an error encrypting or decrypting credentials. Either a credential update is currently being performed, or you must update the farm account credentials on this server before you can perform this task.

• SQL database login failed. Additional error information from SQL Server is included below. Login failed for user ‘DEV\mossservice’.
  
• stsadm -o updatepassword didn’t fix it
  
• stsadm -o upgradefarmcredentials didn’t fix it
  
• The app pool account kept reverting back to the previous account after IISResets

• The Timer Job Service kept reverting back to the previous account after IISResets

• Can’t retract, upgrade, deploy solutions
  

The Fix

So after much sweat, I found this post on the SharePoint forums that linked me to this hotfix (KB 957691). The first time I ran the Configuration Wizard after installing the hotfix, it got stuck on the upgrade part, I believe step 8. I then went and checked all things that ran off of a service account, made sure they all were using the correct account and ran the Config Wizard again. And ta-da! Everything magically works again.

Special thanks to Pav Cherny for his timely article in February 2009’s TechNet Magazine entitled Maintaining Security Account Credentials. It was very educational.