What a mighty doozy this one was… a couple weeks, a few hairs and 7.4 panic attacks later, I think I’ve had one of the most twisted SharePoint issues I’ve ever had to deal with. In my 6 years of working with SharePoint I’ve only had to open up a support case with Microsoft one other time. I take great pride in being able to solve stuff on my own but this was just one of those that had me going in circles. Hopefully this write up will help others in the future.
Environment
- SharePoint 2010 Enterpise December 2011 Cumulative Update 14.0.6114.5000
- 3 Load Balanced WFEs
- 2 Application Servers
- 1 User Profile Service Application
- User Profile Service and User Profile Synchronization Service both running on APP1 server
- User Profile Service Application has 4 Custom User Profile Properties and 1 Property set to export to Active Directory (Picture to thumbnailPhoto)
Since there were quite a few variables in troubleshooting the problems (which I haven’t mentioned yet), I’ll outline all the happenings in a timeline format.
February 2012: Users start to use My Sites to upload their pictures
We’re synching these pictures to Active Directory so that they can be re-used for the users’ Lync and Outlook profile pictures.
February 11, 2012: Deployed December 2011 Cumulative Update
This is a story for another day but in the end everything turned out OK. Lessons learned here: be wary of synchronizations of the User Profile Service after troubleshooting UPA issues after deploying CU. Make sure you disable the “My Site Cleanup Job” and follow the guidance from Joanne Klein here. I learned this the hard way.
February 16, 2012: Installed some Windows Updates
Everything seemed to be normal here.
February 22, 2012: First Reports of Users Not Seeing Their Pictures in Lync & Outlook
Upon inspection of Active Directory and comparing to pictures in SharePoint, there was indeed a mismatch. Photos has not been exported to AD since February 17, 2012. Uh oh, could it have been the Windows Update? Maybe a weekly Timer Job somewhere that regressed from the December 2011 CU? Maybe a combination of both? Errors reported in the FIM MIIS Client and Event Viewer pasted below:
FIMSynchronizationService Event ID 6126
The management agent “MOSS-63649d6d-ab5f-4eda-8c6a-6e2b65a419c7” completed run profile “MOSS_DELTAIMPORT_51f827d4-b836-4851-89de-daf209327762” with a delta import or delta synchronization step type. The rules configuration has changed since the last full import or full synchronization.
User Action
To ensure the updated rules are applied to all objects, a run with step type of full import and full synchronization should be completed.FIMSynchronizationService Event ID 6801
The extensible extension returned an unsupported error.
The stack trace is:“System.Net.WebException: The remote server returned an error: (404) Not Found.
at System.Net.WebClient.DownloadDataInternal(Uri address, WebRequest& request)
at System.Net.WebClient.DownloadData(Uri address)
at Microsoft.Office.Server.UserProfiles.ManagementAgent.ProfileImportExportExtension.DownloadPictures(ProfileChangeData[] profiles)
at Microsoft.Office.Server.UserProfiles.ManagementAgent.ProfileImportExportExtension.Microsoft.MetadirectoryServices.IMAExtensibleFileImport.GenerateImportFile(String fileName, String connectTo, String user, String password, ConfigParameterCollection configParameters, Boolean fFullImport, TypeDescriptionCollection types, String& customData)
Forefront Identity Manager 4.0.2450.34″
After trying to troubleshoot this for a whole day with no leads, I threw in the towel and phoned MSFT support.
February 23-27, 2012: Working with MSFT SharePoint Support Engineer Trying to Restore the UPA
In short, we worked on trying to create new UPAs in production as well as staging with restored Profile and Social Databases to no avail. There was one restore scenario that seemed to work when we left it on the Friday, but when I came in on Monday, the sync errors were happening with the new UPA as well. I also learned that there are quite a bit of URLs that are hard coded into the User Profile and Social Databases. For example, when we restored the databases to the QA environment, users in the QA environment were getting redirected to their Production My Sites. I once again thew in the towel with this Support Engineer when he suggested that there will be no way to recover the User Profile and Social DBs and that his seniors recommend that all of my users will have to recreate all of their profile information. This was completely unacceptable. By the way, if you find yourself on a support call with MSFT or anyone else for that matter, don’t always be so willing to do whatever they recommend. There were quite a few instances where I had to disagree with a troubleshooting step as it would have made my production environment unavailable to my users or result in data loss. So try to use your own best judgement and common sense when working with someone that doesn’t have to deal with your end users.
February 28-March 1, 2012: Waiting Around for Call from Escalation Engineer… But Had Some Revelations
Was supposed to hear back from someone after 48 hours, but did not. Instead, I had to go through some other routes to get some attention via my Technical Account Manager (TAM). In the meantime, I was sleuthing around in regards to the 404 error message and discovered that there was something awry with some of the user profile pictures. I recorded this finding/bug here. I didn’t have a chance to validate that the FIM Sync errors are related to the bad CMYK pictures, but that was the hunch…
March 2, 2012: Got another Support Engineer and… success! Sort of…
We spent about 4 hours in total and eventually reached a semi-conclusion. So this whole entire time, a lot of attention was being paid to the UPA as that was the most probable cause for failed syncs with Active Directory. This time, instead of spending too much time trying to recreate and restore the UPA in various stages, I was able to change the troubleshooting direction to focus more on the user profile pictures instead. With this lead, the Support Engineer suggested that we remove the Picture mapping to Active Directory and then perform a Full Synchronization. Before running the Full Sync, I made mention that the last time I did this, all of the profiles got deleted. After disabling the My Site Cleanup Job, we ran the Full Sync and were indeed able to observe that all the user profiles were marked for deletion using
Select * from userprofile_full (nolock) where bDeleted = 1
on the User Profile Database. That was pretty nerve-racking. We then proceeded to run a few more syncs to confirm that the user profiles were flipped back to the do not delete state.We also confirmed that there were no more sync errors. Woohoo! User profile pictures were definitely the problem, causing the FIM sync to fail.
Resolving the bad CMYK pictures problem
This seems to be a bug with SharePoint and the workaround I’ve found is to delete the offending thumbnail (large thumbnail) generated by SharePoint and then replace it with the medium thumbnail (which works). You can follow that thread here. After resolving the picture issue, I was then again able to successfully export all the user profile images from SharePoint to Active Directory. So in retrospect, if there was an exception in the sync because of one of these images, FIM will tap out and not even attempt to export any other pictures that are working to Active Directory.
Recap & Lessons Learned
- User Profile Service Application and FIM Sync issues do not always require a rebuild of the UPA.
- If you rebuild your Sync DB or Connection to Active Directory, you will lose all of your Profile Property Mappings.
- If you rebuild your Sync DB or Connection to Active Directory, your next sync (either incremental or full, first incremental will force a full) will result in all of your profiles getting marked for deletion.
- To prevent your User Profiles from getting deleted, disable the My Site Cleanup Timer Job.
- Don’t believe the Support Engineer when he says it’s not possible to restore the Profile and Social DBs (YMMV).
- Don’t perform recommended actions that may cause downtime or loss of data. Use your common sense and don’t jeopardize your users’ data.
- You know your environment the best and sometimes you have to go with your gut on an issue. Having a second pair of eyes and helpful suggestions was definitely appreciated but if I had let the Support Engineers continue their scripts, we would still be trying to recreate and restoring the UPA to no avail.
The SharePoint Swiss Army Knife 
Nicest post!
zee
http://walisystemsinc.com
To find out if you have bad picture urls simply connect to your sharepoint database, find your profiles database and run this query.
SELECT RecordID, NTName, PictureUrl
FROM UserProfile_Full
fix the bad urls and update any that are blank with NULL
UPDATE UserProfile_Full
SET PictureUrl = NULL
WHERE (PictureUrl = N”)
informative.
I am having a similar issue. But when I go through the logfiles it seems to occur for some users that don’t even have a picture uploaded into their profiles. I have been resolving this by uploading a blank generic picture and then the import completes with no errors. Then a day or two later the same thing occurs. How can I fix. Isn’t querying the database not allowed and then won’t my environment become unsupported.
Great post, I suspect I’m having the same issue. I rebuilt the UPS, it worked fine on pre-production but then pre-prod environments don’t tend to get the same picture activity. Even more odd three of my test accounts work, but for a couple of prod users I am using a test cases, the pictures in AD get deleted. I restored the UPS using this article http://www.toddklindt.com/blog/Lists/Posts/Post.aspx?ID=300, works like a charm. At least some pictures are going into AD now. I’ll try your post and report back here but it’s certainly looking like exactly the same issue.
P.S I’ve been through 5 escalation engineers at MSFT and two consultants, so I feel your pain, and this has been ongoing for six months. I feel I’m close, I have one or two hairs left I can rip out should the need arise.
Hi Henry, thanks for this post.
We also started receiving this error very recently. We hadn’t had any issues previously. We’d recently deployed the 2012 August CU to our SharePoint farms and then suddenly this problem started popping up. In our case it turned out to be an issue introduced by the CU which Microsoft have subsequently released a Hotfix/CU for:
http://support.microsoft.com/kb/2786675
Your post was great for the initial trouble shooting. Hopefully the support link I’ve added here may help others also.
Cheers
Pete
I concur with Pete; great article. I got me on the right track of trying to resolve this issue. We also applied the hotfix (http://support.microsoft.com/kb/2786675) …
However the errors didn’t seem to go away (The remote server returned an error: (404) Not Found.) … As my team and I were scratching our heads one member suggested that I install fiddler on the server that runs our profile sync service … Duh! … 404 errors are HTTP related and fiddler would catch that … Low and behold … fiddler let me know which items (pictures) had bad URL references; not sure how that happened but we removed them from the UI (Manage Profiles) and ran the process again … Clean as a whistle … Sync process sending pics to AD and finally finishing a complete sync …
If you want to find out from the database (Not supported by MS)
use {Your Profile Database}
SELECT RecordID, NTName, PictureUrl
FROM UserProfile_Full (nolock) where (pictureUrl != N”)
Copy the results to Excel and filter of the URL; making sure they poing to your picutre lib in your MySites Container; this can speed up the process of running the sync over and over to see which picutres are bad … If you find any bad references, go to SCA, Profile App; Manage Profiles and remove the bad referneces by removing the picture
One thing that I did notice about that error was once the sync hit that 404 error … it started over from the beginning (meaning the reading of the picutres) … and every time it hit that 404 error it did the same thing; eventually giving up and stating success … What baffled me was that our QA environment had about 1000 more profiles vs. our Production (QA has no pictures export to AD) … Now we understand b/c of that 404 error and it restarting once it hit that 404 error it never really completed … After fixing all of this (~ 5 days of troubleshooting) … All the manager hierarchal organization items are also back to working state … Thought I would share …
I ran into this issue and was able to fix the issue by opening ADSIedit.msc and removing the offending picture from the user attribute thumbnailphoto. Once I did this, I ran a UPS sync and it succeeded.
I rebuilt the UPSA a few times but after the first synch it would fail with the MOSS Full import stopped-extension-dll-exception and the following would be logged in the app log: System.Net.WebException: The remote server returned an error: (404) Not Found
I noted the existing photos that were uploaded to the user profile picture library were incorrect, but couldn’t find a way to correct them.. The URL in the user profile was the correct format.
Hope this helps someone
To add to the above, it appears that when you upload the picture to Active Directory if it’s not in a supported format it mangles the URL when you synch with SharePoint (at least after the first “successful” synch). I say “successful” because the image never displayed, it was always a Red X. So for whatever reason when this particular image was uploaded to the pictures library the URL was formated /domainuser_picture instead of the correct /domain_user_picture
As I am typing this out I found this blog:
http://blogs.msdn.com/b/cjohnson/archive/2010/05/08/sharepoint-2010-office-2010-and-profile-pictures.aspx
That would have saved me a lot of headache.
Having read this I believed it was extremely enlightening. I appreciate you taking
the time and effort to put this informative article together.
I once again find myself personally spending way too much time both reading and commenting.
But so what, it was still worthwhile!
Useful info. Lucky me I discovered your site by accident, and I’m stunned why this twist of fate didn’t
happened in advance! I bookmarked it.
I always used to study piece of writing in news papers but now as
I am a user of internet thus from now I am using net for
posts, thanks to web.
Hi there, I would like to subscribe for this website to
get latest updates, so where can i do it please help.
The supplier must ensure that they work correctly, so that the insurer is not able
to sign any clients. Depending on the medical coding schools agency or the hospital maintains a history of the patient’s insurance card, front and back. Percentage of Total Collections- This method uses a percentage of the net of all of the pertinent information about the company you work for. Medical coders assign standardized codes to the payer directly or via a clearinghouse.
Thank you for sharing your info. I really appreciate your efforts and
I am waiting for your further write ups thanks once again.
I leave a comment each time I appreciate a post
on a website or if I have something to add to the discussion.
It’s caused by the sincerness communicated in the post I browsed. And on this post Case Study on Troubleshooting a Failed SharePoint User Profile/FIM Synchronization, Bad CMYK User Photos and Disappearing User Profiles | The SharePoint Swiss Army Knife. I was actually excited enough to create a thought 🙂 I actually do have a couple of questions for you if it’s allright.
Could it be simply me or do a few of these remarks appear like they are written by brain dead visitors?
😛 And, if you are posting at additional places, I’d like to follow you. Would you list the complete urls of all your community pages like your Facebook page, twitter feed, or linkedin profile?
Good article. I’m experiencing some of these issues
as well..
Hi there, I found your web site by the use of Google while looking for a related topic, your site
got here up, it seems to be good. I’ve bookmarked it in my
google bookmarks.
Hi there, simply turned into alert to your weblog through Google,
and located that it’s really informative. I’m gonna watch out
for brussels. I’ll be grateful when you proceed this in future.
Numerous other people can be benefited out of your
writing. Cheers!
Good replies in return of this query with real arguments and explaining everything
concerning that.
The Bangalore hotels are equipped with contemporary facilities
and services. Before booking it is essential that you consider a few factors.
Apart, the city outstands evidently with a number of museums, and architectural marvels, and tuneful
groups.
Hi there it’s me, I am also visiting this web page regularly, this web page is in fact pleasant and the
people are truly sharing nice thoughts.
Nice blog here! Additionally your web site a lot up very fast!
What web host are you the usage of? Can I am getting your affiliate link for your host?
I desire my website loaded up as quickly as yours lol
You can choose whatever color and grade of pencil you want.
Producing these marketing items among guests, who aree at the business occasion, enables
your company to really create your product presence visible, felt, and valued.
Small and medium enterprises think of this promotional item as the right product tat helps
for brand building and reaching the targeting audience.
I have an excited synthetic eye just for detail and can anticipate complications just before they take place.
3 To know where the spam originates, you can install encryption, archiving and anti-spam software to
take care in order to delete the email. Is anti spam software is simple to install with easy to use anti spam appliances that can be useful such as the barracuda network anti spam applications, and
other anti spam review materials. The average person gets
about 105 unwanted emails each day, this is the first thing to be lost in combat.
Hi, Neat post. There’s an issue together with your web site
in web explorer, might check this? IE nonetheless is the marketplace leader and a big element of people will omit your magnificent writing due to this
problem.
Wow, this piece of writing is pleasant, my younger
sister is analyzing these things, therefore I
am going to tell her.
w tym
I love to share understanding that I’ve accrued with the calendar year to assist
enhance group overall performance.
I like to disseminate knowledge that I have built up through the
yr to assist enhance team efficiency.
Excellent excited analytical vision with regard to fine detail and may anticipate issues prior to they
will occur.