SharePoint 2010 Enterprise
3 WFE’s each with Query Roles
2 App Servers with 1 Dedicated Indexer

Error in Event Viewer:

Event ID: 8313

SharePoint Web Services Round Robin Service Load Balancer Event: EndpointFailure

Process Name: w3wp

Process ID: 5948

AppDomain Name: /LM/W3SVC/303076968/ROOT-1-129634027688317155

AppDomain ID: 2

Service Application Uri: urn:schemas-microsoft-com:sharepoint:service:767f7d712e064db3bb8945d9d42d0e12#authority=urn:uuid:f7625c7a33924a9981c0206f8bf0054a&authority=https://applicationserver:32844/Topology/topology.svc

Active Endpoints: 3

Failed Endpoints:1

Affected Endpoint: http://applicationserver:32843/767f7d712e064db3bb8945d9d42d0e12/SearchService.svc

Error in Diagnostic Logs:

Internal server error exception: System.ServiceModel.ServerTooBusyException: The HTTP service located at http://applicationserver:32843/767f7d712e064db3bb8945d9d42d0e12/SearchService.svc is too busy.  —> System.Net.WebException: The remote server returned an error: (503) Server Unavailable.     at System.Net.HttpWebRequest.GetResponse()     at System.ServiceModel.Channels.HttpChannelFactory.HttpRequestChannel.HttpChannelRequest.WaitForReply(TimeSpan timeout)

Troubleshooting Steps:

I tried to access the web service that SharePoint was trying to get to via the browser and noticed that it was reporting a 503 error as reported in the Diagnostic Logs. Then I hopped onto IIS on the affected servers and looked at the SharePoint Web Services web application (IIS) in search for a corresponding Service Application GUID that matched. The application didn’t exist on either of the application servers so that would cause the EndpointFailure as reported in the Event Viewer logs. I then double checked the Services on Server page for the application servers to make sure that the search services were turned on as having these turned on would provision the web services. They were all started which threw a little confusion my way but with some hints from this forum post, I went ahead and stopped/started the Search Query and Site Settings Service. After a little while, assuming that some timer jobs had to run, the Service Application with the missing GUID showed up in IIS and the error stopped appearing in the Event logs.

Summarized Solution:

1. Go to your Services on Server page in Central Administration.

2. Stop and Start the Search Query and Site Settings Service on each affected server as reported in the Event Viewer or SharePoint Diagnostic Logs.