SharePoint 2013 Apps do not load CSS properly 403 – defaultcss.ashx
Last week I was confronted with a pretty big SharePoint 2013 Apps Error. After setting up the Environment using my own guide that worked almost a dozen times before, whenever I entered an application, the CSS didn’t load properly and the app looked something like this:
I did check the network monitor when loading and at first, I have found this:
So I thought it was getting a 404 somewhere… however on a second app there were absolutely no 404’s. I then turned on verbose logging and watched ULS in real time and found the following Logs. I know it’s quite a pack of ULS logs, however they will help future victims of this bug find this post!
defaultcss.ashx: using elevated codepath to get css file or other resource because the non-elevated code path failed to get it. System.UnauthorizedAccessException: Access is denied. (Exception from HRESULT: 0x80070005 (E_ACCESSDENIED)) at Microsoft.SharePoint.SPGlobal.HandleUnauthorizedAccessException(UnauthorizedAccessException ex) at Microsoft.SharePoint.Library.SPRequest.SetWebMetainfo(String bstrUrl, Object varMetainfo) at Microsoft.SharePoint.SPWeb.UpdateProperties(StringDictionary data) at Microsoft.SharePoint.SPWeb.set_MasterDefaultCss(String value) at Microsoft.SharePoint.SPWeb.get_MasterDefaultCss() at Microsoft.SharePoint.ApplicationPages.DefaultCss.GetResourceUrl(SPWeb web, Boolean getResource) at Microsoft.SharePoint.ApplicationPages.DefaultCss.ProcessRequest(HttpContext context) System.UnauthorizedAccessException: Access is denied. (Exception from HRESULT: 0x80070005 (E_ACCESSDENIED)), StackTrace: at Microsoft.SharePoint.SPWeb.UpdateProperties(StringDictionary data) at Microsoft.SharePoint.SPWeb.set_MasterDefaultCss(String value) at Microsoft.SharePoint.SPWeb.get_MasterDefaultCss() at Microsoft.SharePoint.ApplicationPages.DefaultCss.GetResourceUrl(SPWeb web, Boolean getResource) at Microsoft.SharePoint.ApplicationPages.DefaultCss.ProcessRequest(HttpContext context) at System.Web.HttpApplication.CallHandlerExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute() at System.Web.HttpApplication.ExecuteStep(IExecutionStep step, Boolean& completedSynchronously) at System.Web.HttpApplication.PipelineStepManager.ResumeSteps(Exception error) at System.Web.HttpApplication.BeginProcessRequestNotification(HttpContext context, AsyncCallback cb) at System.Web.HttpRuntime.ProcessRequestNotificationPrivate(IIS7WorkerRequest wr, HttpContext context) at System.Web.Hosting.PipelineRuntime.ProcessRequestNotificationHelper(IntPtr rootedObjectsPointer, IntPtr nativeRequestContext, IntPtr moduleData, Int32 flags) at System.Web.Hosting.PipelineRuntime.ProcessRequestNotification(IntPtr rootedObjectsPointer, IntPtr nativeRequestContext, IntPtr moduleData, Int32 flags) at System.Web.Hosting.UnsafeIISMethods.MgdIndicateCompletion(IntPtr pHandler, RequestNotificationStatus& notificationStatus) at System.Web.Hosting.UnsafeIISMethods.MgdIndicateCompletion(IntPtr pHandler, RequestNotificationStatus& notificationStatus) at System.Web.Hosting.PipelineRuntime.ProcessRequestNotificationHelper(IntPtr rootedObjectsPointer, IntPtr nativeRequestContext, IntPtr moduleData, Int32 flags) at System.Web.Hosting.PipelineRuntime.ProcessRequestNotification(IntPtr rootedObjectsPointer, IntPtr nativeRequestContext, IntPtr moduleData, Int32 flags) SPRequest.OpenWeb: UserPrincipalName=, AppPrincipalName= ,bstrUrl=http://vladtest.domain.ca/My FAQ List System.UnauthorizedAccessException: Access is denied. (Exception from HRESULT: 0x80070005 (E_ACCESSDENIED)), StackTrace: at Microsoft.SharePoint.SPWeb.InitWeb() at Microsoft.SharePoint.SPWeb.get_Title() at Microsoft.SharePoint.SPSite.OpenWeb(Guid gWebId, Int32 mondoHint) at Microsoft.SharePoint.ApplicationPages.DefaultCss.<>c__DisplayClass3.<ProcessRequest>b__0() at Microsoft.SharePoint.SPSecurity.<>c__DisplayClass5.<RunWithElevatedPrivileges>b__3() at Microsoft.SharePoint.Utilities.SecurityContext.RunAsProcess(CodeToRunElevated secureCode) at Microsoft.SharePoint.SPSecurity.RunWithElevatedPrivileges(WaitCallback secureCode, Object param) at Microsoft.SharePoint.SPSecurity.RunWithElevatedPrivileges(CodeToRunElevated secureCode) at Microsoft.SharePoint.ApplicationPages.DefaultCss.ProcessRequest(HttpContext context) at System.Web.HttpApplication.CallHandlerExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute() at System.Web.HttpApplication.ExecuteStep(IExecutionStep step, Boolean& completedSynchronously) at System.Web.HttpApplication.PipelineStepManager.ResumeSteps(Exception error) at System.Web.HttpApplication.BeginProcessRequestNotification(HttpContext context, AsyncCallback cb) at System.Web.HttpRuntime.ProcessRequestNotificationPrivate(IIS7WorkerRequest wr, HttpContext context) at System.Web.Hosting.PipelineRuntime.ProcessRequestNotificationHelper(IntPtr rootedObjectsPointer, IntPtr nativeRequestContext, IntPtr moduleData, Int32 flags) at System.Web.Hosting.PipelineRuntime.ProcessRequestNotification(IntPtr rootedObjectsPointer, IntPtr nativeRequestContext, IntPtr moduleData, Int32 flags) at System.Web.Hosting.UnsafeIISMethods.MgdIndicateCompletion(IntPtr pHandler, RequestNotificationStatus& notificationStatus) at System.Web.Hosting.UnsafeIISMethods.MgdIndicateCompletion(IntPtr pHandler, RequestNotificationStatus& notificationStatus) at System.Web.Hosting.PipelineRuntime.ProcessRequestNotificationHelper(IntPtr rootedObjectsPointer, IntPtr nativeRequestContext, IntPtr moduleData, Int32 flags) at System.Web.Hosting.PipelineRuntime.ProcessRequestNotification(IntPtr rootedObjectsPointer, IntPtr nativeRequestContext, IntPtr moduleData, Int32 flags) UnauthorizedAccessException for the request. 403 Forbidden will be returned. Error=Access is denied. (Exception from HRESULT: 0x80070005 (E_ACCESSDENIED)) at Microsoft.SharePoint.SPGlobal.HandleUnauthorizedAccessException(UnauthorizedAccessException ex) at Microsoft.SharePoint.Library.SPRequest.OpenWeb(String bstrUrl, String& pbstrServerRelativeUrl, String& pbstrTitle, String& pbstrDescription, String& pbstrTitleResourceId, String& pbstrDescriptionResourceId, Guid& pguidID, DateTime& pdtTimeCreated, String& pbstrRequestAccessEmail, UInt32& pwebVersion, Guid& pguidScopeId, UInt32& pnAuthorID, UInt32& pnLanguage, UInt32& pnLocale, UInt16& pnTimeZone, Boolean& bTime24, Int16& pnCollation, UInt32& pnCollationLCID, Int16& pnCalendarType, Int16& pnAdjustHijriDays, Int16& pnAltCalendarType, Boolean& pbShowWeeks, Int16& pnFirstWeekOfYear, UInt32& pnFirstDayOfWeek, Int16& pnWorkDays, Int16& pnWorkDayStartHour, Int16& pnWorkDayEndHour, Int16& pnMeetingCount, Int32& plFlags, Boolean& bConnectedToPortal, String& pbstrPortalUrl, String& pbstrPortalName, Int32& plWebTemplateId, Int16& pnProvisionConfig, String& pbstrDefaultTheme, String& pbstrDefaultThemeCSSUrl, String& pbstrThemedCssFolderUrl, String& pbstrAlternateCSSUrl, String& pbstrCustomizedCssFileList, String& pbstrCustomJSUrl, String& pbstrAlternateHeaderUrl, String& pbstrMasterUrl, String& pbstrCustomMasterUrl, String& pbstrSiteLogoUrl, String& pbstrSiteLogoDescription, Object& pvarUser, Boolean& pvarIsAuditor, UInt64& ppermMask, Boolean& bUserIsSiteAdmin, Boolean& bHasUniquePerm, Guid& pguidUserInfoListID, Guid& pguidUniqueNavParent, Int32& plSiteFlags, DateTime& pdtLastContentChange, DateTime& pdtLastSecurityChange, String& pbstrWelcomePage, Boolean& pbOverwriteMUICultures, Boolean& pbMUIEnabled, String& pbstrAlternateMUICultures, Int32& plSiteSchemaMajorVersion, Int32& plSiteSchemaMinorVersion, Int32& plSiteSchemaBuildVersion, Int32& plSiteSchemaRevisionVersion, Int32& puiVersion, Int16& pnClientTag, Boolean& pfIsEvalSite, Guid& pgSourceSiteId, DateTime& pdtExpirationDate, Guid& pgEvalSiteId, Guid& pguidAppProductId, String& pbstrRemoteAppUrl, String& pbstrOAuthAppId, String& pbstrAppDatabaseName, Guid& pgAppDatabaseServerReferenceId, String& pbstrAppDatabaseTargetApplicationId, String& pbstrAppWebDomainId, Int32& plUpgradeFlags, DateTime& pdtReminderDate, UInt64& pmaskDeny) at Microsoft.SharePoint.SPWeb.InitWeb() at Microsoft.SharePoint.SPWeb.get_Title() at Microsoft.SharePoint.SPSite.OpenWeb(Guid gWebId, Int32 mondoHint) at Microsoft.SharePoint.ApplicationPages.DefaultCss.<>c__DisplayClass3.<ProcessRequest>b__0() at Microsoft.SharePoint.SPSecurity.<>c__DisplayClass5.<RunWithElevatedPrivileges>b__3() at Microsoft.SharePoint.Utilities.SecurityContext.RunAsProcess(CodeToRunElevated secureCode) at Microsoft.SharePoint.SPSecurity.RunWithElevatedPrivileges(WaitCallback secureCode, Object param) at Microsoft.SharePoint.SPSecurity.RunWithElevatedPrivileges(CodeToRunElevated secureCode) at Microsoft.SharePoint.ApplicationPages.DefaultCss.ProcessRequest(HttpContext context) at System.Web.HttpApplication.CallHandlerExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute() at System.Web.HttpApplication.ExecuteStep(IExecutionStep step, Boolean& completedSynchronously) Application error when access /_layouts/15/defaultcss.ashx, Error=Access is denied. (Exception from HRESULT: 0x80070005 (E_ACCESSDENIED)) at Microsoft.SharePoint.SPGlobal.HandleUnauthorizedAccessException(UnauthorizedAccessException ex) at Microsoft.SharePoint.Library.SPRequest.OpenWeb(String bstrUrl, String& pbstrServerRelativeUrl, String& pbstrTitle, String& pbstrDescription, String& pbstrTitleResourceId, String& pbstrDescriptionResourceId, Guid& pguidID, DateTime& pdtTimeCreated, String& pbstrRequestAccessEmail, UInt32& pwebVersion, Guid& pguidScopeId, UInt32& pnAuthorID, UInt32& pnLanguage, UInt32& pnLocale, UInt16& pnTimeZone, Boolean& bTime24, Int16& pnCollation, UInt32& pnCollationLCID, Int16& pnCalendarType, Int16& pnAdjustHijriDays, Int16& pnAltCalendarType, Boolean& pbShowWeeks, Int16& pnFirstWeekOfYear, UInt32& pnFirstDayOfWeek, Int16& pnWorkDays, Int16& pnWorkDayStartHour, Int16& pnWorkDayEndHour, Int16& pnMeetingCount, Int32& plFlags, Boolean& bConnectedToPortal, String& pbstrPortalUrl, String& pbstrPortalName, Int32& plWebTemplateId, Int16& pnProvisionConfig, String& pbstrDefaultTheme, String& pbstrDefaultThemeCSSUrl, String& pbstrThemedCssFolderUrl, String& pbstrAlternateCSSUrl, String& pbstrCustomizedCssFileList, String& pbstrCustomJSUrl, String& pbstrAlternateHeaderUrl, String& pbstrMasterUrl, String& pbstrCustomMasterUrl, String& pbstrSiteLogoUrl, String& pbstrSiteLogoDescription, Object& pvarUser, Boolean& pvarIsAuditor, UInt64& ppermMask, Boolean& bUserIsSiteAdmin, Boolean& bHasUniquePerm, Guid& pguidUserInfoListID, Guid& pguidUniqueNavParent, Int32& plSiteFlags, DateTime& pdtLastContentChange, DateTime& pdtLastSecurityChange, String& pbstrWelcomePage, Boolean& pbOverwriteMUICultures, Boolean& pbMUIEnabled, String& pbstrAlternateMUICultures, Int32& plSiteSchemaMajorVersion, Int32& plSiteSchemaMinorVersion, Int32& plSiteSchemaBuildVersion, Int32& plSiteSchemaRevisionVersion, Int32& puiVersion, Int16& pnClientTag, Boolean& pfIsEvalSite, Guid& pgSourceSiteId, DateTime& pdtExpirationDate, Guid& pgEvalSiteId, Guid& pguidAppProductId, String& pbstrRemoteAppUrl, String& pbstrOAuthAppId, String& pbstrAppDatabaseName, Guid& pgAppDatabaseServerReferenceId, String& pbstrAppDatabaseTargetApplicationId, String& pbstrAppWebDomainId, Int32& plUpgradeFlags, DateTime& pdtReminderDate, UInt64& pmaskDeny) at Microsoft.SharePoint.SPWeb.InitWeb() at Microsoft.SharePoint.SPWeb.get_Title() at Microsoft.SharePoint.SPSite.OpenWeb(Guid gWebId, Int32 mondoHint) at Microsoft.SharePoint.ApplicationPages.DefaultCss.<>c__DisplayClass3.<ProcessRequest>b__0() at Microsoft.SharePoint.SPSecurity.<>c__DisplayClass5.<RunWithElevatedPrivileges>b__3() at Microsoft.SharePoint.Utilities.SecurityContext.RunAsProcess(CodeToRunElevated secureCode) at Microsoft.SharePoint.SPSecurity.RunWithElevatedPrivileges(WaitCallback secureCode, Object param) at Microsoft.SharePoint.SPSecurity.RunWithElevatedPrivileges(CodeToRunElevated secureCode) at Microsoft.SharePoint.ApplicationPages.DefaultCss.ProcessRequest(HttpContext context) at System.Web.HttpApplication.CallHandlerExecutionStep.System.Web.HttpApplication.IExecutionStep.Execute() at System.Web.HttpApplication.ExecuteStep(IExecutionStep step, Boolean& completedSynchronously)
The important part to see is that in the following logs, we have acces denied on an important app file which is defaultcss.ashx
defaultcss.ashx: using elevated codepath to get css file or other resource because the non-elevated code path failed to get it. System.UnauthorizedAccessException: Access is denied. (Exception from HRESULT: 0x80070005 (E_ACCESSDENIED)) Application error when access /_layouts/15/defaultcss.ashx, Error=Access is denied. (Exception from HRESULT: 0x80070005 (E_ACCESSDENIED)) UnauthorizedAccessException for the request. 403 Forbidden will be returned. Error=Access is denied. (Exception from HRESULT: 0x80070005 (E_ACCESSDENIED))
Also by watching it with Fiddler I had realized there were 403 errors on that file
Screenshot Property of Tobias Wolter because I forgot to take one myself!
Now at least, I had some ULS logs to research on, however the only person I found had this bug had written about it in German and Google translate did a really poor job . That person is Tobias Wolter, and if you understand German, you can also read about this topic on his blog here: http://www.standard-cloud.com/2013/04/sharepoint2013-apps-konfiguration-bug/ .
I tried excluding the permission problem, so I gave ALL SharePoint Service Accounts Full admin on SP Server, SysAdmin on SQL, and full control on Web Application. Did IISReset and reboots but nothing!
To continue debugging, I also tried using ProcMon to find what account was trying to get that file, however it had SUCCES on accessing it… So that didn’t help!
Then, when I was about to give up I finally found the solution. To put you in context, before this moment, when I was installing a SharePoint 2013 farm, I had two Application Pools as per *old* SharePoint Best Practices. One for the MySites and one for the rest of the Web Applications. I wasn’t a firm believer in the Host named Site Collections Model yet. However, I was well aware that Microsoft is advising us to only have 1 Web Application (= 1 App Pool = 1 App Pool Account) in SharePoint 2013.
Solution
Now, as you know if you set up apps in SharePoint 2013, you need an “Empty Host Header” Web application. Well I found out that apps will work only in Web Applications that will use the same Application Pool Account as the Empty Host Header Web Application. To make it safe, I suggest you put it in the same pool (that’s what I did) and it worked. In other words… if you have this error and you’re reading this blog post it’s because you use two application pool accounts! You need to go back to using only one! In other words
I am really glad I finally found the problem and wanted to thank the following people for providing great ideas on twitter and SharePoint Community!
Thank you!