Skip to main content

Limit on number of SXA Sites : Sitecore Site in IIS keeps restarting due to large number of SXA sites

Let me start this blog with a question - Is there a limit on number of sites Sitecore SXA can support on a single Sitecore instance? If you search an answer for this, you will see the below -


Bear with me, the answer to this question is not a simple 'No' in current state of Sitecore and SXA module. We will discuss more on this in this post. 

THE PROBLEM

We have upgraded from Sitecore 8.1 to Sitecore 10.1 and migrated our vanilla Sitecore sites into SXA sites. We started migrating some important sites to SXA and then we planned to move rest of the sites to SXA. Things were fine until the number of SXA sites hosted on our Sitecore instance were small. As we moved our rest of the sites to our new Sitecore instance, we started observing that the Sitecore site in IIS will continuously restart without giving much idea about why it is happening.

THE SOLUTION

We thought this can be a machine specific issue due to memory etc. and may be the issue wont reproduce on other developer machines. But to our surprise, we could reproduce the issue on different machines as well.

Obviously, we opened a Sitecore support ticket to understand this issue and one of our primary questions each time was that is there a limit on number of sites Sitecore or SXA supports. The answer shared with us was obvious - No limits. But we were not convinced because we could see the issue reproducing the moment the number of Sitecore sites reached approx. 540. Yes, that is a large number of sites on a single Sitecore instance and I will explain in upcoming section what caused such large number of sites in our Sitecore instance.

Why such large number of sites in our Sitecore instance?

Ideally the actual number of Sites in our Sitecore instance is lower (lets assume it is close to 100 which is still a high number). But in SXA, a site is equivalent to site definition defined in Site Groupings folder within the Site>Settings.

If a site has n number of languages, we had to create n number of site definitions for the same site to support multiple language. In such case, the only difference between these site definitions is that they have different values in Language field. Even though its single site, it appears as multiple sites due to large number of site definitions it has now to support multiple languages,.

Now, we wanted to incorporate preview functionality for our content authors. Preview functionality will basically show the site preview before the site actually gets published with actual page URLs that appear on live site (I can talk about this separately in a different blog). Such URLs are user friendly and easier to share with business teams and content approval teams (Sitecore's Preview feature doesn't generate user friendly URLs. Hence, we couldn't use it). 

To achieve this, we created a preview version of each site definition. Preview version of  site definition will be different from main site definition as it will point to master database instead of web database (which is used for published sites or live sites only). This means if a site has n languages, it has 2n site definitions now (n site definition to support n languages and then each site definition shall also have a preview version).

For e.g. Lets assume we have 100 sites and at average each site supports 2 languages, we will have 2x100 = 200 site definitions. Now each of these site definitions shall have preview version also which means each site definition will have duplicate now which will use master database to support preview functionality. Hence, number of total site definitions = 2x200 = 400.

I hope the above example helps to explain how 100 sites in Sitecore are getting converted in to 400 site definitions in Sitecore SXA.

So far so good! But our actual count of Sites was higher causing total site definitions to cross 800. Our application worked well until our total number of site definitions didn't cross 540. Once it crossed this limit, the application pool will recycle on its own and the site wont ever come up.

We checked our memories and CPU utilizations and nothing appeared alarming. Our memory + CPU usage was always way below our server's capability.

We started looking in to Event Viewer, we analyzed the memory dumps and logs. We found app crash incidences due to Sitecore waiting to acquire lock on certain resources. Since the Sitecore instance couldn't get resources in time, it used to error out. Error stacks looked as below -

ntdll!NtWaitForMultipleObjects+14

KERNELBASE!WaitForMultipleObjectsEx+fe

clr!WaitForMultipleObjectsEx_SO_TOLERANT+62

clr!Thread::DoAppropriateWaitWorker+205

clr!Thread::DoAppropriateWait+7d

clr!CLREventBase::WaitEx+b6

clr!Thread::Block+2c

clr!SyncBlock::Wait+1c8

clr!ObjectNative::WaitTimeout+e1

Sitecore.Threading.Semaphore.P()+61

Sitecore.Threading.CustomThreadPool.ProcessQueuedItems()+c6

mscorlib_ni!System.Threading.ExecutionContext.RunInternal(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)+172

mscorlib_ni!System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object, Boolean)+15

mscorlib_ni!System.Threading.ExecutionContext.Run(System.Threading.ExecutionContext, System.Threading.ContextCallback, System.Object)+55

mscorlib_ni!System.Threading.ThreadHelper.ThreadStart()+55

clr!CallDescrWorkerInternal+83

clr!CallDescrWorkerWithHandler+4e

clr!MethodDescCallSite::CallTargetWorker+102

clr!ThreadNative::KickOffThread_Worker+fffff02f

clr!ManagedThreadBase_DispatchInner+40

clr!ManagedThreadBase_DispatchMiddle+6c

clr!ManagedThreadBase_DispatchOuter+4c

clr!ManagedThreadBase_DispatchInCorrectAD+15

clr!Thread::DoADCallBack+26b

clr!ManagedThreadBase_DispatchInner+2e2f

clr!ManagedThreadBase_DispatchMiddle+6c

clr!ManagedThreadBase_DispatchOuter+4c

clr!ManagedThreadBase_FullTransitionWithAD+2f

clr!ThreadNative::KickOffThread+e6

clr!Thread::intermediateThreadProc+8b

kernel32!BaseThreadInitThunk+14

ntdll!RtlUserThreadStart+21


We tried to collect the memory dumps of our application when this happens. 

According to the memory dump, the StackOverflowException was caused by a bug #500740: StackOverflowException occurs when using OWIN and the number of sites is over several hundred. The issue occurs because Owin "MapMiddleware" is executed recursively for every site instance. Owin middleware recursion level increases with the number of Sitecore Accelerator sites and may exceed thread stack size. Hence, causing a StackOverflow.

In order to workaround this issue, Sitecore support suggested us to disable this LogoutEndpoint processor if one does not use the following SSO functionality: https://doc.sitecore.com/xp/en/developers/101/sitecore-experience-manager/single-sign-out.html

Alternatively, if this approach isn't suitable for people who use SSO, one should create a custom LogoutEndpoint processor with the changes shown below-



In the sample, we've identified the "Old Code" and "New Code" that resolves the issue as well. To apply the code change please build an assembly with the sample class and override "Sitecore.Owin.Authentication.IdentityServer.Pipelines.Initialize.LogoutEndpoint" processor with new "Sitecore.CS0294067.Owin.Authentication.IdentityServer.Pipelines.Initialize" in the "\App_Config\Sitecore\Owin.Authentication.IdentityServer\Sitecore.Owin.Authentication.IdentityServer.config".


THE VERDICT

In current state, there is known issue in SXA which limits Sitecore capability to host number of SXA sites to couple of hundreds. So, senior developers and architects should consider this fact while planning to add new sites to their existing Sitecore instances.

Hope it helps you guys!

Labels:

Popular posts from this blog

Sitecore PowerShell Script to create all language versions for an item from en version

  We have lots of media items and our business wants to copy the data from en version of media item to all other language versions defined in System/Languages. This ensures that media is available in all the languages. So, we created the below powershell script to achieve the same -  #Get all language versions defined in System/Languages $languages = Get-ChildItem /sitecore/System/Languages -recurse | Select $_.name | Where-Object {$_.name -ne "en"} | Select Name #Ensuring correct items are updated by comparing the template ID  $items = Get-ChildItem -Path "/sitecore/media library/MyProjects" -Recurse | Where-Object {'<media item template id>' -contains $_.TemplateID} #Bulk update context to improve performance New-UsingBlock (New-Object Sitecore.Data.BulkUpdateContext) { foreach($item in $items){    foreach($language in $languages){ $languageVersion = Get-Item -Path $item.Paths.Path -Language $language.Name #Check if language versi
Read more

Export Sitecore media library files to zip using SPE

If you ever require to export Sitecore media files to zip (may be to optimize them), SPE (Sitecore Powershell Extension) has probably the easiest way to do this for you. It's as easy as the below 3 steps -  1. Right click on your folder (icons folder in snap)>Click on Scripts> Click on Download 2. SPE will start zipping all the media files placed within this folder. 3. Once zipping is done, you will see the Download option in the next screen. Click Download Zip containing the media files within is available on your local machine. You can play around with the images now. Hope this helps!! Like and Share ;)
Read more

Make Sitecore instance faster using Roslyn Compiler

When we install the Sitecore instance on local, the first load is slow. After each code deploy also, it takes a while for the Sitecore instance to load and experience editor to come up. For us, the load time for Sitecore instance on local machines was around 4 minutes. We started looking for ways to minimize it and found that if we update our Web.config to use Roslyn compiler and include the relevant Nugets into the project, our load times will improve. We followed the simple steps - Go to the Project you wish to add the NuGet package and right click the project and click 'Manage NuGet Packages'. Make sure your 'Package Source' is set to nuget.org and go to the 'Browse' Tab and search Microsoft.CodeDom.Providers.DotNetCompilerPlatform. Install whichever version you desire, make sure you note which version you installed. You can learn more about it  here . After installation, deploy your project, make sure the Microsoft.CodeDom.Providers.DotNetCompilerPlatform.d
Read more

Experience of a first time Sitecore MVP

The Journey I have been working in Sitecore for almost 10 years now. When I was a beginner in Sitecore, I was highly impressed by the incredible community support. In fact, my initial Sitecore learning path was entirely based on community written blogs on Sitecore. During a discussion with my then technology lead Neeraj Gulia , he proposed the idea that I should start giving back to developer community whenever I get chance. Just like I have been helped by many developers via online blogs, stackoverflow etc., I should also try to help others. Fast forward a few years and I met  Nehemiah Jeyakumar  (now an MVP). He had a big archive of his technical notes in the form Sitecore blogs. I realized my first blog dont have to be perfect and it can be as simple as notes to a specific problem for reference in future. That's when I probably created my first blog post on Sitecore. At that time, I didn't knew about the Sitecore MVP program. Over the years, I gained more confidence to write
Read more

Clean Coding Principles in CSharp

A code shall be easy to read and understand. In this post, I am outlining basic principles  about clean coding after researching through expert recommended books, trainings and based on my experience. A common example to start with is a variable declaration like - int i  The above statement did not clarify the purpose of variable i. However,  the same variable can be declared as -  int pageNumber The moment we declared the variable as int pageNumber, our brain realized that the variable is going to store the value for number of pages. We have set the context in our brain now and it is ready to understand what the code is going to do next with these page numbers. This is one of the basic advantages of clean coding. Reasons for clean coding -  • Reading clean code is easier - Every code is revisited after certain amount of time either by the same or different developer who created it. In both the cases, if the code is unclean, its difficult to understand and update it. • To avoid s
Read more