The EXM dispatch process and performance tuning

Last updated Wednesday, January 10, 2018 in Email Experience Manager for

The dispatch process is a series of pipelines and processors that EXM uses to queue, generate, and send email campaigns. The DispatchNewsletter and Send Email pipelines are the main components of the dispatch process.

You may be able to improve the performance of the dispatch process by following the recommendations outlined in the following areas:

Server roles

An environment can have a varying number of Dedicated Email Dispatch Server roles (DDS roles), including zero, but there is always only a single Primary Content Management server (Primary CM).

  • The DDS role is an extension to the Content Management (CM) role. A server can act solely as a CM role or as both a CM role and a DDS role, but it cannot act as a DDS role alone. The EXM UI cannot be used on a DDS role.
  • The Primary CM role is the only CM role that can use the EXM UI. The Primary CM manages the dispatch process, for example, initializes, queues and starts the dispatch process. By default, the Primary CM also acts as a dispatch server.

    In setups with one or more DDS roles, the Primary CM role can be configured not to participate in the actual dispatch, but only coordinate tasks in order to avoid slowing down content authoring.

The DispatchNewsletter pipeline

The DispatchNewsletter pipeline controls queuing, generating and sending email campaigns. The full DispatchNewsletter pipeline runs only on the Primary CM. A simplified version of the pipeline runs on DDS roles. The SendEmail pipeline always runs on DDS roles. The pipeline can also run on the Primary CM role unless you disable the SendMessage pipeline processor on the server.

In the DispatchNewsletter pipeline, the following pipeline processors impact the dispatch process performance:

  • WaitForDispatchToFinish
  • QueueMessage
  • SendMessage

WaitForDispatchToFinish (Disable dispatch on the Primary CM role)

The WaitForDispatchToFinish pipeline processor queries each DDS role while waiting for the email campaign dispatch to complete. You can enable the WaitForDispatchToFinish pipeline processor if you have at least one DDS role configured.

If you enable this processor, you can disable the SendMessage pipeline processor on the Primary CM role. In this way, the Primary CM role no longer sends any email messages, instead the DDS roles handle the load.

QueueMessage

The QueueMessage pipeline processor uses the segmentation engine to determine the included recipients of the email campaign.

Note

This only applies to regular email campaigns. The QueueMessage pipeline skips automated email campaigns.

The recipients included or excluded from the email campaign are determined by the following approaches:

Included recipients

Excluded recipients

  • The contacts that are in the included lists of the email campaign.
  • The contacts that are in the excluded lists and in the global opt-out list.
  • The contacts that have the DoNotMarket or ConsentRevoked settings set on the ConsentInformation facet.
  • The contacts where the value of the BounceCount setting of the contact's preferred email address on the EmailAddressList facet exceeds the Undelivered Maximum setting on the manager root.

Note

You can override how the included and excluded recipients are determined by creating your own implementation of the Sitecore.EmailCampaign.Cm.Dispatch.IDispatchManager interface.

The QueueMessage pipeline processor adds the contact identifier for each included recipients to the DispatchQueue table in the exm.master database. Then it adds the contacts to queues on multiple threads and in batches according to the following settings in the Sitecore.EmailExperience.ContentManagement.config file. By default, four threads run in parallel and each thread queues 300 contacts at a time:

You can adjust these values to increase performance, for example, to have 10 threads that each queue 1000 contacts. This is recommended if you:

  • Dispatch email campaigns to a large number of contacts, for example, over 1,000,000.
  • Experience that queueing takes a long time.
  • Have problems with the load on your SQL server.

SendMessage

The SendMessage pipeline processor starts the actual dispatching of email messages. It starts the following instances:

  • Message task runner
  • Dispatch task

Message task runner

The message task runner is an instance of the Sitecore.EmailCampaign.Cm.Dispatch.IMessageTaskRunner interface. The default implementation is Sitecore.EmailCampaign.Cm.Dispatch.MessageTaskRunner.

The message task runner starts multiple dispatch tasks according to the following settings specified in the Sitecore.EmailExperience.ContentManagement.config file:

The MaxGenerationThreads setting limits the number of dispatch tasks that can run concurrently. For example:

  • If the values for NumberThreads and MaxGenerationThreads are both set to 16, 16 dispatch tasks process concurrently.
  • If MaxGenerationThreads is set to 8 and NumberThreads is set to 16, only 8 of the dispatch tasks process concurrently while the other 8 tasks are blocked and wait to be processed.

To improve the dispatch speed of an email campaign, you can increase the value in NumberThreads and in MaxGenerationThreads. This, however, increases the load on the CPU.

Dispatch task

A dispatch task is an instance of the Sitecore.EmailCampaign.Cm.Dispatch.IMessageTask interface. The default implementation is Sitecore.EmailCampaign.Cm.Dispatch.DispatchTask.

The dispatch task processes contacts in batches according to the EXM.DispatchBatchSize setting in the Sitecore.EmailExperience.ContentManagement.config file. The default batch size is 100.

For each batch, the following tasks are performed:

  • The dispatch task gets 100 contact identifiers from the DispatchQueue table in the exm.master table.
  • These 100 contact identifiers are then used to load 100 contacts from xConnect, including any relevant facets, for example, the email address.
  • For each contact, the SendEmail pipeline runs.
  • When the pipeline has processed the 100 contacts:
    • An interaction with an Email Sent event is created for each contact that the email campaign was sent to successfully. This is submitted to xConnect in a single batch.
    • A message that contains a list of all the contacts who have changed their email address is submitted to the EmailAddressHistoryBus messaging bus. This updates the EmailAddressHistory facet.

If you increase or decrease the default value of the EXM.DispatchBatchSize setting, it can have a significant impact on the dispatch performance. If you increase the setting value, it decreases the number of requests that EXM makes to xConnect but at some point you may experience diminishing returns. This affects the speed with which EXM loads contacts, creates interactions, and updates facets.

The SendEmail pipeline

The SendEmail pipeline runs for every included recipient of the email campaign.

The FillEmail and SendEmail pipeline processors respectively set the email headers and content, and perform the SMTP dispatch.

The Sleep pipeline processor acts as a throttle and inserts a delay after every email has been sent. By default, this delay is set to 50 milliseconds.

To increase performance, you can decrease this delay or remove the Sleep pipeline processor entirely. This, however, increases the CPU load.

Processing and Reporting

The EXM reporting process relies on the Sitecore Processing role. When you dispatch email messages, the Processing role records any page events and goals as contact interactions and sends them to the processing pool in xConnect. Therefore, if you send email campaigns to large numbers of contacts, it may take EXM a while to report all the information to xConnect and therefore delay the analytics reporting.

You can separate out the Sitecore Processing role to increase the processing performance, reduce the workload on the Primary CM, and provide an opportunity for the Sitecore Processing role to be scaled out.

General performance optimization

To improve the general dispatch performance, consider optimizing the following areas:

  • Email message size
  • Use of personalization
  • Number of contacts
  • Number of dispatch servers
  • The suppression list

Note

To ensure that your EXM module works efficiently and to determine if further performance tuning is necessary, you can run the EXM Performance Measurement tool.

Email message size

The email message size has a considerable impact on the dispatch performance. If your email message contains a lot of data that must be transferred during the dispatch, fewer email messages can be sent per second.

The size of your email message is determined by the amount of content that must be rendered in the message. Therefore, to improve the dispatch performance, you can make sure that there are no superfluous text or images in the message. In addition, ensure that the Embed Images setting on the Manager Root is disabled. Embedded images in an email message often increase the size of the email message considerably.

Note

Determine the size of your email message by sending a quick test to yourself and check the message size in your email client.

Use of personalization

If you use rule-based personalization or any custom code behind personalization in your email message, EXM renders the email message for every recipient. This has a significant negative impact on performance.

If you only use token replacement, ensure that you disable personalization on the Delivery tab of your email campaign. This causes EXM to only render the email campaign once for every dispatch task thread per variant and/or language version of the email campaign, because the output is stored in the EmailCampaign.MessageBodyCache. You can specify the cache size in the EXM.MessageBodyMaxCacheSize setting in the Sitecore.EmailExperience.Core.config file.

Number of contacts

The number of recipients that you add to your email campaign also impacts the dispatch performance. The more recipients you add to your email campaign, the longer it takes for EXM to send the email campaign.

To be able to send your email campaign to many recipients and still maintain high dispatch performance, you can add additional dispatch servers, make more resources (for example, CPU) available, and make sure you maintain high-quality contact data.

EXM uses the List Manager to manage the recipients list in an email campaign. Even though EXM unsubscribes contacts from a contact list, it is important that you groom the lists concurrently. For example:

  • If you synchronize contacts from an external data source, you should ensure that you do not add contacts that have previously unsubscribed.
  • If you use segmented lists, you should ensure that the segmentation conditions are correct and that the facets are properly updated.

Number of dispatch servers

You do not need to add a dedicated email dispatch server, but you may want to do so if you have issues with poor dispatch performance.

If you add a dedicated dispatch server:

  • The load is taken off of the content authoring environment, and editors will not be affected by poor performance during the email campaign dispatch process.
  • The message generation and dispatch processes are significantly improved.

Note

You can add as many dedicated email dispatch servers as you need. The Dedicated email dispatch servers topic provides you with an overview of when it is recommended that you install one or more dedicated dispatch servers.

The suppression list

If you use the Email Cloud provider, the locally cached suppression list checks for each included contact to avoid sending emails to email addresses that the Email Cloud provider suppresses. This saves bandwidth and ensures you are not charged for emails that will be suppressed, at the expense of an additional database lookup for each contact.

You can disable this check by setting the EXM.CheckSuppressionList value to false in the Sitecore.EmailExperience.ContentManagement.config file.

For more information about the EXM suppression list, see Manage the suppression list.

Send feedback about the documentation to docsite@sitecore.net.