The search and indexing log files

Last updated Friday, February 26, 2016 in Sitecore Experience Platform for Developer, Administrator

Sitecore uses log files for tracking search and indexing operations. The crawling log shows information about indexing operations, and the search log shows information about search queries.

Crawling log

The crawling log contains information about the indexing process. You set up the crawling log in the <log4net /> section of the web.config file. The default logging level is INFO:

<logger name="Sitecore.Diagnostics.Crawling" additivity="false">
      <level value="INFO" />
      <appender-ref ref="CrawlingLogFileAppender" />
</logger>

You define the appender for this log in the same section. By default, it writes to a .txt file in the data/logs folder:

<appender name="CrawlingLogFileAppender" type=
          "log4net.Appender.SitecoreLogFileAppender, Sitecore.Logging">
      <file value="$(dataFolder)/logs/Crawling.log.{date}.txt" />
      <appendToFile value="true" />
      <layout type="log4net.Layout.PatternLayout">
        <conversionPattern value="%4t %d{ABSOLUTE} %-5p %m%n" />
      </layout>
</appender>

The implementation uses log4net. The log4net documentation describes how you can change the appender to write to a Windows Event Log, a database, or any other location.

This is an example of crawling log entries:

INFO  [Index=sitecore_core_index] Initializing IntervalAsynchronousUpdateStrategy 
              with interval '00:01:00'.
INFO  [Index=sitecore_core_index] Initializing LuceneDatabaseCrawler. 
              DB:core / Root:/sitecore
INFO  [Index=sitecore_master_index] Initializing SynchronousStrategy.
INFO  [Index=sitecore_master_index] Initializing LuceneDatabaseCrawler. 
              DB:master / Root:/sitecore
INFO  [Index=sitecore_web_index] Initializing OnPublishEndAsynchronousStrategy.
INFO  [Index=sitecore_web_index] Initializing LuceneDatabaseCrawler. 
              DB:web / Root:/sitecore
INFO  [Index=custom_master] Initializing IntervalAsynchronousUpdateStrategy 
              with interval '00:00:05'.
INFO  [Index=custom_master] Initializing LuceneDatabaseCrawler. DB:master / 
              Root:{D70CBEED-6DCF-483F-978F-6FC3C8049512}
INFO  [Index=custom_web] Initializing OnPublishEndAsynchronousStrategy.
INFO  [Index=custom_web] Initializing LuceneDatabaseCrawler. 
              DB:web / Root:{D70CBEED-6DCF-483F-978F-6FC3C8049512}
INFO  [Index=custom_web] Creating primary and secondary directories
INFO  [Index=custom_web] Resolving directories from index property 
              store for index 'custom_web'
INFO [Index=custom_master] IntervalAsynchronousUpdateStrategy executing.

The log begins with information about how indexes are configured and initialized. After that, an entry is written in the crawling log when an index update strategy is applied to an index or when a full rebuild is triggered.

Because the logging level in the example is INFO, there is a limited amount of entries in the log file.

For more detailed logs, for example, for troubleshooting purposes, you can change the logging level to DEBUG:

<logger name="Sitecore.Diagnostics.Crawling" additivity="false">
      <level value="DEBUG" />
      <appender-ref ref="CrawlingLogFileAppender" />
</logger>

With this configuration, and with the verbose logger enabled, Sitecore produces a very detailed item level indexing log.

Verbose logging

You can enable verbose logging in addition to the standard logging. When you enable verbose logging, you can see more detail about the indexing.

You use verbose logging in situations such as:

  • An index is not updated.
  • Some items are not being indexed.
  • A full rebuild is triggered for an index and you need to understand why.
  • You want to explore indexing activity on a specific server.

You want to test how your index is updated across all servers in a multiserver environment

The VerboseLogger is instantiated when ContentSearch.VerboseLogging is true:

The default is false and the setting is not present in the configuration delivered by Sitecore. The Sitecore.ContentSearch.VerboseLogging.config.example file has an example that shows you how to enable verbose logging.

Note

You must only enable the VerboseLogger component in special circumstances and never run it for long periods in a production environment. The log file grows very fast, and this can degrade performance.

You use the verbose logging for troubleshooting. For example, if a specific item is not indexed, the VerboseLogger gives you more context and can help you solve the problem.

The VerboseLogger writes information about a large set of indexing events:

  • indexing:excludedfromindex
  • indexing:start
  • indexing:end
  • indexing:addingrecursive
  • indexing:addedrecursive
  • indexing:adding
  • indexing:added
  • indexing:refreshstart
  • indexing:refreshend
  • indexing:deleteitem
  • indexing:deletegroup
  • indexing:updatingitem
  • indexing:updateditem
  • indexing:updatedependents
  • indexing:refreshstart
  • indexing:refreshend
  • indexing:propertyset
  • indexing:propertyget
  • indexing:propertyadd

Search log

The search log contains information about the search queries that Sitecore executes.

You set up the search log in the <log4net /> section of the web.config file:

    <logger name="Sitecore.Diagnostics.Search" additivity="false">
      <level value="INFO" />
      <appender-ref ref="SearchLogFileAppender" />
    </logger>

You define the appender for this log in the same section. By default, it is set up to write to a .txt file in the data/logs folder:

<appender name="SearchLogFileAppender" 
           type="log4net.Appender.SitecoreLogFileAppender, Sitecore.Logging">
      <file value="$(dataFolder)/logs/Search.log.{date}.txt" />
      <appendToFile value="true" />
      <layout type="log4net.Layout.PatternLayout">
        <conversionPattern value="%4t %d{ABSOLUTE} %-5p %m%n" />
      </layout>
</appender>

The implementation is based on log4net. The log4net documentation describes how you change the appender to write to a Windows Event Log, a database, or any other location.

This is a sample of what the search log entries look like:

3212 19:31:56 INFO  ExecuteQueryAgainstLucene :
         +_datasource:sitecore +(+(+_path:11111111111111111111111111111111 +_latestversion:1)
         +mileagehwy:[1 TO 4mileagecity]) - Filter : 3212 19:31:56 
       INFO  Results from web database :8818

You can enable a full level debug of content searches by setting ContentSearch.EnableSearchDebug = true in the Sitecore.ContentSearch.config file and changing the logging level for the search logger to DEBUG:

<logger name="Sitecore.Diagnostics.Search" additivity="false">
      <level value="DEBUG" />
      <appender-ref ref="SearchLogFileAppender" />
</logger>

The entries in the search are useful for:

  • Developers who want to understand how the Linq code translates into the native search queries that are passed on to the search provider (Lucene or Solr).
  • Administrators who want to understand which search queries are performed on a specific server. They can use this information for further optimization, for example, as input to which queries they should put into the queryWarmup pipeline.