RESOLVED: Solr Exceptions - Document contains at least one immense term in field

If you implemented Solr with Sitecore using Solr 5.x, you may run into the following error when indexing extremely large content in string fields:

org.apache.solr.common.SolrException: Exception writing document id <xxxuniqueid> to the index; possible analysis error.

Caused by: java.lang.IllegalArgumentException: Document contains at least one immense term in field="<xxxfieldname>" (whose UTF8 encoding is longer than the max length 32766), all of which were skipped.  Please correct the analyzer to not produce such terms.  The prefix of the first immense term is: '[10, 9, 9, 10, 32, 32, 32, 32, 32, 32, 82, 84, 82, 83, 32, 70, 97, 99, 105, 108, 105, 116, 121, 32, 10, 32, 32, 32, 32, 84]...', original message: bytes can be at most 32766 in length; got <intgreaterthan32766>

Caused by: org.apache.lucene.util.BytesRefHash$MaxBytesLengthExceededException: bytes can be at most 32766 in length; got <intgreaterthan32766>


This is due to the fact that there was a change in Lucene 4.8.x.  Before then, the exception was just
being eaten...now they throw it up and don't index that document. This exception is actually thrown by Lucene and not Solr as described here: DefaultIndexingChain swallows useful information from MaxBytesLengthExceededException.

After hours of researching the web and turning up with little or nothing to solve the issue, my inclination was to write my pipeline processor to truncate the field being indexed if it exceeded the preset maxFieldLength within Lucene, but then I struck gold. Apparently there is a built-in mechanism in Solr to truncate fields and you can provide your own limit.

Solve:

Step 1: Find the element "updateRequestProcessorChain" and add the following to your Solr.config. If the element does not exists, simply add it directly under <config>.

        <updateRequestProcessorChain name="mychain">
<processor class="solr.TruncateFieldUpdateProcessorFactory">
<str name="typeClass">solr.StrField</str>
<int name="maxLength">10000</int>
</processor>
<processor class="solr.LogUpdateProcessorFactory" />
<processor class="solr.RunUpdateProcessorFactory" />
</updateRequestProcessorChain>


Step 2: Find the element <requestHandler name="/update" class="solr.UpdateRequestHandler"> in Solr.config and insert the following element under it:

       <lst name="defaults">
         <str name="update.chain">mychain</str>
       </lst>

 Restart Apache, and re-build your index. This will ensure your string fields do not exceed the max length set in the processor.

Comments

  1. MartinAugust 8, 2015 at 10:02 AM

    Thanks so much. this got me out of a fix.

    ReplyDelete
  2. HishaamNovember 6, 2017 at 9:17 AM

    This comment has been removed by the author.

    ReplyDelete
  3. HishaamNovember 6, 2017 at 9:18 AM

    In Solr 5.4.1, from the solrconfig.xml, there is no entry of requestHandler name="/update" class="solr.UpdateRequestHandler".

    Should we add it manually?

    ReplyDelete
    Replies
    1. Varun NehraNovember 6, 2017 at 1:13 PM

      You shouldn't need to add it, there should be a requestHandler element for "/update" but may be commented by default with no child elements.

      Delete

Post a Comment

Popular posts from this blog

First look at Sitecore XM Cloud: Part 4 - Creating a new Site

Image
 Once you have successfully create an environment, you can go ahead and create your first site. In order to create a new Site, you need to launch the Sitecore Launchpad by clicking on the "XM Cloud Launchpad": Click on "Sites" menu to create a new Site: Click on "Add your first website" button to start the wizard. I chose the "Basic Site" template: Enter a Site name and pick a language and click on "Create website" button. You will see the following screen while your Site is being created: Once the Site is created, you can edit it in Pages or Explorer using the context menu: Explorer view of the Site: 1. "Details" view of page. 2. "Content" view of page: Unfortunately, trying to load the new Site in pages gives "Unable to connect to remote server"
Read more

Is Rendered Item Valid XHtml Document Could not find schema information warnings during publish item Sitecore 7.2

Image
Note: Since we now live in the HTML5 world, the simple resolution to any XHtml validation errors is to remove the XHtml validation rules on  System->Settings->Validation Rules->Global Rules item. Sitecore should update it's validation rules to be relevant to HTML5. This make the rest of this post obsolete :). I created a simple approval workflow in Sitecore 7.5 and tried to approve an item for publishing and found a bunch of validation errors in the Validation Results. Most of them were related to the "Is Rendered Item Valid XHtml Document" section which looked like this: The problems were related to violations of the W3C HTML5 recommendation An important note here is: The element containing the character encoding declaration must be serialized completely within the first 1024 bytes of the document. Adding the following xmlns attribute to the Html tag fixed most of the errors: <html lang="en" xmlns="http://www.w3.org/1999/...
Read more