Emerse-solr-plugin-7.0.0-alpha1.jar returning old stats data

1

Normally this wouldn’t matter, but we had some documents in our index that had ENCOUNTER_DATE’s that were in the future. It was only 41 in total, and the worst one was in 3017-11-17T00:25:00Z.
This breaks EMERSE because Hibernate rejects this as a valid date when it does its “Update document statistics…” work, which you can force from the Admin->System screen.
So I cleared out the bad documents, and when I use SOLR to check that the dates are all now reasonable all looks good. However, the EMERSE call to …/documents/index-stats which calls into this jar still shows the old data. I tested this with POSTMAN and this is what I see when I do command=minMax, df=ECOUNTER_DATE:
{
“responseHeader”: {
“status”: 0,
“QTime”: 1
},
“min”: “1899-01-11T16:00:00Z”,
“max”: “3017-11-17T00:25:00Z”
}

Is there a way to get this to reset?

2

Yeah, so it looks like we don’t ignore “deleted” documents when calculating the min and max date. When documents are deleted, they are really just marked for deletion, and are ignored in searches etc, but when we coded the min-max endpoint, we didn’t check to see if the documents that have the min and max dates weren’t deleted.

We can fix this, but there’s another thing you can do which makes the bug less of an issue, which is that there’s a setting in EMERSE that will only update the end date of the index to at most today, that way if there’s any future data, it won’t push the index’s end date past today. If you want to use that, you need to set:

batch.updateIndexMaxDateFromSOLRIndexPastToday=false

in the emerse.properties file and restart the server.

3

Thanks, and that worked! Unfortunately the min date time is also illegal! I’ll code something up to fix this temporarily.
Thanks,
Eric

4

So, it doesn’t look like there’s a good way to fix the Solr plugin to ignore deleted documents while still being very fast, which was the intention of the plugin there. However, I forgot we also have another property you can set that might fix your second problem:

batch.updateIndexMinDateFromSOLRIndex=false

This will completely disable changing the min date, so you can set it once in the DB, and then it’ll stay that way.

5

Thanks! And yep, I found that property and used it exactly that way.

Eric