Re: ConcurrentMergeScheduler and MergePolicy question

bq. we've always said to keep the merge factor small for search reasons, at least in the high-update case.

I think we have been wrong. A bunch of segments vs optimized is about the same speed I think. I'd always read that to, but Mike said it didn't make sense once, and some simple testing seemed to prove it out. Which means you probably want a little tail of segments usually. Reopen likes segments (as opposed to an optimized index), and you won't be merging into the largest segments as often. And search speed shouldn't suffer. What did suffer was opening a FieldCache on a multi-segment index - that was a major speed trap - but now, with per segment searching, its not really an issue anymore. I think yonik may have alleviated that issue as well with a patch?

-- 
- Mark

http://www.lucidimagination.com

On Thu, Jul 30, 2009 at 2:48 PM, Grant Ingersoll <gsingers@xxxxxxxxxx> wrote:

Note also response from Mike that talks a little bit about something along these lines: http://www.lucidimagination.com/search/document/fa990adba4d2572b/is_there_a_way_to_control_when_merges_happen#f6f0bfeef4bf9a39

-Grant

On Jul 30, 2009, at 10:35 AM, Grant Ingersoll wrote:

Given a large segment and a bunch of small segments, how does the ConcurrentMergeScheduler (CMS) work?  Does it always merge the smaller segments into the bigger one, or does it merge the smaller segments together?

Something I've been thinking about:  Given a high update environment (and near real time, less than 1 minute, search constraints) and/or a very bursty environment, we've always said to keep the merge factor small for search reasons, at least in the high-update case.  However, I've seen a couple of times where this causes problems because merges can take over and cause pauses, even with CMS, so I am wonder if it makes sense to have a larger merge factor (>10), knowing that I may have a few large segments and then a bunch of small ones and that the CMS will, in the background, be able to keep merging the smaller segments together and in most cases avoid ever having to merge into the large segments (b/c maybe I can just optimize down at slower times or even merge larger segments later. )   Seems like this would allow one to make sure larger merges need not take place, or at least reduce the chances of that happening.

Not sure if I worded that correctly.

Thanks,
Grant

---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@xxxxxxxxxxxxxxxxx
For additional commands, e-mail: java-dev-help@xxxxxxxxxxxxxxxxx

---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@xxxxxxxxxxxxxxxxx
For additional commands, e-mail: java-dev-help@xxxxxxxxxxxxxxxxx