Package org.apache.lucene.index
Class DocumentsWriterFlushControl
- java.lang.Object
-
- org.apache.lucene.index.DocumentsWriterFlushControl
-
- All Implemented Interfaces:
Accountable
final class DocumentsWriterFlushControl extends java.lang.Object implements Accountable
This class controlsDocumentsWriterPerThread
flushing during indexing. It tracks the memory consumption perDocumentsWriterPerThread
and uses a configuredFlushPolicy
to decide if aDocumentsWriterPerThread
must flush.In addition to the
FlushPolicy
the flush control might set certainDocumentsWriterPerThread
as flush pending iff aDocumentsWriterPerThread
exceeds theIndexWriterConfig.getRAMPerThreadHardLimitMB()
to prevent address space exhaustion.
-
-
Nested Class Summary
Nested Classes Modifier and Type Class Description private static class
DocumentsWriterFlushControl.BlockedFlush
-
Field Summary
Fields Modifier and Type Field Description private long
activeBytes
private java.util.Queue<DocumentsWriterFlushControl.BlockedFlush>
blockedFlushes
private boolean
closed
private LiveIndexWriterConfig
config
private DocumentsWriter
documentsWriter
(package private) boolean
flushByRAMWasDisabled
private long
flushBytes
(package private) java.util.concurrent.atomic.AtomicBoolean
flushDeletes
private java.util.IdentityHashMap<DocumentsWriterPerThread,java.lang.Long>
flushingWriters
private FlushPolicy
flushPolicy
private java.util.Queue<DocumentsWriterPerThread>
flushQueue
private boolean
fullFlush
private java.util.List<DocumentsWriterPerThread>
fullFlushBuffer
private long
hardMaxBytesPerDWPT
private InfoStream
infoStream
(package private) double
maxConfiguredRamBuffer
private int
numDocsSinceStalled
private int
numPending
(package private) long
peakActiveBytes
(package private) long
peakDelta
(package private) long
peakFlushBytes
(package private) long
peakNetBytes
private DocumentsWriterPerThreadPool
perThreadPool
(package private) DocumentsWriterStallControl
stallControl
private long
stallStartNS
-
Constructor Summary
Constructors Constructor Description DocumentsWriterFlushControl(DocumentsWriter documentsWriter, LiveIndexWriterConfig config)
-
Method Summary
All Methods Instance Methods Concrete Methods Modifier and Type Method Description (package private) void
abortFullFlushes()
(package private) void
abortPendingFlushes()
long
activeBytes()
(package private) void
addFlushableState(DocumentsWriterPerThreadPool.ThreadState perThread)
java.util.Iterator<DocumentsWriterPerThreadPool.ThreadState>
allActiveThreadStates()
Returns an iterator that provides access to all currently activeDocumentsWriterPerThreadPool.ThreadState
s(package private) boolean
anyStalledThreads()
Returnstrue
iff stalledprivate boolean
assertActiveDeleteQueue(DocumentsWriterDeleteQueue queue)
(package private) boolean
assertBlockedFlushes(DocumentsWriterDeleteQueue flushingQueue)
private boolean
assertMemory()
private boolean
assertNumDocsSinceStalled(boolean stalled)
private DocumentsWriterPerThread
checkout(DocumentsWriterPerThreadPool.ThreadState perThread, boolean markPending)
private void
checkoutAndBlock(DocumentsWriterPerThreadPool.ThreadState perThread)
(package private) DocumentsWriterPerThread
checkoutLargestNonPendingWriter()
Returns the largest non-pending flushable DWPT ornull
if there is none.private void
commitPerThreadBytes(DocumentsWriterPerThreadPool.ThreadState perThread)
(package private) DocumentsWriterPerThread
doAfterDocument(DocumentsWriterPerThreadPool.ThreadState perThread, boolean isUpdate)
(package private) void
doAfterFlush(DocumentsWriterPerThread dwpt)
(package private) void
doOnAbort(DocumentsWriterPerThreadPool.ThreadState state)
(package private) void
doOnDelete()
(package private) DocumentsWriterPerThreadPool.ThreadState
findLargestNonPendingWriter()
(package private) void
finishFullFlush()
boolean
getAndResetApplyAllDeletes()
long
getDeleteBytesUsed()
Returns heap bytes currently consumed by buffered deletes/updates that would be freed if we pushed all deletes.long
getFlushingBytes()
InfoStream
getInfoStream()
Returns theIndexWriter
InfoStream
private java.util.Iterator<DocumentsWriterPerThreadPool.ThreadState>
getPerThreadsIterator(int upto)
private DocumentsWriterPerThread
internalTryCheckOutForFlush(DocumentsWriterPerThreadPool.ThreadState perThread)
(package private) boolean
isFullFlush()
Returnstrue
if a full flush is currently running(package private) long
markForFullFlush()
long
netBytes()
(package private) DocumentsWriterPerThread
nextPendingFlush()
(package private) int
numBlockedFlushes()
Returns the number of flushes that are checked out but not yet available for flushing.(package private) int
numFlushingDWPT()
(package private) int
numQueuedFlushes()
Returns the number of flushes that are already checked out but not yet actively flushing(package private) DocumentsWriterPerThreadPool.ThreadState
obtainAndLock()
private void
pruneBlockedQueue(DocumentsWriterDeleteQueue flushingQueue)
Prunes the blockedQueue by removing all DWPT that are associated with the given flush queue.long
ramBytesUsed()
Return the memory usage of this object in bytes.void
setApplyAllDeletes()
(package private) void
setClosed()
void
setFlushPending(DocumentsWriterPerThreadPool.ThreadState perThread)
Sets flush pending state on the givenDocumentsWriterPerThreadPool.ThreadState
.private long
stallLimitBytes()
java.lang.String
toString()
(package private) DocumentsWriterPerThread
tryCheckoutForFlush(DocumentsWriterPerThreadPool.ThreadState perThread)
private boolean
updatePeaks(long delta)
private boolean
updateStallState()
void
waitForFlush()
(package private) void
waitIfStalled()
This method will block if too many DWPT are currently flushing and no checked out DWPT are available-
Methods inherited from class java.lang.Object
clone, equals, finalize, getClass, hashCode, notify, notifyAll, wait, wait, wait
-
Methods inherited from interface org.apache.lucene.util.Accountable
getChildResources
-
-
-
-
Field Detail
-
hardMaxBytesPerDWPT
private final long hardMaxBytesPerDWPT
-
activeBytes
private long activeBytes
-
flushBytes
private volatile long flushBytes
-
numPending
private volatile int numPending
-
numDocsSinceStalled
private int numDocsSinceStalled
-
flushDeletes
final java.util.concurrent.atomic.AtomicBoolean flushDeletes
-
fullFlush
private boolean fullFlush
-
flushQueue
private final java.util.Queue<DocumentsWriterPerThread> flushQueue
-
blockedFlushes
private final java.util.Queue<DocumentsWriterFlushControl.BlockedFlush> blockedFlushes
-
flushingWriters
private final java.util.IdentityHashMap<DocumentsWriterPerThread,java.lang.Long> flushingWriters
-
maxConfiguredRamBuffer
double maxConfiguredRamBuffer
-
peakActiveBytes
long peakActiveBytes
-
peakFlushBytes
long peakFlushBytes
-
peakNetBytes
long peakNetBytes
-
peakDelta
long peakDelta
-
flushByRAMWasDisabled
boolean flushByRAMWasDisabled
-
stallControl
final DocumentsWriterStallControl stallControl
-
perThreadPool
private final DocumentsWriterPerThreadPool perThreadPool
-
flushPolicy
private final FlushPolicy flushPolicy
-
closed
private boolean closed
-
documentsWriter
private final DocumentsWriter documentsWriter
-
config
private final LiveIndexWriterConfig config
-
infoStream
private final InfoStream infoStream
-
stallStartNS
private long stallStartNS
-
fullFlushBuffer
private final java.util.List<DocumentsWriterPerThread> fullFlushBuffer
-
-
Constructor Detail
-
DocumentsWriterFlushControl
DocumentsWriterFlushControl(DocumentsWriter documentsWriter, LiveIndexWriterConfig config)
-
-
Method Detail
-
activeBytes
public long activeBytes()
-
getFlushingBytes
public long getFlushingBytes()
-
netBytes
public long netBytes()
-
stallLimitBytes
private long stallLimitBytes()
-
assertMemory
private boolean assertMemory()
-
commitPerThreadBytes
private void commitPerThreadBytes(DocumentsWriterPerThreadPool.ThreadState perThread)
-
updatePeaks
private boolean updatePeaks(long delta)
-
doAfterDocument
DocumentsWriterPerThread doAfterDocument(DocumentsWriterPerThreadPool.ThreadState perThread, boolean isUpdate)
-
checkout
private DocumentsWriterPerThread checkout(DocumentsWriterPerThreadPool.ThreadState perThread, boolean markPending)
-
assertNumDocsSinceStalled
private boolean assertNumDocsSinceStalled(boolean stalled)
-
doAfterFlush
void doAfterFlush(DocumentsWriterPerThread dwpt)
-
updateStallState
private boolean updateStallState()
-
waitForFlush
public void waitForFlush()
-
setFlushPending
public void setFlushPending(DocumentsWriterPerThreadPool.ThreadState perThread)
Sets flush pending state on the givenDocumentsWriterPerThreadPool.ThreadState
. TheDocumentsWriterPerThreadPool.ThreadState
must have indexed at least on Document and must not be already pending.
-
doOnAbort
void doOnAbort(DocumentsWriterPerThreadPool.ThreadState state)
-
tryCheckoutForFlush
DocumentsWriterPerThread tryCheckoutForFlush(DocumentsWriterPerThreadPool.ThreadState perThread)
-
checkoutAndBlock
private void checkoutAndBlock(DocumentsWriterPerThreadPool.ThreadState perThread)
-
internalTryCheckOutForFlush
private DocumentsWriterPerThread internalTryCheckOutForFlush(DocumentsWriterPerThreadPool.ThreadState perThread)
-
toString
public java.lang.String toString()
- Overrides:
toString
in classjava.lang.Object
-
nextPendingFlush
DocumentsWriterPerThread nextPendingFlush()
-
setClosed
void setClosed()
-
allActiveThreadStates
public java.util.Iterator<DocumentsWriterPerThreadPool.ThreadState> allActiveThreadStates()
Returns an iterator that provides access to all currently activeDocumentsWriterPerThreadPool.ThreadState
s
-
getPerThreadsIterator
private java.util.Iterator<DocumentsWriterPerThreadPool.ThreadState> getPerThreadsIterator(int upto)
-
doOnDelete
void doOnDelete()
-
getDeleteBytesUsed
public long getDeleteBytesUsed()
Returns heap bytes currently consumed by buffered deletes/updates that would be freed if we pushed all deletes. This does not include bytes consumed by already pushed delete/update packets.
-
ramBytesUsed
public long ramBytesUsed()
Description copied from interface:Accountable
Return the memory usage of this object in bytes. Negative values are illegal.- Specified by:
ramBytesUsed
in interfaceAccountable
-
numFlushingDWPT
int numFlushingDWPT()
-
getAndResetApplyAllDeletes
public boolean getAndResetApplyAllDeletes()
-
setApplyAllDeletes
public void setApplyAllDeletes()
-
obtainAndLock
DocumentsWriterPerThreadPool.ThreadState obtainAndLock()
-
markForFullFlush
long markForFullFlush()
-
assertActiveDeleteQueue
private boolean assertActiveDeleteQueue(DocumentsWriterDeleteQueue queue)
-
addFlushableState
void addFlushableState(DocumentsWriterPerThreadPool.ThreadState perThread)
-
pruneBlockedQueue
private void pruneBlockedQueue(DocumentsWriterDeleteQueue flushingQueue)
Prunes the blockedQueue by removing all DWPT that are associated with the given flush queue.
-
finishFullFlush
void finishFullFlush()
-
assertBlockedFlushes
boolean assertBlockedFlushes(DocumentsWriterDeleteQueue flushingQueue)
-
abortFullFlushes
void abortFullFlushes()
-
abortPendingFlushes
void abortPendingFlushes()
-
isFullFlush
boolean isFullFlush()
Returnstrue
if a full flush is currently running
-
numQueuedFlushes
int numQueuedFlushes()
Returns the number of flushes that are already checked out but not yet actively flushing
-
numBlockedFlushes
int numBlockedFlushes()
Returns the number of flushes that are checked out but not yet available for flushing. This only applies during a full flush if a DWPT needs flushing but must not be flushed until the full flush has finished.
-
waitIfStalled
void waitIfStalled()
This method will block if too many DWPT are currently flushing and no checked out DWPT are available
-
anyStalledThreads
boolean anyStalledThreads()
Returnstrue
iff stalled
-
getInfoStream
public InfoStream getInfoStream()
Returns theIndexWriter
InfoStream
-
findLargestNonPendingWriter
DocumentsWriterPerThreadPool.ThreadState findLargestNonPendingWriter()
-
checkoutLargestNonPendingWriter
final DocumentsWriterPerThread checkoutLargestNonPendingWriter()
Returns the largest non-pending flushable DWPT ornull
if there is none.
-
-