Class DocumentsWriter

  • All Implemented Interfaces:
    java.io.Closeable, java.lang.AutoCloseable, Accountable

    final class DocumentsWriter
    extends java.lang.Object
    implements java.io.Closeable, Accountable
    This class accepts multiple added documents and directly writes segment files. Each added document is passed to the indexing chain, which in turn processes the document into the different codec formats. Some formats write bytes to files immediately, e.g. stored fields and term vectors, while others are buffered by the indexing chain and written only on flush. Once we have used our allowed RAM buffer, or the number of added docs is large enough (in the case we are flushing by doc count instead of RAM usage), we create a real segment and flush it to the Directory. Threads: Multiple threads are allowed into addDocument at once. There is an initial synchronized call to getThreadState which allocates a ThreadState for this thread. The same thread will get the same ThreadState over time (thread affinity) so that if there are consistent patterns (for example each thread is indexing a different content source) then we make better use of RAM. Then processDocument is called on that ThreadState without synchronization (most of the "heavy lifting" is in this call). Finally the synchronized "finishDocument" is called to flush changes to the directory. When flush is called by IndexWriter we forcefully idle all threads and flush only once they are all idle. This means you can call flush with a given thread even while other threads are actively adding/deleting documents. Exceptions: Because this class directly updates in-memory posting lists, and flushes stored fields and term vectors directly to files in the directory, there are certain limited times when an exception can corrupt this state. For example, a disk full while flushing stored fields leaves this file in a corrupt state. Or, an OOM exception while appending to the in-memory posting lists can corrupt that posting list. We call such exceptions "aborting exceptions". In these cases we must call abort() to discard all docs added since the last flush. All other exceptions ("non-aborting exceptions") can still partially update the index structures. These updates are consistent, but, they represent only a part of the document seen up until the exception was hit. When this happens, we immediately mark the document as deleted so that the document is always atomically ("all or none") added to the index.
    • Method Detail

      • deleteQueries

        long deleteQueries​(Query... queries)
                    throws java.io.IOException
        Throws:
        java.io.IOException
      • setLastSeqNo

        void setLastSeqNo​(long seqNo)
      • deleteTerms

        long deleteTerms​(Term... terms)
                  throws java.io.IOException
        Throws:
        java.io.IOException
      • updateDocValues

        long updateDocValues​(DocValuesUpdate... updates)
                      throws java.io.IOException
        Throws:
        java.io.IOException
      • applyDeleteOrUpdate

        private long applyDeleteOrUpdate​(java.util.function.ToLongFunction<DocumentsWriterDeleteQueue> function)
                                  throws java.io.IOException
        Throws:
        java.io.IOException
      • applyAllDeletes

        private boolean applyAllDeletes()
                                 throws java.io.IOException
        If buffered deletes are using too much heap, resolve them and write disk and return true.
        Throws:
        java.io.IOException
      • getNumDocs

        int getNumDocs()
        Returns how many docs are currently buffered in RAM.
      • abort

        void abort()
            throws java.io.IOException
        Called if we hit an exception at a bad time (when updating the index files) and must discard all currently buffered docs. This resets our state, discarding any docs added since last flush.
        Throws:
        java.io.IOException
      • flushOneDWPT

        final boolean flushOneDWPT()
                            throws java.io.IOException
        Throws:
        java.io.IOException
      • lockAndAbortAll

        java.io.Closeable lockAndAbortAll()
                                   throws java.io.IOException
        Locks all currently active DWPT and aborts them. The returned Closeable should be closed once the locks for the aborted DWPTs can be released.
        Throws:
        java.io.IOException
      • abortThreadState

        private int abortThreadState​(DocumentsWriterPerThreadPool.ThreadState perThread)
                              throws java.io.IOException
        Returns how many documents were aborted.
        Throws:
        java.io.IOException
      • getMaxCompletedSequenceNumber

        public long getMaxCompletedSequenceNumber()
        returns the maximum sequence number for all previously completed operations
      • anyChanges

        boolean anyChanges()
      • getBufferedDeleteTermsSize

        public int getBufferedDeleteTermsSize()
      • getNumBufferedDeleteTerms

        public int getNumBufferedDeleteTerms()
      • anyDeletions

        public boolean anyDeletions()
      • close

        public void close()
        Specified by:
        close in interface java.lang.AutoCloseable
        Specified by:
        close in interface java.io.Closeable
      • preUpdate

        private boolean preUpdate()
                           throws java.io.IOException
        Throws:
        java.io.IOException
      • postUpdate

        private boolean postUpdate​(DocumentsWriterPerThread flushingDWPT,
                                   boolean hasEvents)
                            throws java.io.IOException
        Throws:
        java.io.IOException
      • doFlush

        private boolean doFlush​(DocumentsWriterPerThread flushingDWPT)
                         throws java.io.IOException
        Throws:
        java.io.IOException
      • subtractFlushedNumDocs

        void subtractFlushedNumDocs​(int numFlushed)
      • flushAllThreads

        long flushAllThreads()
                      throws java.io.IOException
        Throws:
        java.io.IOException
      • finishFullFlush

        void finishFullFlush​(boolean success)
                      throws java.io.IOException
        Throws:
        java.io.IOException
      • ramBytesUsed

        public long ramBytesUsed()
        Description copied from interface: Accountable
        Return the memory usage of this object in bytes. Negative values are illegal.
        Specified by:
        ramBytesUsed in interface Accountable
      • getFlushingBytes

        public long getFlushingBytes()
        Returns the number of bytes currently being flushed This is a subset of the value returned by ramBytesUsed()