Package org.yamcs.replication

Class ReplicationFile

java.lang.Object
org.yamcs.replication.ReplicationFile
All Implemented Interfaces:
Closeable, AutoCloseable
public class ReplicationFile extends Object implements Closeable
Stores transactions in a memory mapped file. The data is split into pages, each page has a fixed number of transactions.

An index gives a pointer to the beginning of each page to allow to jump faster to a given transaction number.

The metadata transactions form a linked list in order to allow to send them all when a client connects.

Header: 

 12 bytes magic "YAMCS_STREAM"
  1 byte version
  3 bytes spare
  8 bytes first_id =  first transaction in the file = file_id - used for consistency check(if someone renames the file)
  4 bytes page_size - number of transactions per page 
  4 bytes max_pages - max number of pages (and the size of the index)
 
  8 bytes last_mod = last modification time
  4 bytes n =  number of full pages. If n=max_pages, the file is full, cannot be written to it
  4 bytes m = number of transactions on page n
  4 bytes firstMetadataPos - position of the first metadata transaction
  (max_pages+1) x 4 bytes idx - transaction index 
      idx[i] (i=0..max_pages) - offset in the file where transaction with id id_first + i*m starts
      idx[i] = 0 -> no such transaction. this means num_tx < i*m
      idx[max_pages] -> pointer to the end of the file.
transaction data: 
 
 1 byte type
 3 bytes - size of the data that follows
 4 bytes instance_id
 8 bytes transaction_id
 n bytes data
 4 bytes CRC32 calculated over the data including the type and length
 
  for metadata the first 4 bytes of the data is the position of the next metadata record

The methods of this class throw UncheckedIOException instead of IOException. When working with memory mapped files in java, an IO error will cause an unspecified unchecked exception or even crash of Java (because file data is accessed using memory reads/writes). Therefore we prefer not to give a false sense of security by throwing IOException only in some limited situations and converted all these to UncheckedIOException..

The one occasion when Java may crash while no hardware failure is present is when the disk is full.

TODO: add a checker and stop writing data if the disk usage is above a threshold.

  • Method Details

    • getFirstId

      public long getFirstId()

    • newFile

      public static ReplicationFile newFile(String yamcsInstance, Path path, long firstTxId, int pageSize, int maxPages, int maxFileSize)

    • openReadOnly

      public static ReplicationFile openReadOnly(String yamcsInstance, Path path, long firstTxId)

    • openReadWrite

      public static ReplicationFile openReadWrite(String yamcsInstance, Path path, long firstTxId, int maxFileSize)

    • writeData

      public long writeData(Transaction tx)
      Write transaction to the file and returns the transaction id.

      returns -1 if the transaction could not be written because the file is full.

      Parameters:
      tx -
      Returns:

    • writeData

      public long writeData(Transaction tx, boolean sync)
      Write transaction to the file and returns the transaction id.

      returns -1 if the transaction could not be written because the file is full.

      Parameters:
      tx -
      sync - if true, forces the write to disk
      Returns:

    • tail

      public ReplicationTail tail(long txId)
      Returns a ReplicationTail containing a read only ByteBuffer having the position on given txId and with the limit set to the current end of tx data.

      The tail can be sent back in getNewData(ReplicationTail) to obtain more data if available.

      ReplicationTail.eof = true means the file is full so no more data will be available in the future.

      if the txId is smaller than the first transaction of this file, an IllegalArgumentException is thrown.

      If the txId is greater than the highest transaction in this file plus 1, null is returned

      If the txId is the highest transaction in this file plus one, a tail with 0 transactions (i.e. position=limit in the buffer) is returned; it can be used later to get more data.

      Parameters:
      txId -
      Returns:

    • getNewData

      public void getNewData(ReplicationTail rfe)
      Change the limit inside the file tail to the current position in the file buffer. Update the nextTxId

      Also update the eof flag if the file filled up since the last call.

      Parameters:
      rfe -

    • isFull

      public boolean isFull()

    • metadataIterator

      public Iterator<ByteBuffer> metadataIterator()
      Iterate through the metadata
      Returns:

    • close

      public void close()
      Specified by:
      close in interface AutoCloseable
      Specified by:
      close in interface Closeable

    • sync

      public void sync() throws IOException
      Force writing the content on disk.

      The method will call first FileChannel.force(boolean), write the number of transactions to the header and then call again FileChannel.force(boolean) to force also the header on the disk.

      This way should guarantee that the transaction data is written on the disk before the header

      Throws:
      IOException

    • getNextTxId

      public long getNextTxId()
      Returns the last tx id from this file + 1.

      If there is no transaction in this file, return 0.

      Returns:

    • headerSize

      public static int headerSize(int pageSize, int maxPages)

    • numTx

      public int numTx()

    • isSyncRequired

      public boolean isSyncRequired()

    • setSyncRequired

      public void setSyncRequired(boolean syncRequired)
      Set the sync required flag such that the file is synchronized by the ReplicationMaster
      Parameters:
      syncRequired -