Collector - About Store and Forward

Question

What do the settings mean on the Store and Forward tab of a collector?

How does Store and Forward work for a collector?

How can I configure these settings to minimize data loss in the event of a restart?

Answer

During normal operation, the collector will buffer the data it has collected for five seconds by default, then send it to the logger. This five-second batch of data is called a "package". If a collector has lost connectivity to the logger, it will continue to buffer the packages in data until connectivity is reestablished. There is a limit to the amount of data the collector can store in memory.

Once this limit is reached, the collector will begin writing the packages to a buffer on the hard drive. The hard drive buffer also has a limit. If that limit is reached, the collector will either stop collecting or discard the oldest data, depending on the settings.

When connectivity has been reestablished, the collector will forward all the data it has stored in its buffers.

Writing the data to disk is a costly operation, as is reading it to send it to the logger. The collector will be able to forward data fastest upon regaining connectivity if all its data is in memory and none of it is on disk. However, data in the memory buffer would be lost in the event of an unexpected shutdown of the collector, such as during a power failure.

If the collector is allowed to shut down "politely" it should move all its data from the memory buffer to the disk buffer, so no data should be lost regardless of the memory buffer size.

NOTE: For most customers, data loss should be minimal even with the default settings. In order for any data to be lost you would need to have a loss of connectivity for long enough to build up data in the memory buffer, then you would need an unexpected shutdown of the collector machine.

Two settings affect the memory buffer size. If either one of the limits is reached, then collector will start to write to disk.

Max Buffer Size in Memory (MB): This is the maximum amount of data the collector will keep in memory before it starts to write to disk. ICONICS does not recommend setting this value any lower than 128 MB (the default). This value should never be smaller than the amount of data your collector will collect in 5 seconds, and ideally should be several times larger than that.

Max In Memory Packages: This is the maximum amount of five-second packages the collector will keep in memory before it starts to write to disk. A value of zero means there is no package limit. Other than zero, ICONICS does not recommend setting this value any lower than 12 packages (60 seconds' worth of data).

Max Buffer Size On Disk (MB): This is the maximum amount of data the collector will keep on disk.

Discard Oldest Data: Based on this option, when the Max Buffer Size on Disk is reached the collector will either stop collecting or it will discard the oldest data packages to make room for new ones.

Send Most Recent Data First: This option determines whether the collector sends its most recent data or oldest data from the buffers to the logger.