Increasing Log Retention

To increase log retention for specific component, you can modify the configuration file and reset the value of the properties.

Increasing Log Retention for file system

In the /opt/mapr/conf/mfs.conf file, increase the value for the mfs.max.logfile.size.in.mb property. The value for this property is computed using the following formula:

maxSizePerLogFile = maxLogSize / MAX_NUM_OF_LOG_FILES
Here:
  • maxLogSize specifies the total amount of space that file system log files can consume.
  • MAX_NUM_OF_LOG_FILES specifies the total number of file system log files (5 hard coded).

For example, for a value of 10 GB, there will be 5 log files (mfs.log-3 to mfs.log-3.4) of 2GB each.

Increasing Log Retention for CLDB

In the /opt/mapr/conf/log4j.cldb.properties file, modify the value for the following properties:

  • log4j.appender.R.MaxFileSize — specifies the size of each cldb.log before rolling over
  • log4j.appender.R.MaxBackupIndex — specifies the number of cldb.log* before the oldest one is deleted.

For example, suppose the following configuration:

  • log4j.appender.R.MaxFileSize = 1024MB
  • log4j.appender.R.MaxBackupIndex = 10

The CLDB log files will grow to 10GB before they are purged.

NOTE This setting does not impact audit logging.

Increasing Log Retention for Hadoop Services

You can increase log retention for ResourceManager, NodeManager, HistoryServer, and TimelineServer by modifying the following properties in the /opt/mapr/hadoop/hadoop-<Version>/etc/hadoop/log4j.properties file:

  • hadoop.log.maxfilesize — specifies the size of each <service-name>-hostname>.log before rolling over.
  • hadoop.log.maxbackupindex — specifies the number of <service-name>-<hostname>.log* before the oldest one is deleted.

For example, suppose the following configuration:

  • hadoop.log.maxfilesize = 1024Mb
  • hadoop.log.maxbackupindex = 10

There will be 10 GB of total service-specific logs (1GB per file) before the oldest file is purged.