Managing Storage Policies

Data offload is driven by rules, which are configured per volume. Data offload rule can be based on size of file (s), owner (u, g, or p) of the file, and/or file modification timestamp (m). You can apply one rule per volume.

When a rule is associated with a volume, the rule is first applied on the files in the tiering-enabled volume. When applied on the files in the tiering-enabled volume, the offload is triggered for all files in the snapshot chain as well when the criteria in the rule is met. If the file does not exist in the tiering-enabled volume, rule is applied on the latest state of the file in the snapshot chain. If the file exists in the tiering-enabled volume but has no latest state or if the file was deleted in the tiering-enabled volume, offload does not happen.

Rules can be defined using a combination of the following:

u Username or user ID, as configured in the OS registry (such as /etc/passwd file, LDAP, etc.), of a specific user.

Usage: u:<username or user ID>
g Group name or group ID, as configured in the OS registry (such as /etc/group file, LDAP, etc.), of a specific group.

Usage: g:<groupname or group ID>
a (atime) Time (in seconds or days) since the files were last accessed. The number of seconds can be specified by appending s to value and the number of days can be specified by appending d to the value.

Usage:

  • "a:<value>s" — specifies atime in seconds
  • "a:<value>d" — specifies atime in days
Note: If the system time on CLDB and file server nodes are different, the atime rule for offloading data may not work as intended.

This tier rule is matched and files are offloaded, when all of the following conditions are met:

  1. atime tracking is enabled at volume level
  2. Time since atime that is configured on the volume is more than the time specified in the rule
  3. Duration since the file was last accessed is more than the time specified in the rule

Assume that the atime feature is enabled on the volume and that the time in the rule is set to a:300s. Based on this rule, all files that are not accessed since 300s, are offloaded. However, this rule is valid only if time since atime tracking is enabled, is more than 300s. The volume level parameter atimeTrackingStartTime denotes the start time of atime.

For more information, see Tuning Last Access Time.
m (mtime) Time (in seconds or days) since the files were last modified. The number of seconds can be specified by appending s to value and the number of days can be specified by appending d to the value.

Usage:

  • "m:<value>s" — specifies mtime in seconds
  • "m:<value>d" — specifies mtime in days
All files that are not modified since the specified amount of time, are offloaded.
Note: If the system time on CLDB and file server nodes are different, the mtime rule for offloading data may not work as intended.
s The size of the file in bytes, kilobytes, megabytes, or gigabytes. The size of the file can be specified by appending one of the following to the value: b for bytes, k for kilobytes, m for megabytes, or g for gigabytes.

Usage

  • "s:<value>b" — specifies file size in bytes
  • "s:<value>k" — specifies file size in KB
  • "s:<value>m" — specifies file size in MB
  • "s:<value>g" — specifies file size in GB
All files whose size exceeds the specified size are offloaded.
Or, use the following:
p (Default) Specifies all files. Specifies that this operation is applicable to all the files without restriction. This cannot be combined with any other operator.
"" Indicates none of the files. Specifies that this operation cannot be performed on any of the files.
Use the following to string multiple criteria for offload:
& AND operation to combine multiple expressions as the criteria for the rule.
| OR operation to indicate either of the expressions as the criteria for the rule.
() Delimiters for subexpressions.

For volumes configured for erasure coding, a default storage policy, default.ectier.rule (ID 1 and expression p), is applied if one is not specified.

You can create, associate, and remove rules using the MapR Control System, the CLI, and REST API.