FileMigrate.properties
Properties | Default Value | Description | |
---|---|---|---|
UI Setting | Property File | ||
Full Scan Frequency | upload.fullScanFrequencyMin |
60 | Frequency, in minutes, at which full scan of all files detect changes missed by
lite scans and purge (per the configured policy) uploaded files. NOTE: The actual time
of a full scan is computed on a per directory basis and spread randomly around the
specified fullScanFrequency. Thus specifying 60 minutes does not mean that all
directories get a full scan at exactly 60 minutes. Instead, directories randomly get
a full scan on average every 60 minutes (somewhere between 30 and 90 minutes). This
approach avoids load spikes. |
Upload Scan Frequency | upload.scanFrequencySec |
60 | Frequency, in seconds, at which light scans of directories detect changes since the last scan. This value cannot be below 10 seconds. |
Completion Scan Frequency | upload.completionScanFrequencySec |
3 | Frequency, in seconds, for checking upload status. This impacts the time delay before the queue is cleared for more uploads, the reporting time for completion, and statistics collection. NOTE: Values below the default may
consume extra system resources.
|
Minimum Wait Before Upload | upload.minWaitBeforeUploadSecs |
15 | Length of time, in seconds, to wait for uploading a file since the file was last modified. Set a higher value to avoid uploading a file still being modified. |
Maximum Active Uploads | upload.maxActiveUploads |
10 | Maximum number of files to upload concurrently. The scanning component of the file migration service is paused when 10 times this many files are waiting for upload. |
Minimum Wait After Failure | upload.minWaitAfterFailureSecs |
300 | Amount of time, in seconds, to wait (pause) before retrying if upload fails or if there is an error in the file system. |
Maximum Retries Per File | upload.maxErrorsOnOneFileBeforeGiveUp |
10 | Maximum number of times to attempt to upload a file. If the limit is reached, the file is dequeued to allow other files to be uploaded and the file modification time does not change. The dequeued file is rediscovered and queued for upload during the next full scan. |
Maximum System Retries | upload.maxErrorsBeforeAssumeSystemIssue |
10 | Maximum number of errors to allow before pausing all uploads. If this
limit is reached, uploads continue only after the amount of time specified as
value for the minWaitAfterFailureSecs property has elapsed. |
Minimum Idle Time Before Error | upload.minIdleTimeBeforeErrorSec |
300 | Amount of time, in seconds, to wait before logging a warning (in the
filemigrate.log file) if all uploads are not progressing. |
Bucket Location Region | aws.region |
Region information as defined here. For:
NOTE: When you create a bucket in a region, verify that the
bucket is really in that region; in some cases, buckets for one region are actually
hosted elsewhere. |
|
Region Endpoint | aws.regionendpoint |
The AWS region endpoint. | |
Proxy Host | aws.proxyHost |
(Optional) Proxy host information. | |
Proxy Port | aws.proxyPort |
||
Proxy Username | aws.proxyUsername |
||
Proxy Password | aws.proxyPassword |
||
Access Key | aws.accessKey |
Credentials to use for accessing the S3 bucket. | |
Secret Key | aws.secretKey |
||
Directory Path | dir<N>.path |
The path from which the scan of the directory or root directory starts. Replace
<N> with an integer; for each additional directory tree,
increment the number and specify a unique path.NOTE: Do not create multiple
policies or directory configurations pointing to the same file system
location. |
|
Target Bucket | dir<N>.bucket |
The Amazon S3 bucket to upload to. | |
Purge Interval | dir<N>.purgepolicy.hours |
0 | Amount of time, in hours, after the file was last modified to wait before
deleting the file. The default value is never delete (0 ).
NOTE: Deletion is delayed until after the file upload occurs, regardless of
the purge time set. |
Delete Empty Directories | dir<N>.deleteEmptyDirectoryDelayMins |
0 | Amount of time to wait, in minutes, before deleting empty directories.
The default value is never delete (0 ). |
Server Side Encryption | dir<N>.serverSideEncrypt |
false | Specify whether (true ) or not (false )
S3 server side encryption should be used. If the value is true
(enabled), the TransferManager API that specifies AES256 encryption is used
automatically for each uploaded file. |
Ignore Files Regex | dir<N>.ignoreFilesRegex |
File and directory names to ignore, specified as regex pattern, as
defined by java.util.regex . For example:
|
|
X-Attributes | dir<N>.xattrs |
The space-separated list of AWS S3 attributes to copy from MapR-FS to AWS S3
bucket when uploading a file. Make a note of the following:
NOTE: Only extended attributes from MapR file system that start with Extended attributes are uploaded when the file is first uploaded.
If extended attributes are set after file upload, the extended attributes are not
sent to AWS S3 unless the file modification timestamp changes.user. are
considered for upload (since all user settable attributes begin with user. ).
The attribute must be specified without user. . For example, to
upload a file with the extended attribute name user.foo , the
value for this property must be foo because the user. is
implicit. |
|
Authorized Users | authorized.users |
mapr | The comma-separated list of users who can login and use the File
Migration Service. The UI authorizes the following users to login and use the
service:
configure.sh utility. |
Authorized Groups | authorized.groups |
mapr | The comma-separated list of groups authorized to login and use the service. The
default value is the MAPR_USER (mapr, by default) specified with the
configure.sh utility. |