TimeWarp Configurable items

TimeWarp will be configurable via a global config file. It will also be possible to startup the TimeWarp Suite with a alternative cofig file.

User Config

The global config file can optionally point to user config files - if the USER_CONFIG option is set to TRUE then the Suite will also look for a file at <User Home>/.TimeWarp/TimeWarp.conf - this will take the same format as the global config file - and can override or add to some of the global options whe files in that users home area (i.e those files held under /home/<user name>) are archived - any options included in a users local file which are irrelevant at a user config level will be ignored.

If the User config file does not exist then the global settings are used for that user. Note : Unlikely to be implemented prior to or for BETA release

The User config can be used to define a specific set of directories and exclusions/inclusions which apply to directories defined by that user.

Default Config File

Config file will be stored @ /etc/TimeWarp/TimeWarp.conf. Additional advanced configuration items is held in '/etc/TimeWarp/TimeWarpDefaultExclusions.conf' and '<User Home>/.TimeWarp/TimeWarp.conf' (as above)

Configurable items

The Following items are proposed to be configurable :

Use of the configuration items

As delivered the default configuration file will need only minor edits to get the TimeWarp suite working : Only SOURCE_DIRS and BACKUP_ROOT need to be defined. All other items are optional.

Conflicts between system and user local config

Given the ability to create system and user configurations, it seems to appear possible to create conflicts such that files included by the system config is excluded by the user, or vice versa.

These conflicts can only occur if the system wide configuration and the user local configurations define SOURCE_DIR entries which overlap.

If the overlapping directory are outside the users default /home then the system wide settings take priority.

If the overlapping directories are within the users default /home then the user local settings take priority.

This allows a single advanced user to define his own backup settings for his own /home area without impacting the backup settings for any other user /home which uses the system wide settings.

Default Exclusions - proposed

The file /etc/TimeWarp/TimeWarpDefaultExclusions.conf contains a set of config parameters which define sets of regular expression pattern matches. If the DEFAULT_EXCLUSIONS attribute is defined in the config file, then only those exclusions defined in that item will be considered by default - although others files can be excluded by use of the EXCLUDE_MATCH file. If the DEFAULT_EXCLUSIONS is not defined then all patterns under all items defined in /etc/TimeWarp/TimeWarpDefaultExclusions.conf are considered.

This default exclusions file provides a mechanism whereby developers and advanced users are able to re-configure the application to create system wide default exclusions, taking into account installed applications and the temporary files which the application may or may not create.

It is recommended when editing the default exclusions config file that a new config item is added for each application or application suite, and that care is taken when defining the patterns to avoid excluding files which should be backed up.

Priority of Inclusions and Exclusions

Given that a set of different exclusions and includes can be defined via the various system wide and user local config files, then the following priority is applied to each file path that changes :

Overlaps and excluded directories

1. A recursive search is executed on all directories for every entry under SOURCE_DIRS, including those SOURCE_DIRS defined in any user local configurations - overlapping areas are removed from the system and local users watches as above (non user home overlaps are removed from the user watches, user home overlaps are removed from the system defined watches).
2. All paths which match EXCLUDE_PATH are removed from this recursive watch - EXCLUDE_PATH defined in the system wide config is applied to all SOURCE_DIRS defined system wide. EXCLUDE_PATH defined in user local configurations are only applied to those SOURCE_DIRS (and children) defined in the same user local config.

Once the watch is created each file is dealt with on a case by case basis, but further pattern matching is restricted to just applying to files under the watched directories.

Per file matching

1. If the file names matches one of the patterns defined in the default_exclusions.py it is excluded.
2. If the full path matches EXCLUDE_MATCH it is excluded. Any system wide EXCLUDE_MATCH is applied to all files under the system wide watch. Any user defined EXCLUDE_MATCH is only applied to files under a watch defined by the same user.
3. If the full path matches INCLUDE_MATH it is included. Any system wide EXCLUDE_MATCH is applied to all files under the system wide watch. Any user defined EXCLUDE_MATCH is only applied to files under a watch defined by the same user.
4. If the file size is outside the range defined by MIN_SIZE and MAX_SIZE are excluded.

Any file still in the list after the application of all of these rules is recorded as having changed and is scheduled for backup.

It is important to note that the INCLUDE_MATCH parameter cannot be used to include files to be backed up which are already excluded by either the EXCLUDE_PATH rules or the higher priority "Per file matching" rules (i.e the default exclusions or the EXCLUDE_MATCH).

Scheduling Strategies

The following values are envisaged :

• OLDEST_FIRST : The file with the longest time on the QUEUE gets archived first
• LARGEST_FIRST : The file with the largest on disk size gets archived first

Scheduler Batch Size

The scheduler will fetch at most SCHEDULER_BATCHSIZE items from the queue, and wait for all of them to complete their archive operations before fetching another batch.

The SCHEDULER_BATCHSIZE is tunable to allow optimisation especially on busy systems. A low value would allow the system to react very quickly to new items entering the queue, with the downside of more queries on the database. A high value will make the system less responsive to rapid changes but would reduce the number of fetches on the database.

It would make no sense to set this value to lower than MAX_ARCHIVERS.

Prevention of system hogging

Timwearp is designed to ensure that it does not overload either the system it is executing on, or the network it is storing the archives on. To this end a number of paramters can be set :

MAX_ARCHIVERS : This sets the number of archive processes that will run on the system - on a busy system this value can be set lower - at the expense of less reponsive backups - i.e. file changes will wait longer to be recorded.

MAX_CPU : This limits the maximum amount of CPU that the archivers consume as they archive the file changes - as a % of the total CPU_TIME available. Due to the nature of the archiving and measuring process, the CPU usage will be bursty, but the CPU_TIME limit will be met on average.

MAX_BANDWIDTH : This limits the maximum amount of DATA that the archivers can transfer as they archive the file changes - as a bytes per second. Due to the nature of the archiving and measuring process, the CPU usage will be bursty, but the MAX_BANDWIDTH limit will be met on average.

Storage of changes

Two config items directly impact the storage of archives :

BACKUP_ROOT : As discussed this is the root directory for the archive files.

DWELL_TIME : This parameter prevents a rapidly changing file from creating too many archive entries. an update to a file will replace an earlier entry in the archive if the eariler entry is less than DWELL time seconds old.