HadesArchivist - The Archiving tool to save HLD Files on the GSI Tape Storage System

Currently for archiving HLD files we are using a Perl script which runs as a daemon on the DAQ maschines. It comes with it's own web server to monitor the archiving process. In addition it assists the operator to remove already archived files from the disk pool.

Usually there are two daemons running on different maschines - a master and a secondary daemon which takes over control, if the primary daemon is not reachable.

The tool is placed in the directory ~hadaq/hades-archivist on the different DAQ maschines (hadeb02.gsi.de, hadeb03.gsi.de). This is the manual one can read after the tool was started with option -h set: ./hades-archivist -h

 Hades Archiving Daemon

 Use this tool to automatically archive files to the GSI TSM system.
 Usually it runs in the background and looks frequently after new files,
 checks if they match some criteria in terms of size and modification time,
 and then takes the necessary steps to archive them, if not so far.
 Additionally, this daemon comes along with its own web server, providing
 an easy way to retrieve detailed status informations about the archiving
 process.
 The necessary settings are defined in a configuration file whose name 
 has to be given as a parameter. This file must be written in Perl syntax!


 Command Line: hades-archivist [-v] [-f] [-h] [-m master] config-file
   
   -f        : Do not daemonize - stay in foreground.
   -v        : Print verbose messages to stdout.
   -m master : "master" is the hostname of the primary archiving host.
               If the archiving daemon running on this maschine does not
               answer, then this daemon is going to take over the work, to
               make the archiving more failsafe. It returns the archiving 
               control to the master, if it is online again.
   -h        : Print this help.


 Example of a configuration file:

   # Port numbers must be >1024.
   $WebServerPort    = 1977;
   $StatusServerPort = 1978;
   $WebDocRoot       = "./webdocs";

   # Do not use polling times less than 30 seconds. This causes an 
   # tremendous amount of network traffic and CPU load.
   $StatusPollingTime    = 300;
   $ArchivingPollingTime = 3555;

   # Archiving is stopped after TSMArchiveTimeout * NumTSMTimeouts seconds
   $TSMQueryTimeout   = 300;
   $TSMArchiveTimeout = 2700;
   $NumTSMTimeouts    = 3;

   # Use absolute paths for @DataDirectories and list them in quotes, e.g:
   # qw(/d/dir1 /d/dir2 /d/dir3/subdir);
   # Caution: These directories are processed recursively!
   @DataDirectories  = qw(/d/hadeb02/jan05 /d/hadeb03/jan05);

   # HadesArchivivePath and HadesArchiveName define the path on the TSM
   # system
   $HadesArchiveName = "hadesraw";
   $HadesArchivePath = "jan05/hld";
   $TSMClient        = "./tsmcli";    # the client executable
   $TSMFlags         = "0";           # eventually "dev=tape 0"

   # files to be archived have to match all of these criteria
   $MinFileSize     = 1024;           # bytes
   $FileYoungerThan = 28 * 24 * 3600; # seconds
   $FileOlderThan   = 30 * 60;        # seconds
   $FileName        = '^.*\.hld$';    # pattern matched against full path

   # Set these variables to names of Perl scripts which will be hooked into
   # the query and archiving process. This gives you the possibility to do
   # whatever you like before the files are processed by the archiver. The
   # names of the files which will be archived (after the query operation)
   # are saved in the array @NotArchivedFiles. In addition to the variables
   # here, this array can be accessed in these scripts, too.
   $DoBeforeArchiving = undef;
   $DoAfterArchiving  = undef;
   $DoBeforeQuery     = undef;
   $DoAfterQuery      = undef;

-- SimonLang - 16 Mar 2005