Difference: TclToOra (1 vs. 7)

Revision 7
12 Dec 2009 - Main.JanMichel
Line: 1 to 1
Added:
>
>
META TOPICPARENT name="HadesDaqDocumentation"
  Table of contents:
Revision 6
01 Aug 2008 - Main.SergeyYurevich
Line: 1 to 1
  Table of contents:
Line: 13 to 13
 

runinfo2ora.pl

Added:
>
>

Intro

  To simplify the procedure of inserting the information into the Hades database a perl script runinfo2ora.pl was written. It comes with oper CVS repository. The script reads a text file
Line: 26 to 28
  be skipped by the database itself and you will see corresponding error message).
Changed:
<
<
runinfo2ora.pl requires DBI and DBD/Oracle.pm.
>
>

Versions

CVS:

revision date comment
1.1 2007-10-04 works only with one EB. Supports the old format with only one EB
1.2 2008-08-01 works with multiple EBs. Supports new format

Installation guide

runinfo2ora.pl requires Oracle client and Perl modules DBI and DBD/Oracle.pm.
  To be able to install this module from CPAN, an install oracle client is needed.
Changed:
<
<
  • The old installation is here: /data/system/usr/local/oracle. The new installation was put here: /opt/oracle.
  • export ORACLE_HOME=/opt/oracle/product/10.1.0.3
  • compile/install DBD::Oracle (with the force option).
  • nohup ./runinfo2ora.pl &
>
>
  • The old installations of the Oracle client are here: /data/system/usr/local/oracle and here: /opt/oracle. The old installations had a bug: if an uptime of the machine acceeds some 60 days the Oracle client fails to connect to the server. The new bug-free installation is here: /home/hadaq/hadessoftware/oracle/product/10.2.0.1client/
  • Proper Oracle settings:
    • export ORACLE_HOME=/home/hadaq/hadessoftware/oracle/product/10.2.0.1client/
    • ORACLE_SID=orcl
    • export ORAENV_ASK="NO"
    • export LD_LIBRARY_PATH=$ORACLE_HOME/lib:$LD_LIBRARY_PATH
    • export PATH=$ORACLE_HOME/bin:$PATH
  • compile/install DBD::Oracle (with the force option) if missing
    • perl -MCPAN -e shell
    • force install DBD::Oracle
  • check connection to the server: sqlplus daq_pub/hades@db-hades
  • run script: nohup ./runinfo2ora.pl &
 
Changed:
<
<
-- SergeyYurevich - 09 Oct 2007
>
>
-- SergeyYurevich - 01 Aug 2008
 

tcl_to_ora

Revision 5
09 Oct 2007 - Main.SergeyYurevich
Line: 1 to 1
Added:
>
>
Table of contents:


The following programs are used to connect to a Hades database:

  • runinfo2ora.pl
  • tcl_to_ora

runinfo2ora.pl

To simplify the procedure of inserting the information into the Hades database a perl script runinfo2ora.pl was written. It comes with oper CVS repository. The script reads a text file eb_runinfo2ora.txt (created by EB) with necessary information which is then inserted by runinfo2ora.pl into the corresponding table in the database. The program keeps track of the inserted lines. However, in case of the death of the script, runinfo2ora.pl can be run again on the existing eb_runinfo2ora.txt, because the RUN_ID is a unique constraint in the database and all the lines with the same RUN_ID can not be inserted/updated second time (they will be skipped by the database itself and you will see corresponding error message).

runinfo2ora.pl requires DBI and DBD/Oracle.pm. To be able to install this module from CPAN, an install oracle client is needed.
  • The old installation is here: /data/system/usr/local/oracle. The new installation was put here: /opt/oracle.
  • export ORACLE_HOME=/opt/oracle/product/10.1.0.3
  • compile/install DBD::Oracle (with the force option).
  • nohup ./runinfo2ora.pl &

-- SergeyYurevich - 09 Oct 2007

tcl_to_ora

  Here is how to run the program that puts all the runs to the oracle database nearly on-line. The program is called tcl_to_ora and unfortunately it is not really
Revision 4
01 Feb 2005 - Main.MichaelTraxler
Line: 1 to 1
  Here is how to run the program that puts all the runs to the oracle database nearly on-line. The program is called tcl_to_ora and unfortunately it is not really
Line: 93 to 93
  crash of tcl_to_ora) can be found in TABLE daq.active_run and TABLE daq.run).
Added:
>
>

Date: Tue, 1 Feb 2005 23:14:43 +0100 (CET) From: Benjamin Sailer <Benjamin.Sailer@ph.tum.de> To: Michael Traxler <M.Traxler@gsi.de> Cc: Martin Jurkovic <Martin.Jurkovic@ph.tum.de> Subject: Oracle on lxhadesdaq

Hallo zusammen,

hab's kompiliert, der Grund fuer den Fehler war lediglich, dass der Oracle-Praeprozessor nur arbeiten mag, wenn das Oracle-Enviroment komplett gesetzt ist (zu machen mit $ . oraenv , wobei oraenv in /usr/local/bin liegt).

Damit hat dann auch tcl_to_ora kompiliert. Ich habe beides (libOraParam* und tcl_to_ora) nach /usr/local gesteckt, Du kannst es aber natuerlich dort wieder wegwerfen, wenn die Konvention eher ist, dass es in /home/hadaq/aug04 landen soll, der Eindruck ist bei mir beim "make install" entstanden (config.site vom Mathias???).

Gruss und viel Erfolg

Benjamin
  -- BenjaminSailer - Compilation from 27 Aug 2004 to 01 Feb 2005
Revision 3
01 Feb 2005 - Main.MartinJurkovic
Line: 1 to 1
  Here is how to run the program that puts all the runs to the oracle database nearly on-line. The program is called tcl_to_ora and unfortunately it is not really
Line: 11 to 11
 

Basically, the program reads a tcl-file produced by the
Changed:
<
<
eventbuilder (currently /home/hadaq/aut04/oper/eb_s.tcl),
>
>
eventbuilder (currently /home/hadaq/aug04/oper/eb_s.tcl),
  follows it's progress and dumps every line to Oracle. Frequently check if it is running by
Revision 2
01 Feb 2005 - Main.BenjaminSailer
Line: 1 to 1
Changed:
<
<
Hello all,
>
>
Here is how to run the program that puts all the runs to the oracle database nearly on-line. The program is called tcl_to_ora and unfortunately it is not really straight forward to run it as

  • it depends on the state of the database
  • deals with data coming via a pipe to stdin
  • uses an additional program to transform the input
  • needs a defined starting point for operation from the file point of view
 
Deleted:
<
<
once again (and after 2 hours, finally), I managed to again restart the program that fills in runs to Oracle (so that they can appear online in the DAQ logbook and analysis can immediately get parameters for the run id).
 

Basically, the program reads a tcl-file produced by the eventbuilder (currently /home/hadaq/aut04/oper/eb_s.tcl),
Line: 26 to 31
  (null)). Fix that by pure SQL (or ask an expert for that), then cut the part of the file away, that was already been written to Oracle (probably to a file called
Changed:
<
<
"eb_s.tcl__". Now You should stop the
>
>
"eb_s.tcl_<username>_<date>)". Now You should stop the
  eventbuilder process, take the rest and put it to Oracle by
Changed:
<
<
$ cat eb_s.tcl | tcl_to_ora -s hades -e 's/daq_evtbuild/eb1pp/g; s/daq_netmem/eb1gp/g'
>
>
$ cat eb_s.tcl | tcl_to_ora -s hades -e 's/^set [^(]*daq_evtbuild/set eb1pp/g;s/^set [^(]*daq_netmem/set eb1gp/g'
 

then also move this file away, touch a new eb_s.tcl:
Line: 43 to 47
  finally restart tracing the file in background by
Changed:
<
<
nohup sh -c "tail -f eb_s.tcl | tcl_to_ora -s hades -p -e 's/daq_evtbuild/eb1pp/g; s/daq_netmem/eb1gp/g'" &
>
>
nohup sh -c "tail -f eb_s.tcl | tcl_to_ora -s hades -p -e 's/^set [^(]*daq_evtbuild/set eb1pp/g; s/^set [^(]*daq_netmem/set ebgp/g'"
 

Finally, You can start the eventbuilder process again
Line: 52 to 55
 


Changed:
<
<
Hello all,

one more comment to the utility that is writing DAQ events to the database: It can only work if You call the eventbuild process always with the same name (argv[0]). Oracle only takes the information if it is coming from a binary called eb1pp (therefore the argument -e 's/daq_evtbuild/eb1pp/g; s/daq_netmem/eb1gp/g', that changes daq_evtbuild to eb1pp). If You now call the process via ./daq_evtbuild, this has to be changed into eb1pp, so the argument to tcl_to_ora is -e 's/\.\/daq_evtbuild/eb1pp/g; s/\.\/\daq_netmem/eb1gp/g'.

If You call the process once with "daq_evtbuild", once with "./daq_evtbuild", once with "/home/hadaq/bin/daq_evtbuild" and once with "../../bin/daq_evtbuild", the script can never handle this - unless You invent a better sed-script pattern to match all calling variants ...

Looking in the eb_s.tcl-file, I now assume it's ./daq_evtbuild and run tcl_to_ora accordingly.

Best regards

Benjamin
>
>
Just to give a hint about what is going on in these two lines: tcl_to_ora reads some tcl-script from stdin, which in turn should consist of entries of the type

set name(idx) value

where name is the name of a process, e.g. the eventbuild process, idx is the name of the variable to be written, value the value of the variable.

name has to be transformed to be "eb1pp" in case of the evtbuild process and "eb1gp" in case of the netmem process, and therefore stdin can first be filtered by some sed-script supplied with the "-e" option. "-s hades" gives the setup to be consulted to receive additional information like "which beamtime", which "running process", etc.

this process should run permanently while there will be an <EOF> after every open() / close()-sequence on the file eb_p.tcl. Not stopping in case of an <EOF> is caused by the tcl_to_ora-option "-p". So tail -f follows the growing eb_s.tcl and puts it to stdout. nohup sh -c "..." & opens a shell which will not receive any signal in case of the parent process (the interactive shell) dies, so this process of putting tcl-information to the database keeps running even if you close the terminal.
 


Deleted:
<
<
Hello again,

correction to the last entry: I just heard from Mathias that ./daq_evtbuild is not used anymore for official runs.

Anyhow, the argument "-e 's/^set [^(]*daq_evtbuild/set eb1pp/g; s/^set [^(]*daq_netmem/set eb1gp/g'" should cover most possibilities, so I place this instead of the former entry's one.

One more remark: If You try to bring back a long list of files to Oracle, don't get impatient and think anything failed only because the program does not return immediately. In fact, there are only something like 5-10 runs inserted per minute, so if You have a backlog of more than 100 files (even the files going to /dev/null are counted), it may take a while - e.g. while I'm writing these lines, tcl_to_ora is still processing ...

Best regards

Benjamin


Hello again,

correction to the last entry: I just heard from Mathias that ./daq_evtbuild is not used anymore for official runs.

Anyhow, the argument "-e 's/^set [^(]*daq_evtbuild/set eb1pp/g; s/^set [^(]*daq_netmem/set eb1gp/g'" should cover most possibilities, so I place this instead of the former entry's one.

One more remark: If You try to bring back a long list of files to Oracle, don't get impatient and think anything failed only because the program does not return immediately. In fact, there are only something like 5-10 runs inserted per minute, so if You have a backlog of more than 100 files (even the files going to /dev/null are counted), it may take a while - e.g. while I'm writing these lines, tcl_to_ora is still processing ...

Best regards

Benjamin
 
Added:
>
>
In case the beamtime changes, to have correct entries to the database, you have to set the current beamtime for the setup correctly. This can be done by experts directly in the database (TABLE daq.active_experiment), or by the out-of-date gui of the intermediate runctrl (run_manager).

Similar techniques apply to other information like the active operator (TABLE daq.active_operator).

Other useful information about the current state (e.g. in case of a crash of tcl_to_ora) can be found in TABLE daq.active_run and TABLE daq.run).
 
Changed:
<
<
-- BenjaminSailer - 27 Aug 2004
>
>
-- BenjaminSailer - Compilation from 27 Aug 2004 to 01 Feb 2005
 
This site is powered by FoswikiCopyright © by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding Hades Wiki? Send feedback
Imprint (in German)
Privacy Policy (in German)