Difference: EventBuilder2012 (1 vs. 12)

Revision 12
29 Mar 2012 - Main.JoernAdamczewski
Line: 1 to 1
 
META TOPICPARENT name="EventBuilder"

Status of eventbuilder for beamtime 2012

Here documentation of setup and changes
Line: 116 to 116
  click here for full information on Icinga setup

The icinga web interface is at http://hadesdaq02/icinga/ (inside gsi). A tunnel export is available
Changed:
<
<
at http://cerberus.x-matter.uni-frankfurt.de:8888/icinga/
>
>
at http://cerberus.x-matter.uni-frankfurt.de:9999/icinga/
 

DAQ Monitoring frontend

Revision 11
20 Feb 2012 - Main.JoernAdamczewski
Line: 1 to 1
 
META TOPICPARENT name="EventBuilder"

Status of eventbuilder for beamtime 2012

Here documentation of setup and changes
Line: 30 to 30
 
  • use blkid /dev/sde1 to find out partition UUID
  • set this into /etc/fstab
Added:
>
>

NEW: Added 10 new disks of 3 TB each (WD Green caviar), so we have now 14 data disks at lxhadeb05

-- JoernAdamczewski - 20 Feb 2012
  Required adjustments to reduce number of disks:
  • daq_disks demon does not need to explictely exclude upper numbers if they are not present with df command. Besides, new version of daq_disks has autoexclude option for not accessible partitions: =/home/hadaq/bin/daq_disks -a -s 10 =
  • cleanup.pl must be patched in some loops from 22 to 4 disks, now at cleanup_4disks.pl (TODO: improve script)
Revision 10
16 Feb 2012 - Main.JoernAdamczewski
Line: 1 to 1
 
META TOPICPARENT name="EventBuilder"

Status of eventbuilder for beamtime 2012

Here documentation of setup and changes
Line: 31 to 31
 
  • set this into /etc/fstab

Required adjustments to reduce number of disks:
Changed:
<
<
  • daq_disks demon needs to explictely exclude upper numbers: /home/hadaq/bin/daq_disks -s 10 -e 5 -e 6 -e 7 -e 8 -e 9 -e 10 -e 11 -e 12 -e 13 -e 14 -e 15 -e 16 -e 17 -e 18 -e 19 -e 20 -e 21 -e 22
>
>
  • daq_disks demon does not need to explictely exclude upper numbers if they are not present with df command. Besides, new version of daq_disks has autoexclude option for not accessible partitions: =/home/hadaq/bin/daq_disks -a -s 10 =
 
  • cleanup.pl must be patched in some loops from 22 to 4 disks, now at cleanup_4disks.pl (TODO: improve script)
Line: 68 to 68
  New startscript runinfo2orastart_parallel.sh will launch separate processes for each eb file. This workaround seems to do.
Added:
>
>

EPICS iocs for eventbuilders

this is a short summary, see EventBuilderRunControl or chapter 24 of trbnet documentation for full description

Each eventbuilder+netmem process needs one EPICS ioc process; connection is done via shared memory. The EPICS iocs should be started before eventbuilder are started.

This is done by eventbuilder startup script on node lxhadesdaq with option -i (for ioc):
  • start_eb_gbe.pl -i stop -n 1-16 : stops iocs for eventbuilders n
  • start_eb_gbe.pl -i start -n 1-16 : starts iocs for eventbuilders. NOTE: before starting, all running ioc processes are killed. If single eventbuilder number is specified, this will restart all iocs on that server machine

The ioc processes run under account scs, e.g.: SCREEN -dmS ioc_eb12 ../../bin/linux-x86_64/ebctrl st_eb12.cmd

 

Monitoring

Eventbuilder machine control/IPMI

Line: 91 to 106
  login as root, call /usr/StorMan/StorMan.sh
Changed:
<
<

EPICS iocs for eventbuilders

this is a short summary, see EventBuilderRunControl or chapter 24 of trbnet documentation for full description

Each eventbuilder+netmem process needs one EPICS ioc process; connection is done via shared memory. The EPICS iocs should be started before eventbuilder are started.

This is done by eventbuilder startup script with option -i (for ioc):
  • start_eb_gbe.pl -i stop -n 1-16 : stops iocs for eventbuilders n
  • start_eb_gbe.pl -i start -n 1-16 : starts iocs for eventbuilders. NOTE: before starting, all running ioc processes are killed. If single eventbuilder number is specified, this will restart all iocs on that server machine

The ioc processes run under account scs, e.g.: SCREEN -dmS ioc_eb12 ../../bin/linux-x86_64/ebctrl st_eb12.cmd
>
>

Monitoring with ICINGA system

click here for full information on Icinga setup
 
Added:
>
>
The icinga web interface is at http://hadesdaq02/icinga/ (inside gsi). A tunnel export is available at http://cerberus.x-matter.uni-frankfurt.de:8888/icinga/
 
Deleted:
<
<

Monitored properties with ICINGA system

 
Changed:
<
<
click here for full information on Icinga setup
>
>

DAQ Monitoring frontend

The web frontend for lots of HADES monitoring scripts is available at http://hadesdaq02/mon (inside gsi), or exported at http://cerberus.x-matter.uni-frankfurt.de:8888/mon/
 
Deleted:
<
<

ICINGA configuration and restart

  • Icinga server runs on hadesdaq02.
  • Configuration files are at /etc/icinga
  • Restart icinga services with /etc/init.d/icinga reload. The starting error logfile is = /var/log/icinga/config.err=. The regular logfile is = /var/log/icinga/icinga.log=
  • Icinga/nagios plugins are at /usr/lib/nagios/plugins. On Eventbuilder machine, additional plugin scripts are at /home/hadaq/nagios/plugins (connected to cvs)
  • Icinga configuration and plugins are in hadaq cvs repository ext:hadaq@lxi001:/misc/hadesprojects/daq/cvsroot under module nagios
 
Deleted:
<
<

Monitored EB services

 

Changed:
<
<
-- JoernAdamczewski - 09 Feb 2012
>
>
-- JoernAdamczewski - 16 Feb 2012
 

Revision 9
10 Feb 2012 - Main.JoernAdamczewski
Line: 1 to 1
 
META TOPICPARENT name="EventBuilder"

Status of eventbuilder for beamtime 2012

Here documentation of setup and changes
Line: 54 to 54
  NOTE: This script must be started within this directory:
cd /home/hadaq/trbsoft/daq/oracle/
Changed:
<
<
./daq2ora_client.pl &
>
>
./daq2ora_client.pl -o &
 
Added:
>
>
Option -o must be specified for real oracle insert!
 

runinfo to oracle

Done by perl script /home/hadaq/oper/runinfo2ora.pl. Old startscript
Revision 8
09 Feb 2012 - Main.JoernAdamczewski
Line: 1 to 1
 
META TOPICPARENT name="EventBuilder"

Status of eventbuilder for beamtime 2012

Here documentation of setup and changes
Line: 105 to 105
 

Monitored properties with ICINGA system

Changed:
<
<
put here all eventbuilder related icinga information
>
>

click here for full information on Icinga setup
 

ICINGA configuration and restart

  • Icinga server runs on hadesdaq02.
Revision 7
09 Feb 2012 - Main.JoernAdamczewski
Line: 1 to 1
 
META TOPICPARENT name="EventBuilder"

Status of eventbuilder for beamtime 2012

Here documentation of setup and changes
Line: 51 to 51
 

daq 2 oracle

Configuration of DAQ frontends is exported to textfiles in /home/hadaq/oper/daq2ora. This is used as input for /home/hadaq/trbsoft/daq/oracle/daq2ora_client.pl
Changed:
<
<
>
>
NOTE: This script must be started within this directory:
cd /home/hadaq/trbsoft/daq/oracle/
./daq2ora_client.pl &
 

runinfo to oracle

Done by perl script /home/hadaq/oper/runinfo2ora.pl. Old startscript
Line: 101 to 105
 

Monitored properties with ICINGA system

Changed:
<
<
put here all eventbuilder related icinga variables
>
>
put here all eventbuilder related icinga information

ICINGA configuration and restart

  • Icinga server runs on hadesdaq02.
  • Configuration files are at /etc/icinga
  • Restart icinga services with /etc/init.d/icinga reload. The starting error logfile is = /var/log/icinga/config.err=. The regular logfile is = /var/log/icinga/icinga.log=
  • Icinga/nagios plugins are at /usr/lib/nagios/plugins. On Eventbuilder machine, additional plugin scripts are at /home/hadaq/nagios/plugins (connected to cvs)
  • Icinga configuration and plugins are in hadaq cvs repository ext:hadaq@lxi001:/misc/hadesprojects/daq/cvsroot under module nagios

Monitored EB services

-- JoernAdamczewski - 09 Feb 2012
 

Revision 6
03 Feb 2012 - Main.JoernAdamczewski
Line: 1 to 1
 
META TOPICPARENT name="EventBuilder"

Status of eventbuilder for beamtime 2012

Here documentation of setup and changes
Line: 65 to 65
 

Monitoring

Eventbuilder machine control/IPMI

Changed:
<
<
Installed under account hadaq on machine hades30.
>
>
Installed under account hadaq on machine hades30 (vncviewer hades30:1).
  Standalone ipmi viewer is at /home/hadaq/ipmi/IPMIView20-Class_v2.9.4_Build111230. Start in this directoy by calling IPMIView20.sh

Connection to machines:
Changed:
<
<
  • lxahdeb01 - 10.6.99.1
>
>
  • lxhadeb01 - 10.6.99.1 (sm-lxhadeb01.itm.gsi.de)
  • lxhadeb02 - 10.6.104.104 (sm-lxhadeb02.itm.gsi.de)
  • lxhadeb03 - 10.6.104.103 (sm-lxhadeb03.itm.gsi.de)
  • lxhadeb04 - 10.6.104.102 (sm-lxhadeb04.itm.gsi.de)
 
  • lxhadeb05 - 10.6.1.4 (sm-lxhadeb05.itm.gsi.de)
Line: 82 to 85
  login as root, call /usr/StorMan/StorMan.sh
Added:
>
>

EPICS iocs for eventbuilders

this is a short summary, see EventBuilderRunControl or chapter 24 of trbnet documentation for full description

Each eventbuilder+netmem process needs one EPICS ioc process; connection is done via shared memory. The EPICS iocs should be started before eventbuilder are started.

This is done by eventbuilder startup script with option -i (for ioc):
  • start_eb_gbe.pl -i stop -n 1-16 : stops iocs for eventbuilders n
  • start_eb_gbe.pl -i start -n 1-16 : starts iocs for eventbuilders. NOTE: before starting, all running ioc processes are killed. If single eventbuilder number is specified, this will restart all iocs on that server machine

The ioc processes run under account scs, e.g.: SCREEN -dmS ioc_eb12 ../../bin/linux-x86_64/ebctrl st_eb12.cmd

Monitored properties with ICINGA system

put here all eventbuilder related icinga variables

 

Howtos

Revision 5
02 Feb 2012 - Main.JoernAdamczewski
Line: 1 to 1
 
META TOPICPARENT name="EventBuilder"

Status of eventbuilder for beamtime 2012

Here documentation of setup and changes
Line: 22 to 22
 

lxhadeb05

New master event builder
Changed:
<
<
Moved disks 6 to 23 from lxhadeb01
>
>
Did NOT move disks from lxhadeb01, since adaptec controllers are not compatible!
 
Added:
>
>
For the moment, use only first 4 devices ("spares") that were existing before: Partitioned and mounted to ext4
  • use fdisk  /dev/sde to create primary partition (menu)
  • use mkfs -t ext4 /dev/sde1 to format
  • use blkid /dev/sde1 to find out partition UUID
  • set this into /etc/fstab

Required adjustments to reduce number of disks:
  • daq_disks demon needs to explictely exclude upper numbers: /home/hadaq/bin/daq_disks -s 10 -e 5 -e 6 -e 7 -e 8 -e 9 -e 10 -e 11 -e 12 -e 13 -e 14 -e 15 -e 16 -e 17 -e 18 -e 19 -e 20 -e 21 -e 22
  • cleanup.pl must be patched in some loops from 22 to 4 disks, now at cleanup_4disks.pl (TODO: improve script)
 

Configuration

Revision 4
02 Feb 2012 - Main.JoernAdamczewski
Line: 1 to 1
 
META TOPICPARENT name="EventBuilder"

Status of eventbuilder for beamtime 2012

Here documentation of setup and changes
Line: 8 to 8
 

Hardware

Added:
>
>

lxhadesdaq

central machine
 

lxhadeb01

Previous master event builder

Revision 3
02 Feb 2012 - Main.JoernAdamczewski
Line: 1 to 1
 
META TOPICPARENT name="EventBuilder"

Status of eventbuilder for beamtime 2012

Here documentation of setup and changes
Line: 53 to 53
 

Monitoring

Added:
>
>

Eventbuilder machine control/IPMI

Installed under account hadaq on machine hades30. Standalone ipmi viewer is at /home/hadaq/ipmi/IPMIView20-Class_v2.9.4_Build111230. Start in this directoy by calling IPMIView20.sh

Connection to machines:
  • lxahdeb01 - 10.6.99.1
  • lxhadeb05 - 10.6.1.4 (sm-lxhadeb05.itm.gsi.de)

Login as user hadaq with ROOT (!) password!

Eventbuilder disk control/adaptec storage manager

On lxhadeb01 and lxhadeb05: login as root, call=/usr/StorMan/StorMan.sh=
 

Howtos

Revision 2
01 Feb 2012 - Main.JoernAdamczewski
Line: 1 to 1
 
META TOPICPARENT name="EventBuilder"

Status of eventbuilder for beamtime 2012

Here documentation of setup and changes
Added:
>
>
 

Eventbuilder machines

Line: 18 to 19
 

lxhadeb05

* New master event builder*
Added:
>
>
Moved disks 6 to 23 from lxhadeb01
 

Line: 32 to 34
 

Processes and services

Added:
>
>

Data export to oracle

On lxhadesdaq.

daq 2 oracle

Configuration of DAQ frontends is exported to textfiles in /home/hadaq/oper/daq2ora. This is used as input for /home/hadaq/trbsoft/daq/oracle/daq2ora_client.pl

runinfo to oracle

Done by perl script /home/hadaq/oper/runinfo2ora.pl. Old startscript runinfo2orastart.sh. Reads text files eb_runinfo2ora_N.txt produced by eventbuilderN on nfs mounted partitions: /home/hadaq/oper/oper_i (N=eventbuilder number, i=eventbuilder server).

Problem: after upgrading to perl 5.10, crash due to multithreading? New startscript runinfo2orastart_parallel.sh will launch separate processes for each eb file. This workaround seems to do.
 

Monitoring

 
This site is powered by FoswikiCopyright © by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding Hades Wiki? Send feedback
Imprint (in German)
Privacy Policy (in German)