Status of eventbuilder for beamtime 2012

Here documentation of setup and changes

Eventbuilder machines

Hardware

lxhadesdaq

central machine

lxhadeb01

Previous master event builder

lxhadeb02

lxhadeb03

lxhadeb04

lxhadeb05

New master event builder Did NOT move disks from lxhadeb01, since adaptec controllers are not compatible!

For the moment, use only first 4 devices ("spares") that were existing before: Partitioned and mounted to ext4

• use fdisk  /dev/sde to create primary partition (menu)
• use mkfs -t ext4 /dev/sde1 to format
• use blkid /dev/sde1 to find out partition UUID
• set this into /etc/fstab

NEW: Added 10 new disks of 3 TB each (WD Green caviar), so we have now 14 data disks at lxhadeb05

-- JoernAdamczewski - 20 Feb 2012

Required adjustments to reduce number of disks:

• daq_disks demon does not need to explictely exclude upper numbers if they are not present with df command. Besides, new version of daq_disks has autoexclude option for not accessible partitions: /home/hadaq/bin/daq_disks -a -s 10
• cleanup.pl must be patched in some loops from 22 to 4 disks, now at cleanup_4disks.pl (TODO: improve script)

Configuration

Eventbuilder configuration is done by two central files on lxhadesdaq:

• /home/hadaq/trbsoft/daq/evtbuild/eb.conf
• /home/hadaq/trbsoft/daq/hub/register_configgbe_ip.db

Processes and services

Data export to oracle

On lxhadesdaq.

daq 2 oracle

Configuration of DAQ frontends is exported to textfiles in /home/hadaq/oper/daq2ora. This is used as input for /home/hadaq/trbsoft/daq/oracle/daq2ora_client.pl NOTE: This script must be started within this directory:

cd /home/hadaq/trbsoft/daq/oracle/
./daq2ora_client.pl -o &

Option -o must be specified for real oracle insert!

runinfo to oracle

Done by perl script /home/hadaq/oper/runinfo2ora.pl. Old startscript runinfo2orastart.sh. Reads text files eb_runinfo2ora_N.txt produced by eventbuilderN on nfs mounted partitions: /home/hadaq/oper/oper_i (N=eventbuilder number, i=eventbuilder server).

Problem: after upgrading to perl 5.10, crash due to multithreading? New startscript runinfo2orastart_parallel.sh will launch separate processes for each eb file. This workaround seems to do.

EPICS iocs for eventbuilders

this is a short summary, see EventBuilderRunControl or chapter 24 of trbnet documentation for full description

Each eventbuilder+netmem process needs one EPICS ioc process; connection is done via shared memory. The EPICS iocs should be started before eventbuilder are started.

This is done by eventbuilder startup script on node lxhadesdaq with option -i (for ioc):

• start_eb_gbe.pl -i stop -n 1-16 : stops iocs for eventbuilders n
• start_eb_gbe.pl -i start -n 1-16 : starts iocs for eventbuilders. NOTE: before starting, all running ioc processes are killed. If single eventbuilder number is specified, this will restart all iocs on that server machine

The ioc processes run under account scs, e.g.: SCREEN -dmS ioc_eb12 ../../bin/linux-x86_64/ebctrl st_eb12.cmd

Monitoring

Eventbuilder machine control/IPMI

Installed under account hadaq on machine hades30 (vncviewer hades30:1). Standalone ipmi viewer is at /home/hadaq/ipmi/IPMIView20-Class_v2.9.4_Build111230. Start in this directoy by calling IPMIView20.sh

Connection to machines:

• lxhadeb01 - 10.6.99.1 (sm-lxhadeb01.itm.gsi.de)
• lxhadeb02 - 10.6.104.104 (sm-lxhadeb02.itm.gsi.de)
• lxhadeb03 - 10.6.104.103 (sm-lxhadeb03.itm.gsi.de)
• lxhadeb04 - 10.6.104.102 (sm-lxhadeb04.itm.gsi.de)
• lxhadeb05 - 10.6.1.4 (sm-lxhadeb05.itm.gsi.de)

Login as user hadaq with ROOT (!) password!

Eventbuilder disk control/adaptec storage manager

On lxhadeb01 and lxhadeb05: login as root, call /usr/StorMan/StorMan.sh

Monitoring with ICINGA system

click here for full information on Icinga setup

The icinga web interface is at http://hadesdaq02/icinga/ (inside gsi). A tunnel export is available at http://cerberus.x-matter.uni-frankfurt.de:9999/icinga/

DAQ Monitoring frontend

The web frontend for lots of HADES monitoring scripts is available at http://hadesdaq02/mon (inside gsi), or exported at http://cerberus.x-matter.uni-frankfurt.de:8888/mon/

-- JoernAdamczewski - 16 Feb 2012

Howtos

-- JoernAdamczewski - 01 Feb 2012