Event Builder Servers 2025

List of available servers

All machines are Supermicro servers.

hostnames (gsi)	hostnames (hades)	OS	cores	memory	disk /data01	disk other	purpose	remarks
lxhadesdaq	lxhadesdaqp	Deb11 5.10.226-1 (2024-10-03)	32 (AMD EPYC 7351P)	64G	-	7.3 T drbd + 440G /sdai	central server nfs/dhcp/dns; Oracle client gateway	hardware of two machines lxhadesdaq1/2 with shared drbd
lxhadeb08	lxhadeb08p	Deb11 5.10.221-1 (2024-07-14))	32 (AMD EPYC 7351)	252G	73 T	44G /sdai	BNET eb 1/5/9/13
lxhadeb10	lxhadeb10p	Deb11 5.10.223-1 (2024-08-10)	32 (AMD EPYC 7351)	252G	73 T	386G /sdai	BNET eb 3/7/11/15	also hosts /glusterfs/lgad_georep
lxhadeb17o	lxhadeb17po	Deb11 5.10.237-1 (2025-05-19)	48 (AMD EPYC 7443P)	252G	64 T	443G /dev	BNET input 4	additional fast optical nic with different hostname
lxhadeb18o	lxhadeb18po	Deb11 5.10.237-1 (2025-05-19)	48 (AMD EPYC 7443P)	252G	64 T	443G /dev	BNET input 5	additional fast optical nic with different hostname
lxhadeb14	lxhadeb14p	Deb11 5.10.226-1 (2024-10-03)	64 (AMD EPYC 7302)	252G	51 T	224G /sdbi	BNET input 1	also hosts /glusterfs/lgad_georep
lxhadeb15	lxhadeb15p	Deb11 5.10.223-1 (2024-08-10)	64 (AMD EPYC 7302)	252G	51 T	224G /sdai	BNET input 2	also hosts /glusterfs/lgad_georep
lxhadeb16	lxhadeb16p	Deb11 5.10.226-1 (2024-10-03)	64 (AMD EPYC 7302)	252G	51 T	224G /sdbi	BNET input 3	also hosts /glusterfs/lgad_georep
lxhadeb09	lxhadeb09p	Deb11 5.10.223-1 (2024-08-10)	32 (AMD EPYC 7351)	252G	45 T	32G /sdbi	BNET eb 2/6/10/14
lxhadeb11	lxhadeb11p	Deb11 5.10.223-1 (2024-08-10)	32 (AMD EPYC 7351)	252G	44 T	20G /sdai	BNET eb 4/8/12/16
lxhadeb13	lxhadeb13p	Deb12 6.1.140-1 (2025-05-22)	64 (AMD EPYC 7351)	252G	22 T	346G /sdbi	spare for BNET master and raw monitor
lxhadeb12	lxhadeb12p	Deb11 5.10.223-1 (2024-08-10)	32 (AMD EPYC 7351)	252G	7.3 T	25G /sdai	BNET master, dabc raw monitor server

-- Main.JoernAdamczewski - 2025-09-26 - 08:22

Software components to install

example: set up server lxhadeb13 with all required components

lxhadeb13 with Debian GNU/Linux 12 (bookworm), gcc version 12.2.0 (Debian 12.2.0-14+deb12u1), cmake version 3.25.1

system packages

These are missing on a blank lxhadeb* machine- TODO: put already in chef role

• apt-get install aptitude
• aptitude install autoconf
• aptitude install cmake
• aptitude install gcc
• aptitude install g++
• aptitude install gpp
• aptitude install automake
• aptitude install libtool
• aptitude install libz-dev
• aptitude install libattr1-dev
• aptitude install uuid-dev
• aptitude install make

LTSM and FSQ libraries

Sources and all components under /home/hadaq/ltsm/

• /home/hadaq/ltsm/ibm_tsm: ibm tsm system packages (required for ltsm only, not necessarily for fsq). Subfolder with version 8.1.15
  • dpkg -i gskcrypt64_8.0-55.24.linux.x86_64.deb
  • dpkg -i gskssl64_8.0-55.24.linux.x86_64.deb
  • dpkg -i tivsm-api64.amd64.deb
  • dpkg -i tivsm-apicit.amd64.deb
  • dpkg -i tivsm-ba.amd64.deb
  • dpkg -i tivsm-bacit.amd64.deb
  • dpkg -i gskssl64_8.0-55.24.linux.x86_64.deb
• /home/hadaq/ltsm/certs: certificate that must be registered in system by using:
  • dsmcert -add -server ltsmi1 -file  /home/hadaq/ltsm/certs/cert256-ltsmi1-25092023.cert
  • NOTE: dsmcert is part of the above packages, but installation does not provide all libs in search path!
  • had to make some soft links manually:
    • ln -s /usr/local/ibm/gsk8_64/lib64/libgsk8ssl_64.so /lib
    • ln -s /usr/local/ibm/gsk8_64/lib64/libgsk8iccs_64.so /lib
    • ln -s /usr/local/ibm/gsk8_64/lib64/libgsk8km_64.so /lib
    • ln -s /usr/local/ibm/gsk8_64/lib64/libgsk8cms_64.so /lib
• /home/hadaq/ltsm/install: the installation directory (redirects to install2025)
• /home/hadaq/ltsm/ltsm : source of ltsm lib git clone of https://github.com/GSI-HPC/ltsm
  • build with ./autogen.sh
  • ./configure --prefix="/home/hadaq/ltsm/install2025" CFLAGS='-g -DDEBUG -O0' --enable-test
  • make; make install
  • manually provide link for binaries in system path at /usr/local/bin/ltsmc to /home/hadaq/ltsm/install2025/bin/ltsmc
• /home/hadaq/ltsm/fsq : source of fsq lib git clone of https://github.com/GSI-HPC/fsq
  • build with ./autogen.sh
  • ./configure --prefix="/home/hadaq/ltsm/install2025" CFLAGS='-g -DDEBUG -O0' --enable-benchmark
  • make; make install
  • manually provide link for binaries in system path at /usr/local/bin/fsqc to /home/hadaq/ltsm/install2025/bin/fsqc; dito for fsqbench
• For proper compilation of DABC: we need to copy the config.h include files manually to
  • cp /home/hadaq/ltsm/config.h /home/hadaq/ltsm/install/include/ltsm/
  • cp /home/hadaq/fsq/config.h /home/hadaq/ltsm/install/include/fsq/
  • This should be done by "make install", but it is not! TODO: feedback to Samuel.

Adjustment of tivoli config files:

• /opt/tivoli/tsm/client/api/bin64/dsm.sys - same as on other lxhadeb
• /opt/tivoli/tsm/client/api/bin64/dsm.opt - same as on other lxhadeb

DABC and stream frameworks

• Sources and all components under /home/hadaq/soft/
  • /home/hadaq/soft/dabc/git: source of dabc git clone of https://github.com/gsi-ee/dabc.git
  • /home/hadaq/soft/dabc/install/master: most recent installation directory for dabc
  • /home/hadaq/soft/streamframe/git: source of stream analysis framework, clone of https://github.com/gsi-ee/stream.git
  • /home/hadaq/soft/streamframe/install/master: most recent installation directory of stream
• Proper rebuild is done by script /home/hadaq/bin/update_dabc_install.sh (link to repository version)
• The binaries, login scripts and xml configuration files are provided as soft links under /home/hadaq/soft/dabc/bin (directs to installation directory)
• The binaries and login scripts are finally linked into path at /home/hadaq/bin (direct to /home/hadaq/soft/dabc/bin or into installation folders)
• The xml configuration files are finally linked into /home/hadaq/oper/ (direct to /home/hadaq/soft/dabc/bin)

-- Main.JoernAdamczewski - 2025-09-24 - 16:36

Scripts in hadaq path

hadaq@lxhadeb13:~/bin$ ls -l
total 24
lrwxrwxrwx 1 hadaq hades   76 Sep 24 17:30 archivecal_ltsm.sh -> /home/hadaq/soft/dabc/install/master/plugins/ltsm/app/archivecal_ltsm_fsq.sh
-rwxr-x--- 1 hadaq hades 7051 Sep 25 10:40 cleanup.pl
lrwxrwxrwx 1 hadaq hades   76 Sep 24 17:29 cleanup_evtbuild.pl -> /home/hadaq/soft/dabc/install/master/plugins/hadaq/hades/cleanup_evtbuild.pl
lrwxrwxrwx 1 hadaq hades   14 Sep 24 16:01 cmake -> /usr/bin/cmake
lrwxrwxrwx 1 hadaq hades   41 Sep 24 17:33 dabc_master -> /home/hadaq/soft/dabc/bin/dabc_exe.master
lrwxrwxrwx 1 hadaq hades   42 Sep 24 17:29 dabclogin -> /home/hadaq/soft/dabc/bin/dabclogin.master
lrwxrwxrwx 1 hadaq hades   76 Sep 24 17:31 restart_bnetmaster.sh -> /home/hadaq/soft/dabc/install/master/plugins/hadaq/app/restart_bnetmaster.sh
lrwxrwxrwx 1 hadaq hades   72 Sep 24 17:31 restart_rawmon.sh -> /home/hadaq/soft/dabc/install/master/plugins/hadaq/app/restart_rawmon.sh
lrwxrwxrwx 1 hadaq hades   55 Sep 24 17:30 streamlogin -> /home/hadaq/soft/streamframe/install/master/streamlogin
lrwxrwxrwx 1 hadaq hades   79 Sep 24 17:11 update_dabc_install.sh -> /home/hadaq/soft/dabc/install/master/plugins/hadaq/hades/update_dabc_install.sh

Note1: restart_bnetmaster.sh and restart_rawmon.sh are only required on dabc master machine. lxhadeb13 is meant as spare master for lxhadeb12

Note2: cleanup.pl is demon to remove oldest files from /data01 if necessary.

Config files in oper directory

hadaq@lxhadeb13:~/oper$ ls -l
total 0
lrwxrwxrwx 1 hadaq hades 56 Sep 24 17:37 BnetBuilderHadesFSD.master.xml -> /home/hadaq/soft/dabc/bin/BnetBuilderHadesFSD.master.xml
lrwxrwxrwx 1 hadaq hades 51 Sep 24 17:37 BnetInputHades.master.xml -> /home/hadaq/soft/dabc/bin/BnetInputHades.master.xml

Environment for raw monitor

This is on special eventbuilder servers lxhadeb12 (and spares lxhadeb12) only

The raw monitor is started by script /home/hadaq/bin/ restart_rawmon.sh. This used to be done remotely by a script from DAQ operator gui (lxhadesdaq:/home/hadaq/trbsoft/daq/control/gui/), but most recently the script was invoked directly by expert only

• working directory is /data01/monitor/hades/dabc
• This contains first.C with analysis setup
• DABC runtime (with webserver) is provided by hades.xml
• Logfile is written to /home/hadaq/oper/DabcRawMon.log

crontab scripts

For user hadaq:

# m h  dom mon dow   command
@reboot /home/hadaq/bin/cleanup.pl -d &
0 0 * * * /home/hadaq/oper/move_oldruninfos.sh
*/4 *    * * *   /bin/ping -c1 -W1 192.168.104.250 >/dev/null 2>&1

The ping (to a non existing address) will frequently revive certain network switches. The cleanup.pl is old demon to remove oldest hld files on /data01 at high watermark.

For user root:

# m h  dom mon dow   command
*/2 *    * * *   /root/bin/set_eth_affinity_lxhadeb13.pl > /dev/null 2>&1

This script frequently resets the cpu affinity of the network cards interrupts to the desired assignments. Otherwise these may be spoiled by the generic cpu management of the system and can get in conflict with the affinities set for the dabc eventbuilder processes (as defined in /home/hadaq/trbsoft/hadesdaq/evtbuild/start_eb_gbe.pl).

Copied from lxhadeb14 to lxhadeb13 (both are 64 cores now)-> adjusted for debian12 and different nic names!

network interface configurations

/etc/sysctl.d/hades_ip.conf

# JAM 15-06-2020 - tuning for event builders as tested by Michael T.
net.ipv4.ipfrag_high_thresh=41943040 
net.ipv4.ipfrag_max_dist=1024
# JAM 8-2021 this was missing in defaults:
net.core.rmem_max=21299200
net.core.rmem_default=21299200

activate after change with sysctl --system

nfs exports

• the /home/hdaq/oper directory needs to be exported to lxhadesdaq for optionally export runinfo files to Oracle. Also export it to hadesp33 for any hmon issues with TDC calibration files.
• the /data01 may be exported to online analysis servers hadesp63, hadesp66, hadesp67

Done in /etc/exports (copy from any other lxhadeb*)

hadaq@lxhadeb09:~/soft/streamframe/git$ cat /etc/exports
# /etc/exports: the access control list for filesystems which may be exported
#               to NFS clients.  See exports(5).
#
# Example for NFSv2 and NFSv3:
# /srv/homes       hostname1(rw,sync,no_subtree_check) hostname2(ro,sync,no_subtree_check)
#
# Example for NFSv4:
# /srv/nfs4        gss/krb5i(rw,sync,fsid=0,crossmnt,no_subtree_check)
# /srv/nfs4/homes  gss/krb5i(rw,sync,no_subtree_check)
#

#old monitoring machine is now also EBserver/data01 192.168.105.13(rw,sync,no_root_squash,subtree_check)
/data01 192.168.103.126(rw,sync,no_root_squash,subtree_check)
/data01 192.168.103.141(rw,sync,no_root_squash,subtree_check)
/data01 192.168.103.146(rw,sync,no_root_squash,subtree_check)
/home/hadaq/oper 192.168.100.50(rw,sync,no_root_squash,subtree_check)
/home/hadaq/oper 192.168.105.51(rw,sync,no_root_squash,subtree_check)
/home/hadaq/oper 192.168.105.52(rw,sync,no_root_squash,subtree_check)
/home/hadaq/oper 192.168.100.110(rw,sync,no_root_squash,subtree_check)
/home/hadaq/oper hadesp79(rw,sync,no_root_squash,subtree_check)

NOTE: on lxhadeb13 this was already exisiting!

Refresh the export via "exportfs -ra".

redirection of syslog

The local syslog should be redirected to lxhadesdaq, so commone eventbuilder messages show up in file /home/hadeslog/messages. How to do it?

Exchange of BNET master machine

To replace BNET master (currently lxahdeb12) by e.g. lxhadeb13, a number of adjustments must be done:

• in /home/hadaq/trbsoft/hadesdaq/evtbuild/start_eb_gbe.pl : modify variable my $bnetmaster = "lxhadeb12"; (l.413). This is forwarded as environment variable BNETMASTER when the dabc bnet input and builder processes are started and used from the configuration .xml scripts.
• in /home/hadaq/trbsoft/hadesdaq/control/gui/eb/Restart_BNET_control : modify hostname in statement ssh hadaq@lxhadeb12
• several scripts at /home/hadaq/trbsoft/hadesdaq/hmon/
  • hmon_eb_rate.pl: adjust my $masterurl = 'http://lxhadeb12p:8099/';
  • hmon_spill.pl: adjust my $masterurl = 'http://lxhadeb12:8099/';
  • hmon_eberrbits_withreset.pl: adjust my $masterurl = 'http://lxhadeb12:8099/';
  • permanent/hmon_archiver.pl: exchange hostname in my $filetype = get("http://lxhadeb12:8099/Master/...
  • permanent/hmon_eb_missing_data.pl: adjust my $masterurl = 'http://lxhadeb12:8099/';
  • permanent/hmon_eb_run.pl: adjust my $masterurl = 'http://lxhadeb12:8099/';

Exchange of rawmon machine

To replace the rawmon host machine (currently lxahdeb12), a number of adjustments must be done:

• several scripts at /home/hadaq/trbsoft/hadesdaq/hmon/
  • hmon_tdc_totratio.pl adjust hostname in get ("http://lxhadeb12:8090/HADES/Run/...;
• no startup button for rawmon anymore, so adjustments of control/gui script not necessary

---

-- JoernAdamczewski - 2025-09-24