ICINGA

General setup

Icinga is a system and network monitoring application. It watches hosts and services that you specify, alerting you when things go bad and when they get better.

ICINGA configuration and restart

• Icinga server runs on hadesdaq02.
• Restart icinga services with /etc/init.d/icinga reload. The starting error logfile is /var/log/icinga/config.err. The regular logfile is /var/log/icinga/icinga.log
• Configuration files are at /etc/icinga. This is under cvs version control.
• Plugins are at /usr/lib/nagios/plugins. Note that on the remote nodes monitored, additional plugins are located at /home/hadaq/nagios/plugins; these scripts are under cvs version control.
• Config files: Main file is /etc/icinga/icinga.cfg, this will include other config files. The configuration for the commands and servers are in subfolder objects:
  • commands.cfg: Definition of service checks and event handlers
  • localhost.cfg: Definition of services to be tested on localhost, i.e. the icinga webserver hadesdaq02 itself. This is also the central slow control machine!
  • hosts_eb_servers.cfg: Definition of nodes and services for eventbuilder and daq server
  • hosts_power.cfg: Definition of nodes and services for power supplies
  • hosts_etrax.cfg: Definition of nodes and services for etrax frontend cpus: NOTE: this file is auto-generated by script daq2icinga.pl from actual DAQ set up.
  • contacts.cfg: Who will get a notification email from icinga
• NOTE: to update hosts_etrax.cfg, please run on hadesdaq02 the script /etc/icinga/daq2icinga.pl. This will evalulate the epics etrax configuration, and the trb.db or detector components via cross mount of /home/hadaq/trb

Repository for scripts and plugins

All icinga configuration files and user defined plugin scripts are in hadaq repository: hadaq@lxi001.gsi.de:/misc/hadesprojects/daq/cvsroot. Module nagios will contain subfolder plugins for old and new plugins, and subfolder icinga with config files.

What do we monitor?

Hostgroups

On icinga web server go to "Hostgroup Overview". The most important groups are as follows:

• DAQ Servers (daq-servers). Contains the central machine lxhadesdaq
• active EB Servers (eb-servers-active). Eventbuilder servers actually used for data taking.
• EB Servers (eb-servers). ALL Eventbuilder servers, also spares.
• all etrax nodes (etrax). These are all etrax cpus in the HADES experimental area. This nodes are also divided up into hostgroups for the active components:
  • etrax_rpc (rpc).
  • etrax_tof (tof).
  • etrax_start (start).
  • etrax_cts (cts).
  • etrax_scs (scs).
• power supplies (power). Power supply nodes _hadps*_in the HADES experimental area.
• caen crates (caen). Caen HV Power supply nodes hadhvp in the HADES experimental area.
• HADES PCs (hades-pcs). Various interactive machines apart from the DAQ/EB Servers

-- JoernAdamczewski - 16 Feb 2012

Servicegroups

On icinga web server go to "Servicegroup Overview". HINT: Press "View Service Status Grid For All Service Groups" to view all service states at once.

The most important groups are as follows:

General services

• Cpu Load (LOAD). check load (uptime) per number of CPUs
• Check ssh connections (ssh). test of ssh login.
• Linux raid checks (Raid-1). check of /proc/mdstat on machines without adaptec controllers

Eventbuilder and DAQ related

• Eventbuilder data disk status (EB-data-disks). Monitor validity of eventbuilder partitions /data01 .. /data22 with fine granularity.
• Raid controllers (adaptecs). Active check of status of adaptec raid controllers on eventbuilders. NOTE: error may be indicated by state UNKNOWN
• Eventbuilder disks balancing and cleanup (EB-disk-services)). Processes daq_disks and cleanup.pl
• Oracle import services EB and DAQ (oracle-clients). Demons runinfo2oracle and daq2ora that insert ascii files from eventbuilders into hades database.
• Eventbuilder EPICS (EB-epics). Check number of epics iocs for eventbuilders servers and the eventbuilder PV state. NOTE: if EBi-status is CRITICAL, this indicates that eventbuilder process is not running

-- JoernAdamczewski - 16 Feb 2012

Standalone services

There may be single services monitored which are not combined in groups. To display all existing services, on icinga web server go to "Service Detail".

How to act (for beam-time periods)

host	service	status	action
CENTRAL
lxhadesdaq	ping	CRITICAL
lxhadesdaq	/var	CRITICAL	l
lxhadesdaq	CPU load	CRITICAL	wait 20 min. CALL
lxhadesdaq	RUN2ORA	CRITICAL	There should be an icinga eventhandler that automatically solves this problem after 5 minutes. If this fails, login to ssh hadaq@lxhadesdaq and type nohup /home/hadaq/trbsoft/daq/oracle/runinfo2orastart_parallel.sh >/dev/null 2>&1 &
lxhadesdaq	DAQ2ORA	CRITICAL	There should be an icinga eventhandler that automatically solves this problem after 5 minutes. If this fails, login to ssh hadaqlxhadesdaq and type nohup /home/hadaq/trbsoft/daq/oracle/daq2ora_client.pl -d -o &
EVENTBUILDERS
lxhadeb0*	daq_disks	CRITICAL	There should be an icinga eventhandler that automatically solves this problem after 5 minutes. If this fails, login ssh hadaq@lxhadeb0* and type nohup /home/hadaq/bin/daq_disks -a -s 10 >/dev/null 2>&1 &
lxhadeb0*	disks cleanup	CRITICAL	There should be an icinga eventhandler that automatically solves this problem after 5 minutes. If this fails, login to ssh hadaq@xhadeb0* and type nohup /home/hadaq/bin/cleanup.pl >/dev/null 2>&1 &
lxhadeb0*	EB-EPICS procs	CRITICAL	Wait 5 minutes. Login to hadaq@lxhadesdaq and restart all eventbuilder iocs by this: cd /home/hadaq/trbsoft/daq/evtbuild/; ./start_eb_gbe.pl -i start -n 1-16
lxhadeb0*	EBnn-status	CRITICAL	This state means that the eventbuilder process nn itself is not running. This may happen by chance if icinga has just updated status when eventbuilders were being restarted. If this state remains for several minutes during data taking, eventbuilders should be restarted. Do this from operator gui ("StartEB" button), or login to hadaq@lxhadesdaq and restart all eventbuilders by this: cd /home/hadaq/trbsoft/daq/evtbuild/; ./start_eb_gbe.pl -e restart -n 1-16
lxhadeb0*	EBnn-status	UNKNOWN	Means that eventbuilder ioc is not running. Should corresond to CRITICAL state of service EB-EPICS procs on this node. Do as described for EB-EPICS procs.
lxhadeb0*	/dataNN	CRITICAL	This state means that partition /dataNN on eventbuilder lxhadeb0* is not available anymore. This is likely due to xfs filesystem errors. Try to mount partition again: login to root@lxhadeb0* and type umount /dataNN; mount /dataNN

-- JoernAdamczewski - 16 Feb 2012

---

---

Upgrade of icinga2 on hades63

the above documentation is deprecated!

We use this wiki to collect debug information for icinga2 which runs on hades63 since 2020

Existing setup

icinga2

• icinga installation at /etc/icinga2
• local git repository, is mirrored at lxpool:/u/adamczew/git/icinga2 (origin is root@hades63:/etc/icinga2 )
• started with systemctl restart icinga2

icingaweb

• configuration at /etc/icingaweb2
• no repository!
• is configured as apache2 plugin: /etc/apache2/conf.d/icingaweb2.conf
• restart with systemctl restart apache2
• website at http://hades63/icingaweb2 (login as icingaadmin with hadaq PW)

Upgrade problems

After linux upgrade, errors in browser of icingaweb pages:

  
Fatal error: Uncaught PDOException: SQLSTATE[HY000] [2002] 
...
icinga\Application\Web->dispatch() #3 /usr/share/icingaweb2/public/index.php(4): require_once('...') #4 {main} thrown in /usr/share/icinga-php/vendor/vendor/icinga/zf1/library/Zend/Controller/Plugin/Broker.php on line 335

-> it turned out that corresponding mariadb (aka mysql) database could not be started properly!

  
hades63:/etc/icingaweb2 # systemctl status mariadb
à mariadb.service - MariaDB database server
     Loaded: loaded (/usr/lib/systemd/system/mariadb.service; disabled; preset: disabled)
     Active: failed (Result: exit-code) since Fri 2026-06-19 10:32:58 CEST; 25min ago
       Docs: man:mysqld(8)
             https://mariadb.com/kb/en/library/systemd/
    Process: 32055 ExecStartPre=/usr/lib/mysql/mysql-systemd-helper install (code=exited, status=0/SUCCESS)
    Process: 32062 ExecStartPre=/usr/lib/mysql/mysql-systemd-helper upgrade (code=exited, status=1/FAILURE)
        CPU: 535ms

Jun 19 10:31:57 hades63 mysql-systemd-helper[32062]: Checking MySQL configuration for obsolete options...
Jun 19 10:31:57 hades63 mysql-systemd-helper[32062]: Trying to run upgrade of MySQL databases...
Jun 19 10:31:57 hades63 mysql-systemd-helper[32062]: Stale files from previous upgrade detected, cleaned them up
Jun 19 10:31:57 hades63 mysql-systemd-helper[32062]: Running protected MySQL...
Jun 19 10:31:57 hades63 mysql-systemd-helper[32062]: Waiting for MySQL to start
Jun 19 10:32:58 hades63 mysql-systemd-helper[32062]: MySQL is still dead
Jun 19 10:32:58 hades63 mysql-systemd-helper[32062]: MySQL didn't start, can't continue
Jun 19 10:32:58 hades63 systemd[1]: mariadb.service: Control process exited, code=exited, status=1/FAILURE
Jun 19 10:32:58 hades63 systemd[1]: mariadb.service: Failed with result 'exit-code'.
Jun 19 10:32:58 hades63 systemd[1]: Failed to start MariaDB database server.

Serching for logfiles found secret file /var/tmp/mysql-protected.31KkdQ/log_upgrade_run.err (mentioned in the /run/mysql/protecteddir. )

 
2026-06-19 10:31:57 0 [Note] Starting MariaDB 10.11.15-MariaDB source revision cb0d6dd835023a7162ace471cd047161f205dd58 server_uid rq+OkxG3Xfc8Ww307sNqjJ0ztBo= as process 32080
2026-06-19 10:31:57 0 [Note] InnoDB: Compressed tables use zlib 1.2.13
2026-06-19 10:31:57 0 [Note] InnoDB: Number of transaction pools: 1
2026-06-19 10:31:57 0 [Note] InnoDB: Using crc32 + pclmulqdq instructions
2026-06-19 10:31:57 0 [Note] InnoDB: Using Linux native AIO
2026-06-19 10:31:57 0 [Note] InnoDB: innodb_buffer_pool_size_max=128m, innodb_buffer_pool_size=128m
2026-06-19 10:31:57 0 [Note] InnoDB: Completed initialization of buffer pool
2026-06-19 10:31:57 0 [Note] InnoDB: Buffered log writes (block size=512 bytes)
2026-06-19 10:31:57 0 [ERROR] InnoDB: Upgrade after a crash is not supported. The redo log was created with MariaDB 10.5.8. You must start up and shut down MariaDB 10.7 or earlier.
2026-06-19 10:31:57 0 [ERROR] InnoDB: Plugin initialization aborted with error Generic error
2026-06-19 10:31:57 0 [Note] InnoDB: Starting shutdown...
2026-06-19 10:31:57 0 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
2026-06-19 10:31:57 0 [Note] Plugin 'FEEDBACK' is disabled.
2026-06-19 10:31:57 0 [ERROR] Unknown/unsupported storage engine: InnoDB
2026-06-19 10:31:57 0 [ERROR] Aborting

-> current mariadb 10.11.15- is too new to upgrade files created with MariaDB 10.5.8!?

Try zypper install mariadb104 (provides version 10.4):

 
...
2026-06-19 11:23:59 0 [ERROR] InnoDB: Unsupported redo log format. The redo log was created with MariaDB 10.5.8.
2026-06-19 11:23:59 0 [ERROR] InnoDB: Plugin initialization aborted with error Generic error
2026-06-19 11:23:59 0 [Note] InnoDB: Starting shutdown...
2026-06-19 11:24:00 0 [ERROR] Plugin 'InnoDB' registration as a STORAGE ENGINE failed.
2026-06-19 11:24:00 0 [Note] Plugin 'FEEDBACK' is disabled.

This version is too old!

-- Main.JoernAdamczewski - 2026-06-19 - 10:45

Try running older version of mysql from tarball (suse archives are too new): https://mariadb.org/mariadb/all-releases/

• hades63:/home/hadaq/misc/mariadb/mariadb-10.7.8-linux-systemd-x86_64
• backup directory of datadir at /var/lib/mysql-back (copy from /var/lib/mysql)
• backup configuration file at /etc/my.cnf.back (copy from /etc/my.cnf)
• as user mysql: bin/mysqld_safe  --defaults-file=/etc/my.cnf --datadir=/var/lib/mysql --skip-grant-tables
  • need to enable login for mysql in /etc/passwd first by setting login shell
• as user mysql: mysql_upgrade -u root
  • LD_LIBRARY_PATH=/home/hadaq/misc/mariadb/mariadb-10.7.8-linux-systemd-x86_64/lib
  • PATH=/home/hadaq/misc/mariadb/mariadb-10.7.8-linux-systemd-x86_64/bin:/usr/local/bin:/usr/bin:/bin
  • note: mv hidden files and directories from data dir /var/lib/mysql (.local, .bash_history from account home! errror "Incorrect database name '#mysql50#example.database'")
  • result OK
• shutdown mysqld_safe by kill -9 (ctrl-c is blocked) <- better use mysqladmin shutdown
• regular "systemctl restart mariadb" (will use official installation) -> still error "The redo log was created with MariaDB 10.5.8. You must start up and shut down MariaDB 10.7" in /var/tmp/mysql-protected.cyjuRT/log_upgrade_run.err

-> after 2 attempts of this procedure "systemctl restart mariadb" is working again!

-- Main.JoernAdamczewski - 2026-06-23 - 12:56

---

From https://forums.opensuse.org/t/mariadb-doesnt-start-after-upgrade-from-15-4-to-15-5/175878/5

If the start is not working via systemctl, we can also start the instance manually and then execute mariadb_upgrade manually (you should have created a backup of the my.cnf and the data directory / database directory!). So first systemctl stop mariadb Then start mysql as user mysql (mysqld_safe --defaults-file=/my.cnf --datadir=<path where to find the data) Then execute mariadb_uprade

---

-- Main.JoernAdamczewski - 2026-06-19 - 12:50