You are here: Foswiki>DaqSlowControl Web>HadesDaqDocumentation>DetectorControlSystem>SCSLinuxIoc (2018-11-27, PeterZumbruch)Edit Attach

IOC for HADES SCS running on Linux

Introduction - HADES IOC

There is an EPICS IOC running under Linux on machine hadesdaq02.
Right now, it services

the HV for all but MDC,
the LV power supplies,
temperature monitoring (old and new)
connects to the hadcon boards and dreamplugs on the internal HADES VLAN
the stats of all running iocs
as a kind of gateway for process variables available for the GSI LAN
...

Sources and Maintenance

CVS

CVS repository: :ext:scs@lx-pool.gsi.de:/misc/hadesprojects/slowcontrol/cvsroot
CVS module: EPICS/apps/hades

In order to do small maintenance, ...

... connect to hadesdaq02 as user scs using the default scs password.
The configuration is stored under the production directory /home/scs/apps/hades. There you will find a normal IOC directory tree for development and booting the IOC.
- The IOC is named "hades".
To change the database files,
- make your edits in hadesApp/Db
- and afterwards type make.
- Don't forget to commit the changes to CVS.
(re-)start the IOC

In order to do bigger maintenance or development first use the playground ...

... connect to hadesdaq02 as user scs using the default scs password.
The configuration is stored under the development directory /home/scs/playground/EPICS/apps/hades. There you will find a normal IOC directory tree for development and booting the IOC.
- The IOC is named "hades".
To change the database files,
- make your edits in hadesApp/Db
- and afterwards type make.
Make a small dummy test of your database modifications by using softIoc

softIoc -d <Your modified db Files>
On success
- commit the changes to CVS
- change to the production directory /home/scs/apps/hades
- run cvs update
- followed by make
- (re-)start the IOC

Operation

(Re-) Starting the IOC

To restart the server, which is running

hadesdaq02

ssh scs@hadesdaq02

           
       ps x 
     PID   TTY      STAT   TIME COMMAND
    13061 ?        S      0:02 procServ -L ioc-cave-hadesdaq02.log 4813 startEpicsIoc_includingHVSemaphoreCheck.sh ../../bin/linux-x86/hades st.cmd  ### this is the process server 
    13062 pts/4    Ssl+   1:51 startEpicsIoc_includingHVSemaphoreCheck.sh ../../bin/linux-x86/hades st.cmd ### this is the EPICS server

NO:
1. check for remaining semaphores (c.f. chapter "Server Problems")
2. start server from scratch ( c.f. next chapter )
YES: continue

telnet localhost 4813
Switch off auto-restart (procServ), toggle it to be OFF:
<CTRL+T> @@@ Toggled auto restart to ON/OFF
... then hit
<CR>
You should see the epics prompt:
epics>
1. NO you don't see it:
  1. quit telnet session
    <CTRL+]> telnet> quit
  2. check for remaining semaphores (c.f. "Server Problems")
  3. again login to the server
    telnet localhost 4813
2. YES, you have the prompt, i.e. EPICS is running, exit it.
  exit

wait

if the IOC has been running, you first should see the IOC shutting down:

                DEBUG: shutting down crate 0
     Shutdown: successfully disconnected from crate x1
     [...]
     DEBUG: shutting down crate 6 
     Shutdown: successfully disconnected from crate x7 <

then restart the IOC by using:
<CTRL+R>

then you should see it restarting:

                @@@ @@@ @@@ @@@ @@@
     @@@ Received a sigChild for process WXYZ. The process was killed by signal 11
     @@@ Current time: Sun Sep 12 19:29:51 2010
     @@@ Child process is shutting down, auto restart is disabled
     @@@ Use ^R to restart the child, ^Q to quit the server
     @@@ Restarting child "../../bin/linux-x86/hades"
     @@@ The PID of new child "../../bin/linux-x86/hades" is: abcdef
     @@@ @@@ @@@ @@@ @@@
     #!../../bin/linux-x86_64/hades
     ## You may have to change hades to something else
     ## everywhere it appears in this file
     < envPaths
     [...]
     ## Load record instances
     [...]
     iocInit
     Starting iocInit
     ############################################################################
     ## EPICS R3.14.10 $R3-14-10$ $2008/10/27 19:39:04$
     ## EPICS Base built Oct 22 2009
     ############################################################################
     Starting CAEN x527 driver
     pthread_attr_setstacksize error Invalid argument
     iocRun: All initialization complete
     dbl > /home/scs/apps/hades/iocBoot/ioccave/hadesdaq02.dbl
     [...]

To quit from the server without stopping it use the telnet escape (CTRL+]) sequence and quit:
<CTRL+]> telnet> quit
optionally continue with further checks, q.v. below

To start the server ((almost) from scratch)

clean up

make sure procServ is not running by checking for processes:

                     ps x 
        PID   TTY      STAT   TIME COMMAND
       13061 ?        S      0:02 procServ -L ioc-cave-hadesdaq02.log 4813 startEpicsIoc_includingHVSemaphoreCheck.sh ../../bin/linux-x86/hades st.cmd

if it is still there ...
1. brute force: if there is still a procServ process kill it.
2. else: try restart running server

change to scs' procServ director
cd ~scs/procServ && ./ioc-cave.sh
optionally continue with further checks, q.v. below

Further checks

Check the log ~/apps/hades/iocBoot/ioccave/ioc-cave-hadesdaq02.log
1. In case "Semaphore already present" look chapter "Semaphore hanging"
  - obsolete : taken care by startup command:
    startEpicsIoc_includingHVSemaphoreCheck.sh ../../bin/linux-x86/hades st.cmd
2. all other error messages, notify experts,
  - system maybe working, but not completely
check whether server is running,
1. by checking for processes:
  ps x PID TTY STAT TIME COMMAND 13061 ? S 0:02 procServ -L ioc-cave-hadesdaq02.log 4813 startEpicsIoc_includingHVSemaphoreCheck.sh ../../bin/linux-x86/hades st.cmd 13062 pts/4 Ssl+ 1:51 startEpicsIoc_includingHVSemaphoreCheck.sh ../../bin/linux-x86/hades st.cmd
2. You can login to the server:
  1. telnet localhost 4813
  2. ... then hit "Enter", carriage return:
    <CR>
  3. You should see the epics prompt:
    epics>
  4. With the command
    dbl
    you will get a very long list of all process variables.
  5. To quit from the server without stopping it use the telnet escape (CTRL+]) sequence and quit:
    <CTRL+]> telnet> quit

Known server problems

~/apps/hades/iocBoot/ioccave/ioc-cave-hadesdaq02.log .

Semaphore hanging

Automatic procedure at startup

startEpicsIoc_includingHVSemaphoreCheck.sh ../../bin/linux-x86/hades st.cmd

Removing by hand

call script ~/apps/hades/iocBoot/ioccave/removeDeadHVSemaphores.sh
or really by hand

Semaphore already present
 There is another process using the semaphore.
 Or a process using the semaphore exited abnormally.
 In That case try to manually release the semaphore with:
   ipcrm sem XXX.

find the semaphore id

                 ipcs -s 
     ------ Semaphore Arrays --------
    key        semid     owner   perms   nsems      
    0x30222aea <Sem ID>  scs     666     1

Then delete this semaphore by

ipcrm sem <Sem ID>

SY1527 hanging

Ping the crate:

  ping hadhvp05
  PING hadhvp05.gsi.de (192.168.100.69) 56(84) bytes of data.
  64 bytes from hadhvp05.gsi.de (192.168.100.69): icmp_seq=1 ttl=64 time=0.908 ms
  64 bytes from hadhvp05.gsi.de (192.168.100.69): icmp_seq=2 ttl=64 time=0.873 ms 
  ^C

  telnet hadhvp05 1527
  user admin
  password admin
  About menu-->Sessions  (left most menu)

PeterZumbruch

Topic revision: r20 - 2018-11-27, PeterZumbruch

DaqSlowControl

Copyright © by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding Foswiki Send feedback | Imprint | Privacy Policy (in German)