Difference: SCSLinuxIoc (1 vs. 20)

Revision 20
27 Nov 2018 - Main.PeterZumbruch
Line: 1 to 1
Changed:
<
<
META TOPICPARENT name="SlowControl"
>
>
META TOPICPARENT name="DetectorControlSystem"
 

IOC for HADES SCS running on Linux

    Deleted:
    <
    <

    To restart the server, which is running

     
    Added:
    >
    >

    To restart the server, which is running

     
      1. login to hadesdaq02
      ssh scs@hadesdaq02
    Line: 131 to 166
      <CTRL+]>
    telnet> quit
    Changed:
    <
    <
    1. Further checks:
      1. Check the log ~/apps/hades/iocBoot/ioccave/ioc-cave-hadesdaq02.log
        1. In case "Semaphore already present" look chapter "Semaphore hanging"
        2. all other error messages, notify experts,
          • system maybe working, but not completely
      2. check whether server is running,
        1. by checking for processes:
          ps x
          	 PID TTY		 STAT	TIME COMMAND
          	 5615 ?			S		0:00 caRepeater
          	 13061 ?		  S		0:02 procServ -L ioc-cave-hadesdaq02.log 4813 startEpicsIoc_includingHVSemaphoreCheck.sh ../../bin/linux-x86/hades st.cmd
          	 13062 pts/4	 Ssl+	1:51 startEpicsIoc_includingHVSemaphoreCheck.sh ../../bin/linux-x86/hades st.cmd 
        2. You can login to the server:
          1. telnet localhost 4813
          2. ... then hit "Enter", carriage return:
            <CR>
          3. You should see the epics prompt:
            epics>
          4. With the command
            dbl
            you will get a very long list of all process variables.
        3. To quit from the server without stopping it use the telnet escape (CTRL+]) sequence and quit:
          <CTRL+]>
          telnet> quit
    >
    >
    1. optionally continue with further checks, q.v. below
     

    To start the server ((almost) from scratch)

    Line: 168 to 170
     

    To start the server ((almost) from scratch)

    Deleted:
    <
    <
     
      Added:
      >
      >
      Just follow the same procedures crontab would do during restart
        1. login to hadesdaq02
      ssh scs@hadesdaq02
      Changed:
      <
      <
      2. change to IOC's directory and start procServ
      >
      >
      1. clean up
        1. make sure procServ is not running by checking for processes:
          ps x
          		 PID	TTY		STAT	TIME COMMAND
          		 13061 ?		  S		0:02 procServ -L ioc-cave-hadesdaq02.log 4813 startEpicsIoc_includingHVSemaphoreCheck.sh ../../bin/linux-x86/hades st.cmd 
        2. if it is still there ...
          1. brute force: if there is still a procServ process kill it.
          2. else: try restart running server
      2. change to scs' procServ director
       
      Changed:
      <
      <
      cd ~/apps/hades/iocBoot/ioccave
      procServ -L ioc-cave-hadesdaq02.log 4813 startEpicsIoc_includingHVSemaphoreCheck.sh ../../bin/linux-x86/hades st.cmd
      >
      >
      cd ~scs/procServ &&
      ./ioc-cave.sh
       
      Added:
      >
      >
      1. optionally continue with further checks, q.v. below

      Further checks

      1. Check the log ~/apps/hades/iocBoot/ioccave/ioc-cave-hadesdaq02.log
        1. In case "Semaphore already present" look chapter "Semaphore hanging"
          • obsolete : taken care by startup command:
            startEpicsIoc_includingHVSemaphoreCheck.sh ../../bin/linux-x86/hades st.cmd
        2. all other error messages, notify experts,
          • system maybe working, but not completely
        3. check whether server is running,
      1. by checking for processes:
        ps x
        	 PID TTY		 STAT	TIME COMMAND
        
        
      Deleted:
      <
      <
      5615 ? S 0:00 caRepeater
        13061 ? S 0:02 procServ -L ioc-cave-hadesdaq02.log 4813 startEpicsIoc_includingHVSemaphoreCheck.sh ../../bin/linux-x86/hades st.cmd 13062 pts/4 Ssl+ 1:51 startEpicsIoc_includingHVSemaphoreCheck.sh ../../bin/linux-x86/hades st.cmd
      1. You can login to the server:
        1. telnet localhost 4813
      Changed:
      <
      <
        1. ... then hit
      >
      >
          1. ... then hit "Enter", carriage return:
       
      <CR>
      Line: 208 to 229
       
      Changed:
      <
      <

      Starting the Archiver (by Hand)

      >
      >

      Known server problems

       
        Changed:
        <
        <
        ssh scs@lxg0434
        cd ~/epics_apps/Archiver/config/
        engine.sh start [beamTime,default:beamOff]
        
        check whether it is running:
        ps x
        
        look for
        27898 pts/1    Sl     0:00 ArchiveEngine -log /scratch.local/scs/archiver/logs/<beamTime>/Index.log -description Hades Archive /u/scs/epics_apps/Archiver/config/hadesArchive.xml ...
        

        Note: the system produces a lock-file when active and removes it on exit. Incorrect program termination causes the lock-file to stay, that will prevent the data engine from starting next time. NEW: Taken care by engine.sh script

        crontab

        A crontab job calling ~epics_apps/Archiver/config/startEngine.crontab.sh is set up, which is called on reboot.
        • To change the beamtime, edit this file.

        Starting the data server

        >
        >
        If the server is not starting properly read the log file: ~/apps/hades/iocBoot/ioccave/ioc-cave-hadesdaq02.log .

        Semaphore hanging

         
          Changed:
          <
          <
          cd ~/apps/archiver/config/
          . dataserverstart.sh  
          
          check whether it is running:
          ps x
          
          look for
          ArchiveDataServerStandalone /u/scs/epics_apps/Archiver/StandAloneServer/abyss.conf /scratch.local/scs/archiver/jul10/Index
          
          >
          >
          Automatic procedure at startup
          Taken care by startup command:
          startEpicsIoc_includingHVSemaphoreCheck.sh ../../bin/linux-x86/hades st.cmd. But in case this did not work ...
           
          Deleted:
          <
          <

          Server problems

           
            Changed:
            <
            <
            If the server is not starting properly read the log file: ~/apps/hades/iocBoot/ioccave/ioc-cave-hadesdaq02.log .

            Semaphore hanging

            >
            >
            Removing by hand
            • call script ~/apps/hades/iocBoot/ioccave/removeDeadHVSemaphores.sh
            • or really by hand
             
              It could be that a semaphore is not cleaned up from a previous start, this is indicated by the message:
            Line: 266 to 253
             

            In order to cure this do the following:
            Changed:
            <
            <
            • find the semaphore id
            ipcs
            
            ------ Shared Memory Segments --------
            key        shmid      owner      perms      bytes      nattch     status      
            
            >
            >
            • find the semaphore id
              ipcs -s
              
              
             
            Semaphore Arrays -------- key semid owner perms nsems
            Changed:
            <
            <
            0x30222aea 1310720 scs 666 1


            Message Queues -------- key msqid owner perms used-bytes messages

            • Then delete the semaphore by

            ipcrm sem 1310720
            >
            >
            0x30222aea <Sem ID> scs 666 1
            • Then delete this semaphore by
              ipcrm sem <Sem ID>
             

            SY1527 hanging

            Line: 292 to 272
              you should first check if the crates is physically powered on and has Ethernet connection.
            Changed:
            <
            <
            Ping the crate
            >
            >
            • Ping the crate:
               
              ping hadhvp05
              PING hadhvp05.gsi.de (192.168.100.69) 56(84) bytes of data.
              Line: 300 to 281
                64 bytes from hadhvp05.gsi.de (192.168.100.69): icmp_seq=2 ttl=64 time=0.873 ms ^C
              Added:
              >
              >
               
              Changed:
              <
              <
              Login and check hanging CMD sections
              >
              >
              • Login and check hanging CMD sections
                 
                telnet hadhvp05 1527
                user admin
                Line: 307 to 290
                  user admin password admin About menu-->Sessions (left most menu)
                Deleted:
                <
                <
                 
                Added:
                >
                >
                screen shot
                If you see there a CMD TCP/IP connection you have to power cycle the crate.

                 

                Changed:
                <
                <
                screen shot
                >
                >

                Channel Archiver

                The channel archiver is usually running on lxg0434 visible to the GSI LAN
                 
                Changed:
                <
                <
                If you see there a CMD TCP/IP connection you have to power cycle the crate.
                >
                >

                CVS

                  CVS repository
                  :ext:scs@lx-pool.gsi.de:/misc/hadesprojects/slowcontrol/cvsroot
                  CVS module
                  EPICS/tools/Archiver

                Starting the Channel Archiver Engine on lxg0434

                Since the data is stored on a local disk of lxg0434 this service is bound to this machine, but can be easily transfered since besides those data directories all other is global to GSI

                  via crontab

                  A crontab job of user scs calling ~epics_apps/Archiver/config/startEngine.crontab.sh is set up, which is called on reboot.
                  • To change the beamtime, edit the content of that file.

                  by Hand

                    • login and start engine
                      ssh scs@lxg0434
                      cd ~/epics_apps/Archiver/config/
                      engine.sh start [beamTime,default:beamOff]
                    • check whether it is running:
                      ps x
                    • look for successful running job, e.g.
                      27898 pts/1 Sl 0:00 ArchiveEngine -log <Log Dir>/<Beam Time>/Index.log -description "Hades Archive" <Config Dir>/hadesArchive.xml -port <Port>
                 
                Added:
                >
                >
                Note : the system produces a lock-file when active and removes it on exit. Incorrect program termination causes the lock-file to stay, that will prevent the data engine from starting next time.
                NEW : Taken care by engine.sh script on startup, which checks for zombie files and removes them

                Starting the Channel Archiver Data Server on lxg0434

                  by Hand

                    • login and start engine
                      ssh scs@lxg0434
                      cd ~/epics_apps/Archiver/config/
                      . dataserverstart.sh
                    • check whether it is running:
                      ps x
                    • look for successful running job, e.g.
                      ArchiveDataServerStandalone /u/scs/epics_apps/Archiver/StandAloneServer/abyss.conf <Data Dir>/archiver/<Beam Time>/Index

                 
                Added:
                >
                >

                 
                Added:
                >
                >
                -- PeterZumbruch - 29 May 2012
                 
                META FILEATTACHMENT attr="h" comment="screen shot" date="1282906985" name="1527screen.png" path="1527screen.png" size="22690" user="BurkhardKolb" version="1.1"
                META TOPICMOVED by="SimonLang" date="1118326972" from="SimAna.SCSLinuxIoc" to="DaqSlowControl.SCSLinuxIoc"
                Revision 13
                25 May 2012 - Main.PeterZumbruch
                Line: 1 to 1
                 
                META TOPICPARENT name="SlowControl"

                IOC for HADES SCS running on Linux

                Line: 210 to 210
                 

                Changed:
                <
                <

                Starting the Archiver

                >
                >

                Starting the Archiver (by Hand)

                 
                  ssh scs@lxg0434
                  cd ~/epics_apps/Archiver/config/
                Changed:
                <
                <
                . enginestart.sh&
                >
                >
                engine.sh start [beamTime,default:beamOff]
                  check whether it is running:
                Line: 223 to 223
                  look for
                Changed:
                <
                <
                27898 pts/1 Sl 0:00 ArchiveEngine -log /scratch.local/scs/archiver/logs/jul10/Index.log -description Hades Archive /u/scs/epics_apps/Archiver/config/hadesArchive.xml ...
                >
                >
                27898 pts/1 Sl 0:00 ArchiveEngine -log /scratch.local/scs/archiver/logs//Index.log -description Hades Archive /u/scs/epics_apps/Archiver/config/hadesArchive.xml ...
                 
                Changed:
                <
                <
                Note: the system produces a lock-file when active and removes it on exit. Incorrect program termination causes the lock-file to stay, that will prevent the data engine from starting next time. To cure this:
                rm /scratch.local/scs/archiver/jul10/archive_active.lck
                
                >
                >
                Note: the system produces a lock-file when active and removes it on exit. Incorrect program termination causes the lock-file to stay, that will prevent the data engine from starting next time. NEW: Taken care by engine.sh script
                 

                Added:
                >
                >

                crontab

                A crontab job calling ~epics_apps/Archiver/config/startEngine.crontab.sh is set up, which is called on reboot.
                • To change the beamtime, edit this file.
                 

                Starting the data server

                  Revision 12
                  15 Sep 2010 - Main.PeterZumbruch
                  Line: 1 to 1
                   
                  META TOPICPARENT name="SlowControl"

                  IOC for HADES SCS running on Linux

                  Line: 288 to 288
                   

                  SY1527 hanging

                    If you see in the log file that one pf the HV crates does not connect
                  Changed:
                  <
                  <
                  you shoudl first check if the crates is physically powered on and has ethernet connection.
                  >
                  >
                  you should first check if the crates is physically powered on and has Ethernet connection.
                   

                  Ping the crate
                  Revision 11
                  14 Sep 2010 - Main.PeterZumbruch
                  Line: 1 to 1
                   
                  META TOPICPARENT name="SlowControl"

                  IOC for HADES SCS running on Linux

                  Line: 33 to 33
                   
                    Deleted:
                    <
                    <

                    To start the server (from scratch)

                      1. login to hadesdaq02
                      ssh scs@hadesdaq02
                      2. change to IOC's directory and start procServ
                      cd ~/apps/hades/iocBoot/ioccave
                      procServ -L ioc-cave-hadesdaq02.log 4813 ../../bin/linux-x86/hades st.cmd
                      3. check whether server is running,
                      1. by checking for processes:
                        ps x
                        	 PID TTY		 STAT	TIME COMMAND
                        	 5615 ?			S		0:00 caRepeater
                        	 13061 ?		  S		0:02 procServ -L ioc-cave-hadesdaq02.log 4813 ../../bin/linux-x86/hades st.cmd
                        	 13062 pts/4	 Ssl+	1:51 ../../bin/linux-x86/hades st.cmd 
                      2. You can login to the server:
                        1. telnet localhost 4813
                        2. ... then hit
                          <CR>
                        3. You should see the epics prompt:
                          epics>
                        4. With the command
                          dbl
                          you will get a very long list of all process variables.
                      3. To quit from the server without stopping it use the telnet escape (CTRL+]) sequence and quit:
                        ^]
                        telnet> quit
                     

                    To restart the server, which is running

                      1. login to hadesdaq02
                      ssh scs@hadesdaq02
                    Changed:
                    <
                    <
                    2. check whether (procServ) server is running,
                    >
                    >
                    2. check whether the procServ server, which itselfs controls the EPICS server, is running,
                      by checking for processes:
                    ps x
                    	 PID TTY		 STAT	TIME COMMAND
                    
                    
                    Changed:
                    <
                    <
                    5615 ? S 0:00 caRepeater 13061 ? S 0:02 procServ -L ioc-cave-hadesdaq02.log 4813 ../../bin/linux-x86/hades st.cmd 13062 pts/4 Ssl+ 1:51 ../../bin/linux-x86/hades st.cmd
                    1. If not, check for remaining semaphores (c.f. "Server Problems")
                    2. start server from scratch
                    3. You can login to the server:
                    >
                    >
                    13061 ? S 0:02 procServ -L ioc-cave-hadesdaq02.log 4813 startEpicsIoc_includingHVSemaphoreCheck.sh ../../bin/linux-x86/hades st.cmd ### this is the process server 13062 pts/4 Ssl+ 1:51 startEpicsIoc_includingHVSemaphoreCheck.sh ../../bin/linux-x86/hades st.cmd ### this is the EPICS server
                    1. NO:
                      1. check for remaining semaphores (c.f. chapter "Server Problems")
                      2. start server from scratch ( c.f. next chapter )
                    2. YES: continue
                    3. Now you can login to the server:
                     
                    1. telnet localhost 4813
                    Changed:
                    <
                    <
                    1. Switch on auto-restart (procServ), toggle it to be ON:
                    >
                    >
                    1. Switch off auto-restart (procServ), toggle it to be OFF:
                     
                    Changed:
                    <
                    <
                    CTRL+T
                    >
                    >
                    <CTRL+T>
                     
                    @@@ Toggled auto restart to ON/OFF
                    1. ... then hit
                    Line: 108 to 69
                     
                    epics>
                    Changed:
                    <
                    <
                      1. If not
                    >
                    >
                      1. NO you don't see it:
                     
                        1. quit telnet session
                    Changed:
                    <
                    <
                    ^]
                    >
                    >
                    <CTRL+]>
                      telnet> quit
                        1. check for remaining semaphores (c.f. "Server Problems")
                    Changed:
                    <
                    <
                        1. login to the server
                    >
                    >
                        1. again login to the server
                     
                    telnet localhost 4813
                    Changed:
                    <
                    <
                        1. restart the IOC by using:
                          CTRL+R
                      1. Otherwise if you have the prompt, i.e. EPICS is running, exit it.
                    >
                    >
                      1. YES, you have the prompt, i.e. EPICS is running, exit it.
                     
                    exit
                    Line: 136 to 93
                      DEBUG: shutting down crate 6 Shutdown: successfully disconnected from crate x7 <
                    Changed:
                    <
                    <
                      1. and restarting:
                    >
                    >
                      1. then restart the IOC by using:
                        <CTRL+R>
                      2. then you should see it restarting:
                     
                    	  @@@ @@@ @@@ @@@ @@@
                    	  @@@ Received a sigChild for process WXYZ. The process was killed by signal 11
                    
                    
                    Line: 167 to 128
                     
                    1. To quit from the server without stopping it use the telnet escape (CTRL+]) sequence and quit:
                    Changed:
                    <
                    <
                    ^]
                    >
                    >
                    <CTRL+]>
                      telnet> quit
                    Added:
                    >
                    >
                    1. Further checks:
                     
                    1. Check the log ~/apps/hades/iocBoot/ioccave/ioc-cave-hadesdaq02.log
                      1. In case "Semaphore already present" look chapter "Semaphore hanging"
                      2. all other error messages, notify experts,
                    Line: 180 to 142
                      ps x
                    	 PID TTY		 STAT	TIME COMMAND
                    	 5615 ?			S		0:00 caRepeater
                    
                    
                    Changed:
                    <
                    <
                    13061 ? S 0:02 procServ -L ioc-cave-hadesdaq02.log 4813 ../../bin/linux-x86/hades st.cmd 13062 pts/4 Ssl+ 1:51 ../../bin/linux-x86/hades st.cmd
                    >
                    >
                    13061 ? S 0:02 procServ -L ioc-cave-hadesdaq02.log 4813 startEpicsIoc_includingHVSemaphoreCheck.sh ../../bin/linux-x86/hades st.cmd 13062 pts/4 Ssl+ 1:51 startEpicsIoc_includingHVSemaphoreCheck.sh ../../bin/linux-x86/hades st.cmd
                        1. You can login to the server:
                          1. telnet localhost 4813
                          2. ... then hit "Enter", carriage return:
                            <CR>
                          3. You should see the epics prompt:
                            epics>
                          4. With the command
                            dbl
                            you will get a very long list of all process variables.
                        2. To quit from the server without stopping it use the telnet escape (CTRL+]) sequence and quit:
                          <CTRL+]>
                          telnet> quit

                    To start the server ((almost) from scratch)

                      1. login to hadesdaq02
                      ssh scs@hadesdaq02
                      2. change to IOC's directory and start procServ
                      cd ~/apps/hades/iocBoot/ioccave
                      procServ -L ioc-cave-hadesdaq02.log 4813 startEpicsIoc_includingHVSemaphoreCheck.sh ../../bin/linux-x86/hades st.cmd
                      3. check whether server is running,
                      1. by checking for processes:
                        ps x
                        	 PID TTY		 STAT	TIME COMMAND
                        	 5615 ?			S		0:00 caRepeater
                        	 13061 ?		  S		0:02 procServ -L ioc-cave-hadesdaq02.log 4813 startEpicsIoc_includingHVSemaphoreCheck.sh ../../bin/linux-x86/hades st.cmd
                        	 13062 pts/4	 Ssl+	1:51 startEpicsIoc_includingHVSemaphoreCheck.sh ../../bin/linux-x86/hades st.cmd 
                     
                      1. You can login to the server:
                        1. telnet localhost 4813
                    Line: 200 to 203
                      you will get a very long list of all process variables.
                      1. To quit from the server without stopping it use the telnet escape (CTRL+]) sequence and quit:
                    Changed:
                    <
                    <
                    ^]
                    >
                    >
                    <CTRL+]>
                      telnet> quit
                    Revision 10
                    13 Sep 2010 - Main.PeterZumbruch
                    Line: 1 to 1
                     
                    META TOPICPARENT name="SlowControl"

                    IOC for HADES SCS running on Linux

                    Line: 36 to 36
                      -->

                      To start the server (from scratch)

                    Deleted:
                    <
                    <

                      ssh scs@hadesdaq02

                      cd ~/apps/hades/iocBoot/ioccave
                      procServ -L ioc-cave-hadesdaq02.log 4813 ../../bin/linux-x86/hades st.cmd
                     
                    Changed:
                    <
                    <
                    check whether server is running: by checking for processes:

                    scs@hadesdaq02:~/apps/hades/iocBoot/ioccave> ps x
                    
                    
                    >
                    >
                      1. login to hadesdaq02
                      ssh scs@hadesdaq02
                      2. change to IOC's directory and start procServ
                      cd ~/apps/hades/iocBoot/ioccave
                      procServ -L ioc-cave-hadesdaq02.log 4813 ../../bin/linux-x86/hades st.cmd
                      3. check whether server is running,
                      1. by checking for processes:
                        ps x
                        
                        
                      PID TTY STAT TIME COMMAND 5615 ? S 0:00 caRepeater 13061 ? S 0:02 procServ -L ioc-cave-hadesdaq02.log 4813 ../../bin/linux-x86/hades st.cmd
                    Changed:
                    <
                    <
                    13062 pts/4 Ssl+ 1:51 ../../bin/linux-x86/hades st.cmd

                    You can login to the server:

                    telnet localhost 4813

                    hit
                    <CR>
                    

                    You should see the epics prompt

                    epics>

                    With the command
                    >
                    >
                    13062 pts/4 Ssl+ 1:51 ../../bin/linux-x86/hades st.cmd
                    1. You can login to the server:
                      1. telnet localhost 4813
                      2. ... then hit
                        <CR>
                      3. You should see the epics prompt:
                        epics>
                      4. With the command
                      dbl
                    Changed:
                    <
                    <

                    you will get a long list of all process variables.

                    To quit from the server use the telnet escape sequence:

                    ^]
                    quit
                    
                    >
                    >
                    you will get a very long list of all process variables.
                    1. To quit from the server without stopping it use the telnet escape (CTRL+]) sequence and quit:
                      ^]
                      telnet> quit
                     

                    To restart the server, which is running

                      Changed:
                      <
                      <
                      ssh scs@hadesdaq02

                      check whether server (procServ) is running:
                      >
                      >
                      1. login to hadesdaq02
                      ssh scs@hadesdaq02
                      2. check whether (procServ) server is running,
                        by checking for processes:
                      Changed:
                      <
                      <

                      scs@hadesdaq02:%IOC_DIR%> ps x
                      >
                      >
                      ps x
                      
                      
                        PID TTY STAT TIME COMMAND 5615 ? S 0:00 caRepeater
                      Changed:
                      <
                      <
                      13061 ? S 0:02 procServ -L ioc-cave-hadesdaq02.log 4813 ../../bin/linux-x86/hades st.cmd <== this is the telnet server hosting the EPICS IOC 13062 pts/4 Ssl+ 1:51 ../../bin/linux-x86/hades st.cmd <== this is the EPICS IOC

                      You can login to the server:

                      telnet localhost 4813

                      hit
                      <CR>
                      

                      You should see the epics prompt

                      epics>

                      To be sure that any semaphores of HV control are removed you have to do the following steps:

                      1. To switch on auto-restart:
                        • CTRL+T
                          • You will see: @@@ Toggled auto restart to ON/OFF, toggle it to be ON
                      2. exit
                      3. wait
                        1. you should see the IOC shutting down:
                          
                          
                      >
                      >
                      13061 ? S 0:02 procServ -L ioc-cave-hadesdaq02.log 4813 ../../bin/linux-x86/hades st.cmd 13062 pts/4 Ssl+ 1:51 ../../bin/linux-x86/hades st.cmd
                      1. If not, check for remaining semaphores (c.f. "Server Problems")
                      2. start server from scratch
                      3. You can login to the server:
                      1. telnet localhost 4813
                      2. Switch on auto-restart (procServ), toggle it to be ON:
                        CTRL+T
                        @@@ Toggled auto restart to ON/OFF
                      3. ... then hit
                        <CR>
                      4. You should see the epics prompt:
                        epics>
                        1. If not
                          1. quit telnet session
                            ^]
                            telnet> quit
                          2. check for remaining semaphores (c.f. "Server Problems")
                          3. login to the server
                            telnet localhost 4813
                          4. restart the IOC by using:
                            CTRL+R
                        2. Otherwise if you have the prompt, i.e. EPICS is running, exit it.
                          exit
                      5. wait
                        1. if the IOC has been running, you first should see the IOC shutting down:
                          
                          
                        DEBUG: shutting down crate 0 Shutdown: successfully disconnected from crate x1 [...] DEBUG: shutting down crate 6
                      Changed:
                      <
                      <
                      Shutdown: successfully disconnected from crate x7
                        1. and restarting:
                           
                          
                          
                      >
                      >
                      Shutdown: successfully disconnected from crate x7 <
                        1. and restarting:
                          
                          
                        @@@ @@@ @@@ @@@ @@@ @@@ Received a sigChild for process WXYZ. The process was killed by signal 11 @@@ Current time: Sun Sep 12 19:29:51 2010
                      Line: 152 to 164
                        iocRun: All initialization complete dbl > /home/scs/apps/hades/iocBoot/ioccave/hadesdaq02.dbl [...]
                      Changed:
                      <
                      <
                      1. To exit the telnet session:
                        1. CTRL+[
                        2. quit
                      2. Check the log =~/apps/hades/iocBoot/ioccave/ioc-cave-hadesdaq02.log
                      >
                      >
                      1. To quit from the server without stopping it use the telnet escape (CTRL+]) sequence and quit:
                        ^]
                        telnet> quit
                      2. Check the log ~/apps/hades/iocBoot/ioccave/ioc-cave-hadesdaq02.log
                       
                        1. In case "Semaphore already present" look chapter "Semaphore hanging"
                        2. all other error messages, notify experts,
                          • system maybe working, but not completely
                      Added:
                      >
                      >
                      1. check whether server is running,
                        1. by checking for processes:
                          ps x
                          	 PID TTY		 STAT	TIME COMMAND
                          	 5615 ?			S		0:00 caRepeater
                          	 13061 ?		  S		0:02 procServ -L ioc-cave-hadesdaq02.log 4813 ../../bin/linux-x86/hades st.cmd
                          	 13062 pts/4	 Ssl+	1:51 ../../bin/linux-x86/hades st.cmd 
                        2. You can login to the server:
                          1. telnet localhost 4813
                          2. ... then hit
                            <CR>
                          3. You should see the epics prompt:
                            epics>
                          4. With the command
                            dbl
                            you will get a very long list of all process variables.
                        3. To quit from the server without stopping it use the telnet escape (CTRL+]) sequence and quit:
                          ^]
                          telnet> quit
                       

                      Revision 9
                      13 Sep 2010 - Main.PeterZumbruch
                      Line: 1 to 1
                       
                      META TOPICPARENT name="SlowControl"

                      IOC for HADES SCS running on Linux

                      Added:
                      >
                      >
                      <--			   
                      • Style shortcuts - bold green code
                        • Set BOLDGREENCODE =
                        • Set BGC =
                        • Set ENDBGC =
                        • Set BGCSPAN =
                        • Set ENDBGCSPAN =
                        • Set BGCDIV =
                        • Set ENDBGCDIV =
                      • Style shortcuts - small caps (larger) bold
                        • Set SCB =
                        • Set SCE =
                        • Set SCLB =
                        • Set SCLE =
                      -->
                       

                      HADES IOC

                      There is an EPICS IOC running under Linux on machine hadesdaq02. Right now, it services the HV for all but MDC, the LV power supplies, temperature monitoring (old and new) and connects to the hadcon boards on the internal HADES VLAN.
                      Revision 8
                      13 Sep 2010 - Main.PeterZumbruch
                      Line: 1 to 1
                       
                      META TOPICPARENT name="SlowControl"

                      IOC for HADES SCS running on Linux

                      Line: 13 to 13
                        The IOC runs under procServ.

                      Starting the IOC

                      Changed:
                      <
                      <
                      To start the server:
                      >
                      >
                      <-- 
                      • Set IOC_DIR = ~/apps/hades/iocBoot/ioccave
                      • Set IOC_LOG_FILE = ioc-cave-hadesdaq02.log
                      -->

                        To start the server (from scratch)

                       
                      Added:
                      >
                      >
                          ssh scs@hadesdaq02
                        Changed:
                        <
                        <
                        cd ~/apps/hades/iocBoot/ioccave
                        procServ -L ioc-cave-hadesdaq02.log 4813 ../../bin/linux-x86/hades st.cmd
                        >
                        >
                        cd ~/apps/hades/iocBoot/ioccave
                        procServ -L ioc-cave-hadesdaq02.log 4813 ../../bin/linux-x86/hades st.cmd
                         

                        check whether server is running: by checking for processes:
                        Changed:
                        <
                        <
                        scs@hadesdaq02:~/apps/hades/iocBoot/ioccave> ps x
                        >
                        >
                        scs@hadesdaq02:~/apps/hades/iocBoot/ioccave> ps x
                        
                        
                          PID TTY STAT TIME COMMAND 5615 ? S 0:00 caRepeater
                        Changed:
                        <
                        <
                        13061 ? S 0:02 procServ -L ioc-cave-hadesdaq02.log 4813 ../../bin/linux-x86/hades st.cmd
                        >
                        >
                        13061 ? S 0:02 procServ -L ioc-cave-hadesdaq02.log 4813 ../../bin/linux-x86/hades st.cmd
                          13062 pts/4 Ssl+ 1:51 ../../bin/linux-x86/hades st.cmd
                        Changed:
                        <
                        <
                        >
                        >
                         

                        You can login to the server:
                        Line: 58 to 64
                          ^] quit
                        Added:
                        >
                        >

                        To restart the server, which is running

                          ssh scs@hadesdaq02

                          check whether server (procServ) is running: by checking for processes:

                          scs@hadesdaq02:%IOC_DIR%> ps x
                           PID TTY      STAT   TIME COMMAND
                           5615 ?        S      0:00 caRepeater
                          13061 ?        S      0:02 procServ -L %IOC_LOG_FILE% 4813 ../../bin/linux-x86/hades st.cmd  <== this is the telnet server hosting the EPICS IOC
                          13062 pts/4    Ssl+   1:51 ../../bin/linux-x86/hades st.cmd      <== this is the EPICS IOC
                          

                          You can login to the server:

                          telnet localhost 4813

                          hit
                          <CR>
                          
                         
                        Added:
                        >
                        >
                        You should see the epics prompt

                        epics>

                        To be sure that any semaphores of HV control are removed you have to do the following steps:

                        1. To switch on auto-restart:
                          • CTRL+T
                            • You will see: @@@ Toggled auto restart to ON/OFF, toggle it to be ON
                        2. exit
                        3. wait
                          1. you should see the IOC shutting down:
                            DEBUG: shutting down crate 0
                            Shutdown: successfully disconnected from crate x1
                            [...]
                            DEBUG: shutting down crate 6
                            Shutdown: successfully disconnected from crate x7 
                          2. and restarting:
                             
                            @@@ @@@ @@@ @@@ @@@
                            @@@ Received a sigChild for process WXYZ. The process was killed by signal 11
                            @@@ Current time: Sun Sep 12 19:29:51 2010
                            @@@ Child process is shutting down, auto restart is disabled
                            @@@ Use ^R to restart the child, ^Q to quit the server
                            @@@ Restarting child "../../bin/linux-x86/hades"
                            @@@ The PID of new child "../../bin/linux-x86/hades" is: abcdef
                            @@@ @@@ @@@ @@@ @@@
                            #!../../bin/linux-x86_64/hades
                            ## You may have to change hades to something else
                            ## everywhere it appears in this file
                            < envPaths
                            [...]
                            ## Load record instances
                            [...]
                            iocInit
                            Starting iocInit
                            ############################################################################
                            ## EPICS R3.14.10 $R3-14-10$ $2008/10/27 19:39:04$
                            ## EPICS Base built Oct 22 2009
                            ############################################################################
                            Starting CAEN x527 driver
                            pthread_attr_setstacksize error Invalid argument
                            iocRun: All initialization complete
                            dbl > /home/scs/apps/hades/iocBoot/ioccave/hadesdaq02.dbl
                            [...]
                        4. To exit the telnet session:
                          1. CTRL+[
                          2. quit
                        5. Check the log =~/apps/hades/iocBoot/ioccave/ioc-cave-hadesdaq02.log
                          1. In case "Semaphore already present" look chapter "Semaphore hanging"
                          2. all other error messages, notify experts,
                            • system maybe working, but not completely

                         

                        Starting the Archiver

                        Added:
                        >
                        >
                           
                          ssh scs@lxg0434
                          cd ~/epics_apps/Archiver/config/
                          Line: 80 to 167
                            rm /scratch.local/scs/archiver/jul10/archive_active.lck
                          Added:
                          >
                          >
                           

                          Starting the data server

                          Added:
                          >
                          >
                             
                            cd ~/apps/archiver/config/
                            . dataserverstart.sh  
                            Line: 94 to 183
                             
                            ArchiveDataServerStandalone /u/scs/epics_apps/Archiver/StandAloneServer/abyss.conf /scratch.local/scs/archiver/jul10/Index
                            
                            Added:
                            >
                            >

                             

                            Server problems

                            Changed:
                            <
                            <
                            If the server is not starting properly read the log file.
                            >
                            >
                              If the server is not starting properly read the log file: ~/apps/hades/iocBoot/ioccave/ioc-cave-hadesdaq02.log .
                             

                            Semaphore hanging

                            Added:
                            >
                            >
                                It could be that a semaphore is not cleaned up from a previous start, this is indicated by the message:
                              Line: 126 to 219
                               
                              • Then delete the semaphore by

                              ipcrm sem 1310720
                              Added:
                              >
                              >
                               

                              SY1527 hanging

                              Added:
                              >
                              >
                                  If you see in the log file that one pf the HV crates does not connect you shoudl first check if the crates is physically powered on and has ethernet connection.
                                Line: 154 to 249
                                 

                                If you see there a CMD TCP/IP connection you have to power cycle the crate.
                                Added:
                                >
                                >

                                 
                                META FILEATTACHMENT attr="h" comment="screen shot" date="1282906985" name="1527screen.png" path="1527screen.png" size="22690" user="BurkhardKolb" version="1.1"
                                META TOPICMOVED by="SimonLang" date="1118326972" from="SimAna.SCSLinuxIoc" to="DaqSlowControl.SCSLinuxIoc"
                                Revision 7
                                27 Aug 2010 - Main.BurkhardKolb
                                Line: 1 to 1
                                 
                                META TOPICPARENT name="SlowControl"
                                Changed:
                                <
                                <

                                IOC for HADES SCS running on Linux

                                >
                                >

                                IOC for HADES SCS running on Linux

                                 
                                Added:
                                >
                                >

                                HADES IOC

                                  There is an EPICS IOC running under Linux on machine hadesdaq02. Right now, it services the HV for all but MDC, the LV power supplies, temperature monitoring (old and new) and connects to the hadcon boards on the internal HADES VLAN.

                                In order to do maintenance, connect to hadesdaq02 as user scs using the default scs password. The configuration is stored under the directory /home/scs/apps/hades. There you will find a normal IOC directory tree for development and booting the IOC. The IOC is named "hades".
                                Line: 10 to 12
                                 

                                The IOC runs under procServ.
                                Added:
                                >
                                >

                                Starting the IOC

                                  To start the server:

                                ssh scs@hadesdaq02
                                Line: 57 to 60
                                 
                                Changed:
                                <
                                <
                                to start the archive engine:
                                >
                                >

                                Starting the Archiver

                                 
                                ssh scs@lxg0434
                                cd ~/epics_apps/Archiver/config/
                                Line: 77 to 80
                                  rm /scratch.local/scs/archiver/jul10/archive_active.lck
                                Changed:
                                <
                                <
                                to start the data server:
                                >
                                >

                                Starting the data server

                                 
                                cd ~/apps/archiver/config/
                                . dataserverstart.sh  
                                Line: 92 to 96
                                 

                                Server problems

                                If the server is not starting properly read the log file.
                                Added:
                                >
                                >

                                Semaphore hanging

                                  It could be that a semaphore is not cleaned up from a previous start, this is indicated by the message:
                                Line: 122 to 127
                                 

                                ipcrm sem 1310720
                                Added:
                                >
                                >

                                SY1527 hanging

                                If you see in the log file that one pf the HV crates does not connect you shoudl first check if the crates is physically powered on and has ethernet connection.

                                Ping the crate
                                ping hadhvp05
                                PING hadhvp05.gsi.de (192.168.100.69) 56(84) bytes of data.
                                64 bytes from hadhvp05.gsi.de (192.168.100.69): icmp_seq=1 ttl=64 time=0.908 ms
                                64 bytes from hadhvp05.gsi.de (192.168.100.69): icmp_seq=2 ttl=64 time=0.873 ms 
                                ^C
                                

                                Login and check hanging CMD sections
                                telnet hadhvp05 1527
                                user admin
                                password admin
                                About menu-->Sessions  (left most menu)
                                
                                

                                screen shot

                                If you see there a CMD TCP/IP connection you have to power cycle the crate.
                                 
                                Added:
                                >
                                >
                                META FILEATTACHMENT attr="h" comment="screen shot" date="1282906985" name="1527screen.png" path="1527screen.png" size="22690" user="BurkhardKolb" version="1.1"
                                 
                                META TOPICMOVED by="SimonLang" date="1118326972" from="SimAna.SCSLinuxIoc" to="DaqSlowControl.SCSLinuxIoc"
                                Revision 6
                                20 Aug 2010 - Main.BurkhardKolb
                                Line: 1 to 1
                                 
                                META TOPICPARENT name="SlowControl"

                                IOC for HADES SCS running on Linux

                                Changed:
                                <
                                <
                                There is an EPICS IOC running under Linux on machine hadeb01. Right now, it does the temperature monitoring for MDC (the former e7pif), but additional tasks may be added to this IOC.
                                >
                                >
                                There is an EPICS IOC running under Linux on machine hadesdaq02. Right now, it services the HV for all but MDC, the LV power supplies, temperature monitoring (old and new) and connects to the hadcon boards on the internal HADES VLAN.
                                 
                                Changed:
                                <
                                <
                                In order to do maintenance, connect to hadeb01 as user scs using the default scs password. The configuration is stored under the directory $HOME/epics/scs_apps. There you will find a normal IOC directory tree for development and booting the IOC. The IOC is named "slow".
                                >
                                >
                                In order to do maintenance, connect to hadesdaq02 as user scs using the default scs password. The configuration is stored under the directory /home/scs/apps/hades. There you will find a normal IOC directory tree for development and booting the IOC. The IOC is named "hades".
                                 
                                Changed:
                                <
                                <
                                To change the database files, make your edits in slowApp/Db and afterwards type make.
                                >
                                >
                                To change the database files, make your edits in hadesApp/Db and afterwards type make. Don't forget to commit the changes to CVS.
                                 
                                Changed:
                                <
                                <
                                Since the IOC uses the terminal for its console and its impractical to leave the connection to hadeb01 open all the time, the IOC is started under the control of the screen utility (as proposed in EPICS tech-talk). To check, if the IOC session is still running, type screen -ls. If the session is still there, the output will show one "Detached" session. You may connect to the IOC command line with the screen -r command. Then use the usual iocsh commands. To disconnect from the IOC without stopping it, type Ctrl-A Ctrl-D. To stop the IOC, type Ctrl-D (or exit). Stopping the IOC will also close the screen session.
                                >
                                >
                                The IOC runs under procServ.
                                 
                                Changed:
                                <
                                <
                                To start the IOC, change to iocBoot/iocslow, then open a new screen session with a new IOC by typing screen st.cmd.
                                >
                                >
                                To start the server:

                                ssh scs@hadesdaq02

                                cd ~/apps/hades/iocBoot/ioccave
                                procServ -L ioc-cave-hadesdaq02.log 4813 ../../bin/linux-x86/hades st.cmd

                                check whether server is running: by checking for processes:

                                scs@hadesdaq02:~/apps/hades/iocBoot/ioccave> ps x
                                 PID TTY      STAT   TIME COMMAND
                                 5615 ?        S      0:00 caRepeater
                                13061 ?        S      0:02 procServ -L ioc-cave-hadesdaq02.log 4813 ../../bin/linux-x86/hades st.cmd
                                13062 pts/4    Ssl+   1:51 ../../bin/linux-x86/hades st.cmd
                                

                                You can login to the server:

                                telnet localhost 4813

                                hit
                                <CR>
                                

                                You should see the epics prompt

                                epics>

                                With the command

                                dbl

                                you will get a long list of all process variables.

                                To quit from the server use the telnet escape sequence:

                                ^]
                                quit
                                

                                to start the archive engine:
                                ssh scs@lxg0434
                                cd ~/epics_apps/Archiver/config/
                                . enginestart.sh&
                                
                                check whether it is running:
                                ps x
                                
                                look for
                                27898 pts/1    Sl     0:00 ArchiveEngine -log /scratch.local/scs/archiver/logs/jul10/Index.log -description Hades Archive /u/scs/epics_apps/Archiver/config/hadesArchive.xml ...
                                

                                Note: the system produces a lock-file when active and removes it on exit. Incorrect program termination causes the lock-file to stay, that will prevent the data engine from starting next time. To cure this:
                                rm /scratch.local/scs/archiver/jul10/archive_active.lck
                                

                                to start the data server:
                                cd ~/apps/archiver/config/
                                . dataserverstart.sh  
                                
                                check whether it is running:
                                ps x
                                
                                look for
                                ArchiveDataServerStandalone /u/scs/epics_apps/Archiver/StandAloneServer/abyss.conf /scratch.local/scs/archiver/jul10/Index
                                

                                Server problems

                                If the server is not starting properly read the log file. It could be that a semaphore is not cleaned up from a previous start, this is indicated by the message:
                                Semaphore already present
                                 There is another process using the semaphore.
                                 Or a process using the semaphore exited abnormally.
                                 In That case try to manually release the semaphore with:
                                   ipcrm sem XXX.
                                

                                In order to cure this do the following:
                                • find the semaphore id
                                ipcs
                                
                                ------ Shared Memory Segments --------
                                key        shmid      owner      perms      bytes      nattch     status      
                                
                                ------ Semaphore Arrays --------
                                key        semid      owner      perms      nsems     
                                0x30222aea 1310720    scs        666        1         
                                
                                ------ Message Queues --------
                                key        msqid      owner      perms      used-bytes   messages 
                                

                                • Then delete the semaphore by

                                ipcrm sem 1310720
                                 
                                Deleted:
                                <
                                <
                                -- MathiasMuench - 21 Apr 2005
                                 

                                META TOPICMOVED by="SimonLang" date="1118326972" from="SimAna.SCSLinuxIoc" to="DaqSlowControl.SCSLinuxIoc"
                                Revision 5
                                25 Aug 2005 - Main.SimonLang
                                Line: 1 to 1
                                Added:
                                >
                                >
                                META TOPICPARENT name="SlowControl"
                                 

                                IOC for HADES SCS running on Linux

                                There is an EPICS IOC running under Linux on machine hadeb01. Right now, it does the temperature monitoring for MDC (the former e7pif), but additional tasks may be added to this IOC.
                                Revision 4
                                09 Jun 2005 - Main.SimonLang
                                Line: 1 to 1
                                 

                                IOC for HADES SCS running on Linux

                                There is an EPICS IOC running under Linux on machine hadeb01. Right now, it does the temperature monitoring for MDC (the former e7pif), but additional tasks may be added to this IOC.
                                Line: 12 to 12
                                  To start the IOC, change to iocBoot/iocslow, then open a new screen session with a new IOC by typing screen st.cmd.

                                -- MathiasMuench - 21 Apr 2005
                                Added:
                                >
                                >

                                META TOPICMOVED by="SimonLang" date="1118326972" from="SimAna.SCSLinuxIoc" to="DaqSlowControl.SCSLinuxIoc"
                                Revision 3
                                25 Apr 2005 - Main.MathiasMuench
                                Line: 1 to 1
                                 

                                IOC for HADES SCS running on Linux

                                There is an EPICS IOC running under Linux on machine hadeb01. Right now, it does the temperature monitoring for MDC (the former e7pif), but additional tasks may be added to this IOC.
                                Changed:
                                <
                                <
                                In order to do maintenance, connect to hades01 as user scs using the default scs password. The configuration is stored under the directory $HOME/epics/scs_apps. There you will find a normal IOC directory tree for development and booting the IOC. The IOC is named "slow".
                                >
                                >
                                In order to do maintenance, connect to hadeb01 as user scs using the default scs password. The configuration is stored under the directory $HOME/epics/scs_apps. There you will find a normal IOC directory tree for development and booting the IOC. The IOC is named "slow".
                                 

                                To change the database files, make your edits in slowApp/Db and afterwards type make.
                                Revision 2
                                25 Apr 2005 - Main.MathiasMuench
                                Line: 1 to 1
                                 

                                IOC for HADES SCS running on Linux

                                Added:
                                >
                                >
                                There is an EPICS IOC running under Linux on machine hadeb01. Right now, it does the temperature monitoring for MDC (the former e7pif), but additional tasks may be added to this IOC.
                                 
                                Added:
                                >
                                >
                                In order to do maintenance, connect to hades01 as user scs using the default scs password. The configuration is stored under the directory $HOME/epics/scs_apps. There you will find a normal IOC directory tree for development and booting the IOC. The IOC is named "slow".

                                To change the database files, make your edits in slowApp/Db and afterwards type make.

                                Since the IOC uses the terminal for its console and its impractical to leave the connection to hadeb01 open all the time, the IOC is started under the control of the screen utility (as proposed in EPICS tech-talk). To check, if the IOC session is still running, type screen -ls. If the session is still there, the output will show one "Detached" session. You may connect to the IOC command line with the screen -r command. Then use the usual iocsh commands. To disconnect from the IOC without stopping it, type Ctrl-A Ctrl-D. To stop the IOC, type Ctrl-D (or exit). Stopping the IOC will also close the screen session.

                                To start the IOC, change to iocBoot/iocslow, then open a new screen session with a new IOC by typing screen st.cmd.
                                 

                                -- MathiasMuench - 21 Apr 2005
                                 
                                This site is powered by FoswikiCopyright © by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
                                Ideas, requests, problems regarding Hades Wiki? Send feedback
                                Imprint (in German)
                                Privacy Policy (in German)