Difference: EventBuilderSoftware (1 vs. 15)

Revision 15
04 Feb 2009 - Main.SergeyYurevich
Line: 1 to 1
 
META TOPICPARENT name="SoftwareDevelopment"
Deleted:
<
<
Table of contents:

The limitation of maximum 32 subsystems.

The limitation of maximum 32 subsystems was due to the length of the bit mask of a type unsigned long (nettrans.c:
NetTrans_multiRecv
function). New function
NetTrans_multiReceive
does not use bit mask and should be able to handle any number of subsystems.

Tests of EB with many subsystems.

Tested with 60 subsystems:
  • Total event rate = 2 kHz
  • Total data rate = 10 MB/sec
  • All 60 subsystems (soft_rdos) run on hadc07. Distribution of subsystems over different CPUs still run out of synch.
  • No discarded events during 5 hours of running

Tests with 90 subsystems:

Cannot go to 100 subsystems because of memory limitations (in case of 8x2x2 MB buffers).

Conditions:
  • I used TOF0, TOF1 and TOF4 VME CPUs. 30 daq_rdosoft processes per CPU.
  • Water mark was set to 32 kB.
  • No writing to hard disk, no rfio.
  • net.core.netdev_max_backlog = 10000
  • net.core.rmem_max = 4194304 (nettrans.c:openUdp(): requests 2 MB queue)
  • EB buffer = 8 MB (x2x2). Cannot go to 16 MB because 16x2x2x90 = 5.6 GB (more than available memory).

Data rate Event rate Subevent size EB CPU load comment
90x230kB/sec = 20 MB/sec 2 kHz 116 Bytes 0.25 no discarded evts
90x460kB/sec = 40 MB/sec 4 kHz 116 Bytes 0.35 no discarded evts
90x570kB/sec = 50 MB/sec 5 kHz 116 Bytes 0.6 (tends to slowly increase) no discarded evts
60 MB/sec 6 kHz 116 Bytes >1 (tends to increase) some disc. evts

After increasing subevent size from 116 to 252 Bytes I see many discarded events:

Data rate Event rate Subevent size EB CPU load comment
90x246kB/sec = 22 MB/sec 1 kHz 252 Bytes 0.3 20% disc. evts
90x493kB/sec = 44 MB/sec 2 kHz 252 Bytes 0.4 20% disc. evts

The problem of discarded events with larger subevent sizes also happens even only with 40 subsystems. Increasing EB buffer size from 8 to 16 MB for 40 subsystems did not help. Running Event Builder on hadeb06a ended up with the same problem of discarded events. Running daq_netmem under "root" did not help.

The following tests with 30 subsystems give a hint on the roots of the problem.

Data rate Evt rate Subevt size EB buffer size UDP queue size CPU usage Load average comment
30x643 kB/s = 18 MB/s 1 kHz 658 Bytes 32 MB 4 MB 5% 0.25 no discarded evts
30x643 kB/s = 18 MB/s 1 kHz 658 Bytes 16 MB 4 MB 5% 0.25 discarded evts/packets
30x799 kB/s = 22 MB/s 4 kHz 202 Bytes 16 MB 4 MB 5% 0.25 no discarded evts

Conclusions:
  • With 2x32 MB EB double buffer (1-st line in table) - no discarded events.
  • With 2x16 MB EB double buffer (2-nd line in table) - discarded evts/packets.
  • If I just decrease the subevent size and keep the total data rate and EB buffer size, I get no discarded events (3-rd line)! I also noticed that sometimes EB is blocked at a given buffer for few seconds in a case of the larger subevent sizes! Thus, I believe that the problem is most likely with EB algorithm which has a feature to get blocked. I will investigate this further.
  • 5% CPU usage for daq_evtbuild and for daq_netmem (and low average load) mean that it is only the jitter (the fluctuations of the data coming into the buffers) that is responsible for the discarded events.

Expected subevent sizes for different subsystems:

Subsystem Number of TRBs subevent size
RICH 6 960 Bytes
12 480 Bytes
MDC 12 1 kByte
Shower 6 540 Bytes
RPC 24 70 Bytes

Conditions:
  • I used TOF0, TOF1 and TOF4 VME CPUs. 30 daq_rdosoft processes per CPU.
  • Water mark was set to 32 kB.
  • No writing to hard disk, no rfio.
  • net.core.netdev_max_backlog = 10000
  • net.core.rmem_max = 4194304 (nettrans.c:openUdp(): requests 1 MB queue)
  • 0-59 queues: EB buffer = 8 MB (x2x2).
  • 60-89 queues: EB buffer = 16 MB (x2x2).
  • 0-59 sources: subevt size = 168 Bytes (smeared Gaussian-like)
  • 60-89 sources: subevt size = 1.3 kB (smeared Gaussian-like)

Here is a typical fill level of buffers:

------------------ netmem buffer fill levels ----------------------------------------
9                                                                                           
8                                                                                           
7                                                                                           
6                                                                                           
5                                                             ||||||||||||||                
4                                                             ||||||||||||||||||||||||||||||
3                                                             ||||||||||||||||||||||||||||||
2                                                             ||||||||||||||||||||||||||||||
1                                                             ||||||||||||||||||||||||||||||
0                                                  ||| ||||   ||||||||||||||||||||||||||||||
q:012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789
  0         1         2         3         4         5         6         7         8         

------------------ evtbuild buffer fill levels ----------------------------------------
9                                                                                           
8                                                                                           
7                                                                                           
6                                                             ||||||||||||||||||||||||||||||
5                                                             ||||||||||||||||||||||||||||||
4                                                             ||||||||||||||||||||||||||||||
3                                                             ||||||||||||||||||||||||||||||
2                                                             ||||||||||||||||||||||||||||||
1                                                             ||||||||||||||||||||||||||||||
0 |||||||||| ||||| |||||||||||   ||||||||||||| | ||||| |||||||||||||||||||||||||||||||||||||
q:012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789
  0         1         2         3         4         5         6         7         8         
------------------------------------------------------------------------------
evtComp:  602  evtComp/s:  358  bytesWrit:   26M bytesWrit/s:   15M
evtDisc:    0  evtDatErr:    0  evtTagErr:    0 

The buffers 60-89 have 60% fill level due to about 10 times bigger subevt size. The buffers 0-59 are about 10% filled. If I decrease size of 0-59 buffers down to 4x2x2 MB and increase size of 60-89 buffers up to 32x2x2 MB then the fill level becomes almost equal to 20% for all buffers.

Including RFIO to the EB.

In the future DAQ the data will be sent to the gStore via RFIO. The test programs prepared by Horst Goeringer are here:
  • /GSI/staging/adsm/v51/rfioclient/
  • mrawWriteLoopSampleFulln.c - this program sends local file in loop to gStore using RFIO
  • /GSI/staging/adsm/v51/rfioclient/Linux/ - corresponding make files
  • /GSI/staging/adsm/v51/Linux/ - libraries. We need two libraries: librawapiclin.a and librawservn.a
  • /GSI/staging/adsm/v51/inc/ - headers
  • our test setup: rfiodaq:gstore:/hadaqtest/path/file, where hadaqtest is an archive name for the tests

I have implemented an RFIO client in the Event Builder. Now we have a direct data transfer from EB to Data Mover. RFIO clinet code consists of five functions added to evtbuild.c:
  • rfio_openConnection
  • rfio_openFile
  • rfio_writeFile
  • rfio_closeFile
  • rfio_closeConnection

Tests with 30 subsystems showed good performance (no discarded events) of EB with RFIO client under the following conditions:
  • Total event rate = 5 kHz
  • Total data rate = 16 MB/sec
  • EB buffer size = 16 MB (*2*2)

Was also tested with 90 subsystems. No problems were observed.

The sizes of the files transfered to Data Mover look reasonable.

Open questions:
  • Local and remote file names should be identical.
  • runNr and seqNr should be identical between local and remote events.
  • Collect some real data and check with go4.

-- SergeyYurevich - 17 Dec 2007
 

META TOPICMOVED by="MichaelTraxler" date="1197980165" from="DaqSlowControl.EventBuilberSoftware" to="DaqSlowControl.EventBuilderSoftware"
Revision 14
01 Feb 2008 - Main.SergeyYurevich
Line: 1 to 1
 
META TOPICPARENT name="SoftwareDevelopment"
Table of contents:
Line: 66 to 66
 
RPC 24 70 Bytes
Added:
>
>
Conditions:
  • I used TOF0, TOF1 and TOF4 VME CPUs. 30 daq_rdosoft processes per CPU.
  • Water mark was set to 32 kB.
  • No writing to hard disk, no rfio.
  • net.core.netdev_max_backlog = 10000
  • net.core.rmem_max = 4194304 (nettrans.c:openUdp(): requests 1 MB queue)
  • 0-59 queues: EB buffer = 8 MB (x2x2).
  • 60-89 queues: EB buffer = 16 MB (x2x2).
  • 0-59 sources: subevt size = 168 Bytes (smeared Gaussian-like)
  • 60-89 sources: subevt size = 1.3 kB (smeared Gaussian-like)

Here is a typical fill level of buffers:

------------------ netmem buffer fill levels ----------------------------------------
9                                                                                           
8                                                                                           
7                                                                                           
6                                                                                           
5                                                             ||||||||||||||                
4                                                             ||||||||||||||||||||||||||||||
3                                                             ||||||||||||||||||||||||||||||
2                                                             ||||||||||||||||||||||||||||||
1                                                             ||||||||||||||||||||||||||||||
0                                                  ||| ||||   ||||||||||||||||||||||||||||||
q:012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789
  0         1         2         3         4         5         6         7         8         

------------------ evtbuild buffer fill levels ----------------------------------------
9                                                                                           
8                                                                                           
7                                                                                           
6                                                             ||||||||||||||||||||||||||||||
5                                                             ||||||||||||||||||||||||||||||
4                                                             ||||||||||||||||||||||||||||||
3                                                             ||||||||||||||||||||||||||||||
2                                                             ||||||||||||||||||||||||||||||
1                                                             ||||||||||||||||||||||||||||||
0 |||||||||| ||||| |||||||||||   ||||||||||||| | ||||| |||||||||||||||||||||||||||||||||||||
q:012345678901234567890123456789012345678901234567890123456789012345678901234567890123456789
  0         1         2         3         4         5         6         7         8         
------------------------------------------------------------------------------
evtComp:  602  evtComp/s:  358  bytesWrit:   26M bytesWrit/s:   15M
evtDisc:    0  evtDatErr:    0  evtTagErr:    0 

The buffers 60-89 have 60% fill level due to about 10 times bigger subevt size. The buffers 0-59 are about 10% filled. If I decrease size of 0-59 buffers down to 4x2x2 MB and increase size of 60-89 buffers up to 32x2x2 MB then the fill level becomes almost equal to 20% for all buffers.
 

Including RFIO to the EB.

In the future DAQ the data will be sent to the gStore via RFIO.
Revision 13
14 Jan 2008 - Main.SergeyYurevich
Line: 1 to 1
 
META TOPICPARENT name="SoftwareDevelopment"
Table of contents:
Line: 45 to 45
 

The following tests with 30 subsystems give a hint on the roots of the problem.
Changed:
<
<
Data rate Evt rate Subevt size EB buffer size UDP queue size comment
30x643 kB/s = 18 MB/s 1 kHz 658 Bytes 32 MB 4 MB no discarded evts
30x643 kB/s = 18 MB/s 1 kHz 658 Bytes 16 MB 4 MB discarded evts/packets
30x799 kB/s = 22 MB/s 4 kHz 202 Bytes 16 MB 4 MB no discarded evts
>
>
Data rate Evt rate Subevt size EB buffer size UDP queue size CPU usage Load average comment
30x643 kB/s = 18 MB/s 1 kHz 658 Bytes 32 MB 4 MB 5% 0.25 no discarded evts
30x643 kB/s = 18 MB/s 1 kHz 658 Bytes 16 MB 4 MB 5% 0.25 discarded evts/packets
30x799 kB/s = 22 MB/s 4 kHz 202 Bytes 16 MB 4 MB 5% 0.25 no discarded evts
 

Conclusions:
  • With 2x32 MB EB double buffer (1-st line in table) - no discarded events.
  • With 2x16 MB EB double buffer (2-nd line in table) - discarded evts/packets.
  • If I just decrease the subevent size and keep the total data rate and EB buffer size, I get no discarded events (3-rd line)! I also noticed that sometimes EB is blocked at a given buffer for few seconds in a case of the larger subevent sizes! Thus, I believe that the problem is most likely with EB algorithm which has a feature to get blocked. I will investigate this further.
Changed:
<
<

>
>
  • 5% CPU usage for daq_evtbuild and for daq_netmem (and low average load) mean that it is only the jitter (the fluctuations of the data coming into the buffers) that is responsible for the discarded events.
 

Expected subevent sizes for different subsystems:
Revision 12
18 Dec 2007 - Main.MichaelTraxler
Line: 1 to 1
 
META TOPICPARENT name="SoftwareDevelopment"
Table of contents:
Line: 102 to 102
 

-- SergeyYurevich - 17 Dec 2007
Added:
>
>
META TOPICMOVED by="MichaelTraxler" date="1197980165" from="DaqSlowControl.EventBuilberSoftware" to="DaqSlowControl.EventBuilderSoftware"
Revision 11
17 Dec 2007 - Main.SergeyYurevich
Line: 1 to 1
 
META TOPICPARENT name="SoftwareDevelopment"
Table of contents:
Line: 43 to 43
 

The problem of discarded events with larger subevent sizes also happens even only with 40 subsystems. Increasing EB buffer size from 8 to 16 MB for 40 subsystems did not help. Running Event Builder on hadeb06a ended up with the same problem of discarded events. Running daq_netmem under "root" did not help.
Added:
>
>
The following tests with 30 subsystems give a hint on the roots of the problem.

Data rate Evt rate Subevt size EB buffer size UDP queue size comment
30x643 kB/s = 18 MB/s 1 kHz 658 Bytes 32 MB 4 MB no discarded evts
30x643 kB/s = 18 MB/s 1 kHz 658 Bytes 16 MB 4 MB discarded evts/packets
30x799 kB/s = 22 MB/s 4 kHz 202 Bytes 16 MB 4 MB no discarded evts
  Conclusions:
Changed:
<
<
  • Hard to say. Might be that daq_netmem is too slow to extract UDP packets from 2 MB queue??? Requires more investigation.
>
>
  • With 2x32 MB EB double buffer (1-st line in table) - no discarded events.
  • With 2x16 MB EB double buffer (2-nd line in table) - discarded evts/packets.
  • If I just decrease the subevent size and keep the total data rate and EB buffer size, I get no discarded events (3-rd line)! I also noticed that sometimes EB is blocked at a given buffer for few seconds in a case of the larger subevent sizes! Thus, I believe that the problem is most likely with EB algorithm which has a feature to get blocked. I will investigate this further.
 

Expected subevent sizes for different subsystems:
Line: 91 to 100
 
  • Collect some real data and check with go4.
Changed:
<
<
-- SergeyYurevich - 07 Dec 2007
>
>
-- SergeyYurevich - 17 Dec 2007
 
Revision 10
10 Dec 2007 - Main.SergeyYurevich
Line: 1 to 1
 
META TOPICPARENT name="SoftwareDevelopment"
Table of contents:
Line: 13 to 13
 

Tested with 60 subsystems:
  • Total event rate = 2 kHz
Changed:
<
<
  • Total data rate = 10 MHz
>
>
  • Total data rate = 10 MB/sec
 
  • All 60 subsystems (soft_rdos) run on hadc07. Distribution of subsystems over different CPUs still run out of synch.
  • No discarded events during 5 hours of running
Line: 78 to 78
 

Tests with 30 subsystems showed good performance (no discarded events) of EB with RFIO client under the following conditions:
  • Total event rate = 5 kHz
Changed:
<
<
  • Total data rate = 16 MHz
>
>
  • Total data rate = 16 MB/sec
 
  • EB buffer size = 16 MB (*2*2)

Was also tested with 90 subsystems. No problems were observed.
Revision 9
07 Dec 2007 - Main.SergeyYurevich
Line: 1 to 1
 
META TOPICPARENT name="SoftwareDevelopment"
Table of contents:
Line: 9 to 9
  The limitation of maximum 32 subsystems was due to the length of the bit mask of a type unsigned long (nettrans.c:
NetTrans_multiRecv
function). New function
NetTrans_multiReceive
does not use bit mask and should be able to handle any number of subsystems.
Added:
>
>

Tests of EB with many subsystems.

  Tested with 60 subsystems:
  • Total event rate = 2 kHz
  • Total data rate = 10 MHz
Line: 45 to 47
 
  • Hard to say. Might be that daq_netmem is too slow to extract UDP packets from 2 MB queue??? Requires more investigation.
Added:
>
>
Expected subevent sizes for different subsystems:

Subsystem Number of TRBs subevent size
RICH 6 960 Bytes
12 480 Bytes
MDC 12 1 kByte
Shower 6 540 Bytes
RPC 24 70 Bytes
 

Including RFIO to the EB.

Revision 8
07 Dec 2007 - Main.SergeyYurevich
Line: 1 to 1
 
META TOPICPARENT name="SoftwareDevelopment"
Table of contents:
Line: 17 to 17
 

Tests with 90 subsystems:
Added:
>
>
Cannot go to 100 subsystems because of memory limitations (in case of 8x2x2 MB buffers).
  Conditions:
  • I used TOF0, TOF1 and TOF4 VME CPUs. 30 daq_rdosoft processes per CPU.
  • Water mark was set to 32 kB.
Line: 37 to 39
 
90x246kB/sec = 22 MB/sec 1 kHz 252 Bytes 0.3 20% disc. evts
90x493kB/sec = 44 MB/sec 2 kHz 252 Bytes 0.4 20% disc. evts

Changed:
<
<
The problem of discarded events with larger subevent sizes also happens even only with 40 subsystems. Increasing EB buffer size from 8 to 16 MB for 40 subsystems did not help. Running Event Builder on hadeb06a ended up with the same problem of discarded events.
>
>
The problem of discarded events with larger subevent sizes also happens even only with 40 subsystems. Increasing EB buffer size from 8 to 16 MB for 40 subsystems did not help. Running Event Builder on hadeb06a ended up with the same problem of discarded events. Running daq_netmem under "root" did not help.
 

Conclusions:
  • Hard to say. Might be that daq_netmem is too slow to extract UDP packets from 2 MB queue??? Requires more investigation.
Line: 78 to 80
 
  • Collect some real data and check with go4.
Changed:
<
<
-- SergeyYurevich - 06 Dec 2007
>
>
-- SergeyYurevich - 07 Dec 2007
 
Revision 7
06 Dec 2007 - Main.SergeyYurevich
Line: 1 to 1
 
META TOPICPARENT name="SoftwareDevelopment"
Table of contents:
Line: 17 to 17
 

Tests with 90 subsystems:
Changed:
<
<
I used TOF0, TOF1 and TOF4 VME CPUs. 30 daq_rdosoft per CPU. Water mark was set to 32 kB.
>
>
Conditions:
  • I used TOF0, TOF1 and TOF4 VME CPUs. 30 daq_rdosoft processes per CPU.
  • Water mark was set to 32 kB.
  • No writing to hard disk, no rfio.
  • net.core.netdev_max_backlog = 10000
  • net.core.rmem_max = 4194304 (nettrans.c:openUdp(): requests 2 MB queue)
  • EB buffer = 8 MB (x2x2). Cannot go to 16 MB because 16x2x2x90 = 5.6 GB (more than available memory).

Data rate Event rate Subevent size EB CPU load comment
90x230kB/sec = 20 MB/sec 2 kHz 116 Bytes 0.25 no discarded evts
90x460kB/sec = 40 MB/sec 4 kHz 116 Bytes 0.35 no discarded evts
90x570kB/sec = 50 MB/sec 5 kHz 116 Bytes 0.6 (tends to slowly increase) no discarded evts
60 MB/sec 6 kHz 116 Bytes >1 (tends to increase) some disc. evts

After increasing subevent size from 116 to 252 Bytes I see many discarded events:

Data rate Event rate Subevent size EB CPU load comment
90x246kB/sec = 22 MB/sec 1 kHz 252 Bytes 0.3 20% disc. evts
90x493kB/sec = 44 MB/sec 2 kHz 252 Bytes 0.4 20% disc. evts

The problem of discarded events with larger subevent sizes also happens even only with 40 subsystems. Increasing EB buffer size from 8 to 16 MB for 40 subsystems did not help. Running Event Builder on hadeb06a ended up with the same problem of discarded events.

Conclusions:
  • Hard to say. Might be that daq_netmem is too slow to extract UDP packets from 2 MB queue??? Requires more investigation.
 
Deleted:
<
<
Data rate Event rate EB CPU load
90x230kB/sec = 20 MB/sec 2 kHz in 40 min grows upto 5.5
90x460kB/sec = 40 MB/sec 4 kHz in 20 min grows upto 5.5
90x570kB/sec = 50 MB/sec 5 kHz in few sec grows upto 5.5

Once EB CPU load approaches 5.5 I start observing many discarded events. EB CPU load is measured by uptime. CPU usage as provided by top, stays around 16% for daq_netmem and 9% for daq_evtbuild independently of the CPU load. It might be an indication of the following problem: the CPU load on the Event Builder is mostly due to kernel trying to handle large UDP packet rate. Since it does not really belong to a given process this might be not reflected in the information on a CPU usage provided by top.

After discussion with Mathias we came to the following:
  • It would be helpful to see complete usage of CPU by the system. Top gives the following numbers for the overall CPU usage:
    • no EB running: 0.2% (user mode), 0.2% (syst mode)
    • EB running: 2% (user mode), 5.2% (syst mode)
  • What happens when water mark is decreased from 32kB to 16kB?
 

Including RFIO to the EB.

Line: 59 to 68
 
  • Total data rate = 16 MHz
  • EB buffer size = 16 MB (*2*2)
Added:
>
>
Was also tested with 90 subsystems. No problems were observed.
  The sizes of the files transfered to Data Mover look reasonable.

Open questions:
  • Local and remote file names should be identical.
  • runNr and seqNr should be identical between local and remote events.
  • Collect some real data and check with go4.
Deleted:
<
<
  • Breaks for a number of subsystems above 33. Will be investigated.
 
Changed:
<
<
-- SergeyYurevich - 23 Nov 2007
>
>
-- SergeyYurevich - 06 Dec 2007
 
Revision 6
04 Dec 2007 - Main.SergeyYurevich
Line: 1 to 1
 
META TOPICPARENT name="SoftwareDevelopment"
Table of contents:
Line: 15 to 15
 
  • All 60 subsystems (soft_rdos) run on hadc07. Distribution of subsystems over different CPUs still run out of synch.
  • No discarded events during 5 hours of running
Added:
>
>
Tests with 90 subsystems:

I used TOF0, TOF1 and TOF4 VME CPUs. 30 daq_rdosoft per CPU. Water mark was set to 32 kB.

Data rate Event rate EB CPU load
90x230kB/sec = 20 MB/sec 2 kHz in 40 min grows upto 5.5
90x460kB/sec = 40 MB/sec 4 kHz in 20 min grows upto 5.5
90x570kB/sec = 50 MB/sec 5 kHz in few sec grows upto 5.5

Once EB CPU load approaches 5.5 I start observing many discarded events. EB CPU load is measured by uptime. CPU usage as provided by top, stays around 16% for daq_netmem and 9% for daq_evtbuild independently of the CPU load. It might be an indication of the following problem: the CPU load on the Event Builder is mostly due to kernel trying to handle large UDP packet rate. Since it does not really belong to a given process this might be not reflected in the information on a CPU usage provided by top.

After discussion with Mathias we came to the following:
  • It would be helpful to see complete usage of CPU by the system. Top gives the following numbers for the overall CPU usage:
    • no EB running: 0.2% (user mode), 0.2% (syst mode)
    • EB running: 2% (user mode), 5.2% (syst mode)
  • What happens when water mark is decreased from 32kB to 16kB?
 

Including RFIO to the EB.

In the future DAQ the data will be sent to the gStore via RFIO.
Revision 5
27 Nov 2007 - Main.SergeyYurevich
Line: 1 to 1
 
META TOPICPARENT name="SoftwareDevelopment"
Added:
>
>
Table of contents:

 

The limitation of maximum 32 subsystems.

The limitation of maximum 32 subsystems was due to the length of the bit mask of a type unsigned long (nettrans.c:
NetTrans_multiRecv
function). New function
NetTrans_multiReceive
Revision 4
23 Nov 2007 - Main.SergeyYurevich
Line: 1 to 1
 
META TOPICPARENT name="SoftwareDevelopment"

The limitation of maximum 32 subsystems.

Line: 7 to 7
 

Tested with 60 subsystems:
  • Total event rate = 2 kHz
Changed:
<
<
  • Total data rate = 10 MNz
>
>
  • Total data rate = 10 MHz
 
  • All 60 subsystems (soft_rdos) run on hadc07. Distribution of subsystems over different CPUs still run out of synch.
  • No discarded events during 5 hours of running
Line: 22 to 22
 
  • /GSI/staging/adsm/v51/inc/ - headers
  • our test setup: rfiodaq:gstore:/hadaqtest/path/file, where hadaqtest is an archive name for the tests
Changed:
<
<
-- SergeyYurevich - 20 Nov 2007
>
>
I have implemented an RFIO client in the Event Builder. Now we have a direct data transfer from EB to Data Mover. RFIO clinet code consists of five functions added to evtbuild.c:
  • rfio_openConnection
  • rfio_openFile
  • rfio_writeFile
  • rfio_closeFile
  • rfio_closeConnection

Tests with 30 subsystems showed good performance (no discarded events) of EB with RFIO client under the following conditions:
  • Total event rate = 5 kHz
  • Total data rate = 16 MHz
  • EB buffer size = 16 MB (*2*2)

The sizes of the files transfered to Data Mover look reasonable.

Open questions:
  • Local and remote file names should be identical.
  • runNr and seqNr should be identical between local and remote events.
  • Collect some real data and check with go4.
  • Breaks for a number of subsystems above 33. Will be investigated.

-- SergeyYurevich - 23 Nov 2007
 
Revision 3
21 Nov 2007 - Main.SergeyYurevich
Line: 1 to 1
 
META TOPICPARENT name="SoftwareDevelopment"

The limitation of maximum 32 subsystems.

Line: 9 to 9
 
  • Total event rate = 2 kHz
  • Total data rate = 10 MNz
  • All 60 subsystems (soft_rdos) run on hadc07. Distribution of subsystems over different CPUs still run out of synch.
Added:
>
>
  • No discarded events during 5 hours of running
 

Including RFIO to the EB.

Revision 2
20 Nov 2007 - Main.SergeyYurevich
Line: 1 to 1
 
META TOPICPARENT name="SoftwareDevelopment"

The limitation of maximum 32 subsystems.

The limitation of maximum 32 subsystems was due to the length of the bit mask of a type unsigned long (nettrans.c:
NetTrans_multiRecv
function). New function
NetTrans_multiReceive
does not use bit mask and should be able to handle any number of subsystems.
Changed:
<
<
Not fully tested yet.
>
>
Tested with 60 subsystems:
  • Total event rate = 2 kHz
  • Total data rate = 10 MNz
  • All 60 subsystems (soft_rdos) run on hadc07. Distribution of subsystems over different CPUs still run out of synch.
 

Including RFIO to the EB.

Line: 18 to 21
 
  • /GSI/staging/adsm/v51/inc/ - headers
  • our test setup: rfiodaq:gstore:/hadaqtest/path/file, where hadaqtest is an archive name for the tests
Changed:
<
<
-- SergeyYurevich - 14 Nov 2007
>
>
-- SergeyYurevich - 20 Nov 2007
 
 
This site is powered by FoswikiCopyright © by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding Hades Wiki? Send feedback
Imprint (in German)
Privacy Policy (in German)