Streamed Event Building

The name "streamed event building" comes from the fact that inside the network there is no place where we can guarantee to be able to store a complete event. Therefor we have to be able to generate SubEventHeaders on the fly without knowledge of the complete datastream.

The data structure sent by frontends, generated by hubs and the SubEventBuilder is described here:
  • ipu_data_format.pdf: A detailed description of the data format in each stage of the readout system.

What is described in this document, is also shown in the scheme drawn below: A drawing of the data structures through all stages of the network

The original Proposal

The part below was the first proposal to do efficient streamed event building, and may not be in complete agreement with the documents above depicting the current development status.

The mechanism described here are valid for the IPU Data channel only. The trigger LVL1 and slow control channels will be kept with the old behaviour.

A proposal presented on the DAQ meeting on 18. and 19. Dec 2008 is layed out here for discussion. A way of easy subevent building during TRBnet transport is given here. Please contribute in the discussion!

Naming convections

Packet : 80bit of data inside TRBnet, consisting of H0, F0, F1, F2 and F3. The here used packet types are header (HDR), termination (TRM) and data (DAT)

Frame : assembly of packets, starting with a header and ending with a termination.

The normal behaviour of hubs is a simple concatenation of all incoming streams to one long stream: H - D - D - H - D - D - H - D - T

For subevent building we need a slightly different behaviour with a common header in front and with or without the original headers from the FEE.

Example "not packed": H - H - D - D - D - H - D - H - D - H - D - T

Example "packed": H - D - D - D - D - D - D - D - T

A "short" frame consists only of a termination.

Channel usage on TRBnet

TRBnet offers four virtual channels (CH0 - CH3) with decreasing priority in order to serve for different purposes.
Channel usage uplink downlink
0 highest priority, used as trigger distribution channel single packets just termination
1 IPU data channel, used for transporting data from FEs to CTS few packets huge data amounts
2 reserved    
3 slow control moderate data moderate to huge

Return data on channel 1 will be streamed out of the network at a certain point to cross over to Gigabit ethernet towards the event builder. It is recommended to have all network related parts removed from there to get rid of unneeded data which would only occupy expensive disk space. A kind of "subevent building on the fly" is necessary here.

Changes in the packet format

The packet structure needs a small change for increasing data security on the network. A 16bit field in the header will be used to inform about how many TRBnet packets will form the frame. The number of packets will exclude the HDR itself and the TRM, and therefore only denote the number of packets inside this frame.

The length information is by no means a strict limit on TrbNet. It is only needed for building Subevent Headers on channel 2 - there is no impact to TrbNet when the number of announced data packets is exceeded! This also means that the length information is an optional feature and only used on channel 1. A value of xFFFF implies an unknown length of the transfer, which can be used on any channel despite channel 1.

Streamed event building

The streaming event building must fullfill the task of concenating data from on detector system originating from several sources, like in case of RICH from five ADCMs per sector and 30 ADCMs in total. Several layers of hubs must be supported transparently to allow larger systems (like MDC).

Requirements for this task:

  1. each endpoint data transmission on IPU channel must start with a fixed header (DHDR)
  2. each endpoint must return at least the header when being ordered to deliver data
  3. empty datapackets are allowed, but must contain a header
  4. each endpoint must inform the streaming event builder if data can be packed or not (i.e. if data words are complete or need the source address information).

During streaming event building it is mandatory to check trigger information of data segments when concenating data streams. In case of mismatch this event is further processed, but marked as broken.

A mechanism allowing simple tolerance for single broken events should be implemented, but constant errors must stop the DAQ.

DHDR format

Data to be transported on IPU channel is transfered 32bit wide. The smallest unit to be used is 32bit. The DHDR consists of two 32bit words.

The first DHDR word contains the trigger bus information belonging to this data sample, namely:

D[31:29] reserved
D[28] pack bit (0 = don't pack, 1 = pack)
D[27:24] trigger type
D[23:16] trigger random
D[15:0] trigger number
The second DHDR word contains the length of the data packet following, as well as the TRBnet address of the module the data originates from:

D[31:16] length in 32bit words
D[15:0] TRBnet address

This header must be delivered from each endpoint on each data request on channel 1, even when there is no data to be returned.

If the endpoint uses an intrinsic data format with a word size different to 32bit, it has to care for the padding to the next 32bit boundary.

Procedure explained on an example

Let's take an three endpoints of a detector, each one delivering data. Take a look into the PDF attached. The first endpoint delivers three data words, the second none, and the third two.

Data is put into the API and is sent to the hub where it arrives in the receive FIFOs as TrbNet packets.

The streaming event builder logic now removes the HDR packets from all three FIFOs, storing only the length information. Next, the DAT header is removed, and the next two trigger words from all three FIFOs are compared against each other. These two words are also stored.

The next word from the FIFOs delivers the data length of each data seqment. Together with the information from the TrbNet headers the new length of the concenated data segment is calculated. The length calculation depends on the PACK bit from the data header; if it is set to one, the logic knows that all data words are containing unique channel information, so the individual headers can be stripped from data stream. If set to zero, the original headers will stay in the data stream.

A new TrbNet HDR packet is prepared, with a new length information and a new source of data, which is the TrbNet address of the hub.

The first two data words now will build up a new DHDR for the concatenated data stream, as defined above. This allows simple staggering several layers of hubs to concatenate data streams in a transparent way.

Now data from the first FIFO is being transfered to the transmit FIFO, yielding in the data stream depicted on the right side of the sheet (named "TRBnet unpacked" and "TRBnet packed"). The TRM from the first FIFO is discarded, then data from the second FIFO is appended. The third FIFO follows.

Finally a TRM packet is created and inserted into the transmit FIFO. In case anything went wrong during data concenation (like: mismatching trigger information, wrong length information, broken packets, ...) the errorbits in the TRM packet will be used to inform the CTS about this failure.

On the hub endpoint where the concenated data stream is streamed out of TrbNet and delivered to some other entitiy (preferably a TCP/IP connection to event builder) all TrbNet related stuff is taken out (named "TCP/IP unpacked" and "TCP/IP packed"). A simple logic will have to convert the leading two DHDR words to the standard HADES subevent header.

-- MichaelBoehmer - 20 Dec 2008
  • seb.pdf: Example streaming event builder

I Attachment Action Size Date Who Comment
ipu_data_format.pdfpdf ipu_data_format.pdf manage 166 K 2009-07-09 - 12:45 JanMichel A detailed description of the data format in each stage of the readout system.
seb.pdfpdf seb.pdf manage 20 K 2008-12-21 - 16:39 MichaelBoehmer Example streaming event builder
seb2_eb2.pdfpdf seb2_eb2.pdf manage 26 K 2009-07-09 - 12:46 JanMichel A drawing of the data structures through all stages of the network
seb2_eb2.pngpng seb2_eb2.png manage 61 K 2009-07-09 - 12:48 JanMichel A drawing of the data structures through all stages of the network
Topic revision: r9 - 2009-07-09, JanMichel
Copyright © by the contributing authors. All material on this collaboration platform is the property of the contributing authors.
Ideas, requests, problems regarding Foswiki Send feedback | Imprint | Privacy Policy (in German)