DDR4

Purpose of DDR4: when you get the click event on detection, you need to find the angle applied to that qubit (basis information). DDR4 is used to store the angle so that after getting click event, base on value of global counter, you can find the angle. One other reason is that we have constraints over 100km distance between Alice and Bob, the delay on classical channel, so DDR4 is large enough to satisfy these constraints. Below is the overview pictures of modules and IPs in FPGA manage the data flow in DDR4:

• IP DDR4: MIG IP supported by AMD. The core allow you interface directly with the physical Memory. To configure the MIG, follow instructions on opalkelly DDR4 Memory
• axi_virtual_controller_wrapper.v : use AXI Virtual FIFO Controller core from AMD to access DRAM memory as multiple FIFO blocks
• axi_clock_converter_rlt.v use AXI Clock Converter core from AMD as interconnect, change clock domain, because AXI interface on MIG uses 300MHz clock domain
• system_ila_ddr: monitoring AXI, AXIS interface and debug signals
• ddr_data_reg_mngt.v: manages axil registers for commands, settings, status monitoring
• ddr_data.v: manages data flow in/out axi_virtual_controller_wrapper, in/out xdma axistream fifos
• mon_ddr_fifos.v: manages registers to monitor status of AXI Virtual FIFO Controller and axistream fifos
• fifos_out.v: instantiate axistream output fifos. Instantiate in an RTL module allows Vivado changes FREQ_HZ parameter after rebuild block design from Tcl script

Port descriptions

axi_clock_converter_rtl.v

This module instantiates AXI Clock Converter IP of Xilinx. Post description is in Xilinx datasheet.

axi_virtual_controller_wrapper.v

This module instantiates AXI virtual Fifo Controller IP of Xilinx. Post description is in Xilinx datasheet.There are 3 optional ports for monitoring.

ddr4

This is IP of Xilinx. All information is in Xilinx datasheet

ddr4_data.v

ddr_data_reg_mngt.v

mon_ddr_fifos.v

fifos_out.v

This module instantiates 2 fifos: gc_out fifo and alpha fifo in AXIStream mode of FIFO Generator. Description of FIFO Generator is providded by Xilinx

Axil registers

• dq : double qubit, 40MHz
• LSB : Least Significant Bit
• MSB : Most Significant Bit
• Base address: 0x0000_1000
• Offset address slv_reg(n) : 4*n

slv_reg0 - R/W Access - Trigger Control

slv_reg1 - R/W Access - Trigger Control

slv_reg2 - R/W Access - Configuration

slv_reg3 - R/W Access - Trigger Control

slv_reg4 - R/W Access - Configuration

slv_reg5 - R/W Access - Configuration

slv_reg6 - R/W Access - Configuration & Trigger Control

slv_reg7 - R/W Access - Trigger Control

slv_reg8 - R/W Access - Configuration

slv_reg9 - R/W Access - Configuration

slv_reg10 - R/W Access - Configuration

slv_reg11 - R/W Access - Configuration

slv_reg12 - R Access - Monitoring

slv_reg13 - R Access - Monitoring

slv_reg14 - R Access - Monitoring

slv_reg15 - R Access - Monitoring

slv_reg16 - R Access - Monitoring

Data flow

START

Alice sends START command to Bob through Ethernet. They both send the command to their FPGA, the START state will happen at next PPS and synchronise. Network latency has to be small enough, START command on Alice should not be close to the rising edge of PPS.

To make sure START command is not close to rising edge of PPS, Alice will request PPS detection from FPGA, she delays at least 10ms (PPS duty cycle) and send START command. Readback global counter on both Alice and Bob, compare to verify the synchronisation

WRITE MANAGEMENT

In START state, start to count up double global counter and write angles to DDR4. Angles are written as axistream data to AXI Virtual FIFO controller IP. This IP manages the memory map in the MIG, when you want to write or read from DDR4, you just need to manage write/read axistream of AXI Virtual FIFO controller.

The angle includes angle for PM and angle for the second AM. Dedicate 8bits to encode:

• 4 LSB : for PM angle
• next 2 bits : for 2nd AM angle
• 2 MSB : reserved zeros

GC PATH

Bob FPGA gets detection result, sends gc (dq_gc and q_pos) and click result to Bob OS, only output when gc higher than alice-bob fiber_delay. Bob then send gc to Alice (through Ethernet). They sends gc to their FPGA

READ DDR4 MANAGEMENT

When FPGA of each party receives gc, start reading angles from DDR4 based on values of gc and fiber delays value. Make sure fifo_gc_in is not full and AXI Virtual FIFO Controller is not full, by defining fifo_gc_in reading speed higher than click rate, define depth of Virtual FIFO large enough. Fiber delay from angle applied to Alice's 2nd AM is different with the one applied to Alice's PM, reading these angles respectively, then saving to the angles fifo with 4bits encoding:

• 2 LSB: for PM angle
• next 1 bit: for 2nd AM angle
• MSB: reserved zero

SAVE ANGLES

Start saving the angles read from DDR4 to fifo_alpha_out. Choose a moment(value of gc) to start saving, consider the fiber delay between parties. Each party have to read angles before fifo_alpha_out is full.

This is the picture describes the states in FPGA, the path of data between Alice and Bob.

Details in states COUNTING_*. Currently, Alice second AM is placed after Alice PM so the decoy_fiber_delay is shorter than ab_fiber_delay, we jump to COUNTING_AL first

Software control functions

• Ddr_Data_Reg : Set registers

def Ddr_Data_Reg(command,current_gc,read_speed, fiber_delay, pair_mode, de_fiber_delay, de_pair_mode, ab_fiber_delay):
    Write(0x00001000+8,hex(int(command)))
    dq_gc_start = np.int64(current_gc) #+s
    print(hex(dq_gc_start)) 
    gc_lsb = dq_gc_start & 0xffffffff
    gc_msb = (dq_gc_start & 0xffff00000000)>>32
    threshold_full = 50000 #optinal for debug
    Write(0x00001000+16,hex(gc_lsb))
    Write(0x00001000+20,hex(gc_msb))
    Write(0x00001000+32,hex(read_speed))
    Write(0x00001000+36,hex(threshold_full))
    Write(0x00001000+40,hex(de_fiber_delay<<16 | fiber_delay)) #de_fiber_delay only on Alice
    Write(0x00001000+44,hex(ab_fiber_delay)) #Only on Bob
    Write(0x00001000+24,hex(de_pair_mode<<2 | pair_mode<<1)) #de_pair_mode only on Alice
    #Enable register setting
    Write(0x00001000+12,0x0)
    Write(0x00001000+12,0x1)

• Ddr_Data_Init: reset ddr_data module

def Ddr_Data_Init():
    #Reset module
    Write(0x00001000, 0x00) #Start write ddr = 0
    Write(0x00012000 + 16,0x00)
    Write(0x00012000 + 16,0x01)
    time.sleep(1)
    print("Reset ddr data module")

• Ddr_Status: monitoring the fifos flags, monitoring every 0.1s

def Ddr_Status():
   while True:
        ddr_fifos_status = Read(0x00001000 + 52)
        fifos_status = Read(0x00001000 + 56)
        hex_ddr_fifos_status = ddr_fifos_status.decode('utf-8').strip()
        hex_fifos_status = fifos_status.decode('utf-8').strip()
        vfifo_idle = (int(hex_ddr_fifos_status,16) & 0x180)>>7
        vfifo_empty = (int(hex_ddr_fifos_status,16) & 0x60)>>5
        vfifo_full = (int(hex_ddr_fifos_status,16) & 0x18)>>3
        gc_out_full = (int(hex_ddr_fifos_status,16) & 0x4)>>2
        gc_in_empty = (int(hex_ddr_fifos_status,16) & 0x2)>>1
        alpha_out_full = int(hex_ddr_fifos_status,16) & 0x1
        gc_out_empty = (int(hex_fifos_status,16) & 0x4)>>2
        gc_in_full = (int(hex_fifos_status,16) & 0x2)>>1
        alpha_out_empty = int(hex_fifos_status,16) & 0x1
        current_time = datetime.datetime.now()
        print(f"Time: {current_time} VF: {vfifo_full} VE: {vfifo_empty}, VI: {vfifo_idle} | gc_out_f,e: {gc_out_full},{gc_out_empty} | gc_in_f,e: {gc_in_full},{gc_in_empty} | alpha_out_f,e: {alpha_out_full},{alpha_out_empty}", flush=True)
        #print("Time: {current_time}  VF: {vfifo_full}, VE: {vfifo_empty}, VI: {vfifo_idle} | gc_out_f,e: {gc_out_full}, {gc_out_empty} | gc_in_f,e: {gc_in_full}, {gc_in_empty} | alpha_out_f,e: {alpha_out_full}, {alpha_out_empty}                                                                      " ,end ='\r', flush=True)
        time.sleep(0.1)

Last test result List of commands

In step 6, there are some parameters:

• 1999: define speed of read gc_in fifo. This is for click rate more than 50k and less than 100k
• the delay and pair parameters defined from value of returned fiber delay after calibration

• In server and client mechanism, try to START sending gc
• Reading alpha from alpha fifo when status of alpha_out_fifo is not empty. Pay attention to alpha out rate to avoid timeout on xdma

For testing single device, run ddr_loop_test.py. Detail is in Hardware Testing Chapter