UC San Diego SearchMenu

Connections take time & communication

Assembling the connection from San Diego to Amsterdam for the first Prism-CHERuB test took over 5 weeks of group email. This was partly because of the large number of waypoints involved, and partly because of the layer-2 nature of the connection. Each waypoint provider had to be actively involved, confirming vlan numbers and physical connectivity details.

In addition, layer-2 troubleshooting requires visibility that just isn't there end-to-end on these multi-provider vlans. It's necessary to get technical engineers in each waypoint provider involved when diagnostics or performance doesn't meet expectations. Coordination is essential.

Interim and last-mile restrictions constrain

For the San Diego-Amsterdam route, we were able to obtain 40G on one of the interim routes, which was a testbed requiring scheduling. That capped our possible bandwidth and delayed our rollout until the testbed was ready.

For a test for San Diego-New Zealand, the remote site turned out to be 10x10G, rather than 1x100G. Currently the ability to make use of this for greater-than-10G flows is limited, so this test was less interesting than it originally appeared.

Keep your fiber connections clean

Most of the poor performance our initial tests revealed were caused by dirty optics/jumpers. This showed up as framing errors/low-level packet loss on the affected connection. The apparent small effect was enough to significantly limit bandwidth.

Tech specs

A large window and buffer size may be needed to remove effects of latency over long connections. Our scientists calculated a minimum window size of
{round trip time} * 5GB/sec. This was 815MB for the test Prism@UCSD ran; they suggested a 1GB max window.