A few years back I was interested in microsecond-precise timekeeping.
My experience was GPS-backed network clocks were surprisingly cheap - but PTP-aware network switches were surprisingly expensive.
And though I thought I could cut the cost of a grandmaster clock from $2000 down to $200, I figured there'd be no point if I then have to connect it to a $20,000 network switch.
I wonder how Facebook are connecting their precise new time servers to their regular servers?
There is a big push in the Industrial automation world to get TSN/PTP deployed more widely and perhaps universally. The idea is to eliminate the dichotomy of real time vs non real time Ethernet networks in control systems from the plant network down to the machine level.
This way you can send EtherCAT frames at equidistant intervals to servo drives in the microsecond range while casually streaming SPC data or Netflix over the same wire.
This is also in conjunction with the newer single pair Ethernet tech that the Automotive world is also interested in.
> In addition to the more computer-oriented two and four-pair variants, the 100BASE-T1 and 1000BASE-T1 single-pair Ethernet PHYs are intended for automotive applications[17] or as optional data channels in other interconnect applications.[18] The single pair operates at full duplex and has a maximum reach of 15 m or 49 ft (100BASE-T1, 1000BASE-T1 link segment type A) or up to 40 m or 130 ft (1000BASE-T1 link segment type B) with up to four in-line connectors. Both PHYs require a balanced twisted pair with an impedance of 100 Ω. The cable must be capable of transmitting 600 MHz for 1000BASE-T1 and 66 MHz for 100BASE-T1.
> Similar to PoE, Power over Data Lines (PoDL) can provide up to 50 W to a device.[19]
> The IEEE 802.3bu-2016[12] amendment introduced single-pair Power over Data Lines (PoDL) for the single-pair Ethernet standards 100BASE-T1 and 1000BASE-T1 intended for automotive and industrial applications.[13] On the two-pair or four-pair standards, power is transmitted only between pairs, so that within each pair there is no voltage present other than that representing the transmitted data. With single-pair Ethernet, power is transmitted in parallel to the data. PoDL defines 10 power classes, ranging from .5 to 50 W (at PD).
100BASE-T1 is IEEE 802.3bw-2015, 1000BASE-T1 is IEEE 802.3bp-2016. 2.5 Gb/s, 5 Gb/s, and 10 Gb/s over a single pair is 802.3ch-2020: the focus of these is in the embedded automotive space.
Most (if not all) our switches in the datacenter are capable to perform as transparent clocks. With transparent clocks you can keep the time sync error down to less than 5ns per hop
Do the transparent clocks in your datacenter support unicast PTP?
There is a possibility to use PTP as a transport for NTP to take advantage of PTP-specific hardware timestamping. It could also process the correction field, but it seems the switches typically don't support unicast PTP.
The large asymmetry and banding of NTP in the test with Calnex Sentinel suggests it doesn't support the interleaved mode. NTP with hardware timestamping should normally be much more stable and symmetric, closer to PTP.
The unicast PTP support can be limited to boundary clocks and the enterprise profile doesn't require transparent clocks to support the unicast mode.
Also, there are different types of PTP transparent clocks. They can either be end-to-end or peer-to-peer, and either one-step or two-step clocks. To be useful for NTP, I think it would need to be an end-to-end transparent clock and ideally it would be a one-step clock to avoid dealing with with the follow-up messages.
Even the retail price of 1 switch based on their "wedge" design is less than $10k for 32 100gbps ports. Merchant silicon like the Broadcom 56000-series switch ASICs support PTP.
There are consensus protocols that allow a grace period for clock skew, and the narrower you can constrain the clock skew, the more transactions per second you can retire, improving both scalability and latency.
I work in trading, but mostly on the asian markets, so not sure about mifid. Usually the exchanges seem to give milli to second precision (which is very annoying: if Thailand says it's second-precision to timestamp their order executions but Taiwan says it's milli, our system has to use the most precise when ordering them and we get questions from our own multi market clients about why all these executions are seemingly simultaneous or why our system source marked the request at .480 while the exchange replied with 0.0 for the execution.
But, by no mean, does it mean we use special timing - but now that I say that, I realize I have no clue where our time signal comes from, clearly it wouldn't be the internet because our trading servers aren't exactly directly connected there, we must have an internal ntp sever o_O
Not all exchanges are like this. The JPX exchange in Tokyo provides PTP over dedicated cross-connects, and is super accurate, within +/-25ns. It’s pretty neat seeing a PTP clock from Seiko.
There are plenty of practical applications for it.
Imagine super high speed, high precision robotics, maybe a synthetic fiber layup machine or an exotic milling machine, that moves at 10m/s. If you have 1us of precision and accuracy then you can send movement commands that are precise down to 10um.
In these cases mostly not, they require low jitter references and a good counter.
An example that does - I work for IceCube, a big neutrino detector at the South Pole. The detector is essentially a time of flight system that detects emitted light as it propagates through the ice when a neutrino interaction happens. We need good time distribution in the system so we can marry up hits from the 5000 odd sensors in the ice (and successive hits may be nanoseconds apart), and we need good absolute precision because we collaborate with other observatories.
Also pretty much all radio telescopes that do long baseline interferometry need excellent absolute timing. The South Pole Telescope has (I think) the most accurate clock on station, a hydrogen maser-based system. We use GPS.
My experience was GPS-backed network clocks were surprisingly cheap - but PTP-aware network switches were surprisingly expensive.
And though I thought I could cut the cost of a grandmaster clock from $2000 down to $200, I figured there'd be no point if I then have to connect it to a $20,000 network switch.
I wonder how Facebook are connecting their precise new time servers to their regular servers?