Line data Source code
1 : #ifndef HEADER_fd_src_tango_tempo_fd_tempo_h 2 : #define HEADER_fd_src_tango_tempo_fd_tempo_h 3 : 4 : /* APIs for measuring time and tick intervals */ 5 : 6 : #include "../fd_tango_base.h" 7 : 8 : FD_PROTOTYPES_BEGIN 9 : 10 : /* fd_tempo_wallclock_model returns an estimate of t0, the minimum cost 11 : of fd_log_wallclock() in ticks. If opt_tau is non_NULL, on return, 12 : *opt_tau will contain an estimate of typical jitter associated with 13 : fd_log_wallclock() (such that fd_log_wallclock() can be roughly 14 : modeled as a shifted exponential distribution with minimum of t0 and 15 : wait time of tau, average cost of t0 + tau, rms of tau). The first 16 : call of this in a thread group will be slow and all subsequent calls 17 : in the thread group will be fast and return the identical parameters 18 : to the first call. t0 will be finite and positive and the tau will 19 : be finite and non-negative. If the fd_log_wallclock() cannot be 20 : sanely parameterized on the first call, logs a warning and uses a 21 : fallback parameterization. */ 22 : 23 : double 24 : fd_tempo_wallclock_model( double * opt_tau ); 25 : 26 : /* fd_tempo_tickcount_model does the same as fd_tempo_wallclock model 27 : for fd_tickcount(). The model parameter units will be in ticks 28 : instead of nanoseconds. */ 29 : 30 : double 31 : fd_tempo_tickcount_model( double * opt_tau ); 32 : 33 : /* fd_tempo_set_tick_per_ns explicitly sets the return values of 34 : fd_tempo_tick_per_ns below, subsequent calls to that function will 35 : return the values given here. 36 : 37 : These should not be arbitrarily provided, and this function is here 38 : primarily to enable different processes to synchronize their 39 : tick_per_ns value. */ 40 : 41 : void 42 : fd_tempo_set_tick_per_ns( double _mu, 43 : double _sigma ); 44 : 45 : /* fd_tempo_tick_per_ns is the same as the above but gives an estimate 46 : of the rate fd_tickcount() ticks relative to fd_log_wallclock() (this 47 : is in Ghz). The returned value is the observed rate when 48 : fd_tempo_tick_per_ns was first called in the thread group (this call 49 : will take around ~0.5 s). If opt_sigma is non-NULL, on return, 50 : *opt_sigma will have an estimate how much the rate was fluctuating 51 : when observed during the first call. 52 : 53 : IMPORTANT! Though fd_tickcount() is meant to tick at a constant rate 54 : relative to fd_log_wallclock(), the instantaneous rate it ticks can 55 : fluctuate for the usual of clock synchronization reasons (e.g. 56 : thermal and electrical effects from CPU load, CPU clock timing 57 : jitter, similar for the wallclock, etc). As this is an invariant 58 : counter, reasons for it to fluctuate do _NOT_ include directly things 59 : like turbo mode cpu clock frequency changes (it might slightly 60 : indirectly impact it due to correlated changes to system thermal and 61 : electric conditions from the changed power draw). As such, this is 62 : mostly meant for useful for getting a thread group wide consistent 63 : estimate of the number of ticks in a short interval of ns. 64 : 65 : TL;DR This returns an estimate of fd_tickcount()'s clock speed in 66 : GHz. This is _NOT_ the current clock speed of the processor though 67 : it will usually superficially look like it. This is _NOT_ the 68 : instantaneous rate the tickcounter is ticking relative to the 69 : wallclock though it will usually superficially look like it. */ 70 : 71 : double 72 : fd_tempo_tick_per_ns( double * opt_sigma ); 73 : 74 : /* fd_tempo_tick_per_ns_dev is a development-only variant of 75 : fd_tempo_tick_per_ns that measures the rate with a much smaller 76 : wall-clock budget (~1 ms vs ~0.5 s) at the cost of a few ppm of 77 : accuracy. It is intended for firedancer-dev, where boot latency 78 : matters and the rate is not used for anything critical. Like 79 : fd_tempo_tick_per_ns it is FD_ONCE guarded and honors a prior 80 : fd_tempo_set_tick_per_ns; additionally, calling this marks the value 81 : explicit so a later fd_tempo_tick_per_ns() returns this fast 82 : measurement rather than re-running the slow production sampling. Do 83 : NOT use in production. */ 84 : 85 : double 86 : fd_tempo_tick_per_ns_dev( double * opt_sigma ); 87 : 88 : /* fd_tempo_observe_pair observes the fd_log_wallclock() and 89 : fd_tickcount() at the "same time". More precisely, it alternately 90 : observes both a few times and estimates from the "best" wallclock 91 : read what tickcount would have been observed at that time had 92 : fd_tickcount() been called instead. Returns a non-negative measure 93 : of the jitter in ticks in the sense observed tickcount is within 94 : +/-0.5 jitter ticks of the time the wallclock was observed. On 95 : return, if opt_now is non-NULL, *opt_now will contain the actual 96 : fd_log_wallclock() observation and, if opt_tic is non-NULL, *opt_tic 97 : will contain the estimated simultaneous fd_tickcount() observation. 98 : 99 : If anything wonky is detected in the measurement, logs a warning and 100 : returns a best effort. As this does multiple reads under the hood 101 : and uses only one of them, the observed value should be interpreted 102 : as at some point in time between when the call was made and when 103 : the call returned but not always at the same point (can be roughly 104 : modeled as uniformly distributed between when the call was made and 105 : when it returned). 106 : 107 : While this isn't particularly expensive, it isn't particularly cheap 108 : either. Cost is on the order of a few calls to fd_wallclock plus a 109 : few calls to fd_tickcount(). This is mostly meant for doing 110 : precision timing calibrations. */ 111 : 112 : long 113 : fd_tempo_observe_pair( long * opt_now, 114 : long * opt_tic ); 115 : 116 : /* fd_tempo_lazy_default returns a target interval between housekeeping 117 : events in ns (laziness) for a producer / consumer that has a maximum 118 : credits of cr_max / lag behind the producer of lag_max. 119 : 120 : To understand this default, note that a producer should receive / a 121 : consumer should transmit complete flow control credits least as often 122 : as the time it takes a producer to exhaust all its credits / a slow 123 : consumer to process a worst case backlog of lag_max credits. 124 : Otherwise, the communications throughput will be limited by the rate 125 : credits can flow from consumer to producer rather than the rate the 126 : producer can publish / consumer can receive. At the same time, we 127 : don't want to be too eager to return credits to avoid consumer to 128 : producer credit communications competing for NOC resources with 129 : producer to consumer communications. 130 : 131 : This implies we need to update all flow control information on a tile 132 : somewhat faster than: 133 : 134 : cr_max max( typical time it takes a producer to consume a credit, 135 : typical time it takes a consumer to produce a credit ) 136 : 137 : Practical considerations applied to this yield a useful conservative 138 : lower bound: 139 : 140 : Assume credits are network packets (as is often the case), the above 141 : times are the typical time it takes a producer / consumer to generate 142 : / process a packet. Given a producer line-rating minimal sized 143 : Ethernet frames (672 bits) at 100G into a mcache / dcache and 144 : consumers that are keeping up with this producer (both highly 145 : unrealistically harsh situations in the real world as this implies 146 : Ethernet payloads much much smaller than typical real world payloads 147 : and a consumer that can process packets in just a handful of ns), the 148 : above suggests housekeeping done somewhat than: 149 : 150 : ~(cr_max pkt)(672 bit/pkt/100 Gbit/ns) 151 : 152 : will be adequate for all practical purposes. Given that the typical 153 : randomized housekeeping event will be at most ~1.5 lazy, we have: 154 : 155 : lazy < ~cr_max*672/100e9/1.5 ~ 4.48 cr_max 156 : 157 : We use 1+floor( 9*cr_max/4 )) ~ 2.25 cr_max to keep things simple. 158 : Note that that while this might seem aggressive per credit, since 159 : cr_max is typically values in thousands to hundreds of thousands, 160 : this corresponds to default laziness in the tens microseconds to 161 : milliseconds. We also saturate cr_max to keep the returned value in 162 : [1,2^31] ns for all cr_max. */ 163 : 164 : FD_FN_CONST static inline long 165 48 : fd_tempo_lazy_default( ulong cr_max ) { 166 48 : return fd_long_if( cr_max>954437176UL, (long)INT_MAX, (long)(1UL+((9UL*cr_max)>>2)) ); 167 48 : } 168 : 169 : /* fd_tempo_async_min picks a reasonable minimum interval in ticks 170 : between housekeeping events. On success, returns positive integer 171 : power of two in [1,2^31]. On failure, returns zero (logs details). 172 : Reasons for failure include lazy is not in [1,2^31), event_cnt is not 173 : in [1,2^31), tick_per_ns is not in (0,~1.5e29), the combination would 174 : require an unreasonably small (sub-tick) or large (more than 2^31) 175 : async_min. 176 : 177 : More precisely, consider a run loop where event_cnt out-of-band 178 : housekeeping events are cyclicly scheduled to be done with a IID 179 : uniform random interval between events in [async_min,2*async_min] 180 : ticks (as is commonly the case). And suppose we need housekeeping 181 : to complete an event cycle roughly every lazy ns for system 182 : considerations. 183 : 184 : If we were to use a regularly scheduled interval between events (which 185 : is a stunningly bad idea in an distributed system and all too 186 : commonly done), we'd space housekeeping events by: 187 : 188 : async_target ~ tick_per_ns*lazy/event_cnt ticks 189 : 190 : where tick_per_ns is the conversion ratio to use between the 191 : wallclock and the tickrate of whatever counter is used to schedule 192 : housekeeping events. 193 : 194 : Consider using the largest integer power of two less than or equal to 195 : async_target for async_min. In ns then, async_min will be at least 196 : ~0.5*lazy/event_cnt and at most lazy/event_cnt. And since it takes, 197 : on average, 1.5*async_min*event_cnt to process a cycle, this value 198 : for async min will yield an average cycle time of at least ~0.75*lazy 199 : in ns and at most ~1.5*lazy ns. */ 200 : 201 : ulong 202 : fd_tempo_async_min( long lazy, 203 : ulong event_cnt, 204 : float tick_per_ns ); 205 : 206 : /* fd_tempo_async_reload returns a quality random number very quickly in 207 : [async_min,2*async_min). Assumes async_min is an integer power of 2 208 : in [1,2^31]. Consumes exactly 1 rng slot. This is typically used to 209 : randomize the timing of background task processing to avoid auto 210 : synchronization anomalies while providing given strong lower and 211 : upper bounds on the interval between between processing background 212 : tasks. */ 213 : 214 : static inline ulong 215 : fd_tempo_async_reload( fd_rng_t * rng, 216 135667342 : ulong async_min ) { 217 135667342 : return async_min + (((ulong)fd_rng_uint( rng )) & (async_min-1UL)); 218 135667342 : } 219 : 220 : FD_PROTOTYPES_END 221 : 222 : #endif /* HEADER_fd_src_tango_tempo_fd_tempo_h */
