Line data Source code
1 : #ifndef HEADER_fd_src_flamenco_runtime_fd_bank_h
2 : #define HEADER_fd_src_flamenco_runtime_fd_bank_h
3 :
4 : #include "../types/fd_types.h"
5 : #include "../leaders/fd_leaders.h"
6 : #include "../features/fd_features.h"
7 : #include "../rewards/fd_epoch_rewards.h"
8 : #include "../stakes/fd_stake_delegations.h"
9 : #include "../stakes/fd_vote_states.h"
10 : #include "../fd_rwlock.h"
11 : #include "fd_blockhashes.h"
12 : #include "sysvar/fd_sysvar_cache.h"
13 : #include "../../ballet/lthash/fd_lthash.h"
14 :
15 : FD_PROTOTYPES_BEGIN
16 :
17 9 : #define FD_BANKS_MAGIC (0XF17EDA2C7EBA2450) /* FIREDANCER BANKS V0 */
18 :
19 : /* TODO: Some optimizations, cleanups, future work:
20 : 1. Simple data types (ulong, int, etc) should be stored as their
21 : underlying type instead of a byte array.
22 : 2. Perhaps make the query/modify scoping more explicit. Right now,
23 : the caller is free to use the API wrong if there are no locks.
24 : Maybe just expose a different API if there are no locks?
25 : 3. Rename locks to suffix with _query_locking and _query_locking_end
26 : 4. Replace memset with custom constructors for new banks.
27 : 5. Don't templatize out more complex types.
28 : */
29 :
30 : /* A fd_bank_t struct is the representation of the bank state on Solana
31 : for a given block. More specifically, the bank state corresponds to
32 : all information needed during execution that is not stored on-chain,
33 : but is instead cached in a validator's memory. Each of these bank
34 : fields are repesented by a member of the fd_bank_t struct.
35 :
36 : Management of fd_bank_t structs must be fork-aware: the state of each
37 : fd_bank_t must be based on the fd_bank_t of it's parent block. This
38 : state is managed by the fd_banks_t struct.
39 :
40 : In order to support fork-awareness, there are several key features
41 : that fd_banks_t and fd_bank_t MUST support:
42 : 1. Query for any non-rooted block's bank: create a fast lookup
43 : from eslot to bank
44 : 2. Be able to create a new bank for a given block from the bank of
45 : that block's parent and maintain some tree-like structure to
46 : track the parent-child relationships: copy the contents from a
47 : parent bank into a child bank.
48 : 3. Prune the set of active banks to keep the root updated as the
49 : network progresses: free resources of fd_bank_t structs that
50 : are are not direct descendants of the root bank (remove parents
51 : and any competing lineages).
52 : 4. Each bank will have field(s) that are concurrently read/write
53 : from multiple threads: add read-write locks to the fields that are
54 : concurrently written to.
55 : 5. In practice, a bank state for a given block can be very large and
56 : not all of the fields are written to every block. Therefore, it can
57 : be very expensive to copy the entire bank state for a given block
58 : each time a bank is created. In order to avoid large memcpys, we
59 : can use a CoW mechanism for certain fields.
60 : 6. In a similar vein, some fields are very large and are not written
61 : to very often, and are only read at the epoch boundary. The most
62 : notable example is the stake delegations cache. In order to handle
63 : this, we can use a delta-based approach where each bank only has
64 : a delta of the stake delegations. The root bank will own the full
65 : set of stake delegations. This means that the deltas are only
66 : applied to the root bank as each bank gets rooted. If the caller
67 : needs to access the full set of stake delegations for a given
68 : bank, they can assemble the full set of stake delegations by
69 : applying all of the deltas from the current bank and all of its
70 : ancestors up to the root bank.
71 :
72 : Each field of a fd_bank_t has a pre-specified set of fields including
73 : - name: the name of the field
74 : - footprint: the size of the field in bytes
75 : - align: the alignment of the field
76 : - CoW: whether the field is CoW
77 : - has_lock: whether the field has a rw-lock
78 : - type: type of the field
79 :
80 : fd_banks_t is represented by a left-child, right-sibling n-ary tree
81 : (inspired by fd_ghost) to keep track of the parent-child fork tree.
82 : The underlying data structure is a map of fd_bank_t structs that is
83 : keyed by eslot. This map is backed by a simple memory pool.
84 :
85 : NOTE: The reason fd_banks_t is keyed by fd_eslot_t and not by slot is
86 : to handle block equivocation: if there are two different blocks for
87 : the same slot, we need to be able to differentiate and handle both
88 : blocks against different banks. fd_eslot_t is a 64-bit bitfield that
89 : contains both the slot number and a prime counter.
90 :
91 : Each field in fd_bank_t that is not CoW is laid out contiguously in
92 : the fd_bank_t struct as simple uchar buffers. This allows for a simple
93 : memcpy to clone the bank state from a parent to a child.
94 :
95 : Each field that is CoW has its own memory pool. The memory
96 : corresponding to the field is not located in the fd_bank_t struct and
97 : is instead represented by a pool index and a dirty flag. If the field
98 : is modified, then the dirty flag is set, and an element of the pool
99 : is acquired and the data is copied over from the parent pool idx.
100 :
101 : Currently, there is a delta-based field, fd_stake_delegations_t.
102 : Each bank stores a delta-based representation in the form of an
103 : aligned uchar buffer. The full state is stored in fd_banks_t also as
104 : a uchar buffer which corresponds to the full state of stake
105 : delegations for the current root. fd_banks_t also reserves another
106 : buffer which can store the full state of the stake delegations.
107 :
108 : fd_bank_t also holds all of the rw-locks for the fields that have
109 : rw-locks.
110 :
111 : So, when a bank is cloned from a parent, the non CoW fields are copied
112 : over and the CoW fields just copy over a pool index. The CoW behavior
113 : is completely abstracted away from the caller as callers have to
114 : query/modify fields using specific APIs.
115 :
116 : The memory for the banks is based off of two bounds:
117 : 1. the max number of unrooted blocks at any given time. Most fields
118 : can be bounded by this value.
119 : 2. the max number of forks that execute through any 1 block. We bound
120 : fields that are only written to at the epoch boundary by
121 : the max fork width that can execute through the boundary instead of
122 : by the max number of banks. See fd_banks_footprint() for more
123 : details.
124 :
125 : NOTE: An important invariant is that if a field is CoW, then it must
126 : have a rw-lock.
127 :
128 : NOTE: Another important invariant is that if a field is limiting its
129 : fork width, then it must be CoW.
130 :
131 : The usage pattern is as follows:
132 :
133 : To create an initial bank:
134 : fd_bank_t * bank_init = fd_bank_init_bank( banks, eslot );
135 :
136 : To clone bank from parent banks:
137 : fd_bank_t * bank_clone = fd_banks_clone_from_parent( banks, eslot, parent_eslot );
138 :
139 : To publish a bank (aka update the root bank):
140 : fd_bank_t * bank_publish = fd_banks_publish( banks, eslot );
141 :
142 : To query some arbitrary bank:
143 : fd_bank_t * bank_query = fd_banks_get_bank( banks, eslot );
144 :
145 : To access fields in the bank if a field does not have a lock:
146 :
147 : fd_struct_t const * field = fd_bank_field_query( bank );
148 : OR
149 : fd_struct field = fd_bank_field_get( bank );
150 :
151 : To modify fields in the bank if a field does not have a lock:
152 :
153 : fd_struct_t * field = fd_bank_field_modify( bank );
154 : OR
155 : fd_bank_field_set( bank, value );
156 :
157 : To access fields in the bank if the field has a lock:
158 :
159 : fd_struct_t const * field = fd_bank_field_locking_query( bank );
160 : ... use field ...
161 : fd_bank_field_locking_end_query( bank );
162 :
163 : To modify fields in the bank if the field has a lock:
164 :
165 : fd_struct_t * field = fd_bank_field_locking_modify( bank );
166 : ... use field ...
167 : fd_bank_field_locking_end_locking_modify( bank );
168 :
169 : IMPORTANT SAFETY NOTE: fd_banks_t assumes that there is only one bank
170 : being executed against at a time. However, it is safe to call
171 : fd_banks_publish while threads are executing against a bank.
172 :
173 : */
174 :
175 : /* Define additional fields to the bank struct here. If trying to add
176 : a CoW field to the bank, define a pool for it as done below. */
177 :
178 : #define FD_BANKS_ITER(X) \
179 : /* type, name, footprint, align, CoW, limit fork width, has lock */ \
180 78 : X(fd_blockhashes_t, block_hash_queue, sizeof(fd_blockhashes_t), alignof(fd_blockhashes_t), 0, 0, 0 ) /* Block hash queue */ \
181 201 : X(fd_fee_rate_governor_t, fee_rate_governor, sizeof(fd_fee_rate_governor_t), alignof(fd_fee_rate_governor_t), 0, 0, 0 ) /* Fee rate governor */ \
182 201 : X(ulong, capitalization, sizeof(ulong), alignof(ulong), 0, 0, 0 ) /* Capitalization */ \
183 201 : X(ulong, lamports_per_signature, sizeof(ulong), alignof(ulong), 0, 0, 0 ) /* Lamports per signature */ \
184 201 : X(ulong, prev_lamports_per_signature, sizeof(ulong), alignof(ulong), 0, 0, 0 ) /* Previous lamports per signature */ \
185 201 : X(ulong, transaction_count, sizeof(ulong), alignof(ulong), 0, 0, 0 ) /* Transaction count */ \
186 201 : X(ulong, parent_signature_cnt, sizeof(ulong), alignof(ulong), 0, 0, 0 ) /* Parent signature count */ \
187 201 : X(ulong, tick_height, sizeof(ulong), alignof(ulong), 0, 0, 0 ) /* Tick height */ \
188 201 : X(ulong, max_tick_height, sizeof(ulong), alignof(ulong), 0, 0, 0 ) /* Max tick height */ \
189 201 : X(ulong, hashes_per_tick, sizeof(ulong), alignof(ulong), 0, 0, 0 ) /* Hashes per tick */ \
190 201 : X(uint128, ns_per_slot, sizeof(uint128), alignof(uint128), 0, 0, 0 ) /* NS per slot */ \
191 201 : X(ulong, ticks_per_slot, sizeof(ulong), alignof(ulong), 0, 0, 0 ) /* Ticks per slot */ \
192 201 : X(ulong, genesis_creation_time, sizeof(ulong), alignof(ulong), 0, 0, 0 ) /* Genesis creation time */ \
193 201 : X(double, slots_per_year, sizeof(double), alignof(double), 0, 0, 0 ) /* Slots per year */ \
194 201 : X(fd_inflation_t, inflation, sizeof(fd_inflation_t), alignof(fd_inflation_t), 0, 0, 0 ) /* Inflation */ \
195 201 : X(ulong, total_epoch_stake, sizeof(ulong), alignof(ulong), 0, 0, 0 ) /* Total epoch stake */ \
196 201 : /* This is only used for the get_epoch_stake syscall. */ \
197 201 : /* If we are executing in epoch E, this is the total */ \
198 201 : /* stake at the end of epoch E-1. */ \
199 201 : X(ulong, block_height, sizeof(ulong), alignof(ulong), 0, 0, 0 ) /* Block height */ \
200 201 : X(ulong, execution_fees, sizeof(ulong), alignof(ulong), 0, 0, 0 ) /* Execution fees */ \
201 201 : X(ulong, priority_fees, sizeof(ulong), alignof(ulong), 0, 0, 0 ) /* Priority fees */ \
202 201 : X(ulong, signature_count, sizeof(ulong), alignof(ulong), 0, 0, 0 ) /* Signature count */ \
203 201 : X(fd_hash_t, poh, sizeof(fd_hash_t), alignof(fd_hash_t), 0, 0, 0 ) /* PoH */ \
204 201 : X(fd_sol_sysvar_last_restart_slot_t, last_restart_slot, sizeof(fd_sol_sysvar_last_restart_slot_t), alignof(fd_sol_sysvar_last_restart_slot_t), 0, 0, 0 ) /* Last restart slot */ \
205 201 : X(fd_cluster_version_t, cluster_version, sizeof(fd_cluster_version_t), alignof(fd_cluster_version_t), 0, 0, 0 ) /* Cluster version */ \
206 201 : X(fd_hash_t, bank_hash, sizeof(fd_hash_t), alignof(fd_hash_t), 0, 0, 0 ) /* Bank hash */ \
207 201 : X(fd_hash_t, prev_bank_hash, sizeof(fd_hash_t), alignof(fd_hash_t), 0, 0, 0 ) /* Previous bank hash */ \
208 201 : X(ulong, latest_fec_ix_observed, sizeof(ulong), alignof(ulong), 0, 0, 0 ) /* Latest FEC observed */ \
209 201 : X(fd_hash_t, genesis_hash, sizeof(fd_hash_t), alignof(fd_hash_t), 0, 0, 0 ) /* Genesis hash */ \
210 201 : X(fd_epoch_schedule_t, epoch_schedule, sizeof(fd_epoch_schedule_t), alignof(fd_epoch_schedule_t), 0, 0, 0 ) /* Epoch schedule */ \
211 201 : X(fd_rent_t, rent, sizeof(fd_rent_t), alignof(fd_rent_t), 0, 0, 0 ) /* Rent */ \
212 201 : X(fd_lthash_value_t, lthash, sizeof(fd_lthash_value_t), alignof(fd_lthash_value_t), 0, 0, 1 ) /* LTHash */ \
213 201 : X(fd_sysvar_cache_t, sysvar_cache, sizeof(fd_sysvar_cache_t), alignof(fd_sysvar_cache_t), 0, 0, 0 ) /* Sysvar cache */ \
214 201 : X(fd_epoch_rewards_t, epoch_rewards, FD_EPOCH_REWARDS_FOOTPRINT, FD_EPOCH_REWARDS_ALIGN, 1, 1, 1 ) /* Epoch rewards */ \
215 78 : X(fd_cost_tracker_t, cost_tracker, FD_COST_TRACKER_FOOTPRINT, FD_COST_TRACKER_ALIGN, 0, 0, 1 ) /* Cost tracker */ \
216 201 : X(fd_epoch_leaders_t, epoch_leaders, FD_EPOCH_LEADERS_MAX_FOOTPRINT, FD_EPOCH_LEADERS_ALIGN, 1, 1, 1 ) /* Epoch leaders. If our system supports 100k vote accs, */ \
217 42 : /* then there can be 100k unique leaders in the worst */ \
218 42 : /* case. We also can assume 432k slots per epoch. */ \
219 78 : X(fd_features_t, features, sizeof(fd_features_t), alignof(fd_features_t), 0, 0, 0 ) /* Features */ \
220 78 : X(ulong, txn_count, sizeof(ulong), alignof(ulong), 0, 0, 0 ) /* Transaction count */ \
221 78 : X(ulong, nonvote_txn_count, sizeof(ulong), alignof(ulong), 0, 0, 0 ) /* Nonvote transaction count */ \
222 78 : X(ulong, failed_txn_count, sizeof(ulong), alignof(ulong), 0, 0, 0 ) /* Failed transaction count */ \
223 78 : X(ulong, nonvote_failed_txn_count, sizeof(ulong), alignof(ulong), 0, 0, 0 ) /* Nonvote failed transaction count */ \
224 78 : X(ulong, total_compute_units_used, sizeof(ulong), alignof(ulong), 0, 0, 0 ) /* Total compute units used */ \
225 78 : X(ulong, slots_per_epoch, sizeof(ulong), alignof(ulong), 0, 0, 0 ) /* Slots per epoch */ \
226 78 : X(ulong, shred_cnt, sizeof(ulong), alignof(ulong), 0, 0, 0 ) /* Shred count */ \
227 78 : X(ulong, epoch, sizeof(ulong), alignof(ulong), 0, 0, 0 ) /* Epoch */ \
228 78 : X(int, has_identity_vote, sizeof(int), alignof(int), 0, 0, 0 ) /* Has identity vote */ \
229 201 : X(fd_vote_states_t, vote_states, FD_VOTE_STATES_FOOTPRINT, FD_VOTE_STATES_ALIGN, 1, 0, 1 ) /* Vote states for all vote accounts as of epoch E if */ \
230 42 : /* epoch E is the one that is currently being executed */ \
231 201 : X(fd_vote_states_t, vote_states_prev, FD_VOTE_STATES_FOOTPRINT, FD_VOTE_STATES_ALIGN, 1, 1, 1 ) /* Vote states for all vote accounts as of of the end of */ \
232 42 : /* epoch E-1 if epoch E is currently being executed */ \
233 201 : X(fd_vote_states_t, vote_states_prev_prev, FD_VOTE_STATES_FOOTPRINT, FD_VOTE_STATES_ALIGN, 1, 1, 1 ) /* Vote states for all vote accounts as of the end of */ \
234 : /* epoch E-2 if epoch E is currently being executed */
235 :
236 : /* Invariant Every CoW field must have a rw-lock */
237 : #define X(type, name, footprint, align, cow, limit_fork_width, has_lock) \
238 : FD_STATIC_ASSERT( (cow == 1 && has_lock == 1 ) || (cow == 0 ), CoW fields must have a rw-lock ); \
239 : FD_STATIC_ASSERT( (cow == 1 && limit_fork_width == 1) || (limit_fork_width == 0), CoW must be 1 if limit_fork_width is 1 );
240 : FD_BANKS_ITER(X)
241 : #undef X
242 :
243 : /* If a member of the bank is CoW then it needs a corresponding pool
244 : which is defined here. If a type if not a CoW then it does not need
245 : to be in a pool and is laid out contigiously in the bank struct. */
246 :
247 : /* Declare a pool object wrapper for all CoW fields. */
248 : #define HAS_COW_1(name, footprint, align) \
249 : static const ulong fd_bank_##name##_align = align; \
250 : static const ulong fd_bank_##name##_footprint = footprint; \
251 : \
252 : struct fd_bank_##name { \
253 : ulong next; \
254 : uchar data[footprint]__attribute__((aligned(align))); \
255 : }; \
256 : typedef struct fd_bank_##name fd_bank_##name##_t;
257 :
258 : /* Do nothing if CoW is not enabled. */
259 : #define HAS_COW_0(name, footprint, align)
260 :
261 : #define X(type, name, footprint, align, cow, limit_fork_width, has_lock) \
262 : HAS_COW_##cow(name, footprint, align)
263 : FD_BANKS_ITER(X)
264 :
265 : #undef X
266 : #undef HAS_COW_0
267 : #undef HAS_COW_1
268 :
269 : #define POOL_NAME fd_bank_epoch_leaders_pool
270 150 : #define POOL_T fd_bank_epoch_leaders_t
271 : #include "../../util/tmpl/fd_pool.c"
272 :
273 : #define POOL_NAME fd_bank_epoch_rewards_pool
274 129 : #define POOL_T fd_bank_epoch_rewards_t
275 : #include "../../util/tmpl/fd_pool.c"
276 :
277 : #define POOL_NAME fd_bank_vote_states_pool
278 144 : #define POOL_T fd_bank_vote_states_t
279 : #include "../../util/tmpl/fd_pool.c"
280 :
281 : #define POOL_NAME fd_bank_vote_states_prev_pool
282 159 : #define POOL_T fd_bank_vote_states_prev_t
283 : #include "../../util/tmpl/fd_pool.c"
284 :
285 : #define POOL_NAME fd_bank_vote_states_prev_prev_pool
286 129 : #define POOL_T fd_bank_vote_states_prev_prev_t
287 : #include "../../util/tmpl/fd_pool.c"
288 :
289 75 : #define FD_BANK_FLAGS_INIT (0x00000000UL) /* Initialized and replayable. */
290 0 : #define FD_BANK_FLAGS_FROZEN (0x00000001UL) /* Frozen, either because we finished replaying it, or because it was a
291 : snapshot loaded bank. */
292 51 : #define FD_BANK_FLAGS_DEAD (0x00000002UL) /* Dead, meaning we stopped replaying it before we could finish it,
293 : because for example it exceeded the block CU limit, or we decided it
294 : was on a minority fork. */
295 189 : #define FD_BANK_FLAGS_ROOTED (0x00000004UL) /* Rooted because tower said so. */
296 0 : #define FD_BANK_FLAGS_EXEC_RECORDING (0x00000100UL) /* Enable execution recording. */
297 :
298 : /* As mentioned above, the overall layout of the bank struct:
299 : - Fields used for internal pool/bank management
300 : - Non-Cow fields
301 : - CoW fields
302 : - Locks for CoW fields
303 :
304 : The CoW fields are laid out contiguously in the bank struct.
305 : The locks for the CoW fields are laid out contiguously after the
306 : CoW fields.
307 :
308 : (r) Field is owned by the replay tile, and should be updated only by
309 : the replay tile.
310 : */
311 :
312 : struct fd_bank {
313 198 : #define FD_BANK_HEADER_SIZE (offsetof(fd_bank_t, refcnt) + sizeof(ulong))
314 :
315 : /* Fields used for internal pool and bank management */
316 : fd_eslot_t eslot_; /* slot and slot counter that the bank is tracking */
317 : fd_eslot_t parent_eslot_; /* parent slot and slot counter that the bank is tracking */
318 : ulong next; /* reserved for internal use by fd_pool_para, fd_map_chain_para and fd_banks_publish */
319 : ulong parent_idx; /* index of the parent in the node pool */
320 : ulong child_idx; /* index of the left-child in the node pool */
321 : ulong sibling_idx; /* index of the right-sibling in the node pool */
322 : ulong flags; /* (r) keeps track of the state of the bank, as well as some configurations */
323 : ulong refcnt; /* (r) reference count on the bank, see replay for more details */
324 :
325 : /* First, layout all non-CoW fields contiguously. This is done to
326 : allow for cloning the bank state with a simple memcpy. Each
327 : non-CoW field is just represented as a byte array. */
328 :
329 : #define HAS_COW_1(type, name, footprint, align)
330 :
331 : #define HAS_COW_0(type, name, footprint, align) \
332 : uchar name[footprint] __attribute__((aligned(align)));
333 :
334 : #define X(type, name, footprint, align, cow, limit_fork_width, has_lock) \
335 : HAS_COW_##cow(type, name, footprint, align)
336 : FD_BANKS_ITER(X)
337 : #undef X
338 : #undef HAS_COW_0
339 : #undef HAS_COW_1
340 :
341 : /* Now, layout all information needed for CoW fields. These are only
342 : copied when explicitly requested by the caller. The field's data
343 : is located at teh pool idx in the pool. If the dirty flag has been
344 : set, then the element has been copied over for this bank. */
345 :
346 : #define HAS_COW_1(type, name, footprint, align) \
347 : int name##_dirty; \
348 : ulong name##_pool_idx; \
349 : ulong name##_pool_offset;
350 :
351 : #define HAS_COW_0(type, name, footprint, align)
352 :
353 : #define X(type, name, footprint, align, cow, limit_fork_width, has_lock) \
354 : HAS_COW_##cow(type, name, footprint, align)
355 : FD_BANKS_ITER(X)
356 : #undef X
357 : #undef HAS_COW_0
358 : #undef HAS_COW_1
359 :
360 : /* Now emit locks for all fields that need a rwlock. */
361 :
362 : #define HAS_LOCK_1(type, name, footprint, align) \
363 : fd_rwlock_t name##_lock;
364 :
365 : #define HAS_LOCK_0(type, name, footprint, align) /* Do nothing for these. */
366 :
367 : #define X(type, name, footprint, align, cow, limit_fork_width, has_lock) \
368 : HAS_LOCK_##has_lock(type, name, footprint, align)
369 : FD_BANKS_ITER(X)
370 : #undef X
371 : #undef HAS_LOCK_0
372 : #undef HAS_LOCK_1
373 :
374 : /* Stake delegations delta. */
375 :
376 : uchar stake_delegations_delta[FD_STAKE_DELEGATIONS_DELTA_FOOTPRINT] __attribute__((aligned(FD_STAKE_DELEGATIONS_ALIGN)));
377 : int stake_delegations_delta_dirty;
378 : fd_rwlock_t stake_delegations_delta_lock;
379 : };
380 : typedef struct fd_bank fd_bank_t;
381 :
382 : #define HAS_COW_1(type, name, footprint, align) \
383 : static inline void \
384 375 : fd_bank_set_##name##_pool( fd_bank_t * bank, fd_bank_##name##_t * bank_pool ) { \
385 375 : void * bank_pool_mem = fd_bank_##name##_pool_leave( bank_pool ); \
386 375 : if( FD_UNLIKELY( !bank_pool_mem ) ) { \
387 0 : FD_LOG_CRIT(( "Failed to leave bank pool" )); \
388 0 : } \
389 375 : bank->name##_pool_offset = (ulong)bank_pool_mem - (ulong)bank; \
390 375 : } \
391 : static inline fd_bank_##name##_t * \
392 78 : fd_bank_get_##name##_pool( fd_bank_t * bank ) { \
393 78 : return fd_bank_##name##_pool_join( (uchar *)bank + bank->name##_pool_offset ); \
394 78 : }
395 : #define HAS_COW_0(type, name, footprint, align) /* Do nothing for these. */
396 :
397 : #define X(type, name, footprint, align, cow, limit_fork_width, has_lock) \
398 : HAS_COW_##cow(type, name, footprint, align)
399 : FD_BANKS_ITER(X)
400 : #undef X
401 : #undef HAS_COW_0
402 : #undef HAS_COW_1
403 :
404 : /* fd_bank_t is the alignment for the bank state. */
405 :
406 : ulong
407 : fd_bank_align( void );
408 :
409 : /* fd_bank_t is the footprint for the bank state. This does NOT
410 : include the footprint for the CoW state. */
411 :
412 : ulong
413 : fd_bank_footprint( void );
414 :
415 : /**********************************************************************/
416 : /* fd_banks_t is the main struct used to manage the bank state. It can
417 : be used to query/modify/clone/publish the bank state.
418 :
419 : fd_banks_t contains some metadata a map/pool pair to manage the
420 : banks. It also contains pointers to the CoW pools.
421 :
422 : The data is laid out contiguously in memory starting from fd_banks_t;
423 : this can be seen in fd_banks_footprint(). */
424 :
425 : #define POOL_NAME fd_banks_pool
426 321 : #define POOL_T fd_bank_t
427 : #include "../../util/tmpl/fd_pool.c"
428 :
429 : #define MAP_NAME fd_banks_map
430 : #define MAP_ELE_T fd_bank_t
431 : #define MAP_KEY_T fd_eslot_t
432 84 : #define MAP_KEY eslot_
433 270 : #define MAP_KEY_EQ(k0,k1) (k0->id==k1->id)
434 417 : #define MAP_KEY_HASH(key,seed) (fd_ulong_hash( key->id ^ seed ))
435 : #include "../../util/tmpl/fd_map_chain.c"
436 :
437 : struct fd_banks {
438 : ulong magic; /* ==FD_BANKS_MAGIC */
439 : ulong max_total_banks; /* Maximum number of banks */
440 : ulong max_fork_width; /* Maximum fork width executing through
441 : any given slot. */
442 : ulong root_idx; /* root idx */
443 :
444 : /* This lock is only used to serialize banks fork tree reads with
445 : respect to fork tree writes. In other words, tree traversals
446 : cannot happen at the same time as a tree pruning operation or a
447 : tree insertion operation. So the public APIs on banks take either
448 : a read lock or a write lock depending on what they do on the fork
449 : tree. For example, publishing takes a write lock, and bank lookups
450 : take a read lock. Notably, individual banks can still be
451 : concurrently accessed or modified, and this lock does not offer
452 : synchronization on individual fields within a bank. */
453 : fd_rwlock_t rwlock;
454 :
455 : ulong pool_offset; /* offset of pool from banks */
456 : ulong map_offset; /* offset of map from banks */
457 :
458 : /* stake_delegations_root will be the full state of stake delegations
459 : for the current root. It can get updated in two ways:
460 : 1. On boot the snapshot will be directly read into the rooted
461 : stake delegations because we assume that any and all snapshots
462 : are a rooted slot.
463 : 2. Calls to fd_banks_publish() will apply all of the stake
464 : delegation deltas from each of the banks that are about to be
465 : published. */
466 :
467 : uchar stake_delegations_root[FD_STAKE_DELEGATIONS_FOOTPRINT] __attribute__((aligned(FD_STAKE_DELEGATIONS_ALIGN)));
468 :
469 : /* stake_delegations_frontier is reserved memory that can represent
470 : the full state of stake delegations for the current frontier. This
471 : is done by taking the stake_delegations_root and applying all of
472 : the deltas from the current bank and all of its ancestors up to the
473 : root bank. */
474 :
475 : uchar stake_delegations_frontier[FD_STAKE_DELEGATIONS_FOOTPRINT] __attribute__((aligned(FD_STAKE_DELEGATIONS_ALIGN)));
476 :
477 : /* Layout all CoW pools. */
478 :
479 : #define HAS_COW_1(type, name, footprint, align) \
480 : ulong name##_pool_offset; /* offset of pool from banks */
481 :
482 : #define HAS_COW_0(type, name, footprint, align) /* Do nothing for these. */
483 :
484 : #define X(type, name, footprint, align, cow, limit_fork_width, has_lock) \
485 : HAS_COW_##cow(type, name, footprint, align)
486 : FD_BANKS_ITER(X)
487 : #undef X
488 : #undef HAS_COW_0
489 : #undef HAS_COW_1
490 : };
491 : typedef struct fd_banks fd_banks_t;
492 :
493 : /* Bank accesssors. Different accessors are emitted for different types
494 : depending on if the field has a lock or not. */
495 :
496 : #define HAS_LOCK_1(type, name) \
497 : type const * fd_bank_##name##_locking_query( fd_bank_t * bank ); \
498 : void fd_bank_##name##_end_locking_query( fd_bank_t * bank ); \
499 : type * fd_bank_##name##_locking_modify( fd_bank_t * bank ); \
500 : void fd_bank_##name##_end_locking_modify( fd_bank_t * bank );
501 :
502 : #define HAS_LOCK_0(type, name) \
503 : type const * fd_bank_##name##_query( fd_bank_t const * bank ); \
504 : type * fd_bank_##name##_modify( fd_bank_t * bank );
505 :
506 : #define X(type, name, footprint, align, cow, limit_fork_width, has_lock) \
507 : void fd_bank_##name##_set( fd_bank_t * bank, type value ); \
508 : type fd_bank_##name##_get( fd_bank_t const * bank ); \
509 : HAS_LOCK_##has_lock(type, name)
510 : FD_BANKS_ITER(X)
511 : #undef X
512 :
513 : #undef HAS_LOCK_0
514 : #undef HAS_LOCK_1
515 :
516 : /* fd_bank_slot_get() returns the slot of a given bank. */
517 :
518 : static inline ulong
519 669 : fd_bank_slot_get( fd_bank_t const * bank ) {
520 669 : return bank->eslot_.slot;
521 669 : }
522 :
523 : /* fd_bank_prime_get() returns the prime count of a given bank. */
524 :
525 : static inline uint
526 0 : fd_bank_prime_get( fd_bank_t const * bank ) {
527 0 : return bank->eslot_.prime;
528 0 : }
529 :
530 : /* fd_bank_eslot_get() returns the eslot of a given bank. */
531 :
532 : static inline fd_eslot_t
533 138 : fd_bank_eslot_get( fd_bank_t const * bank ) {
534 138 : return bank->eslot_;
535 138 : }
536 :
537 : static inline ulong
538 0 : fd_bank_parent_slot_get( fd_bank_t const * bank ) {
539 0 : return bank->parent_eslot_.slot;
540 0 : }
541 :
542 : static inline uint
543 0 : fd_bank_parent_prime_get( fd_bank_t const * bank ) {
544 0 : return bank->parent_eslot_.prime;
545 0 : }
546 :
547 : static inline fd_eslot_t
548 0 : fd_bank_parent_eslot_get( fd_bank_t const * bank ) {
549 0 : return bank->parent_eslot_;
550 0 : }
551 :
552 : static inline void
553 6 : fd_bank_parent_eslot_set( fd_bank_t * bank, fd_eslot_t parent_eslot ) {
554 6 : bank->parent_eslot_ = parent_eslot;
555 6 : }
556 :
557 : /* Each bank has a fd_stake_delegations_t object which is delta-based.
558 : The usage pattern is the same as other bank fields:
559 : 1. fd_bank_stake_dleegations_delta_locking_modify( bank ) will return
560 : a mutable pointer to the stake delegations delta object. If the
561 : caller has not yet initialized the delta object, then it will
562 : be initialized. Because it is a delta it is not copied over from
563 : a parent bank.
564 : 2. fd_bank_stake_delegations_delta_locking_query( bank ) will return
565 : a const pointer to the stake delegations delta object. If the
566 : delta object has not been initialized, then NULL is returned.
567 : 3. fd_bank_stake_delegations_delta_locking_end_modify( bank ) will
568 : release the write lock on the object.
569 : 4. fd_bank_stake_delegations_delta_locking_end_query( bank ) will
570 : release a read lock on the object.
571 : */
572 :
573 : static inline fd_stake_delegations_t *
574 15 : fd_bank_stake_delegations_delta_locking_modify( fd_bank_t * bank ) {
575 15 : fd_rwlock_write( &bank->stake_delegations_delta_lock );
576 15 : if( !bank->stake_delegations_delta_dirty ) {
577 15 : bank->stake_delegations_delta_dirty = 1;
578 15 : fd_stake_delegations_new( bank->stake_delegations_delta, FD_STAKE_DELEGATIONS_MAX_PER_SLOT, 1 );
579 15 : }
580 15 : return fd_stake_delegations_join( bank->stake_delegations_delta );
581 15 : }
582 :
583 : static inline void
584 15 : fd_bank_stake_delegations_delta_end_locking_modify( fd_bank_t * bank ) {
585 15 : fd_rwlock_unwrite( &bank->stake_delegations_delta_lock );
586 15 : }
587 :
588 : static inline fd_stake_delegations_t *
589 0 : fd_bank_stake_delegations_delta_locking_query( fd_bank_t * bank ) {
590 0 : fd_rwlock_read( &bank->stake_delegations_delta_lock );
591 0 : return bank->stake_delegations_delta_dirty ? fd_stake_delegations_join( bank->stake_delegations_delta ) : NULL;
592 0 : }
593 :
594 : static inline void
595 0 : fd_bank_stake_delegations_delta_end_locking_query( fd_bank_t * bank ) {
596 0 : fd_rwlock_unread( &bank->stake_delegations_delta_lock );
597 0 : }
598 :
599 : /* fd_bank_stake_delegations_frontier_query() will return a pointer to
600 : the full stake delegations for the current frontier. The caller is
601 : responsible that there are no concurrent readers or writers to
602 : the stake delegations returned by this function.
603 :
604 : Under the hood, the function copies the rooted stake delegations and
605 : applies all of the deltas for the direct ancestry from the current
606 : bank up to the rooted bank to the copy. */
607 :
608 : fd_stake_delegations_t *
609 : fd_bank_stake_delegations_frontier_query( fd_banks_t * banks,
610 : fd_bank_t * bank );
611 :
612 : /* fd_banks_stake_delegations_root_query() will return a pointer to the
613 : full stake delegations for the current root. This function should
614 : only be called on boot. */
615 :
616 : fd_stake_delegations_t *
617 : fd_banks_stake_delegations_root_query( fd_banks_t * banks );
618 :
619 : /* Simple getters and setters for the various maps and pools in
620 : fd_banks_t. Notably, the map/pool pairs for the fd_bank_t structs as
621 : well as all of the CoW structs in the banks. */
622 :
623 : static inline fd_bank_t *
624 291 : fd_banks_get_bank_pool( fd_banks_t const * banks ) {
625 291 : return fd_banks_pool_join( ((uchar *)banks + banks->pool_offset) );
626 291 : }
627 :
628 : static inline fd_banks_map_t *
629 225 : fd_banks_get_bank_map( fd_banks_t const * banks ) {
630 225 : return fd_banks_map_join( ((uchar *)banks + banks->map_offset) );
631 225 : }
632 :
633 : static inline void
634 : fd_banks_set_bank_pool( fd_banks_t * banks,
635 9 : fd_bank_t * bank_pool ) {
636 9 : void * bank_pool_mem = fd_banks_pool_leave( bank_pool );
637 9 : if( FD_UNLIKELY( !bank_pool_mem ) ) {
638 0 : FD_LOG_CRIT(( "Failed to leave bank pool" ));
639 0 : }
640 9 : banks->pool_offset = (ulong)bank_pool_mem - (ulong)banks;
641 9 : }
642 :
643 : static inline void
644 : fd_banks_set_bank_map( fd_banks_t * banks,
645 9 : fd_banks_map_t * bank_map ) {
646 9 : void * bank_map_mem = fd_banks_map_leave( bank_map );
647 9 : if( FD_UNLIKELY( !bank_map_mem ) ) {
648 0 : FD_LOG_CRIT(( "Failed to leave bank map" ));
649 0 : }
650 9 : banks->map_offset = (ulong)bank_map_mem - (ulong)banks;
651 9 : }
652 :
653 : #define HAS_COW_1(type, name, footprint, align) \
654 : static inline fd_bank_##name##_t * \
655 483 : fd_banks_get_##name##_pool( fd_banks_t * banks ) { \
656 483 : return fd_bank_##name##_pool_join( (uchar *)banks + banks->name##_pool_offset ); \
657 483 : } \
658 : static inline void \
659 45 : fd_banks_set_##name##_pool( fd_banks_t * banks, fd_bank_##name##_t * bank_pool ) { \
660 45 : void * bank_pool_mem = fd_bank_##name##_pool_leave( bank_pool ); \
661 45 : if( FD_UNLIKELY( !bank_pool_mem ) ) { \
662 0 : FD_LOG_CRIT(( "Failed to leave bank pool" )); \
663 0 : } \
664 45 : banks->name##_pool_offset = (ulong)bank_pool_mem - (ulong)banks; \
665 45 : }
666 :
667 : #define HAS_COW_0(type, name, footprint, align) /* Do nothing for these. */
668 :
669 : #define X(type, name, footprint, align, cow, limit_fork_width, has_lock) \
670 : HAS_COW_##cow(type, name, footprint, align)
671 : FD_BANKS_ITER(X)
672 : #undef X
673 : #undef HAS_COW_0
674 : #undef HAS_COW_1
675 :
676 :
677 : /* fd_banks_root() and fd_banks_root_const() returns a non-const and
678 : const pointer to the root bank respectively. */
679 :
680 : FD_FN_PURE static inline fd_bank_t const *
681 0 : fd_banks_root_const( fd_banks_t const * banks ) {
682 0 : return fd_banks_pool_ele_const( fd_banks_get_bank_pool( banks ), banks->root_idx );
683 0 : }
684 :
685 : FD_FN_PURE static inline fd_bank_t *
686 27 : fd_banks_root( fd_banks_t * banks ) {
687 27 : return fd_banks_pool_ele( fd_banks_get_bank_pool( banks ), banks->root_idx );
688 27 : }
689 :
690 : /* fd_banks_align() returns the alignment of fd_banks_t */
691 :
692 : ulong
693 : fd_banks_align( void );
694 :
695 : /* fd_banks_footprint() returns the footprint of fd_banks_t. This
696 : includes the struct itself but also the footprint for all of the
697 : pools.
698 :
699 : The footprint of fd_banks_t is determined by the total number
700 : of banks that the bank manages. This is an analog for the max number
701 : of unrooted blocks the bank can manage at any given time.
702 :
703 : We can also further bound the memory footprint of the banks by the
704 : max width of forks that can exist at any given time. The reason for
705 : this is that there are several large CoW structs that are only
706 : written to during the epoch boundary (e.g. epoch_rewards,
707 : epoch_stakes, etc.). These structs are read-only afterwards. This
708 : means if we also bound the max number of forks that can execute
709 : through the epoch boundary, we can bound the memory footprint of
710 : the banks. */
711 :
712 : ulong
713 : fd_banks_footprint( ulong max_total_banks,
714 : ulong max_fork_width );
715 :
716 : /* fd_banks_new() creates a new fd_banks_t struct. This function lays
717 : out the memory for all of the constituent fd_bank_t structs and
718 : pools depending on the max_total_banks and the max_fork_width for a
719 : given block. */
720 :
721 : void *
722 : fd_banks_new( void * mem,
723 : ulong max_total_banks,
724 : ulong max_fork_width );
725 :
726 : /* fd_banks_join() joins a new fd_banks_t struct. */
727 :
728 : fd_banks_t *
729 : fd_banks_join( void * mem );
730 :
731 : /* fd_banks_leave() leaves a bank. */
732 :
733 : void *
734 : fd_banks_leave( fd_banks_t * banks );
735 :
736 : /* fd_banks_delete() deletes a bank. */
737 :
738 : void *
739 : fd_banks_delete( void * shmem );
740 :
741 : /* fd_banks_init_bank() initializes a new bank in the bank manager.
742 : This should only be used during bootup. This returns an initial
743 : fd_bank_t with the corresponding eslot.. */
744 :
745 : fd_bank_t *
746 : fd_banks_init_bank( fd_banks_t * banks,
747 : fd_eslot_t eslot );
748 :
749 : /* fd_banks_get_bank() returns a bank for a given eslot. If said eslot
750 : does not exist, NULL is returned.
751 :
752 : The returned pointer is valid so long as the underlying bank does not
753 : get pruned by a publishing operation. Higher level components are
754 : responsible for ensuring that publishing does not happen while a bank
755 : is being accessed. This is done through the reference counter. */
756 :
757 : fd_bank_t *
758 : fd_banks_get_bank( fd_banks_t * banks,
759 : fd_eslot_t eslot );
760 :
761 : /* fd_banks_get_bank_idx returns a bank for a given index into the pool
762 : of banks. This function otherwise has the same behavior as
763 : fd_banks_get_bank(). */
764 :
765 : static inline fd_bank_t *
766 : fd_banks_get_bank_idx( fd_banks_t * banks,
767 0 : ulong idx ) {
768 0 : return fd_banks_pool_ele( fd_banks_get_bank_pool( banks ), idx );
769 0 : }
770 :
771 : /* fd_banks_get_pool_idx returns the index of a bank in the pool. */
772 :
773 : static inline ulong
774 : fd_banks_get_pool_idx( fd_banks_t * banks,
775 0 : fd_bank_t * bank ) {
776 0 : return fd_banks_pool_idx( fd_banks_get_bank_pool( banks ), bank );
777 0 : }
778 :
779 : static inline fd_bank_t *
780 : fd_banks_get_parent( fd_banks_t * banks,
781 0 : fd_bank_t * bank ) {
782 0 : return fd_banks_pool_ele( fd_banks_get_bank_pool( banks ), bank->parent_idx );
783 0 : }
784 :
785 : /* fd_banks_clone_from_parent() clones a bank from a parent bank.
786 : If the bank corresponding to the parent eslot does not exist,
787 : NULL is returned. If a bank is not able to be created, NULL is
788 : returned. The data from the parent bank will copied over into
789 : the new bank.
790 :
791 : A more detailed note: not all of the data is copied over and this
792 : is a shallow clone. All of the CoW fields are not copied over and
793 : will only be done so if the caller explicitly calls
794 : fd_bank_{*}_modify(). This naming was chosen to emulate the
795 : semantics of the Agave client. */
796 :
797 : fd_bank_t *
798 : fd_banks_clone_from_parent( fd_banks_t * banks,
799 : fd_eslot_t eslot,
800 : fd_eslot_t parent_eslot );
801 :
802 : /* fd_banks_publish() publishes a bank to the bank manager. This
803 : should only be used when a bank is no longer needed. This will
804 : prune off the bank from the bank manager. It returns the new root
805 : bank.
806 :
807 : All banks that are ancestors or siblings of the new root bank will be
808 : cancelled and their resources will be released back to the pool. */
809 :
810 : fd_bank_t const *
811 : fd_banks_publish( fd_banks_t * banks,
812 : fd_eslot_t eslot );
813 :
814 : /* fd_bank_clear_bank() clears the contents of a bank. This should ONLY
815 : be used with banks that have no children.
816 :
817 : This function will memset all non-CoW fields to 0.
818 :
819 : For all CoW fields, we will reset the indices to its parent. */
820 :
821 : void
822 : fd_banks_clear_bank( fd_banks_t * banks,
823 : fd_bank_t * bank );
824 :
825 : /* fd_banks_publish_prepare returns the highest block that can be safely
826 : published between the current published root of the fork tree and the
827 : target block. See the note on safe publishing for more details. In
828 : general, a node in the fork tree can be pruned if:
829 : (1) the node itself can be pruned, and
830 : (2) all subtrees (except for the one on the rooted fork) forking off
831 : of the node can be pruned.
832 : The highest publishable block is the highest block on the rooted fork
833 : where the above is true, or the rooted child block of such if there
834 : is one.
835 :
836 : This function assumes that the given target block has been rooted by
837 : consensus. It will mark every block on the rooted fork as rooted, up
838 : to the given target block. It will also mark minority forks as dead.
839 :
840 : Highest publishable block is written to the out pointer. Returns 1
841 : if the publishable block can be advanced beyond the current root.
842 : Returns 0 if no such block can be found. */
843 :
844 : int
845 : fd_banks_publish_prepare( fd_banks_t * banks,
846 : fd_eslot_t target_eslot,
847 : fd_eslot_t * publishable_eslot_out );
848 :
849 : /* fd_banks_rekey_banks updates the bank with eslot old_eslot to have a
850 : new eslot new_eslot. A bank with eslot old_eslot must be a valid
851 : bank or the program will crash. This function should NOT be called
852 : once the current bank has child banks. */
853 :
854 : fd_bank_t *
855 : fd_banks_rekey_bank( fd_banks_t * banks,
856 : fd_eslot_t old_eslot,
857 : fd_eslot_t new_eslot );
858 :
859 : /* fd_banks_mark_bank_dead marks the current bank (and all of its
860 : descendants) as dead. The caller is still responsible for handling
861 : the behavior of the dead bank correctly. */
862 :
863 : void
864 : fd_banks_mark_bank_dead( fd_banks_t * banks,
865 : fd_bank_t * bank );
866 :
867 : /* fd_banks_is_bank_dead returns 1 if the bank is dead, 0 otherwise. */
868 :
869 : static inline int
870 0 : fd_banks_is_bank_dead( fd_bank_t * bank ) {
871 0 : return bank->flags & FD_BANK_FLAGS_DEAD;
872 0 : }
873 :
874 : /* fd_banks_print pretty-prints a formatted banks tree. Printing begins
875 : from the rooted bank. The printer prints out the fork structure
876 : and the slot, eslot, and flags of each bank.
877 :
878 : Calling this function acquires a read lock on the banks struct. The
879 : caller is responsible for making sure that there are no concurrent
880 : writes to the eslot/flag fields for each bank.
881 :
882 : The usage is as follows:
883 : `fd_banks_print( banks )`. */
884 :
885 : void
886 : fd_banks_print( fd_banks_t * banks );
887 :
888 : /* fd_banks_validate does validation on the banks struct to make sure
889 : that there are no corruptions/invariant violations. It returns 0
890 : if no issues have been detected and 1 otherwise.
891 :
892 : List of checks that the function currently performs:
893 : 1. CoW fields have not acquired more elements than the max amount of
894 : allocated banks.
895 : (Add more checks as needed here)
896 : */
897 :
898 : int
899 : fd_banks_validate( fd_banks_t * banks );
900 :
901 : FD_PROTOTYPES_END
902 :
903 : #endif /* HEADER_fd_src_flamenco_runtime_fd_bank_h */
|
