Line data    Source code

       1             : #ifndef HEADER_fd_src_flamenco_stakes_fd_vote_states_h
       2             : #define HEADER_fd_src_flamenco_stakes_fd_vote_states_h
       3             : 
       4             : #include "../../util/fd_util_base.h"
       5             : #include "../../util/tmpl/fd_map.h"
       6             : #include "../types/fd_types_custom.h"
       7             : 
       8        1038 : #define FD_VOTE_STATES_MAGIC (0xF17EDA2CE7601E70UL) /* FIREDANCER VOTER V0 */
       9             : 
      10             : /* fd_vote_states_t is a cache of vote accounts mapping the pubkey of
      11             :    a vote account to various information about the vote account
      12             :    including, stake, last vote slot/timestamp, and commission for the
      13             :    vote account.  The vote states are safe to be used across multiple
      14             :    threads but concurrent reads/writes must be synchronized by the
      15             :    caller.
      16             : 
      17             :    In the runtime, there are 3 instances of fd_vote_states_t that are
      18             :    maintained and used at different points, notably around epoch reward
      19             :    and leader schedule calculations. The 3 instances are:
      20             :    1. vote_states: This is the vote states for the current epoch. This
      21             :       is updated through the course of an epoch as vote accounts are
      22             :       updated.
      23             :    2. vote_states_prev: This is the vote states as of the end of
      24             :       previous epoch E-1 if we are currently executing epoch E.
      25             :       This gets updated at the end of an epoch when vote_states are
      26             :       copied into vote_states_prev.
      27             :    3. vote_states_prev_prev: This is the vote states as of the end of
      28             :       epoch E-2 if we are currently executing epoch E. This only gets
      29             :       updated at the end of an epoch when vote_states_prev is copied
      30             :       into vote_states_prev_prev.
      31             : 
      32             :    The implementation of fd_vote_states_t is a hash map which is backed
      33             :    by a memory pool. Callers are allowed to insert, replace, and remove
      34             :    entries from the map.
      35             : 
      36             :    In practice, fd_vote_states_t are updated in 3 cases:
      37             :    1. They are initially populated from the versioned vote account
      38             :       stake accounts in the snapshot manifest. These are populated from
      39             :       the raw vote account data. This is done in a single pass over the
      40             :       vote account data.
      41             :    2. The vote states for the current epoch can be updated after
      42             :       transaction execution. This is done for vote accounts that are
      43             :       referenced during a transaction.
      44             :    3. Vote states are updated at the epoch boundary. The stake
      45             :       information for the vote states is refreshed at the boundary.
      46             : 
      47             :    The vote states in reality manage a few different sets of
      48             :    information about the vote account:
      49             :    - vote account state: state from the vote account data including
      50             :      the last vote slot/timestamp, commission, and node pubkey.  This is
      51             :      used for clock sysvar and rewards calculations.
      52             :    - stake: stake as of the end of the previous epoch.  This is used
      53             :      eventually for leader schedule calculations.  The stake from epoch
      54             :      T-2 (stake_t_2) is used for the stake in clock calculations and Tower's threshold switch checks.
      55             :      T-1 (stake_t_1)
      56             :    - rewards: this information is only used at the epoch boundary.
      57             : 
      58             :    NOTE: The leader schedule epoch is almost always equal to the current
      59             :    epoch + 1.
      60             :    TODO: highlight the cases where this is not possible.
      61             : 
      62             :    In the agave client, each bank holds its own epoch stakes (agave
      63             :    equivalent of vote_states).  These are keyed by leader schedule epoch
      64             :    (which is almost always equal to epoch + 1).  Old entries are removed
      65             :    and new entries are inserted at the epoch boundary.  When we use the
      66             :    epoch stakes for consensus and for clock timestamp calculations, we
      67             :    query the epoch stakes by the current epoch we are executing in.
      68             : 
      69             :    When we cross from Epoch E-1 to E, we compute our stakes assuming
      70             :    we are in E.  At this point, because we are in epoch E, this means
      71             :    that the leader schedule epoch is E+1 so an epoch_stakes entry for
      72             :    epoch E+1 is inserted into epoch_stakes.
      73             : 
      74             :    After we cross the boundary, in epoch E, we use the epoch stakes as
      75             :    keyed by epoch E for clock/consensus/leader schedule calculations.
      76             :    The stakes keyed by epoch E are actually the ones calculated at the
      77             :    boundary between epoch E-2 and E-1 since that's when the leader
      78             :    schedule epoch is E.  Similiarly, when we are in epoch E+1 we use the
      79             :    stakes caclulated between epoch E-1 and E since that's when the leader
      80             :    schedule epoch is E+1.
      81             : 
      82             :    The firedancer naming of vote states is instead done by reference to
      83             :    the epoch boundary that the stakes were calculated in.  If we are
      84             :    currently in epoch E, we use the vote states calculated at the end of
      85             :    two epochs ago (E-2 -> E-1), hence vote_states_prev_prev.  We store
      86             :    the vote states calculated at the end of the previous epoch
      87             :    (E-1 -> E) as vote_states_prev.  This is also how we cache the stake
      88             :    values in the current vote states which are continually updated. */
      89             : 
      90         207 : #define FD_VOTE_STATES_ALIGN (128UL)
      91             : 
      92             : /* Agave defines the max number of epoch credits to store to be 64.
      93             :    https://github.com/anza-xyz/solana-sdk/blob/vote-interface%40v2.2.6/vote-interface/src/state/mod.rs#L37 */
      94             : #define EPOCH_CREDITS_MAX (64UL)
      95             : 
      96             : struct fd_vote_state_credits {
      97             :   ulong  credits_cnt;
      98             :   ushort epoch       [ EPOCH_CREDITS_MAX ];
      99             :   ulong  credits     [ EPOCH_CREDITS_MAX ];
     100             :   ulong  prev_credits[ EPOCH_CREDITS_MAX ];
     101             : };
     102             : typedef struct fd_vote_state_credits fd_vote_state_credits_t;
     103             : 
     104             : struct fd_vote_state_ele {
     105             :   /* Internal pool/map use */
     106             :   ulong       idx;
     107             :   ulong       next_;
     108             : 
     109             :   /* Vote account stake information which is derived from the stake
     110             :      delegations.  This information is used for leader schedule
     111             :      calculation and clock stake-weighted median calculations.
     112             : 
     113             :      stake_t_2 is used in Tower, for it's threshold switch checks. */
     114             :   ulong       stake;
     115             :   ulong       stake_t_1;
     116             :   ulong       stake_t_2;
     117             : 
     118             :   /* Vote account information which is derived from the vote account
     119             :      data and is used for clock timestamp calculations. */
     120             :   fd_pubkey_t vote_account;
     121             :   fd_pubkey_t node_account;
     122             :   ulong       last_vote_slot;
     123             :   long        last_vote_timestamp;
     124             :   uchar       commission;
     125             : };
     126             : typedef struct fd_vote_state_ele fd_vote_state_ele_t;
     127             : 
     128             : /* Forward declare map iterator API generated by fd_map_chain.c */
     129             : typedef struct fd_vote_state_map_private fd_vote_state_map_t;
     130             : typedef struct fd_map_chain_iter fd_vote_state_map_iter_t;
     131             : struct fd_vote_states_iter {
     132             :   fd_vote_state_map_t *    map;
     133             :   fd_vote_state_ele_t *    pool;
     134             :   fd_vote_state_map_iter_t iter;
     135             : };
     136             : typedef struct fd_vote_states_iter fd_vote_states_iter_t;
     137             : 
     138             : struct __attribute__((aligned(FD_VOTE_STATES_ALIGN))) fd_vote_states {
     139             :   ulong magic;
     140             :   ulong max_vote_accounts_;
     141             :   ulong pool_offset_;
     142             :   ulong map_offset_;
     143             : };
     144             : typedef struct fd_vote_states fd_vote_states_t;
     145             : 
     146             : /* This guarantees that the pool element alignment is at most 128UL. */
     147             : FD_STATIC_ASSERT(alignof(fd_vote_state_ele_t)<=FD_VOTE_STATES_ALIGN, unexpected pool element alignment);
     148             : 
     149             : /* The static footprint of the vote states assumes that there are
     150             :    FD_RUNTIME_MAX_VOTE_ACCOUNTS. It also assumes worst case alignment
     151             :    for each struct. fd_vote_states_t is laid out as first the
     152             :    fd_vote_states_t struct, followed by a pool of fd_vote_state_ele_t
     153             :    structs, followed by a map of fd_vote_state_map_ele_t structs.
     154             :    The pool has FD_RUNTIME_MAX_VOTE_ACCOUNTS elements, and the map
     155             :    has a chain count deteremined by a call to
     156             :    fd_vote_states_chain_cnt_est.
     157             :    NOTE: the footprint is validated to be at least as large as the
     158             :    actual runtime-determined footprint (see test_vote_states.c) */
     159             : 
     160          69 : #define FD_VOTE_STATES_CHAIN_CNT_EST (32768UL)
     161             : #define FD_VOTE_STATES_FOOTPRINT                                                      \
     162             :   /* First, layout the struct with alignment */                                       \
     163          69 :   sizeof(fd_vote_states_t) + alignof(fd_vote_states_t) +                              \
     164          69 :   /* Now layout the pool's data footprint */                                          \
     165          69 :   FD_VOTE_STATES_ALIGN + sizeof(fd_vote_state_ele_t) * FD_RUNTIME_MAX_VOTE_ACCOUNTS + \
     166          69 :   /* Now layout the pool's meta footprint */                                          \
     167          69 :   FD_VOTE_STATES_ALIGN + 128UL /* POOL_ALIGN */ +                                     \
     168          69 :   /* Now layout the map.  We must make assumptions about the chain */                 \
     169          69 :   /* count to be equivalent to chain_cnt_est. */                                      \
     170          69 :   FD_VOTE_STATES_ALIGN + 128UL /* MAP_ALIGN */ + (FD_VOTE_STATES_CHAIN_CNT_EST * sizeof(ulong))
     171             : 
     172             : FD_PROTOTYPES_BEGIN
     173             : 
     174             : /* fd_vote_states_align returns the minimum alignment required for a
     175             :    vote states struct. */
     176             : 
     177             : FD_FN_CONST ulong
     178             : fd_vote_states_align( void );
     179             : 
     180             : /* fd_vote_states_footprint returns the footprint of the vote states
     181             :    struct for a given amount of max vote accounts. */
     182             : 
     183             : FD_FN_CONST ulong
     184             : fd_vote_states_footprint( ulong max_vote_accounts );
     185             : 
     186             : /* fd_vote_states_new creates a new vote states struct with a given
     187             :    number of max vote accounts and a seed. It formats a memory region
     188             :    which is sized based off of the number of vote accounts. */
     189             : 
     190             : void *
     191             : fd_vote_states_new( void * mem,
     192             :                     ulong  max_vote_accounts,
     193             :                     ulong  seed );
     194             : 
     195             : /* fd_vote_states_join joins a vote states struct from a
     196             :    memory region. There can be multiple valid joins for a given memory
     197             :    region but the caller is responsible for accessing memory in a
     198             :    thread-safe manner. */
     199             : 
     200             : fd_vote_states_t *
     201             : fd_vote_states_join( void * mem );
     202             : 
     203             : /* fd_vote_states_init resets the state of a valid join of a
     204             :    vote states struct. */
     205             : 
     206             : void
     207             : fd_vote_states_init( fd_vote_states_t * vote_states );
     208             : 
     209             : /* fd_vote_states_update inserts or updates the vote state corresponding
     210             :    to a given account. */
     211             : 
     212             : fd_vote_state_ele_t *
     213             : fd_vote_states_update( fd_vote_states_t *  vote_states,
     214             :                        fd_pubkey_t const * vote_account );
     215             : 
     216             : /* fd_vote_states_update_from_account inserts or updates the vote state
     217             :    corresponding to a valid vote account. This is the same as
     218             :    fd_vote_states_update but is also responsible for decoding the vote
     219             :    account data into a versioned vote state object and extracing the
     220             :    commission and credits. Kills the client if the vote state cannot
     221             :    be decoded. */
     222             : 
     223             : fd_vote_state_ele_t *
     224             : fd_vote_states_update_from_account( fd_vote_states_t *  vote_states,
     225             :                                     fd_pubkey_t const * vote_account,
     226             :                                     uchar const *       account_data,
     227             :                                     ulong               account_data_len );
     228             : 
     229             : /* fd_vote_states_reset_stakes_t resets the stakes to 0 for each of the
     230             :    vote accounts in fd_vote_states_t. */
     231             : 
     232             : void
     233             : fd_vote_states_reset_stakes( fd_vote_states_t * vote_states );
     234             : 
     235             : /* fd_vote_states_remove removes the vote state corresponding to a given
     236             :    account. Does nothing if the account does not exist. */
     237             : 
     238             : void
     239             : fd_vote_states_remove( fd_vote_states_t *  vote_states,
     240             :                        fd_pubkey_t const * vote_account );
     241             : 
     242             : /* fd_vote_states_query returns the vote state corresponding to a given
     243             :    account. Returns NULL if the account does not exist. This function is
     244             :    safe for concurrent reads, but the caller needs to synchronize
     245             :    concurrent writers to the fd_vote_state_ele_t. */
     246             : 
     247             : fd_vote_state_ele_t *
     248             : fd_vote_states_query( fd_vote_states_t const * vote_states,
     249             :                       fd_pubkey_t const *      vote_account );
     250             : 
     251             : /* fd_vote_states_query_const is the same as fd_vote_states but instead
     252             :    returns a const pointer. */
     253             : 
     254             : fd_vote_state_ele_t const *
     255             : fd_vote_states_query_const( fd_vote_states_t const * vote_states,
     256             :                             fd_pubkey_t const *      vote_account );
     257             : 
     258             : /* fd_vote_states_max returns the maximum number of vote accounts that
     259             :    the vote states struct can support. */
     260             : 
     261             : ulong
     262             : fd_vote_states_max( fd_vote_states_t const * vote_states );
     263             : 
     264             : /* fd_vote_states_cnt returns the number of vote states in the vote
     265             :    states struct. */
     266             : 
     267             : ulong
     268             : fd_vote_states_cnt( fd_vote_states_t const * vote_states );
     269             : 
     270             : /* Iterator API for vote states.  The iterator is initialized with a
     271             :    call to fd_vote_states_iter_init.  The caller is responsible for
     272             :    managing the memory for the iterator.  It is safe to call
     273             :    fd_vote_states_iter_next if the result of
     274             :    fd_vote_states_iter_done() ==0.  It is safe to call
     275             :    fd_vote_states_iter_ele() to get the current vote state. As a note,
     276             :    it is safe to modify the vote state acquired from
     277             :    fd_vote_states_iter_ele() as long as the next_ field is not modified
     278             :    (which the caller should never do).  It is unsafe to insert or remove
     279             :    fd_vote_state_ele_t from the vote states struct while iterating.
     280             : 
     281             :    Under the hood, the iterator is just a wrapper over the iterator in
     282             :    fd_map_chain.c.
     283             : 
     284             :    Example use:
     285             : 
     286             :    fd_vote_states_iter_t iter_[1];
     287             :    for( fd_vote_states_iter_t * iter = fd_vote_states_iter_init( vote_states, iter_ ); !fd_vote_states_iter_done( iter ); fd_vote_states_iter_next( iter ) ) {
     288             :      fd_vote_state_ele_t * vote_state = fd_vote_states_iter_ele( iter );
     289             :      // Do something with the vote state ...
     290             :    }
     291             : */
     292             : 
     293             : fd_vote_state_ele_t *
     294             : fd_vote_states_iter_ele( fd_vote_states_iter_t * iter );
     295             : 
     296             : fd_vote_states_iter_t *
     297             : fd_vote_states_iter_init( fd_vote_states_iter_t *  iter,
     298             :                           fd_vote_states_t const * vote_states );
     299             : 
     300             : int
     301             : fd_vote_states_iter_done( fd_vote_states_iter_t * iter );
     302             : 
     303             : void
     304             : fd_vote_states_iter_next( fd_vote_states_iter_t * iter );
     305             : 
     306             : FD_PROTOTYPES_END
     307             : 
     308             : #endif /* HEADER_fd_src_flamenco_stakes_fd_vote_states_h */