Outbound bandwidth of the leader node?

4.3. Communication Framework
In LibraBFT, the communication framework aims at sharing records with a subset of nodes. Specifically, the framework API consists of three primitive actions:
(i) send, where the node shares the records it has with a fixed number of peers;
(ii) broadcast, where the node sends (pushes) updates to every other node;
(iii) query-all where a node requests (pulls) updates from every other node.

Each validator in the Libra protocol maintains a full membership view of the system and connects directly to any validator it needs to communicate with.

8 Performance
Bandwidth: If we assume that each transaction requires 5 KB of traffic — including the cost of
receiving the transaction via the mempool, rebroadcasting it, receiving blocks from the leader,
and replicating to clients — then validators require a 40 Mbps internet connection to support
1,000 transactions per second. Access to such bandwidth is widely available.

Since leader broadcast proposals to every other nodes, for N nodes, this means leader’s outbound bandwidth is (N-1) * 40Mbps ?
For a cluster with 100 nodes, this means leader’s outbound bandwidth is 99 * 40 = 4Gbps?

Would you consider gossip for broadcasting proposals?

3 Likes

LibraBFTv1 leveraged gossiping for broadcasting of data. In the latest version of the paper (https://developers.libra.org/docs/state-machine-replication-paper) we have moved to a linear communication protocol (see https://developers.libra.org/blog/2019/09/26/libra-bft-protocol for more details). A transaction should be smaller than 5 KB, we typically see about 400 bytes, so the math is closer to 99 validators * 400 bytes / transaction * 1000 transactions / sec = 39600000 bytes / sec = 316.8 Mbps, which is widely available among cloud providers.

We are likely to research other forms of broadcast (including gossip) in the future that scale to larger validator sets after the V1 launch.

3 Likes