Skip to main content

How to Build a Complex Golang VM

This is part of a series of tutorials for building a Virtual Machine (VM):

Introduction​

In this tutorial, we'll walk through how to build a virtual machine by referencing the BlobVM. The BlobVM is a virtual machine that can be used to implement a decentralized key-value store.

A blob (shorthand for "binary large object") is an arbitrary piece of data. BlobVM stores a key-value pair by breaking it apart into multiple chunks stored with their hashes as their keys in the blockchain. A root key-value pair has references to these chunks for lookups. By default, the maximum chunk size is set to 200 KiB.

Prerequisites​

Make sure you have followed the previous tutorials in this series:

Components​

A VM defines how a blockchain should be built. A block is populated with a set of transactions which mutate the state of the blockchain when executed. When a block with a set of transactions is applied to a given state, a state transition occurs by executing all of the transactions in the block in-order and applying it to the previous block of the blockchain. By executing a series of blocks chronologically, anyone can verify and reconstruct the state of the blockchain at an arbitrary point in time.

The BlobVM repository has a few components to handle the lifecycle of tasks from a transaction being issued to a block being accepted across the network:

  • Transaction - A state transition
  • Mempool - Stores pending transactions that haven't been finalized yet
  • Network - Propagates transactions from the mempool other nodes in the network
  • Block - Defines the block format, how to verify it, and how it should be accepted or rejected across the network
  • Block Builder - Builds blocks by including transactions from the mempool
  • Virtual Machine - Application-level logic. Implements the VM interface needed to interact with Avalanche consensus and defines the blueprint for the blockchain.
  • Service - Exposes APIs so users can interact with the VM
  • Factory - Used to initialize the VM

Lifecycle of a Transaction​

A VM will often times expose a set of APIs so users can interact with the it. In the blockchain, blocks can contain a set of transactions which mutate the blockchain's state. Let's dive into the lifecycle of a transaction from its issuance to its finalization on the blockchain.

  • A user makes an API request to service.IssueRawTx to issue their transaction
    • This API will deserialize the user's transaction and forward it to the VM
  • The transaction is submitted to the VM
    • The transaction is added to the VM's mempool
  • The VM asynchronously periodically gossips new transactions in its mempool to other nodes in the network so they can learn about them
  • The VM sends the Avalanche consensus engine a message to indicate that it has transactions in the mempool that are ready to be built into a block
  • The VM proposes the block with to consensus
  • Consensus verifies that the block is valid and well-formed
  • Consensus gets the network to vote on whether the block should be accepted or rejected
    • If a block is rejected, its transactions are reclaimed by the mempool so they can be included in a future block
    • If a block is accepted, it's finalized by writing it to the blockchain

Coding the Virtual Machine​

We'll dive into a few of the packages that are in the The BlobVM repository to learn more about how they work:

  • vm
    • block_builder.go
    • chain_vm.go
    • network.go
    • service.go
    • vm.go
  • chain
    • unsigned_tx.go
    • base_tx.go
    • transfer_tx.go
    • set_tx.go
    • tx.go
    • block.go
    • mempool.go
    • storage.go
    • builder.go
  • mempool
    • mempool.go

Transactions​

The state the blockchain can only be mutated by getting the network to accept a signed transaction. A signed transaction contains the transaction to be executed alongside the signature of the issuer. The signature is required to cryptographically verify the sender's identity. A VM can define an arbitrary amount of unique transactions types to support different operations on the blockchain. The BlobVM implements two different transactions types:

  • TransferTx - Transfers coins between accounts.
  • SetTx - Stores a key-value pair on the blockchain.

UnsignedTransaction​

All transactions in the BlobVM implement the common UnsignedTransaction interface, which exposes shared functionality for all transaction types.

type UnsignedTransaction interface {
    Copy() UnsignedTransaction
    GetBlockID() ids.ID
    GetMagic() uint64
    GetPrice() uint64
    SetBlockID(ids.ID)
    SetMagic(uint64)
    SetPrice(uint64)
    FeeUnits(*Genesis) uint64  // number of units to mine tx
    LoadUnits(*Genesis) uint64 // units that should impact fee rate

    ExecuteBase(*Genesis) error
    Execute(*TransactionContext) error
    TypedData() *tdata.TypedData
    Activity() *Activity
}

BaseTx​

Common functionality and metadata for transaction types are implemented by BaseTx.

  • SetBlockID sets the transaction's block ID.
  • GetBlockID returns the transaction's block ID.
  • SetMagic sets the magic number. The magic number is used to differentiate chains to prevent replay attacks
  • GetMagic returns the magic number. Magic number is defined in genesis.
  • SetPrice sets the price per fee unit for this transaction.
  • GetPrice returns the price for this transaction.
  • FeeUnits returns the fee units this transaction will consume.
  • LoadUnits identical to FeeUnits
  • ExecuteBase executes common validation checks across different transaction types. This validates the transaction contains a valid block ID, magic number, and gas price as defined by genesis.

TransferTx​

TransferTx supports the transfer of tokens from one account to another.

type TransferTx struct {
    *BaseTx `serialize:"true" json:"baseTx"`

    // To is the recipient of the [Units].
    To common.Address `serialize:"true" json:"to"`

    // Units are transferred to [To].
    Units uint64 `serialize:"true" json:"units"`
}

TransferTx embeds BaseTx to avoid re-implementing common operations with other transactions, but implements its own Execute to support token transfers.

This performs a few checks to ensure that the transfer is valid before transferring the tokens between the two accounts.

func (t *TransferTx) Execute(c *TransactionContext) error {
    // Must transfer to someone
    if bytes.Equal(t.To[:], zeroAddress[:]) {
        return ErrNonActionable
    }

    // This prevents someone from transferring to themselves.
    if bytes.Equal(t.To[:], c.Sender[:]) {
        return ErrNonActionable
    }
    if t.Units == 0 {
        return ErrNonActionable
    }
    if _, err := ModifyBalance(c.Database, c.Sender, false, t.Units); err != nil {
        return err
    }
    if _, err := ModifyBalance(c.Database, t.To, true, t.Units); err != nil {
        return err
    }
    return nil
}

SetTx​

SetTx is used to assign a value to a key.

type SetTx struct {
    *BaseTx `serialize:"true" json:"baseTx"`

    Value []byte `serialize:"true" json:"value"`
}

SetTx implements its own FeeUnits method to compensate the network according to the size of the blob being stored.

func (s *SetTx) FeeUnits(g *Genesis) uint64 {
    // We don't subtract by 1 here because we want to charge extra for any
    // value-based interaction (even if it is small or a delete).
    return s.BaseTx.FeeUnits(g) + valueUnits(g, uint64(len(s.Value)))
}

SetTx's Execute method performs a few safety checks to validate that the blob meets the size constraints enforced by genesis and doesn't overwrite an existing key before writing it to the blockchain.

func (s *SetTx) Execute(t *TransactionContext) error {
    g := t.Genesis
    switch {
    case len(s.Value) == 0:
        return ErrValueEmpty
    case uint64(len(s.Value)) > g.MaxValueSize:
        return ErrValueTooBig
    }

    k := ValueHash(s.Value)

    // Do not allow duplicate value setting
    _, exists, err := GetValueMeta(t.Database, k)
    if err != nil {
        return err
    }
    if exists {
        return ErrKeyExists
    }

    return PutKey(t.Database, k, &ValueMeta{
        Size:    uint64(len(s.Value)),
        TxID:    t.TxID,
        Created: t.BlockTime,
    })
}

Signed Transaction​

The unsigned transactions mentioned previously can't be issued to the network without first being signed. BlobVM implements signed transactions by embedding the unsigned transaction alongside its signature in Transaction. In BlobVM, a signature is defined as the ECDSA signature of the issuer's private key of the KECCAK256 hash of the unsigned transaction's data (digest hash).

type Transaction struct {
    UnsignedTransaction `serialize:"true" json:"unsignedTransaction"`
    Signature           []byte `serialize:"true" json:"signature"`

    digestHash []byte
    bytes      []byte
    id         ids.ID
    size       uint64
    sender     common.Address
}

The Transaction type wraps any unsigned transaction. When a Transaction is executed, it calls the Execute method of the underlying embedded UnsignedTx and performs the following sanity checks:

  1. The underlying UnsignedTx must meet the requirements set by genesis
    1. This includes checks to make sure that the transaction contains the correct magic number and meets the minimum gas price as defined by genesis
  2. The transaction's block ID must be a recently accepted block
  3. The transaction must not be a recently issued transaction
  4. The issuer of the transaction must have enough gas
  5. The transaction's gas price must be meet the next expected block's minimum gas price
  6. The transaction must execute without error

If the transaction is successfully verified, it's submitted as a pending write to the blockchain.

func (t *Transaction) Execute(g *Genesis, db database.Database, blk *StatelessBlock, context *Context) error {
    if err := t.UnsignedTransaction.ExecuteBase(g); err != nil {
        return err
    }
    if !context.RecentBlockIDs.Contains(t.GetBlockID()) {
        // Hash must be recent to be any good
        // Should not happen beause of mempool cleanup
        return ErrInvalidBlockID
    }
    if context.RecentTxIDs.Contains(t.ID()) {
        // Tx hash must not be recently executed (otherwise could be replayed)
        //
        // NOTE: We only need to keep cached tx hashes around as long as the
        // block hash referenced in the tx is valid
        return ErrDuplicateTx
    }

    // Ensure sender has balance
    if _, err := ModifyBalance(db, t.sender, false, t.FeeUnits(g)*t.GetPrice()); err != nil {
        return err
    }
    if t.GetPrice() < context.NextPrice {
        return ErrInsufficientPrice
    }
    if err := t.UnsignedTransaction.Execute(&TransactionContext{
        Genesis:   g,
        Database:  db,
        BlockTime: uint64(blk.Tmstmp),
        TxID:      t.id,
        Sender:    t.sender,
    }); err != nil {
        return err
    }
    if err := SetTransaction(db, t); err != nil {
        return err
    }
    return nil
}
Example​

Let's walk through an example on how to issue a SetTx transaction to the BlobVM to write a key-value pair.

  • Create the unsigned transaction for SetTx
utx := &chain.SetTx{
    BaseTx: &chain.BaseTx{},
    Value:  []byte("data"),
}

utx.SetBlockID(lastAcceptedID)
utx.SetMagic(genesis.Magic)
utx.SetPrice(price + blockCost/utx.FeeUnits(genesis))
digest, err := chain.DigestHash(utx)
  • Sign the digest hash with the issuer's private key.
signature, err := chain.Sign(digest, privateKey)
  • Create and initialize the new signed transaction.
tx := chain.NewTx(utx, sig)
if err := tx.Init(g); err != nil {
    return ids.Empty, 0, err
}
  • Issue the request with the client
txID, err = cli.IssueRawTx(ctx, tx.Bytes())

Mempool​

Mempool Overview​

The mempool is a buffer of volatile memory that stores pending transactions. Transactions are stored in the mempool whenever a node learns about a new transaction either through gossip with other nodes or through an API call issued by a user.

The mempool is implemented as a min-max heap ordered by each transaction's gas price. The mempool is created during the initialization of VM.

vm.mempool = mempool.New(vm.genesis, vm.config.MempoolSize)

Whenever a transaction is submitted to VM, it first gets initialized, verified, and executed locally. If the transaction looks valid, then it's added to the mempool.

vm.mempool.Add(tx)

Mempool Methods​

Add​

When a transaction is added to the mempool, Add is called. This performs the following:

  • Checks if the transaction being added already exists in the mempool or not
  • The transaction is added to the min-max heap
  • If the mempool's heap size is larger than the maximum configured value, then the lowest paying transaction is evicted
  • The transaction is added to the list of transactions that are able to be gossiped to other peers
  • A notification is sent through the in the mempool.Pending channel to indicate that the consensus engine should build a new block

Block Builder​

Block Builder Overview​

The TimeBuilder implementation for BlockBuilder acts as an intermediary notification service between the mempool and the consensus engine. It serves the following functions:

  • Periodically gossips new transactions to other nodes in the network
  • Periodically notifies the consensus engine that new transactions from the mempool are ready to be built into blocks

TimeBuilder and can exist in 3 states:

  • dontBuild - There are no transactions in the mempool that are ready to be included in a block
  • building - The consensus engine has been notified that it should build a block and there are currently transactions in the mempool that are eligible to be included into a block
  • mayBuild - There are transactions in the mempool that are eligible to be included into a block, but the consensus engine has not been notified yet

Block Builder Methods​

Gossip​

The Gossip method initiates the gossip of new transactions from the mempool at periodically as defined by vm.config.GossipInterval.

Build​

The Build method consumes transactions from the mempool and signals the consensus engine when it's ready to build a block.

If the mempool signals the TimeBuilder that it has available transactions, TimeBuilder will signal consensus that the VM is ready to build a block by sending the consensus engine a common.PendingTxs message.

When the consensus engine receives the common.PendingTxs message it calls the VM's BuildBlock method. The VM will then build a block from eligible transactions in the mempool.

  • If there are still remaining transactions in the mempool after a block is built, then the TimeBuilder is put into the mayBuild state to indicate that there are still transactions that are eligible to be built into block, but the consensus engine isn't aware of it yet.

Network​

Network handles the workflow of gossiping transactions from a node's mempool to other nodes in the network.

Network Methods​

GossipNewTxs​

GossipNewTxs sends a list of transactions to other nodes in the network. TimeBuilder calls the network's GossipNewTxs function to gossip new transactions in the mempool.

func (n *PushNetwork) GossipNewTxs(newTxs []*chain.Transaction) error {
    txs := []*chain.Transaction{}
    // Gossip at most the target units of a block at once
    for _, tx := range newTxs {
        if _, exists := n.gossipedTxs.Get(tx.ID()); exists {
            log.Debug("already gossiped, skipping", "txId", tx.ID())
            continue
        }
        n.gossipedTxs.Put(tx.ID(), nil)
        txs = append(txs, tx)
    }

    return n.sendTxs(txs)
}

Recently gossiped transactions are maintained in a cache to avoid DDoSing a node from repeated gossip failures.

Other nodes in the network will receive the gossiped transactions through their AppGossip handler. Once a gossip message is received, it's deserialized and the new transactions are submitted to the VM.

func (vm *VM) AppGossip(nodeID ids.NodeID, msg []byte) error {
    txs := make([]*chain.Transaction, 0)
    if _, err := chain.Unmarshal(msg, &txs); err != nil {
        return nil
    }

    // submit incoming gossip
    log.Debug("AppGossip transactions are being submitted", "txs", len(txs))
    if errs := vm.Submit(txs...); len(errs) > 0 {
        for _, err := range errs {

        }
    }

    return nil
}

Block​

Blocks go through a lifecycle of being proposed by a validator, verified, and decided by consensus. Upon acceptance, a block will be committed and will be finalized on the blockchain.

BlobVM implements two types of blocks, StatefulBlock and StatelessBlock.

StatefulBlock​

A StatefulBlock contains strictly the metadata about the block that needs to be written to the database.

type StatefulBlock struct {
    Prnt        ids.ID         `serialize:"true" json:"parent"`
    Tmstmp      int64          `serialize:"true" json:"timestamp"`
    Hght        uint64         `serialize:"true" json:"height"`
    Price       uint64         `serialize:"true" json:"price"`
    Cost        uint64         `serialize:"true" json:"cost"`
    AccessProof common.Hash    `serialize:"true" json:"accessProof"`
    Txs         []*Transaction `serialize:"true" json:"txs"`
}

StatelessBlock​

StatelessBlock is a superset of StatefulBlock and additionally contains fields that are needed to support block-level operations like verification and acceptance throughout its lifecycle in the VM.

type StatelessBlock struct {
    *StatefulBlock   `serialize:"true" json:"block"`
    id                ids.ID
    st                choices.Status
    t                 time.Time
    bytes             []byte
    vm                VM
    children          []*StatelessBlock
    onAcceptDB        *versiondb.Database
}

Let's have a look at the fields of StatelessBlock:

  • StatefulBlock - The metadata about the block that will be written to the database upon acceptance
  • bytes - The serialized form of the StatefulBlock.
  • id - The Keccak256 hash of bytes.
  • st - The status of the block in consensus (i.e Processing, Accepted, or Rejected)
  • children - The children of this block
  • onAcceptDB - The database this block should be written to upon acceptance.

When the consensus engine tries to build a block by calling the VM's BuildBlock, the VM calls the block.NewBlock function to get a new block that is a child of the currently preferred block.

func NewBlock(vm VM, parent snowman.Block, tmstp int64, context *Context) *StatelessBlock {
    return &StatelessBlock{
        StatefulBlock: &StatefulBlock{
            Tmstmp: tmstp,
            Prnt:   parent.ID(),
            Hght:   parent.Height() + 1,
            Price:  context.NextPrice,
            Cost:   context.NextCost,
        },
        vm: vm,
        st: choices.Processing,
    }
}

Some StatelessBlock fields like the block ID, byte representation, and timestamp aren't populated immediately. These are set during the StatelessBlock's init method, which initializes these fields once the block has been populated with transactions.

func (b *StatelessBlock) init() error {
    bytes, err := Marshal(b.StatefulBlock)
    if err != nil {
        return err
    }
    b.bytes = bytes

    id, err := ids.ToID(crypto.Keccak256(b.bytes))
    if err != nil {
        return err
    }
    b.id = id
    b.t = time.Unix(b.StatefulBlock.Tmstmp, 0)
    g := b.vm.Genesis()
    for _, tx := range b.StatefulBlock.Txs {
        if err := tx.Init(g); err != nil {
            return err
        }
    }
    return nil
}

To build the block, the VM will try to remove as many of the highest-paying transactions from the mempool to include them in the new block until the maximum block fee set by genesis is reached.

A block once built, can exist in two states:

  • Rejected - The block was not accepted by consensus
    • In this case, the mempool will reclaim the rejected block's transactions so they can be included in a future block.
  • Accepted - The block was accepted by consensus
    • In this case, we write the block to the blockchain by committing it to the database.

When the consensus engine receives the built block, it calls the block's Verify method to validate that the block is well-formed. In BlobVM, the following constraints are placed on valid blocks:

  • A block must contain at least one transaction and the block's timestamp must be within 10s into the future.
if len(b.Txs) == 0 {
  return nil, nil, ErrNoTxs
}
if b.Timestamp().Unix() >= time.Now().Add(futureBound).Unix() {
  return nil, nil, ErrTimestampTooLate
}
  • The sum of the gas units consumed by the transactions in the block must not exceed the gas limit defined by genesis.
blockSize := uint64(0)
for _, tx := range b.Txs {
  blockSize += tx.LoadUnits(g)
  if blockSize > g.MaxBlockSize {
    return nil, nil, ErrBlockTooBig
  }
}
  • The parent block of the proposed block must exist and have an earlier timestamp.
parent, err := b.vm.GetStatelessBlock(b.Prnt)
if err != nil {
  log.Debug("could not get parent", "id", b.Prnt)
  return nil, nil, err
}
if b.Timestamp().Unix() < parent.Timestamp().Unix() {
  return nil, nil, ErrTimestampTooEarly
}
  • The target block price and minimum gas price must meet the minimum enforced by the VM.
context, err := b.vm.ExecutionContext(b.Tmstmp, parent)
if err != nil {
  return nil, nil, err
}
if b.Cost != context.NextCost {
  return nil, nil, ErrInvalidCost
}
if b.Price != context.NextPrice {
  return nil, nil, ErrInvalidPrice
}

After the results of consensus are complete, the block is either accepted by committing the block to the database or rejected by returning the block's transactions into the mempool.

// implements "snowman.Block.choices.Decidable"
func (b *StatelessBlock) Accept() error {
    if err := b.onAcceptDB.Commit(); err != nil {
        return err
    }
    for _, child := range b.children {
        if err := child.onAcceptDB.SetDatabase(b.vm.State()); err != nil {
            return err
        }
    }
    b.st = choices.Accepted
    b.vm.Accepted(b)
    return nil
}

// implements "snowman.Block.choices.Decidable"
func (b *StatelessBlock) Reject() error {
    b.st = choices.Rejected
    b.vm.Rejected(b)
    return nil
}

API​

Service implements an API server so users can interact with the VM. The VM implements the interface method CreateHandlers that exposes the VM's RPC API.

func (vm *VM) CreateHandlers() (map[string]*common.HTTPHandler, error) {
    apis := map[string]*common.HTTPHandler{}
    public, err := newHandler(Name, &PublicService{vm: vm})
    if err != nil {
        return nil, err
    }
    apis[PublicEndpoint] = public
    return apis, nil
}

One API that's exposed is IssueRawTx to allow users to issue transactions to the BlobVM. It accepts an IssueRawTxArgs that contains the transaction the user wants to issue and forwards it to the VM.

func (svc *PublicService) IssueRawTx(_ *http.Request, args *IssueRawTxArgs, reply *IssueRawTxReply) error {
    tx := new(chain.Transaction)
    if _, err := chain.Unmarshal(args.Tx, tx); err != nil {
        return err
    }

    // otherwise, unexported tx.id field is empty
    if err := tx.Init(svc.vm.genesis); err != nil {
        return err
    }
    reply.TxID = tx.ID()

    errs := svc.vm.Submit(tx)
    if len(errs) == 0 {
        return nil
    }
    if len(errs) == 1 {
        return errs[0]
    }
    return fmt.Errorf("%v", errs)
}

Virtual Machine​

We have learned about all the components used in the BlobVM. Most of these components are referenced in the vm.go file, which acts as the entry point for the consensus engine as well as users interacting with the blockchain. For example, the engine calls vm.BuildBlock(), that in turn calls chain.BuildBlock(). Another example is when a user issues a raw transaction through service APIs, the vm.Submit() method is called.

Let's look at some of the important methods of vm.go that must be implemented:

Virtual Machine Methods​

Initialize​

Initialize is invoked by metalgo when creating the blockchain. metalgo passes some parameters to the implementing VM.

  • ctx - Metadata about the VM's execution
  • dbManager - The database that the VM can write to
  • genesisBytes - The serialized representation of the genesis state of this VM
  • upgradeBytes - The serialized representation of network upgrades
  • configBytes - The serialized VM-specific configuration
  • toEngine - The channel used to send messages to the consensus engine
  • fxs - Feature extensions that attach to this VM
  • appSender - Used to send messages to other nodes in the network

BlobVM upon initialization persists these fields in its own state to use them throughout the lifetime of its execution.

// implements "snowmanblock.ChainVM.common.VM"
func (vm *VM) Initialize(
    ctx *snow.Context,
    dbManager manager.Manager,
    genesisBytes []byte,
    upgradeBytes []byte,
    configBytes []byte,
    toEngine chan<- common.Message,
    _ []*common.Fx,
    appSender common.AppSender,
) error {
    log.Info("initializing blobvm", "version", version.Version)

    // Load config
    vm.config.SetDefaults()
    if len(configBytes) > 0 {
        if err := ejson.Unmarshal(configBytes, &vm.config); err != nil {
            return fmt.Errorf("failed to unmarshal config %s: %w", string(configBytes), err)
        }
    }

    vm.ctx = ctx
    vm.db = dbManager.Current().Database
    vm.activityCache = make([]*chain.Activity, vm.config.ActivityCacheSize)

    // Init channels before initializing other structs
    vm.stop = make(chan struct{})
    vm.builderStop = make(chan struct{})
    vm.doneBuild = make(chan struct{})
    vm.doneGossip = make(chan struct{})
    vm.appSender = appSender
    vm.network = vm.NewPushNetwork()

    vm.blocks = &cache.LRU{Size: blocksLRUSize}
    vm.verifiedBlocks = make(map[ids.ID]*chain.StatelessBlock)

    vm.toEngine = toEngine
    vm.builder = vm.NewTimeBuilder()

    // Try to load last accepted
    has, err := chain.HasLastAccepted(vm.db)
    if err != nil {
        log.Error("could not determine if have last accepted")
        return err
    }

    // Parse genesis data
    vm.genesis = new(chain.Genesis)
    if err := ejson.Unmarshal(genesisBytes, vm.genesis); err != nil {
        log.Error("could not unmarshal genesis bytes")
        return err
    }
    if err := vm.genesis.Verify(); err != nil {
        log.Error("genesis is invalid")
        return err
    }
    targetUnitsPerSecond := vm.genesis.TargetBlockSize / uint64(vm.genesis.TargetBlockRate)
    vm.targetRangeUnits = targetUnitsPerSecond * uint64(vm.genesis.LookbackWindow)
    log.Debug("loaded genesis", "genesis", string(genesisBytes), "target range units", vm.targetRangeUnits)

    vm.mempool = mempool.New(vm.genesis, vm.config.MempoolSize)

    if has { //nolint:nestif
        blkID, err := chain.GetLastAccepted(vm.db)
        if err != nil {
            log.Error("could not get last accepted", "err", err)
            return err
        }

        blk, err := vm.GetStatelessBlock(blkID)
        if err != nil {
            log.Error("could not load last accepted", "err", err)
            return err
        }

        vm.preferred, vm.lastAccepted = blkID, blk
        log.Info("initialized blobvm from last accepted", "block", blkID)
    } else {
        genesisBlk, err := chain.ParseStatefulBlock(
            vm.genesis.StatefulBlock(),
            nil,
            choices.Accepted,
            vm,
        )
        if err != nil {
            log.Error("unable to init genesis block", "err", err)
            return err
        }

        // Set Balances
        if err := vm.genesis.Load(vm.db, vm.AirdropData); err != nil {
            log.Error("could not set genesis allocation", "err", err)
            return err
        }

        if err := chain.SetLastAccepted(vm.db, genesisBlk); err != nil {
            log.Error("could not set genesis as last accepted", "err", err)
            return err
        }
        gBlkID := genesisBlk.ID()
        vm.preferred, vm.lastAccepted = gBlkID, genesisBlk
        log.Info("initialized blobvm from genesis", "block", gBlkID)
    }
    vm.AirdropData = nil

After initializing its own state, BlobVM also starts asynchronous workers to build blocks and gossip transactions to the rest of the network.

    go vm.builder.Build()
    go vm.builder.Gossip()
    return nil
}
GetBlock​

GetBlock returns the block with the provided ID.

GetBlock will attempt to fetch the given block from the database, and return an non-nil error if it wasn't able to get it.

func (vm *VM) GetBlock(id ids.ID) (snowman.Block, error) {
    b, err := vm.GetStatelessBlock(id)
    if err != nil {
        log.Warn("failed to get block", "err", err)
    }
    return b, err
}
ParseBlock​

ParseBlock deserializes a block.

func (vm *VM) ParseBlock(source []byte) (snowman.Block, error) {
    newBlk, err := chain.ParseBlock(
        source,
        choices.Processing,
        vm,
    )
    if err != nil {
        log.Error("could not parse block", "err", err)
        return nil, err
    }
    log.Debug("parsed block", "id", newBlk.ID())

    // If we have seen this block before, return it with the most
    // up-to-date info
    if oldBlk, err := vm.GetBlock(newBlk.ID()); err == nil {
        log.Debug("returning previously parsed block", "id", oldBlk.ID())
        return oldBlk, nil
    }

    return newBlk, nil
}
BuildBlock​

Avalanche consensus calls BuildBlock when it receives a notification from the VM that it has pending transactions that are ready to be issued into a block.

func (vm *VM) BuildBlock() (snowman.Block, error) {
    log.Debug("BuildBlock triggered")
    blk, err := chain.BuildBlock(vm, vm.preferred)
    vm.builder.HandleGenerateBlock()
    if err != nil {
        log.Debug("BuildBlock failed", "error", err)
        return nil, err
    }
    sblk, ok := blk.(*chain.StatelessBlock)
    if !ok {
        return nil, fmt.Errorf("unexpected snowman.Block %T, expected *StatelessBlock", blk)
    }

    log.Debug("BuildBlock success", "blkID", blk.ID(), "txs", len(sblk.Txs))
    return blk, nil
}
SetPreference​

SetPreference sets the block ID preferred by this node. A node votes to accept or reject a block based on its current preference in consensus.

func (vm *VM) SetPreference(id ids.ID) error {
    log.Debug("set preference", "id", id)
    vm.preferred = id
    return nil
}
LastAccepted​

LastAccepted returns the block ID of the block that was most recently accepted by this node.

func (vm *VM) LastAccepted() (ids.ID, error) {
    return vm.lastAccepted.ID(), nil
}

CLI​

BlobVM implements a generic key-value store, but support to read and write arbitrary files into the BlobVM blockchain is implemented in the blob-cli

To write a file, BlobVM breaks apart an arbitrarily large file into many small chunks. Each chunk is submitted to the VM in a SetTx. A root key is generated which contains all of the hashes of the chunks.

func Upload(
    ctx context.Context, cli client.Client, priv *ecdsa.PrivateKey,
    f io.Reader, chunkSize int,
) (common.Hash, error) {
    hashes := []common.Hash{}
    chunk := make([]byte, chunkSize)
    shouldExit := false
    opts := []client.OpOption{client.WithPollTx()}
    totalCost := uint64(0)
    uploaded := map[common.Hash]struct{}{}
    for !shouldExit {
        read, err := f.Read(chunk)
        if errors.Is(err, io.EOF) || read == 0 {
            break
        }
        if err != nil {
            return common.Hash{}, fmt.Errorf("%w: read error", err)
        }
        if read < chunkSize {
            shouldExit = true
            chunk = chunk[:read]

            // Use small file optimization
            if len(hashes) == 0 {
                break
            }
        }
        k := chain.ValueHash(chunk)
        if _, ok := uploaded[k]; ok {
            color.Yellow("already uploaded k=%s, skipping", k)
        } else if exists, _, _, err := cli.Resolve(ctx, k); err == nil && exists {
            color.Yellow("already on-chain k=%s, skipping", k)
            uploaded[k] = struct{}{}
        } else {
            tx := &chain.SetTx{
                BaseTx: &chain.BaseTx{},
                Value:  chunk,
            }
            txID, cost, err := client.SignIssueRawTx(ctx, cli, tx, priv, opts...)
            if err != nil {
                return common.Hash{}, err
            }
            totalCost += cost
            color.Yellow("uploaded k=%s txID=%s cost=%d totalCost=%d", k, txID, cost, totalCost)
            uploaded[k] = struct{}{}
        }
        hashes = append(hashes, k)
    }

    r := &Root{}
    if len(hashes) == 0 {
        if len(chunk) == 0 {
            return common.Hash{}, ErrEmpty
        }
        r.Contents = chunk
    } else {
        r.Children = hashes
    }

    rb, err := json.Marshal(r)
    if err != nil {
        return common.Hash{}, err
    }
    rk := chain.ValueHash(rb)
    tx := &chain.SetTx{
        BaseTx: &chain.BaseTx{},
        Value:  rb,
    }
    txID, cost, err := client.SignIssueRawTx(ctx, cli, tx, priv, opts...)
    if err != nil {
        return common.Hash{}, err
    }
    totalCost += cost
    color.Yellow("uploaded root=%v txID=%s cost=%d totalCost=%d", rk, txID, cost, totalCost)
    return rk, nil
}

Example 1​

blob-cli set-file ~/Downloads/computer.gif -> 6fe5a52f52b34fb1e07ba90bad47811c645176d0d49ef0c7a7b4b22013f676c8

Given the root hash, a file can be looked up by deserializing all of its children chunk values and reconstructing the original file.

// TODO: make multi-threaded
func Download(ctx context.Context, cli client.Client, root common.Hash, f io.Writer) error {
    exists, rb, _, err := cli.Resolve(ctx, root)
    if err != nil {
        return err
    }
    if !exists {
        return fmt.Errorf("%w:%v", ErrMissing, root)
    }
    var r Root
    if err := json.Unmarshal(rb, &r); err != nil {
        return err
    }

    // Use small file optimization
    if contentLen := len(r.Contents); contentLen > 0 {
        if _, err := f.Write(r.Contents); err != nil {
            return err
        }
        color.Yellow("downloaded root=%v size=%fKB", root, float64(contentLen)/units.KiB)
        return nil
    }

    if len(r.Children) == 0 {
        return ErrEmpty
    }

    amountDownloaded := 0
    for _, h := range r.Children {
        exists, b, _, err := cli.Resolve(ctx, h)
        if err != nil {
            return err
        }
        if !exists {
            return fmt.Errorf("%w:%s", ErrMissing, h)
        }
        if _, err := f.Write(b); err != nil {
            return err
        }
        size := len(b)
        color.Yellow("downloaded chunk=%v size=%fKB", h, float64(size)/units.KiB)
        amountDownloaded += size
    }
    color.Yellow("download complete root=%v size=%fMB", root, float64(amountDownloaded)/units.MiB)
    return nil
}

Example 2​

blob-cli resolve-file 6fe5a52f52b34fb1e07ba90bad47811c645176d0d49ef0c7a7b4b22013f676c8 computer_copy.gif

Conclusion​

This documentation covers concepts about Virtual Machine by walking through a VM that implements a decentralized key-value store.

You can learn more about the BlobVM by referencing the README in the GitHub repository.