Release: Polaris - v1.4.8

Polaris

The stars in our galaxy play a crucial role in bringing light and life to the celestial bodies that orbit around them. In much the same way, the star-themed releases of MultiversX will bring light and new life to the features, bug fixes, and improvements that builders and community alike have been eagerly awaiting. With each release, we intend to bring forth new progress milestones, and paradigms, shedding a clearer light on the path forward, and various adoption paths for blockchain technology.

Polaris, will be the first release in this new series of star-themed releases for MultiversX. At the core of this new release, is an aim for better and more proactive communication toward the builders community, and an invitation to engage, contribute and share feedback. The new repositioning and transformation from Elrond to MultiversX was accompanied by a lot of hard work to ensure a smooth transition, new product preparations and initiatives. Our efforts were rewarded in the first month of January, when we outpaced all other blockchains in the ecosystem and scored number 1 in terms of development metrics across several fields.

With Polaris, we are taking our commitment to builders to the next level by making the MultiversX blockchain even more open, user-friendly, and accessible to everyone.

Polaris brings with it a host of exciting new features and improvements, each of which can be thought of as a planet chained to its respective star. We have worked tirelessly to make sure that the hard-to-understand aspects of blockchain technology will be presented in a manner that is a lot easier to understand and accessible. Through this, we are confident that we can continue to set new standards in the blockchain industry and bring the benefits of blockchain technology to a wider audience.

The near future, will hopefully open a new creation phase, bringing tens, hundreds, and eventually thousands of new star systems, each expanding on its own and creating a multiverse of chains and worlds. This will be made possible through the creation of new and expanding sovereign shards, each orbiting around the MultiversX galaxy, and expanding in its own unique way. Thus, each new and significant release like Polaris, will be named after a star, and will henceforth be announced with some significant details in advance, to prepare and get wider feedback from the community.

Just like Polaris(the star) has guided navigators throughout history in their quests, our Polaris will stand as the first release of MultiversX (after rebranding), guiding all the other important releases to come. Next planned release being:

• Altair - with its Guardians feature being the highlight;

Join us on this exciting journey and and help build this future for MultiversX. So let's see what's Polaris about:

Breaking changes

Refactor PeersMiniBlocks data transmission

In the old implementation, the meta chain creates a meta block named "start of epoch meta block" after each epoch, in which it adds all the relevant information, such as rewards and validators information. However, this block contains something suboptimal when it comes to PeersMiniBlocks. In our system, all mini blocks only hold the transaction hashes to keep the block body smaller and improve its transportation over the network. The only exception to this rule is the PeersMiniBlocks contained in each start of epoch meta block. In these blocks, we have the marshalled ValidatorInfo DTO instead of its hash, which results in a larger size for each start of epoch meta block. Currently, each start of epoch meta block contains 3200 marshalled validator info DTOs in its peer mini blocks. As the number of validators increases over time, the size of the start of epoch meta block will continue to grow.

What did we do?

The objective of this refactor is to change the structure and approach of the PeersMiniBlocks in each start of epoch meta block, so that it is consistent with the rest of the mini blocks in the system. Instead of keeping the marshalled ValidatorInfo DTO, each PeersMiniBlock will only contain its hash. All nodes in the network will receive the ValidatorInfo transactions on a separate topic or they can request them on a dedicated topic using the hashes in each PeerMiniBlock. To achieve this new approach, a lot of changes were necessary, including the implementation of dedicated interceptors, resolvers, storages, and pools, as well as changes to the creation and processing procedure. This was a breaking change, so a new dedicated activation flag (RefactorPeersMiniBlocksEnableEpoch) was created to preserve backward compatibility (such as the import-db feature).

Github Link: #4024

Outport driver for events notifier

The outport driver for events notifier has been changed to include more data, the same payload as for Elasticsearch integration. This will be a breaking change in the observer -> events notifier communication, for this, events notifier has to be upgraded to v1.1.8. Services/clients connected to rabbitmq, for example, should not be affected. Please make sure to upgrade events notifier service before mx-chain-go observer node release.

Please check the release notes here.

More information about events notifier service can be found here.

Features

Altered accounts

Altered accounts refers to all accounts that suffered changes in a block. Initially, the purpose of the altered accounts feature was to decouple the indexer from the node. The indexer now receives a map of all the used accounts in a block, allowing it to avoid accessing the trie data. But as a bonus, this feature also enabled the alteration of accounts through the API, opening up new possibilities for integration with other tools and platforms. The implementation of this feature represents a major step towards a more flexible and scalable platform.

Examples:

• <node-ip>:<port>/block/altered-accounts/by-nonce/{nonce}
• <node-ip>:<port>/block/altered-accounts/by-hash/{hash}

KOSK BLS Multisigner

This feature presents advancements in terms of performance, specifically in regards to block signature verification time and CPU usage. By utilizing the staking mechanism and confirming the knowledge of the secret key (KOSK) for each registered BLS public key through a valid signature, we can eliminate certain operations for signature aggregation and aggregated signature verification. These operations were previously used to prevent rogue keys, but the same result is now achieved through the use of KOSK. The removal of these operations results in reduced CPU usage for the verification of metachain and shard blocks.

You can see benchmarking results in here: #4314

Bugfixes and Improvements

We will try to cover most important bugfixes that were implemented in this release. As you know, we place a strong emphasis on maintaining a robust and stable chain. Our goal is to ensure that MultiversX operates smoothly and efficiently, providing users with a seamless experience. Therefore, we've put in significant efforts to address any issues and improve the chain's performance and stability. We are confident that these bugfixes will contribute to a more secure and reliable blockchain experience for you. Going forward, we will continue to focus on enhancing the robustness of MultiversX and delivering a world-class blockchain solution.

State related improvements and fixes

We have made several important changes to the state management system in our latest release. Firstly, to prevent IO bottlenecks in cases where a snapshot takes more than one epoch, we have introduced the ability to avoid multiple snapshots in parallel. To ensure that these snapshots do not conflict with each other, we have added an "isSnapshotInProgress" flag. This flag allows us to skip a snapshot if another one is already running. In addition to this, we have also removed duplicated code from the AccountsDB and PeerAccountsDB.

Secondly, there were some edge cases where trie nodes were not saved in the proper storer. To address this issue, we have implemented a process during the snapshot process that will save trie nodes to the previous storer if they are not present. This ensures that the state is complete and that the node will be able to progress.

Thirdly, to ensure that the node can always progress even if a trie node is missing, we have added a trie syncer to the AccountsDB. This syncer will be triggered if there are any missing nodes during the snapshotting and it will sync missing trie nodes from the network.

Lastly, when getting values from data tries, we now return not only the value, but also the depth from which the value was retrieved. This will allow us to have a more accurate gas model based on the complexity of the trie.

Old NFT storage

In our effort to address the issues encountered during the process of moving old-type NFTs across shards and changing their metadata, we've implemented several changes. First, we updated the NFT metadata every time it was processed to ensure consistency across shards. This was achieved through the integration of this pull request. Second, we refactored the activation flags to be enabled on the mainnet. Finally, we adjusted the blockchain hook counters based on the results of our latest tests. These changes have helped resolve the issue and ensure that moving NFTs across shards and changing their metadata can be done smoothly and efficiently.

Storage fixes

We've also been focused on improving the overall functionality of the system. Three key pull requests were implemented to address specific issues. First, to ensure that the complete state of the node is available, we implemented a change to guarantee that at least two active trie databases are opened when the node is restarting. Secondly, if some trie nodes can not be retrieved from storage (because they are missing), sync those nodes from the network. If the trie nodes are present in storage, but can not be retrieved because the node is closing, do not start the syncing process. Finally, we addressed an error in the process of checking if a trie was empty by adding a check to ensure that the root hash of the trie is equal to the empty trie root hash. These changes have helped to make the system more robust and reliable.

Operation mode cli flag

We recently added a new CLI flag, --operation-mode, which allows users to alter the node's configuration to run in specific modes. However, we encountered an issue where setting the full-archive operation mode did not actually result in a full archive node, as the configuration values were not being overridden. To resolve this issue, we made the necessary changes to override the configuration values if the operation mode is set.

Github Links: #4605, #4646

Termui trie sync status

In this release, we have introduced an improvement on the termui that allows you to monitor the progress of the trie sync on your node. Keeping track of the trie sync is crucial to ensure the smooth operation of your node, and with this improvement, you can easily see the status of the process.

Transaction execution order

We've extended the structure of the transaction that is saved in the ElasticSearch database or used by the notifier with an extra field that specifies the execution order (ExecutionOrder). This field has been added in the Block structure. The notifier and indexer will be the biggest beneficiaries of this addition. For example, this is needed in order to keep track of the correct transaction order notarized on the notifier transactions order channel.

Github Links: #4773

Other bugfixes and improvements worth mentioning

• Fix MetricReceivedProposedBlock overflow
• Fixed the keygenerator binary
• Fix account query options for altered accounts
• Fix custom meta DB remover
• Fix accounts trie remover config value
• Other minor fixes.

Repository code splits and rebranding

As part of our continuous efforts to improve our release processes, architecture and preparation for the release of sovereign chains, we started to split our repositories. This move has allowed us to increase efficiency, reusability (so much needed in case of sovereign shards) and maintainability of our code.

The repository split has been a major undertaking, but it has already begun to pay dividends in terms of better code structure. In keeping with our commitment to promoting and supporting the builder community, we have made some of our previously private repositories public. We believe in the power of collaboration and sharing, and we are proud to encourage and support other builders in their endeavors.

As we move forward with our mission, we will continue to share our progress and updates with the community. Our aim is to provide the best possible experiences for our community of builders so come and let us know of your problems on Discord.
It's important to note that this rebranding and the repository split has consumed a significant portion of our capacity and resources. Despite the challenges, we are proud of the hard work and dedication that has gone into making this change a reality.

Full Github Changelog: Polaris - v1.4.8

Activation epoch

There are 8 flags that will be enabled on epoch 938 which is scheduled to happen on February 23rd, 2023 ~15:30 UTC.

• RuntimeMemStoreLimitEnableEpoch - a maximum hardcap limit for the mem store instruction (invariant);
• SetSenderInEeiOutputTransferEnableEpoch - a flag that will make the EEI subsystem return & store the sender's address for easier traceability & debugging experience;
• RefactorPeersMiniBlocksEnableEpoch - the epoch when the refactoring of the peer's miniblocks is activated: the miniblocks won't contain the marshaled form of the peer change info but a hash of that change. This is now done in the exact manner as the transactions are referenced, transferred & processed. The peer change info remained unchanged and it's processing results will be identical with the initial implementation. So, this change affects only the transmission and the reference of each peer change info;
• MaxBlockchainHookCountersEnableEpoch - will limit the number of blockchain-API calls a SmartContract can perform;
• WipeSingleNFTLiquidityDecreaseEnableEpoch - the SystemAccounts will automatically decrease the value (and do an automatically clean) whenever an NFT is wiped. Without this fix, the SystemAccount would accumulate garbage after the processing of an NFT-wipe operation;
• AlwaysSaveTokenMetaDataEnableEpoch - will trigger the automatic update of an old-type NFT info in order to preserve the consistency of the held information accross shards;
• BLSMultiSignerEnableEpoch.KOSK - will improve the performance of the aggregated signature verification by 2-3x. The security of the BLS signatures is not affected by this change.
• gasScheduleV7.toml - this is a new gas schedule change that will address a bug when computing the developer rewards while executing a multi-esdt-nft-transfer. The cost won't change but the developer fees will be transferred correctly as in any other esdt operation or SC call.

Mainnet config: https://github.com/multiversx/mx-chain-mainnet-config/releases/tag/v1.4.8.1

The full binary release notes are published here: https://github.com/multiversx/mx-chain-go/releases/tag/v1.4.8

We'd like to invite you joining us on Discord, where you can connect with other builders and developers, stay up-to-date on all things happening behind the scenes, and receive support from the community. Furthermore, we're curious to hear your thoughts on our release naming strategy and if you have any questions regarding this release.