Taariq Lewis @ LewisTaariq, CEO of VolumeFi (
volume.finance
) offers a post-mortem on the recent network chain halt in this week’s AMA episode. Nearly three weeks after the Cosmos SDK 50 upgrade to v1.13.1, Paloma faced challenges from AppHash errors, UpgradeHandler, and consensus failure. A vote for a chain halt was passed last week and. the devs team at Paloma quickly buckled down to work and investigated issues causing the failure to restart the chain and get past pre-vote. After issues related to IAVL, commit hash, state sync, and IBC parameters were resolved, the team detected non-determinism due to version mismatch adversely impacting Paloma.
The new release, Paloma v1.13.2 ushers in a new genesis that will have fixes to the build process. Once validators upgrade to the new version and genesis, the network will begin building blocks quickly and pave the way for Racer to come online.
Squeeze, Paloma’s L2 EVM, to be launched on Arbitrum is coming! More bots are in the brew as Paloma Pigeons get ready to take flight. Sequencers and a quorum member of nodes for data availability designated from the Paloma validator set signal more opportunities for Pigeons to further monetize on L2.
Paloma continues to explore the AI space for the best fit. Plans to forge partnerships with AI tool companies are underway to open access for Paloma validators for message relays, AI tooling, and AI execution.
Taariq welcomes everyone to Volume’s Twitter Spaces. He explains that today’s episode will not be about Paloma’s Racer because the network is at a chain halt. He shares a brief background about Volume Finance.
He adds that the focus is to set a path for Paloma validators to make massive profits and revenues.
It has been nearly three weeks since the upgrade to Cosmos SDK 50, and the team continues to face challenges as some validators are unable to get back on-chain. Initially, the Cosmos team thought it was an IAVL upgrade issue as they were seeing edge cases and UpgradeHandler errors. It was decided to do a chain halt and then proceed with the upgrade. However, efforts to restart the chain failed. Attempts to recover from state sync and snapshot also failed.
The team decided to export a new genesis where it was discovered that validators were using the wrong commit hash. After tracking validators to make sure that they are on the right commit hashes, the chain still could not restart. Golang was updated from v.1.21 to v.1.22 with validators now on the same commit hash, and yet, the chain was not producing blocks and was unable to get past pre-vote. Devs did a comparison of the exported state of validators and detected non-determinism due to a version mismatch.
The Paloma team is now going to create a v.1.13.2 that will build upgrades on the makefile so that the build version will always come with the preceding “v” even if it is built manually. The new Cosmos requires the build version number to be exact. IAVL will be updated to 10.12. There will be a new genesis for Racer that will be the required minimum value and will come with a checksum.
Taariq explains that IBC has a new allowed list called
LocalHost
. This allowed list was not properly migrated in the upgrade which might have contributed to the AppHash error during chain start and pre-vote.
He adds that it appears Cosmos 50 is sensitive to these types of matches and once the adjustments are made, he is confident that Paloma will be able to make blocks again.
All Paloma private testnets are downloaded build environments. The team will now have the testnet system automatically deploy manual builds in a mixed-build environment.
He thanks the validators for being patient through testing. They are active and ready to go.
Taariq muses that while the network did not experience these challenges before the upgrade to Cosmos SDK 50, the present issues are clear-cut and workable.
Taariq announces that Squeeze, Paloma’s L2 EVM is coming and will be launched on Arbitrum. The team is working on more bots. Sequencers and a member of the quorum of nodes for data availability will be designated from the Paloma validator set. These opportunities, in addition to relaying messages, will allow validators to participate and further monetize on that layer.
A new engineer will be coming on board to work on the EVM L2 to make sure it is up and running and launched quickly.
Paloma is actively exploring the best fit in the AI space. They will begin talking to AI tool companies to see where Paloma can help with automation so that validators can access relays for AI tooling and AI execution.
Taariq assures that the team will continue to prep for a genesis on Racer. States and balances have been preserved for all validators and all users of Paloma. They will ensure that all builds are consistent and that IBC features are fully allowed.
Stay tuned for the next AMA!
To find out more about Volume, check out Volume Finance (
), join the Discord (
), and follow us on Twitter (@Volumefi). Check out Paloma Chain on
(
), follow them on Twitter (@palomachain), and join the flock on Discord (
). Coo!