Upgrade: the chain halt ongoing at block height 269766

:dove: Pigeons! :dove: COO! COO!


Dear pigeons!

Paloma Testnet-12 is still halted, after the successful chain upgrade, as of this email due to the following:

  1. The Backstory: Paloma was first built on Ignite’s Starport scaffolding. This decision was made early on to bootstrap a network quickly for the pigeons to test and nest in a fast iteration cycle of Paloma’s earliest features.

  2. Paloma’s chain upgrade, this week, from Cosmos-SDK v0.45.8 to v0.45.9 for dragonberry ICS exploit was a success for software upgrade. However, we are unable to restart the chain due to an AppHash non-determinism error.

  3. It appears that IAVL-FAST-NODE feature which is ON in this testnet needs to go OFF as we get ready for mainnet and is causing issues after this type of upgrade.

  4. We turned this feature OFF with all pigeons updating their app.toml today, but the chain still experiences apphash errors for the first new block.

  5. Therefore, we will be refactoing Paloma to migrate off Ignite’s Starport and upgrade to standard Cosmos-SDK and IAVL fix. We expect this will resolve our efforts and make Paloma even more mainnet-ready.

Due to the heavy refactor effort, we will be chainhalt for a few more days until we can deliver upgrades to Paloma by Thursday. We don’t want to downgrade as we have governance at the chain height. We don’t want to create a new testnet as we still have to complete an upgrade without IAVL-FASTNODE issues. We will keep you posted as we continue, but the chain halt state is here until we upgrade and restart from block 269767.

COO! COO! Stay alert pigeons. We’ll be back in flight!
The Bowery King