Dear pigeons,
COO! COO! The Testnest paloma-testnet-10 network halted on 10/2/2022 due to the ongoing instantiation of the EggMinting contracts for developers. This Cosmwasm bug was due to the chain working on wasmd v0.27.0 and not supporting snapshots for statesync of the cosmwasm contracts.
Although the community put together a proposal to upgrade the chain and that proposal passed successfully, the chain was unable to recover to a working chain state before the chain-halt height. All validators eventually experienced AppHash errors, taking them off the network. Today 10/03, we attempted to restart the network with the birdlady validator leading. However, due to a migration bug on the proposed upgrade v0.10.1 which included Cosmwasm wasmd upgrade to v0.29.0-rc1, the chain is still unstarted. At this time, we are still waiting for help from the Cosmwasm team on this issue, but we have been told by Cosmwasm founder Ethan Frey that this is a new issue and that we should downgrade to wasmd v0.28.0.
Next step: We will propose to preserve paloma-testnet-10 with an updated genesis as follows.
-
All pigeons will need to downgrade their pigeon to v0.8.1
-
We will attempt to edit the genesis of a 09/30/2022 snapshot to keep the upgrade proposal, but with an update palomad upgrade to v0.10.0 which includes wasmd upgrade to v0.28.0, the stable version of wasmd. We will share the fork/update of the new genesis for the community to inspect.
-
We will attempt to restart paloma-testnet-10 with this new genesis and preserve chain balances on birdpoop2.
-
We will attempt to edit the chain-halt height so that we are able to quickly upgrade and for all validators to download the new genesis and sync with birdpoop2 and the birdlady.
-
If this works, we will preserve the old chain state, update the network, and get ready for more Paloma Egg Hunting!
This announcement will continue to be edited as we make progress.
Comments are welcomed below.