Feature: Scheduled jobs

R&D
1

Feature Request: Scheduled jobs

Description

This feature represents an extension of the existing scheduler module, adding the option to specify a schedule when creating new jobs, which in turn will be run and monitored according to their schedule, allowing for a more sophisticated control flow.

Business Case

Adding scheduled job execution to the Paloma chain will benefit all users by giving them full control to define and execute cross chain jobs. This feature will allow users to set up recurring jobs on Paloma chain, which can be executed as often as once per block, with the outcome and job history being kept directly on chain. With this level of control and transparency, users can easily manage and execute their cross chain jobs, helping to streamline their operations and increase efficiency.

Requirements

Job creation

Users should be able to specify a set of triggers when creating a job

palomad tx scheduler create-job \
  --job-id robin_job_1 \
  --chain-type evm \
  --chain-ref-id bnb-main \
  --definition $ROBIN_DEFINITION_FILE \
  --payload $ROBIN_PAYLOAD_FILE \
  --payload-modifiable=true \
  --from $CLI_ADDRESS \
  --fee-granter $VALIDATOR_ADDRESS \
  --gas auto \
  --gas-adjustment 1.5 \
  --fees 500ugrain \
  --trigger "*/5 * * * *" \
  -b sync --yes

We will keep the trigger definition in cron format for now, but in the future, this could be expanded to support additional trigger events, i.e.:

  • Fixed time: --trigger "DateTime: 20231001T10:00:00Z"
  • Fixed Block Height: --trigger "BH: 12345"
  • Recurring Block Height: --trigger "RecurringBH: @every 5blocks"
  • Event based: --trigger "Event: ValidatorJailed"
  • and more

Technical architecture breakdown

  1. Update the Job.Trigger definition in the proto files to include scheduling information, including last execution time
  2. Add arg to scheduler tx create-job command, parse and attach as Trigger to the job
  3. Create ListJobs() on the keeper in order to retrieve all jobs.
  4. Stretch goal: Fetch listed by last execution time / scheduled execution time
  5. Extend modules EndBlock logic to iterate over all pending jobs and setup job execution for those matching any trigger criteria, and reschedule if needed according to their trigger spec
  6. Update message attesting verification to update job result on chain

Limitations

List any limitations or constraints that need to be taken into consideration when implementing this feature.

Open Questions

  • Evaluating cron definition is yet to be decided. Evaluating against server timer can lead to unexpected results, i.e. during daylight saving time zone switches, in case the locale doesn’t match the users expectation, leap seconds, etc… To circumvent those problems, time sensitive information is usually kept in UTC. However, this will require the user to keep this in mind when creating triggers.
  • Handling failed jobs can become very tricky. Usually, Pigeons will attempt to redeliver a message for 30 minutes, after which the message is dropped from the queue and considered undeliverable. With this failsafe is place for all messages, adding any sort of retries to the job scheduling itself seems out of scope, especially as it may overlap with additional executions of the same job. Failed message handling should be updated to include monitoring for the source job.
  • Funding: at the moment, users will specify gas, gas adjustment and fees when actually sending the execute-job command. I’m fairly confident this is simply no longer needed, but needs additional research.

Additional Notes

Include any additional information or notes that are relevant to this feature request.

1 Like