Sidekiq — Part II: Processing Jobs and More!

Photo by Lewis Kang'ethe Ngugi on Unsplash

In part-I, we’ve successfully created our first Sidekiq job and pushed it to the queue. Now it’s time to see how Sidekiq processes these jobs and explore additional functionalities!

Processing Jobs

We started a Sidekiq worker in the previous step using the command bundle exec sidekiq. This command instructs Sidekiq to constantly monitor the Redis queue for any new jobs and process them individually.

By default, Sidekiq uses a single-worker process. This means it can only process one job at a time. If you have a high volume of background jobs, you might want to consider running multiple worker processes to improve performance. You can easily achieve this by specifying the number of worker processes while starting Sidekiq:

bundle exec sidekiq -c 4

This command will start four Sidekiq worker processes, allowing Sidekiq to process up to four jobs concurrently.

Sidekiq Web UI

Sidekiq provides a web UI to monitor and manage background jobs. To enable the web UI, you need to add the sidekiq-ui gem to your Gemfile and run bundle install.

gem 'sidekiq-ui'

Once the gem is installed, restart your Rails server and Sidekiq worker. Now you can access the Sidekiq web UI by navigating to http://localhost:3000/sidekiq in your web browser.

The web UI provides a real-time view of your background jobs, including:

• The number of queued, processing, and failed jobs.
• Details about each job, such as its class name, arguments, and retry attempts.
• The ability to retry failed jobs or delete them from the queue.

Retrying Failed Jobs

There might be scenarios where a job fails due to unexpected errors. Sidekiq provides a built-in retry mechanism to handle these situations. By default, Sidekiq retries failed jobs five times with an exponential backoff between retries. This means the time interval between retries will increase after each failed attempt.

You can customize the retry behavior for a specific job by overriding the retry_in method in your job class. Here’s an example:

class MyImportantJob
  include Sidekiq::Job

  def perform(args)
    # Job logic here
    raise "Something went wrong!"
  end

  retry_in 60 * 10, queue: :critical  # Retry the job after 10 minutes in the critical queue
end

In this example, we’ve overridden the retry_in method to retry the failed job after 10 minutes in the critical queue. You can specify a different retry time or even a different queue to place the retried job in.

Sidekiq Queues

By default, Sidekiq uses a single queue named default for all background jobs. However, Sidekiq allows you to define multiple queues to prioritize different types of jobs. For instance, you might have a critical queue for urgent jobs that need to be processed as soon as possible and a low queue for less critical tasks that can be processed with a lower priority.

MyCriticalJob.perform_async(args).enqueue(:critical)

This will enqueue the job in the critical queue. Sidekiq worker processes jobs from different queues based on their priority. By default, the default queue has the highest priority.

Conclusion

In this two-part tutorial, we’ve explored the basics of Sidekiq, including creating jobs, processing them with workers, and managing them through the web UI. We’ve also briefly discussed retries and queues to give you a better understanding of how to handle different job scenarios effectively.

I hope this tutorial has been helpful! Feel free to explore the Sidekiq documentation for more advanced features and functionalities.