Moving from backgrounDrb to DelayedJob
In my earlier post regarding DelayedJob setup I mentioned how to setup DelayedJob and create news tasks in your existing code. This post provides details of how to move from backgrounDrb to delayed_job. First of all its important to know why ?
- BackgroundDrb takes in a lot of memory. It spawns worker at start-up. You can use MiddleMan for dynamically spawing backgrounDrb tasks but in any case, it slows thing down a little. I have 4 workers and overall the parent process consumes over 120MB — which is huge considering my linode.
- Monitoring these jobs is a little bit of a pain. Moreover, running in development / production mode requires a Drb server which adds more memory overhead.
As we speak, github has introduced Resque, which is inspired from DelayedJob but I plan to currently continue with DelayedJob because I dont use Redis (yet). The blog post for Resque has a LOT of details about issues with current background job schedulers. Worth reading!
OK – so you’re now convinced we should use DelayedJob instead of backgrounDrb but have a lot of tasks already configured. These are the steps to follow:
1. Convert your backgrounDrb workers to standard ruby classes:
class BulkUploadWorker < BackgrounDRb::MetaWorker set_worker_name :bulk_upload_worker def upload(args) end
to
class BulkUploadWorker def perform end
2. If you earlier used arguments to be passed to the backgrounDrb jobs, you need to tweak the code a little.
Suppose I have an upload method which takes ‘arg’ as the Hash parameter, it would be invoked in the controller for backgrounDrb like this:
MiddleMan.worker(:bulk_upload_worker).async_upload(:arg => {
'correction' => correction, 'file' => in_file, 'user' => current_user.id} )
Simple change to DelayedJob
Delayed::Job.enqueue BulkUploadWorker.new(correction, infile, current_user.id)
And change the worker to have a perform method (which is the one which gets called on the job):
BulkUploadTask = Struct.new(:correction, :infile, :user_id) do
def perform
file = File.open(infile)
user = User.find(user_id)
...
If you look closely at the code above, even for an experienced Ruby coder – its no piece of cake. Now, I tried the original approch that was on github of
class BulkUploadWorker < Struct.new (:correction, :infile, :user)
but this gives me a type mis-match error. After some searching on the net.. I found the answer to this, quite understandably from one of the ruby greats JEG2 Here James clearly explains how a Struct.new returns a Class and accepts a block of code for all the methods. Note the use of Symbol :infile in the declaration but data member infile in the perform method.
Since my file was in lib/workers/bulk_upload_worker.rb, we need to explicitly require this file for DelayedJob. At this in the config/initializers/delayed_job.rb. Now, before I can get down to brass tags and incorporate it, I really need to know if this works . First ensure that the task works — directly from console:
RAILS_ENV=production ./script/console >> task - BulkUploadWorker.new(false, 'test_file', 3) >> task.perform
Once the task performs as per expectation, start up delayed_job server and test it out from your Web App. If there are errors or exception, delayed_job stores this in the database. So, its a good idea for command-line users with access to the server, to keep an eye out for errors and exceptions in the database.
Enjoy!
