Home > Ruby On Rails > Delayed_job for background processing in Rails

Delayed_job for background processing in Rails

The first thing to do obviously is to install DelayedJob. There are plenty of forked versions available on git-hub. I chose collectiveidea beacuse it was recommend on railscasts. I did refer to this site extensively for setting up delayed_job.

$ sudo gem install collectiveidea-delayed_job

Job half done. I followed instructions on the github page and ensure that I have my environment setup properly. I added the following to the environment.rb:

 config.gem 'collectiveidea-delayed_job', :lib => 'delayed_job',
                     :source => 'http://gems.github.com'

and the following to the Rakefile:

begin
      require 'delayed/tasks'
rescue LoadError
      STDERR.puts "Run `rake gems:install` to install delayed_job"
end

Then issue the command:

./script/generate delayed_job
rake db:migrate

To start the delayed_job server in production mode, issue:

$ RAILS_ENV=production ./script/delayed_job start

To start it in development mode, issue:

$ rake jobs:work

Interesting find – obvious but cost me a lot of time: Nothing stops one from running BOTH the above commands. Infact, in production mode I ran the delayed_job as a daemon AND also using rake. Silly me – I forgot that if I change any code I would need to restart both. I wrongly assumed that jobs:work only ‘showed’ the console — it starts the delayed_job server. So, if you do it this way, you will have twice the number of background jobs floating around 😉

Well, anyway, once I had this properly configured as a daemon, I set about changing code. This was the really aweome part of DelayedJob — no dependencies!! I changed the code from:

user = User.find_by_name('xyz')
user.get_daily_call_statistics

to

user = User.find_by_name('xyz')
user.send_later(:get_daily_call_statistics)

AND IT WORKS !! Awesome.  While digging around a little more, I realized the there is not enough scope for debugging in the default way. I looked up github and google for some help and found this useful tip:

1. Create a config/initializers/delayed_job_config.rb and add:

Delayed::Job.destroy_failed_jobs = false
Delayed::Worker.logger = Rails.logger

This logs all delayed job output to the environment log files and ensures that failed jobs are not destroyed. There are other settings to reduce the failure attempts and the time for the delayed job but I was too excited to try them out immediately.

2. Suppose I have some global variables in the helpers, the are not accessible in the model methods called via delayed_job. Maybe a bug in delayed_job – I do plan to dig deeper into this and figure this one out — either way. I had to break my head trying to figure this one out.

To conclude, what I had earlier was:

Processing DashboardController#explicit_refresh_daily_statistics (for 121.247.65.47 at 2009-10-29 13:16:33) [GET]
Parameters: {"action"=>"explicit_refresh_daily_statistics", "controller"=>"dashboard"}
last seen ...........
Redirected to http://acemoney.in/dashboard
Completed in 74414ms (DB: 16912) | 302 Found [http://acemoney.in/dashboard/explicit_refresh_daily_statistics]

Now after adding the send_later, I have:

Processing DashboardController#explicit_refresh_daily_statistics (for 121.247.65.47 at 2009-10-29 15:16:41) [GET]
Parameters: {"action"=>"explicit_refresh_daily_statistics", "controller"=>"dashboard"}
last seen ...........
Redirected to http://acemoney.in/dashboard
Completed in 420ms (DB: 99) | 302 Found [http://acemoney.in/dashboard/explicit_refresh_daily_statistics]

This means my response time fell from from 74 seconds to 0.5 seconds

Now, I already had backgrounDrb tasks configured earlier and want to migrate them ‘somehow’ to DelayedJob with minimal code. Stay tuned, this post will be updated.

Advertisements
  1. November 4, 2009 at 9:59 am

    I have added a new post for Moving from backgrounDrb to DelayedJob

  2. Sachin
    January 21, 2011 at 3:35 pm

    Hi, I am trying to implement delayed job using mongoid, i have done a simple code for that. When i run rake jobs:work,i can see “1 jobs processed at 94.3685 j/s, 0 failed …” BUT when i check the database, it shows

    “{Job failed to load: uninitialized constant FirstJob. Handler: \”— !ruby/struct:FirstJob \\nmailing_id: 12\\n\”\u000a/home/sachin/.bundler/ruby/1.9.1/delayed_job-6c11015334aa/lib/delayed/backend/base.rb:81:in `rescue in payload_object’\\n/home/sachin/.bundler/ruby/1.9.1/delayed_job-6c11015334aa/lib/delayed/backend/base.rb:79:in `payload_object’\\n/home/sachin/.bundler/ruby/1.9.1/delayed_job-6c11015334aa/lib/delayed/backend/base.rb:87:in `invoke_job’\\n/home/sachin/.bundler/ruby/1.9.1/delayed_job-6c11015334aa/lib/delayed/worker.rb:120:in `block (2 levels) in run’\\n/usr/local/ruby/lib/ruby/1.9.1/timeout.rb:57:in `timeout’\\n/home/sachin/.bundler/ruby/1.9.1/delayed_job-6c11015334aa/lib/delayed/worker.rb:120:in `block in run’\\n/usr/local/ruby/lib/ruby/1.9.1/benchmark.rb:309:in `realtime’\\n/home/sachin/.bundler/ruby/1.9.1/delayed_job-6c11015334aa/lib/delayed/worker.rb:119:in `run’\\n/home/sachin/.bundler/ruby/1.9.1/delayed_job-6c11015334aa/lib/delayed/worker.rb:177:in `reserve_and_run_one_job’\\n/home/sachin/.bundler/ruby/1.9.1/delayed_job-6c11015334aa/lib/delayed/worker.rb:104:in `block in work_off’\\n/home/sachin/.bundler/ruby/1.9.1/delayed_job-6c11015334aa/lib/delayed/worker.rb:103:in `times’\\n/home/sachin/.bundler/ruby/1.9.1/delayed_job-6c11015334aa/lib/delayed/worker.rb:103:in `work_off’\\n/home/sachin/.bundler/ruby/1.9.1/delayed_job-6c11015334aa/lib/delayed/worker.rb:78:in `block (2 levels) in start’\\n/usr/local/ruby/lib/ruby/1.9.1/benchmark.rb:309:in `realtime’\\n/home/sachin/.bundler/ruby/1.9.1/delayed_job-6c11015334aa/lib/delayed/worker.rb:77:in `block in start’\\n/home/sachin/.bundler/ruby/1.9.1/delayed_job-6c11015334aa/lib/delayed/worker.rb:74:in `loop’\\n/home/sachin/.bundler/ruby/1.9.1/delayed_job-6c11015334aa/lib/delayed/worker.rb:74:in `start’\\n/home/sachin/.bundler/ruby/1.9.1/delayed_job-6c11015334aa/lib/delayed/tasks.rb:9:in `block (2 levels) in ‘\\n/usr/local/ruby/lib/ruby/1.9.1/rake.rb:634:in `call’\\n/usr/local/ruby/lib/ruby/1.9.1/rake.rb:634:in `block in execute’\\n/usr/local/ruby/lib/ruby/1.9.1/rake.rb:629:in `each’\\n/usr/local/ruby/lib/ruby/1.9.1/rake.rb:629:in `execute’\\n/usr/local/ruby/lib/ruby/1.9.1/rake.rb:595:in `block in invoke_with_call_chain’\\n/usr/local/ruby/lib/ruby/1.9.1/monitor.rb:201:in `mon_synchronize’\\n/usr/local/ruby/lib/ruby/1.9.1/rake.rb:588:in `invoke_with_call_chain’\\n/usr/local/ruby/lib/ruby/1.9.1/rake.rb:581:in `invoke’\\n/usr/local/ruby/lib/ruby/1.9.1/rake.rb:2041:in `invoke_task’\\n/usr/local/ruby/lib/ruby/1.9.1/rake.rb:2019:in `block (2 levels) in top_level’\\n/usr/local/ruby/lib/ruby/1.9.1/rake.rb:2019:in `each’\\n/usr/local/ruby/lib/ruby/1.9.1/rake.rb:2019:in `block in top_level’\\n/usr/local/ruby/lib/ruby/1.9.1/rake.rb:2058:in `standard_exception_handling’\\n/usr/local/ruby/lib/ruby/1.9.1/rake.rb:2013:in `top_level’\\n/usr/local/ruby/lib/ruby/1.9.1/rake.rb:1992:in `run’\\n/usr/local/ruby/bin/rake:31:in `'”

    Not able to figure what is going where ?

  3. January 21, 2011 at 3:53 pm

    1 job processed, 0 failed — this could the error for an older delayed job.

    rake jobs:clear
    rake jobs:work

    then try this. It should work.

    PS: I have discontinued my blog here — user http://blog.joshsoftware.com instead.
    Cheers!

  4. sharespace
    October 19, 2012 at 2:56 pm

    Hi,
    I have query regarding delayed job pause process. Enqueue the job by setting the run_at time but when i started rails jobs:work it invoke that job and but as run time doesn’t match so it fails but keep retry within 1 second. So for this i have set the Delayed::Worker.const_set(“SLEEP”, 5 * 60 ) on my delayed job instance. But can’t get joy.

    Can you provide solution how to pause the job for some time? – Without using cron via ruby coding only.

    • October 22, 2012 at 11:36 am

      @sharespace – there seems to be something wrong with your run_at configuration.

      1. Can you paste the snippet of code where you configure the run_at?
      2. Workers check the database every 5 seconds, not 1 second. If the job is not scheduled to run, it will not be started.
      3. I suspect there is a mistake in your run_at parameter, which is triggering the job and you are getting a failure.

      Can you paste the failure exception?

  5. November 18, 2012 at 3:52 am

    You actually make it seem so easy with your presentation but I find this
    matter to be actually something which I think I
    would never understand. It seems too complicated and extremely broad for me.

    I’m looking forward for your next post, I’ll try to get the
    hang of it!

  6. April 23, 2016 at 3:49 pm

    I really like reading through a post that will make people think.
    Also, thank you for allowing for me to comment!

  7. June 2, 2016 at 3:54 pm

    Najlepіeј, gdy sam zobaczysz, jaką maǥię kryje w sobie CՏ – Cs 1.6 new era by lukasz v32 downloɑd.

  8. June 15, 2016 at 7:33 pm

    Hi friends, its impressive paragraph on the topic of teachingand fully defined, keep it up all the time.

  9. August 9, 2016 at 8:20 am

    An outstanding share! I’ve just forwarded this onto a colleague who has been doing a little homework
    on this. And he in fact bought me dinner simply because I found it for him…
    lol. So let me reword this…. Thanks for the meal!!
    But yeah, thanx for spending some time to talk about this topic here on your internet site.

  10. August 30, 2016 at 5:40 pm

    I enjoy what you guys are up too. Such cever wor and exposure!
    Keep up the very good works guys I’ve incorporated yyou guys to my own blogroll.

  11. September 6, 2016 at 11:46 am

    magnificent issues altogether, you just gained a new reader.
    What might you suggest about your put up that you just made a few
    days in the past? Anny certain?

  12. October 2, 2016 at 8:22 am

    It’s actually a great and useful piece of info. I am satisfied that you just shared this useful info with us.
    Please stay us up to date like this. Thanks for sharing.

  13. November 14, 2016 at 10:40 am

    I do believe all of the ideas you’ve offered to your post.
    They are very convincing and can certainly work. Still, the posts are too brief for
    newbies. Could you please prolong them a bit from subsequent time?
    Thanks for the post.

  14. December 21, 2016 at 3:31 pm

    Great article.

  15. December 29, 2016 at 11:46 pm

    Excellent post. I used to be checking continuously this weblog and I am inspired!
    Very useful information specifically the ultimate phase 🙂 I deal
    with such information much. I was looking for this
    certain info for a long time. Thanks and good luck.

  16. gay
    December 30, 2016 at 12:18 am

    Excellent way of explaining, and nice article to take information about my presentation topic, which i am going
    to convey in academy.

  17. January 11, 2017 at 8:41 pm

    Having read this I thought it waѕ гeally enlightening.
    I appгeciate yoou spending sopme tkme ɑnd effort to put this
    informative article tоgether. І once again find myseⅼf personally pending ɑ lott
    of tie botҺ reading and posting comments.
    But so wɦаt, it աaѕ stіll worthwhile!

  18. computer software programs microsoft office
    April 27, 2017 at 2:19 pm

    I’d like to find out more? I’d want to find out more details.

  19. May 4, 2017 at 2:28 am

    Currently it seems like Expression Engine is the top blogging platform out there right now.
    (from what I’ve read) Is that what you’re using on your blog?

  1. November 4, 2009 at 9:56 am
  2. April 16, 2010 at 2:11 pm
  3. December 7, 2011 at 1:57 am

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: