Ruby on Rails queueing gem: Runner

4 Oct

A week or so back we started porting projects to Rails 3 and all new projects were created using Rails 3. The project that I was working on (name irrelevant) needed some sort of concurrency. In the past we already tried a few existing Gems but there was always something missing or there was simply too much of something. Is most cases it was also hard to customize the gem or it simply didn’t live up to our expectations.

In the past we had also tried to create our own concurrency implementation but these either-way failed or were just too specifically designed. If you wanted to do something else than the feature it was designed for; you would have to rewrite parts of the code. You may call this design error, but it was simply not the intention to create a flexible concurrency implementation.

So our search for a new concurrency Gem went on; we tried two and came to the conclusion we would have to write something our self. I was given the task to create a new Gem. A new concurrency (queueing) Gem which would have to be flexible, maintainable and would have to be not as specific as we have been in the past. Or in fact this concurrency Gem had to be able to do all sort of tasks. From sending out mails to importing users from a CSV file.

I’d tried delayed_jobs and liked their implementation very much. The one thing we all agreed at here at the office was we didn’t want to have a separate Rake task which would require to be run in order to do some sort of concurrency. We wanted to be able to have the task been done within the application with whatever concurrency we specified; be it Forking or Threading or a simple yield, it all had to be possible. So with delayed_job in mind I started to design the Gem and came up with the name Runner (obvious name!).

I took me two days to implement a very basic version of the Gem. The only thing it could do back then was fork a method in the background without the need of an additional Rake task. At the current time of writing it has the following features:

  • Supports various implementations of Concurrency (forking, threading and yielding. But the last isn’t really concurrency)
  • Developers using the Gem may add different kind of concurrency to gem by simply giving the concurrency class the Runner initializer
  • It has the ability to queue tasks which means you more or less park the tasks for later.
  • It may immediately run tasks in the background using the various concurrency methods.
  • Threading tasks requires no changes to your current classes.
  • Extending the backend (currently comes with ActiveRecord) only requires you to implement database specific code (scopes, etc.)

As you can see after only a couple of days of hard work it can already do quite some things for any tasks you would like to run in the background. But as you probably wonder how does it really work and what’s the catch. Well there really is no catch, you only either way create a new block which you would like to run in the background or you specify a method to be always run in the background or you may call a method on the “chainable” spawn method as you will see below in the example code.

Getting the gem

If you like to try the Gem in an existing or new Rails 3 project you could add Runner to your Gemfile like so:

gem 'runner', '0.1.4', :git => 'git://github.com/stygeo/runner.git' 

Or you could install it from the command line like:

gem install runner

The examples

Please note: the following class implementation of user will be used throughout the examples

# User.rb
class User
  def self.send_mass_mail
    # This tasks requires a lot of processing time
  end

  def say_hello
    puts "Hello"
  end

  def say_goodbye
    puts "Good bye"
  end
  handle_asynch :say_goodbye
end

You may immediately notice the handle_asynch class method. In short what this method does is it creates a new method on the class for the method you specify (in the case `say_goodbye`) which will handle all the background stuff. So if you’d do:

# example.rb
require 'user.rb'

u = User.new
u.say_goodbye

It will automatically run the method `say_goodbye` in the background in a separate process or thread. There are times you’d like to run a method in the background at all times, such as say_goodbye. But there are times you’d like to run a method depending on some condition (background or not). If this is the case you could use the method `spawn` which is usable on every class (it’s included in class Object). This is how you could use spawn:

# example.rb
u = User.new
u.spawn.say_hello

The spawn method uses a ProxyObject which creates a new PerformableMethod. PerformableMethod encapsulates your class (in this case `user`) and serializes the PerformableMethod with your class and the method you’d like to call. Once serialized it’s saved as a task in your database. Once it has been saved to the database it uses a TaskHandler which on its turn forks or threads a new process which deserialize your class’ as a PerformableMethod. PerformableMethod plays a key role in the method calling. Here’s what it holds:

  • Your class
  • The method to call (`say_hello`)

The TaskHandler calls `perform` on the deserialized object (which is a PerformableMethod) in order to run the method you specified for the class you wanted to serialize. In some pseudo code this would be: PerformableMethod (:class => user, :method => say_hello).perform { class.send(method) } This is how Runner runs the tasks in the background in a nice and convenient way.

If you’d like to “park” a task in the queue (meaning you don’t run it immediately) you could pass :queue as the :method to spawn:

u.spawn(:method => :queue).say_hello

or you could do

u.queue.say_hello

The method `queue` is just short-hand for spawn(:method => :queue). When you queue a call it gets parked in a Task which you may resume to run at a later time. For example imagine that you’d like to, after a user signs up, the user receives an instruction manual at night by email. You could queue each `mail` method and, at night, use a Rake tasks (which you could set up with cron to run at night) which handles all tasks that are currently queued.

Early I talked about the different concurrency methods you may use with Runner. Runner supports by defaults 2 real concurrency method and a way to yield. You may use a fork or threading as concurrency methods, but you’re free to improve up on it if you so wish (more on that later). To use other than the default method (defaults can be set in the Runner initializer) you could pass the :with option spawn:

# example.rb
u = User.new
u.spawn(:with => :thread).say_hello

The <code>:with => :thread</code> option forces the task to run as thread instead of the default.  If you created a new concurrency method and added it to Runner (see below) you could also use the :with to force your own concurrency. For example you created the concurrency class Fibers (ConcurrencyFibers, Concurrency prefix is required) you could pass :fibers to the :with options so it will use your ConcurrencyFibers class as concurrency handler. (Please note: This feature is not yet supported, you could however pass your newly created concurrency method as the default, which does work.)

So this is how Runner works and how you may use it. For more information about the Gem or if you have a feature request you could go see the Github page . If you’d like to know more about the Gem’s internals, check out the code from github:

git clone git://github.com/stygeo/runner.git

and start reading from the message.rb file. This is the entry point and holds the spawn method, from there on you could just follow the code.

About these ads

One Response to “Ruby on Rails queueing gem: Runner”

Trackbacks/Pingbacks

  1. Tweets that mention Ruby on Rails queueing gem: Runner « Spiced Ruby -- Topsy.com - 4 October, 2010

    [...] This post was mentioned on Twitter by maranh, Ruby on Rails UK and RubyOnRails Ireland, Jeffrey Wilcke. Jeffrey Wilcke said: Ruby on Rails queueing gem: Runner: http://wp.me/pWhPu-3f [...]

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

Follow

Get every new post delivered to your Inbox.

%d bloggers like this: