Migrating Acemoney onto a different server with nginx+passenger

Acemoney is a hosted application built by us i.e. Josh Software. Currently, its hosted on a linode with nginx+thin configured. The problem here is that there are 3 thin servers which consume humongous ‘stagnant’ memory. We have decided to migrate to nginx+passenger so that we an control in greater detail the number of instances, the memory and the performance.

Some things in the post are specific to Josh Software and its client. Overall, this should give a good clean idea about migrating to passenger.

1. Checkout from the branch from the respository (the 2.3.4 version)

2. Ensure  Rails 2.3.4 is installed

3. Edit the nginx configuration. /opt/nginx/conf/servers/acemoney.in

server {

 listen 80;
 server_name acemoney.in;
 passenger_enabled on;
 root <path-to-deployment>/acemoney/public;
}

4. Restart Nginx. In case you get an error — something like:

2009/10/27 16:02:07 [error] 32685#0: *9 directory index of
"/home/gautam/deployment/acemoney/public/" is forbidden, client: 121.247.65.47,
server: acemoney.in, request: "GET / HTTP/1.1", host: "acemoney.in"

Check syntax in the conf file (I had forgotten a ‘;’) OR check permissions of the root directory to see if you have given r+x permissions

5. The current gem setup was:

$:/opt/nginx/conf/servers$ sudo gem search

*** LOCAL GEMS ***

abstract (1.0.0)
actionmailer (2.3.4, 2.3.3, 2.2.2, 2.1.2)
actionpack (2.3.4, 2.3.3, 2.2.2, 2.1.2)
activerecord (2.3.4, 2.3.3, 2.2.2, 2.1.2)
activeresource (2.3.4, 2.3.3, 2.2.2, 2.1.2)
activesupport (2.3.4, 2.3.3, 2.2.2, 2.1.2)
capistrano (2.5.5)
cgi_multipart_eof_fix (2.5.0)
chronic (0.2.3)
contacts (1.0.13)
daemons (1.0.10)
engineyard-eycap (0.4.7)
erubis (2.6.4)
eventmachine (0.12.6)
fastthread (1.0.7)
gem_plugin (0.2.3)
heywatch (0.0.1)
highline (1.5.0)
hoe (1.12.2)
json (1.1.6)
memcache-client (1.7.2)
mislav-will_paginate (2.3.10)
mongrel (1.1.5)
mongrel_cluster (1.0.5)
mysql (2.7)
net-scp (1.0.2)
net-sftp (2.0.2)
net-ssh (2.0.11)
net-ssh-gateway (1.0.1)
packet (0.1.15)
passenger (2.2.5)
rack (1.0.0, 0.9.1)
rails (2.3.4, 2.3.3, 2.2.2, 2.1.2)
rake (0.8.4)
RedCloth (4.1.9)
right_aws (1.10.0)
right_http_connection (1.2.4)
rubyforge (1.0.3)
rubyist-aasm (2.0.5)
thin (1.0.0)
tidy (1.1.2)
xml-simple (1.0.12)

So, I had to add the following gems:

$ sudo gem install acts_as_reportable

This added the following dependencies:

Successfully installed fastercsv-1.2.3
Successfully installed archive-tar-minitar-0.5.2
Successfully installed color-1.4.0
Successfully installed transaction-simple-1.4.0
Successfully installed pdf-writer-1.1.8
Successfully installed ruport-1.6.1
Successfully installed acts_as_reportable-1.1.1
Successfully installed json_pure-1.1.9
Successfully installed rubyforge-2.0.3
Successfully installed rake-0.8.7

$ sudo gem install prawn
$ sudo gem install ferret
$ sudo gem install acts_as_ferret

6. Then I got the latest database dump from Acemoney server and configured the database locally. Edit the config/database.yml and the following:

production:
  adapter: mysql
  user: acemoney
  password: <password>
  host: localhost

Created a new user in mysql and granted ALL  permissions to acemoney database

mysql> grant all on acemoney.* to 'acemoney'@'localhost' identified by '<password>'

Then create the database and dump the contents from the backup

$ RAILS_ENV=production rake db:create
$ mysql -uacemoney acemoney -p < <backupfile>

Start the ferret_server

$ ./script/ferret_server -eproduction start

Build the index:

$ RAILS_ENV=production rake ace:rebuildFerretIndex

NOW, we are good to go. Make some local changes in /etc/hosts file on your machine to point acemoney.in to the linode IP address. Then http://acemoney.in should take you to the hosted application on nginx+passenger.

Musings on cache-money – Part I

So, I always wanted to find a way of memcaching via ActiveRecord, without having to re-invent the wheel 😉 My investigations initially took me via CachedModel, cache_fu and finally I settled on cache-money. Seems to be *almost exactly* what I wanted – any lookup goes via memcache, any update /edit goes via ActiveRecord + memcache and then if needed to the database.

This is the first of my stunts:

1. Create a basic rails project with a simple posts controller. Being as lazy as I am, I used the ./script/generate scaffold help contents to create my Posts controller! 😉

2. Install the cache-money gem

3. Install memcachd server and configure the rails project (all from the github README of cache-money)

— config/memcache.yml —

production:
    ttl: 604800
    namespace: 'josh1'
    sessions: false
    debug: false
    servers: localhost:11211

— environments/production.rb —

# Use a different cache store in production
config.cache_store = :mem_cache_store
memcache_options = {
   :c_threshold => 10000,
   :compression => true,
   :debug => false,
   :namespace => 'josh1',
   :readonly => false,
 :urlencode => false
}

# require the new gem, this will load up latest memcache
# instead of using the built in 1.5.0
require 'memcache'

# make a CACHE global to use in your controllers instead of
# Rails.cache, this will use the new memcache-client 1.7.2
CACHE = MemCache.new memcache_options

# connect to your server that you started earlier
CACHE.servers = '127.0.0.1:11211'

# this is where you deal with passenger's forking
begin
 PhusionPassenger.on_event(:starting_worker_process) do |forked|
 if forked
 # We're in smart spawning mode, so...
 # Close duplicated memcached connections - they will open themselves
 CACHE.reset
 end
end

# In case you're not running under Passenger (i.e. devmode with mongrel)
rescue NameError => error
end

— config/initializers/cache_money.rb

require 'cache_money'

config = YAML.load(IO.read(File.join(RAILS_ROOT, "config",
                                     "memcached.yml")))[RAILS_ENV]
$memcache = MemCache.new(config)
$memcache.servers = config['servers']

$local = Cash::Local.new($memcache)
$lock = Cash::Lock.new($memcache)
$cache = Cash::Transactional.new($local, $lock)

class ActiveRecord::Base
 is_cached :repository => $cache
end

4. Benchmarking – Instead of running benchmarking, I wrote a few lines of code myself to create a 1000 posts with random text ranging from 1 to 100000 letters.

Setup: Mac OS 1.5 (Leopard) with nginx + passenger + memcached on a MacBook Pro laptop. I used Ruby 1.8.6 (default Mac OS 1.5 Version) and Rails 2.3.3

[term1] $ memcached -uroot -vv

$ ./script/console production
Loading production environment (Rails 2.3.3)
>>  alphanumerics = [('0'..'9'),('A'..'Z'),('a'..'z')].map {
?>       |range| range.to_a}.flatten
=> ["0", "1", "2", "3", "4", "5", "6", "7", "8", "9", "A", "B", "C", "D",
"E","F", "G", "H", "I", "J", "K", "L", "M", "N", "O", "P", "Q", "R", "S",
"T", "U", "V", "W", "X", "Y", "Z", "a", "b", "c", "d", "e", "f", "g", "h",
"i", "j", "k", "l", "m", "n", "o", "p", "q", "r", "s", "t", "u", "v", "w",
"x", "y", "z"]
>>  1000.times do |i|
?>    Post.create(:title => i.to_s, :body => (0...100000).map {
?>      alphanumerics[Kernel.rand(alphanumerics.size)] }.join)
>>  end

Results & Analysis

1. Memcache quickly scaled upto 64MB (default setting) which was cool. I could see the memcache screen scrolling fast to update all objects – in reality it will tries an LRU style purging and gets rid of the oldest objects once its actual memory is ‘full’. Memcache did not crash or stall or thrash — which was great!

2. For each object created in cache, we see the following memcache update:

<22 add lock/Post:1/id/15 0 30 6
>22 STORED
<22 set Post:1/id/15 0 86400 280
>22 STORED
<22 delete lock/Post:1/id/15 0
>22 DELETED

A GET for any posts results in:

<23 get Post:1/id/15
>23 sending key Post:1/id/15
>23 END

An UPDATE / CREATE / DELETE for any posts results in:

<23 add lock/Post:1/id/15 0 30 6
>23 STORED
<23 set Post:1/id/15 0 86400 361
>23 STORED
<23 delete lock/Post:1/id/15 0
>23 DELETED

Excellent!!

2. To confirm if we were getting it right, I checked the logs:

First time for GET:

Processing PostsController#show (for 127.0.0.1 at 2009-10-10 16:06:05) [GET]
 Parameters: {"id"=>"15"}
Rendering template within layouts/posts
Rendering posts/show
Completed in 5ms (View: 2, DB: 95) | 200 OK [http://josh1.local/posts/15]

Second Time for the same GET request:

Processing PostsController#edit (for 127.0.0.1 at 2009-10-10 16:06:42) [GET]
 Parameters: {"id"=>"15"}
Rendering template within layouts/posts
Rendering posts/edit
Completed in 6ms (View: 5, DB: 0) | 200 OK [http://josh1.local/posts/15/edit]

Excellent!!

2. Just to push the pedal, I ran another iteration of 1000 posts with random body text and saw that memcache memory stayed put at 62-64mb. I could see expired cache objects hitting the database and get cached and objects already in the cache NOT hitting the database. Exactly what I wished for.

Some caveats:

1. a. Every ‘find’ request hits the database. Understandable but I wonder if this too can hit via ‘cache’ – its not safe or synch’ed but I wonder.
2. money-cache still does not support joins or includes or nested attributes. (Time to contribute!! )

Overall: VERY VERY GOOD

Next Steps in Part II:

• Test money-cache on a live project with about 1 million records.
• It has currently ~35 Requests Per Minute.
• Some controllers calls take almost 50% of the time. Gotta reduce that to 5% (hopefully)

Cheers!

Phusion Passenger on Nginx – Internal Overview

So, working with nginx and passenger has been really simplified. There is an excellent screen cast about how exactly how to get it working. (http://www.modrails.com/videos/passenger_nginx.mov). What I was really interested in finding out was, what happens under covers.

It turns out that this is the core of what passenger does is irrespective of whether its running via Apache or Nginx. This was interesting as, I believe this is one step we take for granted!

When passenger_enabled is ‘on’, the Rails (railz in Passenger terminology) or the Rack, the processes are not exec’ed but forked! This is HUGE – unlike mongrel or thin. Processes have a different process id but they are child processes – ie. They share all the parents file descriptors. NOW, since we can fork on demand, it makes a huge difference when we are trying to balance the load on servers and reap them on lower load.

There are different types of ways the server is spawned:

• conservative

• smart

• smart lv2 (default)

The conservative method of spawning is the same as what happens with mongrel and thin – the processes are independent. Not recommended at all as this kills the performance.

Smart and smart lv2 are the pre-cached spawners. In the smart way, the framework is spawned and all applications of this framework are spawned. Slightly heavy considering that we do not deploy like this usually.

Smart lv2 is the best of the lot and does a cached application loading – this reduces the load time when the application is requested even for the first time. This is the one we shall dissect.

Questions which arose:

1. Where is the ruby process?

2. How much memory does it take?

3. What is the load time?

If its a smart spawn, I would assume that since one process is forked for every framework application, there should be only one ruby process for rails and one for rack. Need to really investigate this 😉

If its a smart-lv2 spawner, I assumed that there should be at least one ruby process for every application we spawn – however, this is not the case somehow. At all times, I see only one ruby process – how? I created 2 applications and tried to start them from a single nginx server – still only one ruby process – that too consuming 6MB in resident memory. What’s going on here??

Now, I created 2 basic scaffolds, one for each application and invoked the routines – voila, here is my answer! I see 4 ruby processes of 21MB each for 2 applications. So, this explains it – its not rocket science.

Initially, since there are no HTTP requests processed – just the basic application and welcome to rails page, only a single application got spawned. Hence it was only 6MB. When there are some HTTP rails requests, the rails apps were spawned. To verify this:

• I killed the nginx process and the ruby child processes disappeared – expected.

• I restarted nginx and invoked form browser. Only the welcome page – no ruby process spawned

• I invoked a rails request on 1 of the apps – 2 ruby processes (21mb) spawned.

• I invoked a rails request on the second apps – 2 more ruby processes surfaced. Passenger-status shows 2 domains and 1 PID (probably implies 1 process and 1 child worker thread?)

Now, the question remains – when are the processes reaped – is there always 2 instances of the app in memory – which is still huge as each consumes 21mb! Now, I was trying find out if and when the instances get reaped – Still doing some investigation on this.

More later!