Figuring Out Where Ruby is Spending My Memory

Yesterday I mentioned that Starfish and ActiveRecord were eating up memory. Today I decided to dig a little deeper and try to find exactly where my memory is being spent.  If I'm going to figure out how to run as many Ruby processes as possible on a single box I need to know the details. So I just wrote a couple of test Ruby scripts that each run a require on some piece of code that I use and then sleep so I can check memory usage.

I'm just using the command

ps v PID

Where PID is the process id. Here are the stats using the RSS number, which is basically the real or resident memory size in kb of the process (more info on determining memory usage with ps). I'm listing the figures in MB to make it a little more readable. The requires of each test are listed next to the figures.

• 1.36 MB - no requires
• 3.18 MB - require 'open-uri'
• 4.35 MB - require 'rubygems'
  
• 9.23 MB - require 'rubygems'; require 'hpricot'; require 'open-uri'
• 17.82 MB - require 'rubygems'; require 'feed_tools'
• 20.43 MB - require 'config/environment' # RAILS_ENV = 'production'
• 27.15 MB - # all of above except for feed_tools
• 28.80 MB - all of the above

So the Rails environment is certainly a memory hog, but it's not the only culprit. The unfortunate thing for me is that at a minimum for these processes to be useful, I have to run open-uri, rubygems, and hpricot. That means at least 9.23 MB per process and that's before I figure out how to offload their results (database, file, whatever). Doing that would require me writing a bunch of custom code instead of taking advantage of ActiveRecord and FeedTools' ease of use.

All of this is pointing me to the issue that one tiny 256 MB VPS may not be enough for what I want to do...


Technorati Tags: ruby, rubyonrails, ps, memory

Comments

[Lourens Naude]

Hi,

Nice roll of posts :-)

I published a case study some time ago about general Ruby / Rails memory usage : http://blog.methodmissing.com/2007/5/10/rails-memory-usage-case-study/

a) Kudos for decoupling your environment from your MVC stack ...

b) I recently wrote a video transcoder for a client using starfish, with a custom Directory mapreduce type.The first process fires your Drb server up and subsequent clients generall have a 30MB RSS overhead.Not bad at all, considering that potentially scales across your whole subnet to 2 ... X hosts.

c) Dont waste time on designing against your VPS memory limitation, opt for a entry / mid level box with 1 to 2 GB RAM ( there's good deals out there ) which would be sufficient for Production ... and experiments ... off course :-)

[Dr Nic]

It might be interesting to look at running these over jruby, where you can get reuse of the code with free java threading for each jruby instance. It might cost a bunch of memory to get started (JVM + libraries) but you wouldn't need to duplicate this for each "ruby process", I don't think.

That's my understanding anyway. If you investigate it I look forward to the results.