Paul Dix Explains Nothing

Feedzirra release adds simple custom parsing and more!

I finally got around to handling the pull requests and various feature requests people have sent me over the last few weeks. The main new features are: adding new parsing elements across all feeds, ids and guids now get parsed, modified now gets parsed, http basic authentication is now supported, itunes rss is now a supported format, and some zlib errors people were having should now be gone.

Also, when a parser is not available it now calls the onfailure method instead of throwing an error. This is better behavior when you're getting hundreds of feeds and want it to continue without stopping the others.

Parsing of the id/guid was a real debate for me. I intentionally left that out of the library. I had two reasons for this. First, there are many feeds that don't have ids or guids in them. Second, the permalink of the entry should act as a unique identifier. I felt that having another id to track was wasteful and could cause errors. However, enough people have asked about the feature that I've decided to include it in this release.

I also had to do a small release of sax-machine to fix some parsing behavior. The feedzirra gemspec should automatically pull that one down.

Here's what all those new features look like in code:

# access an http basic feed. You can't do this in a bulk feed get. It has to be when getting only one.
Feedzirra::Feed.fetch_and_parse(some_url, :http_authentication => ["username", "password"])

# You can add custom parsing to the feed entry classes. Say you want the wfw:comments fields in an entry
Feedzirra::Feed.add_common_feed_entry_element("wfw:commentRss", :as => :comment_rss)
# The arguments are the same as the SAXMachine arguments for the element method. For more example usage look at the RSSEntry and
# AtomEntry classes. Now you can access those in an atom feed:
Feedzirra::Feed.parse(some_atom_xml).entries.first.comment_rss_ # => wfw:commentRss is now parsed!

# and the new accessors for feed entries
entry.id  # => the atom id or rss guid
entry.modified # => the modified time of the post

Most of these features were thanks to outside contributors. In fact, the only one I did was for adding custom parsing across feed entries. I'd like to thank the following github users for helping out!

• Hamish Rickerby for the itunes stuff
• Daniel Insley for rdoc
• Alexey Dmitriev for the deflate fix
• Nelson Morris for refactoring the tests to mock.
  
• Julien Genestoux for the id, modified, and updated publishing parsing. Also new bahavior on no parser available.

Sorry if I forgot anyone. Please yell at me if I haven't pulled in changes that you think should be included.

Service Oriented Architectures in Rails

Service oriented architectures (SOA) are defined by Wikipedia as "systems [that] group functionality around business processes and package these as interoperable services." In more practical terms this means setting up various services to build out a full system. In the web 2.0 world people tend to think of SOAs as building mashups with public APIs from Google, Twitter, New York Times, Netflix, and countless others. However, more traditional SOAs are built internally for doing things like connecting with legacy systems and sprinkling magic scaling fairy dust to create "enterprise" systems.

One of the most visible examples of this kind of SOA is Amazon.com's architecture, which makes 100-150 different service requests to build a single page. Other high profile examples include LinkedIn's architecture and eBay's scalability best practices. This kind of thing is common in the Java world, but I've heard very little about developers within the Rails community building service architectures. The closest we've come is using message queues for background work, memcached, and sometimes a full text search engine (read: Solr, Sphinx, etc.)

I think the reason for the lack of services architectures in the Ruby community are twofold. First, Rails encourages a monolithic application architecture. Everything is contained within a single application code base. While this enables super fast iteration and development in the beginning, it can get messy and unwieldily later on (just ask anyone that has to wait 15 minutes for their test suite to run or who has had to refactor a big mess of code that contains all sorts of interdependencies.) The second reason is a shortcoming of the Ruby implementation. In Ruby it's impossible to run multiple requests in parallel. You could try threading, but not only will the Ruby threading model kill you, blocking IO will stop your interpreter cold as well.

Let's look at some numbers to put this into perspective. Let's say you have a request that calls out to 10 services to render a page. Further, these services can take anywhere from 100-150 ms to respond. In the regular Ruby world, those service requests would take 1 to 1.5 seconds to return. Inside of the request response life-cycle, this just isn't feasible. That's like making a call to that database that takes 1.5 seconds to come back. It just doesn't work. In the Java parallel world those 10 requests will return in no more time than the longest running (150 ms). With this model it suddenly becomes possible to make multiple service requests inside of the client's request/response chain.

HTTPMachine, a new Ruby library I'm writing for interacting with service architectures addresses the problem of running many requests in parallel. As a test for this I created a local evented http server that simply sleeps for 100ms and then returns a 60 line xml response. Here's some benchmark code that hits the server to test out speeds (in gist form):

calls = 10
@klass = Class.new do
 include HTTPMachine
end
benchmark do |t| 
 t.report("httpmachine") do
 HTTPMachine.service_access do 
 calls.times do
 s = nil
 @klass.get("http://127.0.0.1:3000") do |response_code, response_body|
 s = response_body
 end
 end
 end
 end

 t.report("net::http") do
 calls.times do
 s = open("http://127.0.0.1:3000").read
 end
 end
end

And finally, here are the results of the benchmark:

httpmachine 0.010000 0.000000 0.010000 ( 0.105024)
net::http 0.010000 0.010000 0.020000 ( 1.027697)

This is only the beginning of this library. It's only the test to prove out the concept that making multiple service calls within a client request is feasible. Now that I have that out of the way I'll start building up the other pieces to make a full services architecture a snap to build out. I'd really like to get feedback for the API and some possible use cases of this library. For pure Ruby environments I see something like a bunch of Sinatra services being called inside a main Rails application server. For my environment at work, it's a main Rails application server calling out to a bunch of services written in Java, Python, and C++.

Here's an example of a more fully fleshed out use:

class Post
 include HTTPMachine
 remote_method :search, :server => "http://localhost/search", :reponse_format => :json
end

# and using it
Post.search(params) # => returns fully parsed post results from the search

Thoughts?