« RailsConf this week | Main | On Ruby Interview with me »

May 07, 2009

Comments

Julien

Good Good Good! Could you compare the performance with something like em-http-request?

Thanks

ehsanul

Should be compared with curl-multi for performance, since it sends multiple requests as well, Net::Http isn't a good comparison IMO. But curl-multi doesn't have the functionality of this I'm assuming, doesn't implement many of libcurl's offerings, so I'm sure this will be quite worthwhile even if the speed improvement isn't big. :)

Max

Very COOL!

But why:

class Twitter
include Typhoeus

for me, only work:

class Twitter
require "Typhoeus"
require 'json'
include Typhoeus

Todd A. Fisher

Nice work, but no credit to curb? The multi code looks familiar and the comments point to curb still ;-) I like the typhoeus_easy.c implementation very clean and easy to follow!

Todd A. Fisher

Reminds me of my profiling of Net/HTTP and curb with multi patch applied... see this: http://www.idle-hacking.com/2008/07/updated-curb-multi-interface-patch/ ?

Vishnu Gopal

Would be nice if HTTParty could use this - that has a way cleaner interface imho.

Glenn Gillen

Which version of ruby is this? I believe the Net::HTTP performance is significantly improved in 1.9

Alistair Holt

Wow. I love this. Great job Paul.

Sean

Have you changed out Feedzirra's backend with this? Looks really sweet...

Paul Dix

ehsanul, I didnt' run a comparison against the multi because of it lacking many of these features.

julien, em-http-request is probably as fast. However, I wasn't able to get it to support everything that libcurl does.

Glenn, I tested against 1.8.7. Even in 1.9 I doubt it would compare. The problem would be with blocking IO. Although it would be interesting to test against a NET::HTTP Fibers implementation for parallelism.

Sean, I want to update Feedzirra to use this. I'll hopefully get to that in the next week or so.

BigLove

(Sorry if this somehow ends up being posted multiple times... The commenting wasn't quite working for me.)

Paul,
You've obviously done some great work here, and your interface to the core libraries are fantastic and look like a joy to use. I must admit, though, that I'm a little confused as to why you've compared your parallel request benchmarks to a single threaded, blocking request benchmark?

I suppose I mean to ask, do we even need to benchmark these things to know they're faster? As has been previously mentioned by you and ehsanul, it would be a much more impressive comparison if your implementation was faster than say, Ruby 1.8.7's green threads and having a request in each thread... or having say, Ruby 1.9's fibers in such a scenario (as you mentioned specifically).

I say all that because my current project is calling out to six or seven different web-services, and I've had to implement my own green threading solution with each request. I know I could probably swap out my library with your own, but I wonder a) how much of a performance increase it would give me, and b) how much work it'd be for me :P

If the performance increase is significant, then I'd have to put my nose to the grindstone and update my code with yours. Though, admittedly, I can't see how one could speed up the response time of a server :P Or how using a (probably) faster implementation of a C library instead of Ruby's NET::Http would really increase each request's performance. On that note, it would also be great to see the request time of a single request in your library compared with Ruby's Http library...

Looking forward to your thoughts.

Paul Dix

BigLove,
I suppose I should probably put together a comparison. What would be the proper ones to do? I can think of:
* EMHttpClient
* 1.8.6 Threaded
* 1.8.7 Threaded
* 1.9.1 Fibers

Running all those would be a pretty big pain in the ass though. If I have time I'd love to do that, but the real point of the benchmarks was to showcase the difference of running parallel vs. serial execution.

Konstantin Haase

Not to forget about the latest version of curb!

Andy

But it doesn't support setting custom headers, only a custom user-agent?

John W Higgins

I have a question about the copyright requirement with respect to the usage of the curb code within Typhoeus. Is there a need to explicit add the original copyright of both curb and the patch that Todd Fisher added? I ask not to be an idiot but rather to understand the requirements better in this area. Open source offers so much flexibility but I've never been sure where the lines are in terms of what you need to do to ensure proper attribution to the folks that wrote the original concepts that we then mold into our own creations.

Paul Dix

Hi John,
I don't know what the requirements are. To be honest, I used very little from curb. I definitely used a few pieces in the multi code, but almost everything is completely new.

With open source, everyone builds on everyone else. Neither of these libraries could have been written without libcurl. That couldn't have been written without gcc. I think it's just best efforts to give credit where it is due and not to rip off blatantly.

Have a look at the c code in both libraries. I think you'll find that they are quite different. However, I definitely couldn't have made typhoeus without the help of looking at the curb code.

Chris

So we just started using this in a fairly high volume production env, and we are seeing posts instantly timeout (response code 0), regardless of what the timeout is set to. We built it against the latest libcurl source with --enable-ares and --enable-nonblocking. Seems to happen every 50 or so requests. Has anyone else run into this?

Paul Dix

Hey Chris,
I assume this was you that posted on the Typhoeus list, but just in case it wasn't, you can join in on the thread about this issue here:
http://groups.google.com/group/typhoeus/browse_thread/thread/924d460ebbb240ac

The comments to this entry are closed.

My Photo

Talks

Linkage

Twitter / pauldix