I was working on a project looking to see if Ruby was good enough for responding quickly to HTTP requests. Good thing it, along with Python, and every other language, plays well with C/C++. Anyways, EventMachine apparently blows away Tornado and Twisted. I only tested Tornado because it’s faster, right? What I really wanted to test was if either of these would fall apart under high concurrency or load. For the “Hello World!”, they both survived although as you can see for Tornado, response times became an issue earlier. I’ve also provided ‘ab’ for reference – it’s a little more specific with regard to response times. Clearly both of these are hitting a CPU ceiling – with Tornado hitting it faster. Ftr, I tested on a dual-core 2.33ghz xeon w/RHEL5, python2.6, and ruby1.8.5.

Along my adventure in this hnews thread, I came along this most awesome post: Twisted vs. Tornado: You’re Both Idiots

Anyways, what I’m happy about is there’s a Ruby option for a fast little server which pumps out a bajillion requests per second if you’ve got a farm of servers and it won’t fall on it’s face. Also, I don’t have to use Python and EventMachine is a BREEZE to use. What does suck is the EM HTTP server isn’t RFC compliant but that’s probably just a matter of time and I won’t be using HTTP anyways. ymmv

httperf: Tornado

[root@mail ~]# httperf --port=3002 --num-conns=1000 --num-calls=500 --rate 100 -v
httperf --verbose --client=0/1 --server=localhost --port=3002 --uri=/ --rate=100 --send-buffer=4096 --recv-buffer=16384 --num-conns=1000 --num-calls=500
httperf: maximum number of open descriptors = 1024
reply-rate = 5045.8
reply-rate = 4868.5
reply-rate = 4905.4
reply-rate = 4846.9
reply-rate = 4938.4
reply-rate = 4747.3
reply-rate = 4800.2
reply-rate = 4795.6
reply-rate = 4595.3
reply-rate = 4591.1
reply-rate = 4784.6
reply-rate = 4775.9
reply-rate = 4563.3
reply-rate = 4872.3
reply-rate = 4948.8
reply-rate = 4853.0
reply-rate = 4551.3
reply-rate = 4587.3
reply-rate = 4885.7
reply-rate = 4900.2
Maximum connect burst length: 1
Total: connections 1000 requests 500000 replies 500000 test-duration 104.059 s
Connection rate: 9.6 conn/s (104.1 ms/conn, <=1000 concurrent connections)
Connection time [ms]: min 34704.2 avg 93867.4 max 97177.5 median 95862.5 stddev 6293.6
Connection time [ms]: connect 0.0
Connection length [replies/conn]: 500.000
Request rate: 4805.0 req/s (0.2 ms/req)
Request size [B]: 62.0
Reply rate [replies/s]: min 4551.3 avg 4792.8 max 5045.8 stddev 144.4 (20 samples)
Reply time [ms]: response 187.7 transfer 0.0
Reply size [B]: header 156.0 content 12.0 footer 0.0 (total 168.0)
Reply status: 1xx=0 2xx=500000 3xx=0 4xx=0 5xx=0
CPU time [s]: user 2.97 system 99.60 (user 2.9% system 95.7% total 98.6%)
Net I/O: 1079.2 KB/s (8.8*10^6 bps)
Errors: total 0 client-timo 0 socket-timo 0 connrefused 0 connreset 0
Errors: fd-unavail 0 addrunavail 0 ftab-full 0 other 0

httperf: Ruby EventMachine

[root@mail ~]# httperf --port=3001 --num-conns=1000 --num-calls=500 --rate 100 -v
httperf --verbose --client=0/1 --server=localhost --port=3001 --uri=/ --rate=100 --send-buffer=4096 --recv-buffer=16384 --num-conns=1000 --num-calls=500
httperf: maximum number of open descriptors = 1024
reply-rate = 11631.7
reply-rate = 9769.5
reply-rate = 9352.3
reply-rate = 10086.1
reply-rate = 8899.4
reply-rate = 9759.3
reply-rate = 9985.1
reply-rate = 10152.8
reply-rate = 10383.9
Maximum connect burst length: 1
Total: connections 1000 requests 500000 replies 500000 test-duration 49.590 s
Connection rate: 20.2 conn/s (49.6 ms/conn, <=984 concurrent connections)
Connection time [ms]: min 229.8 avg 39130.7 max 42870.4 median 41409.5 stddev 6775.5
Connection time [ms]: connect 0.0
Connection length [replies/conn]: 500.000
Request rate: 10082.7 req/s (0.1 ms/req)
Request size [B]: 62.0
Reply rate [replies/s]: min 8899.4 avg 10002.2 max 11631.7 stddev 756.9 (9 samples)
Reply time [ms]: response 78.3 transfer 0.0
Reply size [B]: header 65.0 content 12.0 footer 0.0 (total 77.0)
Reply status: 1xx=0 2xx=500000 3xx=0 4xx=0 5xx=0
CPU time [s]: user 2.20 system 46.84 (user 4.4% system 94.4% total 98.9%)
Net I/O: 1368.7 KB/s (11.2*10^6 bps)
Errors: total 0 client-timo 0 socket-timo 0 connrefused 0 connreset 0
Errors: fd-unavail 0 addrunavail 0 ftab-full 0 other 0

ab: Tornado

[root@mail ~]# ab -c1000 -n100000 http://127.0.0.1:3002/
This is ApacheBench, Version 2.0.40-dev <$Revision: 1.146 $> apache-2.0
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Copyright 2006 The Apache Software Foundation, http://www.apache.org/

Benchmarking 127.0.0.1 (be patient)
Completed 10000 requests
Completed 20000 requests
Completed 30000 requests
Completed 40000 requests
Completed 50000 requests
Completed 60000 requests
Completed 70000 requests
Completed 80000 requests
Completed 90000 requests
Finished 100000 requests

Server Software:        TornadoServer/0.1
Server Hostname:        127.0.0.1
Server Port:            3002

Document Path:          /
Document Length:        12 bytes

Concurrency Level:      1000
Time taken for tests:   27.996766 seconds
Complete requests:      100000
Failed requests:        0
Write errors:           0
Total transferred:      16800336 bytes
HTML transferred:       1200024 bytes
Requests per second:    3571.84 [#/sec] (mean)
Time per request:       279.968 [ms] (mean)
Time per request:       0.280 [ms] (mean, across all concurrent requests)
Transfer rate:          586.00 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0  197 1102.8      0   20998
Processing:     1   50  37.3     45    5234
Waiting:        0   49  37.4     44    5234
Total:         18  247 1109.2     45   21253

Percentage of the requests served within a certain time (ms)
  50%     45
  66%     48
  75%     52
  80%     57
  90%     77
  95%   1237
  98%   3074
  99%   3112
 100%  21253 (longest request)

ab: EventMachine

[root@mail ~]# ab -c1000 -n100000 http://127.0.0.1:3001/
This is ApacheBench, Version 2.0.40-dev <$Revision: 1.146 $> apache-2.0
Copyright 1996 Adam Twiss, Zeus Technology Ltd, http://www.zeustech.net/
Copyright 2006 The Apache Software Foundation, http://www.apache.org/

Benchmarking 127.0.0.1 (be patient)
Completed 10000 requests
Completed 20000 requests
Completed 30000 requests
Completed 40000 requests
Completed 50000 requests
Completed 60000 requests
Completed 70000 requests
Completed 80000 requests
Completed 90000 requests
Finished 100000 requests

Server Software:
Server Hostname:        127.0.0.1
Server Port:            3001

Document Path:          /
Document Length:        12 bytes

Concurrency Level:      1000
Time taken for tests:   15.238117 seconds
Complete requests:      100000
Failed requests:        0
Write errors:           0
Total transferred:      7700077 bytes
HTML transferred:       1200012 bytes
Requests per second:    6562.49 [#/sec] (mean)
Time per request:       152.381 [ms] (mean)
Time per request:       0.152 [ms] (mean, across all concurrent requests)
Transfer rate:          493.43 [Kbytes/sec] received

Connection Times (ms)
              min  mean[+/-sd] median   max
Connect:        0   76 603.3      0    9000
Processing:     0   32 264.1     15   10627
Waiting:        0   31 264.1     14   10625
Total:          9  108 752.8     15   14642

Percentage of the requests served within a certain time (ms)
  50%     15
  66%     15
  75%     15
  80%     22
  90%     33
  95%     35
  98%   2999
  99%   3015
 100%  14642 (longest request)

If you’re upgrading to Snow Leopard and develop with Ruby on Rails you’ll need to do a few things.

First, a note about Java. If you depend on Java, 10.6 no longer has Java 1.5. The current workaround for this is to copy a 1.5 install from a non-snow-leopard mac. More instructions here (there’s a d/l for 1.5 as well; yes, Snow Leopard DELETIFIES 1.5).

1) Install XCode from the CD (required for ruby headers)
1a) Install iPhone SDK if that’s something you do
2) gem uninstall ruby-debug-base; gem install ruby-debug-base linecache
3) rebuild database gems and/or drivers (you’ll likely need arch flag, below)

I can’t speak for others but for PostgreSQL I had 8.3 installed and had to download/compile/install the latest 8.3.7 from source then gem uninstall dbd-pg, pg, and ruby-pg and reinstall them – prefixing gem install with ARCHFLAGS=”-arch x86_64″.

For our particular project I had to reinstall json, nokogiri, and rjb (had to export JAVA_HOME=/System/Library/Frameworks/JavaVM.framework/Versions/1.5.0 ).

If you use mod_passenger, reinstall and follow install instructions (passenger-install-apache2-module).

Shameless plug: if this helped you, you live in Texas, and drive < 12k miles a year, checkout my employer.

Errors this stuff fixed for me:

/path/to/gems/ruby-debug-base-0.10.3/lib/ruby_debug.bundle: dlopen(/path/to/gems/ruby-debug-base-0.10.3/lib/ruby_debug.bundle, 9): no suitable image found.  Did find: (LoadError)
	/path/to/gems/ruby-debug-base-0.10.3/lib/ruby_debug.bundle: no matching architecture in universal wrapper - /path/to/gems/ruby-debug-base-0.10.3/lib/ruby_debug.bundle

/path/to/gems/linecache-0.43/lib/../lib/trace_nums.bundle: dlopen(/path/to/gems/linecache-0.43/lib/../lib/trace_nums.bundle, 9): no suitable image found.  Did find: (LoadError)
	/path/to/gems/linecache-0.43/lib/../lib/trace_nums.bundle: no matching architecture in universal wrapper - /path/to/gems/linecache-0.43/lib/../lib/trace_nums.bundle

/path/to/gems/activerecord-2.3.3/lib/active_record/connection_adapters/abstract/connection_specification.rb:76:in `establish_connection': Please install the postgresql adapter: `gem install activerecord-postgresql-adapter` (dlopen(/path/to/gems/pg-0.8.0/lib/pg.bundle, 9): no suitable image found.  Did find: (RuntimeError)
	/path/to/gems/pg-0.8.0/lib/pg.bundle: no matching architecture in universal wrapper - /path/to/gems/pg-0.8.0/lib/pg.bundle)

# This happens when postgresql is old. I had to compile 8.3.7 from source with archflags above.
compat.h:38:2: error: #error PostgreSQL client version too old, requires 7.3 or later.

dlopen(/path/to/gems/json-1.1.7/ext/json/ext/generator.bundle, 9): no suitable image found.  Did find:
	/path/to/gems/json-1.1.7/ext/json/ext/generator.bundle: no matching architecture in universal wrapper - /path/to/gems/json-1.1.7/ext/json/ext/generator.bundle

dlopen(/path/to/gems/nokogiri-1.3.2/lib/nokogiri/nokogiri.bundle, 9): no suitable image found.  Did find:
	/path/to/gems/nokogiri-1.3.2/lib/nokogiri/nokogiri.bundle: mach-o, but wrong architecture - /path/to/gems/nokogiri-1.3.2/lib/nokogiri/nokogiri.bundle

dlopen(/path/to/gems/rjb-1.1.7/lib/rjbcore.bundle, 9): no suitable image found.  Did find: (LoadError)
	/path/to/gems/rjb-1.1.7/lib/rjbcore.bundle: no matching architecture in universal wrapper - /path/to/gems/rjb-1.1.7/lib/rjbcore.bundle

# Java stuff spits out something like this
:in `load': can't create Java VM (RuntimeError)
/Users/alexl/code/gems/gems/ruby-debug-base-0.10.3/lib/ruby_debug.bundle: dlopen(/Users/alexl/code/gems/gems/ruby-debug-base-0.10.3/lib/ruby_debug.bundle, 9): no suitable image found.  Did find: (LoadError)
/Users/alexl/code/gems/gems/ruby-debug-base-0.10.3/lib/ruby_debug.bundle: no matching architecture in universal wrapper – /Users/alexl/code/gems/gems/ruby-debug-base-0.10.3/lib/ruby_debug.bundle

If you work for Heartland Payment Systems, whether you like it or not, you currently represent “what not to do” in the world of Financial Transactions on the Internet.

If  you’re a customer of Heartland then you should take this as an opportunity to verify the security of your own systems. Heartland Payment Systems is a very large organization and if they were more secure you may have been attacked instead. Don’t make their mistake. Be proactive not reactive.

To begin I’d like to immediately point out the DoJ’s release on this event and appreciate the department giving away some details on the attacks which aren’t available on Heartland’s site. I first heard of this story from someone at our office and immediately realized there was more to the story but it wasn’t until I read the DoJ’s release that my suspicions became more than a guess. My response to the email: “Heartland’s negligence was as severe as their assailant’s attack and for this, they should be prosecuted”. According to bloomberg, a shareholder filed suit against Heartland in July and “in a Feb. 24 conference call, Carr [CEO] said the company was the subject of an informal inquiry by the Securities and Exchange Commission, as well as investigations by the Justice Department, the Federal Trade Commission and the Office of the Comptroller of the Currency.”. It’s obvious that the attacks were malefic, but what about Heartland’s security? Was Heartland negligent? Did Heartland employ the “Highest Standards” of security which could have mitigated the severity of their compromise?

Per the DoJ release, Heartland was vulnerable to “sophisticated” SQL Injection attacks which were used in part to compromise more than 130 million credit and debit cards. The attacker, Albert Gonzalez, 28, of Miami, Fla., is now being charged for a “different pattern of hacking activity that targeted different corporate victims and involved different co-conspirators.” – in addition to two other trials for other hacking activities. Thief, criminal, hacker – whatever you want to call him, we can all conclude these attacks were wrong. This guy has been in your Internetz stealing your credit cardz.

Meanwhile, if you’ve been a business who accepts credit cards through Heartland you likely have customers with stolen credit card information and are in a delicate position considering the significant cost required to change payment processors, both technically, and contractually. Changing payment processors often requires programming and the contract agreements required by most payment processors are stifling, costly, or both. So now you’re left with the question as to whether you should trust Heartland with your customers credit card information?

I can’t answer these questions for you but we can at least explore the idea that Heartland’s operations were negligent and ineffective at providing “The Highest Standards” and “The Most Trusted Transactions” as, according to reports, their attacker profiled several systems before determining that Heartland’s system was one to be compromised. If Heartland’s system was as-advertised, would it have withstood these attacks or would the attacker have targeted a different system instead?

Based on the DoJ report of the attack and amount of information compromised I would suspect the attack was preventable if “standard” security systems and practices were adopted. SQL Injection, while it sounds like a complicated attack requiring intricate knowledge, planning, and acrobatic stunts — is really a simple exploit which has been documented and addressed for almost a decade. In fact, a similar attack – code injection – was used in the SQL slammer worm which essentially broke the Internet in 2003. Basically, both attacks depend on a “parameter” not being checked and then being executed; this is akin to letting a stranger drive your car. Being exploited by one of these attacks is like letting an intoxicated stranger drive your car into a pole; maybe he said he was good to drive, and maybe the pole jumped in front of him, but the point is you had no experience to trust a stranger and you shouldn’t ever let an intoxicated driver behind the wheel of a car.

Ignorance is not an excuse for information security just as it isn’t an excuse for tax evasion or violating regulations. If you want to sympathize for Heartland for being attacked then that’s your prerogative and I hope you’re not the person responsible for security at any organization. If you’re concerned for the privacy and security of consumer’s financial and personal information then it’s your right to ask these questions of Heartland Payment Systems, or any company who significantly compromises the security of your information.

links for 2009-03-31

March 31, 2009

links for 2009-03-30

March 30, 2009

links for 2009-03-29

March 29, 2009

links for 2009-03-28

March 28, 2009

  • Tokyo based Appliya is a leading publisher of Japanese iPhone and iPod Touch applications.
    (tags: iphone)
Follow

Get every new post delivered to your Inbox.