Transient Creation Performance - ColdSpring and TransientFactory

2008 November 11
tags: ColdSpring · jMeter · Testing
by Paul Marcotte

In my last post, I demonstrated the generic factory I use to create Transient objects in my Coldfusion apps. Jaime Metcher asked in a comment whether the Transient Factory is faster than CS when instantiating transient objects. I had no hard data to answer Jaime's question, so I took the opportunity to set up jMeter on my new dev machine and run through some load tests to find out.Here's the skinny on my CF installation.

Server Product ColdFusion
Version 8,0,1,195765
Operating System Mac OS X
OS Version 10.5.5
Java Version 1.5.0_13
For the test, I set up a single cfm page that either requests a bean from ColdSpring or from the TransientFactory based on a url param. For each test, jMeter makes 2 page requests passing the factory to use for creating a transient. I varied the tests from 5-50 users, looping between 200-20 times to produce a thousand requests for each factory. Al requests with a 1 second ramp up period. Here are the results 5 users, 200 requests:
Label # Samples Average Median Min Max Std. Dev.
TF Test 1000 5 3 1 81 8.34
CS Test 1000 10 6 2 136 13.63
TOTAL 2000 8 5 1 136 11.53
10 users, 100 requests:
Label # Samples Average Median Min Max Std. Dev.
TF Test 1000 6 4 1 132 12.05
CS Test 1000 18 8 2 258 28.80
TOTAL 2000 12 5 1 258 22.81
20 users, 50 requests:
Label # Samples Average Median Min Max Std. Dev.
TF Test 1000 11 3 1 767 64.16
CS Test 1000 21 9 2 809 45.28
TOTAL 2000 16 5 1 809 55.77
25 users, 40 requests:
Label # Samples Average Median Min Max Std. Dev.
TF Test 1000 35 7 1 2127 205.33
CS Test 1000 33 14 2 2029 74.93
TOTAL 2000 34 10 1 2127 154.56
50 users, 20 requests:
Label # Samples Average Median Min Max Std. Dev.
TF Test 1000 44 5 1 1714 221.58
CS Test 1000 23 10 2 1243 50.62
TOTAL 2000 33 7 1 1714 161.06

The interesting thing to me is that as the number of users increase and requests decrease, CS out-performs the TF for average time, but the TF media is consistently below CS. So, even with a larger standard deviation, it appears that the TransientFactory is marginally faster than ColdSpring for creating transients. So, using the TransientFactory gives you both slightly faster transient creation and the flexibility to pass constructor arguments to init() at runtime and to define a method to invoke after the object is created and autowired.