Performance of creating large collection

oshokodk posted on Monday, July 31, 2006

I am evaluating CSLA framework for the use in large distributed app being converted from C++/COM/Java/… Number of simultaneous user can reach 1000+. Many screens are grid based and some of them display large number of records. Connection speed varies.

I am concern with the performance of creating large collection of business objects.

Have anybody measured how long does it take to create and populate from DataReader of collection BO that has 10 properties? 1000 items in collection? 10000 items?

Thanks

RockfordLhotka replied on Monday, July 31, 2006

I would be concerned as well, not because of the DataReader, since that is entirely unavoidable when loading data from a database in .NET, but because you propose to create 1000-10,000 objects and that is expensive.

If you are doing this as a 2-tier app it wouldn't be so bad, but with 1000+ concurrent users you pretty much need to go with 3-tier, and so there's the overhead of moving the data from the database into your objects and then moving the objects across the network to the client. That's non-trivial.

Of course even if you use a DataSet to do this, it will use a DataReader to load the collection, and the DataSet will have to be serialized across the network from the app server to the client, so much of the overhead is constant... The benefit (potentially) to the DataSet is that they use some tricks internally to store the data in memory - thus avoiding the creation of an object per row. The drawback to the DataSet is that it is notoriously inefficient at serializing to go across the wire...

So in the end there's no simple answer. The only real answer is to test it yourself.

One thing I'll point out, is that in CSLA .NET 2.0 each object (row) loads its own validation rules, and that will add up to a lot of overhead with 1000+ rows. Version 2.1 will help address this with per-type validation and authorization rules. For your testing, I would suggest starting with no validation or authorization rules in the objects for now.

Another thing to consider is your network protocol (data portal channel in CSLA). Of the three options. Enterprise Services is fastest, though it has firewall issues because it uses DCOM. But its performance benefits are non-trivial, and comparatively Remoting and Web Services are both quite slow. That'll be true whether you use CSLA or not - it is just the way the protocols work.

Yet another thing to consider is the idea of using CSLA only on the client, and having your DataPortal_XYZ methods call a set of data services on the app server. While this costs you the benefits of mobile objects, it can have gains in terms of network bandwidth because you can more tightly control the amount of data actually transferred between clients and the server.

In the end however, the only way to know for sure what will and won't work is to do some serious simulation/testing. There are just so many variables involved that anyone else's results can be considered merely a data point in your research, not an answer.

Lakusha replied on Tuesday, August 01, 2006

But, putting aside all the inherent drawbacks of Enterprise Services, and looking at the attractive possibility of a very significant speed boost, there is still one question that remains unanswered: is there a large site somewhere using a CSLA application deployed under Enterprise Services?

From this forum (and the previous one) I know that a lot of CSLA applications are deployed in 2/3 tiers and widely different setups: SQL, ORA, MySQL, WS, Remoting, etc. But I've never seem ES used (not saying it's not, just saying I haven't seen it).

How good a speed boost can we expect with ES? Firewall are not always a problem and having a significant speed boost is worth looking at the option (especially since WCF will deliver many nice things, but no real speed boost).

L

RockfordLhotka replied on Tuesday, August 01, 2006

Honestly, I am not aware of anyone using the ES data portal channel in
production. If you are, please speak up :)

I know that Microsoft had commissioned a performance comparison study a
while back. They hired Ingo Rammer, the widely regarded expert on Remoting,
to do the study. His results showed that, in most cases, ES offered an order
of magnitude better performance than Web Services or Remoting. I believe
that this study did end up getting published on MSDN, but I don't have a
URL.

Rocky

> How good a speed boost can we expect with ES? Firewall are
> not always a problem and having a significant speed boost is
> worth looking at the option (especially since WCF will
> deliver many nice things, but no real speed boost).

Lakusha replied on Tuesday, August 01, 2006

I think you mean this article: http://msdn.microsoft.com/library/default.asp?url=/library/en-us/dnwebsrv/html/asmxremotesperf.asp

In view of such results, why isn't there more ES deployments? Marketing hype about WS (when in fact almost everybody is doing WS-RPC style instead of true messaging) can't be the only reason. Performance is still key to many systems...

Is the support/maintenance pain of ES too great? (I remember components that would not unload and myriads of other pains associated with DCOM :-( )

And I've never used DCOM in a WAN. HTTP is resilient to network latency, but what about DCOM?

L

RockfordLhotka replied on Tuesday, August 01, 2006

Yes, thank you, that's the one.

>
> I think you mean this article:
> http://msdn.microsoft.com/library/default.asp?url=/library/en-
> us/dnwebsrv/html/asmxremotesperf.asp
>
> In view of such results, why isn't there more ES deployments?
> Marketing hype about WS (when in fact almost everybody is
> doing WS-RPC style instead of true messaging) can't be the
> only reason. Performance is still key to many systems...
>
> Is the support/maintenance pain of ES too great? (I remember
> components that would not unload and myriads of other pains
> associated with DCOM :-( )
>
> And I've never used DCOM in a WAN. HTTP is resilient to
> network latency, but what about DCOM?

I think people have too many negative memories of DCOM. But of course ES
isn't straight DCOM. Sure, it uses DCOM on the wire, but it uses COM+ to
host the server, and a COM+ proxy on the client to handle the client
configuration. The result is that most of the DCOM pain is gone. The big
thing is on the server, where things are hosted in COM+ - thus avoiding the
locked-DLL issues you refer to from straight DCOM/COM.

The remaining (real) issues are going through firewalls, and the fact that
there is a client registration step to install the data portal channel proxy
(which makes no-touch or ClickOnce deployment problematic).

You should also remember that COM+ (MTS) predates IIS, Web Services,
Remoting and all the other technologies we're talking about. Because of
this, it has several years more maturity. Additionally, it was designed for
high transaction volumes in an application server, where IIS/ASP.NET was
designed for high transaction volumes in a web server. We just misuse it as
an application server.

DCOM works fine over a WAN, just not over the Internet (at least not without
a VPN). HTTP is only resilient to latency because the timeout values
per-request are high enough to deal with most latencies. DCOM is a little
different, in that it is connection-based. It establishes a TCP connection
to the server and then uses that connection, which means it has less
overhead per-call (part of the perf gain), and thus has a slightly easier
time dealing with latency.

Ultimately, the reason I wrote the ES data portal channel was because Don
Box pushed on it. I was arguing that it was a waste of time, that no one
would care. He suggested that I was ignoring the highest performing network
protocol, and that it was silly not to support it... And of course he was
right - it wasn't hard to implement, and it is silly to ignore the highest
performing (and oldest and most robust) option available.

Rocky

Lakusha replied on Tuesday, August 01, 2006

Thanks for the information.

I don't know that I'll ever deploy our CSLA apps with ES, but the fact that it is possible is quite important for us. It means we might have an escape tunnel if we are stuck with perf issues at a large site and stacking more middle tier pizza boxes is not a good answer. That extra flexibility is worth a lot (and it goes both ways since it opens the door for WCF too).

I am sure one of our salesreps will require us to test/benchmark ES one of these days. If some very large site can go live with 4/5 middle tier servers instead of 10+, the extra work will pay for itself.

L

oshokodk replied on Thursday, August 03, 2006

I did some test. My findings are quite interesting. I measured different ways of restore large XML. XML will be produced by existing web services, compressed, send over wire, decompressed, etc.

It takes 1.2 sec to restores 21k of rows, 10 columns into dataset and extract datatable from it. It takes 0.3 sec to restore collection of BO derived from BusinessObjectBase. This is nice surprise.

However, grid control (Dev Express) bound to datatable is much more responsive. Sorting & filtering is much faster than with the grid bound to collection of business objects. Sorting of grid bound to datatable is sub-second operation. Sorting of the grid bound to collection takes around 3 sec.

Any suggestions?

leotohill replied on Tuesday, September 12, 2006

Lakusha:
I think you mean this article: http://msdn.microsoft.com/library/default.asp?url=/library/en-us/dnwebsrv/html/asmxremotesperf.asp In view of such results, why isn't there more ES deployments? Marketing hype about WS (when in fact almost everybody is doing WS-RPC style instead of true messaging) can't be the only reason. Performance is still key to many systems... Is the support/maintenance pain of ES too great? (I remember components that would not unload and myriads of other pains associated with DCOM :-( ) And I've never used DCOM in a WAN. HTTP is resilient to network latency, but what about DCOM? L

Are we reading the same article? The one that summarizes "the difference between Web services (which are usually perceived as being very slow) and DCOM (which is clearly one of the fastest distributed application technology in this test) is in fact very small for real applications." ?

I wonder, because it seems that you and Rockford still think that ES/DCOM is significantly better, but the conclusion of the article was just the opposite.

vbLuis replied on Wednesday, September 13, 2006

I second the observation. In fact, in a couple of the tests Remoting (using TCP/Binary) outperformed Enterprise Services (see figures 4, 7, and 9). And, in the rest it essentially tied (see figures 1, 2 and 5).

The order of magnitude difference appears to be on loads using a "very small amount of data from server to client" (see figure 7). In that test Remoting using HTTP/Binary (IIS) lags way behind ES, yet Remoting using TCP/Binary blows ES away.

According to the results of the article (not their conclusion), it seems that for optimal performance, you are better off using Remoting TPC/Binary.

What am I missing?

Lakusha replied on Thursday, September 14, 2006

Two things that should be reminded when optimizing n tiers performance (thanks to Rocky who brought our attention on this) are:

a) circular references in the object graph (e.g. like a Parent member)
b) adding serializable events in the graph

Both make a HUGE difference in the size of the object and in the time/cpu necessary to serialize/deserialize it.

I haven't mesured point A yet, but in our case events were 35% of the object(!). We had quite a few event because of the automation model approach.

Point A was fixed by making those references non-serializable and handling them manually at the deserialized event. Point B was fixed by getting rid of most events and handling the rest differently.

Moving a mouse is always faster than moving an elephant...whatever the protocol used.

RockfordLhotka replied on Saturday, September 30, 2006

In 1.53 there are two options you might be using: BrokenRules.Assert(), or CheckRules().

If you are using BrokenRules.Assert() then you are in complete control in terms of when the rules are checked - so you can really do whatever you'd like (though the idea of running it on a background thread is going to cause you way more pain than gain!!)

If you are using CheckRules(), with rule methods, you are still in control, because it is your code that called CheckRules(). Typically you'd make the call in DataPortal_Create/Fetch - but you could make the call in the factory on the client instead. This would take a bit of doing, but I'm sure you can make it work.

The cost you can't avoid, when using the CheckRules() idea, is that AddBusinessRules() is called as each object is created, and in that method you associate rule methods and properties. Really, that's the cost that 2.1 is avoiding with the per-type rule concept.

What you _might_ be able to do, is leave AddBusinessRules() empty, and implement some other method that you call from your client-side factory method to add the rules. Or better yet, override the Deserialized() method and call your custom method to add rules on deserialization - that'll get them added on return from the server, and after a clone or other operation (which is important).

This way the rules wouldn't be associated with properties on the app server, on initial load of the objects, but the rules would get associated as soon as the object graph was deserialized back onto the client - and then your factory could scan through the whole object graph, forcing a call to CheckRules on all objects in the graph (that's the tricky part - might require reflection).

chipso4 replied on Thursday, August 03, 2006

My own results for 45,000 rows from sql using some optimized codesmith templates. Note: I did turn off validataion to compare apples to apples.

    CSLA    DataSets       Diff

       946.2ms     3,440.4ms      2,494.2 Objects Load          3.6 times faster then datasets

       643.6      1,253.6        610.0 Object Iterate          1.9 times faster then datasets

         30.6      3,373.4      3,342.8 Objects Update      110.2 times faster then datasets

     1,517.4      2,903.8      1,386.4 Objects Serialize          1.9 times faster then datasets

The load time shoots up to ~6,000ms if validataion is enabled. And it shoots up to ~16,000ms if every row is validated during load which, for some odd reason, is the default setting on the codesmith templates. I think Rocky's changes in 2.1 will solve or at least minimize the impact of validation on load times. I'm excited to use the new changes.

Brian Criswell replied on Thursday, August 03, 2006

chipso4:
My own results for 45,000 rows from sql using some optimized codesmith templates. Note: I did turn off validataion to compare apples to apples.

    CSLA    DataSets       Diff

       946.2ms     3,440.4ms      2,494.2 Objects Load          3.6 times faster then datasets

       643.6      1,253.6        610.0 Object Iterate          1.9 times faster then datasets

         30.6      3,373.4      3,342.8 Objects Update      110.2 times faster then datasets

     1,517.4      2,903.8      1,386.4 Objects Serialize          1.9 times faster then datasets

The load time shoots up to ~6,000ms if validataion is enabled. And it shoots up to ~16,000ms if every row is validated during load which, for some odd reason, is the default setting on the codesmith templates. I think Rocky's changes in 2.1 will solve or at least minimize the impact of validation on load times. I'm excited to use the new changes.

The default is to validate on load is there in case the data is changed outside of business objects and that change breaks a validation rule or if the validation rules change in such a way as to render the data in the database invalid. If neither of these happens for you, then you could ostensibly skip the validation on load.

oshokodk replied on Thursday, August 03, 2006

I set BO private datamembers instead of set on properties in my load test. This code works much faster than dataset.

Based on your numbers I suspected that something is slowing my sort. That was

CanReadProperty(true) call within each property.

With CanReadProperty(true) sort of 21k records takes 2.2 sec. Without 0.15sec, which is great.

Thanks

Brian Criswell replied on Thursday, August 03, 2006

oshokodk:
I set BO private datamembers instead of set on properties in my load test. This code works much faster than dataset.

Based on your numbers I suspected that something is slowing my sort. That was

CanReadProperty(true) call within each property.

With CanReadProperty(true) sort of 21k records takes 2.2 sec. Without 0.15sec, which is great.

Thanks

If you want to keep CanReadProperty you can try CanReadProperty("PropertyName", true), which would give you a small performance improvement at the cost of some maintainability.

oshokodk replied on Friday, August 11, 2006

Some more perfromance numbers:

Call MarkOld() on 21k objects without any business rules costs around 100ms. MarkOld() calls MarkClean() which calls OnUnknownPropertyChanged().

The only way I see to temporary shuting down OnUnknownPropertyChanged() is to override this method.

I wish MarkOld() and MarkClean() would accept suppressEvent parameter as Mark Dirty() does.

Is this change comming?

RockfordLhotka replied on Friday, August 11, 2006

Have you tried overriding OnUnknownPropertyChanged to only raise PropertyChanged for a property named "" ?

That should change the performance radically, as it would avoid the current loop through to raise PropertyChanged for each property of the object.

Rocky

From: oshokodk [mailto:cslanet@lhotka.net]
Sent: Friday, August 11, 2006 9:21 PM
To: rocky@lhotka.net
Subject: Re: [CSLA .NET] Performance of creating large collection

Some more perfromance numbers:

Call MarkOld() on 21k objects without any business rules costs around 100ms. MarkOld() calls MarkClean() which calls OnUnknownPropertyChanged().

The only way I see to temporary shuting down OnUnknownPropertyChanged() is to override this method.

I wish MarkOld() and MarkClean() would accept suppressEvent parameter as Mark Dirty() does.

Is this change comming?

dileepagarwal replied on Sunday, October 15, 2006

Hi,

==================================================

My own results for 45,000 rows from sql using some optimized codesmith templates. Note: I did turn off validataion to compare apples to apples.

===================================================

I liked your test result. Recently I have started working on CSLA. I have few questions regarding your results.

Was you environment same in both cases?

Did you tried on the same machine or two machines on the LAN or over the Internet?

Could you please let me know which version of dataset you used (ADO 1.x or ADO 2.0)?

Did you use strongly typed dataset or simple dataset?

As CSLA does not have diffgram method like dataset have, how many records(objects) you used while performing your update operation (did you tried update over the network --if answers is yes than my main concern is how much data did you transferred from client to server in both cases).

If anyone did these kind of testing, please share their result (if possible), it will help us a lot.

Thanks,

Dileep Agarwal

Performance of creating large collection

oshokodk posted on Monday, July 31, 2006

RockfordLhotka replied on Monday, July 31, 2006

Lakusha replied on Tuesday, August 01, 2006

RockfordLhotka replied on Tuesday, August 01, 2006

Lakusha replied on Tuesday, August 01, 2006

RockfordLhotka replied on Tuesday, August 01, 2006

Lakusha replied on Tuesday, August 01, 2006

oshokodk replied on Thursday, August 03, 2006

RockfordLhotka replied on Thursday, August 03, 2006

oshokodk replied on Thursday, August 03, 2006

ajj3085 replied on Thursday, August 03, 2006

leotohill replied on Tuesday, September 12, 2006

vbLuis replied on Wednesday, September 13, 2006

RockfordLhotka replied on Wednesday, September 13, 2006

Lakusha replied on Thursday, September 14, 2006

stormc23 replied on Wednesday, September 20, 2006

RockfordLhotka replied on Wednesday, September 20, 2006

Skeeboe replied on Saturday, September 30, 2006

RockfordLhotka replied on Saturday, September 30, 2006

chipso4 replied on Thursday, August 03, 2006

Brian Criswell replied on Thursday, August 03, 2006

oshokodk replied on Thursday, August 03, 2006

Brian Criswell replied on Thursday, August 03, 2006

xal replied on Thursday, August 03, 2006

oshokodk replied on Friday, August 11, 2006

RockfordLhotka replied on Friday, August 11, 2006

oshokodk replied on Sunday, August 13, 2006

dileepagarwal replied on Sunday, October 15, 2006

CSLA	DataSets	Diff
946.2ms	3,440.4ms	2,494.2	Objects Load	3.6	times faster then datasets
643.6	1,253.6	610.0	Object Iterate	1.9	times faster then datasets
30.6	3,373.4	3,342.8	Objects Update	110.2	times faster then datasets
1,517.4	2,903.8	1,386.4	Objects Serialize	1.9	times faster then datasets