How to handle calculation and saving of a huge data

How to handle calculation and saving of a huge data

Old forum URL: forums.lhotka.net/forums/t/11314.aspx

sutono posted on Wednesday, April 18, 2012

Hi Experts,

Currently i'm building a web application which will be used by a lot of users around countries.

I have a feature in my application which is "Data Simulation" of specifications of a Finish Goods Materials. In this feature, users allow to filtering or changing the parameters before doing the simulation so that they can get the final cost to do planning and analysis of their business. Simulation is being done by a quite complicated calculation flow within BO and usually 1 speq record take around 1-2 mins to be done.

Now my customer got 700++ records needs to be simulated, imagine 1 record take 2mins, so 700 records means 1400mins, i'm going crazy handling this huge data on my application and also i can't imagine if there're more than 1 user doing simulation at the same time. Is there any suggestion from you guys which have more experience than me handling such a huge data coming in and perform calculation + saving data?

fyi: I'm using .Net Framework 3.5, with CSLA Framework as the Business Object and MVC 2.0 Framework as the User Interface layer.

Thank you so much in advance for your  suggestion

Stone

bniemyjski replied on Wednesday, April 18, 2012

Do you have to immediately have to show the results? Have you considered moving this to a background service and then show the user any completed calculations (via a db query)? I think this would be the most reliable approach. What happens if your application pool crashes right in the middle of the sims (lets say 200).

JonnyBee replied on Wednesday, April 18, 2012

I'd definitely look into what can be optimized.

  1. Dataaccess with inserts of many records is always best handled by the SqlBulkCopy class.
    Inserting 10.000 rows as an example shouldn't take more than a couple of seconds.
  2. Hugeamounts of data coming in may be optimzed on the wire with compression.
  3. For long running jobs I'd prefer to just save and start an asyncronous process to do the calculation.
    And let the user/application do some polling to get the result.
  4. Quite often you can get better perfomance thoru SQL and/or Index optimization.

Using a good .NET profiler and SQL Profiler is key to finding out what takes time in the application.

Here's a a nice articles on SqlbulkCopy:
http://www.adathedev.co.uk/2010/02/sqlbulkcopy-bulk-load-to-sql-server.html

 

Copyright (c) Marimer LLC