Currently i'm building a web application which will be used by a lot of users around countries.
I have a feature in my application which is "Data Simulation" of specifications of a Finish Goods Materials. In this feature, users allow to filtering or changing the parameters before doing the simulation so that they can get the final cost to do planning and analysis of their business. Simulation is being done by a quite complicated calculation flow within BO and usually 1 speq record take around 1-2 mins to be done.
Now my customer got 700++ records needs to be simulated, imagine 1 record take 2mins, so 700 records means 1400mins, i'm going crazy handling this huge data on my application and also i can't imagine if there're more than 1 user doing simulation at the same time. Is there any suggestion from you guys which have more experience than me handling such a huge data coming in and perform calculation + saving data?
fyi: I'm using .Net Framework 3.5, with CSLA Framework as the Business Object and MVC 2.0 Framework as the User Interface layer.
Thank you so much in advance for your suggestion
Do you have to immediately have to show the results? Have you considered moving this to a background service and then show the user any completed calculations (via a db query)? I think this would be the most reliable approach. What happens if your application pool crashes right in the middle of the sims (lets say 200).
I'd definitely look into what can be optimized.
Using a good .NET profiler and SQL Profiler is key to finding out what takes time in the application.
Here's a a nice articles on SqlbulkCopy:
Copyright (c) Marimer LLC