OT: Desperate plea for help! :)

pelinville posted on Tuesday, April 24, 2007

This a problem I am having. It is a copy/past from my blog so some of the information might be obvious.

Occasionally one user will get another's user data. For example there are assignment types associated with each Section the instructor teaches. Occasionally an instructor will see a list of assignment types from another instructors section.

A bit of background. We are using ASP.NET, IIS6, .NETv2 and AJAXPro.NET. It is an nTier application and CLSA is the middle tier. We use the single page "model" for the development. This means that when an action is performed the user is not taken to another page. In fact, because of AJAX, new controls are written to the page. Also in ASP.NET/IIS each user gets a "session variable" that stores information specific to that user and only that user. This is managed by various technologies and while I, the developer, can screw it up it isn't easy. The main use of session state in our app is to store the principle object to enable role based security. We also use forms authentication.

All calls to the database to get the data for the logged in user goes through what I call an entry point. Basically it gets the Principle from the current thread and associates that principle with a specific instructor. Through that instructor it get all further information such as the Section and Assignment Types for that section.

But, as I said, sometime a Instructor will see Assignment Types from a different section.

Other problems/clues

1. This only happens about once per 100,000 calls. (rough estimated)
2. It has never happened in development or testing.
3. It doesn't happen on every site.
4. Testing is done on both local and the production servers.
5. It has never happened during training. Training is done on the production servers.
6. I have made the same calls done in the web application 1 million times with 10 simulated users and never saw this bug.
7. Because it happens so rarely it is hard to set up logging to capture all the information. What logging I have been able to do doesn't show me anything unexpected.
8. The session isn't "switched". The next AJAX call or refreshing the page shows everything just as it should be.
9. It doesn't cross applications. The bad information only comes from
10. Almost all of the users are behind a firewall/proxy.
11. We use a load balancer that basically keeps sending calls from the same IP to the same server.
12. This load balancer times out after 15 minutes but our session timeout on the server is set at something like 8 hours. Form authentication is also set at 8 hrs.
13. To keep the user going to the same sever we make "dummy" Ajax calls very 14 minutes. We are very certain that the users are being kept on the same server. We verified this through the logs.
13. We have a total of 4 web servers. No web gardens are used.

In the AquireRequestState I do

System.Threading.Thread.CurrentPrincipal = HttpContext.Current.Session("CSLA-principal")

HttpContext.Current.User = HttpContext.Current.Session("CSLA-principal")

This is my wild, crazy guess.

Two users are making a call for the same AJAX method at the same time. Some place and time after the application sends the responses one of the users get the other users response.

Another idea I had is that their could be a some thread switching going on that I don't understand.

Any other ideas?

*Edit Also we use both in process and SQL server session state. Happen with both.

JoeFallon1 replied on Wednesday, April 25, 2007

I have seen this in the past. In my case it was directly related to the cache. I have some BOs which read data from the Db and then I cache them because the values rarely change. (In mnay cases I also have DB cache invalidation set up but that is a topic for a different thread.) These BOs are mostly read only collections (ROCs.)

In one case I had an expensive BO to create which I thought could be cached. The problem is that this BO is a Root BO which can be edited. I figured if I cloned the new cached value and returned that to the user that it would be acceptable. But it was not. The Root BO contained other BOs which were references and when they got updated my cached data was also updated (because it had the same references.) This caused other users to see each others data.

Bottom line - in ASP.Net, never cache a BO that can be edited. Only use Read Only BOs.

Joe

JoeFallon1 replied on Wednesday, April 25, 2007

"Thread switching" - Rocky has mentioned that ASP.Net does have some thread switching occur in some advanced scenarios:

"The only safe way to share global data for a user is by putting it on the current Thread object. There is some complexity introduced by ASP.NET, because the data stored on the Thread object isn’t guaranteed to be consistently available in that environment.When running in ASP.NET, this type of data should be stored in the current HttpContext object."

"... when code is running inside ASP.Net the only reliable way to find the user's principal is through HttpContext.Current.User."

By using the ApplicationContext.User property consistently, you do not need to worry about where the BO is being run - Windows or ASP.Net.

See page 236 of the VB Book for 2.0.

Joe

pelinville replied on Wednesday, April 25, 2007

Just adding this if other shappen upon this topic.

Also in Rocky's blog, that Joe mentions, is this.

"First, ASP.NET doesn’t change your thread at “random” or “without warning” (phrases I’d used in describing my worries). It changes it only in a clearly defined scenario. Specifically, when your thread performs an async IO operation. Scott points out that this is perhaps most common in a page that takes advantage of the new async page capability, or within a HttpModule that uses any async IO."

I don't know if any of this is going on but I am using AJAXPro.Net and it might. Or some other third party control might. Never know do we?

So I am going to try and pass in the HTTPContext.User to the factory methods to get the logged in instructor. Currently I am getting the IPrinciple from the thread. Hopefully this will work.

If it doesn't, and I might not know for a long time since it happens so rarely, I am at a loss. So if anybody else has any suggestions....