Unexpected CSLA multithreading issues.

Unexpected CSLA multithreading issues.

Old forum URL: forums.lhotka.net/forums/t/11638.aspx


rxelizondo posted on Tuesday, October 09, 2012

Hello.

I am very much aware that interacting with one CSLA business object instance from multiple threads can lead to concurrency issues, but what I wasn't expecting was to run into concurrency issues when interacting with multiple CSLA business object instances all running on individual threads.

Rather than trying to explain the issue with words, I will try to illustrate it with some overly simplistic code. Take a look at the snippet below:

static void DoIt()

{

    var cslaObject = new CSLAObject();

    cslaObject.Property1 = "A";

    cslaObject.Property1 = "B";

    cslaObject.Property1 = "C";

    cslaObject.Save();

}

 

As you can see, the code above does nothing other than creating a CSAL based business object, sets up some property values and saves itself. Key to this function is to note that the “cslaObject” is only accessed by the current thread, it’s never shared with other threads.

 

Now let’s move on to the code that creates the multiple threads:

 

static void Main(string[] args)

{

    Thread thread1 = new Thread(DoIt);

    Thread thread2 = new Thread(DoIt);

 

    thread1.Start();

    thread2.Start();

}

 

Nothing interesting happening here, it’s just a method that creates two threads that call the “DoIt()” method.

 

DISCLAIMER: Giving the fact that I am not a multithreading application developer expert and I am certainly no CSLA expert, I will proceed to explain what I think is going on here. It goes something like this:

 

1)      Thread1 kicks off.

2)      Thread1 creates the CSLA object.

3)      Thread1 sets up the Property1 value.

4)      Thread1 pauses for a second….

5)      Thread2 kicks off.

6)      Thread2 creates the CSLA object.

7)      Thread2 sets up the Property1 value.

8)      Thread2 sets up the Property2 value.

9)      Exceptions is thrown (Collection was modified, enumeration operation may not be execute)

 

So it looks to me like although both threads are accessing completely different CSLA objects, some of the CSLA internals are being accessed by both threads. My guess is that this has to do with the optimization done by the CSLA when it sets up the rules since I believe that some of the rule infrastructure is shared through all the objects of the same type.

 

Again, I have no idea if this is the issue or not, I arrived to this conclusion through observation and the process of elimination but I have no conclusive evidence since the problem its hard to reproduce, I did't spent to much time looking into this and I am no CSLA expert. 

 

But before I jump into conclusion, I would like to know if someone can tell me if my assumption that my sample code above is subject to multithreading issues or not. Because if it’s not then I probably need to go back and fix whatever I am doing wrong with my CSLA business objects.

 

Sorry if this has been asked before (I have a feeling is has). I did a quick search on the CSLA forums and only found information regarding concurrency issues when multiple threads access single object and not my specific example.

 

Thanks.

JonnyBee replied on Tuesday, October 09, 2012

Which version of Csla do you use?

Do you have rules attachbusiness to the properties?

rxelizondo replied on Tuesday, October 09, 2012

We are currently using version 4.2.2.0 on a WPF application.

Yes, we have many rules attached to the properties both business and authorization rules.

JonnyBee replied on Tuesday, October 09, 2012

I'd like to dig into this - are you able to reproduce this in a sample with or without rules and if so could you post it here as attachment?

JonnyBee replied on Tuesday, October 09, 2012

I took your simplistic sample and added one atuhz rule and 2 business rules to the property with CSLA 4.3 (trunk) and was unable to recreate the problem.

A couple of updates in CSLA 4.3 (from 4.2.2) to be aware of:

1047 Fix race condition bug with MethodCaller
1026 Add lock on ClearRules in BusinessRules

Especially item 1026 may have solved this issue.

rxelizondo replied on Tuesday, October 09, 2012

Thanks Jonny.

I don't want to mess around with upgrading the CSLA to a newer version at this time so I did't test our current code with the version you suggested. However, I did take a couple of minutes to try to replicate the error we are having and managed to crash the code on two different places. I am posting the crash information here to see if by you looking at where the code crashed you may be able to tell if the fixes you are mentioning may apply to our problem.

I also attached screen shots of the Visual Studio stack trace.

If this does not help (and my guess is that you will want a repro at one point or another) I may put some time aside to try to create a repro.

 

Thanks.

 

-----------------------------------------------------------

Crash 1:

 

Occurred at: 

Csla.Rules.BusinessRules

 

On function: 

private List<string> RunRules(IEnumerable<IBusinessRule> rules, bool cascade, RuleContextModes executionContext)

 

Stack Trace:

System.InvalidOperationException occurred

  Message=Collection was modified; enumeration operation may not execute.

  Source=mscorlib

  StackTrace:

       at System.ThrowHelper.ThrowInvalidOperationException(ExceptionResource resource)

       at System.Collections.Generic.List`1.Enumerator.MoveNextRare()

       at System.Collections.Generic.List`1.Enumerator.MoveNext()

       at System.Linq.Enumerable.WhereListIterator`1.MoveNext()

       at System.Linq.Buffer`1..ctor(IEnumerable`1 source)

       at System.Linq.OrderedEnumerable`1.<GetEnumerator>d__0.MoveNext()

       at Csla.Rules.BusinessRules.RunRules(IEnumerable`1 rules, Boolean cascade, RuleContextModes executionContext)

 

-----------------------------------------------------------

Crash 2:

 

Occurred at:

Csla.Rules.BusinessRules

 

On function: 

private List<string> CheckRulesForProperty(Csla.Core.IPropertyInfo property, bool cascade, RuleContextModes executionContext)

 

Stack Trace:

System.InvalidOperationException occurred

  Message=Collection was modified; enumeration operation may not execute.

  Source=mscorlib

  StackTrace:

       at System.ThrowHelper.ThrowInvalidOperationException(ExceptionResource resource)

       at System.Collections.Generic.List`1.Enumerator.MoveNextRare()

       at System.Collections.Generic.List`1.Enumerator.MoveNext()

       at System.Linq.Enumerable.WhereSelectListIterator`2.MoveNext()

       at Csla.Rules.BusinessRules.CheckRulesForProperty(IPropertyInfo property, Boolean cascade, RuleContextModes executionContext) 

 

rxelizondo replied on Friday, October 12, 2012

 

I just tried to run the application with the latest version of the CSLA (4.3.13) and the problem still persist. So fix 1047 and 1026 do not fix this particular problem.

 

The latest crash that we experienced happened when trying to enumerate the "TypeAuthRules.Rules" collection:

 

    public bool HasPermission(AuthorizationActions action, Csla.Core.IMemberInfo element)

    {

.........

     // Crash occurred here (Collection was modified; enumeration operation may not execute.)

      var rule = TypeAuthRules.Rules.FirstOrDefault(c => c.Element != null && c.Element.Name == element.Name && c.Action == action);

.........

}

Anyway, our workaround is to create an instance of the CSLA object and initialize some of its properties properties on the main thread. We do this in the main thread before any multithreading begins. This seems to give the CSLA object time to initialize itself on a thread safe environment and our problems disappear.

Giving the fact that the exception occurs on the CSLA object I can only assume that this is a CSLA bug.

Maybe in the coming days I will have some time to try to dig deeper into the issue. For now, if you anyone has any ideas then please let me know.

Thanks.

JonnyBee replied on Sunday, October 14, 2012

This was reported earlier too in connection to AddBusinessRules:

http://forums.lhotka.net/forums/p/11511/53368.aspx#53368 

Bugtracker: http://lhotka.net/cslabugs/edit_bug.aspx?id=1078

But I now see that the same should be applied to AddObjectAuthorizationRules (in InitializePerTypeRules method).

rxelizondo replied on Sunday, October 14, 2012

Thanks Jonny,

If is no big deal, could you please post the code with the fix you guys came up?

I would like to copy / paste your fix into the CSLA version we are currently using. We will eventually upgrade to the newest CSLA version but this is not the right time for us to do that.

Thanks.

JonnyBee replied on Monday, October 15, 2012

Open the link to bugtracker and look at the SVN Revisions.

This will show you the diff for each file.

JonnyBee replied on Monday, October 15, 2012

Hi Rene,

I updated the code for both 4.3 and 4.5 to handle race conditions for both Authz and Business rules.

http://www.lhotka.net/cslabugs/edit_bug.aspx?id=1107

Look at the SVN Revisions connected to this item.

rxelizondo replied on Saturday, October 13, 2012

Rocky / Jonny

I think I may have the cause of the problem but you guys will need to verify my assumption, these multithreading issues are real brain teasers Smile

I believe the issue has to do with the way you guys are trying to optimize the call to the "InitializeBusinessRules()" function (the double checked locking optimization). I have copied and pasted the function here for your reference. Please see below:

--------------------------------------------------------------------

private void InitializeBusinessRules()
{
    var rules = BusinessRuleManager.GetRulesForType(this.GetType());
    if (!rules.Initialized)
        lock (rules)
            if (!rules.Initialized)
            {
                rules.Initialized = true;
                try
                {
                    AddBusinessRules();
                }
                catch (Exception)
                {
                    BusinessRuleManager.CleanupRulesForType(this.GetType());
                    throw;  // and rethrow exception
                }
            }
}

--------------------------------------------------------------------

Now consider the following sequence of events happening when a CSLA class gets instantiated for the very *first* time.

1) Thread 1 starts creating an instance of “SomeClass”.

2) Thread 1 enters “InitializeBusinessRules()”.

3) Thread 1 reads that “rules.Initialized” is equals to false.

4) Thread 1 acquires lock.

5) Thread 1 sets “rules.Initialized” to be equal to true (**this is the problem**).

6) Thread 1 enters “AddBusinessRules()”.

7) ….. In the meantime, while Thread 1 is busy working on “AddBusinessRules()”.

8) Thread 2 starts creating an instance of “SomeClass”.

9) Thread 2 enters “InitializeBusinessRules()”.

10) Thread 2 reads that “rules.Initialized” is equals to true.

11) Thread 2 moves on (** oh, oh… but what if Thread 1 is not finished calling the “AddBusinessRules()” function? **).

12) Thread 2 assumes that all business rules are setup so somewhere along the line it calls “CheckRules()”.

13) While Thread 2 is in the middle of calling “CheckRules()” Thread 1 (still executing the “AddBusinessRules()” function) adds more business rules (modifying the rules collection). At some point Thread 2 tries to enumerate the business rule collection but crashes with “Collection was modified; enumeration operation may not execute” exception because Thread 1 modified the collection

I ended up removing the double checked locking optimization as shown by the code below and we stopped having errors.
--------------------------------------------------------------------
private void InitializeBusinessRules()
{
    var rules = BusinessRuleManager.GetRulesForType(this.GetType());
    lock (rules)
    {
        if (!rules.Initialized)
        {
            rules.Initialized = true;
            try
            {
                AddBusinessRules();
            }
            catch (Exception)
            {
                BusinessRuleManager.CleanupRulesForType(this.GetType());
                throw;  // and rethrow exception
            }
        }
    }
}

--------------------------------------------------------------------

I guess you could also move the "rules.Initialized = true" line after the "AddBusinessRules()" instruction but my guess is that having the " if (!rules.Initialized)" line outside the lock is probably looking for trouble.

Please let me know if you guys agree that this my be a problem or not.

Thanks.

 

Copyright (c) Marimer LLC