Aync call from behind the dataportal using remoting

grantj posted on Wednesday, June 24, 2009

We've just started using the async dataportal calls in 3.6.2 and we're noticing some odd behavior while debugging in VS. If we make an async call when we're already behind the dataportal, everything works fine on the dataportal side but control never returns to the client, which is left blocking indefinitely. So for instance, executing the following code:

static Obj GetObj(object crit) {
   return DataPortal.Fetch<Obj>(crit); //client is left blocking here
}

Dataportal_Fetch(object crit) {
    AsyncCommand.BeginExecute();
}
...
 static BeginExecute() {
    DataPortal.BeginExecute(new AsyncCommand(), null);
}

We tried specifying the callback as well as using the BackgroundWorker directly and disposing it in the callback, but got the same results. The weird thing is in a deployed environment it seems to work fine. Furthermore, we've noticed stepping through the code that if the worker thread completes before the main thread returns to the client, it works, otherwise it hangs.

Any ideas?
Thanks,
Grant

RockfordLhotka replied on Wednesday, June 24, 2009

The async data portal isn't designed for that scenario. Maybe it could be, but it isn't.

Stop and think about what you are suggesting.

1. Client calls server, so client is still running, and server is now running on ThreadA.
2. When ThreadA completes, it will respond to client and that's the end of the cycle.
3. Server calls server, so now has ThreadA and ThreadB, but ThreadA is not blocked.
4. Unless something stops ThreadA until ThreadB completes, you can't know if ThreadB will finish before ThreadA returns - and when ThreadA returns that's it - the client moves on and that's that.

As I say, I imagine we could make this work, but enhancing the data portal to use a locking mechanism to block the calling thread (or maybe the original thread handling the client request), and automatically unblock it as appropriate.

But that was outside the scope of our enhancements for 3.6.

grantj replied on Wednesday, June 24, 2009

I understand it doesn't make sense if it's an async call for which you care about the result, but in this particular case it is a logging call and is fire-and-forget. If I did care about the result then I would absolutely accept responsibility for blocking the main thread until it completed.

I may be wrong but I seem to remember one of the earlier principles of CSLA being that it shouldn't matter where a dataportal method was being called from - be it client or server. If I'm right then this scenario breaks that principle - an async call works fine if called from the client, but not if called from the server.

In any case, do you think this is a consequence of using remoting, and if so, could it be alleviated by switching to WCF?

RockfordLhotka replied on Wednesday, June 24, 2009

Oh, I agree that this isn't entirely ideal.

One of the design principals is that it shouldn't matter. But async processing is entirely non-trivial, so for this version I chose to break that principal to maintain reasonable parity with Silverlight. I'll probably live to regret it, but I didn't have the time/budget to make it entirely transparent.

I honestly don't know why you are seeing what you are seeing - async is complex, and it is clearly some unforseen interaction.

Fire-and-forget sounds nice, but is not a safe concept when dealing with general async environments. It is quite possible that the background thread is a child of the calling thread, and it dies when the calling thread completes. Or they may be peers (which I think is probably the case) and the backgroundworker is maintaining a back-reference of some sort to the calling thread.

My guess is that Remoting is getting confused because the 'main thread' completes, but still has some outstanding hook from the backgroundworker, so that thread can't really complete or be released, and this locks up Remoting. Total speculation on my part, but it sounds good :)

Switching to WCF might help - but I'd be nervous, because who knows what might happen under load as background callbacks occur onto a thread that is now running under a different user request, possibly in a different security context, etc.

Your best bet is probably to create and use a synchronization primitive (like an AutoResetEvent) that the background task can set when it completes, so the 'main thread' can block at the bottom of the DataPortal_XYZ method.

That's basically what I'd do if (or when) I address this in the data portal itself. You can be async on a server, but the 'main thread' can never complete until all outstanding async requests complete - that's a general rule of async processing.