Single, centralized collection with heartbeat functionality

Single, centralized collection with heartbeat functionality

Old forum URL: forums.lhotka.net/forums/t/7900.aspx


nhwilly posted on Wednesday, October 28, 2009

I'm not sure where to start with this question, it's a bit of an odd duck.

I am building part of an application that monitors application usage on a public use station. 

Think of DHCP, but for application licenses.  It's actually pretty close.

We have to dynamically assign a license to a user wherever they sit.  The method for doing this is supported by the licensed application itself and encourage by the publisher, so that's not a problem.

But when an application crashes, the system is rebooted or for whatever reason the application isn't in use, the license needs to be freed up for another station.

In an orderly scenario, we would do this all through normal business objects and the database.  But how to detect that an application is no longer in use when it can't tell me that?

My first thought was a class with a shared variable containing a collection, but after reading the posts here, that seems like it could be problematic. 

Rocky wrote that singletons are a problem to deal with locking and so on.

All the traffic (at this point) is LAN based and we probably won't see more than 200 stations making these requests.  We also figure that about 5 seconds is right for the heartbeat.  I doubt that there would be more than 150 application launch events in any given hour (generally MUCH less).

Override the data portal calls to update a heartbeat collection with a heartbeat object?  Maybe keep an anticipated heartbeat timer on the server side and when it fails assume that the application is no longer in use?

Surely someone must have a pattern for this kind of beast.

Any help or ideas would be great,

Thanks in advance, guys.

 

RockfordLhotka replied on Wednesday, October 28, 2009

I think the heartbeat idea is on track - that's a pretty common solution.

But 5 seconds? That seems really frequent to me, but you know your scenario better than I.

There's a game called Supreme Commander (wonderful game), which uses a portal called GPGnet. GPGnet uses a login/logout scheme, but if it crashes you are left "logged in" for a few seconds. Around 30 from what I can tell. So you have to wait 30 seconds to reconnect after a crash, or it says you are still logged in somewhere.

Same basic principal.

To implement a hearbeat model from the client side is relatively easy - you just need a system or dispatch timer that wakes up every n seconds and calls a service on your server. This could be a data portal call with a command object, or you could have a totally seperate WCF service you call. Either way, you need to send the user token in the call.

On the server you'll need some shared memory scheme. If you have just one server you could do this in memory, but if you have a server farm you'll need shared memory, and the easiest way to get that is to use a SQL table. The other advantage of a SQL table is that you don't need to implement a singleton since the database engine will do the locking for you - so personally I'd look strongly at using a SQL table.

And on the server you'll need a service that scans the table looking for dead rows. One would assume it will scan every n seconds as well, or perhaps 2n or 1.5n - something like that. When it finds a dead entry it would remove it, and unlock that key. This could be done using SQL Server features, or a Windows Service.

ajj3085 replied on Thursday, October 29, 2009

RockfordLhotka:
And on the server you'll need a service that scans the table looking for dead rows. One would assume it will scan every n seconds as well, or perhaps 2n or 1.5n - something like that. When it finds a dead entry it would remove it, and unlock that key. This could be done using SQL Server features, or a Windows Service.

We had a web application where invoices could be locked, and we used a similar scheme.

In our implementation though, each "heartbeat" was recorded with a date / time stamp in the database.  If another user attempted to lock, it would fail unless the lock had become "stale."  If a lock was requested and the previous lock had gone stale, the lock would be transfered to the new user.

This worked well, and eliminated the need for a dedicated clean up service to be running, as stale locks are automatically recycled.

nhwilly replied on Thursday, October 29, 2009

Thanks for that.

But how do you control when the locks go stale?

ajj3085 replied on Friday, October 30, 2009

Via a stored procedure, which would get the current date / time and the lock's time.  If the difference between the lock time and the current time is greater than some defined timeout, the lock is stale.  You could probably pass that as a parameter.

dlambert replied on Thursday, October 29, 2009

+1 Heartbeat. I'd recommend that the heartbeat interval be configurable, because you're almost certain to run into instances where you want to change it.

The simplest solution would be to have an asynchronous object on the client that issues "keep-alive" pings to the server at whatever interval you want.

Obviously, this is going to result in more traffic than you really need, so the next level of complexity would be to hook into the normal traffic from client to server such that a licence is auto-renewed every time you hear from the client. In this case, you'd need to decide whether the client or the server initiates the "we haven't spoken in a while" conversation. Either the client tracks "X" time since the last conversation and sends a keep-alive ping, or the server tracks "X" time since it last heard from the client and sends an "are you there" ping. In either case, of course, failure lets the server reclaim the license.

The other part of this pattern is handling a client that comes back after its license is reclaimed - you either give it a new license, or lock it out, I guess. You often see something like this implemented in message queues, where you have guaranteed delivery of messages with associated retries and so on.

Copyright (c) Marimer LLC