Serializing an object to disk to use as an last resort persistence mechanism

Serializing an object to disk to use as an last resort persistence mechanism

Old forum URL: forums.lhotka.net/forums/t/3889.aspx


juddaman posted on Monday, November 12, 2007

Hi guys

If the user fails to save an object (it could even be invalid) before shutting down their PC I serialize the object to disk. When they next open the app I try and restore it, if it restores i tell the user "Data has been rescued, would you like to see it?" if they click yes it's loaded. I've had some issues with this, one being that as the whole object is stored, one private field that I don't want to be stored (for the purpose of emergency back up, it needs to be for cloning) is.  Does anybody have any alternative methods for this kind of saving? Or comments on this method?

Thanks, George

Curelom replied on Monday, November 12, 2007

You could set the property as NonSerialized

[NonSerialized()]

This should work if you don't need that property to be saved to the database, otherwise if you are remoting, it will be lost as remoting objects are serialized before they are transmitted.

juddaman replied on Tuesday, November 13, 2007

Hi, I can't mark the field [NonSerialized()] as I need the value in the DataPortal. It doesn't actually get saved to the database itself but it which values do.

E.g. (in DataPortal_XYZ)

int storeThisValue = -1;

if (FlagX)
{
   
storeThisValue = 5;
}
else
{
    storeThisValue = _ValueFromObject;
}

Where flag X is the field in question.

Any other suggestions?

Curelom replied on Tuesday, November 13, 2007

Could you add an OnDeserialized method and determine how it was deserialized from the streaming context?

protected override void OnDeserialized(System.Runtime.Serialization.StreamingContext context) {

if (context.State == System.Runtime.Serialization.StreamingContextStates.File) {

//do stuff

}

juddaman replied on Monday, November 19, 2007

Thank you both for taking the time to post. I tried using the State enum to get the context for serialization but couldn't get it working. I ended up just adding another property doing some fancy (messy) bits and got it working.

I'm actually more interested in what you guys think of the idea rather than the actual implementation. Anybody got any other ideas for taking a snapshot of the object quickly on shutdown. So if the user shuts down without saving what happens to the data? Also do you agree this is a client issue not a BLL issue? I've seen other posts on saving a invalid object, which at first seamed like what I wanted to do. After thinking about it however it's not, I actually didn't want to allow the user to store this data at all. I was mealy wanted to persist the data to the next Windows login so the user could continue as if they hadn't logged off at all (in a way). Any thoughts?

Thank you! George.

Curelom replied on Monday, November 19, 2007

I think it could become more hairy then it's worth.  I can see sync issues with the datastore the longer they aren't logged in, or if they log on a different box.  You might then need to do a check against the datastore to see how current it is and figure how you deal with the differences.  If your objects have lots of properties it may be worth it, otherwise, I think having the user re-enter isn't to big a deal.  Of course this depends entirely on you business cases.  Perhaps you can have the application prompt the user to save on a shut down.

juddaman replied on Monday, November 19, 2007

Hey cheers for gettings back. Let me explain a bit more...

I can see sync issues with the datastore the longer they aren't logged in, or if they log on a different box.

 I deal with this by using an application settings.So when the users logs on it checks the settings if its true it will try and restore the unsaved data if it doesn;'t find it, it forgets it! It kind of rare for users to change system so this is okay in my case.

You might then need to do a check against the datastore to see how current it is and figure how you deal with the differences.

Not an issue in this case as each user deals with there own bit of data.

I think having the user re-enter isn't to big a deal.

There isn't much data, however the users may have forgot what the data is :-) As the data is mainly concerned with there "leaving work time" which bring me to my next point... The users don't want to use the app lol, its a time tracking solution. ;-)

Perhaps you can have the application prompt the user to save on a shut down.

If the user closes the app, this is what happens and it works great. However on shutdown if the user doesn't  respond to the message box dialog quick enough the application just crashes. This is when I started to think of the store and retrive system.

I'm still very interested in storing another way though. Any ideas?







juddaman replied on Monday, November 19, 2007

I have considered

* using a timer to attempt to call save on the object every few minutes
* saving on property change

Of course both of these options don't deal with saving invalid objects, so I can see lots of message boxes poping up :-)  Also lots of database calls, as the nature of the objects means it will be dirty every 60 seconds! In most cases.










tmg4340 replied on Monday, November 19, 2007

juddaman:
You might then need to do a check against the datastore to see how current it is and figure how you deal with the differences.

Not an issue in this case as each user deals with there own bit of data.

I think having the user re-enter isn't to big a deal.

There isn't much data, however the users may have forgot what the data is :-) As the data is mainly concerned with there "leaving work time" which bring me to my next point... The users don't want to use the app lol, its a time tracking solution. ;-)

Perhaps you can have the application prompt the user to save on a shut down.

If the user closes the app, this is what happens and it works great. However on shutdown if the user doesn't  respond to the message box dialog quick enough the application just crashes. This is when I started to think of the store and retrive system.

OK... let me see if I'm following here.

You're developing, in essence, a timesheet application.  And you're trying to be helpful to the users, who all hate entering their time, and parsing it out by project, etc. etc. etc.

While saying that I applaud your efforts to be helpful, unless it's in the project specs, I'd probably drop the idea.  Unless you're going to allow partially-entered (i.e. invalid) data in the database, the hassle of trying to search for a local copy and attempting to merge it with data returned from the database will likely get you an ulcer and a healthy dose of contempt for your users.  Smile [:)]  Nobody likes to enter their time - but since it's how we get paid, we ought to pay enough attention to it to save our work before we close the app.  A "you have unsaved data - save?" messagebox on shutdown really ought to be enough, with invalid data probably being discarded (possibly with another message box.)  And if the app completely craps the bed - well, you'll have lots of incentive to find and fix the error.  Smile [:)]

Another option is to go the "Outlook route".  This really only works if it's a WinForms app, but I'm assuming that - otherwise you wouldn't be talking about local saves.  Microsoft calls it an Occasionally Connected Client, but the idea is that the user works entirely with a local copy of the data (using SQL Server Compact Edition), and a syncronization process is invoked on a save (or on a recurring schedule) that sends the local data to the server and pulls down new data since the last update.  Microsoft has actually created a halfway-decent framework to do this already - if you go to http://msdn.microsoft.com/sync/, you'll see the latest incarnation.  Using this scenario, there's no such thing as "saving local as a last resort" - it's always saved locally.  This gives you the option of relaxing the local schema to allow for invalid data, or to add additional data fields to mark "partially-saved data", without polluting your database - you control what gets updated to the server store.  It also allows your users to work without being connected to the network, updating the data to the main server when they re-connect to the network.  It's not the Holy Grail of frameworks - I'm not even sure the latest version is considered "RTM".  There's also a healthy dose of "you need VS 2008 and SQL Server 2008 to use this", but I'm pretty sure that's not true.  If nothing else, the previous version works with VS 2005 and SQL Server 2005 pretty well.

I realize that retro-fitting this into an existing project is not necessarily easy - I did it a while ago on a project, and it was a bit of a pain.  But if you really need to do this - and like I said, unless it's in the specs, I'd think long and hard before committing to it - I'd check it out.  I'm not trying to sound harsh - I just see a classic high-risk/low-reward scenario with trying to tackle this, especially if you're going to try and write your own code.

juddaman replied on Tuesday, November 20, 2007

Many thanks for your detailed response.

I'd jus tlike to say, the system I have impemented works already :-) I.e. if you shutdown without saving and restart the system (which is the case 80% of the time) your data will be recovered for the serialized file (80% of the time hehe). That's okay its not vital that this data is restored, just helpful, the users actually like this part of the system:-O

The MS Snyc systems sounds interesting I will take some time to explore it.

This gives you the option of relaxing the local schema to allow for invalid data, or to add additional data fields to mark "partially-saved data".

This sounds like it could work, though I'd like to know, before the data got synd to the central database would it be able to be routed through CSLA buisness rules?

 A "you have unsaved data - save?" messagebox on shutdown really ought to be enough
I agree but I had trouble with this, if the user didn't click Yes or No quick enough Windows would force the app to close and the app would crash. Also if there was invalid data and it couldn't save, Windows would some times crash it in its rush to shutdown the system. Any ways around this? If there is in a future project this would be the way to go I think.

Tar again, George.


tmg4340 replied on Tuesday, November 20, 2007

juddaman:
I'd jus tlike to say, the system I have impemented works already :-) I.e. if you shutdown without saving and restart the system (which is the case 80% of the time) your data will be recovered for the serialized file (80% of the time hehe). That's okay its not vital that this data is restored, just helpful, the users actually like this part of the system:-O

Ah, the infamous 80/20 solution.  It's that last 20% that makes this fall into the "is this really necessary?" category...

juddaman:
The MS Snyc systems sounds interesting I will take some time to explore it.

This gives you the option of relaxing the local schema to allow for invalid data, or to add additional data fields to mark "partially-saved data".

This sounds like it could work, though I'd like to know, before the data got synd to the central database would it be able to be routed through CSLA buisness rules?

The synchronization process is entirely controlled by you - it's not done automatically.  So you can sync with the server whenever you want.  Since it really has to happen after the data is saved to the local copy, I would assume your data has already been through your business rules.

juddaman:
A "you have unsaved data - save?" messagebox on shutdown really ought to be enough
I agree but I had trouble with this, if the user didn't click Yes or No quick enough Windows would force the app to close and the app would crash. Also if there was invalid data and it couldn't save, Windows would some times crash it in its rush to shutdown the system. Any ways around this? If there is in a future project this would be the way to go I think.

So you're trying to save data in case of an unexpected shutdown?  If the user closes your app through normal means - even the "big red X" - Windows won't close the app until the user responds to the message box.  If the messagebox comes up and the user never responds - well, we can only be so helpful.  If the app is hung, and they close it through the "End Task" button on the Task Manager - again, you have much incentive to find and try to fix the problem.  Smile [:)]

If your app is up and running, and Windows closes the app due to shutdown/restart/whatever, then you do have an option, depending on your environment.  In a .NET 2.0 WinForms apps, the FormClosing event on your form gives you a FormClosingEventArgs object, which contains a "CloseReason" property.  This tells you why the form is closing, and would allow you to differentiate between the user closing the form and Windows killing the app.  Depending on what you need, you could bypass the messagebox for certain close reasons and just save the data.  You still have to be quick about it - Windows won't wait forever - but a serialization to disk shouldn't take a ton of time.  From my way of thinking, in those cases, it doesn't make a lot of sense to ask the user about whether they want to save their partially-valid work - if it's that important, just save it for them.

juddaman replied on Tuesday, November 20, 2007

Since it really has to happen after the data is saved to the local copy, I would assume your data has already been through your business rules.

But I need to be able to put Save invalid data locallly then when it is synced to server it need to be validated at that point. But if it is like you say, all manual that shouldn't be a problem right.

I am using the close reason property... exactly a syou describe actually. If the user closes the app by clicking the cross they are ask do you want to save. If its close becuase of shutdown etc I do the
serialization. This happens a lot becuase the app is running minimized most the, so the user forget to open it (show it) and close it before they shut down. "Windows won't wait forever.." this was the problem, if the user never responded it quit, that why I decided do the serialization thing. I was just looking for other approaches, or somebody to say "yeh I've done that before" :-)

"
it doesn't make a lot of sense to ask the user about whether they want to save their partially-valid work - if it's that important, just save it for them."

I can't save it for them if its only partially valid though! Maybe I miss understand you?

George

tmg4340 replied on Tuesday, November 20, 2007

OK... I see what you're getting at.  Some of my answer might depend on how deep your object hierarchy is, though in theory this shouldn't really matter to child objects.  This assumes you went the Sync Framework route, using SQL Server Compact Edition as your local data store.

What I would probably do, in order to make my life simpler, is to create a second set of tables to store the invalid objects.  That way, you aren't mixing your data locally.  That will make the sync process a lot easier, since you'll only synchronize the tables containing valid data.  The invalid objects would be saved to your second set, and upon a restart you would hydrate your objects from the invalid set of tables, clear them out, and apply your normal business rules.  Then, if the data passes muster later, it gets saved to the "clean set" of tables.

Given that no synchronization of the invalid data is involved, your invalid tables could be in an entirely different database - one database equates to one file in SqlCE, and it's fairly easy to create databases in code.  Using that route, your invalid database could be deleted after it performs its startup task, thus providing a bit of simplification to your startup routine.  The only kicker there is that you'd have to re-create the database on shutdown, which could take time, depending on how involved your structure is.

As for handling the actual save, I'd probably add a property to my root object - something like "AllowInvalidSave" - that could be set right before the local save happens.  Then I'd override the "Save" method, checking that flag to know whether I should rely on the base implementation or use my own implementation.  The base "Save" implementation isn't very long, so replicating it while modifying the IsValid checks wouldn't be a whole lot of code copying.  As best as I can remember, the DataPortal doesn't give a whit about "IsValid", so you should be good to go there.

The kicker is that your current implementation can just do a simple Serializer.Serialize() and call it a day; the database technique requires that you take that "AllowInvalidSave" property and propagate it throughout your hierarchy, so your objects know where to save themselves.  But the database technique is a little more robust, since you don't have to worry about the version issues with serialization.  For what it's worth, it will probably also result in a smaller disk file - .NET serialization, even the binary kind, can get pretty bloated.  SqlCE's database overhead is pretty minimal.

Obviously, a lot depends on whether you want to go the sync route.  If your existing solution works 80% of the time - well, maybe that's enough.  There are a few "gotchas" to working with SqlCE - things like no stored procedures or views in the database - that can make a retro-fit somewhat of a pain.  It really changes the way your DAL works - essentially, what you have to do while coding is to forget about the server database and code your DAL against the local store and how that works.  The synchronization is sort of a side piece.  There can be advantages to using native .NET code instead of stored procedures, but it really depends on how your SP's are constructed.  And if you have some complicated views that you rely on a lot, those can be somewhat painful to replicate in a SqlCE environment with .NET code - you could end up writing a lot of DataTable-merging and filtering code.

I know this is getting outside of your original question, but if you really are going to consider going the sync route, take some time to investigate SqlCE.  It's not a bad product, but it imposes certain things on your programming model that you should know before you start.  The documentation will tout this model as a positive, and from a certain perspective it is.  But it started life as a product for embedded databases on mobile devices - it was never designed to be a full-featured DB, and it probably never will be.  It can be a different way of writing database code - almost like a step back - but it performs very well.

juddaman replied on Wednesday, November 21, 2007

I do agree the Sync route would be more robust and the implementation you outline sounds reasonable. The overhead does seam huge though. However, I don’t think it would solving the solution in the correct place. By this I mean I don’t think the BLL should have to change (too much) to accommodate the requirement. The sync method is great for offline apps buts that’s not needed in this case. Synchronization as a concept is not needed at all actually, if the user has altered the data since the “copy” was taken, the “copy” is scraped not merged. It is this reason I think the client should deal with the problem. If the client could “STOP” Windows shutting down and wait for the user to respond to the “Do you want to save?” dialog the ‘backup until next start’ system would not be needed at all. So what can the client do to preserve the data bar serializing the object?

tmg4340 replied on Saturday, November 24, 2007

Well... if you don't think your business objects should change to manage this requirement, then your current solution is probably fine.  It all depends on how you look at this.  As far as I know, there's no way for a client application to override Windows' process timeout - that's why you get the "I'm going to shut this down, you might lose data unless you do something" window.  At the end of the day, Windows has to be able to complete its shutdown process.

One last option to consider - periodic automated saves.  You still have to deal with the problems you have now, but you would be pretty much guaranteed to be within your application session, and once the data is permanently saved, you can dump your "draft copies".  The user might still lose a little information, but you'd get most of it.

tmg4340 replied on Tuesday, November 13, 2007

You know whether you're loading the data via your file persistence - perhaps you can create a second property on your object like "IsLoadedFromFile" that you can then work with inside your object.  That would be marked "NonSerialized()".

If that doesn't work, the only way I see out of this is to do your own custom serialization - i.e. directly implement "ISerializable" and take advantage of the "State" property of the StreamingContext object.  That is supposed to define what kind of serialization operation is taking place, so you should be able to differentiate between serializing to a file and serializing for other processes.  Then you can fall back on the standard serialization for other contexts, but write your own for the file context.

It's a pain - especially if you have an object graph to contend with, since you'll probably have to extend your custom serialization throughout the entire graph - but if you only want the field serialized under certain conditions, and you can't use the second property, this is the only way I know of to get around this.

HTH

Copyright (c) Marimer LLC