Dataportal with compression

Dataportal with compression

Old forum URL: forums.lhotka.net/forums/t/2653.aspx

xal posted on Tuesday, April 03, 2007

Hey all!
I started a new blog over here and to warm things up I wrote a small article about compression in the dataportal. I hope it's of some use...

Have fun!

Andrés

Vaidal replied on Monday, April 09, 2007

Ohhh Great!!!

Can you write here some test results showing the benefits?

xal replied on Monday, April 09, 2007

I haven't found the time to do any thorough testing, but I can tell you that I loaded a huge collection around 1.3 MB when serialized without compression, and the result was ~350KB. That makes a hell of a difference when working over the internet or slow networks, so the processing overhead is insignificant compared to the time you save when transfering the actual data...

There is a better way of doing this, which involves writing a custom sink for remoting. That wouldn't involve creating a custom dataportal, because you could use the sink with the existing one. I'll try to look at that in the future, for now, it was easier to play in csla land since I'm familiarized with the way it works, not so much with remoting. A custom sink may be even better performing, but I'll have to try and see at some point.

Andrés

askywalker replied on Tuesday, April 10, 2007

Nice blog article- looking forward to taking a closer look ... just need to find a round tuit...

My app does occasional 'large' collections where I know that compression would help - but the vast majority of transfers are small.  I don't know anything about a 'custom sink'... I have been considering genuine channels - do you know of it?  I searched the CSLA forums unsucesfully - surprising since it seems to be a popular product and sounds like it could help.  Compression and encryption both worry me - I really don't have much of a clue what is going on with the remoting/dataportal layers...

 

xal replied on Wednesday, April 11, 2007

I'm currently working on an improved version based on Rocky's comments, should be out today or tomorrow.
I haven't heard of genuine channels, but for now, I want to keep it simple. Besides, from what I see in their web page it's not a free product, so I don't think I'll delve into it or even if I do, release something based on that.

The intention is to have a simple way to use a compressed portal without needing to install / buy anything.
The compression algorithm for bzip2 is awesome, and it serves it's purpose, so for now that will be my main focus...

Andrés

xal replied on Monday, April 23, 2007

I finally got around to post the followup. Sorry for the delays.
You can read about the new version here.


Andrés

Cslah replied on Sunday, June 03, 2007

I'm looking into switching this over to .NET 2.0 compression built into the System.IO.Compression namespace to keep things simple (less code to download/compile/keep track of). It seems thte compression namespace will compress the streams just fine, only changes one line of code in each method. Has anyone dabbled with this?

Cslah replied on Sunday, June 03, 2007

OK, I've converted this to .NET Compression and it works great. To do this is a few simple changes:

Compression.vb:
Remove the Imports to the open source compression library
Line 10:
Dim compressedStream As New System.IO.Compression.GZipStream(ms3, IO.Compression.CompressionMode.Compress)
Line 18
Dim decompressedStream As New System.IO.Compression.GZipStream(ms4, IO.Compression.CompressionMode.Decompress)

Thanks Andrés for doing this, it's totally awesome!! We're going to use this instead of the normal portal.

Rocky, would you be willing to put this namespace/code into the next Csla version? I'll convert it to C# & send the code to you if so..

Brian

Copyright (c) Marimer LLC