CSLA Light - Adding JIT object compression to WCF calls

CSLA Light - Adding JIT object compression to WCF calls

Old forum URL: forums.lhotka.net/forums/t/5437.aspx

paupdb posted on Sunday, September 21, 2008

Following on from Ashley's post, I've been looking at ways to compress the objects being sent over the wire between our Silverlight client and the CSLA.Net server.

My first choice was to look for some kind of compression mode/option for the WCF transport itself, but I've found very little information on this and I wasn't too confident that Silverlight would play nicely with the few options I found on the web.

So then I poked around the CSLA WCFProxy itself to see if I could inject some code to do the compression/decompression of the serialised object behind the scenes.
What I've come up with is fairly hacky at the moment and the worst part about it is that I have had to change CSLA framework classes to enable my solution to work.  But I don't think there's much option since once my code hands over to the DataPortal, its all CSLA from there on in.

So what I did is as follows:

1. Added reference to SharpZipLib (normal .Net) into CSLA and SharpZipLib (silverlight version) to CSLALight. 
SharpZipLib seems to be the only compression library available for Silverlight and normal .Net at the moment, and it works great.

2. Added the following compress/decompress code to Csla.Utilities in both CslaLight and CSLA.
Code is ripped straight from Peter Bromberg's excellent Silverlight data tutorial

#region Compression
    public static byte[] Compress(byte[] byteData)
    {
      try {

        MemoryStream ms = new MemoryStream();
        ICSharpCode.SharpZipLib.Zip.Compression.Deflater defl =
           new ICSharpCode.SharpZipLib.Zip.Compression.Deflater(9, false);
        Stream s =
            new ICSharpCode.SharpZipLib.Zip.Compression.Streams.DeflaterOutputStream(ms, defl);
        s.Write(byteData, 0, byteData.Length);
        s.Close();
        byte[] compressedData = (byte[])ms.ToArray();
        return compressedData;
      }
      catch {
        throw;
      }

    }

    public static byte[] Decompress(byte[] byteInput)
    {
      MemoryStream ms = new MemoryStream(byteInput, 0, byteInput.Length);
      byte[] bytResult = null;
      string strResult = String.Empty;
      byte[] writeData = new byte[4096];
      Stream s2 =
         new ICSharpCode.SharpZipLib.Zip.Compression.Streams.InflaterInputStream(ms);
      try {
        bytResult = ReadFullStream(s2);
        s2.Close();
        return bytResult;
      }
      catch {
        throw;
      }
    }

    private static byte[] ReadFullStream(Stream stream)
    {
      byte[] buffer = new byte[32768];
      using (MemoryStream ms = new MemoryStream()) {
        while (true) {
          int read = stream.Read(buffer, 0, buffer.Length);
          if (read <= 0)
            return ms.ToArray();
          ms.Write(buffer, 0, read);
        }
      }
    }
    #endregion

3. Added code intothe getters and setters of Csla.Server.Hosts.Silverlight.UpdateRequest and Csla.Server.Hosts.Silverlight.WcfResponse (these are in the Csla.Net project), along the lines of:

    private byte[] _objectData;
    [DataMember]
    public byte[] ObjectData {
      get
      {
        return Utilities.Decompress(_objectData);
      }
      set
      {
        _objectData = Utilities.Compress(value);
      }
    }

4. Added the same kind of code into the References.cs generated versions of UpdateRequest and WcfResponse on the CslaLight project.  I know this is nasty since this code is generated and so subject to change, but this is a proof of concept so meh.


And thats it.  We've tested a few examples with really good results. 
For example what used to be an 11MB serialised object graph fetch has now shrunk to 100KB over the wire :)
There is definitely a bit of impact on the responsiveness of the app since there is now processing overhead for the compress/decompress actions, but its quite tolerable.

The above is very rudimentary, but with a bit more work I could probably make the compression/decompression actions conditional based on say a web.config appsetting. 
Or I could make the compress/decompress only kick in when the object graph gets larger than a configurable size. That way small object graphs can stay free of compression, and only the bigger ones will incur the compress/decompress overhead.


My question to the Magenic guys is: 

Any chance we can add in some kind of CompressedDataPortal style functionality to CslaLight?

Failing that, what of my solution above?  Any comments, suggestions on what I could do to minimise/remove having to change the CSLA framework directly to implement my desired compression over the wire behaviour?

I would think that given Silverlight's internet leaning , providing some kind of compression options to reduce the size of the over-the-wire objects would be a great feature to add to CSLA.

RockfordLhotka replied on Monday, September 22, 2008

That is a great post - thank you!

We'll look into ways you can do this without modifying existing CSLA code.

It may be possible now - I haven't gone through it. But the primary extensibility mode for the data portal is for you to create your own WcfProxy and WcfHost classes that encapsulate any custom behavior. Like the pure-.NET data portal, you can entirely replace the communication channel used by the data portal.

But we'll look through that to make sure it is possible to achieve this scenario in some manner along that line.

I'm not real interested in taking a dependency on an external library into core CSLA (already have one - don't like it, but can't really avoid it), but I do want to make it practical as an add-on to CSLA.

But can't you just use http 1.1 compression? This was my expectation - that people would just enable http compression through IIS and be done with it?

paupdb replied on Monday, September 22, 2008

Hi Rocky

I agree that putting a SharpZipLib dependency into CSLA is a bad idea. 
I'll definitely have a look into extending the WcfProxy in order to build my compress/decompress functionality on top of CSLA rather than into it.  Do you have any examples of how I should go about this?

Regarding the HTTP 1.1 compression, that was also my first idea but searching the web for some examples of how to implement this with WCF seemed to indicate that WCF does not in fact support HTTP 1.1. compression out the box.

This MSDN forum post in particular turned me off the idea of trying to use IIS to compress the responses.  Additionally, the HTTP 1.1 compression is based on GZip, which Silverlight does not have - so I didn't want to go through trying to get that working only to hit issues when trying to decompress/compress in the SL client.

If I've missed something with the HTTP 1.1 compression I'm all ears :)

Cheers

Paul

RockfordLhotka replied on Tuesday, September 23, 2008

I don’t have any examples of replacing the proxy/host in Silverlight. It is not terribly hard though, because everything is interface-based.

 

You can, for example, copy the WcfProxy and WcfHost classes into your own projects (SL and .NET respectively) and configure the client to use your proxy class, and the server’s WCF endpoint to your use host class and you are set.

 

Rocky

 

 

From: paupdb [mailto:cslanet@lhotka.net]
Sent: Monday, September 22, 2008 9:23 PM
To: rocky@lhotka.net
Subject: Re: [CSLA .NET] CSLA Light - Adding JIT object compression to WCF calls

 

Hi Rocky

I agree that putting a SharpZipLib dependency into CSLA is a bad idea. 
I'll definitely have a look into extending the WcfProxy in order to build my compress/decompress functionality on top of CSLA rather than into it.  Do you have any examples of how I should go about this?

Regarding the HTTP 1.1 compression, that was also my first idea but searching the web for some examples of how to implement this with WCF seemed to indicate that WCF does not in fact support HTTP 1.1. compression out the box.

This MSDN forum post in particular turned me off the idea of trying to use IIS to compress the responses.  Additionally, the HTTP 1.1 compression is based on GZip, which Silverlight does not have - so I didn't want to go through trying to get that working only to hit issues when trying to decompress/compress in the SL client.

If I've missed something with the HTTP 1.1 compression I'm all ears :)


sergeyb replied on Monday, September 29, 2008

Csla version 3.6 has been extended to provide more generalized implementation of the requested compression functionality.  Silverlight WCF Proxy and corresponding host classes have been expanded to contain virtual methods to convert request and response objects prior to sending them across the wire.  As a result, you can create two custom classes (one inherits from WcfProxy<T>, the other one from Csla.Server.Hosts.Silverlight.WcfPortal) and provide additional functionality.  This implementation is not only allowing you to create a compressed proxy, but also opens the door for other potential issues, such as encryption.  There is a sample project under “samples\trunk\CslaLight\cs(and vb)\”  The project is called RemotePortalWithCompressedProxy. It is using ICSharpZipLib to create a compressed proxy.  Please feel free to take a look.  Please note that .ClientCOnfig file has been updated as well as web.config and .svc file to refer to new proxy and server.  Also, Silverlight app upon startup sets up new proxy type.

Thanks for the suggestion and let us know if you have any questions.

Sergey.

paupdb replied on Monday, September 29, 2008

Sergey, thanks for this - sounds excellent.

I'll definitely give that code a run as soon as I'm finished converting my app to RC0 (which has been a crappy experience thus far).

-Paul

paupdb replied on Monday, September 29, 2008

I've implemented the compression functionality as per your sample project and was able to get up and running with it very quickly. 
Thanks Sergey, this is a very easy and clean solution.

One little bug in the Sample project - if you submit a request with no Criteria specified (i.e. request.CriteriaData is null), then the code bombs trying to compress a null object.

FYI, I'm going to probably implement this functionality only on the ObjectData and only when the ObjectData is over a certain size. 
So I'll leave the Criteria, Global and UserContext components uncompressed to try to save unnecessary compress/decompress overhead.
The great thing about your implementation is that I am able to have this level of control :)

sergeyb replied on Tuesday, September 30, 2008

Great to hear that.  I will fix the sample project once I have a second.

 

Sergey Barskiy

Principal Consultant

office: 678.405.0687 | mobile: 404.388.1899

cid:_2_0648EA840648E85C001BBCB886257279
Microsoft Worldwide Partner of the Year | Custom Development Solutions, Technical Innovation

 

From: paupdb [mailto:cslanet@lhotka.net]
Sent: Monday, September 29, 2008 8:15 PM
To: Sergey Barskiy
Subject: Re: [CSLA .NET] RE: CSLA Light - Adding JIT object compression to WCF calls

 

I've implemented the compression functionality as per your sample project and was able to get up and running with it very quickly. 
Thanks Sergey, this is a very easy and clean solution.

One little bug in the Sample project - if you submit a request with no Criteria specified (i.e. request.CriteriaData is null), then the code bombs trying to compress a null object.

FYI, I'm going to probably implement this functionality only on the ObjectData and only when the ObjectData is over a certain size. 
So I'll leave the Criteria, Global and UserContext components uncompressed to try to save unnecessary compress/decompress overhead.
The great thing about your implementation is that I am able to have this level of control :)


JoeFallon1 replied on Tuesday, September 30, 2008

What an awesome thread! An excellent post requesting increased functionality in Csla and showing one way to do it. Compression is something that could be very useful to many developers and we won't all know how to implement it. Then a change to the framework allowing it to be implemented in a flexible fashion just a couple of days later!

Great job and thanks to all involved.

Joe

 

zinovate replied on Tuesday, September 30, 2008

I agree, great post. I can't wait to give it a shot.

James Thomas replied on Friday, April 10, 2009

I've recently started using compression in my application, that was based on Rocky's blog post: http://www.lhotka.net/weblog/SettingUpABasicNtierCSLANETForSilverlightProject.aspx

I discovered the new InventoryDemo sample app and copied how it does compression. Posting this in the hope that it helps someone else, but I'd also be grateful if someone who knows more than me could just check through to see if I've got it right. (My application works ok, so I'm guessing it's ok...)

Steps to implement compression a la InventoryDemo.

1. Copy the ZipBin folder into your solution.

Server business objects project:
1. Create folder called 'Compression'
2. Add CompressedHost.cs and CompressionUtility.cs from the InvLib.Server project.
3. Update the namespaces in these files.
4. Add a reference to ICSharpCode.SharpZipLib.

Client business objects project:
1. Create folder called 'Compression'.
2. Add a link to your CompressionUtility.cs file (in server objects).
3. Add 'CompressedProxy.cs' from the InvLib.Client project.
4. Update the namespace in CompressedProxy.cs.
5. Add a reference to SharpZipLib.

Web service project:
1. Find <service... section in <system.serviceModel> and change the name to 'YourBusinessLibrary.Compression.CompressedHost'.
2. Do the same in WcfPortal.svc.

Silverlight application:
1. Add this line to app.xaml.cs Application_startup: Csla.DataPortal.ProxyTypeName = typeof(BusinessLibrary.Compression.CompressedProxy<>).AssemblyQualifiedName;

RockfordLhotka replied on Friday, April 10, 2009

Thanks James, that's great! I added it to the FAQ too

http://www.lhotka.net/cslanet/faq/SilverlightDataPortalCompression.ashx

dshafer replied on Wednesday, November 04, 2009

You guys rock!  I just used these instructions and in less that 20 minutes took an xml reponse of over 2560 kb down to 88kb.

Thanks for the great posts,

Dustin

marthac replied on Thursday, September 30, 2010

OK I followed those steps exactly and have my project set up just like InventoryDemo. But when I run I get this error with the very first DP call:

"The remote server returned an error: NotFound"

I set a break point in the CompressedProxy.ConvertRequest function and it gets there OK. But it never gets any further from what I can tell.

Does anyone have any ideas as to what I may be doing wrong?

RockfordLhotka replied on Thursday, September 30, 2010

It is almost always a configuration issue. Client config, server config and svc file - they all need to agree.

marthac replied on Thursday, September 30, 2010

I checked that. Here they are in case I missed something obvious:

Web.Config:

 <system.serviceModel>
  <behaviors>
   <serviceBehaviors>
    <behavior name="WcfPortalBehavior">
     <serviceMetadata httpGetEnabled="true"/>
     <serviceDebug includeExceptionDetailInFaults="true"/>
    </behavior>
   </serviceBehaviors>
  </behaviors>
  <bindings>
   <basicHttpBinding>
    <binding name="BasicHttpBinding_IWcfPortal" closeTimeout="00:10:00" openTimeout="00:10:00" receiveTimeout="00:10:00" sendTimeout="00:10:00" maxBufferSize="10000000" maxReceivedMessageSize="10000000">
     <readerQuotas maxStringContentLength="10000000" maxArrayLength="10000000" maxBytesPerRead="10000000" />
     <!--<security mode="TransportCredentialOnly">
      <transport clientCredentialType="Windows" />
     </security>-->
    </binding>
   </basicHttpBinding>
  </bindings>
  <services>
   <service behaviorConfiguration="WcfPortalBehavior" name="Profiler.Core.Compression.CompressedHost">
    <endpoint address="" binding="basicHttpBinding" contract="Csla.Server.Hosts.Silverlight.IWcfPortal" bindingConfiguration="BasicHttpBinding_IWcfPortal">
     <identity>
      <dns value="localhost"/>
     </identity>
    </endpoint>
   </service>
  </services>
 </system.serviceModel>

WcfPortal.svc:

 <% @ServiceHost Service="Profiler.Core.Compression.CompressedHost" %>

ServiceReferences.ClientConfig:

 <system.serviceModel>
  <bindings>
   <basicHttpBinding>
    <binding name="BasicHttpBinding_IWcfPortal" maxBufferSize="10000000"
                    maxReceivedMessageSize="10000000" receiveTimeout="00:10:00" sendTimeout="00:10:00" openTimeout="00:10:00" closeTimeout="00:10:00">
    </binding>
   </basicHttpBinding>
  </bindings>
  <client>
   <endpoint address="http://localhost:2374/WcfPortal.svc"
                binding="basicHttpBinding" bindingConfiguration="BasicHttpBinding_IWcfPortal"
                contract="WcfPortal.IWcfPortal" name="BasicHttpBinding_IWcfPortal" />
  </client>
 </system.serviceModel>

espenrl replied on Friday, January 08, 2010

Great stuff! It was up and running in no time.

I'm using signed assemblies in my projects , and thus all referenced assemblies has to be signed. The SharpZipLib assemblies were not, and I had to make signed versions of them. Thought I would share this with you guys. Files attached.

Cyber replied on Thursday, June 10, 2010

Hi,

regarding the section:

Web service project:
1. Find <service... section in <system.serviceModel> and change the name to 'YourBusinessLibrary.Compression.CompressedHost'.
2. Do the same in WcfPortal.svc.

I currently have:

 

<services>
            <service behaviorConfiguration="HeavonyLight.Web.HeavonyServiceBehavior" name="HeavonyLight.Web.HeavonyService">
                <endpoint address="" binding="customBinding" bindingConfiguration="HeavonyLight.Web.HeavonyService.customBinding0" contract="HeavonyLight.Web.HeavonyService"/>
                <endpoint address="mex" binding="mexHttpBinding" contract="IMetadataExchange"/>
            </service>
        </services>

I use Silverlight 4.0 and have downloaded the relevant 4.0 samples from the project page. I am 10/10 up till this step. Now I'm stuck.

Any clues on a likely configuration with this CustomBinding stuff in place?

Cheers.

EDIT:

I went into the test application and ripped out the plumbing from there.

I'll let you know how my experience goes, Im guessing I need to read up on how channels work some more :)

 

Thanks.

 


tiago replied on Thursday, September 30, 2010

Hi all,

I loved the solution, specially how fast everything can be up and running. I admit I didn't know InventoryDemo uses compression. One point remains to be solved (and not a minor one): the license model.

The Silverlight version of SharpZipLib referenced is SharpZipLib.dll dated Aug 5 2008 and mentions a copyright by HP(???). There are later versions for Silverlight3, Silverlight4 and WindowsPhone7 dated Aug 14 2010.

Both SharpZipLib (the .NET version) and SharpZipLib.Silverlight are GPL licensed but both have an exception.

The former states the exception clearly on their home page http://www.icsharpcode.net/opensource/sharpziplib/ 

The later uses Codeplex that provides limited licensing functionality. So the same exception can be read on a "Discussions" message http://slsharpziplib.codeplex.com/Thread/View.aspx?ThreadId=85286

The exception is the same on both products and states:

// As a special exception, the copyright holders of this library give you
// permission to link this library with independent modules to produce an
// executable, regardless of the license terms of these independent
// modules, and to copy and distribute the resulting executable under
// terms of your choice, provided that you also meet, for each linked
// independent module, the terms and conditions of the license of that
// module.  An independent module is a module which is not derived from
// or based on this library.  If you modify this library, you may extend
// this exception to your version of the library, but you are not
// obligated to do so.  If you do not wish to do so, delete this
// exception statement from your version

So this is kind'a chameleon license. If we distribute/sell a product under CSLA license, we can include this library under the same umbrella and we aren't obliged to open source the applications that use these libraries.

RockfordLhotka replied on Thursday, September 30, 2010

Keep in mind that CSLA has absolutely no dependency on SharpZipLib - we just use it for samples because it is free. If you have issues with that component or its license, there are other (commercial) compression libraries available, and you should be able to use them in place of SharpZipLib in your code with no problem.

Jaans replied on Thursday, September 30, 2010

I'm not sure how its licensing compares but have a look at DotNetZip library

We were a bit frustrated with SharpZipLib (assembly size for download to the Silverlight client, and incompatibility with .NET compression, and a few others). After some looking around found DotNetZip library - we adapted the ZLib component and created a .NET and a Silverlight version. We plugged this into CSLA and ripped out SharpZipLib. (Ps: the ZLib component is the core ZLib compression engine only, without extra other fluff like files/archives/etc.)

The did some extensive testing which showed a twofold improvement that not only reduced the Silverlight DLL size from 139Kb to 79Kb, but it also give a smaller and faster compression speed than SharpZipLib did - (about 3-8% in our experience with CSLA Binary business objects). Been using it since.

I've been trying to work with the project owner to get the Silverlight version to be part of his release library, and offered to do the work. We are not there yet but he agreed to it. Since then I have added the Silverlight project to the solution and checked it in, but unfortunately he doesn't have the VS2010 or the Silverlight tools for VS2008 and not very quick to make changes to the release DLL's. Also, he keeps the key file for signing the project out of the public source (understandably so). The bottom line is you cannot download a signed Silverlight ZLib DLL from the codeplex project yet, but you can easily build your own (using your own key file).

Alternatively, I can send you his .NET and my adapted Silverlight ZLib assembly (signed with a randomly generated key file), just let me know.

Edit: Here's the link to the DotNetZip site: http://dotnetzip.codeplex.com/

tiago replied on Friday, October 01, 2010

Hi Janns,

Under the scope of Remoting I've done a lot of testing of compression libraries:

SharpZipLib resulted in the best compression but it was so slow that the complete transmission time (start of compression - transmission - end of decompression) wasn't good at all. Of course this also depends on the transmission bandwidth; the slower the media, the more important the compression ratio is.

The .NET library itself was rule out as a single test was enough to prove it suffers from a very well known problem at least since MNP5/v.42bis modems (MNP5 compression processes ZIP files but the resulting file is larger than the original; V.42bis solved that problem). But this "inflate on compression" problem was fixed in .NET 4.0

http://msdn.microsoft.com/en-us/library/ms171868(v=VS.100).aspx

"The compression algorithms for the System.IO.Compression.DeflateStream and System.IO.Compression.GZipStream classes have improved so that data that is already compressed is no longer inflated."

So this is now a serious alternative to SharpZipLib.

In the mean time, your solution is the one that is working.

Copyright (c) Marimer LLC