Compression? BinaryFormatter? What are the current techniques?

Compression? BinaryFormatter? What are the current techniques?

Old forum URL: forums.lhotka.net/forums/t/7657.aspx


bt1 posted on Friday, September 18, 2009

I am currently using CSLA 3.6 and am willing to upgrade to 3.8 to obtain any efficiencies.

WinForms VS2008 .NET3.5, CSLA 3.6.X, WCFproxy

I have been building some CSLA objects and started testing them.  I received an error where I needed to increase the maxReceivedMessageSize.   However, the object wasn’t that large, it had several (~600) small child objects.  I quickly came up with an estimate of what size I thought the object should have been and it (the estimate) was lots smaller.  So I used Fiddler2 to look at the packets.  Was I shocked at the amount of overhead the XML  had added,  probably around 10x what the raw data would have been.  After some initial research it looks like CSLA WCF is using the NetDataContractSerializer.

 So after some research here I found several optimizations mentioned:

1.       Compression CSLA

2.       Compression WCF

3.       BinaryFormatter (WcfBfProxyHost) vs NetDataContractSerializer.

4.       BinaryXML (Mobil only?)

The problem is that I am having difficulty understanding

1.       which methods to use (in combination with each other).

2.       Which techniques are the most current (have some been replaced with new better techniques)

3.       Are some techniques only for Mobile/SL .

4.       How to implement them.

I know this is asking a lot but can someone outline what the current recommended techniques are and point me in the direction of some info on how to implement them.

 

RockfordLhotka replied on Friday, September 18, 2009

The .NET serializer (NDCS) does generate some massive amounts of XML, no doubt about it.

The good news is that much of the bulk is in the form of repeated xmlns, assembly and type strings. The reason this is good news is that compression is extremely effective because the repeated strings are eliminated, plus other compression occurs as well of course.

The only place you can easily plug compression into the data portal is with the Silverlight data portal. I didn't bother to add those hooks in the .NET side, because you can just go get a compressed WCF binding instead.

(I rather suspect there are, or will be, compressed WCF bindings for Silverlight too - but they didn't exist when we added the compression hooks to the Silverlight data portal, so some immediate answer was required)

DocJames replied on Saturday, September 19, 2009

RockfordLhotka:

[...] I didn't bother to add those hooks in the .NET side, because you can just go get a compressed WCF binding instead. [...]



Is anyone using compressed WCF binding? I would love to see how it's implemented...

I have a click once application using the DataPortal via WCF - and I really need compression to speed things up :)

Thanks,
Jimmy

RockfordLhotka replied on Saturday, September 19, 2009

>Is anyone using compressed WCF binding? I would love to see how it's
implemented...

I know Magenic has at least one client doing this. They just purchased a
commercial binding component that does the work. Then it is just a matter of
putting the correct lines in the client and server config for WCF to use it
- I'm sure the component came with documentation/help describing exactly how
to do that.

sergeyb replied on Saturday, September 19, 2009

I wrote compressed HTTP/ws-HTTP binding extension for exact same project configuration you have - WPF Click Once with CSLA WCF portal. Unfortunately, client owns the code, so I cannot post it here. I could give you some guidelines on writing one, but they will be at a high-level, again because I cannot post client code here. Let me know if you are interested.


Sergey Barskiy
Principal Consultant
office: 678.405.0687 | mobile: 404.388.1899

Microsoft Worldwide Partner of the Year | Custom Development Solutions, Technical Innovation

-----Original Message-----
From: Rockford Lhotka [mailto:cslanet@lhotka.net]
Sent: Saturday, September 19, 2009 8:27 AM
To: Sergey Barskiy
Subject: RE: [CSLA .NET] Compression? BinaryFormatter? What are the current techniques?

>Is anyone using compressed WCF binding? I would love to see how it's
implemented...

I know Magenic has at least one client doing this. They just purchased a
commercial binding component that does the work. Then it is just a matter of
putting the correct lines in the client and server config for WCF to use it
- I'm sure the component came with documentation/help describing exactly how
to do that.

DocJames replied on Saturday, September 19, 2009

sergeyb:
[...] Let me know if you are interested.


It would be really great with a high-level guideline. It would also point me in the right direction.

I'm also considering trying/buying a component like the one Noemax (http://www.noemax.com) has.

Thanks,
Jimmy

sergeyb replied on Saturday, September 19, 2009

Now that I am thinking about it, I think you can search the forum. I believe someone posted a solution for this. If you are to write your own, then...
You can write a custom behavior extension. You would need to implement a few interfaces that WCF defines: IEndpointBehavior, IDispatchMessageInspector, IClientMessageInspector, IServiceBehavior. You class would need to inherit from an Attribute as well.

IDispatchMessageInspector.AfterReceiveRequest you would decompress a message. In IDispatchMessageInspector.BeforeSendReply you would compress the message. In IClientMessageInspector you would do the opposite.

In IEndpointBehavior.ApplyClientBehavior you would need to add "this" as another message inspector. Same with ApplyDispatchBehavior.

For compress/decompress you can just use any compression library. In the actual Compress routine you would need to create new message, copy headers from the old message, create new header and add it to headers collection.

You can probably find some sample code on MSDN if you search for those interfaces.

The last step is to add new behavior to . node in web/app.config. Then you refer to this extension in node.

Sorry I cannot provide more detailed info.



Sergey Barskiy
Principal Consultant
office: 678.405.0687 | mobile: 404.388.1899

Microsoft Worldwide Partner of the Year | Custom Development Solutions, Technical Innovation


-----Original Message-----
From: DocJames [mailto:cslanet@lhotka.net]
Sent: Saturday, September 19, 2009 12:28 PM
To: Sergey Barskiy
Subject: Re: [CSLA .NET] RE: Compression? BinaryFormatter? What are the currenttechniques?

sergeyb:[...] Let me know if you are interested.

It would be really great with a high-level guideline. It would also point me in the right direction.

I'm also considering trying/buying a component like the one Noemax (http://www.noemax.com) has.

Thanks,
Jimmy

bt1 replied on Monday, September 21, 2009

Update:  I set up the Csla.Server.Hosts.IWcfBfPortal (BinaryFormatter) and got some good performance increases.

 

For the  same Same Object (data) I am getting the following increases using Fiddler to track the http request from the client:

 

NetDataContractSerializer

Request Count:                1

Bytes Sent:         8,716

Bytes Received: 7,213,938

ACTUAL PERFORMANCE

Overall Elapsed:                00:00:04.4218467

 

IWcfBfPortal (Bindary Formatter)

Request Count:                1

Bytes Sent:         7,284

Bytes Received: 918,732

ACTUAL PERFORMANCE

Overall Elapsed:                00:00:02.2031109

 

 

So, the next question is about the compression that has been mentioned.  Will I really get much more of an improvement with compression.  As I understand compression doesn’t work well (as well) with binary data. 

 

Or would I be better off not using a Binary Formatter in WCF and using compression (only) in WCF?

 

RockfordLhotka replied on Monday, September 21, 2009

While that’s true, I suspect you’ll get even better perf increases if you use a compressed binding with NDCS.

 

I created WcfBfProxyHost as an experiment, in response to someone else pointing out the size of the XML created by NDCS. In their testing (so just one real data point) it was faster to use WcfProxy with a compressed binding.

 

Using WcfBfProxy with compressed binding is almost counter-productive btw. This is because the binary data generated by the BinaryFormatter doesn’t compress well – as is true of most binary data.

 

Alexander replied on Thursday, September 24, 2009

Hi everyone!

Since Noemax was mentioned earlier, I will offer some quick info on what solutions are offered by WCF-Xtensions that could be of help in this case:

- Fast Infoset message encoding

Fast Infoset is a binary XML message encoding that is more compact that .NET Binary, available on .NET / .NET CF / Silverlight, and is interoperable with Java. Fast Infoset is an ITU-T/ISO standard.

- Envelope behavior

The Envelope behavior is an out-of-the-box solution that produces a very compact message envelope. It works without making any changes to the existing binding configuration. Introducing this component does not break interop with existing clients that do not support the capabilities of the Envelope behavior.

- HTTP compression

HTTP compression extends the system-provided HTTP transport with standard content negotiation and compression. Can compress/decompress both requests and responses. Enables support of multiple messages encodings on the same endpoint. It is available on .NET and .NET CF, supported by Silverlight, and interoperable with most HTTP servers. Introducing this component does not break interop with existing clients that do not support the new capabilities of the service.

- Message compression binding element

Message compression can be used with any transport and message encoding, including the system-provided TCP transport. Its GZIP implementation provides better compression ratios than the system-provided System.IO.Compression.

You can also combine some of the above solutions for maximum benefits. If you let me know your project's requirements I will be able to offer more specific suggestions.

Alexander

JohnSchneider replied on Friday, September 25, 2009

If you want to get the best performance, take a look at the emerging Efficient XML Interchange (EXI) standard from the World Wide Web Consortium (W3C). It is far more compact than .NET Binary, FastInfoset, BinaryXML (WBXML), compression, etc. and does not incur the processing overhead of compression. The W3C conducted extensive benchmarking of different candidate technologies for the EXI standard and found there was one technology (called "Efficient XML") that consistently achieved the best performance across all their test data, use cases and application classes (see full report). The W3C adopted "Efficient XML" as the basis for the single, global standard for the WWW. FastInfoset was also one the candidates considered; however, it did not achive sufficient compactness or generality to meet the requirements (see requirements matrix).

I work for the small company that developed Efficient XML. We were co-founded by an ex-Microsoft XML veteran and offer a plug-in for Microsoft WCF as well as standard APIs for .NET and .NET CF apps. The plug-in has built-in compression, supports dynamic content-negotiation and interoperates with versions for Java, Java Mobile Edition and native C/C++. If you want to try it out, you can download a trial version of the SDK from here.

   Enjoy!,

   John

Jack replied on Friday, September 25, 2009

John,

 

Is it affordable for a SMB / small app / small user base or is this more geared as an enterprise type implementation?

 

From: JohnSchneider [mailto:cslanet@lhotka.net]
Sent: September-25-09 11:00 AM
To: jaddington@alexandergracie.com
Subject: Re: [CSLA .NET] RE: Compression? BinaryFormatter? What are the current techniques?

 

If you want to get the best performance, take a look at the emerging Efficient XML Interchange (EXI) standard from the World Wide Web Consortium (W3C). It is far more compact than .NET Binary, FastInfoset, BinaryXML (WBXML), compression, etc. and does not incur the processing overhead of compression. The W3C conducted extensive benchmarking of different candidate technologies for the EXI standard and found there was one technology (called "Efficient XML") that consistently achieved the best performance across all their test data, use cases and application classes (see full report). The W3C adopted "Efficient XML" as the basis for the single, global standard for the WWW. FastInfoset was also one the candidates considered; however, it did not achive sufficient compactness or generality to meet the requirements (see requirements matrix).

I work for the small company that developed Efficient XML. We were co-founded by an ex-Microsoft XML veteran and offer a plug-in for Microsoft WCF as well as standard APIs for .NET and .NET CF apps. The plug-in has built-in compression, supports dynamic content-negotiation and interoperates with versions for Java, Java Mobile Edition and native C/C++. If you want to try it out, you can download a trial version of the SDK from here.

   Enjoy!,

   John



JohnSchneider replied on Friday, September 25, 2009

Both. There are per-copy licenses for smaller deployments, per-application licenses for medium-to-large deployments and unlimited enterprise licenses for very large deployments involving many applications. To find out what might work best for you, I recomended shooting a quick e-mail to sales@agiledelta.com.

     All the best!,

     John

jfreeman replied on Monday, November 16, 2009

What did you have to do to get this setup? I have several large BOs that take a long time to retrieve and I just started looking at setting up WCF to ue binary. Thanks.

Jonathan

RockfordLhotka replied on Monday, November 16, 2009

http://www.lhotka.net/cslanet/faq/DataPortalFaq.ashx

Copyright (c) Marimer LLC