Optimizing Wcf dataportal traffic

Optimizing Wcf dataportal traffic

Old forum URL: forums.lhotka.net/forums/t/9777.aspx

maxal posted on Wednesday, November 17, 2010

We looked at the Wcf traffic with fiddler and it's obviously not optimal. NetDataContractSerializer just puts so much data about types, including assembly where the type is defined, and it's for every field and every row.

I suspect that we are not first who tries to solve it. I see following options.

1. Write our own data portal to use Binary serialization. 

2. Use DataContractSerialization

3. Use TcpBinding

4. Use compression on Wcf layer

5. Use compression on DataPortal layer (if we are going to switch to BinarySerialization)

Obviously, compression and changing serialization type can be used together.

Did anyone try anything from this? Any other options? What's your solution?


RockfordLhotka replied on Wednesday, November 17, 2010

Yes, this has been discussed before.

The DCS won't work, at least not without some effort on your part, because the type information does have to flow across the network. Or you need to invent a scheme by which the recipient of the data somehow knows the types of each data field and object in the data stream. That could be done through some pre-determine contract metadata or something, but there aren't any current serializers that do this.

In other words, I didn't choose the BinaryFormatter/NetDataContractSerializer/MobileFormatter by accident - they are only serializers that provide enough information over the wire to actually work.

The simplest solution is to use compression. Due to the large amount of duplicated data in the byte stream, compression is extremely effective at reducing the size of the byte stream. There are third-party compressed bindings for WCF on .NET, and CSLA has a nice model for inserting your own compression library for WCF on Silverlight. That can make a big difference.

Things like TcpBinding seem unlikely to matter - the issue is the size of the data on the wire, not the size of the header information (which is the primary difference between tcp and http).

There's some discussion and links around this in the FAQ (www.lhotka.net/cslanet/faq) incuding the results of a study someone did around the SL data portal, comparing the use of compression to binary xml to a combination of both. As you can imagine, binary xml doesn't compress nearly as well as text xml, so the results were quite interesting.

maxal replied on Wednesday, November 17, 2010

Thank you for the response.

I do realize that DataContract would require special efforts (using DataContract and DataMember attributes). Put it to the list to find if anyone went this route.

I didn't get into details of inserting compression for Silverlight, but as I understand it is not available for WPF anyway, and this is what we are developing WPF 3-tier application.

Couple of questions.

You are saying that tcpbinding is about header only. But isn't it about everything being binary? Sure, if I use NetDataContractSerializer, it doesn't really matter, but what if I use BinaryFormatter? I thought in such case that would be a big difference. Right?

Another question, what do you think about idea of switching to BinarySerialization and using compression right there? I think that would be my own  portal with different WcfResponse WcfRequest objects, so I would contain all my performance improvements and "non-standard" tricks at one place. Or, do you still think that it's better just to buy some third-party Wcf compression. The management hates buying stuff, so I may end up writing it. Doesn't seem to be too complicated anyway. The problem may be in reusing Wcf security features though, as I understand.

RockfordLhotka replied on Thursday, November 18, 2010

It goes beyond that. The DCS simply can't do what's necessary, because it doesn't carry over non-public members or read-only members. It doesn't carry over type information necessary to recreate the object graph. It doesn't preserve the object graph shape (it often creates "phantom" duplicate objects on deserialization.

In short, DCS just isn't useful for this purpose.

TCP vs HTTP is just a protocol difference. You can tell NDCS to use binary xml or regular xml either way. CSLA defaults to using binary xml.

If you use binary xml, compression won't work nearly as well, since there's far less repetition of data in the binary, so the compression just can't do as much. Compression on regular text xml is dramatic, because xml is so amazingly repetative.

That's why I referred you to the FAQ. The study done was pretty extensive, and extablished that using either binary xml or compression was usually the best, and only in some rare cases was both worth the cost.

CSLA 4 does have the same hooks for manual compression in WcfProxy as the SL version. That was one of the features Sergey added for CSLA 4. Basically you create a custom WcfProxy and WcfPortal (subclasses) and override a couple methods where you get the inbound/outbound byte arrays and you can compress them as you see fit.

Copyright (c) Marimer LLC