Ayende @ Rahien

Refunds available at head office

WCF works in mysterious ways

Here is the result of about two hours of trying to figure out what WCF is doing:

class Program
    {
        static private readonly Binding binding = new NetTcpBinding
        {
            OpenTimeout = TimeSpan.FromMilliseconds(500),
            CloseTimeout = TimeSpan.FromMilliseconds(250),
            ReaderQuotas =
                {
                    MaxArrayLength = Int32.MaxValue,
                    MaxBytesPerRead = Int32.MaxValue,
                    MaxNameTableCharCount = Int32.MaxValue,
                    MaxDepth = Int32.MaxValue,
                    MaxStringContentLength = Int32.MaxValue,
                },
            MaxReceivedMessageSize = Int32.MaxValue,
        };
        static void Main()
        {
            try
            {
                var uri = new Uri("net.tcp://" + Environment.MachineName + ":2200/master");
                var serviceHost = new ServiceHost(new DistributedHashTableMaster(new NodeEndpoint
                {
                    Async = uri.ToString(),
                    Sync = uri.ToString()
                }));
                serviceHost.AddServiceEndpoint(typeof(IDistributedHashTableMaster),
                                               binding,
                                               uri);

                serviceHost.Open();

                var channel =
                    new ChannelFactory<IDistributedHashTableMaster>(binding, new EndpointAddress(uri))
                        .CreateChannel();
                channel.Join();
            }
            catch (Exception e)
            {
                Console.WriteLine(e);
            }

        }
    }

    [ServiceBehavior(
        InstanceContextMode = InstanceContextMode.Single,
        ConcurrencyMode = ConcurrencyMode.Single,
        MaxItemsInObjectGraph = Int32.MaxValue
        )]
    public class DistributedHashTableMaster : IDistributedHashTableMaster
    {
        private readonly Segment[] segments;

        public DistributedHashTableMaster(NodeEndpoint endpoint)
        {
            segments = Enumerable.Range(0, 8192).Select(i =>
                                                        new Segment
                                                        {
                                                            AssignedEndpoint = endpoint,
                                                            Index = i
                                                        }).ToArray();
        }

        public Segment[] Join()
        {
            return segments;
        }
    }

    [ServiceContract]
    public interface IDistributedHashTableMaster
    {
        [OperationContract]
        Segment[] Join();
    }

    public class NodeEndpoint
    {
        public string Sync { get; set; }
        public string Async { get; set; }
    }

    public class Segment
    {
        public Guid Version { get; set; }

        public int Index { get; set; }
        public NodeEndpoint AssignedEndpoint { get; set; }
        public NodeEndpoint InProcessOfMovingToEndpoint { get; set; }

        public int WcfHatesMeAndMakeMeSad { get; set; }
    }

The problem? On my machine, executing this results in:

Maximum number of items that can be serialized or deserialized in an object graph is '65536'. Change the object graph or increase the MaxItemsInObjectGraph quota.

The freaky part? Do you see the WcfHatesMeAndMakeMeSad property? If I comment that one out, the problem goes away. Since MaxItemsInObjectGraph is set to int.MaxValue, I don’t know what else to do, and frankly, I am getting mighty tired of WCF doing stuff in unpredictable ways.

Protocol Buffers & TcpClient, here I comes.

Comments

Paul Betts
06/04/2009 05:28 PM by
Paul Betts

That's not freaky at all, you made the Segment object smaller, so there was less objects to serialize, so you manage to get under the 64K limit. Depending on how you count it, there are about 8 objects in total serialized per Segment * 8192 = you're right at the line.

Now I'm not saying WCF is doing the right thing, or that an arbitrary unsettable 64K limit is a good idea, but it's at least not weird.

Ayende Rahien
06/04/2009 05:28 PM by
Ayende Rahien

Paul,

Did you notice that I set just about every limit to int.MaxValue?

Arnon Rotem-Gal-Oz
06/04/2009 05:29 PM by
Arnon Rotem-Gal-Oz

Add [DataContract] attribute to Segment and [DataMember] attributes to its properties (and you might need a KnownType (for segment) attribute on the IDistributedHashTableMaster

Arnon

Krzysztof Kozmic
06/04/2009 05:31 PM by
Krzysztof Kozmic

why on earth would he need known type for that?

Ayende Rahien
06/04/2009 05:33 PM by
Ayende Rahien

Arnon,

Nope, I tried that, it doesn't work.

Ayende Rahien
06/04/2009 05:38 PM by
Ayende Rahien

With or without it, it doesn't work

Arnon Rotem-Gal-Oz
06/04/2009 05:39 PM by
Arnon Rotem-Gal-Oz

Sorry that should be ServiceKnownType not KnownType

NodeEndPoint should also have the [datacontract] and [DataMember] attributes

Ayende Rahien
06/04/2009 05:41 PM by
Ayende Rahien

Not working either :-(

And NodeEndpoint have those

Jesse Ezell
06/04/2009 05:49 PM by
Jesse Ezell

The service behavior attribute only applies to the server side. You need to set it on the ClientSide in the endpointbehaviors config or programmatically or you will hit the limit trying to serialize your object graph.

Krzysztof Kozmic
06/04/2009 05:51 PM by
Krzysztof Kozmic

Try also creating DataContractSerializer and serializing the object graph yourself. If it succeeds then it's a misleading message and the problem lies somewhere else,

Stephen
06/04/2009 05:55 PM by
Stephen

I'm pretty sure the data contract serializer has a limit on it as well, you'll need to specify that (probably as a service behavior).

Arnon Rotem-Gal-Oz
06/04/2009 06:17 PM by
Arnon Rotem-Gal-Oz

Ok, Now I actually ran your code - so I see the problem :)

You also need to set the MaxItemsOnInfoGraph on the client side

so before you CreateChannel on the factory you need to do something like

        var channelFactory =

            new ChannelFactory

<idistributedhashtablemaster(binding, new EndpointAddress(uri));

        foreach (var operationDescription in channelFactory.Endpoint.Contract.Operations)

        {


            var dataContractBehavior =


            operationDescription.Behaviors[typeof(DataContractSerializerOperationBehavior)]


            as DataContractSerializerOperationBehavior;


            if (dataContractBehavior != null)

            {


                dataContractBehavior.MaxItemsInObjectGraph = int.MaxValue;


            }


        }

        var channel=channelFactory.CreateChannel();

        channel.Join();
Rafal
06/04/2009 06:28 PM by
Rafal

Maybe you should set

WCF.IAmSorryForNotHavingAPhD = true

then it will just work

Huseyin Tufekcilerli
06/04/2009 06:55 PM by
Huseyin Tufekcilerli

Alternatively, you can attribute your service contract operations with the following NetDataContractFormat attribute and force them to use NetDataContractSerializer instead of DataContractSerializer:

www.pluralsight.com/.../22284.aspx

Krzysztof Kozmic
06/05/2009 09:23 AM by
Krzysztof Kozmic

Ayende

Did Arnon' suggestion (DataContractSerializerOperationBehavior) fix it for you?

Juanma
06/05/2009 09:53 AM by
Juanma

Protobuf + TcpClient is the way to go.

We migrated a project from WCF to Protobuf + TcpClient in less than a day and got a big performance improvement.

Marc Gravell
06/05/2009 10:59 AM by
Marc Gravell

Re protocol buffers - it isn't 100% there yet, but there is an in-progress interface-based RPC stack in protobuf-net. I've currently only had time to do the http layer, but this is pluggable, so should support TCP (I just haven't had time).

Or you could just use any PB implementation (I gather you're using dotnet-protobufs) and throw the data along the pipe yourself...

Marc Gravell
06/05/2009 11:09 AM by
Marc Gravell

I should also add - if you are talking full .NET to full .NET, there is also a protobuf-net hook directly into WCF - by adding a behaviour (attribute), you can swap the serializer to use protobuf-net. If you are using httpBasic this also uses MTOM for maximum throughput.

Ayende Rahien
06/05/2009 11:48 AM by
Ayende Rahien

Marc,

I gave up on WCF because it got too complex to actually bother using.

Too many things that you have to get just right.

I am currently just throwing data over the wire, and so far, it seems to be pretty cool

Ayende Rahien
06/05/2009 11:50 AM by
Ayende Rahien

Krzysztof,

Yes, but I already made the decision that I am not going to bother with something that brittle

Arnon Rotem-Gal-Oz
06/05/2009 02:27 PM by
Arnon Rotem-Gal-Oz

WCF is not that brittle

I think its main problem is that many (if not all) its defaults are dumb, idiotic, fit only for demos or all of the above

Arnon

Frank
06/05/2009 02:32 PM by
Frank

Sorry to hear that, Ayende. In my personal experience, WCF is not brittle, but one does need a deep understanding about how it is envisioned. Alot like many other frameworks that try to tackle a certain problem but are very flexible.

Ayende Rahien
06/05/2009 02:41 PM by
Ayende Rahien

The problem is that it is setup to make you fail, too many things that you have to keep in mind at once, and way too much complexity at way too early stage of the gaem

Frank
06/05/2009 03:50 PM by
Frank

I have to disagree with that. Your scenario is not one of the more common scenarios. While developing I don't have to think about much, only configure a default binding and off you go. As soon, as you are doing stuff, like sending over alot of items and/or alot of data, you will be running into 'problems'. Many of which are present to prevent people from doing stuff like blowing up your service or receiving client by bombarding it with a never ending XML. ;)

Jeremy Gray
06/05/2009 05:46 PM by
Jeremy Gray

The very fact that this thread got so long and filled with so many little details yet still didn't solve the problem largely proves Ayende's point. There is no reason that he should have to fight that hard to get a large graph through. Sure, there should be limits, and they'll have to be set somewhere, but he did exactly what the exception instructed and it didn't work.

WCF is extremely flexible and extremely powerful, and with enough effort can be used to do excellent things. However, the "with enough effort" part seems to have been the real problem any time I've worked with WCF. As soon as you need to move beyond the defaults you are off chasing giant piles of XML (or its programmatic alternatives) and/or trying to figure out what combination of the myriad attributes to apply to what types on which side of the wire.

Given the power that WCF has, these kinds of things will never be dead easy but I should at least be able to spend more of my effort on direct business value and less on using the library. It doesn't seem to take too long to push WCF past a tipping point and end up at "Protocol Buffers & TcpClient, here I comes" or some other alternative.

Frank
06/05/2009 06:53 PM by
Frank

Oh, but the solution is easy. Configure an endpoint behavior for the client that changes the maximum number of items that the data contract serializer allows.

<behaviors
<endpointbehaviors
<behavior
<datacontractserializer

It is OK if you wanna go down another part, that is your choice.

Frank
06/05/2009 06:55 PM by
Frank

Hmmm, for some reason the blog engine doesn't html encode the XML I posted. View the source of this page if you want to view it. ;-)

John Simons
06/06/2009 12:03 AM by
John Simons

How is this different from enabling 2nd level caching in NHibernate?

Thank god(Ayende) that there is NHProf to see under the hood.

James
06/06/2009 12:28 PM by
James

Interesting, we ran into this exact problem at work on Friday, and were scratching our heads as well.

And the prize for the most misleading exception error message goes to....Team WCF!

In our case, with both endpoints under our control, customers were going WTF, as were we. (Since we send a substantial amount of data back and forth over object graphs, some of them being rather large).

James
06/06/2009 12:32 PM by
James

Frank,

Perhaps the solution is easy for you, but pray tell how one is to ascertain that a custom endpoint behavior to set data contract serliazer options is required, when the exception error message states to change the value of another existing option that has no effect?

WCF is archictecture astronauts gone wild.

KS
06/15/2009 08:35 AM by
KS

We hit various problem with WCF as well, especially the bandwidth it is taking: as both NetData or DataContractSerializer are in XML, they send extra information that we wants.

Looking at ProtoBuffer but I need to define .proto for each entities? Also current Proto.net does not support all C# types right?

Can I just do ISerializable over WCF? Also can WCF take something like 5000 invocation per operation per second kind-of load? We have huge problem with the bandwidth at that rate.

Thank you.

Paul
06/25/2009 11:22 PM by
Paul

You know what's rubbish?

DataContractSerializerOperationBehavior is generated for each of your operations. At the time that it gets created it blissfully ignores all <datacontractserializer options that have been defined, and uses a hardcoded value of 65535.

The ONLY way to change this is with code. Kind of hard when your code is already deployed and the customer refuses to accept a new code change.

WCF is crap. Crap. CRAP.

Never use the pile of garbage, you'll just waste hours of time on plumbing which should Just Work.

Comments have been closed on this topic.