This is what made me leave Subversion:

Grr!

There are some things that I am just not willing to take, and waiting for my machine is one of them.

I had tpo much of this:

I am not sure yet what I am going to use, but it is going to be some sort of DVCS

Merging is one of my least favorite activities, this is especially true when we are talking about a big merge. You might have noticed that I have been talking lately about some big changes that we are making to NH Prof. And now the time has come to merge it all back together.

Well, big changes is quite an understatement, to be fair. what we did is rip apart the entire model the application used to work with. We moved from a push model to a pull model, and that had quite a lot of implications throughout the code base.

Of course, we also did some work on the trunk while we worked on that, so before we can even think about reintegrating the branch, we have to do a merge from the trunk to the branch, which resulted in:

Now, the problem is that this is totally stupid.

Nitpicker corner:
   And yes, I am using Subversion, and before the Git fanboys totally jump on me, I am seriously considering moving to Git to ease this sort of pain.
   And yes, I should have done reverse merges to the  branch all along, so before the Subversion fanboys totally jump on me, I know that.

It is stupid because there are some changes that has been made in parallel in both branches, there are some changes that involve deleting or renaming files that are just not being merged. And yes, I am using SVN 1.5. So, after resolving all the conflicts, I have to do a manual check over this, to make sure that we didn’t miss a merge because of that. I am at the point of so much Argh! that I can’t really keep it inside, hence this post.

A common example that I know is going to hit me is something like FormattedStatement, which is a pretty important class. In the branch, that was changed to be SqlStatement, and I renamed the file as well. Subversion doesn’t merge changes across that. And yes, I used svn rename for that (via Visual SVN).

And, to add insult to injury, doing this manual checking means a lot of going over the network, which means that this is slow.

Yesterday I added a bug to Rhino Service Bus. It was a nasty one, and a slippery one. It relates to threading nastiness with MSMQ, and the details aren’t really interesting.

What is interesting is how I fixed it. You can see the commit log here:

 

At some point, it became quite clear that trying to fix the bug isn’t going to work. I reset the repository back to before I introduced that bug (gotta love source control!) and started reintroducing my change in a very controlled manner.

I am back with the same functionality that I had before I reset the code base, and with no bug. Total time to do this was about 40 minutes or so. I spent quite a bit longer than that just trying to fix up that.

Lesson learned, remember to cut your losses early :-)

It is so much fun to see things that I worked on coming alive. The official announcement is here, with all the boring details. You can skip all of that and go read the code directly using SVN by hitting: https://svnbridge.svn.codeplex.com/svn

Switch svnbridge for your project, and you are done. Note that this is https. And yes, it should work with git-svn as well.

Way cool!

Without getting to the centralized vs. decentralized SCM argument (I understand the differences, I just don't grok them), patch management is important in many scenarios. Contributing to OSS projects is a major one, I admit, but I have previous used these techniques to be able to take emergency fixes on productions and merge them into the development trunk.

The question came up in the NHibernate Contrib mailing list, and Josh Robb has commented on that at length. I thought that it would be a good idea to take that and expand on this a bit.

The problem:

We want to submit a changeset to a project, without having direct access to its source control. The solution is to generate a patch and send it to the destination.

So far, it is simple. It gets complex when you need to deal with more than a single changeset that hasn't been merged to the root.

Let us say that we have several changesets that we have generated. Let us see how we treat them, according to the different scenarios we encounter. A scenario, in this case, is the dependence between the changesets.

Scenario #1 - No dependencies between the patches.

This is a common scenario if you are working on several things in parallel. A classic case is when you are fixing several bugs. In most cases, the changes in each bug fix are unrelated to each other, and can be applied independently.

In this case, you usually generate separate patches for each changeset. This allow to evaluate each patch in isolation, which significantly ease the acceptance of each patch.

This lead us to the First Rule of Patches: keep them small. It is easier to go through seven small patches than 1 big one.

Scenario #2 - No dependencies between the patches, but touching the same files.

This is the case if two changesets has touched the same file, but there is no logical dependency between the patches. In this case, we still want to get separate patches. Usually, I generate one patch, revert to base, work on the second one, generate a patch, etc...

Scenario #3 - Logical dependencies between the patches

One patch relies on behavior / API created in another patch. In this case, the best solution is to create a patch for each distinct behavior, and number them, so it is still possible to review them in isolation, but the merge order is clear.

Scenario #4 - Several revisions of the same patch

In this case, you sumbitted a patch, but continued to work on the same feature/bug and have a new patch before the first one was applied. In this case, the later patch supercede the previous one, which can now be discarded. You need to be careful with this scenario, because too much disconnected work can create huge patches. It is better to review you work and see if you are in situation #3 or really situation #4.

Anything that I missed?

Well, here is how my last few days were spent. Trying to get SvnBridge to work on Mono 1.9.1. (Note that we are talking about working locally, when the hosted version will go live, it will obviously be accessible from everything and anything that can talk to an SVN server).

After getting it to compile and run (not a trivial process, I must say, I used a surprisingly high number of MS CLR stuff, but we will leave that to another post), I had managed to get some basic functionality working, and I was able to successfully checkout the source of SvnBridge itself.

Unfortunately, when I tried to commit those changes, I run into several... problems.

Without beating too much around the bush, the issue was that authentication errors against the TFS web service API. More specifically, in the error handling of those authentications errors.

In several parts of SvnBridge, we have code similar to this:

try
{
    return tfsWebService.DoSomething();
}
catch(WebException we)
{
    HttpWebResponse r = we.Response as r;
    if( r != null && r.StatusCode == HttpStatusCode.Unauthorized )
          throw new NetworkAccessDeniedException(we);       
}

The problematic part is bolded and in red. As I already mentioned, there is a subtle but incredibly annoying difference between Mono and .Net. This means that code such as this will never work. The issue is that the Mono implementation of HttpWebResponse check to verify that no access is done to a disposed object. Precisely like a well behaved object should. The .Net implementation does not.

By the time that the SOAP stack implementation (on both sides) gives you the web exception, the web response has been disposed. But on the Mono side of things, you can't even find out why.

As a result, I made up this logo.

I have reported the issue to the Mono guys, and hopefully they will get this fixed.

Well, it works, finally.

I had major issues getting commits to work. Eventually I figured out that git-svn does not send hashes of the files as all other SVN clients that I have tested so far. I was focused on finding corruption much earlier in the game, and drowned in the details.

I am not sure why it is committing a single change twice, but it is showing the same behavior on standard SVN servers, so I think that I am fine.

And here is the final test:

:-D

First, I really have to say that I had no control over this poster, it just came up and I had to put it here.

Let us just say that this is an encouraging development:

For a few days, some (~4) of SvnBridge integration tests would fail. Not always the same ones (but usually the same group), and only if I run them all as a group, never if I run each test individually, or if I run the entire test class (which rules out most of the test dependencies that causes this). This was incredibly annoying, but several attempts to track down this issue has been less than successful.

Today I got annoyed enough to say that I am not leaving until I solve this. Considering that a full test run of all SvnBridge's tests is... lengthy, that took a while, but I finally tracked down what was going on. The fault was with this method:

protected string Svn(string command)
{
	StringBuilder output = new StringBuilder();
	string err = null;
	ExecuteInternal(command, delegate(Process svn)
	{
		ThreadPool.QueueUserWorkItem(delegate

		{
			err = svn.StandardError.ReadToEnd();
		});
		ThreadPool.QueueUserWorkItem(delegate
		{
			string line;
			while ((line = svn.StandardOutput.ReadLine()) != null)
			{
				Console.WriteLine(line);
				output.AppendLine(line);
			}
		});
	});
	if (string.IsNullOrEmpty(err) == false)
	{
		throw new InvalidOperationException("Failed to execute command: " + err);
	}
	return output.ToString();
}

This will execute svn.exe and gather its input. Only sometimes it would not do so.

I fixed it by changing the implementation to:

protected static string Svn(string command)
{
	var output = new StringBuilder();
	var err = new StringBuilder();
	var readFromStdError = new Thread(prc =>
	{
		string line;
		while ((line = ((Process)prc).StandardError.ReadLine()) != null)
		{
			Console.WriteLine(line);
			err.AppendLine(line);
		}
	});
	var readFromStdOut = new Thread(prc =>
	{
		string line;
		while ((line = ((Process) prc).StandardOutput.ReadLine()) != null)
		{
			Console.WriteLine(line);
			output.AppendLine(line);
		}
	});
	ExecuteInternal(command, svn =>
	{
		readFromStdError.Start(svn);
		readFromStdOut.Start(svn);
	});

	readFromStdError.Join();
	readFromStdOut.Join();

	if (err.Length!=0)
	{
		throw new InvalidOperationException("Failed to execute command: " + err);
	}
	return output.ToString();
}

And that fixed the problem. What was the problem?

There was a known, but irreproducible, in SvnBridge for a long while now.

The essence of the bug is that under some circumstances, updating from SvnBridge would fail. The reason for failure was an assertion failure. Somehow, someone made a change to a file that doesn't exists.

Did you note the some circumstances part? That is the critical issue there. It is the circumstances that I wasn't able to reproduce. As a matter of fact, I wasn't able to narrow what happened. I sometimes got it, and when I had a repro, I could fix it. But all too often, trying to reproduce the same steps would not cause the bug, but work as expected.

Annoying as hell, as you might imagine.

Today I finally managed to figure out what was going on. You can see the repro on the right. This is a timeline, where each user is colored differently.

As you can see, it require a failure unusual set of circumstances, and the way the SVN protocol work make it harder to figure out. (When you ask for changes on an item, you are actually requesting all changes on its parent, which is a critical issue here).

It is also important which side of the rename you are asking about, and... but now I am probably boring you with technical details that are not interesting even to geeks.

I have been in weird bug fixing mode for SvnBridge for a while now, and it is... interesting to see what crops up. Right now most bugs takes quite a long time to track and fix, unfortunately.

Oh, and as a parting shot, take a look here at how I tracked that one:

The bug: SvnBridge will not accept a change to a filename using multi byte format.

Let us start figuring out what is going on, shall we?

Hm... looks like when TortoiseSVN is PUT-ing the file, it uses the directory name instead of the actual file name. Obviously, this is a client bug, and I can close this bug and move on to doing other things. Except that I am pretty sure that SVN can handle non ASCII characters...

Let us compare the network trace from SVN and SvnBridge and see what is going on...

Oh, I got the problem. TortoiseSVN is requesting a CHECKOUT, and we return the wrong URL. Obviously I did something astoundingly stupid there when I processed that request. Indeed, taking a look at what is going on there is... interesting.

Finally I realize that the problem is that while TortoiseSVN sends the URL in seemingly reasonable format, it is obviously being read wrongly by SvnBridge. It is using ASCII encoding to read this, while it probably should use UTF8. Because of that, the URL looks something like /tfsrtm08/test/????.txt, and ? is the query string terminator, no wonder I have issues.

But, how does it work for standard SVN servers. Hm, looks like when TortoiseSVN uses Url Encoding for the URL when taking to a standard SVN server. Why is it doing this?

Hm... looks like the URL that TortoiseSVN uses when it is request a CHECKOUT is the one that is returned in the response to a PROPFIND request.

Sure, that is easy to fix, we will just Url Encode the response from the PROPFIND handler.

Except that it doesn't work...

Oh, we also handle some of that in the file node class, so we will fix it there.

Yeah!

Except that it still doesn't work!

Grr! Looks like the way we handle Url Encoding is too invasive, we need to only UrlEncode non ASCII characters, this means that we should not encode characters such as '/', for example.

Let us try this again, shall we?

Hm... it still doesn't work. What is going on.

Excuse me while I am having a nervous breakdown, please.

Oh! When we send the response to the CHECKOUT request, we send the URL for the PUT request in the location header, and we haven't Url Encoded that yet.

And now it works... :-D

Let us run the test and commit...

We have a broken test.

*Sob*

Well, okay, that is actually a test that is supposed to be broken (it is testing the way we Url Encode urls, after all).

Number of code lines changed, less than 20.

Number of hours spent of this bug: over eight hours.

Conclusion, I am not really being productive today. I am off to visit the code generation wizard...

My personal success criteria for SvnBridge is when I no longer care what I am working against. And I think we got to that point. There are still issues, to be fair, but they won't hit you anywhere near mainline development.

I am listening to Rob Conery MVC Storefront 9 in which he makes the mistake of referring to CodePlex as Subversion. With that, I think that I can say, we are there.

I was very impressed when I saw how Subversion handles the complexity of having data of hierarchical nature that needs to be serialized to XML. Check this out.

<?xml version="1.0" encoding="utf-8"?>
<S:editor-report xmlns:S="svn:">
  <S:target-revision rev="11"/>
  <S:open-root rev="-1"/>
  <S:open-directory name="tags" rev="-1"/>
  <S:add-directory name="tags/asd"/>
  <S:close-directory/>
  <S:close-directory/>
  <S:close-directory/>
</S:editor-report>

And here is another one:

<?xml version="1.0" encoding="utf-8"?>
<S:editor-report xmlns:S="svn:">
  <S:target-revision rev="15"/>
  <S:open-root rev="-1"/>
  <S:open-directory name="trunk" rev="-1"/>
  <S:open-file name="trunk/a.txt" rev="-1"/>
  <S:apply-textdelta checksum="eabc96676e7defda414a1eed33bdfb09">
    U1ZOAAAQEwETk2FzZDENCjINCjMNCjQNCjUNCjY=
  </S:apply-textdelta>
  <S:close-file checksum="c6301e5dad1330a7b9bd5491702c801b"/>
  <S:close-directory/>
  <S:close-directory/>
</S:editor-report>

I was, as they say incredibly happy with this Work Time Fun.

It is amazing how much time you can hunt for the exact cause of a bug. In this case, it took me almost two days (intersperse with other work, however) to track down and find the issue.

Remember, there is no reason to use ASCII, ever. I actually run a blame on the code (a new feature for SvnBridge!) to find out who wrote it. And then I sent a nasty email about it to /dev/null, just to clear my mind.

Let us see how much of a furor that will cause :-)

The reasons that I don't like distributed source control systems have nothing to do with technical reasons, and everything to do with how they are used and promoted.

The main reason that I don't like them is that they encourage isolated work. I don't want anyone in my project to just go off and work on their own for a few weeks, then come back with a huge changeset that now needs to be merged. The way DVCS are promoted, this is a core scenarios, "no one have to see your mistakes".

I completely disagree. I want to see everyone's mistake, and I want them to see mine. There is little to learn from successes, and much to learn from failures. Far worse, I want to be able to peak into other people work, so I can give my input on it, even if it is just "wow, nice", or "yuck, sucks!"

The main advantage of DVCS is speed, trying to browse the repository when you have the entire thing on your HD is lightning fast, compared to doing the same with a remote repository. I tend to use SVK now, but I use it strictly as a local cache, and nothing more.

And that is why I don't like DVCS, I don't want people to work in isolation.

Let me start by stating that you really don't want to do that. This is not something that you want to do, period.

Now that we are over that, I had the chance lately to go fairly deeply into SCM and how they are implemented from two fairly different perspectives. This is a randomly collected set of observations about SCM systems. As usual, the order is arbitrary, and no attempt was made to make any coherent idea out of this.

• It is all about the client. The client in an SCM system has significant responsibilities. It is in charge of reporting the client state, managing all the errors hat the user can cause, and shoulder a lot of the burden.
• It is all about the protocol. Anyone who designs a SCM system should be given a lousy DSL line with disconnects every 15 minutes. Oh, and they should also have to work on a plane a lot.
• On the wire, it is all so simple. It is really surprising to see how the SCM complexity is really just a lot of tiny, easy to handle, details.
• The devil is in the details, though.
• Complexities on the server side:
  • Space management - do you save the diff or the whole file?
  • If just the diff, how do you construct an arbitrary version
  • Keeping history around for branches and copies
  • Cheap copies
  
  
• Complexities on the client side:
  • Do you have one version on the client, per the working copy?
  • Do you have multiple versions, one per each file?
  • Handling inconsistencies between server version, working copy version and original version.
  
  
• What do you optimize for? Bandwidth? Roundtrips?
  • I know of one SCM product who is lousy optimizer for both
  
  
• Distributed SCM can be handled on top of centralized SCM.
• It is not hard at all, except for all the details.
• Don't write your own SCM.
• Trust matters, and you really don't want to be in the situation where you don't trust your SCM.
• Remember that SCM is temporal, you can go backward in time, and even sideway, to a branch.
• There are only three types of operations in SCM:
  • Generate a change set between two paths at two versions
  • Apply a diff to a path, generating a new version
  • Reporting (logs, mostly, and outputting various formats of a changeset)

Overall, it is very simple endeavor. It gets complex when you start talking beyond the wire protocol. As a simple example, how much does it cost you to branch? How much does it cost you to find out if there has been any changes to the working copy?
The other major issue is: How do you ensure that it is reliable?
Now, let me repeat myself, do not write your own source control!

I have just had to make a modification to SvnBridge because someone decided to checkin a change set that included 1,695 changes..

Even ignoring the issue of "your checkin broke my source control" mentality that I have now, this is bad from other perspective.

How do you review such a change? How do you even know what is going on there? In changes of this size, you aren't making a single change, you are making a lot of changes. Now you can't mix & match them either.

Prefer small checkins, they are easier to deal with.

Took a lot of work, and several months, but it is here:

Download Page

A few days ago I posted about profiling results of SvnBridge. I spent the last few days implementing really aggressive caching mechanism, reducing remote calls, and in general working on the performance of the application.

Here are the results:

I think that I am in a good shape if the XML parsing starts to be a high level item in the profiling.

Oh, and just a teaser, check this out. This is a checking out the SvnBridge code from Israel (and the servers are in the US).

No, it is not sustainable, sadly, but it is a really good indication for what good caching will give you.

Following the recent profiling effort, I decided to put far more aggressive caching into SvnBridge.

I set it up so it would cache the full revision from TFS on any query, and then serve it from the cache. When I run it against the test server it worked beautifully. Then I had run into this issue:

I am pretty sure that this is not going to be an acceptable scenario. To be rather exact, I would find it acceptable if it was a one time cost, but the problem is that this is a cost that you have to pay per revision. And that is unacceptable. The major problem is that this uses the underlying QueryItems() method, which returns all of the results, including those from previous revisions. This means that on a busy server (like tfs03), the cost of doing such a query is high.

The number of files returned is actually pretty small (910 in this case), but I assume that it have to check all the files on the server for permission before it allows me to get them.

I wonder how Rhino Security would handle this situation, it wouldn't even get the data out of the DB, and the query enhancement is pretty light weight. I assume it would be pretty fast.

Anyway, this is obviously a bad approach. For now, I made it load only the path (and its descendants) that we need, this mean that we don't get the same benefit of preemptive caching and might talk to the server a bit too much. However, it turn out that the way SvnBridge and SVN makes requests in a way that make this style of caching work fairly well. We always ask for the directory before asking for the descendant, and we have fairly long conversations about the same revision, so that is good candidate.

That isn't optimal for big projects, with a lot of files and a lot of activity, however. Because the way I handle it now, we download the entire project metadata for each revision, that can be a lot for those kind of projects, and having to download them each and every time is a waste.

SvnBridge already contains a very smart piece of code (the UpdateDiffCalculator class) that can figure out the differences between two revisions and only get the items that it needs. The problem is that the caching layer is built mainly in order to support that class.

I think that I'll need to get a bit smarter about this in the future, but for now it seems to be doing the work very well.

This is how I am feeling at the moment (more below):

One of the major disadvantages of having to dog food the source control I am using to write the source control is that it is incredibly fragile.

It is fairly easy to violate some invariant in a non obvious way, which make for problems down the road. The problem is that those problems are symptoms, not the real issue. As a result, I have this:

I try to clean it up often, but at one point I got to SvnBridge5.

Thank god that SVN has such a good patching mechanism.

Okay, here is something that I had to deal with today. I had to implement the dated-rev-report command in SvnBridge. This command sends a date to the server, and get the first revision id that was committed before to that date. Because code is unambiguous, if we were using SQL, it would have been:

CREATE PROC DatedRevReport 
	@requestedDate DATETIME
AS
	SELECT TOP 1 RevisionId
	FROM Revisions
	WHERE CommittedDate < @requestedDate
	ORDER BY CommittedDate DESC

Simple, right? Except that the interface that I have to TFS is not SQL (and I would be pretty sad if it was, so that is a good thing). I have to go through the TFS API in order to get that result.

Here is the API that I have to work with (actually, that is a drastically simplified API, but it contains all the stuff required to solve this issue).

public interface ISourceControl
{
	int GetLatestId();
	LogItem QueryLog(VersionSpec from, VersionSpec to, int countToReturn);
}

public class LogItem
{
	public SourceItemHistory[] History;
}

public class History
{
	public int ChangeSetId;
	public DateTime CommitDateTime;
}

public class DateVersionSpec : VersionSpec
{
	public DateTime Date;
}

public class ChangesetVersionSpec : VersionSpec
{
	public int ChangesetId;
}

Your task, if you agree to accept it, is to build implement the following method:

public int GetVersionForDate(DateTime date)
{
	// your implementation here
}

One important consideration is that you should reduce remote calls. I managed to get it down to 4 remote calls for the worst case scenario. It could have been 3, but TFS has some... finicky date handling.

So, how would you solve the issue?

The first thing is getting the source to your machine. I recommend downloading the latest released version of SvnBridge from the SvnBridge site, and executing the SvnBridge.exe file.

Put https://tfs03.codeplex.com in the Team Foundation Server text box and 8081 in the port text box, then click Ok.

SvnBridge will connect to the remote server and then the the following notification tool tip:

Check out the source using TortoiseSvn:

Note: You can also use the following command line to checkout the code: svn checkout http://localhost:8081/SvnBridge SvnBrdige

Open the SvnBridge.sln in Visual Studio and ensure that you can compile all the projects in the solution.

Note: It is possible that you'll get compilation errors in the TestsRequiringTfsClient project, if you don't have the Team Foundation Client installed. If it fails to compile, you can safely remove the project from the solution. SvnBridge does not require you to have TFS installed, and only this test project has any dependency on TFS.

You can now execute the SvnBridge project and start working with TFS using the SVN protocol.

Running the tests

The following test projects can be run without having a test instance of TFS to work with:

• TestsUnit
• TestsProtocol

The following test projects requires an instance for TFS to work with:

• TestsEndToEnd
• TestsIntegration
• TestsRequiringTfsClient

Setting up TFS test instance

Note: Getting TFS up and running can be tricky, for testing purposes, I generally use the TFS VPC trail.

Open Visual Studio and connect to a TFS server using Team Explorer.

Right click the TFS server and select New Team Project (note that if the New Team Project menu item is missing or disabled, it is likely that you don't have permissions to create a new project):

Name the project SvnBridgeTesting and click next:

On the select process template screen, click Finish:

Important: Ensure that the time zone of the TFS machine match the time zone on the development machine.

For each of the test projects, open the App.config file and paste the following, replacing the configured values with the ones matching your environment.

<?xml version="1.0" encoding="utf-8" ?>
<configuration>
	<configSections>
		<sectionGroup name="applicationSettings" type="System.Configuration.ApplicationSettingsGroup, System, Version=2.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089" >
			<section name="IntegrationTests.Properties.Settings" type="System.Configuration.ClientSettingsSection, System, Version=2.0.0.0, Culture=neutral, PublicKeyToken=b77a5c561934e089" requirePermission="false" />
		</sectionGroup>
	</configSections>
	<applicationSettings>
		<IntegrationTests.Properties.Settings>
			<setting name="ServerUrl" serializeAs="String">
				<value>http://TFSRTM08:8080</value>
			</setting>
			<setting name="Username" serializeAs="String">
				<value>tfsSetup</value>
			</setting>
			<setting name="Password" serializeAs="String">
				<value>P2ssw0rd</value>
			</setting>
			<setting name="Domain" serializeAs="String">
				<value>TFSRTM08</value>
			</setting>
		</IntegrationTests.Properties.Settings>
	</applicationSettings>
</configuration>

Now you can run all the tests, and the development environment is ready to go.

Have fun...