chrisortman : git

Follow Up: How do you manage your open source software

chrisortman — Sun, 06 Apr 2008 15:50:39 GMT

A while ago I asked: How do you manage your open source software?. It seems like Bill Pierce had the same question, so I thought I should get to writing my follow up.

I also asked this same question on the alt.net mailing list (whole thread is here).

A lot of folks suggest not updating once you have picked some library to use. That is a fine approach in some cases, but in my opinion things like Castle and NHibernate are never done and I want to be able to update and take advantage of new features.

So, how do I handle this today?

Import the open source project subversion repository into a git repository.
Build the project and copy its outputs to MyProject/Lib/OpenSourceProjectName using a rakefile I wrote.
Commit the changes to MyProject noting the sha1 (from git) and the svn revision of the open source project.
Create a git tag in the open source project git repository that ties it to the branch in my other project.

What does this look like in practice?

	cd /cygdrive/q/castle
	git svn fetch #gets changes from castle svn
	cd ..
	rake build:castle libcopy:castle_to_myproject
	cd myproject
	git add lib/castle
	git commit -m "Updated castle libs to sha xxxxxxxxxxxxxx svn revision 4000"
	cd ../castle
	git tag -f -a myproject-trunk

So far I have been pretty happy with this approach. It could definitely be a little more automated, but it is doing the job.

The link between the open source code and myproject is maybe a bit too fragile. I think I can do better using either git submodules or braid.

I've been experimenting with braid on my rails projects, I'm not sure yet if it will make sense to use it with for this though.

If you are interested here's the rakefile I use:

require 'rake'

class LibCopy

  def copy(lib_files, lib_build_dir)
    libs_in_use = lib_files
    libs_in_use.each do |lib|
      puts "Check for " + lib
      lib_file = lib_build_dir + "/" +  lib.pathmap("%f") 
      if File.exists?(lib_file)
        puts "Found " + lib_file
        cp lib_file, File.expand_path(lib.pathmap("%p"))
      else
        puts "lib not found:" + lib_file 
      end
    end
  end
end

namespace :build do


  CASTLE_BUILD_DIR = 'q:/castle/build/net-3.5/debug'

  task :castle do |t|
    cd 'castle'
    puts "Building Castle...."
    puts sh("c:\\tools\\nant-0.86-beta1\\bin\\nant.exe -t:net-3.5 -D:common.testrunner.enabled=false -D:mbunit-console=c:\\progra~1\\mbunit\\mbunit.cons.exe")
    puts "Build Finished"
    cd '..'
  end

  task :rhino do |t|
    #Seems like i may need to do an MSBUILD using the generator sln file before the whole build will work
    cd 'rhino-tools'
    puts "Building Rhino-tools"
    puts `msbuild.exe BuildAll.build`
    puts "Done building rhino"

  end

  task :update_rhino_from_castle do
    rhino_castle_libs = FileList['./rhino-tools/SharedLibs/Castle/*']
    LibCopy.new.copy(rhino_castle_libs, CASTLE_BUILD_DIR)
  end
end

namespace :libcopy do 

  task :castle_to_api do
    api_castle_libs = FileList['./OmniaAPI/DependentFiles/Castle/*']
    LibCopy.new.copy(api_castle_libs,CASTLE_BUILD_DIR)
  end

  task :castle_to_mrrest do
    dest_castle_libs = FileList['./MRRestSupport/lib/*']
    LibCopy.new.copy(dest_castle_libs,CASTLE_BUILD_DIR)
  end

  task :castle_to_myproject do
    dest_castle_libs = FileList['./myproject/DependentFiles/Castle/*']
    LibCopy.new.copy(dest_castle_libs,CASTLE_BUILD_DIR)
  end

  task :castle_to_codegenerator do
    dest_castle_libs = FileList['./Castle.MonoRail.CodeGenerator/Libraries/*.dll']
    LibCopy.new.copy(dest_castle_libs,CASTLE_BUILD_DIR)
  end
end

Technorati Tags: git, castle

Why frequent commits are important

chrisortman — Thu, 14 Feb 2008 15:40:49 GMT

Something I have noticed has changed about the way I work now that I use git instead of subversion is that I make a lot more commits.

Let's say that in order to add feature X I need to make the following changes

Add migration
Add ActiveRecord model Foo
Add PersistanceTests for model
Write Tests for feature X
Rename an existing class
Change the way some other model class works
implement some behavior in Foo
Ensure that tests pass

In a lot of code I've seen all these changes would be grouped into a single SVN commit

svn commit -m "Implemented feature X"

But when that is the case a lot of information gets lost about how feature X got implemented.

I think that each of those changes should be their own commit with their own explanation. Centralized version control doesn't help this though because I would get impatient with making 3 minutes worth of changes and waiting 30 seconds to make the commit. I also think that pushing each change out to everyone else before the whole feature is ready would be a bad thing.

Using a DVCS (such as git) it is painless to do this. The commits are fast, and I just because I commit doesn't mean I need to share. I also have the freedom to go back and rewrite history if I realize while I'm implementing some behavior in Foo that I forgot a column that should have been part of my initial migration.

Why I <3 git rebase

chrisortman — Thu, 14 Feb 2008 15:27:07 GMT

I recently started adding a new feature to one of my projects. The first thing I typically do in this scenario:

git checkout -b name-of-feature current-version (replacing name-of-feature and current-version appropriately)

Now as I'm working on this I may bump into something that if I were to refactor would make this new feature a lot a lot simpler to implement.

If I were using only subversion and I started on this refactoring I would have a source history that looks something like:

Added migration for table x

Created new models + tests for feature X

Renamed class foo to bar

Refactored some service into separate services

next change for feature x

fixed bug 370 reported by QA

another change for feature X

Which isn't bad persay, but it would be better if things were in their proper order. I also don't like that I would have to commit some incomplete implementation of feature X just to fix a bug. If I had a little better foresight and planning I might have fixed the bug first, then done that refactoring that will make feature X simpler then actually implemented feature X. Fortunately git makes this pretty easy to do.

Here's what my output of git branch might be:
v1.2
add-feature-x
refactor-some-service
fix-bug-370

v1.2
       |
        -------add-feature-x
                                      |
                                       ---------refactor-some-service-----
                                                                                     |
                                                                                      ----fix-bug-370

In my graph here a - represents a commit and | a branch point
What this says is that I started adding feature x, did some refactoring, stopped to fix a bug and continued my refactoring after.

Just because things happened in this order is no reason that history must say they did. I guess I like the Back to the Future theory of time travel better than Terminator.

Since I haven't shared these changes yet, I can fix up my history.

git co fix-bug-370
git rebase v1.2

I now have

And a couple more times:

git co refactor-some-service
git rebase fix-bug-370
git co add-feature-x
git rebase refactor-some-service

So that I now have

v1.2
       |
        -------fix-bug-370
                                  |
                                   ---------refactor-some-service-----
                                                                                      |
                                                                                      ---- add-feature-x

Which when I eventually do

git co v1.2
git merge add-feature-x

Will produce a nice linear history in the order things should have happened.

Creating Subversion patches with git

chrisortman — Tue, 15 Jan 2008 06:26:58 GMT

Adam expressed his frustration with creating tortoise compatible patches using git.

He mentions a script that will do the job, which I have not yet tried. However if you are using git on windows you most likely have cygwin installed which you can use to install the patch.exe program.

But first you need a patch file.

This is pretty simple

In my example I'm going to be creating a patch to fix FACILITIES-97 in castle project.

First I need my local git repo ready

git co -b bugs/facilities-97 castle-svn/trunk

Now edit / test

When you're all done prepare your index and commit

git add .
git commit -m "fixes (FACILITIES-97)"

Now create your patch

git format-patch castle-svn/trunk

That last command will create a patch file for every commit that is in bugs/facilities-97 but not in castle-svn/trunk
In this case I have a single commit so it creates 0001-fixes-FACILITIES-97.patch

Now that you've got your patches they can be applied to svn like this

patch -p1 -i 0001-fixes-FACILITIES-97.patch

Now here's the gotcha. patch.exe can either read stdin or take a named file. My patches would always fail if I used stdin, so using -i to specify the filename is important.
The -p1 switch tells patch to strip the first part of the path in the patch file because git generates file paths like
/a/InversionOfControl/MicroKernel.....

That's it, you're all done.

Limitations of svn branching

chrisortman — Wed, 09 Jan 2008 17:53:41 GMT

I generally run all my projects that use castle off the trunk. However it has been several months since I've taken any updates, mostly due to lack of time on my part.

As I'm upgrading there are many many changes which require me to update my code. Most due to the introduction of IController and changes from IRailsEngineContext.

One such instance is IResponse.BinaryWrite which seems to no longer be available.

As I dig through svn to try to find if there is a replacement method or why it was removed etc I am hit right in the face with how poorly svn handles branching and merging. The only log message in the trunk is that all these changes were merged back in in one giant commit. If you want to go see what those changes were you need to go look through the branch that was merged, but even then there's no way of knowing exactly which changes were pulled ( I just assume all.)

With git I would be able to see these sort of things and save myself a lot of time.

Getting started with the Castle git repo

chrisortman — Wed, 17 Oct 2007 15:35:09 GMT

Before we dive into commands I feel the need for a little rationale. I'm not going to explain all the virtues of GIT as google will happily do that for you, I am however going to explain why I need to use it.

I have 5 separate projects that all use something from Castle. Keeping track of which project uses which build can be very difficult espcially since I always run from castle trunk. Subversion would have me create a separate vendor branch in each project and stick mirror the castle repo there. That is all well and good until I start fixing bugs or making enhancements. As soon as that happens things get hairy.

My solution is to have a git repository that mirrors the castle svn but contains separate branches for each product that is using it.

On my system git branch -a will list something like

castle-svn/trunk

castle-svn/branches

castle-svn/tags

Product1/current

Product2/current

MyPatches/MR-XXX

MR-Restsupport

.....

Now when I need to jump between projects and dig into castle source it is as simple as

git checkout Product1/current

I can also freely move patches around and stay up to date with the latest and greatest being produced by the castle community.

Setting up the git mirror can be a time consuming process. You do have the option of only pulling history up to the last N revisions, but I like full history so I can visualize how things have changed. Pulling the full history takes a while and puts an increased load on Hammett's servers.

That is why I did the full import and published the mirror to http://repo.or.cz which provides public git hosting.

Now the good stuff. To get started you want to install cygwin + git. Git is a cygwin package so this is easy. I believe there is a MinGW port in progress, but I already had cygwin so this was easy for me.

Now to clone the repo simply do

git clone -n --origin castle-svn git://repo.or.cz/castle.git

These are not the standard options, so let me explain.

-n tells git to not checkout anything after it copies. I like this because a master branch isn't very meaningful to me in this context.

--origin castle-svn tells git to place branches in from the remote repository into /refs/remotes/castle-svn instead of /refs/remotes/origin (the default). This again is just for simplicity on my part. I like doing git checkout trunk castle-svn/trunk better than git checkout trunk origin/trunk

What you have at this point is a git repo that has all the SVN history. On my machine this actually takes up less space than the svn checkout

That is all well and good but you only have the snapshot of the trunk as of when I created the git repo.

At some point I will setup up an automated pull to keep history up to date, if you want to be able send a commit to the castle server you will need svn metadata anyway. To do that we use:

git svn init https://svn.castleproject.org/svn/castle -T trunk -t tags -b branches

This will create a new .git/svn folder and add some options to your .git/config file.

Because we're doing things a little custom from the default behaviour of git-svn we need to edit this .git/config file to tell it where our subversion revisions go.

Open up .git/config in your favorite text editor and make the svn-remote section look like this

[svn-remote "svn"]
    url = https://svn.castleproject.org/svn/castle
    fetch = trunk:refs/remotes/castle-svn/trunk
    branches = branches/*:refs/remotes/castle-svn/branches/*
    tags = tags/*:refs/remotes/castle-svn/tags/*

The stuff to the right of the equals sign is called a refspec and it tells git-svn that heads from svn under trunk goto refs/remotes/castle-svn/trunk (not the wildcard for branches and tags). This is creating remote-tracking branches in git.

With this in place you can now do git svn fetch and git will check your history against all the castle revisions. This does take a minute or so, but it is much faster than having to pull all the svn file data.

So now you want to work with RC3?

git checkout -b RC3 castle-svn/tags/1.0.x-RC3

What about the trunk?

git checkout -b trunk castle-svn/trunk

Maybe you need RC2

git checkout -b RC2 castle-svn/tags/1.0.x-RC2

Try switching between RC2 and RC3 and see how it compares to the paultry svn-switch.

In our next installment we'll look at fixing a bug and creating the patch.

Cheers

PS: Trouble committing? Check here
PPS: My list of git resources so far http://del.icio.us/chrisortman/git http://del.icio.us/chrisortman/git-svn