Since I like to eat, and hence get paid so that I can afford food, I have to work on things that take longer than a week.  But luckily, git can help get a diff for review board based only on a string in the commit comments.

This came about when one of my co-workers posted a bash scripts someone had written that would take a git diff and make it into a svn-like diff.  And that worked great.  My first attempt was to modify it to do multiple files at once, but it ended up making a diff for each revision for each file (if you need that functionality, you can get that here).

So I pulled out python and worked on a new script, which is at the bottom of this post, or you can get it here.

Once you download the file, you run it like this from your git repository:

git-create-review.py (some grep string) > diff.output

The script will take the argument as a string to search for in the commit comments. This is thanks to the git log –grep=foo command available in git. It outputs directly to the terminal, which you can see I’m redirecting into diff.output.  Once you have the file, you can upload to Review Board as you normally would.

I did it by using a search string because we use JIRA to track our work in svn.  We have pre-commit hooks to ensure that we put a JIRA number on each commit.  This helps out git as it has something to search through to get the change list.

Known issues:

First: It will get all the changes between the first and last revision that had the search string.

Example:

I change File.java under JIRA-123 (now at r1).

Another change to File.java under JIRA-321 (now at r2).

I change the file again under JIRA-123 (now r3).

This script will take all changes between 1 and 3, even if I don’t want the JIRA-321 changes to show up.

Second: it doesn’t have any context around removed/moved files. It still should pick up a renamed file as just a completely new file, but it doesn’t have any context of renaming.  But I believe that’s the case with svn anyway.

Third:  I am by no means a good python programmer, and might not know all the conventions that people normally follow. The code below may cause eye hemorrhaging, face hemorrhaging…pretty much any of your organs could be hemorrhaged by looking at the code. CONSIDER THIS YOUR ONLY WARNING.

But I did some testing with newly created files, and removed files, and binary files, but I’m sure I missed some crazy edge cases.  Let me know if you hit any snags.

The script follows:

That person may be tremendously happy…that is, until, you try to “git svn fetch” to get new revisions and branched and what have you, and you get an error message like this:

Last fetched revision of (some branch or tag) was r3421, but we are about to fetch: r3421!

In an attempt to fix it, you may find this blog post, saying to remove the .ref files from your svn directory, which also proves fruitless.

The fix: modify the .git/svn/.metadata file so that the branches-maxRev and tags-maxRev equals the latest commit revision in svn.  So it should look like this after you are done:

; This file is used internally by git-svn
; You should not have to edit it
[svn-remote "svn"]
 reposRoot = (svn server name)
 uuid = c08781d2-f03e-0410-8c7c-e884ea3e41f3
 branches-maxRev = 17224
 tags-maxRev = 17224

Where 17224 was our latest revision.

I got the idea from git-svn’s webpage, where it states:

If the subset of branches or tags is changed after fetching, then .git/svn/.metadata must be manually edited to remove (or reset) branches-maxRev and/or tags-maxRev as appropriate.

I must caveat this with the fact that I have no idea why this works. Git svn is still black magic to me, and making this work is akin to me throwing muskrat bones onto a flaming pyre, and drawing conclusions from the charred remains.  Maybe someone with more git knowledge could drop it off here?

You also have created a set of tests in your DAO layer, using the AbstractTransactionalSpringContextTests.  Good for you, again.

But when you try to mix these two together, you may see tests failing and can’t immediately see why.  This can happen if you try to get data from the materialized view after attempting to save data to a table that the view accesses.

Well, it’s very simple, and probably pretty obvious.  It wasn’t immediately obvious to me why it was failing, and I didn’t really see anything online about it, so hopefully this will stop your search quickly.

First thing to realize is that running your DAO tests through the AbstractTransactionalSpringContextTests is that every test is run in a transaction.  That’s great most of the time.  Each test will run in a little isolated environment and the changes to the database will be rolled back at the end. You don’t have all kinds of test data strewn about afterwards as if it were the island in Cast Away.

The second thing to realize is that a materialized view will only be updated on commit.  There’s a lot more to it than that, but I am not a DBA and that’s all you need to know in general.

So, with those two things in mind, you can see how it might be a problem if you try to write tests against code that uses the materialized view.  All is not lost however.  If you put an endTransaction() call somewhere in your test, all of the DB calls you’ve made up to that point will be committed.  Also, any other calls made after that point will also be committed to the database.  Since that happens, you will be on your own to clean up after the test as you see fit.  Also, placing an endTransaction will only affect the test you place it in.  Any other test will still continue to run and rollback in the normal fashion.

What we ended up doing was making the tests that need to commit to the database small as possible.  We then wrapped the endTransaction() function in a function called setTestToCommit() (my co-worker James’ suggestion), so that it makes more sense to read it, and placed it at the beginning of the test.

There’s lots more information about how it works here.

So you want to work with multiple svn branches (not to be confused with git branches). What you have to do is clone the svn repository in a different way, like this:

$ git svn clone https://somesvnrepo/svn/project -T trunk -b branches -t tags

This should work in most cases. The arguments to the -T, -b, and -t switches are the paths that the trunk, branches, and tags are in respectively. Typically, your branches and tags should be what I put above, but just in case it is different in your case, there you go.

Working with your shiny repo

After a short time in which humans achieve interstellar travel, git finishes cloning, and you have access to the entire svn repository. Now, let’s say you wanted to start working on trunk. Like before, you should checkout a branch, but this time it’s going to use trunk as it’s parent:

$ git checkout -b BranchOfTrunk trunk

But Lo! You have the entire repo, so you can checkout that branch from whatever branch you have in subversion.

$ git checkout -b BranchOfSubversionBranch subversion-branch

Now, you can switch between BranchOfTrunk and BranchOfSubversionBranch, and you’ll get all the code associated with trunk and subversion-branch respectively. No more checkout of a branch ever again, wasting precious keystrokes you could use to post to Facebook.

If a new svn branch is created on the server, make sure to run the following command to see it:

$ git svn fetch
. (updating blah blah)
.
.
$ git branch -a

The last command will show you all the branches, local (your own branches) and remote (every dumb svn branch), for this repository.

NOTE: git svn fetch and git svn rebase are almost exactly the same. fetch will update the entire git repository with anything not received from svn. rebase is concerned only with the parent of the branch you have checked out. If you recall last time, git svn rebase was equivalent to svn update. Now you can work with the same workflow as I mentioned before, working, committing locally, rebasing, etc.

Merging from subversion branch to trunk? BALDERDASH!

Now, let’s say that you want to commit a series of changes into a subversion branch, but you also want to commit the same changes in trunk also. What I’m going to show you is the workflow that I go through most of the time. I think it’s the most effective way to accomplish it in git.

First, you can run a fake dcommit.

$ git svn dcommit --dry-run
Committing to https://svnserver/svn/project/branches/subversion-branch ...
diff-tree b64fdc9250bbe04b3246214ade509e0b0d9d912d~1 b64fdc9250bbe04b3246214ade509e0b0d9d912d

The “–dry-run” is key here, for two reasons. First, you’ll be able to see what branch you are actually committing to in subversion. Not a bad idea, since you’re working in a git repo that has all your subversion branches.

Second, you’ll see a diff tree with some of the commit hashes, one corresponding to each time you did a commit locally, in order. You will want to keep the values of the hash around, because next we are going to “cherry-pick” them into trunk:

$ git checkout BranchOfTrunk
Switched to branch "BranchOfTrunk"
$ git cherry-pick b64fdc9
Finished one cherry-pick.
Created commit e4a7971:  Your commit comments here
 12 files changed, 58 insertions(+), 66 deletions(-)

You’ll notice I only used the first 7 digits of the hash (b64fdc9) to do the cherry-pick. That’s all you need to use…git will pick out the revision from just those seven characters. What this does is pulls the change from the other branch into this branch. If you have more than one commit, you should do this multiple times, in the order of the commits. But then you have the things you did on the branch ready to be checked into trunk!

This is the way I usually work. You may decide that doing a subversion merge is easier for what you are trying to do (I sometimes do also). I’d also like to know if there is an easier way to do this. This doesn’t seem the easiest way, but was the easiest way I could find so far. And I googled for like…20 minutes.

Hopefully this helps…someone. One person, at least.

Simply install ‘xclip’, which is simple also, if you are on ubuntu:

sudo apt-get install xclip

Anything that you pipe to it goes to the X11 clipboard. Once there, you can just hit the middle mouse button to paste it where you want, just as if you had highlighted it. So you can do things like this:

$ cat some.file | xclip
$ nslookup somehost | xclip
$ ./runsomeawesomescriptthatyouwanttheoutputintheclipboard.sh | xclip

I also set up a google code project here for it.  Please let me know if you have any features or feedback on it.

Thanks!

Recently my co-worker James suggested we automate our tests on our project using JMeter and gave me his webpage about the trials and tribulations he went through to get it set up.  It linked to a wiki page, in which someone had created a JMeter plugin for Maven.

Well awesome, it shouldn’t be too hard to set up.  And it wasn’t, I had the JMeter tests up and running in a short amount of time.  But one of the problems was that I wanted to set it up in a more generic way so that we could define different parameters on each of the CI environments we ran. 

For this task, I first tried using filtering in maven, which I got working.  But what you end up with is a JMX file that is basically unusable by default in JMeter. If you ever wanted to run it again standalone in JMeter, you’d have to modify the variables in the JMX file to work correctly. Vice versa, if you set up your test using JMeter, you have to remember to modify it so that the maven variables you need are in the places you need them.

JMeter already has a system in it where you can define variables in the JMX files (and give them defaults, fancy la la!).  For example, say you want to define a variable for a host name.  Inside your JMeter JMX file, you can do something like this:

<stringProp name="HTTPSampler.domain">${__P(someVariableName, localhost)}</stringProp>

Now, you have defined someVariableName to be a variable within your JMX file.  The second value, in this case localhost, is the default if no variable override has been given.

The way to override these variables is with the -J option on the command, in this form:

-JsomeVariableName=yourhost.com

and all the variables called someVariableName will have the value yourhost.com.

Now, with the minor modifications I have made to the JMeter Maven plugin, you can now define the values for these variables inside of your pom.xml, like so:

<jmeterUserProperties>
      <someVariableName>yourhost.com</someVariableName>
</jmeterUserProperties>

and it is just the same as using -J on the JMeter command line.

The way we’re actually going to use it is to combine it with maven variables, like so:

<jmeterUserProperties>
      <someVariableName>${ourhostname}</someVariableName>
</jmeterUserProperties>

So now, we can just do:

mvn -Dourhostname=ourspecialhost ...

and be able to run site specific JMeter tests with only changing the arguments on the command-line.

I will be putting together a page in the next day or so with all the libraries needed to set this up, similar to the wiki page.

But! Not all is lost.  You can use git with Subversion.  This can give you some of the benefits of using a DVCS without having to change everything you do.  I believe you can do this also with Mercurial…I went with git because I had read somewhere it had the best support to working with Subversion. AND RANDOM INTERNET BLOGS ARE ALWAYS RIGHT.

First of all, I will say that I used this as a guide when I was working.  It’s a great start to doing this stuff. I’ll also assume you can find and install git on your own, since you’re all-grow’d up now.

Step 1: Creating a local repo

Ok so let’s get started.  The first step is to get the subversion library.  That webpage I gave you gives you four different ways of doing it, but I decided to go with the fourth option, which is cloning the entire Subversion repository.  You can do that by doing:

git svn clone https://svnserver/project/trunk

Now, you should know that this could take a while.  You might as well go and eat dinner for the next three days while this completes.  Just kidding, this took me about an hour or so while I was working remotely over a VPN, so it’s not so bad.  Our project has about 8000 revisions, and about 230 MBs of files.

So you’re probably wondering “But now if I have the whole history, isn’t it going to take like…a bazillion bytes on my disk?”  To that I say “Non”…it uses compression, and my git repo ended up being about 1 MB bigger than my subversion one.

Step 2: Create a branch

From there, you can start working in git!  The first thing you’ll probably want to do is create a branch:

git branch someNeatoBranch

To change to that branch:

git checkout someNeatoBranch

Now you are working on this particular branch. Note you can do this in one step by doing:

git checkout -b someNeatoBranch

which will create it and put you on it to work.

Step 3: Commit locally

Now you can edit the files to your heart’s content.  And commit locally as often as you like, by doing

git commit -a

You can commit individual files by specifying them, but -a just grabs them all.

Step 4: Commit back to Subversion

Once you’re ready to commit back into the main Subversion repo, first you need to pull all the changes from there, to make sure there are no conflicts. You can do this by using this command:

git svn rebase

This will pull down the latest code and apply your code on top of it, merging if necessary.  Then to commit back to Subversion, you use:

git svn dcommit

And you will notice that your files are committed back to the repo without any more intervention from you!  But where are your comments?  Well, all the comments from the commits you did locally will be used to commit to Subversion.  It will also do that many commits.  So if you did 5 different commits locally, it will turn into 5 different Subversion commits.  As long as your team members don’t worry about the revision number spirialing up towards the heavens, it should be ok.

Step 5: Brunch!

This is a pretty simple workflow to get you started on using git with Subversion, but it’s not even half the story.  There’s so much more powerful stuff with the branching aspect, and I even haven’t tried it yet to share work with my coworkers before commiting to svn (I’m kinda like a git island at the moment).  But try it out, and then check out some of the resources below that helped me get started.

Also, you might want to check how my co-worker James one-upped me.  He was working on some highly experimental code that he wanted to share with a teammate, and created a true git repo in approximately 8 seconds, from existing code.

Resources

• An introduction to git-svn for Subversion/SVK users and deserters
• Git – SVN Crash Course

tar cvzf somefile.tar.gz `svn stat | awk '{print "test -f " $2 " && echo " $2}' | bash`

Word.

To break it down:

The svn stat command is getting all the files that have changed obviously.

The awk command its piped to is the real part that’s doing the work.  The awk command prints

test -f file && echo file

Pipe that through bash, and you have commands, which essentially will echo the file if the file is a file. Simple? No? I mean, it will not print it out if it’s a directory, so you don’t have to worry about it adding entire directories.

Surround that whole thing in backquotes (that’s the one with the tilde (that’s the one next to the ‘1′ key ( I’m not telling you where the ‘1′ key is)))  and give it as the argument to the tar command.