Archive for May, 2008

Psuedocode for server management scripts

May 11, 2008

Along the lines of my post on the Forkolator blog on how to set up the server architecture, I’ve written out some pseudocode for scripts that I think would handle the mongrel servers and the nginx load balancer properly:

% start_nginx
get all mongrels from database
create nginx conf
start nginx

% start_mongrels testproject production 2
mongrel start get_a_free_port()
insert mongrel
pid=278112
port=8004
servername=testproject.forkolator.org
app=testproject
environment=production
mongrel start get_a_free_port()
insert mongrel …
mark server for refresh

% start_mongrels testproject development 1

% reload_nginx (runs as a daemon)
get all mongrels from database
create nginx conf
reload nginx conf

% stop_mongrels testproject production
find the mongrel processes
kill them
remove them from the database
mark the server for refresh

How to get Gitorious running on your own server

May 11, 2008

This is everything I’ve figured out so far about how to get Gitorious up and running on your own server. It’s not everything yet, but it’s a lot of stuff.

OK, so first you want to install the dependencies. I think this are all of them, assuming you’ve installed the basic rails stuff:

apt-get install librmagick-ruby libonig-dev
gem install mime-types oniguruma textpow chronic BlueCloth

Then make a place for Gitorious and download the code:

mkdir /path/for/gitorious; cd /path/for/gitorious
git clone git://gitorious.org/gitorious/mainline.git  
cd mainline

It’s a good idea at this point to read the HACKING file for informational purposes.

The next thing we need to do is create a git user. This will be the user that gets used for SSH connections, running the task queue, and servering the repositories over HTTP:

adduser git

Once we’ve created a user, we can create a place for your git repositories. It has to be owned by the git user, and if you want people to be able to clone over HTTP, it has to be web-accessible. That means your web server has to be running as the git user:

mkdir /path/to/repos
chown git:git /path/to/repos

You are going to need to add your database settings and gitorious configuration. Start with:

cp config/gitorious.sample.yml config/gitorious.yml
nano config/gitorious.yml;

… and put in the necessary info. Next put in the database information:

cp config/database.sample.yml config/database.yml
nano config/database.yml

I called my databases gitorious_development, gitorious_production, and gitorious_test, and I set up a gitorious user in mysql with a fancy password. At this point you want to create the development databases, and then you can start the server:

rake db:migrate
script/server

You should now have the gitorious app running on http://localhost:3000 or wherever you specified in gitorious.yml. Give it a look in your web browser. If you want the server to run in the background, kill the current process (CTRL+C) and then run:

nohup script/server &

That will start the server up in a separate process and spit the output into nohup.out. You can try creating a user account through the web interface. If you don’t get an activation email, there’s something screwy with your mail server. I had to patch Gitorious to use a SMTP server, but that’s a separate post. As a workaround, just go into your gitorious_production database, find your account in the users table, and copy the date in the “created_at” column into the “activated_at” column.

Now you should be able to log in, create a project, and add your SSH public key to your profile. Unfortunately, Gitorious is going to tell you that your key and your project are being created, but they won’t get created without some more intervention.

In order to create the keys and the repositories, you have to set up a separate process that does those tasks regularly. Gitorious has a script set up for this purpose in script/task_performer. Go ahead and run that script, making sure that you run as the “git” user and that you set the correct rails environment. So if you are working in the development environment, run:

su git
RAILS_ENV=development script/task_performer

That will create the repositories and add the keys to /home/git/.ssh/authorized_keys. The authorized_keys file also specifies that before users can be authenticated for git push over SSH, the command “gitorious blah” needs to run (where blah is the users’s username). Except there is no gitorious command in the git user’s path. So let’s link it up:

ln -s RAILS_ROOT/script/gitorious /bin/gitorious

Now if you return to Gitorious, you should see that your keys and repositories have been created. You may want to stick that task_performer script in a cron job… or write a little script that will run it repeatedly in the background.

At this point, you should be able to do a git push (which goes over SSH) and a git clone over http, assuming your repositories are in a web accessible folder. In order to do git clones, you have to start the git daemon. Gitorious has a script for this, but again make sure to run as the git user:

su git
nohup script/git-daemon &

And now you should be able to do git:// clones too.

What’s left to do:

  • I haven’t configured ultrasphinx, so search doesn’t work.
  • I haven’t set up script/graph_generator, so there are no graphs.
  • I have no idea what script/fixup_hooks does. It might be important.

Thanks pygi and teknofire in #gitorious on irc.freenode.net for helpful pointers. Thanks Johan for guidance on getting ssh working correctly for git push.

How to slice up tests

May 7, 2008

One of the hardest things about test driven development is choosing how to slice up your tests. Do you do unit tests (trying to isolate one unit of code) or integrating tests (checking to see that it all works together to lead to some positive outcome)?

I guess you do both, but I still don’t have good instincts about what to do when. Zach Dennis responded to a post I sent to the rspec-users mailing list giving me some good advice about the test I wrote about here.

I’m starting to think my first test should really be to send an email and then see if it got there. Unfortunately, email takes minutes to deliver, and how long should the test suite wait? Maybe I should just give up on testing this component.

How to use mock objects

May 7, 2008

I’ve been really enjoying test driven development lately. It’s more fun, more relaxing, and I think more productive than the classic “just go for it” or “try to plan everything out ahead” styles of development.

For those who don’t know, “test driven development” (TDD) is a style of building stuff. Typically when you want to build something like software, you start off by planning what you’re going to do, or you just start building. TDD is different. You start off by creating an automatic test that tells you whether you succeeded or not. So if I wanted to write some code that added apples to the bin, I would start of writing something like this:

test "when you add apples to the bin, there should be more apples in the bin" do
  start_number = count_apples_in(the_bin)
  add_apples_to(the_bin)
  end_number = count_apples_in(the_bin)

  end_number.should_be_bigger_than(start_number)
end

Note that we haven’t actually written code that adds apples to the bin. What we did was write some code that can tell us if we have something that works. If we run the test, of course the test will fail. But now we have a goal, and we have a good feedback loop. So we write a little code, and see if the test passes. Write a little more, see if it passes. As soon as the test passes, a little green light lights up, and we’re like “yay! our code works!”

The benefits of this are many. First, it’s emotionally gratifying. It makes coding more fun because getting another test to run is never that far away. And if it is far away, you just write another test that’s halfway there. Second, it lets you write really big software and have some confidence that it all works OK together. Because as you write code you build up more and more tests, and you keep running all those old tests, you have a continuous sense of what is working.

Anyway, that was a bit of a digression. What I really wanted to write about is “mock objects”. Once you’ve written a few tests you start to realize that often you want to test a little bit of code that relies on other objects, but you don’t want to test those other objects together with this one. So, if A depends on B, which depends on C, which depends on D, you want to have four separate tests that test each of those things independently.

But how do you test them independently if they are dependent?

The answer is: mock objects. If you’re testing B, you create a “mock” of C, which is just a fake object that does what C is suppose to do in your particular test case. That way your test isn’t dependent on C working properly.

Here’s an example. I want my Mailer object to get set up properly. But the Mailer depends on the GitoriousConfig object, and I don’t want to test how that is getting set up. So really I just want GitoriousConfig to act right for the Mailer. So I use a mock:

  it "uses smtp server if config says so" do
    GitoriousConfig.should_receive(:has_key?).with('smtp_settings').and_return(true)
    GitoriousConfig.should_receive(:[]).with('smtp_settings').and_return({
      :address  => "smtp.postoffice.net",
    })
      
    Kernel.load("config/initializers/mailer.rb")   
    Mailer.smtp_settings[:address].should == "smtp.postoffice.net"
  end

What this code says is that GitoriousConfig should act as if it has a smtp_settings hash in it, and that the address entry points at smtp.postoffice.net. A depends on B, but I’m using a fake version of B so I can isolate the test to A’s behavior.

That allows me to keep my tests nice and clean and separate. Cool.

“This repository is being created”… but it never is.

May 6, 2008

The HACKING file in Gitorious doesn’t quite make this perfectly clear: If you’re getting “This repository is being created”, it’s because Gitorious doesn’t actually do things like create repositories when you create them through the web interface. It adds an entry to the tasks table in the database, and then a separate process comes around to do the work.

In order to actually do those tasks, you run script/task_performer. If you’re working on a development branch, you need to run

RAILS_ENV=development ruby script/task_performer

And that should do the trick. I guess for a production server, you’d just start a separate process that continually checks for new tasks to do. Not sure if that’s somewhere in the gitorious code. It probably is, I just have to find it.

Setting up ruby on rails databases

May 6, 2008

This was not 100% clear to me. But in order to set up your development databases you can just do:

rake db:migrate

But in order to get the database set up for production, you do:

rake db:migrate RAILS_ENV=production

I don’t exactly remember what I did to create the database for testing.

Missing markdown method in ActionView::Base

May 6, 2008

I am trying to get Gitorious running on my own server. Everything is working OK, but I get this error when I try to view a project page:

NoMethodError in Projects#show

Showing projects/show.html.erb where line #5 raised:
undefined method `markdown’ for #

A google search pops up this page which led me to believe that I didn’t have BlueCloth installed properly. Turns out the BlueCloth gem is case sensitive, so I had to do a

gem install BlueCloth

Because, as it turns out, in vendor/rails/actionpack/lib/action_view/helpers/text_helpers.rb file it only adds the markdown method if it finds BlueCloth. I’m still a little sketched out by how much generated code is in a typical rails app. But I guess as you get further along, you start to learn which is generated and which is not.

I did have to restart the mongrel server before the error went away, which kind of sucks. On a production system, restarting the server would be pretty traumatic, and shouldn’t be necessary for adding a new package. Seems like a bug that putting a restart.txt file in the app’s tmp directory didn’t exist.

That’s not something I’m going to take on right now though. Gitorious is working, so I’ll keep working on adding stuff.