Run To The Future

We recently moved from using Hudson to Jenkins (not by choice, our sysadmin just decided to do it one day). I know there's a big nerd-rage about Hudson vs. Jenkins but I honestly couldn't care less. I just want something that works. Well, from what we've seen, Hudson is more stable than Jenkins.Here are some of the problems we are getting (and fixes):

Error	Fix
FATAL: Could not checkout null with start point 1234... Caused by: hudson.plugins.git.GitException: Error performing command: git checkout -f 1234... Command "git checkout -f 1234..." returned status code 1: error: unable to unlink old '' (Permission denied)	Jenkins keeps thinking one file has been modified locally, so (from the jenkins machine) git checkout that file and the tests will run fine.
Sometimes the Xvfb process crashes. You will only know because the tests will not stop trying to run, so any running time above 15 minutes means this has probably happened.	Restart Xvfb (which is supposed to be always running in the background): Xvfb -ac :99 &
Sometimes Jenkins doesn't want to checkout from git (sometimes for the same reason above, a file that has been "modified" locally, but sometimes for no apparent reason): ERROR: Problem fetching from origin / origin - could be unavailable. Continuing anyway ERROR: Could not fetch from any repository FATAL: Could not fetch from any repository	You have to login to the jenkins machine and manually git pull origin dev from the repo, and then run the tests again (via jenkins interface).
Caused by: hudson.plugins.git.GitException: Error performing command: git tag -a -f -m Jenkins Build #309 jenkins-oars-dev-309 Command "git tag -a -f -m Jenkins Build #309 jenkins-oars-dev-309" returned status code 128: error: insufficient permission for adding an object to repository database .git/objects error: unable to write tag file	Not sure about this one. The selenium log was very big so I deleted it and the tests ran. The log is supposed to get rotated but maybe this didn't occur.
error: Your local changes to the following files would be overwritten by merge: Please, commit your changes or stash them before you can merge.	These are local changes to the files on the test machine (not the jenkins machine). So from jenkins, there's a script to remotely update the files on the test machine, but in this case there were local modifications (not sure how that happened either). There's a plan to move the .test repos/installs to the jenkins machine so we won't need to do the remote update. However this plan has been devised by our "sysadmin", so might take a while to eventuate. Solution is to checkout the locally modified files on the test machine then do a git pull.

Please, commit your changes or stash them before you can merge. To be continued...

This is really a note-to-self.

Sometimes when I run my unit tests in Lithium (li3) I get something like:

Exception thrown in lithium\data\Model::connection() on line 929:
The data connection `test` is not configured.

Which doesn't make any sense because our unit test classes inherit from our own Unit class, which determines which data connection to use, however in our Unit test class setUp method we have to tell every class we need to use to use the correct data connection. Also, we never use the data connection "test". (The "test" data connection is the one li3 chooses by default when tests are run, which is why we had to overwrite it in our Unit class.)

So why is this happening? Because a class that I'm using as part of the testing includes another class in itself, and so we also have to manually tell this file which data connection to use.

Probably a better way to do this :D

Lithium Acceptance Tests using Cucumber and Webrat

Introduction

Cucumber (aka. cuke) and webrat are ruby tools for behaviour driven development (BDD) and while they're mostly used on rails apps, they can be applied to PHP apps too. Our motivation for doing this comes from using a virtual machine (vm) for our (work) development. We're using lithium now for all new projects, and we want to do BDD and Test Driven Development (TDD) as well. TDD is fine – lithium supports this really well out of the box (and michaeltwofish is currently writing a lithium plugin to allow the command line testing to output to hudson). Our plan was to use selenium for the acceptance tests (read BDD), but as selenium needs to run a firefox executable, this can't work (easily¹) on the vm. So we've now got webrat (and cuke) running instead.

Most of this post is taken from other sources. The first place to read is aslakhellesoy/cucumber/wiki/php. It's also well worth looking at chits: a real PHP application that uses cucumber/webrat. Also see Acceptance tests on a PHP project with the cucumber / webrat / selenium trio.

I repeated these procedures on darwin and centOS without any problems.

Installing ruby and gems

I downloaded and installed ruby 1.9.2 from source. This also installed the latest version of ruby gems.

$ ruby -v
ruby 1.9.2p0 (2010-08-18 revision 29036) [x86_64-darwin10.5.0]
$ gem -v
1.3.7

Installing required gems (cucumber and webrat)

These are the gems I installed:

$ gem install cucumber
$ gem install webrat
$ gem install mechanize
$ gem install rspec

Layout

You'll need to put your cuke and webrat files somewhere. Typical location is the app/tests/features directory. The feature files go in the top "features" directory:

+––– apps/tests/features/
     +–––feature1.feature
     +–––feature2.feature
     +––– step_definitions/
           +–––webrat_steps.rb
     +–––support/
           +–––env.rb
           +–––hooks.rb
           +–––paths.rb

Features

A feature could map to a high level user story (in the agile method), and be made up of a number of scenarios to be tested. The feature is executed by cucumber and the output will show whether your acceptance tests have passed or failed (or some other error has occurred).

There's plenty of information about how to write these feature files on the cucumber site.

Step Definitions

Each step of a scenario (within the feature file) needs a definition in code of what is to be executed. These steps go in the step_definitions/webrat_steps.rb file. Some of the steps defined here are "standard" eg:

Then /^I should see "(.*)"$/ do |text|
  assert !!(response_body =~ /#{Regexp.escape text}/m), response_body
end

which is used to check if a particular piece of text appears on the page.

See the webrat repo on github, as well as the cucumber wiki page on step definitions, but also take a look at the step definitions of the chits app as this has a lot more examples.²

Support Files

There are three support files. env.rb is used to setup the testing and code here is run before (and possibly after) every scenario. The paths.rb file is a convenient place for the URLs webrat will access. The hooks.rb isn't strictly necessary but if there is code that needs to run before (or after) specific scenarios, this is where it goes.

See the wiki page on using cucumber for PHP apps and the chits support files for examples.

Conclusion

In Part 2 I will talk about getting this to work on a lithium application, in Part 3 I will talk about using cucumber to drive selenium tests and in Part 4 I will talk about putting it under continuous integration with Hudson.

1. All we really need is a headless firefox executable for linux. There is one, we just haven't installed it on our vm yet. But the other big benefit of using webrat over selenium is that it has a much lighter resource footprint. ↩
2. Please note that the first line of this file (from the chits app on github) uses "re". I'm not sure if this is a ruby thing, but I had to change it to "require" to get it to work:

   require File.expand_path(File.join(File.dirname(__FILE__), "..", "support", "paths" ↩

For testing ruby (and ruby on rails) has cucumber and rspec, which seem to be pretty close to de-facto standards.

Note: you can't actually have an rspec-like app with python – this is just the way the language doesn't work like ruby. This is a good article about Ruby vs. Python which mentions

For Python/Django the situation seems to be a little more scattered, but I'm intending to use the following equivalents:

• pyccuracy (there is also lettuce) for cucumber
• nose (extension of python's unittest) for (a sort of replacement for) rspec
• The django-nose plugin for using nose when running django tests
• The nose-xcover for using nose with hudson

I'm also going to use hudson for continuous integration (here is a good description about how to get it running with django and nose).

By the way I installed pip for installing python packages and I've started using it instead of easy_install.