studios.thoughtworks.com - Continuous Integration: what should I automate? Comments

Comment on 'Continuous Integration: what should I automate?' by sohbet

2008-04-27T19:58:59Z

Great article! Maybe when you get a chance you could go into more depth on various database automation tools and practices that are in use, such as schema generation and database migrations.

Comment on 'Continuous Integration: what should I automate?' by Eric

2008-04-15T21:00:04Z

Tagging really isn't required in CVS either. Just do your checkouts from a time stamp (roughly when the trigger fired). Bam. You have a branch and a date-time stamp. It's an absolute set of source code and harder for people to sneak around and modify than the contents at a tag. You add zero seconds to your build process, make it easier to repeat and have all the auditability you want.

Comment on 'Continuous Integration: what should I automate?' by Jez Humble

2007-08-01T02:13:18Z

Hi Bob. Thanks for your comment.

The 10 minute case I discussed in the article is not too difficult to do with Java / .NET, even on quite large systems. However as you point out, things are harder when you're trying to do continous integration on very large C++ projects.

So going back to first principles, the goal is that after a developer does a check-in, they should get feedback on whether they have broken anything important within 10 minutes of doing so. This feedback need not (and indeed it cannot) be definitive - but it should at least include the unit tests (http://en.wikipedia.org/wiki/Unit_testing) for the module concerned in isolation.

So what I would do in your project is have a single CruiseControl project for each module of your software. This would compile just the module, and run the relevant unit tests on it. Once you have this completed for each module, you can move on to the next stage, where you deal with changes that break the external interfaces of your frameworks / libraries.

This is where you need to look at pipelining your builds such that the results of successful builds of upstream (lower level) stages are preserved and available for use by downstream (higher level) stages. In this approach, a successful build of a given module then triggers builds of the modules that depend on it. There are many ways to do this - we'll be covering them in future articles here on the Studios blog - but one possibility is to have your build script publish binaries to (say) a git repository upon a successful build (more on how to do this in our next CCE blog entry).

The dependent stages in your pipeline (say your server code) could check this repository and trigger builds based on that as well as on check-ins to their source code. That way breakages in upstream modules don't affect downstream ones - since only successful builds trigger the next stage in the pipeline - but new versions of upstream modules/libraries automatically trigger builds of the dependent modules. Interface changes may then cause these downstream modules to break - but that is the signal that the upstream dependency built successfully and passed its unit tests, and that it is now time for the developers on the dependent modules to update them.

Clearly with this approach the whole pipeline will take a while to run. However, in this scenario you can start getting feedback without having to wait for the whole codebase to rebuild. You would probably need a final stage to assemble together and deploy the application using the last known good versions of all the modules (which shouldn't be too hard if you published everything into git), and run your full test suite. You could have this scheduled nightly.

Hopefully this addresses some of your concerns. Please get in touch with us for more information - you can email me at jez at thoughtworks dot com.

Comment on 'Continuous Integration: what should I automate?' by billie

2007-07-17T18:27:12Z

Great article! Maybe when you get a chance you could go into more depth on various database automation tools and practices that are in use, such as schema generation and database migrations.

Comment on 'Continuous Integration: what should I automate?' by Bob Hanke

2007-07-14T18:50:23Z

I have been reading these articles about using CI. However I am in the same boat as Anton. Our Server/Client product takes 3+ hours to build on a dedicated build machine with no tests being run. The process involves retrieving source code, building all libraries, servers, desktopapplications, and install software. Our code is mainly C++ which does take longer to build then Java. A lot of our servers and desktop exe's have many dependencies on our internal libraries. So if a major change were to take place to a low level library that almost every other library, server, or dekstop app uses wouldn't that mean a rebuild of the entire system? In addition to that I don't see how I could possibly reduce the entire build to 10 min. Most modules will build independently in 10 min or less. Am I missing something here in understanding how CI is used? I can see it being used if the desktop apps are the ones being modifed on a regular basis. Those are the end point of the dependency chain, are quick to build and can kick off tests. But if we change something that is further up the chain how would we handle that. Also on those lines how do you handle a change to an API of a library that other modules use? You assumptions of CI assume that either what I check in will not break anybody elses module or everyone checks in the updated code at the same time. Neither of these are true. Usally it takes time for the lib to get modified and rebuilt. Then each owner of the dependent modules changed code, unit builds, tests and commits. Thus working down the chain until all modules have had a chance to do this. Then we get a sucessfull build. Please give some guidence. Thanks Bob

Comment on 'Continuous Integration: what should I automate?' by Paul Duvall

2007-06-28T15:18:48Z

"A green light from your continuous integration process should mean that your software is *working*" - Right on, Jez. Nice post. This is a key differentiator of CI - Paul

Comment on 'Continuous Integration: what should I automate?' by Jez Humble

2007-06-28T14:27:00Z

Hi Anton. Thanks for your comments. I am guessing your 3 hour test suite runs in a production-like environment and tests functionality end-to-end. This suite is probably a combination of acceptance tests and integration tests. I would run this as a second cruise project, as described in point five above. The way you get a ten minute unit test suite is by testing the behaviour of your business logic in isolation from your infrastructure. So you might have a class that has a particular responsibility, and you test that, stubbing or mocking out its dependencies on infrastructure like messaging busses or CORBA infrastructure or databases or whatever. This means the tests run very fast, and test *only* the business logic. This kind of testing, known as unit testing, was the main driver behind dependency injection frameworks like Spring and PicoContainer. This kind of design also has the advantage that it isolates your business logic from your infrastructure. This is the way we write large systems here at TW - and yes, we regularly write large, distributed applications in this way. Our unit test suites do sometimes take longer than ten minutes to build, especially with the largest systems we deal with (which sometimes have tens of thousands of unit tests) - but these can generally be broken down into subsystems in any case. Thanks, Jez Humble Delivery Manager, CruiseControl Enterprise

Comment on 'Continuous Integration: what should I automate?' by Anton

2007-06-27T08:32:48Z

Hi! A good initial test suite should cover as much of your codebase as possible, run as fast as possible (your main build should never take more than ten minutes), and not depend on infrastructure like databases or messaging busses. How about "big" distributed applications? E.g., the build process for our system takes more then 3 hours. I don't believe that there can be any reasonable test for a such codebase that takes 10 minutes. In general, have you experience or knowledge about CI and unit testing for distributed applications? Namely, CORBA based ones.