Comments you submit will be routed for moderation. If you have an account, please log in first.
Modify

Ticket #3798 (closed defect: fixed)

Opened 10 years ago

Last modified 10 years ago

Resolve reliability issues with Hudson checkouts

Reported by: wehart Owned by: jdsiiro
Priority: normal Milestone:
Component: Hudson Version:
Keywords: Cc: jdsiiro

Description

The following build failure illustrates a badness in Hudson:

 http://hudson.sandia.gov/hudson/job/acro_trunk_linux64/lastFailedBuild/console

The checkout failed, but hudson continued to attempt a build.

Attachments

Change History

comment:1 Changed 10 years ago by jdsiiro

The checkout failure is almost certainly due to s.s.g restarting the Apache server (which it does frequently). There is not much we can do about that...

I have moved the Hudson checkout process to use svn update. The build process then copies the updated checkout to a temporary directory before building (that way the svn update should never fail due to conflicts or preexisting files). The update should run significantly faster than the full checkout, so we will be less likely to be in the middle of a svn operation when the Apache server restarts. Note: this does not fix the cause -- it only lessens the symptoms. It does, however, drop the chances of a bizarre build failure, as the working copy is being updated instead of a clean checkout, so even if the svn transaction fails, the working copy still has the last version in place.

Another alternative is to have the driver script run svn update until it returns without error. While this would fix the problem of broken checkouts, it would foul up Hudson's revision tracking (i.e. the "changes" section of the build report), as Hudson's original failed update would not have returned the same version numbers as the versions that were actually configured and built.

Maybe the answer is that if the build detected that the Hudson-issued update failed, it would immediately schedule through Hudson another build?

comment:2 Changed 10 years ago by jdsiiro

  • Status changed from new to closed
  • Resolution set to fixed

This has been fixed through an upstream Hudson enhancement: beginning in  1.313, Hudson supports automatically retrying failed SCM checkouts/updates.

View

Add a comment

Modify Ticket

Change Properties
<Author field>
Action
as closed
The resolution will be deleted. Next status will be 'reopened'
Author


E-mail address and user name can be saved in the Preferences.

 
Note: See TracTickets for help on using tickets.