wiki:Hudson/FAQ

Jenkins/Hudson FAQ

What's with this Jenkins/Hudson name confusion?

  • In January, 2011 the Hudson community split into two separate projects: Hudson and Jenkins. The motivations and events leading to the split are not directly pertinent to this FAQ, and it is sufficient to simply note that we have elected to follow the Jenkins project. That said, the Jenkins project carries with it a significant legacy use of the name "Hudson", as does much of our infrastructure (e.g. machine names, entity accounts, slave configurations). While we strive to accurately reflect where you are likely to see the "Hudson" and "Jenkins" names, this is a moving target and errors are likely to occur. For the casual reader it is sufficient to note that within our system, the "Hudson" and "Jenkins" names can be treated as synonymous, and that both refer to the Jenkins project.

How do I log in to Jenkins?

  • Users may log in to the main SRN server using their Kerberos credentials. Login rights to the SON mirror are restricted to Jenkins administrators.

How do I start a Jenkins job?

  • Note: Jobs can only be run from the main Jenkins server (on the SRN). No jobs can be run on or through the SON mirror.
  • After logging in to the SRN server, the main page shows an icon next to each job summary. This icon shows a clock with a green triangle. Clicking on this icon will launch the corresponding Jenkins job. Alternatively, on every job's main page there is a "Build Now" link in the upper left side of the page.

Working with Slaves

How do I restart Jenkins nodes that have hung?

  • The first thing to try is to navigate to that node's page in Jenkins and select the "Disconnect" link on the left-hand side of the page (Note that the link will not be available to most Jenkins users). The node should automatically reconnect within a minute or so.
  • If disconnecting through the Jenkins UI does not work, then:
    1. Log in to the node as either the hudson user, or as a user that has sudo privileges.
    2. Kill the Jenkins java process
    3. On the host status page in Jenkins, click the button in the upper-right corner that says "this node is back online". That should cause Jenkins to start using the node again. If that fails, you can always click the "Disconnect" link to cause Jenkins to kill the slave process, then there should be a "Reconnect" button that appears below the node name/description.

Why is my slave dying immediately upon startup?

  • The log at http://hudson.sandia.gov/computer/<slave-name>/log may help with debugging.
  • The user running the process (typically hudson or jenkins) may not have sufficient permissions on the directory containing the .jar files.
  • A label verifier may be failing, barring the slave from starting. Slave labels are verified as a pre-condition to slave connection. For example, a slave labeled Python2.6 without a working Python 2.6 installation may die.

How do I configure a Linux or OSX slave?

  • The easiest way to set up a slave for Linux or OSX (or any other platform that supports access through SSH) is by using the SSH Slaves plugin.
    • In the node (slave) configuration, select "Launch slave agents on Unix machines via SSH" for the "Launch method"
  • To enable "passwordless" access for Jenkins to the slave:
    1. Add ".ssh/id_rsa" to the "Private Key File" field (Available by pressing the "Advanced" button visible after selecting the SSH launch method.
    2. Add the Jenkins public key (available  here) to "~/.ssh/authorized_keys" for the user that Jenkins will run as on the new slave.

How do I configure a Windows slave?

  • For starters, refer to the  Jenkins Wiki.
    • This installs the Jenkins slave as a Windows service that runs as the Local SYSTEM user. While normally not a problem, this can really cause problems if you need to do anything custom with your Subversion configuration (e.g. accepting self-signed or invalid SSL certificates). Additionally, running Jenkins as a privileged user presents a security risk. To fix this, continue with the following steps to run Jenkins as a non-privileged local user.
    • See also the Vista Installation FAQ
  • There is a Jenkins entity account. Where possible, please have Jenkins run as the Entity. When using an Entity is not possible, create a local hudson user by going to Start -> Settings -> Control Panel -> Administrative Tools -> Computer Management -> Local Users and Groups
  • Edit the permissions for the Jenkins slave workspace (e.g. C:\Hudson) and grant Full Control to the hudson user. Be sure to click the Advanced button and select the check box to "Replace permission entries on all child objects with entries that apply to child objects."
  • Edit the properties for the Hudson Slave service (in Start -> Settings -> Control Panel -> Administrative Tools -> Services). On the "Log On" tab, switch the "Log on as:" from the Local System account to the Jenkins account.
  • To fix Windows services that have broken due to moving the master Jenkins root URL
    • Go to the C:\Hudson or equivalent home directory
    • Edit the hudson-slave.xml configuration file.
    • Look for the -jnlpUrl argument in the <service>...<argument> element. It should match "HUDSON_ROOT/computer/<node>/slave-agent.jnlp" (currently, http://jenkins.sandia.gov:8080/computer/<node>/slave-agent.jnlp).
    • Go to Administrative Tools -> Services
    • Re-start the "Hudson Slave" (or "Jenkins Slave") service
  • If the service has failed to start
    • Check that master Jenkins URL is correct in C:\Hudson\hudson-slave.xml or equivalent
    • If this is correct, go to Administrative Tools -> Services
    • Select the "Hudson Slave" service, right click and open "Properties"
    • Go to the "Log on" tab, and re-enter the user name and password (some Domain managed systems may delete local user privileges to log on as a service; these rights will be automatically re-grated when you re-enter the login information)
    • If this still doesn't work, the slave.jar may be out of date
      • See below for instructions on upgrading the slave.jar.
    • If this still doesn't work, you may have to delete and re-install the service (see the main wiki FAQ)
  • If the node is "offline because it uses old slave.jar", use the following procedure to upgrade the slave.jar
    • Go to Administrative Tools -> Services
    • Stop the "Hudson Slave" (or "Jenkins Slave") service
    • Download an updated  slave.jar and replace the version in C:\hudson.
    • Start the "Hudson Slave" (or "Jenkins Slave") service

How do I get Subversion working on a new Slave?

  • Jenkins uses SVNKit as it's embedded Subversion client. The Subversion configuration is stored in user's home directory.
    • On Linux: ~/.subversion
    • On Windows: %HOMEPATH%\Application Data\Subversion. For most computers, this will be C:\Documents and Settings\hudson\Application Data\Subversion.
      • If you run the Hudson Slave service as the Local SYSTEM user, the Subversion configuration will probably be stored in C:\WINDOWS\system32\config\systemprofile\Application Data\Subversion.
  • If you have a repository that needs authentication or has a "broken" SSL certificate (i.e. expired or self-signed), check out the repository on another machine where you can log in with a working subversion client. Then, copy the relevant files (i.e. from ~/.subversion/auth/svn.simple/ or ~/.subversion/auth/svn.ssl.server/) into the corresponding location in the Jenkins user's configuration.
  • Also, don't forget to correctly configure Subversion for the SRN proxy. The standard servers fragment we use on the SRN is:
    [global]
    http-proxy-exceptions = *.sandia.gov, localhost
    http-proxy-host = <full proxy hostname here>
    http-proxy-port = 80
    
    • Note: The http-proxy-port directive is required in the servers file.
    • IMPORTANT: Jenkins uses SVNKit for interacting with Subversion. There is a weakness in how SVNKit parses your servers file: each section heading may only appear ONCE. If you paste the above into your servers file, be sure to go through the file and comment out any other instances of these section headings.

How do I set up Kerberos-authenticated access to a slave

  • Jenkins's SSH Slaves plugin is not Kerberos-aware. To setup a build slave on a host that only supports Kerberos authentication, you will need to connect to the machine using the system ssh client.
    • In the node configuration, choose the Launch Method, "Launch slave via execution of command on the Master."
    • The Launch command should be something similar to:
      /home/hudson/scripts/launch_krb5_slave.sh {username} {host} {dir} [{java_path}]
      
      where {username} is the remote username that the Jenkins slave should run as, {host} is the remote host name, and {dir} is the directory where you want Jenkins to store the slave.jar file.
    • The launch_krb5_slave.sh script will automatically initialize Kerberos and copy the current slave.jar to the remote slave before launching the slave Java process.
  • As with all new slaves, you need to make sure that the local Subversion configurations are in place (see above).
  • You will need to create and maintain a "{username}.keytab" file in the Jenkins Master $HOME directory.
  • Before this will work, you will need to interactively ssh from the master to the slave in order to accept the remote host's host key.
  • If the default Java on the remote machine is < v1.5, you may provide an alternative path to a Java >= 1.5 as a fourth argument to launch_krb5_slave.sh.