Continuous Integration

Installation

In order to use Hydra on a continuous build server you need a floating key license (see License for details). Floating keys don't require activation and they can be used on any machine, as long as the total number of concurrent users is below the maximum allowed number. To setup a floating key, you need to create a license file, usually ~/.triplequote/hydra.license:

floating-key=XXXXX-XXXXX-XXXXX-XXXXX-XXXXX

You can choose a different file, in which case you need to set hydra.license.store=<path>, for instance when running sbt:

sbt -Dhydra.license.store=/mount/vault/hydra.license

In the rest of this section we discuss how to optimize compilation times on the Continuous Integration (CI) server.

Automatic tuning

If you have read the section about unbalanced workers, you should know that a partition file is automatically generated by Hydra when the workload of the workers is uneven. However, when running a build on the CI, you often start with a fresh checkout, and hence files generated by a build are not available to the next build. If your CI allows it, you should persist the .hydra folder, as it contains the aforementioned generated partition file. Doing so will make your CI compile times optimal and, as a bonus, you will also find data about build times over time.

Changing the location of .hydra

Sometimes it's inconvenient to cache the default location of the .hydra directory, especially if the CI server starts by cloning a Git repository. You can change the location of this directory using hydraBaseDirectory in ThisBuild. For instance, if your CI sets an environment variable, your build may contain:

hydraBaseDirectory in ThisBuild := {
  if (sys.env.isDefinedAt("CI"))
    file("/mount/hydra/MyProject")
  else
    (Global / hydraBaseDirectory).value
}

If persisting the .hydra folder is not possible, you can still optimize the CI build compile times, but that will require manual intervention. Let's see how in the next section.

Manual tuning

By using the "explicit" source partition strategy you can instruct Hydra to use a pre-defined partition file. If you have seen the message about unbalanced workers when compiling locally, then you can use the generated partition file to make sure your CI build will be as fast as your local one!

Let's assume that your project is named portfolio and you have seen the message about unbalanced workers only when compiling its main sources (i.e., the source files under src/main). Start by copying the partition file that was generated by Hydra under .hydra/<build-tool>/portfolio/compile and paste the partition file under the src/main directory of the portfolio project (mind that you will have to commit the partition file copied in src/main as part of your changes).

Next, update your build and set the hydraSourcePartitioner settings to "explicit" for the portfolio project:

lazy val portfolio = (project in file("portfolio"))
  .settings(
    hydraSourcePartitioner in Compile := {
       if (System.getenv("CI").nonEmpty) "explicit"
       else (hydraSourcePartitioner in Compile).value // get the default value, i.e., "auto"
    },
    // your project's settings
  )

Note that the above code snippet assumes that a CI environment variable is set on the CI server used to build. If you are using Jenkins, Travis, or Drone, there is likely an existing environment variable that can be used to detect if a build is being executed on the CI.

Note that if you want to use the "explicit" partition for the project's test sources (i.e., the source files under src/test) then you should copy the partition file under .hydra/<build-tool>/portfolio/test into the src/test directory, and use hydraSourcePartitioner in Test instead of hydraSourcePartitioner in Compile when editing your build file.

Now, when compiling on the CI, the main sources of the portfolio project are going to be compiled using the partition file you just copied into the src/main directory. While, test sources will keep being compiled with the automatic partition strategy.

With the above solution, you are going to have the best possible speedups both locally and on the CI. The only inconvenient is that from time to time you will need to update the explicit partition file used by the CI server. The good news is that Hydra will tell you when the in-use explicit partition file is ineffective. If you see the message:

[info] [portfolio] Hydra workers are unbalanced. Consider using the default "auto" source partition strategy instead for optimal workers' workload.

simply replace the current explicit partition file with a new one, and your CI compilation times will be once again optimal!