Continuous Integration¶
Installation¶
In order to use Hydra on a continuous build server you need a floating key license (see License for details). Floating keys don't require activation and they can be used on any machine, as long as the total number of concurrent users is below the maximum allowed number. To setup a floating key, you need to create a license file, usually ~/.triplequote/hydra.license
:
floating-key=XXXXX-XXXXX-XXXXX-XXXXX-XXXXX
You can choose a different file, in which case you need to set hydra.license.store=<path>
, for instance when running sbt:
sbt -Dhydra.license.store=/mount/vault/hydra.license
In the rest of this section we discuss how to optimize compilation times on the Continuous Integration (CI) server.
Automatic tuning¶
If you have read the section about unbalanced workers, you should know that a partition file is automatically generated by Hydra when the workload of the workers is uneven. However, when running a build on the CI, you often start with a fresh checkout, and hence files generated by a build are not available to the next build. If your CI allows it, you should persist the .hydra
folder, as it contains the aforementioned generated partition file. Doing so will make your CI compile times optimal and, as a bonus, you will also find data about build times over time.
Changing the location of .hydra¶
Sometimes it's inconvenient to cache the default location of the .hydra
directory, especially if the CI server starts by cloning a Git repository. You can change the location of this directory using hydraBaseDirectory in ThisBuild
. For instance, if your CI sets an environment variable, your build may contain:
hydraBaseDirectory in ThisBuild := { if (sys.env.isDefinedAt("CI")) file("/mount/hydra/MyProject") else (Global / hydraBaseDirectory).value }
If persisting the .hydra
folder is not possible, you can still optimize the CI build compile times, but that will require manual intervention. Let's see how in the next section.
Manual tuning¶
By using the "explicit"
source partition strategy you can instruct Hydra to use a pre-defined partition file. If you have seen the message about unbalanced workers when compiling locally, then you can use the generated partition file to make sure your CI build will be as fast as your local one!
Let's assume that your project is named portfolio
and you have seen the message about unbalanced workers only when compiling its main sources (i.e., the source files under src/main
). Start by copying the partition file that was generated by Hydra under .hydra/<build-tool>/portfolio/compile
and paste the partition file under the src/main
directory of the portfolio
project (mind that you will have to commit the partition file copied in src/main
as part of your changes).
Next, update your build and set the hydraSourcePartitioner
settings to "explicit"
for the portfolio
project:
lazy val portfolio = (project in file("portfolio")) .settings( hydraSourcePartitioner in Compile := { if (System.getenv("CI").nonEmpty) "explicit" else (hydraSourcePartitioner in Compile).value // get the default value, i.e., "auto" }, // your project's settings )
Note that the above code snippet assumes that a
CI
environment variable is set on the CI server used to build. If you are using Jenkins, Travis, or Drone, there is likely an existing environment variable that can be used to detect if a build is being executed on the CI.Note that if you want to use the
"explicit"
partition for the project's test sources (i.e., the source files undersrc/test
) then you should copy the partition file under.hydra/<build-tool>/portfolio/test
into thesrc/test
directory, and usehydraSourcePartitioner in Test
instead ofhydraSourcePartitioner in Compile
when editing your build file.
Now, when compiling on the CI, the main sources of the portfolio
project are going to be compiled using the partition file you just copied into the src/main
directory. While, test sources will keep being compiled with the automatic partition strategy.
With the above solution, you are going to have the best possible speedups both locally and on the CI. The only inconvenient is that from time to time you will need to update the explicit partition file used by the CI server. The good news is that Hydra will tell you when the in-use explicit partition file is ineffective. If you see the message:
[info] [portfolio] Hydra workers are unbalanced. Consider using the default "auto" source partition strategy instead for optimal workers' workload.
simply replace the current explicit partition file with a new one, and your CI compilation times will be once again optimal!