Jlink custom jre with scala and sbt

Written on April 26, 2021

In this post will learn how to pack scala play framework application with stripped-down custom JRE using sbt-native-packager with sbt-jlink-plugin.

We will use OpenJDK 11 as the base for custom JRE. I decided to write a blog-post because I struggled with jlink for myself and there’s not so much information on that topic - you can find a lot of articles for java, but those with scala and sbt are less popular. Let’s fix that!

Java jigsaw image

Motivation

JDK 9 ships with the jlink tool which allows us to create stripped-down custom JRE that contains only specific JDK modules. We will find out how to build a play framework application where the JRE size is between 40Mb to 80MB depending upon what dependencies used by the app. Small deployment artifact size is crucial in the age of the containers.

Plan

  1. Create play framework app from the template or take an existing one
  2. Add sbt-native packager plugin and activate JLinkPlugin
  3. Customize JLinkPlugin params
  4. Build custom JRE
  5. Optionally add docker image build
  6. PROFIT!

Play App

Let’s create a play framework app from the template:

  1. Install sbt if you don’t have it
  2. Run sbt new playframework/play-scala-seed.g8.

At this point we have standard files and directories structure. Structure Let’s go straight right off the bat and add sbt-native packager to the project.

Add this line in project/plugins.sbt:

addSbtPlugin("com.typesafe.sbt" % "sbt-native-packager" % "1.8.1")

And add JlinkPlugin to plugins list  in build.sbt:

lazy val root = (project in file(".")).enablePlugins(PlayScala, JlinkPlugin)

Jdeps

At this point, we can run sbt stage to build out app but you will get an error:

[error] Exception in thread "main" java.lang.module.FindException: Module paranamer not found, required by com.fasterxml.jackson.module.paranamer
[error]         at java.base/java.lang.module.Resolver.findFail(Resolver.java:877)
[error]         at java.base/java.lang.module.Resolver.resolve(Resolver.java:191)
[error]         at java.base/java.lang.module.Resolver.resolve(Resolver.java:140)
[error]         at java.base/java.lang.module.Configuration.resolve(Configuration.java:422)
[error]         at java.base/java.lang.module.Configuration.resolve(Configuration.java:256)
[error]         at jdk.jdeps/com.sun.tools.jdeps.JdepsConfiguration.<init>(JdepsConfiguration.java:117)
[error]         at jdk.jdeps/com.sun.tools.jdeps.JdepsConfiguration$Builder.build(JdepsConfiguration.java:563)
[error]         at jdk.jdeps/com.sun.tools.jdeps.JdepsTask.buildConfig(JdepsTask.java:589)
[error]         at jdk.jdeps/com.sun.tools.jdeps.JdepsTask.run(JdepsTask.java:543)
[error]         at jdk.jdeps/com.sun.tools.jdeps.JdepsTask.run(JdepsTask.java:519)
[error]         at jdk.jdeps/com.sun.tools.jdeps.Main.run(Main.java:64)
[error]         at jdk.jdeps/com.sun.tools.jdeps.Main$JDepsToolProvider.run(Main.java:73)
[error]         at java.base/java.util.spi.ToolProvider.run(ToolProvider.java:137)
[error]         at ru.eldis.toollauncher.ToolLauncher.runTool(ToolLauncher.java:68)
[error]         at ru.eldis.toollauncher.ToolLauncher.lambda$main$1(ToolLauncher.java:33)
[error]         at ru.eldis.toollauncher.ToolLauncher.main(ToolLauncher.java:48)

This case is described on the page of sbt-native-packager jlink plugin:

This is often caused by depending on automatic modules. In the example above, com.faterxml.jackson.module.paranamer is an explicit module (as in, it is a JAR with a module descriptor) that defines a dependency on the paranamer module. However, there is no explicit paranamer module - instead, Jackson expects us to use the paranamer JAR file as an automatic module. To do this, the JAR has to be on the module path. At the moment JlinkPlugin does not put it there automatically, so we have to do that ourselves:

So we must add this block to build sbt:

jlinkModulePath := {
  fullClasspath
    .in(jlinkBuildImage)
    .value
    .filter { item =>
      item.get(moduleID.key).exists { modId =>
        modId.name == "paranamer"
      }
    }
    .map(_.data)
}

I also recommend to add 

Global / onChangedBuildSource := ReloadOnSourceChanges

to make sbt reload every time we change something in build.sbt. If we’ll run sbt stage one more time we would get this output:

[error] Dependee packages not found in classpath. You can use jlinkIgnoreMissingDependency to silence these.
[error]   ch.qos.logback.classic -> javax.servlet.http
[error]   ch.qos.logback.classic.boolex -> groovy.lang
[error]   ch.qos.logback.classic.boolex -> org.codehaus.groovy.control
[error]   ch.qos.logback.classic.boolex -> org.codehaus.groovy.reflection
[error]   ch.qos.logback.classic.boolex -> org.codehaus.groovy.runtime
[error]   ch.qos.logback.classic.boolex -> org.codehaus.groovy.runtime.callsite
[error]   ch.qos.logback.classic.boolex -> org.codehaus.groovy.runtime.typehandling
[error]   ch.qos.logback.classic.gaffer -> groovy.lang
[error]   ch.qos.logback.classic.gaffer -> org.codehaus.groovy.control
[error]   ch.qos.logback.classic.gaffer -> org.codehaus.groovy.control.customizers
[error]   ch.qos.logback.classic.gaffer -> org.codehaus.groovy.reflection
[error]   ch.qos.logback.classic.gaffer -> org.codehaus.groovy.runtime
[error]   ch.qos.logback.classic.gaffer -> org.codehaus.groovy.runtime.callsite
[error]   ch.qos.logback.classic.gaffer -> org.codehaus.groovy.runtime.typehandling
[error]   ch.qos.logback.classic.gaffer -> org.codehaus.groovy.runtime.wrappers
[error]   ch.qos.logback.classic.gaffer -> org.codehaus.groovy.transform
[error]   ch.qos.logback.classic.helpers -> javax.servlet
[error]   ch.qos.logback.classic.helpers -> javax.servlet.http
[error]   ch.qos.logback.classic.selector.servlet -> javax.servlet
[error]   ch.qos.logback.classic.servlet -> javax.servlet
[error]   ch.qos.logback.core.boolex -> org.codehaus.janino
[error]   ch.qos.logback.core.joran.conditional -> org.codehaus.commons.compiler
[error]   ch.qos.logback.core.joran.conditional -> org.codehaus.janino
[error]   ch.qos.logback.core.net -> javax.mail
[error]   ch.qos.logback.core.net -> javax.mail.internet
[error]   ch.qos.logback.core.status -> javax.servlet
[error]   ch.qos.logback.core.status -> javax.servlet.http
[error]   io.jsonwebtoken.impl -> android.util
[error]   io.jsonwebtoken.impl.crypto -> org.bouncycastle.jce
[error]   io.jsonwebtoken.impl.crypto -> org.bouncycastle.jce.spec
[error]   javax.transaction -> javax.enterprise.context
[error]   javax.transaction -> javax.enterprise.util
[error]   javax.transaction -> javax.interceptor
[error]   org.joda.time -> org.joda.convert
[error]   org.joda.time.base -> org.joda.convert

We should add this mapping to jlink ignore missing dependency list:

jlinkIgnoreMissingDependency := JlinkIgnore.only(
  "ch.qos.logback.classic" -> "javax.servlet.http",
  "ch.qos.logback.classic.boolex" -> "groovy.lang",
  "ch.qos.logback.classic.boolex" -> "org.codehaus.groovy.control",
  "ch.qos.logback.classic.boolex" -> "org.codehaus.groovy.reflection",
  "ch.qos.logback.classic.boolex" -> "org.codehaus.groovy.runtime",
  "ch.qos.logback.classic.boolex" -> "org.codehaus.groovy.runtime.callsite",
  "ch.qos.logback.classic.boolex" -> "org.codehaus.groovy.runtime.typehandling",
  "ch.qos.logback.classic.gaffer" -> "groovy.lang",
  "ch.qos.logback.classic.gaffer" -> "org.codehaus.groovy.control",
  "ch.qos.logback.classic.gaffer" -> "org.codehaus.groovy.control.customizers",
  "ch.qos.logback.classic.gaffer" -> "org.codehaus.groovy.reflection",
  "ch.qos.logback.classic.gaffer" -> "org.codehaus.groovy.runtime",
  "ch.qos.logback.classic.gaffer" -> "org.codehaus.groovy.runtime.callsite",
  "ch.qos.logback.classic.gaffer" -> "org.codehaus.groovy.runtime.typehandling",
  "ch.qos.logback.classic.gaffer" -> "org.codehaus.groovy.runtime.wrappers",
  "ch.qos.logback.classic.gaffer" -> "org.codehaus.groovy.transform",
  "ch.qos.logback.classic.helpers" -> "javax.servlet",
  "ch.qos.logback.classic.helpers" -> "javax.servlet.http",
  "ch.qos.logback.classic.selector.servlet" -> "javax.servlet",
  "ch.qos.logback.classic.servlet" -> "javax.servlet",
  "ch.qos.logback.core.boolex" -> "org.codehaus.janino",
  "ch.qos.logback.core.joran.conditional" -> "org.codehaus.commons.compiler",
  "ch.qos.logback.core.joran.conditional" -> "org.codehaus.janino",
  "ch.qos.logback.core.net" -> "javax.mail",
  "ch.qos.logback.core.net" -> "javax.mail.internet",
  "ch.qos.logback.core.status" -> "javax.servlet",
  "ch.qos.logback.core.status" -> "javax.servlet.http",
  "io.jsonwebtoken.impl" -> "android.util",
  "io.jsonwebtoken.impl.crypto" -> "org.bouncycastle.jce",
  "io.jsonwebtoken.impl.crypto" -> "org.bouncycastle.jce.spec",
  "javax.transaction" -> "javax.enterprise.context",
  "javax.transaction" -> "javax.enterprise.util",
  "javax.transaction" -> "javax.interceptor",
  "org.joda.time" -> "org.joda.convert",
  "org.joda.time.base" -> "org.joda.convert",
)

After we run sbt stage again we will get packaged application with custom jre in target folder

Jre target

JRE directory size at this point is 80M

stage % du -sh *
 28K    bin
 16K    conf
 80M    jre
 40M    lib
2.6M    share

Let’s try to make it smaller. Add params to jlink build:

jlinkOptions ++= Seq(
  "--no-header-files",
  "--no-man-pages",
  "--compress=2"
)

Results after stage :

stage % du -sh *
 28K    bin
 16K    conf
 48M    jre
 40M    lib
2.6M    share

Much better! Let’s run our application and test if it works. Run sbt runProd which runs both stage and executes app binary using our custom JRE.

[test-jlink] $ runProd -Dplay.http.secret.key=changeme_for_prod_mode_to_work

and we got error:

Oops, cannot start the server.
java.lang.ExceptionInInitializerError
        at akka.dispatch.AbstractNodeQueue.<clinit>(AbstractNodeQueue.java:181)
        at akka.actor.LightArrayRevolverScheduler.<init>(LightArrayRevolverScheduler.scala:190)
        ...
        at play.core.server.ProdServerStart$.main(ProdServerStart.scala:29)
        at play.core.server.ProdServerStart.main(ProdServerStart.scala)
Caused by: java.lang.ExceptionInInitializerError
        at akka.util.Unsafe.<clinit>(Unsafe.java:52)
        at akka.dispatch.AbstractNodeQueue.<clinit>(AbstractNodeQueue.java:179)
        ... 42 more
Caused by: java.lang.NoClassDefFoundError: sun/misc/Unsafe
        at akka.util.Unsafe.<clinit>(Unsafe.java:27)
        ... 43 more
Caused by: java.lang.ClassNotFoundException: sun.misc.Unsafe
        at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:581)
        at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:178)
        at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:522)
        ... 44 more

To make our app work we must add several additional modules to have classes on the classpath. Reason for this that sun.misc.Unsafe is in jdk.unsopported module. Let’s add it:

jlinkModules ++= Seq(
  "jdk.crypto.ec",
  "jdk.unsupported"
)

and re-run runProd. Structure

Yay! Let’s test it in the browser.

Structure

Nice! We have a standard play framework app running on custom JRE build from OpenJDK.

So far so good! New problems arise when we start to add dependencies. Here are some examples:

[error] Exception in thread "main" java.lang.module.FindException: Module java.activation not found, required by java.xml.bind
[error]         at java.base/java.lang.module.Resolver.findFail(Resolver.java:877)
[error]         at java.base/java.lang.module.Resolver.resolve(Resolver.java:191)
[error]         at java.base/java.lang.module.Resolver.resolve(Resolver.java:140)
[error]         at java.base/java.lang.module.Configuration.resolve(Configuration.java:422)
[error]         at java.base/java.lang.module.Configuration.resolve(Configuration.java:256)
[error]         at jdk.jdeps/com.sun.tools.jdeps.JdepsConfiguration.<init>(JdepsConfiguration.java:117)
[error]         at jdk.jdeps/com.sun.tools.jdeps.JdepsConfiguration$Builder.build(JdepsConfiguration.java:563)
[error]         at jdk.jdeps/com.sun.tools.jdeps.JdepsTask.buildConfig(JdepsTask.java:589)
[error]         at jdk.jdeps/com.sun.tools.jdeps.JdepsTask.run(JdepsTask.java:543)
[error]         at jdk.jdeps/com.sun.tools.jdeps.JdepsTask.run(JdepsTask.java:519)
[error]         at jdk.jdeps/com.sun.tools.jdeps.Main.run(Main.java:64)
[error]         at jdk.jdeps/com.sun.tools.jdeps.Main$JDepsToolProvider.run(Main.java:73)
[error]         at java.base/java.util.spi.ToolProvider.run(ToolProvider.java:137)
[error]         at ru.eldis.toollauncher.ToolLauncher.runTool(ToolLauncher.java:68)
[error]         at ru.eldis.toollauncher.ToolLauncher.lambda$main$1(ToolLauncher.java:33)
[error]         at ru.eldis.toollauncher.ToolLauncher.main(ToolLauncher.java:48)

In this situation we must add javax.activation jar to module path:

jlinkModulePath := {
      fullClasspath
        .in(jlinkBuildImage)
        .value
        .filter { item =>
          item.data.toString().contains("javax.activation")
        }
        .map(_.data)
    }

Another example with graalvm.js dependency:

"org.graalvm.js"      % "js" % "21.0.0.2"

[error] Exception in thread "main" java.lang.module.FindException: Module com.ibm.icu not found, required by org.graalvm.js
[error]         at java.base/java.lang.module.Resolver.findFail(Resolver.java:877)
[error]         at java.base/java.lang.module.Resolver.resolve(Resolver.java:191)
[error]         at java.base/java.lang.module.Resolver.resolve(Resolver.java:140)
[error]         at java.base/java.lang.module.Configuration.resolve(Configuration.java:422)
[error]         at java.base/java.lang.module.Configuration.resolve(Configuration.java:256)
[error]         at jdk.jdeps/com.sun.tools.jdeps.JdepsConfiguration.<init>(JdepsConfiguration.java:117)
[error]         at jdk.jdeps/com.sun.tools.jdeps.JdepsConfiguration$Builder.build(JdepsConfiguration.java:563)
[error]         at jdk.jdeps/com.sun.tools.jdeps.JdepsTask.buildConfig(JdepsTask.java:589)
[error]         at jdk.jdeps/com.sun.tools.jdeps.JdepsTask.run(JdepsTask.java:543)
[error]         at jdk.jdeps/com.sun.tools.jdeps.JdepsTask.run(JdepsTask.java:519)
[error]         at jdk.jdeps/com.sun.tools.jdeps.Main.run(Main.java:64)
[error]         at jdk.jdeps/com.sun.tools.jdeps.Main$JDepsToolProvider.run(Main.java:73)
[error]         at java.base/java.util.spi.ToolProvider.run(ToolProvider.java:137)
[error]         at ru.eldis.toollauncher.ToolLauncher.runTool(ToolLauncher.java:68)
[error]         at ru.eldis.toollauncher.ToolLauncher.lambda$main$1(ToolLauncher.java:33)
[error]         at ru.eldis.toollauncher.ToolLauncher.main(ToolLauncher.java:48)

In this case we must add icu4j jar file to module path, just like with “paranamer”

jlinkModulePath := {
  fullClasspath
    .in(jlinkBuildImage)
    .value
    .filter { item =>
      item.get(moduleID.key).exists { modId =>
        modId.name == "paranamer"
      } || 
      item.data.toString().contains("icu4j")
    }
    .map(_.data)
}

Docker

Having zipped application with custom JRE is good but in 2021 the’s a high chance that we will need docker images. Let’s do that! Add another plugin into the build file:

lazy val root = (project in file(".")).enablePlugins(PlayScala, JlinkPlugin, DockerPlugin)

Next step - set some params for docker plugin

import com.typesafe.sbt.packager.docker.DockerChmodType
import com.typesafe.sbt.SbtNativePackager.autoImport.NativePackagerHelper._

mappings in Universal ++= directory(baseDirectory.value / "jre")
dockerBaseImage := "debian:stable-slim"
dockerExposedPorts := Seq(9000)
dockerChmodType := DockerChmodType.UserGroupWriteExecute
dockerAdditionalPermissions += (DockerChmodType.UserGroupPlusExecute, "/opt/docker/jre/bin/java")

Here we add jre dir to docker image,  doing chmod to jre binary, set exposed ports and base image. You can change base image to any other you preffer, but alpine linux will not work here with jdk11. See - jeps/386.

Now we can test out build with sbt command: sbt docker:publishLocal

We got a docker image with our custom JRE bundled inside. The size of the resulting docker image is - 156 MB which is pretty good compared to those with full JRE 250-300 MB. In some situations docker image size can be more than 1GB, looking at you, graalvm docker image.

Let’s run this container and test how it works in the browser:

docker run -p 9000:9000 test-jlink:1.0-SNAPSHOT

Jre target

Jre target

Great, our application works and now we have a docker image with custom slim JRE which we can push to the registry and run on the server. One more thing to keep in mind - to run the app in docker(Linux) you must build it on a Linux host machine. If you build a docker image on mac os and then run it you will get an error:

test-jlink % docker run test-jlink:1.0-SNAPSHOT
/opt/docker/bin/test-jlink: 1: exec: /opt/docker/jre/bin/java: Exec format error

For local development purposes, if you need to build and run a docker image of your app - you can simply skip jlink step and use base image bundled with jre adoptopenjdk/openjdk11:alpine-jre for example. And for production, you can build on CI with Linux and then run on Linux servers.

Or you can build custom jre inside docker, but that’s out of the scope of this article.

Source code on github: https://github.com/dkovalenko/scala-sbt-jlink-demo

Want to discuss?


Tags: scala, sbt, jlink, jigsaw, custom jre