Software security has been a concern for businesses and individuals alike since the very first introduction of basic consumer firewalls. But recent events have highlighted how crucial software supply chain security is becoming.
The SolarWinds hack in 2020 is a prime example, with 18,000 organisations downloading a backdoored version of SolarWinds software. While the ultimate impact of that was small, it could have been massive. More recently, the discovery of the xz Utils backdoor could have been much worse had it not been caught before release. This backdoor, introduced with social engineering into an open source project, took advantage of the size, scope, and lack of monitoring that has made open source so successful and valuable.
The Eclipse Foundation and particularly the Eclipse Adoptium project have been introducing various security measures to address these concerns, including the downstream Eclipse Temurin project, which achieved SLSA Level 3 compliance earlier this year. While security is crucial, independent verification is important for developing industry trust. Eclipse Temurin is proud to announce that Eclipse JDK 21+ releases are now fully independently reproducible.
Fully Independently Reproducible Builds Represent Strongest Possible Assurance
There are, of course, varying degrees of reproducibility, which represent different points along a spectrum of security. The greater the reproducibility, the greater the confidence users can have that the software they’re using hasn’t been tampered with.
What was crucial for Adoptium was to ensure that the builds were fully reproducible, as in bit-for-bit identical. This took some doing, as even something as simple as reproducing a build on a different machine can result in bit-level differences, owing to variations in environment or setup.
Fundamentally, this had three components. Two were already put in place by the project: a well-defined reproducible build pipeline and a secure development Software Bill of Materials (SBOM). The third was a new DevKit that allows users to construct fully reproducible builds.
New GCC DevKit Enables Fully Independent Verification of Supply Chain Security for Linux
The challenge with creating a devkit was that Eclipse Temurin Linux builds were compiled using a custom Adoptium build of the GNU Compiler Collection (GCC). This process pulled GCC from source but meant that users had to use the exact same Adoptium GCC compilers to ensure builds could be reproduced identically.
While this meant that in principle builds could be reproduced bit-for-bit, the problem was that it presented a gap in supply chain security verification. Since all users needed to rely on the same compiler, builds could not be independently reproduced, and if the compiler had been tampered with there would be no way to check.
The solution was to construct a new devkit that would allow users to bypass this restriction. The new GCC DevKit for Linux includes:
- An absolute definition of a GCC toolchain
- The GCC and dependency source version
- The sysroot that was used and how exactly it is built
In other words, users now can independently verify that the compiler itself was not tampered with, closing the final remaining verification gap in the integrity of Eclipse Temurin builds.
Windows Support But Still Reliant on Microsoft Visual Studio
It’s worth noting that while Adoptium has added support for Windows as well, it works slightly differently than it does for Linux.
Rather than using a DevKit as such, a Temurin build on Windows uses a build tools edition of Visual Studio, which is obviously sourced from Microsoft. So, in that respect, consumers are relying to a certain extent on the security of the Microsoft Visual Studio, rather than being able to completely comprehensively evaluate it yourself.
The other difference is that the native binaries, DLLs, and executables in Windows are signed using the Microsoft Windows Signing Tool, which does not reproduce identically. But there is a workaround. We provide a tool for doing a reproducible comparison, which simply removes those signatures before the comparison is made, allowing the rest of the code to be evaluated to make sure it’s all the same.
Verification Goes Beyond Minimum Requirements for SLSA Level 3
One of the things noted at the beginning of this article is that this development was part of Eclipse Temurin meeting SLSA Level 3 compliance. It’s important to note, however, that Temurin had already achieved this level of compliance before the introduction of this latest capability for verification.
The thing that’s important to understand about SLSA is that it generally defines what has to be done to achieve different compliance levels, but not how those steps need to be achieved. In other words, the project only needed to demonstrate reproducibility to qualify for its current level.
The Eclipse Foundation and Adoptium are not looking to merely qualify for security credentials in a technical sense. We’re taking the steps that are necessary to ensure that our software supply chain is actually as secure as it can be. Adoptium’s well-defined build pipeline and SBOM were important and valuable steps in this process, but our goal was always to go further to ensure our builds were actually tamper-proof and completely trustworthy.
We’ve also put together instructions for producing your own Reproducible Verification Build for both Linux and Windows:
Work Ongoing to Further Enhance Security
Security is less of a destination as a goal to strive towards. What constitutes security is always changing. And as such, we’re continuing our efforts to ensure the Adoptium community can rely on the most secure software possible.
One such effort is to expand on the CycloneDX JSON format, which Temurin uses for its SBOMs. That standard has been evolving quickly for the last few years and continues to evolve, and we aim to keep pace. One example of this is the formulation extension of the SBOM, which gives the user information on how exactly their product is built, and that’s something the project is looking to improve later this year.
We plan to introduce the same capability to fully independently recreate builds on Mac OS. That’s one of our major user bases and we want to make sure that builds there are indeed fully reproducible using the Apple toolchain and gcc, which will require further testing.
The AQAvit project will also improve its test monitoring and reporting service. Every Temurin release must be tested and verified before it goes live, which is what this service does, enabling you to look at the test results and success of the various builds. Enhancements to the automatic reporting of the success of our builds will follow.
Get Involved With Eclipse Temurin
Eclipse Temurin is a big project with a lot going on to interest developers. There are many aspects to the project, including secure supply chains and reproducibility, builds and pipelines, and testing.
Whether you’re interested in helping the CI pipeline for building and testing and expanding the range of platforms it supports or updating Temurin build scripts and ensuring we have the toolchain to build it and environments to test it, there’s plenty of interesting work to be done all the time.
If you’re curious and would like to get involved, have a look at the website and GitHub and dive in!
About the Author
Andrew Leonard
Andrew Leonard works at Red Hat and is a committer to the Eclipse Adoptium, Eclipse Temurin, Eclipse Temurin Compliance, and Eclipse AQAvit projects, as well as a PMC member of the Adoptium top-level project.