8293499: Provide jmod --compress option #10213

shipilev · 2022-09-08T07:57:36Z

I have been looking into make clean-images images performance, and realized jmod keeps compressing files with default compression level. Tuning that toward lighter compression levels improves build performance considerably, without a heavy loss in *.jmod sizes.

This PR allows JMOD to select the compression level. Follow-ups would use this in the build system, see #10214.

The interesting asymmetry against jlink is: jlink provides --compress option that only takes 2 for "ZIP compression". (Separately, we could argue if it would be beneficial to extend --compress to jlink as well, so to select the compression level there too.)

Progress

Change must be properly reviewed (1 review required, with at least 1 Reviewer)
Change must not contain extraneous whitespace
Commit message must refer to an issue
Change requires a CSR request to be approved

Issues

JDK-8293499: Provide jmod --compress option
JDK-8293629: Provide jmod --compress option (CSR)

Reviewers

Alan Bateman (@AlanBateman - Reviewer) ⚠️ Review applies to 8f1dc761
Mandy Chung (@mlchung - Reviewer)
Jaikiran Pai (@jaikiran - Reviewer)
Claes Redestad (@cl4es - Reviewer)

Reviewing

Using git

Checkout this PR locally:
$ git fetch https://git.openjdk.org/jdk pull/10213/head:pull/10213
$ git checkout pull/10213

Update a local copy of the PR:
$ git checkout pull/10213
$ git pull https://git.openjdk.org/jdk pull/10213/head

Using Skara CLI tools

Checkout this PR locally:
$ git pr checkout 10213

View PR using the GUI difftool:
$ git pr show -t 10213

Using diff file

Download this PR as a diff file:
https://git.openjdk.org/jdk/pull/10213.diff

bridgekeeper · 2022-09-08T07:59:19Z

👋 Welcome back shade! A progress list of the required criteria for merging this PR into master will be added to the body of your pull request. There are additional pull request commands available for use with this pull request.

openjdk · 2022-09-08T08:01:35Z

@shipilev The following label will be automatically applied to this pull request:

core-libs

When this pull request is ready to be reviewed, an "RFR" email will be sent to the corresponding mailing list. If you would like to change these labels, use the /label pull request command.

mlbridge · 2022-09-08T08:04:22Z

Webrevs

AlanBateman · 2022-09-08T13:34:39Z

I don't have time to look at this closely today but I think an option to configure the compression level wouldn't look out of place on the jmod tool.

Doing the equivalent with jlink would require more thought to see if it makes sense or is really needed. The jimage format is for execution, not a packaging format like jmod that is primarily consumed at link-time. I agree the jlink --compress option is a bit overloaded where --compress=0 and --compress=2 are more like jar --no-compress or its default to deflate. The jar tool doesn't have an option to set the compression level.

mlchung · 2022-09-09T02:00:29Z

It's reasonable for jmod tool to support --compress-level option to specify the compression level. The JMOD file format is not specified but the implementation uses ZIP. Compression level from 0-9 makes sense for ZIP but may not be applicable if the JMOD file format is changed. But it may be okay to worry about that when the file format is changed.

For jlink, when resources are compressed in writing to jimage, there would be performance overhead in reading and decompressing the resources at runtime. I'd be interested in knowing the tradeoff between the smaller jimage vs the runtime overhead in decompressing the resources. I'd advice to separate the jmod compression level change from jlink compression level discussion.

AlanBateman · 2022-09-09T13:03:11Z

I went through the changes and I don't see anything obviously wrong. The command line option doesn't look out of place although as Mandy points us, the format is not documented so there is flexibility to change it in the future. If some future format doesn't support something like compression level then that option could just emit a warning and do nothing.

AlanBateman · 2022-09-09T13:03:16Z

/csr

openjdk · 2022-09-09T13:04:38Z

@AlanBateman has indicated that a compatibility and specification (CSR) request is needed for this pull request.

@shipilev please create a CSR request for issue JDK-8293499 with the correct fix version. This pull request cannot be integrated until the CSR request is approved.

shipilev · 2022-09-12T09:39:59Z

I submitted the CSR request for this change: https://bugs.openjdk.org/browse/JDK-8293629

cl4es · 2022-09-12T10:13:12Z

Could we word this so that the accepted values is compression-dependent? The value appears to be ignored for --compress=0|1 (should it throw if specified or not specified to, say, 0?). We might also add other compressions in the future which allows a wider or narrower range of compression levels (zstd uses 0 - 22).

src/jdk.jlink/share/classes/jdk/tools/jmod/resources/jmod.properties

jaikiran · 2022-09-12T10:18:40Z

I gave this a try locally. It's my understanding that this new option is only relevant when creating the jmod archive. However, right now, in its current form, I can pass this option to commands like jmod extract and jmod list without those commands complaining about that option. For example, this following command doesn't complain about the --compression-level being passed to it:

jmod extract --compression-level=9 jdk/jmods/java.base.jmod

Should we instead complain/fail when this option is used for anything other than create?

shipilev · 2022-09-12T10:26:56Z

Could we word this so that the accepted values is compression-dependent? The value appears to be ignored for --compress=0|1 (should it throw if specified or not specified to, say, 0?). We might also add other compressions in the future which allows a wider or narrower range of compression levels (zstd uses 0 - 22).

Um, this is jmod, it does not have --compress=... option? jlink does. So, I don't quite understand this comment :)

shipilev · 2022-09-12T10:44:23Z

Should we instead complain/fail when this option is used for anything other than create?

Good idea, see new commit.

cl4es · 2022-09-12T10:48:53Z

Um, this is jmod, it does not have --compress=... option? jlink does. So, I don't quite understand this comment :)

Doh! Yes, I mixed up the two and assumed you'd be considering this for jlink as well. The part about being open-ended about alternate compression algorithms still applies, but since the compression for jmod is currently always zip we can defer such wording for if/when that day comes.

src/jdk.jlink/share/classes/jdk/tools/jmod/resources/jmod.properties

jaikiran · 2022-09-12T11:02:54Z

Thank you for the changes Aleksey. They look fine to me (except for that one minor error message related issue which I added a comment for). I haven't given the latest changes a try, but the new test you added to verify that the command fails in other modes when --compression-level will cover it.

shipilev · 2022-09-12T11:20:14Z

The part about being open-ended about alternate compression algorithms still applies, but since the compression for jmod is currently always zip we can defer such wording for if/when that day comes.

Well, we can future-proof it by doing what e.g. ZFS does:

$ zfs set compression
...
	compression     YES      YES   on | off | lzjb | gzip | gzip-[1-9] | zle | lz4 | zstd | zstd-[1-19] | zstd-fast-[1-10,20,30,40,50,60,70,80,90,100,500,1000]
...

Meaning, we allow --compress=zip-[0..9] and default to zip-6. This has the added benefit of clearly stating that zip-0 is still ZIP, but without the compression applied. This is a bit ambiguous with --compression-level=0.

cl4es · 2022-09-12T11:48:50Z

Meaning, we allow --compress=zip-[0..9] and default to zip-6. This has the added benefit of clearly stating that zip-0 is still ZIP, but without the compression applied. This is a bit ambiguous with --compression-level=0.

I like this scheme. It's a good thing there's prior art, too.

mbreinhold · 2022-09-12T15:51:25Z

Meaning, we allow --compress=zip-[0..9] and default to zip-6. This has the added benefit of clearly stating that zip-0 is still ZIP, but without the compression applied. This is a bit ambiguous with --compression-level=0.

I like this scheme. It's a good thing there's prior art, too.

Agreed. With this approach we can also revise jlink --compress to take the same arguments, keeping [012] for compatibility. We can also name this option --compress for jmod, rather than --compression-level, for consistency.

shipilev · 2022-09-12T18:34:41Z

Agreed. With this approach we can also revise jlink --compress to take the same arguments, keeping [012] for compatibility. We can also name this option --compress for jmod, rather than --compression-level, for consistency.

I updated this PR and related CSR with this new option format.

mlchung

I also like the new --compress option taking zip-0, zip-1,... zip-9 values.

src/jdk.jlink/share/classes/jdk/tools/jmod/JmodOutputStream.java

src/jdk.jlink/share/classes/jdk/tools/jmod/JmodTask.java

test/jdk/tools/jmod/JmodTest.java

src/jdk.jlink/share/classes/jdk/tools/jmod/JmodOutputStream.java

src/jdk.jlink/share/classes/jdk/tools/jmod/JmodTask.java

jaikiran

The latest state of this PR looks fine to me.

src/jdk.jlink/share/classes/jdk/tools/jmod/resources/jmod.properties

mlchung

Looks good. The jmod usage should specify the default.

jaikiran · 2022-09-14T01:05:35Z

Hello Aleksey, could you also please create a release note for this? You may already know how to do that, if not, the steps are noted here https://openjdk.org/guide/#release-notes

shipilev · 2022-09-14T08:01:00Z

Hello Aleksey, could you also please create a release note for this? You may already know how to do that, if not, the steps are noted here https://openjdk.org/guide/#release-notes

Yes, but later, once CSR and this PR integrates in its final form.

shipilev · 2022-09-16T06:49:45Z

(posting a comment in the hopes that bots would check the CSR state)

openjdk · 2022-09-16T06:53:14Z

@shipilev This change now passes all automated pre-integration checks.

ℹ️ This project also has non-automated pre-integration requirements. Please see the file CONTRIBUTING.md for details.

After integration, the commit message for the final commit will be:

8293499: Provide jmod --compress option

Reviewed-by: alanb, mchung, jpai, redestad

You can use pull request commands such as /summary, /contributor and /issue to adjust it as needed.

At the time when this comment was updated there had been 66 new commits pushed to the master branch:

01e7b88: 8290917: x86: Memory-operand arithmetic instructions have too low costs
4b8399b: 8293251: Use stringStream::base() instead of as_string() when applicable
a8f0f57: 8278165: Clarify that ZipInputStream does not access the CEN fields for a ZipEntry
746f5f5: 8293816: CI: ciBytecodeStream::get_klass() is not consistent
4b297c1: 8293892: Add links to JVMS 19 and 20 from ClassFileFormatVersion enum constants
dfb9c06: 8293535: jdk/javadoc/doclet/testJavaFX/TestJavaFxMode.java fail with jfx
f42caef: 8293550: Optionally add get-task-allow entitlement to macos binaries
5feca68: 8293840: RISC-V: Remove cbuf parameter from far_call/far_jump/trampoline_call
39cd163: 8293578: Duplicate ldc generated by javac
7765942: 8290367: Update default value and extend the scope of com.sun.jndi.ldap.object.trustSerialData system property
... and 56 more: https://git.openjdk.org/jdk/compare/37df5f56259429482cfdbe38e8b6256f1efaf9e8...master

As there are no conflicts, your changes will automatically be rebased on top of these commits when integrating. If you prefer to avoid this automatic rebasing, please check the documentation for the /integrate command for further details.

➡️ To integrate this PR with the above commit message to the master branch, type /integrate in a new comment.

shipilev · 2022-09-16T07:59:30Z

I did a minor cleanup, unquoted: "zip-6" -> zip-6 to be consistent in help message. Otherwise, this is ready to go and I will integrate it soon. Thanks!

shipilev · 2022-09-19T06:17:15Z

Thank you!

/integrate

openjdk · 2022-09-19T06:20:08Z

Going to push as commit 43f7f47.
Since your change was applied there have been 71 commits pushed to the master branch:

26e08cf: 8293844: C2: Verify Location::{oop,normal} types in PhaseOutput::FillLocArray
357a2cc: 8293937: x86: Drop LP64 conditions from clearly x86_32 code
b1ed40a: 8293466: libjsig should ignore non-modifying sigaction calls
b6ff8fa: 8292073: NMT: remove unused constructor parameter from MallocHeader
cfd44bb: 8293218: serviceability/tmtools/jstat/GcNewTest.java fails with "Error in the percent calculation"
01e7b88: 8290917: x86: Memory-operand arithmetic instructions have too low costs
4b8399b: 8293251: Use stringStream::base() instead of as_string() when applicable
a8f0f57: 8278165: Clarify that ZipInputStream does not access the CEN fields for a ZipEntry
746f5f5: 8293816: CI: ciBytecodeStream::get_klass() is not consistent
4b297c1: 8293892: Add links to JVMS 19 and 20 from ClassFileFormatVersion enum constants
... and 61 more: https://git.openjdk.org/jdk/compare/37df5f56259429482cfdbe38e8b6256f1efaf9e8...master

Your commit was automatically rebased without conflicts.

openjdk · 2022-09-19T06:20:25Z

@shipilev Pushed as commit 43f7f47.

💡 You may see a message that your pull request was closed with unmerged commits. This can be safely ignored.

Fix

8f1dc76

openjdk bot added the rfr Pull request is ready for review label Sep 8, 2022

openjdk bot added the core-libs core-libs-dev@openjdk.org label Sep 8, 2022

openjdk bot mentioned this pull request Sep 8, 2022

8293532: Use lighter jmod compression levels in build config #10214

Closed

3 tasks

openjdk bot added the csr Pull request needs approved CSR before integration label Sep 9, 2022

AlanBateman approved these changes Sep 9, 2022

View reviewed changes

jaikiran reviewed Sep 12, 2022

View reviewed changes

src/jdk.jlink/share/classes/jdk/tools/jmod/resources/jmod.properties Outdated Show resolved Hide resolved

shipilev added 2 commits September 12, 2022 12:27

Merge branch 'master' into JDK-8293499-jmod-compression-level

7117531

Only accept --compression-level when creating the archive

c44ecf0

jaikiran reviewed Sep 12, 2022

View reviewed changes

src/jdk.jlink/share/classes/jdk/tools/jmod/resources/jmod.properties Outdated Show resolved Hide resolved

Adopt "--compress zip-[0-9]" as option format

59eec8c

shipilev changed the title ~~8293499: Provide jmod compression level option~~ 8293499: Provide jmod --compress option Sep 12, 2022

mlchung reviewed Sep 12, 2022

View reviewed changes

More review comments

44b6402

jaikiran reviewed Sep 13, 2022

View reviewed changes

src/jdk.jlink/share/classes/jdk/tools/jmod/JmodOutputStream.java Outdated Show resolved Hide resolved

jaikiran reviewed Sep 13, 2022

View reviewed changes

src/jdk.jlink/share/classes/jdk/tools/jmod/JmodTask.java Show resolved Hide resolved

More review comments

092b0ff

jaikiran approved these changes Sep 13, 2022

View reviewed changes

mlchung reviewed Sep 13, 2022

View reviewed changes

src/jdk.jlink/share/classes/jdk/tools/jmod/resources/jmod.properties Outdated Show resolved Hide resolved

mlchung approved these changes Sep 13, 2022

View reviewed changes

Mention the default level

2729dfb

openjdk bot removed the csr Pull request needs approved CSR before integration label Sep 16, 2022

openjdk bot added the ready Pull request is ready to be integrated label Sep 16, 2022

Cleanup quotes in parameter help

78bf75c

mlchung approved these changes Sep 16, 2022

View reviewed changes

cl4es approved these changes Sep 16, 2022

View reviewed changes

jaikiran approved these changes Sep 17, 2022

View reviewed changes

openjdk bot added the integrated Pull request has been integrated label Sep 19, 2022

openjdk bot closed this Sep 19, 2022

openjdk bot removed ready Pull request is ready to be integrated rfr Pull request is ready for review labels Sep 19, 2022

shipilev deleted the JDK-8293499-jmod-compression-level branch October 21, 2022 09:42

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

8293499: Provide jmod --compress option #10213

8293499: Provide jmod --compress option #10213

shipilev commented Sep 8, 2022 •

edited by openjdk bot

Loading

bridgekeeper bot commented Sep 8, 2022

openjdk bot commented Sep 8, 2022

mlbridge bot commented Sep 8, 2022 •

edited

Loading

AlanBateman commented Sep 8, 2022 •

edited

Loading

mlchung commented Sep 9, 2022

AlanBateman commented Sep 9, 2022

AlanBateman commented Sep 9, 2022

openjdk bot commented Sep 9, 2022

shipilev commented Sep 12, 2022

cl4es commented Sep 12, 2022

jaikiran commented Sep 12, 2022

shipilev commented Sep 12, 2022

shipilev commented Sep 12, 2022

cl4es commented Sep 12, 2022

jaikiran commented Sep 12, 2022

shipilev commented Sep 12, 2022

cl4es commented Sep 12, 2022

mbreinhold commented Sep 12, 2022

shipilev commented Sep 12, 2022

mlchung left a comment •

edited

Loading

jaikiran left a comment

mlchung left a comment

jaikiran commented Sep 14, 2022

shipilev commented Sep 14, 2022

shipilev commented Sep 16, 2022

openjdk bot commented Sep 16, 2022 •

edited

Loading

shipilev commented Sep 16, 2022

shipilev commented Sep 19, 2022

openjdk bot commented Sep 19, 2022

openjdk bot commented Sep 19, 2022

8293499: Provide jmod --compress option #10213

8293499: Provide jmod --compress option #10213

Conversation

shipilev commented Sep 8, 2022 • edited by openjdk bot Loading

Progress

Issues

Reviewers

Reviewing

bridgekeeper bot commented Sep 8, 2022

openjdk bot commented Sep 8, 2022

mlbridge bot commented Sep 8, 2022 • edited Loading

Webrevs

AlanBateman commented Sep 8, 2022 • edited Loading

mlchung commented Sep 9, 2022

AlanBateman commented Sep 9, 2022

AlanBateman commented Sep 9, 2022

openjdk bot commented Sep 9, 2022

shipilev commented Sep 12, 2022

cl4es commented Sep 12, 2022

jaikiran commented Sep 12, 2022

shipilev commented Sep 12, 2022

shipilev commented Sep 12, 2022

cl4es commented Sep 12, 2022

jaikiran commented Sep 12, 2022

shipilev commented Sep 12, 2022

cl4es commented Sep 12, 2022

mbreinhold commented Sep 12, 2022

shipilev commented Sep 12, 2022

mlchung left a comment • edited Loading

Choose a reason for hiding this comment

jaikiran left a comment

Choose a reason for hiding this comment

mlchung left a comment

Choose a reason for hiding this comment

jaikiran commented Sep 14, 2022

shipilev commented Sep 14, 2022

shipilev commented Sep 16, 2022

openjdk bot commented Sep 16, 2022 • edited Loading

shipilev commented Sep 16, 2022

shipilev commented Sep 19, 2022

openjdk bot commented Sep 19, 2022

openjdk bot commented Sep 19, 2022

shipilev commented Sep 8, 2022 •

edited by openjdk bot

Loading

mlbridge bot commented Sep 8, 2022 •

edited

Loading

AlanBateman commented Sep 8, 2022 •

edited

Loading

mlchung left a comment •

edited

Loading

openjdk bot commented Sep 16, 2022 •

edited

Loading