Release Validator Docker Image

classic Classic list List threaded Threaded
21 messages Options
12
Reply | Threaded
Open this post in threaded view
|

Release Validator Docker Image

klcodanr
I've also been working on a docker image to validate and build releases:

https://github.com/apache/sling-whiteboard/tree/master/release-validator

It's relatively easy to use. You'll need to build it, then checking a
release is as easy as:

docker run sling-check-release [build-num]

A couple things I have noticed / want to change are:

 - Support mounting a local folder to save the downloaded artifacts
- It doesn't support the situation where one release depends on another
independent release. It will, however, support dependent releases in the
same repository. I'm not sure there is a clean way around this.
Reply | Threaded
Open this post in threaded view
|

Re: Release Validator Docker Image

Robert Munteanu-2
Hi Dan,

On Thu, 2019-06-13 at 10:20 -0500, Daniel Klco wrote:
> I've also been working on a docker image to validate and build
> releases:
>
> https://github.com/apache/sling-whiteboard/tree/master/release-validator
>
> It's relatively easy to use

That looks nice! One question - do you need to have the Java RPM
locally? If yes, have you considered using one of the OpenJDK docker
images, which already do this for you?

  https://hub.docker.com/_/openjdk

As a side note, I have some local scripts which do the following:

- check the status of a release on Github
- builds a reactor of the whole

I am a bit lazy but I'll attach them here, in case you want to pick
something up for the validator docker image feel free :-)

Thanks,

Robert

check_sling_ci (2K) Download Attachment
check_sling_build (2K) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: Release Validator Docker Image

Daniel Klco-2
Yeah, I've gone back and forth regarding the JDK RPM. On the one hand, it'd
be better not not require the local RPM, on the other hand, to support
validation it'd be good to be able (I would think) to compile against
multiple JDK versions and VMs (e.g. Oracle, IBM, whatever)

I guess it's a question to the overall team, is there any need to validate
against OracleJDK or should I just base this off OpenDJK?

On Thu, Jun 13, 2019 at 10:29 AM Robert Munteanu <[hidden email]> wrote:

> Hi Dan,
>
> On Thu, 2019-06-13 at 10:20 -0500, Daniel Klco wrote:
> > I've also been working on a docker image to validate and build
> > releases:
> >
> > https://github.com/apache/sling-whiteboard/tree/master/release-validator
> >
> > It's relatively easy to use
>
> That looks nice! One question - do you need to have the Java RPM
> locally? If yes, have you considered using one of the OpenJDK docker
> images, which already do this for you?
>
>   https://hub.docker.com/_/openjdk
>
> As a side note, I have some local scripts which do the following:
>
> - check the status of a release on Github
> - builds a reactor of the whole
>
> I am a bit lazy but I'll attach them here, in case you want to pick
> something up for the validator docker image feel free :-)
>
> Thanks,
>
> Robert
>
Reply | Threaded
Open this post in threaded view
|

Re: Release Validator Docker Image

Robert Munteanu-2
On Thu, 2019-06-13 at 10:49 -0500, Daniel Klco wrote:

> Yeah, I've gone back and forth regarding the JDK RPM. On the one
> hand, it'd
> be better not not require the local RPM, on the other hand, to
> support
> validation it'd be good to be able (I would think) to compile against
> multiple JDK versions and VMs (e.g. Oracle, IBM, whatever)
>
> I guess it's a question to the overall team, is there any need to
> validate
> against OracleJDK or should I just base this off OpenDJK?

I always used OpenJDK.

Robert

>
> On Thu, Jun 13, 2019 at 10:29 AM Robert Munteanu <[hidden email]>
> wrote:
>
> > Hi Dan,
> >
> > On Thu, 2019-06-13 at 10:20 -0500, Daniel Klco wrote:
> > > I've also been working on a docker image to validate and build
> > > releases:
> > >
> > > https://github.com/apache/sling-whiteboard/tree/master/release-validator
> > >
> > > It's relatively easy to use
> >
> > That looks nice! One question - do you need to have the Java RPM
> > locally? If yes, have you considered using one of the OpenJDK
> > docker
> > images, which already do this for you?
> >
> >   https://hub.docker.com/_/openjdk
> >
> > As a side note, I have some local scripts which do the following:
> >
> > - check the status of a release on Github
> > - builds a reactor of the whole
> >
> > I am a bit lazy but I'll attach them here, in case you want to pick
> > something up for the validator docker image feel free :-)
> >
> > Thanks,
> >
> > Robert
> >

Reply | Threaded
Open this post in threaded view
|

Re: Release Validator Docker Image

Daniel Klco-2
Sorry for the slow turnaround on this. I updated to use OpenJDK and
rejected the idea of running the integration tests as it makes the
execution time extremely slow. Is this something we'd be interested in
moving out of the Whiteboard? Unfortunately, it does have quite a different
structure than the committer-cli due to being mostly bash scripts.


On Mon, Jun 17, 2019 at 10:38 AM Robert Munteanu <[hidden email]> wrote:

> On Thu, 2019-06-13 at 10:49 -0500, Daniel Klco wrote:
> > Yeah, I've gone back and forth regarding the JDK RPM. On the one
> > hand, it'd
> > be better not not require the local RPM, on the other hand, to
> > support
> > validation it'd be good to be able (I would think) to compile against
> > multiple JDK versions and VMs (e.g. Oracle, IBM, whatever)
> >
> > I guess it's a question to the overall team, is there any need to
> > validate
> > against OracleJDK or should I just base this off OpenDJK?
>
> I always used OpenJDK.
>
> Robert
>
> >
> > On Thu, Jun 13, 2019 at 10:29 AM Robert Munteanu <[hidden email]>
> > wrote:
> >
> > > Hi Dan,
> > >
> > > On Thu, 2019-06-13 at 10:20 -0500, Daniel Klco wrote:
> > > > I've also been working on a docker image to validate and build
> > > > releases:
> > > >
> > > >
> https://github.com/apache/sling-whiteboard/tree/master/release-validator
> > > >
> > > > It's relatively easy to use
> > >
> > > That looks nice! One question - do you need to have the Java RPM
> > > locally? If yes, have you considered using one of the OpenJDK
> > > docker
> > > images, which already do this for you?
> > >
> > >   https://hub.docker.com/_/openjdk
> > >
> > > As a side note, I have some local scripts which do the following:
> > >
> > > - check the status of a release on Github
> > > - builds a reactor of the whole
> > >
> > > I am a bit lazy but I'll attach them here, in case you want to pick
> > > something up for the validator docker image feel free :-)
> > >
> > > Thanks,
> > >
> > > Robert
> > >
>
>
Reply | Threaded
Open this post in threaded view
|

Re: Release Validator Docker Image

Robert Munteanu-2
I was recently thinking about the way we check releases. We currently
have a number of scripts:

- check_staged_release.sh
- check_release_matches_tag.sh

Also I have my own scripts to check the CI status and build locally.
You have proposed a release validator docker image. I would suggest
tying thet up together in the following manner.

1. Have a top-level check_sling_release.sh script which drives the
whole process

2. Create checks-enabled and checks-available directories

The checks-enabled will include

0001_check_staged_release.sh

The checks-available will contain all other checks, e.g.

0002_check_release_matches_tag.sh
0003_check_validator_image.sh
0004_check_ci_status.sh
0005_check_local_build.sh

Enabling a certain check checks to run then means creating (for
example) a symlink from checks-
available/0002_check_release_matches_tag.sh to checks-enabled

This way we can both accumulate more checks and allow flexibility in
which checks are selected.

As for the validator image, we can have in the tooling-release
repository or a different one, I don't have a strong preference either
way.

Thanks,
Robert


On Tue, 2019-08-06 at 09:07 -0400, Daniel Klco wrote:

> Sorry for the slow turnaround on this. I updated to use OpenJDK and
> rejected the idea of running the integration tests as it makes the
> execution time extremely slow. Is this something we'd be interested
> in
> moving out of the Whiteboard? Unfortunately, it does have quite a
> different
> structure than the committer-cli due to being mostly bash scripts.
>
>
> On Mon, Jun 17, 2019 at 10:38 AM Robert Munteanu <[hidden email]>
> wrote:
>
> > On Thu, 2019-06-13 at 10:49 -0500, Daniel Klco wrote:
> > > Yeah, I've gone back and forth regarding the JDK RPM. On the one
> > > hand, it'd
> > > be better not not require the local RPM, on the other hand, to
> > > support
> > > validation it'd be good to be able (I would think) to compile
> > > against
> > > multiple JDK versions and VMs (e.g. Oracle, IBM, whatever)
> > >
> > > I guess it's a question to the overall team, is there any need to
> > > validate
> > > against OracleJDK or should I just base this off OpenDJK?
> >
> > I always used OpenJDK.
> >
> > Robert
> >
> > > On Thu, Jun 13, 2019 at 10:29 AM Robert Munteanu <
> > > [hidden email]>
> > > wrote:
> > >
> > > > Hi Dan,
> > > >
> > > > On Thu, 2019-06-13 at 10:20 -0500, Daniel Klco wrote:
> > > > > I've also been working on a docker image to validate and
> > > > > build
> > > > > releases:
> > > > >
> > > > >
> > https://github.com/apache/sling-whiteboard/tree/master/release-validator
> > > > > It's relatively easy to use
> > > >
> > > > That looks nice! One question - do you need to have the Java
> > > > RPM
> > > > locally? If yes, have you considered using one of the OpenJDK
> > > > docker
> > > > images, which already do this for you?
> > > >
> > > >   https://hub.docker.com/_/openjdk
> > > >
> > > > As a side note, I have some local scripts which do the
> > > > following:
> > > >
> > > > - check the status of a release on Github
> > > > - builds a reactor of the whole
> > > >
> > > > I am a bit lazy but I'll attach them here, in case you want to
> > > > pick
> > > > something up for the validator docker image feel free :-)
> > > >
> > > > Thanks,
> > > >
> > > > Robert
> > > >

Reply | Threaded
Open this post in threaded view
|

Re: Release Validator Docker Image

Daniel Klco-2
Nice idea! I'm thinking we could have a default set of commands, e.g. run
the check staged release and CI status and then a command line option to
attach the symlinks for more.

So I've got an early crack at it in the whiteboard. I am a bit confused
about the behavior of the ci check script you provided. For example,
checking release 2115, it returned an error code as follows:

Status for org.apache.sling.commons.scheduler 2.7.4: pending

See
https://github.com/apache/sling-org-apache-sling-commons-scheduler/commits/org.apache.sling.commons.scheduler-2.7.4
for details

Exit code: 129


When I checked the status it didn't seem very illuminating, so I checked
the build status which was successful:
https://builds.apache.org/blue/organizations/jenkins/Sling%2Fsling-org-apache-sling-commons-scheduler/activity

And Sonar:
https://sonarcloud.io/dashboard?id=apache_sling-org-apache-sling-commons-scheduler

Is there just something about that project that's not configured yet? Is
there some way from the Github API to get the cause?

On Thu, Aug 22, 2019 at 4:04 AM Robert Munteanu <[hidden email]> wrote:

> I was recently thinking about the way we check releases. We currently
> have a number of scripts:
>
> - check_staged_release.sh
> - check_release_matches_tag.sh
>
> Also I have my own scripts to check the CI status and build locally.
> You have proposed a release validator docker image. I would suggest
> tying thet up together in the following manner.
>
> 1. Have a top-level check_sling_release.sh script which drives the
> whole process
>
> 2. Create checks-enabled and checks-available directories
>
> The checks-enabled will include
>
> 0001_check_staged_release.sh
>
> The checks-available will contain all other checks, e.g.
>
> 0002_check_release_matches_tag.sh
> 0003_check_validator_image.sh
> 0004_check_ci_status.sh
> 0005_check_local_build.sh
>
> Enabling a certain check checks to run then means creating (for
> example) a symlink from checks-
> available/0002_check_release_matches_tag.sh to checks-enabled
>
> This way we can both accumulate more checks and allow flexibility in
> which checks are selected.
>
> As for the validator image, we can have in the tooling-release
> repository or a different one, I don't have a strong preference either
> way.
>
> Thanks,
> Robert
>
>
> On Tue, 2019-08-06 at 09:07 -0400, Daniel Klco wrote:
> > Sorry for the slow turnaround on this. I updated to use OpenJDK and
> > rejected the idea of running the integration tests as it makes the
> > execution time extremely slow. Is this something we'd be interested
> > in
> > moving out of the Whiteboard? Unfortunately, it does have quite a
> > different
> > structure than the committer-cli due to being mostly bash scripts.
> >
> >
> > On Mon, Jun 17, 2019 at 10:38 AM Robert Munteanu <[hidden email]>
> > wrote:
> >
> > > On Thu, 2019-06-13 at 10:49 -0500, Daniel Klco wrote:
> > > > Yeah, I've gone back and forth regarding the JDK RPM. On the one
> > > > hand, it'd
> > > > be better not not require the local RPM, on the other hand, to
> > > > support
> > > > validation it'd be good to be able (I would think) to compile
> > > > against
> > > > multiple JDK versions and VMs (e.g. Oracle, IBM, whatever)
> > > >
> > > > I guess it's a question to the overall team, is there any need to
> > > > validate
> > > > against OracleJDK or should I just base this off OpenDJK?
> > >
> > > I always used OpenJDK.
> > >
> > > Robert
> > >
> > > > On Thu, Jun 13, 2019 at 10:29 AM Robert Munteanu <
> > > > [hidden email]>
> > > > wrote:
> > > >
> > > > > Hi Dan,
> > > > >
> > > > > On Thu, 2019-06-13 at 10:20 -0500, Daniel Klco wrote:
> > > > > > I've also been working on a docker image to validate and
> > > > > > build
> > > > > > releases:
> > > > > >
> > > > > >
> > >
> https://github.com/apache/sling-whiteboard/tree/master/release-validator
> > > > > > It's relatively easy to use
> > > > >
> > > > > That looks nice! One question - do you need to have the Java
> > > > > RPM
> > > > > locally? If yes, have you considered using one of the OpenJDK
> > > > > docker
> > > > > images, which already do this for you?
> > > > >
> > > > >   https://hub.docker.com/_/openjdk
> > > > >
> > > > > As a side note, I have some local scripts which do the
> > > > > following:
> > > > >
> > > > > - check the status of a release on Github
> > > > > - builds a reactor of the whole
> > > > >
> > > > > I am a bit lazy but I'll attach them here, in case you want to
> > > > > pick
> > > > > something up for the validator docker image feel free :-)
> > > > >
> > > > > Thanks,
> > > > >
> > > > > Robert
> > > > >
>
>
Reply | Threaded
Open this post in threaded view
|

Re: Release Validator Docker Image

Radu Cotescu-3
Hello,

I’m +1 for any tool that reduces the number of manual steps I have to do when performing, validating or promoting a release.

I have one question though: couldn’t we push these features into the Apache Sling Committer CLI tool [0]? That tool is already based on a minimalistic Docker image with an embedded jlink-customised JRE. Sensible information is provided from the outside anyways (think Apache user and password, JIRA user and password), so if we’re thinking about the keys file we could do something similar.

The only reason for asking such a question is that it would be nice to consolidate all CLI checks into one tool.

Thanks,
Radu

[0] - https://github.com/apache/sling-org-apache-sling-committer-cli <https://github.com/apache/sling-org-apache-sling-committer-cli>
Reply | Threaded
Open this post in threaded view
|

Re: Release Validator Docker Image

Daniel Klco-2
Radu,

In concept I agree, no good reason to have multiple tools to do similar
things. I wonder if we just fork off the release validation process into a
separate script vs the Java-based processes. I'm just not convinced of the
value re-implementing in Java brings vs a few simple bash commands. Let me
poke around and see if I can come up with something.

Thanks,
Dan

On Fri, Aug 23, 2019 at 6:31 AM Radu Cotescu <[hidden email]> wrote:

> Hello,
>
> I’m +1 for any tool that reduces the number of manual steps I have to do
> when performing, validating or promoting a release.
>
> I have one question though: couldn’t we push these features into the
> Apache Sling Committer CLI tool [0]? That tool is already based on a
> minimalistic Docker image with an embedded jlink-customised JRE. Sensible
> information is provided from the outside anyways (think Apache user and
> password, JIRA user and password), so if we’re thinking about the keys file
> we could do something similar.
>
> The only reason for asking such a question is that it would be nice to
> consolidate all CLI checks into one tool.
>
> Thanks,
> Radu
>
> [0] - https://github.com/apache/sling-org-apache-sling-committer-cli <
> https://github.com/apache/sling-org-apache-sling-committer-cli>
Reply | Threaded
Open this post in threaded view
|

Re: Release Validator Docker Image

Bertrand Delacretaz
On Fri, Aug 23, 2019 at 3:00 PM Daniel Klco <[hidden email]> wrote:
> ...I'm just not convinced of the
> value re-implementing in Java brings vs a few simple bash commands...

+1, IMO what's important is for the tools used to be traceable, for
example by including digests of scripts (or Docker images?) in their
output and having people validate those digests before running the
tools.

The OpenWhisk release checking script [1] does that:

  echo "$(basename $0) (script SHA1: $(gpg --print-md SHA1 $0 | cut
-d' ' -f2-))"

So that when someone pastes the script output in their vote message,
it points to the exact version of the tool that was used, assuming
people check the script's digest when running it.

-Bertrand

[1] https://github.com/apache/openwhisk-release/blob/master/tools/rcverify.sh
Reply | Threaded
Open this post in threaded view
|

Re: Release Validator Docker Image

Radu Cotescu-3
In reply to this post by Daniel Klco-2
Hi Daniel,

> On 23 Aug 2019, at 14:59, Daniel Klco <[hidden email]> wrote:
>
> I'm just not convinced of the
> value re-implementing in Java brings vs a few simple bash commands.

Sure, it’s more work to do, but you’d benefit from a nice command line menu and we would unify the tooling. Here are a few examples:

$ docker run -it --env-file=./docker-env apache/sling-cli
Usage: docker run -it --env-file=./docker-env apache/sling-cli [COMMAND]
Apache Sling Committers CLI
Commands:
  help     Displays help information about the specified command
  release  Performs release-related operations



$ docker run -it --env-file=./docker-env apache/sling-cli release help
Usage: docker run -it --env-file=./docker-env apache/sling-cli release [COMMAND]
Performs release-related operations
Commands:
  create-new-jira-version  Creates a new Jira version, if needed, and transitions any unresolved issues from the version being released to
                             the next one
  list                     Lists all open releases
  prepare-email            Prepares an email vote for the releases found in the Nexus staged repository
  tally-votes              Counts votes cast for a release and generates the result email
  update-local-site        Updates the Sling website with the new release information, based on a local checkout
  update-reporter          Updates the Apache Reporter System with the new release information
  help                     Displays help information about the specified command



$ docker run -it --env-file=./docker-env apache/sling-cli release prepare-email help
Usage: docker run -it --env-file=./docker-env apache/sling-cli release prepare-email -r=<repositoryId> [-x=<executionMode>] [COMMAND]
Prepares an email vote for the releases found in the Nexus staged repository
  -r, --repository=<repositoryId>
         Nexus repository id
  -x, --execution-mode=<executionMode>
         execution mode: DRY_RUN, INTERACTIVE, AUTO; default: DRY_RUN
Commands:
  help  Displays help information about the specified command



To help with this I’ve already pushed some code for verifying release signatures in a branch [2], which could easily be integrated with the rest. Checking build status is trivial by comparison.

Regards,
Radu

[2] - https://github.com/apache/sling-org-apache-sling-committer-cli/commit/1c6ea560ddd03947c0fd0b4e14104a1db13b55d6 <https://github.com/apache/sling-org-apache-sling-committer-cli/commit/1c6ea560ddd03947c0fd0b4e14104a1db13b55d6>
Reply | Threaded
Open this post in threaded view
|

Re: Release Validator Docker Image

Daniel Klco-2
Thanks Radu,

I've been looking into this on and off over the last few days and I'm still
struggling to see the value of re-implementing all of this in Java instead
of relying on a simple series of scripts. Sure it's possible to do GPG key
and Hash validation, but I'd still need to write a scraper to download the
artifacts from Nexus and then all of the code around these other functions,
each of which are hundreds of lines of Java code vs a single line in BASH.

What would you think about flipping this the other way around? Instead of
re-implementing in Java, make the docker image more flexible and use
scripts to decide which function to invoke?

I've put together a quick example of what I'm thinking here:

https://github.com/apache/sling-org-apache-sling-committer-cli/tree/bash-validate

On Tue, Aug 27, 2019 at 3:31 AM Radu Cotescu <[hidden email]> wrote:

> Hi Daniel,
>
> > On 23 Aug 2019, at 14:59, Daniel Klco <[hidden email]> wrote:
> >
> > I'm just not convinced of the
> > value re-implementing in Java brings vs a few simple bash commands.
>
> Sure, it’s more work to do, but you’d benefit from a nice command line
> menu and we would unify the tooling. Here are a few examples:
>
> $ docker run -it --env-file=./docker-env apache/sling-cli
> Usage: docker run -it --env-file=./docker-env apache/sling-cli [COMMAND]
> Apache Sling Committers CLI
> Commands:
>   help     Displays help information about the specified command
>   release  Performs release-related operations
>
>
>
> $ docker run -it --env-file=./docker-env apache/sling-cli release help
> Usage: docker run -it --env-file=./docker-env apache/sling-cli release
> [COMMAND]
> Performs release-related operations
> Commands:
>   create-new-jira-version  Creates a new Jira version, if needed, and
> transitions any unresolved issues from the version being released to
>                              the next one
>   list                     Lists all open releases
>   prepare-email            Prepares an email vote for the releases found
> in the Nexus staged repository
>   tally-votes              Counts votes cast for a release and generates
> the result email
>   update-local-site        Updates the Sling website with the new release
> information, based on a local checkout
>   update-reporter          Updates the Apache Reporter System with the new
> release information
>   help                     Displays help information about the specified
> command
>
>
>
> $ docker run -it --env-file=./docker-env apache/sling-cli release
> prepare-email help
> Usage: docker run -it --env-file=./docker-env apache/sling-cli release
> prepare-email -r=<repositoryId> [-x=<executionMode>] [COMMAND]
> Prepares an email vote for the releases found in the Nexus staged
> repository
>   -r, --repository=<repositoryId>
>          Nexus repository id
>   -x, --execution-mode=<executionMode>
>          execution mode: DRY_RUN, INTERACTIVE, AUTO; default: DRY_RUN
> Commands:
>   help  Displays help information about the specified command
>
>
>
> To help with this I’ve already pushed some code for verifying release
> signatures in a branch [2], which could easily be integrated with the rest.
> Checking build status is trivial by comparison.
>
> Regards,
> Radu
>
> [2] -
> https://github.com/apache/sling-org-apache-sling-committer-cli/commit/1c6ea560ddd03947c0fd0b4e14104a1db13b55d6
> <
> https://github.com/apache/sling-org-apache-sling-committer-cli/commit/1c6ea560ddd03947c0fd0b4e14104a1db13b55d6
> >
Reply | Threaded
Open this post in threaded view
|

Re: Release Validator Docker Image

Radu Cotescu-3
Hi Daniel,

> On 6 Sep 2019, at 18:01, Daniel Klco <[hidden email]> wrote:
>
> Thanks Radu,
>
> I've been looking into this on and off over the last few days and I'm still
> struggling to see the value of re-implementing all of this in Java instead
> of relying on a simple series of scripts. Sure it's possible to do GPG key
> and Hash validation, but I'd still need to write a scraper to download the
> artifacts from Nexus and then all of the code around these other functions,
> each of which are hundreds of lines of Java code vs a single line in BASH.
>
> What would you think about flipping this the other way around? Instead of
> re-implementing in Java, make the docker image more flexible and use
> scripts to decide which function to invoke?
>
> I've put together a quick example of what I'm thinking here:
>
> https://github.com/apache/sling-org-apache-sling-committer-cli/tree/bash-validate
>

I’ve managed to implement the missing pieces during last week’s hackathon and on Friday afternoon [3]. The fact that we ship a Docker image with that code is just convenience I guess. The functionality is implemented actually as an OSGi app using the Sling feature model, so we could pack it however we want.

Cheers,
Radu

[3] - https://issues.apache.org/jira/browse/SLING-8684 <https://issues.apache.org/jira/browse/SLING-8684>
Reply | Threaded
Open this post in threaded view
|

Re: Release Validator Docker Image

Daniel Klco-2
Nice! The approach using the Lucene search of the repo contents is much
cleaner than what I was thinking.

On Sun, Sep 8, 2019 at 1:00 PM Radu Cotescu <[hidden email]> wrote:

> Hi Daniel,
>
> > On 6 Sep 2019, at 18:01, Daniel Klco <[hidden email]> wrote:
> >
> > Thanks Radu,
> >
> > I've been looking into this on and off over the last few days and I'm
> still
> > struggling to see the value of re-implementing all of this in Java
> instead
> > of relying on a simple series of scripts. Sure it's possible to do GPG
> key
> > and Hash validation, but I'd still need to write a scraper to download
> the
> > artifacts from Nexus and then all of the code around these other
> functions,
> > each of which are hundreds of lines of Java code vs a single line in
> BASH.
> >
> > What would you think about flipping this the other way around? Instead of
> > re-implementing in Java, make the docker image more flexible and use
> > scripts to decide which function to invoke?
> >
> > I've put together a quick example of what I'm thinking here:
> >
> >
> https://github.com/apache/sling-org-apache-sling-committer-cli/tree/bash-validate
> >
>
> I’ve managed to implement the missing pieces during last week’s hackathon
> and on Friday afternoon [3]. The fact that we ship a Docker image with that
> code is just convenience I guess. The functionality is implemented actually
> as an OSGi app using the Sling feature model, so we could pack it however
> we want.
>
> Cheers,
> Radu
>
> [3] - https://issues.apache.org/jira/browse/SLING-8684 <
> https://issues.apache.org/jira/browse/SLING-8684>
Reply | Threaded
Open this post in threaded view
|

Re: Release Validator Docker Image

Daniel Klco-2
Thanks Radu for doing the hard work. As predicted, once that was in place,
the CI check was relatively straightforward. I added the code to check the
CI status from GitHub, updated the GPG check to download the Sling key file
and added a summary message letting the user know if all of the checks
succeeded:

https://github.com/apache/sling-org-apache-sling-committer-cli/commit/310b49d9d57ba97cc651468ab14143f15433b202

You should now be able to do the following:

bash-3.2$ docker run --env-file=./docker-env apache/sling-cli release
verify --repository=2126


org.apache.sling.fileoptim-0.9.4-source-release.zip

GPG: signed by Dan Klco (CODE SIGNING KEY) <[hidden email]> with key
(id=0xF0EAC1A44C6E4124;
fingerprint=4D78347F4F4F868D8EC2CD13F0EAC1A44C6E4124)

SHA-1: VALID (ea25bb7fb0cb0bff51f20eb4d21cb91d5262259d)

MD-5: VALID (69c30bb2f5e117d1f4ba2d4278b15c0e)


org.apache.sling.fileoptim-0.9.4.jar

GPG: signed by Dan Klco (CODE SIGNING KEY) <[hidden email]> with key
(id=0xF0EAC1A44C6E4124;
fingerprint=4D78347F4F4F868D8EC2CD13F0EAC1A44C6E4124)

SHA-1: VALID (5a5df233209b018420f4255a3a087aeff8bb0a01)

MD-5: VALID (8be14fc95cb74e1d3ff23869dfe9afef)


org.apache.sling.fileoptim-0.9.4.pom

GPG: signed by Dan Klco (CODE SIGNING KEY) <[hidden email]> with key
(id=0xF0EAC1A44C6E4124;
fingerprint=4D78347F4F4F868D8EC2CD13F0EAC1A44C6E4124)

SHA-1: VALID (e08d7a115313af4c10e7006022d730e07a1cfc42)

MD-5: VALID (cfc07c2782b95a3c41e0b8b61fff54f2)

CI Status: VALID:

continuous-integration/jenkins/branch

State: success

Description: This commit looks good

See:
https://builds.apache.org/job/Sling/job/sling-org-apache-sling-file-optimization/job/master/37/display/redirect


org.apache.sling.fileoptim-0.9.4-javadoc.jar

GPG: signed by Dan Klco (CODE SIGNING KEY) <[hidden email]> with key
(id=0xF0EAC1A44C6E4124;
fingerprint=4D78347F4F4F868D8EC2CD13F0EAC1A44C6E4124)

SHA-1: VALID (003ebb552c39c7df07613ed21e7d11d8cc51d27a)

MD-5: VALID (f9b54cca8995a115b978716bb5ac9c14)


org.apache.sling.fileoptim-0.9.4-sources.jar

GPG: signed by Dan Klco (CODE SIGNING KEY) <[hidden email]> with key
(id=0xF0EAC1A44C6E4124;
fingerprint=4D78347F4F4F868D8EC2CD13F0EAC1A44C6E4124)

SHA-1: VALID (ff94c1c349390cd6e9e4316c648ce6909757bef2)

MD-5: VALID (746e62e2e70c006ad083ac76c18da798)



Release Summary: VALID (16 checks executed)

On Mon, Sep 9, 2019 at 9:16 AM Daniel Klco <[hidden email]> wrote:

> Nice! The approach using the Lucene search of the repo contents is much
> cleaner than what I was thinking.
>
> On Sun, Sep 8, 2019 at 1:00 PM Radu Cotescu <[hidden email]> wrote:
>
>> Hi Daniel,
>>
>> > On 6 Sep 2019, at 18:01, Daniel Klco <[hidden email]> wrote:
>> >
>> > Thanks Radu,
>> >
>> > I've been looking into this on and off over the last few days and I'm
>> still
>> > struggling to see the value of re-implementing all of this in Java
>> instead
>> > of relying on a simple series of scripts. Sure it's possible to do GPG
>> key
>> > and Hash validation, but I'd still need to write a scraper to download
>> the
>> > artifacts from Nexus and then all of the code around these other
>> functions,
>> > each of which are hundreds of lines of Java code vs a single line in
>> BASH.
>> >
>> > What would you think about flipping this the other way around? Instead
>> of
>> > re-implementing in Java, make the docker image more flexible and use
>> > scripts to decide which function to invoke?
>> >
>> > I've put together a quick example of what I'm thinking here:
>> >
>> >
>> https://github.com/apache/sling-org-apache-sling-committer-cli/tree/bash-validate
>> >
>>
>> I’ve managed to implement the missing pieces during last week’s hackathon
>> and on Friday afternoon [3]. The fact that we ship a Docker image with that
>> code is just convenience I guess. The functionality is implemented actually
>> as an OSGi app using the Sling feature model, so we could pack it however
>> we want.
>>
>> Cheers,
>> Radu
>>
>> [3] - https://issues.apache.org/jira/browse/SLING-8684 <
>> https://issues.apache.org/jira/browse/SLING-8684>
>
>
Reply | Threaded
Open this post in threaded view
|

Re: Release Validator Docker Image

Carsten Ziegeler
Can we maybe also remove the md5 files and create sha512 files, when the
checksum is correct?

Carsten

Am 09.09.2019 um 19:25 schrieb Daniel Klco:

> Thanks Radu for doing the hard work. As predicted, once that was in place,
> the CI check was relatively straightforward. I added the code to check the
> CI status from GitHub, updated the GPG check to download the Sling key file
> and added a summary message letting the user know if all of the checks
> succeeded:
>
> https://github.com/apache/sling-org-apache-sling-committer-cli/commit/310b49d9d57ba97cc651468ab14143f15433b202
>
> You should now be able to do the following:
>
> bash-3.2$ docker run --env-file=./docker-env apache/sling-cli release
> verify --repository=2126
>
>
> org.apache.sling.fileoptim-0.9.4-source-release.zip
>
> GPG: signed by Dan Klco (CODE SIGNING KEY) <[hidden email]> with key
> (id=0xF0EAC1A44C6E4124;
> fingerprint=4D78347F4F4F868D8EC2CD13F0EAC1A44C6E4124)
>
> SHA-1: VALID (ea25bb7fb0cb0bff51f20eb4d21cb91d5262259d)
>
> MD-5: VALID (69c30bb2f5e117d1f4ba2d4278b15c0e)
>
>
> org.apache.sling.fileoptim-0.9.4.jar
>
> GPG: signed by Dan Klco (CODE SIGNING KEY) <[hidden email]> with key
> (id=0xF0EAC1A44C6E4124;
> fingerprint=4D78347F4F4F868D8EC2CD13F0EAC1A44C6E4124)
>
> SHA-1: VALID (5a5df233209b018420f4255a3a087aeff8bb0a01)
>
> MD-5: VALID (8be14fc95cb74e1d3ff23869dfe9afef)
>
>
> org.apache.sling.fileoptim-0.9.4.pom
>
> GPG: signed by Dan Klco (CODE SIGNING KEY) <[hidden email]> with key
> (id=0xF0EAC1A44C6E4124;
> fingerprint=4D78347F4F4F868D8EC2CD13F0EAC1A44C6E4124)
>
> SHA-1: VALID (e08d7a115313af4c10e7006022d730e07a1cfc42)
>
> MD-5: VALID (cfc07c2782b95a3c41e0b8b61fff54f2)
>
> CI Status: VALID:
>
> continuous-integration/jenkins/branch
>
> State: success
>
> Description: This commit looks good
>
> See:
> https://builds.apache.org/job/Sling/job/sling-org-apache-sling-file-optimization/job/master/37/display/redirect
>
>
> org.apache.sling.fileoptim-0.9.4-javadoc.jar
>
> GPG: signed by Dan Klco (CODE SIGNING KEY) <[hidden email]> with key
> (id=0xF0EAC1A44C6E4124;
> fingerprint=4D78347F4F4F868D8EC2CD13F0EAC1A44C6E4124)
>
> SHA-1: VALID (003ebb552c39c7df07613ed21e7d11d8cc51d27a)
>
> MD-5: VALID (f9b54cca8995a115b978716bb5ac9c14)
>
>
> org.apache.sling.fileoptim-0.9.4-sources.jar
>
> GPG: signed by Dan Klco (CODE SIGNING KEY) <[hidden email]> with key
> (id=0xF0EAC1A44C6E4124;
> fingerprint=4D78347F4F4F868D8EC2CD13F0EAC1A44C6E4124)
>
> SHA-1: VALID (ff94c1c349390cd6e9e4316c648ce6909757bef2)
>
> MD-5: VALID (746e62e2e70c006ad083ac76c18da798)
>
>
>
> Release Summary: VALID (16 checks executed)
>
> On Mon, Sep 9, 2019 at 9:16 AM Daniel Klco <[hidden email]> wrote:
>
>> Nice! The approach using the Lucene search of the repo contents is much
>> cleaner than what I was thinking.
>>
>> On Sun, Sep 8, 2019 at 1:00 PM Radu Cotescu <[hidden email]> wrote:
>>
>>> Hi Daniel,
>>>
>>>> On 6 Sep 2019, at 18:01, Daniel Klco <[hidden email]> wrote:
>>>>
>>>> Thanks Radu,
>>>>
>>>> I've been looking into this on and off over the last few days and I'm
>>> still
>>>> struggling to see the value of re-implementing all of this in Java
>>> instead
>>>> of relying on a simple series of scripts. Sure it's possible to do GPG
>>> key
>>>> and Hash validation, but I'd still need to write a scraper to download
>>> the
>>>> artifacts from Nexus and then all of the code around these other
>>> functions,
>>>> each of which are hundreds of lines of Java code vs a single line in
>>> BASH.
>>>>
>>>> What would you think about flipping this the other way around? Instead
>>> of
>>>> re-implementing in Java, make the docker image more flexible and use
>>>> scripts to decide which function to invoke?
>>>>
>>>> I've put together a quick example of what I'm thinking here:
>>>>
>>>>
>>> https://github.com/apache/sling-org-apache-sling-committer-cli/tree/bash-validate
>>>>
>>>
>>> I’ve managed to implement the missing pieces during last week’s hackathon
>>> and on Friday afternoon [3]. The fact that we ship a Docker image with that
>>> code is just convenience I guess. The functionality is implemented actually
>>> as an OSGi app using the Sling feature model, so we could pack it however
>>> we want.
>>>
>>> Cheers,
>>> Radu
>>>
>>> [3] - https://issues.apache.org/jira/browse/SLING-8684 <
>>> https://issues.apache.org/jira/browse/SLING-8684>
>>
>>
>

--
--
Carsten Ziegeler
Adobe Research Switzerland
[hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: Release Validator Docker Image

Radu Cotescu-3
Hi Carsten,

> On 9 Sep 2019, at 19:29, Carsten Ziegeler <[hidden email]> wrote:
>
> Can we maybe also remove the md5 files and create sha512 files, when the checksum is correct?
>

The sha1 and md5 files seem to be generated by Nexus and not by our Maven release process. I quickly checked how the sha512 files are generated and this seems to be a config in the Apache parent pom [4]. However, those are not attached [5] to the build, most probably due to [6].

I guess we can create the sha512 files with this CLI tool, but the value would come only when we’d be able to push the staged artifacts to dist.

Cheers,
Radu

[4] - https://github.com/apache/maven-apache-parent/blob/apache-21/pom.xml#L440 <https://github.com/apache/maven-apache-parent/blob/apache-21/pom.xml#L440>
[5] - https://checksum-maven-plugin.nicoulaj.net/files-mojo.html#attachChecksums <https://checksum-maven-plugin.nicoulaj.net/files-mojo.html#attachChecksums>[6] - https://issues.apache.org/jira/browse/INFRA-14923 <https://issues.apache.org/jira/browse/INFRA-14923>
Reply | Threaded
Open this post in threaded view
|

Re: Release Validator Docker Image

Carsten Ziegeler
I think the value is already there today, we push manually to dist and
having the sha512 there and not the md5 is already a win.
It would be great to have this mirrored in the maven repository, but
that probably needs those bugs to be fixed.

Carsten

Am 10.09.2019 um 13:16 schrieb Radu Cotescu:

> Hi Carsten,
>
>> On 9 Sep 2019, at 19:29, Carsten Ziegeler <[hidden email]> wrote:
>>
>> Can we maybe also remove the md5 files and create sha512 files, when the checksum is correct?
>>
>
> The sha1 and md5 files seem to be generated by Nexus and not by our Maven release process. I quickly checked how the sha512 files are generated and this seems to be a config in the Apache parent pom [4]. However, those are not attached [5] to the build, most probably due to [6].
>
> I guess we can create the sha512 files with this CLI tool, but the value would come only when we’d be able to push the staged artifacts to dist.
>
> Cheers,
> Radu
>
> [4] - https://github.com/apache/maven-apache-parent/blob/apache-21/pom.xml#L440 <https://github.com/apache/maven-apache-parent/blob/apache-21/pom.xml#L440>
> [5] - https://checksum-maven-plugin.nicoulaj.net/files-mojo.html#attachChecksums <https://checksum-maven-plugin.nicoulaj.net/files-mojo.html#attachChecksums>[6] - https://issues.apache.org/jira/browse/INFRA-14923 <https://issues.apache.org/jira/browse/INFRA-14923>
>

--
--
Carsten Ziegeler
Adobe Research Switzerland
[hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: Release Validator Docker Image

Robert Munteanu-2
On Tue, 2019-09-10 at 22:57 +0200, Carsten Ziegeler wrote:
> I think the value is already there today, we push manually to dist
> and
> having the sha512 there and not the md5 is already a win.
> It would be great to have this mirrored in the maven repository, but
> that probably needs those bugs to be fixed.

So my proposal, echoing discussions at the hackathon would be to
implement a command that publishes a release to dist and that:

- downloads the release artifacts from Nexus (already covered by
existing command)
- computes SHA-256 checksums and adds them to the files to be deployed
- uses the Java API equivalent of `svn remove https://dist.apache.org/
...` to remove all old artifacts of a release
- uses the Java API equivalent of `svn import .
https://dist.apache.org/ ...` to import local set of artifacts to SVN

Would that cover everyone's needs for pushing to dist?

Robert

>
> Carsten
>
> Am 10.09.2019 um 13:16 schrieb Radu Cotescu:
> > Hi Carsten,
> >
> > > On 9 Sep 2019, at 19:29, Carsten Ziegeler <[hidden email]>
> > > wrote:
> > >
> > > Can we maybe also remove the md5 files and create sha512 files,
> > > when the checksum is correct?
> > >
> >
> > The sha1 and md5 files seem to be generated by Nexus and not by our
> > Maven release process. I quickly checked how the sha512 files are
> > generated and this seems to be a config in the Apache parent pom
> > [4]. However, those are not attached [5] to the build, most
> > probably due to [6].
> >
> > I guess we can create the sha512 files with this CLI tool, but the
> > value would come only when we’d be able to push the staged
> > artifacts to dist.
> >
> > Cheers,
> > Radu
> >
> > [4] -
> > https://github.com/apache/maven-apache-parent/blob/apache-21/pom.xml#L440
> > <
> > https://github.com/apache/maven-apache-parent/blob/apache-21/pom.xml#L440
> > >
> > [5] -
> > https://checksum-maven-plugin.nicoulaj.net/files-mojo.html#attachChecksums
> > <
> > https://checksum-maven-plugin.nicoulaj.net/files-mojo.html#attachChecksums>[6
> > ] - https://issues.apache.org/jira/browse/INFRA-14923 <
> > https://issues.apache.org/jira/browse/INFRA-14923>
> >

Reply | Threaded
Open this post in threaded view
|

Re: Release Validator Docker Image

Carsten Ziegeler
Sounds good to me, with the addition that it does not upload the md5
files to SVN and only creates the sha256 (or sha512?) if the check of
the files is successful

Carsten

Am 10.09.2019 um 23:33 schrieb Robert Munteanu:

> On Tue, 2019-09-10 at 22:57 +0200, Carsten Ziegeler wrote:
>> I think the value is already there today, we push manually to dist
>> and
>> having the sha512 there and not the md5 is already a win.
>> It would be great to have this mirrored in the maven repository, but
>> that probably needs those bugs to be fixed.
>
> So my proposal, echoing discussions at the hackathon would be to
> implement a command that publishes a release to dist and that:
>
> - downloads the release artifacts from Nexus (already covered by
> existing command)
> - computes SHA-256 checksums and adds them to the files to be deployed
> - uses the Java API equivalent of `svn remove https://dist.apache.org/
> ...` to remove all old artifacts of a release
> - uses the Java API equivalent of `svn import .
> https://dist.apache.org/ ...` to import local set of artifacts to SVN
>
> Would that cover everyone's needs for pushing to dist?
>
> Robert
>
>>
>> Carsten
>>
>> Am 10.09.2019 um 13:16 schrieb Radu Cotescu:
>>> Hi Carsten,
>>>
>>>> On 9 Sep 2019, at 19:29, Carsten Ziegeler <[hidden email]>
>>>> wrote:
>>>>
>>>> Can we maybe also remove the md5 files and create sha512 files,
>>>> when the checksum is correct?
>>>>
>>>
>>> The sha1 and md5 files seem to be generated by Nexus and not by our
>>> Maven release process. I quickly checked how the sha512 files are
>>> generated and this seems to be a config in the Apache parent pom
>>> [4]. However, those are not attached [5] to the build, most
>>> probably due to [6].
>>>
>>> I guess we can create the sha512 files with this CLI tool, but the
>>> value would come only when we’d be able to push the staged
>>> artifacts to dist.
>>>
>>> Cheers,
>>> Radu
>>>
>>> [4] -
>>> https://github.com/apache/maven-apache-parent/blob/apache-21/pom.xml#L440
>>> <
>>> https://github.com/apache/maven-apache-parent/blob/apache-21/pom.xml#L440
>>>>
>>> [5] -
>>> https://checksum-maven-plugin.nicoulaj.net/files-mojo.html#attachChecksums
>>> <
>>> https://checksum-maven-plugin.nicoulaj.net/files-mojo.html#attachChecksums>[6
>>> ] - https://issues.apache.org/jira/browse/INFRA-14923 <
>>> https://issues.apache.org/jira/browse/INFRA-14923>
>>>
>

--
--
Carsten Ziegeler
Adobe Research Switzerland
[hidden email]
12