Use S3 Bucket with Sling

classic Classic list List threaded Threaded
10 messages Options
Reply | Threaded
Open this post in threaded view
|

Use S3 Bucket with Sling

Saurabh Tiwari
Hello team,

Hope you are doing well.

Just to give you my background, I am beginner in Sling, Felix and
Jackrabbit technologies and in the process of exploring it.

I am planning to use S3 bucket for storage in Sling.
As per my knowledge, the NoSQL storage under the sling is Apache Jackrabbit
Oak,  and Oak supports larger assets storage in S3.

I found these document on Jackrabbit [1],  but unable to get it (may be due
to my limited knowledge of OSGi)

Kindly help me with the steps to be followed to use S3 bucket with sling
and please let me know if I misunderstood anything.

[1] https://jackrabbit.apache.org/oak/docs/osgi_config.html#config-sling


Best Regards,
Saurabh Tiwari
Reply | Threaded
Open this post in threaded view
|

Re: Use S3 Bucket with Sling

Robert Munteanu-2
Hi Saurabh,

On Tue, 2019-07-30 at 11:48 +0530, Saurabh Tiwari wrote:
> Hello team,
>
> Hope you are doing well.
>
> Just to give you my background, I am beginner in Sling, Felix and
> Jackrabbit technologies and in the process of exploring it.

Welcome :-)

>
> I am planning to use S3 bucket for storage in Sling.
> As per my knowledge, the NoSQL storage under the sling is Apache
> Jackrabbit
> Oak,  and Oak supports larger assets storage in S3.
>
> I found these document on Jackrabbit [1],  but unable to get it (may
> be due
> to my limited knowledge of OSGi)
>
> Kindly help me with the steps to be followed to use S3 bucket with
> sling
> and please let me know if I misunderstood anything.
>
> [1]
> https://jackrabbit.apache.org/oak/docs/osgi_config.html#config-sling

We do not have an explicit configuration set for Amazon S3 support in
the Sling starter.

I think that you would need at least:

- to deploy the AWS SDK Jar
- to add a configuration that enables the use of the
org.apache.jackrabbit.oak.plugins.blob.datastore.S3DataStore component

What have you tried so far?

Thanks,

Robert

Reply | Threaded
Open this post in threaded view
|

Re: Use S3 Bucket with Sling

J Logan
 Hi Saurabh,
This might help you get started when working with the Sling starter JAR.  Robert, it'd be great if you could sanity check this but I think this is more or less correct.

1.) Start up the JAR once to get your repository initialized.  Go to the OSGi bundle console and make note of two things:a.) Whether the "org.apache.jackrabbit:oak-blob-cloud" OSGi bundle is presentb.) The version of Oak that is deployed
2.) Stop Sling.  This might not be strictly necessary, but it can't hurt to do this while you're getting started.

3.) If the oak-blob-cloud isn't present in your bundle config, you need to download the "org.apache.jackrabbit:oak-blob-cloud:{oak_version}" JAR and copy it into your deployment.  The "oak_version" value should match the Oak version you saw above.
4.) Add a configuration file "org.apache.jackrabbit.oak.segment.SegmentNodeStoreService.config" that defines
customBlobStore=B"true"
5.) Add a configuration file "" that defines
accessKey="your_s3_access_key"
secretKey="your_s3_secret_key"
s3Bucket="your_s3_bucket"
s3Region="your_s3_region"
s3EndPoint="your_s3_endpoint"
connectionTimeout="120000"
socketTimeout="120000"
maxConnections="40"
writeThreads="30"
maxErrorRetry="10"

If you're using and EC2 instance for Sling, you can use instance profiles to define an access role, and you can leave accessKey and secretKey blank.

5.) Restart Sling.  At this point your setup should be working, or close to working.  Check the Sling logs and S3 console to troubleshoot any issues.
Google "Configuring node stores and data stores in AEM 6" for more info on configuration.
Regards, John


    On Monday, August 5, 2019, 5:44:17 AM PDT, Robert Munteanu <[hidden email]> wrote:  
 
 Hi Saurabh,

On Tue, 2019-07-30 at 11:48 +0530, Saurabh Tiwari wrote:
> Hello team,
>
> Hope you are doing well.
>
> Just to give you my background, I am beginner in Sling, Felix and
> Jackrabbit technologies and in the process of exploring it.

Welcome :-)

>
> I am planning to use S3 bucket for storage in Sling.
> As per my knowledge, the NoSQL storage under the sling is Apache
> Jackrabbit
> Oak,  and Oak supports larger assets storage in S3.
>
> I found these document on Jackrabbit [1],  but unable to get it (may
> be due
> to my limited knowledge of OSGi)
>
> Kindly help me with the steps to be followed to use S3 bucket with
> sling
> and please let me know if I misunderstood anything.
>
> [1]
> https://jackrabbit.apache.org/oak/docs/osgi_config.html#config-sling

We do not have an explicit configuration set for Amazon S3 support in
the Sling starter.

I think that you would need at least:

- to deploy the AWS SDK Jar
- to add a configuration that enables the use of the
org.apache.jackrabbit.oak.plugins.blob.datastore.S3DataStore component

What have you tried so far?

Thanks,

Robert

 
Reply | Threaded
Open this post in threaded view
|

Re: Use S3 Bucket with Sling

Robert Munteanu-2
On Mon, 2019-08-05 at 17:35 +0000, J Logan wrote:
>  Hi Saurabh,
> This might help you get started when working with the Sling starter
> JAR.  Robert, it'd be great if you could sanity check this but I
> think this is more or less correct.

Hi John,

Yes, that looks correct to me. A couple of notes:

a) You could add the required bundles and configs to the provisioning
model files ( src/main/resources/oak.txt ) directly.
b) Besides the oak-blob-cloud bundle you probably also need some
dependenies like the AWS SDK.

Thanks,

Robert

>
> 1.) Start up the JAR once to get your repository initialized.  Go to
> the OSGi bundle console and make note of two things:a.) Whether the
> "org.apache.jackrabbit:oak-blob-cloud" OSGi bundle is presentb.) The
> version of Oak that is deployed
> 2.) Stop Sling.  This might not be strictly necessary, but it can't
> hurt to do this while you're getting started.
>
> 3.) If the oak-blob-cloud isn't present in your bundle config, you
> need to download the "org.apache.jackrabbit:oak-blob-
> cloud:{oak_version}" JAR and copy it into your deployment.  The
> "oak_version" value should match the Oak version you saw above.
> 4.) Add a configuration file
> "org.apache.jackrabbit.oak.segment.SegmentNodeStoreService.config"
> that defines
> customBlobStore=B"true"
> 5.) Add a configuration file "" that defines
> accessKey="your_s3_access_key"
> secretKey="your_s3_secret_key"
> s3Bucket="your_s3_bucket"
> s3Region="your_s3_region"
> s3EndPoint="your_s3_endpoint"
> connectionTimeout="120000"
> socketTimeout="120000"
> maxConnections="40"
> writeThreads="30"
> maxErrorRetry="10"
>
> If you're using and EC2 instance for Sling, you can use instance
> profiles to define an access role, and you can leave accessKey and
> secretKey blank.
>
> 5.) Restart Sling.  At this point your setup should be working, or
> close to working.  Check the Sling logs and S3 console to
> troubleshoot any issues.
> Google "Configuring node stores and data stores in AEM 6" for more
> info on configuration.
> Regards, John
>
>
>     On Monday, August 5, 2019, 5:44:17 AM PDT, Robert Munteanu <
> [hidden email]> wrote:  
>  
>  Hi Saurabh,
>
> On Tue, 2019-07-30 at 11:48 +0530, Saurabh Tiwari wrote:
> > Hello team,
> >
> > Hope you are doing well.
> >
> > Just to give you my background, I am beginner in Sling, Felix and
> > Jackrabbit technologies and in the process of exploring it.
>
> Welcome :-)
>
> > I am planning to use S3 bucket for storage in Sling.
> > As per my knowledge, the NoSQL storage under the sling is Apache
> > Jackrabbit
> > Oak,  and Oak supports larger assets storage in S3.
> >
> > I found these document on Jackrabbit [1],  but unable to get it
> > (may
> > be due
> > to my limited knowledge of OSGi)
> >
> > Kindly help me with the steps to be followed to use S3 bucket with
> > sling
> > and please let me know if I misunderstood anything.
> >
> > [1]
> > https://jackrabbit.apache.org/oak/docs/osgi_config.html#config-sling
>
> We do not have an explicit configuration set for Amazon S3 support in
> the Sling starter.
>
> I think that you would need at least:
>
> - to deploy the AWS SDK Jar
> - to add a configuration that enables the use of the
> org.apache.jackrabbit.oak.plugins.blob.datastore.S3DataStore
> component
>
> What have you tried so far?
>
> Thanks,
>
> Robert
>
>  

Reply | Threaded
Open this post in threaded view
|

Re: Use S3 Bucket with Sling

J Logan
 Yes, you mentioned the AWS bundle in your first reply and I forgot to include that.
The bundle is aws-java-sdk-osgi: https://search.maven.org/search?q=g:com.amazonaws%20AND%20a:aws-java-sdk-osgi&core=gav

    On Tuesday, August 6, 2019, 1:58:11 AM PDT, Robert Munteanu <[hidden email]> wrote:  
 
 On Mon, 2019-08-05 at 17:35 +0000, J Logan wrote:
>  Hi Saurabh,
> This might help you get started when working with the Sling starter
> JAR.  Robert, it'd be great if you could sanity check this but I
> think this is more or less correct.

Hi John,

Yes, that looks correct to me. A couple of notes:

a) You could add the required bundles and configs to the provisioning
model files ( src/main/resources/oak.txt ) directly.
b) Besides the oak-blob-cloud bundle you probably also need some
dependenies like the AWS SDK.

Thanks,

Robert

>
> 1.) Start up the JAR once to get your repository initialized.  Go to
> the OSGi bundle console and make note of two things:a.) Whether the
> "org.apache.jackrabbit:oak-blob-cloud" OSGi bundle is presentb.) The
> version of Oak that is deployed
> 2.) Stop Sling.  This might not be strictly necessary, but it can't
> hurt to do this while you're getting started.
>
> 3.) If the oak-blob-cloud isn't present in your bundle config, you
> need to download the "org.apache.jackrabbit:oak-blob-
> cloud:{oak_version}" JAR and copy it into your deployment.  The
> "oak_version" value should match the Oak version you saw above.
> 4.) Add a configuration file
> "org.apache.jackrabbit.oak.segment.SegmentNodeStoreService.config"
> that defines
> customBlobStore=B"true"
> 5.) Add a configuration file "" that defines
> accessKey="your_s3_access_key"
> secretKey="your_s3_secret_key"
> s3Bucket="your_s3_bucket"
> s3Region="your_s3_region"
> s3EndPoint="your_s3_endpoint"
> connectionTimeout="120000"
> socketTimeout="120000"
> maxConnections="40"
> writeThreads="30"
> maxErrorRetry="10"
>
> If you're using and EC2 instance for Sling, you can use instance
> profiles to define an access role, and you can leave accessKey and
> secretKey blank.
>
> 5.) Restart Sling.  At this point your setup should be working, or
> close to working.  Check the Sling logs and S3 console to
> troubleshoot any issues.
> Google "Configuring node stores and data stores in AEM 6" for more
> info on configuration.
> Regards, John
>
>
>    On Monday, August 5, 2019, 5:44:17 AM PDT, Robert Munteanu <
> [hidden email]> wrote: 

>  Hi Saurabh,
>
> On Tue, 2019-07-30 at 11:48 +0530, Saurabh Tiwari wrote:
> > Hello team,
> >
> > Hope you are doing well.
> >
> > Just to give you my background, I am beginner in Sling, Felix and
> > Jackrabbit technologies and in the process of exploring it.
>
> Welcome :-)
>
> > I am planning to use S3 bucket for storage in Sling.
> > As per my knowledge, the NoSQL storage under the sling is Apache
> > Jackrabbit
> > Oak,  and Oak supports larger assets storage in S3.
> >
> > I found these document on Jackrabbit [1],  but unable to get it
> > (may
> > be due
> > to my limited knowledge of OSGi)
> >
> > Kindly help me with the steps to be followed to use S3 bucket with
> > sling
> > and please let me know if I misunderstood anything.
> >
> > [1]
> > https://jackrabbit.apache.org/oak/docs/osgi_config.html#config-sling
>
> We do not have an explicit configuration set for Amazon S3 support in
> the Sling starter.
>
> I think that you would need at least:
>
> - to deploy the AWS SDK Jar
> - to add a configuration that enables the use of the
> org.apache.jackrabbit.oak.plugins.blob.datastore.S3DataStore
> component
>
> What have you tried so far?
>
> Thanks,
>
> Robert
>

 
Reply | Threaded
Open this post in threaded view
|

Re: Use S3 Bucket with Sling

Saurabh Tiwari
Dear Robert and John,

Thanks so much for your inputs and comments, I highly appreciate your help.

Apologies for the delay in reply, because from last 5 days, I was trying to
implement the steps you suggested and got some success in this (Thanks to
the mailing list of Sling, I refer old mails also :)

So, here is the current status and steps I followed,

# 1. Clone the Sling starter from
https://github.com/apache/sling-org-apache-sling-starter

# 2. Updated the oak.txt file, please refer the link below for the content
of the file
https://justpaste.it/5caki

Basically, I have done three changes in oak.txt file,

## 2.1.) Added these dependencies, (with exploration, I found for
oak-blob-cloud require various dependencies, thus added all the required
ones)

org.apache.jackrabbit/oak-blob-cloud/${oak.version}
com.amazonaws/aws-java-sdk-osgi/1.11.330
io.netty/netty-handler/4.1.17.Final
io.netty/netty-buffer/4.1.17.Final
joda-time/joda-time/2.8.1
software.amazon.ion/ion-java/1.0.2
com.fasterxml.jackson.dataformat/jackson-dataformat-cbor/2.6.7
io.netty/netty-codec-http/4.1.17.Final
io.netty/netty-codec/4.1.17.Final
io.netty/netty-resolver/4.1.17.Final
io.netty/netty-common/4.1.17.Final
io.netty/netty-transport/4.1.17.Final

## 2.2) Added the
org.apache.jackrabbit.oak.plugins.blob.datastore.S3DataStore configuration.

## 2.3) Set the customBlobStore properties as true
(customBlobStore=B"true") in
org.apache.jackrabbit.oak.segment.SegmentNodeStoreService


# 3.) After the above changes, run the command
mvn --update-snapshots clean install

This results in *test cases failure*,
Please refer this link for console logs
https://justpaste.it/4ho0r

# 4.) Here is the content of report file
"org.apache.sling.launchpad.SmokeIT.txt"
generated at "sling-org-apache-sling-starter/target/failsafe-reports"
location
-------------------------------------------------------------------------------
Test set: org.apache.sling.launchpad.SmokeIT
-------------------------------------------------------------------------------
Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 61.585 s
<<< FAILURE! - in org.apache.sling.launchpad.SmokeIT
org.apache.sling.launchpad.SmokeIT  Time elapsed: 61.583 s  <<< ERROR!
java.lang.RuntimeException: Launchpad not ready. Failed check for URL
http://localhost:51038/server/default/jcr:root/content with message 'Status
code is HTTP/1.1 503 Service Unavailable'


Kindly advise me if I am missing anything.
After completion of this work, I am also planning to contribute back all
this information in the form of documentation to the project. :-)


Best Regards,
Saurabh Tiwari

On Tue, Aug 6, 2019 at 8:00 PM J Logan <[hidden email]>
wrote:

>  Yes, you mentioned the AWS bundle in your first reply and I forgot to
> include that.
> The bundle is aws-java-sdk-osgi:
> https://search.maven.org/search?q=g:com.amazonaws%20AND%20a:aws-java-sdk-osgi&core=gav
>
>     On Tuesday, August 6, 2019, 1:58:11 AM PDT, Robert Munteanu <
> [hidden email]> wrote:
>
>  On Mon, 2019-08-05 at 17:35 +0000, J Logan wrote:
> >  Hi Saurabh,
> > This might help you get started when working with the Sling starter
> > JAR.  Robert, it'd be great if you could sanity check this but I
> > think this is more or less correct.
>
> Hi John,
>
> Yes, that looks correct to me. A couple of notes:
>
> a) You could add the required bundles and configs to the provisioning
> model files ( src/main/resources/oak.txt ) directly.
> b) Besides the oak-blob-cloud bundle you probably also need some
> dependenies like the AWS SDK.
>
> Thanks,
>
> Robert
>
> >
> > 1.) Start up the JAR once to get your repository initialized.  Go to
> > the OSGi bundle console and make note of two things:a.) Whether the
> > "org.apache.jackrabbit:oak-blob-cloud" OSGi bundle is presentb.) The
> > version of Oak that is deployed
> > 2.) Stop Sling.  This might not be strictly necessary, but it can't
> > hurt to do this while you're getting started.
> >
> > 3.) If the oak-blob-cloud isn't present in your bundle config, you
> > need to download the "org.apache.jackrabbit:oak-blob-
> > cloud:{oak_version}" JAR and copy it into your deployment.  The
> > "oak_version" value should match the Oak version you saw above.
> > 4.) Add a configuration file
> > "org.apache.jackrabbit.oak.segment.SegmentNodeStoreService.config"
> > that defines
> > customBlobStore=B"true"
> > 5.) Add a configuration file "" that defines
> > accessKey="your_s3_access_key"
> > secretKey="your_s3_secret_key"
> > s3Bucket="your_s3_bucket"
> > s3Region="your_s3_region"
> > s3EndPoint="your_s3_endpoint"
> > connectionTimeout="120000"
> > socketTimeout="120000"
> > maxConnections="40"
> > writeThreads="30"
> > maxErrorRetry="10"
> >
> > If you're using and EC2 instance for Sling, you can use instance
> > profiles to define an access role, and you can leave accessKey and
> > secretKey blank.
> >
> > 5.) Restart Sling.  At this point your setup should be working, or
> > close to working.  Check the Sling logs and S3 console to
> > troubleshoot any issues.
> > Google "Configuring node stores and data stores in AEM 6" for more
> > info on configuration.
> > Regards, John
> >
> >
> >    On Monday, August 5, 2019, 5:44:17 AM PDT, Robert Munteanu <
> > [hidden email]> wrote:
> >
> >  Hi Saurabh,
> >
> > On Tue, 2019-07-30 at 11:48 +0530, Saurabh Tiwari wrote:
> > > Hello team,
> > >
> > > Hope you are doing well.
> > >
> > > Just to give you my background, I am beginner in Sling, Felix and
> > > Jackrabbit technologies and in the process of exploring it.
> >
> > Welcome :-)
> >
> > > I am planning to use S3 bucket for storage in Sling.
> > > As per my knowledge, the NoSQL storage under the sling is Apache
> > > Jackrabbit
> > > Oak,  and Oak supports larger assets storage in S3.
> > >
> > > I found these document on Jackrabbit [1],  but unable to get it
> > > (may
> > > be due
> > > to my limited knowledge of OSGi)
> > >
> > > Kindly help me with the steps to be followed to use S3 bucket with
> > > sling
> > > and please let me know if I misunderstood anything.
> > >
> > > [1]
> > > https://jackrabbit.apache.org/oak/docs/osgi_config.html#config-sling
> >
> > We do not have an explicit configuration set for Amazon S3 support in
> > the Sling starter.
> >
> > I think that you would need at least:
> >
> > - to deploy the AWS SDK Jar
> > - to add a configuration that enables the use of the
> > org.apache.jackrabbit.oak.plugins.blob.datastore.S3DataStore
> > component
> >
> > What have you tried so far?
> >
> > Thanks,
> >
> > Robert
> >
> >
>
>



--
Regards
Saurabh Tiwari
Reply | Threaded
Open this post in threaded view
|

Re: Use S3 Bucket with Sling

Robert Munteanu-2
Hi Saurabh,

On Mon, 2019-08-12 at 12:48 +0530, Saurabh Tiwari wrote:
> Kindly advise me if I am missing anything.
> After completion of this work, I am also planning to contribute back
> all
> this information in the form of documentation to the project. :-)

Looks like you're heading down the right path, something is broken :-)

I would suggest that you run

$ mvn clean package

And then launch the jar file using

$ java -jar target/org.apache.sling.starter-12-SNAPSHOT.jar

This will allow you to inspect the instance by accessing the web
console at

  http://localhost:8080/system/console

and also to get a clear view of the logs at sling/logs/error.log .

Hope this helps,

Robert

Reply | Threaded
Open this post in threaded view
|

Re: Use S3 Bucket with Sling

Joey Smith
In reply to this post by Saurabh Tiwari
Saurabh:
You just leaked your secret key and access key; you should probably rotate
those immediately.

On Mon, Aug 12, 2019 at 1:18 AM Saurabh Tiwari <
[hidden email]> wrote:

> Dear Robert and John,
>
> Thanks so much for your inputs and comments, I highly appreciate your help.
>
> Apologies for the delay in reply, because from last 5 days, I was trying to
> implement the steps you suggested and got some success in this (Thanks to
> the mailing list of Sling, I refer old mails also :)
>
> So, here is the current status and steps I followed,
>
> # 1. Clone the Sling starter from
> https://github.com/apache/sling-org-apache-sling-starter
>
> # 2. Updated the oak.txt file, please refer the link below for the content
> of the file
> https://justpaste.it/5caki
>
> Basically, I have done three changes in oak.txt file,
>
> ## 2.1.) Added these dependencies, (with exploration, I found for
> oak-blob-cloud require various dependencies, thus added all the required
> ones)
>
> org.apache.jackrabbit/oak-blob-cloud/${oak.version}
> com.amazonaws/aws-java-sdk-osgi/1.11.330
> io.netty/netty-handler/4.1.17.Final
> io.netty/netty-buffer/4.1.17.Final
> joda-time/joda-time/2.8.1
> software.amazon.ion/ion-java/1.0.2
> com.fasterxml.jackson.dataformat/jackson-dataformat-cbor/2.6.7
> io.netty/netty-codec-http/4.1.17.Final
> io.netty/netty-codec/4.1.17.Final
> io.netty/netty-resolver/4.1.17.Final
> io.netty/netty-common/4.1.17.Final
> io.netty/netty-transport/4.1.17.Final
>
> ## 2.2) Added the
> org.apache.jackrabbit.oak.plugins.blob.datastore.S3DataStore configuration.
>
> ## 2.3) Set the customBlobStore properties as true
> (customBlobStore=B"true") in
> org.apache.jackrabbit.oak.segment.SegmentNodeStoreService
>
>
> # 3.) After the above changes, run the command
> mvn --update-snapshots clean install
>
> This results in *test cases failure*,
> Please refer this link for console logs
> https://justpaste.it/4ho0r
>
> # 4.) Here is the content of report file
> "org.apache.sling.launchpad.SmokeIT.txt"
> generated at "sling-org-apache-sling-starter/target/failsafe-reports"
> location
>
> -------------------------------------------------------------------------------
> Test set: org.apache.sling.launchpad.SmokeIT
>
> -------------------------------------------------------------------------------
> Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 61.585 s
> <<< FAILURE! - in org.apache.sling.launchpad.SmokeIT
> org.apache.sling.launchpad.SmokeIT  Time elapsed: 61.583 s  <<< ERROR!
> java.lang.RuntimeException: Launchpad not ready. Failed check for URL
> http://localhost:51038/server/default/jcr:root/content with message
> 'Status
> code is HTTP/1.1 503 Service Unavailable'
>
>
> Kindly advise me if I am missing anything.
> After completion of this work, I am also planning to contribute back all
> this information in the form of documentation to the project. :-)
>
>
> Best Regards,
> Saurabh Tiwari
>
> On Tue, Aug 6, 2019 at 8:00 PM J Logan <[hidden email]
> .invalid>
> wrote:
>
> >  Yes, you mentioned the AWS bundle in your first reply and I forgot to
> > include that.
> > The bundle is aws-java-sdk-osgi:
> >
> https://search.maven.org/search?q=g:com.amazonaws%20AND%20a:aws-java-sdk-osgi&core=gav
> >
> >     On Tuesday, August 6, 2019, 1:58:11 AM PDT, Robert Munteanu <
> > [hidden email]> wrote:
> >
> >  On Mon, 2019-08-05 at 17:35 +0000, J Logan wrote:
> > >  Hi Saurabh,
> > > This might help you get started when working with the Sling starter
> > > JAR.  Robert, it'd be great if you could sanity check this but I
> > > think this is more or less correct.
> >
> > Hi John,
> >
> > Yes, that looks correct to me. A couple of notes:
> >
> > a) You could add the required bundles and configs to the provisioning
> > model files ( src/main/resources/oak.txt ) directly.
> > b) Besides the oak-blob-cloud bundle you probably also need some
> > dependenies like the AWS SDK.
> >
> > Thanks,
> >
> > Robert
> >
> > >
> > > 1.) Start up the JAR once to get your repository initialized.  Go to
> > > the OSGi bundle console and make note of two things:a.) Whether the
> > > "org.apache.jackrabbit:oak-blob-cloud" OSGi bundle is presentb.) The
> > > version of Oak that is deployed
> > > 2.) Stop Sling.  This might not be strictly necessary, but it can't
> > > hurt to do this while you're getting started.
> > >
> > > 3.) If the oak-blob-cloud isn't present in your bundle config, you
> > > need to download the "org.apache.jackrabbit:oak-blob-
> > > cloud:{oak_version}" JAR and copy it into your deployment.  The
> > > "oak_version" value should match the Oak version you saw above.
> > > 4.) Add a configuration file
> > > "org.apache.jackrabbit.oak.segment.SegmentNodeStoreService.config"
> > > that defines
> > > customBlobStore=B"true"
> > > 5.) Add a configuration file "" that defines
> > > accessKey="your_s3_access_key"
> > > secretKey="your_s3_secret_key"
> > > s3Bucket="your_s3_bucket"
> > > s3Region="your_s3_region"
> > > s3EndPoint="your_s3_endpoint"
> > > connectionTimeout="120000"
> > > socketTimeout="120000"
> > > maxConnections="40"
> > > writeThreads="30"
> > > maxErrorRetry="10"
> > >
> > > If you're using and EC2 instance for Sling, you can use instance
> > > profiles to define an access role, and you can leave accessKey and
> > > secretKey blank.
> > >
> > > 5.) Restart Sling.  At this point your setup should be working, or
> > > close to working.  Check the Sling logs and S3 console to
> > > troubleshoot any issues.
> > > Google "Configuring node stores and data stores in AEM 6" for more
> > > info on configuration.
> > > Regards, John
> > >
> > >
> > >    On Monday, August 5, 2019, 5:44:17 AM PDT, Robert Munteanu <
> > > [hidden email]> wrote:
> > >
> > >  Hi Saurabh,
> > >
> > > On Tue, 2019-07-30 at 11:48 +0530, Saurabh Tiwari wrote:
> > > > Hello team,
> > > >
> > > > Hope you are doing well.
> > > >
> > > > Just to give you my background, I am beginner in Sling, Felix and
> > > > Jackrabbit technologies and in the process of exploring it.
> > >
> > > Welcome :-)
> > >
> > > > I am planning to use S3 bucket for storage in Sling.
> > > > As per my knowledge, the NoSQL storage under the sling is Apache
> > > > Jackrabbit
> > > > Oak,  and Oak supports larger assets storage in S3.
> > > >
> > > > I found these document on Jackrabbit [1],  but unable to get it
> > > > (may
> > > > be due
> > > > to my limited knowledge of OSGi)
> > > >
> > > > Kindly help me with the steps to be followed to use S3 bucket with
> > > > sling
> > > > and please let me know if I misunderstood anything.
> > > >
> > > > [1]
> > > > https://jackrabbit.apache.org/oak/docs/osgi_config.html#config-sling
> > >
> > > We do not have an explicit configuration set for Amazon S3 support in
> > > the Sling starter.
> > >
> > > I think that you would need at least:
> > >
> > > - to deploy the AWS SDK Jar
> > > - to add a configuration that enables the use of the
> > > org.apache.jackrabbit.oak.plugins.blob.datastore.S3DataStore
> > > component
> > >
> > > What have you tried so far?
> > >
> > > Thanks,
> > >
> > > Robert
> > >
> > >
> >
> >
>
>
>
> --
> Regards
> Saurabh Tiwari
>
Reply | Threaded
Open this post in threaded view
|

Re: Use S3 Bucket with Sling

Saurabh Tiwari
Ah..
Thanks so much Joey for notifying.
I realise the my mistake immediately after sending the email and
precautionary actions are taken care.

Thanks again!

On Mon, Aug 12, 2019 at 7:45 PM Joey Smith <[hidden email]> wrote:

> Saurabh:
> You just leaked your secret key and access key; you should probably rotate
> those immediately.
>
> On Mon, Aug 12, 2019 at 1:18 AM Saurabh Tiwari <
> [hidden email]> wrote:
>
> > Dear Robert and John,
> >
> > Thanks so much for your inputs and comments, I highly appreciate your
> help.
> >
> > Apologies for the delay in reply, because from last 5 days, I was trying
> to
> > implement the steps you suggested and got some success in this (Thanks to
> > the mailing list of Sling, I refer old mails also :)
> >
> > So, here is the current status and steps I followed,
> >
> > # 1. Clone the Sling starter from
> > https://github.com/apache/sling-org-apache-sling-starter
> >
> > # 2. Updated the oak.txt file, please refer the link below for the
> content
> > of the file
> > https://justpaste.it/5caki
> >
> > Basically, I have done three changes in oak.txt file,
> >
> > ## 2.1.) Added these dependencies, (with exploration, I found for
> > oak-blob-cloud require various dependencies, thus added all the required
> > ones)
> >
> > org.apache.jackrabbit/oak-blob-cloud/${oak.version}
> > com.amazonaws/aws-java-sdk-osgi/1.11.330
> > io.netty/netty-handler/4.1.17.Final
> > io.netty/netty-buffer/4.1.17.Final
> > joda-time/joda-time/2.8.1
> > software.amazon.ion/ion-java/1.0.2
> > com.fasterxml.jackson.dataformat/jackson-dataformat-cbor/2.6.7
> > io.netty/netty-codec-http/4.1.17.Final
> > io.netty/netty-codec/4.1.17.Final
> > io.netty/netty-resolver/4.1.17.Final
> > io.netty/netty-common/4.1.17.Final
> > io.netty/netty-transport/4.1.17.Final
> >
> > ## 2.2) Added the
> > org.apache.jackrabbit.oak.plugins.blob.datastore.S3DataStore
> configuration.
> >
> > ## 2.3) Set the customBlobStore properties as true
> > (customBlobStore=B"true") in
> > org.apache.jackrabbit.oak.segment.SegmentNodeStoreService
> >
> >
> > # 3.) After the above changes, run the command
> > mvn --update-snapshots clean install
> >
> > This results in *test cases failure*,
> > Please refer this link for console logs
> > https://justpaste.it/4ho0r
> >
> > # 4.) Here is the content of report file
> > "org.apache.sling.launchpad.SmokeIT.txt"
> > generated at "sling-org-apache-sling-starter/target/failsafe-reports"
> > location
> >
> >
> -------------------------------------------------------------------------------
> > Test set: org.apache.sling.launchpad.SmokeIT
> >
> >
> -------------------------------------------------------------------------------
> > Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 61.585 s
> > <<< FAILURE! - in org.apache.sling.launchpad.SmokeIT
> > org.apache.sling.launchpad.SmokeIT  Time elapsed: 61.583 s  <<< ERROR!
> > java.lang.RuntimeException: Launchpad not ready. Failed check for URL
> > http://localhost:51038/server/default/jcr:root/content with message
> > 'Status
> > code is HTTP/1.1 503 Service Unavailable'
> >
> >
> > Kindly advise me if I am missing anything.
> > After completion of this work, I am also planning to contribute back all
> > this information in the form of documentation to the project. :-)
> >
> >
> > Best Regards,
> > Saurabh Tiwari
> >
> > On Tue, Aug 6, 2019 at 8:00 PM J Logan <[hidden email]
> > .invalid>
> > wrote:
> >
> > >  Yes, you mentioned the AWS bundle in your first reply and I forgot to
> > > include that.
> > > The bundle is aws-java-sdk-osgi:
> > >
> >
> https://search.maven.org/search?q=g:com.amazonaws%20AND%20a:aws-java-sdk-osgi&core=gav
> > >
> > >     On Tuesday, August 6, 2019, 1:58:11 AM PDT, Robert Munteanu <
> > > [hidden email]> wrote:
> > >
> > >  On Mon, 2019-08-05 at 17:35 +0000, J Logan wrote:
> > > >  Hi Saurabh,
> > > > This might help you get started when working with the Sling starter
> > > > JAR.  Robert, it'd be great if you could sanity check this but I
> > > > think this is more or less correct.
> > >
> > > Hi John,
> > >
> > > Yes, that looks correct to me. A couple of notes:
> > >
> > > a) You could add the required bundles and configs to the provisioning
> > > model files ( src/main/resources/oak.txt ) directly.
> > > b) Besides the oak-blob-cloud bundle you probably also need some
> > > dependenies like the AWS SDK.
> > >
> > > Thanks,
> > >
> > > Robert
> > >
> > > >
> > > > 1.) Start up the JAR once to get your repository initialized.  Go to
> > > > the OSGi bundle console and make note of two things:a.) Whether the
> > > > "org.apache.jackrabbit:oak-blob-cloud" OSGi bundle is presentb.) The
> > > > version of Oak that is deployed
> > > > 2.) Stop Sling.  This might not be strictly necessary, but it can't
> > > > hurt to do this while you're getting started.
> > > >
> > > > 3.) If the oak-blob-cloud isn't present in your bundle config, you
> > > > need to download the "org.apache.jackrabbit:oak-blob-
> > > > cloud:{oak_version}" JAR and copy it into your deployment.  The
> > > > "oak_version" value should match the Oak version you saw above.
> > > > 4.) Add a configuration file
> > > > "org.apache.jackrabbit.oak.segment.SegmentNodeStoreService.config"
> > > > that defines
> > > > customBlobStore=B"true"
> > > > 5.) Add a configuration file "" that defines
> > > > accessKey="your_s3_access_key"
> > > > secretKey="your_s3_secret_key"
> > > > s3Bucket="your_s3_bucket"
> > > > s3Region="your_s3_region"
> > > > s3EndPoint="your_s3_endpoint"
> > > > connectionTimeout="120000"
> > > > socketTimeout="120000"
> > > > maxConnections="40"
> > > > writeThreads="30"
> > > > maxErrorRetry="10"
> > > >
> > > > If you're using and EC2 instance for Sling, you can use instance
> > > > profiles to define an access role, and you can leave accessKey and
> > > > secretKey blank.
> > > >
> > > > 5.) Restart Sling.  At this point your setup should be working, or
> > > > close to working.  Check the Sling logs and S3 console to
> > > > troubleshoot any issues.
> > > > Google "Configuring node stores and data stores in AEM 6" for more
> > > > info on configuration.
> > > > Regards, John
> > > >
> > > >
> > > >    On Monday, August 5, 2019, 5:44:17 AM PDT, Robert Munteanu <
> > > > [hidden email]> wrote:
> > > >
> > > >  Hi Saurabh,
> > > >
> > > > On Tue, 2019-07-30 at 11:48 +0530, Saurabh Tiwari wrote:
> > > > > Hello team,
> > > > >
> > > > > Hope you are doing well.
> > > > >
> > > > > Just to give you my background, I am beginner in Sling, Felix and
> > > > > Jackrabbit technologies and in the process of exploring it.
> > > >
> > > > Welcome :-)
> > > >
> > > > > I am planning to use S3 bucket for storage in Sling.
> > > > > As per my knowledge, the NoSQL storage under the sling is Apache
> > > > > Jackrabbit
> > > > > Oak,  and Oak supports larger assets storage in S3.
> > > > >
> > > > > I found these document on Jackrabbit [1],  but unable to get it
> > > > > (may
> > > > > be due
> > > > > to my limited knowledge of OSGi)
> > > > >
> > > > > Kindly help me with the steps to be followed to use S3 bucket with
> > > > > sling
> > > > > and please let me know if I misunderstood anything.
> > > > >
> > > > > [1]
> > > > >
> https://jackrabbit.apache.org/oak/docs/osgi_config.html#config-sling
> > > >
> > > > We do not have an explicit configuration set for Amazon S3 support in
> > > > the Sling starter.
> > > >
> > > > I think that you would need at least:
> > > >
> > > > - to deploy the AWS SDK Jar
> > > > - to add a configuration that enables the use of the
> > > > org.apache.jackrabbit.oak.plugins.blob.datastore.S3DataStore
> > > > component
> > > >
> > > > What have you tried so far?
> > > >
> > > > Thanks,
> > > >
> > > > Robert
> > > >
> > > >
> > >
> > >
> >
> >
> >
> > --
> > Regards
> > Saurabh Tiwari
> >
>


--
Best regards,
Saurabh Tiwari
Reply | Threaded
Open this post in threaded view
|

Re: Use S3 Bucket with Sling

J Logan
Hi Saurabh,

I think you’ll find a Sling run directory if you do:

ls -aR target

from where you ran Maven.  I believe it’s a hidden file that uses the PID of your smoke test run in its name. Have you  looked at the sling.log under that  directory to see why the smoke test failed?

IIRC the smoke test simply starts Sling and tries to connect to it, so if there are any critical missing dependencies or errors, you’ll see something like that 503 error

Sent from my iPhone

> On Aug 12, 2019, at 22:59, Saurabh Tiwari <[hidden email]> wrote:
>
> Ah..
> Thanks so much Joey for notifying.
> I realise the my mistake immediately after sending the email and
> precautionary actions are taken care.
>
> Thanks again!
>
>> On Mon, Aug 12, 2019 at 7:45 PM Joey Smith <[hidden email]> wrote:
>>
>> Saurabh:
>> You just leaked your secret key and access key; you should probably rotate
>> those immediately.
>>
>> On Mon, Aug 12, 2019 at 1:18 AM Saurabh Tiwari <
>> [hidden email]> wrote:
>>
>>> Dear Robert and John,
>>>
>>> Thanks so much for your inputs and comments, I highly appreciate your
>> help.
>>>
>>> Apologies for the delay in reply, because from last 5 days, I was trying
>> to
>>> implement the steps you suggested and got some success in this (Thanks to
>>> the mailing list of Sling, I refer old mails also :)
>>>
>>> So, here is the current status and steps I followed,
>>>
>>> # 1. Clone the Sling starter from
>>> https://github.com/apache/sling-org-apache-sling-starter
>>>
>>> # 2. Updated the oak.txt file, please refer the link below for the
>> content
>>> of the file
>>> https://justpaste.it/5caki
>>>
>>> Basically, I have done three changes in oak.txt file,
>>>
>>> ## 2.1.) Added these dependencies, (with exploration, I found for
>>> oak-blob-cloud require various dependencies, thus added all the required
>>> ones)
>>>
>>> org.apache.jackrabbit/oak-blob-cloud/${oak.version}
>>> com.amazonaws/aws-java-sdk-osgi/1.11.330
>>> io.netty/netty-handler/4.1.17.Final
>>> io.netty/netty-buffer/4.1.17.Final
>>> joda-time/joda-time/2.8.1
>>> software.amazon.ion/ion-java/1.0.2
>>> com.fasterxml.jackson.dataformat/jackson-dataformat-cbor/2.6.7
>>> io.netty/netty-codec-http/4.1.17.Final
>>> io.netty/netty-codec/4.1.17.Final
>>> io.netty/netty-resolver/4.1.17.Final
>>> io.netty/netty-common/4.1.17.Final
>>> io.netty/netty-transport/4.1.17.Final
>>>
>>> ## 2.2) Added the
>>> org.apache.jackrabbit.oak.plugins.blob.datastore.S3DataStore
>> configuration.
>>>
>>> ## 2.3) Set the customBlobStore properties as true
>>> (customBlobStore=B"true") in
>>> org.apache.jackrabbit.oak.segment.SegmentNodeStoreService
>>>
>>>
>>> # 3.) After the above changes, run the command
>>> mvn --update-snapshots clean install
>>>
>>> This results in *test cases failure*,
>>> Please refer this link for console logs
>>> https://justpaste.it/4ho0r
>>>
>>> # 4.) Here is the content of report file
>>> "org.apache.sling.launchpad.SmokeIT.txt"
>>> generated at "sling-org-apache-sling-starter/target/failsafe-reports"
>>> location
>>>
>>>
>> -------------------------------------------------------------------------------
>>> Test set: org.apache.sling.launchpad.SmokeIT
>>>
>>>
>> -------------------------------------------------------------------------------
>>> Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 61.585 s
>>> <<< FAILURE! - in org.apache.sling.launchpad.SmokeIT
>>> org.apache.sling.launchpad.SmokeIT  Time elapsed: 61.583 s  <<< ERROR!
>>> java.lang.RuntimeException: Launchpad not ready. Failed check for URL
>>> http://localhost:51038/server/default/jcr:root/content with message
>>> 'Status
>>> code is HTTP/1.1 503 Service Unavailable'
>>>
>>>
>>> Kindly advise me if I am missing anything.
>>> After completion of this work, I am also planning to contribute back all
>>> this information in the form of documentation to the project. :-)
>>>
>>>
>>> Best Regards,
>>> Saurabh Tiwari
>>>
>>> On Tue, Aug 6, 2019 at 8:00 PM J Logan <[hidden email]
>>> .invalid>
>>> wrote:
>>>
>>>> Yes, you mentioned the AWS bundle in your first reply and I forgot to
>>>> include that.
>>>> The bundle is aws-java-sdk-osgi:
>>>>
>>>
>> https://search.maven.org/search?q=g:com.amazonaws%20AND%20a:aws-java-sdk-osgi&core=gav
>>>>
>>>>    On Tuesday, August 6, 2019, 1:58:11 AM PDT, Robert Munteanu <
>>>> [hidden email]> wrote:
>>>>
>>>>> On Mon, 2019-08-05 at 17:35 +0000, J Logan wrote:
>>>>> Hi Saurabh,
>>>>> This might help you get started when working with the Sling starter
>>>>> JAR.  Robert, it'd be great if you could sanity check this but I
>>>>> think this is more or less correct.
>>>>
>>>> Hi John,
>>>>
>>>> Yes, that looks correct to me. A couple of notes:
>>>>
>>>> a) You could add the required bundles and configs to the provisioning
>>>> model files ( src/main/resources/oak.txt ) directly.
>>>> b) Besides the oak-blob-cloud bundle you probably also need some
>>>> dependenies like the AWS SDK.
>>>>
>>>> Thanks,
>>>>
>>>> Robert
>>>>
>>>>>
>>>>> 1.) Start up the JAR once to get your repository initialized.  Go to
>>>>> the OSGi bundle console and make note of two things:a.) Whether the
>>>>> "org.apache.jackrabbit:oak-blob-cloud" OSGi bundle is presentb.) The
>>>>> version of Oak that is deployed
>>>>> 2.) Stop Sling.  This might not be strictly necessary, but it can't
>>>>> hurt to do this while you're getting started.
>>>>>
>>>>> 3.) If the oak-blob-cloud isn't present in your bundle config, you
>>>>> need to download the "org.apache.jackrabbit:oak-blob-
>>>>> cloud:{oak_version}" JAR and copy it into your deployment.  The
>>>>> "oak_version" value should match the Oak version you saw above.
>>>>> 4.) Add a configuration file
>>>>> "org.apache.jackrabbit.oak.segment.SegmentNodeStoreService.config"
>>>>> that defines
>>>>> customBlobStore=B"true"
>>>>> 5.) Add a configuration file "" that defines
>>>>> accessKey="your_s3_access_key"
>>>>> secretKey="your_s3_secret_key"
>>>>> s3Bucket="your_s3_bucket"
>>>>> s3Region="your_s3_region"
>>>>> s3EndPoint="your_s3_endpoint"
>>>>> connectionTimeout="120000"
>>>>> socketTimeout="120000"
>>>>> maxConnections="40"
>>>>> writeThreads="30"
>>>>> maxErrorRetry="10"
>>>>>
>>>>> If you're using and EC2 instance for Sling, you can use instance
>>>>> profiles to define an access role, and you can leave accessKey and
>>>>> secretKey blank.
>>>>>
>>>>> 5.) Restart Sling.  At this point your setup should be working, or
>>>>> close to working.  Check the Sling logs and S3 console to
>>>>> troubleshoot any issues.
>>>>> Google "Configuring node stores and data stores in AEM 6" for more
>>>>> info on configuration.
>>>>> Regards, John
>>>>>
>>>>>
>>>>>   On Monday, August 5, 2019, 5:44:17 AM PDT, Robert Munteanu <
>>>>> [hidden email]> wrote:
>>>>>
>>>>> Hi Saurabh,
>>>>>
>>>>>> On Tue, 2019-07-30 at 11:48 +0530, Saurabh Tiwari wrote:
>>>>>> Hello team,
>>>>>>
>>>>>> Hope you are doing well.
>>>>>>
>>>>>> Just to give you my background, I am beginner in Sling, Felix and
>>>>>> Jackrabbit technologies and in the process of exploring it.
>>>>>
>>>>> Welcome :-)
>>>>>
>>>>>> I am planning to use S3 bucket for storage in Sling.
>>>>>> As per my knowledge, the NoSQL storage under the sling is Apache
>>>>>> Jackrabbit
>>>>>> Oak,  and Oak supports larger assets storage in S3.
>>>>>>
>>>>>> I found these document on Jackrabbit [1],  but unable to get it
>>>>>> (may
>>>>>> be due
>>>>>> to my limited knowledge of OSGi)
>>>>>>
>>>>>> Kindly help me with the steps to be followed to use S3 bucket with
>>>>>> sling
>>>>>> and please let me know if I misunderstood anything.
>>>>>>
>>>>>> [1]
>>>>>>
>> https://jackrabbit.apache.org/oak/docs/osgi_config.html#config-sling
>>>>>
>>>>> We do not have an explicit configuration set for Amazon S3 support in
>>>>> the Sling starter.
>>>>>
>>>>> I think that you would need at least:
>>>>>
>>>>> - to deploy the AWS SDK Jar
>>>>> - to add a configuration that enables the use of the
>>>>> org.apache.jackrabbit.oak.plugins.blob.datastore.S3DataStore
>>>>> component
>>>>>
>>>>> What have you tried so far?
>>>>>
>>>>> Thanks,
>>>>>
>>>>> Robert
>>>>>
>>>>>
>>>>
>>>>
>>>
>>>
>>>
>>> --
>>> Regards
>>> Saurabh Tiwari
>>>
>>
>
>
> --
> Best regards,
> Saurabh Tiwari