import and export JCR data

classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|

import and export JCR data

lancedolan
Hello,

I'm a former CQ/AEM developer and architect, looking for the best way to import/export data WITHOUT the presence of CRX Package Manager. I'm working on a fresh CMS project using Sling+Oak from scratch, and will never have the benefits of CRX.

So... How do us raw sling developers import / export content data between Sling instances without the CRX Package Manager? What is the community standard approach to this?

I've done this easily with CRX packages in the past, which are apparently using the File Vault api under the hood to map XML-to-repo and repo-to-XML. I might be willing to re-write something like the CRX package manager myself from scratch... I think. But obviously, I could save myself SERIOUS time if there's an existing import/export solution. I'm sure I'm missing something - there's gotta be a slick way.

Is the vlt command line tool my only option? If so, how do I install it and use it? I have read [1], and the result is that I've built the vault-cli from source and when I run vlt export, I get:
"UnsupportedRepositoryOperationException: Missing implementation." I'm guessing I need to do something involving webdav, which I don't understand.

Even if I get vault-cli working, I doubt that it's a valid production-quality solution for moving content.

Also, I have already seen the data import documentation at [2] and I believe it's not a valid repo import/export option.

Help? :)

[1] http://jackrabbit.apache.org/filevault
[2] https://sling.apache.org/documentation/bundles/manipulating-content-the-slingpostservlet-servlets-post.html#importing-content-structures
Reply | Threaded
Open this post in threaded view
|

Re: import and export JCR data

Roy Teeuwen
Hey Lance,

I already gave you this answer on the jcr user list, don't know if you saw it because you haven't responded since...

Seeing as you are going to use Sling, have you looked at Composum[1]. It has all the features you need from CRX, it makes packages, it has a repository browser, it has user management...
It has MIT License and will even be included from the upcoming release of Sling, version 9, on.

Greets,
Roy

[1] https://github.com/ist-dresden/composum <https://github.com/ist-dresden/composum>



> On 6 Jan 2017, at 22:08, lancedolan <[hidden email]> wrote:
>
> Hello,
>
> I'm a former CQ/AEM developer and architect, looking for the best way to
> import/export data WITHOUT the presence of CRX Package Manager. I'm working
> on a fresh CMS project using Sling+Oak from scratch, and will never have the
> benefits of CRX.
>
> So... How do us raw sling developers import / export content data between
> Sling instances without the CRX Package Manager? What is the community
> standard approach to this?
>
> I've done this easily with CRX packages in the past, which are apparently
> using the File Vault api under the hood to map XML-to-repo and repo-to-XML.
> I might be willing to re-write something like the CRX package manager myself
> from scratch... I think. But obviously, I could save myself SERIOUS time if
> there's an existing import/export solution. I'm sure I'm missing something -
> there's gotta be a slick way.
>
> Is the vlt command line tool my only option? If so, how do I install it and
> use it? I have read [1], and the result is that I've built the vault-cli
> from source and when I run vlt export, I get:
> "UnsupportedRepositoryOperationException: Missing implementation." I'm
> guessing I need to do something involving webdav, which I don't understand.
>
> Even if I get vault-cli working, I doubt that it's a valid
> production-quality solution for moving content.
>
> Also, I have already seen the data import documentation at [2] and I believe
> it's not a valid repo import/export option.
>
> Help? :)
>
> [1] http://jackrabbit.apache.org/filevault
> [2]
> https://sling.apache.org/documentation/bundles/manipulating-content-the-slingpostservlet-servlets-post.html#importing-content-structures
>
>
>
>
> --
> View this message in context: http://apache-sling.73963.n3.nabble.com/import-and-export-JCR-data-tp4069390.html
> Sent from the Sling - Users mailing list archive at Nabble.com.

Reply | Threaded
Open this post in threaded view
|

Re: import and export JCR data

lancedolan
Sorry for the double post! *thank you* for the help.

The irony is that, after a full 8 hours of struggle, I found composum on my own only minutes before seeing your post, haha...

I'll probably be active in this community, especially if this P.O.C. goes smoothly enough to get business buy-in.

rock on
Lance
Reply | Threaded
Open this post in threaded view
|

Re: import and export JCR data

lancedolan
Hi All,


Does anybody have some terminal commands or procedure they follow for exporting data without composum? I've downloaded this "jackrabbit-filevault" project, and I've tried some different terminal commands after reading the documentation. Not sure what I'm doing.


*********
For those curious, why not Composum?
It is demonstrating a severe defect preventing me from building a jcr package [https://github.com/ist-dresden/composum/issues/60]. Even if that defect is solved, this instability is concerning enough that we need a separate reliable process for exporting data.
Reply | Threaded
Open this post in threaded view
|

Re: import and export JCR data

Roy Teeuwen
Hey,

I'm not convinced that it is Composum related, if you check the code you can clearly see they use the jackrabbit-filevault api themselves[1], see the imports of the packages.
U could debug that code, it's nothing fancy and I have used the package manager to export content nodes, see an example I just created with it that just has include filter /content [2]

Greets,
Roy

[1] https://github.com/ist-dresden/composum/blob/master/sling/core/pckgmgr/src/main/java/com/composum/sling/core/pckgmgr/PackageServlet.java <https://github.com/ist-dresden/composum/blob/master/sling/core/pckgmgr/src/main/java/com/composum/sling/core/pckgmgr/PackageServlet.java>
[2] http://www68.zippyshare.com/v/4Cc7QOSQ/file.html


> On 7 Feb 2017, at 19:43, lancedolan <[hidden email]> wrote:
>
> Hi All,
>
>
> Does anybody have some terminal commands or procedure they follow for
> exporting data without composum? I've downloaded this "jackrabbit-filevault"
> project, and I've tried some different terminal commands after reading the
> documentation. Not sure what I'm doing.
>
>
> *********
> For those curious, why not Composum?
> It is demonstrating a severe defect preventing me from building a jcr
> package [https://github.com/ist-dresden/composum/issues/60]. Even if that
> defect is solved, this instability is concerning enough that we need a
> separate reliable process for exporting data.
>
>
>
> --
> View this message in context: http://apache-sling.73963.n3.nabble.com/import-and-export-JCR-data-tp4069390p4070156.html
> Sent from the Sling - Users mailing list archive at Nabble.com.

Reply | Threaded
Open this post in threaded view
|

Re: import and export JCR data

lancedolan
If it's not composum, then it's file-vault or JCR related, which is even more concerning.

I'm pretty confident this isn't user error. I've given consulting demonstrations, teaching others the ins and outs of file-vault XML-to-node translation, the various complex rules one can combine, and also written code that dynamically assembles valid file vault packages to be POSTed to CRX package manager, which I believe also uses file vault under the hood... I'm really quite aware, in general, of how this thing should work. Of course I could be missing something, but if this is user error, then it's something esoteric and imperceptible, something very different than the CRX and file vault I'm used to.

Here's what I'm doing:

1) click "+" to create new package
2) click filter tab and "+" to create new filter
3) give root path "/content" and save with default Replace Import Mode and empty filter set
4) Click Build

Result: empty content directory in the package. Nothing in the log file.

I suppose as a next step I can split all com.composum logs to a separate log and set that to DEBUG level and watch for things... Maybe do the same with org.apache.jackrabbit.vault.

One plausible possibility seems that my /content data might be in some invalid state such that file vault can't/won't read it?? I'm really grasping at possibilities at this point.
Reply | Threaded
Open this post in threaded view
|

Re: import and export JCR data

Henry Saginor-2
Hi Lance,

I sometimes use this simple shell script [1] with file vault to import content from one AEM instance to another. I am happy to share if it helps. I am not sure if you are doing this between 2 sling instances though. But you should be able to change it to use export instead of rcp. Of course you would also need to change the command URL since it has /crx/.

Also, this just a guess, but see if your Sling instance has these bundles active. I think your guess is correct. The error indicates that the repository you are connecting to does not support the protocol you are using - WebDav in this case.
Jackrabbit WebDAV Library org.apache.jackrabbit.jackrabbit-webdav
Apache Sling Simple WebDAV Access to repositories org.apache.sling.jcr.webdav
Apache Sling DavEx Access to repositories org.apache.sling.jcr.davex

[1]
#!/bin/sh

help() {
echo
echo "parameters: user:password@sourceHost user:password@destHost cqPath"
echo “example: $0 admin:admin@localhost:4502 <http://localhost:4502/> admin:admin@localhost:4503 <http://localhost:4503/> /content/mysite"
echo
}

case $1 in
--help | -h | -help | -hlep)
help
;;
*)
if [ `echo $@ | wc -w` -gt 2 ]
then
  vlt -v rcp -b 1000 -t 1 -r http://$1/crx/-/jcr:root$3 http://$2/crx/-/jcr:root$3
  echo "Done!"
  echo
else
  help
fi;;
esac

> On Feb 7, 2017, at 3:18 PM, lancedolan <[hidden email]> wrote:
>
> If it's not composum, then it's file-vault or JCR related, which is even more
> concerning.
>
> I'm pretty confident this isn't user error. I've given consulting
> demonstrations, teaching others the ins and outs of file-vault XML-to-node
> translation, the various complex rules one can combine, and also written
> code that dynamically assembles valid file vault packages to be POSTed to
> CRX package manager, which I believe also uses file vault under the hood...
> I'm really quite aware, in general, of how this thing should work. Of course
> I could be missing something, but if this is user error, then it's something
> esoteric and imperceptible, something very different than the CRX and file
> vault I'm used to.
>
> Here's what I'm doing:
>
> 1) click "+" to create new package
> 2) click filter tab and "+" to create new filter
> 3) give root path "/content" and save with default Replace Import Mode and
> empty filter set
> 4) Click Build
>
> Result: empty content directory in the package. Nothing in the log file.
>
> I suppose as a next step I can split all com.composum logs to a separate log
> and set that to DEBUG level and watch for things... Maybe do the same with
> org.apache.jackrabbit.vault.
>
> One plausible possibility seems that my /content data might be in some
> invalid state such that file vault can't/won't read it?? I'm really grasping
> at possibilities at this point.
>
>
>
> --
> View this message in context: http://apache-sling.73963.n3.nabble.com/import-and-export-JCR-data-tp4069390p4070161.html
> Sent from the Sling - Users mailing list archive at Nabble.com.

Reply | Threaded
Open this post in threaded view
|

Re: import and export JCR data

Robert Munteanu-2
In reply to this post by lancedolan
On Tue, 2017-02-07 at 16:18 -0700, lancedolan wrote:

> If it's not composum, then it's file-vault or JCR related, which is
> even more
> concerning. 
>
> I'm pretty confident this isn't user error. I've given consulting
> demonstrations, teaching others the ins and outs of file-vault XML-
> to-node
> translation, the various complex rules one can combine, and also
> written
> code that dynamically assembles valid file vault packages to be
> POSTed to
> CRX package manager, which I believe also uses file vault under the
> hood...
> I'm really quite aware, in general, of how this thing should work. Of
> course
> I could be missing something, but if this is user error, then it's
> something
> esoteric and imperceptible, something very different than the CRX and
> file
> vault I'm used to.

For the record, FileVault as shipped by AEM is open source → you get
the same code.

Also the DavEx entry point is managed by the SlingDavExServlet for both
AEM and Sling so there should be no functional difference.

I suggest that you try using the vlt command-line tool as described at
[1] to see if that works.

Robert

[1]: https://cwiki.apache.org/confluence/display/SLING/Using+FileVault+
with+Sling

>
> Here's what I'm doing:
>
> 1) click "+" to create new package
> 2) click filter tab and "+" to create new filter
> 3) give root path "/content" and save with default Replace Import
> Mode and
> empty filter set
> 4) Click Build
>
> Result: empty content directory in the package. Nothing in the log
> file. 
>
> I suppose as a next step I can split all com.composum logs to a
> separate log
> and set that to DEBUG level and watch for things... Maybe do the same
> with
> org.apache.jackrabbit.vault. 
>
> One plausible possibility seems that my /content data might be in
> some
> invalid state such that file vault can't/won't read it?? I'm really
> grasping
> at possibilities at this point.
>
>
>
> --
> View this message in context: http://apache-sling.73963.n3.nabble.com
> /import-and-export-JCR-data-tp4069390p4070161.html
> Sent from the Sling - Users mailing list archive at Nabble.com.