File system resource provider - Performance

classic Classic list List threaded Threaded
8 messages Options
Reply | Threaded
Open this post in threaded view
|

File system resource provider - Performance

Roy Teeuwen
Hey all,

We are using the file system resource provider to monitor a directory to be attached as resources. Currently in our directory we have around 5 levels deep with in total 200.000 files. The problem we are facing is that when our application starts up, it takes an unusual amount of time to get online. It appears that the file system resource provider is building up the monitor cache and this is the cause of the performance decay. We currently have one file system resource provider config for the entire directory, would it be better to have multiple on lower levels, and if so, how many files maximum per resource provider config should we take?

Or maybe the file system resource provider is not made to work on this large amount of files? For our use-case it looks perfect because we also use the events being triggered and because we like to use the files as resources in our application without having to import them as actual jcr nodes.

Greets,
Roy

signature.asc (849 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: File system resource provider - Performance

Carsten Ziegeler
Hi,

yes, this is an odd behaviour. The whole tree is scanned on startup and
this is blocking. In addition, if you have a large tree the periodic
scanning for changes (this is polling) is probably not optimal either.

I think we could try using newer file features from Java 7 which might
make the scanning obsolete. But I've never looked into it.

Regards

Carsten


Roy Teeuwen wrote
> Hey all,
>
> We are using the file system resource provider to monitor a directory to be attached as resources. Currently in our directory we have around 5 levels deep with in total 200.000 files. The problem we are facing is that when our application starts up, it takes an unusual amount of time to get online. It appears that the file system resource provider is building up the monitor cache and this is the cause of the performance decay. We currently have one file system resource provider config for the entire directory, would it be better to have multiple on lower levels, and if so, how many files maximum per resource provider config should we take?
>
> Or maybe the file system resource provider is not made to work on this large amount of files? For our use-case it looks perfect because we also use the events being triggered and because we like to use the files as resources in our application without having to import them as actual jcr nodes.
>
> Greets,
> Roy
>
--
Carsten Ziegeler
Adobe Research Switzerland
[hidden email]
Reply | Threaded
Open this post in threaded view
|

RE: File system resource provider - Performance

Stefan Seifert

>I think we could try using newer file features from Java 7 which might
>make the scanning obsolete. But I've never looked into it.

you mean with e.g. this?
https://docs.oracle.com/javase/7/docs/api/java/nio/file/WatchService.html

i had this on my todo list some time ago for fsresource - but after some first experiments this seems to behave different depending on the operation system (windows vs. linux), so i dropped it that time. if someone comes up with a concept that works reliable on all operation systems i would be glad to help integrate it in fsresource.

stefan


Reply | Threaded
Open this post in threaded view
|

Re: File system resource provider - Performance

Carsten Ziegeler
 


Stefan Seifert wrote
>
>> I think we could try using newer file features from Java 7 which might
>> make the scanning obsolete. But I've never looked into it.
>
> you mean with e.g. this?
> https://docs.oracle.com/javase/7/docs/api/java/nio/file/WatchService.html
>
> i had this on my todo list some time ago for fsresource - but after some first experiments this seems to behave different depending on the operation system (windows vs. linux), so i dropped it that time. if someone comes up with a concept that works reliable on all operation systems i would be glad to help integrate it in fsresource.
>

Yes, I agree - this doesn't seem to help (I now remember that I looked
into it a long time ago).

I'm not sure if this is a good idea, but we could have a switch that
skips the initial scan and also skips registering the periodic scanners.
Once a file is accessed we lazily add a scanner for that directory to
the list. So once a resource/file is used you get change events. If a
directory is never touched, we don't do anything.

Regards
Carsten
--
Carsten Ziegeler
Adobe Research Switzerland
[hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: File system resource provider - Performance

Roy Teeuwen
In reply to this post by Stefan Seifert
Hey Stefan,

I was planning to give that a try too, could you maybe elaborate on what hick-ups you noticed on the different OSes?

Greets
Roy

> On 29 Sep 2017, at 15:52, Stefan Seifert <[hidden email]> wrote:
>
>
>> I think we could try using newer file features from Java 7 which might
>> make the scanning obsolete. But I've never looked into it.
>
> you mean with e.g. this?
> https://docs.oracle.com/javase/7/docs/api/java/nio/file/WatchService.html
>
> i had this on my todo list some time ago for fsresource - but after some first experiments this seems to behave different depending on the operation system (windows vs. linux), so i dropped it that time. if someone comes up with a concept that works reliable on all operation systems i would be glad to help integrate it in fsresource.
>
> stefan
>
>


signature.asc (849 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: File system resource provider - Performance

Roy Teeuwen
In reply to this post by Carsten Ziegeler
Hey Carsten,

What about doing the initial scanning also async? As in that you do the initial scanning in the first scheduled run, without throwing change events for that first run

Greets,
Roy

> On 29 Sep 2017, at 15:57, Carsten Ziegeler <[hidden email]> wrote:
>
>
>
>
> Stefan Seifert wrote
>>
>>> I think we could try using newer file features from Java 7 which might
>>> make the scanning obsolete. But I've never looked into it.
>>
>> you mean with e.g. this?
>> https://docs.oracle.com/javase/7/docs/api/java/nio/file/WatchService.html
>>
>> i had this on my todo list some time ago for fsresource - but after some first experiments this seems to behave different depending on the operation system (windows vs. linux), so i dropped it that time. if someone comes up with a concept that works reliable on all operation systems i would be glad to help integrate it in fsresource.
>>
>
> Yes, I agree - this doesn't seem to help (I now remember that I looked
> into it a long time ago).
>
> I'm not sure if this is a good idea, but we could have a switch that
> skips the initial scan and also skips registering the periodic scanners.
> Once a file is accessed we lazily add a scanner for that directory to
> the list. So once a resource/file is used you get change events. If a
> directory is never touched, we don't do anything.
>
> Regards
> Carsten
> --
> Carsten Ziegeler
> Adobe Research Switzerland
> [hidden email]


signature.asc (849 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

RE: File system resource provider - Performance

Stefan Seifert
In reply to this post by Roy Teeuwen
unfortunately i do not remember the details right now, but i assume it was something like "does not seem to work at all on windows" or so. i did not invest much time.
but it's probably worth having a second look.

stefan


>-----Original Message-----
>From: Roy Teeuwen [mailto:[hidden email]]
>Sent: Friday, September 29, 2017 3:58 PM
>To: [hidden email]
>Subject: Re: File system resource provider - Performance
>
>Hey Stefan,
>
>I was planning to give that a try too, could you maybe elaborate on what
>hick-ups you noticed on the different OSes?
>
>Greets
>Roy
>
>> On 29 Sep 2017, at 15:52, Stefan Seifert <[hidden email]> wrote:
>>
>>
>>> I think we could try using newer file features from Java 7 which might
>>> make the scanning obsolete. But I've never looked into it.
>>
>> you mean with e.g. this?
>> https://docs.oracle.com/javase/7/docs/api/java/nio/file/WatchService.html
>>
>> i had this on my todo list some time ago for fsresource - but after some
>first experiments this seems to behave different depending on the operation
>system (windows vs. linux), so i dropped it that time. if someone comes up
>with a concept that works reliable on all operation systems i would be glad
>to help integrate it in fsresource.
>>
>> stefan
>>
>>


Reply | Threaded
Open this post in threaded view
|

RE: File system resource provider - Performance

Stefan Seifert
In reply to this post by Roy Teeuwen
we can definitly think about making that startup async, perhaps configurable (with default=async to speed thinks up) - those monitoring is mainly to support the resource events. and for a lot of use cases it may not be importing if file changes during the startup phase are not detected.

stefan

>-----Original Message-----
>From: Roy Teeuwen [mailto:[hidden email]]
>Sent: Friday, September 29, 2017 4:01 PM
>To: [hidden email]
>Cc: Stefan Seifert
>Subject: Re: File system resource provider - Performance
>
>Hey Carsten,
>
>What about doing the initial scanning also async? As in that you do the
>initial scanning in the first scheduled run, without throwing change events
>for that first run
>
>Greets,
>Roy
>
>> On 29 Sep 2017, at 15:57, Carsten Ziegeler <[hidden email]> wrote:
>>
>>
>>
>>
>> Stefan Seifert wrote
>>>
>>>> I think we could try using newer file features from Java 7 which might
>>>> make the scanning obsolete. But I've never looked into it.
>>>
>>> you mean with e.g. this?
>>>
>https://docs.oracle.com/javase/7/docs/api/java/nio/file/WatchService.html
>>>
>>> i had this on my todo list some time ago for fsresource - but after some
>first experiments this seems to behave different depending on the operation
>system (windows vs. linux), so i dropped it that time. if someone comes up
>with a concept that works reliable on all operation systems i would be glad
>to help integrate it in fsresource.
>>>
>>
>> Yes, I agree - this doesn't seem to help (I now remember that I looked
>> into it a long time ago).
>>
>> I'm not sure if this is a good idea, but we could have a switch that
>> skips the initial scan and also skips registering the periodic scanners.
>> Once a file is accessed we lazily add a scanner for that directory to
>> the list. So once a resource/file is used you get change events. If a
>> directory is never touched, we don't do anything.
>>
>> Regards
>> Carsten
>> --
>> Carsten Ziegeler
>> Adobe Research Switzerland
>> [hidden email]