sling-trunk buildbot CI build causing problems

classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

sling-trunk buildbot CI build causing problems

Mark Thomas
Sling developers,

The sling-trunk CI build managed to kill one of the buildbot slaves by
filling this directory with files until the file system ran out of inodes:
/home/buildslave3/slave3/sling-trunk/build/testing/samples/integration-tests/sling/default/jackrabbit/workspaces/default/index

There were so many files ls hung for 5+ minutes without any output.

I have started to clean this up (rm -rf
/home/buildslave3/slave3/sling-trunk) and that looks like it is going to
take at least several hours to complete.

The next CI build should re-checkout sling-trunk so your CI builds
should be unaffected. However, please could you take a look at the
buildbot configuration for this build and figure out a) why this
happened and b) how to stop it happening again.

Cheers,

Mark
on behalf of the ASF Infra team
Reply | Threaded
Open this post in threaded view
|

Re: sling-trunk buildbot CI build causing problems

Robert Munteanu-2
On Fri, Jan 23, 2015 at 1:03 AM, Mark Thomas <[hidden email]> wrote:

> Sling developers,
>
> The sling-trunk CI build managed to kill one of the buildbot slaves by
> filling this directory with files until the file system ran out of inodes:
> /home/buildslave3/slave3/sling-trunk/build/testing/samples/integration-tests/sling/default/jackrabbit/workspaces/default/index
>
> There were so many files ls hung for 5+ minutes without any output.
>
> I have started to clean this up (rm -rf
> /home/buildslave3/slave3/sling-trunk) and that looks like it is going to
> take at least several hours to complete.
>
> The next CI build should re-checkout sling-trunk so your CI builds
> should be unaffected. However, please could you take a look at the
> buildbot configuration for this build and figure out a) why this
> happened and b) how to stop it happening again.

(Moving the discussion to dev@ only while we try to find a solution)

Does anyone have any idea about why this happens?

I'm wondering if we still need the buildbot setup as the Jenkins
builds seem to be (mostly) stable these days.

Robert

>
> Cheers,
>
> Mark
> on behalf of the ASF Infra team
Reply | Threaded
Open this post in threaded view
|

Re: sling-trunk buildbot CI build causing problems

Bertrand Delacretaz
Hi,

On Fri, Jan 23, 2015 at 2:23 PM, Robert Munteanu <[hidden email]> wrote:
> ...I'm wondering if we still need the buildbot setup as the Jenkins
> builds seem to be (mostly) stable these days....

Right, I agree that we could remove the buildbot setup for now.

-Bertrand
Reply | Threaded
Open this post in threaded view
|

Re: sling-trunk buildbot CI build causing problems

Bertrand Delacretaz
In reply to this post by Mark Thomas
Hi Mark,

On Fri, Jan 23, 2015 at 12:03 AM, Mark Thomas <[hidden email]> wrote:
> ...However, please could you take a look at the
> buildbot configuration for this build and figure out a) why this
> happened and b) how to stop it happening again....

For now, feel free to disable those jobs while we discuss a solution
within the Sling team.

-Bertrand
Reply | Threaded
Open this post in threaded view
|

Re: sling-trunk buildbot CI build causing problems

Mark Thomas
In reply to this post by Mark Thomas
Sling developers,

We have just had a re-occurrence of the same problem.

I will clean this up again this time but if it happens again I will
simply remove the sling builds from buildbot.

Mark

On 22/01/2015 23:03, Mark Thomas wrote:

> Sling developers,
>
> The sling-trunk CI build managed to kill one of the buildbot slaves by
> filling this directory with files until the file system ran out of inodes:
> /home/buildslave3/slave3/sling-trunk/build/testing/samples/integration-tests/sling/default/jackrabbit/workspaces/default/index
>
> There were so many files ls hung for 5+ minutes without any output.
>
> I have started to clean this up (rm -rf
> /home/buildslave3/slave3/sling-trunk) and that looks like it is going to
> take at least several hours to complete.
>
> The next CI build should re-checkout sling-trunk so your CI builds
> should be unaffected. However, please could you take a look at the
> buildbot configuration for this build and figure out a) why this
> happened and b) how to stop it happening again.
>
> Cheers,
>
> Mark
> on behalf of the ASF Infra team
>

Reply | Threaded
Open this post in threaded view
|

Re: sling-trunk buildbot CI build causing problems

Felix Meschberger-3
- private@sling, infra@

Hi

Do we know what causes this ? If we don’t I suggest we stop the builds for now and have someone investigate.

Maybe it is some strange indexing configuration in Jackrabbit ?

Regards
Felix

> Am 13.02.2015 um 05:04 schrieb Mark Thomas <[hidden email]>:
>
> Sling developers,
>
> We have just had a re-occurrence of the same problem.
>
> I will clean this up again this time but if it happens again I will
> simply remove the sling builds from buildbot.
>
> Mark
>
> On 22/01/2015 23:03, Mark Thomas wrote:
>> Sling developers,
>>
>> The sling-trunk CI build managed to kill one of the buildbot slaves by
>> filling this directory with files until the file system ran out of inodes:
>> /home/buildslave3/slave3/sling-trunk/build/testing/samples/integration-tests/sling/default/jackrabbit/workspaces/default/index
>>
>> There were so many files ls hung for 5+ minutes without any output.
>>
>> I have started to clean this up (rm -rf
>> /home/buildslave3/slave3/sling-trunk) and that looks like it is going to
>> take at least several hours to complete.
>>
>> The next CI build should re-checkout sling-trunk so your CI builds
>> should be unaffected. However, please could you take a look at the
>> buildbot configuration for this build and figure out a) why this
>> happened and b) how to stop it happening again.
>>
>> Cheers,
>>
>> Mark
>> on behalf of the ASF Infra team
>>
>

Reply | Threaded
Open this post in threaded view
|

Re: sling-trunk buildbot CI build causing problems

Robert Munteanu-3
On Mon, Feb 16, 2015 at 4:16 AM, Felix Meschberger <[hidden email]> wrote:
> - private@sling, infra@
>
> Hi
>
> Do we know what causes this ? If we don’t I suggest we stop the builds for now and have someone investigate.

+1

IIRC we set up buildbot since Jenkins was not stable, but IMO it got
much better in the last months.

Robert

>
> Maybe it is some strange indexing configuration in Jackrabbit ?
>
> Regards
> Felix
>
>> Am 13.02.2015 um 05:04 schrieb Mark Thomas <[hidden email]>:
>>
>> Sling developers,
>>
>> We have just had a re-occurrence of the same problem.
>>
>> I will clean this up again this time but if it happens again I will
>> simply remove the sling builds from buildbot.
>>
>> Mark
>>
>> On 22/01/2015 23:03, Mark Thomas wrote:
>>> Sling developers,
>>>
>>> The sling-trunk CI build managed to kill one of the buildbot slaves by
>>> filling this directory with files until the file system ran out of inodes:
>>> /home/buildslave3/slave3/sling-trunk/build/testing/samples/integration-tests/sling/default/jackrabbit/workspaces/default/index
>>>
>>> There were so many files ls hung for 5+ minutes without any output.
>>>
>>> I have started to clean this up (rm -rf
>>> /home/buildslave3/slave3/sling-trunk) and that looks like it is going to
>>> take at least several hours to complete.
>>>
>>> The next CI build should re-checkout sling-trunk so your CI builds
>>> should be unaffected. However, please could you take a look at the
>>> buildbot configuration for this build and figure out a) why this
>>> happened and b) how to stop it happening again.
>>>
>>> Cheers,
>>>
>>> Mark
>>> on behalf of the ASF Infra team
>>>
>>
>



--
Sent from my (old) computer