[tei-council] Speeding up Jinks builds

[Martin Holmes]

On 12-10-26 09:03 AM, Sebastian Rahtz wrote:
>
> On 26 Oct 2012, at 16:43, Martin Holmes <mholmes at uvic.ca> wrote:
>>
>>   - I'm not sure why P5-Documentation and P5 are two separate jobs.
>> Could they be combined? I think that would be quicker in that it would
>> save copying over lots of stuff from the workspace of one build to the
>> other.
>
> hmm. I don't know why we did 3 jobs instead of 2. I can't see a downside
> to removing Documentation, and letting P5 do the work.

Let's do that then. In the next few days I'll summarize this discussion 
into a ticket, and get started on it in a month or so, when lots of 
other stuff is out of the way.

>>   - A few months ago, all builds were going much more quickly on both
>> Jinkses. I can't think what's changed, except for the move to XInclude
>> from entities. Is it possible that makes a difference? It might be worth
>> doing some tests to find out, and if it does, investigate a way of doing
>> the XIncludes more quickly.
>
> can't explain it, sorry. I am sure its not the XIncludes - that happens once,
> early on, and thereafter everyone uses the same single file.  that should
> have made it faster, not slower.
>
> someone could put in timing messages and see which part of the build
> takes time. I am not volunteering :-}

One benefit of having two Jinkses is that we can use mine for this kind 
of testing and still have yours working normally.

>>   - Both Jinkses are running on virtual machines, so we can presumably
>> push up the memory or change the processing for the VMs if that will
>> help.
>
> maybe.  my gut feeling is that looking at the work its doing is more benefit,
> but i may well be wrong
>
>>   - I suspect there are lots of tweaks I could make to how memory is
>> allocated to ant tasks etc., and those could also be integrated into the
>> Jinks server build script.
> i am curious. why do you think these are an issue?

I keep having to add memory to java processes (like with jing 
yesterday). I haven't made any tweaks for ant, though, so I thought it 
might be worth looking at.

>> Anything else anyone can think of that would help speed up the Jinkses?
>> We're committed to steadily adding more and more tests to P5-Test
>> (Schematron etc.), so that build is going to take progressively longer;
>> anything we can do to mitigate that would be really helpful.
>
> we need profiling. which components of the build take the time?
>
> my observation of slow bits are:
>
>    - the jobs needed for each test file
>    - running LaTeX 3 times

I wonder if XSL:FO and FOP would be faster? (I know that's a HUGE task.)

>    - making .mobi file

That's kindlegen, right? My Jinks isn't doing that, because it's trying 
to be completely open-source, but my jobs are still slower than yours.

>    - validating all the HTML files

Have to do that. :-) But you're right, it takes ages.

Cheers,
Martin

>
> --
> Sebastian Rahtz
> Director (Research Support) of Academic IT Services
> University of Oxford IT Services
> 13 Banbury Road, Oxford OX2 6NN. Phone +44 1865 283431
>
> .
>

-- 
Martin Holmes
University of Victoria Humanities Computing and Media Centre
(mholmes at uvic.ca)