[blfs-support] WebKitGTK+-2.22.3 resource build use is insane

[Christopher Gregory via blfs-support]

Hello,

I have just built WebKitGTK+-2.22.3 and it was the most unpleasant experience I have ever had with building blfs. The build killed off the process a total of 4 times due to running out of memory. I have a 6 core machine and 8 gig of ram with a swap partition of 6 gigs.

This is the first time that ninja has been used in the build process, so I do not know if that is the problem or not. This is my first build on an ssd drive, so I have no idea if swap even gets used as an ssd drive is basicly a giant flash drive.

With this build I did not even have a gui running. This was done solely from directly on console. Also no xwindows.

The build time was also considerably longer, at about 7 hours. On all my other builds of webkit I have done them either in chroot, or via a terminal in lxde.

I think for the next build I will be dropping ninja and directly using cmake with make install.

Has anyone else had this experience with building webkit with ninja?, or has webkit grown so obese that it now requires more than 8 gigs of ram to compile?

Regards,

Christopher

[renodr via blfs-support]

Post by Christopher Gregory via blfs-support
Hello,
I have just built WebKitGTK+-2.22.3 and it was the most unpleasant
experience I have ever had with building blfs. The build killed off
the process a total of 4 times due to running out of memory. I have a
6 core machine and 8 gig of ram with a swap partition of 6 gigs.
This is the first time that ninja has been used in the build process,
so I do not know if that is the problem or not. This is my first
build on an ssd drive, so I have no idea if swap even gets used as an
ssd drive is basicly a giant flash drive.
With this build I did not even have a gui running. This was done
solely from directly on console. Also no xwindows.
The build time was also considerably longer, at about 7 hours. On all
my other builds of webkit I have done them either in chroot, or via a
terminal in lxde.
I think for the next build I will be dropping ninja and directly using
cmake with make install.
Has anyone else had this experience with building webkit with ninja?,
or has webkit grown so obese that it now requires more than 8 gigs of
ram to compile?
Regards,
Christopher

Try using "export NINJAJOBS=1" and recompile WebKit.

[Ken Moffat via blfs-support]

Post by renodr via blfs-support

Post by Christopher Gregory via blfs-support
Hello,
I have just built WebKitGTK+-2.22.3 and it was the most unpleasant
experience I have ever had with building blfs. The build killed off
the process a total of 4 times due to running out of memory. I have a
6 core machine and 8 gig of ram with a swap partition of 6 gigs.

Try using "export NINJAJOBS=1" and recompile WebKit.

I thought I would try this on my 4-core machine - unfortunately, I
accidentally booted the old BLFS-8.3 system instead of the later
build, and had installed all the added extra dependencies before I
realised that.

This is a ryzen3 with 4 cores, and according to top 7.8 GiB of
memory and 6.0 GiB of swap.

Using 2.22.3 with the book's instructions for 2.22.2, and not
forcing NINJAJOBS because I figured running 4+2 jobs would probably
be ok, even if slightly slower than forcing 4). A DESTDIR install.

I was using X, firefox with about 34 tabs, falkon with about 5 tabs,
and 6 instances of urxvt.

These figures from free, and from du for the directory space,
probably give a fairly accurate indication of the space used.

Max real mem noted 7214692
Initial real mem 1363476 (includes the extracted tarball)
change 5851219
max swap 1339648
approx total mem 7190857 plus the build space
added directory space 616852
total 7807709 (7624MiB)

wkit space starts at 181884 (178MiB)
after install 658408 (643MiB)
change 476524
and the install 140328 (128MiB)
directory space at end 616852 (602MiB)

Summary - with (only) 4 jobs, 8GB RAM might be enough if nothing
else is running.

I'm _surprised_ that Christopher's machine OOM'd with a total of
14GB RAM+swap and "only" 6 cores (that implies 8 jobs by default in
ninja) - I guess that hyperthreading, if available, was turned off
so that 'top' only shows 8 CPUs.

ĸen

[Christopher Gregory via blfs-support]

Sent: Saturday, November 10, 2018 at 11:27 AM
Subject: Re: [blfs-support] WebKitGTK+-2.22.3 resource build use is insane

Post by renodr via blfs-support

Post by Christopher Gregory via blfs-support
Hello,
I have just built WebKitGTK+-2.22.3 and it was the most unpleasant
experience I have ever had with building blfs. The build killed off
the process a total of 4 times due to running out of memory. I have a
6 core machine and 8 gig of ram with a swap partition of 6 gigs.

Try using "export NINJAJOBS=1" and recompile WebKit.

I thought I would try this on my 4-core machine - unfortunately, I
accidentally booted the old BLFS-8.3 system instead of the later
build, and had installed all the added extra dependencies before I
realised that.
This is a ryzen3 with 4 cores, and according to top 7.8 GiB of
memory and 6.0 GiB of swap.
Using 2.22.3 with the book's instructions for 2.22.2, and not
forcing NINJAJOBS because I figured running 4+2 jobs would probably
be ok, even if slightly slower than forcing 4). A DESTDIR install.
I was using X, firefox with about 34 tabs, falkon with about 5 tabs,
and 6 instances of urxvt.
These figures from free, and from du for the directory space,
probably give a fairly accurate indication of the space used.
Max real mem noted 7214692
Initial real mem 1363476 (includes the extracted tarball)
change 5851219
max swap 1339648
approx total mem 7190857 plus the build space
added directory space 616852
total 7807709 (7624MiB)
wkit space starts at 181884 (178MiB)
after install 658408 (643MiB)
change 476524
and the install 140328 (128MiB)
directory space at end 616852 (602MiB)
Summary - with (only) 4 jobs, 8GB RAM might be enough if nothing
else is running.
I'm _surprised_ that Christopher's machine OOM'd with a total of
14GB RAM+swap and "only" 6 cores (that implies 8 jobs by default in
ninja) - I guess that hyperthreading, if available, was turned off
so that 'top' only shows 8 CPUs.
ĸen
--
Is it about a bicycle ?
--
http://lists.linuxfromscratch.org/listinfo/blfs-support
FAQ: http://www.linuxfromscratch.org/blfs/faq.html
Unsubscribe: See the above information page

Hello Ken,

I tried the build without ninja, the way that I have built webkit in the past on this machine, with make -j12 and it ran out of memory as well. I ran make -j9 and there were no more out of memory errors, and it built in around 2 hours 20 mins, whereas with ninja it was 7 hours. I am not sure if hyperthreading is on or off.

For me personally, I see absolutely no reason to use ninja for as long as they provide cmake as the build. It is faster for me and less hassle to just drop ninja. I had a look at the ddr3 ram prices, and have noticed that here they are on the climb at $124 for 8 gig. I am still curious to know if swap even gets used on a solid state drive.

I guess I will have to get some more ram before ddr3 goes out of stock due to being obsolete. I am not able to afford the ryzen upgrade kits as they are around $1700NZD.

Regards,

Christopher.

[Ken Moffat via blfs-support]

Post by Christopher Gregory via blfs-support

Sent: Saturday, November 10, 2018 at 11:27 AM
Summary - with (only) 4 jobs, 8GB RAM might be enough if nothing
else is running.
I'm _surprised_ that Christopher's machine OOM'd with a total of
14GB RAM+swap and "only" 6 cores (that implies 8 jobs by default in
ninja) - I guess that hyperthreading, if available, was turned off
so that 'top' only shows 8 CPUs.

Hello Ken,
I tried the build without ninja, the way that I have built webkit in the past on this machine, with make -j12 and it ran out of memory as well. I ran make -j9 and there were no more out of memory errors, and it built in around 2 hours 20 mins, whereas with ninja it was 7 hours. I am not sure if hyperthreading is on or off.

The reason for now using ninja on cmake packages is that it usually
appears to be faster (with recent cmake - it only used to be faster
if rebuilding after a change to a built or part-built source).

But webkitgtk is big, and anything using C++ uses a lot of memory
for each running compile.

I suspect that when you said 6 cores, you meant 6 real cores and in
fact hyperthreading is on, giving 12 apparent cores. My settings in
'top' list a line for each online core, but that might not be the
default. The alternative is to look at /sys/devices/system/cpu to
see how many are reported, or to use lscpu 'the On-line CPU(s)
list:' line, or just 'less /proc/cpuinfo'.

As to the old "use twice the number of cores for the number of make
jobs" advice - that seems to work adequately for the kernel, but all
the code there is C, not C++. When I have tested bigger numbers of
parallel make jobs in big desktop applications, my general
impression has been that N or N+1 is best.

Post by Christopher Gregory via blfs-support
For me personally, I see absolutely no reason to use ninja for as long as they provide cmake as the build. It is faster for me and less hassle to just drop ninja. I had a look at the ddr3 ram prices, and have noticed that here they are on the climb at $124 for 8 gig. I am still curious to know if swap even gets used on a solid state drive.

For swap - yes, it does get used (my figures were from a machine with
only a single 480GB SSD).

As to buying DDR3, whether it is worthwhile depends on the specs - I
have another machine (now mostly powered off) where it uses, I
think, DDR3-1866, 2x4GB. I got 2x8GB but it was slower (and
unstable at expected speeds), and running tests to see how long
things took to build was very disappointing so I went back to the
2x4GB.

Post by Christopher Gregory via blfs-support
I guess I will have to get some more ram before ddr3 goes out of stock due to being obsolete. I am not able to afford the ryzen upgrade kits as they are around $1700NZD.
Regards,
Christopher.

Unfortunately, for some use-cases (e.g. rustc testsuite, and
firefox's Python2 build system) my experience with ryzen is
variable. And of course it's single-threaded performance is
comparatively poor. I think intel chips ramp up their frequency more
quickly, so moving a task across CPUs - I watch 'top' to see that -
appears to cause slow builds as a result. But series-2 ryzens
should be better for the speed.

ĸen