[blfs-support] Perl modules

[Paul Rogers]

The book presents a top layer in alphabetical order, with dependencies
underneath. So our job is made much more tedious and difficult, hand
(re)making the dependency trees to see which are lowest-level to be made
first. May I suggest that if the book order were from lowest level
first, installation order, it would be much easier for us, and eliminate
an error-prone process everybody has to go through separately.

Yes, I know the counter argument is somebody might not have to or want
to install all of them. How would they know that a-priori? I come to
this section with the assumption that if it's in the book that alone is
sufficient reason to install everything there--something or other I may
not discover until later will want me to have installed something here.
I might as well do it now along with everything else.

Failing that argument being persuasive, does anybody here HAVE a list of
the modules in the proper installation dependency order they could
contribute to me? I'd be ever so grateful. TIA
--
Paul Rogers
***@fastmail.fm
Rogers' Second Law: "Everything you do communicates."
(I do not personally endorse any additions after this line. TANSTAAFL
:-)

[Bruce Dubbs]

Post by Paul Rogers
The book presents a top layer in alphabetical order, with dependencies
underneath. So our job is made much more tedious and difficult, hand
(re)making the dependency trees to see which are lowest-level to be made
first. May I suggest that if the book order were from lowest level
first, installation order, it would be much easier for us, and eliminate
an error-prone process everybody has to go through separately.
Yes, I know the counter argument is somebody might not have to or want
to install all of them. How would they know that a-priori? I come to
this section with the assumption that if it's in the book that alone is
sufficient reason to install everything there--something or other I may
not discover until later will want me to have installed something here.
I might as well do it now along with everything else.
Failing that argument being persuasive, does anybody here HAVE a list of
the modules in the proper installation dependency order they could
contribute to me? I'd be ever so grateful. TIA

For perl modules, I generally used the cpan method. It still uses source, but
the builds, including dependencies, are automated.

My usual technique is to build up my system until I reach the need for the first
perl module. Then I install them all in alphabetical order.

-- Bruce

[Ken Moffat]

Post by Paul Rogers
The book presents a top layer in alphabetical order, with dependencies
underneath. So our job is made much more tedious and difficult, hand
(re)making the dependency trees to see which are lowest-level to be made
first. May I suggest that if the book order were from lowest level
first, installation order, it would be much easier for us, and eliminate
an error-prone process everybody has to go through separately.

Many of the dependent modules are external - some are not required for a
build of the module which uses them (e.g. in modules I build for
biber, a lot of extra modules are needed at each stage to get the
testsuites all working - and after my first experience with
biber-1.9 I now run the tests in the modules).

Post by Paul Rogers
Yes, I know the counter argument is somebody might not have to or want
to install all of them. How would they know that a-priori? I come to
this section with the assumption that if it's in the book that alone is
sufficient reason to install everything there--something or other I may
not discover until later will want me to have installed something here.
I might as well do it now along with everything else.

I suggest that you start by identifying which modules you want to
install (i.e. treat them like everything else in the book - some
final package has required and optional dependencies.

And please note that in some parts of the book there are
mutually-exclusive packages (the canonical example is mail servers -
the presence of a sendmail program is often checked for, but in my
case it comes from postfix).

Post by Paul Rogers
Failing that argument being persuasive, does anybody here HAVE a list of
the modules in the proper installation dependency order they could
contribute to me? I'd be ever so grateful. TIA

I build the proverbial shed-load of perl modules, but for different
things. A few of them get used for some audio-video stuff (e.g.
get_iplayer, which will be of no use to people accessing the BBC
from outside GB - unless they work around the restrictions, and for
what is now on the BBC it probably isn't worth it unless you are an
expat Briton), one or two get built on server (test) builds, but the
vast majority are for biber.

For the modules I build on the desktop, I can probably provide some
notes - but they are for my use cases. I'll reply to this in a
while when I'm back at a desktop system (the netbook is too
restrictive to manage this comfortable).

ĸen

[Ken Moffat]

Post by Ken Moffat
For the modules I build on the desktop, I can probably provide some
notes - but they are for my use cases. I'll reply to this in a
while when I'm back at a desktop system (the netbook is too
restrictive to manage this comfortable).

I build a few modules early on:

SGMLSpm gets built as my first main step after booting the new
system - I build all of what is in chapters 45 and 46 (I'm an editor
- I need this stuff).

XML-Simple gets built on the way to building Xorg.

Image::ExifTool (not in the book) gets built when I'm building tools
to make my photos more presentable (gimp, jhead, ufraw and its deps)

libwww-perl is needed for get-iplayer : I used to use an old version
(5.something) which had far fewer deps, but as an editor I get the
fun of trying to keep up to date :) - I'll attach part of my script.
This happens after AV libs have already been built.

Note that km_build is one of my functions to detect if the step
(usually a package) has been already built, and if not it will run
what I call the client script. And it will stop if the client
fails. Note that the reason for attaching it is to show how I
detect that a module is already installed : I only ever have _one_
perl version installed.

Near the end of my main build I install parole and xfburn (although
I very rarely use the latter) - it looks as if URI is needed,
probably for exo, and in a normal build it will already be
installed, but I check anyway:

test -f /usr/lib/perl5/site_perl/*/URI.pm ||
km_build URI ${KM_SCRIPTS}/perl.d/URI

I also do that if I'm doing a less-minimal xfce install (not full -
I fell out of love with moidori, and for the moment I can avoid
webkitgtk).

After that, there are two desktop scripts which build perl modules -
extra-biber (for biber, obviously) and extra-font-analysis (for
working out what is in fonts, particularly ttf/otf : I keep
promising myself I'm going to get back to doing that!). I'll attach
the full scripts because the comments and tests might be useful.
The biber script requires that I have already installed texlive -
nowadays I only build the source (although I'll still do at least
one binary install, on a lower-powered machine, when 2015 is
released). Obviously some of the boilerplate (particularly, the
test for ppc64) is out of date.

Similarly, extra-font-analysis gets built on a completed desktop
system: usually, I only do it on one machine. Again, most people
will not want this. For me, 'fret' gets used when I'm looking at a
new font to get an idea of what is in it - and fontforge is used if
I wish to extract a ttf from a ttc (TrueType Font and Collection) so
that I can check what it contains.

Äžen

[Paul Rogers]

Post by Bruce Dubbs
For perl modules, I generally used the cpan method. It still uses
source, but the builds, including dependencies, are automated.
My usual technique is to build up my system until I reach the need for
the first perl module. Then I install them all in alphabetical order.

The only one I've needed so far has been the XML-Parser, and I did that
one individually. AFAIK, there's no need on the immediate horizon, just
seems like a good point to take care of it--"in for a penny, in for a
pound." I did get and tarball the bunch at
http://anduin.linuxfromscratch.org/sources/BLFS/svn/perl-modules/.

What would be the best way to capture the order CPAN does things in,
thinking along the lines of getting it from a tee'd console log perhaps?
--
Paul Rogers
***@fastmail.fm
Rogers' Second Law: "Everything you do communicates."
(I do not personally endorse any additions after this line. TANSTAAFL :-)

[Bruce Dubbs]

Post by Paul Rogers

Post by Bruce Dubbs
For perl modules, I generally used the cpan method. It still uses
source, but the builds, including dependencies, are automated.
My usual technique is to build up my system until I reach the need for
the first perl module. Then I install them all in alphabetical order.

The only one I've needed so far has been the XML-Parser, and I did that
one individually. AFAIK, there's no need on the immediate horizon, just
seems like a good point to take care of it--"in for a penny, in for a
pound." I did get and tarball the bunch at
http://anduin.linuxfromscratch.org/sources/BLFS/svn/perl-modules/.
What would be the best way to capture the order CPAN does things in,
thinking along the lines of getting it from a tee'd console log perhaps?

That would work or you could just create a build tree manually from the
dependencies in the book.

-- Bruce

[Paul Rogers]

Post by Ken Moffat
Near the end of my main build I install parole and xfburn (although
I very rarely use the latter) - it looks as if URI is needed,
probably for exo, and in a normal build it will already be
test -f /usr/lib/perl5/site_perl/*/URI.pm ||
km_build URI ${KM_SCRIPTS}/perl.d/URI
. ${KM_SCRIPTS}/functions

Umm, at first blush, anything in km_build that'd be useful to me?
What's in functions?
--
Paul Rogers
***@fastmail.fm
Rogers' Second Law: "Everything you do communicates."
(I do not personally endorse any additions after this line. TANSTAAFL
:-)

[Ken Moffat]

Post by Paul Rogers

Post by Ken Moffat
Near the end of my main build I install parole and xfburn (although
I very rarely use the latter) - it looks as if URI is needed,
probably for exo, and in a normal build it will already be
test -f /usr/lib/perl5/site_perl/*/URI.pm ||
km_build URI ${KM_SCRIPTS}/perl.d/URI
. ${KM_SCRIPTS}/functions

Umm, at first blush, anything in km_build that'd be useful to me?
What's in functions?

The functions script contains all the things I reference. km_build
gets called with a name (URI in that case) and mostly one other parm
(script name to execute, any parms to pass to that, e.g. the target
name if I'm building blfs-bootscripts. It looks to see if I created
a stamp for this (in which case, the build/install completed) and
if not then it invokes the build using eval, and (a recent change)
invokes a km_failed script for non-zero status, so that even I
cannot overlook that e.g. openssl -j4 failed.

More generally, routines for deciding how to untar (nowadays tar
copes with everything, but when I started you had to pass -j or -z
or whatever) or unzip (and also to handle packages without a
directory - mostly zip files, but tzdata is like that). All sorts of
infrastructure for logging (I hesitate to say that is now perfect,
because I thought that a week ago but then found out that
/usr/bin/firefox (a symlink) was not getting handled correctly on an
upgrade) and for measuring space and time.

The routines for identifying the directory name might give people
useful ideas, and I generally use awk and sed : before this, I'd
been doing perl stuff to compare a build with the results of
building itself, and sometimes I miss that - but generally my perl
is very rusty, and in any case my scripts all need to work
throughout the build of LFS.

Much of it is specific to how I build (/sources is an nfs mount
here). And, of course, things to measure space and time (in SBUs)
although I would not use those measurements for an edit.

So no, it is unlikely to be useful to you. At one time, I used to
upload my scripts from time to time, but I don't think anybody was
ever interested.

And just in case it isn't obvious - my scripts have always been
buggy. The important thing is to keep them working well-enough, and
to understand the failure modes. So writing your own is encouraged.
Some of it is anachronistic (e.g. I start log names with desktop-
or general- once I've built the packages I always need, which are in
a separate pre-boot script and built from chroot). And I prize
verbosity, so that in most cases I can look at the term and see
where I am (but for slow packages, knowing the package name is not
especially helpful).

But am I ashamed of any of the functions ? No, not enough to worry.
I'm sure most people don't want to read all 31K, so I'll attach a
.xz version.

These are designed for my normal urxvt term (100 characters wide),
and I deliberately put redundant whitespace in some places, because
my .vimrc has

"for redundant whitespace, from Jeremy Kerr via Olof Johanson
"highlight RedundantWhitespace ctermbg=red guibg=red
"match RedundantWhitespace /\s\+$\| \+\ze\t/
highlight RedundantSpaces term=standout ctermbg=red guibg=red
match RedundantSpaces /\s\+$\| \+\ze\t/

(found on lkml) and that allows me to use the red markings as an
identifier of things I wish to highlight, or for debugging stuff.
That works in the term and in urxvt (I think my vim colour scheme is
elflord) - I normally use a black background in terms - but on my
current experimental built I have real xterm (usually, it's a
symlink to urxvt) and there it does not work.

ISTR that when I uploaded the earlier versions I labelled them as
GPL v2 or later, but the WTFPL would be equally valid (some places
do not allow Public Domain). So it's like anything else you find on
the web - free to use, maybe sort-of works, and you will need to do
the legwork to solve any problems.

If anybody is misguided enough to try this, 'template' is what I
typically use as the base for a new package.
https://en.wikipedia.org/wiki/Phrases_from_The_Hitchhiker%27s_Guide_to_the_Galaxy#Share_and_Enjoy

Äžen

[Ken Moffat]

Post by Ken Moffat
And just in case it isn't obvious - my scripts have always been
buggy.

[...]

Post by Ken Moffat
But am I ashamed of any of the functions ? No, not enough to worry.
I'm sure most people don't want to read all 31K, so I'll attach a
.xz version.

And saturday/sunday I used it to build LFS - some obvious changes
are needed for chapter 5 ($LFS/var does not exist) and I had some
odd results on the way into chroot which might be related to some
procedural errors after the first failure (I _need_ egrep, which is
a script using /bin/sh, plain grep is inadequate for that approach).
Actually, I suppose I could use 'grep -e' : didn't think of that!

But the logs for coreutils omitted quite a few programs - I had
overlooked what the traditional form of 'diff' does when some changed
lines in a file have extra (or, perhaps, deleted) lines amongst them.

So, definitely buggy and I'm sure that could also apply in BLFS if
somebody built a package, then later rebuilt it after adding extra
dependencies.

Oh well, that just means it will be a bit longer before I'm ready to
do the tickets I have. As with all coding, scripts need to be
tested.

ĸen

[Ken Moffat]

Post by Ken Moffat
I suppose I could use 'grep -e' : didn't think of that!

Wrong, of course (look at man grep or the egrep script to spot the
error)

ĸen

[Bruce Dubbs]

Post by Ken Moffat

Post by Ken Moffat
I suppose I could use 'grep -e' : didn't think of that!

Wrong, of course (look at man grep or the egrep script to spot the
error)

I didn't look. grep -E ?

-- Bruce

[Fernando de Oliveira]

Post by Bruce Dubbs

Post by Ken Moffat

Post by Ken Moffat
I suppose I could use 'grep -e' : didn't think of that!

Wrong, of course (look at man grep or the egrep script to spot the
error)

I didn't look. grep -E ?

Neither did I.

I use it almost every day:

grep -vE 'html|archive|blfs-full'

to avoid these results when grepping something in BLFS/trunk/BOOK.

Another daily use:

for i in usr/bin/*
do
echo $i && ldd $i | grep -E 'nspr|libplc|libplds'
done; unset i

where I change 'nspr|libplc|libplds' to meet the needs of present
package being analysed.

[Fernando de Oliveira]

Post by Fernando de Oliveira

Post by Bruce Dubbs

Post by Ken Moffat

Post by Ken Moffat
I suppose I could use 'grep -e' : didn't think of that!

Wrong, of course (look at man grep or the egrep script to spot the
error)

I didn't look. grep -E ?

Neither did I.
grep -vE 'html|archive|blfs-full'
to avoid these results when grepping something in BLFS/trunk/BOOK.
for i in usr/bin/*
do
echo $i && ldd $i | grep -E 'nspr|libplc|libplds'
done; unset i
where I change 'nspr|libplc|libplds' to meet the needs of present
package being analysed.

FWIW, this post was meant as a joke between editors.

I never compare Ken knowledge with mine. You, Ken, have much deeper and
more extensive knowledge than me, in these matters. This was a rare
occasion where I daily use some basic knowledge that you know, but had
just forgotten the option was with capital E.

Bruce's post surely was a joke, too.

[Ken Moffat]

Post by Fernando de Oliveira

Post by Fernando de Oliveira

Post by Bruce Dubbs

Post by Ken Moffat

Post by Ken Moffat
I suppose I could use 'grep -e' : didn't think of that!

Wrong, of course (look at man grep or the egrep script to spot the
error)

I didn't look. grep -E ?

Neither did I.
grep -vE 'html|archive|blfs-full'
to avoid these results when grepping something in BLFS/trunk/BOOK.
for i in usr/bin/*
do
echo $i && ldd $i | grep -E 'nspr|libplc|libplds'
done; unset i
where I change 'nspr|libplc|libplds' to meet the needs of present
package being analysed.

FWIW, this post was meant as a joke between editors.
I never compare Ken knowledge with mine. You, Ken, have much deeper and
more extensive knowledge than me, in these matters. This was a rare
occasion where I daily use some basic knowledge that you know, but had
just forgotten the option was with capital E.
Bruce's post surely was a joke, too.

Personally, I found the 'foo|bar|baz' style of your example
educational, and that is why we are here.

And I don't have particularly extensive knowledge, but I have been
lucky enough to be able to devote time to LFS for a few years, and
I'm slowly learning from my errors. We do what we can, we all have
different strengths.

Going back to my scripts, which is where this started, I think they
are now *adequately* logging the installed and modified files (and
any that were deleted - in a running system, those are normally not
related to the build). I now use a hack to not try to log modified
or deleted files when installing /etc/passwd and /etc/groups (still
don't understand those issues) and I manage to record /mnt/lfs/etc
as created (I do these before entering chroot, so I can run that in
one go) although passwd is correctly shown as '/etc/passwd'.

My big change this week has been using comm (I'd never heard of it
before) to compare the results of 'ls' (directories) and 'ls -l'
(files, to pick up modified files. Oh, and in British Summertime
(UTC +1) the 'ls -l output' shows different times before and after
coreutils is installed, both in chapter 5 and in chapter 6. Solved
by 'find ... | grep -v ... | TZ=UTC ls -l >listing'. Very
educational for me, and a lot of false ideas along the way.

ĸen

[Paul Rogers]

I played around some. I did the CPAN with the console teed (and my
package manager so I can rip it all out), then one can grep the
"CPAN.pm: Building xyz" lines it announces when it starts a
module--except that then it discovers all the unsatisfied dependencies,
stacks this, does those, then pops it when they're all done--so ignore
the first of any pairs. That should give one a proper order. I got
most of the way there, but there's a problem.

Finance::Quote has a bunch of dependencies, so I followed that to see
what went on. It fails to install. There's a problem with its use of
MSchilli's IO::Socket::SSL v1.5.4, and the CPAN version is v2.something.
(Not the first time I've seen that name! ;-) )

I have to think about this, whether I want to go ahead, what to do about
the failures. Unless you have a fix, Finance::Quote maybe shouldn't be
in the book.
--
Paul Rogers
***@fastmail.fm
Rogers' Second Law: "Everything you do communicates."
(I do not personally endorse any additions after this line. TANSTAAFL
:-)

[Paul Rogers]

Obviously, I cannot speak for packages which are outside the book. But
for everything non-perl that is in the book, we either
require/recommend any perl modules it uses, in which case they are in
the book, or we note tham as optional, and those might have external
references.

What I have is what was on anduin back in January. In looking at the
CPAN console log some more, it's still a bit confusing. But usually
just letting puzzles stew in my brain for a while results in a solution
of some sort.

And although I rarely use a desktop, there's XFCE to install! Maybe by
then I'll have a plan.

It is hard to compare python, because there is no single unified
source. Personally, I find it hard to find the source even when I
think I know the module name, and fairly recently I was looking for
something that turned out to have been defunct for a few years (there
was a different module which replaced it for that application).

Yes, making it just that much harder trying to make a "basic module
library". I can see now why you wait for need.

And unless you use modern forms of \TeX you will have no use for biber
which is the package responsible for many of the perl modules

I might have had some use for some of the documentation, but if I don't
"have to have" I generally avoid packages.

in BLFS. Most other BLFS packages using perl do not need many
modules, but anything using libwww-perl pulls in a load.

Seems like that one has some utility. I'm inclined to give it the
benefit of the doubt.

Thanks, Ken, for your input. It has been instructive.

I have an idea that may make the tree from what's in the book's
files in less than exponential time. But I'll just leave that to
stew for a while. I don't generally NEED a desktop environment, but
it is expected, isn't it. ;-) After that I'll see what I think
about perl modules.
--
Paul Rogers
***@fastmail.fm
Rogers' Second Law: "Everything you do communicates."
(I do not personally endorse any additions after this line. TANSTAAFL :-)

[Ken Moffat]

Post by Paul Rogers

It is hard to compare python, because there is no single unified
source. Personally, I find it hard to find the source even when I
think I know the module name, and fairly recently I was looking for
something that turned out to have been defunct for a few years (there
was a different module which replaced it for that application).

Yes, making it just that much harder trying to make a "basic module
library". I can see now why you wait for need.

I think you misinterpreted what I wrote. That paragraph was a
comment on python.

ĸen

[Paul Rogers]

Post by Paul Rogers
Yes, making it just that much harder trying to make a "basic module
library". I can see now why you wait for need.

I think you misinterpreted what I wrote. That paragraph was a comment
on python.

Perhaps, but I found it true in the CPAN also. As reported, the
Finance::Quote module has some obsolete dependencies as I read the

Post by Paul Rogers

Post by Ken Moffat
I suggest that you start by identifying which modules you want to
install (i.e. treat them like everything else in the book - some
final package has required and optional dependencies.

--
Paul Rogers
***@fastmail.fm
Rogers' Second Law: "Everything you do communicates."
(I do not personally endorse any additions after this line. TANSTAAFL
:-)

[Paul Rogers]

Plus, for a lot of people, Python is just a much nicer language to
code in. It's a bit more verbose - it doesn't have as much of the
concise little syntax that Perl does for some common tasks - but
overall more readable.

HAHAHAHA! You remind me of a project I worked on 35 years ago on a
USNavy contract. It was required we code in "CMS-2", the Navy's
standard programming language. The best way I found to describe CMS-2
was: "Imagine an old COBOL programmer had heard about Pascal, and
decided to write a compiler for such a language. CMS-2 would be the
language he imagined." Of course, interpretting that means one would
have had to have read COBOL sometime, and there's a generation of
programmers who never have.
--
Paul Rogers
***@fastmail.fm
Rogers' Second Law: "Everything you do communicates."
(I do not personally endorse any additions after this line. TANSTAAFL
:-)

[Paul Rogers]

related to the build). I now use a hack to not try to log modified or
deleted files when installing /etc/passwd and /etc/groups (still don't
understand those issues) and I manage to record /mnt/lfs/etc as
created (I do these before entering chroot, so I can run that in one
go) although passwd is correctly shown as '/etc/passwd'.

Ken, FWIW in my (B)LFS build scripts I have five functions, describe,
setup, build, finish, remove, and a case statement at the end to take
an argument for what to do (with a 6th option "all" which is setup,
build, finish).

The idea is: build is pretty much what the book says to do, produce
"binaries" that I can restore when I'm cloning a fresh system. It
starts my package manager watching just before "make install", then
generates the "removal script", and tarballs all those files just
before the function ends. Most often setup is the useradd/groupadd
stuff, because restoring /etc/passwd is history dependent. Finish is
usually where I customize config files. Remove is where I undo what
setup & finish did before having my package manager remove the binaries
it saw being installed, i.e. userdel, rm /etc/yadayada.conf. When I am
cloning, my clone script runs the setup function, restores the
immutable binaries saved by my package manager at the end of build, and
then finish function. This separation gives me enough flexibility to
use the same scripts for building and cloning. It recognizes there are
some things that can and some that can't be just laid down from an
original tarball.

My package manager does recognize new files from changed files, and
won't remove one that was already there. But neither will it restore
the file. It's up to me to decide to restore the original package, if
that's right. Maybe yes, maybe no.

And everything is in bash scripts where I can get at it when I
need to! ;-)
--
Paul Rogers
***@fastmail.fm
Rogers' Second Law: "Everything you do communicates."
(I do not personally endorse any additions after this line. TANSTAAFL :-)

[Ken Moffat]

Post by Paul Rogers
My package manager does recognize new files from changed files, and
won't remove one that was already there. But neither will it restore
the file. It's up to me to decide to restore the original package, if
that's right. Maybe yes, maybe no.

I'm impressed - until recently I thought that my scripts were
correctly identifying changed files, but playing with new versions
of perl modules showed me they were not picking up the .pm modules
(all dated in the past). Now I'm back in business, at least until
the next bug shows up.

Post by Paul Rogers
And everything is in bash scripts where I can get at it when I
need to! ;-)

Good :-)

ĸen