Installation guide is not updated in some languages

[Hideki Yamane]

Hi (please CC:me),

Could someone tell me why installation guide for Stretch in
some languages is not updated, please?

OK)
https://www.debian.org/releases/testing/amd64/index.html.en
https://www.debian.org/releases/testing/amd64/index.html.it
https://www.debian.org/releases/testing/amd64/index.html.el
https://www.debian.org/releases/testing/amd64/index.html.cs
https://www.debian.org/releases/testing/amd64/index.html.es
https://www.debian.org/releases/testing/amd64/index.html.de
https://www.debian.org/releases/testing/amd64/index.html.fr

Not update)
https://www.debian.org/releases/testing/amd64/index.html.ja
https://www.debian.org/releases/testing/amd64/index.html.sv
https://www.debian.org/releases/testing/amd64/index.html.ko
https://www.debian.org/releases/testing/amd64/index.html.zh-cn
https://www.debian.org/releases/testing/amd64/index.html.ru
https://www.debian.org/releases/testing/amd64/index.html.pt
https://www.debian.org/releases/testing/amd64/index.html.vi

[Hideki Yamane]

Hi,

On Sat, 3 Jun 2017 11:07:13 +0900

This is not here. Please contact d-i folks. They build and publish
them as I understand.
Let me explain...

(snip)

This points to https://www.debian.org/devel/debian-installer/ which
states the latest version is at: https://d-i.debian.org/manual/

Oh, those are updated one. I should ask d-i team as you said.
Thank you very much!

[Holger Wansing]

[debian-boot in CC]

Post by Hideki Yamane
Hi,
On Sat, 3 Jun 2017 11:07:13 +0900

This is not here. Please contact d-i folks. They build and publish
them as I understand.
Let me explain...

(snip)

This points to https://www.debian.org/devel/debian-installer/ which
states the latest version is at: https://d-i.debian.org/manual/

Oh, those are updated one. I should ask d-i team as you said.
Thank you very much!

Basically there are different methods in building the guide:
https://d-i.debian.org/manual/ is built from svn, while the one at
https://www.debian.org/releases/testing/ is built from a package version
in the archive.

However, there is no such version in the archive, which has that content
shown on https://www.debian.org/releases/testing/. This can be easily seen
at the copyright term, as an example:
https://www.debian.org/releases/testing/amd64/index.html.en shows
"Copyright © 2004 – 2017 the Debian Installer team" (which is correct),
and
https://www.debian.org/releases/testing/amd64/index.html.ja shows
"製作著作 © 2004 – 2015 the Debian Installer team" (which is not correct).

But the version 20170525
(https://packages.debian.org/stretch/installation-guide-amd64)
has
"製作著作 © 2004 – 2017 the Debian Installer team" in
../ja/index.html.

So the issue is within the webwml/english/doc/ code, that publishs the
manual onto the website.



Holger

[Holger Wansing]

Hi,

It's in cron for lessoften. It downloads the source package and runs
./buildweb.sh found in build directory. This is d-i team to update if
it is caused by it.
lessoften.log at https://www-master.debian.org/build-logs/webwml/
run-parts: executing
/srv/www.debian.org/cron/lessoften-parts/1installation-guide
no new installation-guide for stretch, skipping build
Hmmm... the fetch script 1installation-guide uses ftp to get the latest
source package.
Anyway, the last source package upload is [2017-05-25] Accepted 20170525
in unstable (medium) (Samuel Thibault). So there should be no update
since then.
So with new testing/unstable upload, this should be updated.

If everything had worked fine, it should have been updated 1-2 weeks ago,
when 20170525 version was uploaded, right?
The content at
https://www.debian.org/releases/testing/amd64/index.html.ja
for example is from a much older version.

So there seems to be some problem somewhere ...

I had similar problems with the manual build at
http://d-i.alioth.debian.org/manual/:
some files were not rsynced to alioth; I had to clean out all those problematic
files from the path, and then they were newly and correctly created via rsync.
Maybe someone could try that? (I'm lacking permissions for that.)


Holger

[Samuel Thibault]

Hello,

Post by Holger Wansing
If everything had worked fine, it should have been updated 1-2 weeks ago,
when 20170525 version was uploaded, right?
The content at
https://www.debian.org/releases/testing/amd64/index.html.ja
for example is from a much older version.
So there seems to be some problem somewhere ...
I had similar problems with the manual build at
some files were not rsynced to alioth; I had to clean out all those problematic
files from the path, and then they were newly and correctly created via rsync.
Maybe someone could try that? (I'm lacking permissions for that.)

It could also be some sort of timeout or ENOSPC: the outdated pages
match exactly the end of the language list.

Actually I don't know: how are the www.debian.org pages generated? I
guess it's the buildweb.sh script, but on which host? How is it
triggered?

Samuel

[Samuel Thibault]

Post by Samuel Thibault
Actually I don't know: how are the www.debian.org pages generated? I
guess it's the buildweb.sh script, but on which host? How is it
triggered?

The installation-guide documentation says

The check for a new version in unstable is done by the 'lessoften'
script. This will fetch the source tarball for the new version, extract
that and then calls the build/buildweb.sh script.

But I don't know what "lessoften" is.

Samuel

[Samuel Thibault]

Post by Samuel Thibault

Post by Samuel Thibault
Actually I don't know: how are the www.debian.org pages generated? I
guess it's the buildweb.sh script, but on which host? How is it
triggered?

The installation-guide documentation says
The check for a new version in unstable is done by the 'lessoften'
script. This will fetch the source tarball for the new version, extract
that and then calls the build/buildweb.sh script.
But I don't know what "lessoften" is.

Apparently that happens on www-master.

/srv/www.debian.org/installmanual/stretch.log

shows:

kpathsea: Running mktextfm VL-PGothic-Regular/OT
mkdir: cannot create directory 'build.tmp.ja.i386/dblatex/mt22600.tmp': No such file or directory
xelatex failed

I don't know why that happens, investigating, perhaps it's just because
it's a Jessie box only.

Samuel

[Samuel Thibault]

Post by Samuel Thibault

Post by Samuel Thibault

Post by Samuel Thibault
Actually I don't know: how are the www.debian.org pages generated? I
guess it's the buildweb.sh script, but on which host? How is it
triggered?

The installation-guide documentation says
The check for a new version in unstable is done by the 'lessoften'
script. This will fetch the source tarball for the new version, extract
that and then calls the build/buildweb.sh script.
But I don't know what "lessoften" is.

Apparently that happens on www-master.
/srv/www.debian.org/installmanual/stretch.log
kpathsea: Running mktextfm VL-PGothic-Regular/OT
mkdir: cannot create directory 'build.tmp.ja.i386/dblatex/mt22600.tmp': No such file or directory
xelatex failed
I don't know why that happens, investigating, perhaps it's just because
it's a Jessie box only.

Some build dependencies are missing on www-master:

fonts-wqy-microhei fonts-vlgothic

Samuel

[Samuel Thibault]

Hello doc people,

Post by Samuel Thibault
fonts-wqy-microhei fonts-vlgothic

Could you apply the attached patch on www-master, so that we way more
easily catch the issue?

And also please run

touch -t 201701010000 /srv/www.debian.org/installmanual/stretch.log

so that lessoften triggers the build again.

Thanks,
Samuel

[Samuel Thibault]

Hello doc people,

Post by Samuel Thibault

Post by Samuel Thibault
fonts-wqy-microhei fonts-vlgothic

Could you apply the attached patch on www-master, so that we way more
easily catch the issue?

Ping?

Post by Samuel Thibault
And also please run
touch -t 201701010000 /srv/www.debian.org/installmanual/stretch.log
so that lessoften triggers the build again.

That should now get it to succeed, I have tested with the amd64 arch.
Could some doc people do this one at last?

Thanks,
Samuel

[Cyril Brulebois]

Hi,

Post by Samuel Thibault

Post by Samuel Thibault

Post by Samuel Thibault
fonts-wqy-microhei fonts-vlgothic

Could you apply the attached patch on www-master, so that we way
more easily catch the issue?

Ping?

I'm working on a different patch to get mail notifications whenever the
build fails.

However, seeing the cron.git repository, I think you wanted debian-www?

Post by Samuel Thibault

Post by Samuel Thibault
And also please run
touch -t 201701010000 /srv/www.debian.org/installmanual/stretch.log
so that lessoften triggers the build again.

That should now get it to succeed, I have tested with the amd64 arch.
Could some doc people do this one at last?

I've poked #debian-www to see whether it might be a good idea to get a
debwww sudo a bit wider than just update-part on devel/debian-installer
for me.


KiBi.

[Paul Wise]

Post by Cyril Brulebois
However, seeing the cron.git repository, I think you wanted debian-www?

Some folks on debian-doc have debwww access IIRC.

Post by Cyril Brulebois
I've poked #debian-www to see whether it might be a good idea to get a
debwww sudo a bit wider than just update-part on devel/debian-installer
for me.

I've added the ability to run the installation-guide script as debwww
for the d-i group, but that would not have helped in this case because
the script checks that a new upload of the installation-guide was done
since the last build. I manually removed the log and rebuilt it to
work around that.

At this point I think the right solution here is to be doing builds
during package build time and shipping the results into packages,
which are then unpacked on www-master.d.o. IIRC this is how other
documents on the website are built, so I'm not sure why the
installation-guide is different.

[Samuel Thibault]

Hello,

BTW, I have just updated the list of languages in
english/releases/stretch/release.data . I guess something needs to be
run to update
https://www.debian.org/releases/testing/installmanual.en.html
to include the new languages.

Also, with that run the PDF versions of the chinese, japanese, and
vietnamese translations of the manual should be showing up now that the
PDF files got built.

Samuel

[Holger Wansing]

Hi,

Post by Samuel Thibault
Hello,
BTW, I have just updated the list of languages in
english/releases/stretch/release.data . I guess something needs to be
run to update
https://www.debian.org/releases/testing/installmanual.en.html
to include the new languages.
Also, with that run the PDF versions of the chinese, japanese, and
vietnamese translations of the manual should be showing up now that the
PDF files got built.

Looks good, thanks!


Holger

[Cyril Brulebois]

Hi,

Hmmm... I don't have shell access but I can run this command via cron.
https://www.debian.org/releases/testing/amd64/index.html.ja
This is good work around for now. But this is very high CPU load on
debian-www and not a good idea. That's why it is pushed aside to
lessoften. The inherent problem is debian-www is running stable
distribution while the installation-guide source package is mainly
developed on the testing/unstable tool chain.
The better way is unpack binary packages to build files. Some file name
changes and URL rewrite is needed but that has been done successfully
for debian-handbook debian-handbook_*.deb in often cron script. This is
very light weight. (This time s/html/html.en/ etc. is needed. sed
script for URL is much lighter CPU load than stabdard build process.)
Even now fetching of the source and binary packages are done in often
cron script using the ftp protocol. It is still working but I don't
know how long we support the ftp protocol. Our mirror stopped using the
ftp protocol. So maybe soon this access may be killed.
This is not just a problem for installation-guide but almost all other
debian-doc files.

Given we're already building docs to ship in binary packages, it seems
reasonable to re-use those build results on www-master indeed. There's
maybe some history behind building it there but I'm not aware of it. So
working on an update of the “build” process looks good to me. Any takers
on the debian-boot side?


KiBi.

[Beatrice Torracca]

Post by Cyril Brulebois
Given we're already building docs to ship in binary packages, it seems
reasonable to re-use those build results on www-master indeed. There's
maybe some history behind building it there but I'm not aware of it. So
working on an update of the “build” process looks good to me. Any takers
on the debian-boot side?

Hi all,

isn't there also the fact that binary packages are created only so
often, while the sources in the VCS maybe more up-to-date?

Would not the documentation on the website be more up-to-date if they
were generated from the VCS sources?

I think I see something like that happening with the Debian FAQ that,
as far as I know, has the documentation online taken from the Debian
packages.

Hope I am not missing something obvious.

Thanks for the work on this,

beatrice

[Samuel Thibault]

Post by Cyril Brulebois
Given we're already building docs to ship in binary packages, it seems
reasonable to re-use those build results on www-master indeed. There's
maybe some history behind building it there but I'm not aware of it.

AFAICT, the notable difference is that files are named e.g. .html.en on
the website, while they are named .html in the package. These names make
sense in their respective contexts: we need a .html extension in the
package so that it opens automatically, and we need the .en extension on
the website to get language negociation. Perhaps one can use .en.html
on the website?

(and it's not only a question of file renaming, links in the .html files
need to be updated accordingly).

Samuel

[Paul Wise]

Post by Samuel Thibault
AFAICT, the notable difference is that files are named e.g. .html.en on
the website, while they are named .html in the package. These names make
sense in their respective contexts: we need a .html extension in the
package so that it opens automatically, and we need the .en extension on
the website to get language negociation. Perhaps one can use .en.html
on the website?

The files for the main part of the website use .en.html:

https://www.debian.org/index.en.html

I think the website mainly uses index.en.html for language
negotiation, I logged into our static mirrors and the only area that
uses index.html.en are the release notes in
www.d.o/releases/*/*/*.html.* and an April Fools page from a while
back.

https://www.debian.org/misc/canterbury

Post by Samuel Thibault
(and it's not only a question of file renaming, links in the .html files
need to be updated accordingly).

That would be the main issue.

[Samuel Thibault]

Post by Paul Wise

Post by Samuel Thibault
AFAICT, the notable difference is that files are named e.g. .html.en on
the website, while they are named .html in the package. These names make
sense in their respective contexts: we need a .html extension in the
package so that it opens automatically, and we need the .en extension on
the website to get language negociation. Perhaps one can use .en.html
on the website?

https://www.debian.org/index.en.html

Ah, cool :)

Post by Paul Wise

Post by Samuel Thibault
(and it's not only a question of file renaming, links in the .html files
need to be updated accordingly).

That would be the main issue.

Or not: we can simply change the rules that produce .html.en into
producing .en.html, and drop the rules which produce .html

Samuel

[Cyril Brulebois]

Post by Samuel Thibault

Post by Paul Wise

Post by Samuel Thibault
(and it's not only a question of file renaming, links in the .html
files need to be updated accordingly).

That would be the main issue.

Or not: we can simply change the rules that produce .html.en into
producing .en.html, and drop the rules which produce .html

I didn't check yet, but do we have different contents in .html and
.CC.html? If we do, oops. If we don't, we could keep symlinks instead,
and ignore them when building the website?


KiBi.

[Samuel Thibault]

Post by Cyril Brulebois

Post by Samuel Thibault

Post by Paul Wise

Post by Samuel Thibault
(and it's not only a question of file renaming, links in the .html
files need to be updated accordingly).

That would be the main issue.

Or not: we can simply change the rules that produce .html.en into
producing .en.html, and drop the rules which produce .html

I didn't check yet, but do we have different contents in .html and
.CC.html?

I guess in my earlier mail I failed to be clear: AFAICT, the name of the
file (and thus the links) seem to be about the only change.

Post by Cyril Brulebois
If we do, oops. If we don't, we could keep symlinks instead,
and ignore them when building the website?

Which symlink precisely?

ln -s foo.en.html foo.html

?

Samuel

[Samuel Thibault]

Post by Samuel Thibault

Post by Cyril Brulebois

Post by Samuel Thibault

Post by Paul Wise

Post by Samuel Thibault
(and it's not only a question of file renaming, links in the .html
files need to be updated accordingly).

That would be the main issue.

Or not: we can simply change the rules that produce .html.en into
producing .en.html, and drop the rules which produce .html

I didn't check yet, but do we have different contents in .html and
.CC.html?

I guess in my earlier mail I failed to be clear: AFAICT, the name of the
file (and thus the links) seem to be about the only change.

The buildweb.sh script is basically just forcing

export official_build="1" (which is set for package upload anyway)
export web_build="1"
export manual_target="for_wdo"


The only differences are

- these links on the website:

example preconfiguration file from <span class="phrase"><a class="ulink" href="http://www.debian.org/releases/stretch/example-preseed.txt" target="_top">http://www.debian.org/releases/stretch/example-preseed.txt</a></span>.

which are made

example preconfiguration file from <span class="phrase"><a class="ulink" href="../example-preseed.txt" target="_top">../example-preseed.txt</a></span>.

in the package. I'd say we can use relative links on the website as
well.

- the link to the various formats:

The document you are now reading, which is the official version of the
Installation Guide for the stretch release of Debian; available
in <a class="ulink" href="http://www.debian.org/releases/stretch//installmanual" target="_top">various formats and
translations</a>.

in the website, and

This document you are now reading, in plain ASCII, HTML or PDF format.

<div class="itemizedlist"><ul class="itemizedlist" type="disc">
<li class="listitem"><p>
<a class="ulink" href="install.en.txt" target="_top">install.en.txt</a>
</p></li>
<li class="listitem"><p>
<a class="ulink" href="install.en.html" target="_top">install.en.html</a>
</p></li>
<li class="listitem"><p>
<a class="ulink" href="install.en.pdf" target="_top">install.en.pdf</a>
</p></li>
</ul></div>

in the package. The real difference is "and [various] translations". I
don't think this is worth making a difference between the website and
the package?

Samuel

[Cyril Brulebois]

Post by Samuel Thibault
The buildweb.sh script is basically just forcing
export official_build="1" (which is set for package upload anyway)
export web_build="1"
export manual_target="for_wdo"
The only differences are
example preconfiguration file from <span class="phrase"><a class="ulink" href="http://www.debian.org/releases/stretch/example-preseed.txt" target="_top">http://www.debian.org/releases/stretch/example-preseed.txt</a></span>.
which are made
example preconfiguration file from <span class="phrase"><a class="ulink" href="../example-preseed.txt" target="_top">../example-preseed.txt</a></span>.
in the package. I'd say we can use relative links on the website as
well.

Looks good to me, yes.

Post by Samuel Thibault
The document you are now reading, which is the official version of the
Installation Guide for the stretch release of Debian; available
in <a class="ulink" href="http://www.debian.org/releases/stretch//installmanual" target="_top">various formats and
translations</a>.

(nitpick: Probably a slash too many before installmanual.)

Post by Samuel Thibault
in the website, and
This document you are now reading, in plain ASCII, HTML or PDF format.
<div class="itemizedlist"><ul class="itemizedlist" type="disc">
<li class="listitem"><p>
<a class="ulink" href="install.en.txt" target="_top">install.en.txt</a>
</p></li>
<li class="listitem"><p>
<a class="ulink" href="install.en.html" target="_top">install.en.html</a>
</p></li>
<li class="listitem"><p>
<a class="ulink" href="install.en.pdf" target="_top">install.en.pdf</a>
</p></li>
</ul></div>
in the package. The real difference is "and [various] translations". I
don't think this is worth making a difference between the website and
the package?

I don't think it's required, but it might be nice to give doc/www
people a chance to voice their opinion here.


KiBi.

[Samuel Thibault]

Hello DSA,

Post by Samuel Thibault
fonts-wqy-microhei fonts-vlgothic

I have attached a debian.org.git patch.

Samuel

[Aurelien Jarno]

Post by Samuel Thibault
Hello DSA,

Post by Samuel Thibault
fonts-wqy-microhei fonts-vlgothic

I have attached a debian.org.git patch.

Thanks, I merged the patch and installed the packages on wolkenstein.

Aurelien

[Holger Wansing]

Hi,

Post by Samuel Thibault

Post by Samuel Thibault
/srv/www.debian.org/installmanual/stretch.log
kpathsea: Running mktextfm VL-PGothic-Regular/OT
mkdir: cannot create directory 'build.tmp.ja.i386/dblatex/mt22600.tmp': No such file or directory
xelatex failed
I don't know why that happens, investigating, perhaps it's just because
it's a Jessie box only.

fonts-wqy-microhei fonts-vlgothic

That is then caused by switching the pdf building to dblatex.
Is dblatex installed on that box?

Holg

[Samuel Thibault]

Post by Holger Wansing

Post by Samuel Thibault

Post by Samuel Thibault
/srv/www.debian.org/installmanual/stretch.log
kpathsea: Running mktextfm VL-PGothic-Regular/OT
mkdir: cannot create directory 'build.tmp.ja.i386/dblatex/mt22600.tmp': No such file or directory
xelatex failed
I don't know why that happens, investigating, perhaps it's just because
it's a Jessie box only.

fonts-wqy-microhei fonts-vlgothic

That is then caused by switching the pdf building to dblatex.

Something like this, yes.

Post by Holger Wansing
Is dblatex installed on that box?

Yes. dpkg-checkbuilddeps only reports these two packages as missing.

Samuel

[victory]

Post by Samuel Thibault
But I don't know what "lessoften" is.

debwww/cron.git