Change upstream repo for weblate debian-reference

[Osamu Aoki]

Hi,

I am the upstream of debian-reference
https://salsa.debian.org/debian/debian-reference
https://tracker.debian.org/pkg/debian-reference
https://www.debian.org/doc/manuals/debian-reference/

Someone set up weblate service previously pointing to alioth git repo.
As you know Debian is moving repository to salsa.debian.org.
debian-reference is no exception, here.

Problem is I don't know how to update current weblate setting.
https://hosted.weblate.org/projects/debian-reference/translations/#information

Specifically, URL listed for "Instructions for translators" and
"Repository" in the above web page is wrong and they need to be updated
to the new location.

"Instructions for translators"
https://salsa.debian.org/debian/debian-reference

"Repository"
https://salsa.debian.org/debian/debian-reference.git

Otherwise, translation effort ion weblate are not used effectively.

Your help to fix this situation is most appreciated.

Regards,

Osamu

[Michal Čihař]

Hello

Post by Osamu Aoki
I am the upstream of debian-reference
https://salsa.debian.org/debian/debian-reference
https://tracker.debian.org/pkg/debian-reference
https://www.debian.org/doc/manuals/debian-reference/
Someone set up weblate service previously pointing to alioth git repo.
As you know Debian is moving repository to salsa.debian.org.
debian-reference is no exception, here.
Problem is I don't know how to update current weblate setting.
https://hosted.weblate.org/projects/debian-reference/translations/#
information
Specifically, URL listed for "Instructions for translators" and
"Repository" in the above web page is wrong and they need to be updated
to the new location.
"Instructions for translators"
https://salsa.debian.org/debian/debian-reference
"Repository"
https://salsa.debian.org/debian/debian-reference.git
Otherwise, translation effort ion weblate are not used effectively.
Your help to fix this situation is most appreciated.

I've just changed both and I've also added you to administrators to the
project on Weblate.

Best regards

[Osamu Aoki]

Hi,

Post by Michal Čihař
Hello

...

Post by Michal Čihař
I've just changed both and I've also added you to administrators to the
project on Weblate.

I see more control. Thanks.

Osamu

[Laura Arjona Reina]

Hello Osamu and Debian Documentation Team

Post by Osamu Aoki
Hi,
I am the upstream of debian-reference
https://salsa.debian.org/debian/debian-reference
https://tracker.debian.org/pkg/debian-reference
https://www.debian.org/doc/manuals/debian-reference/
Someone set up weblate service previously pointing to alioth git repo.
As you know Debian is moving repository to salsa.debian.org.
debian-reference is no exception, here.

I've updated the URLs in the website repository:

https://anonscm.debian.org/viewvc/webwml/webwml/english/doc/user-manuals.defs?r1=1.25&r2=1.26&sortby=date

Please correct me if I did it wrong.

If there are more manuals already moved to Salsa, just tell.

Regards,

[Osamu Aoki]

Hi,

Post by Laura Arjona Reina
Hello Osamu and Debian Documentation Team

Post by Osamu Aoki
Hi,
I am the upstream of debian-reference
https://salsa.debian.org/debian/debian-reference
https://tracker.debian.org/pkg/debian-reference
https://www.debian.org/doc/manuals/debian-reference/
Someone set up weblate service previously pointing to alioth git repo.
As you know Debian is moving repository to salsa.debian.org.
debian-reference is no exception, here.

https://anonscm.debian.org/viewvc/webwml/webwml/english/doc/user-manuals.defs?r1=1.25&r2=1.26&sortby=date
Please correct me if I did it wrong.

Thanks (I should have done this since I have write access to webwml ;-)

Post by Laura Arjona Reina
If there are more manuals already moved to Salsa, just tell.

That is on my agenda to do ;-)

By the way, we have a fundamental problem with www.debian.org

The contents of debian-reference are published to the www.debian.org
servers from the latest released packages by the cron script run on the
master www.debian.org server and they are mirrored to all the
www.debian.org servers.

https://anonscm.debian.org/git/debwww/cron.git (Maybe moving to salsa soon)
parts/1ftpfiles
parts/7doc

As the 1ftpfiles indicates, packages are gatherted from ftp package site
which is now non-sup[ported officially. (Now we are supposed to use
http sites)

Since this requires significant coding, this code updates requires to be
done by someone with shell access to www.debian.org.

I know you were thinking to migrate www.debian.org building
infrastructure to git based system from the current CVS. Did you also
looked into parts/1ftpfiles issues?

Osamu

Post by Laura Arjona Reina
Regards,
--
Laura Arjona Reina
https://wiki.debian.org/LauraArjona

[Laura Arjona Reina]

Package: www.debian.org
User: ***@packages.debian.org
Usertag: scripts
Severity: normal
X-Debbugs-CC: debian-***@lists.debian.org

Hi!

Turning this conversation into a bug report so it does not get lost.

Post by Osamu Aoki
Hi,

[...]

Post by Osamu Aoki
By the way, we have a fundamental problem with www.debian.org
The contents of debian-reference are published to the www.debian.org
servers from the latest released packages by the cron script run on
the master www.debian.org server and they are mirrored to all the
www.debian.org servers.
https://anonscm.debian.org/git/debwww/cron.git (Maybe moving to salsa soon)
parts/1ftpfiles
parts/7doc
As the 1ftpfiles indicates, packages are gatherted from ftp package site
which is now non-sup[ported officially. (Now we are supposed to use
http sites)
Since this requires significant coding,

AFAIK we just change ftp:// for http:// and it should work isn't it?
In any case I'll have a deeper look later.


this code updates requires to be

Post by Osamu Aoki
done by someone with shell access to www.debian.org.

I think that the copy of cron scripts in www-master.debian.org is
updated automatically by some of those scripts, but if we need to do
anything else in www-master.debian.org, I'll try to help.

Post by Osamu Aoki
I know you were thinking to migrate www.debian.org building
infrastructure to git based system from the current CVS. Did you also
looked into parts/1ftpfiles issues?

Not yet. If you know other issues different than the ftp:// -> http://
one, please let us (debian-***@lists.debian.org) know or file bugs
against www.debian.org pseudopackages.

Thanks for caring!

[Laura Arjona Reina]

Hello
I've been doing some tests and this is what I have, for now:

* the current script is in:
https://anonscm.debian.org/cgit/debwww/cron.git/tree/parts/1ftpfiles
(and it works, not sure when/if it will stop working...).

* There, changing ftp:// for http:// is not enough, because we were getting
files using wildcards (which I guess worked for ftp:// but it does not work for
http://).

* Relevant code is a function "wgetfiles" which is called in 3 ways:

wgetfiles "" "" ftp://${ftpsite}/debian/doc/ 2 doc
wgetfiles emacsen-common emacsen-common_*.deb
wgetfiles dpkg dpkg-doc_1.9.21_all.deb
http://snapshot.debian.org/archive/debian/20050312T000000Z/pool/main 7

And the specific wget call inside the function wgetfiles is, currently:

wget --timeout=60 --quiet --recursive --timestamping --no-host-directories \
--cut-dirs=${cutdirs} --directory-prefix=${prefix} \
${ftpurl}/${initial}/${namesrc}/${namebin}

I've learned that with wget we can use wildcards with --recursive and -A
"pattern" and we should probably add --no-parent to avoid the recursive go to
other places.

Then I've transformed the wget call into this:

wget -e robots=off --no-parent --timeout=60 --recursive --timestamping
--no-host-directories \
--cut-dirs=${cutdirs} --directory-prefix=${prefix} \
--reject "*.html*" -A "${namebin}" ${ftpurl}/${initial}/${namesrc}/

(see also the complete diff and the 'new' 1ftpfiles, attached).

This seems to work (I've run the script in local and later checked that the
files were downloaded in /srv/www.debian.org/cron/ftpfiles), but it needs
improvements, because:

* I've workarounded the robots.txt with "-e robots=off" but I guess that this is
not the correct/elegant/respectful way?

* wget downloads all the files and then removes the ones that don't match the
pattern specified with -A. Maybe there is a more efficient way to do this?


Cheers

[Osamu Aoki]

What you did is one way ;-)

Post by Laura Arjona Reina
Hello
https://anonscm.debian.org/cgit/debwww/cron.git/tree/parts/1ftpfiles
(and it works, not sure when/if it will stop working...).

...

Post by Laura Arjona Reina
This seems to work (I've run the script in local and later checked that the
files were downloaded in /srv/www.debian.org/cron/ftpfiles), but it needs
* I've workarounded the robots.txt with "-e robots=off" but I guess that this is
not the correct/elegant/respectful way?
* wget downloads all the files and then removes the ones that don't match the
pattern specified with -A. Maybe there is a more efficient way to do this?

Most wgetfiles are downloading the latest unstable binary packages.
Why re-invent the binary package downloader when we have "apt-get"?

apt-get has -C option to use non-standard /etc/apt/sources.list
listing unstable distribution:

deb http://deb.debian.org/debian/ sid main contrib non-free

We need to use non-standard directories and not to contaminate the
system ones:
/var/cache/apt/archives/
/var/lib/apt/lists/

This can be done by setting through
Item: Dir::Cache::Archives
Item: Dir::State::Lists
specified via -o option.

By setting all these and few more options as needed, we should be
able to download the latest binary package from the archive using the
proven tool.

Of course, obsolete dpkg-doc from snapshot should use original wgetfiles

Installation guide: I need to check how it should be handled.

What do you think?

Regards,

Osamu

[Osamu Aoki]

Hi,

Post by Osamu Aoki
What you did is one way ;-)

...

Post by Osamu Aoki

Post by Laura Arjona Reina
pattern specified with -A. Maybe there is a more efficient way to do this?

Most wgetfiles are downloading the latest unstable binary packages.
Why re-invent the binary package downloader when we have "apt-get"?
apt-get has -C option to use non-standard /etc/apt/sources.list
deb http://deb.debian.org/debian/ sid main contrib non-free
We need to use non-standard directories and not to contaminate the
/var/cache/apt/archives/
/var/lib/apt/lists/
This can be done by setting through
Item: Dir::Cache::Archives
Item: Dir::State::Lists
specified via -o option.
By setting all these and few more options as needed, we should be
able to download the latest binary package from the archive using the
proven tool.

Just to be clear, I was thinking to do the following to get the latest
binary package in the previous post:

apt-get update
apt-get -d install $PACKAGENAME

Then go to redirected /var/cache/apt/archives/ location to pick up the
latest package.

This is just an outline. There may needs to do few more chores to get
this working.

Post by Osamu Aoki
Of course, obsolete dpkg-doc from snapshot should use original wgetfiles
Installation guide: I need to check how it should be handled.
What do you think?
Regards,
Osamu

[Paul Wise]

Post by Osamu Aoki
apt-get has -C option to use non-standard /etc/apt/sources.list

This is the wrong way to completely override the system apt
configuration. The right way is to set the APT_CONFIG environment
variable. See the apt.conf(5) manual page, APT_CONFIG is the first
method that apt uses to find the config file and the -C command-line
option is the very last one so it cannot prevent loading all the
system config. The chdist tool (from devscripts), the apt-venv package
and the derivatives census code get this correct.

I definitely agree that apt-get could be used to download packages
from repositories, especially since it will verify hashes and
signatures.

[Osamu Aoki]

Hi,

Post by Paul Wise

Post by Osamu Aoki
apt-get has -C option to use non-standard /etc/apt/sources.list

This is the wrong way to completely override the system apt
configuration. The right way is to set the APT_CONFIG environment
variable. See the apt.conf(5) manual page, APT_CONFIG is the first
method that apt uses to find the config file and the -C command-line
option is the very last one so it cannot prevent loading all the
system config. The chdist tool (from devscripts), the apt-venv package
and the derivatives census code get this correct.

Great point!

Post by Paul Wise
I definitely agree that apt-get could be used to download packages
from repositories, especially since it will verify hashes and
signatures.

One negative point of above approach is that it always downloads doc deb
packages even if there are no changes. Also unpacking code is still
complicated to make it future proof.

Then I thought it may be good idea to create very small Debian unstable
chroot. Then install pertinent doc packages. For old deb from
snapshot, we can wget and dpkg -i. Then this chroot can be updated and
dist-upgraded. This causes to download all base package updates but
skips repeated download of doc packages. Hmmm... which is worse?

I think that chroot may need to mount few parent filesystem points.
mounting /proc filesystem
mounting /sys filesystem
creating /{dev,run}/shm
mounting /dev/pts filesystem
redirecting /dev/ptmx to /dev/pts/ptmx

Osamu

[Paul Wise]

Post by Osamu Aoki
One negative point of above approach is that it always downloads doc deb
packages even if there are no changes.

This will prevent downloading the same package twice:

chdist apt old install --download-only emacsen-common

Post by Osamu Aoki
Also unpacking code is still complicated to make it future proof.

Hmm, do you have any more details? Is dpkg -x not enough?

Post by Osamu Aoki
Then I thought it may be good idea to create very small Debian unstable
chroot. Then install pertinent doc packages. For old deb from
snapshot, we can wget and dpkg -i. Then this chroot can be updated and
dist-upgraded. This causes to download all base package updates but
skips repeated download of doc packages. Hmmm... which is worse?

A chroot needs root so probably isn't going to work here.

[Osamu Aoki]

Hi,

Post by Paul Wise

Post by Osamu Aoki
One negative point of above approach is that it always downloads doc deb
packages even if there are no changes.

chdist apt old install --download-only emacsen-common

Oh, that's it. I should have known ... this is simple enough.
(I always had real chroot systems .... so never used it)

Post by Paul Wise

Post by Osamu Aoki
Also unpacking code is still complicated to make it future proof.

Hmm, do you have any more details? Is dpkg -x not enough?

Probably enough.

Post by Paul Wise

Post by Osamu Aoki
Then I thought it may be good idea to create very small Debian unstable
chroot. Then install pertinent doc packages. For old deb from
snapshot, we can wget and dpkg -i. Then this chroot can be updated and
dist-upgraded. This causes to download all base package updates but
skips repeated download of doc packages. Hmmm... which is worse?

A chroot needs root so probably isn't going to work here.

That's fair.

So now agree. Thanks for the idea.

Osamu