[PATCH] makepkg: use builtin globbing to print files in package

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|

[PATCH] makepkg: use builtin globbing to print files in package

Eli Schwartz-2
- it comes with free collation when moving the LC_ALL declaration up a bit;
  this fixes a bug where the .FILES were not being properly sorted and
  their order depended on directory creation order, which broke
  reproducible builds in the wild.
- it handles sorting null-delimited output everywhere, without sort -z;
  this lets us get rid of sed hacks
- it is faster than invoking multiple find subprocesses
- dotfiles can be automatically printed *and the C locale sorts them first*
  with a single ** glob

Signed-off-by: Eli Schwartz <[hidden email]>
---
 scripts/makepkg.sh.in | 8 ++++++--
 1 file changed, 6 insertions(+), 2 deletions(-)

diff --git a/scripts/makepkg.sh.in b/scripts/makepkg.sh.in
index ae1ef01b..1325b019 100644
--- a/scripts/makepkg.sh.in
+++ b/scripts/makepkg.sh.in
@@ -711,10 +711,14 @@ write_buildinfo() {
 # database files are placed at the beginning of the package regardless of
 # sorting
 list_package_files() {
- (find . -path './.*' \! -name '.'; find . \! -path './.*' \! -name '.' | LC_ALL=C sort) |
- sed -e 's|^\./||' | tr '\n' '\0'
+ (
+ export LC_COLLATE=C
+ shopt -s dotglob globstar
+ printf '%s\0' **
+ )
 }
 
+
 create_package() {
  (( NOARCHIVE )) && return 0
 
--
2.18.0
Reply | Threaded
Open this post in threaded view
|

Re: [PATCH] makepkg: use builtin globbing to print files in package

Luke Shumaker-2
On Tue, 21 Aug 2018 10:15:12 -0400,
Eli Schwartz wrote:

> diff --git a/scripts/makepkg.sh.in b/scripts/makepkg.sh.in
> index ae1ef01b..1325b019 100644
> --- a/scripts/makepkg.sh.in
> +++ b/scripts/makepkg.sh.in
> @@ -711,10 +711,14 @@ write_buildinfo() {
>  # database files are placed at the beginning of the package regardless of
>  # sorting
>  list_package_files() {
> - (find . -path './.*' \! -name '.'; find . \! -path './.*' \! -name '.' | LC_ALL=C sort) |
> - sed -e 's|^\./||' | tr '\n' '\0'
> + (
> + export LC_COLLATE=C
> + shopt -s dotglob globstar
> + printf '%s\0' **
> + )

Since globbing is done in the same process, it should be sufficient to
set LC_COLLATE; no need to export it.

--
Happy hacking,
~ Luke Shumaker
Reply | Threaded
Open this post in threaded view
|

Re: [PATCH] makepkg: use builtin globbing to print files in package

Eli Schwartz-2
On 8/21/18 11:41 AM, Luke Shumaker wrote:

> On Tue, 21 Aug 2018 10:15:12 -0400,
> Eli Schwartz wrote:
>> diff --git a/scripts/makepkg.sh.in b/scripts/makepkg.sh.in
>> index ae1ef01b..1325b019 100644
>> --- a/scripts/makepkg.sh.in
>> +++ b/scripts/makepkg.sh.in
>> @@ -711,10 +711,14 @@ write_buildinfo() {
>>  # database files are placed at the beginning of the package regardless of
>>  # sorting
>>  list_package_files() {
>> - (find . -path './.*' \! -name '.'; find . \! -path './.*' \! -name '.' | LC_ALL=C sort) |
>> - sed -e 's|^\./||' | tr '\n' '\0'
>> + (
>> + export LC_COLLATE=C
>> + shopt -s dotglob globstar
>> + printf '%s\0' **
>> + )
>
> Since globbing is done in the same process, it should be sufficient to
> set LC_COLLATE; no need to export it.
I guess....

My initial version used LC_COLLATE without globstar and just replaced
the first find command with printf, so it needed to be exported to sort
as well. Then dreisner pointed out using globstar could totally drop the
find subprocess.

--
Eli Schwartz
Bug Wrangler and Trusted User


signature.asc (849 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|

Re: [PATCH] makepkg: use builtin globbing to print files in package

Andrew Gregory
In reply to this post by Eli Schwartz-2
On 08/21/18 at 10:15am, Eli Schwartz wrote:
> - dotfiles can be automatically printed *and the C locale sorts them first*
>   with a single ** glob

The glob will sort dotfiles before alphanumeric names, but there are several
characters that will still sort before a period.
Reply | Threaded
Open this post in threaded view
|

Re: [PATCH] makepkg: use builtin globbing to print files in package

Eli Schwartz-2
On 8/21/18 11:53 AM, Andrew Gregory wrote:
> On 08/21/18 at 10:15am, Eli Schwartz wrote:
>> - dotfiles can be automatically printed *and the C locale sorts them first*
>>   with a single ** glob
>
> The glob will sort dotfiles before alphanumeric names, but there are several
> characters that will still sort before a period.

Eh. In the wise words of dreisner (when I asked if I should amend it to
fix that):

yes technically if someone were to create a root directory starting with
! "  # $ & %  ' ( ) * + , - it would sort before the dotfiles
the day someone does that, i'll perfect my implementation of stabbing
people over standard tcp/ip connections.

I rather agree, since none of those filenames are remotely sane, at
least not as direct children of / -- which is why this matters, since
our sorting goal is IIRC just to ensure our metadata dotfiles are the
most efficient files to extract. If that fails... oh well, some crazy
person's package will have somewhat less efficiently extracted metadata.

Although, this could be fixed I guess with:

printf '%s\0' .!(|.); printf '%s\0' !(.*) */** | sort -uz

--
Eli Schwartz
Bug Wrangler and Trusted User


signature.asc (849 bytes) Download Attachment