[PATCH] makepkg: respect $SOURCE_DATE_EPOCH to activate reproducible builds

classic Classic list List threaded Threaded
5 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

[PATCH] makepkg: respect $SOURCE_DATE_EPOCH to activate reproducible builds

Eli Schwartz
If SOURCE_DATE_EPOCH is set, `touch` all source files before running
build() to fix the modification times. This works around build systems
and compilers that embed the file modification times into the file
contents of release artifacts.

Signed-off-by: Eli Schwartz <[hidden email]>
---

Guarded by checking for the variable, because this stomps all over
incremental builds.

I prefer looking for SOURCE_DATE_EPOCH in the environment because less
bloat IMHO, but perhaps this would be preferable as a makepkg flag or
makepkg.conf option?

Is there anything else that needs to be done to force reproducible
builds, other than merely setting SOURCE_DATE_EPOCH which should be
harmless as a global thing?

 scripts/makepkg.sh.in | 15 ++++++++++++++-
 1 file changed, 14 insertions(+), 1 deletion(-)

diff --git a/scripts/makepkg.sh.in b/scripts/makepkg.sh.in
index 20e9dd7e..fb1d40a9 100644
--- a/scripts/makepkg.sh.in
+++ b/scripts/makepkg.sh.in
@@ -79,15 +79,22 @@ PKGFUNC=0
 PKGVERFUNC=0
 PREPAREFUNC=0
 REPKG=0
+REPRODUCIBLE=0
 RMDEPS=0
 SKIPCHECKSUMS=0
 SKIPPGPCHECK=0
 SIGNPKG=''
 SPLITPKG=0
 SOURCEONLY=0
+
 VERIFYSOURCE=0
 
-export SOURCE_DATE_EPOCH=${SOURCE_DATE_EPOCH:-$(date +%s)}
+if [[ -n $SOURCE_DATE_EPOCH ]]; then
+ REPRODUCIBLE=1
+else
+ SOURCE_DATE_EPOCH=$(date +%s)
+fi
+export SOURCE_DATE_EPOCH
 
 PACMAN_OPTS=()
 
@@ -475,6 +482,12 @@ run_prepare() {
 }
 
 run_build() {
+ if (( REPRODUCIBLE )); then
+ # We have activated reproducible builds, so unify source times before
+ # building
+ find "$srcdir" -exec touch -h -d @$SOURCE_DATE_EPOCH {} +
+ fi
+
  run_function_safe "build"
 }
 
--
2.13.2
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [PATCH] makepkg: respect $SOURCE_DATE_EPOCH to activate reproducible builds

Allan McRae
On 07/07/17 03:26, Eli Schwartz wrote:
> If SOURCE_DATE_EPOCH is set, `touch` all source files before running
> build() to fix the modification times. This works around build systems
> and compilers that embed the file modification times into the file
> contents of release artifacts.
>

I think this is a reasonable compromise.

@Levente:  are you happy with this approach?
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

[PATCH v2] makepkg: respect $SOURCE_DATE_EPOCH to activate reproducible builds

Eli Schwartz-2
In reply to this post by Eli Schwartz
If SOURCE_DATE_EPOCH is set, `touch` all source files before running
build() to fix the modification times. This works around build systems
and compilers that embed the file modification times into the file
contents of release artifacts.

Signed-off-by: Eli Schwartz <[hidden email]>
---

v2: Fix accidental extra blank line that sneaked into v1

 scripts/makepkg.sh.in | 14 +++++++++++++-
 1 file changed, 13 insertions(+), 1 deletion(-)

diff --git a/scripts/makepkg.sh.in b/scripts/makepkg.sh.in
index 20e9dd7e..77d39ca5 100644
--- a/scripts/makepkg.sh.in
+++ b/scripts/makepkg.sh.in
@@ -79,6 +79,7 @@ PKGFUNC=0
 PKGVERFUNC=0
 PREPAREFUNC=0
 REPKG=0
+REPRODUCIBLE=0
 RMDEPS=0
 SKIPCHECKSUMS=0
 SKIPPGPCHECK=0
@@ -87,7 +88,12 @@ SPLITPKG=0
 SOURCEONLY=0
 VERIFYSOURCE=0
 
-export SOURCE_DATE_EPOCH=${SOURCE_DATE_EPOCH:-$(date +%s)}
+if [[ -n $SOURCE_DATE_EPOCH ]]; then
+ REPRODUCIBLE=1
+else
+ SOURCE_DATE_EPOCH=$(date +%s)
+fi
+export SOURCE_DATE_EPOCH
 
 PACMAN_OPTS=()
 
@@ -475,6 +481,12 @@ run_prepare() {
 }
 
 run_build() {
+ if (( REPRODUCIBLE )); then
+ # We have activated reproducible builds, so unify source times before
+ # building
+ find "$srcdir" -exec touch -h -d @$SOURCE_DATE_EPOCH {} +
+ fi
+
  run_function_safe "build"
 }
 
--
2.13.3
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [PATCH] makepkg: respect $SOURCE_DATE_EPOCH to activate reproducible builds

Levente Polyak-3
In reply to this post by Allan McRae
On 07/28/2017 04:36 AM, Allan McRae wrote:

> On 07/07/17 03:26, Eli Schwartz wrote:
>> If SOURCE_DATE_EPOCH is set, `touch` all source files before running
>> build() to fix the modification times. This works around build systems
>> and compilers that embed the file modification times into the file
>> contents of release artifacts.
>>
>
> I think this is a reasonable compromise.
>
> @Levente:  are you happy with this approach?
>

The conditional REPRODUCIBLE touch block should be outside of run_build
as former would imply that a build() function is mandatory inside every
PKGBUILD to make this work.
Its quite common (f.e. for python) to not have a build() function and I
believe it should generally work without enforcing to have such function.
Therefor I would recommend we move that block between run_build and
run_prepare, f.e. a place where this works is line 1696 in commit bcc9c417.
A tested/verified adjustment doing the above would be:
https://github.com/anthraxx/pacman/commit/520acf93d83774c4b74ec8c19c1ba31fddbdb8da


Technically this approach will work, but personally i believe this is a
bit too much hidden requirement to make it work. So for clarity I would
either recommend we add a section describing this behaviour inside the
manpage (as manually defining SOURCE_DATE_EPOCH will be mandatory to
make it work) or we use makepkg flags plus config (which will also make
it more transparent).
I don't have any strong favorites, Eli implementation will do its job,
the only difference will be that manually setting SOURCE_DATE_EPOCH is
mandatory for the first initial invocation as well.

cheers,
Levente

PS: thanks :P


signature.asc (849 bytes) Download Attachment
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: [PATCH] makepkg: respect $SOURCE_DATE_EPOCH to activate reproducible builds

Eli Schwartz-2
On 07/28/2017 07:59 AM, Levente Polyak wrote:

> On 07/28/2017 04:36 AM, Allan McRae wrote:
>> On 07/07/17 03:26, Eli Schwartz wrote:
>>> If SOURCE_DATE_EPOCH is set, `touch` all source files before running
>>> build() to fix the modification times. This works around build systems
>>> and compilers that embed the file modification times into the file
>>> contents of release artifacts.
>>>
>>
>> I think this is a reasonable compromise.
>>
>> @Levente:  are you happy with this approach?
>>
>
>
> The conditional REPRODUCIBLE touch block should be outside of run_build
> as former would imply that a build() function is mandatory inside every
> PKGBUILD to make this work.
> Its quite common (f.e. for python) to not have a build() function and I
> believe it should generally work without enforcing to have such function.
> Therefor I would recommend we move that block between run_build and
> run_prepare, f.e. a place where this works is line 1696 in commit bcc9c417.
> A tested/verified adjustment doing the above would be:
> https://github.com/anthraxx/pacman/commit/520acf93d83774c4b74ec8c19c1ba31fddbdb8da
I kind of feel like this is a sign that people are doing the wrong
thing, since you shouldn't be modifying $srcdir during package().

Personally, I am very careful about making sure `python setup.py build`
is run during build(), the same way I am careful about making sure
`make` is run during build() even when `make install` would cause `make
all` to run anyway.

I don't feel like we should make a split between GNU Make and other
build systems, or between a build system that executes a C compiler vs.
one that merely runs pod2man and copies files around.

I won't even mention npm except to say, look at aur/rapydscript-ng-git.

> Technically this approach will work, but personally i believe this is a
> bit too much hidden requirement to make it work. So for clarity I would
> either recommend we add a section describing this behaviour inside the
> manpage (as manually defining SOURCE_DATE_EPOCH will be mandatory to
> make it work) or we use makepkg flags plus config (which will also make
> it more transparent).
> I don't have any strong favorites, Eli implementation will do its job,
> the only difference will be that manually setting SOURCE_DATE_EPOCH is
> mandatory for the first initial invocation as well.

Okay. I personally prefer the implementation that doesn't add more flags
or makepkg.conf options, because at a minimum anyone who wants to make
use of this will need to have tooling to read from a previous invocation
regardless or on top of those flags for the second invocation, so we
might as well follow the same approach the whole time.

Perhaps a REPRODUCIBILITY section in makepkg(1) right after ADDITIONAL
FEATURES, then, to describe what efforts makepkg will go to to ensure a
reproducible build?
And then add SOURCE_DATE_EPOCH for environment variables makepkg will
respect.

--
Eli Schwartz


signature.asc (849 bytes) Download Attachment
Loading...