[PATCH 0/4] build: minimize rebuilds

[PATCH 0/4] build: minimize rebuilds

[Sam Ravnborg]

Following patchset minimizes re-builds.
With the setup I use at the moment where I run
build tests on 100+ arm related boards I see that the
build time is reduced to half - for an otherwise
clean build with no changes.
In other words - the wall clock time for a
simple "try to build all boards" where nothing
has changed goes from ~2:35 minutes to 1:21 minutes

The patches has been tested in my local setup - and
no missing rebuilds was seen.

Sam Ravnborg (4):
      build: fix that defaultenv is always rebuilt
      build: fix that passwd.h is always built
      build: fix that the bbenv file is always rebuilt
      build: fix that LZO file is always rebuilt

 common/Makefile        | 28 +++++++++++++++-------------
 defaultenv/Makefile    |  4 ++--
 scripts/Kbuild.include | 12 +++++++-----
 scripts/Makefile.lib   |  2 +-
 4 files changed, 25 insertions(+), 21 deletions(-)

_______________________________________________
barebox mailing list
barebox@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/barebox

[PATCH 1/4] build: fix that defaultenv is always rebuilt

[Sam Ravnborg]

From fcffd0176b3394cc664d9bf5b31b053253323bfb Mon Sep 17 00:00:00 2001
From: Sam Ravnborg <sam@ravnborg.org>
Date: Tue, 26 Dec 2017 16:47:16 +0100
Subject: [PATCH 1/4] build: fix that defaultenv is always rebuilt

Using following pattern:

	target: prerequisite FORCE
		$(call cmd,foo)

is almost always wrong. Due to FORCE target will be rebuilt
using cmd_foo no mater the timestamps.

The following pattern:

	target: prerequisite FORCE
        	$(call if_changed,foo)

Will call cmd_foo only if:
- target is not updated
- or command to built target has changed

But to make this work target must be added to the
list af targets built.
This can be done by assigning target to extra-y or targets.

Use the latter pattern for defaultenv to avoid rebuilding
the environmentt for each build.

Signed-off-by: Sam Ravnborg <sam@ravnborg.org>
---
 defaultenv/Makefile | 4 ++--
 1 file changed, 2 insertions(+), 2 deletions(-)

diff --git a/defaultenv/Makefile b/defaultenv/Makefile
index 33e0eceef..f313b04e8 100644
--- a/defaultenv/Makefile
+++ b/defaultenv/Makefile
@@ -14,7 +14,7 @@ quiet_cmd_env_default = ENV     $@
 cmd_env_default = ($(srctree)/scripts/genenv $(srctree) $(objtree) $@ $(CONFIG_DEFAULT_ENVIRONMENT_PATH))
 
 $(obj)/barebox_default_env: FORCE
-	$(call cmd,env_default)
+	$(call if_changed,env_default)
 
 quiet_cmd_env_h = ENVH    $@
 cmd_env_h = cat $< | (cd $(obj) && $(objtree)/scripts/bin2c "__aligned(4) default_environment") > $@; \
@@ -27,4 +27,4 @@ quiet_cmd_env_zero = ENVZ    $@
 cmd_env_zero = ($(objtree)/scripts/bareboxenv -z $(CONFIG_DEFAULT_ENVIRONMENT_PATH) $@)
 
 $(obj)/barebox_zero_env: FORCE
-	$(call cmd,env_zero)
+	$(call if_changed,env_zero)
-- 
2.12.0


_______________________________________________
barebox mailing list
barebox@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/barebox

[PATCH 2/4] build: fix that passwd.h is always built

[Sam Ravnborg]

From 48fe20e2bf2249b2f89d96c9787e0b489c015054 Mon Sep 17 00:00:00 2001
From: Sam Ravnborg <sam@ravnborg.org>
Date: Tue, 26 Dec 2017 18:02:17 +0100
Subject: [PATCH 2/4] build: fix that passwd.h is always built

Use the kbuild provided support for generated
files to avoid that passwd.h is always generated
thus triggering further re-builds

Signed-off-by: Sam Ravnborg <sam@ravnborg.org>
---
 common/Makefile | 28 +++++++++++++++-------------
 1 file changed, 15 insertions(+), 13 deletions(-)

diff --git a/common/Makefile b/common/Makefile
index 8cd0ab300..5351ef0f7 100644
--- a/common/Makefile
+++ b/common/Makefile
@@ -63,27 +63,29 @@ obj-$(CONFIG_BAREBOX_UPDATE_IMX_NAND_FCB) += imx-bbu-nand-fcb.o
 obj-$(CONFIG_CONSOLE_RATP)	+= ratp.o
 obj-$(CONFIG_BOOT)		+= boot.o
 
-quiet_cmd_pwd_h = PWDH    $@
 ifdef CONFIG_PASSWORD
-ifneq ($(CONFIG_PASSWORD_DEFAULT),"")
-PASSWD_FILE := $(shell cd $(srctree); find $(CONFIG_PASSWORD_DEFAULT) -type f)
-cmd_pwd_h = echo -n "static const char default_passwd[] = \"" > $@; \
-	cat $< | tr -d '\n' >> $@; \
-	echo "\";" >> $@
 
-include/generated/passwd.h: $(PASSWD_FILE)
-	$(call if_changed,pwd_h)
+ifeq ($(CONFIG_PASSWORD_DEFAULT),"")
+define filechk_passwd
+	echo "static const char default_passwd[] = \"\";"
+endef
 else
-cmd_pwd_h = echo "static const char default_passwd[] = \"\";" > $@
-
-include/generated/passwd.h: FORCE
-	$(call if_changed,pwd_h)
+define filechk_passwd
+(							\
+echo -n "static const char default_passwd[] = \"";	\
+cat $(shell find $(srctree) -name $(CONFIG_PASSWORD_DEFAULT) -type f) | tr -d '\n';\
+echo "\";"						\
+)
+endef
 endif
 
+include/generated/passwd.h: $(srctree)/$(src)/Makefile FORCE
+	$(call filechk,passwd)
+
 targets += include/generated/passwd.h
 
 $(obj)/password.o: include/generated/passwd.h
-endif
+endif # CONFIG_PASSWORD
 
 # dependencies on generated files need to be listed explicitly
 $(obj)/version.o: include/generated/compile.h
-- 
2.12.0


_______________________________________________
barebox mailing list
barebox@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/barebox

[PATCH 3/4] build: fix that the bbenv file is always rebuilt

[Sam Ravnborg]

From ac9ca6505d5b887c351117d9c033c8a76cc77125 Mon Sep 17 00:00:00 2001
From: Sam Ravnborg <sam@ravnborg.org>
Date: Tue, 26 Dec 2017 18:05:14 +0100
Subject: [PATCH 3/4] build: fix that the bbenv file is always rebuilt

Use if_changed in rule for bbenv file.
This avoids re-builds.
The target is already assigned to extra-y - so the
kbuild logic will work as expected.

Signed-off-by: Sam Ravnborg <sam@ravnborg.org>
---
 scripts/Makefile.lib | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/scripts/Makefile.lib b/scripts/Makefile.lib
index b084bacab..d202a7299 100644
--- a/scripts/Makefile.lib
+++ b/scripts/Makefile.lib
@@ -292,7 +292,7 @@ quiet_cmd_env = ENV     $@
 cmd_env=$(srctree)/scripts/genenv $(srctree) $(objtree) $@ $<
 
 %.bbenv$(DEFAULT_COMPRESSION_SUFFIX): % FORCE
-	$(call cmd,env)
+	$(call if_changed,env)
 
 # Bzip2
 # ---------------------------------------------------------------------------
-- 
2.12.0


_______________________________________________
barebox mailing list
barebox@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/barebox

[PATCH 4/4] build: fix that LZO file is always rebuilt

[Sam Ravnborg]

From e398a00f84db33ea5ae7f6ee12c54511ef7a94fc Mon Sep 17 00:00:00 2001
From: Sam Ravnborg <sam@ravnborg.org>
Date: Tue, 26 Dec 2017 18:09:35 +0100
Subject: [PATCH 4/4] build: fix that LZO file is always rebuilt

Port the make-cmd from linux kernel.
with the updated version $$ is porperly escaped, thus
the LZO rule works as intended.
And we avoid rebuilds when not required

Signed-off-by: Sam Ravnborg <sam@ravnborg.org>
---
 scripts/Kbuild.include | 12 +++++++-----
 1 file changed, 7 insertions(+), 5 deletions(-)

diff --git a/scripts/Kbuild.include b/scripts/Kbuild.include
index 310dcdc94..fb3fa88fb 100644
--- a/scripts/Kbuild.include
+++ b/scripts/Kbuild.include
@@ -205,11 +205,13 @@ else
 arg-check = $(if $(strip $(cmd_$@)),,1)
 endif
 
-# >'< substitution is for echo to work,
-# >$< substitution to preserve $ when reloading .cmd file
-# note: when using inline perl scripts [perl -e '...$$t=1;...']
-# in $(cmd_xxx) double $$ your perl vars
-make-cmd = $(subst \\,\\\\,$(subst \#,\\\#,$(subst $$,$$$$,$(call escsq,$(cmd_$(1))))))
+# Replace >$< with >$$< to preserve $ when reloading the .cmd file
+# (needed for make)
+# Replace >#< with >\#< to avoid starting a comment in the .cmd file
+# (needed for make)
+# Replace >'< with >'\''< to be able to enclose the whole string in '...'
+# (needed for the shell)
+make-cmd = $(call escsq,$(subst \#,\\\#,$(subst $$,$$$$,$(cmd_$(1)))))
 
 # Find any prerequisites that is newer than target or that does not exist.
 # PHONY targets skipped in both cases.
-- 
2.12.0


_______________________________________________
barebox mailing list
barebox@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/barebox

Re: [PATCH 4/4] build: fix that LZO file is always rebuilt

[Sascha Hauer]

Hi Sam,

On Tue, Dec 26, 2017 at 07:36:53PM +0100, Sam Ravnborg wrote:
> From e398a00f84db33ea5ae7f6ee12c54511ef7a94fc Mon Sep 17 00:00:00 2001
> From: Sam Ravnborg <sam@ravnborg.org>
> Date: Tue, 26 Dec 2017 18:09:35 +0100
> Subject: [PATCH 4/4] build: fix that LZO file is always rebuilt
> 
> Port the make-cmd from linux kernel.
> with the updated version $$ is porperly escaped, thus
> the LZO rule works as intended.
> And we avoid rebuilds when not required
> 
> Signed-off-by: Sam Ravnborg <sam@ravnborg.org>
> ---
>  scripts/Kbuild.include | 12 +++++++-----
>  1 file changed, 7 insertions(+), 5 deletions(-)
> 
> diff --git a/scripts/Kbuild.include b/scripts/Kbuild.include
> index 310dcdc94..fb3fa88fb 100644
> --- a/scripts/Kbuild.include
> +++ b/scripts/Kbuild.include
> @@ -205,11 +205,13 @@ else
>  arg-check = $(if $(strip $(cmd_$@)),,1)
>  endif
>  
> -# >'< substitution is for echo to work,
> -# >$< substitution to preserve $ when reloading .cmd file
> -# note: when using inline perl scripts [perl -e '...$$t=1;...']
> -# in $(cmd_xxx) double $$ your perl vars
> -make-cmd = $(subst \\,\\\\,$(subst \#,\\\#,$(subst $$,$$$$,$(call escsq,$(cmd_$(1))))))
> +# Replace >$< with >$$< to preserve $ when reloading the .cmd file
> +# (needed for make)
> +# Replace >#< with >\#< to avoid starting a comment in the .cmd file
> +# (needed for make)
> +# Replace >'< with >'\''< to be able to enclose the whole string in '...'
> +# (needed for the shell)
> +make-cmd = $(call escsq,$(subst \#,\\\#,$(subst $$,$$$$,$(cmd_$(1)))))
>  

This doesn't work for me. Without this patch I do not see any rebuilds
of lzo files, but with this patch I do.

Seems you are using another shell, dash vs. bash? Porting the second
hunk of the kernel commit 164f0d2efaaef83 fixes it for me.

Sascha

-- 
Pengutronix e.K.                           |                             |
Industrial Linux Solutions                 | http://www.pengutronix.de/  |
Peiner Str. 6-8, 31137 Hildesheim, Germany | Phone: +49-5121-206917-0    |
Amtsgericht Hildesheim, HRA 2686           | Fax:   +49-5121-206917-5555 |

_______________________________________________
barebox mailing list
barebox@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/barebox

Re: [PATCH 4/4] build: fix that LZO file is always rebuilt

[Sascha Hauer]

On Fri, Jan 05, 2018 at 11:49:46AM +0100, Sascha Hauer wrote:
> Hi Sam,
> 
> On Tue, Dec 26, 2017 at 07:36:53PM +0100, Sam Ravnborg wrote:
> > From e398a00f84db33ea5ae7f6ee12c54511ef7a94fc Mon Sep 17 00:00:00 2001
> > From: Sam Ravnborg <sam@ravnborg.org>
> > Date: Tue, 26 Dec 2017 18:09:35 +0100
> > Subject: [PATCH 4/4] build: fix that LZO file is always rebuilt
> > 
> > Port the make-cmd from linux kernel.
> > with the updated version $$ is porperly escaped, thus
> > the LZO rule works as intended.
> > And we avoid rebuilds when not required
> > 
> > Signed-off-by: Sam Ravnborg <sam@ravnborg.org>
> > ---
> >  scripts/Kbuild.include | 12 +++++++-----
> >  1 file changed, 7 insertions(+), 5 deletions(-)
> > 
> > diff --git a/scripts/Kbuild.include b/scripts/Kbuild.include
> > index 310dcdc94..fb3fa88fb 100644
> > --- a/scripts/Kbuild.include
> > +++ b/scripts/Kbuild.include
> > @@ -205,11 +205,13 @@ else
> >  arg-check = $(if $(strip $(cmd_$@)),,1)
> >  endif
> >  
> > -# >'< substitution is for echo to work,
> > -# >$< substitution to preserve $ when reloading .cmd file
> > -# note: when using inline perl scripts [perl -e '...$$t=1;...']
> > -# in $(cmd_xxx) double $$ your perl vars
> > -make-cmd = $(subst \\,\\\\,$(subst \#,\\\#,$(subst $$,$$$$,$(call escsq,$(cmd_$(1))))))
> > +# Replace >$< with >$$< to preserve $ when reloading the .cmd file
> > +# (needed for make)
> > +# Replace >#< with >\#< to avoid starting a comment in the .cmd file
> > +# (needed for make)
> > +# Replace >'< with >'\''< to be able to enclose the whole string in '...'
> > +# (needed for the shell)
> > +make-cmd = $(call escsq,$(subst \#,\\\#,$(subst $$,$$$$,$(cmd_$(1)))))
> >  
> 
> This doesn't work for me. Without this patch I do not see any rebuilds
> of lzo files, but with this patch I do.
> 
> Seems you are using another shell, dash vs. bash? Porting the second
> hunk of the kernel commit 164f0d2efaaef83 fixes it for me.

Ok, applied with the missing hunk from 164f0d2efaaef83:

@@ -220,7 +222,7 @@ any-prereq = $(filter-out $(PHONY),$?) $(filter-out $(PHONY) $(wildcard $^),$^)
 if_changed = $(if $(strip $(any-prereq) $(arg-check)),                       \
        @set -e;                                                             \
        $(echo-cmd) $(cmd_$(1));                                             \
-       echo 'cmd_$@ := $(make-cmd)' > $(dot-target).cmd)
+       printf '%s\n' 'cmd_$@ := $(make-cmd)' > $(dot-target).cmd)
 

Please let me know if I have overlooked something.

Sascha

-- 
Pengutronix e.K.                           |                             |
Industrial Linux Solutions                 | http://www.pengutronix.de/  |
Peiner Str. 6-8, 31137 Hildesheim, Germany | Phone: +49-5121-206917-0    |
Amtsgericht Hildesheim, HRA 2686           | Fax:   +49-5121-206917-5555 |

_______________________________________________
barebox mailing list
barebox@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/barebox

Re: [PATCH 4/4] build: fix that LZO file is always rebuilt

[Sam Ravnborg]

Hi Sascha.
> > > +# (needed for the shell)
> > > +make-cmd = $(call escsq,$(subst \#,\\\#,$(subst $$,$$$$,$(cmd_$(1)))))
> > >  
> > 
> > This doesn't work for me. Without this patch I do not see any rebuilds
> > of lzo files, but with this patch I do.
> > 
> > Seems you are using another shell, dash vs. bash? Porting the second
> > hunk of the kernel commit 164f0d2efaaef83 fixes it for me.

$ sh --version
GNU bash, version 4.2.46(1)-release (x86_64-redhat-linux-gnu)

This is a CentOS based system.

> 
> Ok, applied with the missing hunk from 164f0d2efaaef83:
> 
> @@ -220,7 +222,7 @@ any-prereq = $(filter-out $(PHONY),$?) $(filter-out $(PHONY) $(wildcard $^),$^)
>  if_changed = $(if $(strip $(any-prereq) $(arg-check)),                       \
>         @set -e;                                                             \
>         $(echo-cmd) $(cmd_$(1));                                             \
> -       echo 'cmd_$@ := $(make-cmd)' > $(dot-target).cmd)
> +       printf '%s\n' 'cmd_$@ := $(make-cmd)' > $(dot-target).cmd)
>

The above snip have no impact on my system - which is why I left it out.
To try to do minimal changes only.
I should have mentioned this - as well as the specific reference to the
kernel commit.

but in other words - tested with the above change. OK.

	Sam

_______________________________________________
barebox mailing list
barebox@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/barebox