mail archive of the barebox mailing list
 help / color / mirror / Atom feed
* [PATCH] genenv: create a gcc-like .d file for depenencies.
@ 2019-02-18 15:57 Tomaz Solc
  2019-02-20  8:57 ` Sascha Hauer
  0 siblings, 1 reply; 6+ messages in thread
From: Tomaz Solc @ 2019-02-18 15:57 UTC (permalink / raw)
  To: barebox; +Cc: Tomaz Solc

Commit 658af1ca mentions that environment build dependencies are not tracked
and the complete environment is rebuilt on every build.

However commit 105201e0 added if_changed to the make rule, so this is currently
not the case. Environment is only rebuilt if barebox_default_env is missing or
if CONFIG_DEFAULT_ENVIRONMENT_PATH has changed. genenv is not re-run if any
individual source files are modified. So if environment source files are
edited, Barebox is rebuilt with a stale barebox_default_env unless it is
manually deleted first.

With this commit genenv creates a .d file, similar to those created by gcc to
track C header dependencies. This is then passed to Kbuild with if_changed_dep
in the Makefile. This makes make re-run genenv if any source environment files are
changed. However, new environment files are still not detected automatically.
---
 defaultenv/Makefile |  2 +-
 scripts/genenv      | 18 +++++++++---------
 2 files changed, 10 insertions(+), 10 deletions(-)

diff --git a/defaultenv/Makefile b/defaultenv/Makefile
index f313b04e8..383b48e29 100644
--- a/defaultenv/Makefile
+++ b/defaultenv/Makefile
@@ -14,7 +14,7 @@ quiet_cmd_env_default = ENV     $@
 cmd_env_default = ($(srctree)/scripts/genenv $(srctree) $(objtree) $@ $(CONFIG_DEFAULT_ENVIRONMENT_PATH))
 
 $(obj)/barebox_default_env: FORCE
-	$(call if_changed,env_default)
+	$(call if_changed_dep,env_default)
 
 quiet_cmd_env_h = ENVH    $@
 cmd_env_h = cat $< | (cd $(obj) && $(objtree)/scripts/bin2c "__aligned(4) default_environment") > $@; \
diff --git a/scripts/genenv b/scripts/genenv
index 5ebe69963..756a2522b 100755
--- a/scripts/genenv
+++ b/scripts/genenv
@@ -30,29 +30,29 @@ abspath() {
 export -f abspath
 
 tempdir=$(abspath "${target}.genenv.tmp")
-tmpfile="$(mktemp)"
+depfile=$(dirname "${target}")/.$(basename "${target}").d
 
 mkdir -p "$tempdir"
 
 (cd $basedir
+# First listed dependency is ignored by fixdep, but must be a readable file.
+deps=scripts/genenv
 for i in $*; do
 	if [ -d $i ]; then
 		cp -r $i/* $tempdir
+		deps="$deps $(find $i -type f)"
 	else
 		cp -a $i $tempdir
+		deps="$deps $i"
 	fi
 done
+# This doesn't work with spaces in filenames, but Kbuild's fixdep doesn't seem
+# to support such filenames in any case.
+echo "${target}: $(echo ${deps})" > ${depfile}
 )
 
 find $tempdir -name '.svn' -o -name '*~' -delete
 
-$objtree/scripts/bareboxenv -s $tempdir ${tmpfile}
-
-diff "${tmpfile}" "${target}" >/dev/null 2>/dev/null
-if [ $? != 0 ]; then
-	mv "${tmpfile}" "${target}"
-else
-	rm ${tmpfile}
-fi
+$objtree/scripts/bareboxenv -s $tempdir ${target}
 
 rm -r $tempdir
-- 
2.11.0


_______________________________________________
barebox mailing list
barebox@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/barebox

^ permalink raw reply	[flat|nested] 6+ messages in thread

* Re: [PATCH] genenv: create a gcc-like .d file for depenencies.
  2019-02-18 15:57 [PATCH] genenv: create a gcc-like .d file for depenencies Tomaz Solc
@ 2019-02-20  8:57 ` Sascha Hauer
  2019-02-20 11:14   ` Tomaž Šolc
  0 siblings, 1 reply; 6+ messages in thread
From: Sascha Hauer @ 2019-02-20  8:57 UTC (permalink / raw)
  To: Tomaz Solc; +Cc: barebox

Hi Tomaz,

On Mon, Feb 18, 2019 at 04:57:12PM +0100, Tomaz Solc wrote:
> Commit 658af1ca mentions that environment build dependencies are not tracked
> and the complete environment is rebuilt on every build.
> 
> However commit 105201e0 added if_changed to the make rule, so this is currently
> not the case. Environment is only rebuilt if barebox_default_env is missing or
> if CONFIG_DEFAULT_ENVIRONMENT_PATH has changed. genenv is not re-run if any
> individual source files are modified. So if environment source files are
> edited, Barebox is rebuilt with a stale barebox_default_env unless it is
> manually deleted first.
> 
> With this commit genenv creates a .d file, similar to those created by gcc to
> track C header dependencies. This is then passed to Kbuild with if_changed_dep
> in the Makefile. This makes make re-run genenv if any source environment files are
> changed. However, new environment files are still not detected automatically.

Please try the attached patch. It should solve this issue once and for
all. With this the environments are always built, but only if they have
actually changed then the corresponding .S files are built.

Sascha

------------------------------8<--------------------------------

From 6c1275c02d9635ccdf7c263cbb3f853a20c670b3 Mon Sep 17 00:00:00 2001
From: Sascha Hauer <s.hauer@pengutronix.de>
Date: Wed, 20 Feb 2019 09:45:32 +0100
Subject: [PATCH] defaultenv: Fix dependencies

The defaultenv should be rebuilt once a file in it has changed.
the genenv script always generates the environment file to a temporary
file. Only if it has changed to the last target file the temporary file
is moved over the target file. This means we always have to call genenv,
thus replace "if_changed" with "cmd".
With this dependencies are correctly tracked. New or changed files
result in new image builds whereas unchanged environments do not
unnecessarily result in new images.

Signed-off-by: Sascha Hauer <s.hauer@pengutronix.de>
---
 scripts/Makefile.lib | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/scripts/Makefile.lib b/scripts/Makefile.lib
index 7b8643bf57..561247ab34 100644
--- a/scripts/Makefile.lib
+++ b/scripts/Makefile.lib
@@ -325,7 +325,7 @@ quiet_cmd_env = ENV     $@
 cmd_env=$(srctree)/scripts/genenv $(srctree) $(objtree) $@ $<
 
 %.bbenv$(DEFAULT_COMPRESSION_SUFFIX): % FORCE
-	$(call if_changed,env)
+	$(call cmd,env)
 
 # Bzip2
 # ---------------------------------------------------------------------------
-- 
2.20.1

-- 
Pengutronix e.K.                           |                             |
Industrial Linux Solutions                 | http://www.pengutronix.de/  |
Peiner Str. 6-8, 31137 Hildesheim, Germany | Phone: +49-5121-206917-0    |
Amtsgericht Hildesheim, HRA 2686           | Fax:   +49-5121-206917-5555 |

_______________________________________________
barebox mailing list
barebox@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/barebox

^ permalink raw reply	[flat|nested] 6+ messages in thread

* Re: [PATCH] genenv: create a gcc-like .d file for depenencies.
  2019-02-20  8:57 ` Sascha Hauer
@ 2019-02-20 11:14   ` Tomaž Šolc
  2019-02-20 11:28     ` Sascha Hauer
  0 siblings, 1 reply; 6+ messages in thread
From: Tomaž Šolc @ 2019-02-20 11:14 UTC (permalink / raw)
  To: Sascha Hauer; +Cc: barebox

Hi Sascha,

On 20. 02. 19 09:57, Sascha Hauer wrote:
> Hi Tomaz,
> 
> On Mon, Feb 18, 2019 at 04:57:12PM +0100, Tomaz Solc wrote:
>> With this commit genenv creates a .d file, similar to those created by gcc to
>> track C header dependencies. This is then passed to Kbuild with if_changed_dep
>> in the Makefile. This makes make re-run genenv if any source environment files are
>> changed. However, new environment files are still not detected automatically.
> 
> Please try the attached patch. It should solve this issue once and for
> all. With this the environments are always built, but only if they have
> actually changed then the corresponding .S files are built.

Unfortunately with your patch the final image still gets rebuilt with a 
stale environment unless I delete defaultenv/barebox_default_env.

See transcript bellow, where I'm using your patch. I modify one 
environment file for Raspberry Pi. After just calling "make", the image 
doesn't include the change.

Best regards
Tomaž

$ make rpi_defconfig
$ make
[...]
$ strings images/barebox-raspberry-pi-3.img | grep global.test
$ echo global.test="foo" >> 
arch/arm/boards/raspberry-pi/env/init/bootargs-base
$ make
make[1]: 'include/generated/mach-types.h' is up to date.
   CHK     include/generated/version.h
   CHK     include/generated/utsrelease.h
   UPD     include/generated/utsrelease.h
   CREATE  include/config.h
   CHK     include/generated/compile.h
   CC      common/version.o
   CC      common/globalvar.o
   CHK     include/generated/passwd.h
   LD      common/built-in.o
   ENV     defaultenv/defaultenv-2-base.bbenv
   GEN     .version
   CHK     include/generated/compile.h
   UPD     include/generated/compile.h
   CC      common/version.o
   CHK     include/generated/passwd.h
   LD      common/built-in.o
   LD      .tmp_barebox1
   KSYM    .tmp_kallsyms1.S
   AS      .tmp_kallsyms1.o
   LD      .tmp_barebox2
   KSYM    .tmp_kallsyms2.S
   AS      .tmp_kallsyms2.o
   LD      barebox
   SYSMAP  System.map
   OBJCOPY barebox.bin
   SHIPPED_S images/barebox.z
   AS      images/piggy.o
   LD      images/start_raspberry_pi1.pbl
   OBJCOPYB images/start_raspberry_pi1.pblb
   SHIPPED images/barebox-raspberry-pi-1.img
   LD      images/start_raspberry_pi2.pbl
   OBJCOPYB images/start_raspberry_pi2.pblb
   SHIPPED images/barebox-raspberry-pi-2.img
   LD      images/start_raspberry_pi3.pbl
   OBJCOPYB images/start_raspberry_pi3.pblb
   SHIPPED images/barebox-raspberry-pi-3.img
   LN      images/../barebox-flash-image
images built:
barebox-raspberry-pi-1.img
barebox-raspberry-pi-2.img
barebox-raspberry-pi-3.img
$ strings images/barebox-raspberry-pi-3.img | grep global.test
$ rm defaultenv/barebox_default_env
$ make
make[1]: 'include/generated/mach-types.h' is up to date.
   CHK     include/generated/version.h
   CHK     include/generated/utsrelease.h
   CREATE  include/config.h
   CHK     include/generated/compile.h
   CHK     include/generated/passwd.h
   ENV     defaultenv/barebox_default_env
   ENVH    defaultenv/barebox_default_env.h
   CC      defaultenv/defaultenv.o
   ENV     defaultenv/defaultenv-2-base.bbenv
   LD      defaultenv/built-in.o
   GEN     .version
   CHK     include/generated/compile.h
   UPD     include/generated/compile.h
   CC      common/version.o
   CHK     include/generated/passwd.h
   LD      common/built-in.o
   LD      .tmp_barebox1
   KSYM    .tmp_kallsyms1.S
   AS      .tmp_kallsyms1.o
   LD      .tmp_barebox2
   KSYM    .tmp_kallsyms2.S
   AS      .tmp_kallsyms2.o
   LD      barebox
   SYSMAP  System.map
   OBJCOPY barebox.bin
   SHIPPED_S images/barebox.z
   AS      images/piggy.o
   LD      images/start_raspberry_pi1.pbl
   OBJCOPYB images/start_raspberry_pi1.pblb
   SHIPPED images/barebox-raspberry-pi-1.img
   LD      images/start_raspberry_pi2.pbl
   OBJCOPYB images/start_raspberry_pi2.pblb
   SHIPPED images/barebox-raspberry-pi-2.img
   LD      images/start_raspberry_pi3.pbl
   OBJCOPYB images/start_raspberry_pi3.pblb
   SHIPPED images/barebox-raspberry-pi-3.img
   LN      images/../barebox-flash-image
images built:
barebox-raspberry-pi-1.img
barebox-raspberry-pi-2.img
barebox-raspberry-pi-3.img
$ strings images/barebox-raspberry-pi-3.img | grep global.test
global.test=foo
$

_______________________________________________
barebox mailing list
barebox@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/barebox

^ permalink raw reply	[flat|nested] 6+ messages in thread

* Re: [PATCH] genenv: create a gcc-like .d file for depenencies.
  2019-02-20 11:14   ` Tomaž Šolc
@ 2019-02-20 11:28     ` Sascha Hauer
  2019-02-21  8:34       ` Tomaž Šolc
  0 siblings, 1 reply; 6+ messages in thread
From: Sascha Hauer @ 2019-02-20 11:28 UTC (permalink / raw)
  To: Tomaž Šolc; +Cc: barebox

On Wed, Feb 20, 2019 at 12:14:43PM +0100, Tomaž Šolc wrote:
> Hi Sascha,
> 
> On 20. 02. 19 09:57, Sascha Hauer wrote:
> > Hi Tomaz,
> > 
> > On Mon, Feb 18, 2019 at 04:57:12PM +0100, Tomaz Solc wrote:
> > > With this commit genenv creates a .d file, similar to those created by gcc to
> > > track C header dependencies. This is then passed to Kbuild with if_changed_dep
> > > in the Makefile. This makes make re-run genenv if any source environment files are
> > > changed. However, new environment files are still not detected automatically.
> > 
> > Please try the attached patch. It should solve this issue once and for
> > all. With this the environments are always built, but only if they have
> > actually changed then the corresponding .S files are built.
> 
> Unfortunately with your patch the final image still gets rebuilt with a
> stale environment unless I delete defaultenv/barebox_default_env.
> 
> See transcript bellow, where I'm using your patch. I modify one environment
> file for Raspberry Pi. After just calling "make", the image doesn't include
> the change.

Right, I missed the environment snippet that is generated from
CONFIG_DEFAULT_ENVIRONMENT_PATH which is generated at a different place.
Please try this updated patch.

Sascha

-----------------------------------8<-------------------------------

From c41a478e59a74b66a3de5c597224f02e0b39fa12 Mon Sep 17 00:00:00 2001
From: Sascha Hauer <s.hauer@pengutronix.de>
Date: Wed, 20 Feb 2019 09:45:32 +0100
Subject: [PATCH] defaultenv: Fix dependencies

The defaultenv should be rebuilt once a file in it has changed.
the genenv script always generates the environment file to a temporary
file. Only if it has changed to the last target file the temporary file
is moved over the target file. This means we always have to call genenv,
thus replace "if_changed" with "cmd".
With this dependencies are correctly tracked. New or changed files
result in new image builds whereas unchanged environments do not
unnecessarily result in new images.

Signed-off-by: Sascha Hauer <s.hauer@pengutronix.de>
---
 defaultenv/Makefile  | 2 +-
 scripts/Makefile.lib | 2 +-
 2 files changed, 2 insertions(+), 2 deletions(-)

diff --git a/defaultenv/Makefile b/defaultenv/Makefile
index f313b04e84..03b329c4dd 100644
--- a/defaultenv/Makefile
+++ b/defaultenv/Makefile
@@ -14,7 +14,7 @@ quiet_cmd_env_default = ENV     $@
 cmd_env_default = ($(srctree)/scripts/genenv $(srctree) $(objtree) $@ $(CONFIG_DEFAULT_ENVIRONMENT_PATH))
 
 $(obj)/barebox_default_env: FORCE
-	$(call if_changed,env_default)
+	$(call cmd,env_default)
 
 quiet_cmd_env_h = ENVH    $@
 cmd_env_h = cat $< | (cd $(obj) && $(objtree)/scripts/bin2c "__aligned(4) default_environment") > $@; \
diff --git a/scripts/Makefile.lib b/scripts/Makefile.lib
index 7b8643bf57..561247ab34 100644
--- a/scripts/Makefile.lib
+++ b/scripts/Makefile.lib
@@ -325,7 +325,7 @@ quiet_cmd_env = ENV     $@
 cmd_env=$(srctree)/scripts/genenv $(srctree) $(objtree) $@ $<
 
 %.bbenv$(DEFAULT_COMPRESSION_SUFFIX): % FORCE
-	$(call if_changed,env)
+	$(call cmd,env)
 
 # Bzip2
 # ---------------------------------------------------------------------------
-- 
2.20.1


-- 
Pengutronix e.K.                           |                             |
Industrial Linux Solutions                 | http://www.pengutronix.de/  |
Peiner Str. 6-8, 31137 Hildesheim, Germany | Phone: +49-5121-206917-0    |
Amtsgericht Hildesheim, HRA 2686           | Fax:   +49-5121-206917-5555 |

_______________________________________________
barebox mailing list
barebox@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/barebox

^ permalink raw reply	[flat|nested] 6+ messages in thread

* Re: [PATCH] genenv: create a gcc-like .d file for depenencies.
  2019-02-20 11:28     ` Sascha Hauer
@ 2019-02-21  8:34       ` Tomaž Šolc
  2019-02-22  7:34         ` Sascha Hauer
  0 siblings, 1 reply; 6+ messages in thread
From: Tomaž Šolc @ 2019-02-21  8:34 UTC (permalink / raw)
  To: Sascha Hauer; +Cc: barebox

Hi Sascha,

On 20. 02. 19 12:28, Sascha Hauer wrote:
> Right, I missed the environment snippet that is generated from
> CONFIG_DEFAULT_ENVIRONMENT_PATH which is generated at a different place.
> Please try this updated patch.

Thanks, as far as I can see this patch works correctly now.

I suggest adding a comment in defaultenv/Makefile that rebuilding the 
environment on every build is the intended behavior. It seems this was 
already once erroneously considered a bug (see commit 105201e0).

Best regards
Tomaž

_______________________________________________
barebox mailing list
barebox@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/barebox

^ permalink raw reply	[flat|nested] 6+ messages in thread

* Re: [PATCH] genenv: create a gcc-like .d file for depenencies.
  2019-02-21  8:34       ` Tomaž Šolc
@ 2019-02-22  7:34         ` Sascha Hauer
  0 siblings, 0 replies; 6+ messages in thread
From: Sascha Hauer @ 2019-02-22  7:34 UTC (permalink / raw)
  To: Tomaž Šolc; +Cc: barebox

On Thu, Feb 21, 2019 at 09:34:47AM +0100, Tomaž Šolc wrote:
> Hi Sascha,
> 
> On 20. 02. 19 12:28, Sascha Hauer wrote:
> > Right, I missed the environment snippet that is generated from
> > CONFIG_DEFAULT_ENVIRONMENT_PATH which is generated at a different place.
> > Please try this updated patch.
> 
> Thanks, as far as I can see this patch works correctly now.
> 
> I suggest adding a comment in defaultenv/Makefile that rebuilding the
> environment on every build is the intended behavior. It seems this was
> already once erroneously considered a bug (see commit 105201e0).

Good idea, just did that.

Regards,
  Sascha

-- 
Pengutronix e.K.                           |                             |
Industrial Linux Solutions                 | http://www.pengutronix.de/  |
Peiner Str. 6-8, 31137 Hildesheim, Germany | Phone: +49-5121-206917-0    |
Amtsgericht Hildesheim, HRA 2686           | Fax:   +49-5121-206917-5555 |

_______________________________________________
barebox mailing list
barebox@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/barebox

^ permalink raw reply	[flat|nested] 6+ messages in thread

end of thread, other threads:[~2019-02-22  7:34 UTC | newest]

Thread overview: 6+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2019-02-18 15:57 [PATCH] genenv: create a gcc-like .d file for depenencies Tomaz Solc
2019-02-20  8:57 ` Sascha Hauer
2019-02-20 11:14   ` Tomaž Šolc
2019-02-20 11:28     ` Sascha Hauer
2019-02-21  8:34       ` Tomaž Šolc
2019-02-22  7:34         ` Sascha Hauer

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox