Oselas 2018.12.0 toolchain seems to break barebox for AM335x

Oselas 2018.12.0 toolchain seems to break barebox for AM335x

[Andreas Geisenhainer]

Hello barebox mailinglist.

While updating my local ptxdist setup I ran into the following problem.

Updating the toolchain to OSELAS_2018.12.0 and copiling barebox
it decided not to start anymore. To my surprise, compiling the
exact same source with the previous OSELAS toolchain (2018.02.0)
just works.
I installed the toolchains through the official debian repository.

We're still running an older barebox (2015.09), so i grabed a
stock, barebox 2019.06.1 and it produces the exact same behaviour.

We're running barebox an a Phytec PhyCORE AM335x platform, and the
behaviour does emerge for the default "barebox-am335x-phytec-phycore.img"
image from barebox.

* Exhibit A: barebox 2019.06.1 compiled with OSELAS_2018.12.0 (gcc 8.2.1)
----------------------------------8<-------------------------------------
barebox@sbo800:/mnt/10.0.1.6/rootbootm -vv -c barebox_2019.06.1_k1.img

Loading ARM barebox image 'barebox_2019.06.1_k1.img'
using internal devicetree
OS image not yet relocated
Passing control to OMAP barebox handler
exitcall-> devices_shutdown+0x1/0x24
exitcall-> arch_shutdown+0x1/0x1c
---------------------------------->8-------------------------------------

* Exhibit B: barebox 2019.06.1 compiled with OSELAS_2018.02.0 (gcc 7.3.1)
----------------------------------8<-------------------------------------
barebox@sbo800:/mnt/10.0.1.6/rootbootm -vv -c barebox_2019.06.1_k2.img

Loading ARM barebox image 'barebox_2019.06.1_k2.img'
using internal devicetree
OS image not yet relocated
Passing control to OMAP barebox handler
exitcall-> devices_shutdown+0x1/0x24
exitcall-> arch_shutdown+0x1/0x1c
start.c: memory at 0x80000000, size 0x20000000
start.c: found Compressed DTB in boarddata, copying to 0x9fdfd040
start.c: initializing malloc pool at 0x8fefe820 (size 0x0fefe820)
start.c: starting barebox...
----------------------------------<8-------------------------------------

What's the best course of action to debug this further?

There's a slight possibility for a barebox config option to be the culprit,
since i copied our local config into the extracted archive.
But it does not really explain why this breaks.

kind regards
Andreas Geisenhainer

-- 
        _
   __ _| |_ ___
  / _` | __/ __|
| (_| | |_\__ \
  \__,_|\__|___/
Elektronik GmbH


_______________________________________________
barebox mailing list
barebox@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/barebox

Re: Oselas 2018.12.0 toolchain seems to break barebox for AM335x

[Sascha Hauer]

Hi Andreas,

On Mon, Jul 08, 2019 at 04:15:00PM +0200, Andreas Geisenhainer wrote:
> Hello barebox mailinglist.
> 
> While updating my local ptxdist setup I ran into the following problem.
> 
> Updating the toolchain to OSELAS_2018.12.0 and copiling barebox
> it decided not to start anymore. To my surprise, compiling the
> exact same source with the previous OSELAS toolchain (2018.02.0)
> just works.
> I installed the toolchains through the official debian repository.
> 
> We're still running an older barebox (2015.09), so i grabed a
> stock, barebox 2019.06.1 and it produces the exact same behaviour.
> 
> We're running barebox an a Phytec PhyCORE AM335x platform, and the
> behaviour does emerge for the default "barebox-am335x-phytec-phycore.img"
> image from barebox.

I just tried to reproduce this on a beaglebone black as this is at least
the same SoC. Unfortunately I was not successful. I saw this one time:

| barebox@TI AM335x BeagleBone black:/ bootm mnt/tftp/b
| eth0: 100Mbps full duplex link detected
| eth0: DHCP client bound to address 192.168.24.94
| 
| Loading ARM barebox image 'mnt/tftp/b'
| unable to handle paging request at address 0x020c8260
| pc : [<900a7ba6>]    lr : [<900a7c3b>]
| sp : 9ffefd70  ip : 00000000  fp : 00000000
| r10: 900a4b50  r9 : 900a7abd  r8 : 9fe6aba2
| r7 : 9ffefd70  r6 : 40000000  r5 : 00000d80  r4 : 402f0400
| r3 : 020c8260  r2 : 00000006  r1 : 00000d80  r0 : 40000000
| Flags: nZCv  IRQs off  FIQs on  Mode SVC_32
| 
| WARNING: [<9fe58229>] (unwind_backtrace+0x1/0x68) from [<9fe00ff5>] (panic+0x1d/0x34)
| WARNING: [<9fe00ff5>] (panic+0x1d/0x34) from [<9fe5657d>] (do_exception+0xd/0x10)
| WARNING: [<9fe5657d>] (do_exception+0xd/0x10) from [<9fe565dd>] (do_data_abort+0x21/0x2c)
| WARNING: [<9fe565dd>] (do_data_abort+0x21/0x2c) from [<9fe561f4>] (do_abort_6+0x48/0x54)

Unfortunately this disappeared after some changes and never came back.

Does your barebox always crash or does the behaviour change when you try
adding debugging? you might want to add some putc_ll() at earlier places
to see how far you get.

Sascha

-- 
Pengutronix e.K.                           |                             |
Industrial Linux Solutions                 | http://www.pengutronix.de/  |
Peiner Str. 6-8, 31137 Hildesheim, Germany | Phone: +49-5121-206917-0    |
Amtsgericht Hildesheim, HRA 2686           | Fax:   +49-5121-206917-5555 |

_______________________________________________
barebox mailing list
barebox@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/barebox

Re: Oselas 2018.12.0 toolchain seems to break barebox for AM335x

[Andreas Geisenhainer]

Hello Sascha, hello ML.

On 2019-07-10 9:58 a.m., Sascha Hauer wrote:
> Hi Andreas,
>
> On Mon, Jul 08, 2019 at 04:15:00PM +0200, Andreas Geisenhainer wrote:
>> We're running barebox an a Phytec PhyCORE AM335x platform, and the
>> behaviour does emerge for the default "barebox-am335x-phytec-phycore.img"
>> image from barebox.
> I just tried to reproduce this on a beaglebone black as this is at least
> the same SoC. Unfortunately I was not successful.
>
> Unfortunately this disappeared after some changes and never came back.
Thanks for trying.
> Does your barebox always crash or does the behaviour change when you try
> adding debugging? you might want to add some putc_ll() at earlier places
> to see how far you get.
It crashes every time, as far as i can tell, since there is no more output.
I'm working around the the problem for a bit, but will investigate further
into it. There should be a Beaglebone Black around here. I'll try to
reproduce it on this platform and get back to you.

kind regards
Andreas

-- 
        _
   __ _| |_ ___
  / _` | __/ __|
| (_| | |_\__ \
  \__,_|\__|___/
Elektronik GmbH


_______________________________________________
barebox mailing list
barebox@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/barebox

Re: Oselas 2018.12.0 toolchain seems to break barebox for AM335x

[Andreas Geisenhainer]

[-- Attachment #1: Type: text/plain, Size: 2123 bytes --]

Hello Sascha, hello barebox-ml.

 > On 2019-07-10 9:58 a.m., Sascha Hauer wrote:
 >> On Mon, Jul 08, 2019 at 04:15:00PM +0200, Andreas Geisenhainer wrote:
 >>> We're running barebox an a Phytec PhyCORE AM335x platform, and the
 >>> behaviour does emerge for the default
 >>> "barebox-am335x-phytec-phycore.img"
 >>> image from barebox.
 >> I just tried to reproduce this on a beaglebone black as this is at least
 >> the same SoC. Unfortunately I was not successful.

I finally found the time to reproduce this on a beaglebone black.
I tried two different barebox versions (v2018.03 and v2019.10)
without any changes to the source.

My current steps where

1) make am335x_mlo_defconfig / am335x_defconfig
	(depending on the version)
2) make
3) copied two files onto the uSD card I booted the boneblack
    from:
     - barebox-am33xx-beaglebone.img (I)
     - barebox-am33xx-phytec-phycore.img (II)
4) booted the boneblack and loaded each file manually using the
    `bootm` command.


I'm working under the following assumption:
  a) the beagleboneblack image is working for our AM335x-phycore
  b) the phytec-phycore image should work on a boneblack
      (at least rudimentary)

When I'm using the OSELAS.2018.02.0 toolchain a) und b) hold up. 
Additionally we get
   c) on the boneblack both images (I) and (II) are working

With the OSELAS.2019.09.0 toolchain, the boneblack image (I) is
working on the boneblack, but the phycore-image (II) is not.

There seem to be at least two slightly different behaviors
I've been observing:
Yesterday it justs stops working, no further output, nothing (see Part 
B) within attachment.
Two days ago it complained about a "unhandled NULL pointer dereference", 
followed by a reset of the chip (i captured that within the attachment 
at C).

One more point: I've chosen to build the toolchains for 
"arm-cortexa8-linux-gnueabihf".
To rule out problems with this decision I build a more generic 
"arm-v7a-linux-gnueabihf"
one, but the problem persists with it, too.

Is there anything more I can do on my end?
I'm open for any ideas, here. :)

with kind regards
Andreas Geisenhainer

[-- Attachment #2: barebox_observations.txt --]
[-- Type: text/plain, Size: 2484 bytes --]


###########################
# A) working barebox 2018.03.0 compiled with OSELAS_2018.02
###############

Loading ARM barebox image '/mnt/10.0.1.6/root/barebox-am33xx-phytec-phycore-no-spi.img'
exitcall-> nv_exit+0x1/0x20
exitcall-> devices_shutdown+0x1/0x28
exitcall-> arch_shutdown+0x1/0x1c


barebox 2018.03.0-gcfddd8e026db #14 Wed Oct 23 10:48:06 CEST 2019


Board: Phytec phyCORE AM335x
initcall-> globalvar_init+0x1/0x2c
initcall-> platform_init+0x1/0xc
initcall-> mdio_bus_init+0x1/0xc
initcall-> usb_bus_init+0x1/0xc
initcall-> spi_bus_init+0x1/0xc
initcall-> i2c_bus_init+0x1/0x10
initcall-> gpio_desc_alloc+0x1/0x18
initcall-> fs_bus_init+0x1/0xc
initcall-> dmtimer_init+0x1/0x30
initcall-> of_arm_init+0x1/0x2c
initcall-> unwind_init+0x1/0x24
initcall-> pcs_init+0x1/0xc
initcall-> net_init+0x1/0x64
initcall-> init_fs+0x1/0x2c
initcall-> omap_init+0x1/0x5c
initcall-> ns16550_serial_driver_register+0x1/0xc
initcall-> dos_partition_init+0x1/0xc
initcall-> am33xx_scrm_init+0x1/0xc
initcall-> mmu_init+0x1/0x1e8
[...]


###########################
# B) broken barebox 2019.10.0 compiled with OSELAS_2019.09
########
barebox@TI AM335x BeagleBone black:/mnt/mmc0.0 bootm -vv -c barebox-am33xx-phytec-phycore.img

Loading ARM barebox image 'barebox-am33xx-phytec-phycore.img'
OS image not yet relocated
Passing control to OMAP barebox handler
exitcall-> nv_exit+0x1/0x20
exitcall-> state_shutdown+0x1/0x20
exitcall-> devices_shutdown+0x1/0x24
exitcall-> arch_shutdown+0x1/0x1c

-> waits here (nothing seems to happen)


########################
# C) broken barebox 2018.03.0 compiled with OSELAS_2019.09
#########

Loading ARM barebox image '/mnt/10.0.1.6/root/barebox-am33xx-phytec-phycore-no-spi.img'
exitcall-> nv_exit+0x1/0x20
exitcall-> devices_shutdown+0x1/0x28
exitcall-> arch_shutdown+0x1/0x1c
unable to handle NULL pointer dereference at address 0x000009db
pc : [<88009d1a>]    lr : [<88009d35>]
sp : 8ffef918  ip : 00000000  fp : 00000000
r10: 00000000  r9 : 88009c21  r8 : 87fea400
r7 : 00000004  r6 : 8ffef954  r5 : 88009c18  r4 : 0000000e
r3 : 01000000  r2 : 00000003  r1 : 000009d8  r0 : 00000002
Flags: nzcv  IRQs off  FIQs on  Mode SVC_32

[<8fe52ca5>] (unwind_backtrace+0x1/0x58) from [<8fe00c11>] (panic+0x1d/0x34)
[<8fe00c11>] (panic+0x1d/0x34) from [<8fe512cd>] (do_exception+0xd/0x10)
[<8fe512cd>] (do_exception+0xd/0x10) from [<8fe5132d>] (do_data_abort+0x21/0x2c)
[<8fe5132d>] (do_data_abort+0x21/0x2c) from [<8fe50dd4>] (do_abort_6+0x48/0x54)

->reset

[-- Attachment #3: Type: text/plain, Size: 149 bytes --]

_______________________________________________
barebox mailing list
barebox@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/barebox

Re: Oselas 2018.12.0 toolchain seems to break barebox for AM335x

[Andreas Geisenhainer]

Good day, everyone.

*bump*

On 2019-10-25 10:48 a.m., Andreas Geisenhainer wrote:
> Hello Sascha, hello barebox-ml.
> 
>  > On 2019-07-10 9:58 a.m., Sascha Hauer wrote:
>  >> On Mon, Jul 08, 2019 at 04:15:00PM +0200, Andreas Geisenhainer wrote:
>  >>> We're running barebox an a Phytec PhyCORE AM335x platform, and the
>  >>> behaviour does emerge for the default
>  >>> "barebox-am335x-phytec-phycore.img"
>  >>> image from barebox.
>  >> I just tried to reproduce this on a beaglebone black as this is at 
> least
>  >> the same SoC. Unfortunately I was not successful.
> 
> I finally found the time to reproduce this on a beaglebone black.
> I tried two different barebox versions (v2018.03 and v2019.10)
> without any changes to the source.
> 
> My current steps where
> 
> 1) make am335x_mlo_defconfig / am335x_defconfig
>      (depending on the version)
> 2) make
> 3) copied two files onto the uSD card I booted the boneblack
>     from:
>      - barebox-am33xx-beaglebone.img (I)
>      - barebox-am33xx-phytec-phycore.img (II)
> 4) booted the boneblack and loaded each file manually using the
>     `bootm` command.
> 
> 
> I'm working under the following assumption:
>   a) the beagleboneblack image is working for our AM335x-phycore
>   b) the phytec-phycore image should work on a boneblack
>       (at least rudimentary)
> 
> When I'm using the OSELAS.2018.02.0 toolchain a) und b) hold up. 
> Additionally we get
>    c) on the boneblack both images (I) and (II) are working
> 
> With the OSELAS.2019.09.0 toolchain, the boneblack image (I) is
> working on the boneblack, but the phycore-image (II) is not.
> 
> There seem to be at least two slightly different behaviors
> I've been observing:
> Yesterday it justs stops working, no further output, nothing (see Part 
> B) within attachment.
> Two days ago it complained about a "unhandled NULL pointer dereference", 
> followed by a reset of the chip (i captured that within the attachment 
> at C).
> 
> One more point: I've chosen to build the toolchains for 
> "arm-cortexa8-linux-gnueabihf".
> To rule out problems with this decision I build a more generic 
> "arm-v7a-linux-gnueabihf"
> one, but the problem persists with it, too.
> 
> Is there anything more I can do on my end?
> I'm open for any ideas, here. :)

Was anyone able find something regarding this?

Do some crazy ideas exist anywhere?

kind regards,
Andreas Geisenhainer

_______________________________________________
barebox mailing list
barebox@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/barebox

Re: Oselas 2018.12.0 toolchain seems to break barebox for AM335x

[Sascha Hauer]

Hi Andreas,

On Tue, Nov 12, 2019 at 08:48:04AM +0100, Andreas Geisenhainer wrote:
> Good day, everyone.
> 
> *bump*
> 
> On 2019-10-25 10:48 a.m., Andreas Geisenhainer wrote:
> > Hello Sascha, hello barebox-ml.
> > 
> >  > On 2019-07-10 9:58 a.m., Sascha Hauer wrote:
> >  >> On Mon, Jul 08, 2019 at 04:15:00PM +0200, Andreas Geisenhainer wrote:
> >  >>> We're running barebox an a Phytec PhyCORE AM335x platform, and the
> >  >>> behaviour does emerge for the default
> >  >>> "barebox-am335x-phytec-phycore.img"
> >  >>> image from barebox.
> >  >> I just tried to reproduce this on a beaglebone black as this is at
> > least
> >  >> the same SoC. Unfortunately I was not successful.
> > 
> > I finally found the time to reproduce this on a beaglebone black.
> > I tried two different barebox versions (v2018.03 and v2019.10)
> > without any changes to the source.
> > 
> > My current steps where
> > 
> > 1) make am335x_mlo_defconfig / am335x_defconfig
> >      (depending on the version)
> > 2) make
> > 3) copied two files onto the uSD card I booted the boneblack
> >     from:
> >      - barebox-am33xx-beaglebone.img (I)
> >      - barebox-am33xx-phytec-phycore.img (II)
> > 4) booted the boneblack and loaded each file manually using the
> >     `bootm` command.
> > 
> > 
> > I'm working under the following assumption:
> >   a) the beagleboneblack image is working for our AM335x-phycore
> >   b) the phytec-phycore image should work on a boneblack
> >       (at least rudimentary)
> > 
> > When I'm using the OSELAS.2018.02.0 toolchain a) und b) hold up.
> > Additionally we get
> >    c) on the boneblack both images (I) and (II) are working
> > 
> > With the OSELAS.2019.09.0 toolchain, the boneblack image (I) is
> > working on the boneblack, but the phycore-image (II) is not.
> > 
> > There seem to be at least two slightly different behaviors
> > I've been observing:
> > Yesterday it justs stops working, no further output, nothing (see Part
> > B) within attachment.
> > Two days ago it complained about a "unhandled NULL pointer dereference",
> > followed by a reset of the chip (i captured that within the attachment
> > at C).
> > 
> > One more point: I've chosen to build the toolchains for
> > "arm-cortexa8-linux-gnueabihf".
> > To rule out problems with this decision I build a more generic
> > "arm-v7a-linux-gnueabihf"
> > one, but the problem persists with it, too.
> > 
> > Is there anything more I can do on my end?
> > I'm open for any ideas, here. :)
> 
> Was anyone able find something regarding this?
> 
> Do some crazy ideas exist anywhere?

Sorry for the delay, I probably knew this issue will be something
unfortunate to look at...

Anyway, I can reproduce this issue here. The problem goes down to
am335x_sdram_size() which is executed at an address which it is not
linked at. Beginning from a certain compiler version gcc is clever
enough to compile the switch/case used there to assign a variable
as a lookup table placed in the data section. This table is looked for
not at the address we are running at (somewhere in SDRAM), but at
an offset to the address we are linked at (0x0). There are two solutions
to this, first one I'd like to apply to master, see appended patch. This
is only a local change to the affected file. However, other files
may suffer from the same issue.
The second solution is to add -fPIE to CPPFLAGS which is a more general
approach. I'd like to give this a try in -next.

Sascha

-----------------------------8<---------------------------

From 3b8b70bfd743c303349f67c892631cb04c2b3c68 Mon Sep 17 00:00:00 2001
From: Sascha Hauer <s.hauer@pengutronix.de>
Date: Tue, 12 Nov 2019 11:53:48 +0100
Subject: [PATCH] ARM: am335x: Fix am335x_sdram_size() not running at link
 address

am335x_sdram_size() may be called when we are not running at the
address we are linked at. This means tree switch conversions and
jump tables will not work. Disable these in the CFLAGS for this
file.
This fixes a crash in am335x_sdram_size() with newer gcc versions.

Signed-off-by: Sascha Hauer <s.hauer@pengutronix.de>
---
 arch/arm/mach-omap/Makefile | 2 ++
 1 file changed, 2 insertions(+)

diff --git a/arch/arm/mach-omap/Makefile b/arch/arm/mach-omap/Makefile
index 36b2aa090e..06cd191f95 100644
--- a/arch/arm/mach-omap/Makefile
+++ b/arch/arm/mach-omap/Makefile
@@ -22,6 +22,8 @@ pbl-$(CONFIG_ARCH_OMAP3) += omap3_generic.o auxcr.o
 obj-$(CONFIG_ARCH_OMAP4) += omap4_generic.o omap4_clock.o
 pbl-$(CONFIG_ARCH_OMAP4) += omap4_generic.o omap4_clock.o
 obj-pbl-$(CONFIG_ARCH_AM33XX) += am33xx_generic.o am33xx_clock.o am33xx_mux.o am3xxx.o
+CFLAGS_pbl-am33xx_generic.o := -fno-tree-switch-conversion -fno-jump-tables
+CFLAGS_am33xx_generic.o := -fno-tree-switch-conversion -fno-jump-tables
 obj-pbl-$(CONFIG_ARCH_AM35XX) += am3xxx.o am35xx_emif4.o
 obj-$(CONFIG_ARCH_AM33XX) += am33xx_scrm.o
 obj-$(CONFIG_ARCH_OMAP3) += omap3_clock.o
-- 
2.24.0.rc1



-- 
Pengutronix e.K.                           |                             |
Steuerwalder Str. 21                       | http://www.pengutronix.de/  |
31137 Hildesheim, Germany                  | Phone: +49-5121-206917-0    |
Amtsgericht Hildesheim, HRA 2686           | Fax:   +49-5121-206917-5555 |

_______________________________________________
barebox mailing list
barebox@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/barebox

Re: Oselas 2018.12.0 toolchain seems to break barebox for AM335x

[Andreas Geisenhainer]

On 2019-11-12 12:00 p.m., Sascha Hauer wrote:
> 
> Sorry for the delay, I probably knew this issue will be something
> unfortunate to look at...

No Problem, thanks for your hard work. :)

> The second solution is to add -fPIE to CPPFLAGS which is a more general
> approach. I'd like to give this a try in -next.

I tried this more generic, second approach on our local setup and it 
works. \o/

Thanks and kind regards
Andreas

_______________________________________________
barebox mailing list
barebox@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/barebox