Re: [PATCH v2 0/6] ARM: Rockchip: Read amount of memory from DDR controller

[Michael Riesch <michael.riesch@wolfvision.net>]

Hi Sascha,

On 4/3/23 09:32, Sascha Hauer wrote:
> On Fri, Mar 31, 2023 at 05:42:54PM +0200, Michael Riesch wrote:
>> Hi Sascha,
>>
>> On 3/28/23 09:40, Sascha Hauer wrote:
>>> This series adds support for reading the amount of memory from
>>> the DDR controller. This helps on boards which come with
>>> different amounts of memory like the Radxa Rock3a.
>>>
>>> This series also fixes issues with an upstream TF-A firmware. With this
>>> the IRAM where the bootsource is stored is no longer accessible in
>>> normal mode. We have to read its contents before starting the TF-A.
>>> For this it became necessary to add a common barebox entry function
>>> for rk3568, to get a common place to read the IRAM contents.
>>
>> Nice, thanks for your efforts! Now it should be possible to remove the
>> memory nodes from arch/arm/dts/rk356*, right?
>>
>> I tried this on a ROCK3A with 8 GB and the memory calculation returned
>> the correct result. There are now two ram devices
>>
>>    `-- mem0
>>       `-- 0x00000000-0x10fffffff (   4.3 GiB): /dev/ram1
>>    `-- mem1
>>       `-- 0x00000000-0xef5fffff (   3.7 GiB): /dev/ram0
>>    `-- mem2
>>       `-- 0x00000000-0xffffffffffffffff (   0 Bytes): /dev/mem
> 
> At first sight I was a bit puzzled that both memory regions start a 0x0.
> That's correct for the devinfo output of course. The output of 'iomem'
> would show the situation better here.

barebox@Radxa ROCK3 Model A:/ iomem
0x0000000000000000 - 0xffffffffffffffff (size 0x0000000000000000) iomem
  0x0000000000a00000 - 0x00000000efffffff (size 0x00000000ef600000) ram0
    0x00000000afcf0e00 - 0x00000000efcf0dff (size 0x0000000040000000)
malloc space
    0x00000000efcf0e00 - 0x00000000efcfffd1 (size 0x000000000000f1d2)
board data
    0x00000000efd00000 - 0x00000000efd99a4f (size 0x0000000000099a50)
barebox
    0x00000000efd99a50 - 0x00000000efde59ff (size 0x000000000004bfb0)
barebox data
    0x00000000efde5a00 - 0x00000000efde8d57 (size 0x0000000000003358) bss
    0x00000000effe8000 - 0x00000000effeffff (size 0x0000000000008000) stack
  0x00000000fd000000 - 0x00000000fd3fffff (size 0x0000000000400000) xHCI0
  0x00000000fdd40000 - 0x00000000fdd40fff (size 0x0000000000001000)
fdd40000.i2c@fdd40000.of
  0x00000000fdd60000 - 0x00000000fdd600ff (size 0x0000000000000100)
pinctrl.of
  0x00000000fe000000 - 0x00000000fe003fff (size 0x0000000000004000)
fe000000.mmc@fe000000.of
  0x00000000fe010000 - 0x00000000fe01ffff (size 0x0000000000010000)
fe010000.ethernet@fe010000.of
  0x00000000fe2b0000 - 0x00000000fe2b3fff (size 0x0000000000004000)
fe2b0000.mmc@fe2b0000.of
  0x00000000fe310000 - 0x00000000fe31ffff (size 0x0000000000010000)
fe310000.mmc@fe310000.of
  0x00000000fe5e0000 - 0x00000000fe5e0fff (size 0x0000000000001000)
fe5e0000.i2c@fe5e0000.of
  0x00000000fe650000 - 0x00000000fe6500ff (size 0x0000000000000100)
fe650000.serial@fe650000.of
  0x00000000fe660000 - 0x00000000fe6600ff (size 0x0000000000000100)
fe660000.serial@fe660000.of
  0x00000000fe720000 - 0x00000000fe7200ff (size 0x0000000000000100)
fe720000.saradc@fe720000.of
  0x00000000fe740000 - 0x00000000fe7400ff (size 0x0000000000000100)
pinctrl.of
  0x00000000fe750000 - 0x00000000fe7500ff (size 0x0000000000000100)
pinctrl.of
  0x00000000fe760000 - 0x00000000fe7600ff (size 0x0000000000000100)
pinctrl.of
  0x00000000fe770000 - 0x00000000fe7700ff (size 0x0000000000000100)
pinctrl.of
  0x00000000fe820000 - 0x00000000fe8200ff (size 0x0000000000000100)
fe820000.phy@fe820000.of
  0x00000000fe830000 - 0x00000000fe8300ff (size 0x0000000000000100)
fe830000.phy@fe830000.of
  0x00000000fe840000 - 0x00000000fe8400ff (size 0x0000000000000100)
fe840000.phy@fe840000.of
  0x00000000fe8a0000 - 0x00000000fe8affff (size 0x0000000000010000)
fe8a0000.usb2phy@fe8a0000.of
  0x00000000fe8b0000 - 0x00000000fe8bffff (size 0x0000000000010000)
fe8b0000.usb2phy@fe8b0000.of
  0x0000000100000000 - 0x000000020fffffff (size 0x0000000110000000) ram1
barebox@Radxa ROCK3 Model A:/

>> that start at SZ_4G and 0xa00000, respectively.
>>
>> However, after loading the kernel the system hangs:
>>
>> Loaded kernel to 0x100000000, devicetree at 0x101970000
>>
>> Is this a bug in barebox or is something special required to boot the
>> kernel from SZ_4G?
> 
> I would have guessed that barebox puts the kernel below the malloc area
> just as usual. Ok, apparently it doesn't. I haven't found anything in
> Documentation/arm64/booting.rst that forbids placing the kernel above
> the 32bit boundary. You could verify that the memory nodes barebox
> writes meet your expectations by adding some -v to the boot command.
> Also earlycon might help to get some output from the kernel.

After having a quick look at booti.c I think that
memory_bank_first_find_space is invoked to determine the start address.
Apparently, this returns the start and end of ram1. I tried to override
the start and end variables with the boundaries of ram0, which caused
the kernel to boot from 0xa00000 but didn't bring any further success.

With the earlycon parameter set, the boot log now shows:

Loaded kernel to 0x00a00000, devicetree at 0x02370000
[    0.000000] Booting Linux on physical CPU 0x0000000000 [0x412fd050]
[    0.000000] Linux version 6.3.0-rc2-20230316-1+ (ptxdist@ptxdist)
(aarch64-v8a-linux-gnu-gcc (OSELAS.0
[    0.000000] Machine model: Radxa ROCK3 Model A
[    0.000000] earlycon: uart0 at MMIO32 0x00000000fe660000 (options
'1500000n8')
[    0.000000] printk: bootconsole [uart0] enabled
[    0.000000] SError Interrupt on CPU0, code 0x00000000be000011 -- SError
[    0.000000] CPU: 0 PID: 0 Comm: swapper Not tainted
6.3.0-rc2-20230316-1+ #17
[    0.000000] Hardware name: Radxa ROCK3 Model A (DT)
[    0.000000] pstate: 200000c9 (nzCv daIF -PAN -UAO -TCO -DIT -SSBS
BTYPE=--)
[    0.000000] pc : __memset+0x16c/0x188
[    0.000000] lr : early_pgtable_alloc+0xb0/0x130
[    0.000000] sp : ffffffc009753b10
[    0.000000] x29: ffffffc009753b10 x28: 0000000000000000 x27:
0000000000a10000
[    0.000000] x26: ffffffc008c9ffff x25: ffffffc008010000 x24:
0000000000000000
[    0.000000] x23: ffffffc008010000 x22: fffffffdfda36000 x21:
ffffffc0097dd000
[    0.000000] x20: 000000020ffff000 x19: caaead2f6f13289c x18:
0000000000000000
[    0.000000] x17: 386e303030303035 x16: 312720736e6f6974 x15:
0000000000000000
[    0.000000] x14: 0000000000000000 x13: 0000000000000000 x12:
000000000002667a
[    0.000000] x11: 656c6f736e6f6374 x10: ffffffc007600000 x9 :
0000000000000000
[    0.000000] x8 : fffffffdfda39780 x7 : 0000000000000000 x6 :
000000000000003f
[    0.000000] x5 : 0000000000000040 x4 : 0000000000000000 x3 :
0000000000000004
[    0.000000] x2 : 0000000000000840 x1 : 0000000000000000 x0 :
fffffffdfda39000
[    0.000000] Kernel panic - not syncing: Asynchronous SError Interrupt
[    0.000000] CPU: 0 PID: 0 Comm: swapper Not tainted
6.3.0-rc2-20230316-1+ #17
[    0.000000] Hardware name: Radxa ROCK3 Model A (DT)
[    0.000000] Call trace:
[    0.000000]  dump_backtrace+0xb4/0x138
[    0.000000]  show_stack+0x30/0x48
[    0.000000]  dump_stack_lvl+0x60/0xb0
[    0.000000]  dump_stack+0x18/0x28
[    0.000000]  panic+0x394/0x3a8
[    0.000000]  nmi_panic+0xbc/0xc8
[    0.000000]  arm64_serror_panic+0x6c/0x88
[    0.000000]  do_serror+0x44/0x88
[    0.000000]  el1h_64_error_handler+0x38/0x50
[    0.000000]  el1h_64_error+0x64/0x68
[    0.000000]  __memset+0x16c/0x188
[    0.000000]  create_kpti_ng_temp_pgd+0x110/0x640
[    0.000000]  map_kernel_segment+0xd0/0x1b0
[    0.000000]  paging_init+0x1c8/0xdb0
[    0.000000]  setup_arch+0x3c8/0xa40
[    0.000000]  start_kernel+0xe0/0xcf8
[    0.000000]  __primary_switched+0xb4/0xc8

and boot -vvvv prints the memory nodes:

        memory@100000000 {
                device_type = "memory";
                reg = <0x1 0x0 0x1 0x10000000>;
        };
        memory@a00000 {
                device_type = "memory";
                reg = <0x0 0xa00000 0x0 0xef600000>;
        };

FWIW: I commented out the arm_add_mem_device call for ram1. With that
change, only the memory@a00000 is appended and the kernel boots normally
on the ROCK3A 8GB.

Thanks and best regards,
Michael