Re: [PATCH v2] Porting barebox to a new SoC

[Ahmad Fatoum <a.fatoum@pengutronix.de>]

Hello Lior,

On 14.06.23 08:42, Lior Weintraub wrote:
> Hi Ahmad,
> 
> Looks like I found the issue :-)
> 
> When I declared the flash@0 node I used about 16MB for the kernel partition even though currently my kernel image is around 11MB:
> flash@0 {
>     compatible = "mtd-rom";
>     probe-type = "map_rom";
>     reg = <0x0 0x20000000 0x0 0x10000000>; /* Offset 512MB, size 256MB */
>     bank-width = <4>;
>     device-width = <1>;
> 
>     #address-cells = <1>;
>     #size-cells = <1>;
>     kernel@0 { /* 16MB (minus 32KB) */
>         label = "kernel";
>         //reg = <0x0 0x00FF8000>; // This line caused the error
>         reg =   <0x0 0x00b30000>;
>     };
>     oftree@00FF8000 { /* 32KB for DTB image */
>         label = "oftree";
>         reg = <0x00FF8000 0x00008000>;
>     };
>     rootfs@1000000 { /* Offset 512MB+16MB with size 8MB */
>         label = "rootfs";
>         reg = <0x01000000 0x00800000>;
>     };
> };
> 
> From the error message I saw that barebox tried to place the rootfs on address 0xb30000 and that conflicted with the kernel:
> mtdrom0.kernel: read ofs: 0x00fe0000 count: 0x00018000
> __request_region ok: 0x00000000:0x00ff7fff flags=0x200
> mtdrom0.rootfs: read ofs: 0x00000000 count: 0x00000800
> __request_region: 0x00b30000:0x00b4ffff (image) conflicts with 0x00000000:0x00ff7fff (image)

This error message still doesn't make sense to me. No idea why this conflicts.
I see now though that I made this more complicated that it needs be. Please
scratch all that, remove the mtd-rom and revert the /memory node to encompass
all memory. Then you can at runtime, just create the partitions with the addpart
command (see help addpart for info). Then you can boot normally.

> 
> So I changed the kernel@0 node and used size 0x00b30000 and now kernel starts running.
> The kernel is producing all kinds of errors but at least it's a start :-)
> The errors are now related to pieces of HW that we didn't implement on QEMU so it is understandable.
> 
> How can we control where the rootfs is copied to?

Normally, global.bootm.initrd.loadaddr, but you shouldn't need to do
that. Try it with addpart and let me know.

Cheers,
Ahmad

>  
> Thanks again for your kind advise and patience.
> Cheers,
> Lior.
> 
>> -----Original Message-----
>> From: Lior Weintraub
>> Sent: Tuesday, June 13, 2023 4:28 PM
>> To: Ahmad Fatoum <a.fatoum@pengutronix.de>; Ahmad Fatoum
>> <ahmad@a3f.at>; barebox@lists.infradead.org
>> Subject: RE: [PATCH v2] Porting barebox to a new SoC
>>
>> Hi Ahmad,
>>
>> I also thought that load address was suspicious (0x8000000) but then I saw
>> the following comment on nxp_imx8mq_evk_start:
>> 		/*
>> 		 * On completion the TF-A will jump to
>> MX8MQ_ATF_BL33_BASE_ADDR
>> 		 * in EL2. Copy the image there, but replace the PBL part of
>> 		 * that image with ourselves. On a high assurance boot only
>> the
>> 		 * currently running code is validated and contains the
>> checksum
>> 		 * for the piggy data, so we need to ensure that we are
>> running
>> 		 * the same code in DRAM.
>> 		 */
>> Our code was based on this function and as a result it copies the barebox
>> image into address 0x8000000 (which is our settings of
>> ATF_BL33_BASE_ADDR).
>> The fact that Linux image is also copied there seemed reasonable (I might be
>> wrong here) because once loading is done and we jump to Linux there is no
>> need for barebox code anymore.
>>
>> In any case, at that time when I thought it was wrong I also tried to run the
>> bootm command with "-a 0" but I got exactly the same error:
>> barebox:/ bootm -o /dev/mtdrom0.oftree -r /dev/mtdrom0.rootfs -a 0
>> /dev/mtdrom0.kernel
>> getopt: optindex: 1 nonopts: 0 optind: 1
>> getopt: optindex: 1 nonopts: 0 optind: 3
>> getopt: optindex: 1 nonopts: 0 optind: 5
>> getopt: optindex: 1 nonopts: 0 optind: 7
>> mtdrom0.kernel: read ofs: 0x00000000 count: 0x00001000
>>
>> Loading ARM aarch64 Linux image '/dev/mtdrom0.kernel'
>> header 00000000: 4d 5a 40 fa 3f 1a 23 14 00 00 00 00 00 00 00 00
>> MZ@.?.#.........
>> header 00000010: 00 00 b3 00 00 00 00 00 0a 00 00 00 00 00 00 00
>> ................
>> header 00000020: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
>> ................
>> header 00000030: 00 00 00 00 00 00 00 00 41 52 4d 64 40 00 00 00
>> ........ARMd@...
>> header 00000040: 50 45 00 00 64 aa 02 00 00 00 00 00 00 00 00 00
>> PE..d...........
>> booti: Kernel to be loaded to 0+b30000
>> __request_region ok: 0x00000000:0x0001ffff flags=0x200
>> mtdrom0.kernel: read ofs: 0x00000000 count: 0x00020000
>> __request_region ok: 0x00000000:0x0003ffff flags=0x200
>> mtdrom0.kernel: read ofs: 0x00020000 count: 0x00020000
>> __request_region ok: 0x00000000:0x0005ffff flags=0x200
>> .
>> .
>> mtdrom0.kernel: read ofs: 0x00fe0000 count: 0x00018000
>> __request_region ok: 0x00000000:0x00ff7fff flags=0x200
>> mtdrom0.rootfs: read ofs: 0x00000000 count: 0x00000800
>> __request_region: 0x00b30000:0x00b4ffff (image) conflicts with
>> 0x00000000:0x00ff7fff (image)
>> unable to request SDRAM 0x00b30000-0x00b4ffff
>> handler failed with: Out of memory
>>
>>
>> The iomem before:
>> barebox:/ iomem
>> 0x0000000000000000 - 0xffffffffffffffff (size 0x0000000000000000) iomem
>>   0x0000000000000000 - 0x000000001fffffff (size 0x0000000020000000)
>> ram0
>>     0x00000000079fef00 - 0x0000000007dfeeff (size 0x0000000000400000)
>> malloc space
>>     0x0000000007dfef00 - 0x0000000007dfffd1 (size 0x00000000000010d2)
>> board data
>>     0x0000000007e00000 - 0x0000000007e3d95f (size
>> 0x000000000003d960) barebox
>>     0x0000000007e3d960 - 0x0000000007e50bb7 (size
>> 0x0000000000013258) barebox data
>>     0x0000000007e50bb8 - 0x0000000007e53b3f (size
>> 0x0000000000002f88) bss
>>     0x0000000007ff0000 - 0x0000000007ff7fff (size 0x0000000000008000)
>> stack
>>   0x0000000020000000 - 0x000000002fffffff (size 0x0000000010000000)
>> 20000000.flash@0.of
>>   0x000000d000307000 - 0x000000d000307fff (size
>> 0x0000000000001000) amba
>>
>> After:
>> barebox:/ iomem
>> 0x0000000000000000 - 0xffffffffffffffff (size 0x0000000000000000) iomem
>>   0x0000000000000000 - 0x000000001fffffff (size 0x0000000020000000)
>> ram0
>>     0x00000000079fef00 - 0x0000000007dfeeff (size 0x0000000000400000)
>> malloc space
>>     0x0000000007dfef00 - 0x0000000007dfffd1 (size 0x00000000000010d2)
>> board data
>>     0x0000000007e00000 - 0x0000000007e3d95f (size
>> 0x000000000003d960) barebox
>>     0x0000000007e3d960 - 0x0000000007e50bb7 (size
>> 0x0000000000013258) barebox data
>>     0x0000000007e50bb8 - 0x0000000007e53b3f (size
>> 0x0000000000002f88) bss
>>     0x0000000007ff0000 - 0x0000000007ff7fff (size 0x0000000000008000)
>> stack
>>   0x0000000020000000 - 0x000000002fffffff (size 0x0000000010000000)
>> 20000000.flash@0.of
>>   0x000000d000307000 - 0x000000d000307fff (size
>> 0x0000000000001000) amba
>>
>> I did not set global.bootm.image.loadaddr
>> Speaking of env, the last few lines on the barebox terminal are:
>> initcall-> load_environment+0x0/0x4c
>> loading defaultenv
>> environment load /dev/env0: No such file or directory
>> Maybe you have to create the partition.
>> initcalls done
>>
>> When I run "printenv" I am getting:
>> barebox:/ printenv
>> locals:
>> globals:
>> bootsource=unknown
>> bootsource_instance=unknown
>> PATH=/env/bin
>>
>> Hope this info can help.
>> Cheers,
>> Lior.
>>
>>
>>> -----Original Message-----
>>> From: Ahmad Fatoum <a.fatoum@pengutronix.de>
>>> Sent: Tuesday, June 13, 2023 3:51 PM
>>> To: Lior Weintraub <liorw@pliops.com>; Ahmad Fatoum <ahmad@a3f.at>;
>>> barebox@lists.infradead.org
>>> Subject: Re: [PATCH v2] Porting barebox to a new SoC
>>>
>>> CAUTION: External Sender
>>>
>>> On 13.06.23 14:39, Lior Weintraub wrote:
>>>> Hi Ahmad,
>>>>
>>>> As usual, your comments are spot on :-)
>>>>
>>>> After fixing the DT, devinfo command shows correct flash node:
>>>>    `-- 20000000.flash@0.of
>>>>       `-- mtdrom0
>>>>          `-- 0x00000000-0x0fffffff (   256 MiB): /dev/mtdrom0
>>>>          `-- mtdrom0.kernel
>>>>             `-- 0x00000000-0x00ff7fff (    16 MiB): /dev/mtdrom0.kernel
>>>>          `-- mtdrom0.oftree
>>>>             `-- 0x00000000-0x00007fff (    32 KiB): /dev/mtdrom0.oftree
>>>>          `-- mtdrom0.rootfs
>>>>             `-- 0x00000000-0x007fffff (     8 MiB): /dev/mtdrom0.rootfs
>>>>
>>>> As can be seen above, we chose to map this "flash" node on DRAM
>> address
>>> 0x20000000 (offset 512MB).
>>>> The BL1 code correctly copy the 3 artifacts into those location (verified the
>>> content via "md -l -s /dev/mtdrom0.kernel").
>>>> Running "drvinfo" shows:
>>>> mtdram
>>>>         20000000.flash@0.of
>>>>
>>>> Now when I try to run "bootm -o /dev/mtdrom0.oftree -r
>>> /dev/mtdrom0.rootfs /dev/mtdrom0.kernel" I am getting this error:
>>>>
>>>> barebox:/ bootm -o /dev/mtdrom0.oftree -r /dev/mtdrom0.rootfs
>>> /dev/mtdrom0.kernel
>>>> getopt: optindex: 1 nonopts: 0 optind: 1
>>>> getopt: optindex: 1 nonopts: 0 optind: 3
>>>> getopt: optindex: 1 nonopts: 0 optind: 5
>>>> mtdrom0.kernel: read ofs: 0x00000000 count: 0x00001000
>>>>
>>>> Loading ARM aarch64 Linux image '/dev/mtdrom0.kernel'
>>>> header 00000000: 4d 5a 40 fa 3f 1a 23 14 00 00 00 00 00 00 00 00
>>> MZ@.?.#.........
>>>> header 00000010: 00 00 b3 00 00 00 00 00 0a 00 00 00 00 00 00 00
>>> ................
>>>> header 00000020: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
>>> ................
>>>> header 00000030: 00 00 00 00 00 00 00 00 41 52 4d 64 40 00 00 00
>>> ........ARMd@...
>>>> header 00000040: 50 45 00 00 64 aa 02 00 00 00 00 00 00 00 00 00
>>> PE..d...........
>>>> booti: Kernel to be loaded to 8000000+b30000
>>>
>>> Kernel image is relocatable and ARM64 header in hexdump says that
>>> load offset is zero. Yet, barebox wants to load your image to
>>> 8000000, which is already reserved.
>>>
>>> What's the output of the iomem command before and after the failed
>>> boot attempt?
>>>
>>>> __request_region ok: 0x08000000:0x0801ffff flags=0x200
>>>> mtdrom0.kernel: read ofs: 0x00000000 count: 0x00020000
>>>> __request_region ok: 0x08000000:0x0803ffff flags=0x200
>>>> mtdrom0.kernel: read ofs: 0x00020000 count: 0x00020000
>>>> .
>>>> .
>>>> mtdrom0.kernel: read ofs: 0x00fe0000 count: 0x00018000
>>>> __request_region ok: 0x08000000:0x08ff7fff flags=0x200
>>>> mtdrom0.rootfs: read ofs: 0x00000000 count: 0x00000800
>>>> __request_region: 0x08b30000:0x08b4ffff (image) conflicts with
>>> 0x08000000:0x08ff7fff (image)
>>>> unable to request SDRAM 0x08b30000-0x08b4ffff
>>>> handler failed with: Out of memory
>>>
>>> Just to be sure: You don't set global.bootm.image.loadaddr, right?
>>>
>>> Cheers,
>>> Ahmad
>>>
>>>>
>>>> Appreciate your advice,
>>>> Cheers,
>>>> Lior.
>>>>
>>>>
>>>>> -----Original Message-----
>>>>> From: Ahmad Fatoum <a.fatoum@pengutronix.de>
>>>>> Sent: Monday, June 12, 2023 6:08 PM
>>>>> To: Lior Weintraub <liorw@pliops.com>; Ahmad Fatoum
>> <ahmad@a3f.at>;
>>>>> barebox@lists.infradead.org
>>>>> Subject: Re: [PATCH v2] Porting barebox to a new SoC
>>>>>
>>>>> CAUTION: External Sender
>>>>>
>>>>> Hi,
>>>>>
>>>>> On 12.06.23 16:59, Lior Weintraub wrote:
>>>>>> Hello Ahmad,
>>>>>>
>>>>>> Just for simple simulations we make the ROM size 192MB so it can
>> include
>>> all
>>>>> needed artifacts.
>>>>>
>>>>> I am not convinced that this is much of a simplification over having
>>>>> your Qemu machine provide cfi-flash.
>>>>>
>>>>>> When we get this simple system to work we will move the relevant parts
>>> to
>>>>> BL2.
>>>>>
>>>>> Ok.
>>>>>
>>>>>> DRAM is also simulated now as SRAM so we are not worried about
>>>>> initializations.
>>>>>>
>>>>>> So if I understand correctly, we can decide that address 0x10000000 will
>>> be
>>>>> reserved for "flash" and add the following node:
>>>>>> flash@0 {
>>>>>
>>>>> flash@10000000, but that's just for informational purposes.
>>>>>
>>>>>>     compatible = "mtd-rom";
>>>>>>     probe-type = "map_rom";
>>>>>>     reg = <0x10000000 0x10000000>;
>>>>>>     bank-width = <4>;
>>>>>>     device-width = <1>;
>>>>>>
>>>>>>     #address-cells = <1>;
>>>>>>     #size-cells = <1>;
>>>>>>     kernel@0 {
>>>>>>         label = "kernel";
>>>>>>         reg = <0x0 0x01000000>;
>>>>>>     };
>>>>>>     rootfs@1000000 {
>>>>>>         label = "rootfs";
>>>>>>         reg = <0x01000000 0x00800000>;
>>>>>>     };
>>>>>> };
>>>>>>
>>>>>> When I use this node on our DT I see the following devinfo:
>>>>>> barebox:/ devinfo
>>>>>> `-- global
>>>>>> `-- nv
>>>>>> `-- platform
>>>>>>    `-- machine
>>>>>>    `-- psci.of
>>>>>>    `-- 1000000010000000.flash@0.of
>>>>>
>>>>> As you can see the address is wrong. That's because you have
>>>>> #address-cells = <2>;
>>>>> #size-cells = <2>;
>>>>>
>>>>> Yet, you wrote reg as if it was <1>/<1>. Change it to
>>>>>
>>>>>   reg = <0x0 0x10000000 0x0 0x10000000>;
>>>>>
>>>>> Remember all device tree integers are 32-bit big endian.
>>>>>
>>>>>>    `-- soc.of
>>>>>>       `-- c000000000.sram@c000000000.of
>>>>>>       `-- soc:pmu.of
>>>>>>       `-- soc:timer.of
>>>>>>       `-- e000000000.interrupt-controller@E000000000.of
>>>>>>    `-- mem0
>>>>>>       `-- 0x00000000-0x0fffffff (   256 MiB): /dev/ram0
>>>>>>    `-- mem1
>>>>>>       `-- 0x00000000-0xffffffffffffffff (   0 Bytes): /dev/mem
>>>>>> `-- amba
>>>>>>    `-- d000307000.serial@d000307000.of
>>>>>>       `-- cs0
>>>>>>          `-- 0x00000000-0xffffffffffffffff (   0 Bytes): /dev/cs0
>>>>>> `-- spi
>>>>>> `-- fs
>>>>>>    `-- ramfs0
>>>>>>    `-- devfs0
>>>>>>    `-- pstore0
>>>>>>
>>>>>> Not sure how to proceed from here...
>>>>>
>>>>> Type drvinfo command to see what drivers are bound to what devices.
>>>>> If you see no driver bound to your flash device, then maybe you need
>>>>> to enable the correct driver, that would be CONFIG_MTD_MTDRAM
>> (which
>>>>> handles both read-only and read-write memory mapped MTD).
>>>>>
>>>>> Cheers,
>>>>> Ahmad
>>>>>
>>>>>> Cheers,
>>>>>> Lior.
>>>>>>
>>>>>>> -----Original Message-----
>>>>>>> From: Ahmad Fatoum <a.fatoum@pengutronix.de>
>>>>>>> Sent: Monday, June 12, 2023 3:29 PM
>>>>>>> To: Lior Weintraub <liorw@pliops.com>; Ahmad Fatoum
>>> <ahmad@a3f.at>;
>>>>>>> barebox@lists.infradead.org
>>>>>>> Subject: Re: [PATCH v2] Porting barebox to a new SoC
>>>>>>>
>>>>>>> CAUTION: External Sender
>>>>>>>
>>>>>>> Hello Lior,
>>>>>>>
>>>>>>> On 12.06.23 11:27, Lior Weintraub wrote:
>>>>>>>> Hi Ahmad,
>>>>>>>>
>>>>>>>> Regarding the rootfs and Linux run question:
>>>>>>>> Our board doesn't include eMMC\SD card.
>>>>>>>> We only have our NOR Flash to store the binaries and our BL1 code
>> will
>>>>> make
>>>>>>> sure to copy those blobs into DRAM.
>>>>>>>
>>>>>>> How could BL1 copy artifacts in DRAM when BL2 first needs to set up
>>>>> DRAM?
>>>>>>>
>>>>>>>> The use of QEMU needs to be as simple as possible so no hidden virtio
>>>>>>> drivers and such because we would like to simulate the real HW flow.
>>>>>>>
>>>>>>> cfi-flash is just memory-mapped flash, which is the next simplest thing
>>>>>>> after BL1 copies stuff into (limited-to-4M) on-chip SRAM.
>>>>>>>
>>>>>>>> Our DTS file now includes the GIC, timer and arm,psci-1.0.
>>>>>>>> We have an initial build of Linux kernel Image (using buildroot) and a
>>>>>>> rootfs.cpio.xz.
>>>>>>>> I assume we can somehow reserve a portion of the DRAM to store
>>> those
>>>>>>> binaries and let barebox boot this Linux.
>>>>>>>
>>>>>>> You can make this work, but this is not how your actual system will
>>>>>>> look like and trying to make this work is harder than it needs to be.
>>>>>>>
>>>>>>> Just add a cfi-flash that corresponds to the Linux/rootfs flash in
>>>>>>> your actual system, then boot with bootm (type help bootm for info
>>>>>>> on how to use it). You don't need to hardcode the load addresses,
>>>>>>> barebox will determine them automatically.
>>>>>>>
>>>>>>>> Can you please advise how to make it work?
>>>>>>>
>>>>>>> For completion's sake, if you have 64M of RAM that's preloaded with
>>>>>>> boot images:
>>>>>>>
>>>>>>>   - Remove the 64M from the barebox /memory node. You can use a
>>>>> different
>>>>>>>     DT for kernel, but if you have memory that barebox shouldn't
>> override,
>>>>>>>     you need to tell it that.
>>>>>>>
>>>>>>>   - Add a mtd-rom node that describes these 64M that you have. You
>> can
>>>>>>>     add partitions for the region used for kernel and oftree
>>>>>>>
>>>>>>>   - boot with bootm -o /dev/mtdrom0.oftree -r /dev/mtdrom0.initrd \
>>>>>>>     /dev/mtdrom0.kernel
>>>>>>>
>>>>>>> That's admittedly cumbersome, but necessary, so barebox knows what
>>>>>>> memory
>>>>>>> it may use for itself and what memory may be used for boot.
>>>>>>>
>>>>>>> Correct solution is to use cfi-flash or similar.
>>>>>>>
>>>>>>> Cheers,
>>>>>>> Ahmad
>>>>>>> --
>>>>>>> Pengutronix e.K.                           |                             |
>>>>>>> Steuerwalder Str. 21                       | http://www.pengutronix.de/  |
>>>>>>> 31137 Hildesheim, Germany                  | Phone: +49-5121-206917-0    |
>>>>>>> Amtsgericht Hildesheim, HRA 2686           | Fax:   +49-5121-206917-
>> 5555
>>> |
>>>>>>
>>>>>
>>>>> --
>>>>> Pengutronix e.K.                           |                             |
>>>>> Steuerwalder Str. 21                       | http://www.pengutronix.de/  |
>>>>> 31137 Hildesheim, Germany                  | Phone: +49-5121-206917-0    |
>>>>> Amtsgericht Hildesheim, HRA 2686           | Fax:   +49-5121-206917-5555
>> |
>>>>
>>>
>>> --
>>> Pengutronix e.K.                           |                             |
>>> Steuerwalder Str. 21                       | http://www.pengutronix.de/  |
>>> 31137 Hildesheim, Germany                  | Phone: +49-5121-206917-0    |
>>> Amtsgericht Hildesheim, HRA 2686           | Fax:   +49-5121-206917-5555 |
> 

-- 
Pengutronix e.K.                           |                             |
Steuerwalder Str. 21                       | http://www.pengutronix.de/  |
31137 Hildesheim, Germany                  | Phone: +49-5121-206917-0    |
Amtsgericht Hildesheim, HRA 2686           | Fax:   +49-5121-206917-5555 |