RE: [PATCH v2] Porting barebox to a new SoC

[Lior Weintraub <liorw@pliops.com>]

Hi Ahmad,

Looks like I found the issue :-)

When I declared the flash@0 node I used about 16MB for the kernel partition even though currently my kernel image is around 11MB:
flash@0 {
    compatible = "mtd-rom";
    probe-type = "map_rom";
    reg = <0x0 0x20000000 0x0 0x10000000>; /* Offset 512MB, size 256MB */
    bank-width = <4>;
    device-width = <1>;

    #address-cells = <1>;
    #size-cells = <1>;
    kernel@0 { /* 16MB (minus 32KB) */
        label = "kernel";
        //reg = <0x0 0x00FF8000>; // This line caused the error
        reg =   <0x0 0x00b30000>;
    };
    oftree@00FF8000 { /* 32KB for DTB image */
        label = "oftree";
        reg = <0x00FF8000 0x00008000>;
    };
    rootfs@1000000 { /* Offset 512MB+16MB with size 8MB */
        label = "rootfs";
        reg = <0x01000000 0x00800000>;
    };
};

From the error message I saw that barebox tried to place the rootfs on address 0xb30000 and that conflicted with the kernel:
mtdrom0.kernel: read ofs: 0x00fe0000 count: 0x00018000
__request_region ok: 0x00000000:0x00ff7fff flags=0x200
mtdrom0.rootfs: read ofs: 0x00000000 count: 0x00000800
__request_region: 0x00b30000:0x00b4ffff (image) conflicts with 0x00000000:0x00ff7fff (image)

So I changed the kernel@0 node and used size 0x00b30000 and now kernel starts running.
The kernel is producing all kinds of errors but at least it's a start :-)
The errors are now related to pieces of HW that we didn't implement on QEMU so it is understandable.

How can we control where the rootfs is copied to?
 
Thanks again for your kind advise and patience.
Cheers,
Lior.

> -----Original Message-----
> From: Lior Weintraub
> Sent: Tuesday, June 13, 2023 4:28 PM
> To: Ahmad Fatoum <a.fatoum@pengutronix.de>; Ahmad Fatoum
> <ahmad@a3f.at>; barebox@lists.infradead.org
> Subject: RE: [PATCH v2] Porting barebox to a new SoC
> 
> Hi Ahmad,
> 
> I also thought that load address was suspicious (0x8000000) but then I saw
> the following comment on nxp_imx8mq_evk_start:
> 		/*
> 		 * On completion the TF-A will jump to
> MX8MQ_ATF_BL33_BASE_ADDR
> 		 * in EL2. Copy the image there, but replace the PBL part of
> 		 * that image with ourselves. On a high assurance boot only
> the
> 		 * currently running code is validated and contains the
> checksum
> 		 * for the piggy data, so we need to ensure that we are
> running
> 		 * the same code in DRAM.
> 		 */
> Our code was based on this function and as a result it copies the barebox
> image into address 0x8000000 (which is our settings of
> ATF_BL33_BASE_ADDR).
> The fact that Linux image is also copied there seemed reasonable (I might be
> wrong here) because once loading is done and we jump to Linux there is no
> need for barebox code anymore.
> 
> In any case, at that time when I thought it was wrong I also tried to run the
> bootm command with "-a 0" but I got exactly the same error:
> barebox:/ bootm -o /dev/mtdrom0.oftree -r /dev/mtdrom0.rootfs -a 0
> /dev/mtdrom0.kernel
> getopt: optindex: 1 nonopts: 0 optind: 1
> getopt: optindex: 1 nonopts: 0 optind: 3
> getopt: optindex: 1 nonopts: 0 optind: 5
> getopt: optindex: 1 nonopts: 0 optind: 7
> mtdrom0.kernel: read ofs: 0x00000000 count: 0x00001000
> 
> Loading ARM aarch64 Linux image '/dev/mtdrom0.kernel'
> header 00000000: 4d 5a 40 fa 3f 1a 23 14 00 00 00 00 00 00 00 00
> MZ@.?.#.........
> header 00000010: 00 00 b3 00 00 00 00 00 0a 00 00 00 00 00 00 00
> ................
> header 00000020: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> ................
> header 00000030: 00 00 00 00 00 00 00 00 41 52 4d 64 40 00 00 00
> ........ARMd@...
> header 00000040: 50 45 00 00 64 aa 02 00 00 00 00 00 00 00 00 00
> PE..d...........
> booti: Kernel to be loaded to 0+b30000
> __request_region ok: 0x00000000:0x0001ffff flags=0x200
> mtdrom0.kernel: read ofs: 0x00000000 count: 0x00020000
> __request_region ok: 0x00000000:0x0003ffff flags=0x200
> mtdrom0.kernel: read ofs: 0x00020000 count: 0x00020000
> __request_region ok: 0x00000000:0x0005ffff flags=0x200
> .
> .
> mtdrom0.kernel: read ofs: 0x00fe0000 count: 0x00018000
> __request_region ok: 0x00000000:0x00ff7fff flags=0x200
> mtdrom0.rootfs: read ofs: 0x00000000 count: 0x00000800
> __request_region: 0x00b30000:0x00b4ffff (image) conflicts with
> 0x00000000:0x00ff7fff (image)
> unable to request SDRAM 0x00b30000-0x00b4ffff
> handler failed with: Out of memory
> 
> 
> The iomem before:
> barebox:/ iomem
> 0x0000000000000000 - 0xffffffffffffffff (size 0x0000000000000000) iomem
>   0x0000000000000000 - 0x000000001fffffff (size 0x0000000020000000)
> ram0
>     0x00000000079fef00 - 0x0000000007dfeeff (size 0x0000000000400000)
> malloc space
>     0x0000000007dfef00 - 0x0000000007dfffd1 (size 0x00000000000010d2)
> board data
>     0x0000000007e00000 - 0x0000000007e3d95f (size
> 0x000000000003d960) barebox
>     0x0000000007e3d960 - 0x0000000007e50bb7 (size
> 0x0000000000013258) barebox data
>     0x0000000007e50bb8 - 0x0000000007e53b3f (size
> 0x0000000000002f88) bss
>     0x0000000007ff0000 - 0x0000000007ff7fff (size 0x0000000000008000)
> stack
>   0x0000000020000000 - 0x000000002fffffff (size 0x0000000010000000)
> 20000000.flash@0.of
>   0x000000d000307000 - 0x000000d000307fff (size
> 0x0000000000001000) amba
> 
> After:
> barebox:/ iomem
> 0x0000000000000000 - 0xffffffffffffffff (size 0x0000000000000000) iomem
>   0x0000000000000000 - 0x000000001fffffff (size 0x0000000020000000)
> ram0
>     0x00000000079fef00 - 0x0000000007dfeeff (size 0x0000000000400000)
> malloc space
>     0x0000000007dfef00 - 0x0000000007dfffd1 (size 0x00000000000010d2)
> board data
>     0x0000000007e00000 - 0x0000000007e3d95f (size
> 0x000000000003d960) barebox
>     0x0000000007e3d960 - 0x0000000007e50bb7 (size
> 0x0000000000013258) barebox data
>     0x0000000007e50bb8 - 0x0000000007e53b3f (size
> 0x0000000000002f88) bss
>     0x0000000007ff0000 - 0x0000000007ff7fff (size 0x0000000000008000)
> stack
>   0x0000000020000000 - 0x000000002fffffff (size 0x0000000010000000)
> 20000000.flash@0.of
>   0x000000d000307000 - 0x000000d000307fff (size
> 0x0000000000001000) amba
> 
> I did not set global.bootm.image.loadaddr
> Speaking of env, the last few lines on the barebox terminal are:
> initcall-> load_environment+0x0/0x4c
> loading defaultenv
> environment load /dev/env0: No such file or directory
> Maybe you have to create the partition.
> initcalls done
> 
> When I run "printenv" I am getting:
> barebox:/ printenv
> locals:
> globals:
> bootsource=unknown
> bootsource_instance=unknown
> PATH=/env/bin
> 
> Hope this info can help.
> Cheers,
> Lior.
> 
> 
> > -----Original Message-----
> > From: Ahmad Fatoum <a.fatoum@pengutronix.de>
> > Sent: Tuesday, June 13, 2023 3:51 PM
> > To: Lior Weintraub <liorw@pliops.com>; Ahmad Fatoum <ahmad@a3f.at>;
> > barebox@lists.infradead.org
> > Subject: Re: [PATCH v2] Porting barebox to a new SoC
> >
> > CAUTION: External Sender
> >
> > On 13.06.23 14:39, Lior Weintraub wrote:
> > > Hi Ahmad,
> > >
> > > As usual, your comments are spot on :-)
> > >
> > > After fixing the DT, devinfo command shows correct flash node:
> > >    `-- 20000000.flash@0.of
> > >       `-- mtdrom0
> > >          `-- 0x00000000-0x0fffffff (   256 MiB): /dev/mtdrom0
> > >          `-- mtdrom0.kernel
> > >             `-- 0x00000000-0x00ff7fff (    16 MiB): /dev/mtdrom0.kernel
> > >          `-- mtdrom0.oftree
> > >             `-- 0x00000000-0x00007fff (    32 KiB): /dev/mtdrom0.oftree
> > >          `-- mtdrom0.rootfs
> > >             `-- 0x00000000-0x007fffff (     8 MiB): /dev/mtdrom0.rootfs
> > >
> > > As can be seen above, we chose to map this "flash" node on DRAM
> address
> > 0x20000000 (offset 512MB).
> > > The BL1 code correctly copy the 3 artifacts into those location (verified the
> > content via "md -l -s /dev/mtdrom0.kernel").
> > > Running "drvinfo" shows:
> > > mtdram
> > >         20000000.flash@0.of
> > >
> > > Now when I try to run "bootm -o /dev/mtdrom0.oftree -r
> > /dev/mtdrom0.rootfs /dev/mtdrom0.kernel" I am getting this error:
> > >
> > > barebox:/ bootm -o /dev/mtdrom0.oftree -r /dev/mtdrom0.rootfs
> > /dev/mtdrom0.kernel
> > > getopt: optindex: 1 nonopts: 0 optind: 1
> > > getopt: optindex: 1 nonopts: 0 optind: 3
> > > getopt: optindex: 1 nonopts: 0 optind: 5
> > > mtdrom0.kernel: read ofs: 0x00000000 count: 0x00001000
> > >
> > > Loading ARM aarch64 Linux image '/dev/mtdrom0.kernel'
> > > header 00000000: 4d 5a 40 fa 3f 1a 23 14 00 00 00 00 00 00 00 00
> > MZ@.?.#.........
> > > header 00000010: 00 00 b3 00 00 00 00 00 0a 00 00 00 00 00 00 00
> > ................
> > > header 00000020: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
> > ................
> > > header 00000030: 00 00 00 00 00 00 00 00 41 52 4d 64 40 00 00 00
> > ........ARMd@...
> > > header 00000040: 50 45 00 00 64 aa 02 00 00 00 00 00 00 00 00 00
> > PE..d...........
> > > booti: Kernel to be loaded to 8000000+b30000
> >
> > Kernel image is relocatable and ARM64 header in hexdump says that
> > load offset is zero. Yet, barebox wants to load your image to
> > 8000000, which is already reserved.
> >
> > What's the output of the iomem command before and after the failed
> > boot attempt?
> >
> > > __request_region ok: 0x08000000:0x0801ffff flags=0x200
> > > mtdrom0.kernel: read ofs: 0x00000000 count: 0x00020000
> > > __request_region ok: 0x08000000:0x0803ffff flags=0x200
> > > mtdrom0.kernel: read ofs: 0x00020000 count: 0x00020000
> > > .
> > > .
> > > mtdrom0.kernel: read ofs: 0x00fe0000 count: 0x00018000
> > > __request_region ok: 0x08000000:0x08ff7fff flags=0x200
> > > mtdrom0.rootfs: read ofs: 0x00000000 count: 0x00000800
> > > __request_region: 0x08b30000:0x08b4ffff (image) conflicts with
> > 0x08000000:0x08ff7fff (image)
> > > unable to request SDRAM 0x08b30000-0x08b4ffff
> > > handler failed with: Out of memory
> >
> > Just to be sure: You don't set global.bootm.image.loadaddr, right?
> >
> > Cheers,
> > Ahmad
> >
> > >
> > > Appreciate your advice,
> > > Cheers,
> > > Lior.
> > >
> > >
> > >> -----Original Message-----
> > >> From: Ahmad Fatoum <a.fatoum@pengutronix.de>
> > >> Sent: Monday, June 12, 2023 6:08 PM
> > >> To: Lior Weintraub <liorw@pliops.com>; Ahmad Fatoum
> <ahmad@a3f.at>;
> > >> barebox@lists.infradead.org
> > >> Subject: Re: [PATCH v2] Porting barebox to a new SoC
> > >>
> > >> CAUTION: External Sender
> > >>
> > >> Hi,
> > >>
> > >> On 12.06.23 16:59, Lior Weintraub wrote:
> > >>> Hello Ahmad,
> > >>>
> > >>> Just for simple simulations we make the ROM size 192MB so it can
> include
> > all
> > >> needed artifacts.
> > >>
> > >> I am not convinced that this is much of a simplification over having
> > >> your Qemu machine provide cfi-flash.
> > >>
> > >>> When we get this simple system to work we will move the relevant parts
> > to
> > >> BL2.
> > >>
> > >> Ok.
> > >>
> > >>> DRAM is also simulated now as SRAM so we are not worried about
> > >> initializations.
> > >>>
> > >>> So if I understand correctly, we can decide that address 0x10000000 will
> > be
> > >> reserved for "flash" and add the following node:
> > >>> flash@0 {
> > >>
> > >> flash@10000000, but that's just for informational purposes.
> > >>
> > >>>     compatible = "mtd-rom";
> > >>>     probe-type = "map_rom";
> > >>>     reg = <0x10000000 0x10000000>;
> > >>>     bank-width = <4>;
> > >>>     device-width = <1>;
> > >>>
> > >>>     #address-cells = <1>;
> > >>>     #size-cells = <1>;
> > >>>     kernel@0 {
> > >>>         label = "kernel";
> > >>>         reg = <0x0 0x01000000>;
> > >>>     };
> > >>>     rootfs@1000000 {
> > >>>         label = "rootfs";
> > >>>         reg = <0x01000000 0x00800000>;
> > >>>     };
> > >>> };
> > >>>
> > >>> When I use this node on our DT I see the following devinfo:
> > >>> barebox:/ devinfo
> > >>> `-- global
> > >>> `-- nv
> > >>> `-- platform
> > >>>    `-- machine
> > >>>    `-- psci.of
> > >>>    `-- 1000000010000000.flash@0.of
> > >>
> > >> As you can see the address is wrong. That's because you have
> > >> #address-cells = <2>;
> > >> #size-cells = <2>;
> > >>
> > >> Yet, you wrote reg as if it was <1>/<1>. Change it to
> > >>
> > >>   reg = <0x0 0x10000000 0x0 0x10000000>;
> > >>
> > >> Remember all device tree integers are 32-bit big endian.
> > >>
> > >>>    `-- soc.of
> > >>>       `-- c000000000.sram@c000000000.of
> > >>>       `-- soc:pmu.of
> > >>>       `-- soc:timer.of
> > >>>       `-- e000000000.interrupt-controller@E000000000.of
> > >>>    `-- mem0
> > >>>       `-- 0x00000000-0x0fffffff (   256 MiB): /dev/ram0
> > >>>    `-- mem1
> > >>>       `-- 0x00000000-0xffffffffffffffff (   0 Bytes): /dev/mem
> > >>> `-- amba
> > >>>    `-- d000307000.serial@d000307000.of
> > >>>       `-- cs0
> > >>>          `-- 0x00000000-0xffffffffffffffff (   0 Bytes): /dev/cs0
> > >>> `-- spi
> > >>> `-- fs
> > >>>    `-- ramfs0
> > >>>    `-- devfs0
> > >>>    `-- pstore0
> > >>>
> > >>> Not sure how to proceed from here...
> > >>
> > >> Type drvinfo command to see what drivers are bound to what devices.
> > >> If you see no driver bound to your flash device, then maybe you need
> > >> to enable the correct driver, that would be CONFIG_MTD_MTDRAM
> (which
> > >> handles both read-only and read-write memory mapped MTD).
> > >>
> > >> Cheers,
> > >> Ahmad
> > >>
> > >>> Cheers,
> > >>> Lior.
> > >>>
> > >>>> -----Original Message-----
> > >>>> From: Ahmad Fatoum <a.fatoum@pengutronix.de>
> > >>>> Sent: Monday, June 12, 2023 3:29 PM
> > >>>> To: Lior Weintraub <liorw@pliops.com>; Ahmad Fatoum
> > <ahmad@a3f.at>;
> > >>>> barebox@lists.infradead.org
> > >>>> Subject: Re: [PATCH v2] Porting barebox to a new SoC
> > >>>>
> > >>>> CAUTION: External Sender
> > >>>>
> > >>>> Hello Lior,
> > >>>>
> > >>>> On 12.06.23 11:27, Lior Weintraub wrote:
> > >>>>> Hi Ahmad,
> > >>>>>
> > >>>>> Regarding the rootfs and Linux run question:
> > >>>>> Our board doesn't include eMMC\SD card.
> > >>>>> We only have our NOR Flash to store the binaries and our BL1 code
> will
> > >> make
> > >>>> sure to copy those blobs into DRAM.
> > >>>>
> > >>>> How could BL1 copy artifacts in DRAM when BL2 first needs to set up
> > >> DRAM?
> > >>>>
> > >>>>> The use of QEMU needs to be as simple as possible so no hidden virtio
> > >>>> drivers and such because we would like to simulate the real HW flow.
> > >>>>
> > >>>> cfi-flash is just memory-mapped flash, which is the next simplest thing
> > >>>> after BL1 copies stuff into (limited-to-4M) on-chip SRAM.
> > >>>>
> > >>>>> Our DTS file now includes the GIC, timer and arm,psci-1.0.
> > >>>>> We have an initial build of Linux kernel Image (using buildroot) and a
> > >>>> rootfs.cpio.xz.
> > >>>>> I assume we can somehow reserve a portion of the DRAM to store
> > those
> > >>>> binaries and let barebox boot this Linux.
> > >>>>
> > >>>> You can make this work, but this is not how your actual system will
> > >>>> look like and trying to make this work is harder than it needs to be.
> > >>>>
> > >>>> Just add a cfi-flash that corresponds to the Linux/rootfs flash in
> > >>>> your actual system, then boot with bootm (type help bootm for info
> > >>>> on how to use it). You don't need to hardcode the load addresses,
> > >>>> barebox will determine them automatically.
> > >>>>
> > >>>>> Can you please advise how to make it work?
> > >>>>
> > >>>> For completion's sake, if you have 64M of RAM that's preloaded with
> > >>>> boot images:
> > >>>>
> > >>>>   - Remove the 64M from the barebox /memory node. You can use a
> > >> different
> > >>>>     DT for kernel, but if you have memory that barebox shouldn't
> override,
> > >>>>     you need to tell it that.
> > >>>>
> > >>>>   - Add a mtd-rom node that describes these 64M that you have. You
> can
> > >>>>     add partitions for the region used for kernel and oftree
> > >>>>
> > >>>>   - boot with bootm -o /dev/mtdrom0.oftree -r /dev/mtdrom0.initrd \
> > >>>>     /dev/mtdrom0.kernel
> > >>>>
> > >>>> That's admittedly cumbersome, but necessary, so barebox knows what
> > >>>> memory
> > >>>> it may use for itself and what memory may be used for boot.
> > >>>>
> > >>>> Correct solution is to use cfi-flash or similar.
> > >>>>
> > >>>> Cheers,
> > >>>> Ahmad
> > >>>> --
> > >>>> Pengutronix e.K.                           |                             |
> > >>>> Steuerwalder Str. 21                       | http://www.pengutronix.de/  |
> > >>>> 31137 Hildesheim, Germany                  | Phone: +49-5121-206917-0    |
> > >>>> Amtsgericht Hildesheim, HRA 2686           | Fax:   +49-5121-206917-
> 5555
> > |
> > >>>
> > >>
> > >> --
> > >> Pengutronix e.K.                           |                             |
> > >> Steuerwalder Str. 21                       | http://www.pengutronix.de/  |
> > >> 31137 Hildesheim, Germany                  | Phone: +49-5121-206917-0    |
> > >> Amtsgericht Hildesheim, HRA 2686           | Fax:   +49-5121-206917-5555
> |
> > >
> >
> > --
> > Pengutronix e.K.                           |                             |
> > Steuerwalder Str. 21                       | http://www.pengutronix.de/  |
> > 31137 Hildesheim, Germany                  | Phone: +49-5121-206917-0    |
> > Amtsgericht Hildesheim, HRA 2686           | Fax:   +49-5121-206917-5555 |