Re: PCIE on LS1021A

[Ahmad Fatoum <a.fatoum@pengutronix.de>]

Hi,

(Adding mailing list to Cc)

On 12/5/25 5:58 PM, Renaud Barbier wrote:
> I read about the MMU in ARMv7 with LPAE
> My understanding is that for  LPAE:
> - There are 3 different sizes 4K, 2MB or 1G to map memory associated to 3 table level with 512 entries:
> - in U-boot   arch/arm/cpu/armv7/ls102xa/cpu.c:mmu_setup , the boot loader uses two levels of table to map the full memory DDR or device memory with 1G blocks in level 1 or the PCIe with 2MiB block2 in level 2.. It is not clear to me how devices registers get map to virtual address though the function map_physmem afterwards though.

I am not familiar either with the MMU setup in U-Boot.

> For barebox,  I would think, 3 levels of table are needed. Am I right?

I think it would be acceptable for map_io_sections() to fail with an
error code if called to remap a too small region.

So if supporting remap of only 2M for non-SDRAM makes things easier for
you, go for it.

Cheers,
Ahmad

> 
> 
> 
> 
> 
>> -----Original Message-----
>> From: Ahmad Fatoum <a.fatoum@pengutronix.de>
>> Sent: 28 November 2025 10:15
>> To: Renaud Barbier <Renaud.Barbier@ametek.com>; Barebox List
>> <barebox@lists.infradead.org>
>> Cc: Lucas Stach <lst@pengutronix.de>
>> Subject: Re: PCIE on LS1021A
>>
>> ***NOTICE*** This came from an external source. Use caution when
>> replying, clicking links, or opening attachments.
>>
>> Hi Renaud,
>>
>> On 11/18/25 6:42 PM, Renaud Barbier wrote:
>>> Hello Ahmad,
>>> This has been a while that I asked you about a question on the PCIE for the
>> LS1021A.
>>> We are developing a new product with a NVME device using the LS1021A
>>> CPU and I would like to add LPAE support to barebox so that PCI child
>> devices can be probed.
>>
>> Sounds good!
>>
>>> I am trying to figure out how to assemble LPAE support based on the
>> information you gave me below and what I see in U-boot.
>>>
>>> A grep of LPAE in U-boot shows several files with specific code for LPAE:
>>> arch/arm/cpu/armv7/start.S
>>> arch/arm/lib/cache-cp15.c
>>> arch/arm/cpu/armv7/ls102xa/cpu.c
>>>
>>> I do  recognize similar pieces of code between barebox and U-boot From
>>> the message below, I understand I would need to update map_io_sections
>>> to map the PCI address as in U-boot mmu_setup from the file
>>> arch/arm/cpu/armv7/ls102xa/cpu.c
>>
>> I imagine, you'll want to to basically duplicate mmu_32.c as mmu_32l or
>> something and reimplement it for long descriptor use.
>>
>> Types like phys_addr_t would become 64-bit (and map_io_sections should
>> take a 64-bit physical address in your case) and then you need to map the
>> PCI regions below < 4G, so the CPU may access them.
>>
>> Cheers,
>> Ahmad
>>
>>>
>>> Cheers,
>>> Renaud
>>>
>>>
>>>
>>>> -----Original Message-----
>>>> From: Ahmad Fatoum <a.fatoum@pengutronix.de>
>>>> Sent: 09 December 2022 19:18
>>>> To: Renaud Barbier <Renaud.Barbier@ametek.com>; Barebox List
>>>> <barebox@lists.infradead.org>
>>>> Cc: Lucas Stach <lst@pengutronix.de>
>>>> Subject: Re: PCIE on LS1021A
>>>>
>>>> ***NOTICE*** This came from an external source. Use caution when
>>>> replying, clicking links, or opening attachments.
>>>>
>>>> On 09.12.22 19:37, Ahmad Fatoum wrote:
>>>>>> From my debugging I can see that the Layerscape PCIE driver use VA
>>>>>> address = PA address = 0x24000000
>>>>>>
>>>>>> So  Is the problem I am seeing an issue with mapping the correct
>>>>>> physical
>>>> address for a 32-bit processor?
>>>>>>
>>>>>> If yes, how can I map the 64-bit PA to a 32-bit VA?
>>>>>
>>>>> Normally, you would call map_io_sections as pci-tegra does, but in
>>>>> your case this alone is insufficient as you will need to implement
>>>>> ARM32 LPAE support first. Once that's in place, you can use
>>>>> map_io_sections and map it to e.g. 0x24000000 as U-Boot does
>>>> arch/arm/cpu/armv7/ls102xa/cpu.c mmu_setup().
>>>>>
>>>>> U-Boot LPAE support was added to support Rpi2, which starts in HYP
>>>>> mode, but we had worked around that in barebox to not require LPAE.
>>>>> For your case however, I don't believe there's a way around using
>>>>> LPAE
>>>> page tables.
>>>>>
>>>>> Tangentially related: I don't know how the PCI controller maintains
>>>>> cache coherency, but if it does write back through CPU caches, you
>>>>> may
>>>> observe memory corruption.
>>>>>
>>>>> It may be the safest for you to disable cache snooping for PCIe
>>>>> until that's resolved (We've this planned, but it will probably not
>>>>> happen this
>>>> year.
>>>>> If you're interested I can elaborate).
>>>>
>>>> I should have shortened the context a bit. Posting again in case you
>>>> missed it.
>>>>
>>>>>
>>>>> Cheers,
>>>>> Ahmad
>>>>>
>>>>>
>>>>>
>>>>>>
>>>>>> Cheers,
>>>>>> Renaud
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>
>>>> --
>>>> Pengutronix e.K.                           |                             |
>>>> Steuerwalder Str. 21                       |
>>>>
>> https://urldefense.com/v3/__http://www.pengutronix.de/__;!!HKOSU0g!Cf
>>>> A
>>>> u4L9MQvIuEvfQmVymcZYkrdbrn98J2QbYMFsHhqFrxp8fDzjT6-
>>>> h9nSajsgYcNvsCjJVWd98oYvdyj7pvR-ZfDY0$   |
>>>> 31137 Hildesheim, Germany                  | Phone: +49-5121-206917-0    |
>>>> Amtsgericht Hildesheim, HRA 2686           | Fax:   +49-5121-206917-5555
>> |
>>>
>>
>> --
>> Pengutronix e.K.                  |                             |
>> Steuerwalder Str. 21              |
>> https://urldefense.com/v3/__http://www.pengutronix.de/__;!!HKOSU0g!Gr
>> qIPFb9B_Y1IapEh1Za8-
>> VKxnpv9bH1ejrQAh_s0Zf5MypqMk7BEOM9anjygqIVmKWxo04Tbhpm4L3YtQ
>> C2G3bKq8E$   |
>> 31137 Hildesheim, Germany         | Phone: +49-5121-206917-0    |
>> Amtsgericht Hildesheim, HRA 2686  | Fax:   +49-5121-206917-5555 |
> 

-- 
Pengutronix e.K.                  |                             |
Steuerwalder Str. 21              | http://www.pengutronix.de/  |
31137 Hildesheim, Germany         | Phone: +49-5121-206917-0    |
Amtsgericht Hildesheim, HRA 2686  | Fax:   +49-5121-206917-5555 |