Re: PCIE on LS1021A

[Ahmad Fatoum <a.fatoum@pengutronix.de>]

Hello Renaud,

On 09.12.22 18:31, Renaud Barbier wrote:
> 
> Hello,
> We have added support for the LS1021A in barebox (from v2022.03)

That's great. Do you have the patches public somewhere?

> At present neither Linux and barebox are able to probe the PCIE device connected to PE1
> The board has a switch fabric connected to PCE1.
> Using U-boot we are able to see this device and the NXP bridge

Does Linux PCIe works when booted with U-Boot? At least the LS1046A has some special
PCIe fixups. I am not sure how applicable these are to the LS1021A.

Sascha can say more on that, but I will focus on the other parts of your question.

> Using barebox we see only the Bridge. Then, it fails on the first read to get the header type from the deivice on bus 1.
> 
> We know this driver works on the LS1046A as it can detects a PCI card on the LS1046A-RDB.
> Both the LS1021A (32-bit cpu) and LS1046 (64-bit cpu) have their PCIE space to access the device conf, I/O and mem space in 64-bit address space
> 
> On the LS1046 I do see access at 0x40.xxxx.xxxx while on the LS1021A, it is only a 32-bit access using the lower 32-bit.
> 
> As an experiemnt in U-boot, I have disabled the PCI driver and configured the bridge to access the device.
> To my surprise I could see the device not using the 40-bit address. So I am not sure it gets mapped (I send a question to NXP)
> 
> => md 0x24000000
> 24000000: b86114e4 00100000 02000002 00000000  ..a.............
> 
> 
> Doint the same operation on barebox, the data are only 
> barebox:/ md 0x24000000
> 24000000: xxxxxxxx xxxxxxxx xxxxxxxx xxxxxxxx                ................
> barebox:/ md 0x4024000000
> 4024000000: xxxxxxxx xxxxxxxx xxxxxxxx xxxxxxxx              ................
> 4024000010: xxxxxxxx xxxxxxxx xxxxxxxx xxxxxxxx              ................
> 
> From my debugging I can see that the Layerscape PCIE driver use VA address = PA address = 0x24000000
> 
> So  Is the problem I am seeing an issue with mapping the correct physical address for a 32-bit processor?
> 
> If yes, how can I map the 64-bit PA to a 32-bit VA?

Normally, you would call map_io_sections as pci-tegra does, but in your
case this alone is insufficient as you will need to implement ARM32 LPAE
support first. Once that's in place, you can use map_io_sections and map
it to e.g. 0x24000000 as U-Boot does arch/arm/cpu/armv7/ls102xa/cpu.c mmu_setup().

U-Boot LPAE support was added to support Rpi2, which starts in HYP mode, but we had
worked around that in barebox to not require LPAE. For your case however, I don't believe
there's a way around using LPAE page tables.

Tangentially related: I don't know how the PCI controller maintains cache coherency,
but if it does write back through CPU caches, you may observe memory corruption.

It may be the safest for you to disable cache snooping for PCIe until that's
resolved (We've this planned, but it will probably not happen this year.
If you're interested I can elaborate).

Cheers,
Ahmad



> 
> Cheers,
> Renaud
> 
> 
> 
> 
> 

-- 
Pengutronix e.K.                           |                             |
Steuerwalder Str. 21                       | http://www.pengutronix.de/  |
31137 Hildesheim, Germany                  | Phone: +49-5121-206917-0    |
Amtsgericht Hildesheim, HRA 2686           | Fax:   +49-5121-206917-5555 |