PCIE on LS1021A

PCIE on LS1021A

[Renaud Barbier]

Hello,
We have added support for the LS1021A in barebox (from v2022.03)
At present neither Linux and barebox are able to probe the PCIE device connected to PE1
The board has a switch fabric connected to PCE1.
Using U-boot we are able to see this device and the NXP bridge

Using barebox we see only the Bridge. Then, it fails on the first read to get the header type from the deivice on bus 1.

We know this driver works on the LS1046A as it can detects a PCI card on the LS1046A-RDB.
Both the LS1021A (32-bit cpu) and LS1046 (64-bit cpu) have their PCIE space to access the device conf, I/O and mem space in 64-bit address space

On the LS1046 I do see access at 0x40.xxxx.xxxx while on the LS1021A, it is only a 32-bit access using the lower 32-bit.

As an experiemnt in U-boot, I have disabled the PCI driver and configured the bridge to access the device.
To my surprise I could see the device not using the 40-bit address. So I am not sure it gets mapped (I send a question to NXP)

=> md 0x24000000
24000000: b86114e4 00100000 02000002 00000000  ..a.............


Doint the same operation on barebox, the data are only 
barebox:/ md 0x24000000
24000000: xxxxxxxx xxxxxxxx xxxxxxxx xxxxxxxx                ................
barebox:/ md 0x4024000000
4024000000: xxxxxxxx xxxxxxxx xxxxxxxx xxxxxxxx              ................
4024000010: xxxxxxxx xxxxxxxx xxxxxxxx xxxxxxxx              ................

>From my debugging I can see that the Layerscape PCIE driver use VA address = PA address = 0x24000000

So  Is the problem I am seeing an issue with mapping the correct physical address for a 32-bit processor?

If yes, how can I map the 64-bit PA to a 32-bit VA?

Cheers,
Renaud

RE: PCIE on LS1021A

[Renaud Barbier]

Ok. I did found the piece of code in U-boot that maps the 40-bit space for the LS1021A.
Is barebox able to do an iomap on a 40-bit address or should I add the U-boot code for t?

Cheers,
Renaud

-----Original Message-----
From: barebox <barebox-bounces@lists.infradead.org> On Behalf Of Renaud Barbier
Sent: 09 December 2022 17:31
To: Barebox List <barebox@lists.infradead.org>
Subject: PCIE on LS1021A

***NOTICE*** This came from an external source. Use caution when replying, clicking links, or opening attachments.

Hello,
We have added support for the LS1021A in barebox (from v2022.03) At present neither Linux and barebox are able to probe the PCIE device connected to PE1 The board has a switch fabric connected to PCE1.
Using U-boot we are able to see this device and the NXP bridge

Using barebox we see only the Bridge. Then, it fails on the first read to get the header type from the deivice on bus 1.

We know this driver works on the LS1046A as it can detects a PCI card on the LS1046A-RDB.
Both the LS1021A (32-bit cpu) and LS1046 (64-bit cpu) have their PCIE space to access the device conf, I/O and mem space in 64-bit address space

On the LS1046 I do see access at 0x40.xxxx.xxxx while on the LS1021A, it is only a 32-bit access using the lower 32-bit.

As an experiemnt in U-boot, I have disabled the PCI driver and configured the bridge to access the device.
To my surprise I could see the device not using the 40-bit address. So I am not sure it gets mapped (I send a question to NXP)

=> md 0x24000000
24000000: b86114e4 00100000 02000002 00000000  ..a.............


Doint the same operation on barebox, the data are only barebox:/ md 0x24000000
24000000: xxxxxxxx xxxxxxxx xxxxxxxx xxxxxxxx                ................
barebox:/ md 0x4024000000
4024000000: xxxxxxxx xxxxxxxx xxxxxxxx xxxxxxxx              ................
4024000010: xxxxxxxx xxxxxxxx xxxxxxxx xxxxxxxx              ................

>From my debugging I can see that the Layerscape PCIE driver use VA address = PA address = 0x24000000

So  Is the problem I am seeing an issue with mapping the correct physical address for a 32-bit processor?

If yes, how can I map the 64-bit PA to a 32-bit VA?

Cheers,
Renaud

Re: PCIE on LS1021A

[Ahmad Fatoum]

Hello Renaud,

On 09.12.22 18:31, Renaud Barbier wrote:
> 
> Hello,
> We have added support for the LS1021A in barebox (from v2022.03)

That's great. Do you have the patches public somewhere?

> At present neither Linux and barebox are able to probe the PCIE device connected to PE1
> The board has a switch fabric connected to PCE1.
> Using U-boot we are able to see this device and the NXP bridge

Does Linux PCIe works when booted with U-Boot? At least the LS1046A has some special
PCIe fixups. I am not sure how applicable these are to the LS1021A.

Sascha can say more on that, but I will focus on the other parts of your question.

> Using barebox we see only the Bridge. Then, it fails on the first read to get the header type from the deivice on bus 1.
> 
> We know this driver works on the LS1046A as it can detects a PCI card on the LS1046A-RDB.
> Both the LS1021A (32-bit cpu) and LS1046 (64-bit cpu) have their PCIE space to access the device conf, I/O and mem space in 64-bit address space
> 
> On the LS1046 I do see access at 0x40.xxxx.xxxx while on the LS1021A, it is only a 32-bit access using the lower 32-bit.
> 
> As an experiemnt in U-boot, I have disabled the PCI driver and configured the bridge to access the device.
> To my surprise I could see the device not using the 40-bit address. So I am not sure it gets mapped (I send a question to NXP)
> 
> => md 0x24000000
> 24000000: b86114e4 00100000 02000002 00000000  ..a.............
> 
> 
> Doint the same operation on barebox, the data are only 
> barebox:/ md 0x24000000
> 24000000: xxxxxxxx xxxxxxxx xxxxxxxx xxxxxxxx                ................
> barebox:/ md 0x4024000000
> 4024000000: xxxxxxxx xxxxxxxx xxxxxxxx xxxxxxxx              ................
> 4024000010: xxxxxxxx xxxxxxxx xxxxxxxx xxxxxxxx              ................
> 
> From my debugging I can see that the Layerscape PCIE driver use VA address = PA address = 0x24000000
> 
> So  Is the problem I am seeing an issue with mapping the correct physical address for a 32-bit processor?
> 
> If yes, how can I map the 64-bit PA to a 32-bit VA?

Normally, you would call map_io_sections as pci-tegra does, but in your
case this alone is insufficient as you will need to implement ARM32 LPAE
support first. Once that's in place, you can use map_io_sections and map
it to e.g. 0x24000000 as U-Boot does arch/arm/cpu/armv7/ls102xa/cpu.c mmu_setup().

U-Boot LPAE support was added to support Rpi2, which starts in HYP mode, but we had
worked around that in barebox to not require LPAE. For your case however, I don't believe
there's a way around using LPAE page tables.

Tangentially related: I don't know how the PCI controller maintains cache coherency,
but if it does write back through CPU caches, you may observe memory corruption.

It may be the safest for you to disable cache snooping for PCIe until that's
resolved (We've this planned, but it will probably not happen this year.
If you're interested I can elaborate).

Cheers,
Ahmad



> 
> Cheers,
> Renaud
> 
> 
> 
> 
> 

-- 
Pengutronix e.K.                           |                             |
Steuerwalder Str. 21                       | http://www.pengutronix.de/  |
31137 Hildesheim, Germany                  | Phone: +49-5121-206917-0    |
Amtsgericht Hildesheim, HRA 2686           | Fax:   +49-5121-206917-5555 |

RE: PCIE on LS1021A

[Renaud Barbier]

> That's great. Do you have the patches public somewhere?

I do have support for theLS1021A-IOT. The machine support is living in its own mach-ls102x (as u-boot did).
I would need to review the code and prepare patches.  I will ask my manager when I can spend time doing that.,


> Does Linux PCIe works when booted with U-Boot? At least the LS1046A has some special PCIe fixups. I am not sure how applicable these are to the LS1021A.
Not sure why but Linux does not start in my U-boot. I need to fix that.

Re: PCIE on LS1021A

[Ahmad Fatoum]

Hi,

On 09.12.22 19:58, Renaud Barbier wrote:
> 
> 
>> That's great. Do you have the patches public somewhere?
> 
> I do have support for theLS1021A-IOT. The machine support is living in its own mach-ls102x (as u-boot did).
> I would need to review the code and prepare patches.  I will ask my manager when I can spend time doing that.,

Even if you don't get around yet to submit them formally for inclusion,
it's good to know where they are if someone wants to continue work
on them.

I still highly recommend you post your LPAE patches to the mailing list
when you done implementing them.


>> Does Linux PCIe works when booted with U-Boot? At least the LS1046A has some special PCIe fixups. I am not sure how applicable these are to the LS1021A.
> Not sure why but Linux does not start in my U-boot. I need to fix that.

Or add LPAE to barebox ;)

Cheers,
Ahmad
 

-- 
Pengutronix e.K.                           |                             |
Steuerwalder Str. 21                       | http://www.pengutronix.de/  |
31137 Hildesheim, Germany                  | Phone: +49-5121-206917-0    |
Amtsgericht Hildesheim, HRA 2686           | Fax:   +49-5121-206917-5555 |

RE: PCIE on LS1021A

[Renaud Barbier]

.>> Does Linux PCIe works when booted with U-Boot? At least the LS1046A has some special PCIe fixups. I am not sure how applicable these are to the LS1021A.
>> Not sure why but Linux does not start in my U-boot. I need to fix that.

>Or add LPAE to barebox ;)
Thanks. I enabled LPAE in Linux and  the device now is detected.
Hopefully, I will get you the basic of the LS1021A-IOT early next year. 


Cheers,
Renaud

Re: PCIE on LS1021A

[Ahmad Fatoum]

On 09.12.22 19:37, Ahmad Fatoum wrote:
>> From my debugging I can see that the Layerscape PCIE driver use VA address = PA address = 0x24000000
>>
>> So  Is the problem I am seeing an issue with mapping the correct physical address for a 32-bit processor?
>>
>> If yes, how can I map the 64-bit PA to a 32-bit VA?
> 
> Normally, you would call map_io_sections as pci-tegra does, but in your
> case this alone is insufficient as you will need to implement ARM32 LPAE
> support first. Once that's in place, you can use map_io_sections and map
> it to e.g. 0x24000000 as U-Boot does arch/arm/cpu/armv7/ls102xa/cpu.c mmu_setup().
> 
> U-Boot LPAE support was added to support Rpi2, which starts in HYP mode, but we had
> worked around that in barebox to not require LPAE. For your case however, I don't believe
> there's a way around using LPAE page tables.
> 
> Tangentially related: I don't know how the PCI controller maintains cache coherency,
> but if it does write back through CPU caches, you may observe memory corruption.
> 
> It may be the safest for you to disable cache snooping for PCIe until that's
> resolved (We've this planned, but it will probably not happen this year.
> If you're interested I can elaborate).

I should have shortened the context a bit. Posting again in case you missed it.

> 
> Cheers,
> Ahmad
> 
> 
> 
>>
>> Cheers,
>> Renaud
>>
>>
>>
>>
>>
> 

-- 
Pengutronix e.K.                           |                             |
Steuerwalder Str. 21                       | http://www.pengutronix.de/  |
31137 Hildesheim, Germany                  | Phone: +49-5121-206917-0    |
Amtsgericht Hildesheim, HRA 2686           | Fax:   +49-5121-206917-5555 |