RE: PCIE on LS1021A

[Renaud Barbier <Renaud.Barbier@ametek.com>]

Based on your information and U-boot and I have started to work on the LPAE support. So far full of debugging and hacks.

It is based on the mmu_32.c file. As I have failed to use the 3 MMU tables,  at present I am using only 2 as  in u-boot.
The 64-bit PCI space is remapped with:
map_io_sections(0x4000000000ULL ,IOMEM(0x24000000UL), 192 << 20);

To detect the NVME device, the virtulal address 0x24000000 is hard-coded into the functions dw_pcie_[wr|rd]_other_conf of drivers/pci/pcie-designware-host.c as follows:
if (bus->primary == pp->root_bus_nr) {
                  type = PCIE_ATU_TYPE_CFG0;
                  cpu_addr = pp->cfg0_base;
                  cfg_size = pp->cfg0_size;
                  pp->va_cfg0_base = IOMEM(0x24000000); /* XXX */
                  va_cfg_base = pp->va_cfg0_base;

What is the method to pass the address to the driver?

And I get the following:
layerscape-pcie 3400000.pcie@3400000.of: host bridge /soc/pcie@3400000 ranges:
layerscape-pcie 3400000.pcie@3400000.of: Parsing ranges property...
layerscape-pcie 3400000.pcie@3400000.of:       IO 0x4000010000..0x400001ffff -> 0x0000000000
layerscape-pcie 3400000.pcie@3400000.of:      MEM 0x4040000000..0x407fffffff -> 0x0040000000

ERROR: io_bus_addr = 0x0, io_base = 0x4000010000
ERROR: mem_bus_addr = 0x4040000000 -> Based on Linux output, the mem_bus_addr should be as above 0x4000.0000 to be programmed in the ATU target register.
ERROR: mem_base = 0x4040000000, offset = 0x0

ERROR: layerscape-pcie 3400000.pcie@3400000.of: iATU unroll: disabled

pci: pci_scan_bus for bus 0
pci:  last_io = 0x00010000, last_mem = 0x40000000, last_mem_pref = 0x00000000
pci: class = 00000604, hdr_type = 00000001
pci: 00:00 [1957:0e0a]
pci: pci_scan_bus for bus 1
pci:  last_io = 0x00010000, last_mem = 0x40000000, last_mem_pref = 0x00000000

pci: class = 00000108, hdr_type = 00000000
pci: 01:00 [126f:2263] -> NVME device found
pci: pbar0: mask=ffffc004 NP-MEM 16384 bytes
ERROR: pci: &&&  sub = 0x2263, 0x126f kind = NP-MEM&&&
ERROR: pci: &&& write BAR 0x10 = 0x40000000 &&&
...
pci: pci_scan_bus returning with max=02
pci: bridge NP limit at 0x40100000
pci: bridge IO limit at 0x00010000
pci: pbar0: mask=ff000000 NP-MEM 16777216 bytes
pci: pbar1: mask=fc000000 NP-MEM 67108864 bytes
pci: pci_scan_bus returning with max=02
ERROR: nvme pci-126f:2263.0: enabling bus mastering

Then, the system hangs on the instruction 3 lines below:
ERROR: nvme_pci_enable : 0x4000001c -> Fails to access the NVME CSTS register. It does not matter if mem_bus_addr is set to 0x4000.0000 to program the ATU to translate the address 0x40.4000.0000 to 0x4000.0000.
if (readl(dev->bar + NVME_REG_CSTS) == -1)

0x4000.0000 is also the quadSPI memory area. So I guess I should remap the access too.

Unhappily, my work is now at a stop as there is a hardware failure on my system.

Note: the MMU may not be set properly as the out of-band fails to on TX timeout. I can reach the prompt after the NVME probing failed.