i.MX6 PCIe support

i.MX6 PCIe support

[Andrey Smirnov]

Hi All,

I am working on adding EEPROM programming capabilities to e1000 driver
on a custom i.MX6Q (automotive) board with i210 (8086:1531) connected
over PCIe link. Unfortunately I am observing some bad/unexpected
behavior that might indicate PCIe issues.

What I am seeing is device(i210) enumerating on PCIe bus, however any
accesses to its registers via its memory "window" result in either
0xFFFFFFFF or 0xDEADBEEF. I tried accessing various PCIe configuration
registers of that device and it seem to work fine (as a part of that
test I read out some AER registers and they didn't indicate presence
of any PCIe link related errors).

However when I boot Linux kernel (with 'igb' driver patched to
recognize 8086:1531 VID:PID) on that board and run 'eepromARMtool'
(Intel provided EEPROM programming tool) I can see it reading various
registers correctly

My questions are:
   - Is anyone currently using i.MX6 PCIe driver in 'next' branch?
   - Has anyone seen similar behavior on this particular controller
and discovered the root cause?

Any thought or advise are greatly appreciated.

Thanks,
Andrey Smirnov

_______________________________________________
barebox mailing list
barebox@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/barebox

Re: i.MX6 PCIe support

[Lucas Stach]

Hi Andrey,

Am Montag, den 30.11.2015, 10:56 -0800 schrieb Andrey Smirnov:
> Hi All,
> 
> I am working on adding EEPROM programming capabilities to e1000 driver
> on a custom i.MX6Q (automotive) board with i210 (8086:1531) connected
> over PCIe link. Unfortunately I am observing some bad/unexpected
> behavior that might indicate PCIe issues.
> 
> What I am seeing is device(i210) enumerating on PCIe bus, however any
> accesses to its registers via its memory "window" result in either
> 0xFFFFFFFF or 0xDEADBEEF. I tried accessing various PCIe configuration
> registers of that device and it seem to work fine (as a part of that
> test I read out some AER registers and they didn't indicate presence
> of any PCIe link related errors).
> 
> However when I boot Linux kernel (with 'igb' driver patched to
> recognize 8086:1531 VID:PID) on that board and run 'eepromARMtool'
> (Intel provided EEPROM programming tool) I can see it reading various
> registers correctly
> 
> My questions are:
>    - Is anyone currently using i.MX6 PCIe driver in 'next' branch?

I didn't try the latest revision, but with v2015.10.0 PCIe on i.MX6 was
working fine.

>    - Has anyone seen similar behavior on this particular controller
> and discovered the root cause?
> 
The symptoms you describe do look to me like the firmware on the i210
device doesn't get initialized. Unfortunately I don't have any specific
pointers where to start searching for the root cause.

Regards,
Lucas
-- 
Pengutronix e.K.             | Lucas Stach                 |
Industrial Linux Solutions   | http://www.pengutronix.de/  |


_______________________________________________
barebox mailing list
barebox@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/barebox

Re: i.MX6 PCIe support

[Andrey Smirnov]

>> However when I boot Linux kernel (with 'igb' driver patched to
>> recognize 8086:1531 VID:PID) on that board and run 'eepromARMtool'
>> (Intel provided EEPROM programming tool) I can see it reading various
>> registers correctly
>>
>> My questions are:
>>    - Is anyone currently using i.MX6 PCIe driver in 'next' branch?
>
> I didn't try the latest revision, but with v2015.10.0 PCIe on i.MX6 was
> working fine.
>
>>    - Has anyone seen similar behavior on this particular controller
>> and discovered the root cause?
>>
> The symptoms you describe do look to me like the firmware on the i210
> device doesn't get initialized. Unfortunately I don't have any specific
> pointers where to start searching for the root cause.
>

I kept looking into the issue and noticed one thing that Linux kernel
does differently -- when handling PCI bridges it allocates bridge
window before that PCI bridge's BAR windows in address space as can be
seen below:

[    0.831099] imx6q-pcie 1ffc000.pcie: PCI host bridge to bus 0000:00
[    0.837480] pci_bus 0000:00: root bus resource [io  0x1000-0x10000]
[    0.843878] pci_bus 0000:00: root bus resource [mem 0x01000000-0x01efffff]
[    0.850881] pci_bus 0000:00: root bus resource [bus 00-ff]
[    0.856913] PCI: bus0: Fast back to back transfers disabled
[    0.880182] PCI: bus1: Fast back to back transfers disabled
[    0.885980] pci 0000:00:00.0: BAR 8: assigned [mem 0x01000000-0x01bfffff]
[    0.892902] pci 0000:00:00.0: BAR 0: assigned [mem 0x01c00000-0x01cfffff]
[    0.899806] pci 0000:00:00.0: BAR 6: assigned [mem
0x01d00000-0x01d0ffff pref]
[    0.907154] pci 0000:00:00.0: BAR 7: assigned [io  0x1000-0x1fff]
[    0.913387] pci 0000:01:00.0: BAR 0: assigned [mem 0x01000000-0x017fffff]
[    0.920324] pci 0000:01:00.0: BAR 3: assigned [mem 0x01800000-0x01803fff]
[    0.927241] pci 0000:01:00.0: BAR 2: assigned [io  0x1000-0x101f]
[    0.933480] pci 0000:00:00.0: PCI bridge to [bus 01]
[    0.938553] pci 0000:00:00.0:   bridge window [io  0x1000-0x1fff]
[    0.944772] pci 0000:00:00.0:   bridge window [mem 0x01000000-0x01bfffff]
[    0.951984] pcieport 0000:00:00.0: Signaling PME through PCIe PME interrupt
[    0.959056] pci 0000:01:00.0: Signaling PME through PCIe PME interrupt


When I tried emulating this behavior in Barebox by moving
"setup_device(dev, 2)" from line 384 to be right after
"postscan_setup_bridge(dev)" in "drivers/pci/pci.c" I was able to
communicate with i210.

I am bit hesitant to make a patch yet, since for the life of me I
cannot explain this behavior, any thoughts on why this can be
happening?

Andrey

_______________________________________________
barebox mailing list
barebox@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/barebox

Re: i.MX6 PCIe support

[Lucas Stach]

Am Sonntag, den 06.12.2015, 22:03 -0800 schrieb Andrey Smirnov:
> >> However when I boot Linux kernel (with 'igb' driver patched to
> >> recognize 8086:1531 VID:PID) on that board and run 'eepromARMtool'
> >> (Intel provided EEPROM programming tool) I can see it reading various
> >> registers correctly
> >>
> >> My questions are:
> >>    - Is anyone currently using i.MX6 PCIe driver in 'next' branch?
> >
> > I didn't try the latest revision, but with v2015.10.0 PCIe on i.MX6 was
> > working fine.
> >
> >>    - Has anyone seen similar behavior on this particular controller
> >> and discovered the root cause?
> >>
> > The symptoms you describe do look to me like the firmware on the i210
> > device doesn't get initialized. Unfortunately I don't have any specific
> > pointers where to start searching for the root cause.
> >
> 
> I kept looking into the issue and noticed one thing that Linux kernel
> does differently -- when handling PCI bridges it allocates bridge
> window before that PCI bridge's BAR windows in address space as can be
> seen below:
> 
> [    0.831099] imx6q-pcie 1ffc000.pcie: PCI host bridge to bus 0000:00
> [    0.837480] pci_bus 0000:00: root bus resource [io  0x1000-0x10000]
> [    0.843878] pci_bus 0000:00: root bus resource [mem 0x01000000-0x01efffff]
> [    0.850881] pci_bus 0000:00: root bus resource [bus 00-ff]
> [    0.856913] PCI: bus0: Fast back to back transfers disabled
> [    0.880182] PCI: bus1: Fast back to back transfers disabled
> [    0.885980] pci 0000:00:00.0: BAR 8: assigned [mem 0x01000000-0x01bfffff]
> [    0.892902] pci 0000:00:00.0: BAR 0: assigned [mem 0x01c00000-0x01cfffff]
> [    0.899806] pci 0000:00:00.0: BAR 6: assigned [mem
> 0x01d00000-0x01d0ffff pref]
> [    0.907154] pci 0000:00:00.0: BAR 7: assigned [io  0x1000-0x1fff]
> [    0.913387] pci 0000:01:00.0: BAR 0: assigned [mem 0x01000000-0x017fffff]
> [    0.920324] pci 0000:01:00.0: BAR 3: assigned [mem 0x01800000-0x01803fff]
> [    0.927241] pci 0000:01:00.0: BAR 2: assigned [io  0x1000-0x101f]
> [    0.933480] pci 0000:00:00.0: PCI bridge to [bus 01]
> [    0.938553] pci 0000:00:00.0:   bridge window [io  0x1000-0x1fff]
> [    0.944772] pci 0000:00:00.0:   bridge window [mem 0x01000000-0x01bfffff]
> [    0.951984] pcieport 0000:00:00.0: Signaling PME through PCIe PME interrupt
> [    0.959056] pci 0000:01:00.0: Signaling PME through PCIe PME interrupt
> 
> 
> When I tried emulating this behavior in Barebox by moving
> "setup_device(dev, 2)" from line 384 to be right after
> "postscan_setup_bridge(dev)" in "drivers/pci/pci.c" I was able to
> communicate with i210.
> 
> I am bit hesitant to make a patch yet, since for the life of me I
> cannot explain this behavior, any thoughts on why this can be
> happening?
> 
Can you please compile barebox with config PCI_DEBUG set and provide the
output of that? Mybe with and without your change applied?

Thanks,
Lucas

-- 
Pengutronix e.K.             | Lucas Stach                 |
Industrial Linux Solutions   | http://www.pengutronix.de/  |


_______________________________________________
barebox mailing list
barebox@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/barebox

Re: i.MX6 PCIe support

[Andrey Smirnov]

> Can you please compile barebox with config PCI_DEBUG set and provide the
> output of that? Mybe with and without your change applied?

Here's the output (I also added the code to print bridge's *_BASE and
*_LIMIT values):

Good (with my change present):

imx6-pcie 1ffc000.pcie: Link up, Gen=1
pci: pci_scan_bus for bus 0
pci:  last_io = 0x00000000, last_mem = 0x01000000, last_mem_pref = 0x00000000
pci: class = 00000604, hdr_type = 00000001
pci: 00:00 [16c3:abcd]
pci: pci_scan_bus for bus 1
pci:  last_io = 0x00000000, last_mem = 0x01000000, last_mem_pref = 0x00000000
pci: class = 00000200, hdr_type = 00000000
pci: 01:00 [8086:1531]
pci: pbar0: mask=ff800000 NP memory 8388608 bytes
pci: pbar1 set bad mask
pci: pbar2: mask=ffffffe1 io 32 bytes
pci: pbar3: mask=ffffc000 NP memory 16384 bytes
pci: pbar4 set bad mask
pci: pbar5 set bad mask
pci: pci_scan_bus returning with max=02
pci: pbar0: mask=fff0000e NP memory 1048576 bytes
pci: pbar1 set bad mask
pci-16c3:abcd.0: bridge window [mem 0x10000000-0x00000000 pref]
pci-16c3:abcd.0: bridge window [mem 0x01000000-0x01800000]
pci-16c3:abcd.0: bridge window [io 0x00000000-0x00000000]

Bad (without my change present):

imx6-pcie 1ffc000.pcie: Link up, Gen=1
pci: pci_scan_bus for bus 0
pci:  last_io = 0x00000000, last_mem = 0x01000000, last_mem_pref = 0x00000000
pci: class = 00000604, hdr_type = 00000001
pci: 00:00 [16c3:abcd]
pci: pbar0: mask=fff0000e NP memory 1048576 bytes
pci: pbar1 set bad mask
pci: pci_scan_bus for bus 1
pci:  last_io = 0x00000000, last_mem = 0x01100000, last_mem_pref = 0x00000000
pci: class = 00000200, hdr_type = 00000000
pci: 01:00 [8086:1531]
pci: pbar0: mask=ff800000 NP memory 8388608 bytes
pci: pbar1 set bad mask
pci: pbar2: mask=ffffffe1 io 32 bytes
pci: pbar3: mask=ffffc000 NP memory 16384 bytes
pci: pbar4 set bad mask
pci: pbar5 set bad mask
pci: pci_scan_bus returning with max=02
pci-16c3:abcd.0: bridge window [mem 0x10000000-0x00000000 pref]
pci-16c3:abcd.0: bridge window [mem 0x01100000-0x01900000]
pci-16c3:abcd.0: bridge window [io 0x00000000-0x00000000]


Regards,
Andrey

_______________________________________________
barebox mailing list
barebox@lists.infradead.org
http://lists.infradead.org/mailman/listinfo/barebox