Look. I know 4+ people with the same issues. All using Linux. On various different machines. All of them are currently making the switch away from Saleae. Please take this seriously. The jump from v1 to v2 was somewhat of an inconvenience to some because of the change in file formats. The new GUI looks great, sure. But apparently it is quite unstable. I did not need to use a Saleae for a year and now have some work where I expected this to be my daily driver for the next couple of months. I considered buying another Saleae for a student of mine. But the extreme price hike combined with such annoying problems is deeply frustrating. Reading about a new internal hardware being used … but otherwise same features; seems like a serious chance missed to add a hardware trigger input/output …
I was also checking that the USB port used for the Saleae is directly attached to its own host controller. Based on the lsusb -t output, I think that is the case:
Bus 04.Port 1: Dev 1, Class=root_hub, Driver=xhci_hcd/6p, 20000M
|__ Port 1: Dev 49, If 0, Class=Vendor Specific Class, Driver=usbfs, 5000M
<no further entries for Bus 04>
and a simple lsusb gives:
Bus 004 Device 049: ID 21a9:1006 Saleae Logic Pro
Bus 004 Device 001: ID 1d6b:0003 Linux Foundation 3.0 root hub
Thanks for all of the information here and through email! We’re working on those crash dumps next, while we wait for the Lenovo we ordered to arrive.
Quick update. We’ve poured through a good number of stack traces so far, and we have fixes internally for more than half of the crashes so far! We need to do more testing, but these fixes will be in the next release.
However, it appears as if the root cause for these crashes was the device disconnecting and reconnecting in the span of a few seconds. I suspect that this is not physically happening, and something else is going wrong.
Could anyone observing crashes on Linux do a few things?
- send us the latest dmesg output around the time of the crash. We’re looking for any USB connectivity issues, although I’m not exactly sure what we’re looking for yet.
- Check the physical connection to the device. Do the USB connections feel well seated? Do you observe device disconnections & reconnections when wiggling cables?
- If you notice any pattern to device device disconnections, e.g. when performing a specific action in our software, or while interacting with another device connected via USB?
I’m so happy to see the progress on this, thank you!
-
Honestly, it is too much of a hassle to collect logs without direction, because it corrupts my entire USB controller. This means, I have to reboot each time this occurs.
-
Both my Saleae and PC are gently used, and the connections feel very tight and secure. There is no wiggling when this occurs.
-
The software pattern is very straight-forward. Record three (3) lines (e.g. I2C - 3V3, SCL, SDA) with both analog and digital samples at the highest speeds for both. This is 100% reproducible on my machine. If, for some reason, that doesn’t work, then just keep adding lines to sample until it repros.
This method doesn’t require any other device to be plugged in. When I originally reported this ages ago, I was noticing that using the Saleae while debugging with STLINK-V3MINI was causing corruption. Later, I was able to observe the same behavior, and, more generally, it appears to point to the USB controller simply being overwhelmed.
Hi all,
We’ve just released 2.3.53, which you can get here: Logic 2.3.53
This fixes the application crashed caused by the device disconnecting and reconnecting quickly. However, we haven’t yet identified the underlying problem of why it’s falling off the bus and reconnecting.
@Zak, could you send us the output of lspci -k? I’d like to check which USB 3 host controllers are in that machine.
Also, if you’re seeing capture failures at the start of the capture, could you run the tests I mentioned above using the SALEAE_USB_TRANSFER_LIMIT environment variable? If the capture fails right out of the gate at higher sample rates, it might just be that we’re failing to submit all the URBs. We submit more and more URBs at once for higher and higher sample rates, until we hit a cap. That environment variable allows you to lower that cap, which could potentially solve that issue.
Here is the result of the command you wanted to see, but this is a busy week and I can’t run the test at this time.
$ lspci -k
00:00.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Root Complex
Subsystem: CLEVO/KAPOK Computer Starship/Matisse Root Complex
00:00.2 IOMMU: Advanced Micro Devices, Inc. [AMD] Starship/Matisse IOMMU
Subsystem: CLEVO/KAPOK Computer Starship/Matisse IOMMU
00:01.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
00:01.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse GPP Bridge
Kernel driver in use: pcieport
00:01.3 PCI bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse GPP Bridge
Kernel driver in use: pcieport
00:02.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
00:03.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
00:03.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse GPP Bridge
Kernel driver in use: pcieport
00:04.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
00:05.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
00:07.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
00:07.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B]
Kernel driver in use: pcieport
00:08.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Host Bridge
00:08.1 PCI bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B]
Kernel driver in use: pcieport
00:08.2 PCI bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B]
Kernel driver in use: pcieport
00:08.3 PCI bridge: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Internal PCIe GPP Bridge 0 to bus[E:B]
Kernel driver in use: pcieport
00:14.0 SMBus: Advanced Micro Devices, Inc. [AMD] FCH SMBus Controller (rev 61)
Subsystem: CLEVO/KAPOK Computer FCH SMBus Controller
Kernel driver in use: piix4_smbus
Kernel modules: i2c_piix4, sp5100_tco
00:14.3 ISA bridge: Advanced Micro Devices, Inc. [AMD] FCH LPC Bridge (rev 51)
Subsystem: CLEVO/KAPOK Computer FCH LPC Bridge
00:18.0 Host bridge: Advanced Micro Devices, Inc. [AMD] Matisse/Vermeer Data Fabric: Device 18h; Function 0
00:18.1 Host bridge: Advanced Micro Devices, Inc. [AMD] Matisse/Vermeer Data Fabric: Device 18h; Function 1
00:18.2 Host bridge: Advanced Micro Devices, Inc. [AMD] Matisse/Vermeer Data Fabric: Device 18h; Function 2
00:18.3 Host bridge: Advanced Micro Devices, Inc. [AMD] Matisse/Vermeer Data Fabric: Device 18h; Function 3
Kernel driver in use: k10temp
Kernel modules: k10temp
00:18.4 Host bridge: Advanced Micro Devices, Inc. [AMD] Matisse/Vermeer Data Fabric: Device 18h; Function 4
00:18.5 Host bridge: Advanced Micro Devices, Inc. [AMD] Matisse/Vermeer Data Fabric: Device 18h; Function 5
00:18.6 Host bridge: Advanced Micro Devices, Inc. [AMD] Matisse/Vermeer Data Fabric: Device 18h; Function 6
00:18.7 Host bridge: Advanced Micro Devices, Inc. [AMD] Matisse/Vermeer Data Fabric: Device 18h; Function 7
01:00.0 Non-Volatile memory controller: Samsung Electronics Co Ltd NVMe SSD Controller SM981/PM981/PM983
Subsystem: Samsung Electronics Co Ltd SSD 970 EVO Plus 1TB
Kernel driver in use: nvme
Kernel modules: nvme
02:00.0 USB controller: Advanced Micro Devices, Inc. [AMD] 400 Series Chipset USB 3.1 XHCI Controller (rev 01)
Subsystem: CLEVO/KAPOK Computer 400 Series Chipset USB 3.1 XHCI Controller
Kernel driver in use: xhci_hcd
Kernel modules: xhci_pci
02:00.1 SATA controller: Advanced Micro Devices, Inc. [AMD] 400 Series Chipset SATA Controller (rev 01)
Subsystem: CLEVO/KAPOK Computer 400 Series Chipset SATA Controller
Kernel driver in use: ahci
Kernel modules: ahci
02:00.2 PCI bridge: Advanced Micro Devices, Inc. [AMD] 400 Series Chipset PCIe Bridge (rev 01)
Kernel driver in use: pcieport
03:00.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] 400 Series Chipset PCIe Port (rev 01)
DeviceName: Broadcom 5762
Kernel driver in use: pcieport
03:01.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] 400 Series Chipset PCIe Port (rev 01)
Kernel driver in use: pcieport
03:04.0 PCI bridge: Advanced Micro Devices, Inc. [AMD] 400 Series Chipset PCIe Port (rev 01)
Kernel driver in use: pcieport
04:00.0 Unassigned class [ff00]: Realtek Semiconductor Co., Ltd. RTL8411B PCI Express Card Reader (rev 01)
Subsystem: CLEVO/KAPOK Computer RTL8411B PCI Express Card Reader
Kernel driver in use: rtsx_pci
Kernel modules: rtsx_pci
04:00.1 Ethernet controller: Realtek Semiconductor Co., Ltd. RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller (rev 12)
Subsystem: CLEVO/KAPOK Computer RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller
Kernel driver in use: r8169
Kernel modules: r8169
05:00.0 Network controller: Intel Corporation Wi-Fi 6 AX200 (rev 1a)
Subsystem: Intel Corporation Wi-Fi 6 AX200NGW
Kernel driver in use: iwlwifi
Kernel modules: iwlwifi
07:00.0 VGA compatible controller: NVIDIA Corporation TU106M [GeForce RTX 2070 Mobile / Max-Q Refresh] (rev a1)
Subsystem: CLEVO/KAPOK Computer TU106M [GeForce RTX 2070 Mobile / Max-Q Refresh]
Kernel driver in use: nvidia
Kernel modules: nvidiafb, nouveau, nvidia_drm, nvidia
07:00.1 Audio device: NVIDIA Corporation TU106 High Definition Audio Controller (rev a1)
Subsystem: CLEVO/KAPOK Computer TU106 High Definition Audio Controller
Kernel driver in use: snd_hda_intel
Kernel modules: snd_hda_intel
07:00.2 USB controller: NVIDIA Corporation TU106 USB 3.1 Host Controller (rev a1)
Subsystem: NVIDIA Corporation TU106 USB 3.1 Host Controller
Kernel driver in use: xhci_hcd
Kernel modules: xhci_pci
07:00.3 Serial bus controller: NVIDIA Corporation TU106 USB Type-C UCSI Controller (rev a1)
Subsystem: NVIDIA Corporation TU106 USB Type-C UCSI Controller
Kernel modules: i2c_nvidia_gpu
08:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse PCIe Dummy Function
Subsystem: CLEVO/KAPOK Computer Starship/Matisse PCIe Dummy Function
09:00.0 Non-Essential Instrumentation [1300]: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Reserved SPP
Subsystem: CLEVO/KAPOK Computer Starship/Matisse Reserved SPP
09:00.1 Encryption controller: Advanced Micro Devices, Inc. [AMD] Starship/Matisse Cryptographic Coprocessor PSPCPP
Subsystem: CLEVO/KAPOK Computer Starship/Matisse Cryptographic Coprocessor PSPCPP
Kernel driver in use: ccp
Kernel modules: ccp
09:00.3 USB controller: Advanced Micro Devices, Inc. [AMD] Matisse USB 3.0 Host Controller
Subsystem: Advanced Micro Devices, Inc. [AMD] Matisse USB 3.0 Host Controller
Kernel driver in use: xhci_hcd
Kernel modules: xhci_pci
09:00.4 Audio device: Advanced Micro Devices, Inc. [AMD] Starship/Matisse HD Audio Controller
Subsystem: CLEVO/KAPOK Computer Starship/Matisse HD Audio Controller
Kernel driver in use: snd_hda_intel
Kernel modules: snd_hda_intel
0a:00.0 SATA controller: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] (rev 51)
Subsystem: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode]
Kernel driver in use: ahci
Kernel modules: ahci
0b:00.0 SATA controller: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode] (rev 51)
Subsystem: Advanced Micro Devices, Inc. [AMD] FCH SATA Controller [AHCI mode]
Kernel driver in use: ahci
Kernel modules: ahci
Thanks Zak! If/when you have a chance to test 2.3.53, please let us know if it’s improved - same for anyone else having this issue.
We still haven’t heard back from anyone directly about this, but so far we’ve seen exactly 1 of these crashes since the update, and previously we were seeing about 50 crash reports a week.
The new prompt is pretty cool, and does a good job of explaining the problem. Unfortunately, it leaves the device in a weird state.
It appears to have crashed my USB driver again.
Here is the log from dmesg:
Crash event:
[ 464.338797] xhci_hcd 0000:02:00.0: ERROR unknown event type 37
Unpluged the Saleae:
[ 546.281520] usb 2-2: USB disconnect, device number 2
Replugged in the Saleae:
[ 556.519174] usb 1-2: new high-speed USB device number 5 using xhci_hcd
[ 561.862055] usb 1-2: device descriptor read/64, error -110
[ 569.404750] usb 1-10: Failed to suspend device, error -110
[ 577.478952] usb 1-2: device descriptor read/64, error -110
[ 577.922822] usb 1-2: new high-speed USB device number 6 using xhci_hcd
[ 583.101923] usb 1-2: device descriptor read/64, error -110
[ 598.719306] usb 1-2: device descriptor read/64, error -110
[ 598.829167] usb usb1-port2: attempt power cycle
[ 599.243198] usb 1-2: new high-speed USB device number 7 using xhci_hcd
[ 604.470710] usb 1-2: device descriptor read/8, error -110
[ 609.846004] usb 1-2: device descriptor read/8, error -110
[ 610.289519] usb 1-2: new high-speed USB device number 8 using xhci_hcd
[ 615.477111] usb 1-2: device descriptor read/8, error -110
[ 620.856350] usb 1-2: device descriptor read/8, error -110
[ 620.965965] usb usb1-port2: unable to enumerate USB device
[ 626.365000] usb 1-1: reset high-speed USB device number 2 using xhci_hcd
[ 631.738731] usb 1-1: device descriptor read/64, error -110
[ 633.650742] usb 1-10: Failed to suspend device, error -110
[ 647.344968] usb 1-1: device descriptor read/64, error -110
[ 647.792917] usb 1-1: reset high-speed USB device number 2 using xhci_hcd
[ 652.980377] usb 1-1: device descriptor read/64, error -110
[ 668.594877] usb 1-1: device descriptor read/64, error -110
[ 669.038840] usb 1-1: reset high-speed USB device number 2 using xhci_hcd
[ 674.094678] usb 1-1: device descriptor read/8, error -110
[ 679.470318] usb 1-1: device descriptor read/8, error -110
[ 679.926750] usb 1-1: reset high-speed USB device number 2 using xhci_hcd
[ 685.105766] usb 1-1: device descriptor read/8, error -110
[ 690.481311] usb 1-1: device descriptor read/8, error -110
[ 690.593065] usb 1-1: USB disconnect, device number 2
[ 690.797003] usb 1-1: new high-speed USB device number 9 using xhci_hcd
[ 695.988656] usb 1-1: device descriptor read/64, error -110
[ 711.595601] usb 1-1: device descriptor read/64, error -110
[ 712.043535] usb 1-1: new high-speed USB device number 10 using xhci_hcd
[ 717.227252] usb 1-1: device descriptor read/64, error -110
[ 732.842366] usb 1-1: device descriptor read/64, error -110
[ 732.951845] usb usb1-port1: attempt power cycle
[ 733.362285] usb 1-1: new high-speed USB device number 11 using xhci_hcd
[ 738.602338] usb 1-1: device descriptor read/8, error -110
[ 743.978158] usb 1-1: device descriptor read/8, error -110
[ 744.433723] usb 1-1: new high-speed USB device number 12 using xhci_hcd
[ 749.609798] usb 1-1: device descriptor read/8, error -110
[ 754.989528] usb 1-1: device descriptor read/8, error -110
[ 755.098756] usb usb1-port1: unable to enumerate USB device
I tried both power cycling the Saleae and restarting the application, no change.
After rebooting my computer, the USB driver is restored and the Saleae works again. I really appreciate the work you guys are doing here, because this type of crash is a painful and horrible experience.
On an unrelated note, I’m seeing the following ERROR on the command line. It occurs every time I launch the Logic application and has nothing to do with the crash.
[325752:0526/074153.714697:ERROR:nss_util.cc(286)] After loading Root Certs, loaded==false: NSS error code: -8018
Thanks Zak, this could help a lot! I think this is a separate issue than the device disconnect/reconnect race that we’ve been seeing with other users, but I can’t say for sure yet. We also haven’t heard reports of the USB controller getting into a bad state, I think that’s unique to your case so far.
Error type 37 appears to indicate that the event ring is full. I wonder if this is related to the number of USB transfers our software is queueing. Above, I have instructions to test the environment variable SALEAE_USB_TRANSFER_LIMIT.
There is a lot of instructions up there, and I know your time is quite valuable. If you can run one test, please try the following:
before launching logic from the command line:
export SALEAE_USB_TRANSFER_LIMIT=1024000
Then, run the following captures for 10 seconds each:
3 digital channels @ 500 MS/s. (50% max USB throughput)
4 digital channels @ 500 MS/s. (67% max USB throughput)
5 digital channels @ 500 MS/s. (83% max USB throughput)
6 digital channels @ 500 MS/s. (100% max USB throughput)
Let me if those captures complete successfully, or result in an error, and if you still encounter the USB host lockup and the error “ERROR unknown event type 37”
I ran a quick test on a separate Linux machine (21.10) and it worked great!
It still starts with these troubling warnings and errors:
mesa: for the --simplifycfg-sink-common option: may only occur zero or one times!
mesa: for the --global-isel-abort option: may only occur zero or one times!
mesa: for the --amdgpu-atomic-optimizations option: may only occur zero or one times!
[132473:0527/111817.813738:ERROR:sandbox_linux.cc(374)] InitializeSandbox() called with multiple threads in process gpu-process.
It would be nice to see those cleaned up, but bravo for the MUCH improved experience!
I will test with export SALEAE_USB_TRANSFER_LIMIT=1024000 on my other Linux machine (22.04) later this weekend and report back in this thread.