Automating long recordings? Using multiprocessing for the saves?

[bu5]

Hello!

I’ve got a Saleae Logic Pro 8 I would like to acquire multiple hours of data from after a digital trigger.

I did find mention in the documentation about being able to automate long recordings in Python, but no real information other than this short snippet:

The software already has exceptionally deep buffer capabilities. However, there are still cases where longer recording would be preferred, for hours or even days. Since the software can't be used to record continuously for that length of time, the long capture must be broken into a series of shorter captures that are saved to disk. That results in small delays between captures that will result in lost data; however, in most cases, the save and capture restart time is well under 1 second. This operation usually only needs to be performed once an hour.

For this operation, you can either use the existing sample code or create your own application from scratch. The basic process is to use one command over and over again. That command is "CAPUTRE_TO_FILE". See the documentation for more details. Once the capture has completed and the file has been saved, the software will reply over the socket "ACK". Then the software is ready to receive a new capture to file command.

However, I could not find the existing sample code it mentioned, nor any other mention of that command anywhere else other than that single spot.

The issue I’m trying to solve right now is when the file is saving, the acquisition stops far longer than 1 second. Splitting the capture files into multiple saves with a tiny pause in between resuming capture would be fine. But the current wait misses a ton of data.

I was considering moving the save/export steps to a separate process, but trying to move to multiple processes has proved difficult. I’m having trouble pulling info from the capture session to the save process. Trying to redefine things in the save process gives me an error “Cannot use Manager after it has been closed”

Has anyone done anything like this, or have advice for me as I try to?

Here’s what I’ve got right now:

This code will read the trigger, record for 5 seconds, but isn’t pulling the file to analyze. I’m probably doing something wrong with the queue, but thought I’d include this in case it would be helpful.

This code is meant to start the recording in one process, then save it in a second process. However, I’m getting that “Cannot use Manager…” error message when I try it this way.

Any help would be appreciated, thanks!

[timreyes]

Sorry about that! The documentation seemed to refer to our older (no longer supported) Socket API interface for our legacy Logic 1.x software.

I’ve updated the support article below:

Automating Long Overnight Captures

@bu5

The issue I’m trying to solve right now is when the file is saving, the acquisition stops far longer than 1 second. Splitting the capture files into multiple saves with a tiny pause in between resuming capture would be fine. But the current wait misses a ton of data.

Hm… we’ll get this on our backlog to review your Python script and will run some tests on our end. I’ll keep you updated on our findings and recommendations.

[markgarrison]

Hi @bu5,

Good news, we did test that this can be done with Logic 2. We recommend using the python main thread to perform the captures, and use threads to take care of saving and closing finished captures. Here is an example I tested a while ago which allows you to export, save, and close captures in the background while the next capture is running. Let me know if you have any trouble with it!

from saleae import automation
import os
import os.path
from datetime import datetime
import threading

# Connect to the running Logic 2 Application on port `10430`.
# Alternatively you can use automation.Manager.launch() to launch a new Logic 2 process - see
# the API documentation for more details.
# Using the `with` statement will automatically call manager.close() when exiting the scope. If you
# want to use `automation.Manager` outside of a `with` block, you will need to call `manager.close()` manually.
with automation.Manager.connect(port=10430) as manager:

    # Configure the capturing device to record on digital channels 0, 1, 2, and 3,
    # with a sampling rate of 10 MSa/s, and a logic level of 3.3V.
    # The settings chosen here will depend on your device's capabilities and what
    # you can configure in the Logic 2 UI.
    device_configuration = automation.LogicDeviceConfiguration(
        enabled_digital_channels=[0, 1, 2, 3],
        digital_sample_rate=10_000_000,
        digital_threshold_volts=3.3,
    )

    # Record 5 seconds of data before stopping the capture
    capture_configuration = automation.CaptureConfiguration(
        capture_mode=automation.TimedCaptureMode(duration_seconds=0.5)
    )

    # Start a capture - the capture will be automatically closed when leaving the `with` block
    # Note: We are using serial number 'F4241' here, which is the serial number for
    #       the Logic Pro 16 demo device. You can remove the device_id and the first physical
    #       device found will be used, or you can use your device's serial number.
    #       See the "Finding the Serial Number of a Device" section for information on finding your
    #       device's serial number.

    threads = []
    for i in range(5):
        print(f'starting capture {i}...')

        capture = manager.start_capture(
            device_id='F4241',
            device_configuration=device_configuration,
            capture_configuration=capture_configuration)

        # Wait until the capture has finished
        # This will take about 5 seconds because we are using a timed capture mode
        capture.wait()

        def _worker(cap):
            # Store output in a timestamped directory
            output_dir = os.path.join(
                os.getcwd(), f'output-{datetime.now().strftime("%Y-%m-%d_%H-%M-%S")}-{i}')
            os.makedirs(output_dir)

            # Export raw digital data to a CSV file
            cap.export_raw_data_csv(
                directory=output_dir, digital_channels=[0, 1, 2, 3])

            # Finally, save the capture to a file
            capture_filepath = os.path.join(output_dir, 'example_capture.sal')
            cap.save_capture(filepath=capture_filepath)

            cap.close()
            pass

        thread = threading.Thread(target=_worker, args=(capture, ))
        threads.append(thread)
        thread.start()
    for th in threads:
        th.join()

[saleae]

I have tested this and there is still a gap of about 600ms between the captures.
Is there a way to make the gap smaller or even make it no gap ?

Benno

[markgarrison]

Unfortunately no, there is no way to completely eliminate the gap between recordings.

What total duration are you trying to record? Could you tell me more about your application? I might be able to suggest alternative solutions.

[saleae]

Depending the problem it could bu up for a week or so at sample rate of at least 5Msamples/sec preferred 10Msamples/sec analog. Usually it is the analog monitoring of CAN_H and CAN_L on ha 500kbps CAN bus.
I have a seperate program reading the saved data back and analyse the samples, substract CAN_L and CAN_H so you have a nice differential signal and decode the CAN, throw it away if there is noting interesting in the data.

[BitBob]

Is your issue actually in the analog data, that you need to capture it at higher resolution? I’m guessing that the higher the resolution of data, the more latency you might have between captures?

Here’s a post with a python script to hack your analog filtering to reduce sample rate w/o losing as much bandwidth:

Saleae Logic2 Analog filtering technical details?

I’ll give that a try … Kind regards, Sebastian

Finally, if you can probe the RX pin vs. CANH/L (digital only), you might be able to capture a lot more in one capture w/o needing to split up the sessions, assuming the issue doesn’t need to see the analog data and is a bit-level thing.

[saleae]

See the image on the problem that we had on that occasion.
It took a long time to get this capture.
Sometimes the CAN_L part was gone (later discovered it was a bad contact due constant torsion of the can connectors)

This is impossible to detect with digital only.
We now face a similar problem where some device is putting dominant state on the bus for 10 bits or so, not always but sometimes.

To trigger on this you need to do Realtime math on the data and then also Realtime decoding of the calculated results.

[BitBob]

Hi @saleae –

From your example, I understand why you’d like to capture the analog data, too. However, have you tried triggering on the error frame pulse on a digital channel?

Assumptions:

• Using a typical 5V CAN bus electrical interface
  (i.e., the bus pins are biased to 2.5V at idle)
• Using a Logic Pro 8 or 16 hardware to capture the CAN bus
  Note: Logic 8 hardware should also work, or any Logic hardware will work if you’re on the digital side (RX pin) of a CAN transceiver vs. on the analog side of the CAN bus (CAN_L pin)
• CAN bus is running within normal operational timing tolerances
  (or else adjust the Pulse width range value(s) accordingly)
• CAN bus has at least two active nodes on the network, and these nodes are capable of generating CAN error frames when any bus error is detected
• The issue you want to capture results in an error frame on the CAN bus, due to any of the following CAN errors:
  • CAN bit error
  • CAN stuff error (bit stuffing error)
  • CAN CRC error
  • CAN form error
  • CAN acknowledgement error (ACK error)

Setup:

• Digital channel capture: use an RX pin on micro/transceiver (preferred), or capture the CAN_L pin on the CAN bus
  (for more detailed explanation see the support article on decoding differential and high-voltage data)
• Digital threshold (if using CAN_L pin or 3.3+ V digital I/O RX pin): use 3.3+ volts setting (actual threshold limit is 1.65 V on Logic Pro)
  (see support article about supported voltage thresholds for more details)
• Use the Trigger Mode setting to capture the data
• Select Low Pulse for the Pattern and Channel (using digital channel on RX or CAN_L pin)
  
  
• Select an appropriate pulse width range based on the bit-rate setting and a CAN error frame
  (i.e., a minimum of 6 consecutive active bits)
  • min pulse width: (6 x <1 bit-period>) - (~10% of <1 bit period>) (margin for node(s) running a little faster)
  • max pulse width: 20(+) x <1 bit-period> (arbitrarily large upper-bound margin)
    Note: looking for a minimum pulse width to trigger, so the max is N/A – should pick something generously large, but not ‘too large’ if the CAN bus is going off/asleep within the capture)
  • Example settings (CAN bit-rate of 500 kb/s):
    • bit-period: 1 / = 1 / (500 kb/s) = 2 us
    • min pulse width: (6 x 2 us) - (0.1 x 2 us) = 12 us - 0.2 us = 11.8 us]
    • max pulse width: 20 x 2 us = 40 us
      
      
• Set an appropriate Capture duration after trigger value
  (if you want more data captured after the error frame)
• Set a large enough Memory buffer size value
  (depending on how many other channels you are capturing, the Analog capture rate setting and how much CAN data/edges are on the bus)

When any bus error occurs and at least one node detects & generates an error-frame, you should trigger and store as much information as needed (based on the Memory buffer size and Capture duration after trigger settings set above). Thus, as long as the CAN bus errors/failures aren’t occurring more frequently than the time between capture saves/restarts (i.e., > ~600 ms apart, per your info above?), you could hopefully capture all of the failure events without missing anything important?

Likewise, your comment:

@saleae

We now face a similar problem where some device is putting dominant state on the bus for 10 bits or so, not always but sometimes.

… could also be captured using the above setup?

Specifically, I think a “dominant state on the bus for 10 bits or so” might be an error-frame caused by one (or more) of the CAN errors listed above.

To troubleshoot this in more detail, you might want to observe the CAN TX pins (and possibly RX pins) of all nodes on the CAN bus to isolate which node(s) are generating the error flag(s) on the bus (observable as a low pulse >= 6 bits on the TX pin of the node(s) generating the active error flag).

Likewise, you could use the Saleae CAN protocol analyzer (if using “Classical” CAN bus protocol), or else install a 3rd party analyzer (if using CAN FD bus protocol) since Saleae doesn’t have native CAN FD analyzer support yet. Finally, I would also suggest adding the CAN protocol analyzer during post-capture analysis to avoid any extra overhead or latencies during data capture, as it isn’t required for triggering/storing the initial captures.

[Edit/Follow-up:]
Assuming that access to the internal RX/TX pins might be more difficult to access or capture during testing, you might be able to infer enough information with just the CAN bus monitoring and the CAN protocol analyzer – since the trigger of the ‘error active flag’ (error-frame pulse) should get you to the exact offending CAN frame, and (hopefully) the error occurs sometime after the CAN ID / SOF is transmitted, so you can infer which node was transmitting (assuming each CAN ID is transmitted by only one node on the network).

However, certain CAN errors might require access to the TX/RX pins of the node causing the error, since the error may not be visible on the CAN bus side of the transceiver (e.g., a CAN bit error due to transmitting node not seeing a TX dominant bit on the RX pin within the expected delay time). Other errors may have to do with other low-level timings of a given node that may only be seen on the digital side of that node’s transceiver and knowing the CAN peripheral bus clock speed and/or CAN register settings of that specific node (some of which might be inferred by monitoring only the CAN bus, but not as robustly or easily as using the internal digital signals directly, if available).