Converting an LLA to HLA - hints/tricks?

[KurtE]

Sorry if a lot of this has been asked before, and I will continue to search, but thought I would ask a few questions anyway…

Decided to try to better learn Python and see about converting my Dynamixel servo LLA into an HLA using the the Serial LLA, which makes a lot of sense as I often wanted to see the actual bytes as well as the actual packet information. Besides it is one way to get a better understanding of Python. I am faking my way through it with the help of google.

While I am doing this, I am still fumbling along with a few current questions. Some sort of Python related, some your HLA classes and some how to do similar stuff to LLA.

Questions like:

If I have a something like ChoicesSetting

    ChooseServoTypes = ChoicesSetting(
        label='Protocol 1 Servo Type',
        choices=('AX Servos (default)', 'MX Servos', 'XL 320 Servos', 'X Servos')
    )

In order to find the users settings, Do I have to do something like:

        if self.ChooseServoTypes == 'AX Servos (default)':
            self.prot1TableIndex = 0
        elif self.ChooseServoTypes == 'MX Servos':
            self.prot1TableIndex = 1
...

Or is there a way to ask that object for the current selection by index? So have code like:

In my c++ analyzer I keep a state table for which byte I am processing next, and I then setup an enum {} with symbolic names for each state. Something like:

	enum DECODE_STEP
	{
	  DE_HEADER1,
	  DE_HEADER2,
	  DE_ID,
	  DE_LENGTH,
	  DE_INSTRUCTION,
	  DE_DATA,
	  DE_CHECKSUM
	};
...

		switch ( DecodeIndex )
		{
			case DE_HEADER1:
				if ( current_byte == 0xFF )
				{
'''

What is a reasonable way to do similar in Python? Right now I simply am doing

        if self.frame_state == 0: # find first 0xff
            if ch == b'\xff':
                self.frame_start_time = frame.start_time
                self.frame_state = 1 
        elif self.frame_state == 1:  # find second 0xff
            if ch == b'\xff':
                self.frame_second_time = frame.start_time
                self.frame_state = 2
            else:
                self.frame_state = 0 # not a packet                
...

There is probably a cleaner way?

Then questions like: With LLA, I would generate multiple result strings for the same packet.

For example for a Read operation there was code like:

	else if ((packet_type == DynamixelAnalyzer::READ) && (Packet_length == 4))
	{
		AddResultString("R");
		AddResultString("READ");
		AddResultString("RD(", id_str, ")");
		ss << "RD(" << id_str << ") REG: " << reg_start_str;
		AddResultString(ss.str().c_str());
		
		if ((pregister_name = GetServoRegisterName(servo_id, reg_start)))
		{
			ss << "(" << pregister_name << ")";
			AddResultString(ss.str().c_str());
		}

		ss << " LEN:" << reg_count_str;
		AddResultString(ss.str().c_str());

		Package_Handled = true;
	}

And then depending on how big the packet display area is, you would choose the largest one that would fit…

Is there something like this for the HLA?

HLA Table output, Is there a psuedo standard for HLAs? Example I am no where near done yet, but I broke out the write commands from the others so far: So I have some data like:

I have the write and dynamixel split out in the result types:

    result_types = {
        'simple': {
            'format': '{{data.cmd}} ID:{{data.id}}'
        },
        'write': {
            'format': '{{data.cmd}} ID:{{data.id}} REG:{{data.reg}} = {{data.data}}'
        },
        'dynamixel': {
            'format': 'ID:{{data.id}} {{data.cmd}} {{data.data}}'
        }
    }

Is this a preferred way? or is it better to have most/all of one HLA all show up under the same “type” in the table?

That is all for now, for sure will probably have other questions. Again sorry if this is a repeat of some other thread.

[P.Jaquiery]

My go to languages are C++ and Perl and I dabble in Python - mostly for writing HLAs. In each of the three languages I use associative arrays a lot. In Python they are called dictionaries and look like:

    # Build a dispatch table
    self.dispatch = {
        "FE":     self.DecodeFE,
        "EEProm": self.DecodeEEProm,
        "POD":    self.DecodePOD,
        }
    
    # Test to see if an entry is in the table
    if device in self.dispatch:
        # Call the handler
        result = self.dispatch[self.device](params)

This example uses a dictionary to dispatch to a handler function to deal with a particular device type having first checked that a handler exists. You can use integers and strings as keys (strictly speaking, any immutable value) and anything you like as values.

Dictionaries are the go to tool for handling what we’d often use a switch statement for in C++, using dispatch code as above. They are also useful for mapping strings to indexes - your user choice scenario.

[markgarrison]

I’m actually pretty new to python myself, but the other two members of our software team are pretty experienced with it.

For your first question, you can see what I did in the text messages HLA here:

saleae/hla-text-messages/blob/master/text_messages.py#L41

Where I used a dictionary to hold both the string option to display (the key) and the actual delimiter character to use (the value).

The new HLA’s don’t support generating multiple strings per frame like the LLA system did. The HLA frame display strings are very, very limited at the moment, not even supporting different display radixes. This is something we want to dramatically improve, given that the handlebars template system we’re using is quite powerful.

You can however change the format strings based on the settings in the init function, as demonstrated in the text_messages HLA.

HLA Table output, Is there a psuedo standard for HLAs?
Is this a preferred way?

We don’t have a standard, in fact with the last HLAs we wrote, we tried to explore a few different directions for different parts of the HLA.

or is it better to have most/all of one HLA all show up under the same “type” in the table?
No, I recommend using as many unique types as make sense for your data. More specifically, I suggest having a unique type for each “schema” of keys. I’d really like to add optional schemas to the produced frames in a future version, so that we can dramatically improve data table display.

For example, our USB PD analyzer produces 26 different frame types.

The sidebar data table is pretty limited right now, but we have big plans for that. Ideally you could pop it out into a new window, and have some powerful organization, search, and filter features to find what you’re looking for.

For your last message, you can do the same thing in python. What you described can be seen here in python:

Python function pointer

And it sounds like a good idea to me - basically maintain a state, and call the correct function based on the current state.

There are a few alternatives to writing what’s basically a general purpose state machine.

1. You could buffer input frames until you are sure you have a complete packet, then you can decode that packet in one shot.

e.g.:

  def __init__(self):
    self.frame_buffer = []

  def decode(self, frame):
    self.frame_buffer.append(frame)
    if(self.have_complete_packet()):
      return self.process_packet()

  def have_complete_packet(self):
    # check self.frame_buffer to see if we have a complete packet.
    return True

  def process_packet(self):
    return [] # you can return None, a single FrameV2 instance, or an array of FrameV2 instances.

In our USB PD HLA, I tested out using coroutines to make packet decoding look synchronous. I couldn’t find a way to make it look simple and clean, but the end result basically looked like this:

byte1 = yield from self.get_byte()
byte2 = yield from self.get_byte()
byte3 = yield from self.get_byte()
byte4 = yield from self.get_byte()

However it was my first experience with Python’s coroutines, which was quite the challenge. However, basically my decoder became a suspendible function. the decode(self, frame) function would then pass 1 frame into the decoder each time it was called.

[KurtE]

Thanks @markgarrison @P.Jaquiery :

Looks like a few tricks I should be trying out soon, like setup the choices to have it then linked to other data structure…

So far the stuff is sort of brute force, like using states of 0, 1, 2, …

But at least parts of it are looking like they are starting to work. Wish there was more advanced bubbles…

The data tables:

The terminal mode is showing data like:

Settings: AX Servos (default) X Servos (default) yes Unknown (default)
DXL  Read  ID: 2  Reg: MODE#  Cnt: 0x2
  DXL Reply ID: 2  Err:   Data:  1060
DXL  Read  ID: 2  Reg: PPOS  Cnt: 0x4
  DXL Reply ID: 2  Err:   Data:  2075
DXL  Read  ID: 8  Reg: MODE#  Cnt: 0x2
  DXL Reply ID: 8  Err:   Data:  1060
DXL  Read  ID: 8  Reg: PPOS  Cnt: 0x4
  DXL Reply ID: 8  Err:   Data:  2077
DXL  Read  ID: 14  Reg: MODE#  Cnt: 0x2
  DXL Reply ID: 14  Err:   Data:  1060
DXL  Read  ID: 14  Reg: PPOS  Cnt: 0x4
  DXL Reply ID: 14  Err:   Data:  1962
DXL  Read  ID: 2  Reg: PPOS  Cnt: 0x4
  DXL Reply ID: 2  Err:   Data:  2075
DXL  Read  ID: 7  Reg: MODE#  Cnt: 0x2
  DXL Reply ID: 7  Err:   Data:  1060
DXL  Read  ID: 7  Reg: PPOS  Cnt: 0x4
...
DXL  SWrite  ID: 254  Reg: GOAL  Cnt: 0x4
        Data:   0x8: 2076 0xe: 1963 0x2: 2074 0x7: 1954 0xd: 2061 0x13: 1952 0xa: 2780 0x10: 2769 0x4: 2782 0x9: 1359 0xf: 1310 0x3: 1296 0xc: 1427 0x12: 1437 0x6: 1418 0xb: 2615 0x11: 2676 0x5: 2696
DXL  SWrite  ID: 254  Reg: GOAL  Cnt: 0x4
        Data:   0x8: 2075 0xe: 1965 0x2: 2073 0x7: 1956 0xd: 2061 0x13: 1954 0xa: 2786 0x10: 2775 0x4: 2788 0x9: 1352 0xf: 1304 0x3: 1290 0xc: 1419 0x12: 1429 0x6: 1411 0xb: 2624 0x11: 2684 0x5: 2703
DXL  SWrite  ID: 254  Reg: GOAL  Cnt: 0x4
        Data:   0x8: 2075 0xe: 1967 0x2: 2073 0x7: 1958 0xd: 2060 0x13: 1956 0xa: 2792 0x10: 2781 0x4: 2794 0x9: 1345 0xf: 1298 0x3: 1284 0xc: 1411 0x12: 1421 0x6: 1403 0xb: 2633 0x11: 2691 0x5: 2710

The Syncwrite data ones I probably want to break up into multiple lines of text…

Likewise bubbles:

Still lots of work in robustness and the like, had issue where I saw a little blip in data which caused a 0 to read as 0x40 and code was expecting 0… So need to check and handle cases like that, plus probably check for delta time between bytes. If gap > X assume bad packet set state back to look for new packet.

Also right now the code is not trying to validate the checksum (protocol 1) or CRC protocol 2…

Code is a mess, but if anyone wishes to take peek. It is up at: GitHub - KurtE/SaleaeDynamixelAnalyzer_HLA: Saleae High Level Analyzer for Robotis Dynamixel Servos

Next up pull out Hexapod with AX servos and try it with protocol 1… Was experimenting with XL430-W250 servos (Protocol 2) hexapod.

Now back to playing.

And again thanks for hints. And more hints are always appreciated.

[P.Jaquiery]

Just had a quick look at your code. I’d be inclined to use a dispatch table for the state machine, so something like:

    def frame_start(self, frame: AnalyzerFrame)
        self.frame_start_time = frame.start_time
        self.frame_state = 'sync'

    def frame_sync(self, frame: AnalyzerFrame)
        if ch == b'\xff':
            self.frame_second_time = frame.start_time
            self.frame_state = 'id'
        else:
            self.frame_state = 'start' # not a packet

    def __init__(self):
        ...
        self.dispatch = {
            'start': self.frame_start,
            'sync':  self.frame_sync,
            'id':    self.frame_id,
            ...
            }
        self.frame_state = 'start'
    ...

    def decode(self, frame: AnalyzerFrame):
        ...
        if self.frame_start in self.dispatch:
            return self.dispatch[self.frame_state](frame, AnalyzerFrame)
        else:
            return "Bad state: " + self.frame_state

[KurtE]

Thanks,

Will try that out soon. I also believe I need to do similar to the generation of the resultant packets
May also break that part up again for Protocol1 and Protocol2 as enough casing in the code.

As you probably noticed sort of written quick and dirty, learning things…

[P.Jaquiery]

Learning things is good.

Yes, I’d use a similar dispatch technique in generate_result_frame to handle cmd.

An interesting option for Protocol1 and Protocol2 is to handle them using sub states:

        self.dispatch = {
            'start': self.frame_start,
            'sync':  self.frame_sync,
            'id':    self.frame_id,
            'prot1': {
                'inst': self.prot1_inst,
                'data': self.prot1_data,
                },
            'prot2': {
                'id':   self.prot2_id,
                'len1': self.prot2_len1,
                }
            }

    def decode(self):
        if not self.state in self.dispatch:
            return 'Bad state ' + self.state
        
        if 'dict' != type(self.dispatch[self.state]):
            return self.dispatch[self.state]()
        
        if not self.substate in self.dispatch[self.state]:
            return 'Bad substate ' + self.state + ':' + self.substate
        
        return self.dispatch[self.state][self.substate]()

The values in a Dictionary don’t have to be of the same type so we can mix dispatch entries and dispatch tables then use type() to figure out what we got. The slight advantage is we can then see the relationship between the protocol handlers in the dispatch table, but the substates live in their own name space so there is no need to bend over backwards to avoid name collisions for states.

[KurtE]

Thanks,

I did a quick and dirty conversion of the decode to dispatch functions. Appears to be running again,
will play some next with doing the processing of the packets.

[KurtE]

I have changed over the processing of the packets and generating both most of the protocol 1 and Protocol 2 messages.

Also added in checksum computations for protocol 1, or CRC on protocol 2. And do prints if they don’t match and add a checksum/crc column in table…

The next thing I was trying to do, is to detect if I think things went bad. That is if I believe I am in some parsing state and the gap between receiving characters exceeds some thread hold, then assume packet went bad and reset back to looking for first 0xff…

But first attempts at this don’t compile. I thought maybe N times the time to receive a character gap would signal this.

Problem is that I keep getting errors on the subtraction of times in code like:

        # lets add in a timeout if there is too much of a gap between characters
        if self.frame_state != '1stFF':
            char_time = frame.end_time - frame.start_time
            char_gap = frame.start_time - self.last_char_end_time

            if char_gap > (self.packet_timeout_char_count * char_time):
                print("$$ packet timeout")
                self.frame_state = '1stFF'
        self.last_char_end_time = frame.end_time  

It errors on the char_gap line

Do I need to declare self.last_char_end_time as some type? Or is there some better way to check
the timing?

Thanks

[KurtE]

Found another posting, solved like:

        if self.frame_state != '1stFF':
            char_time = float(frame.end_time - frame.start_time)
            char_gap = float(frame.start_time - self.last_char_end_time)
            if char_gap > (self.packet_timeout_char_count * char_time):
                print("$$ packet timeout")
                self.frame_state = '1stFF'
        self.last_char_end_time = frame.end_time  

[P.Jaquiery]

@KurtE

packet_timeout_char_count

See https://support.saleae.com/extensions/api-documentation. GraphTimeDelta is an object. .end_time and .start_time are GraphTime objects and subtracting them generates a GraphTimeDelta object (char_time and char_gap). GraphTimeDelta objects can be turned into seconds as floats using:

char_time = float(frame.end_time - frame.start_time)

Update: sigh, didn’t scroll down far enough

[KurtE]

Thanks,

Took me a bit to find a message that described casting it to float…

Code is working reasonably right now. Could use more cleanup, like reducing duplicate code, by creating simple functions…

I may soon put it out as a version 0.0.1 ore the like to see if others using Robotis Servos are interested.

Also a few more experiments to see things I can or can not do… (Or if I like the results).

Again mostly talking to myself, which is not unusual

The Dynamixel have a command Fast Sync Read, where you can query several servos for N Register bytes starting at some starting register… I have that decoded currently looks like:

In this case I am asking for register 0x84 and I am asking for 4 bytes which I know for the XL servos is Present Position and that this value is 4 bytes…

Now it comes back with one composite packet that I decode that looks like:

Now I remember the starting index and count from the request. And as such I can walk through
the data. This composite data has for each of the servos:
An error field (first one I believe is stored in the normal error position of packet.
followed by: servoID, bytes for this servo, and 2byte CRC data…

Currently I return one packet for this whole thing where data is encoded:
like : 19:2051 (if there is an error) it will be here as well…

But I am wondering if it would make sense to have the HLA return multiple new frames for this one actual Servo packet? That is one for initial part of packet, followed logically with packet per servo…

Questions to test, can one call to decode return a collection of packets?
Can the timings of these overlap?

I do show some of them split off in the terminal view.

DXL  SRead  ID: 254  Reg: PPOS  Cnt: 0x4 IDs: 0x13 0x3 0x5 0x2 0x4 0x6
  DXL Reply ID: 19  Data:  1534
  DXL Reply ID: 3  Data:  1544
  DXL Reply ID: 5  Data:  1539
  DXL Reply ID: 2  Data:  1534
  DXL Reply ID: 4  Data:  1529
  DXL Reply ID: 6  Data:  1539
DXL  FSRead  ID: 254  Reg: PPOS  Cnt: 0x4 IDs: 0x13 0x3 0x5 0x2 0x4 0x6
DXL  Reply  ID: 254  Reg:
        ID: 19  Data:   1535 
        ID: 3  Data:   1544 
        ID: 5  Data:   1539 
        ID: 2  Data:   1534 
        ID: 4  Data:   1529 
        ID: 6  Data:   1539 

Now back to experimenting

[przemek.klosowski]

Sorry, I am late to the game and I see that you are well ahead, but for your specific question

Do I have to do something like:
        choices=('AX Servos (default)', 'MX Servos', 'XL 320 Servos', 'X Servos')
        if self.ChooseServoTypes == 'AX Servos (default)':
            self.prot1TableIndex = 0

You can simply get the index (assign it to a variable, or simply use it to dispatch code).

choices.index('X Servos')

[markgarrison]

@KurtE

But I am wondering if it would make sense to have the HLA return multiple new frames for this one actual Servo packet? That is one for initial part of packet, followed logically with packet per servo…

Questions to test, can one call to decode return a collection of packets?
Can the timings of these overlap?

Yes, the decode function can return no frames, one frame, or an array of frames!

Please don’t return overlapping or out-of-order frames. The frames have to be in order because all collection access assumes that the frames are in time order.

We designed it with non-overlapping frames in mind, I’m not sure what will happen if they overlap, but the front-end bubble rendering will have some problems.

[KurtE]

Thanks both of you,

Right now, I am sort of doing brute force and also a lot more often then I need to …

    def UpdateServoNamesTable(self): 
        if self.frame_protocol == 1:
            if self.ChooseServoTypes1 == 'MX Servos':
               self.ServoNameTable = self.s_mx_register_names
            elif self.ChooseServoTypes1 == 'XL320 Servos':
                self.ServoNameTable = self.s_xl320_register_names
            else:
                self.ServoNameTable = self.s_ax_register_names
        else:
            if self.ChooseServoTypes2 == 'MX Servos':
                self.ServoNameTable = self.s_mx_register_names
            elif self.ChooseServoTypes2 == 'XL320 Servos':
                self.ServoNameTable = s_xl320_register_names
            else:
                self.ServoNameTable = self.s_x_register_names

Will play with it again soon, and maybe split into two parts… Protocol1 and Protocol2, and only set the table once at startup time, and then have functions to do the name translation and size of registers, maybe specific to which protocol.

As for returning multiple frames from the decode. Will need to experiment on where to get the timings from. i.e. do I keep the start/stop times for each character packet that came in or is there a way from decode to ask for those timings.

that is suppose I ask for the FastSyncRead for 4 bytes of data (Current position) for all 18 servos of my hexapod. The response packet will be about 10+8*18 bytes in it like about 154 bytes.
Will still have to experiment to see if I looks right to create 19 frames (one for the main message and one for each of the servo responses. Again not sure if I will like it better or not.

[P.Jaquiery]

As it happens, right now I’m writing a general regular expression match HLA to match what amounts to the bubble text from LLAs. That has the same problem of figuring out the time span of the matched text. My plan is to write a StrBlockBuffer class that stores blocks of text as passed into the HLA in frames with their start and stop times, then allow matching on the concatenation of the text in the blocks. After a successful match I’ll use the start and end match character indexes to find the block and position in the block for the start and end points and then calculate the start and end times.

I’m just at the start of this so nothing to show yet, but watch a market place near you for a new Match HLA “real soon now”.

[KurtE]

Thanks for the hints.

Was playing today with saving a simple object per data packet:

class dataTimeRecord(object):
    "Stores data and time info"
    def __init__(self, data, start_time, end_time):
        self.data = data
        self.start_time = start_time
        self.end_time = end_time

So far I have just hacked it into the Protocol 2, and only use it with the Fast Sync Read:

I am still playing, but it is showing some promise, still not sure how I want to handle the first part of it:

[P.Jaquiery]

Depending on what the LLA does self.data can end up with just one byte/char per object which can get pretty cumbersome! I’ve uploaded my new HLA to the marketplace - turns out to be called Re Search. The extension contains a StrBlockBuffer class that may have some helpful ideas in it. See GitHub - GrandFatherADI/ReSearch: Saleae HLA for highlighting matched text from LLAs using a regular expression match (or install it in Logic and find the source).