Zniffer capture of a 4-hop route, working, but it takes 110ms

[petergebruers 1,261]

Z-Wave routing is a blessing and a curse... And more of a curse when you get "long routes", which are often defined as 3 and 4 hop routes (that is the longest route possible).

 

I regularly point out to users... When you use a Zniffer to capture routes, it is very likely you won't see the complete conversation.

 

Why is that? Because it stands to reason, the node is far away and the hops are necessary to get there so that would mean wherever you place your Zniffer, it won't be able to hear the complete conversation.

 

Imagine A is in your garden shed and it cannot reach the controller and B is somewhat halfway..

 

A <-------> B <------> Controller & Zniffer

 

Your Zniffer probably cannot "hear" node A because your controller has decided it needs B to repeat the message. I say "probably" because the Zniffer antenna is not the antenna on your controller, and they might not have the same sensitivity. This means, they do not hear identical radio signals. You can have as much as 10 dB difference (that is the RSSI column in Zniffer). Probably even more.

 

Now imagine you move your Zniffer closer to B, you should be able to hear both "A" and the controller and thus observe all packets involved in a message exchange.

 

A <-------> B & Zniffer <------> Controller

 

With longer routes, it becomes less likely to "hear" everything, because supposedly "longer route" means "more distance". But Z-Wave does not really have the concept of distance and has no 3D vision.  

 

Are you with me so far?

 

But... sometimes, under specific circumstances, the routing algorithm makes weird decisions and sometimes nodes that would not need routing at all can decide to take (unreasonably) long routes.

 

While analyzing a capture of user @Tony270570 this caught my eye. The controller tries to set the dimming level of node 15 to 0 = OFF and this is what happens "on air"

 

22:25:00.771    Routed:(1)->15 - 140 - 63 - 37 - (46)    Switch Multilevel Set
22:25:00.782    Routed:(1) - 15->140 - 63 - 37 - (46)    Switch Multilevel Set
22:25:00.795    Routed:(1) - 15 - 140->63 - 37 - (46)    Switch Multilevel Set
22:25:00.807    Routed:(1) - 15 - 140 - 63->37 - (46)    Switch Multilevel Set
22:25:00.816    Routed:(1) - 15 - 140 - 63 - 37->(46)    Switch Multilevel Set
22:25:00.824    Routed:(1) - 15 - 140 - 63 - 37<-(46)    Routed Ack
22:25:00.832    Routed:(1) - 15 - 140 - 63<-37 - (46)    Routed Ack
22:25:00.844    Routed:(1) - 15 - 140<-63 - 37 - (46)    Routed Ack

(missing packet here)
22:25:00.866    Routed:(1)<-15 - 140 - 63 - 37 - (46)    Routed Ack
22:25:00.877    Ack

 

Without much explanation you can probably guess a packet is travelling between node 1 and node 46 and it succeeds. In this case, Zniffer only missed one packet, the routed ACK from node 140 -> 150. This might not be a signal strength issue, Silabs warns that Zniffer sometimes misses a packet.

 

I did not print the RSSI column but all packets are between 44 and 68, lower numbers are better because this is actually a negative number. Zniffer can capture packets with 70-80 dB reliably. So we can conclude that Zniffer was "near all these nodes". This also suggests that the route is unnecessarily long - because Tony told me he is rather close to the controller I'd say it can work without routing. But don't take my word for it, I can prove it

 

19:18:45.515    46    1    Singlecast    Multilevel Sensor Report
19:18:45.525    1    46    Ack    

 

Say "DOH"!

 

BTW that was capture made another day. The dimmer reports its level to the controller.

 

Do we care about this? 
 

If you look at the long route, you'll see it takes about 110 ms to complete. The direct only takes 10-20 ms, that is as fast as ZW300/500 can get. If this is the only command happening, you won't notice much. But many devices report "status" back, that is another 110 ms. And they'll also report power... It all adds up.

 

EDIT

 

I left out many of the details of Z-Wave routing, and I probably does not work the way you think it does.

If you want more in-depth info about Z-Wave routing, I recommend this series written by @robmac

 

Please login or register to see this link.

 

I've been working with him an a few other users for a very long time...

Edited June 9, 2020 by petergebruers

[Momos 101]

5 hours ago, petergebruers said:

Do we care about this? 

 

Yes we care a lot !

 

Very good explanation as always Peter  thank you.

 

Indeed we want to keep zwave traffic as low as possible to have a fast network,  BUT regarding routes there is a big issue for which i have no explanation as to why it happens.

 

It would seem logical that reconfiguring the mesh network of a device would make the device connect to the controller directly if it sees it with good enough signal, but unfortunatelly this is not the case.  As it happens I am in the process of tweaking my network after the migration to HC3.  There are devices well in rage of the controller - same room, but reconfiguring their mesh dozen of times still adds 2-3 hops, sometimes through a device on a different floor ?  I wish there was a way to force them to connect to the controller first.

 

 

 

[Tony270570 92]

Yes thank you Peter as @Momo said, excellent explanations and witing for next one  !   

@Momo did you notice direct connection issues with old zwave 300 serie? I'm asking that because i've replaced few old devices by new ones (500) at the same place they now get the hc3 in direct and for few when far from the hc with one hope, but none with 4 hopes like before. 

 

[Momos 101]

Some are Zwave 300 (like 70% of them) but there still are Zwave 500 devices that chose a longer route instead the obvious shorter one ? 

Soft / hard reconfiguring does not solve it. They just like the longer path ?

[petergebruers 1,261]

If you see long, unreasonable routes, that is probably caused by the device going into "explorer frame routing mode" and learning a long but valid route, then considers this to be the "Last Working Route". If you want to diagnose this kind of issue, you  may need to run a capture for a very long time, to see when that "explorer" search algorithm kicked in, it might have happen long before you have noticed. You can mesh reconfigure the node, that will cause new routing to be tried and it will get a shorter one, but the effect might be temporary, until the next "disturbance" causes the device to learn a new route. It really depends on your network and I cannot tell what "very long" and "temporary" mean in this context.

 

11 hours ago, Momos said:

t would seem logical that reconfiguring the mesh network of a device would make the device connect to the controller directly if it sees it with good enough signal

Correct, partly, a mesh update will make the controller calculate new routes, send 4 candidates to the device and the device will try direct, then the 4 routes, then explorer frames. So the sending node decides what happens, not the controller, it simply handed 4 alternative routes to the device to choose from.

 

I simplified the explanation a bit, if you want full detail I recommend this series about routing written by @robmac

 

Please login or register to see this link.

 

11 hours ago, Momos said:

but unfortunatelly this is not the case.

If "direct" fails, the device will try one of its available route, if that route works, the node tends to "stick" with that route. I don't know when or how or why a node would try "direct" again instead of the last working route. Power cycling might help, sending from controller to device or vice versa might help and "mesh reconfigure" would work but is very impractical because of the extra network load.

 

11 hours ago, Momos said:

I wish there was a way to force them to connect to the controller first.

There is a way to force routing, but it is not very obvious because neither HC2 nor HC3 have direct support for this. You can write an "Application Priority Route" which means a route to be tried first. You need an extra controller (for example, the Silabs UZB3) and the (free) Silabs PC Controller to do this. I would not recommend it though, although both @robmac have tried and used it I would first check if (a) the long routes cause "real trouble" are just a nuisance and (b) check other possible causes

 

BTW I am in contact with @Tony270570 - we have seen weird issues since he migrated to HC3 but we have no conclusion yet, but I'll keep you informed.

 

9 hours ago, Tony270570 said:

i've replaced few old devices by new ones (500) at the same place they now get the hc3 in direct and for few when far from the hc with one hope, but none with 4 hopes like before. 

The ZW500 has better range indeed, I have tested that and I would say the 30% claim is real... I would not jump to conclusions because we haven't explained every possible issue you have seen yet, but if a ZW300 causes constant trouble and is "far away" and money is not an issue then I'd replace the older device. I have a network with plenty of ZW300 devices but I live in a "compact" house.

IMHO the first step is still reduce "chatter" of devices eg lower or disable power reporting.

[Momos 101]

Hehe "money not an issue "... I wish it were true ? I am slowly replacing very old devices with new ones but it is a process that will take a long time...

Coming back to the long routes: the explorer frames must be the culprit here. I'll try to make some very long captures to see when it's happening.

 

Some further inquiries about mesh reconfiguration and when /how it is triggered.

- by doing it from the GUI 

- by doing a full hard reset of the module 

- by doing a soft reset ?! 

- by cutting the power to the device or gateway ?! 

 

Reducing traffic in the network is the way to go.  Does disabling for example the temperature sensor in a Fibaro motion sensor disables all temperature related traffic or one has to adjust every parameter ? I know that disabling the device is enough but a confirmation is welcomed ? 

 

 

 

 

 

 

[Tony270570 92]

11 hours ago, Momos said:

I wish it were true

I wish it were true too ? 

 

14 hours ago, petergebruers said:

I have tested that and I would say the 30% claim is real

Yes! we can really notice the difference, it also take time but i've decided to move all my devices to zw500, still have  bunches of them be they have started disapearing.

 

14 hours ago, petergebruers said:

but we have no conclusion yet, but I'll keep you informed.

Knowing you, they will come. For sure !

Edited June 9, 2020 by Tony270570

[petergebruers 1,261]

Part 2 - Explaining Z-Wave Routing and what "mesh update" does by analysing  a Zniffer capture: