Device "transfer failed" following by "transfer OK"

[AutoFrank 372]

Hi 

I have a good many devices that when they process an action (switch on/off)

I get a Transfer failed

Please login or register to see this image.

/monthly_2017_04/failed.PNG.17dc6d978046edf12731b1389da8181f.PNG" />

 

followed a few seconds later with a transfer OK

 

 

The screenshots above are on device masters (as I was naming them) but the same is seen on device slaves

 

The action gets completed but I don't recall seeing the failed before

It's across a number of device types

 

Is anybody seeing the same or does anybody know what this is happening

 

HC2 running 4.120

 

Thanks

_f

 

Edited April 25, 2017 by AutoFrank

[AutoFrank 372]

@petergebruers, @chaicka, 

Any idea with this one ?

 

 

I also have a few lights (fibaro relays and dimmers) where I can switch them on via the Web UI and the device will switch on, it will indicate transfer failed but when I go to switch them off it won't switch off. Most of these I can switch off locally but one I cannot

 

Edited April 25, 2017 by AutoFrank

[petergebruers 1,265]

@AutoFrank if you find the time, can you run the script I posted here, on one of your trouble-making-devices? I'd like to know what statuses it prints.

 

 

[AutoFrank 372]

26 minutes ago, petergebruers said:

@AutoFrank if you find the time, can you run the script I posted here, on one of your trouble-making-devices? I'd like to know what statuses it prints.

 

 

 

Thanks @petergebruers

 

I ran it on two devices that are exhibiting the behaviour

253 is a fibaro dimmer 1 (pizzaAreaLight)

710 is one side of Fibaro relay (pixxaAreaSpotLight)

 

result as follows..

 

Please login or register to see this code.

thoughts ?

[petergebruers 1,265]

• pizzaAreaLight is bad, but still turns on. The script tells me that the HC2 did not get confirmation of the dimmer after 5 seconds, and that's after several transmission attempts, so the script started telling you "Z-wave transfer failed". From that point on, my script sends a new "turnOn" command every 3 seconds. After a few attempts, and after 16 seconds from the start of the script, the HC2 finally got confirmation. Question: When you ran this script, when did the light turn on?
• pizzaAreaSpotlight puzzles me, because that one doesn't seem to get confirmation within the limit of the script = 45 seconds. That's possible, but then I'd expect it to be marked "dead". Question1: did this light turn on and when, and question 2: is  the global option "Mark nodes as dead" unchecked at the moment?

Possible explanations that come to mind:

• You have a script that sends lots of redundant commands to a device or devices, filling op the Z-Wave queue and causing global slowdown. Disable scripts and put "fibaro:abort()" at the start of VD main loops.
• pizzaAreaSpotlight does not have Z-Wave coverage, the device is dead or was accidentally reset. I'd take it out and bring it closer, or, if possible, take your HC2 closer to the module so you can update the mash around it (don't update the full mesh please, only specific devices). Parameter for "toggle" versus "momentary" switch may be wrong, giving the impression that the switch doesn't work.
• You have a broken (internal) SSD disk. This causes slowdown and weird issues. Do you get web page issues? Or app issues? It might explain why your HC2 somethings hangs and has weird LED combinations lit. This is an issue for Fibaro support.

[AutoFrank 372]

8 minutes ago, petergebruers said:

• pizzaAreaLight is bad, but still turns on. The script tells me that the HC2 did not get confirmation of the dimmer after 5 seconds, and that's after several transmission attempts, so the script started telling you "Z-wave transfer failed". From that point on, my script sends a new "turnOn" command every 3 seconds. After a few attempts, and after 16 seconds from the start of the script, the HC2 finally got confirmation. Question: When you ran this script, when did the light turn on?
• pizzaAreaSpotlight puzzles me, because that one doesn't seem to get confirmation within the limit of the script = 45 seconds. That's possible, but then I'd expect it to be marked "dead". Question1: did this light turn on and when, and question 2: is  the global option "Mark nodes as dead" unchecked at the moment?

Possible explanations that come to mind:

• You have a script that sends lots of redundant commands to a device or devices, filling op the Z-Wave queue and causing global slowdown. Disable scripts and put "fibaro:abort()" at the start of VD main loops.
• pizzaAreaSpotlight does not have Z-Wave coverage, the device is dead or was accidentally reset. I'd take it out and bring it closer, or, if possible, take your HC2 closer to the module so you can update the mash around it (don't update the full mesh please, only specific devices). Parameter for "toggle" versus "momentary" switch may be wrong, giving the impression that the switch doesn't work.
• You have a broken (internal) SSD disk. This causes slowdown and weird issues. Do you get web page issues? Or app issues? It might explain why your HC2 somethings hangs and has weird LED combinations lit. This is an issue for Fibaro support.

 

Thanks @petergebruers

I appreciate the help and insight...

 

I just disabled all the scenes and re ran it remotely and the spotlight was still failing after 45 seconds

I'll try and do the same with the VD's

When I ran the script last evening I didn't observe but I will repeat this evening and note at what point the light turns on

 

Spotlight - Mark if dead was 'No' which surprised me so I've changed that for Yes and will observe

Coverage should be okay - the module in in a small utility cabinet in the garage next to 4 other modules with another 4 within 10 feet.

 

Re the SSD disk possibility I'm not having major issue but I do see it slow from time to time and especially if I have multiple tabs open in a browser showing different scenes or devices. Also if I try and save a lot of changes together I get the sinning blue balls for a while

- Any idea where I could find the discussion on that ?

 

I was trying to think back last evening to when these all these issues started and think it was when I added all the Aeotec MS6 devices. I got a lot of failed inclusions and ended up with all the unwanted / not configured devices. Fibaro Support removed all but 1 device as it had a reference in the zwave chip. 

I wonder are the MS6 devices causing an issue with the zwave queue

 

I am seeing this issue on a number of devices and TBH there is really no pattern, pointing to one or a group of devices.  I think I have an issue impacting the solution system wide.  I don't seem to have a resource issue (RAM, memory and storage are all healthy)

I'm inclined to go with the theory that something is flooding the zwave queue and I assume that may be a device or bad code (I can check the code) 

- Is there anyway to observe directly or indirectly the zwave queue and if it is filling up fast or a long queue all the time.

 

 

_f

 

 

 

 

 

 

[petergebruers 1,265]

6 minutes ago, AutoFrank said:

Spotlight - Mark if dead was 'No' which surprised me so I've changed that for Yes and will observe

 

I think it's best to set it to yes, because if it's really dead, sending data to a dead device causes a lot of traffic. When you run my script, you should see it become "dead".

 

6 minutes ago, AutoFrank said:

Coverage should be okay - the module in in a small utility cabinet in the garage next to 4 other modules with another 4 within 10 feet.

(...)

Try mesh reconfiguration on this device. Maybe this (or a neighbouring device) has been used or installed elsewhere? That could explain why it doesn't have proper routes.

 

6 minutes ago, AutoFrank said:

 

Re the SSD disk possibility I'm not having major issue but I do see it slow from time to time and especially if I have multiple tabs open in a browser showing different scenes or devices. Also if I try and save a lot of changes together I get the sinning blue balls for a while

- Any idea where I could find the discussion on that ?

If it's really bad, the operating system of your HC dumps some info to the console, on the HDMI port. Fibaro support can go through logs to see if read/write errors have occurred.

 

6 minutes ago, AutoFrank said:

 

I was trying to think back last evening to when these all these issues started and think it was when I added all the Aeotec MS6 devices. I got a lot of failed inclusions and ended up with all the unwanted / not configured devices. Fibaro Support removed all but 1 device as it had a reference in the zwave chip. 

I wonder are the MS6 devices causing an issue with the zwave queue

I'm not sure, but one thing that's interesting is that you use a lot of them (10? 16?) on USB power and that means that you can *probably* set the device parameters in such a way that it updates constantly. I can tell you that an RGBW set to input mode can send many updates per second, if you've got the settings wrong (or your input is noisy). In the manual of the MS6 you'll find parameter 40 to 44 are used to limit updates. Quote: "p40 Enable/disable the selective reporting only when measurements reach a certain threshold or percentage set in 41-44 below. This is used to reduce network traffic. (0 = disable, 1 = enable)". Default = 0 = disable.

 

6 minutes ago, AutoFrank said:

I am seeing this issue on a number of devices and TBH there is really no pattern, pointing to one or a group of devices.  I think I have an issue impacting the solution system wide.

I don't seem to have a resource issue (RAM, memory and storage are all healthy)

I'm inclined to go with the theory that something is flooding the zwave queue and I assume that may be a device or bad code (I can check the code) 

- Is there anyway to observe directly or indirectly the zwave queue and if it is filling up fast or a long queue all the time.

 

As far as I can tell, only Fibaro support has access to the queues and the logs.

I use a "software defined radio" = SDR to literally "listen to the network traffic". It doesn't decode anything, but it gives an idea of what's going on. At the moment I have no issues. The only odd thing I observed is that one of my D/W sensor chatters a lot when it changes status. I mean, it chatters more then when I open another door, with a seemingly identical sensor. I haven't found an explanation yet...

 

Poor man's queue test? I sometimes run a script, that is nothing but a for loop (to limit the number of possible hung items in the queue) that turns on a device for 2 seconds, then turns it off for 2 seconds. It should do that at a fairly regular pace.

[AutoFrank 372]

1 hour ago, petergebruers said:

 

I think it's best to set it to yes, because if it's really dead, sending data to a dead device causes a lot of traffic. When you run my script, you should see it become "dead".

 

Try mesh reconfiguration on this device. Maybe this (or a neighbouring device) has been used or installed elsewhere? That could explain why it doesn't have proper routes.

 

If it's really bad, the operating system of your HC dumps some info to the console, on the HDMI port. Fibaro support can go through logs to see if read/write errors have occurred.

 

I'm not sure, but one thing that's interesting is that you use a lot of them (10? 16?) on USB power and that means that you can *probably* set the device parameters in such a way that it updates constantly. I can tell you that an RGBW set to input mode can send many updates per second, if you've got the settings wrong (or your input is noisy). In the manual of the MS6 you'll find parameter 40 to 44 are used to limit updates. Quote: "p40 Enable/disable the selective reporting only when measurements reach a certain threshold or percentage set in 41-44 below. This is used to reduce network traffic. (0 = disable, 1 = enable)". Default = 0 = disable.

 

 

As far as I can tell, only Fibaro support has access to the queues and the logs.

I use a "software defined radio" = SDR to literally "listen to the network traffic". It doesn't decode anything, but it gives an idea of what's going on. At the moment I have no issues. The only odd thing I observed is that one of my D/W sensor chatters a lot when it changes status. I mean, it chatters more then when I open another door, with a seemingly identical sensor. I haven't found an explanation yet...

 

Poor man's queue test? I sometimes run a script, that is nothing but a for loop (to limit the number of possible hung items in the queue) that turns on a device for 2 seconds, then turns it off for 2 seconds. It should do that at a fairly regular pace.

 

Thanks @petergebruers

Lots of food for thought as usual 

Yes - I have 14 of the MS6, all running on usb power. I'll take a look at the parameters you reference

 

edit 

There are certainly some I can tweak but I found this which seems odd

 

Please login or register to see this attachment.

 

 

I'll create the test scene see if I can get any insight from that.

Re the SDR - Is this the type of device your talking about  (

Please login or register to see this link.

 ) did you buy or make yours, I may be interested in going after one of those

 

_f

 

Edited April 26, 2017 by AutoFrank

[AutoFrank 372]

@petergebruers

 

I did a but more checking and noticed that many of my fibaro devices were not enabled for "Mark if dead" so I went trough them all and enabled them. I am now seeing a few more dead devices.

 

I ran your script and seem to get a different result each time

Is this what you'd expect 

 

Here is the result from three lights in the kitchen (repeated twice)

Please login or register to see this code.

 

and this is the result from multiple lights 

Some strange ones here - utility light was dead and then OK. I was watching the device in the WebUI and the message toggled between no communication, transfer failed, transfer OK

 

Please login or register to see this code.

TBH, I'm not sure what to make of these results.

 

[jimicr 89]

Maybe ask support to check your HC2?.... I don't think this is something you can fix yourself

[AutoFrank 372]

3 minutes ago, jimicr said:

Maybe ask support to check your HC2?.... I don't think this is something you can fix yourself

 

@jimicr

Do you think it looks that bad ?

[jimicr 89]

Just a gut feeling, at least ask them look at the logs we cant see.

And who knows what happend during your recovery/ restore session. 

 

Hope my feeling is wrong and you can pinpoint a offending device or aeotec MS6.

And maybe I'm seeing things a bit gloomy after spending over an hour cleaning after our 3 year old who got very sick all over her sisters room

[AutoFrank 372]

1 minute ago, jimicr said:

Just a gut feeling, at least ask them look at the logs we cant see.

And who knows what happend during your recovery/ restore session. 

 

Hope my feeling is wrong and you can pinpoint a offending device or aeotec MS6.

And maybe I'm seeing things a bit gloomy after spending over an hour cleaning after our 3 year old who got very sick all over her sisters room

 

@jimicr

I definitely got the better job between those 2 ... but been there and done that and it's not pleasant

hope the little one is okay

i'm trying to avoid getting support because my HC2 is in a different subnet to internet connection and I have to patch things over and then open the ports so I have to be at home when they connect

 

I was thinking of doing a recovery and a restore just to see if it dislodges anything

something is definitely not right and at this stage I don't think I have anything to loose

[jimicr 89]

1 minute ago, AutoFrank said:

 

@jimicr

I definitely got the better job between those 2 ... but been there and done that and it's not pleasant

hope the little one is okay

i'm trying to avoid getting support because my HC2 is in a different subnet to internet connection and I have to patch things over and then open the ports so I have to be at home when they connect

 

I was thinking of doing a recovery and a restore just to see if it dislodges anything

something is definitely not right and at this stage I don't think I have anything to loose

Thanks,

 

Maybe try with out recovery first, just restore a backup.. Who knows it's as simple as "a bit falling over" during last restore

 

[pos 22]

On 25/04/2017 at 4:08 PM, AutoFrank said:

Hi 

I have a good many devices that when they process an action (switch on/off)

I get a Transfer failed

Please login or register to see this link.

 

followed a few seconds later with a transfer OK

 

Please login or register to see this link.

 

The screenshots above are on device masters (as I was naming them) but the same is seen on device slaves

 

The action gets completed but I don't recall seeing the failed before

It's across a number of device types

 

Is anybody seeing the same or does anybody know what this is happening

 

HC2 running 4.120

 

Thanks

_f

 

 

 

This is a bug. I have logged that to Fibaro running 4.110. It is stil there on 4.120. The Fibaro bug id is #75656
" Bug report -- AES encryption multi sensor <-> HC2" . It was logged feb 2 2017. They have taken it to backline.

 

If you have the same problem, it it related to AES. If you re-bind your device with no encryption and it works, you have the same...

 

Peo
 

[AutoFrank 372]

Just now, jimicr said:

Thanks,

 

Maybe try with out recovery first, just restore a backup.. Who knows it's as simple as "a bit falling over" during last restore

 

true .. well here goes nothing

 

[jimicr 89]

1 minute ago, pos said:

 

 

This is a bug. I have logged that to Fibaro running 4.110. It is stil there on 4.120. The Fibaro bug id is #75656
" Bug report -- AES encryption multi sensor <-> HC2" . It was logged feb 2 2017. They have taken it to backline.

 

If you have the same problem, it it related to AES. If you re-bind your device with no encryption and it works, you have the same...

 

Peo
 

@pos please keep us posted on this one...

 

1 minute ago, AutoFrank said:

true .. well here goes nothing

 

@AutoFrank good luck!

[pos 22]

The logged info to Fibaro on the bug id 75656 I mention above started when I thought it was a mutlisensor problem. But later on it turned out that it was related to almost everything. I can see it on my popp thermostats, multisenors etc... 

 

This is what was logged on #75656

 

--snip--

Hi

 

Bug report

 

I think there is a AES encryption bug in the Fibaro multi sensors 5gen ver 3.2 or in the HC2 4.110.

 

It started with that I saw “transfer failed” sometimes. when I saw that in the HC2 GUI it was for all slaves on that unit at the same time. This occurs between 1 of 20 to 1 out of 30 motion triggers. It seems it happens with a non successful transmission for Basic Set “0” OFF from the sensor.

 

 

What I have done to pin point this an come to this conclusion follows…

 

• I have removed the multisensor from the network and set it to factory reset (success checked with a faded red light). After that I have included it again with a distance from HC2 of 1m. “Still sometimes “transfer failed”…
• After factory reset the “blind time” is 2 and “alarm cancel delay” is 5 whereas the PDF manual  FGMS-001-EN-T-v2.0.PDF says default is “blind time”  3 and “alarm cancel delay”  30. The sensor default is not in accordance with what the PDF says. And at every reset these values goes back to 2 and 5. “Still sometimes transfer failed”…
• Changing to blind time”  3 and “alarm cancel delay”  30 in accordance to what the PDF says is default. Makes sure the unit receives the parameters. Still sometimes “transfer failed”…
• Did a range test that shows solid green. If I got it right it shows a direct and stable communication with the HC2 unit. Still sometimes “transfer failed”…
• Looking deeper into it and can state that this sometimes happens on A L L my Fibaro multi sensors no matter where they are places or what parameters they have. Not often, but in the rate as mentioned at the top of this post. I therefor can conclude that it is NOT a single broken Fibaro multi sensor. So Still sometimes “transfer failed” on ALL multi sensors. 

 

 

Now…. I did as the first bullet again and removed the unit from the network, did a reset and included it again but did NOT check “Add in security mode if device supports it”. NOW IT WORKS !No errors at all ever. I have checked this with two others sensors as well. This is for sure a bug that can easily be reproduced. I have also verified this with a guy running another installation. He also get "transfer failed" at the same trigger rate as me. Probably not a correctly encrypted frame sent from the sensor or problem to decode it at the HC2 end.

 

Best Regards

Per-Olov Sjöholm

Sweden

--snip--

 

/Peo

[AutoFrank 372]

8 minutes ago, pos said:

 

 

This is a bug. I have logged that to Fibaro running 4.110. It is stil there on 4.120. The Fibaro bug id is #75656
" Bug report -- AES encryption multi sensor <-> HC2" . It was logged feb 2 2017. They have taken it to backline.

 

If you have the same problem, it it related to AES. If you re-bind your device with no encryption and it works, you have the same...

 

Peo
 

 

Thanks @pos

 

Is this what you mean by encryption ?  (3rd checkbox)

Please login or register to see this attachment.

 

I'm 99% sure I haven't included any device with encryption

Do you of any way I can check ?

 

Does one bad device impact others, I have 6 or 7 with this issue.

 

Eddit - I just read your post

 

I have one fibaro multi sensor

Are you suggesting this would be causing all my issues ?.

 

Edited April 26, 2017 by AutoFrank

[pos 22]

@AutoFrank Don´t know... Did not get if you have permanent issues or intermittent issues. The bug I logged is confirmed by me plus  a Zwave online store owner. But as I wrote, it is happening in 1:20 to 1:30 of the times, i.e intermittent. But.... I found a picture of your home system, and it seems you have a lot more VDs and stuff than me . So m a y b e it can be worse on your system than on mine. Therefor a good guess is that, yes it could be the very same bug. But of course we can never be sure...

 

Peo

[AutoFrank 372]

Just now, pos said:

@AutoFrank Don´t know... Did not get if you have permanent issues or intermittent issues. The bug I logged is confirmed by me plus  a Zwave online store owner. But as I wrote, it is happening in 1:20 to 1:30 of the times, i.e intermittent. But.... I found a picture of your home system, and it seems you have a lot more VDs and stuff than me . So m a y b e it can be worse on your system than on mine. Therefor a good guess is that, yes it could be the very same bug. But of course we can never be sure...

 

Peo

 

Thanks @pos

my fibaro multi sensor was fw 2.7

I tried to remove it but it didn't remove fully.. so I've started the recovery process as a last ditch effort before I ask Fibaro to dial in and take a look