HC3 Crashes

[Sjakie 156]

Hi all,

My HC3 crashes sometimes.

Processors goes  from 50-100%,  System slow.

It happens 3 times now after creating a  new back up HC3 starts services and goes in repair mode and comes online with an older version.

Fibaro blames my 3rd party devices for me this is a joke! but my problem stays.

All those 3rd party devices where running without any problem on my HC2! I installed Aeotec, doorsenor and pir Quibino, curtain relais and Coolcam a few wcd plugs all other is Fibaroe relais, wcd plugs and 20 HUE lights

I have no clue how to detect where what is going wrong only exclude 3rd part devices and add them one by one.

Half of my devices are in the HC3 the other half sits running in HC2.

Any one a good idea?

//Sjakie

 

[Bodyart 231]

Maybe try older backup and observe... Most crashes and slow downs are caused by programing is my experience...

[petergebruers 1,268]

My opinion as an end user... There is a "hidden" command to gather diagnostic info and we are not supposed to know but in rare cases I think it is justified to try it 

 

When it happens again, asap do this:

 

<your ip>/api/service/logsDump

 

So in my case that is:

 

http://192.168.0.51/api/service/logsDump

 

Wait several minutes... If your HC3 still works (I mean if it hasn't locked up completely), you'll get an encrypted file called "LogsDump-HC3-......tar.gz". It contains a ton of information and you can then send that file to [email protected] for analysis.

 

There is a way to decrypt this file but I am not sure if you would gain anything by that because, like I said, it has "a ton of information"  but also "red herrings" and I think it is best to let Fibaro have a look at it.

[tinman 1,070]

remove hue (for now), if you have any secured included devices exclude them and include without security, remove all "refreshstate", "all-in-one", mqtt things, remove all other plugins - so basically naked HC3 with incesure included zwave devs

[10der 832]

16 minutes ago, tinman said:

remove hue (for now), if you have any secured included devices exclude them and include without security, remove all "refreshstate", "all-in-one", mqtt things, remove all other plugins - so basically naked HC3 with incesure included zwave devs

@Sjakie also you may do not "power on" box at all. HC3 unallotted for it.

 am I correct   @tinman,?

 

PS: I'm so tired...

[tinman 1,070]

12 minutes ago, 10der said:

do not "power on" box at all. PS: I'm so tired...

 

for some ppl here in forum might be better to not use any system at all. 

 

19 minutes ago, 10der said:

do not "power on" box at all. PS: I'm so tired...

 

if your brand new car is running hot, after you added new wheels, one need to remove them first to get stable and proven situation, to be able to run diagnostics and compare results to known system. The statement you made shows once again that you have no idea about complex system and their diagnostics. @Sjakie asked for help, not for stupid comment.

[10der 832]

@tinman another question 

why my rpi4 with

20 ZigBee devices 

ZigBee dongle 

node-red

node-red UI 

grafana + influx 

nginx server 

mailserver postmail / imap / smtp

Has 40-60% (max) CPU
and last question 

why @Sjakie what spend 600USD for HC3 should has naked devices? Or we are who bought HC3 like Pinocchio - «the fool himself» what bought this device

Edited September 3, 2020 by 10der

[tinman 1,070]

15 minutes ago, 10der said:

@tinman another question 

 

no, stop spaming and see/read below

 

15 minutes ago, 10der said:

Or we are who bought HC3 like Pinocchio - «the fool himself» what bought this device

 

if you don't have any idea, how to help @Sjakie please don't post, it is really useless for users who might look for help.

 

24 minutes ago, 10der said:

Pinocchio

 

if you wish discuss pinocchio, node-red, and compare system loads (i have 800+ devices on my HC3 with under 70% cpu load, any better here?) please create separate topic in off-topic subforum.

[Sjakie 156]

Guys,

I like some humor too in this stressfull time.

No device is added in secure mode all just add click yes and go

No plugins

Only Event Runner and that cant be the issue because Jan knows what he is doing.

Hue is my main thing in the house.

I have been back wards now 4 times and that should be enough

Wierd is its running for a few hours excellent and suddenly  terible.

Another observation because Fibaro did remote assistance.

I logged in also remotely  because when trying to login I gor waithing for available web socket

Remotely HC3 reacted much faster as local so that I dont understand either should be reverse in my opinion.

//Sjakie

[Sjakie 156]

Guys,

33 devices

30 Hue

 

Ram 79%

CPU 1-50% has been much higher

If I see CPU usage high I do the same as in HC2 restore the latest version.

Need to add another 50 devices

//Sjakie

[Sjakie 156]

Hi Guys,

HC3 System message Ram 85% 

[tinman 1,070]

79% ram is high, i don't have such high load with 800 devices, so something is leaking memory.

 

9 minutes ago, Sjakie said:

Only Event Runner and that cant be the issue because Jan knows what he is doing.

 

Jan is not almighty God (Jesus was, remember his "termination"?), so there is always chance something eats resources ... maybe in combination with something else.

 

9 minutes ago, Sjakie said:

Hue is my main thing in the house.

 

honestly, i noticed issues with hue immediatelly, have only two of them (3rd part models on old bridge) - which should be never ever a problem, but "one can feel how the HC3 slowed down", with all GA versions. As latest alpha runs with no speed gap after hue plugin installed, i can only assume something got optimized.

                                        

 

9 minutes ago, Sjakie said:

I logged in also remotely  because when trying to login I gor waithing for available web socket

Remotely HC3 reacted much faster as local so that I dont understand either should be reverse in my opinion.

 

that one is always "bad omen".

6 minutes ago, Sjakie said:

Hi Guys,

HC3 System message Ram 85% 

 

ohh even worse .. take that log now. A quick look on that could at least shows what process is eating memory

 

[akatar 208]

@tinman to be honest, with my hc3 all cores jumps to 90%

With the same amount of devices on my hc2 it was about 20%, something must be wrong with this firmware (or hc3 is just a raspberry pi for a high price)

Ram is at 60%, also higher then on my old hc2

 

 

Please login or register to see this attachment.

Edited September 3, 2020 by akatar

[akatar 208]

and my second hc3, 19 devices and ram at 54%

[tinman 1,070]

great, and now customer screenshot: 151 devices, ~50 scenes, - before HC2, slow as hell, crashing from time to time, now HC3, with bit load, everything fast (actually perfect time to take screenshot, as around midnight some larger scenes are running there):

 

Please login or register to see this spoiler.

 

Generally i observed following things with 5.040.37

- ipad with old app -> bad idea, blocking (scene and energy panel) resources, unstable system
 

- alarm is sometimes wired, same scene with activated zone crashing one HC3, while on another one HC3 works. Scene re--created, everything fine. I got that multiple times, block scenes, so hard to get copy/paste trap (bug). In principle backup before one start to create alarm system  just to be prepared, if works, fine, if not, the data is there.
 

- securely included devices -> bad idea, blocking HC3 up to 20mins. Not every device seems to be problematic, while danalock runs, keyfob blocked me multiple times. 
 

- migrations from HC2 to HC3, without soft-reconfigured devices -> all kind of routing issues, slow HC3, HCServer crash, zwave stack crash (i can see in installer app that). 
 

- hue plugins -> not really crashing something, but one can "feel" that HC3 react slower (at least BUI). After hours of blocked packages, no discovery, re-connections of bridge, plugin stopped to respond, blocking BUI a bit more. QA, doing lot of more communication with bridge, was the solution for me (i like anywaqy QAs much more than plugins, i wish to have the control). However, the latest HC3 alpha firmware seems to use better hue plugin, it react fast, so i will test once again with GA once available (while i have no issues with Hue QA, some customers just don't like them, becuase it's lua and dark matter ...).

 

 

 

[jgab 2,645]

7 hours ago, tinman said:

79% ram is high, i don't have such high load with 800 devices, so something is leaking memory.

Jan is not almighty God (Jesus was, remember his "termination"?), so there is always chance something eats resources ... maybe in combination with something else.

For sure - my stuff has bugs and crashes like anyone else (you should know that by now @Sjakie  ) and the HC3 is unchartered territory  still.

Yes, my EventRunner4 QA is a big honking QA that runs an /refreshStates loop. It does use some memory proportional to number of rules but doesn't build up any "log" data structures that should grow infinitely... at least running it in my simulator gives no indication of that. On the HC3 it could of course be "in combination with something else". 

 

It would be nice if collectgarabage() would come back so that we could keep an eye on how our QAs/scenes use memory.

 

 

Edited September 4, 2020 by jgab

[10der 832]

@jgab Jan IMHO memory is not a root cause issue here but CPU usage.

[Sjakie 156]

Hi Guys,

Unfortuanly if I try to do a logfile as Peter adviced the HC3 freezes, mmhh after 3 minutes it shows to download

So after one week fighting to get something usefull out off this HC3 (first floor and toilet is in HC3) WAF is below zero.

I will put it back to HC2 and see what to do with HC3

//Sjakie

 

[petergebruers 1,268]

3 hours ago, Sjakie said:

Unfortuanly if I try to do a logfile as Peter adviced the HC3 freezes, mmhh after 3 minutes it shows to download

Yes, it takes "unusually long" to create that  log dump, it is probably some timeout on the HC3. I forgot to report this slowdown... Will do so now.

 

Let's hope you get some useful feedback!

 

3 hours ago, Sjakie said:

So after one week fighting to get something usefull out off this HC3 (first floor and toilet is in HC3) WAF is below zero.

I will put it back to HC2 and see what to do with HC3

I understand your frustration.

[Sjakie 156]

Thanks Peter!

Yeah I noticed it that it will take a while but unfortuanaly i couldt take the log while CPU was very high

Frustration is a kind off  soft expression.

//Sjakie