exploring simple main event scenes

[Theo 7]

As I anounced already, I want to explore a simple main control scene. The main goal is to have as few scenes as possible that run continously ('autostart') to improve overview and avoid contradictions and racing conditions. Because it is meant as an exploration I will keep it very simple. At some time I will probably join either the @Sankotronic stream or the @jgab event model. They are far beyond my experience on this platform for this moment. And too complex for me as a beginner also. 

I already presented my time calculation model, based on integers from 0 to 2359, with two very simple functions to work with decimal hours (7+1.5 -> 830) and timesteps 759+5 -> 804.

One thing I learned from the last month or so was "zwave freeze". Causes can be multiple, but one of them certainly may be poor programming practices.
Therefore I want to fire fibaro calls at low speed and log them when needed.

Please login or register to see this code.

 

The deviceCall function can handle most of the commands at a beginners level anyway, so fine for now. A built-in sleep certifies a low pace. It checks testmode on the log funtion. Without testmode no logging.

Edited January 27, 2019 by Theo
typos

[Theo 7]

When you convert a block scene to lua you will see that identical code can be seen at two places. On first principles I want to maintain one version of code at all times. In this case the identical blocks are just what you want to do in the scene: the payload. My main scene has no autotriggers (for now) but we can add them later, based on converted block scenes. The AutoFunc() starts the payload() when the scene starts automatically.

Please login or register to see this code.

Normally the AutoFunc() just waits for 60 seconds, as shown in any converted block scene. As our payload() can and will contain multiple timed fibaro calls which each extend the cycle, when actually executed, we have to reduce the waiting time in AutoFunc(). An added complication is that we only know the time with a seconds resolution which adds an uncertainty of 1 second. So I want to be on the safe side there.

Any comments @jgab or @Sankotronic?

[jgab 1,951]

17 minutes ago, Theo said:

Any comments @jgab or @Sankotronic?

 

Your AutoFunc function tries to create a loop calling the payload every minute (60s). The "compensation" calculation is uneccessary complicated. There is a os.time() function in Lua giving you the current time in seconds since Jan 1st 1970 (epoch time). The idea behind doing a non-drifting loop is to initialise a variable X with the current epoch. Every time in the loop add whatever time delay in seconds to X and wait X-os.time()

Please login or register to see this code.

important here that AFtime is global variable, even though we initialise it the first time in a local scope. You have a problem if payload always takes more than 60s, but as long as the average payload takes less than 60s it will calibrate right.

Be aware, I saw in your previous post that you wanted to add delays to your fibaro:calls. In general that is a bad idea and fibaro:sleep halts everything else in your scene, including setTimeouts...

[Sankotronic 2,086]

I agree with @jgab  and I have to rewrite some of my work because setTimeout and fibaro:sleep really does not like each other. Using them together in one code can also significantly increase CPU usage at least that is what I'm experiencing with some of my code.

 

[Theo 7]

2 hours ago, jgab said:

Be aware, I saw in your previous post that you wanted to add delays to your fibaro:calls. In general that is a bad idea and fibaro:sleep halts everything else in your scene, including setTimeouts...

Thank you for your valuable comment. The fibaro:sleep command appears to have a crappy implementation. Given the zwave freeze discussions I steel feel some need to introduce delays. For some reason sometimes ( once per two weeks or so) one of my Fibaro wallplugs becomes unresponsive.

[jgab 1,951]

I strongly believe that it should be the underlying platform and the z-wave stacks responsibility to pace out the commands. I believe that was the intention of the people specifying the z-wave protocol too.

Assume you do manage to pace them in one of your scenes, you can’t sync across many scenes anyway. On the other hand I haven’t had problem with “freeze” during my 5 years with the system. When I have had issues it has been faulty devices that I then tossed in the bin - not tried to compensate with coding.

Having said that, you could queue up commands and have a separate thread that execute them - but then you have the problem with asynchronicity; knowing when commands have completed.

If there are some devices that needs pacing of commands I would leave that to be done explicitly case by case and not build it into every command. Having your 5 lamps in the room light one by one with a seconds interval when entering the room is a cool effect the first time but annoying in the long run.

[Bodyart 229]

2 hours ago, jgab said:

On the other hand I haven’t had problem with “freeze” during my 5 years with the system.

Hi @jgab,

 

do you think there is a correlation between your (high in comparision to most published code here) level of programming approach and and your non-freezing system?

Woul'd we have much less freezing cases than we have now?

I'm having my HC2 frozen since 4.5.30 .... experienced it about 7 times until now.

I also had to rebuild it once from scratch because the fibaro support coul'd only tell me that i may have unwanted characters in remarks (whatever that may mean)

So as i understand from your postings you're going much deeper to underlaying stuctures and fibaro's implementations so i wouldn't be surprised at all if you were a Master of Computer Science

 

Hope to read much more from you in the future and thank you beeing with us here. Your teachings are most valuable.

 

Thank you.

[jgab 1,951]

1 hour ago, Bodyart said:

Hi @jgab,

 

do you think there is a correlation between your (high in comparision to most published code here) level of programming approach and and your non-freezing system?

Woul'd we have much less freezing cases than we have now?

I don't know. 

 

I do note that many report extreme high CPU loads and RAM etc. I run huge complex scenes (framework with 2000 lines of code) and have never had issues with that. Lua is a really efficient language and the box quite capable, it's not like we are running Win10 on it...

I have made coding errors that I have sworn was the HC2's fault, and I have had coding errors that only showed themselves every other week. Coding is not easy. It's more often my fault than Fibaro's, but I'm as frustrated as anyone when there is a problem on their side and it takes months to fix. I tend to fix my own bugs pretty quick.

 

What I may do different is that I stay away from sleeps and VDs - and most have a lot of VDs (with main loops) and scenes that trigger and then polls for state changes in sleep loops. VDs seems to be based on code that God forgot (why 5.1? and when was there last a new API available for VDs?) , and sleep seems to be a busy wait. I have no good experience of either.

 

What I do instead is that I have long running scenes that only run on threads/timeouts. Long running means that I can do tasks like parsing big json tables when I start up because it only happens once - and not every time a scene gets triggered. Because I don't have a main loop polling every x second but instead many "threads" (and I have a gazillion of them) that execute when they need to it tend to smooth out the CPU load. I used to have a scheduler in GEA style that run my rules every X second and the load was much spikier back then. I do have spikes currently too, but they are further between and I'm almost certain they relate to other task (Lua's garbage-collecting in running scenes and other HC2 house-keeping tasks)

 

I guess that stressing the box can starve some processes like the Z-wave stack, and there could be symptoms of "freezing". I'm also sure that there are devices that are faulty or not to spec and cause network congestion.

 

Anyway, my programming approach don't use sleeps, only run rules when needed, and despite 2000 lines of code is still pretty conservative on resources. Go figure. I recently added support for Hue devices and to get state changes for buttons, switches and lights the bridge needs to be polled and I really dislike it. The Philips Hue team is another developer team that have chosen an approach that doesn't scale. They have a secret web socket API for specific partners but the rest of us are spamming the bridge with status polls.. but I'm digressing...

 

All that said, my programming approach still need coding skills and I still make errors as noted above. Add to that that home automation tasks are highly asynchronous to their nature and thus highly challenging to get right. Just look at the proverbial when-to-turn-off-the-light-in-the-bathroom, and  how family members' behaviours easily poke hole in any code logic. I guess we won't know but for sure the hC2 must have caused some divorces? digressing agin.

 

Another thing that help me run less buggy code is that I have made sure that my framework work the same offline (in Zerobrane studio) as it does on the HC2. I tend to do no development on the HC2 these day. I develop everything offline. I can stress test it by simulating massive amount of triggers. I can run it for a simulated month in a second. I can run predefined test-cases. I can do simulated "inter-scene" communication via fibaro:globals or fibaro:startScene(id, args). So when it works offline in ZeroBrane, it works directly when copied to the HC2. When it doesn't it's not because of the scene logic, but because I forgot some offline support code that can't run on the HC2.

So, when I finally deploy a scene on the HC2 it usually runs pretty ok.

 

I've said it before but I think Fibaro chose a deceptively simple coding model for scenes, that turns out to be complex for beginners wanting to do little more. I'm also surprised that they haven't provided a better development environment - it's almost like they aren't interested in having advanced scenes being developed - just simple scripts for installers to provide basic customisations. I think that will turn out to be a bad decision because the market race will be about who has the smartest gateway - and then they will need all the help they can get.

 

 

[Theo 7]

mrs. Theo complained that this afternoon that the curtains were not closed at the supposed time. That never happened when this was contolled by a separate block scene.

As curtain open/close events are critical to our feeling of safety this is not allowed to happen in a main scene. How to overcome this?

One reason might be that AutoFunc might jump over a minute, i.e. going from 15:55.90 to 15:57.10, currentTime would then jump from 15:55 to 15:57. Just missed a "closing event"...
Therefore two changes today: let AutoFunc() make a smaller setTimeout (in ms) delay, and check for "timejumps" in payload()

 

Please login or register to see this code.

 

[jgab 1,951]

10 hours ago, Theo said:

mrs. Theo complained that this afternoon that the curtains were not closed at the supposed time. That never happened when this was contolled by a separate block scene.

As curtain open/close events are critical to our feeling of safety this is not allowed to happen in a main scene. How to overcome this?

One reason might be that AutoFunc might jump over a minute, i.e. going from 15:55.90 to 15:57.10, currentTime would then jump from 15:55 to 15:57. Just missed a "closing event"...
Therefore two changes today: let AutoFunc() make a smaller setTimeout (in ms) delay, and check for "timejumps" in payload()

 

Please login or register to see this code.

 

 

The way you "compensate" introduces a "random drift". If you wanted intervals of 59s, but I don't think you want that, you should do

Please login or register to see this code.

In fact, I would keep it at 60, because that is not your problem. I suspect that you have some calls, like "deviceCall", that do fibaro:sleep. That halts everything in your scene including setTimeout timers, and could easily push it the over some minute barrier and you drop a minute. Don't mix them. Use only setTimeout. 

Another thing you could do is make sure you don't start the AutoFunc loop xx:yy:59 so you easily spill over to the next minute. Make it start on the next xx:yy:30.

[Theo 7]

3 hours ago, jgab said:

I suspect that you have some calls, like "deviceCall", that do fibaro:sleep.

I already had taken out the fibaro:sleep; deviceCall() now only combines the call and logging. Your suggestion of forcing the loop initially halfway a minute is interesting.
Also I have another idea that would work for any loop cycle time. I will publish that tonight. Thanks for your fruitful comments.

[Theo 7]

On 1/29/2019 at 8:51 AM, jgab said:

Another thing you could do is make sure you don't start the AutoFunc loop xx:yy:59 so you easily spill over to the next minute. Make it start on the next xx:yy:30.

I changed AutoFunc()'s initialisation folowing your suggestions:

Please login or register to see this code.

It rounds AFtime to whole minutes and add 30 seconds. Now the first setTimeout will be longer than a minute, but then it will repeat steadily halfway a minute. As mentioned before: all other sleepy things were removed.

Quote

[DEBUG] 15:05:11: start
[DEBUG] 15:05:11: payload start: 1505
[DEBUG] 15:05:11: time jump: now=1505, previous=0
[DEBUG] 15:06:30: payload start: 1506

 

[jgab 1,951]

38 minutes ago, Theo said:

I changed AutoFunc()'s initialisation folowing your suggestions:

Please login or register to see this code.

It rounds AFtime to whole minutes and add 30 seconds. Now the first setTimeout will be longer than a minute, but then it will repeat steadily halfway a minute. As mentioned before: all other sleepy things were removed.

 

 

Ok.

You don't need to the AFtime calculation every time

Quote

AFtime=AFtime or 60*math.floor(os.time()/60)+30 -- initialize global AFtime first time around to whole minute+30s

 

[jgab 1,951]

A alternative is to give payload the time it should operate on - then you can run on whole minutes

Please login or register to see this code.

send AFtime to payload, os.date takes epoch as option. You will always get a time that is on the minute and never any skip (in reality you may be a second or 2 off, but it will recalibrate, and your time tests are easier). You can check if AFtime falls behind real time too much and recalibrate. Problem is that your rules can miss a time.

[jgab 1,951]

 

So, you seems to be doing some kind of scheduler that run every minute checking if it should carry out some tasks.

An alternative approach is to schedule each task with a setTimeout, and then let the task reschedule itself for the next day. The advantage is that you don't need to constantly check in a loop against many specific times, worrying about accidentally missing a minute. 

Example of that style of programming you can see here

 

or a minimal scheduler that is more "data driven" here

 

 

[Theo 7]

So here is what is a kind of end point for me for a simple main event that is open for experiments. I started by converting a block scene to lua, therefore I stayed close to that structure. I experienced "missed" event times in previous versions. Earlier on in this thread I expermented with somewhat reducing the cycle time to avoid that. Here I have returned to an excat pace (on average) of 1 minute ("CycleMinutes=1). This is used to increment the AFepochSec in AutoFunc(), initialised in the main part.

Now I do not want to depend on an exact time, but an time window. With my integer time format (0 - 2359) this is pretty easy, see TimeWindow(). On the other side, I do want to have it done once per day regardless the size of the time window. For this reason I introduce an event number, "currentEvent". The integer time format can be manipulated with through addTime().

 

Now the whole structure can be described in a few lines:

1) Main starts the AutoFunc()

2) AutoFunc() calls payload() and repeats itself every minute

3) payload() contains a list of wanted events:

 

if CheckEvent(timeWindow(currentTime,{some time number}),{some value}) then...

means that if the current time falls within the window and the currentEvent is lower than the given number than currentEvent is set to the given number and the result of checkEvent() will be true.

 

The only caveat here is that the given numbers should be ordered with the given times:

CheckEvent(TimeWindow(currentTime,15),1)

CheckEvent(TimeWindow(currentTime,730),2)
CheckEvent(TimeWindow(currentTime,1900),3)
CheckEvent(TimeWindow(currentTime,2300),4)

Also once per day you have to reset the currentEvent number ( if (TimeWindow(currentTime,0) and (currentEvent ~= 0)) ). You can also use this to recalculate season dependent times. My curtain opening time varies roughly 90 minutes, while the sunUp time varies much more.

 

This scene also supports a manual start, which can be used for extra reports and analysis.
Apart from that logging can be switched on/off (testmode=true) and there is a teststates variable. If true I can simulate the whole day by setting alternative times and move the curtains just a little bit. [see attachment for that version]

 

The event concept could easily be rewritten in some table but for now I prefer to keep it simple. The scene has been running flawlessly for a week.

 

Please login or register to see this code.

 

Please login or register to see this attachment.

[jgab 1,951]

Congrats! Nothing beats code written by yourself and that you understand.

A bit unfortunate that your self confidence got hurt by some skipped events in the past, and you feel forced to rely on time windows. I think it complicates the code a bit. Time windows can be good for catchup logic but that's anther matter...

Anway, it seems to hum away - I tested it for 2 months in my

Please login or register to see this link.

and it looks ok.

Please login or register to see this spoiler.

 

[Theo 7]

Made another step on my simple approach. Still the essential point is that I do not want to miss any events by using a TimeWindow function. It happened also with a block scene for whatever reason. For now I have brought all my regular ('timed') activities into this main scene. So there is no other scene running automaticalle anymore.

My approach in the in the previous version, passing "events", has some limitations. Now reentry through a TimeWindow is evaded by doing a test on something that is changed in a event: if TimeWindow(someTime) and (somelight == "off") then turnon(somelight); end;

Please login or register to see this code.

 

Edited March 11, 2019 by Theo
in v05 delay could cause missed event, now v07

[Theo 7]

Maybe not new to experienced LUA-guru's but to me it proved a relevant simplification to enable/disable other scenes from my central script.

Please login or register to see this code.

Now that scene (68) itself does not have to depend on times, making its logics much simpler.