HC2 scripting limitations and challenges

[konbaasiang 0]

Hi!

 

Although I am a programmer by profession, I am new to LUA and HC2 scene/VD scripting.

Being new to the platform, I'm noticing several limitations that are working together to make simple tasks extremely complex, and making it difficult to manage and reuse code, in a way that one might not as one gets used to the platform and accepts things as they are. That said, I could also be missing something, or multiple somethings.

 

Here's what I'm trying to do.

 

I have an 8-channel "web relay" connected to the lights on the wall surrounding my house. See the attached picture of a web relay. They're $30 on aliexpress so they offer very good value for the money, so to speak.

 

The lights on the inside of the wall are connected alternately to three circuits, like so: 1-2-3-1-2-3-1-2-3-1-2-3 etc

The lights on the outside of the wall (facing the street) are connected the same way: 4-5-6-4-5-6-4-5-6 ...

The last two remaining ones are connected to other lights but they're easy and can be ignored for the purpose of this discussion.

 

I would like to implement the following modes:

Off

On, meaning 1 of 3 circuits on, alternating daily

Bright - All 3 circuits on. For yard work in the evening, or security.

Chase - this is the fun one, for parties or just to show off. Circuit 1 for 300ms, Circuit 2 for 300ms, Circuit 3 for 300ms, repeat. The lights will chase each other around the property.

 

The Web Relay is controlled by TCP. You open a connection, send a text string. Easy as pie.

So, what's the problem?

 

In a Virtual Device, the main loop only runs once every 3 seconds. This is too slow. Even once per second (like it claims) is too slow.

This is easily solved with setTimeout but setTimeout is not available in a VD.

Okay, fine. We can make the VD trigger a scene to handle the Chase mode. The scene can use setTimeout to reliably get a 300ms timer.

But, in a scene, net.FTcpSocket is again missing, so it cannot send the command to a web relay in the first place!

So then, do we make the scene trigger a different VD button to send the command each time?

I guess we could, in theory, but then this is becoming a hot mess. How would we keep track of all that without making a mess in the HC2 gui, with all these extra VD buttons and scenes we had to create just to work around limitations? Not to mention all the global variables I would need to handle "communication" between the VD and the scenes..

and even if I manage to do all that -- likely hours of work, for flashing some lights ... how would I reuse it for the three circuits on the outside wall?

 

I must be missing something, I think. It cannot possibly be this hard.

 

I would love to hear your suggestions about how to accomplish the task with HC2 scripting.

 

At this point it will be a lot quicker and easier for me to implement this in C++ on another machine and just have a HC2 VD trigger the different modes by HTTP, so I'm going to do that, but I would still like to learn how others would have solved the problem, and perhaps suggest that these limitations are unnecessary and that it could save a lot of development headache if they were simply addressed.

 

As it is, there seem to be just the right amount of limitations in just the right places to make simple things very, very difficult. I have to believe that Fibaro wants to improve their product and make it easier to use and develop for, so here are my suggestions. I'm sure many of these have been requested before, but here goes:

 

• We need "static" variables, local to a script but static between executions, so that a script can keep internal variables between executions without having to result to using global variables. Globals are messy because they are impossible to organize, and any seasoned programmer does their best to avoid them. The only time true global variables are needed is when two different scripts need to share the same value.
• Make setTimeout available in a VD. Alternatively, make the main loop interval customizable. Every 3 seconds is too often for many tasks, and too slow for others. It's difficult to see a reason why this cannot be customizable. VD main loops could have a header to customize this sort of thing, just like Scenes do. No need to add anything in the user interface itself.
  setTimeout would be preferable though, because the main loop likely does not need to run periodically in all modes, so this way it could be left up to the script writer, which would save CPU resources.
• Make net.FTcpSocket available in Scenes. Why is the syntax for HTTP/TCP/UDP different between Scenes and VDs to begin with? From what I've read in other forum post, this has been a problem for years, and thus could have been addressed years ago. Obviously it can't be changed at this point without breaking existing scripts, but one could add the functionality to both Scenes and VDs so that they match, thus maintaining functionality of all existing scripts while making it easier and cleaner to write new ones.

 

Again, I'm sure I'm missing some things, would be glad to receive some pointers. Thank you for reading!

 

Please login or register to see this spoiler.

 

[konbaasiang 0]

Okay, so maybe it isn't quite as bad as I first thought:

 

--[[
%% properties
%% events
%% globals
--]]

 

local counter=0;

local vd=186;
local btn=6;

function myfunc()
 
  counter = counter + 1;
  fibaro:call(vd, "pressButton", btn+(counter % 3))
  setTimeout(myfunc,350);
 
end

 

myfunc();

 

 

Turns out, local variables actually do what I want, without needing a global.

 

Any way to hide the A/B/C buttons from the VD, which are really only there to allow the scene to send TCP commands?

 

Please login or register to see this spoiler.

 

[jgab 2,641]

 

Quote

• We need "static" variables, local to a script but static between executions, so that a script can keep internal variables between executions without having to result to using global variables. Globals are messy because they are impossible to organize, and any seasoned programmer does their best to avoid them. The only time true global variables are needed is when two different scripts need to share the same value.

Yes, globals are messy, and using them as message boxes when

Please login or register to see this link.

.

For scenes, I have moved to a "

Please login or register to see this link.

to have long running scripts that can keep state in local Lua variables - much more fun (and natural) then juggling fibaro globals. It also resembles the more traditional "event loop" programming style that I believe Fibaro should have chosen (or made an option)

 

Quote

• Make setTimeout available in a VD. Alternatively, make the main loop interval customizable. Every 3 seconds is too often for many tasks, and too slow for others. It's difficult to see a reason why this cannot be customizable. VD main loops could have a header to customize this sort of thing, just like Scenes do. No need to add anything in the user interface itself.

setTimeout would be preferable though, because the main loop likely does not need to run periodically in all modes, so this way it could be left up to the script writer, which would save CPU resources.

Yes, they have made some strange design choices...

 

Quote

• Make net.FTcpSocket available in Scenes. Why is the syntax for HTTP/TCP/UDP different between Scenes and VDs to begin with? From what I've read in other forum post, this has been a problem for years, and thus could have been addressed years ago. Obviously it can't be changed at this point without breaking existing scripts, but one could add the functionality to both Scenes and VDs so that they match, thus maintaining functionality of all existing scripts while making it easier and cleaner to write new ones.

Yes, it's a mess. VD's run Lua 5.1 and Scene's Lua 5.2... network functions in one is synchronous, the other is asynchronous. I don't really know how they were thinking but obviously one case was to write device components for integrating external systems/devices - the other case was of course to allow for user GUIs for these external systems as well as for HC2 scenes. I'm not sure it make sense to combine them in the "VD" concept.

Trying to use the VDs as a way to have a common place for interfaces to all kind of external systems I can sympathise with, but I don't think it makes sense as many systems have excellent apps and the shortcomings in what can be expressed in a VD GUI are too many at the moment.

The programming model in VDs is also too light for some more advanced integration as you have stumbled upon. Now I see you solved it by pressing "pre-defined" buttons on the VD. But imagine if you need to pass a parameter (sending through a fibaro global or a VD label) and you have race conditions. The other thing to watch up for is what happens if you press buttons too fast... (or the HC2 takes a pause, which is known to happen...). 

 

I tend to make all my logic in scenes and only use VDs as light GUI components to the scenes (if I really need a GUI). The one thing I really miss is the socket api though. 

 

However, I have integrated my scenes with a

Please login or register to see this link.

running on a pi, allowing me to do more advanced integration stuff. I agree that it is a pity to have to resort to that, but the HC2 - or any other HA system - will not be best at everything (and node-red is kind of fun). However, I agree, the HC2 could easily be a bit more competent.

 

Quote

Again, I'm sure I'm missing some things, would be glad to receive some pointers. Thank you for reading!

 

Working on the HC2 is to understand the shortcomings and find ways around them  

 

 

 

 

 

 

Edited January 6, 2019 by jgab

[konbaasiang 0]

Thanks so much for your reply, jgab. Makes more sense now. I read your entire post about the event model, I am gaining a new appreciation for just how in need of servicing the back-end is.

I wonder, is there any particular reason why Fibaro don't seem to be doing much about it?

 

[Guest spazpeker]

I also use node red, I find the Node Red Alexa better than Fibaro, I can also control my Screenline Blinds (with RFXCOM) via node red as well as Nest and weather Station sensors,

i can also query the heating state of zones via Alexa and turn on holiday mode

 

Please login or register to see this spoiler.

 

[jgab 2,641]

On 1/6/2019 at 10:25 AM, konbaasiang said:

Thanks so much for your reply, jgab. Makes more sense now. I read your entire post about the event model, I am gaining a new appreciation for just how in need of servicing the back-end is.

I wonder, is there any particular reason why Fibaro don't seem to be doing much about it?

 

I'm guessing but my feeling is that their hart is more into devices then the gateway (HomeKit push, CES 2019 etc). Most updates to the gateway is new device support, and when there are bugs in the gateway it usually takes some time to get a fix (time drift, exceptions in scenes, etc). It was years since we saw some really interesting feature added for developers. I seen some projects evolve like this and then they are usually "consultant" driven. Nothing bad about consultants, but they are contracted on a per-needs bases to fix issues in an (old) code base where the original developers (and visionaries) have left. Probably similar thing with the app. ...or maybe its a mix. So yes, they seem to lack some visionary lead architect for the gateway software - someone that wants to make it to a thriving development platform and build a large ecosystem - someone that understands what software developers wants and wants to please them. If they don't come up with features and capabilities that will attract the best software developers, the logic  of homeautomation will move out into other platforms (and the HCX will just only be an expensive z.wave gateway at the end of the day). Attracting the best developers is how you stay relevant in the competition for homecontrol servers.

If their hart (and margins) is in devices I would suggest to open source the framework - they would still need to hire a "Linus" to be the guard and still have "releases", but maybe manage to attract good contributors and a faster evolution of the platform in a direction that pleases software developers....

 

 

[konbaasiang 0]

Than you again, @jgab -- well said, sir!

 

It sounds like I may have gotten onto the Fibaro HC2 bandwagon too late?

 

My initial reason to get into z-wave was that I wanted to be able to measure how much power my entire house was using, and how much of that power was being generated by my solar panels, and how much power my air conditioning is using, in an automated fashion.

I bought DIN-rail power meters from Qubino to accomplish this. Other Z-Wave meters exist, but only Qubino ma kes DIN-rail compatible ones:

 

Please login or register to see this spoiler.

 

 

I bought three of the single phase meters, and one of the three phase meters... and then, joy of joys, turns out they are completely unsupposed by FIbaro HC2. The single phase meters kind-of sort-of work, but the three phase will not configure properly.

 

After reading some forum posts and noticing that others had the same problem with no solution in sight, I also ordered a UZB1 Z-wave USB stick, and a license for the Z-Way software Z-wave controller.

I ordered these as a backup (in case Fibaro never adds support -- i'm already $550 deep in Qubino meters, which is almost the cost of a HC2, so spending another $70 on another solution was a no-brainer).

 

Do you have any experience with Z-way?

 

At the very least I will eventually get that running and pair the Qubino meters with it (keeping two separate Z-Wave networks), but if it turns out to be better, and HC2 continues to feel like a dead end, I may just dump the HC2 completely before I've wasted too much time working around arbitrary and unnecessary limitations.

 

[jgab 2,641]

Hmm, re-reading my post it maybe came through as too doom and gloom.

Supported devices and templates is always an issue - don't know how that is prioritized or why it's seems so difficult. Unfortunately I'm not into the z-wave spec (others in this forum are the experts), but I wonder how it is similar or differs from something like say USB, and devices classes - USB seems to have reasonable generic support for may classes of devices.

Some of the open source platforms seems to be quicker to implement support for devices because they are driven by the priorities of the users themselves. 

Anyway, I think the HC2 is a nice box to run Lua scripts on and has a reasonable z-wave support. I don't have a lot of different devices and they are all fibaro devices (and I integrate towards Hue and node-red) - so it kind of works reasonable me, and I have implemented my own scene framework that I think is fun to develop, and develop on top of.

Other (open source) platforms may have their advantages, but it seems the stability is an issue there too. So, in summary, the market is still immature. I expect the big players, Amazon, Apple, Microsoft to increasingly embrace this - but then z-wave will just be one of many protocols. However, I'm not sure if enthusiast like me will be let in to hack my own scripts and "frameworks"...

 

Edited January 8, 2019 by jgab

[konbaasiang 0]

@jgab, your post was indeed pretty much doom and gloom, but the worst part is, from what I've learned so far, I'm afraid you hit the nail on the head. Just yesterday I explained some of the most obvious limitations of HC2 scripting to a coworker of mine, who is also a programmer but with even more experience than myself (30 years vs 20 years) and his reaction was also pretty much WTF. It indeed seems like it may have been developed by consultants who didn't talk to each other.

The house I just built and moved into is a smaller, temporary house. It's a prototype / experiment in pretty much every way, from building technique (building a house to nordic insulation standards, but in Thailand, and with materials commonly available locally), to ventilation, central air conditioning, and home automation, which i've never ventured into before. I'm making sure to make as many mistakes as possible on this house, so that I don't have to make them when building the main, much bigger house.

My HC2 is pretty much set up to handle the needs of the small house now. Construction of the main house is about to start, but it'll probably be two years until it's move-in ready. So, that means, for me the clock is ticking. If HC2 scripting has been revamped by then, I'll buy another HC2 and link them by LAN, to make it one big z-wave network. If it still has the same limitations, I will have no choice to look for another solution, because the current limitations would make it too labour- and maintenance intensive for a system that large. Let's see if anyone is listening.

Edited January 9, 2019 by konbaasiang

[LordOfFalcons 5]

super interesting conversation gents. just posting so the system will keep me informed if more is said on this. i'm also experiencing some frustrating limitations with HC2 (though i'm new and from what i read not as skilled as you too), but my various internet searches have also not suggested there is anything much better out there that has the flexibility / configurability that Fibaro offers with HC2 and lua... if you find anything. please let me know... 

[jgab 2,641]

12 hours ago, LordOfFalcons said:

super interesting conversation gents. just posting so the system will keep me informed if more is said on this. i'm also experiencing some frustrating limitations with HC2 (though i'm new and from what i read not as skilled as you too), but my various internet searches have also not suggested there is anything much better out there that has the flexibility / configurability that Fibaro offers with HC2 and lua... if you find anything. please let me know... 

I kind of like the HC2 environment. It's pretty vanilla with scripts that get triggered by devices and the possibility to "autostart" a script to run in a loop. On-top of that you can build pretty much anything using basic Lua. 

 

I have built a

Please login or register to see this link.

that runs in a loop *and* get triggered by external events. This allows me to experiment with implementing my own script languages, that suits the task of home automation in a way I like. If I could ask for something it would be that the HC2 framework  had an event queue that could be polled or could have call-backs registered with, similar to Windows or Mac frameworks - it would make my live much easier - but hey, there are ways around it. Maybe also some more Lua features like loadstring, coroutines and some of the "standard" Lua libraries out there (a bit tedious to hand-code parsers for xml or script languages you want to experiment with). Its not difficult to see that some minor tweaks and additions to their framework could make it a really good platform for developers - but they seem to lack interest or vision, or both.

 

The problem with many other frameworks that I have looked at is that they usually have too much "framework" and have made too many decisions for you (OpenHAB, Domoticz, Home Assistant etc). Lots of features but often a too cumbersome script model based on their own invention or some limitations of an existing language. Not many allows for implementing your own script language  

They also have more integration with external systems in general, but since I integrated to node-red, almost everything I need to integrate with is out there. Because many of these other frameworks are open-source they do attract some good programmers and developers - which I can be a bit jealous of. OpenHAB has some 55 contributors and an average of 3-5 commits a week on GitHub. Domoticz have 236 contributors and ~20 commits a week.

 

The "VD model" offered by the HC2 I'm not especially impressed with. Clunky programming model for serious integration towards external systems. An to use it as a GUI to external devices?  Why write a worse interface to a device than what's usually available in the app for that device (i.e. Hue). If you want a single dashboard to monitor the home, a page full of button-and-label limited VDs is so 1990's. My philosophy is that smart homes don't require a lot of buttons to be pressed - smart homes work in the background doing the right things at the right time. That is also a model more compatible with my wife. .

 

Because I have a stateful scene model (something that HC2 should provide for us) and have chosen an event model, it is easy to implement things like a

Please login or register to see this link.

using some 30 lines of Lua. The fantasy is the limit. Looking at the other framework I usually get the feeling that they have been too clever and would limit my options to do things the way I prefer. 

 

To summarise, the HC2 allows me to play with frameworks (systems) concepts (single scenes, event models, pubsub, scene management etc). In fact, I have been starting to treat the HC2 as a micro service platform (couldn't we have kubernetes for scene management  ) with scenes being services that send events to each other. It makes you think different about the system as a whole. The other thing is that it  allows me to experiment with my own home-brewed "home automation" script languages on top of these frameworks. These things keep me occupied 100% of the time I spend with the systems and I'm having a lot of fun... so that's why I'm still sticking with the HC2.

 

If you just want a framework where you can click and configure your home without programming I can see that the HC2 is starting to lag behind other frameworks - but click-and-configure will only get you an automated home, never a smart home  

Edited February 17, 2019 by jgab

[konbaasiang 0]

That is indeed the frustrating part for me.. there isn't very much missing for it to be so much more flexible and easy to use with regards to scripting. It's really only a handful of things. Jgab's framework sounds like something I should look into for the future, because indeed having a slow idle loop that can be triggered at will could be useful for all sorts of things.. but that also begs the question: why doesn't fibaro just build something like it in?

 

Is there any interest from Fibaro's side in making it better? Certainly some of us in the forum could reach a consensus and distill lua-related script feature requests down to a reasonable, concise list. Of course, if there's no interest then there's no sense in taking the time.

 

 

@jgab, is your framework public?

 

[jgab 2,641]

2 hours ago, konbaasiang said:

That is indeed the frustrating part for me.. there isn't very much missing for it to be so much more flexible and easy to use with regards to scripting. It's really only a handful of things. Jgab's framework sounds like something I should look into for the future, because indeed having a slow idle loop that can be triggered at will could be useful for all sorts of things.. but that also begs the question: why doesn't fibaro just build something like it in?

 

Is there any interest from Fibaro's side in making it better? Certainly some of us in the forum could reach a consensus and distill lua-related script feature requests down to a reasonable, concise list. Of course, if there's no interest then there's no sense in taking the time.

 

 

@jgab, is your framework public?

 

 

Sure, it's described and available in this thread 

and there is nothing secret with how it works. I truly believe that the basic "single scene model" is a much better programming model on the HC2 (also because it's possible to debug offline). I have been very open in discussing and documenting the framework - in hope to get feedback if there are deficiencies or improvements - but very few have taken the bait. There are a few and I'm very grateful for their feedback - and some of them even seems to build some pretty advanced scenes on top of the framework.

There are some documentation in

Please login or register to see this link.

and some discussions

Please login or register to see this link.

when we improved the throughput. However, it's a balance act as there is no true locking mechanism for access to fibaro:globals (or any global changeable resource for that matter). Fortunately the model we have now for the message box seems to work quite well in all practical cases. However, all the magic we are forced to now could easily been solved by fibaro with support for an event queue. ... and I would be happy to throw out my framework and focus on my EventScript language  

Please login or register to see this code.

or polling:

Please login or register to see this code.

This may need some more careful programming. Probably a variable sleep interval like in the EventRunner framework. However, if the fibaro:nextSourceTrigger() call takes 5ms and you sleep 250ms, the scene would have a pretty low idle consumption anyway. 

Now this may look like a what people do; a loop polling for devices' state changes and sleeping between. However, this is much more efficient if there are many devices we need to monitor and there is no way to integrate it with 'other' triggers from other scenes as we can't poll for them.

 

So, there would be no need for starting multiple scene instances or have limits of 10 instances. If the scene is running, just give the trigger to the scene, otherwise, start the scene and give the trigger to the scene. New triggers are added to the scene's trigger queue. There could still be the standard fibaro:getSourceTrigger() to be backwards compatible. And while they are at it, make 'other' a trigger with arguments so we can scrap the fibaro:args()... with coroutines and a "peek/remove" for the event/trigger queue we could even implement something that looks synchronous calls to functions in other scenes with return arguments (using startScene and peeking for the argument to come back as a trigger). 

Edited February 17, 2019 by jgab

[jgab 2,641]

A minimal version that supports sourceTrigger callbacks or nextSourceTrigger could look like this: We need to wrap the "scene code" inside a main() function as we need to do some setup before user code can execute. This would of course not be needed if the support was in the HC2 scene framework  and it would be much safer and efficient. However, there is

Please login or register to see this link.

...

Please login or register to see this code.

 

Edited February 17, 2019 by jgab

[jack.daniels 103]

On 2/17/2019 at 12:18 AM, jgab said:

 

The "VD model" offered by the HC2 I'm not especially impressed with.

Thats definitely an epic fail of the Fibaro.

There was a plugin support before and it was very flexible. But Fibaro decided thats too much for usere and left it for internal use only. You can still make a plugin but need to use a trick to install it. On hc2 root password is enough but hcl needs to be reflashed (readonly rootfs)

[Theo 7]

In a lot of topics for VD's is advised to not use them at all. Is that also true for VD's with only button-triggered events, i.e. without main code?