HC3 coding - first impressions

[jgab 2,009]

I have been playing around with the HC3 and tried to program some scenes/devices. This is my experience so far.

Disclaimer1: My review is based on the latest beta firmware, v5.020.xx, and there may be improvements coming before the ink in this post has dried.

disclaimer2: My personal preferences when it comes to API design and programming styles shines through in my review… others may have other preferences.

 

Scenes

They have changed quite a lot, some Lua functions and some fibaro* functions have disappeared -

Please login or register to see this link.

 

So scenes have a new way of declaring triggers. No more the Lua comment section in the top of the scene. Instead a separate section of the scene that contains a "Lua table" expression that checks various events/states. If the total expression is true, the action part of the scene is executed. Technically, the scene structure that you get from api.get("/scenes/<SceneID>").content returns a struct with two parts; a conditions part with the condition structure and an actions part with the Lua code that makes up the action part.

Is this an improvement? I don’t know - seasoned Lua programmers could make these tests in the Lua code previously anyway and without having been able to dig too deep into this there is usually some interesting trade-offs and subtle sources of errors/unintended consequences when doing these kind of trigger expressions when it comes to stateless and stateful events….

 

Is it easier for non-Lua programmers? You judge, here is a condition for a device and a time:

Please login or register to see this code.

(block scenes make use of the conditions support - but I'm not reviewing the block scenes as the implementation seem to be a bit unfinished in the current release)

 

Because scenes miss net.HTTPClient(), fibaro.getSourceTrigger(), fibaro.sleep(), os.time(), os.date(), clearTimeout() I haven’t played around so much with scenes, as some of my standard scenes were not portable without major rework.

Edit: The above functions are now part of scenes. Note that fibaro:getSourceTrigger() is now a variable, 'sourceTrigger'.

 

Noted though that they changed from fibaro:* to fibaro.* and some names have changed, like  fibaro:getGlobalValue has changed to fibaro.getGlobalVariable etc. deviceID can be a list, so fibaro.call({66,77},"turnOff") works.

 

Devices

This is a new animal and has received  most of my attention.

 

Devices you code yourself are called QuickApps.

 

When you create a new QuickApp device, you can choose what type of device it should be; binary switch, binary sensor, multilevel switch, temperature sensor etc. …and it will be treated as a “native” device of that type.

You can call fibaro.call(<myDeviceID>,”turnOn”) on a QuickApp device of type binary switch etc.

 

However, you are limited to the available types - more on that later.

 

To code a QuickApp device you now have QuickApp “class” that provides a some functions that makes up the Devices interface.

Please login or register to see this code.

Above is a simple binary switch you can turn on/off. :onInit() is called when the device is saved or restarted - good place to setup stuff or start loops (with setTimeout)

 

You extend the QuickApp class with methods being the actions that the device should accept. Here ‘turnOn’ and ’turnOff’.

 

It turns out that you can define

Please login or register to see this code.

and then from another scene/device call fibaro.call(<deviceID>,”doWhatever”,5,10)

and your function gets called. This is seriously cool as it means that we have a nice way to pass arguments between devices and from scenes to devices - something we didn’t have before.

The other cool thing is that the function is called in the same thread as is started with :onInit()…. This requires some effort from the design perspective (not like in the old scenes when just another instance was created).

This also means that

Please login or register to see this code.

will count up x every time fibaro.call(<deviceID>,”incX”) is called. We have stateful devices...

 

So, ‘QuickApp’ is a “class” that we can extends with some methods to handle actions on devices.

 

There are also possibilities to define handlers for the QDs UI elements (button etc).

If you have a button named ‘button1’ you can define

Please login or register to see this code.

and it will be called when the button is pressed (similar for sliders)

 

There maybe other QuickApp:* methods too but I haven’t discovered any more yet.

 

Then there is the self:* methods…

 

Inside the QuickApp:functions() there is 'self', a variable available with our QuickApps methods.

‘self’ is not the same as ‘QuickApp’ and should not be confused (however in some prototype based object implementations for Lua, the “class” and the “self” is the same object, so it can be confusing). QuickApp is the class, and 'self' is a variable holding our instance of that class.

 

I have found some useful methods.

Please login or register to see this code.

Note here that the UI that can be defined have the same buttons in sets of 1 - 5 buttons, slider, and static text - doesn’t seem to be any enhancements here. It may be extendable but I guess it needs to be supported in the mobile apps too.

 

Another improvement is that Devices have their own “global variables”. In practice a property table associated with the Device structure. We have functions like:

Please login or register to see this code.

 

We can do setVariable/getVariable for variables belonging to our device. They survive restarts and they are also saved when we download and distribute with a QuickApp VD file. In that way we can see them as a generalisation of the IP: and Port fields in old VDs - Here we can define them ourselves and use them as initialisation parameters for QDs.

 

There may be more self:* functions but I haven’t seen them so far.

 

Besides this we have almost the same Lua and fibaro functions as we had in scenes in the past. No synchronous HTTP functions anymore - only the asynchronous net.HTTPClient().

 

The api.get(“/refreshState?<lastID>”) api seems a little more extensive - it seems to report more types of events - like global variables changing states.

 

And there is a new concept <CustomEvents>.

They are of type {name = <string>, userDescription=<string>}

They can be defined with api.post("/customEvents",{name=name,userDescription=descr})

and “emitted” with api.post("/customEvents/"..name)

I think there is a fibaro.emitCustomEvent(<name>) function available too.

 

Scenes can probably trigger on them and they show up in api.get(“/refreshState?<lastID>”)

 

Because events are “statically” defined - name and user description - you have to be a bit creative with naming schemas to use them as general message events - but it’s doable. I will come back on this topic and why you would like to have that... I have played with them to implement a kind of broadcast function...

 

Summary

So, all-in-all, QDs opens up for some new interesting programming paradigms.

- What I miss is a more extensive network library to easier integrate with external systems/devices. Nothing much have happened with the net.HTTPClient() library in 7 years…

- I would have liked  more UI elements and layout options than the buttons and sliders we have had for some years now.

- I would have liked Lua metatables, loadstring, and coroutines too - it’s 2020 and the security issues should be solvable (there are so many other more low hanging security risks anyway)

- Possibility to enable remote Lua debugging - using

Please login or register to see this link.

?

- Now, some more advanced programming has to be realised as QDs, and thus need to have device types like binary switch. For some code like schedulers or other helper scenes it doesn't make sense to squeeze it into a device model - they are not really devices. At least an 'other' device type would be nice to have.

 

This was a first impression - some good news and some stuff that could Improve.

Some small issues I have been wrestling with:

- There are some glitches in the code editors and sometimes the code for a device just stops working and the only solutions is to remove the device definition and create a new.

- Hit and miss what line an error is reported on...

- Some mistakes just hangs/give no visible errors - Ex. by mistake calling QuickApp:debug("FOO")

- api.get(“/refreshState?<lastID>”) seems to hang for ~30s if there is no new event available, instead of immediately returning an empty event/change table.

- net.HTTPClient() seems to have an issue with ‘POST’ command and json payloads in many cases as neither the success or the error handler seems to get called (no problem with GET commands) Turned out that I needed to add a 'Connection: keep-alive' header when doing a POST to get the result back - never needed that in the past...

 

However, I expect these to be fixed over time… and people start coding will report in more bugs...

 

Disclaimer 3: I may have missed stuff that are available and I haven’t been able to discover   

 

So, have anyone else discovered some new programming concepts on the HC3? or additional methods/tricks available in QuickApps ?

Meanwhile, I will update this thread with new stuff as I continue to make discoveries in this undocumented landscape...  

 

Edited April 5, 2021 by jgab

[petergebruers 1,265]

11 hours ago, jgab said:

Disclaimer1: My review is based on the latest beta firmware, v5.020.xx, and there may be improvements coming before the ink in this post has dried.

 

First devices ship with 5.020.60 to be exact.

 

11 hours ago, jgab said:

They have changed quite a lot, some Lua functions and some fibaro* functions have disappeared -

Please login or register to see this link.

Yes, they've promised that.

 

11 hours ago, jgab said:

This is a new animal and has received  most of my attention.

 

Devices you code yourself are called QuickApps.

 

Yes, IMHO, it is best to think of it this way
 
"I am a power user on HC2"  -------> "I port my scenes to Quick Apps on HC3"
"I am a new user" ----> "Use block scenes on HC3 (graphic or Lua)"

 

There is no direct conversion of either scenes nor of VDs so this is going to upset people...

 

11 hours ago, jgab said:

We have stateful devices...

Yes, that is a big thing imho.

 

11 hours ago, jgab said:

- I would have liked  more UI elements and layout options than the buttons and sliders we have had for some years now.

I am not sure that will happen, but we might see more device types

 

11 hours ago, jgab said:

Possibility to enable remote Lua debugging - using

Please login or register to see this link.

?

 

Anything that looks like a possible security risk or a way to break out the locked down environment is a no-go, HC3 is much, much more closed (in all possible ways) than HC3.

 

I was used to debug "offline" and I miss that. I think, debugging is harder, at least for now, because we don't have an offline emulation like we had for HC2 + the glitches you've mentioned.

 

BTW all issues have been reported already and are tracked by Fibaro.

 

11 hours ago, jgab said:

Disclaimer 3: I may have missed stuff that are available and I haven’t been able to discover   

I think that's about it...

I could add... No "sleep" so setTimeout

This means no "mainloop" that runs every 3 seconds, but imho that's not a big loss - you code that by using setTimeout

[petergebruers 1,265]

4 hours ago, petergebruers said:

I could add... No "sleep" so setTimeout

Just got notified: fibaro.sleep() should be available with next update.

[jgab 2,009]

16 hours ago, petergebruers said:

Just got notified: fibaro.sleep() should be available with next update.

I’m not a fan of fibaro.sleep() but I think it makes sense to bring it back. So many people have small scenes like <trigger on x do y and wait until x is safe for z seconds>. Makes it easier for them to port.

I would like fibaro to start the scene as a coroutine (they probably do?) and then let sleep yield - you could then sleep and wait for other setTimeout calls without freezing everything.

[A.Socha 113]

W dniu 5.02.2020 o 10:42, petergebruers napisał:

 

 

 

I am not sure that will happen, but we might see more device types

 

 

You're right, but first we want to restore the deficiencies and remove the problems that came out, and I can say that it's not over.

There will be more types, at the moment there is a lack of thermostats, remote controls, multimedia, unlike in the previous version we intend to provide much more extensive possibilities for creating integration

[Macschimmi 35]

I don't get the Fibaro thermostats that I can control them via the app, but are they carried out in the heating settings? I'm desperate

Edited February 7, 2020 by Macschimmi

[tinman 1,027]

7 hours ago, Macschimmi said:

I don't get the Fibaro thermostats that I can control them via the app, but are they carried out in the heating settings? I'm desperate

 

just think different, heating/cooling is not via each every thermostate but via each every zone:

- create as much you need, climate zones, it can be more than one per room

- add to these zones your thermostates - here you can select what kind of mode is supported by your thermostate, e.g. only heating/off, or cooling/heating, etc.  If you select not supported mode, you will get nice warning 
- if you have none, or need to use unique combination like temp sensor from your TV and wifi Relais mounted to your IR Heater, then use QuickApp and build within max 25 clicks a thermostate  
- use Home Center app -> Heating shortcut -> to set zone values, to run on manual or scheduled values - no need care about "do i have to set eco or heating to get xxx thermostate to run properly with Fibaro?"  

If that not enought, create as much you need, profiles and simple block scenes to activate specific heating zones based on the profile you enabled.

Edited February 7, 2020 by tinman

[Macschimmi 35]

Thank you for the info ?

[petrkl12 208]

@jgab Can I expect that you will develop EventRunner for HC3?

 

In one of my HC2 virtual device I have Net.FUdpSocket() but in HC3 QA it doesn't work. Is it possible to use another command?

 

[jgab 2,009]

14 minutes ago, petrkl12 said:

@jgab Can I expect that you will develop EventRunner for HC3?

 

In one of my HC2 virtual device I have Net.FUdpSocket() but in HC3 QA it doesn't work. Is it possible to use another command?

 

I’ve made some progress in porting ER to HC3 - there may be some changes but most of the ER script can stay the same.  Still issues to work out...
 

Rumor is that they are coming out with an update next week - hope we get some more of the old functions back.

[petrkl12 208]

Super, Thanks

[Sjakie 137]

Jan,

That is great news!

Wow what a program (Multiple) Event Runner.

Great support you are giving us!

Thanks and have a nice weekend,

//Sjakie

[RH_Dreambox 179]

I follow this thread with great interest as it seems that @jgab is the person who has the ability and capacity to give us interesting and relevant information about HC3 and its possibilities.
We users are grateful for jgab's clear and easy-to-understand examples that make it easier for those who are thinking of switching from HC2 to HC3.
For my own part, I wait with new hardware until there is a working EventRunner for HC3?

[jgab 2,009]

Here are Fibaro's documentation for

Please login or register to see this link.

I don't know if this is the latest or not (they do mention fibaro.getSourceTrigger that is not available in 5.020.60)

[RH_Dreambox 179]

Thanks @jgab, Fibaro has hidden that path well, because I haven't been able to find that information myself.

[pnutp0wer 22]

@RH_Dreambox they updated the documentation, there is a header referring to the documentation of block scenes and lua

Please login or register to see this link.

[jgab 2,009]

I been looking into the 'conditions' feature of scenes.

This is not a tutorial on scene and conditions. You can read the fibaro scene documentation that I linked to in my previous post for a tutorial. However, I need to establish some terminology so that I can explain my experience of this so far ...  

 

"conditions" is separate section of the scene where the trigger condition is declared - the main scene code is referred to as the "actions"

Here is an example of a "conditions part"

Please login or register to see this code.

It's a Lua table format.

-operator can be of type 'any' or 'all'

-isTrigger can be of type true or false

-and <Trigger expression> is various kind of trigger conditions with device/times/weather/customevents etc.

 

We can nest the conditions, so what we have here is a logical expression of trigger conditions. 'any' being 'or', and 'all' being 'and'.

Ex.

Please login or register to see this code.

The 'isTrigger' property marks if the Trigger condition should trigger the check of the scene condition, or if it's just used as a test.

Ex.

T1 returns true if the temperature of device 20 is greater than 20

T2 returns true if the temperature of device 30 is greater than 20

 

If T1's isTrigger' is true, it means that the (whole) scene condition will be checked every time T1's state changes

If T2's isTrigger' is false then state changes it will not trigger the scene. T2 is only used as a test in the condition when the scene is triggered by some other trigger.

 

Did that makes sense?

(For you that have used EventRunner, devices/globals/events in the expression on the left of the '=>' trigger devices, on the right hand side they don't.)

 

A device trigger can specify what property should be the trigger, and a condition that the value of this property need to satisfy.

Ex. a trigger for a device (ID 20) to check if it's value property is true looks like

Please login or register to see this code.

This value return true/false but for numeric values you can use >,<,>=,<=,!= etc . also works for string value. (See documentation for more).

 

The time test is of two types. cron expression and sunset/sunrise.

Here is a time test using 'cron'

Please login or register to see this code.

The test is only true at 06:00 on Monday,Wednesday, and Friday.

The operator can be 'match', 'match>' ,match<' etc.

...and "matchInterval'

Please login or register to see this code.

According to the manual; "Condition true every 5 minutes, starting at 12:00 5.04.2019"

So when the date value is true, it will match every 300s from that match. (I tested it and the interval seems to be in minutes, so this is 300minutes interval - will they allow intervals in seconds?)

 

Ok, now it's starts to be a bit funky...

 

Let's start with the simple time test. What does 'match>' mean? 

If the value is  { "0", "12", "5", "4", "*", "2019" } it makes sense that it should be true after 12:00 5.04.2019

If the value is   { "0", "*", "*", "*", "*", "*" } - what does it mean? True after every whole hour, but for how long? Until it's true again? nah, until midnight? nah.

It obviously works sometimes, this is an expression being true between 09:00 and 10:00

Please login or register to see this code.

I guess they clear the condition at... midnight? so the first test <09:00 can be false again...

If so, it's a hack and what about other periods -  if you want to test something is between 1st of january to 5th of april?

 

In the general case you can see a cron expression as a list of times that it is valid for. When comparing the current time against that list, what value do you chose? 

 

The interval version, when does the interval stop? When the scene is triggered and condition false?, or at midnight or? (I didn't have the patience to wait it out  )

 

sunset and sunrise tests are a bit more straightforward and they have the semantics of daily test.

 

The time tests' semantics are funky and hopefully we will begin to understand what they mean by exploring them.

(Btw, it would be nice if for testing one could speed-run the conditions over a certain time period, hours, weeks ´, months and get a log when they trigger)

 

If we have a condition of type (my own simplified notation of a condition)

Please login or register to see this code.

Will that work of the device is breached before 09:00 and continues to be breached when the time becomes 09:00 ?

Will it work if we set 'isTrigger'=true on the first time test? I couldn't make it work but it may just be a bug.

 

Ok, conditions operators are 'any' and 'all' - I think that they should add 'not' for completeness.

 

Another thing is the constants used in the 'conditions' expression. device ID numbers and value constants.

It would be nice to have variables (fibaro's global variables?) as substitutes for constants. If you have a temperature trigger that should trigger if it's over 20 degrees, it would be nice to fetch and set that that value using a variable. 

 

The very common use-case that you want to trigger the scene if a device has been safe for x minutes (or the condition has been true for x minutes) is not possible to express in the conditions as I understand it.

...but should be possible to add as another conditions operator with some work...  

 

Well, I'm not really convinced that this is much easier to use than coding it in Lua (in the "actions" sections) - and there are still limitations/inflexibilities in the expressiveness that makes me wonder if people will not struggle with this as much as they did with Lua... and it's still programming (getting those operators right), just yet another idiosyncratic syntax...

 

Well, fortunately we will get fibaro.getSourceTrigger() back  

 

P.S However, it's very easy to compile the block scene tests to a 'conditions' expression... 

 

Edited February 9, 2020 by jgab

[petrkl12 208]

@jgab

Do you know how to call basic http autentification in Quick Apps?

I have tried this but I have received 401 error - but url in Chrome is OK

Please login or register to see this code.

 

[petergebruers 1,265]

44 minutes ago, jgab said:

The very common use-case that you want to trigger the scene if a device has been safe for x minutes (or the condition has been true for x minutes) is not possible to express in the conditions as I understand it.

I have a feature request similar to that, and it is under consideration by Fibaro. Try to express cleanly: "turn off a lamp that has been on for X hours". This is tricky to implement because we cannot cancel timers.

[jgab 2,009]

20 minutes ago, petergebruers said:

I have a feature request similar to that, and it is under consideration by Fibaro. Try to express cleanly: "turn off a lamp that has been on for X hours". This is tricky to implement because we cannot cancel timers.

We used to have clearTimeout.... Why don't we have t in scenes, but we have it in devices?

...and why did they in their wisdom create fibaro.setTimeout with the arguments swapped?

 

There are two approaches; The GEA way where they test the expression every 30s and when it has been true for the given period it's a go and they execute the action (they do some tricks with lastModified too to make sure stuff don't turns on/off/on in the period)

 

In my EventRunner framework, I run the expression every time it is triggered by some device/global/event being part of the expression, and I start/and stop timers to keep track of it.

Ex. rule(trueFor(00:05,device1:lux > 200 & device2:value < 10) & 10:00..11:00 => device3:on")

Every time device1 or device2 changes state I run the rule and 'trueFor' checks the value for the expression,

- if it's true and already was true it does nothing, and returns false (timer already started),

- if it's true and it was false before it starts a timer with a continuation that returns true and executes the rest of the rule.

- if it's false it cancel eventual timers and return false.

 

They mark all the triggering parts of the condition so it should be easy for them

 

Then of course you want to re-trigger the rule, so if a door is open for 10min sens a message, and continue to send a message every 10min it's open etc.

 

We have clearTimeout in QuickApps and I have been able to move my framework over there at the moment - but it would be much nicer to have it as a scene,

I could probably run it in a scene too (if we get os.time and os.date) and I make a stupid implementation of clearTimeout (but it's very wasteful resource wise to let timers continue and then inhibit them when they expire)

 

Btw, if they would let us have coroutines we could do really nice code and helper functions - we can hide the asynchronous nature of a lot of stuff in the framework and make it look synchronous for the user.

coroutines are deemed safe for sandbox lua's these days (at least as safe as math.*)

Edited February 9, 2020 by jgab

[jgab 2,009]

37 minutes ago, petrkl12 said:

@jgab

Do you know how to call basic http autentification in Quick Apps?

I have tried this but I have received 401 error - but url in Chrome is OK

Please login or register to see this code.

 

 

Have you tried

Please login or register to see this code.

 

You may also need to set headers ['Accept']=xxx ['Content-Type']=yyy to get it right too.

I also had to set ['Connection']='keep-alive' to get value back from the Telegram server - don't know why.