Jump to content

Welcome to Smart Home Forum by FIBARO

Dear Guest,

 

as you can notice parts of Smart Home Forum by FIBARO is not available for you. You have to register in order to view all content and post in our community. Don't worry! Registration is a simple free process that requires minimal information for you to sign up. Become a part of of Smart Home Forum by FIBARO by creating an account.

 

As a member you can:

  •     Start new topics and reply to others
  •     Follow topics and users to get email updates
  •     Get your own profile page and make new friends
  •     Send personal messages
  •     ... and learn a lot about our system!

 

Regards,

Smart Home Forum by FIBARO Team


HC3 QuickApps coding - tips and tricks


jgab

Recommended Posts

On 3/18/2020 at 5:34 PM, jgab said:

I made a long post on how I thought QAs worked (internally) and why it is a cooperative multitasking environment and why we need to use setTimeout to give time to other tasks , like clicking buttons, calling QuickApp:functions etc.

@jgab

Even using all your recommendations when one QuickApp busy (for example waits for refreshState) all QuickApp are stuck.  This is something that Fibaro needs to solve.

Since http available in QuickApp only, imagine what can happened...!!!

By the way this is not multitasking... even UI stuck while QuickApp busy.

Edited by cag014
Link to comment
Share on other sites

  • Topic Author
  • You mean within one QuickApp - different QuickApps run independent of each other. 

    6 hours ago, cag014 said:

    @jgab

    Even using all your recommendations when one QuickApp busy (for example waits for refreshState) all QuickApp are stuck.  This is something that Fibaro needs to solve.

    Since http available in QuickApp only, imagine what can happened...!!!

     

    Yes, I think the behaviour with refreshState is not optimal. @petergebruers thinks it's a feature. I can see that side of the matter too (the example below actually uses that "feature")

    You say "for example waits for refreshSates" but I think it's only the refreshStates that becomes an issue as other api.* calls are very fast. All the other issues with call QuickApp:functions is handled by giving up time with setTimeout. (or realising that calling your self with a QuickApp:function gives no time for yourself to respond :-) )

    So besides that api.get have a timeout of ~30s if there is no value ready (as can happen for refreshStates) which one can have opinions about (I would like to add an "timeout" parameter) the other behaviour of the HC3 QuickApp framework is quite logical and predictable (scenes and conditions is another matter :-) )

     

    I don't think the average QuickApp developer will encounter the refreshStates issue - and if so we just need to document ways around it.

     

    So, specifically for refreshStates:

     

    -One hack I use is to generate an event last in the refreshStates loop, like setting a global variable, then there is always at least one event next time i call api.get and I will not hang.

     

    -net.HTTPClient() requests gives time to other other "threads" (cooperative tasks) when it's waiting for the callback. (this is also true for net.TCPsocket() )

    So, if you use net.HTTPClient():request instead you can "poll" refreshStates without hanging the rest. However, I don't like having to provide the credentials to the HC3.(forgot about :11111..)

     

    -Have you used my recommendation and tried polling refreshStates in a separate QuickApp?

    This works for me.

    I have one QuickApp with this code - this one "hangs" on refreshStates until there are some new events (or it times out).

    Please login or register to see this code.

    and other QuickApps that wants events just need to have the function QuickApp:EVENTS defined in the code

    Ex.

    Please login or register to see this code.

    Side note. It can be tempting to add a QuickApp:function like 

    Please login or register to see this code.

    In the event watcher quickapp to allow other quickapps tell it that they are interested in events (instead of the event watcher scanning the devices' code for the EVENTS function).

    However, that doesn't work that well as the event watcher will not get the call while it's busy waiting for refreshStates.

     

    6 hours ago, cag014 said:

    @jgab

    By the way this is not multitasking... even UI stuck while QuickApp busy.

    Please define "busy". Waiting for refreshStates? That we discussed above.

    Running in a tight loop or waiting for refreshStates? - no time for anything else in the QA and no time to update UI or handle button clicks and that's expected.

    etc.

    If something is stuck it's usually not giving time to cooperative tasks. Nothing is ever running in parallell in a single QA. It's only cooperative tasks giving up time to each other.

    Then there is another matter that there are bugs in the framework so that UI elements are not updated when we call updateView etc. 

    Edited by jgab
    Link to comment
    Share on other sites

    55 minutes ago, jgab said:

    Have you used my recommendation and tried polling refreshStates in a separate QuickApp?

    This works for me.

    I have one QuickApp with this code - this one "hangs" on refreshStates until there are some new events (or it times out).

    Yes, I do use separate QuickApp... and when it stuck all other activities (including system UI) are stuck.

    So what you're saying that moving forward with newest (faster and more powerful) box we have this issue while on HC2 it work fine.

    Link to comment
    Share on other sites

  • Topic Author
  • 1 minute ago, cag014 said:

    Yes, I do use separate QuickApp... and when it stuck all other activities (including system UI) are stuck.

    So what you're saying that moving forward with newest (faster and more powerful) box we have this issue while on HC2 it work fine.

    Stuck in what? refreshState?

    My example in the previous post has the event watcher "stuck" in waiting for refreshState and all other QuickApps works just fine for me. Have you tried to use net.HTTPClient() instead?

    I'm running some extremely complicated scenes (multiple EventRunners, each polling refreshStates, and calling each other) and I have no performance issue. The 4 cores are actually not that bad to have...

    Link to comment
    Share on other sites

  • Topic Author
  • 10 minutes ago, cag014 said:

    Yes, I do use separate QuickApp... and when it stuck all other activities (including system UI) are stuck.

    So what you're saying that moving forward with newest (faster and more powerful) box we have this issue while on HC2 it work fine.

     

    Try to code some simple condensed examples that gives you the problem you refer to and post them here so we can find ways how to get around it.

    Link to comment
    Share on other sites

    18 minutes ago, jgab said:

    My example in the previous post has the event watcher "stuck" in waiting for refreshState and all other QuickApps works just fine for me.

    I have one scene using api.get on refreshStates and ather FQAs work as expected, nothing hangs.

     

    Code in my "refreshStates" scene, stripped to the "refreshStates" part and leaving out all the irrelevant parts

     

    Please login or register to see this code.

    This gives only "crude timing" because the worst case time resolution is 30 seconds but I can live with that. If you want to do better than that, you would have to use httpClient.

     

    I wrote this a very long ago on HC2. I think the most interesting bit is the part that checks if states.last changed due to backup. I cannot explain the "rand" stuff, I am imitating what Fibaro does but I haven't seen any adverse effects on HC2 if you leave out that "rand" part. It might have to do with detecting multiple clients coming from same IP/PORT combination (that is pure speculation).

    Link to comment
    Share on other sites

    @jgab, @petergebruers

    May I ask you how many devices you have... probably more than two?

    In addition as jgab mentioned, he has a lot of QuickApp, scenes and etc. which I believe update properties and views.

    All these create a lot of refershState  activities, so you cannot see what I'm talking about....

    Believe me... when I update view in Quick:App every second... no issues everything runs properly, but the problem is still there...

    Link to comment
    Share on other sites

  • Topic Author
  • 10 minutes ago, petergebruers said:

    I have one scene using api.get on refreshStates and ather FQAs work as expected, nothing hangs.

     

    Code in my "refreshStates" scene, stripped to the "refreshStates" part and leaving out all the irrelevant parts

     

    This gives only "crude timing" because the worst case time resolution is 30 seconds but I can live with that. If you want to do better than that, you would have to use httpClient.

     

    I wrote this a very long ago on HC2. I think the most interesting bit is the part that checks if states.last changed due to backup. I cannot explain the "rand" stuff, I am imitating what Fibaro does but I haven't seen any adverse effects on HC2 if you leave out that "rand" part. It might have to do with detecting multiple clients coming from same IP/PORT combination (that is pure speculation).

    Yes, I think your guess on rand is correct and probably something we should start to use just to be on the safe side. However, it's still a bit strange not to be able to pair up requests with responses...

    ...and yes I see the same behaviour as you do. However, when hanging on refreshStates,  if an event happens (or changes or alarmChanges) it will return immediately with the event (at least on the HC3) so the "response" time is as good as it gets.

    Edited by jgab
    Link to comment
    Share on other sites

    Just now, jgab said:

    However, when hanging on refreshStat,  if an event happens (or changes or alarmChanges) it will return immediately with the event (at least on the HC3) so the "response" time is as good as it gets.

    Exactly, that was IMHO the "design intent" behind the refreshStates API. It uses the property that http is actually a TCP socket, and when there is no data on the socket the handler goes to sleep, the O/S manages that. As soon as the HC2 sends data, the TCP socket receives data and the receiving thread wakes up. So the other end, your quick app in this case, resumes ASAP. It is as good as it gets using plain TCP communication protocol. Also, neither side is ever waisting CPU cycles (eg by "spinning" in a tight loop). If refsreshStates was non-blocking, you would have to write a while loop with sleep or setTimeout and you would get worse response times and higher CPU (depending on sleep interval).

    Link to comment
    Share on other sites

  • Topic Author
  • 1 minute ago, petergebruers said:

    Exactly, that was IMHO the "design intent" behind the refreshStates API. It uses the property that http is actually a TCP socket, and when there is no data on the socket the handler goes to sleep, the O/S manages that. As soon as the HC2 sends data, the TCP socket receives data and the receiving thread wakes up. So the other end, your quick app in this case, resumes ASAP. It is as good as it gets using plain TCP communication protocol. Also, neither side is ever waisting CPU cycles (eg by "spinning" in a tight loop). If refsreshStates was non-blocking, you would have to write a while loop with sleep or setTimeout and you would get worse response times and higher CPU (depending on sleep interval).

    Yes, we agree on that. It just requires some extra coding effort to not get stuck when trying to retrieve the events and accidentally block other things going on in the QA.

    But it's doable and that's what the last posts have been about. Different approaches to do it will fit different use cases. net.HTTPClient() or generate an event if inside the same QA. A separate "event watcher" could be reasonable if you have a lot of QA's polling for refreshStates.

    Link to comment
    Share on other sites

  • Topic Author
  • 2 hours ago, petergebruers said:

    Exactly, that was IMHO the "design intent" behind the refreshStates API. It uses the property that http is actually a TCP socket, and when there is no data on the socket the handler goes to sleep, the O/S manages that. As soon as the HC2 sends data, the TCP socket receives data and the receiving thread wakes up.

    Yes, the problem is that it hangs the (Linux) thread and Lua is single threaded so it hangs the whole calling QA. (there are ways for Luabind calls from different Linux threads to lock the luastate, but because here is only one luastate it becomes in reality single-threaded access)

    A more elegant approach would have been to run the QA as a Lua coroutine and suspend the coroutine that calls api.get. (similar to what I outlined in the toy "

    Please login or register to see this link.

    ")

    That would allow other tasks (that would also be coroutines) like setTImeouts or externally called QuickApp:functions to still run. There are LuaSocket type libraries that are non-blocking and work with coroutines (NGINX/openresty uses it) . and it's not that difficult to write from scratch using LuaSocket.

     

    The "main loop" scheduling the coroutines could make use of a system timer to suspend when no coroutines need to run, thus saving CPU and be as efficient when nothing needs to run. That timer could be interrupted and woken up when an external event arrives - like a UI click or QuickApp:function call  etc - so a new coroutine for that task could be scheduled.

     

    I'm having the feeling that Fibaro have made it too easy for themselves and instead put a lot of burden on programmers not so familiar with a cooperative multitasking model.

    Edited by jgab
    Link to comment
    Share on other sites

    @jgab yeah sure the whole discussion centres around the absence "coroutines" sou you really have to choose between

     

    - non blocking with callbacks, so setTimeOut and net.httpClient

    - blocking with sleep and api.get

     

    I would like to have coroutines as well because of the reasons you've mentioned...

    Link to comment
    Share on other sites

  • Topic Author
  • 7 minutes ago, petergebruers said:

    @jgab yeah sure the whole discussion centres around the absence "coroutines" sou you really have to choose between

     

    - non blocking with callbacks, so setTimeOut and net.httpClient

    - blocking with sleep and api.get

     

    I would like to have coroutines as well because of the reasons you've mentioned...

    Yes, and then maybe Fibaro should produce some documentation how to navigate this blocking/non-blocking environment to educate the users and reduce future support calls due to strange behaving QAs....

     

    ...or they can spend some efforts in trying to hide it (e.g with coroutines). 

    I would have spent my money on the latter...

    Edited by jgab
    • Like 1
    Link to comment
    Share on other sites

    1 hour ago, cag014 said:

    May I ask you how many devices you have... probably more than two?

    No, that is about right, I have many Z-Wave devices but only e few FQAs

     

    - One FQA using refreshStates to turn on/off lights in kitchen.

    - One FQA dumping event log to influxdb, does that once per minute

    - One FQA controlling a RGBW light, connected to a DMX bus and using http to set RGBW levels

     

    That's it. All three use some form of http access.

     

    I have a "refreshstates" logging script on my MAC and it dumps events, the 3d column is time delta between events:

     

    Please login or register to see this code.

    As you can see... Mys system is rather quiet. Except that 198 ms interval, all should be very noticeable, if all FQAs hang while waiting on such an event.

     

    But I don't notice any such delay when I toggle my RGBW light from my phone (so phone -> HC2 -> triggers FQA -> calls specific http URL -> code on my server sends DMX command -> RGBW light toggles). Tried 50 times...

    Link to comment
    Share on other sites

    @cag014 (quick and dirty) refresh states "dumper" using http client instead of api.get. You've been around here for a while so you probably know api.get urls get translated to http://127.0.0.1:11111/api but for other forum members that may not be obvious.

     

    IMHO this stays responsive and avoids blocking issues. Would require a bit more error handling for production because in rare case you might get an error from that call. The request never leaves the HC3 so it is quite stable, but like api.get it can happen though the number of posts on this forum abut this issue is very low.

     

    Please login or register to see this code.

    I am not an elegant coder like @jgab - I am only posting here, hoping that it will be of use in your hunt for a better "refreshStates" experience on your system.

    Edited by petergebruers
    • Like 2
    Link to comment
    Share on other sites

    On 2/17/2020 at 6:54 PM, jgab said:

    api.get("/devices/"..id).properties.quickAppVariables

     

    In general it would be nice if the call fibaro.getValue(deviceID,<property>) would call the function QuickApp:getValue(<property>) .... end

    and a value could be returned...

     

    On 2/17/2020 at 6:58 PM, cag014 said:

    I know, but it force you to deal with jSON that includes all global variables... I thought finally fibaro did something that we can receive value and not a table.

     

    If it can help...

     

    Please login or register to see this code.

    Use:

    Please login or register to see this code.

     

    Link to comment
    Share on other sites

    On 2/29/2020 at 7:39 AM, jgab said:

    some 5.021.38 issues

    Please login or register to see this code.

     

    I use in QA the following...

     

    Please login or register to see this code.

    In a QA method :

    Please login or register to see this code.

    i call the QA from scene, the scene is triggered herself by GlobalVariable if result needed

    • Like 1
    Link to comment
    Share on other sites

    18 hours ago, Krikroff said:

    I use in QA the following...

    Thanks for sharing. Your code inspired me to turn that into a "Binary Sensor" quick app and apply a few "optimisations". Well, I think they are optimisations... Please tell me if you do not agree. Firstly I've moved the http client to "self" and create it in "onInit" only once. Because you only need one of those, per app, and it can handle all your requests. No need to create and destroy that HTTPClient each time. Also moved the "header" with authentication because that one also does not change and makes it easier to see where you define the password (if you has multiple http requests, in this example there is only one so not a good example). Actually, you could take it a bit further and store the options or the complete request, because it uses always the same options and url.

     

    I am not sure if it matters, but if it turns out that FQAs ore some things on HC3 leak, then this sort of optimisation might buy you more time. I am not sure. I have no proof of that.

     

    The choice of 10 seconds in the loop is completely arbitrary... That number might be a bit to low (too fast) in case you only want to detect up/down of internet link in code. On the other hand, if you want some visual feedback 10s may be good.

     

    I did not test the "error(result.status)" part, I copied that from your code and it appears to me it would get called in case the API does not return the expected data... You probably have a very good reason to do that (apart from the fact that this is "good practice") - can tell me what can cause this kind of error?

     

    Please login or register to see this code.

     

    • Like 1
    Link to comment
    Share on other sites

    2 hours ago, petergebruers said:

    Thanks for sharing. Your code inspired me to turn that into a "Binary Sensor" quick app and apply a few "optimisations". Well, I think they are optimisations... Please tell me if you do not agree. Firstly I've moved the http client to "self" and create it in "onInit" only once. Because you only need one of those, per app, and it can handle all your requests. No need to create and destroy that HTTPClient each time. Also moved the "header" with authentication because that one also does not change and makes it easier to see where you define the password (if you has multiple http requests, in this example there is only one so not a good example). Actually, you could take it a bit further and store the options or the complete request, because it uses always the same options and url.

     

    I'am glad to have inspired you ;)

    The piece of code (rewritten and taken out of context) posted is deliberately structured like this in order to offer something extremely simple, it obviously does not look like that in my code ;) and your comments are judicious.

     

    2 hours ago, petergebruers said:

    I am not sure if it matters, but if it turns out that FQAs ore some things on HC3 leak, then this sort of optimisation might buy you more time. I am not sure. I have no proof of that.


    for sure :)  ... Personally I develop all of my components in class, this allowing to optimize, rationalize and reuse the developments.
     

    2 hours ago, petergebruers said:

    I did not test the "error(result.status)" part, I copied that from your code and it appears to me it would get called in case the API does not return the expected data... You probably have a very good reason to do that (apart from the fact that this is "good practice") - can tell me what can cause this kind of error?

     

    I rewrote blindly and this is not suitable, I offer the following thing which will be able to deal with possible errors  (missing auth for example ...) using the callback. 

     

    Please login or register to see this code.

    and the callbacks:

    Please login or register to see this code.

     

    • Thanks 1
    Link to comment
    Share on other sites

    i am trying to press a button in a quickapp from a scene but i do not know the action to use.
    in the quickapp it is self:button2Clicked() but i cant seem to get it working from a lua scene to the quickapp

    Link to comment
    Share on other sites

    Join the conversation

    You can post now and register later. If you have an account, sign in now to post with your account.

    Guest
    Reply to this topic...

    ×   Pasted as rich text.   Paste as plain text instead

      Only 75 emoji are allowed.

    ×   Your link has been automatically embedded.   Display as a link instead

    ×   Your previous content has been restored.   Clear editor

    ×   You cannot paste images directly. Upload or insert images from URL.

    ×
    ×
    • Create New...