Jump to content

Welcome to Smart Home Forum by FIBARO

Dear Guest,

 

as you can notice parts of Smart Home Forum by FIBARO is not available for you. You have to register in order to view all content and post in our community. Don't worry! Registration is a simple free process that requires minimal information for you to sign up. Become a part of of Smart Home Forum by FIBARO by creating an account.

 

As a member you can:

  •     Start new topics and reply to others
  •     Follow topics and users to get email updates
  •     Get your own profile page and make new friends
  •     Send personal messages
  •     ... and learn a lot about our system!

 

Regards,

Smart Home Forum by FIBARO Team


  • 0

Capture All Event to Send to Log Server


dowoco

Question

I am trying to find a way to capture the events in the Events Panel ("/fibaro/en/panels/events.html") so that I can record them in a standalone Log server. This way I can manage all my servers logs from one place. I have the code to pass the log information, but just no way to extract it from Fibaro HC2.

 

I have added logging to my Scenes and to the Virtual Devices. All the Debug and Error events are being captured and logged. I thought it would be useful to also capture the device events in the Event Panel as this would help me to uncover issues.

 

Is there an API or Scene that someone is aware of that would allow me to capture this information? I assume there is in the fact that the event panel exists and its updating in real-time.

 

I am using InfluxDb with telegraf to capture the syslogs as I already have an instance running. I use Grafana as the visualisation tool.

Link to comment
Share on other sites

16 answers to this question

Recommended Posts

  • 0
  • Inquirer
  • I found this topic: 

    and this topic.

     

     

    Both are useful and work looking into further.

    Link to comment
    Share on other sites

    • 0
    4 hours ago, dowoco said:

    Is there an API or Scene that someone is aware of that would allow me to capture this information? I assume there is in the fact that the event panel exists and its updating in real-time.

    I don't know what exactly you need to catch and why, but please download Z-wave Monitor ( and Z-wave Viewer) to see if it fits your request to capture  Z-wave events in real time, regardless if they're logging to event panel.

    Going further we can work together to change the script in order to send this info to your log server...

    Edited by cag014
    Link to comment
    Share on other sites

    • 0

    Just write scene and in header put * instead of number of device

    this scene will calling by Fibaro on each event 

    after you may send this event to own server 

     

    another way grab event via “refresh”, @cag014 for example have a good script for it and add in this script calling your own server and pass here events 

     

    another way is node-red and Fibaro plug-in for node red, just setup this node and pass all event t to your MQTT server 

     

    btw, dear Fibaro,  @T.Konopka   can you provide a  simple answer for my simple question 

    when you add MQTT client what 2 year you have in private methods to public LUA API

    1) never. 

    2) in next release 

    3) when pigs fly

     

    thank you

    Link to comment
    Share on other sites

    • 0

    The API I use is:

     

    Please login or register to see this code.

    But please also have a look at the ideas posted by @cag014 and @10der

     

    I wrote my own script to copy and synchronize all events to influxdb, based on that API, but it is for private use only. I wrote it because none of the scripts I found here did exactly what I wanted (they usually introduce more data points than actually exist). There are different ways to do it and that is one of the reasons why it is private-only. Send me a PM if you're interested.

    Link to comment
    Share on other sites

    • 0
  • Inquirer
  • @cag014The Z-wave Monitor is awesome, this is part of what I was looking for. I will have a dig around and see how I can attach the Logserver to it. But the visuals on the Z-wave Monitor are impressive. 

    Link to comment
    Share on other sites

    • 0

    Thank you

    As I mentioned, if you can provide the code which sends that data to server, I can try to embedded in the script

    Link to comment
    Share on other sites

    • 0

    I'm not the OP, so I hope you don't mind me commenting on this... I had similar questions when I developed my script and want to save you some time.

     

    Storing data in influxdb uses a very simple http protocol. The question is: how do you want to store the data? Influxdb is not sql and does not have joins. So I tried 5 different scenario's before picking one. You also have to think about either storing by ID or by "some meaningful name" possibly using a translation table. I store everything by ID because I sometimes rename devices and that would break history. But my script also can translate names, it is an option. Lastly, I am not sure how long you can stop a scene based on refreshstates, before you start missing events. My script is based on the event log, and in my case I can stop it for about 10 days (then HC  purges records after 40.000 like every HC, in my case that is about 10 days). Maybe the ideal solution is based on both refreshstates and eventlog api...

    • Like 1
    Link to comment
    Share on other sites

    • 0

    I'm curious, what the purpose to save this information at external Server.. this is a huge amount of events  to review (it probably will take few hours to review one day of events).

     

     

    Link to comment
    Share on other sites

    • 0

    In my case... Check if a certain device from a certain manufacturer meets my expectations ;)

    My database has about 2 million data points.

    Link to comment
    Share on other sites

    • 0
    2 minutes ago, petergebruers said:

    My database has about 2 million data points.

     it's becoming surreal! Old Peter. blanket and some wine, rocking chair. Peter separating 2 million data points on print paper...

    Link to comment
    Share on other sites

    • 0
    On 2/20/2019 at 7:10 AM, 10der said:

    Just write scene and in header put * instead of number of device

    this scene will calling by Fibaro on each event 

    after you may send this event to own server 

     

    another way grab event via “refresh”, @cag014 for example have a good script for it and add in this script calling your own server and pass here events 

     

    another way is node-red and Fibaro plug-in for node red, just setup this node and pass all event t to your MQTT server 

     

    btw, dear Fibaro,  @T.Konopka   can you provide a  simple answer for my simple question 

    when you add MQTT client what 2 year you have in private methods to public LUA API

    1) never. 

    2) in next release 

    3) when pigs fly

     

    thank you

    Hi 10der is it possible that i can't save the scene anymore when I put a * as device id?

    Link to comment
    Share on other sites

    • 0
    On 2/20/2019 at 2:37 PM, petergebruers said:

    I store everything by ID because I sometimes rename devices and that would break history. But my script also can translate names, it is an option.

     

    How did you do it, to put everything in Influx. Can you share some ideas?

    Link to comment
    Share on other sites

    • 0
    52 minutes ago, SmartHomeEddy said:

    How did you do it, to put everything in Influx. Can you share some ideas?

    There are a few aspects to this, you may want to think about theme before implementing this because it is hard to change once you have production data in your DB.

     

    - how to name things

    - how many databases/measurements/fields/tags do you want... It is related to the first question but it also dictates which "things" are indices and which is data and how you can manipulate them.

    - What exactly do you want to represent... Do you want to run a "check" of sensors every X minutes and store that? Or do you want a "copy of the event log" which tells you exactly what happened when? I chose the latter.

     

    Note.. Everything you'll find only talks about "monitoring of machines" and the uses case is totally unlike IoT. The main thing with systems monitoring is you have a regular time series and a steady flow of data: ever minute you check memory, CPU and free disk space and turn that into a graph. With IoT something happens at a random time, eg motion sensor reports safety at 10:00, light is turned on at 10:03, a temperature sensor reports at 10:15 and does not report anything until temp changes... It is an irregular aka "unevenly spaced" time series. They are harder to work with (but they are "true" data sets). One "solution" most people use it pretend the event log is a regular time series. To take the temperature sensor example, you could store the cached data on your HC every 2 minutes. So although the temp sensor reports at 10:15, you only record the new temperature at 10:16. And at 10:18 you store it again, and at 10:20 again the same temperature. It is easy to work with, but it is also... somewhat wrong...

     

    I have a somewhat peculiar way of getting the data into InfluxDB because I wanted to be able to do "joins" between measurements and that is not supported on InfluxDB. I know the language looks deceptively like SQL but is quite a dialect and quite a specific database as well. Excellent for time series, but definitely not a general purpose database. For most users on this forum it does not matter because the goal is to make graphs in ... "grafana".

     

    I never published this script for several reasons, one being the choice's I have made (I want to use the command line interface of InfluxDB) and the data format of the event log has changed a few times and may still be "in flux" (pun intended) which means I would have to be under pressure to fix this after each upgrade. You may ask... Why can't people fix it when they have the source code? We'll... I did share it with a few users and the consensus is that it is "quite hard to read and understand" although the underlying principle is quite easy to grasp. I have to use "httpClient", that means my code is using callbacks and that is a confusing concept if you don't use this often...

     

    Give me 1/2 hour and I'll share the main thoughts here, see if it can get you started or at least make informed decisions.

    Edited by petergebruers
    • Like 1
    Link to comment
    Share on other sites

    • 0

    Thanks @petergebruers I was considering this weekend how to do it and indeed there are different ways. At this moment I only send some values every one minute and therefor I am limited in Influx or Grafana to what I now considered to send. But I want to be more flexible and use Influx / Grafana more for its own purpose, what it is good at. Fibaro only gives us a few graphics up til now and the underlaying data is deleted now and then. So the HC3 is not the right place to do it, I think Infux / Grafana can be the right place. 

     

    So I am looking for different considerations, ideas. 

     

    I was thinking to send data from weather (including air quality), energy consumption, refreshstates, events and or diagnostics to Influx. 

    Link to comment
    Share on other sites

    • 0
    17 minutes ago, SmartHomeEddy said:

    At this moment I only send some values every one minute and therefor I am limited in Influx or Grafana to what I now considered to send.

    Indeed, it is easy to get your first graph "up and running" that way but you've probably gone through the same thought process like me: it is "limited" and it feels as if "all the data is available on your HCx" but you cannot get it in InfluxDB "in the right way".

     

    It depends what you ant to do... Someone asked my "what is a temperature chart used for" and in my case the answer if "to check performance of my system and see if a certain TRV misbehaves". Not everyone is interested in that. In my case, 14 days of storage of a few devices would be sufficient. On the other hand... if I notice an issue I might want to compare with much older data.

     

    This lead me to the idea... what if I could keep the "event log data forever". It is possible, but there are a few gotchas.

     

    23 minutes ago, SmartHomeEddy said:

    I think Infux / Grafana can be the right place. 

    Just for sake of completeness... I started with InfluxDB 0.9 and at that time certainly "grafana" was the way to go. There weren't many popular options back then. The 2.X version is a bit new IMHO so now I am at V 1.8. Since that old version, Influx has its own kind of dashboard called "chronograph" but I have never used it.

     

    Please login or register to see this link.

     

    Also, some of my data is in a "plain old SQL database" and I do miss some of the specific InfluxDB stuff (like "fill" of missing data). If you already have a PostgreSQL database, this is worth investigating. 

    Please login or register to see this link.

     - I ran a few tests on my Mac but never tried in production.

     

    I haven't felt the urge to test InfluxDB 2.X yet

     

    31 minutes ago, SmartHomeEddy said:

    I was thinking to send data from weather (including air quality), energy consumption, refreshstates, events and or diagnostics to Influx.

    refreshStates and events are very similar. The refreshstates has a clear advantage: it is instant. The downside: if InfluxDB is not running, data is lost. That's why I transfer events by running a FQA every minute and it checks "last timestamp transferred".

     

    Not sure if "weather" is a special case but I guess it is, I cannot remember seeing it in "events". Diagnostics like memory/cpu should be transferred periodically I guess.

     

    The broad lines of my "events to influx" script are

     

    local url =
                  ("/events/history?from=%d&to=%d&numberOfRecords=200000"):format(
                      from, to)

     

    The "from" is based on a single timestamp stored in single spot in InfluxDB. The "to" is not always "now", it represents a sane maximum to allow for batching.

     

    The second thing, worthy of note, is the fact that InfluxDB uses "time" as a unique hue, if two measurements have the same timestamp then "last write wins". The timestamps in the event log are "integer seconds" so using them straight up would not work.

     

          if (timestamp == timestampPrev) then
              timestampAdd = timestampAdd + 1
     

    Meaning... if an event has the same timestamp as the previous one, make it unique by adding a small amount. I have set the transfer format to millis, so if, for example, a motion sensor triggers at 09:00:00 and the light turns on 0.5 second later, also at 09:00:00 that is, I actually write 09:00:00.000 and 09:00:00.001

     

    Then, lastly, per event you have to extract the useful data and that is quite a long if/then else. Here is a part to give some you some inspiration.

     

      if e.type == "SceneActivationEvent" then
          line = line_format.event_SceneActivation:format(fieldname,
                                                          e.data.sceneId,
                                                          timestamp, timestampAdd)
      elseif e.type == "CentralSceneEvent" then
          line = line_format.event_CentralScene:format(fieldname,
                                                       escape_field_value(
                                                           e.data.keyAttribute),
                                                       fieldname, e.data.keyId,
                                                       timestamp, timestampAdd)
     

    You don't need to worry about the exact meaning of the "format" commands, the key point is SceneActivationEvent and CentralSceneEvent are different enough to warrant separate treatment.

     

    When you have a "DevicePropertyUpdatedEvent" the actual may be in property "value"... But it depends... A thermostat will report under "heatingThermostatSetpoint" and variants of that.

     

    Lastly, before writing data, you have to decide on naming stuff. I would argue that IDs are the only "stable" representation of devices, as long as you do not reconfigure the devices. On the other hand, naming field D117 is not very user friendly and error prone. On the other hand, base the names on device name and room name (and possibly section name as well) might be "fragile". A simple rename on HCx  writes data to the wrong field.

     

    So I came up with a comprise. By default, use the ID. But allow for a mapping in code...

     

    local mapping =
    {
    [773]="Boiler Out",
     [774]="Temp_Ext",
     [772]="Temp_ttt",
    ...

     

    I hope this inspires you to look at your "problem" - but with fresh inspiration. I designed this a long time ago (started on HC2, it took 6 revisions mostly because of changes in the API) and it is certainly not the only way to do it.

    • Like 1
    Link to comment
    Share on other sites

    • 0
    3 hours ago, petergebruers said:

    Influx has its own kind of dashboard called "chronograph" but I have never used it.

    Ah, thanks. I didn't see that one yet. I will have a look at it. At first glance its looks like it would cover Grafana. 

     

     

    3 hours ago, petergebruers said:

    I haven't felt the urge to test InfluxDB 2.X yet

    I looked at it, but I am still not sure it works on e Raspberry Pi 4B setup. When I install the last stable release, Influx still comes with 1.8.

     

     

    3 hours ago, petergebruers said:

    The second thing, worthy of note, is the fact that InfluxDB uses "time" as a unique hue, if two measurements have the same timestamp then "last write wins". The timestamps in the event log are "integer seconds" so using them straight up would not work.

    Even if they are measurements of two different devices?

     

     

    3 hours ago, petergebruers said:

    So I came up with a comprise. By default, use the ID. But allow for a mapping in code...

    I was thinking about that issue. I already have Devices with the same name in different rooms. And now and then a reconfigure takes place. This is a smart solution. In case of a reconfigure you just have to change de mapping. 

     

     

    3 hours ago, petergebruers said:

    I hope this inspires you to look at your "problem" - but with fresh inspiration.

    Thanks Peter, this inspires me a lot. You had the "quest" some time ago, I recognize a lot of your issues and understand your solutions ???

    Link to comment
    Share on other sites

    Join the conversation

    You can post now and register later. If you have an account, sign in now to post with your account.

    Guest
    Answer this question...

    ×   Pasted as rich text.   Paste as plain text instead

      Only 75 emoji are allowed.

    ×   Your link has been automatically embedded.   Display as a link instead

    ×   Your previous content has been restored.   Clear editor

    ×   You cannot paste images directly. Upload or insert images from URL.

    ×
    ×
    • Create New...