auto::cerr2 – Automatic Capture of Errors version 2

[Łukasz997 117]

This thread is a continuation of thread 

Please login or register to see this link.

 – it’s worth reading to see the background of this QuickApp idea.

Now, this is  Automatic Capture of Errors 2.
It’s based on the idea of version 1, but it’s a completely new program. I think it deserves a new thread.

What’s the same/similar:

• captures www console messages, deduplicates them & shows them in the QuickApp UI and in the Yubii mobile app with statistics,

• uses colors to indicate message age,

• holds a large database of messages,

• sends PUSH notifications so you never miss important info; prevents a “storm” of repeating messages. PUSH pause/relaxation is dynamic, based on the actual situation,

• can send PUSH summaries,

• DST-aware,

• …and the rest of the minor functions described in the

  Please login or register to see this link.

  thread.

What’s new:

• continuously tracks not only errors, but all messages displayed in the web console,

• provides a filter DSL – a simple language for writing filters that limit which messages are processed (displayed & PUSHed),

• allows you to write as many filters as you want and change them without restarting the QuickApp (via drop-down menu),

• precise error and warning reporting for the filter DSL,

• many minor improvements (clearing messages, help, UI...),

and

• one more “secret” feature: the ability to append new filters remotely without editing this QA code or using local/global HC3 variables.
  This will be described in a separate thread, because it can be used much more broadly than just with this QA.

 

Benefits over the standard web console:

• stores more messages,

• deduplicates messages,

• sends PUSH on new messages,

• powerful filtering (on HC3 www console entering phrase in filter field only highlight match, not do filtering). User can, via defining filters, catch rare, unusual but important message and receive immediate notification when it arrive,

• works on Yubii app on smartphone.

 

In the next messages I’ll describe configuration, UI, and the filter DSL.
In a separate thread – the remote config option.

Please login or register to see this attachment.

Edited October 2, 2025 by Łukasz997

[Łukasz997 117]

1. Configuration

The QA structure contains two files: __main_config and __filters_config.

 

In __main_config you’ll find a detailed description of all QA parameters. In most cases, you don’t need to change anything.
You may want to adjust:

• user_id – who receives PUSHes. If set to "auto", a superuser will be selected (may not be named admin). You can also specify a user ID (improved vs. old version) or disable PUSH at all (nil or 0).

• purge_after_sec – time in seconds after which a message is permanently purged from the database. New messages with the same source & text reset the timer. Tracked independently per message.

Rarely, you may consider changing:

• database_limit – max number of messages stored. Don’t overdo it to avoid HC3 slowdown (I mean > ~5k  ).

• ui_row_limit – max messages displayed in UI. Can’t exceed 148 (HTML library limit). Table generation is CPU consuming, but there’s caching at both HTML-render and UI generation level (table is reused when possible).

CONF.colors defines colors for message age. Unlike in v1, it’s fully automatic and not sensitive to changes in purge_after_sec. No need to edit it.

 

__filters_config holds definitions of filter sets.
Each entry is a text block (enclosed in [[...]]). Later, all filters can be accessed via a drop-down menu.

At program start filters are translated into internal format with structure check only – all must pass. Otherwise QA won’t start.
In the next stage, rules are compiled into “byte-code” and fully syntax-checked. Detailed info about errors is provided. Invalid filters are silently removed from drop-down menu.

 

DSL (domain specific language) for creating filters will be described later.

[Łukasz997 117]

2. User Interface (UI)

Please login or register to see this image.

/monthly_2025_10/mess.jpg.10e2a5b0c956ef8e30b7c349ff6512ca.jpg" />

 

Section 1 - Messages from QA itself (e.g. filter compilation errors or warnings). It disappears after a while.
Buttons have hidden functions when long-pressed. Help is displayed after long-pressing ON & Unmute.

 

Section 2 – short QA status. First two lines are self-explanatory. The third line reflects filters and need some knowledge about filters:

• for ID 111 → all message types are hidden,

• for 222 → warnings, debug and trace messages are hidden,

• for 333 → there's complex rule that will be evaluated when message of type debug or trace comes from this source (eval result true = show, false = hide),

• for all sources -> there's also complex rule but only for warning.
  Everything else is shown (OTHERS: ...).

 

 

Section 3 – list of messages, ordered as follows (simplified explanation):

1. Storms on top

   • red-hot issues (in storm, not yet informed),

   • then stormed but already informed.

2. All remaining (non-storm) items.
   Items are sorted by last event time, newest at the top (descending)

In first column there's message source - ID or name.
If it's a scene -> color is orange. Devices, QA -> plain color. Named sources as 'ZWAVE' -> blue.
As table can contain not only errors, at begin of second column is colored small dot with letter (🅓🅣🅦🅔) - it means: DEBUG, ERROR, WARNING, TRACE; Hard to read, but colors are the same as in www console.
same as TYPES in www console 

 

All rest columns has the same meaning as in auto:cerr - see description in that thread.

Please remember that QA always tracks (memorises) all messages - but when you change the filter, some may show up and others may disappear.

 

Edited October 1, 2025 by Łukasz997

[Łukasz997 117]

3. Filters DSL

Little complicated but interesting and powerful feature, I hope.

In the __Filter_config file you will find FILTERS = {...} with definitions of built-in filters. You can add filters or change them as you want.
Each filter block is enclosed in [[...]] and has the following syntax:

Please login or register to see this code.

• setname must be the first non-empty line. This is the name under which the filter will be visible in the drop-down menu.

• default must be the second (valid values are allow or deny). It means: “when there’s no rule defined for a given source of message – this is the default behaviour”.

  • allow → show message

  • deny → hide message (not visible in UI, no PUSH).

• These two lines are mandatory – missing, invalid, or duplicated entries cause a fatal error at QA start.

 

Next, there are one or more rule definitions.

Please login or register to see this code.

KEY is the source of messages – like TAG in the www console.
It can be:

• a number representing the ID of QA, scene, or device (just 456, 768 etc., without QUICKAPP or SCENE),

• 'NAME' of source like 'ZWAVE', 'ZIGBEE',

• * (star) representing all sources of messages.

SCOPE is like TYPE in the www console. It can be either:

• *

• d, t, w, e (like DEBUG, TRACE, WARNING, ERROR).
  A single letter or a combination of more is allowed.

Meaning: limit messages to given type(s), or * acts as all types (like dtwe).

Examples:

Please login or register to see this code.

 

Edited October 1, 2025 by Łukasz997

[Łukasz997 117]

RULE (what to show/hide within that scope)

Either:

• + means ALLOW all for that key & scope 

• - means DENY all for that key & scope

or a boolean expression containing:

• text – word(s) we want to match in a message. Spaces inside are allowed, leading and trailing spaces are trimmed. Case-insensitive.

• text can contain exact characters (message == exact text) and wildcards:

  • * – matches any characters

  • ? – matches any single character

Expression can also contain operators:

• & – logical AND

• | – logical OR

• ! – logical NOT

• precedence: ! > & > | (first !, last |)

• () – to group and change operator priority

Examples:

Please login or register to see this code.

 

It can be as many rules per block as you want.
Please consider that only one rule per (key, type) is allowed!
For the same key (* / ID / NAME) and the same message type, duplicates are rejected (error).
So, combine conditions in one expression.

Default mode

Rules can only be useful in combination with default – to create a blacklist or whitelist of messages.

• default = allow → everything is shown unless you hide it (blacklist mode).

• default = deny → nothing is shown unless you allow it (whitelist mode).

When a message comes in, the filter system doesn’t test all rules.
It picks the single most specific rule that matches the message (precedence: ID > 'NAMETAG' > * global).

That one rule alone decides:

• true = show,

• false = hide.

If no rule exists for this message, the default value (allow/deny) applies.

 

So, if arrives a message from ID=100 - rule with this ID (100 = ...) decides if exists (and no further evaluation), otherwise - if exists * (global rule) - it decides (precedence allow ID and * to coexists) and last - fallback is default field (always exists). This provides a clear model for resolving conflicts.

Edited October 4, 2025 by Łukasz997

[Łukasz997 117]

How to think about filters (practical guide)

Step 1 – Pick default mode

• default = allow → everything is shown unless you hide it (blacklist mode).

• default = deny → nothing is shown unless you allow it (whitelist mode).

Step 2 – Write rules

In blacklist mode (allow): write rules with NOT (!) to hide unwanted messages.
Example:

Please login or register to see this code.

In whitelist mode (deny): write rules without NOT to allow what you need.
Example:

Please login or register to see this code.

 

Don’t mix styles

• In allow mode, positive rules are redundant (everything is shown by default).

• In deny mode, negative rules are redundant (everything is hidden by default).

Example:

Please login or register to see this code.

 

When you type:
111 = *: abcd
it means that and message from 111 is EXACTLY abcd
If you mean: contains abcd, use *abcd*, ends with abcd, use *abcd

Edited October 2, 2025 by Łukasz997

[jgab 2,641]

admit that it's fun to write compilers?  

[Łukasz997 117]

7 godzin temu, jgab napisał:

admit that it's fun to write compilers?  

Yes, it is a wonderful adventure in computer science. The mini-language itself is quite simple, calculator-like - but it still has its own grammar, lexer, and AST. I had absolutely minimum prior experience with compiler construction, so I’m especially happy with how it came out.

 

In the end it’s just a bit over a hundred lines of code, instead of pages filled with if-find-match-then logic.

 

Right now, the parser only analyzes what comes after the ‘:’. Everything before the ‘:’ - a later addition (since filters originally had a different, less user-friendly format) - is translated straight into that old notation without going through the grammar. I could now eliminate this step, because I’ve already adapted the parser rules to handle the entire filter line, but doing a direct re-integration seemed a bit tricky than translating into the legacy format.

But that wasn’t the key part of the project. The crucial thing wasn’t so much inventing the syntax as inventing the rules by which the filters operate. Without taking into account the possibilities of collisions or the need for fallbacks, one could easily write the whole system, test it, and then conclude it belongs in the trash. I was one step away from that myself - this was my first attempt at building something like this.

[jgab 2,641]

Yes, I guess technically it's not a compiler, more a parser-evaluator as you interpret the ast - on the other hand the definition of a compiler is not so stringent...

Do you allow spaces inside the patterns? Why not use the lua string match format directly?

[Łukasz997 117]

I would say: this is parser-evaluator with pattern pre-compile. Spaces are allowed in the middle of pattern, edge spaces are cut off (*settimeout error* will work as expected).

 

Yes, you’re absolutely right – with plain string.match this can be achieved too. Under some assumptions maybe its simpler. But there are a few catches: Lua patterns don’t support alternation (|), logical operators, or proper grouping – so for anything beyond a single substring match you quickly end up writing multiple :find calls glued with ifs. They also have quirky escaping rules (% everywhere…) which are not very user-friendly.

 

At the end - this is written like thousands of server messages analyser with amount of filters to compile per one second... 

 

But everything that was secondary...

I did it this way precisely because it was an attempt to tackle the problem as a formal language. I always wanted to write a small parser/engine, and earlier attempts never had any real application – they were just toys. Here it feels useful, even if only for a narrow group of users.

[Łukasz997 117]

Due to changes in fibaro.alert(...) mentioned by @jayrock, I am attaching an updated version of auto::cerr2.
The change is minor, but without it a::c2 will generate an error on the latest firmware 5.200.

Please login or register to see this attachment.