TableMan: HTML Table Server solution

[Łukasz997 117]

Welcome to the completely unnecessary applications series! Part N+1.

TableMan: An HTML Table Server for Nice/Fibaro HC3

The Fibaro UI has quite a few limitations. But it also has some real possibilities: it supports simple HTML tags, it can display SVG, and that opens up quite a few ways to present data sensibly.

 

One of the simplest and most commonly used ways of presenting data is a table. We can see tables in many user applications. I think quite a few people have written their own libraries for handling HTML tables and then include them in their next QuickApps.

I did the same. Over time I ended up with five “generations” of increasingly sophisticated libraries, plus one mini version. One of those libraries used SVG instead of HTML and could display tables stacked on top of each other like windows (they even cast shadows!). This turned out to be very useful, for example, for preference screens (very long preference sections in many applications I have seen are a problem).

The last HTML approach already had table filtering and sorting, cell merging, formatting, a two-level cache, and a few other bells and whistles. The code exceeded 900 lines.

 

There were 15 user applications, and each of them had to carry those 900 lines and meet their memory requirements. That meant about 13,000 lines of repeated code: some applications used this part, others used that part, and none of them used everything. I do not even want to mention how much work updating it required. And yes, bugs did happen.

 

That led to the question: can tables in Fibaro be done in a client-server model? Can large libraries included in every application be replaced with one central QuickApp that handles tables for everyone, while leaving only a thin client inside each application? It turned out that yes, they can. After a few weeks of work, a working version of TableMan was born.

 

Command Language

The assumption was simple: it should be possible to build complex tables with single lines of code, and writing a value to a table cell should be almost as simple as print. The whole thing comes down to a small command language. It is a set of simple commands that can be used to construct and update tables. The currently available commands are:

 

link, display, hide, delete, set, style, setRow, setCol, styleRow, styleCol, styleAll, clear, clearCell, clearValue, clearStyle, header, merge, unmerge, addRows, deleteRows.

 

You can combine several tables into one view, merge ranges of cells, format individual cells, whole rows and columns, change colors, clear values or styles, add and delete rows. The server can also automatically choose a contrasting text color for the background. Thanks to this, quite sophisticated tables can be created in a very simple way.

Typical usage looks like this (first you need to register with the service and create a table using new😞

Please login or register to see this code.

 

Server

It is one central QuickApp that receives commands from its clients. It stores private table data for each client, executes commands, and, when needed, updates the client’s UI directly.

There is no classic return communication here. The client sends a command (fire and forget), and the server does the rest. There are no confirmations, negotiations, callbacks, or additional protocol. If something needs to be displayed, the server sends ready HTML directly to the appropriate UI element of the client.

This is a deliberate simplification. It is hard to expect execution confirmation from print.

The server has extensive error diagnostics, but it does not return errors to the client. It logs them on its own console. There are more than thirty types of errors, so it is visible whether the problem is an invalid cell range, a nonexistent table, bad parameters, and so on.

It also has its own administrative tables: it shows clients, their activity, number of operations, traffic profile, and the load generated by individual applications and tables. This is very helpful for finding the chattiest applications and trimming them down.

Writing to a table does not automatically display it. So you can build or update a table with many commands (at minimal CPU cost), and only at the end execute the display command.

 

Client

On the application side there are two client variants.

The first one is very simple. It is suitable for applications that rarely update data or have small tables. It has about 80 lines of code. It collects commands into batches and sends them to the server in one go, to reduce the number of hub.call

 

The second client is larger, about 220 lines, but it contains a simple yet very effective cache mechanism: if a given command has already been sent to the server and nothing in it has changed, the client does not send it again. It also cuts off display if there were no real changes in the table. This greatly reduces communication for semi-static tables, where some cells do not change and only individual values are updated. The application no longer needs its own layer for limiting communication.

The cache is fast. It is based on calculating a hash, which is cheaper than ordinary cache searching: the table index is the command and address (which is also based on hashing, just internally in LUA) and the value is a hash of the parameters. One has to be very careful, because once every ~30 years or so there may be a hash collision (2^32). The effectiveness in reducing communication is impressive, considering the simplicity of the solution.

 

Results

The work is still in progress, but I think the results are very good.

Thanks to the optimizations, the CPU load of the hub did not increase, or increased only very slightly. On the other hand, for reasons that are not entirely clear, memory demand visibly calmed down. There is about six times less code (overall), but the data still has to be stored somewhere.

 

The most important benefit, however, is architectural.

The server model allows functionality to be added and improved without interfering with client applications. This is achieved by the fact that the client has only three functions, completely independent of the command language: register, new, and exec. Client is completely stupid. It does not know the commands, parameters, it checks nothing.

Commands are sent like this:

Please login or register to see this code.

 

If a new feature appears in the server, an application can use it without replacing the client. It is enough to call the new command through exec.

The server can grow, receive new capabilities and fixes, while the client remains unchanged. Only very large architectural changes would require replacing the client.

 

Please login or register to see this attachment.

 

Please login or register to see this attachment.