TableMan: HTML Table Server solution

[Łukasz997 132]

Welcome to the completely unnecessary applications series! Part N+1.

TableMan: An HTML Table Server for Nice/Fibaro HC3

The Fibaro UI has quite a few limitations. But it also has some real possibilities: it supports simple HTML tags, it can display SVG, and that opens up quite a few ways to present data sensibly.

 

One of the simplest and most commonly used ways of presenting data is a table. We can see tables in many user applications. I think quite a few people have written their own libraries for handling HTML tables and then include them in their next QuickApps.

I did the same. Over time I ended up with five “generations” of increasingly sophisticated libraries, plus one mini version. One of those libraries used SVG instead of HTML and could display tables stacked on top of each other like windows (they even cast shadows!). This turned out to be very useful, for example, for preference screens (very long preference sections in many applications I have seen are a problem).

The last HTML approach already had table filtering and sorting, cell merging, formatting, a two-level cache, and a few other bells and whistles. The code exceeded 900 lines.

 

There were 15 user applications, and each of them had to carry those 900 lines and meet their memory requirements. That meant about 13,000 lines of repeated code: some applications used this part, others used that part, and none of them used everything. I do not even want to mention how much work updating it required. And yes, bugs did happen.

 

That led to the question: can tables in Fibaro be done in a client-server model? Can large libraries included in every application be replaced with one central QuickApp that handles tables for everyone, while leaving only a thin client inside each application? It turned out that yes, they can. After a few weeks of work, a working version of TableMan was born.

 

Command Language

The assumption was simple: it should be possible to build complex tables with single lines of code, and writing a value to a table cell should be almost as simple as print. The whole thing comes down to a small command language. It is a set of simple commands that can be used to construct and update tables. The currently available commands are:

 

link, display, hide, delete, set, style, setRow, setCol, styleRow, styleCol, styleAll, clear, clearCell, clearValue, clearStyle, header, merge, unmerge, addRows, deleteRows.

 

You can combine several tables into one view, merge ranges of cells, format individual cells, whole rows and columns, change colors, clear values or styles, add and delete rows. The server can also automatically choose a contrasting text color for the background. Thanks to this, quite sophisticated tables can be created in a very simple way.

Typical usage looks like this (first you need to register with the service and create a table using new😞

Please login or register to see this code.

 

Most commands accept many attributes that make it possible to control practically every aspect of the table’s appearance: text and background colors, font and font size, horizontal and vertical alignment, and so on.

Importantly, only the explicitly provided attribute is changed. Everything else remains as it was. This makes it possible to shape the table’s appearance incrementally, without having to set all attributes at once.

 

Server

It is one central QuickApp that receives commands from its clients. It stores private table data for each client, executes commands, and, when needed, updates the client’s UI directly.

There is no classic return communication here. The client sends a command (fire and forget), and the server does the rest. There are no confirmations, negotiations, callbacks, or additional protocol. If something needs to be displayed, the server sends ready HTML directly to the appropriate UI element of the client.

This is a deliberate simplification. It is hard to expect execution confirmation from print.

The server has extensive error diagnostics, but it does not return errors to the client. It logs them on its own console. There are more than thirty types of errors, so it is visible whether the problem is an invalid cell range, a nonexistent table, bad parameters, and so on.

It also has its own administrative tables: it shows clients, their activity, number of operations, traffic profile, and the load generated by individual applications and tables. This is very helpful for finding the chattiest applications and trimming them down.

Writing to a table does not automatically display it. So you can build or update a table with many commands (at minimal CPU cost), and only at the end execute the display command.

 

Client

On the application side there are two client variants.

The first one is very simple. It is suitable for applications that rarely update data or have small tables. It has about 80 lines of code. It collects commands into batches and sends them to the server in one go, to reduce the number of hub.call

 

The second client is larger, about 220 lines, but it contains a simple yet very effective cache mechanism: if a given command has already been sent to the server and nothing in it has changed, the client does not send it again. It also cuts off display if there were no real changes in the table. This greatly reduces communication for semi-static tables, where some cells do not change and only individual values are updated. The application no longer needs its own layer for limiting communication.

The cache is fast. It is based on calculating a hash, which is cheaper than ordinary cache searching: the table index is the command and address (which is also based on hashing, just internally in LUA) and the value is a hash of the parameters. One has to be very careful, because once every ~30 years or so there may be a hash collision (2^32). The effectiveness in reducing communication is impressive, considering the simplicity of the solution.

 

Results

The work is still in progress, but I think the results are very good.

Thanks to the optimizations, the CPU load of the hub did not increase, or increased only very slightly. On the other hand, for reasons that are not entirely clear, memory demand visibly calmed down. There is about six times less code (overall), but the data still has to be stored somewhere.
The consequence of no back-communication to client is that cells data cannot be read, only write. If needed it's application responsibility to remember such data. But The client have to be very simple, so it's the price. The second reason is that it would have to be asynchronous, which I personally do not like.

 

The most important benefit, however, is architectural.

The server model allows functionality to be added and improved without interfering with client applications. This is achieved by the fact that the client has only three functions, completely independent of the command language: register, new, and exec. Client is completely stupid. It does not know the commands, parameters, it checks nothing. 

Only sends an "intention": execute the commands, show the result. Rest is server-sided.

 

If a new feature appears in the server, an application can use it without replacing the client. It is enough to call the new command through exec.

Only very large architectural changes would require replacing the client.

 

Please login or register to see this attachment.

 

Please login or register to see this attachment.

 

 

 

 

Edited June 27 by Łukasz997

[Christb 33]

Will you share your code when finished?

[Łukasz997 132]

 

13 minut temu, Christb napisał:

Will you share your code when finished?

I will think about it and answer soon.

[Łukasz997 132]

Let me answer at some length, because it may indeed seem strange that I have been showing off various ideas, but somehow never include the code.

The programs I write, and in general the area of programming that interests me, are not particularly popular in our community. For example, I am not interested at all in creating integrations with various devices, which are desired by the community, and rightly so. What I do interests, I think, only a handful of people, who nevertheless provide me with a certain amount of support and satisfaction through their likes.

Before I sit down to program, I spend many hours constructing the contract, deciding how it is all supposed to work. I am not a professional programmer, only an amateur, but this is what the DNSfD project taught me: a concept that is wrong in one small but crucial place means either throwing the code away or going through the torments of Tantalus while trying to “repair” it.
 

You should also know that this is not a simple program, such as a controller for blinds or a fireplace. It is around 3,000 lines of code. It would have to be tested thoroughly, because bugs and failures in a server-based model can be mysterious and difficult to track down. I didn't do this to the end, yet.

Creating client-server software requires a particular kind of experience. It is an area which, it seems to me, really follows its own rules. Something always appears that you had not anticipated, even after spending a long time thinking about how everything should work. TableMan happens to be a relatively simple project, although it has quite an extensive “construction language.” But even there I have already added things I had not expected.

Another project, DNSfD, showed very clearly how difficult it is to explain to the community a concept that is systemic, tool-oriented. It also showed that more sophisticated, automated and elegant solutions do not necessarily win. Not because they are bad, but because first you still have to explain why they exist at all and how they are supposed to be used.

Documentation would also have to be written. AI would probably help with that, but it would not do everything. I simply cannot find the time for it anymore. It would probably be a solid 20 to 30 pages in two languages.

On the other hand, the era of sweet idleness in my life came to an end, suddenly and unexpectedly, and I had to turn programming into an additional source of income.

In short, in my situation these would have to be paid solutions, so that I could be fully available to users. That, in turn, creates other problems which I am not actually in a position to deal with.

I will think some more about this particular project, but for now I have to admit that, as a general rule, I had no intention of publishing these programs. I only intended to publish information about them, which may also be valuable in some way.

I apologize for this somewhat chaotic and, as usual, long reply, but I am simply writing what I think.

Edited 1 hour ago by Łukasz997