Visual Studio Code (vscode) for QuickApp development

[jgab 1,972]

This setup is designed for QA development using Visual Studio Code (VSCode). It includes an emulator, allowing you to run and debug QAs directly in VSCode. This setup is compatible with HC3, offering features such as code completion, syntax highlighting, and workflow tasks for downloading, editing, packaging, and uploading QAs to/from HC3.

Tested on Windows 11 and MacOS, this setup should also work on Linux.

 

You can download the repository from

Please login or register to see this link.

. Documentation, still in progress, is available

Please login or register to see this link.

. Additionally, there's a video tutorial on installing this setup on Windows 11.

 

 

The setup in VSCode looks as shown in the image.

Please login or register to see this attachment.

 

This configuration allows you to develop and run a QuickApp (QA) offline on your PC, Mac, or Linux system, and interact with the HC3 in a controlled manner.

There's also a simple Web user interface for interacting with the QAs and monitoring events
(available at  

Please login or register to see this link.

)

 

Please login or register to see this attachment.

 

To get started:

1. Install Python 3 on your machine. NOTE! Only Python v3.11 and 3.12 is supported. 
2. in settings.json set

   Please login or register to see this code.

    

3. fibpython should point to your executable python. In the example it points to a python venv in the workspace, but it could point anywhere you have installed python.
   There is a settings.templ provided as a base for your own settings.json (should be in the .vscode directory)

4. To test if the installation is ok , in the vscode terminal window, type what you set the fibpython path to. ex:

   Please login or register to see this code.

   and see if it works.
5. Install the required Python libraries from requirements.txt using >pip install -r requirements.txt.
   Make sure they install for the python interpreter that is pointed out by fibpython.
6. Create a config.json file with credentials to access the HC3, based on the config.json.example.
7. Install the VSCode extension "

   Please login or register to see this link.

   " by Tom Blind.
8. Create a QA file in the directory, choose the launcher "Fibenv QA file (remote)," and start debugging with F5.
9. For more examples, refer to the files in the examples/ directory.

 

The included .gitignore file excludes the ./dev and ./test subdirectories, allowing you to test your own code without affecting the rest of the repository. For installing Python libraries, it's recommended to create a virtual environment in the folder first.

 

This setup includes a Python wrapper for the Lua runtime (Lupa), addressing dependencies on luasocket and others, eliminating the need for Lua installation on your machine. To instruct the emulator about the type of QA, directives similar to TQAE can be used in the QA file, but with slight differences, as shown in the example.

 

Ex.

Please login or register to see this code.

Note the --%%remote directive

It instructs the emulator that it's ok to call device 788,789 o the HC3. As a default, the emulator treats all resources as local (we can read from HC3 but then treat them as local copies) and we enable resources we want to interact with on the HC3 as 'remote'. This goes for other resources also like 'globalVariables'.

 

The setup integrates with the Lua debugger, allowing for breakpoint setting and more. Note that this project is still in progress, and updates will be provided as improvements are made.

 

The emulator UI can be accessed at

Please login or register to see this link.

, with a Swagger page for implemented APIs at

Please login or register to see this link.

. The port (5004) can be changed as needed.

 

Supported APIs include various fibaro and net functions, along with api calls and plugin functions. The setup also includes classes for QuickApp development and management.

 

Supported APIs

Please login or register to see this code.

 

For your own QA development have the repo 'fibemu' downloaded in a separate directory. Then create your own vscode development directory, and initialise that as a git repo if you want.

With that open, do File->Add Folder to Workspace, and add the fibemu folder.

This is what Vscode calls a "multi-root" workspace and it.

Create a lua (QuickApp file) in your directory.

First line in your QuickApp file should be the offset to your directory

Please login or register to see this code.

So if you have a multi-root workspace now that looks like

Please login or register to see this code.

You set the root to ../myDirectory/

Then press F5 to start to debug. The launch commands and other will be taken from the fibemu folder.

This way you can pull down new version of fibemu without messing up your own directory and development.

Also, don't forget to setup your config.json with credentials (copy from fibemu/config.json.example)

 

Please login or register to see this link.

(may include a video in the future)

 

Implementation

• Please login or register to see this link.

  .

• Please login or register to see this link.

 

Workflow

There are some defined vscode Tasks that help in remotely uploading and updating the QA on the HC3 from within the vscode environment 

• "QA, download fqa" downloads an QA from the HC3 and saves it as a .fqa file. The task will prompt for deviceID and path where to store. The path/dir needs to exist
• "QA, download and unpack" downloads an QA from the HC3 and saves all QA files as .lua files. It also adds fibemu headers in the main file so it can be opened and run with the emulator . The task will prompt for deviceId and path where to store. The path/dir needs to exist
• "QA, upload" will upload the QA to the HC3. It will prompt for QA file. If '.' is given as argument it will upload the current opened file. This will create a new QA, with a new deviceId on the HC3.
• "QA, update" will try to update QA files, viewLayout, uiCallbacks, and quickAppVariables of an existing QA on the HC3. If '.' is given as argument the file must have set the fibemu header --%%id=<ID> so it knows what QA to update. One can also give the deviceId of the QA on the HC3 that should be updated. This is convenient when developing and avoiding new IDs being "consumed". Sometimes when you update a QA you would not like to update the quickAppVariables. In that case give '-' instead '.' for the current opened file, or -deviceId for an exiting QA on the HC3.

Please login or register to see this link.

 

Scripts

An advantage with the emulator is that we have access to more lua functions than on the HC3 which allow us to write some maintenance scripts

• Please login or register to see this link.

  . Backs up QAs from the HC3 to a local directory, keeping the 3 last versions

• Please login or register to see this link.

  . Packs a development file to a .fqa, initialises some quickAppVariables, adds readme.txt file and zips it to an archive.

 

Known issues

• While the QA is running, break-points can't be added. This is a limitation of the debugger used. Just add the break-point and restart the QA.
• When the emulator crashes, it may leave a process open that keeps the port 5004 in use. The emulator will complain at restart that the port is already bound. You may need to manually kill the process.
  On Mac:  >kill -9 $(lsof -ti:5004)

Please login or register to see this attachment.

Edited May 9 by jgab

[Joep 181]

Nice! What happened that you go the Visual Studio Code route? 😀

 

I tested some development with VScode and GitHub CoPilot and it blows your mind. It speeds up creating the QA classes and functions framework very fast.

Edited June 27, 2023 by Joep

[jgab 1,972]

4 minutes ago, Joep said:

Nice! What happened that you go the Visual Studio Code route? 😀

Well, vscode has started to grow on me  
Also started to develop other stuff in other languages so I would like to keep everything in the same development environment. TQAE is runnable in vscode now but a specially developed vscode emulator will have some advantages... 

6 minutes ago, Joep said:

Nice! What happened that you go the Visual Studio Code route? 😀

 

I tested some development with VScode and GitHub CoPilot and it blows your mind. It speeds up creating the QA classes and functions framework very fast.

yes, and we can add fibaro api completion/tooltip with the lua language server

Edited June 27, 2023 by jgab

[jgab 1,972]

Improvement,

- support for child devices

- simple UI to test buttons/slides/labels

 

It runs EventRunner4 now so it's pretty usable...

[michalt 3]

@jgabas usual thanks for your work. For me it also required jinja2 to run so it might be a good idea to add it to requirements. 

[jgab 1,972]

50 minutes ago, michalt said:

@jgabas usual thanks for your work. For me it also required jinja2 to run so it might be a good idea to add it to requirements. 

Yes, right. Updated it now.

I used pipreqs to generate requirements.txt but it missed jinja2 - and so did I.

[jgab 1,972]

net.TCPSocket and net.UDPSocket in place. 

[jgab 1,972]

Current APIs implemented (click to make it larger)

Please login or register to see this attachment.

[jgab 1,972]

The environment is a mix of Python and Lua. The blue parts in the picture below is Python and the green is implemented in Lua.

The Lua debugger is started in the green environment when we run the emulator and the vscode IDE can then lets us debug our QA like any other Lua code.

The model is based on that we have a copy of the HC3 resources in our own "Resource DB" and all manipulation of devices, globalvariables, rooms, sections etc are done to towards that database. 

If some resource is manipulated in the HC3, we get an event (refreshStates) and update our copy in our database.

If a resource is marked as "remote", e.g. we are allowed to change the resource on the HC3 too, we will "sync" back the change from our database, back to the HC3.

Ex. if we set a globalvariable in the emulator and it is marked "remote" we will  sync that back to the HC3. Same for device property changes etc.

The exception is fibaro.call(ID,...) that will call the device on the HC3 if the device is marked "remote".

 

Please login or register to see this attachment.

 

The picture shows one QA, but the emulator can load several QAs and run them at the "same" time. There will still be only one fibemu thread, and the QAs will share the same timer queue (e.g. they run as Lua coroutines). However, they will have separate fibaro.sleep that don't block each other.

Edited July 2, 2023 by jgab

[jgab 1,972]

Version bumped from 0.0.3 to 0.3  

Lots of small and large bug fixes.
TCPSocket, UDPSocket and WebSocket tested and they behave the same as on the HC3 as far as I can see. Also started to build up a collection of test QAs in the example directory.

 

Shifting my focus from TQAE to this project...

Edited July 3, 2023 by jgab

[jgab 1,972]

Tips & tricks

• Debug console colors can be setup in config.json (see config.json.example). If you run vscode in dark mode that can be nice to modify to your liking.
  A quick fix is to set the config parameter dark=true, then the text color will be set to light grey.
• All system logs in the Debug console is tagged as [SYS  ] or [SYSERR] and can thus easily be filtered out in the Debug console's filter field with !SYS
• Break-points in Lua can not be set while the program is running. To add a new break-point, add and restart the QA. This is a limitation of the Lua debugger used. Break-points can be removed while the program is running though.
• The are 3 ways to add configuration. In ~/.fibemu.json. That file, if available, is read first.Can be a good place to store HC3 credentials like ip, user,password.
  After that, config.json in the vscode project directory is read in and merged with the previous config data.
  Lastly, config parameters set in the QA source file is added to the config data.
• The vscode launch.json file contains a number of different launch options when debugging. There is usually a "remote" and a "local" version, and the "local" version will run without accessing the HC3. Good to have if developing where there is no access to an HC3, like on the beach...
• The launch option marked with "emu files" include the Lua emulator files in the path for the debugger so that one can step into these files. Normally, you don't want that, instead you only want to step through your own code. Ex. with "emu files" and you step into a 'setTimeout' you will step into the setTimeout implementation...
• There are also 2 python launch options used to run the program with the python debugger ("remote" and "local"). Mainly used for debugging the framework. However, you can't see your Lua code this way.
• There is an option in the launch file to add an Lua init file with the "-I" option. This is a Lua file run before the QAs. It can be a place to setup som "virtual" QAs or other things....

Edited July 3, 2023 by jgab

[jgab 1,972]

Added support for Lua Language Server.

Install vscode extension "Lua Language Server" by sumneko

In ${workspace}/.vscode/emufiles/lspaddon are the setup and libs for custom API IntelliSense support.

I've started to add some stuff on QuickApp but if anyone is willing to dive deeper into this I'm happy for any support.

Note, you need to add the settings from ${workspace}/.vscode/.settings.json to point to the lspaddon dir.

[jgab 1,972]

Lots of bug fixes and improvements the last days.

I have added 4 tasks to the vscode project.

• "QA, download fqa" downloads an QA from the HC3 and saves it as a .fqa file. The task will prompt for deviceID and path where to store. The path/dir needs to exist
• "QA, download and unpack" downloads an QA from the HC3 and saves all QA files as .lua files. It also adds fibemu headers in the main file so it can be opened and run with the emulator . The task will prompt for deviceId and path where to store. The path/dir needs to exist
• "QA, upload" will upload the QA to the HC3. It will prompt for QA file. If '.' will upload the current opened file. This will create a new QA, with a new deviceId on the HC3.
• "QA, update" will try to update QA files, viewLayout, uiCallbacks, and quickAppVariables of an existing QA on the HC3. The file must have set the fibemu header --%%id=<ID> so it knows what QA to update. This is convenient when developing and avoiding new IDs being "consumed"....

Please login or register to see this attachment.

Edited July 13, 2023 by jgab

[michalt 3]

Hi @jab, very nice! Will experiment a bit and let you know my thoughts! 

[jgab 1,972]

2 hours ago, michalt said:

Hi @jab, very nice! Will experiment a bit and let you know my thoughts! 

Great!, let me know if there are any strangeness - like strange error messages that are hard to understand. Because the "QA framework" is written in a mix of python and Lua, sometimes the python part can throw error that can be hard to map to what's happening in your Lua code. I'm trying to close those "holes" to keep the "Lua" feeling when coding...

I will also write a better "manual" in the weeks that comes as the features implemented are stacking up faster then I post here...

[michalt 3]

Hi @jgab for this moment the only thing that found is non deterministic error:

Error : :onInit() [string "lib/fibaroExtra.lua"]:833: attempt to call a nil value (field 'encodeFast') - restarting in 5s[0m

 

I'm running EH4 in emu and this happens sometimes when I run to test the rules, usually this gets resolved just by restarting the runner - its annoyance but noting blocking. 

[jgab 1,972]

2 hours ago, michalt said:

Hi @jgab for this moment the only thing that found is non deterministic error:

Error : :onInit() [string "lib/fibaroExtra.lua"]:833: attempt to call a nil value (field 'encodeFast') - restarting in 5s[0m

 

I'm running EH4 in emu and this happens sometimes when I run to test the rules, usually this gets resolved just by restarting the runner - its annoyance but noting blocking. 

I think that could be a fibaroExtra problem... what version do you run? There were some issues some time back with modules not loading in the right order.

[michalt 3]

@jgab

fibaro.FIBARO_EXTRA = "v0.956"

Edited July 15, 2023 by michalt

[jgab 1,972]

40 minutes ago, michalt said:

@jgab

fibaro.FIBARO_EXTRA = "v0.956"

Yes, we are up to v0.960 now. Good idea to update.

Ok, the other thing to check for is that none of the fibaroExtra functions are available before we enter the QuickApp:onInit() function.

The reason is that the order of which the files are loaded are more or less random (the non deterministic behaviour you experienced)

 

However, in this case I think it's enough to update.

 

[PeterV959 6]

I installed Python, the Python requirements, and VS Code along with the Tom Blind debugger extensions. Then I git cloned the fibemu repository and I am trying to run any of the examples in the examples folder. I get the following errors on the Debug Console:

 

 

Please login or register to see this attachment.

 

So it looks like it can't find 'websocket'. What have I missed?

 

Looks like an exciting environment. Can't wait to try it out.

 

Thanks

 

Update: I saw @jgab had updated something three hours ago so I tried to "git fetch" fibemu. After that I ran the pip install for requirements. Still no effect. But when I poked around I saw a requirements.txt file in the ./.vscode/fibemu folder. I tried to install that and was rewarded with the addition of websockets. I am now up and running, although I need to turn off colors for the debug console output. I am using a dark high contrast theme and I am getting black letters on a black background. Should be an easy fix, right? I will poke around and figure out how to correct that for my environment (and maybe learn something in the process).

 

Thanks @jgab

Edited July 17, 2023 by PeterV959
Update