AI Agent QuickApp

[jgab 2,645]

I have put together a small Lua SDK that can be used in QuickApps to call openAI.

It supports tool/function calls so the AI can ask back to the QA to carry out actions.

Ex. If I call

Please login or register to see this code.

we get the result

Please login or register to see this attachment.

Here I run it in my emulator offline, but it works the same on the HC3...

 

I use the gpt4.1-mini that is one of the cheapest on OpenAI

The AI calls back to 3 functions,

1. get_current_date_and_time, so it knows what time it is on the HC3

2. device_name_to_id so it can translate the QA name "kkkk" to a deviceId (I sometimes don't name my QAs that well...)

3. schedule_action, with the tool/function it would like to run at a later time.

 

The schedule_action, written in Lua will start a setTimeout to run the function at the given time...

 

So the AI is quite clever breaking down the request in the steps needed to fulfill the scheduling.

It also decided that Saturday afternoon should be 3PM, and that's ok.

 

Running this took 2147 tokens at the price of 1/10 of an US cent, with the current pricing for the gpt4.1-mini model.

I have uploaded $5 to my OpenAI account so I should be able to run it ~5000 times. 

Is it expensive? Well maybe, maybe not. I guess the usefulness will decide, and this example is probably not the most useful...

 

I have also tried with a local hosted AI (llm) with Ollama. It's for free but the performance on models possible to run on a local PC

is not that great, and I had trouble getting consequent results (function calls) back from he AI....

 

I'm still thinkering with the code, but the main loop looks like

Please login or register to see this code.

 

and the tool definitions

Please login or register to see this code.

schedule_task doesn't do anything here but the real implementation is pretty straight forward...

Edited June 13, 2025 by jgab

[jgab 2,645]

There are a couple of ways/strategies to use AI in HA context.

 

1. Like above, when we ask the AI to automate something in natural language.

    Things that should happen in the future need to be scheduled. AI can have "memory"/context that remembers the rule, but it needs to be reminded that the time has happened (We don't want to tell it the time every minute everyday).

    Therefore it makes sense to ask the AI to schedule it at a specific time using our schedule function. We can then remind the AI when the time is up.

    We can also ask for scheduling at recurring time intervals, like "every Saturday".

    A complication is 'exceptions'. "every Saturday except Hollidays". In the example above the AI schedules function calls for the given time, and if the function calls have to be able to express exceptions it can become complex.

    It may be better to ask the AI to break the future request into a time in json and action, in natural language. {time="15:00",day="Saturday",recurring=true}, "turn on lamp in kitchen if it's not a Holliday". Then our scheduler will ask the AI to "turn on lamp in kitchen if it's not a Holliday" when it becomes 15:00 on a Saturday...
 

    Trigger rules may be easier. We tell the AI that the lamp in the kitchen should be turned on when the sensor is breached except on Saturday mornings. We later tell it that the sensor is breached and it should carry out appropriate actions. Generalized, a scheduled rule is a trigger rule, where the trigger is a time trigger.

 

2. Have the AI monitor all sensors and actuators in the Home over time, and propose new actuator actions given the past history. Here the issue is to have enough history/training data. There would also probably be a need for some kind of feedback system to tell the AI if its actions was right. This is a bit more complicated to code but all AI tools are there....

 

3. We could ask the AI to create a Scene for us, that we specify in natural language. HASS has plugins that does this, and it's reasonable easy has HASS has a common YAML/json format for actions/rules. The difficulty here is to understand if the AI has solved the task satisfactory (also an issue with 1.) - but when your wife gets mad you can blame the AI...

 

4. Use AI's coding skill to verify and debug QA code that we write and highlight potential problems. Some of the AI models are quite good at coding incl. Lua. So one could have an AI that watches if a QA is saved and looks at the code and makes warnings of something looks strange. Coding in an IDE like Cursor or VSCode with agent support already gives us some of this today (offline)

 

5. Have the AI to monitor the system overall. Load, memory usage, errors from QAs /Scenes etc. and warn when things start to look bad and even maybe be able to diagnose and find faulty QAs/Scenes...

[jgab 2,645]

I have played a little more with the AI for HC3. 

The strategy changed, instead of having the AI doing tool calls (that are unreliable on limited locally run AIs), I resorted to letting the AI respond with a JSON struct that the QuickApp interprets as a command/task.

A key command is the 'Schedule task". that the AI responds with when it discovers that something should be done in the future.

The QuickApp will then just resend what should be done at that specific time to the AI and let it carry it out.

Ex.

Please login or register to see this code.

So the QuickApp will schedule a user prompt and come back later with "User> It's Friday, 07:00. Turn on the light ..."

 

Triggers are noted, so that when a registered decide triggers it will generate a user prompt "User> It's Friday, 07:00. Device 712 is breached"

If the AI has been told in the past to do something when device 712 is breached it will do that then.

 

The setup code looks like 

Please login or register to see this code.

 

The system prompt
 

Please login or register to see this code.

And a run with the rules in the example above the run looks like below.

It also logs the AI thinking so it turns into a novella, a bit entertaining.
Here I also run it in my emulator and speed up the time, so we can run shelling for next sunset without waiting forever (yes, sunset is 22:08 in Stockholm now)
User prompt (User>) generated by QuickApp.
AI prompt (AI>) response from AI
Code prompt (Code>) JSON code from AI interpreted by the QuickApp

Please login or register to see this code.