I’ve been experimenting with giving LLMs the ability to manipulate a web app’s UI in real time. Not just streaming text into a chat window, but actually calling JavaScript functions in the browser - swapping panels, dropping pins on a map, updating lists. This is a writeup of how I got there, the problems I ran into, and what I ended up building.
The Problem
LLMs are good at calling backend tools - querying a database, hitting an API, writing a file. But triggering something visual in the user’s browser in real time is a different problem. There’s no clean standard way to do it, especially in a Django app. The typical approach is to return data from the LLM and have the frontend poll or re-render. That works but it’s clunky and slow.
What I wanted was for the LLM to be able to call frontend functions directly - open a panel, drop a pin on a map, log a message to a console - the same way it calls backend tools, without the frontend having to know in advance what the LLM decided to do.
Demo
The demo app is a simple notes and map interface. You can ask the LLM to add notes, drop pins on the map, switch between views, and log its reasoning to an on-screen console - all driven by natural language.
How Normal Tool Use Works
In a standard LLM integration, tools are defined as JSON schemas and passed to the model. When the LLM decides to use one, it returns a tool call with the function name and arguments. Your backend then executes the function and returns the result.
Frontend tools follow exactly the same pattern - they use the same schema format and can be mixed freely with regular backend tools. The only difference is what happens when the tool is called.
Defining Frontend Tools
Frontend tools are defined using the standard OpenAI tool schema format, the same as any other tool:
FRONTEND_TOOLS = [
...
{
"type": "function",
"function": {
"name": "open_console",
"description": "Open the LLM console panel so the user can see log output. Use close_console to hide it.",
"parameters": {
"type": "object",
"properties": {},
"required": [],
},
},
},
{
"type": "function",
"function": {
"name": "drop_pin",
"description": "Drop a marker on the map at a given location with an optional label.",
"parameters": {
"type": "object",
"properties": {
"lat": {"type": "number", "description": "Latitude"},
"lng": {"type": "number", "description": "Longitude"},
"label": {"type": "string", "description": "Popup label shown on the marker (optional)"},
},
"required": ["lat", "lng"],
},
},
},
...
]
FRONTEND_TOOL_NAMES = {t["function"]["name"] for t in FRONTEND_TOOLS}
Dispatching to the Frontend
When a tool call comes in, the backend checks whether it’s a frontend tool. If it is, instead of executing a function on the server it publishes the tool call to the frontend over SSE. If it’s a backend tool it executes normally.
def _execute(name: str, arguments: dict) -> str:
if name in FRONTEND_TOOL_NAMES:
publish({"type": "tool_call", "tool": name, "args": arguments})
return json.dumps({"status": "dispatched"})
return execute_tool(name, arguments)
This is fire and forget for now - the backend doesn’t wait for acknowledgement from the frontend. For a production implementation you’d want some form of confirmation but for this demo it works reliably enough.
The Relay Object
On the frontend, a small Relay class handles the SSE connection and routes incoming tool calls to registered handler functions. Handlers are registered using the .on() method, making the interface clean and readable.
export class Relay {
constructor() {
this._handlers = new Map()
}
on(name, fn) {
this._handlers.set(name, fn)
return this
}
connect() {
this._startSSE()
return this
}
_startSSE() {
...
es.addEventListener('tool_call', (event) => {
const data = JSON.parse(event.data)
this._dispatch(data)
})
...
}
_dispatch({ tool, args }) {
this._handlers.get(tool)?.(args)
}
}
Here is the Relay instance from the demo app, showing all the registered handlers:
window._relay = new Relay()
.registerHtmxTriggers()
.on("open_console", () => {
if (document.getElementById("console-panel").classList.contains("collapsed")) toggleConsole();
})
.on("close_console", () => {
if (!document.getElementById("console-panel").classList.contains("collapsed")) toggleConsole();
})
.on("log", ({ message }) => appendToConsole(message))
.on("refresh_note_list", () => loadNotes())
.on("set_dark_mode", ({ dark }) => setDark(dark))
.on("go_to_coordinates", ({ lat, lng, zoom = 14 }) =>
map?.flyTo([lat, lng], zoom),
)
.on("drop_pin", (args) => dropPin(args))
.connect();
HTMX Integration
One useful pattern that emerged is triggering HTMX swaps from the LLM. Rather than the frontend knowing in advance which view to show, the LLM can swap panels based on context - switching to the map when a location is mentioned, switching back to notes otherwise.
Frontend tools for HTMX triggers are defined on the backend:
FRONTEND_TOOLS = [
{
"type": "function",
"function": {
"name": "htmx_trigger",
"description": (
'Fire an HTMX action by name. Available triggers: '
'"swap-to-map" — Switch the left panel to the map view; '
'"swap-to-notes" — Switch the left panel back to the note list'
),
"parameters": {
"type": "object",
"properties": {
"trigger": {"type": "string", "description": "One of: swap-to-map, swap-to-notes"},
},
"required": ["trigger"],
},
},
},
...
]
The Relay class handles this generically by sending HX trigger to the front-end with the relevant value:
export class Relay {
...
registerHtmxTriggers() {
return this.on('htmx_trigger', ({ trigger }) => {
document.querySelector(`[data-trigger="${trigger}"]`)?.dispatchEvent(new Event(trigger))
})
}
...
}
The SSE Tab Limit Problem
SSE works well until you open more than six tabs on the same domain. Browsers limit each domain to six concurrent HTTP connections, so additional tabs queue waiting for a connection to free up. In practice this means tabs hang silently with no error message, which is confusing to debug.
The solution uses two browser APIs that aren’t widely known: the Web Locks API and BroadcastChannel.
The idea is straightforward. Each tab races to acquire a named lock. Whichever tab wins becomes the “tab leader” and opens the SSE connection. When a tool call arrives, the leader dispatches it locally and broadcasts it to all other tabs via BroadcastChannel. If the leader tab closes, the lock is released and another tab picks it up, reopening the SSE connection automatically.
This means only one SSE connection is ever open per domain regardless of how many tabs are open.
_startSSE() {
// navigator.locks ensures only one tab holds the SSE connection
navigator.locks.request('relay_lock', () => {
const es = new EventSource('/api/events/')
let release
const held = new Promise((resolve) => { release = resolve })
es.addEventListener('tool_call', (event) => {
const data = JSON.parse(event.data)
// Call this _dispatch since BroadcastChannel doesn't deliver to the sender
this._dispatch(data)
// Notify other tabs
this._channel.postMessage(data)
})
es.onerror = () => {
es.close()
// Release the lock before retrying. Without this, the tab holds the
// lock forever while also queuing a new request for it, deadlocking
// all other tabs that try to open.
release()
setTimeout(() => this._startSSE(), 3000)
}
// Close the SSE connection on unload so the server frees the thread immediately.
// The browser releases the lock naturally when the tab is gone.
window.addEventListener('beforeunload', () => es.close())
return held
})
// Other tabs receive via BroadcastChannel
this._channel.onmessage = ({ data }) => this._dispatch(data)
}
Gunicorn Configuration
SSE requires long-lived connections which means threads are held open for the duration. The default Gunicorn sync workers time out and close these connections. The fix is to use threaded workers with a timeout of zero:
web:
build: .
command: sh -c "python manage.py migrate --no-input && gunicorn demo.wsgi:application --bind 0.0.0.0:8000 --worker-class gthread --workers 1 --threads 50 --timeout 0"
...
--worker-class gthread switches to threaded workers, --threads 50 allows up to 50 concurrent connections, and --timeout 0 disables the worker timeout so long-lived SSE connections aren’t killed.
How It Compares to AG-UI
AG-UI is a full protocol for agent-user interaction backed by CopilotKit and supported by LangGraph and CrewAI. It handles streaming, shared state, multi-agent coordination, cancellation, and more. It’s the right choice for complex agentic applications.
This is not that. This is a minimal pattern for Django developers who want to add LLM frontend tool calling to an existing app without adopting a new protocol or framework. If you have a Django app with a chat interface and you want the LLM to be able to manipulate your UI, this gets you there with a small amount of code and no new dependencies beyond what you’re already using.
Repo
[Link here]