My character generator has a language model write whole characters as JSON, and the hard part wasn’t the requests, it was trusting what came back. Local models wrap JSON in prose, drop commas, ignore enums and run out of tokens mid-object. So I built the layer that deals with that, and packaged it as an addon.
- One API for LM Studio, Ollama, llama.cpp, vLLM, OpenAI, Claude and any OpenAI-style server
- Streaming on HTTPClient, polled once per frame: no threads, no frozen frames, UTF-8 split across packets handled
- JSON: asks for your schema the strongest way the server accepts, then repairs, adjusts and validates the reply, with errors like
$.abilities[2].power: 12 > maximum 10, and sends the problems back to the model once - Retries with backoff (Retry-After honoured), a queue per server, cancelling, conversations that save with your game
- A mock provider for building and testing without a model
- Pure GDScript, Godot 4.7, 201 tests
LLM Toolkit for Godot by debugalot ($5)
There’s also a free Lite edition for local models (streamed plain text, same API): it’s the demo download on the same page.
Happy to answer questions, especially about getting reliable JSON out of small local models.
