Tool Calling with Local Models

What is tool calling?

Tool calling lets the model ask your program to run a function, then uses the result in its answer. It is how agents read files, search code, or call APIs.

Which local models support it?

Tool calling works best on models trained for it:

Qwen 2.5
Llama 3.1+
Mistral Nemo
Hermes 3

Check LM Studio's Tool Calling template in the model settings.

Minimal Python example

import requests, json

def read_file(path):
    with open(path) as f:
        return f.read()

tools = [{
    "type": "function",
    "function": {
        "name": "read_file",
        "parameters": {
            "type": "object",
            "properties": {"path": {"type": "string"}},
            "required": ["path"]
        }
    }
}]

r = requests.post("http://localhost:1234/v1/chat/completions", json={
    "model": "local-model",
    "messages": [{"role": "user", "content": "Summarize main.py"}],
    "tools": tools
}).json()

msg = r["choices"][0]["message"]
if msg.get("tool_calls"):
    args = json.loads(msg["tool_calls"][0]["function"]["arguments"])
    content = read_file(args["path"])
    print(content)

Expect imperfections

Local models are less reliable at tool calling than Claude. Always validate arguments and handle missing parameters gracefully.