Local LLMs

​

News

• 2026/05/21: We now recommend Qwen3.6-35B-A3B as the first local model to try with OpenHands. It is an open-weight MoE model built for agentic coding, supports a large context window, and is available through LM Studio, Ollama, vLLM, and SGLang.

​

Quickstart: Running OpenHands with a Local LLM using LM Studio

• LM Studio as the local model server, which handles metadata downloads automatically and offers a simple, user-friendly interface for configuration.
• Qwen3.6-35B-A3B as the LLM for software development. This model is optimized for agentic coding and works well with tool-heavy workflows like OpenHands.

​

Hardware Requirements

• A recent GPU with at least 24GB of VRAM for quantized variants, or multiple GPUs for full precision and larger context windows, or
• A Mac with Apple Silicon with at least 64GB of unified memory for quantized variants

​

1. Install LM Studio

​

2. Download the Model

1. Make sure to set the User Interface Complexity Level to “Power User”, by clicking on the appropriate label at the bottom of the window.
2. Click the “Discover” button (Magnifying Glass icon) on the left navigation bar to open the Models download page.

3. Search for “Qwen3.6-35B-A3B”, confirm you’re downloading from the official Qwen publisher, then proceed to download.

4. Wait for the download to finish.

​

3. Load the Model

1. Click the “Developer” button (Console icon) on the left navigation bar to open the Developer Console.
2. Click the “Select a model to load” dropdown at the top of the application window.

3. Enable the “Manually choose model load parameters” switch.
4. Select Qwen3.6-35B-A3B from the model list.

5. Enable the “Show advanced settings” switch at the bottom of the Model settings flyout to show all the available settings.
6. Set “Context Length” to at least 22000 (for lower VRAM systems) or 32768 (recommended for better performance) and enable Flash Attention.
7. Click “Load Model” to start loading the model.

​

4. Start the LLM server

1. Enable the switch next to “Status” at the top-left of the Window.
2. Take note of the Model API Identifier shown on the sidebar on the right.

docker exec -it openhands-app curl -s http://host.docker.internal:1234/v1/models

​

5. Start OpenHands

1. Check the installation guide and ensure all prerequisites are met before running OpenHands, then run:

docker run -it --rm --pull=always \
    -e AGENT_SERVER_IMAGE_REPOSITORY=ghcr.io/openhands/agent-server \
    -e AGENT_SERVER_IMAGE_TAG=1.26.0-python \
    -e LOG_ALL_EVENTS=true \
    -v /var/run/docker.sock:/var/run/docker.sock \
    -v ~/.openhands:/.openhands \
    -p 3000:3000 \
    --add-host host.docker.internal:host-gateway \
    --name openhands-app \
    docker.openhands.dev/openhands/openhands:1.8

2. Wait until the server is running (see log below):

Digest: sha256:e72f9baecb458aedb9afc2cd5bc935118d1868719e55d50da73190d3a85c674f
Status: Image is up to date for docker.openhands.dev/openhands/openhands:1.8
Starting OpenHands...
Running OpenHands as root
14:22:13 - openhands:INFO: server_config.py:50 - Using config class None
INFO:     Started server process [8]
INFO:     Waiting for application startup.
INFO:     Application startup complete.
INFO:     Uvicorn running on http://0.0.0.0:3000 (Press CTRL+C to quit)

3. Visit http://localhost:3000 in your browser.

​

6. Configure OpenHands to use the LLM server

1. Click “see advanced settings” to open the LLM Settings page.

2. Enable the “Advanced” switch at the top of the page to show all the available settings.
3. Set the following values:
   • Custom Model: openai/qwen/qwen3.6-35b-a3b (the Model API identifier from LM Studio, prefixed with “openai/”)
   • Base URL: http://host.docker.internal:1234/v1
   • API Key: local-llm
4. Click “Save Settings” to save the configuration.

​

Community-Reported Notes and Troubleshooting

• qwen2.5-coder-14b-instruct — reported to resolve chatbot-like behavior
• qwopus3.5-27b-v3 Q8_0 (and similar retrained qwopus variants) — reported to work well with tool calls

​

Advanced: Alternative LLM Backends

​

Create an OpenAI-Compatible Endpoint with Ollama

• Install Ollama following the official documentation.
• Example launch command for Qwen3.6-35B-A3B:

# ⚠️ WARNING: OpenHands requires a large context size to work properly.
# When using Ollama, set OLLAMA_CONTEXT_LENGTH to at least 22000.
# The default (4096) is way too small — not even the system prompt will fit, and the agent will not behave correctly.
OLLAMA_CONTEXT_LENGTH=32768 OLLAMA_HOST=0.0.0.0:11434 OLLAMA_KEEP_ALIVE=-1 nohup ollama serve &
ollama pull qwen3.6:35b-a3b

​

Create an OpenAI-Compatible Endpoint with vLLM or SGLang

huggingface-cli download Qwen/Qwen3.6-35B-A3B --local-dir Qwen/Qwen3.6-35B-A3B

​

Serving the model using SGLang

• Install SGLang following the official documentation.
• Example launch command (with at least 2 GPUs):

SGLANG_ALLOW_OVERWRITE_LONGER_CONTEXT_LEN=1 python3 -m sglang.launch_server \
    --model Qwen/Qwen3.6-35B-A3B \
    --served-model-name Qwen3.6-35B-A3B \
    --port 8000 \
    --tp 2 --dp 1 \
    --host 0.0.0.0 \
    --api-key mykey --context-length 131072

​

Serving the model using vLLM

• Install vLLM following the official documentation.
• Example launch command (with at least 2 GPUs):

vllm serve Qwen/Qwen3.6-35B-A3B \
    --host 0.0.0.0 --port 8000 \
    --api-key mykey \
    --tensor-parallel-size 2 \
    --served-model-name Qwen3.6-35B-A3B \
    --enable-prefix-caching

1. Install the Arctic Inference library that automatically patches vLLM:

pip install git+https://github.com/snowflakedb/ArcticInference.git

2. Run the launch command with speculative decoding enabled:

vllm serve Qwen/Qwen3.6-35B-A3B \
    --host 0.0.0.0 --port 8000 \
    --api-key mykey \
    --tensor-parallel-size 2 \
    --served-model-name Qwen3.6-35B-A3B \
    --speculative-config '{"method": "suffix"}'

​

Run OpenHands (Alternative Backends)

​

Using Docker

​

Using Development Mode

​

Configure OpenHands (Alternative Backends)

1. Click “see advanced settings” to access the full configuration panel.
2. Enable the Advanced toggle at the top of the page.
3. Set the following parameters, if you followed the examples above:
   • Custom Model: openai/<served-model-name>
     • For Ollama: openai/qwen3.6:35b-a3b
     • For SGLang/vLLM: openai/Qwen3.6-35B-A3B
   • Base URL: http://host.docker.internal:<port>/v1 Use port 11434 for Ollama, or 8000 for SGLang and vLLM.
   • API Key:
     • For Ollama: any placeholder value (e.g. dummy, local-llm)
     • For SGLang or vLLM: use the same key provided when starting the server (e.g. mykey)