Software Sushi

Ubuntu 24.04 · x86_64 · CPU inference

LocalAI CPU Model API - Software Sushi

Getting started · authenticated local model API

1. Launch and connect

Choose t3.large, 8 GiB RAM, with at least 48 GiB encrypted gp3 root storage. The image targets Ubuntu 24.04 x86_64. Require IMDSv2 and attach your own instance profile with the permissions needed for Session Manager. Your network must provide connectivity to Systems Manager.

Keep inbound ports 8080 and 50051 closed. No inbound application rule is needed for a Session Manager connection. In the EC2 console, select your instance, choose Connect → Session Manager and open a shell.

sudo cloud-init status --wait

Wait for automatic first boot. It creates the per-instance credential and the ubuntu client configuration. The API is an HTTP service with no web UI.

2. Run the included task

From the Session Manager shell, run the installed task as ubuntu:

sudo -iu ubuntu bash /opt/sushi/docs/demo.sh

The task checks that an unauthenticated request is rejected, lists the selected model and asks the actual local model “What is 2 + 2?” with a 64-token output limit. It checks the answer and retains the request, response and result under ~/localai-demo/ for ubuntu. A successful task prints Local model answered: 4 and its result directory. A failure exits nonzero; keep the available results for diagnosis.

3. Call the API with curl

The API base is http://127.0.0.1:8080/v1, and the model name is sushi-qwen. These curl commands run on the instance as ubuntu. First switch to its login shell:

sudo -iu ubuntu

Run the entire following block in an unrecorded shell. The Python producer sends the authorization header directly through an anonymous pipe to curl --config -. The credential value is absent from shell arguments, environment variables and terminal output. Do not run the producer separately or enable tracing, verbose HTTP output or header logging. The example disables curl’s default configuration, proxies and redirects and uses only the fixed loopback endpoint.

set +x
(
  set -euo pipefail
  umask 077
  test "$(id -un)" = ubuntu
  mkdir -p "$HOME/localai-curl"
  result_dir=$(mktemp -d "$HOME/localai-curl/request.XXXXXXXX")
  chmod 700 "$result_dir"
  cd "$result_dir"
  cat > request.json <<'JSON'
{"model":"sushi-qwen","messages":[{"role":"system","content":"Answer arithmetic questions with a single digit and no explanation."},{"role":"user","content":"What is 2 + 2?"}],"temperature":0,"max_tokens":64,"stream":false}
JSON
  python3 - <<'PYCONFIG' | curl --disable --config - \
    --noproxy '*' --proto '=http' --max-redirs 0 \
    --connect-timeout 5 --max-time 180 --fail --silent --show-error \
    --header 'Content-Type: application/json' \
    --data-binary @request.json --output response.json \
    http://127.0.0.1:8080/v1/chat/completions
import json, os, re, stat
from pathlib import Path
p = Path.home() / '.config/localai/client.json'
try:
    info = p.lstat()
    if (not stat.S_ISREG(info.st_mode) or info.st_nlink != 1
            or info.st_uid != os.getuid() or info.st_mode & 0o077
            or info.st_size > 4096):
        raise ValueError()
    client = json.loads(p.read_bytes())
    if (client.get('endpoint') != 'http://127.0.0.1:8080'
            or client.get('model') != 'sushi-qwen'
            or re.fullmatch('[a-f0-9]{64}', client.get('api_key', '')) is None):
        raise ValueError()
except Exception:
    raise SystemExit('Client configuration invalid; original preserved.') from None
# This output goes only into curl stdin, never into argv or the terminal.
print('header = ' + json.dumps('Authorization: Bearer ' + client['api_key']))
PYCONFIG
  python3 - <<'PYRESPONSE'
import json
from pathlib import Path
response = json.loads(Path('response.json').read_bytes())
print(response['choices'][0]['message']['content'])
PYRESPONSE
  printf 'Saved request and response: %s\n' "$result_dir"
)

The block sends one bounded completion request, prints only the returned message content and saves the request and response in a new protected ~/localai-curl/ directory. The code is an optional client example; the installed demo above supplies the separate answer check. Treat generated text as model output and review it before use.

4. Optional workstation tunnel

To connect a workstation application, install and configure AWS CLI and the Session Manager plugin with your own authorized AWS identity. Run the following on your workstation and enter your actual instance ID and Region. Keep that terminal open:

printf 'EC2 instance ID: '
IFS= read -r LOCALAI_INSTANCE_ID
printf 'AWS Region: '
IFS= read -r LOCALAI_AWS_REGION
aws ssm start-session --region "$LOCALAI_AWS_REGION" \
  --target "$LOCALAI_INSTANCE_ID" \
  --document-name AWS-StartPortForwardingSession \
  --parameters '{"portNumber":["8080"],"localPortNumber":["8080"]}'

Your local API base is then http://127.0.0.1:8080/v1. Opening it in a browser does not provide a chat interface. Authentication is still required. The curl example above intentionally runs on the instance, where the client configuration remains. If you configure a separate workstation client, use your organization’s secure credential handling; do not paste the credential into shell commands or URLs.

5. Model and resource limits

The selected model is Qwen2.5-0.5B-Instruct Q4_K_M, exposed as sushi-qwen. It is included in the image; local inference requires no post-launch model download or external inference provider account. AWS access for instance management is separate.

The configuration uses one concurrent backend inference, two CPU threads and a 2048-token context. The included task limits output to 64 tokens. Model files are read-only; mutable state and caches are under /var/lib/localai. Galleries, automatic backend downloads, agents, MCP and runtime settings are disabled.

The recommended launch profile provides headroom for the operating system, context, caches and application state. It is not a measured physical minimum or a throughput/latency guarantee. Measure your own workload before changing the profile.

6. Identity, results and recovery

First boot writes the ubuntu client configuration at ~/.config/localai/client.json with mode 0600. The included client reads it without printing its token. Do not display this file, copy it into logs, or attach it to support requests.

Back up /etc/localai, /var/lib/localai and the ubuntu client configuration together using your protected backup process. Restore the matching identity and state together. Credentials and client configuration survive service restarts. Existing state without its original credentials causes a startup failure rather than a silent identity reset.

For a nonsecret service summary, run:

systemctl is-active sushi-localai-firstboot.service sushi-localai-backend.service sushi-localai.service

If startup or a request fails, preserve the protected result directory and inspect the instance through your administrative session. Do not delete identity files to force a new credential. Read the installed /opt/sushi/docs/README.md for the supplied workflow.

7. Charges and licenses

The software fee is USD 0.03 per running instance-hour. EC2, EBS and data transfer are billed separately. The supplied local model needs no paid inference-provider API.

LocalAI uses MIT terms; the selected Qwen model has separate Apache-2.0 terms. Read the installed notices and model card in /usr/share/doc/localai/. Software Sushi packages this software independently and does not claim endorsement by the upstream projects or AWS.