pch/festbench-llm
1
0
Fork 0
Code Issues Pull requests Projects Releases 6 Packages Wiki Activity Actions

6 releases 6 tags

RSS feed
  • v0.3.6 430c0dd644

    v0.3.6 Stable

    pch released this 2026-10-17 11:45:33 +00:00 | 1 commits to main since this release

    scenariobench: fix oracle crash on non-UTF-8 model output (errors=replace on all subprocess decoders); kubelet_config_repair verify regex (double = matched nothing); complete podman->docker migration. selftest 16/16.

    Downloads
  • v0.3.4 430c0dd644

    v0.3.4 Stable

    pch released this 2026-09-21 12:11:00 +00:00 | 1 commits to main since this release

    Hosted OpenAI-compatible providers (Ollama Cloud, and anything speaking the same protocol), plus --inference-args.

    benchlib assumed llama.cpp on every axis: /health for liveness, /props for
    identity, a single served model so the request body could say "x", and
    reasoning_content as the only spelling of reasoning. A hosted provider fails
    all four, quietly -- a run filed under whichever model sorts first, reasoning
    counted as zero, and TTFT timing the reasoning phase as prefill.

    --model name the model to request, e.g. deepseek-v4-flash:cloud
    --engine record the inference host, e.g. ollama-cloud
    --inference-args renamed from --server-config; the old flag still works

    Every change is a widening: llama.cpp keeps its existing path and stays
    zero-config. A hosted endpoint now reports thinking level n/a rather than
    claiming a level it has no mechanism to honour.

    Downloads
  • v0.3.3 66da94523a

    v0.3.3 Stable

    pch released this 2026-09-16 15:29:09 +00:00 | 2 commits to main since this release

    The check now ends by naming what is wrong.

    Previously the last line was only a count -- '1 blocking problem(s), 3 warning(s)' -- after a list that may have scrolled away. It now restates each failure and warning beneath the count, e.g.

    1 blocking problem, 3 warnings
  FAIL  container runtime: no docker or podman found
  WARN  fs.inotify.max_user_watches: 65536 (kind wants >= 524288)

    The missing-runtime advice now gives the install and docker-group commands rather than repeating 'install docker or podman'.

    No image change: the runner image is identical to v0.3.1.

    sha256sum -c SHA256SUMS
chmod +x festbench_0.3.3_linux_amd64
./festbench_0.3.3_linux_amd64 --check --endpoint <your endpoint>
    Downloads
  • v0.3.2 99a37c33e1

    v0.3.2 Stable

    pch released this 2026-09-16 13:44:00 +00:00 | 3 commits to main since this release

    Fixes the docker-group diagnosis.

    A socket that refuses permission has two causes with opposite fixes. v0.3.1 always advised newgrp docker, which only works if your account is already in the docker group. It now checks, and says either:

    • you ARE in the group but this shell predates it -> newgrp docker, or log out and back in
    • you are NOT in the group -> sudo usermod -aG docker $USER, then log out and back in

    No image change: the runner image is identical to v0.3.1.

    sha256sum -c SHA256SUMS
chmod +x festbench_0.3.2_linux_amd64
./festbench_0.3.2_linux_amd64 --check --endpoint <your endpoint>
    Downloads
  • v0.3.1 28b8e682a8

    v0.3.1 Stable

    pch released this 2026-09-12 03:36:57 +00:00 | 4 commits to main since this release

    Guided setup reworked, and four fixes found by using it.

    • the prerequisite check now tests the docker SOCKET, not just that the binary exists. Previously it reported 'container runtime ok' and then failed several prompts later with permission denied.
    • results are written to the working directory. They previously landed beside the binary, which breaks once it is installed to /usr/local/bin.
    • the guided run asks about the connection first and no longer asks for a model twice.
    • API keys are supported for endpoints that require one. Without this a keyed endpoint answers 401 and the suites scored it as model failures.
    • thinking levels are read from the model's own chat template rather than assumed.

    Requires only a container runtime. Linux x86_64.

    sha256sum -c SHA256SUMS
chmod +x festbench_0.3.1_linux_amd64
./festbench_0.3.1_linux_amd64 --check --endpoint <your endpoint>
    Downloads
  • v0.3.0 28b8e682a8

    v0.3.0 Stable

    pch released this 2026-09-12 02:44:49 +00:00 | 4 commits to main since this release

    First public build. Linux x86_64, static, requires only a container runtime.

    Verify and install:

    sha256sum -c SHA256SUMS
chmod +x festbench_0.3.0_linux_amd64

    Run 'festbench --check' before the first benchmark: it reports the problems that fail confusingly rather than clearly.

    Downloads