orchard

mirror

Every site I host, in one repo, along with the Cloudflare Tunnel and Caddy that front them. It's all Go, Vite, and SQLite, and it runs on a desktop at home with nothing listening on an inbound port.

blogbuncaddycloudflare-tunneldockergogolanghomelabhtml-templatemonorepoself-hostedseosqlitestatic-sitetypstuptime-monitoringviteweb-analytics

 1<!doctype html>
 2<html lang="en">
 3<head>
 4<meta charset="utf-8">
 5<meta name="viewport" content="width=device-width, initial-scale=1">
 6<meta name="robots" content="noindex, nofollow">
 7<title>llm &middot; one model, one card, every service</title>
 8<link rel="stylesheet" href="{{asset "static/app.css"}}">
 9  <!-- Analytics, first party and on this estate. The id is identity rather than
10       a credential, so it is a constant here like every other site's. -->
11  <script>
12    (function (m, e, t, r, i, c, s) {
13      m.collectorQueue = m.collectorQueue || r;
14      m.collectorServer = c;
15      m.collectorId = s;
16      m.collectorScript = e.createElement(t);
17      m.collectorScript.src = c + i;
18      e.head.appendChild(m.collectorScript);
19    })(window, document, "script", [], "/static/collector.js",
20       "https://analytics.bythewood.me", "e432c02b-1537-4703-bc5a-6aa2e8573613");
21  </script>
22</head>
23<body>
24<main class="landing">
25  <a class="brand" href="/"><span class="key">L</span><span class="brand-name">llm</span></a>
26
27  <p class="eyebrow">Self hosted &middot; {{.Model}} &middot; api key required</p>
28
29  <h1>One set of weights on one card, in front of <span class="accent">every service</span>.</h1>
30
31  <p class="lede">
32    An OpenAI shaped endpoint over llama.cpp. Before this each site carried its own
33    model server, which on an 8GB card meant whichever one you used last evicted the
34    other. This holds the model, hands out keys, and writes down every prompt and
35    every completion that passes through.
36  </p>
37
38  <ul class="points">
39    <li><b>A proxy, not a library.</b> Anything that already speaks chat completions needs a base url and a header, not a rewrite.</li>
40    <li><b>Keys per caller</b>, so usage is attributable and revoking one stops it on the next request.</li>
41    <li><b>Every call logged</b> in full, prompt and completion, with tokens and throughput, aged out on a retention window.</li>
42    <li><b>Loaded on demand</b> and unloaded after three idle minutes, so a desktop that is also a workstation gets its card back.</li>
43  </ul>
44
45  <div class="ctas">
46    <a class="btn" href="/login">Open dashboard &rarr;</a>
47    <a class="btn ghost" href="{{.Source}}" target="_blank" rel="noopener">View source</a>
48  </div>
49
50</main>
51</body>
52</html>