Archive Search

Built by gpt-6-astra low. Search powered by glm-5.3. may contain hallucins

Search Samuel Shadrach’s English writing with GLM-5.3 via OpenRouter.

Live app: https://search.samuelshadrach.com/

The backend uses Python/FastAPI; the frontend is plain HTML, CSS, and JavaScript. No frontend build step is required.

Run locally

Requires Python 3.11+ and an OpenRouter API key with access to z-ai/glm-5.3.

cd archive-search
python3 -m venv .venv
.venv/bin/pip install -r requirements.txt
cp .env.example .env
# Edit .env and set OPENROUTER_API_KEY.
.venv/bin/uvicorn app:app --env-file .env --host 127.0.0.1 --port 8000

Open http://localhost:8000. Alternatively, export OPENROUTER_API_KEY in the server environment and omit --env-file .env. The key is never sent to the browser. Real environment files, credentials, and private keys are excluded from Git; .env.example contains only a placeholder.

How search works

1. Discover the latest text_english_singlepage_YYYYMMDD.md from the archive directory.
2. Read GLM-5.3’s current context limit from OpenRouter and tokenize the document with its published tokenizer. Input stays below half the context window, including prompt overhead, a 2,048-token buffer, and a further 12% margin for OpenRouter’s token estimate. The other half is available for reasoning and output.
3. Pack chunks of at most 100,000 tokens each (including post metadata), or less if required by the half-context safety limit. Count posts and verify final sections using native batch tokenization. Move whole posts between sections; only oversized posts need splitting. Every preparation counts the current content without a chunk cache.
4. Start two independent pipelines in parallel: low-reasoning search → low-reasoning reranking, and high-reasoning search → high-reasoning reranking. Both share the same prepared archive chunks. Each searches all chunks concurrently with the user’s query and the instruction only include posts in my_research/ folder. The backend also filters matches to that folder. Context-limit rejections repack only the rejected chunk; other requests continue.
5. Rerank all collected matches using the same model. The model returns only result IDs; an exact-permutation check prevents additions, deletions, merging, or rewritten content. Duplicate post matches remain separate. Zero or one match requires no ranking request. Each pipeline retries a failed reranking step once, with identical inputs and reasoning effort; cancellation does not trigger a retry.
6. Show results only after a whole pipeline finishes successfully, with excerpts and HTML post links. Show each completed pipeline in its own result group while the other continues. When both are ready, show high results above low results. A failed pipeline never publishes partial results; the other can still finish. Full model responses are excluded from API events, the interface, and downloads.

At completion, one estimated reasoning + output token total covers both pipelines, including reported usage from failed attempts and reranking retries. OpenRouter completion tokens already include reasoning tokens, so they are not added twice. Missing usage is labelled as a lower bound.

Search progress uses server-sent events. Stopping a search cancels outstanding requests in both pipelines and retains any completed results already displayed. A provider may still bill for work already performed. Downloads contain the currently displayed results in order and include both completed groups with their reasoning levels.

Search and reranking use z-ai/glm-5.3:floor with their respective low or high reasoning effort to prioritize lower-priced providers through OpenRouter’s Floor routing. Context limits and tokenizer selection still use the base GLM-5.3 model.

The archive and model metadata are cached in memory for 15 minutes. The pinned tokenizer is downloaded to .cache/; no model weights are downloaded. Internet access to the archive, Hugging Face, and OpenRouter is required. Queries incur OpenRouter usage charges.

Optional MAX_COMPLETION_TOKENS lowers the completion ceiling, including reasoning tokens. Its default is half the context window, capped by the provider’s advertised maximum. The same input and output limits apply to reranking; oversized combined results are reported rather than silently truncated.

Checks

.venv/bin/pip install -r requirements-dev.txt
.venv/bin/python -m pytest -q
node --check static/app.js

# With the app running locally:
.venv/bin/playwright install chromium
.venv/bin/python tests/browser_check.py

Browser checks mock inference responses and incur no API charges. They use the running app’s /api/info endpoint for archive metadata. Set ARCHIVE_BASE_URL to check another deployment. Screenshots are generated in the ignored test-results/ directory.

Deployment

See deployment notes for the systemd service, Nginx configuration, HTTPS renewal, and updates. The public deployment uses a server-owned API key for visitor searches.

Repository contents are limited to the current source, tests, dependency manifests, configuration examples, and documentation. Runtime credentials and TLS material belong outside the repository.