# SearchHub

A local search engine for your browser bookmarks and history. Import bookmarks/history from Firefox, Zen, Chrome, or Chromium, search them with full-text queries, and optionally forward searches to external engines like Wikipedia or SearXNG (aggregates results from dozens of backends). Content can be automatically tagged via a two-stage ONNX pipeline — a fast bi-encoder scores all tags, then a cross-encoder reranker refines the top candidates (opt-in; see auto-tagging section).

The ONNX models are embedded via the optional `tagging` feature (included by default). If you don't need auto-tagging, you can build a ~30% smaller binary by excluding it.

## Quick start

**Prerequisites:** [Rust](https://rustup.rs/) (or grab a [pre-built binary](https://vit.am/~ololduck/search_hub/latest/)).

```sh
git clone https://vit.am/~ololduck/search_hub/repository.git search_hub
cd search_hub
cargo install --path .
```

Then import your browser bookmarks and start the web UI:

```sh
search_hub import firefox     # or "chrome", "zen"
search_hub serve              # open http://127.0.0.1:8080
```

**Smaller build (no ONNX tagging):**

```sh
cargo install --path . --no-default-features
```

## CLI reference

| Command | What it does |
|---------|-------------|
| `search_hub serve` | Start web UI on port 8080 |
| `search_hub serve --port 3000` | Start on a custom port |
| `search_hub import firefox` | Import bookmarks from Firefox |
| `search_hub import chrome` | Import from Chrome/Chromium |
| `search_hub import zen` | Import from Zen Browser |
| `search_hub search "query"` | Search bookmarks from the terminal |
| `search_hub list` | List all bookmarks |
| `search_hub insert "Title" https://..."` | Add a bookmark (fetches content, auto-tags if enabled) |
| `search_hub remove --id 1` | Delete a bookmark by ID |
| `search_hub retag --all` | Re-run auto-tagging (requires `[tagging] enabled = true` in config) |
| `search_hub init-config` | Create a default config file at `~/.config/search_hub/config.toml` |
| `search_hub config-schema` | Print the JSON Schema for the config file format |
| `search_hub self-update` | Check abbaye Atom feed and update to the latest release |
| `search_hub self-update --dry-run` | Check for updates without downloading |
| `search_hub self-update --target x86_64-unknown-linux-gnu` | Override the target triple |

All commands use `~/.local/share/search_hub/bookmarks.db` by default. Override with `--db-path` or set `db_path` in the config file.

## Web UI

Search queries are forwarded to external engines: Wikipedia, [crates.io](https://crates.io), and optionally [SearXNG](https://searx.space) if configured. Works as a custom search provider in Firefox/Zen via OpenSearch (auto-discovered at `/opensearch.xml`).

When JavaScript is available, results load progressively: local bookmark results appear instantly, then external engine results stream in as each engine responds. The server supports a `?js=true` parameter that skips external engines during the initial page render (~50ms instead of ~1700ms), letting the client fetch them via a streaming connection instead. The inline JavaScript automatically adds `?js=true` to the URL after the first load.

For the fastest load from your browser's search bar (before JS runs), configure the search engine URL with `?js=true`:

- **Firefox:** Settings → Search → Manage Search Engines → Add, use `http://localhost:8080/search?q=%s&js=true`
- **OpenSearch template:** `http://localhost:8080/search?q={searchTerms}&js=true`

Without `?js=true`, the server waits for all external engines before sending the page — a fallback for browsers without JavaScript (still works, just slower on first load).

## Configuration

Run `search_hub init-config` to create `~/.config/search_hub/config.toml` with all available options commented out. The config supports:

- `[tagging]` — auto-tagging settings (enable, threshold, model selection, reranker)
- `[[engines]]` — external search engines (SearXNG, Wikipedia, MDN, crates.io, generic)
- `[[bangs]]` — custom search shortcuts
- `[[tagging.tags]]` — custom tag definitions with optional negatives
- `[web]` — server bind address, page size, worker threads
- `exclude_urls` — hosts to skip when fetching content
- `db_path` — bookmark database location

See `search_hub config-schema` for the complete JSON Schema of the config file format.

## Search shortcuts

Shortcodes are compact aliases auto-generated from configured `[[engines]]`.

| Prefix | Example | Behavior |
|--------|---------|----------|
| `!` | `!w Rust` | HTTP 302 redirect to the site's own search results page |
| `@` | `@w Rust` | Show search results from that engine only (bookmarks still shown) |

### Auto-generated shortcodes

| Engine | Shortcode | Bang URL |
|--------|-----------|----------|
| Wikipedia (lang=en) | `w` | `https://en.wikipedia.org/w/index.php?search={}` |
| Wikipedia (lang=fr) | `wfr` | `https://fr.wikipedia.org/w/index.php?search={}` |
| MDN (locale=en-US) | `mdn` | `https://developer.mozilla.org/en-US/search?q={}` |
| MDN (locale=fr) | `mdnfr` | `https://developer.mozilla.org/fr/search?q={}` |
| crates.io | `crates` | `https://crates.io/search?q={}` |
| SearXNG | `sx` | `{instance}/search?q={}` |
| Generic | slugified name | the engine's own URL template |

Override `shortcode`, `bang_url`, or `bang_enabled` per engine:

```toml
[[engines]]
type = "wikipedia"
lang = "fr"
shortcode = "wikifr"       # overrides "wfr"
bang_enabled = false       # disable ! redirect (still searchable via @)
```

Custom bangs (standalone `!` redirects without `@` support):

```toml
[[bangs]]
trigger = "gh"
url = "https://github.com/search?q={}"
name = "GitHub"
```

If two engines share a shortcode, SearchHub panics at startup naming both. Set `shortcode` on one to resolve.

## Auto-tagging

When `[tagging] enabled = true`, SearchHub automatically tags bookmarks during insert and import using a two-stage ONNX pipeline:

1. **Bi-encoder** — `BAAI/bge-small-en-v1.5` (~33 MB) embeds both the bookmark content and each tag's example set. Scores use **mean-pooling** (average similarity across all tag examples), requiring consensus rather than a single match.

2. **Cross-encoder reranker** — `BAAI/bge-reranker-v2-m3` (~2 GB, optional) re-ranks the top 10 candidates for higher precision. Falls back to bi-encoder scores on failure. Disable with `reranker_enabled = false` to save disk space.

Tags can include **negative examples**. The score is `pos_mean - neg_mean * 0.3`, so negative matches penalize a tag without dominating. Tags without negatives are unaffected. A **confidence threshold** (`[tagging] threshold`, default: 0.6) controls which tags are applied.

### Tag taxonomy

SearchHub ships with 21 built-in tags. Add or override them with `[[tagging.tags]]` in config.

| Tag | Domain |
|-----|--------|
| `rust` | Rust language projects and tooling |
| `python` | Python language projects and tooling |
| `javascript` | JS/TS, Node.js, browser APIs |
| `web` | Web frameworks, HTML, CSS, frontend |
| `api` | REST, GraphQL, RPC protocols |
| `database` | SQL, NoSQL, ORMs, migrations |
| `testing` | Unit, integration, E2E, property-based |
| `linux` | Kernel, systemd, shell, config |
| `security` | Auth, crypto, CVE, hardening |
| `ai` | ML, LLMs, computer vision, NLP |
| `data` | ETL, pipelines, formats (CSV, Parquet) |
| `audio` | Sound processing, codecs, DSP |
| `video` | Video codecs, streaming, rendering |
| `gaming` | Game engines, graphics, physics |
| `hardware` | CPU, GPU, memory, peripherals |
| `mobile` | iOS, Android, PWA, responsive |
| `design` | UI/UX, typography, color, icons |
| `documentation` | API refs, ADRs, READMEs, docstrings |
| `news` | Release announcements, changelogs, events |
| `social` | Forums, chat, federated platforms |
| `productivity` | Note-taking, task management, tools |

### Custom tags with negatives

```toml
[[tagging.tags]]
name = "my-special-tag"
examples = ["example text one", "example text two"]
negatives = ["something similar but different", "another close miss"]
```

### Disk space

The bi-encoder model is ~33 MB. Enabling the reranker adds ~2 GB (~2.1 GB total in cache). Set `reranker_enabled = false` to minimize disk usage. The first tagging command downloads models to `~/.cache/search_hub/`.

## Deployment

### Systemd user services

**Web server** (auto-start on login):

```sh
VERSION=$(search_hub --version | cut -d\  -f2)
mkdir -p ~/.config/systemd/user
wget -O ~/.config/systemd/user/search-hub-web.service \
  https://vit.am/~ololduck/search_hub/repository/browse/v$VERSION/contrib/search-hub-web.service
systemctl --user daemon-reload
systemctl --user enable --now search-hub-web.service
```

**Auto-import** (daily Firefox/Zen bookmark import):

```sh
wget -O ~/.config/systemd/user/search-hub-import.service \
  https://vit.am/~ololduck/search_hub/repository/browse/v$VERSION/contrib/search-hub-import.service
wget -O ~/.config/systemd/user/search-hub-import.timer \
  https://vit.am/~ololduck/search_hub/repository/browse/v$VERSION/contrib/search-hub-import.timer
systemctl --user daemon-reload
systemctl --user enable --now search-hub-import.timer
```

**Auto-update** (weekly release check):

```sh
wget -O ~/.config/systemd/user/search-hub-update.service \
  https://vit.am/~ololduck/search_hub/repository/browse/v$VERSION/contrib/search-hub-update.service
wget -O ~/.config/systemd/user/search-hub-update.timer \
  https://vit.am/~ololduck/search_hub/repository/browse/v$VERSION/contrib/search-hub-update.timer
systemctl --user daemon-reload
systemctl --user enable --now search-hub-self-update.timer
```

Check status with `systemctl --user status search-hub-web`. View logs with `journalctl --user -u search-hub-web -f`.

### Podman / Docker

A container image is available at `oci.vit.am/search-hub:latest` (port 8080, runs as `search_hub` user).

```sh
podman run -d --name search-hub \
  -p 8080:8080 \
  -v ~/.config/search_hub:/home/search_hub/.config/search_hub:ro \
  -v ~/.local/share/search_hub:/home/search_hub/.local/share/search_hub \
  oci.vit.am/search-hub:latest serve
```

SIGHUP reloads the config without restarting: `podman kill -s HUP search-hub`

Or use `docker compose up -d` (see `docker-compose.yaml` at the project root, includes a commented SearXNG service). For systemd-native container management, see the Quadlet file at `contrib/search-hub.container`.

## Resources

- **Downloads:** [vit.am/~ololduck/search_hub/latest](https://vit.am/~ololduck/search_hub/latest/)
- **Repository browser:** [vit.am/~ololduck/search_hub/repository](https://vit.am/~ololduck/search_hub/repository)
- **Git clone:** `git clone https://vit.am/~ololduck/search_hub/repository.git search_hub`