| @@ -1,3 +1,178 @@ |
| +//! Builder types, the [`Builder`] trait, and the parallel execution model. |
| +//! |
| +//! Each `[[builders]]` entry in `abbaye.toml` deserialises into a |
| +//! [`BuilderEntry`] and is run as an independent Tokio task. This module |
| +//! owns the [`AnyBuilder`] enum (the TOML type-tag dispatch), the |
| +//! [`LogEvent`] channel used for progress-bar updates, and the [`Builder`] |
| +//! trait that every concrete builder must implement. |
| +//! |
| +//! --- |
| +//! |
| +//! # Parallel Builder Execution |
| +//! |
| +//! This section explains how Abbaye runs the `[[builders]]` entries defined in |
| +//! `abbaye.toml` in parallel, how inter-builder dependencies are enforced, and |
| +//! how the progress UI is kept in sync with the running tasks. |
| +//! |
| +//! ## Overview |
| +//! |
| +//! Every `[[builders]]` entry is executed as an independent Tokio async task. |
| +//! Tasks are launched at the same time (via a shared `JoinSet`) and may |
| +//! therefore run fully concurrently. When a builder declares `depends_on`, |
| +//! its task simply waits — without blocking any thread — until all named |
| +//! prerequisites have completed successfully before invoking the underlying |
| +//! build logic. |
| +//! |
| +//! ## Configuration Model |
| +//! |
| +//! ### `BuilderEntry` |
| +//! |
| +//! Each `[[builders]]` table in the TOML file deserialises into a |
| +//! [`BuilderEntry`]: |
| +//! |
| +//! | Field | Type | Purpose | |
| +//! |--------------|-------------------|-------------------------------------------------------------------| |
| +//! | `builder` | [`AnyBuilder`] | Type-tagged union holding the builder's own config (flattened). | |
| +//! | `id` | `Option<String>` | Stable name that other builders can reference in `depends_on`. | |
| +//! | `depends_on` | `Vec<String>` | IDs of builders that must succeed before this one starts. | |
| +//! |
| +//! Example: |
| +//! |
| +//! ```toml |
| +//! [[builders]] |
| +//! type = "cargo" |
| +//! id = "compile" |
| +//! |
| +//! [[builders]] |
| +//! type = "script" |
| +//! script = ["strip target/release/mybin"] |
| +//! outputs = ["target/release/mybin"] |
| +//! depends_on = ["compile"] # waits for the cargo builder above |
| +//! ``` |
| +//! |
| +//! ### `AnyBuilder` variants |
| +//! |
| +//! | TOML `type` | Rust variant | What it does | |
| +//! |--------------|---------------|-----------------------------------------------------------| |
| +//! | `archive` | `Archive` | Creates a `.tar.gz` of the source tree. | |
| +//! | `cargo` | `Cargo` | `cargo build --release`, optionally for multiple targets. | |
| +//! | `cargo_doc` | `CargoDoc` | `cargo doc`. | |
| +//! | `markdown` | `Markdown` | Renders a directory of `.md` files to HTML. | |
| +//! | `script` | `Script` | Runs an arbitrary sequence of `sh -c` commands. | |
| +//! |
| +//! ## Orchestration in `build_site` (`src/site.rs`) |
| +//! |
| +//! The entire builder pipeline lives inside `build_site`, in three phases. |
| +//! |
| +//! ### Phase 1 — Validation |
| +//! |
| +//! Before any task is spawned, two checks are performed: |
| +//! |
| +//! 1. **Reference check** — every string in `depends_on` must match an `id` |
| +//! of some other builder in the same config. Unknown IDs are reported as a |
| +//! hard error immediately. |
| +//! |
| +//! 2. **Cycle detection** — a depth-first search visits the dependency graph |
| +//! and returns an error if a cycle is found. |
| +//! |
| +//! The DFS tracks three states per node: `0 = unvisited`, `1 = in the current |
| +//! stack`, `2 = fully processed`. Encountering a node in state `1` means |
| +//! there is a back-edge, i.e. a cycle. |
| +//! |
| +//! ### Phase 2 — Completion channels |
| +//! |
| +//! For each builder that carries an `id`, a `tokio::sync::watch` channel is |
| +//! created: |
| +//! |
| +//! ```text |
| +//! watch::channel(None::<bool>) |
| +//! ↑ |
| +//! └─ initial value: None (pending) |
| +//! ``` |
| +//! |
| +//! The channel is later sent `Some(true)` (success) or `Some(false)` |
| +//! (failure). Dependents hold a cloned `Receiver` and call |
| +//! `wait_for(|v| v.is_some())`, which suspends the task cooperatively until |
| +//! the value changes. |
| +//! |
| +//! If a dependency's `watch::Sender` is dropped without ever sending a value |
| +//! (e.g. the task panicked), `wait_for` returns an `Err`, which is treated as |
| +//! a failure and the dependent is skipped. |
| +//! |
| +//! ### Phase 3 — Task spawning and collection |
| +//! |
| +//! All tasks are submitted to a single `tokio::task::JoinSet`. Every task |
| +//! follows the same lifecycle: |
| +//! |
| +//! - **Spawned** → waits for all declared dependencies. |
| +//! - **Running** → all deps succeeded; the builder executes. |
| +//! - **Skipped** → a dependency failed; returns `Ok(vec![])` and signals |
| +//! `Some(false)` so its own dependents also skip. |
| +//! - **Done / Failed** → signals `Some(true)` or `Some(false)`. |
| +//! |
| +//! After all tasks finish, collected [`ArtifactPath`]s are classified: |
| +//! |
| +//! - **File** artifacts → `dist/` (distribution binaries, archives, etc.). |
| +//! - **Directory** artifacts → `docs/` (rustdoc output, rendered Markdown, …). |
| +//! |
| +//! If any builder returned an error, the first error is propagated to the |
| +//! caller after all spinners have settled, so the full UI is always rendered |
| +//! to completion. |
| +//! |
| +//! ## Progress UI |
| +//! |
| +//! The UI is built with the `indicatif` crate. Each builder task owns: |
| +//! |
| +//! - A **parent spinner** inserted *above* a shared summary bar. |
| +//! - An **mpsc log channel** ([`LogSender`]) over which the builder streams |
| +//! events. |
| +//! - A **log-consumer task** that receives [`LogEvent`]s and updates the |
| +//! spinner. |
| +//! |
| +//! ### `LogEvent` variants |
| +//! |
| +//! | Variant | Meaning | |
| +//! |------------------------------|---------------------------------------------------| |
| +//! | `Line(String)` | Update the parent spinner message. | |
| +//! | `ChildStart { id, label }` | Create a sub-spinner below the parent. | |
| +//! | `ChildLine { id, line }` | Update a child spinner's message. | |
| +//! | `ChildFinish { id, … }` | Close a child spinner with ✓ or ✗. | |
| +//! |
| +//! The `ChildStart` / `ChildLine` / `ChildFinish` events are used only by the |
| +//! `cargo` builder when it compiles for multiple targets in parallel (see |
| +//! below). |
| +//! |
| +//! ## Inner Parallelism: `CargoBuilder` |
| +//! |
| +//! When a `cargo` builder lists multiple `targets`, it spawns one Tokio task |
| +//! per target inside its own inner `JoinSet` — a second level of concurrency |
| +//! nested inside the outer builder task. |
| +//! |
| +//! Each inner task: |
| +//! |
| +//! 1. Creates a [`tempfile::TempDir`] and passes it as `--target-dir` so the |
| +//! `cargo build` process does not contend with sibling builds on cargo's |
| +//! file lock. |
| +//! 2. Emits a `ChildStart` event so the UI creates a dedicated sub-spinner. |
| +//! 3. Uses `line_bridge` to adapt the plain-string stderr stream from |
| +//! `run_cargo_build` into `ChildLine` events on the parent [`LogSender`]. |
| +//! 4. After the build, copies artifacts to stable `target/<triple>/release/` |
| +//! paths before the `TempDir` is dropped. |
| +//! 5. Emits a `ChildFinish` event when done. |
| +//! |
| +//! Within `run_cargo_build` itself, stderr and the JSON stdout are consumed |
| +//! concurrently in separate `tokio::spawn` tasks so neither stream blocks the |
| +//! other. |
| +//! |
| +//! ## Other Builders |
| +//! |
| +//! | Builder | Async strategy | |
| +//! |-------------|---------------------------------------------------------------------------------| |
| +//! | `archive` | `tokio::task::spawn_blocking` for the CPU-bound tar walk; sends `Line` events. | |
| +//! | `cargo_doc` | Spawns one `tokio::spawn` to stream stderr; awaits `cargo doc` exit. | |
| +//! | `markdown` | Sequential per-file render loop; `spawn_blocking` for the directory walk. | |
| +//! | `script` | Runs commands sequentially; streams stdout **and** stderr in parallel tasks. | |
| + |
| use miette::Result; |
| use schemars::JsonSchema; |
| use serde::{Deserialize, Serialize}; |
| @@ -185,6 +360,324 @@ pub struct ArtifactPath { |
| pub hash: Option<String>, |
| } |
| |
| +/// # Implementing a New Builder |
| +/// |
| +/// This section walks you through adding a brand-new builder type to Abbaye, |
| +/// from an empty file to a fully working `[[builders]]` entry that users can |
| +/// drop into their `abbaye.toml`. |
| +/// |
| +/// The worked example is a **`make` builder** that runs one or more `make` |
| +/// targets and collects declared output paths as release artifacts. |
| +/// |
| +/// ## How Builders Plug In |
| +/// |
| +/// Every builder type is one variant of the [`AnyBuilder`] enum. The enum |
| +/// acts as the TOML-visible type tag (`type = "make"`) and routes calls down |
| +/// to the concrete implementation. The [`Builder`] trait is the interface |
| +/// every implementation must satisfy: |
| +/// |
| +/// ```text |
| +/// abbaye.toml ──► AnyBuilder enum ──► BuilderEntry |
| +/// │ |
| +/// ▼ |
| +/// Builder trait |
| +/// │ |
| +/// ▼ |
| +/// src/builders/make.rs |
| +/// ``` |
| +/// |
| +/// ## Step 1 — Create the builder file |
| +/// |
| +/// Create `src/builders/make.rs`. Start with imports that mirror what the |
| +/// existing builders use: |
| +/// |
| +/// ```rust,ignore |
| +/// use std::{path::PathBuf, process::Stdio}; |
| +/// |
| +/// use miette::{IntoDiagnostic, Result, miette}; |
| +/// use schemars::JsonSchema; |
| +/// use serde::{Deserialize, Serialize}; |
| +/// use tokio::io::{AsyncBufReadExt, BufReader}; |
| +/// use tokio::process::Command; |
| +/// |
| +/// use crate::builders::{ArtifactPath, Builder, LogEvent, LogSender}; |
| +/// ``` |
| +/// |
| +/// | Import | Purpose | |
| +/// |-----------------|-------------------------------------------------------------------| |
| +/// | `ArtifactPath` | The return type: a path plus display name and optional hash. | |
| +/// | `Builder` | The trait your struct must implement. | |
| +/// | `LogEvent` | The structured event type sent over the log channel. | |
| +/// | `LogSender` | An `mpsc::UnboundedSender<LogEvent>` — your handle to the UI. | |
| +/// |
| +/// ## Step 2 — Write the config struct |
| +/// |
| +/// The config struct holds every field that can appear in the TOML table. |
| +/// It must derive exactly these traits (all are required for the builder |
| +/// machinery): |
| +/// |
| +/// ```rust,ignore |
| +/// /// Configuration for [`MakeBuilder`]. |
| +/// #[derive(Debug, Default, Clone, Deserialize, Serialize, JsonSchema)] |
| +/// pub struct MakeBuilderConfig { |
| +/// /// `make` target to build. Defaults to the default target when absent. |
| +/// pub target: Option<String>, |
| +/// |
| +/// /// Path to the Makefile. Passed as `-f <path>`. |
| +/// pub makefile: Option<PathBuf>, |
| +/// |
| +/// /// Paths of the files produced by `make` to treat as release artifacts. |
| +/// #[serde(default)] |
| +/// pub outputs: Vec<PathBuf>, |
| +/// } |
| +/// ``` |
| +/// |
| +/// | Derive | Why it is required | |
| +/// |---------------|-------------------------------------------------------------------| |
| +/// | `Debug` | Diagnostic printing. | |
| +/// | `Default` | The [`Builder`] trait bound requires it for the associated type. | |
| +/// | `Clone` | [`BuilderEntry::clone`] is used when spawning the builder task. | |
| +/// | `Deserialize` | Figment deserialises the TOML table into this struct. | |
| +/// | `Serialize` | Needed for `abbaye init` and JSON Schema generation. | |
| +/// | `JsonSchema` | Powers `abbaye dump-schema` so editors get completion/validation. | |
| +/// |
| +/// Every public field should have a doc comment — these become the field |
| +/// descriptions in the generated JSON Schema and appear in editor tooltips. |
| +/// |
| +/// If `Default` cannot be derived because a field has a non-false/zero/empty |
| +/// default (e.g. a `bool` that defaults to `true`), implement `Default` |
| +/// manually and provide a `fn default_fieldname() -> T` free function for the |
| +/// `#[serde(default = "...")]` attribute. See [`crate::builders::cargo`] for |
| +/// an example. |
| +/// |
| +/// ## Step 3 — Implement the `Builder` trait |
| +/// |
| +/// Define a zero-sized marker struct and implement [`Builder`] for it: |
| +/// |
| +/// ```rust,ignore |
| +/// pub struct MakeBuilder; |
| +/// |
| +/// impl Builder for MakeBuilder { |
| +/// type ConfigType = MakeBuilderConfig; |
| +/// |
| +/// async fn build( |
| +/// &self, |
| +/// config: Self::ConfigType, |
| +/// version: &str, |
| +/// log: LogSender, |
| +/// ) -> Result<Vec<ArtifactPath>> { |
| +/// let mut cmd = Command::new("make"); |
| +/// |
| +/// // Always expose the version being built. |
| +/// cmd.env("ABBAYE_BUILDING_VERSION", version); |
| +/// |
| +/// if let Some(ref makefile) = config.makefile { |
| +/// cmd.arg("-f").arg(makefile); |
| +/// } |
| +/// if let Some(ref target) = config.target { |
| +/// cmd.arg(target); |
| +/// } |
| +/// |
| +/// let mut child = cmd |
| +/// .stdout(Stdio::piped()) |
| +/// .stderr(Stdio::piped()) |
| +/// .spawn() |
| +/// .into_diagnostic()?; |
| +/// |
| +/// // Stream stdout and stderr concurrently so neither pipe fills up |
| +/// // and deadlocks the child process. |
| +/// let stdout = child.stdout.take().expect("stdout was piped"); |
| +/// let stderr = child.stderr.take().expect("stderr was piped"); |
| +/// let log_out = log.clone(); |
| +/// let log_err = log.clone(); |
| +/// let stdout_task = tokio::spawn(async move { |
| +/// let mut lines = BufReader::new(stdout).lines(); |
| +/// while let Ok(Some(line)) = lines.next_line().await { |
| +/// let _ = log_out.send(LogEvent::Line(line)); |
| +/// } |
| +/// }); |
| +/// let stderr_task = tokio::spawn(async move { |
| +/// let mut lines = BufReader::new(stderr).lines(); |
| +/// while let Ok(Some(line)) = lines.next_line().await { |
| +/// let _ = log_err.send(LogEvent::Line(line)); |
| +/// } |
| +/// }); |
| +/// |
| +/// // Drain I/O tasks *before* checking the exit code to avoid |
| +/// // losing the last lines of output. |
| +/// let status = child.wait().await.into_diagnostic()?; |
| +/// let _ = tokio::join!(stdout_task, stderr_task); |
| +/// |
| +/// if !status.success() { |
| +/// return Err(miette!( |
| +/// "make failed with exit status: {}", |
| +/// status.code().unwrap_or(-1) |
| +/// )); |
| +/// } |
| +/// |
| +/// // Collect declared outputs as artifacts. |
| +/// let mut artifacts = Vec::new(); |
| +/// for output in &config.outputs { |
| +/// if !output.exists() { |
| +/// return Err(miette!( |
| +/// "declared output does not exist after make: {}", |
| +/// output.display() |
| +/// )); |
| +/// } |
| +/// let name = output |
| +/// .file_name() |
| +/// .ok_or_else(|| miette!("output path has no file name"))? |
| +/// .to_string_lossy() |
| +/// .into_owned(); |
| +/// artifacts.push(ArtifactPath { path: output.clone(), name, hash: None }); |
| +/// } |
| +/// Ok(artifacts) |
| +/// } |
| +/// } |
| +/// ``` |
| +/// |
| +/// ### Key conventions |
| +/// |
| +/// **Always set `ABBAYE_BUILDING_VERSION`.** Every builder must expose the |
| +/// version being built through this environment variable so that build scripts |
| +/// can embed it without extra plumbing. |
| +/// |
| +/// **Always drain I/O tasks before checking the exit status.** If you |
| +/// `.await` the child first, the subprocess's pipes may fill up and cause it |
| +/// to block forever before it can exit. |
| +/// |
| +/// **Errors from `log.send(…)` are silently ignored** (`let _ = …`). The |
| +/// receiver may already be gone by the time a builder finishes — that is |
| +/// expected and harmless. |
| +/// |
| +/// **File artifacts → `dist/`, directory artifacts → `docs/`.** The |
| +/// orchestrator in `site.rs` distinguishes them by `path.is_dir()`. Return |
| +/// a directory path only for documentation trees. |
| +/// |
| +/// **CPU-bound or blocking I/O must use `spawn_blocking`.** Tokio runs on a |
| +/// small thread pool; blocking it starves other tasks. |
| +/// |
| +/// ## Step 4 — Register in `mod.rs` |
| +/// |
| +/// Open `src/builders/mod.rs` and make four additions. |
| +/// |
| +/// **4a — Declare the module:** |
| +/// |
| +/// ```rust,ignore |
| +/// pub mod make; |
| +/// ``` |
| +/// |
| +/// **4b — Import the types:** |
| +/// |
| +/// ```rust,ignore |
| +/// use make::{MakeBuilder, MakeBuilderConfig}; |
| +/// ``` |
| +/// |
| +/// **4c — Add a variant to [`AnyBuilder`]** with a doc comment containing a |
| +/// canonical TOML usage example (this becomes the JSON Schema description): |
| +/// |
| +/// ```rust,ignore |
| +/// /// Runs `make [target]` and collects declared output paths as artifacts. |
| +/// /// |
| +/// /// ```toml |
| +/// /// [[builders]] |
| +/// /// type = "make" |
| +/// /// target = "release" # optional |
| +/// /// outputs = ["dist/mybin"] |
| +/// /// ``` |
| +/// Make(MakeBuilderConfig), |
| +/// ``` |
| +/// |
| +/// The variant name is lowercased by `rename_all = "snake_case"`, so `Make` |
| +/// becomes `type = "make"` in TOML. Multi-word names use `CamelCase` in |
| +/// Rust and become `snake_case` in TOML (e.g. `WasmPack` → `wasm_pack`). |
| +/// |
| +/// **4d — Wire up `label()` and `build()`:** |
| +/// |
| +/// ```rust,ignore |
| +/// Self::Make(_) => "make", // in label() |
| +/// Self::Make(c) => MakeBuilder.build(c.clone(), version, log).await, // in build() |
| +/// ``` |
| +/// |
| +/// `label()` provides the default spinner prefix when the user hasn't given |
| +/// the builder an `id`. Keep it short and lowercase-kebab. |
| +/// |
| +/// ## Step 5 — Verify |
| +/// |
| +/// ```sh |
| +/// cargo build |
| +/// cargo run -- dump-schema | python3 -m json.tool | grep -A 20 '"make"' |
| +/// ``` |
| +/// |
| +/// ## Step 6 — Use it in `abbaye.toml` |
| +/// |
| +/// ```toml |
| +/// [[builders]] |
| +/// type = "make" |
| +/// target = "release" |
| +/// outputs = ["build/myprogram"] |
| +/// |
| +/// # With dependency ordering: |
| +/// [[builders]] |
| +/// type = "make" |
| +/// id = "make-release" |
| +/// target = "release" |
| +/// outputs = ["build/myprogram"] |
| +/// |
| +/// [[builders]] |
| +/// type = "script" |
| +/// script = ["upx build/myprogram"] |
| +/// outputs = ["build/myprogram"] |
| +/// depends_on = ["make-release"] |
| +/// ``` |
| +/// |
| +/// ## Advanced — Parallel sub-tasks with child spinners |
| +/// |
| +/// If your builder naturally fans out into independent sub-tasks (like the |
| +/// `cargo` builder running one `cargo build` per target triple), you can run |
| +/// them in parallel and give each its own sub-spinner in the UI. |
| +/// |
| +/// The pattern is always the same: |
| +/// |
| +/// 1. Send [`LogEvent::ChildStart`] `{ id, label }` to create the sub-spinner. |
| +/// 2. Forward output lines as [`LogEvent::ChildLine`] `{ id, line }`. |
| +/// 3. Send [`LogEvent::ChildFinish`] `{ id, success, summary }` to close it. |
| +/// |
| +/// ```rust,ignore |
| +/// join_set.spawn(async move { |
| +/// let _ = log.send(LogEvent::ChildStart { |
| +/// id: target.clone(), |
| +/// label: target.clone(), |
| +/// }); |
| +/// |
| +/// // ... do work, sending ChildLine events ... |
| +/// |
| +/// let _ = log.send(LogEvent::ChildFinish { |
| +/// id: target.clone(), |
| +/// success: result.is_ok(), |
| +/// summary: "done".to_owned(), |
| +/// }); |
| +/// }); |
| +/// ``` |
| +/// |
| +/// The `line_bridge` helper in [`crate::builders::cargo`] is a reusable |
| +/// pattern for adapting a plain-string sender to structured [`LogEvent`]s — |
| +/// feel free to copy or generalise it. |
| +/// |
| +/// ## Reviewer checklist |
| +/// |
| +/// - Config struct derives `Debug`, `Default` (or has a manual impl), |
| +/// `Clone`, `Deserialize`, `Serialize`, `JsonSchema`. |
| +/// - Every public config field has a doc comment. |
| +/// - The builder sets `ABBAYE_BUILDING_VERSION` in every subprocess |
| +/// environment. |
| +/// - Stdout and stderr are consumed concurrently to prevent pipe deadlocks. |
| +/// - Log send errors are silently ignored (`let _ = log.send(…)`). |
| +/// - Blocking I/O uses `tokio::task::spawn_blocking`. |
| +/// - Both `label()` and `build()` arms are added to [`AnyBuilder`]. |
| +/// - `pub mod` declaration and `use` imports are present in `mod.rs`. |
| +/// - `cargo build` succeeds with no new warnings. |
| +/// - `abbaye dump-schema` shows the new variant and all its fields. |
| #[allow(async_fn_in_trait)] |
| pub trait Builder { |
| type ConfigType: Default + for<'de> Deserialize<'de> + Clone; |