Add a package manager: rex init/add/remove/get/update/deps

Packages come from git (a registry can be added later; a bare version in
rex.toml is reserved for it). rex.toml [deps] names them, rex.lock pins exact
commits, and run/build/check/test download what is missing. `import json`
reads the package's json.rex; package files get `@json...` module keys so they
never mix with the project's own files. Newest tag of the same major version
wins when packages disagree. The language server reads downloaded packages.
Docs: docs/14-packages.md.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This commit is contained in:
Your NameandClaude Opus 5.5 committed 2026-11-16 10:30:11 +01:00
1 parent 300d767124
commit 15da1e2379
16 files changed
+1588 -61

No files matched your search

+4 -2
View File
@@ -65,8 +65,10 @@ error: age cannot be negative
closures, maps, characters, modules. closures, maps, characters, modules.
- **C and C++ libraries.** `import "raylib.h"` and call it: Rex reads the header itself, - **C and C++ libraries.** `import "raylib.h"` and call it: Rex reads the header itself,
links the library, and builds your own `.c`/`.cpp` files next to the program. links the library, and builds your own `.c`/`.cpp` files next to the program.
- **Tooling included.** A formatter, a language server, and editor files, all in the one `rex` - **Packages from git.** `rex add git.joji.li/joji/json`, then `import json`. Exact versions are
program. kept in `rex.lock`; `rex update` moves to newer ones when you ask.
- **Tooling included.** A formatter, a language server, a package manager and editor files, all
in the one `rex` program.
## Get started ## Get started
+13
View File
@@ -212,6 +212,19 @@ Targets: game dev, CLI tools, desktop apps.
`CLITERAL(Color){...}` (emitted as the macro). Programs that import headers are built with `-std=gnu11`. `CLITERAL(Color){...}` (emitted as the macro). Programs that import headers are built with `-std=gnu11`.
Not supported (a clear error when called): variadic functions, callbacks, unions/arrays by value, lists of Not supported (a clear error when called): variadic functions, callbacks, unions/arrays by value, lists of
structs, `long double`, function-like macros. Rex never frees C memory. `rex header x.h [word]` lists what Rex sees. structs, `long double`, function-like macros. Rex never frees C memory. `rex header x.h [word]` lists what Rex sees.
- **Packages** (`src/pkg.rs`): git repositories (a registry may come later; `json = "1.2.0"` is reserved for
it and is an error now). `rex.toml` `[deps]` maps a name to `"host/owner/repo@rev"` (https is added; `://`
and `git@host:` addresses kept as written; rev = tag, branch or commit; none = main branch) or a folder
(`"../lib"`). `rex.lock` (lines `name url rev commit`) pins commits. Cache: `$REX_HOME` or `~/.cache/rex`
(`git/` bare mirrors, `pkg/<url>/<commit>/` unpacked with `git archive`). `run/build/check/test` download what
is missing and rewrite the lock when rex.toml changed; the LSP only reads what is downloaded. Versions: one per
package name in a build; same repo + same major -> the newest tag wins (re-walk), else an error naming both.
The project's folder deps override. `import json` -> the package's `json.rex`, `import json.p` -> its `p.rex`;
inside the package, imports count from its top folder. Module keys of package files start with `@`
(`@json.p`, see `loader::Scope`), and the loader sets `Import.alias` to the written last word. A project file
`json.rex` next to a package `json` is an error. Commands: `rex init [name]`, `rex add <addr>[@rev] [--name n]`
(newest tag when none given; name from the package's `[package] name`, else the last address part),
`rex remove`, `rex get`, `rex update [names]` (newest tag of the same major; branch deps move), `rex deps`.
- **String slicing**: `s[a..b]`, `s[a..]`, `s[..b]` (byte positions, bounds checked; panics with "the length is N"). - **String slicing**: `s[a..b]`, `s[a..]`, `s[..b]` (byte positions, bounds checked; panics with "the length is N").
- **Running**: `rex run file.rex a b c` passes `a b c` to the program (`os.args`). - **Running**: `rex run file.rex a b c` passes `a b c` to the program (`os.args`).
- **Example**: `examples/calc/` is a multi-file calculator (tokenizer, parser, evaluator, shared scope); `session.txt` / `session.out` are checked by a test. - **Example**: `examples/calc/` is a multi-file calculator (tokenizer, parser, evaluator, shared scope); `session.txt` / `session.out` are checked by a test.
+3
View File
@@ -58,6 +58,9 @@ Klein/
import engine.engine // the same line works in game/app/main.rex and in any other file import engine.engine // the same line works in game/app/main.rex and in any other file
``` ```
`rex init` writes one for you. The same file lists the packages the project uses
(see [chapter 14](14-packages.md)).
Rex finds the nearest `rex.toml` above the file you run (and above each file you open in an Rex finds the nearest `rex.toml` above the file you run (and above each file you open in an
editor). Without one, imports are counted from the folder of the file you run, as above. With editor). Without one, imports are counted from the folder of the file you run, as above. With
one, even a file next to your main file is imported with its full path (`import game.app.util`). one, even a file next to your main file is imported with its full path (`import game.app.util`).
+3
View File
@@ -124,6 +124,7 @@ import tools.fs // tools/fs.rex, used as fs.name
import ..shared // ../shared.rex (...x = two folders up) import ..shared // ../shared.rex (...x = two folders up)
// rex.toml in the top folder: imports count from there, in every file // rex.toml in the top folder: imports count from there, in every file
import os // built-in: os math time rand import os // built-in: os math time rand
import json // a package: `rex add git.joji.li/joji/json` (see chapter 14)
pub func f() {} // visible to other files pub func f() {} // visible to other files
``` ```
@@ -147,6 +148,8 @@ lib.fill(buf, n) // T* parameters take a variable or a list
## Commands ## Commands
`rex run f.rex args…` · `rex build f.rex -o out` · `rex check f.rex` · `rex test [path]` · `rex emit-c f.rex` · `rex fmt path` (`--check` to only report) · `rex lsp` (editor server) · `rex header x.h [word]` (what a C header offers) `rex run f.rex args…` · `rex build f.rex -o out` · `rex check f.rex` · `rex test [path]` · `rex emit-c f.rex` · `rex fmt path` (`--check` to only report) · `rex lsp` (editor server) · `rex header x.h [word]` (what a C header offers)
Packages: `rex init` · `rex add git.host/owner/name[@v1.2.0]` (or `../folder`) · `rex remove name` · `rex update [name]` · `rex get` · `rex deps`
## Things that stop the program (exit code 2) ## Things that stop the program (exit code 2)
Divide by zero · list index/slice out of range · `pop` on an empty list · reading a missing Divide by zero · list index/slice out of range · `pop` on an empty list · reading a missing
map key with `m[k]` · using a `nil` or gone `weak` value · calling a `nil` function or interface · map key with `m[k]` · using a `nil` or gone `weak` value · calling a `nil` function or interface ·
+132
View File
@@ -0,0 +1,132 @@
# 14. Packages
A **package** is Rex code from another project that you use in yours: a JSON reader,
a game library, helpers a friend wrote. Rex gets packages straight from **git**: any git
repository with Rex files in it is a package. There is no website you have to upload to.
## Using a package
Start a project (once), then add the package with its git address:
```text
$ rex init
wrote rex.toml (the project `game`)
$ rex add git.joji.li/joji/json
fetching git.joji.li/joji/json
downloading json v1.2.0
added json v1.2.0 (use it with `import json`)
```
Now import it like one of your own files:
```rex
import json
func main() {
var v = json.parse("[1, 2]")?
print(v)
}
```
`rex run`, `rex build`, `rex check` and `rex test` download whatever is missing by themselves.
- The address is written without `https://` (it is added for you). `https://...`,
`ssh://...` and `git@host:owner/repo` work too, for private repositories.
- Without a version, `rex add` takes the newest one. Pick one with `@`:
`rex add git.joji.li/joji/json@v1.1.0`. After the `@` can also come a branch
(`@main`) or a commit.
- A repository without any versions is followed on its main branch.
- `rex remove json` stops using it.
## rex.toml and rex.lock
`rex add` writes the package into `rex.toml`, in the top folder of your project:
```toml
[package]
name = "game"
[deps]
json = "git.joji.li/joji/json@v1.2.0"
mylib = "../mylib"
```
You can edit this file yourself too; the next `rex run` picks it up.
Next to it, rex writes **`rex.lock`**. It remembers the exact commit of every package,
so your program builds the same tomorrow, on another computer, even if a package's author
changes something. Keep both files with your code (commit them). Don't edit `rex.lock`.
## Newer versions
Versions are git tags like `v1.2.0`: **major.minor.patch**. A change in the first number
(`v2.0.0`) means the package changed in ways that may break your code.
```text
$ rex update
json v1.2.0 -> v1.4.1
json: v2.0.0 is out too (a new major version may need changes in your code: `rex add git.joji.li/joji/json@v2.0.0`)
```
`rex update` moves every package to its newest version **with the same first number**, and
packages that follow a branch to its newest commit. `rex update json` does only that one.
Jumping to a new major version is up to you, with `rex add ...@v2.0.0`.
`rex deps` shows what your project uses, and what those packages use in turn:
```text
$ rex deps
json v1.4.1
strs v1.0.3
mylib (folder ../mylib)
```
## A package in a folder
A folder on your computer works as a package too: `rex add ../mylib`. Changes in that
folder show up at the next run, with nothing to update. It is the easy way to share code
between your own projects, or to try changes to a package before you publish them.
## Making a package
A package is a folder (a git repository, to share it) with a `rex.toml` and a file with the
package's name:
```text
json/
rex.toml [package] name = "json"
json.rex what `import json` gives
parser.rex what `import json.parser` gives
```
```rex
// json.rex
import parser // the package's own parser.rex
pub func parse(text: string): (parser.Value, error) {
return parser.read(text)
}
```
- Imports inside a package count from the package's top folder, as in any project. A
package's `parser.rex` and your own `parser.rex` are different files, so names never clash.
- A package can use other packages: list them in its own `rex.toml` under `[deps]`.
Rex downloads those too. If two packages want different versions of the same package, the
newer one is used (as long as the first number is the same; otherwise it's an error that
names who wants what).
- Only `pub` things can be used from outside, as with your own files.
- To publish a version: `git tag v1.0.0` and `git push --tags`.
## Good to know
- Packages are kept in `~/.cache/rex` (or the folder in `$REX_HOME`). Deleting it is safe:
they come back at the next run.
- Once everything is downloaded, building needs no internet.
- Your editor sees packages that are downloaded. After adding one by hand to `rex.toml`, run
`rex get` (or any `rex run`) so the editor can find it.
- A package's name is the same as its `[package] name`, written in lowercase letters,
digits and `_`. If yours has a file `json.rex` and you add a package called `json`, Rex
says so: rename one of them.
- There is no package website yet. `json = "1.2.0"` (a version without an address) is kept
for that and is an error for now.
+1
View File
@@ -22,6 +22,7 @@ These notes teach it from zero. Read them in order, or jump to the
| 11 | [Cheatsheet](11-cheatsheet.md) | Everything on one page | | 11 | [Cheatsheet](11-cheatsheet.md) | Everything on one page |
| 12 | [A real program](12-calculator-walkthrough.md) | A calculator, split over five files | | 12 | [A real program](12-calculator-walkthrough.md) | A calculator, split over five files |
| 13 | [C and C++ libraries](13-c-libraries.md) | `import "raylib.h"`, your own C code | | 13 | [C and C++ libraries](13-c-libraries.md) | `import "raylib.h"`, your own C code |
| 14 | [Packages](14-packages.md) | `rex add`, using code from other projects |
## How to read the examples ## How to read the examples
+12 -2
View File
@@ -1075,7 +1075,9 @@ impl Checker {
if i.c { if i.c {
return self.import_c(i); return self.import_c(i);
} }
let alias = i.path.rsplit('.').next().unwrap().to_string(); // a package's files have keys like `@json.parser` (see `loader::Scope`); the loader then
// says the name it is used by
let alias = i.alias.clone().unwrap_or_else(|| i.path.rsplit('.').next().unwrap().to_string());
if self.imports.contains_key(&alias) { if self.imports.contains_key(&alias) {
return Err(Diag::new(format!("`{alias}` is imported twice"), i.span)); return Err(Diag::new(format!("`{alias}` is imported twice"), i.span));
} }
@@ -3713,7 +3715,15 @@ impl Checker {
Some(m) => self.expr_as(m, Ty::Str)?, Some(m) => self.expr_as(m, Ty::Str)?,
None => mk(HExprKind::Str(String::new()), Ty::Str), None => mk(HExprKind::Str(String::new()), Ty::Str),
}; };
let file = if self.cur_mod.is_empty() { String::new() } else { format!("{}.rex ", self.cur_mod.replace('.', "/")) }; let file = match self.cur_mod.strip_prefix('@') {
_ if self.cur_mod.is_empty() => String::new(),
// a package's file: `json: parser.rex`
Some(k) => match k.split_once('.') {
Some((pkg, rest)) => format!("{pkg}: {}.rex ", rest.replace('.', "/")),
None => format!("{k}: {k}.rex "),
},
None => format!("{}.rex ", self.cur_mod.replace('.', "/")),
};
let place = mk(HExprKind::Str(format!("{file}line {}", span.line)), Ty::Str); let place = mk(HExprKind::Str(format!("{file}line {}", span.line)), Ty::Str);
Ok(mk(HExprKind::Builtin { c: "rex_assert", shape: BuiltinShape::Plain, args: vec![cond, place, msg] }, Ty::Unit)) Ok(mk(HExprKind::Builtin { c: "rex_assert", shape: BuiltinShape::Plain, args: vec![cond, place, msg] }, Ty::Unit))
} }
+36 -37
View File
@@ -7,19 +7,15 @@ use std::time::{SystemTime, UNIX_EPOCH};
use crate::ast::{Item, Program}; use crate::ast::{Item, Program};
use crate::error::{Diag, PResult}; use crate::error::{Diag, PResult};
use crate::{cheader, check, codegen_c, loader, parser}; use crate::{cheader, check, codegen_c, loader, parser, pkg};
/// Read the C headers imported by the program. `dir` is the folder of the main file. /// Read the C headers imported by the program. `dir(key)` is the folder module `key` is in
pub fn load_c_headers(main: &Program, modules: &[(String, Program)], dir: &Path) -> PResult<Vec<cheader::Lib>> { /// (`""`: the main file), where its headers are looked for first.
pub fn load_c_headers(main: &Program, modules: &[(String, Program)], dir: &dyn Fn(&str) -> PathBuf) -> PResult<Vec<cheader::Lib>> {
let mut libs: Vec<cheader::Lib> = Vec::new(); let mut libs: Vec<cheader::Lib> = Vec::new();
let all = std::iter::once(("", main)).chain(modules.iter().map(|(k, p)| (k.as_str(), p))); let all = std::iter::once(("", main)).chain(modules.iter().map(|(k, p)| (k.as_str(), p)));
for (key, p) in all { for (key, p) in all {
// `import a.b` lives in a/b.rex: its headers are found next to it let here = dir(key);
let mut here = dir.to_path_buf();
let parts: Vec<&str> = key.split('.').collect();
for part in &parts[..parts.len().saturating_sub(1)] {
here.push(part);
}
for item in &p.items { for item in &p.items {
let Item::Import(i) = item else { continue }; let Item::Import(i) = item else { continue };
if !i.c || libs.iter().any(|l| l.header == i.path) { if !i.c || libs.iter().any(|l| l.header == i.path) {
@@ -33,11 +29,20 @@ pub fn load_c_headers(main: &Program, modules: &[(String, Program)], dir: &Path)
Ok(libs) Ok(libs)
} }
/// The folder of module `key` in a project without packages whose files are counted from `dir`.
fn plain_dir(dir: &Path, key: &str) -> PathBuf {
if key.is_empty() {
return dir.to_path_buf();
}
let file = dir.join(loader::key_file(key));
file.parent().map(Path::to_path_buf).unwrap_or_default()
}
/// Rex source -> C source, for a program in a single file. C headers it imports are looked /// Rex source -> C source, for a program in a single file. C headers it imports are looked
/// for in the current folder and the system's. /// for in the current folder and the system's.
pub fn compile_to_c(src: &str) -> PResult<String> { pub fn compile_to_c(src: &str) -> PResult<String> {
let ast = parser::parse(src)?; let ast = parser::parse(src)?;
let libs = load_c_headers(&ast, &[], Path::new("."))?; let libs = load_c_headers(&ast, &[], &|k| plain_dir(Path::new("."), k))?;
let hir = check::check_project_c(&ast, &[], libs)?; let hir = check::check_project_c(&ast, &[], libs)?;
Ok(codegen_c::generate(&hir)) Ok(codegen_c::generate(&hir))
} }
@@ -49,18 +54,20 @@ pub fn compile_project(
main_src: &str, main_src: &str,
read: impl FnMut(&str) -> Option<(String, String)>, read: impl FnMut(&str) -> Option<(String, String)>,
) -> Result<String, String> { ) -> Result<String, String> {
Ok(codegen_c::generate(&check_project(main_path, main_src, read, false)?)) let project = pkg::Project::bare(loader::project_dir(Path::new(main_path)));
Ok(codegen_c::generate(&check_project(main_path, main_src, &project, read, false)?))
} }
/// Load and check a program; `module_ok`: the main file may lack `main` (it is a module). /// Load and check a program; `module_ok`: the main file may lack `main` (it is a module).
fn check_project( fn check_project(
main_path: &str, main_path: &str,
main_src: &str, main_src: &str,
project: &pkg::Project,
mut read: impl FnMut(&str) -> Option<(String, String)>, mut read: impl FnMut(&str) -> Option<(String, String)>,
module_ok: bool, module_ok: bool,
) -> Result<crate::hir::HProgram, String> { ) -> Result<crate::hir::HProgram, String> {
let mut seen: HashMap<String, (String, String)> = HashMap::new(); let mut seen: HashMap<String, (String, String)> = HashMap::new();
let loaded = loader::load(main_src, &mut |key| { let loaded = loader::load_in(main_src, &project.scope(), &mut |key| {
let found = read(key); let found = read(key);
if let Some((path, src)) = &found { if let Some((path, src)) = &found {
seen.insert(key.to_string(), (path.clone(), src.clone())); seen.insert(key.to_string(), (path.clone(), src.clone()));
@@ -77,8 +84,7 @@ fn check_project(
let loaded = loaded.map_err(&render)?; let loaded = loaded.map_err(&render)?;
let modules: Vec<(String, crate::ast::Program)> = let modules: Vec<(String, crate::ast::Program)> =
loaded.modules.into_iter().map(|m| (m.key, m.program)).collect(); loaded.modules.into_iter().map(|m| (m.key, m.program)).collect();
let dir = loader::project_dir(Path::new(main_path)); let libs = load_c_headers(&loaded.main, &modules, &|k| project.module_dir(k)).map_err(&render)?;
let libs = load_c_headers(&loaded.main, &modules, &dir).map_err(&render)?;
if module_ok { if module_ok {
check::check_module_c(&loaded.main, &modules, libs).map_err(&render) check::check_module_c(&loaded.main, &modules, libs).map_err(&render)
} else { } else {
@@ -86,39 +92,32 @@ fn check_project(
} }
} }
/// The program at `path` with its project: the source, and the packages (downloaded if missing).
fn open(path: &str) -> Result<(String, pkg::Project), String> {
let src = std::fs::read_to_string(path).map_err(|e| format!("error: cannot read {path}: {e}"))?;
let project = pkg::Project::for_file(Path::new(path), pkg::Mode::Online).map_err(|e| format!("error: {e}"))?;
Ok((src, project))
}
/// Compile the program whose main file is `path`; imports are found next to it. /// Compile the program whose main file is `path`; imports are found next to it.
pub fn compile_file(path: &str) -> Result<String, String> { pub fn compile_file(path: &str) -> Result<String, String> {
let src = std::fs::read_to_string(path).map_err(|e| format!("error: cannot read {path}: {e}"))?; let (src, project) = open(path)?;
let dir = loader::project_dir(Path::new(path)); Ok(codegen_c::generate(&check_project(path, &src, &project, |k| project.read(k), false)?))
compile_project(path, &src, |key| {
let file = dir.join(loader::key_file(key));
std::fs::read_to_string(&file).ok().map(|s| (file.display().to_string(), s))
})
} }
/// Check the file at `path` (and what it imports) without building it. Unlike /// Check the file at `path` (and what it imports) without building it. Unlike
/// `compile_file`, a file without `main` is fine: it may be a module. /// `compile_file`, a file without `main` is fine: it may be a module.
pub fn check_file(path: &str) -> Result<(), String> { pub fn check_file(path: &str) -> Result<(), String> {
let src = std::fs::read_to_string(path).map_err(|e| format!("error: cannot read {path}: {e}"))?; let (src, project) = open(path)?;
let dir = loader::project_dir(Path::new(path)); check_project(path, &src, &project, |k| project.read(k), true).map(|_| ())
let read = |key: &str| {
let file = dir.join(loader::key_file(key));
std::fs::read_to_string(&file).ok().map(|s| (file.display().to_string(), s))
};
check_project(path, &src, read, true).map(|_| ())
} }
/// The program in `path` built for `rex test`: the C code, and the names of its test functions /// The program in `path` built for `rex test`: the C code, and the names of its test functions
/// (`func test_...()` in that file, which take nothing and return nothing or an `error`), in the /// (`func test_...()` in that file, which take nothing and return nothing or an `error`), in the
/// order they were written. The C program runs test number `n` when started with argument `n`. /// order they were written. The C program runs test number `n` when started with argument `n`.
pub fn compile_tests(path: &str) -> Result<(String, Vec<String>), String> { pub fn compile_tests(path: &str) -> Result<(String, Vec<String>), String> {
let src = std::fs::read_to_string(path).map_err(|e| format!("error: cannot read {path}: {e}"))?; let (src, project) = open(path)?;
let dir = loader::project_dir(Path::new(path)); let hir = check_project(path, &src, &project, |k| project.read(k), true)?;
let read = |key: &str| {
let file = dir.join(loader::key_file(key));
std::fs::read_to_string(&file).ok().map(|s| (file.display().to_string(), s))
};
let hir = check_project(path, &src, read, true)?;
// in the order written in the file // in the order written in the file
let ast = parser::parse(&src).map_err(|d| d.render(path, &src))?; let ast = parser::parse(&src).map_err(|d| d.render(path, &src))?;
let mut tests = Vec::new(); let mut tests = Vec::new();
@@ -242,13 +241,13 @@ pub fn build_exe_with(c_code: &str, out: &Path, flags: &[&str]) -> Result<(), St
/// The error's `file` says which module it is in (`None`: the main file). /// The error's `file` says which module it is in (`None`: the main file).
pub fn diagnose( pub fn diagnose(
main_src: &str, main_src: &str,
dir: &Path, project: &pkg::Project,
mut read: impl FnMut(&str) -> Option<(String, String)>, mut read: impl FnMut(&str) -> Option<(String, String)>,
) -> Result<(), Diag> { ) -> Result<(), Diag> {
let loaded = loader::load(main_src, &mut |key| read(key))?; let loaded = loader::load_in(main_src, &project.scope(), &mut |key| read(key))?;
let modules: Vec<(String, crate::ast::Program)> = let modules: Vec<(String, crate::ast::Program)> =
loaded.modules.into_iter().map(|m| (m.key, m.program)).collect(); loaded.modules.into_iter().map(|m| (m.key, m.program)).collect();
let libs = load_c_headers(&loaded.main, &modules, dir)?; let libs = load_c_headers(&loaded.main, &modules, &|k| project.module_dir(k))?;
check::check_module_c(&loaded.main, &modules, libs)?; check::check_module_c(&loaded.main, &modules, libs)?;
Ok(()) Ok(())
} }
+1
View File
@@ -15,5 +15,6 @@ pub mod loader;
pub mod lsp; pub mod lsp;
pub mod moves; pub mod moves;
pub mod parser; pub mod parser;
pub mod pkg;
pub mod stdlib; pub mod stdlib;
pub mod types; pub mod types;
+77 -12
View File
@@ -2,6 +2,7 @@
//! folder of the project (the main file's folder, or the nearest one above it with a //! folder of the project (the main file's folder, or the nearest one above it with a
//! `rex.toml`); `import a.b` loads `a/b.rex`; `import ..a` loads `../a.rex`. //! `rex.toml`); `import a.b` loads `a/b.rex`; `import ..a` loads `../a.rex`.
use std::collections::HashMap;
use std::path::{Path, PathBuf}; use std::path::{Path, PathBuf};
use crate::ast::{Item, Program}; use crate::ast::{Item, Program};
@@ -48,21 +49,74 @@ pub struct Loaded {
pub modules: Vec<Module>, pub modules: Vec<Module>,
} }
/// Which imports go to packages (see `pkg`). A package's files get keys starting with `@`:
/// `import json.parser` in the project is `@json.parser` when `json` is one of its packages, and
/// inside the package `json`, `import parser` (its own file) is `@json.parser` too. So a
/// package's files never mix with the project's, or with another package's.
#[derive(Debug, Default, Clone)]
pub struct Scope {
/// The project's own packages.
pub root_deps: Vec<String>,
/// Every package in the build, with the packages it uses.
pub packages: HashMap<String, Vec<String>>,
}
impl Scope {
/// The key for `import path` written in the module `from` (`""`: the main file).
pub fn key(&self, from: &str, path: &str) -> Result<String, String> {
let first = path.split('.').next().unwrap_or("");
let owner = from.strip_prefix('@').map(|k| k.split('.').next().unwrap_or(k));
match owner.and_then(|o| self.packages.get_key_value(o)) {
None if self.root_deps.iter().any(|d| d == first) => Ok(format!("@{path}")),
None => Ok(path.to_string()),
Some(_) if path.starts_with('.') => {
Err(format!("a package can only import its own files and its packages, not `{path}`"))
}
Some((o, deps)) if first == o || deps.iter().any(|d| d == first) => Ok(format!("@{path}")),
Some((o, _)) => Ok(format!("@{o}.{path}")),
}
}
}
/// `read(key)` returns the display path and the source of a module, if it exists. /// `read(key)` returns the display path and the source of a module, if it exists.
pub fn load(main_src: &str, read: &mut dyn FnMut(&str) -> Option<(String, String)>) -> PResult<Loaded> { pub fn load(main_src: &str, read: &mut dyn FnMut(&str) -> Option<(String, String)>) -> PResult<Loaded> {
let main = parser::parse(main_src)?; load_in(main_src, &Scope::default(), read)
}
/// Like `load`, for a project with packages.
pub fn load_in(main_src: &str, scope: &Scope, read: &mut dyn FnMut(&str) -> Option<(String, String)>) -> PResult<Loaded> {
let mut main = parser::parse(main_src)?;
resolve(&mut main, "", scope)?;
let mut modules = Vec::new(); let mut modules = Vec::new();
let mut stack = Vec::new(); let mut stack = Vec::new();
load_imports(&main, "", read, &mut modules, &mut stack)?; load_imports(&main, "", read, &mut modules, &mut stack, scope)?;
Ok(Loaded { main, modules }) Ok(Loaded { main, modules })
} }
/// Turn the import paths of module `from` into module keys. The name the module is used by
/// stays the last word that was written.
fn resolve(p: &mut Program, from: &str, scope: &Scope) -> PResult<()> {
for item in &mut p.items {
let Item::Import(i) = item else { continue };
if i.c || stdlib::MODULES.contains(&i.path.as_str()) {
continue;
}
let key = scope.key(from, &i.path).map_err(|m| Diag::new(m, i.span).in_module(from))?;
if key != i.path {
i.alias = Some(i.path.rsplit('.').next().unwrap().to_string());
i.path = key;
}
}
Ok(())
}
fn load_imports( fn load_imports(
p: &Program, p: &Program,
from: &str, from: &str,
read: &mut dyn FnMut(&str) -> Option<(String, String)>, read: &mut dyn FnMut(&str) -> Option<(String, String)>,
modules: &mut Vec<Module>, modules: &mut Vec<Module>,
stack: &mut Vec<String>, stack: &mut Vec<String>,
scope: &Scope,
) -> PResult<()> { ) -> PResult<()> {
for item in &p.items { for item in &p.items {
let Item::Import(i) = item else { continue }; let Item::Import(i) = item else { continue };
@@ -79,19 +133,30 @@ fn load_imports(
return Err(Diag::new(format!("circular import: {}", cycle.join(" -> ")), i.span).in_module(from)); return Err(Diag::new(format!("circular import: {}", cycle.join(" -> ")), i.span).in_module(from));
} }
let Some((_, src)) = read(&key) else { let Some((_, src)) = read(&key) else {
let file = key_file(&key); let msg = if let Some(k) = key.strip_prefix('@') {
return Err(Diag::new( let (pkg, rest) = k.split_once('.').unwrap_or((k, ""));
format!( let file = if rest.is_empty() { format!("{pkg}.rex") } else { key_file(rest) };
"cannot find module `{key}` (looked for {file} next to your program; built-in modules: {})", format!("the package `{pkg}` has no file {file}")
} else {
let mut msg = format!(
"cannot find module `{key}` (looked for {} next to your program; built-in modules: {})",
key_file(&key),
stdlib::MODULES.join(", ") stdlib::MODULES.join(", ")
), );
i.span, let first = key.split('.').next().unwrap_or("");
) if scope.packages.contains_key(first) {
.in_module(from)); msg.push_str(&format!(
"\n`{first}` is a package that one of your packages uses; to use it yourself, add it to your rex.toml too"
));
}
msg
};
return Err(Diag::new(msg, i.span).in_module(from));
}; };
let program = parser::parse(&src).map_err(|d| d.in_module(&key))?; let mut program = parser::parse(&src).map_err(|d| d.in_module(&key))?;
resolve(&mut program, &key, scope)?;
stack.push(key.clone()); stack.push(key.clone());
load_imports(&program, &key, read, modules, stack)?; load_imports(&program, &key, read, modules, stack, scope)?;
stack.pop(); stack.pop();
modules.push(Module { key, program }); modules.push(Module { key, program });
} }
+16 -6
View File
@@ -166,12 +166,13 @@ impl Server {
/// Check the file and build the `publishDiagnostics` message. /// Check the file and build the `publishDiagnostics` message.
fn diagnostics(&self, uri: &str) -> Json { fn diagnostics(&self, uri: &str) -> Json {
let text = &self.docs[uri]; let text = &self.docs[uri];
let dir: Option<PathBuf> = uri_to_path(uri).map(|p| crate::loader::project_dir(&p)); let path = uri_to_path(uri);
let project = project_of(path.as_deref());
let docs = &self.docs; let docs = &self.docs;
let result = std::panic::catch_unwind(std::panic::AssertUnwindSafe(|| { let result = std::panic::catch_unwind(std::panic::AssertUnwindSafe(|| {
let here = dir.clone().unwrap_or_default(); driver::diagnose(text, &project, |key| {
driver::diagnose(text, &here, |key| { path.as_ref()?;
let file = dir.as_ref()?.join(crate::loader::key_file(key)); let file = project.module_file(key);
// an import that is open in the editor counts with its unsaved changes // an import that is open in the editor counts with its unsaved changes
for (u, t) in docs { for (u, t) in docs {
if uri_to_path(u).as_deref() == Some(file.as_path()) { if uri_to_path(u).as_deref() == Some(file.as_path()) {
@@ -355,6 +356,14 @@ fn completion_kind(k: Kind) -> i64 {
} }
} }
/// The project a file belongs to, with the packages that are already downloaded (an editor
/// never downloads anything; `rex get` does).
fn project_of(file: Option<&std::path::Path>) -> crate::pkg::Project {
let Some(file) = file else { return crate::pkg::Project::bare(PathBuf::new()) };
crate::pkg::Project::for_file(file, crate::pkg::Mode::Offline)
.unwrap_or_else(|_| crate::pkg::Project::bare(crate::loader::project_dir(file)))
}
impl Server { impl Server {
/// The C header behind an import, read the same way the compiler reads it. /// The C header behind an import, read the same way the compiler reads it.
fn c_lib(&self, uri: &str, imp: &crate::index::Import) -> Option<crate::cheader::Lib> { fn c_lib(&self, uri: &str, imp: &crate::index::Import) -> Option<crate::cheader::Lib> {
@@ -398,8 +407,9 @@ impl Server {
/// The source of the module `key` as seen from the file `uri`: open in the editor, or on disk. /// The source of the module `key` as seen from the file `uri`: open in the editor, or on disk.
fn module_source(&self, uri: &str, key: &str) -> Option<(String, String)> { fn module_source(&self, uri: &str, key: &str) -> Option<(String, String)> {
let dir = crate::loader::project_dir(&uri_to_path(uri)?); let project = project_of(Some(&uri_to_path(uri)?));
let file = dir.join(crate::loader::key_file(key)); let key = project.scope().key("", key).ok()?;
let file = project.module_file(&key);
let target = path_to_uri(&file); let target = path_to_uri(&file);
for (u, t) in &self.docs { for (u, t) in &self.docs {
if uri_to_path(u).as_deref() == Some(file.as_path()) { if uri_to_path(u).as_deref() == Some(file.as_path()) {
+136 -1
View File
@@ -1,7 +1,7 @@
use std::path::PathBuf; use std::path::PathBuf;
use std::{env, fs, process}; use std::{env, fs, process};
use rex::{driver, fmt, lexer, lsp, parser}; use rex::{driver, fmt, lexer, loader, lsp, parser, pkg};
fn usage() -> ! { fn usage() -> ! {
eprintln!("Rex compiler (work in progress)\n"); eprintln!("Rex compiler (work in progress)\n");
@@ -12,6 +12,12 @@ fn usage() -> ! {
eprintln!(" rex check <file.rex> check for errors without building"); eprintln!(" rex check <file.rex> check for errors without building");
eprintln!(" rex test [file|dir]... run the test_ functions (default: this folder)"); eprintln!(" rex test [file|dir]... run the test_ functions (default: this folder)");
eprintln!(" rex fmt [--check] <file|dir>.. format code in place (--check only reports; `-` = stdin to stdout)"); eprintln!(" rex fmt [--check] <file|dir>.. format code in place (--check only reports; `-` = stdin to stdout)");
eprintln!(" rex init [name] start a project here (writes rex.toml)");
eprintln!(" rex add <git-address>[@version] [--name n] use a package (or a folder: rex add ../mylib)");
eprintln!(" rex remove <name> stop using a package");
eprintln!(" rex get download the packages in rex.toml / rex.lock");
eprintln!(" rex update [name]... move packages to their newest versions");
eprintln!(" rex deps list the packages this project uses");
eprintln!(" rex lsp run the language server (for editors)"); eprintln!(" rex lsp run the language server (for editors)");
eprintln!(" rex header <file.h> [name] show what Rex can use from a C header"); eprintln!(" rex header <file.h> [name] show what Rex can use from a C header");
eprintln!(" rex parse <file.rex> print the syntax tree"); eprintln!(" rex parse <file.rex> print the syntax tree");
@@ -90,6 +96,132 @@ fn run_header(args: &[String]) -> ! {
process::exit(0); process::exit(0);
} }
/// The project the current folder is in: the nearest folder with a `rex.toml`.
fn project_root() -> Option<PathBuf> {
let cwd = env::current_dir().ok()?;
cwd.ancestors().find(|d| d.join(loader::ROOT_FILE).is_file()).map(PathBuf::from)
}
fn need_root() -> PathBuf {
project_root().unwrap_or_else(|| die("error: there is no rex.toml here or in a folder above; start a project with `rex init`"))
}
/// `rex init|add|remove|get|update|deps`.
fn run_pkg(cmd: &str, args: &[String]) -> ! {
let mut fetch = pkg::Fetcher::default();
let sync = |root: &PathBuf, mode: pkg::Mode, fetch: &mut pkg::Fetcher| -> pkg::Project {
pkg::Project::open(root, mode, fetch).unwrap_or_else(|e| die(&format!("error: {e}")))
};
match (cmd, args) {
("init", [] | [_]) => {
let cwd = env::current_dir().unwrap_or_default();
let toml = cwd.join(loader::ROOT_FILE);
if toml.exists() {
die(&format!("error: {} already exists", toml.display()));
}
let name = match args.first() {
Some(n) => n.clone(),
None => cwd.file_name().map(|n| n.to_string_lossy().to_lowercase().replace(|c: char| !c.is_ascii_alphanumeric(), "_")).unwrap_or_default(),
};
if !pkg::valid_name(&name) {
die(&format!("error: `{name}` can't be a package name: use lowercase letters, digits and `_` (`rex init mygame`)"));
}
fs::write(&toml, format!("[package]\nname = \"{name}\"\n\n[deps]\n")).unwrap_or_else(|e| die(&format!("error: cannot write {}: {e}", toml.display())));
println!("wrote {} (the project `{name}`)", loader::ROOT_FILE);
}
("add", [_, ..]) => {
let mut spec = None;
let mut name = None;
let mut it = args.iter();
while let Some(a) = it.next() {
match a.as_str() {
"--name" => name = Some(it.next().cloned().unwrap_or_else(|| usage())),
_ if spec.is_none() => spec = Some(a.clone()),
_ => usage(),
}
}
let spec = spec.unwrap_or_else(|| usage());
let root = project_root().unwrap_or_else(|| env::current_dir().unwrap_or_default());
let (name, src) = pkg::add(&root, &spec, name.as_deref(), &mut fetch).unwrap_or_else(|e| die(&format!("error: {e}")));
let toml = root.join(loader::ROOT_FILE);
let old = fs::read_to_string(&toml).unwrap_or_default();
let new = pkg::edit_manifest(&old, &name, Some(&pkg::spec_text(&src)));
fs::write(&toml, &new).unwrap_or_else(|e| die(&format!("error: cannot write {}: {e}", toml.display())));
match pkg::Project::open(&root, pkg::Mode::Online, &mut fetch) {
Ok(p) => {
let version = p.packages.iter().find(|q| q.name == name).map(|q| q.version()).unwrap_or_default();
println!("added {name} {version} (use it with `import {name}`)");
}
Err(e) => {
// leave rex.toml as it was
let _ = if old.is_empty() { fs::remove_file(&toml) } else { fs::write(&toml, &old) };
die(&format!("error: {e}"));
}
}
}
("remove", [name]) => {
let root = need_root();
let toml = root.join(loader::ROOT_FILE);
let old = fs::read_to_string(&toml).unwrap_or_default();
let m = pkg::parse_manifest(&old, &toml.display().to_string()).unwrap_or_else(|e| die(&format!("error: {e}")));
if !m.deps.iter().any(|d| d.0 == *name) {
die(&format!("error: `{name}` is not in {}", toml.display()));
}
fs::write(&toml, pkg::edit_manifest(&old, name, None)).unwrap_or_else(|e| die(&format!("error: cannot write {}: {e}", toml.display())));
sync(&root, pkg::Mode::Online, &mut fetch);
println!("removed {name}");
}
("get", []) => {
let root = need_root();
let p = sync(&root, pkg::Mode::Online, &mut fetch);
println!("{} package{} ready", p.packages.len(), if p.packages.len() == 1 { "" } else { "s" });
}
("update", names) => {
let root = need_root();
let (text, changes) = pkg::newer_tags(&root, names, &mut fetch).unwrap_or_else(|e| die(&format!("error: {e}")));
let toml = root.join(loader::ROOT_FILE);
let old = fs::read_to_string(&toml).unwrap_or_default();
if text != old {
fs::write(&toml, &text).unwrap_or_else(|e| die(&format!("error: cannot write {}: {e}", toml.display())));
}
let before: Vec<(String, Option<String>)> = pkg::read_lock(&root).into_iter().map(|l| (l.name, Some(l.commit))).collect();
let p = match pkg::Project::open(&root, pkg::Mode::Update(names.to_vec()), &mut fetch) {
Ok(p) => p,
Err(e) => {
let _ = fs::write(&toml, &old);
die(&format!("error: {e}"));
}
};
let mut said = !changes.is_empty();
for c in &changes {
println!("{c}");
}
// packages that follow a branch move without rex.toml changing
for q in &p.packages {
let was = before.iter().find(|b| b.0 == q.name).and_then(|b| b.1.clone());
if matches!(q.source, pkg::Source::Git { rev: None, .. }) && was.is_some() && was != q.commit {
println!("{} -> {}", q.name, q.version());
said = true;
}
}
if !said {
println!("everything is up to date");
}
}
("deps", []) => {
let root = need_root();
let p = sync(&root, pkg::Mode::Online, &mut fetch);
if p.packages.is_empty() {
println!("no packages (add one with `rex add <git-address>`)");
} else {
print!("{}", pkg::tree_text(&p));
}
}
_ => usage(),
}
process::exit(0);
}
/// `rex fmt [--check] files-or-folders...` /// `rex fmt [--check] files-or-folders...`
fn run_fmt(args: &[String]) -> ! { fn run_fmt(args: &[String]) -> ! {
// `rex fmt -`: text in on stdin, tidy text out on stdout (for editors) // `rex fmt -`: text in on stdin, tidy text out on stdout (for editors)
@@ -236,6 +368,9 @@ fn main() {
if args.first().map(String::as_str) == Some("fmt") { if args.first().map(String::as_str) == Some("fmt") {
run_fmt(&args[1..]); run_fmt(&args[1..]);
} }
if let Some(cmd @ ("init" | "add" | "remove" | "get" | "update" | "deps")) = args.first().map(String::as_str) {
run_pkg(cmd, &args[1..]);
}
if args.first().map(String::as_str) == Some("header") { if args.first().map(String::as_str) == Some("header") {
run_header(&args[1..]); run_header(&args[1..]);
} }
+832
View File
@@ -0,0 +1,832 @@
//! Packages: Rex code from other projects, fetched with git.
//!
//! A project names its packages in `rex.toml`:
//!
//! ```toml
//! [package]
//! name = "game"
//!
//! [deps]
//! json = "git.joji.li/joji/json@v1.2.0" # a git repository at a tag (or a branch or a commit)
//! util = "../util" # a folder on this computer
//! ```
//!
//! `rex.lock` (written by rex) pins every git package to an exact commit, so a build never
//! changes by surprise; `rex update` moves to newer versions on purpose. Packages are unpacked
//! into a cache (`$REX_HOME`, else `$XDG_CACHE_HOME/rex`, else `~/.cache/rex`), one folder per
//! commit. `import json` reads `json.rex` at the top of the package, `import json.parser` its
//! `parser.rex`.
//!
//! Inside the compiler a package's files have keys starting with `@` (`@json`, `@json.parser`),
//! so they can never mix with the project's own files (see `loader::Scope`).
//!
//! A bare version (`json = "1.2.0"`) is kept for a package registry later; for now it is an error.
use std::collections::{HashMap, HashSet};
use std::path::{Path, PathBuf};
use std::process::{Command, Stdio};
use crate::{loader, stdlib};
pub const LOCK_FILE: &str = "rex.lock";
// ---------------------------------------------------------------------------------------------
// rex.toml
/// The parts of a `rex.toml` that rex understands.
#[derive(Debug, Default, Clone)]
pub struct Manifest {
/// `[package] name = "..."`.
pub name: Option<String>,
/// `[deps]`: the name, the text on the right, and the line it is on.
pub deps: Vec<(String, String, usize)>,
}
/// Read `rex.toml` (a small part of TOML: `[sections]`, `key = "text"` and `# comments`).
pub fn parse_manifest(text: &str, shown: &str) -> Result<Manifest, String> {
let mut m = Manifest::default();
let mut section = String::new();
for (n, raw) in text.lines().enumerate() {
let line = strip_comment(raw).trim();
let at = format!("{shown}:{}", n + 1);
if line.is_empty() {
continue;
}
if let Some(s) = line.strip_prefix('[') {
let Some(s) = s.strip_suffix(']') else {
return Err(format!("{at}: a section name ends with `]`, like `[deps]`"));
};
section = s.trim().to_string();
continue;
}
let Some((k, v)) = line.split_once('=') else {
return Err(format!("{at}: expected `name = \"value\"`"));
};
let key = k.trim().trim_matches('"').to_string();
match section.as_str() {
"package" if key == "name" => m.name = Some(text_value(v.trim(), &at)?),
"deps" | "dependencies" => {
let value = text_value(v.trim(), &at).map_err(|_| {
format!("{at}: write where `{key}` comes from in quotes, like `{key} = \"git.example.com/someone/{key}@v1.0.0\"`")
})?;
if m.deps.iter().any(|d| d.0 == key) {
return Err(format!("{at}: `{key}` is listed twice"));
}
m.deps.push((key, value, n + 1));
}
// other settings are for later versions of rex
_ => {}
}
}
Ok(m)
}
fn strip_comment(line: &str) -> &str {
let mut quoted = false;
for (i, c) in line.char_indices() {
match c {
'"' => quoted = !quoted,
'#' if !quoted => return &line[..i],
_ => {}
}
}
line
}
fn text_value(v: &str, at: &str) -> Result<String, String> {
match v.strip_prefix('"').and_then(|v| v.strip_suffix('"')) {
Some(s) if !s.contains('"') && !s.contains('\\') => Ok(s.to_string()),
_ => Err(format!("{at}: expected text in quotes")),
}
}
/// The manifest in `dir`, or an empty one if there is no `rex.toml`.
pub fn read_manifest(dir: &Path) -> Result<Manifest, String> {
let path = dir.join(loader::ROOT_FILE);
match std::fs::read_to_string(&path) {
Ok(text) => parse_manifest(&text, &path.display().to_string()),
Err(_) => Ok(Manifest::default()),
}
}
/// Set `name = "value"` in the `[deps]` of the `rex.toml` text (`None` removes it), keeping
/// everything else in the file as it was.
pub fn edit_manifest(text: &str, name: &str, value: Option<&str>) -> String {
let mut lines: Vec<String> = text.lines().map(str::to_string).collect();
let new_line = value.map(|v| format!("{name} = \"{v}\""));
let mut section = String::new();
let mut deps_end: Option<usize> = None; // where a new entry goes: after the last line of [deps]
let mut found: Option<usize> = None;
for (i, l) in lines.iter().enumerate() {
let t = strip_comment(l).trim();
if let Some(s) = t.strip_prefix('[').and_then(|s| s.strip_suffix(']')) {
section = s.trim().to_string();
if section == "deps" || section == "dependencies" {
deps_end = Some(i + 1);
}
continue;
}
if section == "deps" || section == "dependencies" {
if !t.is_empty() {
deps_end = Some(i + 1);
}
if t.split_once('=').map_or(false, |(k, _)| k.trim().trim_matches('"') == name) {
found = Some(i);
}
}
}
match (found, new_line) {
(Some(i), Some(l)) => lines[i] = l,
(Some(i), None) => {
lines.remove(i);
}
(None, Some(l)) => match deps_end {
Some(e) => lines.insert(e, l),
None => {
while lines.last().map_or(false, |l| l.trim().is_empty()) {
lines.pop();
}
if !lines.is_empty() {
lines.push(String::new());
}
lines.push("[deps]".to_string());
lines.push(l);
}
},
(None, None) => {}
}
let mut out = lines.join("\n");
out.push('\n');
out
}
// ---------------------------------------------------------------------------------------------
// where a package comes from
#[derive(Debug, Clone, PartialEq)]
pub enum Source {
/// A git repository and, maybe, a tag, branch or commit (`None`: its main branch).
Git { url: String, rev: Option<String> },
/// A folder on this computer.
Path(PathBuf),
}
impl Source {
pub fn show(&self) -> String {
match self {
Source::Git { url, rev: Some(r) } => format!("{url}@{r}"),
Source::Git { url, rev: None } => url.clone(),
Source::Path(p) => p.display().to_string(),
}
}
}
/// The text on the right of `json = "..."`; a folder is counted from `base`.
pub fn parse_source(spec: &str, base: &Path) -> Result<Source, String> {
let spec = spec.trim();
if spec.is_empty() {
return Err("it is empty".to_string());
}
if spec.starts_with('.') || spec.starts_with('/') {
return Ok(Source::Path(base.join(spec)));
}
if let Some(home) = spec.strip_prefix("~/") {
return Ok(Source::Path(PathBuf::from(std::env::var("HOME").unwrap_or_default()).join(home)));
}
let looks_like_version = spec.chars().next().map_or(false, |c| c.is_ascii_digit() || "^~=<>".contains(c));
if looks_like_version && !spec.contains('/') {
return Err(format!(
"`{spec}` is a version without a place to get it from: there is no package registry yet, so write the git address, like \"git.example.com/someone/name@v{}\"",
spec.trim_start_matches(|c| "^~=<>v".contains(c))
));
}
// `url@rev`: an `@` after the last `/` or `:` (so `git@host:a/b` is not split)
let cut = spec.rfind('@').filter(|&i| {
let after = &spec[i + 1..];
!after.is_empty() && !after.contains('/') && !after.contains(':') && spec[..i].contains('/')
});
let (url, rev) = match cut {
Some(i) => (&spec[..i], Some(spec[i + 1..].to_string())),
None => (spec, None),
};
let url = url.trim_end_matches('/').trim_end_matches(".git").to_string();
if !url.contains('/') {
return Err(format!("`{spec}` is not a git address, like \"git.example.com/someone/name\""));
}
Ok(Source::Git { url, rev })
}
/// What `git clone` gets for an address: `host/path` means https.
pub fn clone_url(url: &str) -> String {
if url.contains("://") || url.starts_with("git@") || Path::new(url).is_absolute() {
url.to_string()
} else {
format!("https://{url}")
}
}
/// A name that works after `import`: the last part of the address, if it is one.
pub fn guess_name(src: &Source) -> String {
let last = match src {
Source::Git { url, .. } => url.rsplit(['/', ':']).next().unwrap_or("").to_string(),
Source::Path(p) => {
let p = std::fs::canonicalize(p).unwrap_or_else(|_| p.clone());
p.file_name().map(|n| n.to_string_lossy().to_string()).unwrap_or_default()
}
};
last.trim_end_matches(".git").to_string()
}
pub fn valid_name(name: &str) -> bool {
let mut cs = name.chars();
cs.next().map_or(false, |c| c.is_ascii_lowercase() || c == '_')
&& cs.all(|c| c.is_ascii_lowercase() || c.is_ascii_digit() || c == '_')
}
// ---------------------------------------------------------------------------------------------
// versions
/// `v1.2.3` or `1.2.3`, maybe with `-beta.1` after it.
#[derive(Debug, Clone, PartialEq, Eq, PartialOrd, Ord)]
pub struct Version {
pub major: u64,
pub minor: u64,
pub patch: u64,
/// `None` sorts after any pre-release, as it should.
pub stable: Option<()>,
pub pre: String,
}
pub fn version(tag: &str) -> Option<Version> {
let t = tag.strip_prefix('v').unwrap_or(tag);
let (nums, pre) = match t.split_once('-') {
Some((n, p)) => (n, p.to_string()),
None => (t, String::new()),
};
let mut parts = nums.split('.').map(|p| p.parse::<u64>().ok());
let (major, minor, patch) = (parts.next()??, parts.next().flatten().unwrap_or(0), parts.next().flatten().unwrap_or(0));
if parts.next().is_some() {
return None;
}
Some(Version { major, minor, patch, stable: if pre.is_empty() { Some(()) } else { None }, pre })
}
/// The newest of the version tags, leaving out pre-releases unless there is nothing else;
/// `major`: only that major version.
pub fn newest_tag(tags: &[String], major: Option<u64>) -> Option<String> {
let mut vs: Vec<(Version, &String)> =
tags.iter().filter_map(|t| version(t).map(|v| (v, t))).filter(|(v, _)| major.map_or(true, |m| v.major == m)).collect();
vs.sort();
let stable = vs.iter().rev().find(|(v, _)| v.stable.is_some());
stable.or(vs.last()).map(|(_, t)| (*t).clone())
}
// ---------------------------------------------------------------------------------------------
// git and the cache
pub fn cache_dir() -> PathBuf {
if let Some(d) = std::env::var_os("REX_HOME") {
return PathBuf::from(d);
}
if let Some(d) = std::env::var_os("XDG_CACHE_HOME") {
return PathBuf::from(d).join("rex");
}
PathBuf::from(std::env::var_os("HOME").unwrap_or_default()).join(".cache").join("rex")
}
fn slug(url: &str) -> String {
let u = url.split_once("://").map_or(url, |(_, rest)| rest);
u.chars().map(|c| if c.is_ascii_alphanumeric() || c == '.' || c == '-' { c } else { '_' }).collect()
}
fn git(git_dir: Option<&Path>, args: &[&str]) -> Result<String, String> {
let mut cmd = Command::new("git");
if let Some(d) = git_dir {
cmd.arg("--git-dir").arg(d);
}
// fail instead of waiting for a password nobody will type
cmd.args(args).env("GIT_TERMINAL_PROMPT", "0").stdin(Stdio::null());
match cmd.output() {
Err(e) => Err(format!("cannot run git ({e}); install git to use packages")),
Ok(o) if o.status.success() => Ok(String::from_utf8_lossy(&o.stdout).trim().to_string()),
Ok(o) => Err(String::from_utf8_lossy(&o.stderr).trim().to_string()),
}
}
/// Downloads, remembering which repositories were already brought up to date in this run.
#[derive(Default)]
pub struct Fetcher {
fresh: HashSet<String>,
/// Say what is being downloaded (on stderr).
pub quiet: bool,
}
impl Fetcher {
/// A local copy of the repository with all its branches and tags; `refresh`: ask the
/// server for news (once per run).
fn mirror(&mut self, url: &str, refresh: bool) -> Result<PathBuf, String> {
let dir = cache_dir().join("git").join(slug(url));
if !dir.join("HEAD").is_file() {
if !self.quiet {
eprintln!("fetching {url}");
}
std::fs::create_dir_all(dir.parent().unwrap()).map_err(|e| format!("cannot create {}: {e}", cache_dir().display()))?;
let tmp = dir.with_extension(format!("tmp{}", std::process::id()));
let _ = std::fs::remove_dir_all(&tmp);
git(None, &["clone", "--mirror", "--quiet", &clone_url(url), &tmp.to_string_lossy()])
.map_err(|e| format!("cannot download {url}:\n{e}"))?;
if std::fs::rename(&tmp, &dir).is_err() {
// someone else was faster
let _ = std::fs::remove_dir_all(&tmp);
}
self.fresh.insert(url.to_string());
} else if refresh && !self.fresh.contains(url) {
if !self.quiet {
eprintln!("fetching {url}");
}
git(Some(&dir), &["fetch", "--quiet", "--prune", "--tags", "origin"]).map_err(|e| format!("cannot update {url}:\n{e}"))?;
self.fresh.insert(url.to_string());
}
Ok(dir)
}
pub fn tags(&mut self, url: &str) -> Result<Vec<String>, String> {
let m = self.mirror(url, true)?;
Ok(git(Some(&m), &["tag", "--list"])?.lines().map(str::to_string).collect())
}
/// The commit a tag, branch or commit stands for (`None`: the main branch).
fn commit(&mut self, url: &str, rev: Option<&str>) -> Result<String, String> {
let what = format!("{}^{{commit}}", rev.unwrap_or("HEAD"));
// a version tag doesn't move, so one already here needs no trip to the server
if rev.and_then(version).is_some() {
let m = self.mirror(url, false)?;
if let Ok(c) = git(Some(&m), &["rev-parse", "--verify", "--quiet", &format!("refs/tags/{what}")]) {
return Ok(c);
}
}
let m = self.mirror(url, true)?;
git(Some(&m), &["rev-parse", "--verify", "--quiet", &what]).map_err(|_| match rev {
Some(r) => {
let tags = self.tags(url).unwrap_or_default();
let hint = match newest_tag(&tags, None) {
Some(t) => format!(" (the newest version is {t})"),
None => " (it has no version tags; leave out the `@...` to use its main branch)".to_string(),
};
format!("{url} has no tag, branch or commit called `{r}`{hint}")
}
None => format!("{url} has no commits"),
})
}
/// The files of `url` at `commit`, unpacked in the cache.
fn tree(&mut self, url: &str, commit: &str, label: &str) -> Result<PathBuf, String> {
let dir = cache_dir().join("pkg").join(slug(url)).join(commit);
if dir.is_dir() {
return Ok(dir);
}
let mut m = self.mirror(url, false)?;
if git(Some(&m), &["cat-file", "-e", &format!("{commit}^{{commit}}")]).is_err() {
m = self.mirror(url, true)?;
}
if !self.quiet {
eprintln!("downloading {label}");
}
let tmp = dir.with_extension(format!("tmp{}", std::process::id()));
let _ = std::fs::remove_dir_all(&tmp);
std::fs::create_dir_all(&tmp).map_err(|e| format!("cannot create {}: {e}", tmp.display()))?;
let unpack = || -> Result<(), String> {
let mut archive = Command::new("git")
.arg("--git-dir")
.arg(&m)
.args(["archive", "--format=tar", commit])
.stdout(Stdio::piped())
.stderr(Stdio::piped())
.spawn()
.map_err(|e| format!("cannot run git: {e}"))?;
let out = archive.stdout.take().unwrap();
let tar = Command::new("tar").arg("-x").arg("-C").arg(&tmp).stdin(out).output().map_err(|e| format!("cannot run tar: {e}"))?;
let a = archive.wait_with_output().map_err(|e| e.to_string())?;
if !a.status.success() {
return Err(String::from_utf8_lossy(&a.stderr).trim().to_string());
}
if !tar.status.success() {
return Err(String::from_utf8_lossy(&tar.stderr).trim().to_string());
}
Ok(())
};
if let Err(e) = unpack() {
let _ = std::fs::remove_dir_all(&tmp);
return Err(format!("cannot unpack {url} at {commit}: {e}"));
}
if std::fs::rename(&tmp, &dir).is_err() {
let _ = std::fs::remove_dir_all(&tmp);
}
Ok(dir)
}
}
// ---------------------------------------------------------------------------------------------
// rex.lock
#[derive(Debug, Clone, PartialEq)]
pub struct Locked {
pub name: String,
pub url: String,
pub rev: Option<String>,
pub commit: String,
}
pub fn read_lock(root: &Path) -> Vec<Locked> {
let text = std::fs::read_to_string(root.join(LOCK_FILE)).unwrap_or_default();
text.lines()
.filter(|l| !l.trim_start().starts_with('#'))
.filter_map(|l| {
let w: Vec<&str> = l.split_whitespace().collect();
let [name, url, rev, commit] = w[..] else { return None };
Some(Locked { name: name.into(), url: url.into(), rev: (rev != "-").then(|| rev.to_string()), commit: commit.into() })
})
.collect()
}
fn lock_text(entries: &[Locked]) -> String {
let mut s = String::from("# Written by rex: the exact version of every package. Keep it with your code; don't edit it.\n");
for e in entries {
s.push_str(&format!("{} {} {} {}\n", e.name, e.url, e.rev.as_deref().unwrap_or("-"), e.commit));
}
s
}
// ---------------------------------------------------------------------------------------------
// putting a project together
#[derive(Debug, Clone)]
pub struct Package {
pub name: String,
pub dir: PathBuf,
/// The packages this one imports.
pub deps: Vec<String>,
pub source: Source,
/// Where it comes from, as written in `rex.toml`.
pub spec: String,
/// Git packages: the exact commit.
pub commit: Option<String>,
}
impl Package {
/// `v1.2.0`, `main branch @ 3f9a1c2`, or the folder.
pub fn version(&self) -> String {
match (&self.source, &self.commit) {
(Source::Git { rev: Some(r), .. }, _) => r.clone(),
(Source::Git { rev: None, .. }, Some(c)) => format!("main branch @ {}", &c[..c.len().min(7)]),
_ => format!("(folder {})", self.spec),
}
}
}
/// How much `Project::open` may do.
#[derive(Debug, Clone, PartialEq)]
pub enum Mode {
/// Only what is already downloaded (for editors).
Offline,
/// Download what is missing and update `rex.lock` when `rex.toml` changed.
Online,
/// Like `Online`, but ignore what `rex.lock` says about these packages (all if empty) and
/// look for news on their servers.
Update(Vec<String>),
}
#[derive(Debug, Default)]
pub struct Project {
/// The folder with `rex.toml` (or the main file's folder).
pub root: PathBuf,
pub root_deps: Vec<String>,
pub packages: Vec<Package>,
}
impl Project {
/// A project without packages.
pub fn bare(root: PathBuf) -> Project {
Project { root, ..Default::default() }
}
/// The project `file` belongs to, with its packages.
pub fn for_file(file: &Path, mode: Mode) -> Result<Project, String> {
Project::open(&loader::project_dir(file), mode, &mut Fetcher::default())
}
pub fn open(root: &Path, mode: Mode, fetch: &mut Fetcher) -> Result<Project, String> {
let manifest = read_manifest(root)?;
if manifest.deps.is_empty() {
return Ok(Project::bare(root.to_path_buf()));
}
let lock = read_lock(root);
let ignore = |name: &str| match &mode {
Mode::Update(names) => names.is_empty() || names.iter().any(|n| n == name),
_ => false,
};
let toml = root.join(loader::ROOT_FILE).display().to_string();
// Who wants which package. When two want different versions of the same repository
// the newer one wins (and everything is walked again), as long as the major version is
// the same.
let mut chosen: HashMap<String, (Source, String)> = HashMap::new();
loop {
let mut changed = false;
let mut packages: Vec<Package> = Vec::new();
let mut todo: Vec<(String, PathBuf, Manifest)> = vec![(toml.clone(), root.to_path_buf(), manifest.clone())];
let mut root_deps = Vec::new();
let mut first = true;
while let Some((who, base, m)) = todo.pop() {
let mut names = Vec::new();
for (name, spec, line) in &m.deps {
let at = format!("{who}:{line}");
if !valid_name(name) {
return Err(format!("{at}: `{name}` can't be a package name: use lowercase letters, digits and `_`"));
}
if stdlib::MODULES.contains(&name.as_str()) {
return Err(format!("{at}: `{name}` is a built-in module; call the package something else"));
}
let src = parse_source(spec, &base).map_err(|e| format!("{at}: `{name}`: {e}"))?;
names.push(name.clone());
match chosen.get(name) {
None => {
chosen.insert(name.clone(), (src, who.clone()));
}
Some((old, _)) if *old == src => {}
// the project's own folder packages win (for working on a package locally)
Some((Source::Path(_), by)) if *by == toml => {}
Some((old, by)) => match (old, &src) {
(Source::Git { url: u1, rev: Some(r1) }, Source::Git { url: u2, rev: Some(r2) }) if u1 == u2 => {
match (version(r1), version(r2)) {
(Some(v1), Some(v2)) if v1.major == v2.major => {
if v2 > v1 {
chosen.insert(name.clone(), (src, who.clone()));
changed = true;
}
}
_ => return Err(conflict(name, old, by, &src, &who)),
}
}
_ => return Err(conflict(name, old, by, &src, &who)),
},
}
}
if first {
root_deps = names.clone();
first = false;
}
for name in names {
if packages.iter().any(|p| p.name == name) {
continue;
}
let (src, by) = chosen[&name].clone();
let spec = m.deps.iter().find(|d| d.0 == name).filter(|_| by == who).map_or_else(|| src.show(), |d| d.1.clone());
let (dir, commit) = match &src {
Source::Path(p) => {
if !p.is_dir() {
return Err(format!("{by}: the folder {} for `{name}` doesn't exist", p.display()));
}
(std::fs::canonicalize(p).unwrap_or_else(|_| p.clone()), None)
}
Source::Git { url, rev } => {
let label = format!("{name} {}", rev.as_deref().unwrap_or("(main branch)"));
let pinned = lock
.iter()
.find(|l| l.name == name && l.url == *url && l.rev == *rev)
.filter(|_| !ignore(&name))
.map(|l| l.commit.clone());
let commit = match (pinned, &mode) {
(Some(c), _) => c,
(None, Mode::Offline) => {
return Err(format!("the package `{name}` is not downloaded yet; run `rex get`"))
}
(None, _) => fetch.commit(url, rev.as_deref()).map_err(|e| format!("{by}: `{name}`: {e}"))?,
};
let tree = cache_dir().join("pkg").join(slug(url)).join(&commit);
let dir = if mode == Mode::Offline {
if !tree.is_dir() {
return Err(format!("the package `{name}` is not downloaded yet; run `rex get`"));
}
tree
} else {
fetch.tree(url, &commit, &label)?
};
(dir, Some(commit))
}
};
let pm = read_manifest(&dir)?;
if let Some(own) = &pm.name {
if *own != name {
return Err(format!(
"{by}: the package at {} calls itself `{own}`; name it that way: `{own} = \"{}\"`",
src.show(),
src.show()
));
}
}
let shown = format!("{} ({name})", dir.join(loader::ROOT_FILE).display());
let deps = pm.deps.iter().map(|d| d.0.clone()).collect();
packages.push(Package { name: name.clone(), dir: dir.clone(), deps, source: src, spec, commit });
todo.push((shown, dir, pm));
}
}
if changed {
continue;
}
for name in &root_deps {
let own = root.join(format!("{name}.rex"));
if own.is_file() {
return Err(format!(
"both your file {} and the package `{name}` would be `import {name}`; rename the file, or give the package another name in rex.toml",
own.display()
));
}
}
packages.sort_by(|a, b| a.name.cmp(&b.name));
if mode != Mode::Offline {
let entries: Vec<Locked> = packages
.iter()
.filter_map(|p| match (&p.source, &p.commit) {
(Source::Git { url, rev }, Some(c)) => {
Some(Locked { name: p.name.clone(), url: url.clone(), rev: rev.clone(), commit: c.clone() })
}
_ => None,
})
.collect();
let text = lock_text(&entries);
let path = root.join(LOCK_FILE);
let old = std::fs::read_to_string(&path).ok();
if old.as_deref() != Some(text.as_str()) && !(entries.is_empty() && old.is_none()) {
std::fs::write(&path, text).map_err(|e| format!("cannot write {}: {e}", path.display()))?;
}
}
return Ok(Project { root: root.to_path_buf(), root_deps, packages });
}
}
/// The file behind a module key (see `loader::Scope`): `@json.parser` is `parser.rex` in
/// the package `json`, `@json` its `json.rex`; other keys are the project's own files.
pub fn module_file(&self, key: &str) -> PathBuf {
if let Some(k) = key.strip_prefix('@') {
let (name, rest) = k.split_once('.').unwrap_or((k, ""));
if let Some(p) = self.packages.iter().find(|p| p.name == name) {
return if rest.is_empty() { p.dir.join(format!("{name}.rex")) } else { p.dir.join(loader::key_file(rest)) };
}
}
self.root.join(loader::key_file(key))
}
/// The folder C headers imported by module `key` are looked for in (`""`: the main file).
pub fn module_dir(&self, key: &str) -> PathBuf {
if key.is_empty() {
return self.root.clone();
}
self.module_file(key).parent().map(Path::to_path_buf).unwrap_or_default()
}
/// Which imports go to packages, for the loader.
pub fn scope(&self) -> loader::Scope {
loader::Scope {
root_deps: self.root_deps.clone(),
packages: self.packages.iter().map(|p| (p.name.clone(), p.deps.clone())).collect(),
}
}
/// Read a module the compiler asks for: its display path and its text.
pub fn read(&self, key: &str) -> Option<(String, String)> {
let file = self.module_file(key);
std::fs::read_to_string(&file).ok().map(|s| (file.display().to_string(), s))
}
}
fn conflict(name: &str, a: &Source, a_by: &str, b: &Source, b_by: &str) -> String {
format!(
"two different packages called `{name}` are needed:\n {} wants {}\n {} wants {}\nOnly one can be used; make them agree.",
a_by,
a.show(),
b_by,
b.show()
)
}
/// `rex update`: the newest tag with the same major version for every tagged package in
/// `rex.toml` (only `names`, if given). Returns the new `rex.toml` text and what changed.
pub fn newer_tags(root: &Path, names: &[String], fetch: &mut Fetcher) -> Result<(String, Vec<String>), String> {
let path = root.join(loader::ROOT_FILE);
let mut text = std::fs::read_to_string(&path).map_err(|_| format!("there is no {} here", loader::ROOT_FILE))?;
let m = parse_manifest(&text, &path.display().to_string())?;
for n in names {
if !m.deps.iter().any(|d| d.0 == *n) {
return Err(format!("`{n}` is not in {}", path.display()));
}
}
let mut changes = Vec::new();
for (name, spec, _) in &m.deps {
if !names.is_empty() && !names.contains(name) {
continue;
}
let Ok(Source::Git { url, rev: Some(rev) }) = parse_source(spec, root) else { continue };
let Some(v) = version(&rev) else { continue };
let tags = fetch.tags(&url)?;
if let Some(t) = newest_tag(&tags, Some(v.major)) {
if version(&t).map_or(false, |nv| nv > v) {
text = edit_manifest(&text, name, Some(&format!("{url}@{t}")));
changes.push(format!("{name} {rev} -> {t}"));
}
if let Some(newer) = newest_tag(&tags, None).filter(|n| version(n).map_or(false, |nv| nv.major > v.major)) {
changes.push(format!("{name}: {newer} is out too (a new major version may need changes in your code: `rex add {url}@{newer}`)"));
}
}
}
Ok((text, changes))
}
/// `rex add`: where to get `spec` from and the name to import it by. Downloads it to find out.
pub fn add(root: &Path, spec: &str, name: Option<&str>, fetch: &mut Fetcher) -> Result<(String, Source), String> {
let src = parse_source(spec, &std::env::current_dir().unwrap_or_default())?;
// a folder is written relative to the project when it can be
let src = match src {
Source::Path(p) => {
let abs = std::fs::canonicalize(&p).map_err(|_| format!("the folder {} doesn't exist", p.display()))?;
Source::Path(relative(&abs, &std::fs::canonicalize(root).unwrap_or_else(|_| root.to_path_buf())))
}
Source::Git { url, rev: None } => {
// the newest version, if it has any
let tags = fetch.tags(&url)?;
Source::Git { rev: newest_tag(&tags, None), url }
}
s => s,
};
let dir = match &src {
Source::Path(p) => root.join(p),
Source::Git { url, rev } => {
let c = fetch.commit(url, rev.as_deref())?;
fetch.tree(url, &c, &format!("{} {}", guess_name(&src), rev.as_deref().unwrap_or("(main branch)")))?
}
};
let own = read_manifest(&dir)?.name;
let name = match (name, own) {
(Some(n), Some(o)) if n != o => return Err(format!("that package calls itself `{o}`, so it is imported as `{o}`")),
(Some(n), _) => n.to_string(),
(None, Some(o)) => o,
(None, None) => guess_name(&src),
};
if !valid_name(&name) {
return Err(format!(
"`{name}` can't be used after `import`; pick a name with lowercase letters, digits and `_`: `rex add {spec} --name {}`",
name.to_lowercase().replace(|c: char| !c.is_ascii_alphanumeric(), "_")
));
}
if !dir.join(format!("{name}.rex")).is_file() {
return Err(format!("{} has no {name}.rex at its top, so there is nothing to `import {name}`", src.show()));
}
Ok((name, src))
}
/// `target` written relative to `base` (`../json`), or as it is if that doesn't work out.
fn relative(target: &Path, base: &Path) -> PathBuf {
let t: Vec<_> = target.components().collect();
let b: Vec<_> = base.components().collect();
let same = t.iter().zip(&b).take_while(|(x, y)| x == y).count();
if same == 0 {
return target.to_path_buf();
}
let mut out = PathBuf::new();
for _ in same..b.len() {
out.push("..");
}
for c in &t[same..] {
out.push(c);
}
if !out.starts_with("..") {
out = Path::new(".").join(out);
}
out
}
/// The text to put in `rex.toml` for a source.
pub fn spec_text(src: &Source) -> String {
match src {
Source::Path(p) => p.display().to_string(),
s => s.show(),
}
}
/// `rex deps`: the packages as a tree.
pub fn tree_text(p: &Project) -> String {
let mut out = String::new();
fn walk(p: &Project, names: &[String], depth: usize, seen: &mut Vec<String>, out: &mut String) {
for n in names {
let Some(pkg) = p.packages.iter().find(|q| q.name == *n) else { continue };
let again = seen.contains(n);
out.push_str(&format!("{}{} {}{}\n", " ".repeat(depth), n, pkg.version(), if again && !pkg.deps.is_empty() { " (see above)" } else { "" }));
if !again {
seen.push(n.clone());
walk(p, &pkg.deps, depth + 1, seen, out);
}
}
}
walk(p, &p.root_deps, 0, &mut Vec::new(), &mut out);
out
}
+24
View File
@@ -160,6 +160,30 @@ fn imports_are_read_next_to_the_file() {
assert_eq!(out[0].path(&["params", "diagnostics"]), Some(&Json::Arr(vec![])), "{:?}", out[0]); assert_eq!(out[0].path(&["params", "diagnostics"]), Some(&Json::Arr(vec![])), "{:?}", out[0]);
} }
#[test]
fn packages_are_seen_by_the_editor() {
let dir = std::env::temp_dir().join(format!("rex-lsp-pkg-{}", std::process::id()));
std::fs::create_dir_all(dir.join("app")).unwrap();
std::fs::create_dir_all(dir.join("mylib")).unwrap();
std::fs::write(dir.join("app/rex.toml"), "[deps]\nmylib = \"../mylib\"\n").unwrap();
std::fs::write(dir.join("mylib/mylib.rex"), "pub func n(): int { return 1 }\n").unwrap();
let uri = format!("file://{}/app/main.rex", dir.display());
let good = "import mylib\nfunc main() {\n print(mylib.n())\n}\n";
let bad = "import mylib\nfunc main() {\n print(mylib.n(5))\n}\n";
let mut msgs = vec![request(1, "initialize", Json::obj(vec![])), open(&uri, good), request(2, "textDocument/definition", at(&uri, 2, 17))];
let other = format!("file://{}/app/other.rex", dir.display());
msgs.push(open(&other, bad));
msgs.extend(exit_messages());
let (out, _) = talk(msgs);
let _ = std::fs::remove_dir_all(&dir);
let diags: Vec<&Json> = out.iter().filter_map(|m| m.path(&["params", "diagnostics"])).collect();
assert_eq!(diags[0], &Json::Arr(vec![]), "{out:?}");
assert!(matches!(diags[1], Json::Arr(a) if a.len() == 1), "{out:?}");
let def = out.iter().find(|m| m.get("id") == Some(&Json::Num(2.0))).unwrap();
let target = def.path(&["result", "uri"]).map(|u| u.to_string()).unwrap_or_default();
assert!(target.contains("mylib/mylib.rex"), "{def:?}");
}
#[test] #[test]
fn unknown_requests_get_an_error() { fn unknown_requests_get_an_error() {
let mut msgs = vec![request(5, "textDocument/codeLens", Json::obj(vec![]))]; let mut msgs = vec![request(5, "textDocument/codeLens", Json::obj(vec![]))];
+289
View File
@@ -0,0 +1,289 @@
//! Packages: `rex add/remove/get/update/deps`, `rex.toml` / `rex.lock`, and imports that reach
//! into packages. Git repositories are made on the fly and used through `file://` addresses;
//! the package cache goes to a temp folder (`REX_HOME`).
use std::path::{Path, PathBuf};
use std::process::{Command, Output};
use rex::loader::Scope;
use rex::pkg::{edit_manifest, newest_tag, parse_manifest, parse_source, Source};
fn temp_dir() -> PathBuf {
static N: std::sync::atomic::AtomicU32 = std::sync::atomic::AtomicU32::new(0);
let n = N.fetch_add(1, std::sync::atomic::Ordering::Relaxed);
let dir = std::env::temp_dir().join(format!("rex-packages-{}-{n}", std::process::id()));
let _ = std::fs::remove_dir_all(&dir);
std::fs::create_dir_all(&dir).unwrap();
dir
}
fn write(dir: &Path, files: &[(&str, &str)]) {
for (name, text) in files {
let f = dir.join(name);
std::fs::create_dir_all(f.parent().unwrap()).unwrap();
std::fs::write(f, text).unwrap();
}
}
fn git(dir: &Path, args: &[&str]) {
let o = Command::new("git")
.args(args)
.current_dir(dir)
.env("GIT_AUTHOR_NAME", "t")
.env("GIT_AUTHOR_EMAIL", "t@t")
.env("GIT_COMMITTER_NAME", "t")
.env("GIT_COMMITTER_EMAIL", "t@t")
.output()
.unwrap();
assert!(o.status.success(), "git {args:?}: {}", String::from_utf8_lossy(&o.stderr));
}
/// Commit everything in `dir` (a new repository the first time) and tag it.
fn release(dir: &Path, files: &[(&str, &str)], tag: Option<&str>) {
std::fs::create_dir_all(dir).unwrap();
if !dir.join(".git").exists() {
git(dir, &["init", "-q"]);
}
write(dir, files);
git(dir, &["add", "."]);
git(dir, &["commit", "-q", "--allow-empty", "-m", "x"]);
if let Some(t) = tag {
git(dir, &["tag", t]);
}
}
fn url(dir: &Path) -> String {
format!("file://{}", dir.display())
}
struct Env {
root: PathBuf,
}
impl Env {
fn new() -> Env {
let root = temp_dir();
std::fs::create_dir_all(root.join("app")).unwrap();
Env { root }
}
fn rex(&self, args: &[&str]) -> Output {
Command::new(env!("CARGO_BIN_EXE_rex"))
.args(args)
.current_dir(self.root.join("app"))
.env("REX_HOME", self.root.join("cache"))
.output()
.unwrap()
}
fn ok(&self, args: &[&str]) -> String {
let o = self.rex(args);
assert!(o.status.success(), "rex {args:?}:\n{}{}", String::from_utf8_lossy(&o.stdout), String::from_utf8_lossy(&o.stderr));
String::from_utf8_lossy(&o.stdout).to_string()
}
fn fails(&self, args: &[&str]) -> String {
let o = self.rex(args);
assert!(!o.status.success(), "rex {args:?} should fail:\n{}", String::from_utf8_lossy(&o.stdout));
String::from_utf8_lossy(&o.stderr).to_string()
}
fn app(&self, files: &[(&str, &str)]) {
write(&self.root.join("app"), files);
}
fn read(&self, file: &str) -> String {
std::fs::read_to_string(self.root.join("app").join(file)).unwrap_or_default()
}
}
impl Drop for Env {
fn drop(&mut self) {
let _ = std::fs::remove_dir_all(&self.root);
}
}
/// `strs` (v1.0.0, v1.1.0) with its own `util.rex`, and `json` (v0.3.0), which uses strs v1.0.0.
fn two_packages(e: &Env) -> (PathBuf, PathBuf) {
let strs = e.root.join("strs");
release(
&strs,
&[
("rex.toml", "[package]\nname = \"strs\"\n"),
("strs.rex", "import util\n\npub func shout(s: string): string { return util.bang(s.upper()) }\n"),
("util.rex", "pub func bang(s: string): string { return s + \"!\" }\n"),
],
Some("v1.0.0"),
);
release(&strs, &[("util.rex", "pub func bang(s: string): string { return s + \"!!\" }\n")], Some("v1.1.0"));
let json = e.root.join("json");
release(
&json,
&[
("rex.toml", &format!("[package]\nname = \"json\"\n\n[deps]\nstrs = \"{}@v1.0.0\"\n", url(&strs))),
("json.rex", "import strs\nimport parser\n\npub func hello(): string { return strs.shout(parser.word()) }\n"),
("parser.rex", "pub func word(): string { return \"hi\" }\n"),
],
Some("v0.3.0"),
);
(strs, json)
}
#[test]
fn add_a_package_and_use_it() {
let e = Env::new();
let (strs, json) = two_packages(&e);
e.ok(&["init"]);
let out = e.ok(&["add", &url(&json)]);
assert!(out.contains("added json v0.3.0"), "{out}");
assert!(e.read("rex.toml").contains(&format!("json = \"{}@v0.3.0\"", url(&json))), "{}", e.read("rex.toml"));
let lock = e.read("rex.lock");
assert!(lock.contains(&format!("json {} v0.3.0 ", url(&json))) && lock.contains(&format!("strs {} v1.0.0 ", url(&strs))), "{lock}");
// the project's own util.rex and the package's util.rex are different files
e.app(&[
("main.rex", "import json\nimport util\n\nfunc main() {\n print(json.hello(), util.mine())\n}\n"),
("util.rex", "pub func mine(): int { return 5 }\n"),
]);
assert_eq!(e.ok(&["run", "main.rex"]), "HI! 5\n");
// asking for a newer strs moves json to it too
let out = e.ok(&["add", &format!("{}@v1.1.0", url(&strs))]);
assert!(out.contains("added strs v1.1.0"), "{out}");
assert_eq!(e.ok(&["run", "main.rex"]), "HI!! 5\n");
assert_eq!(e.ok(&["deps"]), "json v0.3.0\n strs v1.1.0\nstrs v1.1.0\n");
e.ok(&["remove", "strs"]);
assert!(!e.read("rex.toml").contains("strs"));
assert_eq!(e.ok(&["run", "main.rex"]), "HI! 5\n");
}
#[test]
fn the_lock_file_keeps_the_build_the_same() {
let e = Env::new();
let lib = e.root.join("lib");
release(&lib, &[("lib.rex", "pub func n(): int { return 1 }\n")], None);
e.ok(&["init"]);
let out = e.ok(&["add", &url(&lib)]);
assert!(out.contains("added lib main branch @"), "{out}");
e.app(&[("main.rex", "import lib\n\nfunc main() { print(lib.n()) }\n")]);
assert_eq!(e.ok(&["run", "main.rex"]), "1\n");
// new commits on the branch don't change the build until `rex update`
release(&lib, &[("lib.rex", "pub func n(): int { return 2 }\n")], None);
assert_eq!(e.ok(&["run", "main.rex"]), "1\n");
let out = e.ok(&["update"]);
assert!(out.contains("lib -> main branch @"), "{out}");
assert_eq!(e.ok(&["run", "main.rex"]), "2\n");
}
#[test]
fn update_moves_to_the_newest_tag_of_the_same_major_version() {
let e = Env::new();
let (strs, _) = two_packages(&e);
e.ok(&["init"]);
e.ok(&["add", &format!("{}@v1.0.0", url(&strs))]);
release(&strs, &[], Some("v2.0.0"));
let out = e.ok(&["update", "strs"]);
assert!(out.contains("strs v1.0.0 -> v1.1.0") && out.contains("v2.0.0 is out too"), "{out}");
assert!(e.read("rex.toml").contains("@v1.1.0"));
assert!(e.ok(&["update"]).contains("v2.0.0 is out too"));
}
#[test]
fn a_package_in_a_folder() {
let e = Env::new();
write(&e.root.join("mylib"), &[("mylib.rex", "import helper\n\npub func n(): int { return helper.k() }\n"), ("helper.rex", "pub func k(): int { return 42 }\n")]);
e.ok(&["init"]);
assert!(e.ok(&["add", "../mylib"]).contains("added mylib (folder ../mylib)"));
assert!(e.read("rex.toml").contains("mylib = \"../mylib\""));
assert!(!e.read("rex.lock").contains("mylib"));
e.app(&[("main.rex", "import mylib\n\nfunc main() { print(mylib.n()) }\n")]);
assert_eq!(e.ok(&["run", "main.rex"]), "42\n");
// changes in the folder show up right away
write(&e.root.join("mylib"), &[("helper.rex", "pub func k(): int { return 7 }\n")]);
assert_eq!(e.ok(&["run", "main.rex"]), "7\n");
}
#[test]
fn package_mistakes_are_explained() {
let e = Env::new();
let (strs, json) = two_packages(&e);
e.ok(&["init"]);
let err = e.fails(&["add", &format!("{}@v9.0.0", url(&strs))]);
assert!(err.contains("no tag, branch or commit called `v9.0.0`") && err.contains("newest version is v1.1.0"), "{err}");
assert!(!e.read("rex.toml").contains("strs"));
e.ok(&["add", &url(&json)]);
// a package that only json uses
e.app(&[("main.rex", "import strs\n\nfunc main() { }\n")]);
let err = e.fails(&["run", "main.rex"]);
assert!(err.contains("`strs` is a package that one of your packages uses"), "{err}");
// a file of your own with the same name as a package
e.app(&[("main.rex", "import json\n\nfunc main() { }\n"), ("json.rex", "pub func x() {}\n")]);
let err = e.fails(&["run", "main.rex"]);
assert!(err.contains("both your file") && err.contains("the package `json`"), "{err}");
std::fs::remove_file(e.root.join("app/json.rex")).unwrap();
// a missing file inside a package
e.app(&[("main.rex", "import json.nope\n\nfunc main() { }\n")]);
let err = e.fails(&["run", "main.rex"]);
assert!(err.contains("the package `json` has no file nope.rex"), "{err}");
// a version without a git address (kept for a registry)
e.app(&[("rex.toml", "[deps]\njson = \"1.2.0\"\n")]);
let err = e.fails(&["run", "main.rex"]);
assert!(err.contains("there is no package registry yet"), "{err}");
}
#[test]
fn two_major_versions_of_one_package_are_an_error() {
let e = Env::new();
let (strs, json) = two_packages(&e);
release(&strs, &[], Some("v2.0.0"));
e.ok(&["init"]);
e.ok(&["add", &url(&json)]);
let err = e.fails(&["add", &format!("{}@v2.0.0", url(&strs))]);
assert!(err.contains("two different packages called `strs`") && err.contains("v1.0.0") && err.contains("v2.0.0"), "{err}");
assert!(!e.read("rex.toml").contains("strs"), "rex.toml is left as it was");
}
#[test]
fn import_keys() {
let mut s = Scope::default();
s.root_deps = vec!["json".into()];
s.packages.insert("json".into(), vec!["strs".into()]);
s.packages.insert("strs".into(), vec![]);
assert_eq!(s.key("", "json").unwrap(), "@json");
assert_eq!(s.key("", "json.parser").unwrap(), "@json.parser");
assert_eq!(s.key("", "util").unwrap(), "util");
assert_eq!(s.key("", "strs").unwrap(), "strs"); // not the project's own package
assert_eq!(s.key("@json", "parser").unwrap(), "@json.parser");
assert_eq!(s.key("@json.parser", "json").unwrap(), "@json");
assert_eq!(s.key("@json", "strs").unwrap(), "@strs");
assert_eq!(s.key("@strs", "util").unwrap(), "@strs.util");
assert!(s.key("@json", "..up").is_err());
}
#[test]
fn reading_and_editing_rex_toml() {
let m = parse_manifest("# mine\n[package]\nname = \"game\" # the name\n\n[deps]\njson = \"a.com/b/json@v1.0.0\"\n", "rex.toml").unwrap();
assert_eq!(m.name.as_deref(), Some("game"));
assert_eq!(m.deps, vec![("json".to_string(), "a.com/b/json@v1.0.0".to_string(), 6)]);
assert!(parse_manifest("[deps]\njson = 1\n", "rex.toml").unwrap_err().contains("rex.toml:2"));
let t = "[package]\nname = \"g\"\n\n[deps]\njson = \"x/y/json\"\n\n[other]\nk = \"v\"\n";
let added = edit_manifest(t, "strs", Some("x/y/strs@v1"));
assert_eq!(added, "[package]\nname = \"g\"\n\n[deps]\njson = \"x/y/json\"\nstrs = \"x/y/strs@v1\"\n\n[other]\nk = \"v\"\n");
assert_eq!(edit_manifest(&added, "json", Some("x/y/json@v2")).lines().nth(4), Some("json = \"x/y/json@v2\""));
assert_eq!(edit_manifest(&added, "strs", None), t);
assert_eq!(edit_manifest("", "a", Some("x/a")), "[deps]\na = \"x/a\"\n");
assert_eq!(edit_manifest("[package]\nname = \"g\"\n", "a", Some("x/a")), "[package]\nname = \"g\"\n\n[deps]\na = \"x/a\"\n");
}
#[test]
fn where_packages_come_from() {
let here = Path::new("/p");
let g = |u: &str, r: Option<&str>| Source::Git { url: u.into(), rev: r.map(Into::into) };
assert_eq!(parse_source("git.joji.li/joji/json@v1.2.0", here).unwrap(), g("git.joji.li/joji/json", Some("v1.2.0")));
assert_eq!(parse_source("github.com/a/b.git", here).unwrap(), g("github.com/a/b", None));
assert_eq!(parse_source("git@host.com:a/b@main", here).unwrap(), g("git@host.com:a/b", Some("main")));
assert_eq!(parse_source("git@host.com:a/b", here).unwrap(), g("git@host.com:a/b", None));
assert_eq!(parse_source("../lib", here).unwrap(), Source::Path(PathBuf::from("/p/../lib")));
assert!(parse_source("1.2.0", here).unwrap_err().contains("registry"));
assert!(parse_source("json", here).is_err());
let tags: Vec<String> = ["v1.0.0", "v1.10.0", "v1.9.0", "v2.0.0-beta.1", "junk"].iter().map(|s| s.to_string()).collect();
assert_eq!(newest_tag(&tags, None).as_deref(), Some("v1.10.0"));
assert_eq!(newest_tag(&tags, Some(2)).as_deref(), Some("v2.0.0-beta.1"));
}
+9 -1
View File
@@ -2208,7 +2208,15 @@ fn docs_examples_run() {
let end = after.find("```").unwrap(); let end = after.find("```").unwrap();
let code = &after[..end]; let code = &after[..end];
rest = &after[end + 3..]; rest = &after[end + 3..];
if !code.contains("func main") || code.contains("import utils") { // programs that import files or packages that aren't in the text can't run alone
let foreign = code.lines().any(|l| {
// `import "zlib.h"` and `import z "zlib.h"` are C headers
l.strip_prefix("import ").map_or(false, |rest| {
let w = rest.split_whitespace().next().unwrap_or("");
!rest.contains('"') && !rex::stdlib::MODULES.contains(&w)
})
});
if !code.contains("func main") || foreign {
continue; continue;
} }
// the output block, if one follows right away // the output block, if one follows right away