From c371b7353bef29a7b10ca0c55a401e5e8401a752 Mon Sep 17 00:00:00 2001 From: tannevaled Date: Fri, 13 Nov 2026 14:48:49 +0100 Subject: [PATCH 1/2] conformance: an oracle that cannot load its gems scores zero, and a sweep calls that a result MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit ⛔ WHAT HAPPENED. The pkgx MRI 4.0.6 on this machine starts with RubyGems absent from its load path: `RubyGems' were not loaded. cannot load such file -- rubygems (LoadError) net/ftp, net/http, stringio and most of the rest have been BUNDLED GEMS since Ruby 3.1, so every library/net-ftp spec raised LoadError before running a single example. That is not an error the sweep notices: the interpreter exits, contributes 0, and cmd/specdelta prints 56 file(s): control 0 passing, treatment 331 passing, +331 which reads exactly like rbgo beating the reference. I compared the whole module that way before reading the interpreter's stderr. A per-file score cannot tell "failed every example" from "never started", and the silence of a broken instrument wears the shape of a measurement. cmd/mriref puts every gem's lib directory on the load path -- what RubyGems would have resolved anyway, rather than hand-resolving net/ftp's closure (net-protocol, timeout, resolv, io-wait, stringio, ...). It takes one script, so it drops into cmd/specdelta and run.sh wherever an interpreter is named. ⛔ IT REFUSES AN EMPTY GEM ROOT, which is the whole point and is what the tests assert first: a gem root yielding no lib directory is not a gem-free Ruby, it is the wrong path, and running anyway reproduces precisely the failure above. Three cases are covered (empty directory, gems that have no lib, a path that does not exist), plus both settings being required and the -I list being SORTED -- a load path whose order moves between runs can change which gem wins a name collision, and an oracle has to be reproducible. THE CHECK THAT IT WORKS, and the reason this is worth having rather than a note in someone's shell history: with the gems on the path that interpreter scores 459 on library/net-ftp's 56 files -- the same figure this repository's own dated sweep recorded for MRI. An oracle that reproduces a number measured independently is one you can lean on. Co-Authored-By: Claude Opus 5 --- scripts/conformance/rubyspec/README.md | 48 +++++++ .../conformance/rubyspec/cmd/mriref/main.go | 117 ++++++++++++++++++ .../rubyspec/cmd/mriref/main_test.go | 102 +++++++++++++++ 3 files changed, 267 insertions(+) create mode 100644 scripts/conformance/rubyspec/cmd/mriref/main.go create mode 100644 scripts/conformance/rubyspec/cmd/mriref/main_test.go diff --git a/scripts/conformance/rubyspec/README.md b/scripts/conformance/rubyspec/README.md index 740a2776..dbbd1d12 100644 --- a/scripts/conformance/rubyspec/README.md +++ b/scripts/conformance/rubyspec/README.md @@ -295,6 +295,54 @@ Those 50 files are the surface any claim about the move has to cover. It remains the right instrument for the other half too: a `UPDATE_BASELINE=1` that absorbs a loss silently. +## Making the reference interpreter able to answer — `cmd/mriref` + +⛔ **An MRI that cannot load its bundled gems scores a clean ZERO on every file, +and a sweep reports that as a result.** + +A pkgx-installed MRI 4.0.6 on a developer machine starts with RubyGems absent +from its load path: + +```text +`RubyGems' were not loaded. +cannot load such file -- rubygems (LoadError) +``` + +`net/ftp`, `net/http`, `stringio`, `bigdecimal` and most of the rest have been +**bundled gems** since Ruby 3.1, so on such an interpreter every spec file that +needs one raises `LoadError` before a single example runs. It does not error +out of the sweep — it contributes 0, 56 times, and `cmd/specdelta` prints: + +```text +56 file(s): control 0 passing, treatment 331 passing, +331 +``` + +which reads exactly like rbgo beating the reference. The whole of +`library/net-ftp` was compared that way once before anyone read the +interpreter's stderr: **a per-file score cannot tell "failed every example" +from "never started".** + +`cmd/mriref` puts every gem's `lib` on the load path — what RubyGems would have +resolved anyway — and refuses to run if the gem root yields none, since an +empty `-I` list reproduces the exact failure it exists to prevent: + +```sh +MRIREF_RUBY=~/.pkgx/ruby-lang.org/v4.0.6/bin/ruby \ +MRIREF_GEMS=~/.pkgx/ruby-lang.org/v4.0.6/lib/ruby/gems/4.0.0/gems \ +go run ./scripts/conformance/rubyspec/cmd/specdelta \ + -root ~/ruby-spec-corpus -dir library/net-ftp -move 4.0.6:4.0.9 \ + $(go env GOPATH)/bin/mriref ./rbgo +``` + +It takes one script, so it drops into `cmd/specdelta` and `run.sh` wherever an +interpreter is named. + +**The check that it is working:** with the gems on the path that interpreter +scores **459** on `library/net-ftp`'s 56 files — the same figure this +repository's own dated sweep recorded for MRI. An oracle that reproduces a +number measured elsewhere is one you can lean on; one that scores zero is not +an oracle at all. + ## Asking the reference interpreter — `cmd/oracle` Every parity fix starts with *"what does MRI actually do here?"*, and the obvious diff --git a/scripts/conformance/rubyspec/cmd/mriref/main.go b/scripts/conformance/rubyspec/cmd/mriref/main.go new file mode 100644 index 00000000..2b2db035 --- /dev/null +++ b/scripts/conformance/rubyspec/cmd/mriref/main.go @@ -0,0 +1,117 @@ +// Copyright (c) the go-embedded-ruby/ruby authors +// +// SPDX-License-Identifier: BSD-3-Clause + +// Command mriref runs a reference MRI with every bundled gem on the load path, +// so it can serve as the conformance oracle. +// +// ⛔ WHY IT EXISTS, AND WHAT IT COST TO FIND. A pkgx-installed MRI 4.0.6 on a +// developer machine starts with RubyGems absent from its load path: +// +// `RubyGems' were not loaded. +// cannot load such file -- rubygems (LoadError) +// +// `net/ftp`, `net/http`, `stringio`, `bigdecimal` and most of the rest have +// been BUNDLED GEMS since Ruby 3.1, so on such an interpreter every ruby/spec +// file that needs one raises LoadError before a single example runs. +// +// Used as an oracle that way it does not error out: it scores a clean ZERO on +// every file. A sweep then reports `control 0 passing, treatment 331 passing, +// +331`, which reads exactly like the implementation under test beating the +// reference. It is the silence of a broken instrument wearing the shape of a +// measurement. The whole of library/net-ftp was compared that way once before +// anyone looked at the interpreter's stderr -- a per-file score cannot tell +// "failed every example" from "never started". +// +// With the gem lib directories on the path the same interpreter scores 459 on +// those 56 files, which is the figure the repository's own dated sweep +// recorded, so the oracle is checkable against something. +// +// Rather than resolve net/ftp's dependency closure by hand (net-protocol, +// timeout, resolv, io-wait, stringio, …) it adds EVERY gem's lib directory, +// which is what RubyGems would have resolved anyway. +// +// Usage, as an interpreter that takes one script -- the shape cmd/specdelta +// and run.sh expect: +// +// MRIREF_RUBY=/path/to/bin/ruby \ +// MRIREF_GEMS=/path/to/lib/ruby/gems//gems \ +// mriref library/net-ftp/acct_spec.rb +package main + +import ( + "errors" + "fmt" + "os" + "os/exec" + "path/filepath" + "sort" +) + +func main() { + args, err := buildArgs(os.Getenv("MRIREF_RUBY"), os.Getenv("MRIREF_GEMS"), os.Args[1:]) + if err != nil { + fmt.Fprintf(os.Stderr, "mriref: %v\n", err) + os.Exit(2) + } + cmd := exec.Command(os.Getenv("MRIREF_RUBY"), args...) + cmd.Stdin, cmd.Stdout, cmd.Stderr = os.Stdin, os.Stdout, os.Stderr + if err := cmd.Run(); err != nil { + var ee *exec.ExitError + if errors.As(err, &ee) { + os.Exit(ee.ExitCode()) + } + fmt.Fprintf(os.Stderr, "mriref: %v\n", err) + os.Exit(2) + } +} + +// errNoGems is returned when the gem root holds no lib directory at all. +var errNoGems = errors.New("no gem lib directory") + +// buildArgs turns a gem root into the -I list, then appends the caller's own +// arguments. +// +// ⛔ IT REFUSES AN EMPTY RESULT. A gem root that yields nothing is not a +// gem-free Ruby, it is the wrong path -- and running anyway would reproduce +// precisely the failure this command exists to prevent: an interpreter that +// cannot load its bundled gems, scoring zero on every file while looking like +// it ran. +func buildArgs(ruby, gems string, rest []string) ([]string, error) { + if ruby == "" || gems == "" { + return nil, errors.New("MRIREF_RUBY and MRIREF_GEMS must both be set") + } + libs, err := gemLibs(gems) + if err != nil { + return nil, err + } + if len(libs) == 0 { + return nil, fmt.Errorf("%w under %s -- refusing to run as if there were none", errNoGems, gems) + } + out := make([]string, 0, 2*len(libs)+len(rest)) + for _, l := range libs { + out = append(out, "-I", l) + } + return append(out, rest...), nil +} + +// gemLibs lists the lib directory of every gem under root, sorted so a run is +// reproducible. +func gemLibs(root string) ([]string, error) { + entries, err := os.ReadDir(root) + if err != nil { + return nil, err + } + var libs []string + for _, e := range entries { + if !e.IsDir() { + continue + } + lib := filepath.Join(root, e.Name(), "lib") + if st, err := os.Stat(lib); err == nil && st.IsDir() { + libs = append(libs, lib) + } + } + sort.Strings(libs) + return libs, nil +} diff --git a/scripts/conformance/rubyspec/cmd/mriref/main_test.go b/scripts/conformance/rubyspec/cmd/mriref/main_test.go new file mode 100644 index 00000000..bb9b4398 --- /dev/null +++ b/scripts/conformance/rubyspec/cmd/mriref/main_test.go @@ -0,0 +1,102 @@ +// Copyright (c) the go-embedded-ruby/ruby authors +// +// SPDX-License-Identifier: BSD-3-Clause + +package main + +import ( + "errors" + "os" + "path/filepath" + "strings" + "testing" +) + +// gemRoot builds a gem tree: each name becomes //lib, except names +// ending in "!" which get the gem directory and NO lib (a gem with nothing to +// add to the load path, which must be skipped rather than counted). +func gemRoot(t *testing.T, names ...string) string { + t.Helper() + root := t.TempDir() + for _, n := range names { + if strings.HasSuffix(n, "!") { + if err := os.MkdirAll(filepath.Join(root, strings.TrimSuffix(n, "!")), 0o755); err != nil { + t.Fatal(err) + } + continue + } + if err := os.MkdirAll(filepath.Join(root, n, "lib"), 0o755); err != nil { + t.Fatal(err) + } + } + return root +} + +// ⛔ THE REFUSAL IS THE POINT OF THIS COMMAND. An interpreter that cannot load +// its bundled gems does not fail loudly -- it scores ZERO on every spec file, +// which a sweep reports as a clean result for the control and reads as the +// implementation under test beating the reference. So a gem root that yields +// no lib directory must stop the run, not start it. +func TestBuildArgsRefusesAGemRootThatYieldsNothing(t *testing.T) { + for _, tc := range []struct { + name string + root string + }{ + {"an empty directory", gemRoot(t)}, + {"only gems with no lib", gemRoot(t, "a!", "b!")}, + {"a path that does not exist", filepath.Join(t.TempDir(), "absent")}, + } { + t.Run(tc.name, func(t *testing.T) { + if _, err := buildArgs("/bin/ruby", tc.root, []string{"x.rb"}); err == nil { + t.Fatal("it must refuse, not run with an empty -I list") + } + }) + } + // The empty-but-readable case names its own reason, so the message says what + // to fix rather than only that something failed. + _, err := buildArgs("/bin/ruby", gemRoot(t), []string{"x.rb"}) + if !errors.Is(err, errNoGems) { + t.Errorf("err = %v, want it to wrap errNoGems", err) + } +} + +// Both environment variables are required: a half-configured oracle would run +// the wrong interpreter, or the right one with no gems, and both score zero. +func TestBuildArgsRequiresBothSettings(t *testing.T) { + root := gemRoot(t, "net-ftp-0.3.9") + if _, err := buildArgs("", root, nil); err == nil { + t.Error("an unset MRIREF_RUBY must be refused") + } + if _, err := buildArgs("/bin/ruby", "", nil); err == nil { + t.Error("an unset MRIREF_GEMS must be refused") + } +} + +// The -I list carries every gem that has a lib, in a stable order, and the +// caller's own arguments come last so the script is still the script. +func TestBuildArgsPutsEveryGemLibBeforeTheScript(t *testing.T) { + root := gemRoot(t, "timeout-0.6.0", "net-ftp-0.3.9", "nolib!", "net-protocol-0.2.2") + got, err := buildArgs("/bin/ruby", root, []string{"library/net-ftp/acct_spec.rb", "--extra"}) + if err != nil { + t.Fatal(err) + } + // 3 gems with a lib -> 6 elements, then the 2 caller arguments. + if len(got) != 8 { + t.Fatalf("got %d argument(s) %v, want 8 (3 × -I plus 2 of the caller's)", len(got), got) + } + for i := 0; i < 6; i += 2 { + if got[i] != "-I" { + t.Errorf("argument %d = %q, want -I", i, got[i]) + } + } + // ⛔ Sorted, because a load path that changes order between runs can change + // which gem wins a name collision, and an oracle has to be reproducible. + if !strings.HasSuffix(got[1], "net-ftp-0.3.9/lib") || + !strings.HasSuffix(got[3], "net-protocol-0.2.2/lib") || + !strings.HasSuffix(got[5], "timeout-0.6.0/lib") { + t.Errorf("the -I list is not in sorted order: %v", got[:6]) + } + if got[6] != "library/net-ftp/acct_spec.rb" || got[7] != "--extra" { + t.Errorf("the caller's arguments must come last, got %v", got[6:]) + } +} From c111edff9dc01d2d47b1e4328f54679261534789 Mon Sep 17 00:00:00 2001 From: tannevaled Date: Fri, 13 Nov 2026 16:23:01 +0100 Subject: [PATCH 2/2] mriref: the order assertion was right about the order and wrong about the separator MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The Windows lane failed, and it was the TEST, not the command: main_test.go:97: the -I list is not in sorted order: [-I …\net-ftp-0.3.9\lib -I …\net-protocol-0.2.2\lib -I …\timeout-0.6.0\lib] The list printed in the error is in exactly the order the test wanted. The assertion compared `strings.HasSuffix(got[1], "net-ftp-0.3.9/lib")` — a forward slash — so it passed on macOS and Linux and could not pass on Windows, where filepath.Join builds `\`. Now compared by the GEM DIRECTORY'S NAME via filepath.Base(filepath.Dir(...)), which is separator-independent by construction and is also what the check actually means: entry i must be the i-th gem in sorted order. The error message names the gem it found instead of printing three long paths. ⛔ I cannot run the Windows lane here, so the claim is that the new assertion contains no separator at all, not that I have watched it pass on Windows. Co-Authored-By: Claude Opus 5 --- .../conformance/rubyspec/cmd/mriref/main_test.go | 15 +++++++++++---- 1 file changed, 11 insertions(+), 4 deletions(-) diff --git a/scripts/conformance/rubyspec/cmd/mriref/main_test.go b/scripts/conformance/rubyspec/cmd/mriref/main_test.go index bb9b4398..2f24ab26 100644 --- a/scripts/conformance/rubyspec/cmd/mriref/main_test.go +++ b/scripts/conformance/rubyspec/cmd/mriref/main_test.go @@ -91,10 +91,17 @@ func TestBuildArgsPutsEveryGemLibBeforeTheScript(t *testing.T) { } // ⛔ Sorted, because a load path that changes order between runs can change // which gem wins a name collision, and an oracle has to be reproducible. - if !strings.HasSuffix(got[1], "net-ftp-0.3.9/lib") || - !strings.HasSuffix(got[3], "net-protocol-0.2.2/lib") || - !strings.HasSuffix(got[5], "timeout-0.6.0/lib") { - t.Errorf("the -I list is not in sorted order: %v", got[:6]) + // + // Compared by the GEM DIRECTORY'S NAME, not by a "/lib" suffix: the + // first version of this test used the suffix and passed on macOS and Linux + // while failing on Windows, where the separator is `\`. The order was right + // all along; the assertion was not portable. Naming the directory is also + // what the check actually means. + want := []string{"net-ftp-0.3.9", "net-protocol-0.2.2", "timeout-0.6.0"} + for i, w := range want { + if gem := filepath.Base(filepath.Dir(got[2*i+1])); gem != w { + t.Errorf("-I entry %d is %q, want the gem %q (list: %v)", i, gem, w, got[:6]) + } } if got[6] != "library/net-ftp/acct_spec.rb" || got[7] != "--extra" { t.Errorf("the caller's arguments must come last, got %v", got[6:])