Nix VM integration tests (#8339)

This adds a couple of Nix-based VM integration tests:

- Does `ghostty +version` run successfully?
- Can we create a new terminal window? (This is detected by setting the
background to a color that doesn't appear
normally on the desktop and seeing if we can detect that color in a
screenshot).

Obviously more can be done but I thought that these would be a couple of
good first steps.

The whole test suite can be run with `nix flake check`. Individual tests
can be run with a command like this:

```
nix run .#check.x86_64-linux.<test name>.driver
```

.gitignore

@@ -11,6 +11,7 @@ zig-cache/
 .zig-cache/
 zig-out/
 /result*
+/.nixos-test-history
 example/*.wasm
 test/ghostty
 test/cases/**/*.actual.png

CONTRIBUTING.md

@@ -232,3 +232,266 @@ pull request will be accepted with a high degree of certainty.
 > **Pull requests are NOT a place to discuss feature design.** Please do
 > not open a WIP pull request to discuss a feature. Instead, use a discussion
 > and link to your branch.
+
+# Developer Guide
+
+> [!NOTE]
+>
+> **The remainder of this file is dedicated to developers actively
+> working on Ghostty.** If you're a user reporting an issue, you can
+> ignore the rest of this document.
+
+## Including and Updating Translations
+
+See the [Contributor's Guide](po/README_CONTRIBUTORS.md) for more details.
+
+## Checking for Memory Leaks
+
+While Zig does an amazing job of finding and preventing memory leaks,
+Ghostty uses many third-party libraries that are written in C. Improper usage
+of those libraries or bugs in those libraries can cause memory leaks that
+Zig cannot detect by itself.
+
+### On Linux
+
+On Linux the recommended tool to check for memory leaks is Valgrind. The
+recommended way to run Valgrind is via `zig build`:
+
+```sh
+zig build run-valgrind
+```
+
+This builds a Ghostty executable with Valgrind support and runs Valgrind
+with the proper flags to ensure we're suppressing known false positives.
+
+You can combine the same build args with `run-valgrind` that you can with
+`run`, such as specifying additional configurations after a trailing `--`.
+
+## Input Stack Testing
+
+The input stack is the part of the codebase that starts with a
+key event and ends with text encoding being sent to the pty (it
+does not include _rendering_ the text, which is part of the
+font or rendering stack).
+
+If you modify any part of the input stack, you must manually verify
+all the following input cases work properly. We unfortunately do
+not automate this in any way, but if we can do that one day that'd
+save a LOT of grief and time.
+
+Note: this list may not be exhaustive, I'm still working on it.
+
+### Linux IME
+
+IME (Input Method Editors) are a common source of bugs in the input stack,
+especially on Linux since there are multiple different IME systems
+interacting with different windowing systems and application frameworks
+all written by different organizations.
+
+The following matrix should be tested to ensure that all IME input works
+properly:
+
+1. Wayland, X11
+2. ibus, fcitx, none
+3. Dead key input (e.g. Spanish), CJK (e.g. Japanese), Emoji, Unicode Hex
+4. ibus versions: 1.5.29, 1.5.30, 1.5.31 (each exhibit slightly different behaviors)
+
+> [!NOTE]
+>
+> This is a **work in progress**. I'm still working on this list and it
+> is not complete. As I find more test cases, I will add them here.
+
+#### Dead Key Input
+
+Set your keyboard layout to "Spanish" (or another layout that uses dead keys).
+
+1. Launch Ghostty
+2. Press `'`
+3. Press `a`
+4. Verify that `á` is displayed
+
+Note that the dead key may or may not show a preedit state visually.
+For ibus and fcitx it does but for the "none" case it does not. Importantly,
+the text should be correct when it is sent to the pty.
+
+We should also test canceling dead key input:
+
+1. Launch Ghostty
+2. Press `'`
+3. Press escape
+4. Press `a`
+5. Verify that `a` is displayed (no diacritic)
+
+#### CJK Input
+
+Configure fcitx or ibus with a keyboard layout like Japanese or Mozc. The
+exact layout doesn't matter.
+
+1. Launch Ghostty
+2. Press `Ctrl+Shift` to switch to "Hiragana"
+3. On a US physical layout, type: `konn`, you should see `こん` in preedit.
+4. Press `Enter`
+5. Verify that `こん` is displayed in the terminal.
+
+We should also test switching input methods while preedit is active, which
+should commit the text:
+
+1. Launch Ghostty
+2. Press `Ctrl+Shift` to switch to "Hiragana"
+3. On a US physical layout, type: `konn`, you should see `こん` in preedit.
+4. Press `Ctrl+Shift` to switch to another layout (any)
+5. Verify that `こん` is displayed in the terminal as committed text.
+
+## Nix Virtual Machines
+
+Several Nix virtual machine definitions are provided by the project for testing
+and developing Ghostty against multiple different Linux desktop environments.
+
+Running these requires a working Nix installation, either Nix on your
+favorite Linux distribution, NixOS, or macOS with nix-darwin installed. Further
+requirements for macOS are detailed below.
+
+VMs should only be run on your local desktop and then powered off when not in
+use, which will discard any changes to the VM.
+
+The VM definitions provide minimal software "out of the box" but additional
+software can be installed by using standard Nix mechanisms like `nix run nixpkgs#<package>`.
+
+### Linux
+
+1. Check out the Ghostty source and change to the directory.
+2. Run `nix run .#<vmtype>`. `<vmtype>` can be any of the VMs defined in the
+   `nix/vm` directory (without the `.nix` suffix) excluding any file prefixed
+   with `common` or `create`.
+3. The VM will build and then launch. Depending on the speed of your system, this
+   can take a while, but eventually you should get a new VM window.
+4. The Ghostty source directory should be mounted to `/tmp/shared` in the VM. Depending
+   on what UID and GID of the user that you launched the VM as, `/tmp/shared` _may_ be
+   writable by the VM user, so be careful!
+
+### macOS
+
+1. To run the VMs on macOS you will need to enable the Linux builder in your `nix-darwin`
+   config. This _should_ be as simple as adding `nix.linux-builder.enable=true` to your
+   configuration and then rebuilding. See [this](https://nixcademy.com/posts/macos-linux-builder/)
+   blog post for more information about the Linux builder and how to tune the performance.
+2. Once the Linux builder has been enabled, you should be able to follow the Linux instructions
+   above to launch a VM.
+
+### Custom VMs
+
+To easily create a custom VM without modifying the Ghostty source, create a new
+directory, then create a file called `flake.nix` with the following text in the
+new directory.
+
+```
+{
+  inputs = {
+    nixpkgs.url = "nixpkgs/nixpkgs-unstable";
+    ghostty.url = "github:ghostty-org/ghostty";
+  };
+  outputs = {
+    nixpkgs,
+    ghostty,
+    ...
+  }: {
+   nixosConfigurations.custom-vm = ghostty.create-gnome-vm {
+     nixpkgs = nixpkgs;
+     system = "x86_64-linux";
+     overlay = ghostty.overlays.releasefast;
+     # module = ./configuration.nix # also works
+     module = {pkgs, ...}: {
+       environment.systemPackages = [
+         pkgs.btop
+       ];
+     };
+    };
+  };
+}
+```
+
+The custom VM can then be run with a command like this:
+
+```
+nix run .#nixosConfigurations.custom-vm.config.system.build.vm
+```
+
+A file named `ghostty.qcow2` will be created that is used to persist any changes
+made in the VM. To "reset" the VM to default delete the file and it will be
+recreated the next time you run the VM.
+
+### Contributing new VM definitions
+
+#### VM Acceptance Criteria
+
+We welcome the contribution of new VM definitions, as long as they meet the following criteria:
+
+1. They should be different enough from existing VM definitions that they represent a distinct
+   user (and developer) experience.
+2. There's a significant Ghostty user population that uses a similar environment.
+3. The VMs can be built using only packages from the current stable NixOS release.
+
+#### VM Definition Criteria
+
+1. VMs should be as minimal as possible so that they build and launch quickly.
+   Additional software can be added at runtime with a command like `nix run nixpkgs#<package name>`.
+2. VMs should not expose any services to the network, or run any remote access
+   software like SSH daemons, VNC or RDP.
+3. VMs should auto-login using the "ghostty" user.
+
+## Nix VM Integration Tests
+
+Several Nix VM tests are provided by the project for testing Ghostty in a "live"
+environment rather than just unit tests.
+
+Running these requires a working Nix installation, either Nix on your
+favorite Linux distribution, NixOS, or macOS with nix-darwin installed. Further
+requirements for macOS are detailed below.
+
+### Linux
+
+1. Check out the Ghostty source and change to the directory.
+2. Run `nix run .#check.<system>.<test-name>.driver`. `<system>` should be
+   `x86_64-linux` or `aarch64-linux` (even on macOS, this launches a Linux
+   VM, not a macOS one). `<test-name>` should be one of the tests defined in
+   `nix/tests.nix`. The test will build and then launch. Depending on the speed
+   of your system, this can take a while. Eventually though the test should
+   complete. Hopefully successfully, but if not error messages should be printed
+   out that can be used to diagnose the issue.
+3. To run _all_ of the tests, run `nix flake check`.
+
+### macOS
+
+1. To run the VMs on macOS you will need to enable the Linux builder in your `nix-darwin`
+   config. This _should_ be as simple as adding `nix.linux-builder.enable=true` to your
+   configuration and then rebuilding. See [this](https://nixcademy.com/posts/macos-linux-builder/)
+   blog post for more information about the Linux builder and how to tune the performance.
+2. Once the Linux builder has been enabled, you should be able to follow the Linux instructions
+   above to launch a test.
+
+### Interactively Running Test VMs
+
+To run a test interactively, run `nix run
+.#check.<system>.<test-name>.driverInteractive`. This will load a Python console
+that can be used to manage the test VMs. In this console run `start_all()` to
+start the VM(s). The VMs should boot up and a window should appear showing the
+VM's console.
+
+For more information about the Nix test console, see [the NixOS manual](https://nixos.org/manual/nixos/stable/index.html#sec-call-nixos-test-outside-nixos)
+
+### SSH Access to Test VMs
+
+Some test VMs are configured to allow outside SSH access for debugging. To
+access the VM, use a command like the following:
+
+```
+ssh -o StrictHostKeyChecking=accept-new -o UserKnownHostsFile=/dev/null -p 2222 root@192.168.122.1
+ssh -o StrictHostKeyChecking=accept-new -o UserKnownHostsFile=/dev/null -p 2222 ghostty@192.168.122.1
+```
+
+The SSH options are important because the SSH host keys will be regenerated
+every time the test is started. Without them, your personal SSH known hosts file
+will become difficult to manage. The port that is needed to access the VM may
+change depending on the test.
+
+None of the users in the VM have passwords so do not expose these VMs to the Internet.

flake.lock

@@ -34,6 +34,27 @@
         "type": "github"
       }
     },
+    "home-manager": {
+      "inputs": {
+        "nixpkgs": [
+          "nixpkgs"
+        ]
+      },
+      "locked": {
+        "lastModified": 1755776884,
+        "narHash": "sha256-CPM7zm6csUx7vSfKvzMDIjepEJv1u/usmaT7zydzbuI=",
+        "owner": "nix-community",
+        "repo": "home-manager",
+        "rev": "4fb695d10890e9fc6a19deadf85ff79ffb78da86",
+        "type": "github"
+      },
+      "original": {
+        "owner": "nix-community",
+        "ref": "release-25.05",
+        "repo": "home-manager",
+        "type": "github"
+      }
+    },
     "nixpkgs": {
       "locked": {
         "lastModified": 1763191728,
@@ -51,6 +72,7 @@
       "inputs": {
         "flake-compat": "flake-compat",
         "flake-utils": "flake-utils",
+        "home-manager": "home-manager",
         "nixpkgs": "nixpkgs",
         "zig": "zig",
         "zon2nix": "zon2nix"

flake.nix

@@ -33,6 +33,13 @@
         nixpkgs.follows = "nixpkgs";
       };
     };
+
+    home-manager = {
+      url = "github:nix-community/home-manager?ref=release-25.05";
+      inputs = {
+        nixpkgs.follows = "nixpkgs";
+      };
+    };
   };
 
   outputs = {
@@ -40,6 +47,7 @@
     nixpkgs,
     zig,
     zon2nix,
+    home-manager,
     ...
   }:
     builtins.foldl' nixpkgs.lib.recursiveUpdate {} (
@@ -47,7 +55,7 @@
         system: let
           pkgs = nixpkgs.legacyPackages.${system};
         in {
-          devShell.${system} = pkgs.callPackage ./nix/devShell.nix {
+          devShells.${system}.default = pkgs.callPackage ./nix/devShell.nix {
             zig = zig.packages.${system}."0.15.2";
             wraptest = pkgs.callPackage ./nix/pkgs/wraptest.nix {};
             zon2nix = zon2nix;
@@ -79,6 +87,10 @@
 
           formatter.${system} = pkgs.alejandra;
 
+          checks.${system} = import ./nix/tests.nix {
+            inherit home-manager nixpkgs self system;
+          };
+
           apps.${system} = let
             runVM = (
               module: let
@@ -95,6 +107,9 @@
               in {
                 type = "app";
                 program = "${program}";
+                meta = {
+                  description = "start a vm from ${toString module}";
+                };
               }
             );
           in {
@@ -120,11 +135,6 @@
           ghostty = self.packages.${prev.stdenv.hostPlatform.system}.ghostty-debug;
         };
       };
-      create-vm = import ./nix/vm/create.nix;
-      create-cinnamon-vm = import ./nix/vm/create-cinnamon.nix;
-      create-gnome-vm = import ./nix/vm/create-gnome.nix;
-      create-plasma6-vm = import ./nix/vm/create-plasma6.nix;
-      create-xfce-vm = import ./nix/vm/create-xfce.nix;
     };
 
   nixConfig = {

nix/tests.nix

@@ -0,0 +1,283 @@
+{
+  self,
+  system,
+  nixpkgs,
+  home-manager,
+  ...
+}: let
+  nixos-version = nixpkgs.lib.trivial.release;
+
+  pkgs = import nixpkgs {
+    inherit system;
+    overlays = [
+      self.overlays.debug
+    ];
+  };
+
+  pink_value = "#FF0087";
+
+  color_test = ''
+    import tempfile
+    import subprocess
+
+    def check_for_pink(final=False) -> bool:
+        with tempfile.NamedTemporaryFile() as tmpin:
+            machine.send_monitor_command("screendump {}".format(tmpin.name))
+
+            cmd = 'convert {} -define histogram:unique-colors=true -format "%c" histogram:info:'.format(
+                tmpin.name
+            )
+            ret = subprocess.run(cmd, shell=True, capture_output=True)
+            if ret.returncode != 0:
+                raise Exception(
+                    "image analysis failed with exit code {}".format(ret.returncode)
+                )
+
+            text = ret.stdout.decode("utf-8")
+            return "${pink_value}" in text
+  '';
+
+  mkNodeGnome = {
+    config,
+    pkgs,
+    settings,
+    sshPort ? null,
+    ...
+  }: {
+    imports = [
+      ./vm/wayland-gnome.nix
+      settings
+    ];
+
+    virtualisation = {
+      forwardPorts = pkgs.lib.optionals (sshPort != null) [
+        {
+          from = "host";
+          host.port = sshPort;
+          guest.port = 22;
+        }
+      ];
+
+      vmVariant = {
+        virtualisation.host.pkgs = pkgs;
+      };
+    };
+
+    services.openssh = {
+      enable = true;
+      settings = {
+        PermitRootLogin = "yes";
+        PermitEmptyPasswords = "yes";
+      };
+    };
+
+    security.pam.services.sshd.allowNullPassword = true;
+
+    users.groups.ghostty = {
+      gid = 1000;
+    };
+
+    users.users.ghostty = {
+      uid = 1000;
+    };
+
+    home-manager = {
+      users = {
+        ghostty = {
+          home = {
+            username = config.users.users.ghostty.name;
+            homeDirectory = config.users.users.ghostty.home;
+            stateVersion = nixos-version;
+          };
+          programs.ssh = {
+            enable = true;
+            extraOptionOverrides = {
+              StrictHostKeyChecking = "accept-new";
+              UserKnownHostsFile = "/dev/null";
+            };
+          };
+        };
+      };
+    };
+
+    system.stateVersion = nixos-version;
+  };
+
+  mkTestGnome = {
+    name,
+    settings,
+    testScript,
+    ocr ? false,
+  }:
+    pkgs.testers.runNixOSTest {
+      name = name;
+
+      enableOCR = ocr;
+
+      extraBaseModules = {
+        imports = [
+          home-manager.nixosModules.home-manager
+        ];
+      };
+
+      nodes = {
+        machine = {
+          config,
+          pkgs,
+          ...
+        }:
+          mkNodeGnome {
+            inherit config pkgs settings;
+            sshPort = 2222;
+          };
+      };
+
+      testScript = testScript;
+    };
+in {
+  basic-version-check = pkgs.testers.runNixOSTest {
+    name = "basic-version-check";
+    nodes = {
+      machine = {pkgs, ...}: {
+        users.groups.ghostty = {};
+        users.users.ghostty = {
+          isNormalUser = true;
+          group = "ghostty";
+          extraGroups = ["wheel"];
+          hashedPassword = "";
+          packages = [
+            pkgs.ghostty
+          ];
+        };
+      };
+    };
+    testScript = {...}: ''
+      machine.succeed("su - ghostty -c 'ghostty +version'")
+    '';
+  };
+
+  basic-window-check-gnome = mkTestGnome {
+    name = "basic-window-check-gnome";
+    settings = {
+      home-manager.users.ghostty = {
+        xdg.configFile = {
+          "ghostty/config".text = ''
+            background = ${pink_value}
+          '';
+        };
+      };
+    };
+    ocr = true;
+    testScript = {nodes, ...}: let
+      user = nodes.machine.users.users.ghostty;
+      bus_path = "/run/user/${toString user.uid}/bus";
+      bus = "DBUS_SESSION_BUS_ADDRESS=unix:path=${bus_path}";
+      gdbus = "${bus} gdbus";
+      ghostty = "${bus} ghostty";
+      su = command: "su - ${user.name} -c '${command}'";
+      gseval = "call --session -d org.gnome.Shell -o /org/gnome/Shell -m org.gnome.Shell.Eval";
+      wm_class = su "${gdbus} ${gseval} global.display.focus_window.wm_class";
+    in ''
+      ${color_test}
+
+      with subtest("wait for x"):
+          start_all()
+          machine.wait_for_x()
+
+      machine.wait_for_file("${bus_path}")
+
+      with subtest("Ensuring no pink is present without the terminal."):
+          assert (
+              check_for_pink() == False
+          ), "Pink was present on the screen before we even launched a terminal!"
+
+      machine.systemctl("enable app-com.mitchellh.ghostty-debug.service", user="${user.name}")
+      machine.succeed("${su "${ghostty} +new-window"}")
+      machine.wait_until_succeeds("${wm_class} | grep -q 'com.mitchellh.ghostty-debug'")
+
+      machine.sleep(2)
+
+      with subtest("Have the terminal display a color."):
+          assert(
+              check_for_pink() == True
+          ), "Pink was not found on the screen!"
+
+      machine.systemctl("stop app-com.mitchellh.ghostty-debug.service", user="${user.name}")
+    '';
+  };
+
+  ssh-integration-test = pkgs.testers.runNixOSTest {
+    name = "ssh-integration-test";
+    extraBaseModules = {
+      imports = [
+        home-manager.nixosModules.home-manager
+      ];
+    };
+    nodes = {
+      server = {...}: {
+        users.groups.ghostty = {};
+        users.users.ghostty = {
+          isNormalUser = true;
+          group = "ghostty";
+          extraGroups = ["wheel"];
+          hashedPassword = "";
+          packages = [];
+        };
+        services.openssh = {
+          enable = true;
+          settings = {
+            PermitRootLogin = "yes";
+            PermitEmptyPasswords = "yes";
+          };
+        };
+        security.pam.services.sshd.allowNullPassword = true;
+      };
+      client = {
+        config,
+        pkgs,
+        ...
+      }:
+        mkNodeGnome {
+          inherit config pkgs;
+          settings = {
+            home-manager.users.ghostty = {
+              xdg.configFile = {
+                "ghostty/config".text = let
+                in ''
+                  shell-integration-features = ssh-terminfo
+                '';
+              };
+            };
+          };
+          sshPort = 2222;
+        };
+    };
+    testScript = {nodes, ...}: let
+      user = nodes.client.users.users.ghostty;
+      bus_path = "/run/user/${toString user.uid}/bus";
+      bus = "DBUS_SESSION_BUS_ADDRESS=unix:path=${bus_path}";
+      gdbus = "${bus} gdbus";
+      ghostty = "${bus} ghostty";
+      su = command: "su - ${user.name} -c '${command}'";
+      gseval = "call --session -d org.gnome.Shell -o /org/gnome/Shell -m org.gnome.Shell.Eval";
+      wm_class = su "${gdbus} ${gseval} global.display.focus_window.wm_class";
+    in ''
+      with subtest("Start server and wait for ssh to be ready."):
+          server.start()
+          server.wait_for_open_port(22)
+
+      with subtest("Start client and wait for ghostty window."):
+          client.start()
+          client.wait_for_x()
+          client.wait_for_file("${bus_path}")
+          client.systemctl("enable app-com.mitchellh.ghostty-debug.service", user="${user.name}")
+          client.succeed("${su "${ghostty} +new-window"}")
+          client.wait_until_succeeds("${wm_class} | grep -q 'com.mitchellh.ghostty-debug'")
+
+      with subtest("SSH from client to server and verify that the Ghostty terminfo is copied.")
+          client.sleep(2)
+          client.send_chars("ssh ghostty@server\n")
+          server.wait_for_file("${user.home}/.terminfo/x/xterm-ghostty", timeout=30)
+    '';
+  };
+}

nix/vm/common-gnome.nix

@@ -22,6 +22,19 @@
     };
   };
 
+  systemd.user.services = {
+    "org.gnome.Shell@wayland" = {
+      serviceConfig = {
+        ExecStart = [
+          # Clear the list before overriding it.
+          ""
+          # Eval API is now internal so Shell needs to run in unsafe mode.
+          "${pkgs.gnome-shell}/bin/gnome-shell --unsafe-mode"
+        ];
+      };
+    };
+  };
+
   environment.systemPackages = [
     pkgs.gnomeExtensions.no-overview
   ];

nix/vm/common.nix

@@ -4,9 +4,6 @@
 
   documentation.nixos.enable = false;
 
-  networking.hostName = "ghostty";
-  networking.domain = "mitchellh.com";
-
   virtualisation.vmVariant = {
     virtualisation.memorySize = 2048;
   };
@@ -28,17 +25,11 @@
   users.groups.ghostty = {};
 
   users.users.ghostty = {
+    isNormalUser = true;
     description = "Ghostty";
     group = "ghostty";
     extraGroups = ["wheel"];
-    isNormalUser = true;
+    hashedPassword = "";
-    initialPassword = "ghostty";
-  };
-
-  environment.etc = {
-    "xdg/autostart/com.mitchellh.ghostty.desktop" = {
-      source = "${pkgs.ghostty}/share/applications/com.mitchellh.ghostty.desktop";
-    };
   };
 
   environment.systemPackages = [
@@ -61,6 +52,7 @@
 
   services.displayManager = {
     autoLogin = {
+      enable = true;
       user = "ghostty";
     };
   };