Considerably more search internals (#9585)

Chugging along towards #189

This adds significantly more internal work for searching. A long time
ago, I added #2885 which had a hint of what I was thinking of. This
simultaneously builds on this and changes direction.

The change of direction is that instead of making PageList fully
concurrency safe and having a search thread access it concurrently, I'm
now making an architectural shift where our search thread will grab the
big lock (blocking all IO/rendering), but with the bet that we can make
our critical areas small enough and time them well enough that the
performance hit while actively searching will be minimal. **Results yet
to be seen, but the path to implement this is much, much simpler.**

## Rearchitecting Search

To that end, this PR builds on #2885 by making `src/terminal/search` and
entire package (rather than a single file).

```mermaid
graph TB
    subgraph Layer5 ["<b>Layer 5: Thread Orchestration</b>"]
        Thread["<b>Thread</b><br/>━━━━━━━━━━━━━━━━━━━━━<br/>• MPSC queue management<br/>• libxev event loop<br/>• Message handling<br/>• Surface mailbox communication<br/>• Forward progress coordination"]
    end
    
    subgraph Layer4 ["<b>Layer 4: Screen Coordination</b>"]
        ScreenSearch["<b>ScreenSearch</b><br/>━━━━━━━━━━━━━━━━━━━━━<br/>• State machine (tick + feed)<br/>• Result caching<br/>• Per-screen (alt/primary)<br/>• Composes Active + History search<br/>• Interrupt handling"]
    end
    
    subgraph Layer3 ["<b>Layer 3: Domain-Specific Search</b>"]
        ActiveSearch["<b>ActiveSearch</b><br/>━━━━━━━━━━━━━━━━━━━━━<br/>• Active area only<br/>• Invalidate & re-search<br/>• Small, volatile data"]
        
        PageListSearch["<b>PageListSearch</b><br/>━━━━━━━━━━━━━━━━━━━━━<br/>• History search (reverse order)<br/>• Separated tick/feed ops<br/>• Immutable PageList assumption<br/>• Garbage pin detection"]
    end
    
    subgraph Layer2 ["<b>Layer 1: Primitive Operations</b>"]
        SlidingWindow["<b>SlidingWindow</b><br/>━━━━━━━━━━━━━━━━━━━━━<br/>• Manual linked list node management<br/>• Circular buffer maintenance<br/>• Zero-allocation search<br/>• Match yielding<br/>• Page boundary handling"]
    end
    
    Thread --> ScreenSearch
    ScreenSearch --> ActiveSearch
    ScreenSearch --> PageListSearch
    ActiveSearch --> SlidingWindow
    PageListSearch --> SlidingWindow
    
    classDef layer5 fill:#0a0a0a,stroke:#ff0066,stroke-width:3px,color:#ffffff
    classDef layer4 fill:#0f0f0f,stroke:#ff6600,stroke-width:3px,color:#ffffff
    classDef layer3 fill:#141414,stroke:#ffaa00,stroke-width:3px,color:#ffffff
    classDef layer2 fill:#1a1a1a,stroke:#00ff00,stroke-width:3px,color:#ffffff
    
    class Thread layer5
    class ScreenSearch layer4
    class ActiveSearch,PageListSearch layer3
    class SlidingWindow layer2
    
    style Layer5 fill:#050505,stroke:#ff0066,stroke-width:2px,color:#ffffff
    style Layer4 fill:#080808,stroke:#ff6600,stroke-width:2px,color:#ffffff
    style Layer3 fill:#0c0c0c,stroke:#ffaa00,stroke-width:2px,color:#ffffff
    style Layer2 fill:#101010,stroke:#00ff00,stroke-width:2px,color:#ffffff
```

Within the package, we have composable layers that let us test each
point:

- `SlidingWindow`: The lowest layer, the caller manually adds linked
list page nodes and it maintains a sliding window we search over,
yielding results without allocation (besides the circular buffers to
maintain the sliding window).
- `PageListSearch`: Searches a PageList structure in reverse order
(assumption: more recent matches are more valuable than older), but
separates out the `tick` (search, but no PageList access) and `feed`
(PageList access, prep data for search but don't search) operations.
This lets us `feed` in a critical area and `tick` outside. **This
assumes an immutable PageList, so this is for history.**
- `ActiveSearch`: Searches only the active area of a PageList. The
expectation is that the active area changes much more regularly, but it
is also very small (relative to scrollback). Throws away and re-searches
the active area as necessary.
- `ScreenSearch`: Composes the previous three components to coordinate
searching an active terminal screen. You'd have one of these per screen
(alt vs primary). This also caches results unlike the other components,
with the expectation that the caller will revisit the results as screens
change (so if you switch from neovim back to your shell and vice versa
with a search active, it won't start over).
- `Thread`: A dedicated search thread that will receive messages via
MPSC queues while managing the forward progress of a `ScreenSearch` and
sending matches back to the surface mailbox for apprt rendering. **The
thread component is not functional, just boilerplate, in this PR.**

ScreenSearch is a state machine that moves in an iterative `tick` +
`feed` fashion. This will let us "interrupt" the search with updates on
the search thread (read our mailbox via libxev loops for example) and
will let us minimize critical areas with locks (only `feed`).

Each component is significantly unit tested, especially around page
boundary cases. Given the complexity, there is no way this is perfect,
but the architecture is such that we can easily add regression tests as
we find issues.

## Other Changes, Notes

The only change to actually used code is that tracked pins in a
`PageList` can now be flagged as "garbage." A garbage tracked pin is one
that had to be moved in a non-sensical way because the previous location
it tracked has been deleted. This is used by the searcher to detect that
our history was pruned.

**If my assumption about the big lock is wrong** and this ends up being
godawful for performance, then it should still be okay because more
granular locking and reference counting such as that down by @dave-fl in
#8850 can be pushed into these components and reused. So this work is
still valuable on its own.

## Future

This PR is still just a bunch of internals, split out into its own PR so
I don't make one huge 10K diff PR. There are a number of future tasks:

- Flesh out `ScreenSearch` and hook it up to `Thread`
- Pull search thread management into `Surface` (or possibly the render
thread or shared render state since active area changes can be
synchronized with renderer frame rebuilds. Not sure yet.)
- Send updates back to the surface thread so that apprts can update UI.
- Apprt actions, input bindings, etc. to hook this all up (the easy
part, really).

The next step is to continue to flesh out the `ScreenSearch` as required
and hook it up to `Thread`.

**AI disclosure:** AI reviewed the code and assisted with some tests,
but didn't write any of the logic or design. This is beyond its ability
(or my ability to spec it out clearly enough for AI to succeed).
pull/9335/merge
Mitchell Hashimoto 2025-11-14 12:38:06 -08:00 committed by GitHub
commit a5a914c2b8
No known key found for this signature in database
GPG Key ID: B5690EEEBB952194
8 changed files with 2536 additions and 885 deletions

View File

@ -371,6 +371,9 @@ fn verifyIntegrity(self: *const PageList) IntegrityError!void {
if (comptime !build_options.slow_runtime_safety) return;
if (self.pause_integrity_checks > 0) return;
// Our viewport pin should never be garbage
assert(!self.viewport_pin.garbage);
// Verify that our cached total_rows matches the actual row count
const actual_total = self.totalRows();
if (actual_total != self.total_rows) {
@ -528,6 +531,8 @@ pub fn reset(self: *PageList) void {
self.total_rows = self.rows;
// Update all our tracked pins to point to our first page top-left
// and mark them as garbage, because it got mangled in a way where
// semantically it really doesn't make sense.
{
var it = self.tracked_pins.iterator();
while (it.next()) |entry| {
@ -535,7 +540,11 @@ pub fn reset(self: *PageList) void {
p.node = self.pages.first.?;
p.x = 0;
p.y = 0;
p.garbage = true;
}
// Our viewport pin is never garbage
self.viewport_pin.garbage = false;
}
// Move our viewport back to the active area since everything is gone.
@ -2428,7 +2437,9 @@ pub fn grow(self: *PageList) !?*List.Node {
p.node = self.pages.first.?;
p.y = 0;
p.x = 0;
p.garbage = true;
}
self.viewport_pin.garbage = false;
// In this case we do NOT need to update page_size because
// we're reusing an existing page so nothing has changed.
@ -3047,13 +3058,16 @@ pub fn eraseRows(
fn erasePage(self: *PageList, node: *List.Node) void {
assert(node.next != null or node.prev != null);
// Update any tracked pins to move to the next page.
// Update any tracked pins to move to the previous or next page.
const pin_keys = self.tracked_pins.keys();
for (pin_keys) |p| {
if (p.node != node) continue;
p.node = node.next orelse node.prev orelse unreachable;
p.node = node.prev orelse node.next orelse unreachable;
p.y = 0;
p.x = 0;
// This doesn't get marked garbage because the tracked pin
// movement is sensical.
}
// Remove the page from the linked list
@ -3844,8 +3858,9 @@ fn totalRows(self: *const PageList) usize {
return rows;
}
/// The total number of pages in this list.
fn totalPages(self: *const PageList) usize {
/// The total number of pages in this list. This should only be used
/// for tests since it is O(N) over the list of pages.
pub fn totalPages(self: *const PageList) usize {
var pages: usize = 0;
var node_ = self.pages.first;
while (node_) |node| {
@ -3903,6 +3918,13 @@ pub const Pin = struct {
y: size.CellCountInt = 0,
x: size.CellCountInt = 0,
/// This is flipped to true for tracked pins that were tracking
/// a page that got pruned for any reason and where the tracked pin
/// couldn't be moved to a sensical location. Users of the tracked
/// pin could use this data and make their own determination of
/// semantics.
garbage: bool = false,
pub inline fn rowAndCell(self: Pin) struct {
row: *pagepkg.Row,
cell: *pagepkg.Cell,
@ -5757,6 +5779,7 @@ test "PageList grow prune scrollback" {
try testing.expect(p.node == s.pages.first.?);
try testing.expect(p.x == 0);
try testing.expect(p.y == 0);
try testing.expect(p.garbage);
// Verify the viewport offset cache was invalidated. After pruning,
// the offset should have changed because we removed rows from
@ -10641,6 +10664,29 @@ test "PageList reset across two pages" {
try testing.expectEqual(@as(usize, s.rows), s.totalRows());
}
test "PageList reset moves tracked pins and marks them as garbage" {
const testing = std.testing;
const alloc = testing.allocator;
var s = try init(alloc, 80, 24, null);
defer s.deinit();
// Create a tracked pin into the active area
const p = try s.trackPin(s.pin(.{ .active = .{
.x = 42,
.y = 12,
} }).?);
defer s.untrackPin(p);
s.reset();
// Our added pin should now be garbage
try testing.expect(p.garbage);
// Viewport pin should not be garbage because it makes sense.
try testing.expect(!s.viewport_pin.garbage);
}
test "PageList clears history" {
const testing = std.testing;
const alloc = testing.allocator;

View File

@ -849,7 +849,7 @@ pub const PageFormatter = struct {
/// Initializes a page formatter. Other options can be set directly on the
/// struct after initialization and before calling `format()`.
pub fn init(page: *const Page, opts: Options) PageFormatter {
return PageFormatter{
return .{
.page = page,
.opts = opts,
.start_x = 0,

View File

@ -1,885 +1,13 @@
//! Search functionality for the terminal.
//!
//! At the time of writing this comment, this is a **work in progress**.
//!
//! Search at the time of writing is implemented using a simple
//! boyer-moore-horspool algorithm. The suboptimal part of the implementation
//! is that we need to encode each terminal page into a text buffer in order
//! to apply BMH to it. This is because the terminal page is not laid out
//! in a flat text form.
//!
//! To minimize memory usage, we use a sliding window to search for the
//! needle. The sliding window only keeps the minimum amount of page data
//! in memory to search for a needle (i.e. `needle.len - 1` bytes of overlap
//! between terminal pages).
//!
//! Future work:
//!
//! - PageListSearch on a PageList concurrently with another thread
//! - Handle pruned pages in a PageList to ensure we don't keep references
//! - Repeat search a changing active area of the screen
//! - Reverse search so that more recent matches are found first
//!
const std = @import("std");
const Allocator = std.mem.Allocator;
const assert = std.debug.assert;
const CircBuf = @import("../datastruct/main.zig").CircBuf;
const terminal = @import("main.zig");
const point = terminal.point;
const Page = terminal.Page;
const PageList = terminal.PageList;
const Pin = PageList.Pin;
const Selection = terminal.Selection;
const Screen = terminal.Screen;
const PageFormatter = @import("formatter.zig").PageFormatter;
pub const Active = @import("search/active.zig").ActiveSearch;
pub const PageList = @import("search/pagelist.zig").PageListSearch;
pub const Screen = @import("search/screen.zig").ScreenSearch;
pub const Thread = @import("search/Thread.zig");
/// Searches for a term in a PageList structure.
///
/// At the time of writing, this does not support searching a pagelist
/// simultaneously as its being used by another thread. This will be resolved
/// in the future.
pub const PageListSearch = struct {
/// The list we're searching.
list: *PageList,
test {
@import("std").testing.refAllDecls(@This());
/// The sliding window of page contents and nodes to search.
window: SlidingWindow,
/// Initialize the page list search.
///
/// The needle is not copied and must be kept alive for the duration
/// of the search operation.
pub fn init(
alloc: Allocator,
list: *PageList,
needle: []const u8,
) Allocator.Error!PageListSearch {
var window = try SlidingWindow.init(alloc, needle);
errdefer window.deinit();
return .{
.list = list,
.window = window,
};
}
pub fn deinit(self: *PageListSearch) void {
self.window.deinit();
}
/// Find the next match for the needle in the pagelist. This returns
/// null when there are no more matches.
pub fn next(self: *PageListSearch) Allocator.Error!?Selection {
// Try to search for the needle in the window. If we find a match
// then we can return that and we're done.
if (self.window.next()) |sel| return sel;
// Get our next node. If we have a value in our window then we
// can determine the next node. If we don't, we've never setup the
// window so we use our first node.
var node_: ?*PageList.List.Node = if (self.window.meta.last()) |meta|
meta.node.next
else
self.list.pages.first;
// Add one pagelist node at a time, look for matches, and repeat
// until we find a match or we reach the end of the pagelist.
// This append then next pattern limits memory usage of the window.
while (node_) |node| : (node_ = node.next) {
try self.window.append(node);
if (self.window.next()) |sel| return sel;
}
// We've reached the end of the pagelist, no matches.
return null;
}
};
/// Searches page nodes via a sliding window. The sliding window maintains
/// the invariant that data isn't pruned until (1) we've searched it and
/// (2) we've accounted for overlaps across pages to fit the needle.
///
/// The sliding window is first initialized empty. Pages are then appended
/// in the order to search them. If you're doing a reverse search then the
/// pages should be appended in reverse order and the needle should be
/// reversed.
///
/// All appends grow the window. The window is only pruned when a searc
/// is done (positive or negative match) via `next()`.
///
/// To avoid unnecessary memory growth, the recommended usage is to
/// call `next()` until it returns null and then `append` the next page
/// and repeat the process. This will always maintain the minimum
/// required memory to search for the needle.
const SlidingWindow = struct {
/// The allocator to use for all the data within this window. We
/// store this rather than passing it around because its already
/// part of multiple elements (eg. Meta's CellMap) and we want to
/// ensure we always use a consistent allocator. Additionally, only
/// a small amount of sliding windows are expected to be in use
/// at any one time so the memory overhead isn't that large.
alloc: Allocator,
/// The data buffer is a circular buffer of u8 that contains the
/// encoded page text that we can use to search for the needle.
data: DataBuf,
/// The meta buffer is a circular buffer that contains the metadata
/// about the pages we're searching. This usually isn't that large
/// so callers must iterate through it to find the offset to map
/// data to meta.
meta: MetaBuf,
/// Offset into data for our current state. This handles the
/// situation where our search moved through meta[0] but didn't
/// do enough to prune it.
data_offset: usize = 0,
/// The needle we're searching for. Does not own the memory.
needle: []const u8,
/// A buffer to store the overlap search data. This is used to search
/// overlaps between pages where the match starts on one page and
/// ends on another. The length is always `needle.len * 2`.
overlap_buf: []u8,
const DataBuf = CircBuf(u8, 0);
const MetaBuf = CircBuf(Meta, undefined);
const Meta = struct {
node: *PageList.List.Node,
cell_map: std.ArrayList(point.Coordinate),
pub fn deinit(self: *Meta, alloc: Allocator) void {
self.cell_map.deinit(alloc);
}
};
pub fn init(
alloc: Allocator,
needle: []const u8,
) Allocator.Error!SlidingWindow {
var data = try DataBuf.init(alloc, 0);
errdefer data.deinit(alloc);
var meta = try MetaBuf.init(alloc, 0);
errdefer meta.deinit(alloc);
const overlap_buf = try alloc.alloc(u8, needle.len * 2);
errdefer alloc.free(overlap_buf);
return .{
.alloc = alloc,
.data = data,
.meta = meta,
.needle = needle,
.overlap_buf = overlap_buf,
};
}
pub fn deinit(self: *SlidingWindow) void {
self.alloc.free(self.overlap_buf);
self.data.deinit(self.alloc);
var meta_it = self.meta.iterator(.forward);
while (meta_it.next()) |meta| meta.deinit(self.alloc);
self.meta.deinit(self.alloc);
}
/// Clear all data but retain allocated capacity.
pub fn clearAndRetainCapacity(self: *SlidingWindow) void {
var meta_it = self.meta.iterator(.forward);
while (meta_it.next()) |meta| meta.deinit(self.alloc);
self.meta.clear();
self.data.clear();
self.data_offset = 0;
}
/// Search the window for the next occurrence of the needle. As
/// the window moves, the window will prune itself while maintaining
/// the invariant that the window is always big enough to contain
/// the needle.
pub fn next(self: *SlidingWindow) ?Selection {
const slices = slices: {
// If we have less data then the needle then we can't possibly match
const data_len = self.data.len();
if (data_len < self.needle.len) return null;
break :slices self.data.getPtrSlice(
self.data_offset,
data_len - self.data_offset,
);
};
// Search the first slice for the needle.
if (std.mem.indexOf(u8, slices[0], self.needle)) |idx| {
return self.selection(
idx,
self.needle.len,
);
}
// Search the overlap buffer for the needle.
if (slices[0].len > 0 and slices[1].len > 0) overlap: {
// Get up to needle.len - 1 bytes from each side (as much as
// we can) and store it in the overlap buffer.
const prefix: []const u8 = prefix: {
const len = @min(slices[0].len, self.needle.len - 1);
const idx = slices[0].len - len;
break :prefix slices[0][idx..];
};
const suffix: []const u8 = suffix: {
const len = @min(slices[1].len, self.needle.len - 1);
break :suffix slices[1][0..len];
};
const overlap_len = prefix.len + suffix.len;
assert(overlap_len <= self.overlap_buf.len);
@memcpy(self.overlap_buf[0..prefix.len], prefix);
@memcpy(self.overlap_buf[prefix.len..overlap_len], suffix);
// Search the overlap
const idx = std.mem.indexOf(
u8,
self.overlap_buf[0..overlap_len],
self.needle,
) orelse break :overlap;
// We found a match in the overlap buffer. We need to map the
// index back to the data buffer in order to get our selection.
return self.selection(
slices[0].len - prefix.len + idx,
self.needle.len,
);
}
// Search the last slice for the needle.
if (std.mem.indexOf(u8, slices[1], self.needle)) |idx| {
return self.selection(
slices[0].len + idx,
self.needle.len,
);
}
// No match. We keep `needle.len - 1` bytes available to
// handle the future overlap case.
var meta_it = self.meta.iterator(.reverse);
prune: {
var saved: usize = 0;
while (meta_it.next()) |meta| {
const needed = self.needle.len - 1 - saved;
if (meta.cell_map.items.len >= needed) {
// We save up to this meta. We set our data offset
// to exactly where it needs to be to continue
// searching.
self.data_offset = meta.cell_map.items.len - needed;
break;
}
saved += meta.cell_map.items.len;
} else {
// If we exited the while loop naturally then we
// never got the amount we needed and so there is
// nothing to prune.
assert(saved < self.needle.len - 1);
break :prune;
}
const prune_count = self.meta.len() - meta_it.idx;
if (prune_count == 0) {
// This can happen if we need to save up to the first
// meta value to retain our window.
break :prune;
}
// We can now delete all the metas up to but NOT including
// the meta we found through meta_it.
meta_it = self.meta.iterator(.forward);
var prune_data_len: usize = 0;
for (0..prune_count) |_| {
const meta = meta_it.next().?;
prune_data_len += meta.cell_map.items.len;
meta.deinit(self.alloc);
}
self.meta.deleteOldest(prune_count);
self.data.deleteOldest(prune_data_len);
}
// Our data offset now moves to needle.len - 1 from the end so
// that we can handle the overlap case.
self.data_offset = self.data.len() - self.needle.len + 1;
self.assertIntegrity();
return null;
}
/// Return a selection for the given start and length into the data
/// buffer and also prune the data/meta buffers if possible up to
/// this start index.
///
/// The start index is assumed to be relative to the offset. i.e.
/// index zero is actually at `self.data[self.data_offset]`. The
/// selection will account for the offset.
fn selection(
self: *SlidingWindow,
start_offset: usize,
len: usize,
) Selection {
const start = start_offset + self.data_offset;
assert(start < self.data.len());
assert(start + len <= self.data.len());
// meta_consumed is the number of bytes we've consumed in the
// data buffer up to and NOT including the meta where we've
// found our pin. This is important because it tells us the
// amount of data we can safely deleted from self.data since
// we can't partially delete a meta block's data. (The partial
// amount is represented by self.data_offset).
var meta_it = self.meta.iterator(.forward);
var meta_consumed: usize = 0;
const tl: Pin = pin(&meta_it, &meta_consumed, start);
// Store the information required to prune later. We store this
// now because we only want to prune up to our START so we can
// find overlapping matches.
const tl_meta_idx = meta_it.idx - 1;
const tl_meta_consumed = meta_consumed;
// We have to seek back so that we reinspect our current
// iterator value again in case the start and end are in the
// same segment.
meta_it.seekBy(-1);
const br: Pin = pin(&meta_it, &meta_consumed, start + len - 1);
assert(meta_it.idx >= 1);
// Our offset into the current meta block is the start index
// minus the amount of data fully consumed. We then add one
// to move one past the match so we don't repeat it.
self.data_offset = start - tl_meta_consumed + 1;
// meta_it.idx is br's meta index plus one (because the iterator
// moves one past the end; we call next() one last time). So
// we compare against one to check that the meta that we matched
// in has prior meta blocks we can prune.
if (tl_meta_idx > 0) {
// Deinit all our memory in the meta blocks prior to our
// match.
const meta_count = tl_meta_idx;
meta_it.reset();
for (0..meta_count) |_| meta_it.next().?.deinit(self.alloc);
if (comptime std.debug.runtime_safety) {
assert(meta_it.idx == meta_count);
assert(meta_it.next().?.node == tl.node);
}
self.meta.deleteOldest(meta_count);
// Delete all the data up to our current index.
assert(tl_meta_consumed > 0);
self.data.deleteOldest(tl_meta_consumed);
}
self.assertIntegrity();
return .init(tl, br, false);
}
/// Convert a data index into a pin.
///
/// The iterator and offset are both expected to be passed by
/// pointer so that the pin can be efficiently called for multiple
/// indexes (in order). See selection() for an example.
///
/// Precondition: the index must be within the data buffer.
fn pin(
it: *MetaBuf.Iterator,
offset: *usize,
idx: usize,
) Pin {
while (it.next()) |meta| {
// meta_i is the index we expect to find the match in the
// cell map within this meta if it contains it.
const meta_i = idx - offset.*;
if (meta_i >= meta.cell_map.items.len) {
// This meta doesn't contain the match. This means we
// can also prune this set of data because we only look
// forward.
offset.* += meta.cell_map.items.len;
continue;
}
// We found the meta that contains the start of the match.
const map = meta.cell_map.items[meta_i];
return .{
.node = meta.node,
.y = @intCast(map.y),
.x = map.x,
};
}
// Unreachable because it is a precondition that the index is
// within the data buffer.
unreachable;
}
/// Add a new node to the sliding window. This will always grow
/// the sliding window; data isn't pruned until it is consumed
/// via a search (via next()).
pub fn append(
self: *SlidingWindow,
node: *PageList.List.Node,
) Allocator.Error!void {
// Initialize our metadata for the node.
var meta: Meta = .{
.node = node,
.cell_map = .empty,
};
errdefer meta.deinit(self.alloc);
// This is suboptimal but we need to encode the page once to
// temporary memory, and then copy it into our circular buffer.
// In the future, we should benchmark and see if we can encode
// directly into the circular buffer.
var encoded: std.Io.Writer.Allocating = .init(self.alloc);
defer encoded.deinit();
// Encode the page into the buffer.
const formatter: PageFormatter = formatter: {
var formatter: PageFormatter = .init(&meta.node.data, .plain);
formatter.point_map = .{
.alloc = self.alloc,
.map = &meta.cell_map,
};
break :formatter formatter;
};
formatter.format(&encoded.writer) catch {
// writer uses anyerror but the only realistic error on
// an ArrayList is out of memory.
return error.OutOfMemory;
};
assert(meta.cell_map.items.len == encoded.written().len);
// Ensure our buffers are big enough to store what we need.
try self.data.ensureUnusedCapacity(self.alloc, encoded.written().len);
try self.meta.ensureUnusedCapacity(self.alloc, 1);
// Append our new node to the circular buffer.
try self.data.appendSlice(encoded.written());
try self.meta.append(meta);
self.assertIntegrity();
}
fn assertIntegrity(self: *const SlidingWindow) void {
if (comptime !std.debug.runtime_safety) return;
// We don't run integrity checks on Valgrind because its soooooo slow,
// Valgrind is our integrity checker, and we run these during unit
// tests (non-Valgrind) anyways so we're verifying anyways.
if (std.valgrind.runningOnValgrind() > 0) return;
// Integrity check: verify our data matches our metadata exactly.
var meta_it = self.meta.iterator(.forward);
var data_len: usize = 0;
while (meta_it.next()) |m| data_len += m.cell_map.items.len;
assert(data_len == self.data.len());
// Integrity check: verify our data offset is within bounds.
assert(self.data_offset < self.data.len());
}
};
test "PageListSearch single page" {
const testing = std.testing;
const alloc = testing.allocator;
var s = try Screen.init(alloc, 80, 24, 0);
defer s.deinit();
try s.testWriteString("hello. boo! hello. boo!");
try testing.expect(s.pages.pages.first == s.pages.pages.last);
var search = try PageListSearch.init(alloc, &s.pages, "boo!");
defer search.deinit();
// We should be able to find two matches.
{
const sel = (try search.next()).?;
try testing.expectEqual(point.Point{ .active = .{
.x = 7,
.y = 0,
} }, s.pages.pointFromPin(.active, sel.start()).?);
try testing.expectEqual(point.Point{ .active = .{
.x = 10,
.y = 0,
} }, s.pages.pointFromPin(.active, sel.end()).?);
}
{
const sel = (try search.next()).?;
try testing.expectEqual(point.Point{ .active = .{
.x = 19,
.y = 0,
} }, s.pages.pointFromPin(.active, sel.start()).?);
try testing.expectEqual(point.Point{ .active = .{
.x = 22,
.y = 0,
} }, s.pages.pointFromPin(.active, sel.end()).?);
}
try testing.expect((try search.next()) == null);
try testing.expect((try search.next()) == null);
}
test "SlidingWindow empty on init" {
const testing = std.testing;
const alloc = testing.allocator;
var w = try SlidingWindow.init(alloc, "boo!");
defer w.deinit();
try testing.expectEqual(0, w.data.len());
try testing.expectEqual(0, w.meta.len());
}
test "SlidingWindow single append" {
const testing = std.testing;
const alloc = testing.allocator;
var w = try SlidingWindow.init(alloc, "boo!");
defer w.deinit();
var s = try Screen.init(alloc, 80, 24, 0);
defer s.deinit();
try s.testWriteString("hello. boo! hello. boo!");
// We want to test single-page cases.
try testing.expect(s.pages.pages.first == s.pages.pages.last);
const node: *PageList.List.Node = s.pages.pages.first.?;
try w.append(node);
// We should be able to find two matches.
{
const sel = w.next().?;
try testing.expectEqual(point.Point{ .active = .{
.x = 7,
.y = 0,
} }, s.pages.pointFromPin(.active, sel.start()).?);
try testing.expectEqual(point.Point{ .active = .{
.x = 10,
.y = 0,
} }, s.pages.pointFromPin(.active, sel.end()).?);
}
{
const sel = w.next().?;
try testing.expectEqual(point.Point{ .active = .{
.x = 19,
.y = 0,
} }, s.pages.pointFromPin(.active, sel.start()).?);
try testing.expectEqual(point.Point{ .active = .{
.x = 22,
.y = 0,
} }, s.pages.pointFromPin(.active, sel.end()).?);
}
try testing.expect(w.next() == null);
try testing.expect(w.next() == null);
}
test "SlidingWindow single append no match" {
const testing = std.testing;
const alloc = testing.allocator;
var w = try SlidingWindow.init(alloc, "nope!");
defer w.deinit();
var s = try Screen.init(alloc, 80, 24, 0);
defer s.deinit();
try s.testWriteString("hello. boo! hello. boo!");
// We want to test single-page cases.
try testing.expect(s.pages.pages.first == s.pages.pages.last);
const node: *PageList.List.Node = s.pages.pages.first.?;
try w.append(node);
// No matches
try testing.expect(w.next() == null);
try testing.expect(w.next() == null);
// Should still keep the page
try testing.expectEqual(1, w.meta.len());
}
test "SlidingWindow two pages" {
const testing = std.testing;
const alloc = testing.allocator;
var w = try SlidingWindow.init(alloc, "boo!");
defer w.deinit();
var s = try Screen.init(alloc, 80, 24, 1000);
defer s.deinit();
// Fill up the first page. The final bytes in the first page
// are "boo!"
const first_page_rows = s.pages.pages.first.?.data.capacity.rows;
for (0..first_page_rows - 1) |_| try s.testWriteString("\n");
for (0..s.pages.cols - 4) |_| try s.testWriteString("x");
try s.testWriteString("boo!");
try testing.expect(s.pages.pages.first == s.pages.pages.last);
try s.testWriteString("\n");
try testing.expect(s.pages.pages.first != s.pages.pages.last);
try s.testWriteString("hello. boo!");
// Add both pages
const node: *PageList.List.Node = s.pages.pages.first.?;
try w.append(node);
try w.append(node.next.?);
// Search should find two matches
{
const sel = w.next().?;
try testing.expectEqual(point.Point{ .active = .{
.x = 76,
.y = 22,
} }, s.pages.pointFromPin(.active, sel.start()).?);
try testing.expectEqual(point.Point{ .active = .{
.x = 79,
.y = 22,
} }, s.pages.pointFromPin(.active, sel.end()).?);
}
{
const sel = w.next().?;
try testing.expectEqual(point.Point{ .active = .{
.x = 7,
.y = 23,
} }, s.pages.pointFromPin(.active, sel.start()).?);
try testing.expectEqual(point.Point{ .active = .{
.x = 10,
.y = 23,
} }, s.pages.pointFromPin(.active, sel.end()).?);
}
try testing.expect(w.next() == null);
try testing.expect(w.next() == null);
}
test "SlidingWindow two pages match across boundary" {
const testing = std.testing;
const alloc = testing.allocator;
var w = try SlidingWindow.init(alloc, "hello, world");
defer w.deinit();
var s = try Screen.init(alloc, 80, 24, 1000);
defer s.deinit();
// Fill up the first page. The final bytes in the first page
// are "boo!"
const first_page_rows = s.pages.pages.first.?.data.capacity.rows;
for (0..first_page_rows - 1) |_| try s.testWriteString("\n");
for (0..s.pages.cols - 4) |_| try s.testWriteString("x");
try s.testWriteString("hell");
try testing.expect(s.pages.pages.first == s.pages.pages.last);
try s.testWriteString("o, world!");
try testing.expect(s.pages.pages.first != s.pages.pages.last);
// Add both pages
const node: *PageList.List.Node = s.pages.pages.first.?;
try w.append(node);
try w.append(node.next.?);
// Search should find a match
{
const sel = w.next().?;
try testing.expectEqual(point.Point{ .active = .{
.x = 76,
.y = 22,
} }, s.pages.pointFromPin(.active, sel.start()).?);
try testing.expectEqual(point.Point{ .active = .{
.x = 7,
.y = 23,
} }, s.pages.pointFromPin(.active, sel.end()).?);
}
try testing.expect(w.next() == null);
try testing.expect(w.next() == null);
// We shouldn't prune because we don't have enough space
try testing.expectEqual(2, w.meta.len());
}
test "SlidingWindow two pages no match prunes first page" {
const testing = std.testing;
const alloc = testing.allocator;
var w = try SlidingWindow.init(alloc, "nope!");
defer w.deinit();
var s = try Screen.init(alloc, 80, 24, 1000);
defer s.deinit();
// Fill up the first page. The final bytes in the first page
// are "boo!"
const first_page_rows = s.pages.pages.first.?.data.capacity.rows;
for (0..first_page_rows - 1) |_| try s.testWriteString("\n");
for (0..s.pages.cols - 4) |_| try s.testWriteString("x");
try s.testWriteString("boo!");
try testing.expect(s.pages.pages.first == s.pages.pages.last);
try s.testWriteString("\n");
try testing.expect(s.pages.pages.first != s.pages.pages.last);
try s.testWriteString("hello. boo!");
// Add both pages
const node: *PageList.List.Node = s.pages.pages.first.?;
try w.append(node);
try w.append(node.next.?);
// Search should find nothing
try testing.expect(w.next() == null);
try testing.expect(w.next() == null);
// We should've pruned our page because the second page
// has enough text to contain our needle.
try testing.expectEqual(1, w.meta.len());
}
test "SlidingWindow two pages no match keeps both pages" {
const testing = std.testing;
const alloc = testing.allocator;
var s = try Screen.init(alloc, 80, 24, 1000);
defer s.deinit();
// Fill up the first page. The final bytes in the first page
// are "boo!"
const first_page_rows = s.pages.pages.first.?.data.capacity.rows;
for (0..first_page_rows - 1) |_| try s.testWriteString("\n");
for (0..s.pages.cols - 4) |_| try s.testWriteString("x");
try s.testWriteString("boo!");
try testing.expect(s.pages.pages.first == s.pages.pages.last);
try s.testWriteString("\n");
try testing.expect(s.pages.pages.first != s.pages.pages.last);
try s.testWriteString("hello. boo!");
// Imaginary needle for search. Doesn't match!
var needle_list: std.ArrayList(u8) = .empty;
defer needle_list.deinit(alloc);
try needle_list.appendNTimes(alloc, 'x', first_page_rows * s.pages.cols);
const needle: []const u8 = needle_list.items;
var w = try SlidingWindow.init(alloc, needle);
defer w.deinit();
// Add both pages
const node: *PageList.List.Node = s.pages.pages.first.?;
try w.append(node);
try w.append(node.next.?);
// Search should find nothing
try testing.expect(w.next() == null);
try testing.expect(w.next() == null);
// No pruning because both pages are needed to fit needle.
try testing.expectEqual(2, w.meta.len());
}
test "SlidingWindow single append across circular buffer boundary" {
const testing = std.testing;
const alloc = testing.allocator;
var w = try SlidingWindow.init(alloc, "abc");
defer w.deinit();
var s = try Screen.init(alloc, 80, 24, 0);
defer s.deinit();
try s.testWriteString("XXXXXXXXXXXXXXXXXXXboo!XXXXX");
// We are trying to break a circular buffer boundary so the way we
// do this is to duplicate the data then do a failing search. This
// will cause the first page to be pruned. The next time we append we'll
// put it in the middle of the circ buffer. We assert this so that if
// our implementation changes our test will fail.
try testing.expect(s.pages.pages.first == s.pages.pages.last);
const node: *PageList.List.Node = s.pages.pages.first.?;
try w.append(node);
try w.append(node);
{
// No wrap around yet
const slices = w.data.getPtrSlice(0, w.data.len());
try testing.expect(slices[0].len > 0);
try testing.expect(slices[1].len == 0);
}
// Search non-match, prunes page
try testing.expect(w.next() == null);
try testing.expectEqual(1, w.meta.len());
// Change the needle, just needs to be the same length (not a real API)
w.needle = "boo";
// Add new page, now wraps
try w.append(node);
{
const slices = w.data.getPtrSlice(0, w.data.len());
try testing.expect(slices[0].len > 0);
try testing.expect(slices[1].len > 0);
}
{
const sel = w.next().?;
try testing.expectEqual(point.Point{ .active = .{
.x = 19,
.y = 0,
} }, s.pages.pointFromPin(.active, sel.start()).?);
try testing.expectEqual(point.Point{ .active = .{
.x = 21,
.y = 0,
} }, s.pages.pointFromPin(.active, sel.end()).?);
}
try testing.expect(w.next() == null);
}
test "SlidingWindow single append match on boundary" {
const testing = std.testing;
const alloc = testing.allocator;
var w = try SlidingWindow.init(alloc, "abcd");
defer w.deinit();
var s = try Screen.init(alloc, 80, 24, 0);
defer s.deinit();
try s.testWriteString("o!XXXXXXXXXXXXXXXXXXXbo");
// We are trying to break a circular buffer boundary so the way we
// do this is to duplicate the data then do a failing search. This
// will cause the first page to be pruned. The next time we append we'll
// put it in the middle of the circ buffer. We assert this so that if
// our implementation changes our test will fail.
try testing.expect(s.pages.pages.first == s.pages.pages.last);
const node: *PageList.List.Node = s.pages.pages.first.?;
try w.append(node);
try w.append(node);
{
// No wrap around yet
const slices = w.data.getPtrSlice(0, w.data.len());
try testing.expect(slices[0].len > 0);
try testing.expect(slices[1].len == 0);
}
// Search non-match, prunes page
try testing.expect(w.next() == null);
try testing.expectEqual(1, w.meta.len());
// Change the needle, just needs to be the same length (not a real API)
w.needle = "boo!";
// Add new page, now wraps
try w.append(node);
{
const slices = w.data.getPtrSlice(0, w.data.len());
try testing.expect(slices[0].len > 0);
try testing.expect(slices[1].len > 0);
}
{
const sel = w.next().?;
try testing.expectEqual(point.Point{ .active = .{
.x = 21,
.y = 0,
} }, s.pages.pointFromPin(.active, sel.start()).?);
try testing.expectEqual(point.Point{ .active = .{
.x = 1,
.y = 0,
} }, s.pages.pointFromPin(.active, sel.end()).?);
}
try testing.expect(w.next() == null);
// Non-public APIs
_ = @import("search/sliding_window.zig");
}

View File

@ -0,0 +1,63 @@
//! Search thread that handles searching a terminal for a string match.
//! This is expected to run on a dedicated thread to try to prevent too much
//! overhead to other terminal read/write operations.
//!
//! The current architecture of search does acquire global locks for accessing
//! terminal data, so there's still added contention, but we do our best to
//! minimize this by trading off memory usage (copying data to minimize lock
//! time).
pub const Thread = @This();
const std = @import("std");
const Allocator = std.mem.Allocator;
const BlockingQueue = @import("../../datastruct/main.zig").BlockingQueue;
const log = std.log.scoped(.search_thread);
/// Allocator used for some state
alloc: std.mem.Allocator,
/// The mailbox that can be used to send this thread messages. Note
/// this is a blocking queue so if it is full you will get errors (or block).
mailbox: *Mailbox,
/// Initialize the thread. This does not START the thread. This only sets
/// up all the internal state necessary prior to starting the thread. It
/// is up to the caller to start the thread with the threadMain entrypoint.
pub fn init(alloc: Allocator) Thread {
// The mailbox for messaging this thread
var mailbox = try Mailbox.create(alloc);
errdefer mailbox.destroy(alloc);
return .{
.alloc = alloc,
.mailbox = mailbox,
};
}
/// Clean up the thread. This is only safe to call once the thread
/// completes executing; the caller must join prior to this.
pub fn deinit(self: *Thread) void {
// Nothing can possibly access the mailbox anymore, destroy it.
self.mailbox.destroy(self.alloc);
}
/// The main entrypoint for the thread.
pub fn threadMain(self: *Thread) void {
// Call child function so we can use errors...
self.threadMain_() catch |err| {
// In the future, we should expose this on the thread struct.
log.warn("search thread err={}", .{err});
};
}
fn threadMain_(self: *Thread) !void {
defer log.debug("search thread exited", .{});
_ = self;
}
/// The type used for sending messages to the thread.
pub const Mailbox = BlockingQueue(Message, 64);
/// The messages that can be sent to the thread.
pub const Message = union(enum) {};

View File

@ -0,0 +1,172 @@
const std = @import("std");
const testing = std.testing;
const Allocator = std.mem.Allocator;
const point = @import("../point.zig");
const size = @import("../size.zig");
const PageList = @import("../PageList.zig");
const Selection = @import("../Selection.zig");
const SlidingWindow = @import("sliding_window.zig").SlidingWindow;
const Terminal = @import("../Terminal.zig");
/// Searches for a substring within the active area of a PageList.
///
/// The distinction for "active area" is important because it is the
/// only part of a PageList that is mutable. Therefore, its the only part
/// of the terminal that needs to be repeatedly searched as the contents
/// change.
///
/// This struct specializes in searching only within that active area,
/// and handling the active area moving as new lines are added to the bottom.
pub const ActiveSearch = struct {
window: SlidingWindow,
pub fn init(
alloc: Allocator,
needle: []const u8,
) Allocator.Error!ActiveSearch {
// We just do a forward search since the active area is usually
// pretty small so search results are instant anyways. This avoids
// a small amount of work to reverse things.
var window: SlidingWindow = try .init(alloc, .forward, needle);
errdefer window.deinit();
return .{ .window = window };
}
pub fn deinit(self: *ActiveSearch) void {
self.window.deinit();
}
/// Update the active area to reflect the current state of the PageList.
///
/// This doesn't do the search, it only copies the necessary data
/// to perform the search later. This lets the caller hold the lock
/// on the PageList for a minimal amount of time.
///
/// This returns the first page (in reverse order) covered by this
/// search. This allows the history search to overlap and search history.
/// There CAN BE duplicates, and this page CAN BE mutable, so the history
/// search results should prune anything that's in the active area.
///
/// If the return value is null it means the active area covers the entire
/// PageList, currently.
pub fn update(
self: *ActiveSearch,
list: *const PageList,
) Allocator.Error!?*PageList.List.Node {
// Clear our previous sliding window
self.window.clearAndRetainCapacity();
// First up, add enough pages to cover the active area.
var rem: usize = list.rows;
var node_ = list.pages.last;
var last_node: ?*PageList.List.Node = null;
while (node_) |node| : (node_ = node.prev) {
_ = try self.window.append(node);
last_node = node;
// If we reached our target amount, then this is the last
// page that contains the active area. We go to the previous
// page once more since its the first page of our required
// overlap.
if (rem <= node.data.size.rows) {
node_ = node.prev;
break;
}
rem -= node.data.size.rows;
}
// Next, add enough overlap to cover needle.len - 1 bytes (if it
// exists) so we can cover the overlap.
while (node_) |node| : (node_ = node.prev) {
// If the last row of this node isn't wrapped we can't overlap.
const row = node.data.getRow(node.data.size.rows - 1);
if (!row.wrap) break;
// We could be more accurate here and count bytes since the
// last wrap but its complicated and unlikely multiple pages
// wrap so this should be fine.
const added = try self.window.append(node);
if (added >= self.window.needle.len - 1) break;
}
// Return the last node we added to our window.
return last_node;
}
/// Find the next match for the needle in the active area. This returns
/// null when there are no more matches.
pub fn next(self: *ActiveSearch) ?Selection {
return self.window.next();
}
};
test "simple search" {
const alloc = testing.allocator;
var t: Terminal = try .init(alloc, .{ .cols = 10, .rows = 10 });
defer t.deinit(alloc);
var s = t.vtStream();
defer s.deinit();
try s.nextSlice("Fizz\r\nBuzz\r\nFizz\r\nBang");
var search: ActiveSearch = try .init(alloc, "Fizz");
defer search.deinit();
_ = try search.update(&t.screen.pages);
{
const sel = search.next().?;
try testing.expectEqual(point.Point{ .active = .{
.x = 0,
.y = 0,
} }, t.screen.pages.pointFromPin(.active, sel.start()).?);
try testing.expectEqual(point.Point{ .active = .{
.x = 3,
.y = 0,
} }, t.screen.pages.pointFromPin(.active, sel.end()).?);
}
{
const sel = search.next().?;
try testing.expectEqual(point.Point{ .active = .{
.x = 0,
.y = 2,
} }, t.screen.pages.pointFromPin(.active, sel.start()).?);
try testing.expectEqual(point.Point{ .active = .{
.x = 3,
.y = 2,
} }, t.screen.pages.pointFromPin(.active, sel.end()).?);
}
try testing.expect(search.next() == null);
}
test "clear screen and search" {
const alloc = testing.allocator;
var t: Terminal = try .init(alloc, .{ .cols = 10, .rows = 10 });
defer t.deinit(alloc);
var s = t.vtStream();
defer s.deinit();
try s.nextSlice("Fizz\r\nBuzz\r\nFizz\r\nBang");
var search: ActiveSearch = try .init(alloc, "Fizz");
defer search.deinit();
_ = try search.update(&t.screen.pages);
try s.nextSlice("\x1b[2J"); // Clear screen
try s.nextSlice("\x1b[H"); // Move cursor home
try s.nextSlice("Buzz\r\nFizz\r\nBuzz");
_ = try search.update(&t.screen.pages);
{
const sel = search.next().?;
try testing.expectEqual(point.Point{ .active = .{
.x = 0,
.y = 1,
} }, t.screen.pages.pointFromPin(.active, sel.start()).?);
try testing.expectEqual(point.Point{ .active = .{
.x = 3,
.y = 1,
} }, t.screen.pages.pointFromPin(.active, sel.end()).?);
}
try testing.expect(search.next() == null);
}

View File

@ -0,0 +1,390 @@
const std = @import("std");
const Allocator = std.mem.Allocator;
const assert = std.debug.assert;
const testing = std.testing;
const CircBuf = @import("../../datastruct/main.zig").CircBuf;
const terminal = @import("../main.zig");
const point = terminal.point;
const Page = terminal.Page;
const PageList = terminal.PageList;
const Pin = PageList.Pin;
const Selection = terminal.Selection;
const Screen = terminal.Screen;
const PageFormatter = @import("../formatter.zig").PageFormatter;
const Terminal = @import("../Terminal.zig");
const SlidingWindow = @import("sliding_window.zig").SlidingWindow;
/// Searches for a term in a PageList structure.
///
/// This searches in reverse order starting from the given node.
///
/// This assumes that nodes do not change contents. For nodes that change
/// contents, look at ActiveSearch, which is designed to re-search the active
/// area since it assumed to change. When integrating ActiveSearch with
/// PageListSearch, the caller should start the PageListSearch from the
/// returned node from ActiveSearch.update().
///
/// Concurrent access to a PageList or nodes in a PageList are not allowed,
/// so the caller should ensure that necessary locks are held. Each function
/// documents whether it accesses the PageList or not. For example, you can
/// safely call `next()` without holding a lock, but you must hold a lock
/// while calling `feed()`.
pub const PageListSearch = struct {
/// The list we're searching.
list: *PageList,
/// The sliding window of page contents and nodes to search.
window: SlidingWindow,
/// The tracked pin for our current position in the pagelist. This
/// will always point to the CURRENT node we're searching from so that
/// we can track if we move.
pin: *Pin,
/// Initialize the page list search. The needle is copied so it can
/// be freed immediately.
///
/// Accesses the PageList/Node so the caller must ensure it is safe
/// to do so if there is any concurrent access.
pub fn init(
alloc: Allocator,
needle: []const u8,
list: *PageList,
start: *PageList.List.Node,
) Allocator.Error!PageListSearch {
// We put a tracked pin into the node that we're starting from.
// By using a tracked pin, we can keep our pagelist references safe
// because if the pagelist prunes pages, the tracked pin will
// be moved somewhere safe.
const pin = try list.trackPin(.{
.node = start,
.y = start.data.size.rows - 1,
.x = start.data.size.cols - 1,
});
errdefer list.untrackPin(pin);
// Create our sliding window we'll use for searching.
var window: SlidingWindow = try .init(alloc, .reverse, needle);
errdefer window.deinit();
// We always feed our initial page data into the window, because
// we have the lock anyways and this lets our `pin` point to our
// current node and feed to work properly.
_ = try window.append(start);
return .{
.list = list,
.window = window,
.pin = pin,
};
}
/// Modifies the PageList (to untrack a pin) so the caller must ensure
/// that it is safe to do so.
pub fn deinit(self: *PageListSearch) void {
self.window.deinit();
self.list.untrackPin(self.pin);
}
/// Return the next match in the loaded page nodes. If this returns
/// null then the PageList search needs to be fed the next node(s).
/// Call, `feed` to do this.
///
/// Beware that the selection returned may point to a node that
/// is freed if the caller does not hold necessary locks on the
/// PageList while searching. The pins should be validated prior to
/// final use.
///
/// This does NOT access the PageList, so it can be called without
/// a lock held.
pub fn next(self: *PageListSearch) ?Selection {
return self.window.next();
}
/// Feed more data to the sliding window from the pagelist. This will
/// feed enough data to cover at least one match (needle length) if it
/// exists; this doesn't perform the search, it only feeds data.
///
/// This accesses nodes in the PageList, so the caller must ensure
/// it is safe to do so (i.e. hold necessary locks).
///
/// This returns false if there is no more data to feed. This essentially
/// means we've searched the entire pagelist.
pub fn feed(self: *PageListSearch) Allocator.Error!bool {
// Add at least enough data to find a single match.
var rem = self.window.needle.len;
// Start at our previous node and then continue adding until we
// get our desired amount of data.
var node_: ?*PageList.List.Node = self.pin.node.prev;
while (node_) |node| : (node_ = node.prev) {
rem -|= try self.window.append(node);
// Move our tracked pin to the new node.
self.pin.node = node;
if (rem == 0) break;
}
// True if we fed any data.
return rem < self.window.needle.len;
}
};
test "simple search" {
const alloc = testing.allocator;
var t: Terminal = try .init(alloc, .{ .cols = 10, .rows = 10 });
defer t.deinit(alloc);
var s = t.vtStream();
defer s.deinit();
try s.nextSlice("Fizz\r\nBuzz\r\nFizz\r\nBang");
var search: PageListSearch = try .init(
alloc,
"Fizz",
&t.screen.pages,
t.screen.pages.pages.last.?,
);
defer search.deinit();
{
const sel = search.next().?;
try testing.expectEqual(point.Point{ .active = .{
.x = 0,
.y = 2,
} }, t.screen.pages.pointFromPin(.active, sel.start()).?);
try testing.expectEqual(point.Point{ .active = .{
.x = 3,
.y = 2,
} }, t.screen.pages.pointFromPin(.active, sel.end()).?);
}
{
const sel = search.next().?;
try testing.expectEqual(point.Point{ .active = .{
.x = 0,
.y = 0,
} }, t.screen.pages.pointFromPin(.active, sel.start()).?);
try testing.expectEqual(point.Point{ .active = .{
.x = 3,
.y = 0,
} }, t.screen.pages.pointFromPin(.active, sel.end()).?);
}
try testing.expect(search.next() == null);
// We should not be able to feed since we have one page
try testing.expect(!try search.feed());
}
test "feed multiple pages with matches" {
const alloc = testing.allocator;
var t: Terminal = try .init(alloc, .{ .cols = 10, .rows = 10 });
defer t.deinit(alloc);
var s = t.vtStream();
defer s.deinit();
// Fill up first page
const first_page_rows = t.screen.pages.pages.first.?.data.capacity.rows;
for (0..first_page_rows - 1) |_| try s.nextSlice("\r\n");
try s.nextSlice("Fizz");
try testing.expect(t.screen.pages.pages.first == t.screen.pages.pages.last);
// Create second page
try s.nextSlice("\r\n");
try testing.expect(t.screen.pages.pages.first != t.screen.pages.pages.last);
try s.nextSlice("Buzz\r\nFizz");
var search: PageListSearch = try .init(
alloc,
"Fizz",
&t.screen.pages,
t.screen.pages.pages.last.?,
);
defer search.deinit();
// First match on the last page
const sel1 = search.next();
try testing.expect(sel1 != null);
try testing.expect(search.next() == null);
// Feed should succeed and load the first page
try testing.expect(try search.feed());
// Now we should find the match on the first page
const sel2 = search.next();
try testing.expect(sel2 != null);
try testing.expect(search.next() == null);
// No more pages to feed
try testing.expect(!try search.feed());
}
test "feed multiple pages no matches" {
const alloc = testing.allocator;
var t: Terminal = try .init(alloc, .{ .cols = 10, .rows = 10 });
defer t.deinit(alloc);
var s = t.vtStream();
defer s.deinit();
// Fill up first page
const first_page_rows = t.screen.pages.pages.first.?.data.capacity.rows;
for (0..first_page_rows - 1) |_| try s.nextSlice("\r\n");
try s.nextSlice("Hello");
// Create second page
try s.nextSlice("\r\n");
try testing.expect(t.screen.pages.pages.first != t.screen.pages.pages.last);
try s.nextSlice("World");
var search: PageListSearch = try .init(
alloc,
"Nope",
&t.screen.pages,
t.screen.pages.pages.last.?,
);
defer search.deinit();
// No matches on last page
try testing.expect(search.next() == null);
// Feed first page
try testing.expect(try search.feed());
// Still no matches
try testing.expect(search.next() == null);
// No more pages
try testing.expect(!try search.feed());
}
test "feed iteratively through multiple matches" {
const alloc = testing.allocator;
var t: Terminal = try .init(alloc, .{ .cols = 80, .rows = 24 });
defer t.deinit(alloc);
var s = t.vtStream();
defer s.deinit();
const first_page_rows = t.screen.pages.pages.first.?.data.capacity.rows;
// Fill first page with a match at the end
for (0..first_page_rows - 1) |_| try s.nextSlice("\r\n");
try s.nextSlice("Page1Test");
try testing.expect(t.screen.pages.pages.first == t.screen.pages.pages.last);
// Create second page with a match
try s.nextSlice("\r\n");
try testing.expect(t.screen.pages.pages.first != t.screen.pages.pages.last);
try s.nextSlice("Page2Test");
var search: PageListSearch = try .init(
alloc,
"Test",
&t.screen.pages,
t.screen.pages.pages.last.?,
);
defer search.deinit();
// Match on page 2
try testing.expect(search.next() != null);
try testing.expect(search.next() == null);
// Feed page 1
try testing.expect(try search.feed());
try testing.expect(search.next() != null);
try testing.expect(search.next() == null);
// No more pages
try testing.expect(!try search.feed());
}
test "feed with match spanning page boundary" {
const alloc = testing.allocator;
var t: Terminal = try .init(alloc, .{ .cols = 80, .rows = 24 });
defer t.deinit(alloc);
var s = t.vtStream();
defer s.deinit();
const first_page_rows = t.screen.pages.pages.first.?.data.capacity.rows;
// Fill first page ending with "Te"
for (0..first_page_rows - 1) |_| try s.nextSlice("\r\n");
for (0..t.screen.pages.cols - 2) |_| try s.nextSlice("x");
try s.nextSlice("Te");
try testing.expect(t.screen.pages.pages.first == t.screen.pages.pages.last);
// Second page starts with "st"
try s.nextSlice("st");
try testing.expect(t.screen.pages.pages.first != t.screen.pages.pages.last);
var search: PageListSearch = try .init(
alloc,
"Test",
&t.screen.pages,
t.screen.pages.pages.last.?,
);
defer search.deinit();
// No complete match on last page alone (only has "st")
try testing.expect(search.next() == null);
// Feed first page - this should give us enough data to find "Test"
try testing.expect(try search.feed());
// Should find the spanning match
const sel = search.next().?;
try testing.expect(sel.start().node != sel.end().node);
{
const str = try t.screen.selectionString(
alloc,
.{ .sel = sel },
);
defer alloc.free(str);
try testing.expectEqualStrings(str, "Test");
}
// No more matches
try testing.expect(search.next() == null);
// No more pages
try testing.expect(!try search.feed());
}
test "feed with match spanning page boundary with newline" {
const alloc = testing.allocator;
var t: Terminal = try .init(alloc, .{ .cols = 80, .rows = 24 });
defer t.deinit(alloc);
var s = t.vtStream();
defer s.deinit();
const first_page_rows = t.screen.pages.pages.first.?.data.capacity.rows;
// Fill first page ending with "Te"
for (0..first_page_rows - 1) |_| try s.nextSlice("\r\n");
for (0..t.screen.pages.cols - 2) |_| try s.nextSlice("x");
try s.nextSlice("Te");
try testing.expect(t.screen.pages.pages.first == t.screen.pages.pages.last);
// Second page starts with "st"
try s.nextSlice("\r\n");
try testing.expect(t.screen.pages.pages.first != t.screen.pages.pages.last);
try s.nextSlice("st");
var search: PageListSearch = try .init(
alloc,
"Test",
&t.screen.pages,
t.screen.pages.pages.last.?,
);
defer search.deinit();
// Should not find any matches since we broke with an explicit newline.
try testing.expect(search.next() == null);
try testing.expect(try search.feed());
try testing.expect(search.next() == null);
try testing.expect(!try search.feed());
}

View File

@ -0,0 +1,594 @@
const std = @import("std");
const assert = std.debug.assert;
const testing = std.testing;
const Allocator = std.mem.Allocator;
const point = @import("../point.zig");
const PageList = @import("../PageList.zig");
const Pin = PageList.Pin;
const Screen = @import("../Screen.zig");
const Selection = @import("../Selection.zig");
const Terminal = @import("../Terminal.zig");
const ActiveSearch = @import("active.zig").ActiveSearch;
const PageListSearch = @import("pagelist.zig").PageListSearch;
const SlidingWindow = @import("sliding_window.zig").SlidingWindow;
/// Searches for a needle within a Screen, handling active area updates,
/// pages being pruned from the screen (e.g. scrollback limits), and more.
///
/// Unlike our lower-level searchers (like ActiveSearch and PageListSearch),
/// this will cache and store all search results so the caller can re-access
/// them as needed. This structure does this because it is intended to help
/// the caller handle the case where the Screen is changing while the user
/// is searching.
///
/// An inactive screen can continue to be searched in the background, and when
/// screen state changes, the renderer/caller can access the existing search
/// results without needing to re-search everything. This prevents a particularly
/// nasty UX where going to alt screen (e.g. neovim) and then back would
/// restart the full scrollback search.
pub const ScreenSearch = struct {
/// The screen being searched.
screen: *Screen,
/// The active area search state
active: ActiveSearch,
/// The history (scrollback) search state. May be null if there is
/// no history yet.
history: ?HistorySearch,
/// Current state of the search, a state machine.
state: State,
/// The results found so far. These are stored separately because history
/// is mostly immutable once found, while active area results may
/// change. This lets us easily reset the active area results for a
/// re-search scenario.
history_results: std.ArrayList(Selection),
active_results: std.ArrayList(Selection),
/// History search state.
const HistorySearch = struct {
/// The actual searcher state.
searcher: PageListSearch,
/// The pin for the first node that this searcher is searching from.
/// We use this when the active area changes to find the diff between
/// the top of the new active area and the previous start point
/// to determine if we need to search more history.
start_pin: *Pin,
pub fn deinit(self: *HistorySearch, screen: *Screen) void {
self.searcher.deinit();
screen.pages.untrackPin(self.start_pin);
}
};
/// Search state machine
const State = enum {
/// Currently searching the active area
active,
/// Currently searching the history area
history,
/// History search is waiting for more data to be fed before
/// it can progress.
history_feed,
/// Search is complete given the current terminal state.
complete,
};
// Initialize a screen search for the given screen and needle.
pub fn init(
alloc: Allocator,
screen: *Screen,
needle: []const u8,
) Allocator.Error!ScreenSearch {
var result: ScreenSearch = .{
.screen = screen,
.active = try .init(alloc, needle),
.history = null,
.state = .active,
.active_results = .empty,
.history_results = .empty,
};
errdefer result.deinit();
// Update our initial active area state
try result.reloadActive();
return result;
}
pub fn deinit(self: *ScreenSearch) void {
const alloc = self.allocator();
self.active.deinit();
if (self.history) |*h| h.deinit(self.screen);
self.active_results.deinit(alloc);
self.history_results.deinit(alloc);
}
fn allocator(self: *ScreenSearch) Allocator {
return self.active.window.alloc;
}
pub const TickError = Allocator.Error || error{
FeedRequired,
SearchComplete,
};
/// Returns all matches as an owned slice (caller must free).
/// The matches are ordered from most recent to oldest (e.g. bottom
/// of the screen to top of the screen).
pub fn matches(
self: *ScreenSearch,
alloc: Allocator,
) Allocator.Error![]Selection {
const active_results = self.active_results.items;
const history_results = self.history_results.items;
const results = try alloc.alloc(
Selection,
active_results.len + history_results.len,
);
errdefer alloc.free(results);
// Active does a forward search, so we add the active results then
// reverse them. There are usually not many active results so this
// is fast enough compared to adding them in reverse order.
assert(self.active.window.direction == .forward);
@memcpy(
results[0..active_results.len],
active_results,
);
std.mem.reverse(Selection, results[0..active_results.len]);
// History does a backward search, so we can just append them
// after.
@memcpy(
results[active_results.len..],
history_results,
);
return results;
}
/// Search the full screen state. This will block until the search
/// is complete. For performance, it is recommended to use `tick` and
/// `feed` to incrementally make progress on the search instead.
pub fn searchAll(self: *ScreenSearch) Allocator.Error!void {
while (true) {
self.tick() catch |err| switch (err) {
error.OutOfMemory => return error.OutOfMemory,
error.FeedRequired => try self.feed(),
error.SearchComplete => return,
};
}
}
/// Make incremental progress on the search without accessing any
/// screen state (so no lock is required).
///
/// This will return error.FeedRequired if the search cannot make progress
/// without being fed more data. In this case, the caller should call
/// the `feed` function to provide more data to the searcher.
///
/// This will return error.SearchComplete if the search is fully complete.
/// This is to signal to the caller that it can move to a more efficient
/// sleep/wait state until there is more work to do (e.g. new data to feed).
pub fn tick(self: *ScreenSearch) TickError!void {
switch (self.state) {
.active => try self.tickActive(),
.history => try self.tickHistory(),
.history_feed => return error.FeedRequired,
.complete => return error.SearchComplete,
}
}
/// Feed more data to the searcher so it can continue searching. This
/// accesses the screen state, so the caller must hold the necessary locks.
pub fn feed(self: *ScreenSearch) Allocator.Error!void {
const history: *PageListSearch = if (self.history) |*h| &h.searcher else {
// No history to feed, search is complete.
self.state = .complete;
return;
};
// Future: we may want to feed multiple pages at once here to
// lower the frequency of lock acquisitions.
if (!try history.feed()) {
// No more data to feed, search is complete.
self.state = .complete;
return;
}
// Depending on our state handle where feed goes
switch (self.state) {
// If we're searching active or history, then feeding doesn't
// change the state.
.active, .history => {},
// Feed goes back to searching history.
.history_feed => self.state = .history,
// If we're complete then the feed call above should always
// return false and we can't reach this.
.complete => unreachable,
}
}
fn tickActive(self: *ScreenSearch) Allocator.Error!void {
// For the active area, we consume the entire search in one go
// because the active area is generally small.
const alloc = self.allocator();
while (self.active.next()) |sel| {
// If this fails, then we miss a result since `active.next()`
// moves forward and prunes data. In the future, we may want
// to have some more robust error handling but the only
// scenario this would fail is OOM and we're probably in
// deeper trouble at that point anyways.
try self.active_results.append(alloc, sel);
}
// We've consumed the entire active area, move to history.
self.state = .history;
}
fn tickHistory(self: *ScreenSearch) Allocator.Error!void {
const history: *HistorySearch = if (self.history) |*h| h else {
// No history to search, we're done.
self.state = .complete;
return;
};
// Try to consume all the loaded matches in one go, because
// the search is generally fast for loaded data.
const alloc = self.allocator();
while (history.searcher.next()) |sel| {
// Ignore selections that are found within the starting
// node since those are covered by the active area search.
if (sel.start().node == history.start_pin.node) continue;
// Same note as tickActive for error handling.
try self.history_results.append(alloc, sel);
}
// We need to be fed more data.
self.state = .history_feed;
}
/// Reload the active area because it has changed.
///
/// Since it is very fast, this will also do the full active area
/// search again, too. This avoids any complexity around the search
/// state machine.
///
/// The caller must hold the necessary locks to access the screen state.
pub fn reloadActive(self: *ScreenSearch) Allocator.Error!void {
const list: *PageList = &self.screen.pages;
if (try self.active.update(list)) |history_node| history: {
// We need to account for any active area growth that would
// cause new pages to move into our history. If there are new
// pages then we need to re-search the pages and add it to
// our history results.
const history_: ?*HistorySearch = if (self.history) |*h| state: {
// If our start pin became garbage, it means we pruned all
// the way up through it, so we have no history anymore.
// Reset our history state.
if (h.start_pin.garbage) {
h.deinit(self.screen);
self.history = null;
self.history_results.clearRetainingCapacity();
break :state null;
}
break :state h;
} else null;
const history = history_ orelse {
// No history search yet, but we now have history. So let's
// initialize.
// Our usage of needle below depends on this
assert(self.active.window.direction == .forward);
var search: PageListSearch = try .init(
self.allocator(),
self.active.window.needle,
list,
history_node,
);
errdefer search.deinit();
const pin = try list.trackPin(.{ .node = history_node });
errdefer list.untrackPin(pin);
self.history = .{
.searcher = search,
.start_pin = pin,
};
// We don't need to update any history since we had no history
// before, so we can break out of the whole conditional.
break :history;
};
if (history.start_pin.node == history_node) {
// No change in the starting node, we're done.
break :history;
}
// Do a forward search from our prior node to this one. We
// collect all the results into a new list. We ASSUME that
// reloadActive is being called frequently enough that there isn't
// a massive amount of history to search here.
const alloc = self.allocator();
var window: SlidingWindow = try .init(
alloc,
.forward,
self.active.window.needle,
);
defer window.deinit();
while (true) {
_ = try window.append(history.start_pin.node);
if (history.start_pin.node == history_node) break;
const next = history.start_pin.node.next orelse break;
history.start_pin.node = next;
}
assert(history.start_pin.node == history_node);
var results: std.ArrayList(Selection) = try .initCapacity(
alloc,
self.history_results.items.len,
);
errdefer results.deinit(alloc);
while (window.next()) |sel| {
if (sel.start().node == history_node) continue;
try results.append(
alloc,
sel,
);
}
// If we have no matches then there is nothing to change
// in our history (fast path)
if (results.items.len == 0) break :history;
// Matches! Reverse our list then append all the remaining
// history items that didn't start on our original node.
std.mem.reverse(Selection, results.items);
try results.appendSlice(alloc, self.history_results.items);
self.history_results.deinit(alloc);
self.history_results = results;
}
// Reset our active search results and search again.
self.active_results.clearRetainingCapacity();
switch (self.state) {
// If we're in the active state we run a normal tick so
// we can move into a better state.
.active => try self.tickActive(),
// Otherwise, just tick it and move back to whatever state
// we were in.
else => {
const old_state = self.state;
defer self.state = old_state;
try self.tickActive();
},
}
}
};
test "simple search" {
const alloc = testing.allocator;
var t: Terminal = try .init(alloc, .{ .cols = 10, .rows = 2 });
defer t.deinit(alloc);
var s = t.vtStream();
defer s.deinit();
try s.nextSlice("Fizz\r\nBuzz\r\nFizz\r\nBang");
var search: ScreenSearch = try .init(alloc, &t.screen, "Fizz");
defer search.deinit();
try search.searchAll();
try testing.expectEqual(2, search.active_results.items.len);
// We don't test history results since there is overlap
// Get all matches
const matches = try search.matches(alloc);
defer alloc.free(matches);
try testing.expectEqual(2, matches.len);
{
const sel = matches[0];
try testing.expectEqual(point.Point{ .screen = .{
.x = 0,
.y = 2,
} }, t.screen.pages.pointFromPin(.screen, sel.start()).?);
try testing.expectEqual(point.Point{ .screen = .{
.x = 3,
.y = 2,
} }, t.screen.pages.pointFromPin(.screen, sel.end()).?);
}
{
const sel = matches[1];
try testing.expectEqual(point.Point{ .screen = .{
.x = 0,
.y = 0,
} }, t.screen.pages.pointFromPin(.screen, sel.start()).?);
try testing.expectEqual(point.Point{ .screen = .{
.x = 3,
.y = 0,
} }, t.screen.pages.pointFromPin(.screen, sel.end()).?);
}
}
test "simple search with history" {
const alloc = testing.allocator;
var t: Terminal = try .init(alloc, .{
.cols = 10,
.rows = 2,
.max_scrollback = std.math.maxInt(usize),
});
defer t.deinit(alloc);
const list: *PageList = &t.screen.pages;
var s = t.vtStream();
defer s.deinit();
try s.nextSlice("Fizz\r\n");
while (list.totalPages() < 3) try s.nextSlice("\r\n");
for (0..list.rows) |_| try s.nextSlice("\r\n");
try s.nextSlice("hello.");
var search: ScreenSearch = try .init(alloc, &t.screen, "Fizz");
defer search.deinit();
try search.searchAll();
try testing.expectEqual(0, search.active_results.items.len);
// Get all matches
const matches = try search.matches(alloc);
defer alloc.free(matches);
try testing.expectEqual(1, matches.len);
{
const sel = matches[0];
try testing.expectEqual(point.Point{ .screen = .{
.x = 0,
.y = 0,
} }, t.screen.pages.pointFromPin(.screen, sel.start()).?);
try testing.expectEqual(point.Point{ .screen = .{
.x = 3,
.y = 0,
} }, t.screen.pages.pointFromPin(.screen, sel.end()).?);
}
}
test "reload active with history change" {
const alloc = testing.allocator;
var t: Terminal = try .init(alloc, .{
.cols = 10,
.rows = 2,
.max_scrollback = std.math.maxInt(usize),
});
defer t.deinit(alloc);
const list: *PageList = &t.screen.pages;
var s = t.vtStream();
defer s.deinit();
try s.nextSlice("Fizz\r\n");
// Start up our search which will populate our initial active area.
var search: ScreenSearch = try .init(alloc, &t.screen, "Fizz");
defer search.deinit();
try search.searchAll();
{
const matches = try search.matches(alloc);
defer alloc.free(matches);
try testing.expectEqual(1, matches.len);
}
// Grow into two pages so our history pin will move.
while (list.totalPages() < 2) try s.nextSlice("\r\n");
for (0..list.rows) |_| try s.nextSlice("\r\n");
try s.nextSlice("2Fizz");
// Active area changed so reload
try search.reloadActive();
try search.searchAll();
// Get all matches
{
const matches = try search.matches(alloc);
defer alloc.free(matches);
try testing.expectEqual(2, matches.len);
{
const sel = matches[1];
try testing.expectEqual(point.Point{ .screen = .{
.x = 0,
.y = 0,
} }, t.screen.pages.pointFromPin(.screen, sel.start()).?);
try testing.expectEqual(point.Point{ .screen = .{
.x = 3,
.y = 0,
} }, t.screen.pages.pointFromPin(.screen, sel.end()).?);
}
{
const sel = matches[0];
try testing.expectEqual(point.Point{ .active = .{
.x = 1,
.y = 1,
} }, t.screen.pages.pointFromPin(.active, sel.start()).?);
try testing.expectEqual(point.Point{ .active = .{
.x = 4,
.y = 1,
} }, t.screen.pages.pointFromPin(.active, sel.end()).?);
}
}
// Reset the screen which will make our pin garbage.
t.fullReset();
try s.nextSlice("WeFizzing");
try search.reloadActive();
try search.searchAll();
{
const matches = try search.matches(alloc);
defer alloc.free(matches);
try testing.expectEqual(1, matches.len);
{
const sel = matches[0];
try testing.expectEqual(point.Point{ .active = .{
.x = 2,
.y = 0,
} }, t.screen.pages.pointFromPin(.active, sel.start()).?);
try testing.expectEqual(point.Point{ .active = .{
.x = 5,
.y = 0,
} }, t.screen.pages.pointFromPin(.active, sel.end()).?);
}
}
}
test "active change contents" {
const alloc = testing.allocator;
var t: Terminal = try .init(alloc, .{ .cols = 10, .rows = 5 });
defer t.deinit(alloc);
var s = t.vtStream();
defer s.deinit();
try s.nextSlice("Fuzz\r\nBuzz\r\nFizz\r\nBang");
var search: ScreenSearch = try .init(alloc, &t.screen, "Fizz");
defer search.deinit();
try search.searchAll();
try testing.expectEqual(1, search.active_results.items.len);
// Erase the screen, move our cursor to the top, and change contents.
try s.nextSlice("\x1b[2J\x1b[H"); // Clear screen and move home
try s.nextSlice("Bang\r\nFizz\r\nHello!");
try search.reloadActive();
try search.searchAll();
try testing.expectEqual(1, search.active_results.items.len);
// Get all matches
const matches = try search.matches(alloc);
defer alloc.free(matches);
try testing.expectEqual(1, matches.len);
{
const sel = matches[0];
try testing.expectEqual(point.Point{ .screen = .{
.x = 0,
.y = 1,
} }, t.screen.pages.pointFromPin(.screen, sel.start()).?);
try testing.expectEqual(point.Point{ .screen = .{
.x = 3,
.y = 1,
} }, t.screen.pages.pointFromPin(.screen, sel.end()).?);
}
}

File diff suppressed because it is too large Load Diff