Mirrors
/
ripgrep
mirror of https://github.com/BurntSushi/ripgrep.git
1
0
Fork 0
Code Issues Projects Releases Wiki Activity

cli: replace clap with lexopt and supporting code 

ripgrep began it's life with docopt for argument parsing. Then it moved
to Clap and stayed there for a number of years. Clap has served ripgrep
well, and it probably could continue to serve ripgrep well, but I ended
up deciding to move off of it.

Why?

The first time I had the thought of moving off of Clap was during the
2->3->4 transition. I thought the 3.x and 4.x releases were great, but
for me, it ended up moving a little too quickly. Since the release of
4.x was telegraphed around when 3.x came out, I decided to just hold off
and wait to migrate to 4.x instead of doing a 3.x migration followed
shortly by another 4.x migration. Of course, I just never ended up doing
the migration at all. I never got around to it and there just wasn't a
compelling reason for me to upgrade. While I never investigated it, I
saw an upgrade as a non-trivial amount of work in part because I didn't
encapsulate the usage of Clap enough.

The above is just what got me started thinking about it. It wasn't
enough to get me to move off of it on its own. What ended up pushing me
over the edge was a combination of factors:

* As mentioned above, I didn't want to run on the migration treadmill.
This has proven to not be much of an issue, but at the time of the
2->3->4 releases, I didn't know how long Clap 4.x would be out before a
5.x would come out.
* The release of lexopt[1] caught my eye. IMO, that crate demonstrates
exactly how something new can arrive on the scene and just thoroughly
solve a problem minimalistically. It has the docs, the reasoning, the
simple API, the tests and good judgment. It gets all the weird corner
cases right that Clap also gets right (and is part of why I was
originally attracted to Clap).
* I have an overall desire to reduce the size of my dependency tree. In
part because a smaller dependency tree tends to correlate with better
compile times, but also in part because it reduces my reliance and trust
on others. It lets me be the "master" of ripgrep's destiny by reducing
the amount of behavior that is the result of someone else's decision
(whether good or bad).
* I perceived that Clap solves a more general problem than what I
actually need solved. Despite the vast number of flags that ripgrep has,
its requirements are actually pretty simple. We just need simple
switches and flags that support one value. No multi-value flags. No
sub-commands. And probably a lot of other functionality that Clap has
that makes it so flexible for so many different use cases. (I'm being
hand wavy on the last point.)

With all that said, perhaps most importantly, the future of ripgrep
possibly demands a more flexible CLI argument parser. In today's world,
I would really like, for example, flags like `--type` and `--type-not`
to be able to accumulate their repeated values into a single sequence
while respecting the order they appear on the CLI. For example, prior
to this migration, `rg regex-automata -Tlock -ttoml` would not return
results in `Cargo.lock` in this repository because the `-Tlock` always
took priority even though `-ttoml` appeared after it. But with this
migration, `-ttoml` now correctly overrides `-Tlock`. We would like to
do similar things for `-g/--glob` and `--iglob` and potentially even
now introduce a `-G/--glob-not` flag instead of requiring users to use
`!` to negate a glob. (Which I had done originally to work-around this
problem.) And some day, I'd like to add some kind of boolean matching to
ripgrep perhaps similar to how `git grep` does it. (Although I haven't
thought too carefully on a design yet.) In order to do that, I perceive
it would be difficult to implement correctly in Clap.

I believe that this last point is possible to implement correctly in
Clap 2.x, although it is awkward to do so. I have not looked closely
enough at the Clap 4.x API to know whether it's still possible there. In
any case, these were enough reasons to move off of Clap and own more of
the argument parsing process myself.

This did require a few things:

* I had to write my own logic for how arguments are combined into one
single state object. Of course, I wanted this. This was part of the
upside. But it's still code I didn't have to write for Clap.
* I had to write my own shell completion generator.
* I had to write my own `-h/--help` output generator.
* I also had to write my own man page generator. Well, I had to do this
with Clap 2.x too, although my understanding is that Clap 4.x supports
this. With that said, without having tried it, my guess is that I
probably wouldn't have liked the output it generated because I
ultimately had to write most of the roff by hand myself to get the man
page I wanted. (This also had the benefit of dropping the build
dependency on asciidoc/asciidoctor.)

While this is definitely a fair bit of extra work, it overall only cost
me a couple days. IMO, that's a good trade off given that this code is
unlikely to change again in any substantial way. And it should also
allow for more flexible semantics going forward.

Fixes #884, Fixes #1648, Fixes #1701, Fixes #1814, Fixes #1966

[1]: https://docs.rs/lexopt/0.3.0/lexopt/index.html
pull/2648/head
Andrew Gallant 2023-10-16 18:05:39 -04:00
parent c33f623719
commit 082245dadb
47 changed files with 12730 additions and 6147 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

33
.github/workflows/ci.yml vendored
View File

@ -53,7 +53,7 @@ jobs:
include:
- build: pinned
os: ubuntu-latest
rust: 1.72.1
rust: 1.74.0
- build: stable
os: ubuntu-latest
rust: stable
@ -94,7 +94,7 @@ jobs:
rust: nightly-x86_64-gnu
steps:
- name: Checkout repository
uses: actions/checkout@v3
uses: actions/checkout@v4
- name: Install packages (Ubuntu)
if: matrix.os == 'ubuntu-latest'
@ -131,6 +131,7 @@ jobs:
run: |
echo "cargo command is: ${{ env.CARGO }}"
echo "target flag is: ${{ env.TARGET_FLAGS }}"
echo "target dir is: ${{ env.TARGET_DIR }}"
- name: Build ripgrep and all crates
run: ${{ env.CARGO }} build --verbose --workspace ${{ env.TARGET_FLAGS }}
@ -164,26 +165,6 @@ jobs:
if: matrix.target != ''
run: ${{ env.CARGO }} test --verbose --workspace ${{ env.TARGET_FLAGS }}
- name: Test for existence of build artifacts (Windows)
if: matrix.os == 'windows-2022'
shell: bash
run: |
outdir="$(ci/cargo-out-dir "${{ env.TARGET_DIR }}")"
ls "$outdir/_rg.ps1" && file "$outdir/_rg.ps1"
- name: Test for existence of build artifacts (Unix)
if: matrix.os != 'windows-2022'
shell: bash
run: |
outdir="$(ci/cargo-out-dir "${{ env.TARGET_DIR }}")"
# TODO: Check for the man page generation here. For whatever reason,
# it seems to be intermittently failing in CI. No idea why.
# for f in rg.bash rg.fish rg.1; do
for f in rg.bash rg.fish; do
# We could use file -E here, but it isn't supported on macOS.
ls "$outdir/$f" && file "$outdir/$f"
done
- name: Test zsh shell completions (Unix, sans cross)
# We could test this when using Cross, but we'd have to execute the
# 'rg' binary (done in test-complete) with qemu, which is a pain and
@ -197,11 +178,15 @@ jobs:
shell: bash
run: ${{ env.CARGO }} test --manifest-path crates/cli/Cargo.toml ${{ env.TARGET_FLAGS }} --lib print_hostname -- --nocapture
- name: Print available short flags
shell: bash
run: ${{ env.CARGO }} test --bin rg ${{ env.TARGET_FLAGS }} flags::defs::tests::available_shorts -- --nocapture
rustfmt:
runs-on: ubuntu-latest
steps:
- name: Checkout repository
uses: actions/checkout@v3
uses: actions/checkout@v4
- name: Install Rust
uses: dtolnay/rust-toolchain@master
with:
@ -214,7 +199,7 @@ jobs:
runs-on: ubuntu-latest
steps:
- name: Checkout repository
uses: actions/checkout@v3
uses: actions/checkout@v4
- name: Install Rust
uses: dtolnay/rust-toolchain@master
with:

233
.github/workflows/release.yml vendored
View File

@ -1,47 +1,39 @@
# The way this works is the following:
#
# The create-release job runs purely to initialize the GitHub release itself
# and to output upload_url for the following job.
#
# The build-release job runs only once create-release is finished. It gets the
# release upload URL from create-release job outputs, then builds the release
# executables for each supported platform and attaches them as release assets
# to the previously created release.
#
# The key here is that we create the release only once.
#
# Reference:
# https://eugene-babichenko.github.io/blog/2020/05/09/github-actions-cross-platform-auto-releases/
name: release
# Only do the release on x.y.z tags.
on:
push:
# Enable when testing release infrastructure on a branch.
# branches:
# - ag/work
tags:
- "[0-9]+.[0-9]+.[0-9]+"
# We need this to be able to create releases.
permissions:
contents: write
jobs:
# The create-release job runs purely to initialize the GitHub release itself,
# and names the release after the `x.y.z` tag that was pushed. It's separate
# from building the release so that we only create the release once.
create-release:
name: create-release
runs-on: ubuntu-latest
# env:
# Set to force version number, e.g., when no tag exists.
# RG_VERSION: TEST-0.0.0
outputs:
rg_version: ${{ env.RG_VERSION }}
# VERSION: TEST-0.0.0
steps:
- uses: actions/checkout@v3
- uses: actions/checkout@v4
- name: Get the release version from the tag
shell: bash
if: env.RG_VERSION == ''
if: env.VERSION == ''
run: echo "VERSION=${{ github.ref_name }}" >> $GITHUB_ENV
- name: Show the version
run: |
echo "RG_VERSION=$GITHUB_REF_NAME" >> $GITHUB_ENV
echo "version is: ${{ env.RG_VERSION }}"
echo "version is: $VERSION"
- name: Create GitHub release
env:
GH_TOKEN: ${{ github.token }}
run: gh release create ${{ env.RG_VERSION }}
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
run: gh release create $VERSION --draft --verify-tag --title $VERSION
outputs:
version: ${{ env.VERSION }}
build-release:
name: build-release
@ -52,25 +44,50 @@ jobs:
# systems.
CARGO: cargo
# When CARGO is set to CROSS, this is set to `--target matrix.target`.
TARGET_FLAGS: ""
TARGET_FLAGS:
# When CARGO is set to CROSS, TARGET_DIR includes matrix.target.
TARGET_DIR: ./target
# Bump this as appropriate. We pin to a version to make sure CI
# continues to work as cross releases in the past have broken things
# in subtle ways.
CROSS_VERSION: v0.2.5
# Emit backtraces on panics.
RUST_BACKTRACE: 1
# Build static releases with PCRE2.
PCRE2_SYS_STATIC: 1
strategy:
fail-fast: false
matrix:
build: [linux, linux-arm, macos, win-msvc, win-gnu, win32-msvc]
include:
- build: linux
os: ubuntu-latest
rust: nightly
target: x86_64-unknown-linux-musl
- build: linux-arm
strip: x86_64-linux-musl-strip
- build: stable-x86
os: ubuntu-latest
rust: nightly
target: arm-unknown-linux-gnueabihf
rust: stable
target: i686-unknown-linux-gnu
strip: x86_64-linux-gnu-strip
qemu: i386
- build: stable-aarch64
os: ubuntu-latest
rust: stable
target: aarch64-unknown-linux-gnu
strip: aarch64-linux-gnu-strip
qemu: qemu-aarch64
- build: stable-powerpc64
os: ubuntu-latest
rust: stable
target: powerpc64-unknown-linux-gnu
strip: powerpc64-linux-gnu-strip
qemu: qemu-ppc64
- build: stable-s390x
os: ubuntu-latest
rust: stable
target: s390x-unknown-linux-gnu
strip: s390x-linux-gnu-strip
qemu: qemu-s390x
- build: macos
os: macos-latest
rust: nightly
@ -90,15 +107,17 @@ jobs:
steps:
- name: Checkout repository
uses: actions/checkout@v3
uses: actions/checkout@v4
- name: Install packages (Ubuntu)
if: matrix.os == 'ubuntu-latest'
shell: bash
run: |
ci/ubuntu-install-packages
- name: Install packages (macOS)
if: matrix.os == 'macos-latest'
shell: bash
run: |
ci/macos-install-packages
@ -109,64 +128,140 @@ jobs:
target: ${{ matrix.target }}
- name: Use Cross
if: matrix.os == 'ubuntu-latest' && matrix.target != ''
shell: bash
run: |
cargo install cross
# In the past, new releases of 'cross' have broken CI. So for now, we
# pin it. We also use their pre-compiled binary releases because cross
# has over 100 dependencies and takes a bit to compile.
dir="$RUNNER_TEMP/cross-download"
mkdir "$dir"
echo "$dir" >> $GITHUB_PATH
cd "$dir"
curl -LO "https://github.com/cross-rs/cross/releases/download/$CROSS_VERSION/cross-x86_64-unknown-linux-musl.tar.gz"
tar xf cross-x86_64-unknown-linux-musl.tar.gz
echo "CARGO=cross" >> $GITHUB_ENV
- name: Set target variables
shell: bash
run: |
echo "TARGET_FLAGS=--target ${{ matrix.target }}" >> $GITHUB_ENV
echo "TARGET_DIR=./target/${{ matrix.target }}" >> $GITHUB_ENV
- name: Show command used for Cargo
shell: bash
run: |
echo "cargo command is: ${{ env.CARGO }}"
echo "target flag is: ${{ env.TARGET_FLAGS }}"
echo "target dir is: ${{ env.TARGET_DIR }}"
- name: Build release binary
run: ${{ env.CARGO }} build --verbose --release --features pcre2 ${{ env.TARGET_FLAGS }}
shell: bash
run: |
${{ env.CARGO }} build --verbose --release --features pcre2 ${{ env.TARGET_FLAGS }}
if [ "${{ matrix.os }}" = "windows-latest" ]; then
bin="target/${{ matrix.target }}/release/rg.exe"
else
bin="target/${{ matrix.target }}/release/rg"
fi
echo "BIN=$bin" >> $GITHUB_ENV
- name: Strip release binary (linux, macos and macos-arm)
if: matrix.build == 'linux' || matrix.os == 'macos'
run: strip "target/${{ matrix.target }}/release/rg"
- name: Strip release binary (macos)
if: matrix.os == 'macos'
shell: bash
run: strip "$BIN"
- name: Strip release binary (arm)
if: matrix.build == 'linux-arm'
- name: Strip release binary (cross)
if: env.CARGO == 'cross'
shell: bash
run: |
docker run --rm -v \
"$PWD/target:/target:Z" \
rustembedded/cross:arm-unknown-linux-gnueabihf \
arm-linux-gnueabihf-strip \
/target/arm-unknown-linux-gnueabihf/release/rg
"rustembedded/cross:${{ matrix.target }}" \
"${{ matrix.strip }}" \
"/target/${{ matrix.target }}/release/rg"
- name: Build archive
- name: Determine archive name
shell: bash
run: |
outdir="$(ci/cargo-out-dir "${{ env.TARGET_DIR }}")"
staging="ripgrep-${{ needs.create-release.outputs.rg_version }}-${{ matrix.target }}"
mkdir -p "$staging"/{complete,doc}
version="${{ needs.create-release.outputs.version }}"
echo "ARCHIVE=ripgrep-$version-${{ matrix.target }}" >> $GITHUB_ENV
cp {README.md,COPYING,UNLICENSE,LICENSE-MIT} "$staging/"
cp {CHANGELOG.md,FAQ.md,GUIDE.md} "$staging/doc/"
cp "$outdir"/{rg.bash,rg.fish,_rg.ps1} "$staging/complete/"
cp complete/_rg "$staging/complete/"
- name: Creating directory for archive
shell: bash
run: |
mkdir -p "$ARCHIVE"/{complete,doc}
cp "$BIN" "$ARCHIVE"/
cp {README.md,COPYING,UNLICENSE,LICENSE-MIT} "$ARCHIVE"/
cp {CHANGELOG.md,FAQ.md,GUIDE.md} "$ARCHIVE"/doc/
if [ "${{ matrix.os }}" = "windows-latest" ]; then
cp "target/${{ matrix.target }}/release/rg.exe" "$staging/"
7z a "$staging.zip" "$staging"
certutil -hashfile "$staging.zip" SHA256 > "$staging.zip.sha256"
echo "ASSET=$staging.zip" >> $GITHUB_ENV
echo "ASSET_SUM=$staging.zip.sha256" >> $GITHUB_ENV
else
# The man page is only generated on Unix systems. ¯\_(ツ)_/¯
cp "$outdir"/rg.1 "$staging/doc/"
cp "target/${{ matrix.target }}/release/rg" "$staging/"
tar czf "$staging.tar.gz" "$staging"
shasum -a 256 "$staging.tar.gz" > "$staging.tar.gz.sha256"
echo "ASSET=$staging.tar.gz" >> $GITHUB_ENV
echo "ASSET_SUM=$staging.tar.gz.sha256" >> $GITHUB_ENV
fi
- name: Generate man page and completions (no emulation)
if: matrix.qemu == ''
shell: bash
run: |
"$BIN" --version
"$BIN" --generate complete-bash > "$ARCHIVE/complete/rg.bash"
"$BIN" --generate complete-fish > "$ARCHIVE/complete/rg.fish"
"$BIN" --generate complete-powershell > "$ARCHIVE/complete/_rg.ps1"
"$BIN" --generate complete-zsh > "$ARCHIVE/complete/_rg"
"$BIN" --generate man > "$ARCHIVE/doc/rg.1"
- name: Generate man page and completions (emulation)
if: matrix.qemu != ''
shell: bash
run: |
docker run --rm -v \
"$PWD/target:/target:Z" \
"rustembedded/cross:${{ matrix.target }}" \
"${{ matrix.qemu }}" "/$BIN" --version
docker run --rm -v \
"$PWD/target:/target:Z" \
"rustembedded/cross:${{ matrix.target }}" \
"${{ matrix.qemu }}" "/$BIN" \
--generate complete-bash > "$ARCHIVE/complete/rg.bash"
docker run --rm -v \
"$PWD/target:/target:Z" \
"rustembedded/cross:${{ matrix.target }}" \
"${{ matrix.qemu }}" "/$BIN" \
--generate complete-fish > "$ARCHIVE/complete/rg.fish"
docker run --rm -v \
"$PWD/target:/target:Z" \
"rustembedded/cross:${{ matrix.target }}" \
"${{ matrix.qemu }}" "/$BIN" \
--generate complete-powershell > "$ARCHIVE/complete/_rg.ps1"
docker run --rm -v \
"$PWD/target:/target:Z" \
"rustembedded/cross:${{ matrix.target }}" \
"${{ matrix.qemu }}" "/$BIN" \
--generate complete-zsh > "$ARCHIVE/complete/_rg"
docker run --rm -v \
"$PWD/target:/target:Z" \
"rustembedded/cross:${{ matrix.target }}" \
"${{ matrix.qemu }}" "/$BIN" \
--generate man > "$ARCHIVE/doc/rg.1"
- name: Build archive (Windows)
shell: bash
if: matrix.os == 'windows-latest'
run: |
7z a "$ARCHIVE.zip" "$ARCHIVE"
certutil -hashfile "$ARCHIVE.zip" SHA256 > "$ARCHIVE.zip.sha256"
echo "ASSET=$ARCHIVE.zip" >> $GITHUB_ENV
echo "ASSET_SUM=$ARCHIVE.zip.sha256" >> $GITHUB_ENV
- name: Build archive (Unix)
shell: bash
if: matrix.os != 'windows-latest'
run: |
tar czf "$ARCHIVE.tar.gz" "$ARCHIVE"
shasum -a 256 "$ARCHIVE.tar.gz" > "$ARCHIVE.tar.gz.sha256"
echo "ASSET=$ARCHIVE.tar.gz" >> $GITHUB_ENV
echo "ASSET_SUM=$ARCHIVE.tar.gz.sha256" >> $GITHUB_ENV
- name: Upload release archive
env:
GH_TOKEN: ${{ github.token }}
run: gh release upload ${{ needs.create-release.outputs.rg_version }} ${{ env.ASSET }} ${{ env.ASSET_SUM }}
GITHUB_TOKEN: ${{ secrets.GITHUB_TOKEN }}
shell: bash
run: |
version="${{ needs.create-release.outputs.version }}"
gh release upload "$version" ${{ env.ASSET }} ${{ env.ASSET_SUM }}

10
CHANGELOG.md
View File

@ -21,6 +21,8 @@ Feature enhancements:
Gradle, GraphQL, Markdown, Prolog, Raku, TypeScript, USD, V
* [FEATURE #1790](https://github.com/BurntSushi/ripgrep/issues/1790):
Add new `--stop-on-nonmatch` flag.
* [FEATURE #1814](https://github.com/BurntSushi/ripgrep/issues/1814):
Flags are now categorized in `-h/--help` output and ripgrep's man page.
* [FEATURE #2195](https://github.com/BurntSushi/ripgrep/issues/2195):
When `extra-verbose` mode is enabled in zsh, show extra file type info.
* [FEATURE #2409](https://github.com/BurntSushi/ripgrep/pull/2409):
@ -28,14 +30,22 @@ Feature enhancements:
Bug fixes:
* [BUG #884](https://github.com/BurntSushi/ripgrep/issues/884):
Don't error when `-v/--invert-match` is used multiple times.
* [BUG #1275](https://github.com/BurntSushi/ripgrep/issues/1275):
Fix bug with `\b` assertion in the regex engine.
* [BUG #1648](https://github.com/BurntSushi/ripgrep/issues/1648):
Fix bug where sometimes short flags with values, e.g., `-M 900`, would fail.
* [BUG #1701](https://github.com/BurntSushi/ripgrep/issues/1701):
Fix bug where some flags could not be repeated.
* [BUG #1757](https://github.com/BurntSushi/ripgrep/issues/1757):
Fix bug when searching a sub-directory didn't have ignores applied correctly.
* [BUG #1891](https://github.com/BurntSushi/ripgrep/issues/1891):
Fix bug when using `-w` with a regex that can match the empty string.
* [BUG #1911](https://github.com/BurntSushi/ripgrep/issues/1911):
Disable mmap searching in all non-64-bit environments.
* [BUG #1966](https://github.com/BurntSushi/ripgrep/issues/1966):
Fix bug where ripgrep can panic when printing to stderr.
* [BUG #2108](https://github.com/BurntSushi/ripgrep/issues/2108):
Improve docs for `-r/--replace` syntax.
* [BUG #2198](https://github.com/BurntSushi/ripgrep/issues/2198):

50
Cargo.lock generated
View File

@ -29,12 +29,6 @@ version = "0.21.4"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "9ba43ea6f343b788c8764558649e08df62f86c6ef251fdaeb1ffd010a9ae50a2"
[[package]]
name = "bitflags"
version = "1.3.2"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "bef38d45163c2f1dde094a7dfd33ccf595c92905c8f8f4fdc18d06fb1037718a"
[[package]]
name = "bstr"
version = "1.7.0"
@ -62,18 +56,6 @@ version = "1.0.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "baf1de4339761588bc0619e3cbc0120ee582ebb74b53b4efbf79117bd2da40fd"
[[package]]
name = "clap"
version = "2.34.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "a0610544180c38b88101fecf2dd634b174a62eef6946f84dfc6a7127512b381c"
dependencies = [
"bitflags",
"strsim",
"textwrap",
"unicode-width",
]
[[package]]
name = "crossbeam-channel"
version = "0.5.8"
@ -291,6 +273,12 @@ dependencies = [
"libc",
]
[[package]]
name = "lexopt"
version = "0.3.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "baff4b617f7df3d896f97fe922b64817f6cd9a756bb81d40f8883f2f66dcb401"
[[package]]
name = "libc"
version = "0.2.149"
@ -434,15 +422,16 @@ version = "13.0.0"
dependencies = [
"anyhow",
"bstr",
"clap",
"grep",
"ignore",
"jemallocator",
"lexopt",
"log",
"serde",
"serde_derive",
"serde_json",
"termcolor",
"textwrap",
"walkdir",
]
@ -498,12 +487,6 @@ dependencies = [
"serde",
]
[[package]]
name = "strsim"
version = "0.8.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "8ea5119cdb4c55b55d432abb513a0429384878c15dde60cc77b1c99de1a95a6a"
[[package]]
name = "syn"
version = "2.0.38"
@ -517,21 +500,18 @@ dependencies = [
[[package]]
name = "termcolor"
version = "1.3.0"
version = "1.4.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "6093bad37da69aab9d123a8091e4be0aa4a03e4d601ec641c327398315f62b64"
checksum = "ff1bc3d3f05aff0403e8ac0d92ced918ec05b666a43f83297ccef5bea8a3d449"
dependencies = [
"winapi-util",
]
[[package]]
name = "textwrap"
version = "0.11.0"
version = "0.16.0"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "d326610f408c7a4eb6f51c37c330e496b08506c9457c9d34287ecc38809fb060"
dependencies = [
"unicode-width",
]
checksum = "222a222a5bfe1bba4a77b45ec488a741b3cb8872e5e499451fd7d0129c9c7c3d"
[[package]]
name = "unicode-ident"
@ -539,12 +519,6 @@ version = "1.0.12"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "3354b9ac3fae1ff6755cb6db53683adb661634f67557942dea4facebec0fee4b"
[[package]]
name = "unicode-width"
version = "0.1.11"
source = "registry+https://github.com/rust-lang/crates.io-index"
checksum = "e51733f11c9c4f72aa0c160008246859e340b00807569a0da0e7a1079b27ba85"
[[package]]
name = "walkdir"
version = "2.4.0"

14
Cargo.toml
View File

@ -50,26 +50,18 @@ members = [
[dependencies]
anyhow = "1.0.75"
bstr = "1.6.0"
bstr = "1.7.0"
grep = { version = "0.2.12", path = "crates/grep" }
ignore = { version = "0.4.19", path = "crates/ignore" }
lexopt = "0.3.0"
log = "0.4.5"
serde_json = "1.0.23"
termcolor = "1.1.0"
[dependencies.clap]
version = "2.33.0"
default-features = false
features = ["suggestions"]
textwrap = { version = "0.16.0", default-features = false }
[target.'cfg(all(target_env = "musl", target_pointer_width = "64"))'.dependencies.jemallocator]
version = "0.5.0"
[build-dependencies.clap]
version = "2.33.0"
default-features = false
features = ["suggestions"]
[dev-dependencies]
serde = "1.0.77"
serde_derive = "1.0.77"

16
RELEASE-CHECKLIST.md
View File

@ -1,11 +1,12 @@
Release Checklist
-----------------
# Release Checklist
* Ensure local `master` is up to date with respect to `origin/master`.
* Run `cargo update` and review dependency updates. Commit updated
`Cargo.lock`.
* Run `cargo outdated` and review semver incompatible updates. Unless there is
a strong motivation otherwise, review and update every dependency. Also
run `--aggressive`, but don't update to crates that are still in beta.
* Update date in `crates/core/flags/doc/template.rg.1`.
* Review changes for every crate in `crates` since the last ripgrep release.
If the set of changes is non-empty, issue a new release for that crate. Check
crates in the following order. After updating a crate, ensure minimal
@ -39,8 +40,10 @@ Release Checklist
> tool that recursively searches the current directory for a regex pattern.
> By default, ripgrep will respect gitignore rules and automatically skip
> hidden files/directories and binary files.
* Run `ci/build-deb` locally and manually upload the deb package to the
release.
* Run `git checkout $version && ci/build-and-publish-deb $version` on a Linux
system that has `cargo deb` installed.
* Run `git checkout $version && ci/build-and-publish-m2 $version` on a macOS
system with Apple silicon.
* Run `cargo publish`.
* Run `ci/sha256-releases {VERSION} >> pkg/brew/ripgrep-bin.rb`. Then edit
`pkg/brew/ripgrep-bin.rb` to update the version number and sha256 hashes.
@ -52,5 +55,6 @@ Release Checklist
Unreleased changes. Release notes have not yet been written.
```
Note that
[`cargo-up` can be found in BurntSushi's dotfiles](https://github.com/BurntSushi/dotfiles/blob/master/bin/cargo-up).
Note that [`cargo-up` can be found in BurntSushi's dotfiles][dotfiles].
[dotfiles]: https://github.com/BurntSushi/dotfiles/blob/master/bin/cargo-up

261
build.rs
View File

@ -1,70 +1,24 @@
use std::{
env,
fs::{self, File},
io::{self, Read, Write},
path::Path,
process,
};
use clap::Shell;
use crate::app::{RGArg, RGArgKind};
#[allow(dead_code)]
#[path = "crates/core/app.rs"]
mod app;
fn main() {
// OUT_DIR is set by Cargo and it's where any additional build artifacts
// are written.
let Some(outdir) = env::var_os("OUT_DIR") else {
eprintln!(
"OUT_DIR environment variable not defined. \
Please file a bug: \
https://github.com/BurntSushi/ripgrep/issues/new"
);
process::exit(1);
};
fs::create_dir_all(&outdir).unwrap();
let stamp_path = Path::new(&outdir).join("ripgrep-stamp");
if let Err(err) = File::create(&stamp_path) {
panic!("failed to write {}: {}", stamp_path.display(), err);
}
if let Err(err) = generate_man_page(&outdir) {
eprintln!("failed to generate man page: {}", err);
}
// Use clap to build completion files.
let mut app = app::app();
app.gen_completions("rg", Shell::Bash, &outdir);
app.gen_completions("rg", Shell::Fish, &outdir);
app.gen_completions("rg", Shell::PowerShell, &outdir);
// Note that we do not use clap's support for zsh. Instead, zsh completions
// are manually maintained in `complete/_rg`.
// Make the current git hash available to the build.
if let Some(rev) = git_revision_hash() {
println!("cargo:rustc-env=RIPGREP_BUILD_GIT_HASH={}", rev);
}
// Embed a Windows manifest and set some linker options. The main reason
// for this is to enable long path support on Windows. This still, I
// believe, requires enabling long path support in the registry. But if
// that's enabled, then this will let ripgrep use C:\... style paths that
// are longer than 260 characters.
set_git_revision_hash();
set_windows_exe_options();
}
/// Embed a Windows manifest and set some linker options.
///
/// The main reason for this is to enable long path support on Windows. This
/// still, I believe, requires enabling long path support in the registry. But
/// if that's enabled, then this will let ripgrep use C:\... style paths that
/// are longer than 260 characters.
fn set_windows_exe_options() {
static MANIFEST: &str = "pkg/windows/Manifest.xml";
let Ok(target_os) = env::var("CARGO_CFG_TARGET_OS") else { return };
let Ok(target_env) = env::var("CARGO_CFG_TARGET_ENV") else { return };
let Ok(target_os) = std::env::var("CARGO_CFG_TARGET_OS") else { return };
let Ok(target_env) = std::env::var("CARGO_CFG_TARGET_ENV") else { return };
if !(target_os == "windows" && target_env == "msvc") {
return;
}
let Ok(mut manifest) = env::current_dir() else { return };
let Ok(mut manifest) = std::env::current_dir() else { return };
manifest.push(MANIFEST);
let Some(manifest) = manifest.to_str() else { return };
@ -77,189 +31,16 @@ fn set_windows_exe_options() {
println!("cargo:rustc-link-arg-bin=rg=/WX");
}
fn git_revision_hash() -> Option<String> {
let output = process::Command::new("git")
.args(&["rev-parse", "--short=10", "HEAD"])
.output()
.ok()?;
let v = String::from_utf8_lossy(&output.stdout).trim().to_string();
if v.is_empty() {
None
} else {
Some(v)
/// Make the current git hash available to the build as the environment
/// variable `RIPGREP_BUILD_GIT_HASH`.
fn set_git_revision_hash() {
use std::process::Command;
let args = &["rev-parse", "--short=10", "HEAD"];
let Ok(output) = Command::new("git").args(args).output() else { return };
let rev = String::from_utf8_lossy(&output.stdout).trim().to_string();
if rev.is_empty() {
return;
}
}
fn generate_man_page<P: AsRef<Path>>(outdir: P) -> io::Result<()> {
// If asciidoctor isn't installed, fallback to asciidoc.
if let Err(err) = process::Command::new("asciidoctor").output() {
eprintln!(
"Could not run 'asciidoctor' binary, falling back to 'a2x'."
);
eprintln!("Error from running 'asciidoctor': {}", err);
return legacy_generate_man_page::<P>(outdir);
}
// 1. Read asciidoctor template.
// 2. Interpolate template with auto-generated docs.
// 3. Save interpolation to disk.
// 4. Use asciidoctor to convert to man page.
let outdir = outdir.as_ref();
let cwd = env::current_dir()?;
let tpl_path = cwd.join("doc").join("rg.1.txt.tpl");
let txt_path = outdir.join("rg.1.txt");
let mut tpl = String::new();
File::open(&tpl_path)?.read_to_string(&mut tpl)?;
let options =
formatted_options()?.replace("&#123;", "{").replace("&#125;", "}");
tpl = tpl.replace("{OPTIONS}", &options);
let githash = git_revision_hash();
let githash = githash.as_ref().map(|x| &**x);
tpl = tpl.replace("{VERSION}", &app::long_version(githash, false));
File::create(&txt_path)?.write_all(tpl.as_bytes())?;
let result = process::Command::new("asciidoctor")
.arg("--doctype")
.arg("manpage")
.arg("--backend")
.arg("manpage")
.arg(&txt_path)
.spawn()?
.wait()?;
if !result.success() {
let msg =
format!("'asciidoctor' failed with exit code {:?}", result.code());
return Err(ioerr(msg));
}
Ok(())
}
fn legacy_generate_man_page<P: AsRef<Path>>(outdir: P) -> io::Result<()> {
// If asciidoc isn't installed, then don't do anything.
if let Err(err) = process::Command::new("a2x").output() {
eprintln!("Could not run 'a2x' binary, skipping man page generation.");
eprintln!("Error from running 'a2x': {}", err);
return Ok(());
}
// 1. Read asciidoc template.
// 2. Interpolate template with auto-generated docs.
// 3. Save interpolation to disk.
// 4. Use a2x (part of asciidoc) to convert to man page.
let outdir = outdir.as_ref();
let cwd = env::current_dir()?;
let tpl_path = cwd.join("doc").join("rg.1.txt.tpl");
let txt_path = outdir.join("rg.1.txt");
let mut tpl = String::new();
File::open(&tpl_path)?.read_to_string(&mut tpl)?;
tpl = tpl.replace("{OPTIONS}", &formatted_options()?);
let githash = git_revision_hash();
let githash = githash.as_ref().map(|x| &**x);
tpl = tpl.replace("{VERSION}", &app::long_version(githash, false));
File::create(&txt_path)?.write_all(tpl.as_bytes())?;
let result = process::Command::new("a2x")
.arg("--no-xmllint")
.arg("--doctype")
.arg("manpage")
.arg("--format")
.arg("manpage")
.arg(&txt_path)
.spawn()?
.wait()?;
if !result.success() {
let msg = format!("'a2x' failed with exit code {:?}", result.code());
return Err(ioerr(msg));
}
Ok(())
}
fn formatted_options() -> io::Result<String> {
let mut args = app::all_args_and_flags();
args.sort_by(|x1, x2| x1.name.cmp(&x2.name));
let mut formatted = vec![];
for arg in args {
if arg.hidden {
continue;
}
// ripgrep only has two positional arguments, and probably will only
// ever have two positional arguments, so we just hardcode them into
// the template.
if let app::RGArgKind::Positional { .. } = arg.kind {
continue;
}
formatted.push(formatted_arg(&arg)?);
}
Ok(formatted.join("\n\n"))
}
fn formatted_arg(arg: &RGArg) -> io::Result<String> {
match arg.kind {
RGArgKind::Positional { .. } => {
panic!("unexpected positional argument")
}
RGArgKind::Switch { long, short, multiple } => {
let mut out = vec![];
let mut header = format!("--{}", long);
if let Some(short) = short {
header = format!("-{}, {}", short, header);
}
if multiple {
header = format!("*{}* ...::", header);
} else {
header = format!("*{}*::", header);
}
writeln!(out, "{}", header)?;
writeln!(out, "{}", formatted_doc_txt(arg)?)?;
Ok(String::from_utf8(out).unwrap())
}
RGArgKind::Flag { long, short, value_name, multiple, .. } => {
let mut out = vec![];
let mut header = format!("--{}", long);
if let Some(short) = short {
header = format!("-{}, {}", short, header);
}
if multiple {
header = format!("*{}* _{}_ ...::", header, value_name);
} else {
header = format!("*{}* _{}_::", header, value_name);
}
writeln!(out, "{}", header)?;
writeln!(out, "{}", formatted_doc_txt(arg)?)?;
Ok(String::from_utf8(out).unwrap())
}
}
}
fn formatted_doc_txt(arg: &RGArg) -> io::Result<String> {
let paragraphs: Vec<String> = arg
.doc_long
.replace("{", "&#123;")
.replace("}", r"&#125;")
// Hack to render ** literally in man page correctly. We can't put
// these crazy +++ in the help text directly, since that shows
// literally in --help output.
.replace("*-g 'foo/**'*", "*-g +++'foo/**'+++*")
.split("\n\n")
.map(|s| s.to_string())
.collect();
if paragraphs.is_empty() {
return Err(ioerr(format!("missing docs for --{}", arg.name)));
}
let first = format!(" {}", paragraphs[0].replace("\n", "\n "));
if paragraphs.len() == 1 {
return Ok(first);
}
Ok(format!("{}\n+\n{}", first, paragraphs[1..].join("\n+\n")))
}
fn ioerr(msg: String) -> io::Error {
io::Error::new(io::ErrorKind::Other, msg)
println!("cargo:rustc-env=RIPGREP_BUILD_GIT_HASH={}", rev);
}

33
ci/build-deb → ci/build-and-publish-deb
View File

@ -5,7 +5,7 @@ D="$(cd "$(dirname "${BASH_SOURCE[0]}")" >/dev/null 2>&1 && pwd)"
# This script builds a binary dpkg for Debian based distros. It does not
# currently run in CI, and is instead run manually and the resulting dpkg is
# uploaded to GitHub via the web UI.
# uploaded to GitHub at the end of this script.
#
# Note that this requires 'cargo deb', which can be installed with
# 'cargo install cargo-deb'.
@ -13,13 +13,19 @@ D="$(cd "$(dirname "${BASH_SOURCE[0]}")" >/dev/null 2>&1 && pwd)"
# This should be run from the root of the ripgrep repo.
if ! command -V cargo-deb > /dev/null 2>&1; then
echo "cargo-deb command missing" >&2
exit 1
echo "cargo-deb command missing" >&2
exit 1
fi
if ! command -V asciidoctor > /dev/null 2>&1; then
echo "asciidoctor command missing" >&2
exit 1
version="$1"
if [ -z "$version" ]; then
echo "missing version" >&2
echo "Usage: "$(basename "$0")" <version>" >&2
exit 1
fi
if ! grep -q "version = \"$version\"" Cargo.toml; then
echo "version does not match Cargo.toml" >&2
exit 1
fi
# 'cargo deb' does not seem to provide a way to specify an asset that is
@ -30,13 +36,20 @@ fi
cargo build
DEPLOY_DIR=deployment/deb
OUT_DIR="$("$D"/cargo-out-dir target/debug/)"
mkdir -p "$DEPLOY_DIR"
# Copy man page and shell completions.
cp "$OUT_DIR"/{rg.1,rg.bash,rg.fish} "$DEPLOY_DIR/"
cp complete/_rg "$DEPLOY_DIR/"
# Generate man page and shell completions. `cargo deb` knows how to find these
# files via the manifest configuration in `Cargo.toml`.
"target/debug/rg" --generate complete-bash > "$DEPLOY_DIR/rg.bash"
"target/debug/rg" --generate complete-fish > "$DEPLOY_DIR/rg.fish"
"target/debug/rg" --generate complete-zsh > "$DEPLOY_DIR/_rg"
"target/debug/rg" --generate man > "$DEPLOY_DIR/rg.1"
# Since we're distributing the dpkg, we don't know whether the user will have
# PCRE2 installed, so just do a static build.
PCRE2_SYS_STATIC=1 cargo deb --target x86_64-unknown-linux-musl
target="target/x86_64-unknown-linux-musl/debian"
deb="$target/ripgrep_$version-1_amd64.deb"
debsum="$deb.sha256"
shasum -a 256 "$deb" > "$debsum"
gh release upload "$version" "$deb" "$debsum"

34
ci/build-and-publish-m2 Executable file
View File

@ -0,0 +1,34 @@
#!/bin/bash
set -e
version="$1"
if [ -z "$version" ]; then
echo "missing version" >&2
echo "Usage: "$(basename "$0")" <version>" >&2
exit 1
fi
if ! grep -q "version = \"$version\"" Cargo.toml; then
echo "version does not match Cargo.toml" >&2
exit 1
fi
cargo build --release --features pcre2
BIN=target/release/rg
NAME=ripgrep-$version-aarch64-apple-darwin
ARCHIVE="deployment/m2/$NAME"
mkdir -p "$ARCHIVE"/{complete,doc}
cp target/release/rg "$ARCHIVE"/
strip "$ARCHIVE/rg"
cp {README.md,COPYING,UNLICENSE,LICENSE-MIT} "$ARCHIVE"/
cp {CHANGELOG.md,FAQ.md,GUIDE.md} "$ARCHIVE"/doc/
"$BIN" --generate complete-bash > "$ARCHIVE/complete/rg.bash"
"$BIN" --generate complete-fish > "$ARCHIVE/complete/rg.fish"
"$BIN" --generate complete-powershell > "$ARCHIVE/complete/_rg.ps1"
"$BIN" --generate complete-zsh > "$ARCHIVE/complete/_rg"
"$BIN" --generate man > "$ARCHIVE/doc/rg.1"
tar c -C deployment/m2 -z -f "$ARCHIVE.tar.gz" "$NAME"
shasum -a 256 "$ARCHIVE.tar.gz" > "$ARCHIVE.tar.gz.sha256"
gh release upload "$version" "$ARCHIVE.tar.gz" "$ARCHIVE.tar.gz.sha256"

19
ci/cargo-out-dir
View File

@ -1,19 +0,0 @@
#!/bin/bash
# Finds Cargo's `OUT_DIR` directory from the most recent build.
#
# This requires one parameter corresponding to the target directory
# to search for the build output.
if [ $# != 1 ]; then
echo "Usage: $(basename "$0") <target-dir>" >&2
exit 2
fi
# This works by finding the most recent stamp file, which is produced by
# every ripgrep build.
target_dir="$1"
find "$target_dir" -name ripgrep-stamp -print0 \
| xargs -0 ls -t \
| head -n1 \
| xargs dirname

2
ci/test-complete
View File

@ -19,7 +19,7 @@ get_comp_args() {
main() {
local diff
local rg="${0:a:h}/../${TARGET_DIR:-target}/release/rg"
local _rg="${0:a:h}/../complete/_rg"
local _rg="${0:a:h}/../crates/core/flags/complete/rg.zsh"
local -a help_args comp_args
[[ -e $rg ]] || rg=${rg/%\/release\/rg/\/debug\/rg}

2
crates/cli/src/wtr.rs
View File

@ -3,6 +3,7 @@ use std::io::{self, IsTerminal};
use termcolor::{self, HyperlinkSpec};
/// A writer that supports coloring with either line or block buffering.
#[derive(Debug)]
pub struct StandardStream(StandardStreamKind);
/// Returns a possibly buffered writer to stdout for the given color choice.
@ -57,6 +58,7 @@ pub fn stdout_buffered_block(
StandardStream(StandardStreamKind::BlockBuffered(out))
}
#[derive(Debug)]
enum StandardStreamKind {
LineBuffered(termcolor::StandardStream),
BlockBuffered(termcolor::BufferedStandardStream),

3251
crates/core/app.rs
View File

File diff suppressed because it is too large Load Diff

1975
crates/core/args.rs
View File

File diff suppressed because it is too large Load Diff

107
crates/core/flags/complete/bash.rs Normal file
View File

@ -0,0 +1,107 @@
/*!
Provides completions for ripgrep's CLI for the bash shell.
*/
use crate::flags::defs::FLAGS;
const TEMPLATE_FULL: &'static str = "
_rg() {
local i cur prev opts cmds
COMPREPLY=()
cur=\"${COMP_WORDS[COMP_CWORD]}\"
prev=\"${COMP_WORDS[COMP_CWORD-1]}\"
cmd=\"\"
opts=\"\"
for i in ${COMP_WORDS[@]}; do
case \"${i}\" in
rg)
cmd=\"rg\"
;;
*)
;;
esac
done
case \"${cmd}\" in
rg)
opts=\"!OPTS!\"
if [[ ${cur} == -* || ${COMP_CWORD} -eq 1 ]] ; then
COMPREPLY=($(compgen -W \"${opts}\" -- \"${cur}\"))
return 0
fi
case \"${prev}\" in
!CASES!
esac
COMPREPLY=($(compgen -W \"${opts}\" -- \"${cur}\"))
return 0
;;
esac
}
complete -F _rg -o bashdefault -o default rg
";
const TEMPLATE_CASE: &'static str = "
!FLAG!)
COMPREPLY=($(compgen -f \"${cur}\"))
return 0
;;
";
const TEMPLATE_CASE_CHOICES: &'static str = "
!FLAG!)
COMPREPLY=($(compgen -W \"!CHOICES!\" -- \"${cur}\"))
return 0
;;
";
/// Generate completions for Bash.
///
/// Note that these completions are based on what was produced for ripgrep <=13
/// using Clap 2.x. Improvements on this are welcome.
pub(crate) fn generate() -> String {
let mut opts = String::new();
for flag in FLAGS.iter() {
opts.push_str("--");
opts.push_str(flag.name_long());
opts.push(' ');
if let Some(short) = flag.name_short() {
opts.push('-');
opts.push(char::from(short));
opts.push(' ');
}
if let Some(name) = flag.name_negated() {
opts.push_str("--");
opts.push_str(name);
opts.push(' ');
}
}
opts.push_str("<PATTERN> <PATH>...");
let mut cases = String::new();
for flag in FLAGS.iter() {
let template = if !flag.doc_choices().is_empty() {
let choices = flag.doc_choices().join(" ");
TEMPLATE_CASE_CHOICES.trim_end().replace("!CHOICES!", &choices)
} else {
TEMPLATE_CASE.trim_end().to_string()
};
let name = format!("--{}", flag.name_long());
cases.push_str(&template.replace("!FLAG!", &name));
if let Some(short) = flag.name_short() {
let name = format!("-{}", char::from(short));
cases.push_str(&template.replace("!FLAG!", &name));
}
if let Some(negated) = flag.name_negated() {
let name = format!("--{negated}");
cases.push_str(&template.replace("!FLAG!", &name));
}
}
TEMPLATE_FULL
.replace("!OPTS!", &opts)
.replace("!CASES!", &cases)
.trim_start()
.to_string()
}

47
crates/core/flags/complete/fish.rs Normal file
View File

@ -0,0 +1,47 @@
/*!
Provides completions for ripgrep's CLI for the fish shell.
*/
use crate::flags::defs::FLAGS;
const TEMPLATE: &'static str =
"complete -c rg -n '__fish_use_subcommand' !SHORT! !LONG! !DOC!\n";
const TEMPLATE_CHOICES: &'static str =
"complete -c rg -n '__fish_use_subcommand' !SHORT! !LONG! !DOC! -r -f -a '!CHOICES!'\n";
/// Generate completions for Fish.
///
/// Note that these completions are based on what was produced for ripgrep <=13
/// using Clap 2.x. Improvements on this are welcome.
pub(crate) fn generate() -> String {
let mut out = String::new();
for flag in FLAGS.iter() {
let short = match flag.name_short() {
None => "".to_string(),
Some(byte) => format!("-s {}", char::from(byte)),
};
let long = format!("-l '{}'", flag.name_long().replace("'", "\\'"));
let doc = format!("-d '{}'", flag.doc_short().replace("'", "\\'"));
let template = if flag.doc_choices().is_empty() {
TEMPLATE.to_string()
} else {
TEMPLATE_CHOICES
.replace("!CHOICES!", &flag.doc_choices().join(" "))
};
out.push_str(
&template
.replace("!SHORT!", &short)
.replace("!LONG!", &long)
.replace("!DOC!", &doc),
);
if let Some(negated) = flag.name_negated() {
out.push_str(
&template
.replace("!SHORT!", "")
.replace("!LONG!", &negated)
.replace("!DOC!", &doc),
);
}
}
out
}

8
crates/core/flags/complete/mod.rs Normal file
View File

@ -0,0 +1,8 @@
/*!
Modules for generating completions for various shells.
*/
pub(super) mod bash;
pub(super) mod fish;
pub(super) mod powershell;
pub(super) mod zsh;

86
crates/core/flags/complete/powershell.rs Normal file
View File

@ -0,0 +1,86 @@
/*!
Provides completions for ripgrep's CLI for PowerShell.
*/
use crate::flags::defs::FLAGS;
const TEMPLATE: &'static str = "
using namespace System.Management.Automation
using namespace System.Management.Automation.Language
Register-ArgumentCompleter -Native -CommandName 'rg' -ScriptBlock {
param($wordToComplete, $commandAst, $cursorPosition)
$commandElements = $commandAst.CommandElements
$command = @(
'rg'
for ($i = 1; $i -lt $commandElements.Count; $i++) {
$element = $commandElements[$i]
if ($element -isnot [StringConstantExpressionAst] -or
$element.StringConstantType -ne [StringConstantType]::BareWord -or
$element.Value.StartsWith('-')) {
break
}
$element.Value
}) -join ';'
$completions = @(switch ($command) {
'rg' {
!FLAGS!
}
})
$completions.Where{ $_.CompletionText -like \"$wordToComplete*\" } |
Sort-Object -Property ListItemText
}
";
const TEMPLATE_FLAG: &'static str =
"[CompletionResult]::new('!DASH_NAME!', '!NAME!', [CompletionResultType]::ParameterName, '!DOC!')";
/// Generate completions for PowerShell.
///
/// Note that these completions are based on what was produced for ripgrep <=13
/// using Clap 2.x. Improvements on this are welcome.
pub(crate) fn generate() -> String {
let mut flags = String::new();
for (i, flag) in FLAGS.iter().enumerate() {
let doc = flag.doc_short().replace("'", "''");
let dash_name = format!("--{}", flag.name_long());
let name = flag.name_long();
if i > 0 {
flags.push('\n');
}
flags.push_str(" ");
flags.push_str(
&TEMPLATE_FLAG
.replace("!DASH_NAME!", &dash_name)
.replace("!NAME!", &name)
.replace("!DOC!", &doc),
);
if let Some(byte) = flag.name_short() {
let dash_name = format!("-{}", char::from(byte));
let name = char::from(byte).to_string();
flags.push_str("\n ");
flags.push_str(
&TEMPLATE_FLAG
.replace("!DASH_NAME!", &dash_name)
.replace("!NAME!", &name)
.replace("!DOC!", &doc),
);
}
if let Some(negated) = flag.name_negated() {
let dash_name = format!("--{}", negated);
flags.push_str("\n ");
flags.push_str(
&TEMPLATE_FLAG
.replace("!DASH_NAME!", &dash_name)
.replace("!NAME!", &negated)
.replace("!DOC!", &doc),
);
}
}
TEMPLATE.trim_start().replace("!FLAGS!", &flags)
}

26
complete/_rg → crates/core/flags/complete/rg.zsh
View File

@ -73,6 +73,7 @@ _rg() {
{-c,--count}'[only show count of matching lines for each file]'
'--count-matches[only show count of individual matches for each file]'
'--include-zero[include files with zero matches in summary]'
$no"--no-include-zero[don't include files with zero matches in summary]"
+ '(encoding)' # Encoding options
{-E+,--encoding=}'[specify text encoding of files to search]: :_rg_encodings'
@ -108,6 +109,15 @@ _rg() {
{-L,--follow}'[follow symlinks]'
$no"--no-follow[don't follow symlinks]"
+ '(generate)' # Options for generating ancillary data
'--generate=[generate man page or completion scripts]:when:((
man\:"man page"
complete-bash\:"shell completions for bash"
complete-zsh\:"shell completions for zsh"
complete-fish\:"shell completions for fish"
complete-powershell\:"shell completions for PowerShell"
))'
+ glob # File-glob options
'*'{-g+,--glob=}'[include/exclude files matching specified glob]:glob'
'*--iglob=[include/exclude files matching specified case-insensitive glob]:glob'
@ -125,8 +135,8 @@ _rg() {
$no"--no-hidden[don't search hidden files and directories]"
+ '(hybrid)' # hybrid regex options
'--auto-hybrid-regex[dynamically use PCRE2 if necessary]'
$no"--no-auto-hybrid-regex[don't dynamically use PCRE2 if necessary]"
'--auto-hybrid-regex[DEPRECATED: dynamically use PCRE2 if necessary]'
$no"--no-auto-hybrid-regex[DEPRECATED: don't dynamically use PCRE2 if necessary]"
+ '(ignore)' # Ignore-file options
"(--no-ignore-global --no-ignore-parent --no-ignore-vcs --no-ignore-dot)--no-ignore[don't respect ignore files]"
@ -183,6 +193,7 @@ _rg() {
+ '(max-depth)' # Directory-depth options
'--max-depth=[specify max number of directories to descend]:number of directories'
'--maxdepth=[alias for --max-depth]:number of directories'
'!--maxdepth=:number of directories'
+ '(messages)' # Error-message options
@ -210,15 +221,15 @@ _rg() {
+ '(passthru)' # Pass-through options
'(--vimgrep)--passthru[show both matching and non-matching lines]'
'!(--vimgrep)--passthrough'
'(--vimgrep)--passthrough[alias for --passthru]'
+ '(pcre2)' # PCRE2 options
{-P,--pcre2}'[enable matching with PCRE2]'
$no'(pcre2-unicode)--no-pcre2[disable matching with PCRE2]'
+ '(pcre2-unicode)' # PCRE2 Unicode options
$no'(--no-pcre2 --no-pcre2-unicode)--pcre2-unicode[enable PCRE2 Unicode mode (with -P)]'
'(--no-pcre2 --pcre2-unicode)--no-pcre2-unicode[disable PCRE2 Unicode mode (with -P)]'
$no'(--no-pcre2 --no-pcre2-unicode)--pcre2-unicode[DEPRECATED: enable PCRE2 Unicode mode (with -P)]'
'(--no-pcre2 --pcre2-unicode)--no-pcre2-unicode[DEPRECATED: disable PCRE2 Unicode mode (with -P)]'
+ '(pre)' # Preprocessing options
'(-z --search-zip)--pre=[specify preprocessor utility]:preprocessor utility:_command_names -e'
@ -252,7 +263,8 @@ _rg() {
accessed\:"sort by last accessed time"
created\:"sort by creation time"
))'
'!(threads)--sort-files[sort results by file path (disables parallelism)]'
'(threads)--sort-files[DEPRECATED: sort results by file path (disables parallelism)]'
$no"--no-sort-files[DEPRECATED: do not sort results]"
+ '(stats)' # Statistics options
'(--files file-match)--stats[show search statistics]'
@ -293,6 +305,7 @@ _rg() {
+ misc # Other options — no need to separate these at the moment
'(-b --byte-offset)'{-b,--byte-offset}'[show 0-based byte offset for each matching line]'
$no"--no-byte-offset[don't show byte offsets for each matching line]"
'--color=[specify when to use colors in output]:when:((
never\:"never use colors"
auto\:"use colors or not based on stdout, TERM, etc."
@ -312,6 +325,7 @@ _rg() {
"(1 stats)--files[show each file that would be searched (but don't search)]"
'*--ignore-file=[specify additional ignore file]:ignore file:_files'
'(-v --invert-match)'{-v,--invert-match}'[invert matching]'
$no"--no-invert-match[do not invert matching]"
'(-M --max-columns)'{-M+,--max-columns=}'[specify max length of lines to print]:number of bytes'
'(-m --max-count)'{-m+,--max-count=}'[specify max number of matches per file]:number of matches'
'--max-filesize=[specify size above which files should be ignored]:file size (bytes)'

23
crates/core/flags/complete/zsh.rs Normal file
View File

@ -0,0 +1,23 @@
/*!
Provides completions for ripgrep's CLI for the zsh shell.
Unlike completion short for other shells (at time of writing), zsh's
completions for ripgrep are maintained by hand. This is because:
1. They are lovingly written by an expert in such things.
2. Are much higher in quality than the ones below that are auto-generated.
Namely, the zsh completions take application level context about flag
compatibility into account.
3. There is a CI script that fails if a new flag is added to ripgrep that
isn't included in the zsh completions.
4. There is a wealth of documentation in the zsh script explaining how it
works and how it can be extended.
In principle, I'd be open to maintaining any completion script by hand so
long as it meets criteria 3 and 4 above.
*/
/// Generate completions for zsh.
pub(crate) fn generate() -> String {
include_str!("rg.zsh").to_string()
}

9
crates/core/config.rs → crates/core/flags/config.rs
View File

@ -1,6 +1,9 @@
// This module provides routines for reading ripgrep config "rc" files. The
// primary output of these routines is a sequence of arguments, where each
// argument corresponds precisely to one shell argument.
/*!
This module provides routines for reading ripgrep config "rc" files.
The primary output of these routines is a sequence of arguments, where each
argument corresponds precisely to one shell argument.
*/
use std::{
ffi::OsString,

7625
crates/core/flags/defs.rs Normal file
View File

File diff suppressed because it is too large Load Diff

259
crates/core/flags/doc/help.rs Normal file
View File

@ -0,0 +1,259 @@
/*!
Provides routines for generating ripgrep's "short" and "long" help
documentation.
The short version is used when the `-h` flag is given, while the long version
is used when the `--help` flag is given.
*/
use std::{collections::BTreeMap, fmt::Write};
use crate::flags::{defs::FLAGS, doc::version, Category, Flag};
const TEMPLATE_SHORT: &'static str = include_str!("template.short.help");
const TEMPLATE_LONG: &'static str = include_str!("template.long.help");
/// Wraps `std::write!` and asserts there is no failure.
///
/// We only write to `String` in this module.
macro_rules! write {
($($tt:tt)*) => { std::write!($($tt)*).unwrap(); }
}
/// Generate short documentation, i.e., for `-h`.
pub(crate) fn generate_short() -> String {
let mut cats: BTreeMap<Category, (Vec<String>, Vec<String>)> =
BTreeMap::new();
let (mut maxcol1, mut maxcol2) = (0, 0);
for flag in FLAGS.iter().copied() {
let columns =
cats.entry(flag.doc_category()).or_insert((vec![], vec![]));
let (col1, col2) = generate_short_flag(flag);
maxcol1 = maxcol1.max(col1.len());
maxcol2 = maxcol2.max(col2.len());
columns.0.push(col1);
columns.1.push(col2);
}
let mut out =
TEMPLATE_SHORT.replace("!!VERSION!!", &version::generate_digits());
for (cat, (col1, col2)) in cats.iter() {
let var = format!("!!{name}!!", name = cat.as_str());
let val = format_short_columns(col1, col2, maxcol1, maxcol2);
out = out.replace(&var, &val);
}
out
}
/// Generate short for a single flag.
///
/// The first element corresponds to the flag name while the second element
/// corresponds to the documentation string.
fn generate_short_flag(flag: &dyn Flag) -> (String, String) {
let (mut col1, mut col2) = (String::new(), String::new());
// Some of the variable names are fine for longer form
// docs, but they make the succinct short help very noisy.
// So just shorten some of them.
let var = flag.doc_variable().map(|s| {
let mut s = s.to_string();
s = s.replace("SEPARATOR", "SEP");
s = s.replace("REPLACEMENT", "TEXT");
s = s.replace("NUM+SUFFIX?", "NUM");
s
});
// Generate the first column, the flag name.
if let Some(byte) = flag.name_short() {
let name = char::from(byte);
write!(col1, r"-{name}");
write!(col1, r", ");
}
write!(col1, r"--{name}", name = flag.name_long());
if let Some(var) = var.as_ref() {
write!(col1, r"={var}");
}
// And now the second column, with the description.
write!(col2, "{}", flag.doc_short());
(col1, col2)
}
/// Write two columns of documentation.
///
/// `maxcol1` should be the maximum length (in bytes) of the first column,
/// while `maxcol2` should be the maximum length (in bytes) of the second
/// column.
fn format_short_columns(
col1: &[String],
col2: &[String],
maxcol1: usize,
_maxcol2: usize,
) -> String {
assert_eq!(col1.len(), col2.len(), "columns must have equal length");
const PAD: usize = 2;
let mut out = String::new();
for (i, (c1, c2)) in col1.iter().zip(col2.iter()).enumerate() {
if i > 0 {
write!(out, "\n");
}
let pad = maxcol1 - c1.len() + PAD;
write!(out, " ");
write!(out, "{c1}");
write!(out, "{}", " ".repeat(pad));
write!(out, "{c2}");
}
out
}
/// Generate long documentation, i.e., for `--help`.
pub(crate) fn generate_long() -> String {
let mut cats = BTreeMap::new();
for flag in FLAGS.iter().copied() {
let mut cat = cats.entry(flag.doc_category()).or_insert(String::new());
if !cat.is_empty() {
write!(cat, "\n\n");
}
generate_long_flag(flag, &mut cat);
}
let mut out =
TEMPLATE_LONG.replace("!!VERSION!!", &version::generate_digits());
for (cat, value) in cats.iter() {
let var = format!("!!{name}!!", name = cat.as_str());
out = out.replace(&var, value);
}
out
}
/// Write generated documentation for `flag` to `out`.
fn generate_long_flag(flag: &dyn Flag, out: &mut String) {
if let Some(byte) = flag.name_short() {
let name = char::from(byte);
write!(out, r" -{name}");
if let Some(var) = flag.doc_variable() {
write!(out, r" {var}");
}
write!(out, r", ");
} else {
write!(out, r" ");
}
let name = flag.name_long();
write!(out, r"--{name}");
if let Some(var) = flag.doc_variable() {
write!(out, r"={var}");
}
write!(out, "\n");
let doc = flag.doc_long().trim();
let doc = super::render_custom_markup(doc, "flag", |name, out| {
let Some(flag) = crate::flags::parse::lookup(name) else {
unreachable!(r"found unrecognized \flag{{{name}}} in --help docs")
};
if let Some(name) = flag.name_short() {
write!(out, r"-{}/", char::from(name));
}
write!(out, r"--{}", flag.name_long());
});
let doc = super::render_custom_markup(&doc, "flag-negate", |name, out| {
let Some(flag) = crate::flags::parse::lookup(name) else {
unreachable!(
r"found unrecognized \flag-negate{{{name}}} in --help docs"
)
};
let Some(name) = flag.name_negated() else {
let long = flag.name_long();
unreachable!(
"found \\flag-negate{{{long}}} in --help docs but \
{long} does not have a negation"
);
};
write!(out, r"--{name}");
});
let mut cleaned = remove_roff(&doc);
if let Some(negated) = flag.name_negated() {
// Flags that can be negated that aren't switches, like
// --context-separator, are somewhat weird. Because of that, the docs
// for those flags should discuss the semantics of negation explicitly.
// But for switches, the behavior is always the same.
if flag.is_switch() {
write!(cleaned, "\n\nThis flag can be disabled with --{negated}.");
}
}
let indent = " ".repeat(8);
let wrapopts = textwrap::Options::new(71)
// Normally I'd be fine with breaking at hyphens, but ripgrep's docs
// includes a lot of flag names, and they in turn contain hyphens.
// Breaking flag names across lines is not great.
.word_splitter(textwrap::WordSplitter::NoHyphenation);
for (i, paragraph) in cleaned.split("\n\n").enumerate() {
if i > 0 {
write!(out, "\n\n");
}
let mut new = paragraph.to_string();
if paragraph.lines().all(|line| line.starts_with(" ")) {
// Re-indent but don't refill so as to preserve line breaks
// in code/shell example snippets.
new = textwrap::indent(&new, &indent);
} else {
new = new.replace("\n", " ");
new = textwrap::refill(&new, &wrapopts);
new = textwrap::indent(&new, &indent);
}
write!(out, "{}", new.trim_end());
}
}
/// Removes roff syntax from `v` such that the result is approximately plain
/// text readable.
///
/// This is basically a mish mash of heuristics based on the specific roff used
/// in the docs for the flags in this tool. If new kinds of roff are used in
/// the docs, then this may need to be updated to handle them.
fn remove_roff(v: &str) -> String {
let mut lines = vec![];
for line in v.trim().lines() {
assert!(!line.is_empty(), "roff should have no empty lines");
if line.starts_with(".") {
if line.starts_with(".IP ") {
let item_label = line
.split(" ")
.nth(1)
.expect("first argument to .IP")
.replace(r"\(bu", r"•")
.replace(r"\fB", "")
.replace(r"\fP", ":");
lines.push(format!("{item_label}"));
} else if line.starts_with(".IB ") || line.starts_with(".BI ") {
let pieces = line
.split_whitespace()
.skip(1)
.collect::<Vec<_>>()
.concat();
lines.push(format!("{pieces}"));
} else if line.starts_with(".sp")
|| line.starts_with(".PP")
|| line.starts_with(".TP")
{
lines.push("".to_string());
}
} else if line.starts_with(r"\fB") && line.ends_with(r"\fP") {
let line = line.replace(r"\fB", "").replace(r"\fP", "");
lines.push(format!("{line}:"));
} else {
lines.push(line.to_string());
}
}
// Squash multiple adjacent paragraph breaks into one.
lines.dedup_by(|l1, l2| l1.is_empty() && l2.is_empty());
lines
.join("\n")
.replace(r"\fB", "")
.replace(r"\fI", "")
.replace(r"\fP", "")
.replace(r"\-", "-")
.replace(r"\\", r"\")
}

110
crates/core/flags/doc/man.rs Normal file
View File

@ -0,0 +1,110 @@
/*!
Provides routines for generating ripgrep's man page in `roff` format.
*/
use std::{collections::BTreeMap, fmt::Write};
use crate::flags::{defs::FLAGS, doc::version, Flag};
const TEMPLATE: &'static str = include_str!("template.rg.1");
/// Wraps `std::write!` and asserts there is no failure.
///
/// We only write to `String` in this module.
macro_rules! write {
($($tt:tt)*) => { std::write!($($tt)*).unwrap(); }
}
/// Wraps `std::writeln!` and asserts there is no failure.
///
/// We only write to `String` in this module.
macro_rules! writeln {
($($tt:tt)*) => { std::writeln!($($tt)*).unwrap(); }
}
/// Returns a `roff` formatted string corresponding to ripgrep's entire man
/// page.
pub(crate) fn generate() -> String {
let mut cats = BTreeMap::new();
for flag in FLAGS.iter().copied() {
let mut cat = cats.entry(flag.doc_category()).or_insert(String::new());
if !cat.is_empty() {
writeln!(cat, ".sp");
}
generate_flag(flag, &mut cat);
}
let mut out = TEMPLATE.replace("!!VERSION!!", &version::generate_digits());
for (cat, value) in cats.iter() {
let var = format!("!!{name}!!", name = cat.as_str());
out = out.replace(&var, value);
}
out
}
/// Writes `roff` formatted documentation for `flag` to `out`.
fn generate_flag(flag: &'static dyn Flag, out: &mut String) {
if let Some(byte) = flag.name_short() {
let name = char::from(byte);
write!(out, r"\fB\-{name}\fP");
if let Some(var) = flag.doc_variable() {
write!(out, r" \fI{var}\fP");
}
write!(out, r", ");
}
let name = flag.name_long();
write!(out, r"\fB\-\-{name}\fP");
if let Some(var) = flag.doc_variable() {
write!(out, r"=\fI{var}\fP");
}
write!(out, "\n");
writeln!(out, ".RS 4");
let doc = flag.doc_long().trim();
// Convert \flag{foo} into something nicer.
let doc = super::render_custom_markup(doc, "flag", |name, out| {
let Some(flag) = crate::flags::parse::lookup(name) else {
unreachable!(r"found unrecognized \flag{{{name}}} in roff docs")
};
out.push_str(r"\fB");
if let Some(name) = flag.name_short() {
write!(out, r"\-{}/", char::from(name));
}
write!(out, r"\-\-{}", flag.name_long());
out.push_str(r"\fP");
});
// Convert \flag-negate{foo} into something nicer.
let doc = super::render_custom_markup(&doc, "flag-negate", |name, out| {
let Some(flag) = crate::flags::parse::lookup(name) else {
unreachable!(
r"found unrecognized \flag-negate{{{name}}} in roff docs"
)
};
let Some(name) = flag.name_negated() else {
let long = flag.name_long();
unreachable!(
"found \\flag-negate{{{long}}} in roff docs but \
{long} does not have a negation"
);
};
out.push_str(r"\fB");
write!(out, r"\-\-{name}");
out.push_str(r"\fP");
});
writeln!(out, "{doc}");
if let Some(negated) = flag.name_negated() {
// Flags that can be negated that aren't switches, like
// --context-separator, are somewhat weird. Because of that, the docs
// for those flags should discuss the semantics of negation explicitly.
// But for switches, the behavior is always the same.
if flag.is_switch() {
writeln!(out, ".sp");
writeln!(
out,
r"This flag can be disabled with \fB\-\-{negated}\fP."
);
}
}
writeln!(out, ".RE");
}

38
crates/core/flags/doc/mod.rs Normal file
View File

@ -0,0 +1,38 @@
/*!
Modules for generating documentation for ripgrep's flags.
*/
pub(crate) mod help;
pub(crate) mod man;
pub(crate) mod version;
/// Searches for `\tag{...}` occurrences in `doc` and calls `replacement` for
/// each such tag found.
///
/// The first argument given to `replacement` is the tag value, `...`. The
/// second argument is the buffer that accumulates the full replacement text.
///
/// Since this function is only intended to be used on doc strings written into
/// the program source code, callers should panic in `replacement` if there are
/// any errors or unexpected circumstances.
fn render_custom_markup(
mut doc: &str,
tag: &str,
mut replacement: impl FnMut(&str, &mut String),
) -> String {
let mut out = String::with_capacity(doc.len());
let tag_prefix = format!(r"\{tag}{{");
while let Some(offset) = doc.find(&tag_prefix) {
out.push_str(&doc[..offset]);
let start = offset + tag_prefix.len();
let Some(end) = doc[start..].find('}').map(|i| start + i) else {
unreachable!(r"found {tag_prefix} without closing }}");
};
let name = &doc[start..end];
replacement(name, &mut out);
doc = &doc[end + 1..];
}
out.push_str(doc);
out
}

61
crates/core/flags/doc/template.long.help Normal file
View File

@ -0,0 +1,61 @@
ripgrep !!VERSION!!
Andrew Gallant <jamslam@gmail.com>
ripgrep (rg) recursively searches the current directory for a regex pattern.
By default, ripgrep will respect gitignore rules and automatically skip hidden
files/directories and binary files.
Use -h for short descriptions and --help for more details.
Project home page: https://github.com/BurntSushi/ripgrep
USAGE:
rg [OPTIONS] PATTERN [PATH ...]
rg [OPTIONS] -e PATTERN ... [PATH ...]
rg [OPTIONS] -f PATTERNFILE ... [PATH ...]
rg [OPTIONS] --files [PATH ...]
rg [OPTIONS] --type-list
command | rg [OPTIONS] PATTERN
rg [OPTIONS] --help
rg [OPTIONS] --version
POSITIONAL ARGUMENTS:
<PATTERN>
A regular expression used for searching. To match a pattern beginning
with a dash, use the -e/--regexp flag.
For example, to search for the literal '-foo', you can use this flag:
rg -e -foo
You can also use the special '--' delimiter to indicate that no more
flags will be provided. Namely, the following is equivalent to the
above:
rg -- -foo
<PATH>...
A file or directory to search. Directories are searched recursively.
File paths specified on the command line override glob and ignore
rules.
INPUT OPTIONS:
!!input!!
SEARCH OPTIONS:
!!search!!
FILTER OPTIONS:
!!filter!!
OUTPUT OPTIONS:
!!output!!
OUTPUT MODES:
!!output-modes!!
LOGGING OPTIONS:
!!logging!!
OTHER BEHAVIORS:
!!other-behaviors!!

415
crates/core/flags/doc/template.rg.1 Normal file
View File

@ -0,0 +1,415 @@
.TH RG 1 2023-11-13 "!!VERSION!!" "User Commands"
.
.
.SH NAME
rg \- recursively search the current directory for lines matching a pattern
.
.
.SH SYNOPSIS
.\" I considered using GNU troff's .SY and .YS "synopsis" macros here, but it
.\" looks like they aren't portable. Specifically, they don't appear to be in
.\" BSD's mdoc used on macOS.
.sp
\fBrg\fP [\fIOPTIONS\fP] \fIPATTERN\fP [\fIPATH\fP...]
.sp
\fBrg\fP [\fIOPTIONS\fP] \fB\-e\fP \fIPATTERN\fP... [\fIPATH\fP...]
.sp
\fBrg\fP [\fIOPTIONS\fP] \fB\-f\fP \fIPATTERNFILE\fP... [\fIPATH\fP...]
.sp
\fBrg\fP [\fIOPTIONS\fP] \fB\-\-files\fP [\fIPATH\fP...]
.sp
\fBrg\fP [\fIOPTIONS\fP] \fB\-\-type\-list\fP
.sp
\fIcommand\fP | \fBrg\fP [\fIOPTIONS\fP] \fIPATTERN\fP
.sp
\fBrg\fP [\fIOPTIONS\fP] \fB\-\-help\fP
.sp
\fBrg\fP [\fIOPTIONS\fP] \fB\-\-version\fP
.
.
.SH DESCRIPTION
ripgrep (rg) recursively searches the current directory for a regex pattern.
By default, ripgrep will respect your \fB.gitignore\fP and automatically skip
hidden files/directories and binary files.
.sp
ripgrep's default regex engine uses finite automata and guarantees linear
time searching. Because of this, features like backreferences and arbitrary
look-around are not supported. However, if ripgrep is built with PCRE2,
then the \fB\-P/\-\-pcre2\fP flag can be used to enable backreferences and
look-around.
.sp
ripgrep supports configuration files. Set \fBRIPGREP_CONFIG_PATH\fP to a
configuration file. The file can specify one shell argument per line. Lines
starting with \fB#\fP are ignored. For more details, see \fBCONFIGURATION
FILES\fP below.
.sp
ripgrep will automatically detect if stdin exists and search stdin for a regex
pattern, e.g. \fBls | rg foo\fP. In some environments, stdin may exist when
it shouldn't. To turn off stdin detection, one can explicitly specify the
directory to search, e.g. \fBrg foo ./\fP.
.sp
Tip: to disable all smart filtering and make ripgrep behave a bit more like
classical grep, use \fBrg -uuu\fP.
.
.
.SH REGEX SYNTAX
ripgrep uses Rust's regex engine by default, which documents its syntax:
\fIhttps://docs.rs/regex/1.*/regex/#syntax\fP
.sp
ripgrep uses byte-oriented regexes, which has some additional documentation:
\fIhttps://docs.rs/regex/1.*/regex/bytes/index.html#syntax\fP
.sp
To a first approximation, ripgrep uses Perl-like regexes without look-around or
backreferences. This makes them very similar to the "extended" (ERE) regular
expressions supported by *egrep*, but with a few additional features like
Unicode character classes.
.sp
If you're using ripgrep with the \fB\-P/\-\-pcre2\fP flag, then please consult
\fIhttps://www.pcre.org\fP or the PCRE2 man pages for documentation on the
supported syntax.
.
.
.SH POSITIONAL ARGUMENTS
.TP 12
\fIPATTERN\fP
A regular expression used for searching. To match a pattern beginning with a
dash, use the \fB\-e/\-\-regexp\fP option.
.TP 12
\fIPATH\fP
A file or directory to search. Directories are searched recursively. File paths
specified explicitly on the command line override glob and ignore rules.
.
.
.SH OPTIONS
This section documents all flags that ripgrep accepts. Flags are grouped into
categories below according to their function.
.sp
Note that many options can be turned on and off. In some cases, those flags are
not listed explicitly below. For example, the \fB\-\-column\fP flag (listed
below) enables column numbers in ripgrep's output, but the \fB\-\-no\-column\fP
flag (not listed below) disables them. The reverse can also exist. For example,
the \fB\-\-no\-ignore\fP flag (listed below) disables ripgrep's \fBgitignore\fP
logic, but the \fB\-\-ignore\fP flag (not listed below) enables it. These
flags are useful for overriding a ripgrep configuration file (or alias) on the
command line. Each flag's documentation notes whether an inverted flag exists.
In all cases, the flag specified last takes precedence.
.
.SS INPUT OPTIONS
!!input!!
.
.SS SEARCH OPTIONS
!!search!!
.
.SS FILTER OPTIONS
!!filter!!
.
.SS OUTPUT OPTIONS
!!output!!
.
.SS OUTPUT MODES
!!output-modes!!
.
.SS LOGGING OPTIONS
!!logging!!
.
.SS OTHER BEHAVIORS
!!other-behaviors!!
.
.
.SH EXIT STATUS
If ripgrep finds a match, then the exit status of the program is \fB0\fP.
If no match could be found, then the exit status is \fB1\fP. If an error
occurred, then the exit status is always \fB2\fP unless ripgrep was run with
the \fB\-q/\-\-quiet\fP flag and a match was found. In summary:
.sp
.IP \(bu 3n
\fB0\fP exit status occurs only when at least one match was found, and if
no error occurred, unless \fB\-q/\-\-quiet\fP was given.
.
.IP \(bu 3n
\fB1\fP exit status occurs only when no match was found and no error occurred.
.
.IP \(bu 3n
\fB2\fP exit status occurs when an error occurred. This is true for both
catastrophic errors (e.g., a regex syntax error) and for soft errors (e.g.,
unable to read a file).
.
.
.SH AUTOMATIC FILTERING
ripgrep does a fair bit of automatic filtering by default. This section
describes that filtering and how to control it.
.sp
\fBTIP\fP: To disable automatic filtering, use \fBrg -uuu\fP.
.sp
ripgrep's automatic "smart" filtering is one of the most apparent
differentiating features between ripgrep and other tools like \fBgrep\fP. As
such, its behavior may be surprising to users that aren't expecting it.
.sp
ripgrep does four types of filtering automatically:
.sp
.
.IP 1. 3n
Files and directories that match ignore rules are not searched.
.IP 2. 3n
Hidden files and directories are not searched.
.IP 3. 3n
Binary files (files with a \fBNUL\fP byte) are not searched.
.IP 4. 3n
Symbolic links are not followed.
.PP
The first type of filtering is the most sophisticated. ripgrep will attempt to
respect your \fBgitignore\fP rules as faithfully as possible. In particular,
this includes the following:
.
.IP \(bu 3n
Any global rules, e.g., in \fB$HOME/.config/git/ignore\fP.
.
.IP \(bu 3n
Any rules in relevant \fB.gitignore\fP files.
.
.IP \(bu 3n
Any local rules, e.g., in \fB.git/info/exclude\fP.
.PP
In some cases, ripgrep and \fBgit\fP will not always be in sync in terms
of which files are ignored. For example, a file that is ignored via
\fB.gitignore\fP but is tracked by \fBgit\fP would not be searched by ripgrep
even though \fBgit\fP tracks it. This is unlikely to ever be fixed. Instead,
you should either make sure your exclude rules match the files you track
precisely, or otherwise use \fBgit grep\fP for search.
.sp
Additional ignore rules can be provided outside of a \fBgit\fP context:
.
.IP \(bu 3n
Any rules in \fB.ignore\fP.
.
.IP \(bu 3n
Any rules in \fB.rgignore\fP.
.
.IP \(bu 3n
Any rules in files specified with the \fB\-\-ignore\-file\fP flag.
.PP
The precedence of ignore rules is as follows, with later items overriding
earlier items:
.
.IP \(bu 3n
Files given by \fB\-\-ignore\-file\fP.
.
.IP \(bu 3n
Global gitignore rules, e.g., from \fB$HOME/.config/git/ignore\fP.
.
.IP \(bu 3n
Local rules from \fB.git/info/exclude\fP.
.
.IP \(bu 3n
Rules from \fB.gitignore\fP.
.
.IP \(bu 3n
Rules from \fB.ignore\fP.
.
.IP \(bu 3n
Rules from \fB.rgignore\fP.
.PP
So for example, if \fIfoo\fP were in a \fB.gitignore\fP and \fB!\fP\fIfoo\fP
were in an \fB.rgignore\fP, then \fIfoo\fP would not be ignored since
\fB.rgignore\fP takes precedence over \fB.gitignore\fP.
.sp
Each of the types of filtering can be configured via command line flags:
.
.IP \(bu 3n
There are several flags starting with \fB\-\-no\-ignore\fP that toggle which,
if any, ignore rules are respected. \fB\-\-no\-ignore\fP by itself will disable
all
of them.
.
.IP \(bu 3n
\fB\-./\-\-hidden\fP will force ripgrep to search hidden files and directories.
.
.IP \(bu 3n
\fB\-\-binary\fP will force ripgrep to search binary files.
.
.IP \(bu 3n
\fB\-L/\-\-follow\fP will force ripgrep to follow symlinks.
.PP
As a special short hand, the \fB\-u\fP flag can be specified up to three times.
Each additional time incrementally decreases filtering:
.
.IP \(bu 3n
\fB\-u\fP is equivalent to \fB\-\-no\-ignore\fP.
.
.IP \(bu 3n
\fB\-uu\fP is equivalent to \fB\-\-no\-ignore \-\-hidden\fP.
.
.IP \(bu 3n
\fB\-uuu\fP is equivalent to \fB\-\-no\-ignore \-\-hidden \-\-binary\fP.
.PP
In particular, \fBrg -uuu\fP should search the same exact content as \fBgrep
-r\fP.
.
.
.SH CONFIGURATION FILES
ripgrep supports reading configuration files that change ripgrep's default
behavior. The format of the configuration file is an "rc" style and is very
simple. It is defined by two rules:
.
.IP 1. 3n
Every line is a shell argument, after trimming whitespace.
.
.IP 2. 3n
Lines starting with \fB#\fP (optionally preceded by any amount of whitespace)
are ignored.
.PP
ripgrep will look for a single configuration file if and only if the
\fBRIPGREP_CONFIG_PATH\fP environment variable is set and is non-empty.
ripgrep will parse arguments from this file on startup and will behave as if
the arguments in this file were prepended to any explicit arguments given to
ripgrep on the command line. Note though that the \fBrg\fP command you run
must still be valid. That is, it must always contain at least one pattern at
the command line, even if the configuration file uses the \fB\-e/\-\-regexp\fP
flag.
.sp
For example, if your ripgreprc file contained a single line:
.sp
.EX
\-\-smart\-case
.EE
.sp
then the following command
.sp
.EX
RIPGREP_CONFIG_PATH=wherever/.ripgreprc rg foo
.EE
.sp
would behave identically to the following command:
.sp
.EX
rg \-\-smart-case foo
.EE
.sp
Another example is adding types, like so:
.sp
.EX
\-\-type-add
web:*.{html,css,js}*
.EE
.sp
The above would behave identically to the following command:
.sp
.EX
rg \-\-type\-add 'web:*.{html,css,js}*' foo
.EE
.sp
The same applies to using globs. This:
.sp
.EX
\-\-glob=!.git
.EE
.sp
or this:
.sp
.EX
\-\-glob
!.git
.EE
.sp
would behave identically to the following command:
.sp
.EX
rg \-\-glob '!.git' foo
.EE
.sp
The bottom line is that every shell argument needs to be on its own line. So
for example, a config file containing
.sp
.EX
\-j 4
.EE
.sp
is probably not doing what you intend. Instead, you want
.sp
.EX
\-j
4
.EE
.sp
or
.sp
.EX
\-j4
.EE
.sp
ripgrep also provides a flag, \fB\-\-no\-config\fP, that when present will
suppress any and all support for configuration. This includes any future
support for auto-loading configuration files from pre-determined paths.
.sp
Conflicts between configuration files and explicit arguments are handled
exactly like conflicts in the same command line invocation. That is, assuming
your config file contains only \fB\-\-smart\-case\fP, then this command:
.sp
.EX
RIPGREP_CONFIG_PATH=wherever/.ripgreprc rg foo \-\-case\-sensitive
.EE
.sp
is exactly equivalent to
.sp
.EX
rg \-\-smart\-case foo \-\-case\-sensitive
.EE
.sp
in which case, the \fB\-\-case\-sensitive\fP flag would override the
\fB\-\-smart\-case\fP flag.
.
.
.SH SHELL COMPLETION
Shell completion files are included in the release tarball for Bash, Fish, Zsh
and PowerShell.
.sp
For \fBbash\fP, move \fBrg.bash\fP to \fB$XDG_CONFIG_HOME/bash_completion\fP or
\fB/etc/bash_completion.d/\fP.
.sp
For \fBfish\fP, move \fBrg.fish\fP to \fB$HOME/.config/fish/completions\fP.
.sp
For \fBzsh\fP, move \fB_rg\fP to one of your \fB$fpath\fP directories.
.
.
.SH CAVEATS
ripgrep may abort unexpectedly when using default settings if it searches a
file that is simultaneously truncated. This behavior can be avoided by passing
the \fB\-\-no\-mmap\fP flag which will forcefully disable the use of memory
maps in all cases.
.sp
ripgrep may use a large amount of memory depending on a few factors. Firstly,
if ripgrep uses parallelism for search (the default), then the entire
output for each individual file is buffered into memory in order to prevent
interleaving matches in the output. To avoid this, you can disable parallelism
with the \fB\-j1\fP flag. Secondly, ripgrep always needs to have at least a
single line in memory in order to execute a search. A file with a very long
line can thus cause ripgrep to use a lot of memory. Generally, this only occurs
when searching binary data with the \fB\-a/\-\-text\fP flag enabled. (When the
\fB\-a/\-\-text\fP flag isn't enabled, ripgrep will replace all NUL bytes with
line terminators, which typically prevents exorbitant memory usage.) Thirdly,
when ripgrep searches a large file using a memory map, the process will likely
report its resident memory usage as the size of the file. However, this does
not mean ripgrep actually needed to use that much heap memory; the operating
system will generally handle this for you.
.
.
.SH VERSION
!!VERSION!!
.
.
.SH HOMEPAGE
\fIhttps://github.com/BurntSushi/ripgrep\fP
.sp
Please report bugs and feature requests to the issue tracker. Please do your
best to provide a reproducible test case for bugs. This should include the
corpus being searched, the \fBrg\fP command, the actual output and the expected
output. Please also include the output of running the same \fBrg\fP command but
with the \fB\-\-debug\fP flag.
.sp
If you have questions that don't obviously fall into the "bug" or "feature
request" category, then they are welcome in the Discussions section of the
issue tracker: \fIhttps://github.com/BurntSushi/ripgrep/discussions\fP.
.
.
.SH AUTHORS
Andrew Gallant <\fIjamslam@gmail.com\fP>

38
crates/core/flags/doc/template.short.help Normal file
View File

@ -0,0 +1,38 @@
ripgrep !!VERSION!!
Andrew Gallant <jamslam@gmail.com>
ripgrep (rg) recursively searches the current directory for a regex pattern.
By default, ripgrep will respect gitignore rules and automatically skip hidden
files/directories and binary files.
Use -h for short descriptions and --help for more details.
Project home page: https://github.com/BurntSushi/ripgrep
USAGE:
rg [OPTIONS] PATTERN [PATH ...]
POSITIONAL ARGUMENTS:
<PATTERN> A regular expression used for searching.
<PATH>... A file or directory to search.
INPUT OPTIONS:
!!input!!
SEARCH OPTIONS:
!!search!!
FILTER OPTIONS:
!!filter!!
OUTPUT OPTIONS:
!!output!!
OUTPUT MODES:
!!output-modes!!
LOGGING OPTIONS:
!!logging!!
OTHER BEHAVIORS:
!!other-behaviors!!

148
crates/core/flags/doc/version.rs Normal file
View File

@ -0,0 +1,148 @@
/*!
Provides routines for generating version strings.
Version strings can be just the digits, an overall short one-line description
or something more verbose that includes things like CPU target feature support.
*/
use std::fmt::Write;
/// Generates just the numerical part of the version of ripgrep.
///
/// This includes the git revision hash.
pub(crate) fn generate_digits() -> String {
let semver = option_env!("CARGO_PKG_VERSION").unwrap_or("N/A");
match option_env!("RIPGREP_BUILD_GIT_HASH") {
None => semver.to_string(),
Some(hash) => format!("{semver} (rev {hash})"),
}
}
/// Generates a short version string of the form `ripgrep x.y.z`.
pub(crate) fn generate_short() -> String {
let digits = generate_digits();
format!("ripgrep {digits}")
}
/// Generates a longer multi-line version string.
///
/// This includes not only the version of ripgrep but some other information
/// about its build. For example, SIMD support and PCRE2 support.
pub(crate) fn generate_long() -> String {
let (compile, runtime) = (compile_cpu_features(), runtime_cpu_features());
let mut out = String::new();
writeln!(out, "{}", generate_short()).unwrap();
writeln!(out, "features:{}", features().join(",")).unwrap();
if !compile.is_empty() {
writeln!(out, "simd(compile):{}", compile.join(",")).unwrap();
}
if !runtime.is_empty() {
writeln!(out, "simd(runtime):{}", runtime.join(",")).unwrap();
}
out
}
/// Returns the relevant SIMD features supported by the CPU at runtime.
///
/// This is kind of a dirty violation of abstraction, since it assumes
/// knowledge about what specific SIMD features are being used by various
/// components.
fn runtime_cpu_features() -> Vec<String> {
#[cfg(target_arch = "x86_64")]
{
let mut features = vec![];
let sse2 = is_x86_feature_detected!("sse2");
features.push(format!("{sign}SSE2", sign = sign(sse2)));
let ssse3 = is_x86_feature_detected!("ssse3");
features.push(format!("{sign}SSSE3", sign = sign(ssse3)));
let avx2 = is_x86_feature_detected!("avx2");
features.push(format!("{sign}AVX2", sign = sign(avx2)));
features
}
#[cfg(target_arch = "aarch64")]
{
let mut features = vec![];
// memchr and aho-corasick only use NEON when it is available at
// compile time. This isn't strictly necessary, but NEON is supposed
// to be available for all aarch64 targets. If this isn't true, please
// file an issue at https://github.com/BurntSushi/memchr.
let neon = cfg!(target_feature = "neon");
features.push(format!("{sign}NEON", sign = sign(neon)));
features
}
#[cfg(not(any(target_arch = "x86_64", target_arch = "aarch64")))]
{
vec![]
}
}
/// Returns the SIMD features supported while compiling ripgrep.
///
/// In essence, any features listed here are required to run ripgrep correctly.
///
/// This is kind of a dirty violation of abstraction, since it assumes
/// knowledge about what specific SIMD features are being used by various
/// components.
///
/// An easy way to enable everything available on your current CPU is to
/// compile ripgrep with `RUSTFLAGS="-C target-cpu=native"`. But note that
/// the binary produced by this will not be portable.
fn compile_cpu_features() -> Vec<String> {
#[cfg(target_arch = "x86_64")]
{
let mut features = vec![];
let sse2 = cfg!(target_feature = "sse2");
features.push(format!("{sign}SSE2", sign = sign(sse2)));
let ssse3 = cfg!(target_feature = "ssse3");
features.push(format!("{sign}SSSE3", sign = sign(ssse3)));
let avx2 = cfg!(target_feature = "avx2");
features.push(format!("{sign}AVX2", sign = sign(avx2)));
features
}
#[cfg(target_arch = "aarch64")]
{
let mut features = vec![];
let neon = cfg!(target_feature = "neon");
features.push(format!("{sign}NEON", sign = sign(neon)));
features
}
#[cfg(not(any(target_arch = "x86_64", target_arch = "aarch64")))]
{
vec![]
}
}
/// Returns a list of "features" supported (or not) by this build of ripgrpe.
fn features() -> Vec<String> {
let mut features = vec![];
let simd_accel = cfg!(feature = "simd-accel");
features.push(format!("{sign}simd-accel", sign = sign(simd_accel)));
let pcre2 = cfg!(feature = "pcre2");
features.push(format!("{sign}pcre2", sign = sign(pcre2)));
features
}
/// Returns `+` when `enabled` is `true` and `-` otherwise.
fn sign(enabled: bool) -> &'static str {
if enabled {
"+"
} else {
"-"
}
}

1409
crates/core/flags/hiargs.rs Normal file
View File

File diff suppressed because it is too large Load Diff

758
crates/core/flags/lowargs.rs Normal file
View File

@ -0,0 +1,758 @@
/*!
Provides the definition of low level arguments from CLI flags.
*/
use std::{
ffi::{OsStr, OsString},
path::PathBuf,
};
use {
bstr::{BString, ByteVec},
grep::printer::{HyperlinkFormat, UserColorSpec},
};
/// A collection of "low level" arguments.
///
/// The "low level" here is meant to constrain this type to be as close to the
/// actual CLI flags and arguments as possible. Namely, other than some
/// convenience types to help validate flag values and deal with overrides
/// between flags, these low level arguments do not contain any higher level
/// abstractions.
///
/// Another self-imposed constraint is that populating low level arguments
/// should not require anything other than validating what the user has
/// provided. For example, low level arguments should not contain a
/// `HyperlinkConfig`, since in order to get a full configuration, one needs to
/// discover the hostname of the current system (which might require running a
/// binary or a syscall).
///
/// Low level arguments are populated by the parser directly via the `update`
/// method on the corresponding implementation of the `Flag` trait.
#[derive(Debug, Default)]
pub(crate) struct LowArgs {
// Essential arguments.
pub(crate) special: Option<SpecialMode>,
pub(crate) mode: Mode,
pub(crate) positional: Vec<OsString>,
pub(crate) patterns: Vec<PatternSource>,
// Everything else, sorted lexicographically.
pub(crate) binary: BinaryMode,
pub(crate) boundary: Option<BoundaryMode>,
pub(crate) buffer: BufferMode,
pub(crate) byte_offset: bool,
pub(crate) case: CaseMode,
pub(crate) color: ColorChoice,
pub(crate) colors: Vec<UserColorSpec>,
pub(crate) column: Option<bool>,
pub(crate) context: ContextMode,
pub(crate) context_separator: ContextSeparator,
pub(crate) crlf: bool,
pub(crate) dfa_size_limit: Option<usize>,
pub(crate) encoding: EncodingMode,
pub(crate) engine: EngineChoice,
pub(crate) field_context_separator: FieldContextSeparator,
pub(crate) field_match_separator: FieldMatchSeparator,
pub(crate) fixed_strings: bool,
pub(crate) follow: bool,
pub(crate) glob_case_insensitive: bool,
pub(crate) globs: Vec<String>,
pub(crate) heading: Option<bool>,
pub(crate) hidden: bool,
pub(crate) hostname_bin: Option<PathBuf>,
pub(crate) hyperlink_format: HyperlinkFormat,
pub(crate) iglobs: Vec<String>,
pub(crate) ignore_file: Vec<PathBuf>,
pub(crate) ignore_file_case_insensitive: bool,
pub(crate) include_zero: bool,
pub(crate) invert_match: bool,
pub(crate) line_number: Option<bool>,
pub(crate) logging: Option<LoggingMode>,
pub(crate) max_columns: Option<u64>,
pub(crate) max_columns_preview: bool,
pub(crate) max_count: Option<u64>,
pub(crate) max_depth: Option<usize>,
pub(crate) max_filesize: Option<u64>,
pub(crate) mmap: MmapMode,
pub(crate) multiline: bool,
pub(crate) multiline_dotall: bool,
pub(crate) no_config: bool,
pub(crate) no_ignore_dot: bool,
pub(crate) no_ignore_exclude: bool,
pub(crate) no_ignore_files: bool,
pub(crate) no_ignore_global: bool,
pub(crate) no_ignore_messages: bool,
pub(crate) no_ignore_parent: bool,
pub(crate) no_ignore_vcs: bool,
pub(crate) no_messages: bool,
pub(crate) no_require_git: bool,
pub(crate) no_unicode: bool,
pub(crate) null: bool,
pub(crate) null_data: bool,
pub(crate) one_file_system: bool,
pub(crate) only_matching: bool,
pub(crate) path_separator: Option<u8>,
pub(crate) pre: Option<PathBuf>,
pub(crate) pre_glob: Vec<String>,
pub(crate) quiet: bool,
pub(crate) regex_size_limit: Option<usize>,
pub(crate) replace: Option<BString>,
pub(crate) search_zip: bool,
pub(crate) sort: Option<SortMode>,
pub(crate) stats: bool,
pub(crate) stop_on_nonmatch: bool,
pub(crate) threads: Option<usize>,
pub(crate) trim: bool,
pub(crate) type_changes: Vec<TypeChange>,
pub(crate) unrestricted: usize,
pub(crate) vimgrep: bool,
pub(crate) with_filename: Option<bool>,
}
/// A "special" mode that supercedes everything else.
///
/// When one of these modes is present, it overrides everything else and causes
/// ripgrep to short-circuit. In particular, we avoid converting low-level
/// argument types into higher level arguments types that can fail for various
/// reasons related to the environment. (Parsing the low-level arguments can
/// fail too, but usually not in a way that can't be worked around by removing
/// the corresponding arguments from the CLI command.) This is overall a hedge
/// to ensure that version and help information are basically always available.
#[derive(Clone, Copy, Debug, Eq, PartialEq)]
pub(crate) enum SpecialMode {
/// Show a condensed version of "help" output. Generally speaking, this
/// shows each flag and an extremely terse description of that flag on
/// a single line. This corresponds to the `-h` flag.
HelpShort,
/// Shows a very verbose version of the "help" output. The docs for some
/// flags will be paragraphs long. This corresponds to the `--help` flag.
HelpLong,
/// Show condensed version information. e.g., `ripgrep x.y.z`.
VersionShort,
/// Show verbose version information. Includes "short" information as well
/// as features included in the build.
VersionLong,
/// Show PCRE2's version information, or an error if this version of
/// ripgrep wasn't compiled with PCRE2 support.
VersionPCRE2,
}
/// The overall mode that ripgrep should operate in.
///
/// If ripgrep were designed without the legacy of grep, these would probably
/// be sub-commands? Perhaps not, since they aren't as frequently used.
///
/// The point of putting these in one enum is that they are all mutually
/// exclusive and override one another.
///
/// Note that -h/--help and -V/--version are not included in this because
/// they always overrides everything else, regardless of where it appears
/// in the command line. They are treated as "special" modes that short-circuit
/// ripgrep's usual flow.
#[derive(Clone, Copy, Debug, Eq, PartialEq)]
pub(crate) enum Mode {
/// ripgrep will execute a search of some kind.
Search(SearchMode),
/// Show the files that *would* be searched, but don't actually search
/// them.
Files,
/// List all file type definitions configured, including the default file
/// types and any additional file types added to the command line.
Types,
/// Generate various things like the man page and completion files.
Generate(GenerateMode),
}
impl Default for Mode {
fn default() -> Mode {
Mode::Search(SearchMode::Standard)
}
}
impl Mode {
/// Update this mode to the new mode while implementing various override
/// semantics. For example, a search mode cannot override a non-search
/// mode.
pub(crate) fn update(&mut self, new: Mode) {
match *self {
// If we're in a search mode, then anything can override it.
Mode::Search(_) => *self = new,
_ => {
// Once we're in a non-search mode, other non-search modes
// can override it. But search modes cannot. So for example,
// `--files -l` will still be Mode::Files.
if !matches!(*self, Mode::Search(_)) {
*self = new;
}
}
}
}
}
/// The kind of search that ripgrep is going to perform.
#[derive(Clone, Copy, Debug, Eq, PartialEq)]
pub(crate) enum SearchMode {
/// The default standard mode of operation. ripgrep looks for matches and
/// prints them when found.
///
/// There is no specific flag for this mode since it's the default. But
/// some of the modes below, like JSON, have negation flags like --no-json
/// that let you revert back to this default mode.
Standard,
/// Show files containing at least one match.
FilesWithMatches,
/// Show files that don't contain any matches.
FilesWithoutMatch,
/// Show files containing at least one match and the number of matching
/// lines.
Count,
/// Show files containing at least one match and the total number of
/// matches.
CountMatches,
/// Print matches in a JSON lines format.
JSON,
}
/// The thing to generate via the --generate flag.
#[derive(Clone, Copy, Debug, Eq, PartialEq)]
pub(crate) enum GenerateMode {
/// Generate the raw roff used for the man page.
Man,
/// Completions for bash.
CompleteBash,
/// Completions for zsh.
CompleteZsh,
/// Completions for fish.
CompleteFish,
/// Completions for PowerShell.
CompletePowerShell,
}
/// Indicates how ripgrep should treat binary data.
#[derive(Debug, Eq, PartialEq)]
pub(crate) enum BinaryMode {
/// Automatically determine the binary mode to use. Essentially, when
/// a file is searched explicitly, then it will be searched using the
/// `SearchAndSuppress` strategy. Otherwise, it will be searched in a way
/// that attempts to skip binary files as much as possible. That is, once
/// a file is classified as binary, searching will immediately stop.
Auto,
/// Search files even when they have binary data, but if a match is found,
/// suppress it and emit a warning.
///
/// In this mode, `NUL` bytes are replaced with line terminators. This is
/// a heuristic meant to reduce heap memory usage, since true binary data
/// isn't line oriented. If one attempts to treat such data as line
/// oriented, then one may wind up with impractically large lines. For
/// example, many binary files contain very long runs of NUL bytes.
SearchAndSuppress,
/// Treat all files as if they were plain text. There's no skipping and no
/// replacement of `NUL` bytes with line terminators.
AsText,
}
impl Default for BinaryMode {
fn default() -> BinaryMode {
BinaryMode::Auto
}
}
/// Indicates what kind of boundary mode to use (line or word).
#[derive(Debug, Eq, PartialEq)]
pub(crate) enum BoundaryMode {
/// Only allow matches when surrounded by line bounaries.
Line,
/// Only allow matches when surrounded by word bounaries.
Word,
}
/// Indicates the buffer mode that ripgrep should use when printing output.
///
/// The default is `Auto`.
#[derive(Debug, Eq, PartialEq)]
pub(crate) enum BufferMode {
/// Select the buffer mode, 'line' or 'block', automatically based on
/// whether stdout is connected to a tty.
Auto,
/// Flush the output buffer whenever a line terminator is seen.
///
/// This is useful when wants to see search results more immediately,
/// for example, with `tail -f`.
Line,
/// Flush the output buffer whenever it reaches some fixed size. The size
/// is usually big enough to hold many lines.
///
/// This is useful for maximum performance, particularly when printing
/// lots of results.
Block,
}
impl Default for BufferMode {
fn default() -> BufferMode {
BufferMode::Auto
}
}
/// Indicates the case mode for how to interpret all patterns given to ripgrep.
///
/// The default is `Sensitive`.
#[derive(Debug, Eq, PartialEq)]
pub(crate) enum CaseMode {
/// Patterns are matched case sensitively. i.e., `a` does not match `A`.
Sensitive,
/// Patterns are matched case insensitively. i.e., `a` does match `A`.
Insensitive,
/// Patterns are automatically matched case insensitively only when they
/// consist of all lowercase literal characters. For example, the pattern
/// `a` will match `A` but `A` will not match `a`.
Smart,
}
impl Default for CaseMode {
fn default() -> CaseMode {
CaseMode::Sensitive
}
}
/// Indicates whether ripgrep should include color/hyperlinks in its output.
///
/// The default is `Auto`.
#[derive(Debug, Eq, PartialEq)]
pub(crate) enum ColorChoice {
/// Color and hyperlinks will never be used.
Never,
/// Color and hyperlinks will be used only when stdout is connected to a
/// tty.
Auto,
/// Color will always be used.
Always,
/// Color will always be used and only ANSI escapes will be used.
///
/// This only makes sense in the context of legacy Windows console APIs.
/// At time of writing, ripgrep will try to use the legacy console APIs
/// if ANSI coloring isn't believed to be possible. This option will force
/// ripgrep to use ANSI coloring.
Ansi,
}
impl Default for ColorChoice {
fn default() -> ColorChoice {
ColorChoice::Auto
}
}
impl ColorChoice {
/// Convert this color choice to the corresponding termcolor type.
pub(crate) fn to_termcolor(&self) -> termcolor::ColorChoice {
match *self {
ColorChoice::Never => termcolor::ColorChoice::Never,
ColorChoice::Auto => termcolor::ColorChoice::Auto,
ColorChoice::Always => termcolor::ColorChoice::Always,
ColorChoice::Ansi => termcolor::ColorChoice::AlwaysAnsi,
}
}
}
/// Indicates the line context options ripgrep should use for output.
///
/// The default is no context at all.
#[derive(Debug, Eq, PartialEq)]
pub(crate) enum ContextMode {
/// All lines will be printed. That is, the context is unbounded.
Passthru,
/// Only show a certain number of lines before and after each match.
Limited(ContextModeLimited),
}
impl Default for ContextMode {
fn default() -> ContextMode {
ContextMode::Limited(ContextModeLimited::default())
}
}
impl ContextMode {
/// Set the "before" context.
///
/// If this was set to "passthru" context, then it is overridden in favor
/// of limited context with the given value for "before" and `0` for
/// "after."
pub(crate) fn set_before(&mut self, lines: usize) {
match *self {
ContextMode::Passthru => {
*self = ContextMode::Limited(ContextModeLimited {
before: Some(lines),
after: None,
both: None,
})
}
ContextMode::Limited(ContextModeLimited {
ref mut before,
..
}) => *before = Some(lines),
}
}
/// Set the "after" context.
///
/// If this was set to "passthru" context, then it is overridden in favor
/// of limited context with the given value for "after" and `0` for
/// "before."
pub(crate) fn set_after(&mut self, lines: usize) {
match *self {
ContextMode::Passthru => {
*self = ContextMode::Limited(ContextModeLimited {
before: None,
after: Some(lines),
both: None,
})
}
ContextMode::Limited(ContextModeLimited {
ref mut after, ..
}) => *after = Some(lines),
}
}
/// Set the "both" context.
///
/// If this was set to "passthru" context, then it is overridden in favor
/// of limited context with the given value for "both" and `None` for
/// "before" and "after".
pub(crate) fn set_both(&mut self, lines: usize) {
match *self {
ContextMode::Passthru => {
*self = ContextMode::Limited(ContextModeLimited {
before: None,
after: None,
both: Some(lines),
})
}
ContextMode::Limited(ContextModeLimited {
ref mut both, ..
}) => *both = Some(lines),
}
}
/// A convenience function for use in tests that returns the limited
/// context. If this mode isn't limited, then it panics.
#[cfg(test)]
pub(crate) fn get_limited(&self) -> (usize, usize) {
match *self {
ContextMode::Passthru => unreachable!("context mode is passthru"),
ContextMode::Limited(ref limited) => limited.get(),
}
}
}
/// A context mode for a finite number of lines.
///
/// Namely, this indicates that a specific number of lines (possibly zero)
/// should be shown before and/or after each matching line.
///
/// Note that there is a subtle difference between `Some(0)` and `None`. In the
/// former case, it happens when `0` is given explicitly, where as `None` is
/// the default value and occurs when no value is specified.
///
/// `both` is only set by the -C/--context flag. The reason why we don't just
/// set before = after = --context is because the before and after context
/// settings always take precedent over the -C/--context setting, regardless of
/// order. Thus, we need to keep track of them separately.
#[derive(Debug, Default, Eq, PartialEq)]
pub(crate) struct ContextModeLimited {
before: Option<usize>,
after: Option<usize>,
both: Option<usize>,
}
impl ContextModeLimited {
/// Returns the specific number of contextual lines that should be shown
/// around each match. This takes proper precedent into account, i.e.,
/// that `before` and `after` both partially override `both` in all cases.
///
/// By default, this returns `(0, 0)`.
pub(crate) fn get(&self) -> (usize, usize) {
let (mut before, mut after) =
self.both.map(|lines| (lines, lines)).unwrap_or((0, 0));
// --before and --after always override --context, regardless
// of where they appear relative to each other.
if let Some(lines) = self.before {
before = lines;
}
if let Some(lines) = self.after {
after = lines;
}
(before, after)
}
}
/// Represents the separator to use between non-contiguous sections of
/// contextual lines.
///
/// The default is `--`.
#[derive(Clone, Debug, Eq, PartialEq)]
pub(crate) struct ContextSeparator(Option<BString>);
impl Default for ContextSeparator {
fn default() -> ContextSeparator {
ContextSeparator(Some(BString::from("--")))
}
}
impl ContextSeparator {
/// Create a new context separator from the user provided argument. This
/// handles unescaping.
pub(crate) fn new(os: &OsStr) -> anyhow::Result<ContextSeparator> {
let Some(string) = os.to_str() else {
anyhow::bail!(
"separator must be valid UTF-8 (use escape sequences \
to provide a separator that is not valid UTF-8)"
)
};
Ok(ContextSeparator(Some(Vec::unescape_bytes(string).into())))
}
/// Creates a new separator that intructs the printer to disable contextual
/// separators entirely.
pub(crate) fn disabled() -> ContextSeparator {
ContextSeparator(None)
}
/// Return the raw bytes of this separator.
///
/// If context separators were disabled, then this returns `None`.
///
/// Note that this may return a `Some` variant with zero bytes.
pub(crate) fn into_bytes(self) -> Option<Vec<u8>> {
self.0.map(|sep| sep.into())
}
}
/// The encoding mode the searcher will use.
///
/// The default is `Auto`.
#[derive(Debug, Eq, PartialEq)]
pub(crate) enum EncodingMode {
/// Use only BOM sniffing to auto-detect an encoding.
Auto,
/// Use an explicit encoding forcefully, but let BOM sniffing override it.
Some(grep::searcher::Encoding),
/// Use no explicit encoding and disable all BOM sniffing. This will
/// always result in searching the raw bytes, regardless of their
/// true encoding.
Disabled,
}
impl Default for EncodingMode {
fn default() -> EncodingMode {
EncodingMode::Auto
}
}
/// The regex engine to use.
///
/// The default is `Default`.
#[derive(Debug, Eq, PartialEq)]
pub(crate) enum EngineChoice {
/// Uses the default regex engine: Rust's `regex` crate.
///
/// (Well, technically it uses `regex-automata`, but `regex-automata` is
/// the implementation of the `regex` crate.)
Default,
/// Dynamically select the right engine to use.
///
/// This works by trying to use the default engine, and if the pattern does
/// not compile, it switches over to the PCRE2 engine if it's available.
Auto,
/// Uses the PCRE2 regex engine if it's available.
PCRE2,
}
impl Default for EngineChoice {
fn default() -> EngineChoice {
EngineChoice::Default
}
}
/// The field context separator to use to between metadata for each contextual
/// line.
///
/// The default is `-`.
#[derive(Clone, Debug, Eq, PartialEq)]
pub(crate) struct FieldContextSeparator(BString);
impl Default for FieldContextSeparator {
fn default() -> FieldContextSeparator {
FieldContextSeparator(BString::from("-"))
}
}
impl FieldContextSeparator {
/// Create a new separator from the given argument value provided by the
/// user. Unescaping it automatically handled.
pub(crate) fn new(os: &OsStr) -> anyhow::Result<FieldContextSeparator> {
let Some(string) = os.to_str() else {
anyhow::bail!(
"separator must be valid UTF-8 (use escape sequences \
to provide a separator that is not valid UTF-8)"
)
};
Ok(FieldContextSeparator(Vec::unescape_bytes(string).into()))
}
/// Return the raw bytes of this separator.
///
/// Note that this may return an empty `Vec`.
pub(crate) fn into_bytes(self) -> Vec<u8> {
self.0.into()
}
}
/// The field match separator to use to between metadata for each matching
/// line.
///
/// The default is `:`.
#[derive(Clone, Debug, Eq, PartialEq)]
pub(crate) struct FieldMatchSeparator(BString);
impl Default for FieldMatchSeparator {
fn default() -> FieldMatchSeparator {
FieldMatchSeparator(BString::from(":"))
}
}
impl FieldMatchSeparator {
/// Create a new separator from the given argument value provided by the
/// user. Unescaping it automatically handled.
pub(crate) fn new(os: &OsStr) -> anyhow::Result<FieldMatchSeparator> {
let Some(string) = os.to_str() else {
anyhow::bail!(
"separator must be valid UTF-8 (use escape sequences \
to provide a separator that is not valid UTF-8)"
)
};
Ok(FieldMatchSeparator(Vec::unescape_bytes(string).into()))
}
/// Return the raw bytes of this separator.
///
/// Note that this may return an empty `Vec`.
pub(crate) fn into_bytes(self) -> Vec<u8> {
self.0.into()
}
}
/// The type of logging to do. `Debug` emits some details while `Trace` emits
/// much more.
#[derive(Debug, Eq, PartialEq)]
pub(crate) enum LoggingMode {
Debug,
Trace,
}
/// Indicates when to use memory maps.
///
/// The default is `Auto`.
#[derive(Debug, Eq, PartialEq)]
pub(crate) enum MmapMode {
/// This instructs ripgrep to use heuristics for selecting when to and not
/// to use memory maps for searching.
Auto,
/// This instructs ripgrep to always try memory maps when possible. (Memory
/// maps are not possible to use in all circumstances, for example, for
/// virtual files.)
AlwaysTryMmap,
/// Never use memory maps under any circumstances. This includes even
/// when multi-line search is enabled where ripgrep will read the entire
/// contents of a file on to the heap before searching it.
Never,
}
impl Default for MmapMode {
fn default() -> MmapMode {
MmapMode::Auto
}
}
/// Represents a source of patterns that ripgrep should search for.
///
/// The reason to unify these is so that we can retain the order of `-f/--flag`
/// and `-e/--regexp` flags relative to one another.
#[derive(Debug, Eq, PartialEq)]
pub(crate) enum PatternSource {
/// Comes from the `-e/--regexp` flag.
Regexp(String),
/// Comes from the `-f/--file` flag.
File(PathBuf),
}
/// The sort criteria, if present.
#[derive(Debug, Eq, PartialEq)]
pub(crate) struct SortMode {
/// Whether to reverse the sort criteria (i.e., descending order).
pub(crate) reverse: bool,
/// The actual sorting criteria.
pub(crate) kind: SortModeKind,
}
/// The criteria to use for sorting.
#[derive(Debug, Eq, PartialEq)]
pub(crate) enum SortModeKind {
/// Sort by path.
Path,
/// Sort by last modified time.
LastModified,
/// Sort by last accessed time.
LastAccessed,
/// Sort by creation time.
Created,
}
impl SortMode {
/// Checks whether the selected sort mode is supported. If it isn't, an
/// error (hopefully explaining why) is returned.
pub(crate) fn supported(&self) -> anyhow::Result<()> {
match self.kind {
SortModeKind::Path => Ok(()),
SortModeKind::LastModified => {
let md = std::env::current_exe()
.and_then(|p| p.metadata())
.and_then(|md| md.modified());
let Err(err) = md else { return Ok(()) };
anyhow::bail!(
"sorting by last modified isn't supported: {err}"
);
}
SortModeKind::LastAccessed => {
let md = std::env::current_exe()
.and_then(|p| p.metadata())
.and_then(|md| md.accessed());
let Err(err) = md else { return Ok(()) };
anyhow::bail!(
"sorting by last accessed isn't supported: {err}"
);
}
SortModeKind::Created => {
let md = std::env::current_exe()
.and_then(|p| p.metadata())
.and_then(|md| md.created());
let Err(err) = md else { return Ok(()) };
anyhow::bail!(
"sorting by creation time isn't supported: {err}"
);
}
}
}
}
/// A single instance of either a change or a selection of one ripgrep's
/// file types.
#[derive(Debug, Eq, PartialEq)]
pub(crate) enum TypeChange {
/// Clear the given type from ripgrep.
Clear { name: String },
/// Add the given type definition (name and glob) to ripgrep.
Add { def: String },
/// Select the given type for filtering.
Select { name: String },
/// Select the given type for filtering but negate it.
Negate { name: String },
}

282
crates/core/flags/mod.rs Normal file
View File

@ -0,0 +1,282 @@
/*!
Defines ripgrep's command line interface.
This modules deals with everything involving ripgrep's flags and positional
arguments. This includes generating shell completions, `--help` output and even
ripgrep's man page. It's also responsible for parsing and validating every
flag (including reading ripgrep's config file), and manages the contact points
between these flags and ripgrep's cast of supporting libraries. For example,
once [`HiArgs`] has been created, it knows how to create a multi threaded
recursive directory traverser.
*/
use std::{
ffi::OsString,
fmt::Debug,
panic::{RefUnwindSafe, UnwindSafe},
};
pub(crate) use crate::flags::{
complete::{
bash::generate as generate_complete_bash,
fish::generate as generate_complete_fish,
powershell::generate as generate_complete_powershell,
zsh::generate as generate_complete_zsh,
},
doc::{
help::{
generate_long as generate_help_long,
generate_short as generate_help_short,
},
man::generate as generate_man_page,
version::{
generate_long as generate_version_long,
generate_short as generate_version_short,
},
},
hiargs::HiArgs,
lowargs::{GenerateMode, Mode, SearchMode, SpecialMode},
parse::{parse, ParseResult},
};
mod complete;
mod config;
mod defs;
mod doc;
mod hiargs;
mod lowargs;
mod parse;
/// A trait that encapsulates the definition of an optional flag for ripgrep.
///
/// This trait is meant to be used via dynamic dispatch. Namely, the `defs`
/// module provides a single global slice of `&dyn Flag` values correspondings
/// to all of the flags in ripgrep.
///
/// ripgrep's required positional arguments are handled by the parser and by
/// the conversion from low-level arguments to high level arguments. Namely,
/// all of ripgrep's positional arguments are treated as file paths, except
/// in certain circumstances where the first argument is treated as a regex
/// pattern.
///
/// Note that each implementation of this trait requires a long flag name,
/// but can also optionally have a short version and even a negation flag.
/// For example, the `-E/--encoding` flag accepts a value, but it also has a
/// `--no-encoding` negation flag for reverting back to "automatic" encoding
/// detection. All three of `-E`, `--encoding` and `--no-encoding` are provided
/// by a single implementation of this trait.
///
/// ripgrep only supports flags that are switches or flags that accept a single
/// value. Flags that accept multiple values are an unsupported abberation.
trait Flag: Debug + Send + Sync + UnwindSafe + RefUnwindSafe + 'static {
/// Returns true if this flag is a switch. When a flag is a switch, the
/// CLI parser will look for a value after the flag is seen.
fn is_switch(&self) -> bool;
/// A short single byte name for this flag. This returns `None` by default,
/// which signifies that the flag has no short name.
///
/// The byte returned must be an ASCII codepoint that is a `.` or is
/// alpha-numeric.
fn name_short(&self) -> Option<u8> {
None
}
/// Returns the long name of this flag. All flags must have a "long" name.
///
/// The long name must be at least 2 bytes, and all of its bytes must be
/// ASCII codepoints that are either `-` or alpha-numeric.
fn name_long(&self) -> &'static str;
/// Returns a list of aliases for this flag.
///
/// The aliases must follow the same rules as `Flag::name_long`.
///
/// By default, an empty slice is returned.
fn aliases(&self) -> &'static [&'static str] {
&[]
}
/// Returns a negated name for this flag. The negation of a flag is
/// intended to have the opposite meaning of a flag or to otherwise turn
/// something "off" or revert it to its default behavior.
///
/// Negated flags are not listed in their own section in the `-h/--help`
/// output or man page. Instead, they are automatically mentioned at the
/// end of the documentation section of the flag they negated.
///
/// The aliases must follow the same rules as `Flag::name_long`.
///
/// By default, a flag has no negation and this returns `None`.
fn name_negated(&self) -> Option<&'static str> {
None
}
/// Returns the variable name describing the type of value this flag
/// accepts. This should always be set for non-switch flags and never set
/// for switch flags.
///
/// For example, the `--max-count` flag has its variable name set to `NUM`.
///
/// The convention is to capitalize variable names.
///
/// By default this returns `None`.
fn doc_variable(&self) -> Option<&'static str> {
None
}
/// Returns the category of this flag.
///
/// Every flag must have a single category. Categories are used to organize
/// flags in the generated documentation.
fn doc_category(&self) -> Category;
/// A (very) short documentation string describing what this flag does.
///
/// This may sacrifice "proper English" in order to be as terse as
/// possible. Generally, we try to ensure that `rg -h` doesn't have any
/// lines that exceed 79 columns.
fn doc_short(&self) -> &'static str;
/// A (possibly very) longer documentation string describing in full
/// detail what this flag does. This should be in mandoc/mdoc format.
fn doc_long(&self) -> &'static str;
/// If this is a non-switch flag that accepts a small set of specific
/// values, then this should list them.
///
/// This returns an empty slice by default.
fn doc_choices(&self) -> &'static [&'static str] {
&[]
}
/// Given the parsed value (which might just be a switch), this should
/// update the state in `args` based on the value given for this flag.
///
/// This may update state for other flags as appropriate.
///
/// The `-V/--version` and `-h/--help` flags are treated specially in the
/// parser and should do nothing here.
///
/// By convention, implementations should generally not try to "do"
/// anything other than validate the value given. For example, the
/// implementation for `--hostname-bin` should not try to resolve the
/// hostname to use by running the binary provided. That should be saved
/// for a later step. This convention is used to ensure that getting the
/// low-level arguments is as reliable and quick as possible. It also
/// ensures that "doing something" occurs a minimal number of times. For
/// example, by avoiding trying to find the hostname here, we can do it
/// once later no matter how many times `--hostname-bin` is provided.
///
/// Implementations should not include the flag name in the error message
/// returned. The flag name is included automatically by the parser.
fn update(
&self,
value: FlagValue,
args: &mut crate::flags::lowargs::LowArgs,
) -> anyhow::Result<()>;
}
/// The category that a flag belongs to.
///
/// Categories are used to organize flags into "logical" groups in the
/// generated documentation.
#[derive(Clone, Copy, Debug, Eq, Hash, PartialEq, PartialOrd, Ord)]
enum Category {
/// Flags related to how ripgrep reads its input. Its "input" generally
/// consists of the patterns it is trying to match and the haystacks it is
/// trying to search.
Input,
/// Flags related to the operation of the search itself. For example,
/// whether case insensitive matching is enabled.
Search,
/// Flags related to how ripgrep filters haystacks. For example, whether
/// to respect gitignore files or not.
Filter,
/// Flags related to how ripgrep shows its search results. For example,
/// whether to show line numbers or not.
Output,
/// Flags related to changing ripgrep's output at a more fundamental level.
/// For example, flags like `--count` suppress printing of individual
/// lines, and instead just print the total count of matches for each file
/// searched.
OutputModes,
/// Flags related to logging behavior such as emitting non-fatal error
/// messages or printing search statistics.
Logging,
/// Other behaviors not related to ripgrep's core functionality. For
/// example, printing the file type globbing rules, or printing the list
/// of files ripgrep would search without actually searching them.
OtherBehaviors,
}
impl Category {
/// Returns a string representation of this category.
///
/// This string is the name of the variable used in various templates for
/// generated documentation. This name can be used for interpolation.
fn as_str(&self) -> &'static str {
match *self {
Category::Input => "input",
Category::Search => "search",
Category::Filter => "filter",
Category::Output => "output",
Category::OutputModes => "output-modes",
Category::Logging => "logging",
Category::OtherBehaviors => "other-behaviors",
}
}
}
/// Represents a value parsed from the command line.
///
/// This doesn't include the corresponding flag, but values come in one of
/// two forms: a switch (on or off) or an arbitrary value.
///
/// Note that the CLI doesn't directly support negated switches. For example,
/// you can'd do anything like `-n=false` or any of that nonsense. Instead,
/// the CLI parser knows about which flag names are negations and which aren't
/// (courtesy of the `Flag` trait). If a flag given is known as a negation,
/// then a `FlagValue::Switch(false)` value is passed into `Flag::update`.
#[derive(Debug)]
enum FlagValue {
/// A flag that is either on or off.
Switch(bool),
/// A flag that comes with an arbitrary user value.
Value(OsString),
}
impl FlagValue {
/// Return the yes or no value of this switch.
///
/// If this flag value is not a switch, then this panics.
///
/// This is useful when writing the implementation of `Flag::update`.
/// namely, callers usually know whether a switch or a value is expected.
/// If a flag is something different, then it indicates a bug, and thus a
/// panic is acceptable.
fn unwrap_switch(self) -> bool {
match self {
FlagValue::Switch(yes) => yes,
FlagValue::Value(_) => {
unreachable!("got flag value but expected switch")
}
}
}
/// Return the user provided value of this flag.
///
/// If this flag is a switch, then this panics.
///
/// This is useful when writing the implementation of `Flag::update`.
/// namely, callers usually know whether a switch or a value is expected.
/// If a flag is something different, then it indicates a bug, and thus a
/// panic is acceptable.
fn unwrap_value(self) -> OsString {
match self {
FlagValue::Switch(_) => {
unreachable!("got switch but expected flag value")
}
FlagValue::Value(v) => v,
}
}
}

392
crates/core/flags/parse.rs Normal file
View File

@ -0,0 +1,392 @@
/*!
Parses command line arguments into a structured and typed representation.
*/
use std::ffi::OsString;
use anyhow::Context;
use crate::flags::{
defs::FLAGS,
hiargs::HiArgs,
lowargs::{LoggingMode, LowArgs, SpecialMode},
Flag, FlagValue,
};
/// The result of parsing CLI arguments.
///
/// This is basically a `anyhow::Result<T>`, but with one extra variant that is
/// inhabited whenever ripgrep should execute a "special" mode. That is, when a
/// user provides the `-h/--help` or `-V/--version` flags.
///
/// This special variant exists to allow CLI parsing to short circuit as
/// quickly as is reasonable. For example, it lets CLI parsing avoid reading
/// ripgrep's configuration and converting low level arguments into a higher
/// level representation.
#[derive(Debug)]
pub(crate) enum ParseResult<T> {
Special(SpecialMode),
Ok(T),
Err(anyhow::Error),
}
impl<T> ParseResult<T> {
/// If this result is `Ok`, then apply `then` to it. Otherwise, return this
/// result unchanged.
fn and_then<U>(
self,
mut then: impl FnMut(T) -> ParseResult<U>,
) -> ParseResult<U> {
match self {
ParseResult::Special(mode) => ParseResult::Special(mode),
ParseResult::Ok(t) => then(t),
ParseResult::Err(err) => ParseResult::Err(err),
}
}
}
/// Parse CLI arguments and convert then to their high level representation.
pub(crate) fn parse() -> ParseResult<HiArgs> {
parse_low().and_then(|low| match HiArgs::from_low_args(low) {
Ok(hi) => ParseResult::Ok(hi),
Err(err) => ParseResult::Err(err),
})
}
/// Parse CLI arguments only into their low level representation.
///
/// This takes configuration into account. That is, it will try to read
/// `RIPGREP_CONFIG_PATH` and prepend any arguments found there to the
/// arguments passed to this process.
///
/// This will also set one-time global state flags, such as the log level and
/// whether messages should be printed.
fn parse_low() -> ParseResult<LowArgs> {
if let Err(err) = crate::logger::Logger::init() {
let err = anyhow::anyhow!("failed to initialize logger: {err}");
return ParseResult::Err(err);
}
let parser = Parser::new();
let mut low = LowArgs::default();
if let Err(err) = parser.parse(std::env::args_os().skip(1), &mut low) {
return ParseResult::Err(err);
}
// Even though we haven't parsed the config file yet (assuming it exists),
// we can still use the arguments given on the CLI to setup ripgrep's
// logging preferences. Even if the config file changes them in some way,
// it's really the best we can do. This way, for example, folks can pass
// `--trace` and see any messages logged during config file parsing.
set_log_levels(&low);
// Before we try to take configuration into account, we can bail early
// if a special mode was enabled. This is basically only for version and
// help output which shouldn't be impacted by extra configuration.
if let Some(special) = low.special.take() {
return ParseResult::Special(special);
}
// If the end user says no config, then respect it.
if low.no_config {
log::debug!("not reading config files because --no-config is present");
return ParseResult::Ok(low);
}
// Look for arguments from a config file. If we got nothing (whether the
// file is empty or RIPGREP_CONFIG_PATH wasn't set), then we don't need
// to re-parse.
let config_args = crate::flags::config::args();
if config_args.is_empty() {
log::debug!("no extra arguments found from configuration file");
return ParseResult::Ok(low);
}
// The final arguments are just the arguments from the CLI appending to
// the end of the config arguments.
let mut final_args = config_args;
final_args.extend(std::env::args_os().skip(1));
// Now do the CLI parsing dance again.
let mut low = LowArgs::default();
if let Err(err) = parser.parse(final_args.into_iter(), &mut low) {
return ParseResult::Err(err);
}
// Reset the message and logging levels, since they could have changed.
set_log_levels(&low);
ParseResult::Ok(low)
}
/// Sets global state flags that control logging based on low-level arguments.
fn set_log_levels(low: &LowArgs) {
crate::messages::set_messages(!low.no_messages);
crate::messages::set_ignore_messages(!low.no_ignore_messages);
match low.logging {
Some(LoggingMode::Trace) => {
log::set_max_level(log::LevelFilter::Trace)
}
Some(LoggingMode::Debug) => {
log::set_max_level(log::LevelFilter::Debug)
}
None => log::set_max_level(log::LevelFilter::Warn),
}
}
/// Parse the sequence of CLI arguments given a low level typed set of
/// arguments.
///
/// This is exposed for testing that the correct low-level arguments are parsed
/// from a CLI. It just runs the parser once over the CLI arguments. It doesn't
/// setup logging or read from a config file.
///
/// This assumes the iterator given does *not* begin with the binary name.
#[cfg(test)]
pub(crate) fn parse_low_raw(
rawargs: impl IntoIterator<Item = impl Into<OsString>>,
) -> anyhow::Result<LowArgs> {
let mut args = LowArgs::default();
Parser::new().parse(rawargs, &mut args)?;
Ok(args)
}
/// Return the metadata for the flag of the given name.
pub(super) fn lookup(name: &str) -> Option<&'static dyn Flag> {
// N.B. Creating a new parser might look expensive, but it only builds
// the lookup trie exactly once. That is, we get a `&'static Parser` from
// `Parser::new()`.
match Parser::new().find_long(name) {
FlagLookup::Match(&FlagInfo { flag, .. }) => Some(flag),
_ => None,
}
}
/// A parser for turning a sequence of command line arguments into a more
/// strictly typed set of arguments.
#[derive(Debug)]
struct Parser {
/// A single map that contains all possible flag names. This includes
/// short and long names, aliases and negations. This maps those names to
/// indices into `info`.
map: FlagMap,
/// A map from IDs returned by the `map` to the corresponding flag
/// information.
info: Vec<FlagInfo>,
}
impl Parser {
/// Create a new parser.
///
/// This always creates the same parser and only does it once. Callers may
/// call this repeatedly, and the parser will only be built once.
fn new() -> &'static Parser {
use std::sync::OnceLock;
// Since a parser's state is immutable and completely determined by
// FLAGS, and since FLAGS is a constant, we can initialize it exactly
// once.
static P: OnceLock<Parser> = OnceLock::new();
P.get_or_init(|| {
let mut infos = vec![];
for &flag in FLAGS.iter() {
infos.push(FlagInfo {
flag,
name: Ok(flag.name_long()),
kind: FlagInfoKind::Standard,
});
for alias in flag.aliases() {
infos.push(FlagInfo {
flag,
name: Ok(alias),
kind: FlagInfoKind::Alias,
});
}
if let Some(byte) = flag.name_short() {
infos.push(FlagInfo {
flag,
name: Err(byte),
kind: FlagInfoKind::Standard,
});
}
if let Some(name) = flag.name_negated() {
infos.push(FlagInfo {
flag,
name: Ok(name),
kind: FlagInfoKind::Negated,
});
}
}
let map = FlagMap::new(&infos);
Parser { map, info: infos }
})
}
/// Parse the given CLI arguments into a low level representation.
///
/// The iterator given should *not* start with the binary name.
fn parse<I, O>(&self, rawargs: I, args: &mut LowArgs) -> anyhow::Result<()>
where
I: IntoIterator<Item = O>,
O: Into<OsString>,
{
let mut p = lexopt::Parser::from_args(rawargs);
while let Some(arg) = p.next().context("invalid CLI arguments")? {
let lookup = match arg {
lexopt::Arg::Value(value) => {
args.positional.push(value);
continue;
}
lexopt::Arg::Short(ch) if ch == 'h' => {
// Special case -h/--help since behavior is different
// based on whether short or long flag is given.
args.special = Some(SpecialMode::HelpShort);
continue;
}
lexopt::Arg::Short(ch) if ch == 'V' => {
// Special case -V/--version since behavior is different
// based on whether short or long flag is given.
args.special = Some(SpecialMode::VersionShort);
continue;
}
lexopt::Arg::Short(ch) => self.find_short(ch),
lexopt::Arg::Long(name) if name == "help" => {
// Special case -h/--help since behavior is different
// based on whether short or long flag is given.
args.special = Some(SpecialMode::HelpLong);
continue;
}
lexopt::Arg::Long(name) if name == "version" => {
// Special case -V/--version since behavior is different
// based on whether short or long flag is given.
args.special = Some(SpecialMode::VersionLong);
continue;
}
lexopt::Arg::Long(name) => self.find_long(name),
};
let mat = match lookup {
FlagLookup::Match(mat) => mat,
FlagLookup::UnrecognizedShort(name) => {
anyhow::bail!("unrecognized flag -{name}")
}
FlagLookup::UnrecognizedLong(name) => {
anyhow::bail!("unrecognized flag --{name}")
}
};
let value = if matches!(mat.kind, FlagInfoKind::Negated) {
// Negated flags are always switches, even if the non-negated
// flag is not. For example, --context-separator accepts a
// value, but --no-context-separator does not.
FlagValue::Switch(false)
} else if mat.flag.is_switch() {
FlagValue::Switch(true)
} else {
FlagValue::Value(p.value().with_context(|| {
format!("missing value for flag {mat}")
})?)
};
mat.flag
.update(value, args)
.with_context(|| format!("error parsing flag {mat}"))?;
}
Ok(())
}
/// Look for a flag by its short name.
fn find_short(&self, ch: char) -> FlagLookup<'_> {
if !ch.is_ascii() {
return FlagLookup::UnrecognizedShort(ch);
}
let byte = u8::try_from(ch).unwrap();
let Some(index) = self.map.find(&[byte]) else {
return FlagLookup::UnrecognizedShort(ch);
};
FlagLookup::Match(&self.info[index])
}
/// Look for a flag by its long name.
///
/// This also works for aliases and negated names.
fn find_long(&self, name: &str) -> FlagLookup<'_> {
let Some(index) = self.map.find(name.as_bytes()) else {
return FlagLookup::UnrecognizedLong(name.to_string());
};
FlagLookup::Match(&self.info[index])
}
}
/// The result of looking up a flag name.
#[derive(Debug)]
enum FlagLookup<'a> {
/// Lookup found a match and the metadata for the flag is attached.
Match(&'a FlagInfo),
/// The given short name is unrecognized.
UnrecognizedShort(char),
/// The given long name is unrecognized.
UnrecognizedLong(String),
}
/// The info about a flag associated with a flag's ID in the the flag map.
#[derive(Debug)]
struct FlagInfo {
/// The flag object and its associated metadata.
flag: &'static dyn Flag,
/// The actual name that is stored in the Aho-Corasick automaton. When this
/// is a byte, it corresponds to a short single character ASCII flag. The
/// actual pattern that's in the Aho-Corasick automaton is just the single
/// byte.
name: Result<&'static str, u8>,
/// The type of flag that is stored for the corresponding Aho-Corasick
/// pattern.
kind: FlagInfoKind,
}
/// The kind of flag that is being matched.
#[derive(Debug)]
enum FlagInfoKind {
/// A standard flag, e.g., --passthru.
Standard,
/// A negation of a standard flag, e.g., --no-multiline.
Negated,
/// An alias for a standard flag, e.g., --passthrough.
Alias,
}
impl std::fmt::Display for FlagInfo {
fn fmt(&self, f: &mut std::fmt::Formatter) -> std::fmt::Result {
match self.name {
Ok(long) => write!(f, "--{long}"),
Err(short) => write!(f, "-{short}", short = char::from(short)),
}
}
}
/// A map from flag names (short, long, negated and aliases) to their ID.
///
/// Once an ID is known, it can be used to look up a flag's metadata in the
/// parser's internal state.
#[derive(Debug)]
struct FlagMap {
map: std::collections::HashMap<Vec<u8>, usize>,
}
impl FlagMap {
/// Create a new map of flags for the given flag information.
///
/// The index of each flag info corresponds to its ID.
fn new(infos: &[FlagInfo]) -> FlagMap {
let mut map = std::collections::HashMap::with_capacity(infos.len());
for (i, info) in infos.iter().enumerate() {
match info.name {
Ok(name) => {
assert_eq!(None, map.insert(name.as_bytes().to_vec(), i));
}
Err(byte) => {
assert_eq!(None, map.insert(vec![byte], i));
}
}
}
FlagMap { map }
}
/// Look for a match of `name` in the given Aho-Corasick automaton.
///
/// This only returns a match if the one found has a length equivalent to
/// the length of the name given.
fn find(&self, name: &[u8]) -> Option<usize> {
self.map.get(name).copied()
}
}

119
crates/core/subject.rs → crates/core/haystack.rs
View File

@ -1,108 +1,111 @@
/*!
Defines a builder for haystacks.
A "haystack" represents something we want to search. It encapsulates the logic
for whether a haystack ought to be searched or not, separate from the standard
ignore rules and other filtering logic.
Effectively, a haystack wraps a directory entry and adds some light application
level logic around it.
*/
use std::path::Path;
/// A configuration for describing how subjects should be built.
#[derive(Clone, Debug)]
struct Config {
strip_dot_prefix: bool,
}
impl Default for Config {
fn default() -> Config {
Config { strip_dot_prefix: false }
}
}
/// A builder for constructing things to search over.
#[derive(Clone, Debug)]
pub struct SubjectBuilder {
config: Config,
pub(crate) struct HaystackBuilder {
strip_dot_prefix: bool,
}
impl SubjectBuilder {
/// Return a new subject builder with a default configuration.
pub fn new() -> SubjectBuilder {
SubjectBuilder { config: Config::default() }
impl HaystackBuilder {
/// Return a new haystack builder with a default configuration.
pub(crate) fn new() -> HaystackBuilder {
HaystackBuilder { strip_dot_prefix: false }
}
/// Create a new subject from a possibly missing directory entry.
/// Create a new haystack from a possibly missing directory entry.
///
/// If the directory entry isn't present, then the corresponding error is
/// logged if messages have been configured. Otherwise, if the subject is
/// deemed searchable, then it is returned.
pub fn build_from_result(
/// logged if messages have been configured. Otherwise, if the directory
/// entry is deemed searchable, then it is returned as a haystack.
pub(crate) fn build_from_result(
&self,
result: Result<ignore::DirEntry, ignore::Error>,
) -> Option<Subject> {
) -> Option<Haystack> {
match result {
Ok(dent) => self.build(dent),
Err(err) => {
err_message!("{}", err);
err_message!("{err}");
None
}
}
}
/// Create a new subject using this builder's configuration.
/// Create a new haystack using this builder's configuration.
///
/// If a subject could not be created or should otherwise not be searched,
/// then this returns `None` after emitting any relevant log messages.
pub fn build(&self, dent: ignore::DirEntry) -> Option<Subject> {
let subj =
Subject { dent, strip_dot_prefix: self.config.strip_dot_prefix };
if let Some(ignore_err) = subj.dent.error() {
ignore_message!("{}", ignore_err);
/// If a directory entry could not be created or should otherwise not be
/// searched, then this returns `None` after emitting any relevant log
/// messages.
fn build(&self, dent: ignore::DirEntry) -> Option<Haystack> {
let hay = Haystack { dent, strip_dot_prefix: self.strip_dot_prefix };
if let Some(err) = hay.dent.error() {
ignore_message!("{err}");
}
// If this entry was explicitly provided by an end user, then we always
// want to search it.
if subj.is_explicit() {
return Some(subj);
if hay.is_explicit() {
return Some(hay);
}
// At this point, we only want to search something if it's explicitly a
// file. This omits symlinks. (If ripgrep was configured to follow
// symlinks, then they have already been followed by the directory
// traversal.)
if subj.is_file() {
return Some(subj);
if hay.is_file() {
return Some(hay);
}
// We got nothing. Emit a debug message, but only if this isn't a
// directory. Otherwise, emitting messages for directories is just
// noisy.
if !subj.is_dir() {
if !hay.is_dir() {
log::debug!(
"ignoring {}: failed to pass subject filter: \
"ignoring {}: failed to pass haystack filter: \
file type: {:?}, metadata: {:?}",
subj.dent.path().display(),
subj.dent.file_type(),
subj.dent.metadata()
hay.dent.path().display(),
hay.dent.file_type(),
hay.dent.metadata()
);
}
None
}
/// When enabled, if the subject's file path starts with `./` then it is
/// When enabled, if the haystack's file path starts with `./` then it is
/// stripped.
///
/// This is useful when implicitly searching the current working directory.
pub fn strip_dot_prefix(&mut self, yes: bool) -> &mut SubjectBuilder {
self.config.strip_dot_prefix = yes;
pub(crate) fn strip_dot_prefix(
&mut self,
yes: bool,
) -> &mut HaystackBuilder {
self.strip_dot_prefix = yes;
self
}
}
/// A subject is a thing we want to search. Generally, a subject is either a
/// file or stdin.
/// A haystack is a thing we want to search.
///
/// Generally, a haystack is either a file or stdin.
#[derive(Clone, Debug)]
pub struct Subject {
pub(crate) struct Haystack {
dent: ignore::DirEntry,
strip_dot_prefix: bool,
}
impl Subject {
/// Return the file path corresponding to this subject.
impl Haystack {
/// Return the file path corresponding to this haystack.
///
/// If this subject corresponds to stdin, then a special `<stdin>` path
/// If this haystack corresponds to stdin, then a special `<stdin>` path
/// is returned instead.
pub fn path(&self) -> &Path {
pub(crate) fn path(&self) -> &Path {
if self.strip_dot_prefix && self.dent.path().starts_with("./") {
self.dent.path().strip_prefix("./").unwrap()
} else {
@ -111,21 +114,21 @@ impl Subject {
}
/// Returns true if and only if this entry corresponds to stdin.
pub fn is_stdin(&self) -> bool {
pub(crate) fn is_stdin(&self) -> bool {
self.dent.is_stdin()
}
/// Returns true if and only if this entry corresponds to a subject to
/// Returns true if and only if this entry corresponds to a haystack to
/// search that was explicitly supplied by an end user.
///
/// Generally, this corresponds to either stdin or an explicit file path
/// argument. e.g., in `rg foo some-file ./some-dir/`, `some-file` is
/// an explicit subject, but, e.g., `./some-dir/some-other-file` is not.
/// an explicit haystack, but, e.g., `./some-dir/some-other-file` is not.
///
/// However, note that ripgrep does not see through shell globbing. e.g.,
/// in `rg foo ./some-dir/*`, `./some-dir/some-other-file` will be treated
/// as an explicit subject.
pub fn is_explicit(&self) -> bool {
/// as an explicit haystack.
pub(crate) fn is_explicit(&self) -> bool {
// stdin is obvious. When an entry has a depth of 0, that means it
// was explicitly provided to our directory iterator, which means it
// was in turn explicitly provided by the end user. The !is_dir check
@ -135,7 +138,7 @@ impl Subject {
self.is_stdin() || (self.dent.depth() == 0 && !self.is_dir())
}
/// Returns true if and only if this subject points to a directory after
/// Returns true if and only if this haystack points to a directory after
/// following symbolic links.
fn is_dir(&self) -> bool {
let ft = match self.dent.file_type() {
@ -150,7 +153,7 @@ impl Subject {
self.dent.path_is_symlink() && self.dent.path().is_dir()
}
/// Returns true if and only if this subject points to a file.
/// Returns true if and only if this haystack points to a file.
fn is_file(&self) -> bool {
self.dent.file_type().map_or(false, |ft| ft.is_file())
}

16
crates/core/logger.rs
View File

@ -1,7 +1,10 @@
// This module defines a super simple logger that works with the `log` crate.
// We don't need anything fancy; just basic log levels and the ability to
// print to stderr. We therefore avoid bringing in extra dependencies just
// for this functionality.
/*!
Defines a super simple logger that works with the `log` crate.
We don't do anything fancy. We just need basic log levels and the ability to
print to stderr. We therefore avoid bringing in extra dependencies just for
this functionality.
*/
use log::{self, Log};
@ -10,15 +13,16 @@ use log::{self, Log};
/// This logger does no filtering. Instead, it relies on the `log` crates
/// filtering via its global max_level setting.
#[derive(Debug)]
pub struct Logger(());
pub(crate) struct Logger(());
/// A singleton used as the target for an implementation of the `Log` trait.
const LOGGER: &'static Logger = &Logger(());
impl Logger {
/// Create a new logger that logs to stderr and initialize it as the
/// global logger. If there was a problem setting the logger, then an
/// error is returned.
pub fn init() -> Result<(), log::SetLoggerError> {
pub(crate) fn init() -> Result<(), log::SetLoggerError> {
log::set_logger(LOGGER)
}
}

563
crates/core/main.rs
View File

@ -1,21 +1,20 @@
use std::{
io::{self, Write},
time::Instant,
};
/*!
The main entry point into ripgrep.
*/
use std::{io::Write, process::ExitCode};
use ignore::WalkState;
use crate::{args::Args, subject::Subject};
use crate::flags::{HiArgs, SearchMode};
#[macro_use]
mod messages;
mod app;
mod args;
mod config;
mod flags;
mod haystack;
mod logger;
mod search;
mod subject;
// Since Rust no longer uses jemalloc by default, ripgrep will, by default,
// use the system allocator. On Linux, this would normally be glibc's
@ -40,143 +39,163 @@ mod subject;
#[global_allocator]
static ALLOC: jemallocator::Jemalloc = jemallocator::Jemalloc;
fn main() {
if let Err(err) = Args::parse().and_then(try_main) {
eprintln_locked!("{:#}", err);
std::process::exit(2);
}
}
fn try_main(args: Args) -> anyhow::Result<()> {
use args::Command::*;
let matched = match args.command() {
Search => search(&args),
SearchParallel => search_parallel(&args),
SearchNever => Ok(false),
Files => files(&args),
FilesParallel => files_parallel(&args),
Types => types(&args),
PCRE2Version => pcre2_version(&args),
}?;
if matched && (args.quiet() || !messages::errored()) {
std::process::exit(0)
} else if messages::errored() {
std::process::exit(2)
} else {
std::process::exit(1)
}
}
/// The top-level entry point for single-threaded search. This recursively
/// steps through the file list (current directory by default) and searches
/// each file sequentially.
fn search(args: &Args) -> anyhow::Result<bool> {
/// The meat of the routine is here. This lets us call the same iteration
/// code over each file regardless of whether we stream over the files
/// as they're produced by the underlying directory traversal or whether
/// they've been collected and sorted (for example) first.
fn iter(
args: &Args,
subjects: impl Iterator<Item = Subject>,
started_at: std::time::Instant,
) -> anyhow::Result<bool> {
let quit_after_match = args.quit_after_match()?;
let mut stats = args.stats()?;
let mut searcher = args.search_worker(args.stdout())?;
let mut matched = false;
let mut searched = false;
for subject in subjects {
searched = true;
let search_result = match searcher.search(&subject) {
Ok(search_result) => search_result,
// A broken pipe means graceful termination.
Err(err) if err.kind() == io::ErrorKind::BrokenPipe => break,
Err(err) => {
err_message!("{}: {}", subject.path().display(), err);
continue;
/// Then, as it was, then again it will be.
fn main() -> ExitCode {
match run(flags::parse()) {
Ok(code) => code,
Err(err) => {
// Look for a broken pipe error. In this case, we generally want
// to exit "gracefully" with a success exit code. This matches
// existing Unix convention. We need to handle this explicitly
// since the Rust runtime doesn't ask for PIPE signals, and thus
// we get an I/O error instead. Traditional C Unix applications
// quit by getting a PIPE signal that they don't handle, and thus
// the unhandled signal causes the process to unceremoniously
// terminate.
for cause in err.chain() {
if let Some(ioerr) = cause.downcast_ref::<std::io::Error>() {
if ioerr.kind() == std::io::ErrorKind::BrokenPipe {
return ExitCode::from(0);
}
}
};
matched |= search_result.has_match();
if let Some(ref mut stats) = stats {
*stats += search_result.stats().unwrap();
}
if matched && quit_after_match {
break;
}
eprintln_locked!("{:#}", err);
ExitCode::from(2)
}
if args.using_default_path() && !searched {
eprint_nothing_searched();
}
if let Some(ref stats) = stats {
let elapsed = Instant::now().duration_since(started_at);
// We don't care if we couldn't print this successfully.
let _ = searcher.print_stats(elapsed, stats);
}
Ok(matched)
}
let started_at = Instant::now();
let subject_builder = args.subject_builder();
let subjects = args
.walker()?
.filter_map(|result| subject_builder.build_from_result(result));
if args.needs_stat_sort() {
let subjects = args.sort_by_stat(subjects).into_iter();
iter(args, subjects, started_at)
} else {
iter(args, subjects, started_at)
}
}
/// The top-level entry point for multi-threaded search. The parallelism is
/// itself achieved by the recursive directory traversal. All we need to do is
/// feed it a worker for performing a search on each file.
/// The main entry point for ripgrep.
///
/// The given parse result determines ripgrep's behavior. The parse
/// result should be the result of parsing CLI arguments in a low level
/// representation, and then followed by an attempt to convert them into a
/// higher level representation. The higher level representation has some nicer
/// abstractions, for example, instead of representing the `-g/--glob` flag
/// as a `Vec<String>` (as in the low level representation), the globs are
/// converted into a single matcher.
fn run(result: crate::flags::ParseResult<HiArgs>) -> anyhow::Result<ExitCode> {
use crate::flags::{Mode, ParseResult};
let args = match result {
ParseResult::Err(err) => return Err(err),
ParseResult::Special(mode) => return special(mode),
ParseResult::Ok(args) => args,
};
let matched = match args.mode() {
Mode::Search(_) if !args.matches_possible() => false,
Mode::Search(mode) if args.threads() == 1 => search(&args, mode)?,
Mode::Search(mode) => search_parallel(&args, mode)?,
Mode::Files if args.threads() == 1 => files(&args)?,
Mode::Files => files_parallel(&args)?,
Mode::Types => return types(&args),
Mode::Generate(mode) => return generate(mode),
};
Ok(if matched && (args.quiet() || !messages::errored()) {
ExitCode::from(0)
} else if messages::errored() {
ExitCode::from(2)
} else {
ExitCode::from(1)
})
}
/// The top-level entry point for single-threaded search.
///
/// This recursively steps through the file list (current directory by default)
/// and searches each file sequentially.
fn search(args: &HiArgs, mode: SearchMode) -> anyhow::Result<bool> {
let started_at = std::time::Instant::now();
let haystack_builder = args.haystack_builder();
let unsorted = args
.walk_builder()?
.build()
.filter_map(|result| haystack_builder.build_from_result(result));
let haystacks = args.sort(unsorted);
let mut matched = false;
let mut searched = false;
let mut stats = args.stats();
let mut searcher = args.search_worker(
args.matcher()?,
args.searcher()?,
args.printer(mode, args.stdout()),
)?;
for haystack in haystacks {
searched = true;
let search_result = match searcher.search(&haystack) {
Ok(search_result) => search_result,
// A broken pipe means graceful termination.
Err(err) if err.kind() == std::io::ErrorKind::BrokenPipe => break,
Err(err) => {
err_message!("{}: {}", haystack.path().display(), err);
continue;
}
};
matched = matched || search_result.has_match();
if let Some(ref mut stats) = stats {
*stats += search_result.stats().unwrap();
}
if matched && args.quit_after_match() {
break;
}
}
if args.has_implicit_path() && !searched {
eprint_nothing_searched();
}
if let Some(ref stats) = stats {
let wtr = searcher.printer().get_mut();
let _ = print_stats(mode, stats, started_at, wtr);
}
Ok(matched)
}
/// The top-level entry point for multi-threaded search.
///
/// The parallelism is itself achieved by the recursive directory traversal.
/// All we need to do is feed it a worker for performing a search on each file.
///
/// Requesting a sorted output from ripgrep (such as with `--sort path`) will
/// automatically disable parallelism and hence sorting is not handled here.
fn search_parallel(args: &Args) -> anyhow::Result<bool> {
use std::sync::atomic::{AtomicBool, Ordering::SeqCst};
fn search_parallel(args: &HiArgs, mode: SearchMode) -> anyhow::Result<bool> {
use std::sync::atomic::{AtomicBool, Ordering};
let quit_after_match = args.quit_after_match()?;
let started_at = Instant::now();
let subject_builder = args.subject_builder();
let bufwtr = args.buffer_writer()?;
let stats = args.stats()?.map(std::sync::Mutex::new);
let started_at = std::time::Instant::now();
let haystack_builder = args.haystack_builder();
let bufwtr = args.buffer_writer();
let stats = args.stats().map(std::sync::Mutex::new);
let matched = AtomicBool::new(false);
let searched = AtomicBool::new(false);
let mut searcher_err = None;
args.walker_parallel()?.run(|| {
let mut searcher = args.search_worker(
args.matcher()?,
args.searcher()?,
args.printer(mode, bufwtr.buffer()),
)?;
args.walk_builder()?.build_parallel().run(|| {
let bufwtr = &bufwtr;
let stats = &stats;
let matched = &matched;
let searched = &searched;
let subject_builder = &subject_builder;
let mut searcher = match args.search_worker(bufwtr.buffer()) {
Ok(searcher) => searcher,
Err(err) => {
searcher_err = Some(err);
return Box::new(move |_| WalkState::Quit);
}
};
let haystack_builder = &haystack_builder;
let mut searcher = searcher.clone();
Box::new(move |result| {
let subject = match subject_builder.build_from_result(result) {
Some(subject) => subject,
let haystack = match haystack_builder.build_from_result(result) {
Some(haystack) => haystack,
None => return WalkState::Continue,
};
searched.store(true, SeqCst);
searched.store(true, Ordering::SeqCst);
searcher.printer().get_mut().clear();
let search_result = match searcher.search(&subject) {
let search_result = match searcher.search(&haystack) {
Ok(search_result) => search_result,
Err(err) => {
err_message!("{}: {}", subject.path().display(), err);
err_message!("{}: {}", haystack.path().display(), err);
return WalkState::Continue;
}
};
if search_result.has_match() {
matched.store(true, SeqCst);
matched.store(true, Ordering::SeqCst);
}
if let Some(ref locked_stats) = *stats {
let mut stats = locked_stats.lock().unwrap();
@ -184,128 +203,110 @@ fn search_parallel(args: &Args) -> anyhow::Result<bool> {
}
if let Err(err) = bufwtr.print(searcher.printer().get_mut()) {
// A broken pipe means graceful termination.
if err.kind() == io::ErrorKind::BrokenPipe {
if err.kind() == std::io::ErrorKind::BrokenPipe {
return WalkState::Quit;
}
// Otherwise, we continue on our merry way.
err_message!("{}: {}", subject.path().display(), err);
err_message!("{}: {}", haystack.path().display(), err);
}
if matched.load(SeqCst) && quit_after_match {
if matched.load(Ordering::SeqCst) && args.quit_after_match() {
WalkState::Quit
} else {
WalkState::Continue
}
})
});
if let Some(err) = searcher_err.take() {
return Err(err);
}
if args.using_default_path() && !searched.load(SeqCst) {
if args.has_implicit_path() && !searched.load(Ordering::SeqCst) {
eprint_nothing_searched();
}
if let Some(ref locked_stats) = stats {
let elapsed = Instant::now().duration_since(started_at);
let stats = locked_stats.lock().unwrap();
let mut searcher = args.search_worker(args.stdout())?;
// We don't care if we couldn't print this successfully.
let _ = searcher.print_stats(elapsed, &stats);
let mut wtr = searcher.printer().get_mut();
let _ = print_stats(mode, &stats, started_at, &mut wtr);
let _ = bufwtr.print(&mut wtr);
}
Ok(matched.load(SeqCst))
Ok(matched.load(Ordering::SeqCst))
}
fn eprint_nothing_searched() {
err_message!(
"No files were searched, which means ripgrep probably \
applied a filter you didn't expect.\n\
Running with --debug will show why files are being skipped."
);
}
/// The top-level entry point for file listing without searching.
///
/// This recursively steps through the file list (current directory by default)
/// and prints each path sequentially using a single thread.
fn files(args: &HiArgs) -> anyhow::Result<bool> {
let haystack_builder = args.haystack_builder();
let unsorted = args
.walk_builder()?
.build()
.filter_map(|result| haystack_builder.build_from_result(result));
let haystacks = args.sort(unsorted);
/// The top-level entry point for listing files without searching them. This
/// recursively steps through the file list (current directory by default) and
/// prints each path sequentially using a single thread.
fn files(args: &Args) -> anyhow::Result<bool> {
/// The meat of the routine is here. This lets us call the same iteration
/// code over each file regardless of whether we stream over the files
/// as they're produced by the underlying directory traversal or whether
/// they've been collected and sorted (for example) first.
fn iter(
args: &Args,
subjects: impl Iterator<Item = Subject>,
) -> anyhow::Result<bool> {
let quit_after_match = args.quit_after_match()?;
let mut matched = false;
let mut path_printer = args.path_printer(args.stdout())?;
for subject in subjects {
matched = true;
if quit_after_match {
let mut matched = false;
let mut path_printer = args.path_printer_builder().build(args.stdout());
for haystack in haystacks {
matched = true;
if args.quit_after_match() {
break;
}
if let Err(err) = path_printer.write(haystack.path()) {
// A broken pipe means graceful termination.
if err.kind() == std::io::ErrorKind::BrokenPipe {
break;
}
if let Err(err) = path_printer.write(subject.path()) {
// A broken pipe means graceful termination.
if err.kind() == io::ErrorKind::BrokenPipe {
break;
}
// Otherwise, we have some other error that's preventing us from
// writing to stdout, so we should bubble it up.
return Err(err.into());
}
// Otherwise, we have some other error that's preventing us from
// writing to stdout, so we should bubble it up.
return Err(err.into());
}
Ok(matched)
}
let subject_builder = args.subject_builder();
let subjects = args
.walker()?
.filter_map(|result| subject_builder.build_from_result(result));
if args.needs_stat_sort() {
let subjects = args.sort_by_stat(subjects).into_iter();
iter(args, subjects)
} else {
iter(args, subjects)
}
Ok(matched)
}
/// The top-level entry point for listing files without searching them. This
/// recursively steps through the file list (current directory by default) and
/// prints each path sequentially using multiple threads.
/// The top-level entry point for multi-threaded file listing without
/// searching.
///
/// This recursively steps through the file list (current directory by default)
/// and prints each path sequentially using multiple threads.
///
/// Requesting a sorted output from ripgrep (such as with `--sort path`) will
/// automatically disable parallelism and hence sorting is not handled here.
fn files_parallel(args: &Args) -> anyhow::Result<bool> {
use std::sync::atomic::AtomicBool;
use std::sync::atomic::Ordering::SeqCst;
use std::sync::mpsc;
use std::thread;
fn files_parallel(args: &HiArgs) -> anyhow::Result<bool> {
use std::{
sync::{
atomic::{AtomicBool, Ordering},
mpsc,
},
thread,
};
let quit_after_match = args.quit_after_match()?;
let subject_builder = args.subject_builder();
let mut path_printer = args.path_printer(args.stdout())?;
let haystack_builder = args.haystack_builder();
let mut path_printer = args.path_printer_builder().build(args.stdout());
let matched = AtomicBool::new(false);
let (tx, rx) = mpsc::channel::<Subject>();
let (tx, rx) = mpsc::channel::<crate::haystack::Haystack>();
let print_thread = thread::spawn(move || -> io::Result<()> {
for subject in rx.iter() {
path_printer.write(subject.path())?;
// We spawn a single printing thread to make sure we don't tear writes.
// We use a channel here under the presumption that it's probably faster
// than using a mutex in the worker threads below, but this has never been
// seriously litigated.
let print_thread = thread::spawn(move || -> std::io::Result<()> {
for haystack in rx.iter() {
path_printer.write(haystack.path())?;
}
Ok(())
});
args.walker_parallel()?.run(|| {
let subject_builder = &subject_builder;
args.walk_builder()?.build_parallel().run(|| {
let haystack_builder = &haystack_builder;
let matched = &matched;
let tx = tx.clone();
Box::new(move |result| {
let subject = match subject_builder.build_from_result(result) {
Some(subject) => subject,
let haystack = match haystack_builder.build_from_result(result) {
Some(haystack) => haystack,
None => return WalkState::Continue,
};
matched.store(true, SeqCst);
if quit_after_match {
matched.store(true, Ordering::SeqCst);
if args.quit_after_match() {
WalkState::Quit
} else {
match tx.send(subject) {
match tx.send(haystack) {
Ok(_) => WalkState::Continue,
Err(_) => WalkState::Quit,
}
@ -317,18 +318,18 @@ fn files_parallel(args: &Args) -> anyhow::Result<bool> {
// A broken pipe means graceful termination, so fall through.
// Otherwise, something bad happened while writing to stdout, so bubble
// it up.
if err.kind() != io::ErrorKind::BrokenPipe {
if err.kind() != std::io::ErrorKind::BrokenPipe {
return Err(err.into());
}
}
Ok(matched.load(SeqCst))
Ok(matched.load(Ordering::SeqCst))
}
/// The top-level entry point for --type-list.
fn types(args: &Args) -> anyhow::Result<bool> {
/// The top-level entry point for `--type-list`.
fn types(args: &HiArgs) -> anyhow::Result<ExitCode> {
let mut count = 0;
let mut stdout = args.stdout();
for def in args.type_defs()? {
for def in args.types().definitions() {
count += 1;
stdout.write_all(def.name().as_bytes())?;
stdout.write_all(b": ")?;
@ -343,32 +344,156 @@ fn types(args: &Args) -> anyhow::Result<bool> {
}
stdout.write_all(b"\n")?;
}
Ok(count > 0)
Ok(ExitCode::from(if count == 0 { 1 } else { 0 }))
}
/// The top-level entry point for --pcre2-version.
fn pcre2_version(args: &Args) -> anyhow::Result<bool> {
#[cfg(feature = "pcre2")]
fn imp(args: &Args) -> anyhow::Result<bool> {
use grep::pcre2;
/// Implements ripgrep's "generate" modes.
///
/// These modes correspond to generating some kind of ancillary data related
/// to ripgrep. At present, this includes ripgrep's man page (in roff format)
/// and supported shell completions.
fn generate(mode: crate::flags::GenerateMode) -> anyhow::Result<ExitCode> {
use crate::flags::GenerateMode;
let mut stdout = args.stdout();
let output = match mode {
GenerateMode::Man => flags::generate_man_page(),
GenerateMode::CompleteBash => flags::generate_complete_bash(),
GenerateMode::CompleteZsh => flags::generate_complete_zsh(),
GenerateMode::CompleteFish => flags::generate_complete_fish(),
GenerateMode::CompletePowerShell => {
flags::generate_complete_powershell()
}
};
writeln!(std::io::stdout(), "{}", output.trim_end())?;
Ok(ExitCode::from(0))
}
/// Implements ripgrep's "special" modes.
///
/// A special mode is one that generally short-circuits most (not all) of
/// ripgrep's initialization logic and skips right to this routine. The
/// special modes essentially consist of printing help and version output. The
/// idea behind the short circuiting is to ensure there is as little as possible
/// (within reason) that would prevent ripgrep from emitting help output.
///
/// For example, part of the initialization logic that is skipped (among
/// other things) is accessing the current working directory. If that fails,
/// ripgrep emits an error. We don't want to emit an error if it fails and
/// the user requested version or help information.
fn special(mode: crate::flags::SpecialMode) -> anyhow::Result<ExitCode> {
use crate::flags::SpecialMode;
let output = match mode {
SpecialMode::HelpShort => flags::generate_help_short(),
SpecialMode::HelpLong => flags::generate_help_long(),
SpecialMode::VersionShort => flags::generate_version_short(),
SpecialMode::VersionLong => flags::generate_version_long(),
// --pcre2-version is a little special because it emits an error
// exit code if this build of ripgrep doesn't support PCRE2.
SpecialMode::VersionPCRE2 => return version_pcre2(),
};
writeln!(std::io::stdout(), "{}", output.trim_end())?;
Ok(ExitCode::from(0))
}
/// The top-level entry point for `--pcre2-version`.
fn version_pcre2() -> anyhow::Result<ExitCode> {
let mut stdout = std::io::stdout().lock();
#[cfg(feature = "pcre2")]
{
use grep::pcre2;
let (major, minor) = pcre2::version();
writeln!(stdout, "PCRE2 {}.{} is available", major, minor)?;
if cfg!(target_pointer_width = "64") && pcre2::is_jit_available() {
writeln!(stdout, "JIT is available")?;
}
Ok(true)
Ok(ExitCode::from(0))
}
#[cfg(not(feature = "pcre2"))]
fn imp(args: &Args) -> anyhow::Result<bool> {
let mut stdout = args.stdout();
{
writeln!(stdout, "PCRE2 is not available in this build of ripgrep.")?;
Ok(false)
Ok(ExitCode::from(1))
}
}
/// Prints a heuristic error messages when nothing is searched.
///
/// This can happen if an applicable ignore file has one or more rules that
/// are too broad and cause ripgrep to ignore everything.
///
/// We only show this error message when the user does *not* provide an
/// explicit path to search. This is because the message can otherwise be
/// noisy, e.g., when it is intended that there is nothing to search.
fn eprint_nothing_searched() {
err_message!(
"No files were searched, which means ripgrep probably \
applied a filter you didn't expect.\n\
Running with --debug will show why files are being skipped."
);
}
/// Prints the statistics given to the writer given.
///
/// The search mode given determines whether the stats should be printed in
/// a plain text format or in a JSON format.
///
/// The `started` time should be the time at which ripgrep started working.
///
/// If an error occurs while writing, then writing stops and the error is
/// returned. Note that callers should probably ignore this errror, since
/// whether stats fail to print or not generally shouldn't cause ripgrep to
/// enter into an "error" state. And usually the only way for this to fail is
/// if writing to stdout itself fails.
fn print_stats<W: Write>(
mode: SearchMode,
stats: &grep::printer::Stats,
started: std::time::Instant,
mut wtr: W,
) -> std::io::Result<()> {
let elapsed = std::time::Instant::now().duration_since(started);
if matches!(mode, SearchMode::JSON) {
// We specifically match the format laid out by the JSON printer in
// the grep-printer crate. We simply "extend" it with the 'summary'
// message type.
serde_json::to_writer(
&mut wtr,
&serde_json::json!({
"type": "summary",
"data": {
"stats": stats,
"elapsed_total": {
"secs": elapsed.as_secs(),
"nanos": elapsed.subsec_nanos(),
"human": format!("{:0.6}s", elapsed.as_secs_f64()),
},
}
}),
)?;
write!(wtr, "\n")
} else {
write!(
wtr,
"
{matches} matches
{lines} matched lines
{searches_with_match} files contained matches
{searches} files searched
{bytes_printed} bytes printed
{bytes_searched} bytes searched
{search_time:0.6} seconds spent searching
{process_time:0.6} seconds
",
matches = stats.matches(),
lines = stats.matched_lines(),
searches_with_match = stats.searches_with_match(),
searches = stats.searches(),
bytes_printed = stats.bytes_printed(),
bytes_searched = stats.bytes_searched(),
search_time = stats.elapsed().as_secs_f64(),
process_time = elapsed.as_secs_f64(),
)
}
imp(args)
}

61
crates/core/messages.rs
View File

@ -1,21 +1,59 @@
/*!
This module defines some macros and some light shared mutable state.
This state is responsible for keeping track of whether we should emit certain
kinds of messages to the user (such as errors) that are distinct from the
standard "debug" or "trace" log messages. This state is specifically set at
startup time when CLI arguments are parsed and then never changed.
The other state tracked here is whether ripgrep experienced an error
condition. Aside from errors associated with invalid CLI arguments, ripgrep
generally does not abort when an error occurs (e.g., if reading a file failed).
But when an error does occur, it will alter ripgrep's exit status. Thus, when
an error message is emitted via `err_message`, then a global flag is toggled
indicating that at least one error occurred. When ripgrep exits, this flag is
consulted to determine what the exit status ought to be.
*/
use std::sync::atomic::{AtomicBool, Ordering};
/// When false, "messages" will not be printed.
static MESSAGES: AtomicBool = AtomicBool::new(false);
/// When false, "messages" related to ignore rules will not be printed.
static IGNORE_MESSAGES: AtomicBool = AtomicBool::new(false);
/// Flipped to true when an error message is printed.
static ERRORED: AtomicBool = AtomicBool::new(false);
/// Like eprintln, but locks STDOUT to prevent interleaving lines.
/// Like eprintln, but locks stdout to prevent interleaving lines.
///
/// This locks stdout, not stderr, even though this prints to stderr. This
/// avoids the appearance of interleaving output when stdout and stderr both
/// correspond to a tty.)
#[macro_export]
macro_rules! eprintln_locked {
($($tt:tt)*) => {{
{
use std::io::Write;
// This is a bit of an abstraction violation because we explicitly
// lock STDOUT before printing to STDERR. This avoids interleaving
// lock stdout before printing to stderr. This avoids interleaving
// lines within ripgrep because `search_parallel` uses `termcolor`,
// which accesses the same STDOUT lock when writing lines.
// which accesses the same stdout lock when writing lines.
let stdout = std::io::stdout();
let _handle = stdout.lock();
eprintln!($($tt)*);
// We specifically ignore any errors here. One plausible error we
// can get in some cases is a broken pipe error. And when that
// occurs, we should exit gracefully. Otherwise, just abort with
// an error code because there isn't much else we can do.
//
// See: https://github.com/BurntSushi/ripgrep/issues/1966
if let Err(err) = writeln!(std::io::stderr(), $($tt)*) {
if err.kind() == std::io::ErrorKind::BrokenPipe {
std::process::exit(0);
} else {
std::process::exit(2);
}
}
}
}}
}
@ -52,19 +90,19 @@ macro_rules! ignore_message {
}
/// Returns true if and only if messages should be shown.
pub fn messages() -> bool {
pub(crate) fn messages() -> bool {
MESSAGES.load(Ordering::SeqCst)
}
/// Set whether messages should be shown or not.
///
/// By default, they are not shown.
pub fn set_messages(yes: bool) {
pub(crate) fn set_messages(yes: bool) {
MESSAGES.store(yes, Ordering::SeqCst)
}
/// Returns true if and only if "ignore" related messages should be shown.
pub fn ignore_messages() -> bool {
pub(crate) fn ignore_messages() -> bool {
IGNORE_MESSAGES.load(Ordering::SeqCst)
}
@ -75,16 +113,19 @@ pub fn ignore_messages() -> bool {
/// Note that this is overridden if `messages` is disabled. Namely, if
/// `messages` is disabled, then "ignore" messages are never shown, regardless
/// of this setting.
pub fn set_ignore_messages(yes: bool) {
pub(crate) fn set_ignore_messages(yes: bool) {
IGNORE_MESSAGES.store(yes, Ordering::SeqCst)
}
/// Returns true if and only if ripgrep came across a non-fatal error.
pub fn errored() -> bool {
pub(crate) fn errored() -> bool {
ERRORED.load(Ordering::SeqCst)
}
/// Indicate that ripgrep has come across a non-fatal error.
pub fn set_errored() {
///
/// Callers should not use this directly. Instead, it is called automatically
/// via the `err_message` macro.
pub(crate) fn set_errored() {
ERRORED.store(true, Ordering::SeqCst);
}

264
crates/core/search.rs
View File

@ -1,59 +1,47 @@
use std::{
io,
path::{Path, PathBuf},
time::Duration,
};
/*!
Defines a very high level "search worker" abstraction.
use {
grep::{
cli,
matcher::Matcher,
printer::{Standard, Stats, Summary, JSON},
regex::RegexMatcher as RustRegexMatcher,
searcher::{BinaryDetection, Searcher},
},
ignore::overrides::Override,
serde_json::{self as json, json},
termcolor::WriteColor,
};
A search worker manages the high level interaction points between the matcher
(i.e., which regex engine is used), the searcher (i.e., how data is actually
read and matched using the regex engine) and the printer. For example, the
search worker is where things like preprocessors or decompression happens.
*/
#[cfg(feature = "pcre2")]
use grep::pcre2::RegexMatcher as PCRE2RegexMatcher;
use std::{io, path::Path};
use crate::subject::Subject;
use {grep::matcher::Matcher, termcolor::WriteColor};
/// The configuration for the search worker. Among a few other things, the
/// configuration primarily controls the way we show search results to users
/// at a very high level.
/// The configuration for the search worker.
///
/// Among a few other things, the configuration primarily controls the way we
/// show search results to users at a very high level.
#[derive(Clone, Debug)]
struct Config {
json_stats: bool,
preprocessor: Option<PathBuf>,
preprocessor_globs: Override,
preprocessor: Option<std::path::PathBuf>,
preprocessor_globs: ignore::overrides::Override,
search_zip: bool,
binary_implicit: BinaryDetection,
binary_explicit: BinaryDetection,
binary_implicit: grep::searcher::BinaryDetection,
binary_explicit: grep::searcher::BinaryDetection,
}
impl Default for Config {
fn default() -> Config {
Config {
json_stats: false,
preprocessor: None,
preprocessor_globs: Override::empty(),
preprocessor_globs: ignore::overrides::Override::empty(),
search_zip: false,
binary_implicit: BinaryDetection::none(),
binary_explicit: BinaryDetection::none(),
binary_implicit: grep::searcher::BinaryDetection::none(),
binary_explicit: grep::searcher::BinaryDetection::none(),
}
}
}
/// A builder for configuring and constructing a search worker.
#[derive(Clone, Debug)]
pub struct SearchWorkerBuilder {
pub(crate) struct SearchWorkerBuilder {
config: Config,
command_builder: cli::CommandReaderBuilder,
decomp_builder: cli::DecompressionReaderBuilder,
command_builder: grep::cli::CommandReaderBuilder,
decomp_builder: grep::cli::DecompressionReaderBuilder,
}
impl Default for SearchWorkerBuilder {
@ -64,11 +52,11 @@ impl Default for SearchWorkerBuilder {
impl SearchWorkerBuilder {
/// Create a new builder for configuring and constructing a search worker.
pub fn new() -> SearchWorkerBuilder {
let mut cmd_builder = cli::CommandReaderBuilder::new();
pub(crate) fn new() -> SearchWorkerBuilder {
let mut cmd_builder = grep::cli::CommandReaderBuilder::new();
cmd_builder.async_stderr(true);
let mut decomp_builder = cli::DecompressionReaderBuilder::new();
let mut decomp_builder = grep::cli::DecompressionReaderBuilder::new();
decomp_builder.async_stderr(true);
SearchWorkerBuilder {
@ -80,10 +68,10 @@ impl SearchWorkerBuilder {
/// Create a new search worker using the given searcher, matcher and
/// printer.
pub fn build<W: WriteColor>(
pub(crate) fn build<W: WriteColor>(
&self,
matcher: PatternMatcher,
searcher: Searcher,
searcher: grep::searcher::Searcher,
printer: Printer<W>,
) -> SearchWorker<W> {
let config = self.config.clone();
@ -99,29 +87,17 @@ impl SearchWorkerBuilder {
}
}
/// Forcefully use JSON to emit statistics, even if the underlying printer
/// is not the JSON printer.
///
/// This is useful for implementing flag combinations like
/// `--json --quiet`, which uses the summary printer for implementing
/// `--quiet` but still wants to emit summary statistics, which should
/// be JSON formatted because of the `--json` flag.
pub fn json_stats(&mut self, yes: bool) -> &mut SearchWorkerBuilder {
self.config.json_stats = yes;
self
}
/// Set the path to a preprocessor command.
///
/// When this is set, instead of searching files directly, the given
/// command will be run with the file path as the first argument, and the
/// output of that command will be searched instead.
pub fn preprocessor(
pub(crate) fn preprocessor(
&mut self,
cmd: Option<PathBuf>,
cmd: Option<std::path::PathBuf>,
) -> anyhow::Result<&mut SearchWorkerBuilder> {
if let Some(ref prog) = cmd {
let bin = cli::resolve_binary(prog)?;
let bin = grep::cli::resolve_binary(prog)?;
self.config.preprocessor = Some(bin);
} else {
self.config.preprocessor = None;
@ -132,9 +108,9 @@ impl SearchWorkerBuilder {
/// Set the globs for determining which files should be run through the
/// preprocessor. By default, with no globs and a preprocessor specified,
/// every file is run through the preprocessor.
pub fn preprocessor_globs(
pub(crate) fn preprocessor_globs(
&mut self,
globs: Override,
globs: ignore::overrides::Override,
) -> &mut SearchWorkerBuilder {
self.config.preprocessor_globs = globs;
self
@ -147,7 +123,10 @@ impl SearchWorkerBuilder {
///
/// Note that if a preprocessor command is set, then it overrides this
/// setting.
pub fn search_zip(&mut self, yes: bool) -> &mut SearchWorkerBuilder {
pub(crate) fn search_zip(
&mut self,
yes: bool,
) -> &mut SearchWorkerBuilder {
self.config.search_zip = yes;
self
}
@ -155,13 +134,14 @@ impl SearchWorkerBuilder {
/// Set the binary detection that should be used when searching files
/// found via a recursive directory search.
///
/// Generally, this binary detection may be `BinaryDetection::quit` if
/// we want to skip binary files completely.
/// Generally, this binary detection may be
/// `grep::searcher::BinaryDetection::quit` if we want to skip binary files
/// completely.
///
/// By default, no binary detection is performed.
pub fn binary_detection_implicit(
pub(crate) fn binary_detection_implicit(
&mut self,
detection: BinaryDetection,
detection: grep::searcher::BinaryDetection,
) -> &mut SearchWorkerBuilder {
self.config.binary_implicit = detection;
self
@ -170,14 +150,14 @@ impl SearchWorkerBuilder {
/// Set the binary detection that should be used when searching files
/// explicitly supplied by an end user.
///
/// Generally, this binary detection should NOT be `BinaryDetection::quit`,
/// since we never want to automatically filter files supplied by the end
/// user.
/// Generally, this binary detection should NOT be
/// `grep::searcher::BinaryDetection::quit`, since we never want to
/// automatically filter files supplied by the end user.
///
/// By default, no binary detection is performed.
pub fn binary_detection_explicit(
pub(crate) fn binary_detection_explicit(
&mut self,
detection: BinaryDetection,
detection: grep::searcher::BinaryDetection,
) -> &mut SearchWorkerBuilder {
self.config.binary_explicit = detection;
self
@ -191,14 +171,14 @@ impl SearchWorkerBuilder {
/// every search also has some aggregate statistics or meta data that may be
/// useful to higher level routines.
#[derive(Clone, Debug, Default)]
pub struct SearchResult {
pub(crate) struct SearchResult {
has_match: bool,
stats: Option<Stats>,
stats: Option<grep::printer::Stats>,
}
impl SearchResult {
/// Whether the search found a match or not.
pub fn has_match(&self) -> bool {
pub(crate) fn has_match(&self) -> bool {
self.has_match
}
@ -206,103 +186,36 @@ impl SearchResult {
///
/// It can be expensive to compute statistics, so these are only present
/// if explicitly enabled in the printer provided by the caller.
pub fn stats(&self) -> Option<&Stats> {
pub(crate) fn stats(&self) -> Option<&grep::printer::Stats> {
self.stats.as_ref()
}
}
/// The pattern matcher used by a search worker.
#[derive(Clone, Debug)]
pub enum PatternMatcher {
RustRegex(RustRegexMatcher),
pub(crate) enum PatternMatcher {
RustRegex(grep::regex::RegexMatcher),
#[cfg(feature = "pcre2")]
PCRE2(PCRE2RegexMatcher),
PCRE2(grep::pcre2::RegexMatcher),
}
/// The printer used by a search worker.
///
/// The `W` type parameter refers to the type of the underlying writer.
#[derive(Debug)]
pub enum Printer<W> {
#[derive(Clone, Debug)]
pub(crate) enum Printer<W> {
/// Use the standard printer, which supports the classic grep-like format.
Standard(Standard<W>),
Standard(grep::printer::Standard<W>),
/// Use the summary printer, which supports aggregate displays of search
/// results.
Summary(Summary<W>),
Summary(grep::printer::Summary<W>),
/// A JSON printer, which emits results in the JSON Lines format.
JSON(JSON<W>),
JSON(grep::printer::JSON<W>),
}
impl<W: WriteColor> Printer<W> {
fn print_stats(
&mut self,
total_duration: Duration,
stats: &Stats,
) -> io::Result<()> {
match *self {
Printer::JSON(_) => self.print_stats_json(total_duration, stats),
Printer::Standard(_) | Printer::Summary(_) => {
self.print_stats_human(total_duration, stats)
}
}
}
fn print_stats_human(
&mut self,
total_duration: Duration,
stats: &Stats,
) -> io::Result<()> {
write!(
self.get_mut(),
"
{matches} matches
{lines} matched lines
{searches_with_match} files contained matches
{searches} files searched
{bytes_printed} bytes printed
{bytes_searched} bytes searched
{search_time:0.6} seconds spent searching
{process_time:0.6} seconds
",
matches = stats.matches(),
lines = stats.matched_lines(),
searches_with_match = stats.searches_with_match(),
searches = stats.searches(),
bytes_printed = stats.bytes_printed(),
bytes_searched = stats.bytes_searched(),
search_time = fractional_seconds(stats.elapsed()),
process_time = fractional_seconds(total_duration)
)
}
fn print_stats_json(
&mut self,
total_duration: Duration,
stats: &Stats,
) -> io::Result<()> {
// We specifically match the format laid out by the JSON printer in
// the grep-printer crate. We simply "extend" it with the 'summary'
// message type.
let fractional = fractional_seconds(total_duration);
json::to_writer(
self.get_mut(),
&json!({
"type": "summary",
"data": {
"stats": stats,
"elapsed_total": {
"secs": total_duration.as_secs(),
"nanos": total_duration.subsec_nanos(),
"human": format!("{:0.6}s", fractional),
},
}
}),
)?;
write!(self.get_mut(), "\n")
}
/// Return a mutable reference to the underlying printer's writer.
pub fn get_mut(&mut self) -> &mut W {
pub(crate) fn get_mut(&mut self) -> &mut W {
match *self {
Printer::Standard(ref mut p) => p.get_mut(),
Printer::Summary(ref mut p) => p.get_mut(),
@ -316,29 +229,32 @@ impl<W: WriteColor> Printer<W> {
/// It is intended for a single worker to execute many searches, and is
/// generally intended to be used from a single thread. When searching using
/// multiple threads, it is better to create a new worker for each thread.
#[derive(Debug)]
pub struct SearchWorker<W> {
#[derive(Clone, Debug)]
pub(crate) struct SearchWorker<W> {
config: Config,
command_builder: cli::CommandReaderBuilder,
decomp_builder: cli::DecompressionReaderBuilder,
command_builder: grep::cli::CommandReaderBuilder,
decomp_builder: grep::cli::DecompressionReaderBuilder,
matcher: PatternMatcher,
searcher: Searcher,
searcher: grep::searcher::Searcher,
printer: Printer<W>,
}
impl<W: WriteColor> SearchWorker<W> {
/// Execute a search over the given subject.
pub fn search(&mut self, subject: &Subject) -> io::Result<SearchResult> {
let bin = if subject.is_explicit() {
/// Execute a search over the given haystack.
pub(crate) fn search(
&mut self,
haystack: &crate::haystack::Haystack,
) -> io::Result<SearchResult> {
let bin = if haystack.is_explicit() {
self.config.binary_explicit.clone()
} else {
self.config.binary_implicit.clone()
};
let path = subject.path();
let path = haystack.path();
log::trace!("{}: binary detection: {:?}", path.display(), bin);
self.searcher.set_binary_detection(bin);
if subject.is_stdin() {
if haystack.is_stdin() {
self.search_reader(path, &mut io::stdin().lock())
} else if self.should_preprocess(path) {
self.search_preprocessor(path)
@ -350,28 +266,10 @@ impl<W: WriteColor> SearchWorker<W> {
}
/// Return a mutable reference to the underlying printer.
pub fn printer(&mut self) -> &mut Printer<W> {
pub(crate) fn printer(&mut self) -> &mut Printer<W> {
&mut self.printer
}
/// Print the given statistics to the underlying writer in a way that is
/// consistent with this searcher's printer's format.
///
/// While `Stats` contains a duration itself, this only corresponds to the
/// time spent searching, where as `total_duration` should roughly
/// approximate the lifespan of the ripgrep process itself.
pub fn print_stats(
&mut self,
total_duration: Duration,
stats: &Stats,
) -> io::Result<()> {
if self.config.json_stats {
self.printer().print_stats_json(total_duration, stats)
} else {
self.printer().print_stats(total_duration, stats)
}
}
/// Returns true if and only if the given file path should be
/// decompressed before searching.
fn should_decompress(&self, path: &Path) -> bool {
@ -399,10 +297,11 @@ impl<W: WriteColor> SearchWorker<W> {
&mut self,
path: &Path,
) -> io::Result<SearchResult> {
use std::{fs::File, process::Stdio};
let bin = self.config.preprocessor.as_ref().unwrap();
let mut cmd = std::process::Command::new(bin);
cmd.arg(path)
.stdin(std::process::Stdio::from(std::fs::File::open(path)?));
cmd.arg(path).stdin(Stdio::from(File::open(path)?));
let mut rdr = self.command_builder.build(&mut cmd).map_err(|err| {
io::Error::new(
@ -478,7 +377,7 @@ impl<W: WriteColor> SearchWorker<W> {
/// searcher and printer.
fn search_path<M: Matcher, W: WriteColor>(
matcher: M,
searcher: &mut Searcher,
searcher: &mut grep::searcher::Searcher,
printer: &mut Printer<W>,
path: &Path,
) -> io::Result<SearchResult> {
@ -514,7 +413,7 @@ fn search_path<M: Matcher, W: WriteColor>(
/// and printer.
fn search_reader<M: Matcher, R: io::Read, W: WriteColor>(
matcher: M,
searcher: &mut Searcher,
searcher: &mut grep::searcher::Searcher,
printer: &mut Printer<W>,
path: &Path,
mut rdr: R,
@ -546,8 +445,3 @@ fn search_reader<M: Matcher, R: io::Read, W: WriteColor>(
}
}
}
/// Return the given duration as fractional seconds.
fn fractional_seconds(duration: Duration) -> f64 {
(duration.as_secs() as f64) + (duration.subsec_nanos() as f64 * 1e-9)
}

2
crates/printer/src/json.rs
View File

@ -447,7 +447,7 @@ impl JSONBuilder {
/// }
/// }
/// ```
#[derive(Debug)]
#[derive(Clone, Debug)]
pub struct JSON<W> {
config: Config,
wtr: CounterWriter<W>,

2
crates/printer/src/standard.rs
View File

@ -489,7 +489,7 @@ impl StandardBuilder {
/// then the `new_no_color` constructor can be used, or, alternatively,
/// the `termcolor::NoColor` adapter can be used to wrap any `io::Write`
/// implementation without enabling any colors.
#[derive(Debug)]
#[derive(Clone, Debug)]
pub struct Standard<W> {
config: Config,
wtr: RefCell<CounterWriter<W>>,

2
crates/printer/src/summary.rs
View File

@ -350,7 +350,7 @@ impl SummaryBuilder {
///
/// This type is generic over `W`, which represents any implementation of
/// the `termcolor::WriteColor` trait.
#[derive(Debug)]
#[derive(Clone, Debug)]
pub struct Summary<W> {
config: Config,
wtr: RefCell<CounterWriter<W>>,

16
crates/printer/src/util.rs
View File

@ -46,7 +46,7 @@ impl<M: Matcher> Replacer<M> {
Replacer { space: None }
}
/// Executes a replacement on the given subject string by replacing all
/// Executes a replacement on the given haystack string by replacing all
/// matches with the given replacement. To access the result of the
/// replacement, use the `replacement` method.
///
@ -55,7 +55,7 @@ impl<M: Matcher> Replacer<M> {
&'a mut self,
searcher: &Searcher,
matcher: &M,
mut subject: &[u8],
mut haystack: &[u8],
range: std::ops::Range<usize>,
replacement: &[u8],
) -> io::Result<()> {
@ -63,8 +63,8 @@ impl<M: Matcher> Replacer<M> {
// do this dance.
let is_multi_line = searcher.multi_line_with_matcher(&matcher);
if is_multi_line {
if subject[range.end..].len() >= MAX_LOOK_AHEAD {
subject = &subject[..range.end + MAX_LOOK_AHEAD];
if haystack[range.end..].len() >= MAX_LOOK_AHEAD {
haystack = &haystack[..range.end + MAX_LOOK_AHEAD];
}
} else {
// When searching a single line, we should remove the line
@ -72,8 +72,8 @@ impl<M: Matcher> Replacer<M> {
// look-around) to observe the line terminator and not match
// because of it.
let mut m = Match::new(0, range.end);
trim_line_terminator(searcher, subject, &mut m);
subject = &subject[..m.end()];
trim_line_terminator(searcher, haystack, &mut m);
haystack = &haystack[..m.end()];
}
{
let &mut Space { ref mut dst, ref mut caps, ref mut matches } =
@ -83,7 +83,7 @@ impl<M: Matcher> Replacer<M> {
replace_with_captures_in_context(
matcher,
subject,
haystack,
range.clone(),
caps,
dst,
@ -91,7 +91,7 @@ impl<M: Matcher> Replacer<M> {
let start = dst.len();
caps.interpolate(
|name| matcher.capture_index(name),
subject,
haystack,
replacement,
dst,
);

2
crates/searcher/src/searcher/mod.rs
View File

@ -124,7 +124,7 @@ impl BinaryDetection {
/// source data from an encoding to UTF-8 before searching.
///
/// An `Encoding` will always be cheap to clone.
#[derive(Clone, Debug)]
#[derive(Clone, Debug, Eq, PartialEq)]
pub struct Encoding(&'static encoding_rs::Encoding);
impl Encoding {

38
tests/feature.rs
View File

@ -411,7 +411,8 @@ rgtest!(
|dir: Dir, mut cmd: TestCommand| {
dir.create("sherlock", SHERLOCK);
let lines = cmd.arg("--stats").arg("Sherlock").stdout();
let lines = cmd.arg("-j1").arg("--stats").arg("Sherlock").stdout();
assert!(lines.contains("Sherlock"));
assert!(lines.contains("2 matched lines"));
assert!(lines.contains("1 files contained matches"));
assert!(lines.contains("1 files searched"));
@ -423,7 +424,40 @@ rgtest!(f411_parallel_search_stats, |dir: Dir, mut cmd: TestCommand| {
dir.create("sherlock_1", SHERLOCK);
dir.create("sherlock_2", SHERLOCK);
let lines = cmd.arg("--stats").arg("Sherlock").stdout();
let lines = cmd.arg("-j2").arg("--stats").arg("Sherlock").stdout();
dbg!(&lines);
assert!(lines.contains("4 matched lines"));
assert!(lines.contains("2 files contained matches"));
assert!(lines.contains("2 files searched"));
assert!(lines.contains("seconds"));
});
rgtest!(
f411_single_threaded_quiet_search_stats,
|dir: Dir, mut cmd: TestCommand| {
dir.create("sherlock", SHERLOCK);
let lines = cmd
.arg("--quiet")
.arg("-j1")
.arg("--stats")
.arg("Sherlock")
.stdout();
assert!(!lines.contains("Sherlock"));
assert!(lines.contains("2 matched lines"));
assert!(lines.contains("1 files contained matches"));
assert!(lines.contains("1 files searched"));
assert!(lines.contains("seconds"));
}
);
rgtest!(f411_parallel_quiet_search_stats, |dir: Dir, mut cmd: TestCommand| {
dir.create("sherlock_1", SHERLOCK);
dir.create("sherlock_2", SHERLOCK);
let lines =
cmd.arg("-j2").arg("--quiet").arg("--stats").arg("Sherlock").stdout();
assert!(!lines.contains("Sherlock"));
assert!(lines.contains("4 matched lines"));
assert!(lines.contains("2 files contained matches"));
assert!(lines.contains("2 files searched"));

13
tests/json.rs
View File

@ -189,6 +189,19 @@ rgtest!(basic, |dir: Dir, mut cmd: TestCommand| {
assert_eq!(msgs[4].unwrap_summary().stats.bytes_printed, 494);
});
rgtest!(quiet_stats, |dir: Dir, mut cmd: TestCommand| {
dir.create("sherlock", SHERLOCK);
cmd.arg("--json")
.arg("--quiet")
.arg("--stats")
.arg("Sherlock Holmes")
.arg("sherlock");
let msgs = json_decode(&cmd.stdout());
assert_eq!(msgs[0].unwrap_summary().stats.searches_with_match, 1);
assert_eq!(msgs[0].unwrap_summary().stats.bytes_searched, 367);
});
#[cfg(unix)]
rgtest!(notutf8, |dir: Dir, mut cmd: TestCommand| {
use std::ffi::OsStr;

1
tests/regression.rs
View File

@ -402,6 +402,7 @@ rgtest!(r428_unrecognized_style, |dir: Dir, mut cmd: TestCommand| {
let output = cmd.cmd().output().unwrap();
let stderr = String::from_utf8_lossy(&output.stderr);
let expected = "\
error parsing flag --colors: \
unrecognized style attribute ''. Choose from: nobold, bold, nointense, \
intense, nounderline, underline.
";

4
tests/util.rs
View File

@ -78,7 +78,7 @@ impl Dir {
nice_err(&dir, fs::remove_dir_all(&dir));
}
nice_err(&dir, repeat(|| fs::create_dir_all(&dir)));
Dir { root: root, dir: dir, pcre2: false }
Dir { root, dir, pcre2: false }
}
/// Use PCRE2 for this test.
@ -167,7 +167,7 @@ impl Dir {
if self.is_pcre2() {
cmd.arg("--pcre2");
}
TestCommand { dir: self.clone(), cmd: cmd }
TestCommand { dir: self.clone(), cmd }
}
/// Returns the path to the ripgrep executable.