1. The macOS grep Has No Multi-Pattern Automaton

The first thing to check, and the thing I checked far too late, is that the grep on macOS is not GNU grep:

$ grep --version
grep (BSD grep, GNU compatible) 2.6.0-FreeBSD

Nearly everything in this post follows from that, because the two have different multi-pattern implementations.

GNU grep compiles the entire pattern set into one automaton and makes a single pass, and BSD grep has no such automaton. That is the FreeBSD maintainers' own description of their own code. I measured the actual behavior.

So why would 1122 patterns cost a month? Because without one, grep reads a line and then evaluates the pattern set against it a pattern at a time, so with 1.27 billion lines that's a multiplication instead of a single pass.

The multiplication is measurable. Three pattern counts against three file sizes, each cell the mean of two runs:

Four times the bytes costs four times the time, and ten times the patterns costs nine to ten times the time, so what you pay for is the product: about 6.5 milliseconds per pattern per mebibyte. Multiply your pattern count by your file size in mebibytes and you have the time: 1,500 pattern-mebibytes is ten seconds and 9,000 is a minute, whatever combination gets you there.

That is what a run costs when nothing matches, which makes it a ceiling. A line that matches lets grep stop trying the rest of the patterns on it, and a run where 99.9% of lines matched came in 27% under.

Against GNU grep, the honest comparison is a cell where neither tool is near its own floor: over 128 MiB with a thousand patterns, BSD grep takes 13m56s and GNU grep 0.401s, 2084x faster. ripgrep takes 0.171s over that same 128 MiB with a thousand patterns, 4887x faster, which is the number on the short version's lead image. Both are five-run medians, measured the way BSD grep's 835.69s was. Pattern scaling separates the two greps the same way: on that same 128 MiB, a hundred times the patterns costs BSD grep 96 times the time and GNU grep 3.2.

Locale is not the explanation either: LC_ALL=C moves BSD grep's 56.2s to 54.8s, a difference inside the run-to-run spread. If you want GNU grep on a Mac, brew install grep installs it as ggrep, and that is where the 2084x speed-up is. Anchoring every pattern to ^uuid, is worth a further 2.31x against its own unanchored run over the full file at its 12.6% match rate, and 6.38x at a 1.5% match rate: the sparser the matches, the bigger the gain. It needs -E, because ^ is a literal caret under -F, and unanchored -E costs nothing by itself, within 1% for both greps, so the anchor is the whole effect.

The commit that made BSD grep the FreeBSD default, in December 2020, calls the slowdown against gnugrep

the price to pay for fewer bugs

before pointing anyone who needs speed at ripgrep or the_silver_searcher.

I have not read either grep's source.

What ripgrep Does Instead

ripgrep has an automaton and doesn't pay for that multiplication. One flag is important: the -a in rg -aF -f tells it to treat the input as text, and without it a stray byte sequence in the user-agent column can make ripgrep decide the stream is binary and stop early. A file that is quietly short is a worse outcome than a run that is slow.

None of ripgrep's own accelerations applied to a set of 1122 UUIDs, and it still came in around 7 minutes, because Aho-Corasick reads each byte of the input once and only once however many patterns it happens to be carrying. Telling it where to look barely helps either: anchoring every pattern to ^uuid,, so that only byte 1 is tested, is worth 1.06x over the full file, and between 1.06x and 1.08x at every match rate from 1.5% to 12.6%. Every anchored figure in this post comes from rounds that re-measured the unanchored baseline beside it. There is also a layer between -F -f and the matcher that I hadn't expected.

grep_regex::config: assembling HIR from 1122 fixed string literals
grep_regex::matcher: final regex: "(?:7faedcb6-90c3-4e9a-9c6f-e2fbf0b998e7)|(?:5fe23ef0-ef4b-...)|..."

2. The macOS awk Costs 3,200 ns per Record

The awk was next. macOS ships Brian Kernighan's "one true awk", so I benchmarked it against the sample, cached, with wc -l on the end:

Each bar is a median of three runs, six for mawk. That 3,200 ns is the cost per record on this file's 259-byte lines, nine tenths of it paid before the program body runs at all:

• find the newline,
• copy the bytes into $0, and
• step the loop.

My first mawk timing was 6.8x the repeat, 0.483s wall against 0.088s of user time where the repeat was 0.071s / 0.058s. The ratio is the tell: a single-threaded program that is not getting the CPU shows a wall time far above its user time, where one that is merely slow shows the two close together. Had I taken 0.483s at face value, mawk would have looked slower than ripgrep's 0.260s and I would have built the pipeline around an rg prefilter it does not need.

On Apple silicon you cannot pin a thread to the performance or the efficiency cores, so a starved run is something you detect afterwards rather than prevent. That is why every timing here is a median of repeated runs with user time recorded beside the wall clock.

A bare record loop, END{print NR} with no program body at all, takes 2.256s against the 2.501s above, so about 90% of BSD awk's time on this sample goes on the loop rather than on anything I wrote. Under mawk the same loop is 0.032s of 0.071s, which is why replacing the awk is the whole fix.

Neither field splitting (1.6%) nor the locale specification (4.5%) made any significant difference here, but LC_ALL=C, which has no impact on mawk, has a significant impact on other tools. Outside the C locale these tools treat the input as characters rather than bytes, so each one has to be decoded from its multibyte form before it can be compared, and ranges and case folding follow the locale's rules rather than the byte values. LC_ALL=C says to treat bytes as bytes. The same program under gawk takes 0.314s without the C locale and 0.195s with it, a 1.6x speed-up. If your data has multibyte characters in it, it is not safe.

Field splitting is not free, and the next section measures it at 81 ns a line, but BSD awk spends 3,200 ns per record, forty times as much, so the splitting is not what makes BSD awk slow.

3. Field Splitting Costs 81 ns per Line

The chart above suggests field splitting costs nothing, but that doesn't generalize. Under mawk, on the full file:

103 seconds over 1.27 billion lines is 81 ns a line. The two outputs are byte-identical, so on this file the two expressions select the same rows, which is one more confirmation that field 1 is always a clean 36-byte UUID.

Why would naming a field cost anything, when the record is already in memory? Because awk splits fields lazily, on first reference. The moment you touch $1, awk has to scan the entire record, including the 101-character quoted user agent, and allocate every field string the row splits into, only to read the first one. By contrast substr($0,1,36) reads 36 bytes and stops, and because the program never mentions a numbered field, the record is never tokenized at all.

So the same 81 ns is 2.5% of BSD awk's 3,200 ns per record, lost in its noise, and 47% of mawk's runtime.

1. Quoted Fields Break awk -F, Silently

awk -F, doesn't understand quoting, so one comma inside "Mozilla/5.0 (KHTML, like Gecko)" shifts every field index after it on that row, and since different user agents carry different numbers of commas, different rows shift by different amounts. You get wrong data rather than a parse error. That is why the pipeline matches on substr($0,1,36): center_id is field 1, ahead of the first quoted column, so it never touches the problem. Anything between the first quoted column and the last is unsafe; $NF is safe as long as nothing after the last quoted column contains a comma.

If you do need a column that sits past a quoted one, gawk 5.3 or newer with --csv is the answer: brew install gawk, Request for Comments (RFC) 4180 as its spec, and 3.3x the cost of a plain -F, split for a correct answer. mawk cannot help at all, having neither FPAT nor a CSV mode in any release.

pigz -dc analytics.csv.gz | awk -F, 'NF!=8{n++} END{print n+0}'

2. The Filter Eats the Header Row

The header row matches none of the patterns, so every filter here throws it away along with the rest of the non-matching lines.

mawk can put it back on its own, which is what the FNR==1|| in the pipeline at the top is doing: awk numbers its records, so line 1 is identifiable from inside the program.

grep and ripgrep have no record number and no way to say "keep the first one", so the shell has to peel the line off the front of the stream before the filter ever sees it:

pigz -dc analytics.csv.gz \
  | { IFS= read -r h; printf '%s\n' "$h"; grep -F -f ids.txt; } \
  | pigz -p 8 > out.gz

pigz -dc analytics.csv.gz \
  | { IFS= read -r h; printf '%s\n' "$h"; rg -aF -f ids.txt; } \
  | pigz -p 8 > out.gz

read takes exactly one line from the shared stdin, and the filter picks up from line 2. The IFS= keeps the header's leading and trailing whitespace from being stripped, and the -r stops backslashes being read as escapes.

3. A Newline Inside a Data Field Splits One Record in Two

This one is about the data file, not the pattern file. RFC 4180 allows a newline inside a quoted field, and mawk, grep and ripgrep all read one line at a time. Where the data contains such a newline, one record arrives as two lines: the second begins in the middle of a record, so substr($0,1,36) reads whatever happens to sit there, and an identifier split across the break matches nothing at all. None of the three reports it.

The analytics file has no such newlines, which is what makes the byte offset safe on it. Counting the same file two ways tells you: wc -l counts lines and gives 1,272,998,467, and DuckDB's CSV reader, which understands quoting, counted 1,272,998,467 records when it loaded the file for the DuckDB post. Equal line and record counts mean no field contains a newline.

4. The Pattern File Matches Everything, or Nothing

Two things to check in ids.txt whichever grep you have. Both fail silently, and in opposite directions.

One empty line in the pattern file matches every input line, so the filter returns the whole input instead of your rows. The tell runs the wrong way: the run gets faster, because an empty pattern matches before any comparison work happens and the job stops filtering and starts copying. BSD grep drops from 10.65s to 0.01s on a 2 MiB slice with 1122 patterns, GNU grep from 0.64s to 0.12s over 200 MiB, ripgrep from 0.27s to 0.09s. More empty lines change nothing and neither does where they sit, so one stray newline at the end of ids.txt does it. mawk is immune, because an empty string is never equal to a 36-byte prefix.

grep -c '^$' ids.txt      # want 0

The wrong line ending breaks it the other way. These are Unix tools, so a line ends at a line feed (LF) and everything in front of it is the pattern. A pattern file saved on Windows ends each line with a carriage return and a line feed (CRLF), so every pattern picks up a trailing carriage return that the data does not have, and nothing matches.

BSD grep returns nothing with exit status 1, and takes longer than a non-CRLF run, 14.30s vs. 10.65s, because a line that matches lets it break out of the pattern loop and a line that cannot match never does. mawk writes your header row and nothing else. ripgrep strips the carriage return and is unaffected.

The old Mac convention of a carriage return with no line feed (CR) fails worse, and it takes ripgrep down with the others: there are no line feeds at all, so the whole file reads as one enormous pattern that matches nothing. wc -l says 0 lines, which is the tell.

file ids.txt              # want "ASCII text", not "with CRLF line terminators"
wc -l < ids.txt           # want your pattern count, not 0

tr -d '\r'   < ids.txt > ids.unix.txt   # CRLF
tr '\r' '\n' < ids.txt > ids.unix.txt   # CR only

5. A Leading Slice Is Accurate for Size and Worthless for Density or Timing

My sample was the leading 200 MiB, taken with head -c. For line length it was fine: 267.2 bytes a line against the file's real 259.0, and a projected 1.234 billion lines against 1,272,998,467, both close enough to plan with.

For match rate it was worthless. 7794 of its 784,857 lines matched, 0.99%, where the finished job matched 12.6%. The file is written in runs of one center at a time, so a slice off the front covers only the centers that happen to start it. One command tells you before you trust it for anything density-related:

pigz -dc analytics.csv.gz | head -c $((200*1024*1024)) | cut -d, -f1 | sort -u | wc -l

Here that is 9 against the 3555 centers in the whole file. A handful of distinct values means the file is grouped by that column, and every projection made from the head of it will be wrong in the direction of whatever the head happens to contain. It was also too small: by the first 1% of the file the cumulative match rate is 13.9%, overshooting the real 12.6% rather than undershooting it.

The same slice is worthless for timing anything whose cost moves with the match rate. Anchored GNU grep takes 2m54s over the full file at a 1.5% match rate and 7m27s at 12.6%, a 2.56x rise. The same two runs over the leading 128 MiB take 0.075s and 0.077s, flat to within 3%. Measure there and you would never know the cost moves. I cut four 128 MiB windows, one from the head and the others a quarter, half and three quarters of the way in, and anchoring made GNU grep about 4.8x faster on every one of them at the job's match rate; over the whole file it is 2.31x.

6. gzip -l Misreports Any Output of 4 GiB or More

The gzip trailer holds the uncompressed size in four bytes, so anything at or above 4 GiB comes back modulo 2^32 with no warning. This file's real 44.2 GB is reported as a ratio of -8.8%, which would mean compressing made it bigger. pigz -l at least refuses to guess, printing unk where the reduction goes when the stored length and the compressed size cannot both be true. Only pigz -dc filtered.csv.gz | wc -c actually answers, in about fourteen seconds.