I need to scan very large JSONL files efficiently and am considering a parallel grep-style approach over line-delimited text.
Would love to hear how you would design it.
I need to scan very large JSONL files efficiently and am considering a parallel grep-style approach over line-delimited text.
Would love to hear how you would design it.
parsing will make it orders of magnitude slower
It will not if your parser is not overcomplicated and does not populate some huge structures with data. You only need to find tokens and compare them with field names and values you are looking for. Regexes are slower and don’t allow processing escaped characters correctly.