Join with more flexible matching criteria #2358

terefang · 2024-12-10T11:36:29Z

terefang
Dec 10, 2024

Is your feature request related to a problem? Please describe.

Today qsv join allows a full complement of structural join options (left, right, outer, etc etc) but only a "case-insensitive" match option.

for the following use cases (join criteria), i had to import/join/export via sqlite instead of using the excellent qsv:

right starts-with left – in sql: right_val LIKE CONCAT(left_val,'%')
right contains left – in sql: right_val LIKE CONCAT('%',left_val,'%')
right ends-with left – in sql: right_val LIKE CONCAT('%',left_val)
left starts-with right – in sql: left_val LIKE CONCAT(right_val,'%')
left contains right – in sql: left_val LIKE CONCAT('%',right_val,'%')
left ends-with right – in sql: left_val LIKE CONCAT('%',right_val)

Describe the solution you'd like

an option to specify the match direction — eg --match-direction=LTR|RTL
an option to sepcify the match type — eg --match-type=startswith|contains|endswith

Describe alternatives you've considered

the current workaround is to import both csv files into sqlite and export the joined product.

jqnatividad · 2024-12-10T12:44:03Z

jqnatividad
Dec 10, 2024
Maintainer

Thanks for the detailed request @terefang .

Out of curiosity, have you tried the joinp command or even doing the join using sqlp?

0 replies

terefang · 2024-12-10T13:31:24Z

terefang
Dec 10, 2024
Author

if i understand joinp correctly, it does not solve my use-cases.

0 replies

terefang · 2024-12-10T14:33:05Z

terefang
Dec 10, 2024
Author

just tried out sqlp.

The error Message is "Unsupported SQL join constraint" for select * from "report" r left join "report2" r2 on (r2.val2 like concat(r.val1,'%'))

0 replies

jqnatividad · 2024-12-11T11:38:15Z

jqnatividad
Dec 11, 2024
Maintainer

Hi @terefang , given that join uses a hashmap as its main data structure, it'd be a major refactor to implement pattern matching there.
I'm currently investigating implementing your request in joinp instead as it uses Polars and it has built-ins for pattern matching.

0 replies

jqnatividad · 2024-12-18T15:50:35Z

jqnatividad
Dec 18, 2024
Maintainer

Hi @terefang , I created some tests illustrating how to do joins per your request using sqlp and joinp...

They both use cross joins though...

0 replies

ondohotola · 2024-12-18T16:39:03Z

ondohotola
Dec 18, 2024

Where are the tests?

0 replies

jqnatividad · 2024-12-18T18:16:05Z

jqnatividad
Dec 18, 2024
Maintainer

They're here @ondohotola
23938d4

0 replies

ondohotola · 2024-12-18T18:35:08Z

ondohotola
Dec 18, 2024

Thank you.

0 replies

jqnatividad · 2024-12-18T21:18:58Z

jqnatividad
Dec 18, 2024
Maintainer

Closing this for now...

FYI, I did try to extend joinp to have --match-type and --match-dir but it was not feasible as it ultimately required a cross join internally as the --filter-left and --filter-right can only refer to columns on the left and right respectively.

I also attempted to do it in sqlp, but Polars SQL currently only allows literals after LIKE and ILIKE and even CONCAT caused a run-time Polars SQL error.

As Polars SQL expands over time, I'll revisit this.

0 replies

terefang · 2024-12-18T21:33:32Z

terefang
Dec 18, 2024
Author

i will test this after my vacation

0 replies

jqnatividad · 2024-12-19T10:27:34Z

jqnatividad
Dec 19, 2024
Maintainer

In the interim, I created a discussion based on your original request so other users can find the workarounds.

7 replies

terefang Jan 7, 2025
Author

and that is a bummer

$ qsv joinp --filter-left "Location != '' and Location != 'undefined'" --filer-right "name != ''" --non-equi "starts_with(Location_left,name_right)" colt_svc.csv cmn_location.utf.csv 
Killed

jqnatividad Jan 7, 2025
Maintainer

there's a typo in the last command... --filer-right should be --filter-right.

Also, before running the command, can you set the env variable QSV_LOG_LEVEL=debug?

On Linux/macOS - export QSV_LOG_LEVEL=debug and it will produce a log file in the current working directory.

terefang Jan 7, 2025
Author

there's a typo in the last command... --filer-right should be --filter-right.

Also, before running the command, can you set the env variable QSV_LOG_LEVEL=debug?

On Linux/macOS - export QSV_LOG_LEVEL=debug and it will produce a log file in the current working directory.

[2025-01-07 15:06:31.417112 +00:00] INFO [qsv::util] src/util.rs:843: START: joinp --filter-left Location != '' and Location != 'undefined' --filter-right name!='' --non-equi starts_with(Location_left,name_right) colt_svc.csv cmn_location.utf.csv
[2025-01-07 15:06:39.822489 +00:00] INFO [qsv::util] src/util.rs:1435: No valid .env file found. Proceeding with default settings and current environment variable settings.
[2025-01-07 15:06:39.869835 +00:00] DEBUG [sqlparser::parser] /home/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/sqlparser-0.52.0/src/parser/mod.rs:387: Parsing sql 'Location != '' and Location != 'undefined''...
[2025-01-07 15:06:39.879169 +00:00] DEBUG [sqlparser::parser] /home/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/sqlparser-0.52.0/src/parser/mod.rs:914: parsing expr
[2025-01-07 15:06:39.879901 +00:00] DEBUG [sqlparser::parser] /home/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/sqlparser-0.52.0/src/parser/mod.rs:916: prefix: Identifier(Ident { value: "Location", quote_style: None })
[2025-01-07 15:06:39.880453 +00:00] DEBUG [sqlparser::dialect] /home/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/sqlparser-0.52.0/src/dialect/mod.rs:381: get_next_precedence_full() TokenWithLocation { token: Neq, location: Location { line: 1, column: 10 } }
[2025-01-07 15:06:39.885359 +00:00] DEBUG [sqlparser::parser] /home/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/sqlparser-0.52.0/src/parser/mod.rs:919: next precedence: 20
[2025-01-07 15:06:39.885368 +00:00] DEBUG [sqlparser::parser] /home/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/sqlparser-0.52.0/src/parser/mod.rs:914: parsing expr
[2025-01-07 15:06:39.885379 +00:00] DEBUG [sqlparser::parser] /home/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/sqlparser-0.52.0/src/parser/mod.rs:916: prefix: Value(SingleQuotedString(""))
[2025-01-07 15:06:39.885392 +00:00] DEBUG [sqlparser::dialect] /home/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/sqlparser-0.52.0/src/dialect/mod.rs:381: get_next_precedence_full() TokenWithLocation { token: Word(Word { value: "and", quote_style: None, keyword: AND }), location: Location { line: 1, column: 16 } }
[2025-01-07 15:06:39.885407 +00:00] DEBUG [sqlparser::parser] /home/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/sqlparser-0.52.0/src/parser/mod.rs:919: next precedence: 10
[2025-01-07 15:06:39.885410 +00:00] DEBUG [sqlparser::dialect] /home/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/sqlparser-0.52.0/src/dialect/mod.rs:381: get_next_precedence_full() TokenWithLocation { token: Word(Word { value: "and", quote_style: None, keyword: AND }), location: Location { line: 1, column: 16 } }
[2025-01-07 15:06:39.885415 +00:00] DEBUG [sqlparser::parser] /home/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/sqlparser-0.52.0/src/parser/mod.rs:919: next precedence: 10
[2025-01-07 15:06:39.885419 +00:00] DEBUG [sqlparser::parser] /home/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/sqlparser-0.52.0/src/parser/mod.rs:914: parsing expr
[2025-01-07 15:06:39.885426 +00:00] DEBUG [sqlparser::parser] /home/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/sqlparser-0.52.0/src/parser/mod.rs:916: prefix: Identifier(Ident { value: "Location", quote_style: None })
[2025-01-07 15:06:39.885430 +00:00] DEBUG [sqlparser::dialect] /home/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/sqlparser-0.52.0/src/dialect/mod.rs:381: get_next_precedence_full() TokenWithLocation { token: Neq, location: Location { line: 1, column: 29 } }
[2025-01-07 15:06:39.885434 +00:00] DEBUG [sqlparser::parser] /home/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/sqlparser-0.52.0/src/parser/mod.rs:919: next precedence: 20
[2025-01-07 15:06:39.885437 +00:00] DEBUG [sqlparser::parser] /home/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/sqlparser-0.52.0/src/parser/mod.rs:914: parsing expr
[2025-01-07 15:06:39.885444 +00:00] DEBUG [sqlparser::parser] /home/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/sqlparser-0.52.0/src/parser/mod.rs:916: prefix: Value(SingleQuotedString("undefined"))
[2025-01-07 15:06:39.885447 +00:00] DEBUG [sqlparser::dialect] /home/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/sqlparser-0.52.0/src/dialect/mod.rs:381: get_next_precedence_full() TokenWithLocation { token: EOF, location: Location { line: 0, column: 0 } }
[2025-01-07 15:06:39.885460 +00:00] DEBUG [sqlparser::parser] /home/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/sqlparser-0.52.0/src/parser/mod.rs:919: next precedence: 0
[2025-01-07 15:06:39.885463 +00:00] DEBUG [sqlparser::dialect] /home/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/sqlparser-0.52.0/src/dialect/mod.rs:381: get_next_precedence_full() TokenWithLocation { token: EOF, location: Location { line: 0, column: 0 } }
[2025-01-07 15:06:39.885467 +00:00] DEBUG [sqlparser::parser] /home/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/sqlparser-0.52.0/src/parser/mod.rs:919: next precedence: 0
[2025-01-07 15:06:39.885470 +00:00] DEBUG [sqlparser::dialect] /home/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/sqlparser-0.52.0/src/dialect/mod.rs:381: get_next_precedence_full() TokenWithLocation { token: EOF, location: Location { line: 0, column: 0 } }
[2025-01-07 15:06:39.885473 +00:00] DEBUG [sqlparser::parser] /home/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/sqlparser-0.52.0/src/parser/mod.rs:919: next precedence: 0
[2025-01-07 15:06:39.893253 +00:00] DEBUG [sqlparser::parser] /home/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/sqlparser-0.52.0/src/parser/mod.rs:387: Parsing sql 'name!='''...
[2025-01-07 15:06:39.893273 +00:00] DEBUG [sqlparser::parser] /home/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/sqlparser-0.52.0/src/parser/mod.rs:914: parsing expr
[2025-01-07 15:06:39.893282 +00:00] DEBUG [sqlparser::parser] /home/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/sqlparser-0.52.0/src/parser/mod.rs:916: prefix: Identifier(Ident { value: "name", quote_style: None })
[2025-01-07 15:06:39.893288 +00:00] DEBUG [sqlparser::dialect] /home/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/sqlparser-0.52.0/src/dialect/mod.rs:381: get_next_precedence_full() TokenWithLocation { token: Neq, location: Location { line: 1, column: 5 } }
[2025-01-07 15:06:39.893293 +00:00] DEBUG [sqlparser::parser] /home/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/sqlparser-0.52.0/src/parser/mod.rs:919: next precedence: 20
[2025-01-07 15:06:39.893297 +00:00] DEBUG [sqlparser::parser] /home/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/sqlparser-0.52.0/src/parser/mod.rs:914: parsing expr
[2025-01-07 15:06:39.893303 +00:00] DEBUG [sqlparser::parser] /home/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/sqlparser-0.52.0/src/parser/mod.rs:916: prefix: Value(SingleQuotedString(""))
[2025-01-07 15:06:39.893307 +00:00] DEBUG [sqlparser::dialect] /home/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/sqlparser-0.52.0/src/dialect/mod.rs:381: get_next_precedence_full() TokenWithLocation { token: EOF, location: Location { line: 0, column: 0 } }
[2025-01-07 15:06:39.893311 +00:00] DEBUG [sqlparser::parser] /home/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/sqlparser-0.52.0/src/parser/mod.rs:919: next precedence: 0
[2025-01-07 15:06:39.893314 +00:00] DEBUG [sqlparser::dialect] /home/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/sqlparser-0.52.0/src/dialect/mod.rs:381: get_next_precedence_full() TokenWithLocation { token: EOF, location: Location { line: 0, column: 0 } }
[2025-01-07 15:06:39.893318 +00:00] DEBUG [sqlparser::parser] /home/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/sqlparser-0.52.0/src/parser/mod.rs:919: next precedence: 0
[2025-01-07 15:06:39.893817 +00:00] DEBUG [sqlparser::parser] /home/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/sqlparser-0.52.0/src/parser/mod.rs:387: Parsing sql 'starts_with(Location_left,name_right)'...
[2025-01-07 15:06:39.893836 +00:00] DEBUG [sqlparser::parser] /home/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/sqlparser-0.52.0/src/parser/mod.rs:914: parsing expr
[2025-01-07 15:06:39.893848 +00:00] DEBUG [sqlparser::parser] /home/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/sqlparser-0.52.0/src/parser/mod.rs:914: parsing expr
[2025-01-07 15:06:39.893887 +00:00] DEBUG [sqlparser::parser] /home/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/sqlparser-0.52.0/src/parser/mod.rs:916: prefix: Identifier(Ident { value: "Location_left", quote_style: None })
[2025-01-07 15:06:39.893892 +00:00] DEBUG [sqlparser::dialect] /home/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/sqlparser-0.52.0/src/dialect/mod.rs:381: get_next_precedence_full() TokenWithLocation { token: Comma, location: Location { line: 1, column: 26 } }
[2025-01-07 15:06:39.893896 +00:00] DEBUG [sqlparser::parser] /home/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/sqlparser-0.52.0/src/parser/mod.rs:919: next precedence: 0
[2025-01-07 15:06:39.893906 +00:00] DEBUG [sqlparser::parser] /home/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/sqlparser-0.52.0/src/parser/mod.rs:914: parsing expr
[2025-01-07 15:06:39.893911 +00:00] DEBUG [sqlparser::parser] /home/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/sqlparser-0.52.0/src/parser/mod.rs:916: prefix: Identifier(Ident { value: "name_right", quote_style: None })
[2025-01-07 15:06:39.893915 +00:00] DEBUG [sqlparser::dialect] /home/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/sqlparser-0.52.0/src/dialect/mod.rs:381: get_next_precedence_full() TokenWithLocation { token: RParen, location: Location { line: 1, column: 37 } }
[2025-01-07 15:06:39.893918 +00:00] DEBUG [sqlparser::parser] /home/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/sqlparser-0.52.0/src/parser/mod.rs:919: next precedence: 0
[2025-01-07 15:06:39.893923 +00:00] DEBUG [sqlparser::parser] /home/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/sqlparser-0.52.0/src/parser/mod.rs:916: prefix: Function(Function { name: ObjectName([Ident { value: "starts_with", quote_style: None }]), parameters: None, args: List(FunctionArgumentList { duplicate_treatment: None, args: [Unnamed(Expr(Identifier(Ident { value: "Location_left", quote_style: None }))), Unnamed(Expr(Identifier(Ident { value: "name_right", quote_style: None })))], clauses: [] }), filter: None, null_treatment: None, over: None, within_group: [] })
[2025-01-07 15:06:39.893945 +00:00] DEBUG [sqlparser::dialect] /home/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/sqlparser-0.52.0/src/dialect/mod.rs:381: get_next_precedence_full() TokenWithLocation { token: EOF, location: Location { line: 0, column: 0 } }
[2025-01-07 15:06:39.893949 +00:00] DEBUG [sqlparser::parser] /home/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/sqlparser-0.52.0/src/parser/mod.rs:919: next precedence: 0

jqnatividad Jan 7, 2025
Maintainer

Can you also set RUST_BACKTRACE=1?

terefang Jan 7, 2025
Author

it looks like it is killed by the kernal for OOME

$ cat qsv_rCURRENT.log
[2025-01-07 22:16:20.985522 +00:00] INFO [qsv::util] src/util.rs:843: START: joinp --filter-left Location != '' and Location != 'undefined' --filter-right name!='' --non-equi starts_with(Location_left,name_right) colt_svc.csv cmn_location.utf.csv
[2025-01-07 22:16:21.268734 +00:00] INFO [qsv::util] src/util.rs:1435: No valid .env file found. Proceeding with default settings and current environment variable settings.
[2025-01-07 22:16:21.326386 +00:00] DEBUG [sqlparser::parser] /home/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/sqlparser-0.52.0/src/parser/mod.rs:387: Parsing sql 'Location != '' and Location != 'undefined''...
[2025-01-07 22:16:21.335973 +00:00] DEBUG [sqlparser::parser] /home/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/sqlparser-0.52.0/src/parser/mod.rs:914: parsing expr
[2025-01-07 22:16:21.336608 +00:00] DEBUG [sqlparser::parser] /home/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/sqlparser-0.52.0/src/parser/mod.rs:916: prefix: Identifier(Ident { value: "Location", quote_style: None })
[2025-01-07 22:16:21.341371 +00:00] DEBUG [sqlparser::dialect] /home/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/sqlparser-0.52.0/src/dialect/mod.rs:381: get_next_precedence_full() TokenWithLocation { token: Neq, location: Location { line: 1, column: 10 } }
[2025-01-07 22:16:21.342386 +00:00] DEBUG [sqlparser::parser] /home/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/sqlparser-0.52.0/src/parser/mod.rs:919: next precedence: 20
[2025-01-07 22:16:21.342394 +00:00] DEBUG [sqlparser::parser] /home/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/sqlparser-0.52.0/src/parser/mod.rs:914: parsing expr
[2025-01-07 22:16:21.342405 +00:00] DEBUG [sqlparser::parser] /home/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/sqlparser-0.52.0/src/parser/mod.rs:916: prefix: Value(SingleQuotedString(""))
[2025-01-07 22:16:21.342420 +00:00] DEBUG [sqlparser::dialect] /home/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/sqlparser-0.52.0/src/dialect/mod.rs:381: get_next_precedence_full() TokenWithLocation { token: Word(Word { value: "and", quote_style: None, keyword: AND }), location: Location { line: 1, column: 16 } }
[2025-01-07 22:16:21.342435 +00:00] DEBUG [sqlparser::parser] /home/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/sqlparser-0.52.0/src/parser/mod.rs:919: next precedence: 10
[2025-01-07 22:16:21.342440 +00:00] DEBUG [sqlparser::dialect] /home/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/sqlparser-0.52.0/src/dialect/mod.rs:381: get_next_precedence_full() TokenWithLocation { token: Word(Word { value: "and", quote_style: None, keyword: AND }), location: Location { line: 1, column: 16 } }
[2025-01-07 22:16:21.342445 +00:00] DEBUG [sqlparser::parser] /home/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/sqlparser-0.52.0/src/parser/mod.rs:919: next precedence: 10
[2025-01-07 22:16:21.342450 +00:00] DEBUG [sqlparser::parser] /home/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/sqlparser-0.52.0/src/parser/mod.rs:914: parsing expr
[2025-01-07 22:16:21.342457 +00:00] DEBUG [sqlparser::parser] /home/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/sqlparser-0.52.0/src/parser/mod.rs:916: prefix: Identifier(Ident { value: "Location", quote_style: None })
[2025-01-07 22:16:21.342461 +00:00] DEBUG [sqlparser::dialect] /home/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/sqlparser-0.52.0/src/dialect/mod.rs:381: get_next_precedence_full() TokenWithLocation { token: Neq, location: Location { line: 1, column: 29 } }
[2025-01-07 22:16:21.342466 +00:00] DEBUG [sqlparser::parser] /home/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/sqlparser-0.52.0/src/parser/mod.rs:919: next precedence: 20
[2025-01-07 22:16:21.342469 +00:00] DEBUG [sqlparser::parser] /home/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/sqlparser-0.52.0/src/parser/mod.rs:914: parsing expr
[2025-01-07 22:16:21.342476 +00:00] DEBUG [sqlparser::parser] /home/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/sqlparser-0.52.0/src/parser/mod.rs:916: prefix: Value(SingleQuotedString("undefined"))
[2025-01-07 22:16:21.342489 +00:00] DEBUG [sqlparser::dialect] /home/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/sqlparser-0.52.0/src/dialect/mod.rs:381: get_next_precedence_full() TokenWithLocation { token: EOF, location: Location { line: 0, column: 0 } }
[2025-01-07 22:16:21.342494 +00:00] DEBUG [sqlparser::parser] /home/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/sqlparser-0.52.0/src/parser/mod.rs:919: next precedence: 0
[2025-01-07 22:16:21.342497 +00:00] DEBUG [sqlparser::dialect] /home/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/sqlparser-0.52.0/src/dialect/mod.rs:381: get_next_precedence_full() TokenWithLocation { token: EOF, location: Location { line: 0, column: 0 } }
[2025-01-07 22:16:21.342501 +00:00] DEBUG [sqlparser::parser] /home/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/sqlparser-0.52.0/src/parser/mod.rs:919: next precedence: 0
[2025-01-07 22:16:21.342505 +00:00] DEBUG [sqlparser::dialect] /home/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/sqlparser-0.52.0/src/dialect/mod.rs:381: get_next_precedence_full() TokenWithLocation { token: EOF, location: Location { line: 0, column: 0 } }
[2025-01-07 22:16:21.342509 +00:00] DEBUG [sqlparser::parser] /home/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/sqlparser-0.52.0/src/parser/mod.rs:919: next precedence: 0
[2025-01-07 22:16:21.344620 +00:00] DEBUG [sqlparser::parser] /home/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/sqlparser-0.52.0/src/parser/mod.rs:387: Parsing sql 'name!='''...
[2025-01-07 22:16:21.344634 +00:00] DEBUG [sqlparser::parser] /home/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/sqlparser-0.52.0/src/parser/mod.rs:914: parsing expr
[2025-01-07 22:16:21.344641 +00:00] DEBUG [sqlparser::parser] /home/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/sqlparser-0.52.0/src/parser/mod.rs:916: prefix: Identifier(Ident { value: "name", quote_style: None })
[2025-01-07 22:16:21.344646 +00:00] DEBUG [sqlparser::dialect] /home/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/sqlparser-0.52.0/src/dialect/mod.rs:381: get_next_precedence_full() TokenWithLocation { token: Neq, location: Location { line: 1, column: 5 } }
[2025-01-07 22:16:21.344650 +00:00] DEBUG [sqlparser::parser] /home/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/sqlparser-0.52.0/src/parser/mod.rs:919: next precedence: 20
[2025-01-07 22:16:21.344653 +00:00] DEBUG [sqlparser::parser] /home/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/sqlparser-0.52.0/src/parser/mod.rs:914: parsing expr
[2025-01-07 22:16:21.344659 +00:00] DEBUG [sqlparser::parser] /home/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/sqlparser-0.52.0/src/parser/mod.rs:916: prefix: Value(SingleQuotedString(""))
[2025-01-07 22:16:21.344662 +00:00] DEBUG [sqlparser::dialect] /home/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/sqlparser-0.52.0/src/dialect/mod.rs:381: get_next_precedence_full() TokenWithLocation { token: EOF, location: Location { line: 0, column: 0 } }
[2025-01-07 22:16:21.344666 +00:00] DEBUG [sqlparser::parser] /home/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/sqlparser-0.52.0/src/parser/mod.rs:919: next precedence: 0
[2025-01-07 22:16:21.344669 +00:00] DEBUG [sqlparser::dialect] /home/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/sqlparser-0.52.0/src/dialect/mod.rs:381: get_next_precedence_full() TokenWithLocation { token: EOF, location: Location { line: 0, column: 0 } }
[2025-01-07 22:16:21.344672 +00:00] DEBUG [sqlparser::parser] /home/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/sqlparser-0.52.0/src/parser/mod.rs:919: next precedence: 0
[2025-01-07 22:16:21.345124 +00:00] DEBUG [sqlparser::parser] /home/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/sqlparser-0.52.0/src/parser/mod.rs:387: Parsing sql 'starts_with(Location_left,name_right)'...
[2025-01-07 22:16:21.345557 +00:00] DEBUG [sqlparser::parser] /home/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/sqlparser-0.52.0/src/parser/mod.rs:914: parsing expr
[2025-01-07 22:16:21.345565 +00:00] DEBUG [sqlparser::parser] /home/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/sqlparser-0.52.0/src/parser/mod.rs:914: parsing expr
[2025-01-07 22:16:21.345568 +00:00] DEBUG [sqlparser::parser] /home/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/sqlparser-0.52.0/src/parser/mod.rs:916: prefix: Identifier(Ident { value: "Location_left", quote_style: None })
[2025-01-07 22:16:21.345570 +00:00] DEBUG [sqlparser::dialect] /home/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/sqlparser-0.52.0/src/dialect/mod.rs:381: get_next_precedence_full() TokenWithLocation { token: Comma, location: Location { line: 1, column: 26 } }
[2025-01-07 22:16:21.345572 +00:00] DEBUG [sqlparser::parser] /home/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/sqlparser-0.52.0/src/parser/mod.rs:919: next precedence: 0
[2025-01-07 22:16:21.345577 +00:00] DEBUG [sqlparser::parser] /home/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/sqlparser-0.52.0/src/parser/mod.rs:914: parsing expr
[2025-01-07 22:16:21.345580 +00:00] DEBUG [sqlparser::parser] /home/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/sqlparser-0.52.0/src/parser/mod.rs:916: prefix: Identifier(Ident { value: "name_right", quote_style: None })
[2025-01-07 22:16:21.345581 +00:00] DEBUG [sqlparser::dialect] /home/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/sqlparser-0.52.0/src/dialect/mod.rs:381: get_next_precedence_full() TokenWithLocation { token: RParen, location: Location { line: 1, column: 37 } }
[2025-01-07 22:16:21.345583 +00:00] DEBUG [sqlparser::parser] /home/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/sqlparser-0.52.0/src/parser/mod.rs:919: next precedence: 0
[2025-01-07 22:16:21.345586 +00:00] DEBUG [sqlparser::parser] /home/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/sqlparser-0.52.0/src/parser/mod.rs:916: prefix: Function(Function { name: ObjectName([Ident { value: "starts_with", quote_style: None }]), parameters: None, args: List(FunctionArgumentList { duplicate_treatment: None, args: [Unnamed(Expr(Identifier(Ident { value: "Location_left", quote_style: None }))), Unnamed(Expr(Identifier(Ident { value: "name_right", quote_style: None })))], clauses: [] }), filter: None, null_treatment: None, over: None, within_group: [] })
[2025-01-07 22:16:21.345599 +00:00] DEBUG [sqlparser::dialect] /home/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/sqlparser-0.52.0/src/dialect/mod.rs:381: get_next_precedence_full() TokenWithLocation { token: EOF, location: Location { line: 0, column: 0 } }
[2025-01-07 22:16:21.345601 +00:00] DEBUG [sqlparser::parser] /home/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/sqlparser-0.52.0/src/parser/mod.rs:919: next precedence: 0

jqnatividad · 2025-01-30T18:58:56Z

jqnatividad
Jan 30, 2025
Maintainer

Hi @terefang , can you give it a try with qsv v2.2.1? Also, what does qsv --version say?

Also, how big are the files? Number of rows? Number of columns? File size?

0 replies

terefang · 2025-02-02T00:26:43Z

terefang
Feb 2, 2025
Author

the left files are about 200k+ rows at about 35-50MB
the right files are about 20 to 500 rows of up to 256kB

$ qsv --version
qsv 2.0.0-mimalloc-apply;fetch;foreach;geocode;polars-0.45.1-py-1.19.0-e0ef7f9;self_update-12-12;50.19 GiB-5.65 GiB-48.75 GiB-62.74 GiB (x86_64-unknown-linux-musl compiled with Rust 1.83) prebuilt-older

will update to 2.2.1 and post result

0 replies

terefang · 2025-02-26T07:17:56Z

terefang
Feb 26, 2025
Author

i tested the command from above with the exact same data files:

$ qsv --version
qsv 2.2.1-mimalloc-apply;fetch;foreach;geocode;polars-0.46.0:release;self_update-12-12;50.19 GiB-251.69 MiB-36.20 GiB-62.74 GiB (x86_64-unknown-linux-musl compiled with Rust 1.84) prebuilt-older
$ wc -l colt_svc.csv
1548 colt_svc.csv
$ wc -c colt_svc.csv
142921 colt_svc.csv
$ wc -l cmn_location.utf.csv
253754 cmn_location.utf.csv
$ wc -c cmn_location.utf.csv
148469652 cmn_location.utf.csv
$ qsv joinp --filter-left "Location != '' and Location != 'undefined'" --filter-right "name != ''" --non-equi "starts_with(Location_left,name_right)" colt_svc.csv cmn_location.utf.csv 
Killed

0 replies

terefang · 2025-02-26T07:22:47Z

terefang
Feb 26, 2025
Author

log from running with RUST_BACKTRACE=1 QSV_LOG_LEVEL=debug

[2025-02-26 07:19:31.204257 +00:00] INFO [qsv::util] src/util.rs:854: START: joinp --filter-left Location != '' and Location != 'undefined' --filter-right name != '' --non-equi starts_with(Location_left,name_right) colt_svc.csv cmn_location.utf.csv
[2025-02-26 07:19:31.530088 +00:00] INFO [qsv::util] src/util.rs:1446: No valid .env file found. Proceeding with default settings and current environment variable settings.
[2025-02-26 07:19:31.666107 +00:00] DEBUG [sqlparser::parser] /home/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/sqlparser-0.53.0/src/parser/mod.rs:388: Parsing sql 'Location != '' and Location != 'undefined''...
[2025-02-26 07:19:31.691942 +00:00] DEBUG [sqlparser::parser] /home/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/sqlparser-0.53.0/src/parser/mod.rs:963: parsing expr
[2025-02-26 07:19:31.692078 +00:00] DEBUG [sqlparser::parser] /home/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/sqlparser-0.53.0/src/parser/mod.rs:965: prefix: Identifier(Ident { value: "Location", quote_style: None, span: Span(Location(1,1)..Location(1,9)) })
[2025-02-26 07:19:31.693012 +00:00] DEBUG [sqlparser::dialect] /home/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/sqlparser-0.53.0/src/dialect/mod.rs:433: get_next_precedence_full() TokenWithSpan { token: Neq, span: Span(Location(1,10)..Location(1,12)) }
[2025-02-26 07:19:31.693436 +00:00] DEBUG [sqlparser::parser] /home/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/sqlparser-0.53.0/src/parser/mod.rs:968: next precedence: 20
[2025-02-26 07:19:31.693444 +00:00] DEBUG [sqlparser::parser] /home/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/sqlparser-0.53.0/src/parser/mod.rs:963: parsing expr
[2025-02-26 07:19:31.693454 +00:00] DEBUG [sqlparser::parser] /home/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/sqlparser-0.53.0/src/parser/mod.rs:965: prefix: Value(SingleQuotedString(""))
[2025-02-26 07:19:31.694071 +00:00] DEBUG [sqlparser::dialect] /home/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/sqlparser-0.53.0/src/dialect/mod.rs:433: get_next_precedence_full() TokenWithSpan { token: Word(Word { value: "and", quote_style: None, keyword: AND }), span: Span(Location(1,16)..Location(1,19)) }
[2025-02-26 07:19:31.694089 +00:00] DEBUG [sqlparser::parser] /home/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/sqlparser-0.53.0/src/parser/mod.rs:968: next precedence: 10
[2025-02-26 07:19:31.694094 +00:00] DEBUG [sqlparser::dialect] /home/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/sqlparser-0.53.0/src/dialect/mod.rs:433: get_next_precedence_full() TokenWithSpan { token: Word(Word { value: "and", quote_style: None, keyword: AND }), span: Span(Location(1,16)..Location(1,19)) }
[2025-02-26 07:19:31.694099 +00:00] DEBUG [sqlparser::parser] /home/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/sqlparser-0.53.0/src/parser/mod.rs:968: next precedence: 10
[2025-02-26 07:19:31.694103 +00:00] DEBUG [sqlparser::parser] /home/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/sqlparser-0.53.0/src/parser/mod.rs:963: parsing expr
[2025-02-26 07:19:31.694110 +00:00] DEBUG [sqlparser::parser] /home/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/sqlparser-0.53.0/src/parser/mod.rs:965: prefix: Identifier(Ident { value: "Location", quote_style: None, span: Span(Location(1,20)..Location(1,28)) })
[2025-02-26 07:19:31.694115 +00:00] DEBUG [sqlparser::dialect] /home/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/sqlparser-0.53.0/src/dialect/mod.rs:433: get_next_precedence_full() TokenWithSpan { token: Neq, span: Span(Location(1,29)..Location(1,31)) }
[2025-02-26 07:19:31.694119 +00:00] DEBUG [sqlparser::parser] /home/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/sqlparser-0.53.0/src/parser/mod.rs:968: next precedence: 20
[2025-02-26 07:19:31.694122 +00:00] DEBUG [sqlparser::parser] /home/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/sqlparser-0.53.0/src/parser/mod.rs:963: parsing expr
[2025-02-26 07:19:31.694129 +00:00] DEBUG [sqlparser::parser] /home/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/sqlparser-0.53.0/src/parser/mod.rs:965: prefix: Value(SingleQuotedString("undefined"))
[2025-02-26 07:19:31.694141 +00:00] DEBUG [sqlparser::dialect] /home/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/sqlparser-0.53.0/src/dialect/mod.rs:433: get_next_precedence_full() TokenWithSpan { token: EOF, span: Span(Location(0,0)..Location(0,0)) }
[2025-02-26 07:19:31.694145 +00:00] DEBUG [sqlparser::parser] /home/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/sqlparser-0.53.0/src/parser/mod.rs:968: next precedence: 0
[2025-02-26 07:19:31.694148 +00:00] DEBUG [sqlparser::dialect] /home/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/sqlparser-0.53.0/src/dialect/mod.rs:433: get_next_precedence_full() TokenWithSpan { token: EOF, span: Span(Location(0,0)..Location(0,0)) }
[2025-02-26 07:19:31.694152 +00:00] DEBUG [sqlparser::parser] /home/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/sqlparser-0.53.0/src/parser/mod.rs:968: next precedence: 0
[2025-02-26 07:19:31.694155 +00:00] DEBUG [sqlparser::dialect] /home/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/sqlparser-0.53.0/src/dialect/mod.rs:433: get_next_precedence_full() TokenWithSpan { token: EOF, span: Span(Location(0,0)..Location(0,0)) }
[2025-02-26 07:19:31.694158 +00:00] DEBUG [sqlparser::parser] /home/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/sqlparser-0.53.0/src/parser/mod.rs:968: next precedence: 0
[2025-02-26 07:19:31.707979 +00:00] DEBUG [sqlparser::parser] /home/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/sqlparser-0.53.0/src/parser/mod.rs:388: Parsing sql 'name != '''...
[2025-02-26 07:19:31.707992 +00:00] DEBUG [sqlparser::parser] /home/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/sqlparser-0.53.0/src/parser/mod.rs:963: parsing expr
[2025-02-26 07:19:31.708000 +00:00] DEBUG [sqlparser::parser] /home/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/sqlparser-0.53.0/src/parser/mod.rs:965: prefix: Identifier(Ident { value: "name", quote_style: None, span: Span(Location(1,1)..Location(1,5)) })
[2025-02-26 07:19:31.708005 +00:00] DEBUG [sqlparser::dialect] /home/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/sqlparser-0.53.0/src/dialect/mod.rs:433: get_next_precedence_full() TokenWithSpan { token: Neq, span: Span(Location(1,6)..Location(1,8)) }
[2025-02-26 07:19:31.708009 +00:00] DEBUG [sqlparser::parser] /home/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/sqlparser-0.53.0/src/parser/mod.rs:968: next precedence: 20
[2025-02-26 07:19:31.708013 +00:00] DEBUG [sqlparser::parser] /home/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/sqlparser-0.53.0/src/parser/mod.rs:963: parsing expr
[2025-02-26 07:19:31.708019 +00:00] DEBUG [sqlparser::parser] /home/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/sqlparser-0.53.0/src/parser/mod.rs:965: prefix: Value(SingleQuotedString(""))
[2025-02-26 07:19:31.708022 +00:00] DEBUG [sqlparser::dialect] /home/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/sqlparser-0.53.0/src/dialect/mod.rs:433: get_next_precedence_full() TokenWithSpan { token: EOF, span: Span(Location(0,0)..Location(0,0)) }
[2025-02-26 07:19:31.708026 +00:00] DEBUG [sqlparser::parser] /home/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/sqlparser-0.53.0/src/parser/mod.rs:968: next precedence: 0
[2025-02-26 07:19:31.708029 +00:00] DEBUG [sqlparser::dialect] /home/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/sqlparser-0.53.0/src/dialect/mod.rs:433: get_next_precedence_full() TokenWithSpan { token: EOF, span: Span(Location(0,0)..Location(0,0)) }
[2025-02-26 07:19:31.708033 +00:00] DEBUG [sqlparser::parser] /home/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/sqlparser-0.53.0/src/parser/mod.rs:968: next precedence: 0
[2025-02-26 07:19:31.708348 +00:00] DEBUG [sqlparser::parser] /home/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/sqlparser-0.53.0/src/parser/mod.rs:388: Parsing sql 'starts_with(Location_left,name_right)'...
[2025-02-26 07:19:31.708401 +00:00] DEBUG [sqlparser::parser] /home/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/sqlparser-0.53.0/src/parser/mod.rs:963: parsing expr
[2025-02-26 07:19:31.708415 +00:00] DEBUG [sqlparser::parser] /home/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/sqlparser-0.53.0/src/parser/mod.rs:963: parsing expr
[2025-02-26 07:19:31.708421 +00:00] DEBUG [sqlparser::parser] /home/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/sqlparser-0.53.0/src/parser/mod.rs:965: prefix: Identifier(Ident { value: "Location_left", quote_style: None, span: Span(Location(1,13)..Location(1,26)) })
[2025-02-26 07:19:31.708426 +00:00] DEBUG [sqlparser::dialect] /home/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/sqlparser-0.53.0/src/dialect/mod.rs:433: get_next_precedence_full() TokenWithSpan { token: Comma, span: Span(Location(1,26)..Location(1,27)) }
[2025-02-26 07:19:31.708430 +00:00] DEBUG [sqlparser::parser] /home/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/sqlparser-0.53.0/src/parser/mod.rs:968: next precedence: 0
[2025-02-26 07:19:31.708436 +00:00] DEBUG [sqlparser::parser] /home/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/sqlparser-0.53.0/src/parser/mod.rs:963: parsing expr
[2025-02-26 07:19:31.708441 +00:00] DEBUG [sqlparser::parser] /home/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/sqlparser-0.53.0/src/parser/mod.rs:965: prefix: Identifier(Ident { value: "name_right", quote_style: None, span: Span(Location(1,27)..Location(1,37)) })
[2025-02-26 07:19:31.708446 +00:00] DEBUG [sqlparser::dialect] /home/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/sqlparser-0.53.0/src/dialect/mod.rs:433: get_next_precedence_full() TokenWithSpan { token: RParen, span: Span(Location(1,37)..Location(1,38)) }
[2025-02-26 07:19:31.708449 +00:00] DEBUG [sqlparser::parser] /home/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/sqlparser-0.53.0/src/parser/mod.rs:968: next precedence: 0
[2025-02-26 07:19:31.708455 +00:00] DEBUG [sqlparser::parser] /home/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/sqlparser-0.53.0/src/parser/mod.rs:965: prefix: Function(Function { name: ObjectName([Ident { value: "starts_with", quote_style: None, span: Span(Location(1,1)..Location(1,12)) }]), uses_odbc_syntax: false, parameters: None, args: List(FunctionArgumentList { duplicate_treatment: None, args: [Unnamed(Expr(Identifier(Ident { value: "Location_left", quote_style: None, span: Span(Location(1,13)..Location(1,26)) }))), Unnamed(Expr(Identifier(Ident { value: "name_right", quote_style: None, span: Span(Location(1,27)..Location(1,37)) })))], clauses: [] }), filter: None, null_treatment: None, over: None, within_group: [] })
[2025-02-26 07:19:31.708693 +00:00] DEBUG [sqlparser::dialect] /home/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/sqlparser-0.53.0/src/dialect/mod.rs:433: get_next_precedence_full() TokenWithSpan { token: EOF, span: Span(Location(0,0)..Location(0,0)) }
[2025-02-26 07:19:31.708698 +00:00] DEBUG [sqlparser::parser] /home/runner/.cargo/registry/src/index.crates.io-6f17d22bba15001f/sqlparser-0.53.0/src/parser/mod.rs:968: next precedence: 0

0 replies

jqnatividad · 2025-02-26T11:41:41Z

jqnatividad
Feb 26, 2025
Maintainer

You're on qsv v2.2.1. Can you try on v3.1.1?

4 replies

terefang Mar 2, 2025
Author

$ qsv --version
qsv 3.1.1-mimalloc-apply;fetch;foreach;geocode;polars-0.46.0:py-1.23.0;self_update-12-12;50.19 GiB-570.25 MiB-50.25 GiB-62.74 GiB (x86_64-unknown-linux-musl compiled with Rust 1.85) prebuilt-older
$ rm qsv_rCURRENT.log 
$ export RUST_BACKTRACE=1
$ export QSV_LOG_LEVEL=debug
$ qsv joinp --filter-left "Location != '' and Location != 'undefined'" --filter-right "name != ''" --non-equi "starts_with(Location_left,name_right)" colt_svc.csv cmn_location.utf.csv 
Killed

terefang Mar 2, 2025
Author

$ cat qsv_rCURRENT.log 
[2025-03-02 10:00:33.826955 +00:00] INFO [qsv::util] src/util.rs:947: START: joinp --filter-left Location != '' and Location != 'undefined' --filter-right name != '' --non-equi starts_with(Location_left,name_right) colt_svc.csv cmn_location.utf.csv
[2025-03-02 10:00:34.047740 +00:00] INFO [qsv::util] src/util.rs:1539: No valid .env file found. Proceeding with default settings and current environment variable settings.
[2025-03-02 10:00:34.090138 +00:00] DEBUG [sqlparser::parser] /home/runner/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/sqlparser-0.53.0/src/parser/mod.rs:388: Parsing sql 'Location != '' and Location != 'undefined''...
[2025-03-02 10:00:34.114141 +00:00] DEBUG [sqlparser::parser] /home/runner/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/sqlparser-0.53.0/src/parser/mod.rs:963: parsing expr
[2025-03-02 10:00:34.114705 +00:00] DEBUG [sqlparser::parser] /home/runner/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/sqlparser-0.53.0/src/parser/mod.rs:965: prefix: Identifier(Ident { value: "Location", quote_style: None, span: Span(Location(1,1)..Location(1,9)) })
[2025-03-02 10:00:34.130421 +00:00] DEBUG [sqlparser::dialect] /home/runner/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/sqlparser-0.53.0/src/dialect/mod.rs:433: get_next_precedence_full() TokenWithSpan { token: Neq, span: Span(Location(1,10)..Location(1,12)) }
[2025-03-02 10:00:34.130905 +00:00] DEBUG [sqlparser::parser] /home/runner/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/sqlparser-0.53.0/src/parser/mod.rs:968: next precedence: 20
[2025-03-02 10:00:34.130921 +00:00] DEBUG [sqlparser::parser] /home/runner/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/sqlparser-0.53.0/src/parser/mod.rs:963: parsing expr
[2025-03-02 10:00:34.130943 +00:00] DEBUG [sqlparser::parser] /home/runner/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/sqlparser-0.53.0/src/parser/mod.rs:965: prefix: Value(SingleQuotedString(""))
[2025-03-02 10:00:34.130949 +00:00] DEBUG [sqlparser::dialect] /home/runner/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/sqlparser-0.53.0/src/dialect/mod.rs:433: get_next_precedence_full() TokenWithSpan { token: Word(Word { value: "and", quote_style: None, keyword: AND }), span: Span(Location(1,16)..Location(1,19)) }
[2025-03-02 10:00:34.130957 +00:00] DEBUG [sqlparser::parser] /home/runner/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/sqlparser-0.53.0/src/parser/mod.rs:968: next precedence: 10
[2025-03-02 10:00:34.130962 +00:00] DEBUG [sqlparser::dialect] /home/runner/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/sqlparser-0.53.0/src/dialect/mod.rs:433: get_next_precedence_full() TokenWithSpan { token: Word(Word { value: "and", quote_style: None, keyword: AND }), span: Span(Location(1,16)..Location(1,19)) }
[2025-03-02 10:00:34.130967 +00:00] DEBUG [sqlparser::parser] /home/runner/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/sqlparser-0.53.0/src/parser/mod.rs:968: next precedence: 10
[2025-03-02 10:00:34.130971 +00:00] DEBUG [sqlparser::parser] /home/runner/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/sqlparser-0.53.0/src/parser/mod.rs:963: parsing expr
[2025-03-02 10:00:34.130981 +00:00] DEBUG [sqlparser::parser] /home/runner/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/sqlparser-0.53.0/src/parser/mod.rs:965: prefix: Identifier(Ident { value: "Location", quote_style: None, span: Span(Location(1,20)..Location(1,28)) })
[2025-03-02 10:00:34.130986 +00:00] DEBUG [sqlparser::dialect] /home/runner/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/sqlparser-0.53.0/src/dialect/mod.rs:433: get_next_precedence_full() TokenWithSpan { token: Neq, span: Span(Location(1,29)..Location(1,31)) }
[2025-03-02 10:00:34.130990 +00:00] DEBUG [sqlparser::parser] /home/runner/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/sqlparser-0.53.0/src/parser/mod.rs:968: next precedence: 20
[2025-03-02 10:00:34.130994 +00:00] DEBUG [sqlparser::parser] /home/runner/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/sqlparser-0.53.0/src/parser/mod.rs:963: parsing expr
[2025-03-02 10:00:34.131001 +00:00] DEBUG [sqlparser::parser] /home/runner/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/sqlparser-0.53.0/src/parser/mod.rs:965: prefix: Value(SingleQuotedString("undefined"))
[2025-03-02 10:00:34.131013 +00:00] DEBUG [sqlparser::dialect] /home/runner/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/sqlparser-0.53.0/src/dialect/mod.rs:433: get_next_precedence_full() TokenWithSpan { token: EOF, span: Span(Location(0,0)..Location(0,0)) }
[2025-03-02 10:00:34.131017 +00:00] DEBUG [sqlparser::parser] /home/runner/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/sqlparser-0.53.0/src/parser/mod.rs:968: next precedence: 0
[2025-03-02 10:00:34.131020 +00:00] DEBUG [sqlparser::dialect] /home/runner/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/sqlparser-0.53.0/src/dialect/mod.rs:433: get_next_precedence_full() TokenWithSpan { token: EOF, span: Span(Location(0,0)..Location(0,0)) }
[2025-03-02 10:00:34.131024 +00:00] DEBUG [sqlparser::parser] /home/runner/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/sqlparser-0.53.0/src/parser/mod.rs:968: next precedence: 0
[2025-03-02 10:00:34.131027 +00:00] DEBUG [sqlparser::dialect] /home/runner/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/sqlparser-0.53.0/src/dialect/mod.rs:433: get_next_precedence_full() TokenWithSpan { token: EOF, span: Span(Location(0,0)..Location(0,0)) }
[2025-03-02 10:00:34.131031 +00:00] DEBUG [sqlparser::parser] /home/runner/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/sqlparser-0.53.0/src/parser/mod.rs:968: next precedence: 0
[2025-03-02 10:00:34.159229 +00:00] DEBUG [sqlparser::parser] /home/runner/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/sqlparser-0.53.0/src/parser/mod.rs:388: Parsing sql 'name != '''...
[2025-03-02 10:00:34.159259 +00:00] DEBUG [sqlparser::parser] /home/runner/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/sqlparser-0.53.0/src/parser/mod.rs:963: parsing expr
[2025-03-02 10:00:34.159271 +00:00] DEBUG [sqlparser::parser] /home/runner/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/sqlparser-0.53.0/src/parser/mod.rs:965: prefix: Identifier(Ident { value: "name", quote_style: None, span: Span(Location(1,1)..Location(1,5)) })
[2025-03-02 10:00:34.159280 +00:00] DEBUG [sqlparser::dialect] /home/runner/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/sqlparser-0.53.0/src/dialect/mod.rs:433: get_next_precedence_full() TokenWithSpan { token: Neq, span: Span(Location(1,6)..Location(1,8)) }
[2025-03-02 10:00:34.159286 +00:00] DEBUG [sqlparser::parser] /home/runner/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/sqlparser-0.53.0/src/parser/mod.rs:968: next precedence: 20
[2025-03-02 10:00:34.159291 +00:00] DEBUG [sqlparser::parser] /home/runner/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/sqlparser-0.53.0/src/parser/mod.rs:963: parsing expr
[2025-03-02 10:00:34.159300 +00:00] DEBUG [sqlparser::parser] /home/runner/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/sqlparser-0.53.0/src/parser/mod.rs:965: prefix: Value(SingleQuotedString(""))
[2025-03-02 10:00:34.159304 +00:00] DEBUG [sqlparser::dialect] /home/runner/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/sqlparser-0.53.0/src/dialect/mod.rs:433: get_next_precedence_full() TokenWithSpan { token: EOF, span: Span(Location(0,0)..Location(0,0)) }
[2025-03-02 10:00:34.159308 +00:00] DEBUG [sqlparser::parser] /home/runner/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/sqlparser-0.53.0/src/parser/mod.rs:968: next precedence: 0
[2025-03-02 10:00:34.159312 +00:00] DEBUG [sqlparser::dialect] /home/runner/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/sqlparser-0.53.0/src/dialect/mod.rs:433: get_next_precedence_full() TokenWithSpan { token: EOF, span: Span(Location(0,0)..Location(0,0)) }
[2025-03-02 10:00:34.159315 +00:00] DEBUG [sqlparser::parser] /home/runner/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/sqlparser-0.53.0/src/parser/mod.rs:968: next precedence: 0
[2025-03-02 10:00:34.159740 +00:00] DEBUG [sqlparser::parser] /home/runner/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/sqlparser-0.53.0/src/parser/mod.rs:388: Parsing sql 'starts_with(Location_left,name_right)'...
[2025-03-02 10:00:34.159792 +00:00] DEBUG [sqlparser::parser] /home/runner/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/sqlparser-0.53.0/src/parser/mod.rs:963: parsing expr
[2025-03-02 10:00:34.159808 +00:00] DEBUG [sqlparser::parser] /home/runner/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/sqlparser-0.53.0/src/parser/mod.rs:963: parsing expr
[2025-03-02 10:00:34.159815 +00:00] DEBUG [sqlparser::parser] /home/runner/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/sqlparser-0.53.0/src/parser/mod.rs:965: prefix: Identifier(Ident { value: "Location_left", quote_style: None, span: Span(Location(1,13)..Location(1,26)) })
[2025-03-02 10:00:34.159821 +00:00] DEBUG [sqlparser::dialect] /home/runner/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/sqlparser-0.53.0/src/dialect/mod.rs:433: get_next_precedence_full() TokenWithSpan { token: Comma, span: Span(Location(1,26)..Location(1,27)) }
[2025-03-02 10:00:34.159825 +00:00] DEBUG [sqlparser::parser] /home/runner/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/sqlparser-0.53.0/src/parser/mod.rs:968: next precedence: 0
[2025-03-02 10:00:34.159832 +00:00] DEBUG [sqlparser::parser] /home/runner/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/sqlparser-0.53.0/src/parser/mod.rs:963: parsing expr
[2025-03-02 10:00:34.159837 +00:00] DEBUG [sqlparser::parser] /home/runner/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/sqlparser-0.53.0/src/parser/mod.rs:965: prefix: Identifier(Ident { value: "name_right", quote_style: None, span: Span(Location(1,27)..Location(1,37)) })
[2025-03-02 10:00:34.159842 +00:00] DEBUG [sqlparser::dialect] /home/runner/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/sqlparser-0.53.0/src/dialect/mod.rs:433: get_next_precedence_full() TokenWithSpan { token: RParen, span: Span(Location(1,37)..Location(1,38)) }
[2025-03-02 10:00:34.159846 +00:00] DEBUG [sqlparser::parser] /home/runner/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/sqlparser-0.53.0/src/parser/mod.rs:968: next precedence: 0
[2025-03-02 10:00:34.159851 +00:00] DEBUG [sqlparser::parser] /home/runner/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/sqlparser-0.53.0/src/parser/mod.rs:965: prefix: Function(Function { name: ObjectName([Ident { value: "starts_with", quote_style: None, span: Span(Location(1,1)..Location(1,12)) }]), uses_odbc_syntax: false, parameters: None, args: List(FunctionArgumentList { duplicate_treatment: None, args: [Unnamed(Expr(Identifier(Ident { value: "Location_left", quote_style: None, span: Span(Location(1,13)..Location(1,26)) }))), Unnamed(Expr(Identifier(Ident { value: "name_right", quote_style: None, span: Span(Location(1,27)..Location(1,37)) })))], clauses: [] }), filter: None, null_treatment: None, over: None, within_group: [] })
[2025-03-02 10:00:34.162999 +00:00] DEBUG [sqlparser::dialect] /home/runner/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/sqlparser-0.53.0/src/dialect/mod.rs:433: get_next_precedence_full() TokenWithSpan { token: EOF, span: Span(Location(0,0)..Location(0,0)) }
[2025-03-02 10:00:34.163006 +00:00] DEBUG [sqlparser::parser] /home/runner/.cargo/registry/src/index.crates.io-1949cf8c6b5b557f/sqlparser-0.53.0/src/parser/mod.rs:968: next precedence: 0

jqnatividad Mar 2, 2025
Maintainer

Hi @terefang ,
Can you give a minimum reproducible example, including some sample files?

Does it work for a small test subset of your existing data? Or does it fail even for a small subset?

terefang Mar 20, 2025
Author

i will try to find a dataset that reproduces the error.

Join with more flexible matching criteria #2358

Uh oh!

terefang Dec 10, 2024

Replies: 16 comments · 11 replies

Uh oh!

jqnatividad Dec 10, 2024 Maintainer

Uh oh!

terefang Dec 10, 2024 Author

Uh oh!

terefang Dec 10, 2024 Author

Uh oh!

jqnatividad Dec 11, 2024 Maintainer

Uh oh!

jqnatividad Dec 18, 2024 Maintainer

Uh oh!

ondohotola Dec 18, 2024

Uh oh!

jqnatividad Dec 18, 2024 Maintainer

Uh oh!

ondohotola Dec 18, 2024

Uh oh!

jqnatividad Dec 18, 2024 Maintainer

Uh oh!

terefang Dec 18, 2024 Author

Uh oh!

jqnatividad Dec 19, 2024 Maintainer

Uh oh!

terefang Jan 7, 2025 Author

Uh oh!

Uh oh!

jqnatividad Jan 7, 2025 Maintainer

Uh oh!

terefang Jan 7, 2025 Author

Uh oh!

Uh oh!

jqnatividad Jan 7, 2025 Maintainer

Uh oh!

Uh oh!

terefang Jan 7, 2025 Author

Uh oh!

Uh oh!

jqnatividad Jan 30, 2025 Maintainer

Uh oh!

terefang Feb 2, 2025 Author

Uh oh!

Uh oh!

terefang Feb 26, 2025 Author

Uh oh!

terefang Feb 26, 2025 Author

Uh oh!

jqnatividad Feb 26, 2025 Maintainer

Uh oh!

terefang Mar 2, 2025 Author

Uh oh!

terefang Mar 2, 2025 Author

Uh oh!

jqnatividad Mar 2, 2025 Maintainer

Uh oh!

terefang Mar 20, 2025 Author

terefang
Dec 10, 2024

Replies: 16 comments 11 replies

jqnatividad
Dec 10, 2024
Maintainer

terefang
Dec 10, 2024
Author

terefang
Dec 10, 2024
Author

jqnatividad
Dec 11, 2024
Maintainer

jqnatividad
Dec 18, 2024
Maintainer

ondohotola
Dec 18, 2024

jqnatividad
Dec 18, 2024
Maintainer

ondohotola
Dec 18, 2024

jqnatividad
Dec 18, 2024
Maintainer

terefang
Dec 18, 2024
Author

jqnatividad
Dec 19, 2024
Maintainer

terefang Jan 7, 2025
Author

jqnatividad Jan 7, 2025
Maintainer

terefang Jan 7, 2025
Author

jqnatividad Jan 7, 2025
Maintainer

terefang Jan 7, 2025
Author

jqnatividad
Jan 30, 2025
Maintainer

terefang
Feb 2, 2025
Author

terefang
Feb 26, 2025
Author

terefang
Feb 26, 2025
Author

jqnatividad
Feb 26, 2025
Maintainer

terefang Mar 2, 2025
Author

terefang Mar 2, 2025
Author

jqnatividad Mar 2, 2025
Maintainer

terefang Mar 20, 2025
Author