How to handle multi-line values in CSVs

[alexkreidler]

---

Describe the bug
qsv select fails when it comes across a CSV multiline string (e.g. quote, then multiple newlines and endquote)
qsv stats fails with SIGSEGV (Address boundary error)

To Reproduce
Steps to reproduce the behavior:

Create a file simple.csv with this data:

id,date_created,date_modified,judges,date_filed,date_filed_is_approximate,slug,case_name_short,case_name,case_name_full,scdb_id,scdb_decision_direction,scdb_votes_majority,scdb_votes_minority,source,procedural_history,attorneys,nature_of_suit,posture,syllabus,headnotes,summary,disposition,history,other_dates,cross_reference,correction,citation_count,precedential_status,date_blocked,blocked,filepath_json_harvard,filepath_pdf_harvard,docket_id,arguments,headmatter
"1553297","2014-10-30 18:18:48.065126+00","2024-11-02 17:57:19.548229+00","Cecelia H. Goetz","1985-10-29","f","ambico-inc-v-aic-photo-inc-in-re-aic-photo-inc","In re AIC","Ambico, Inc. v. AIC Photo, Inc. (In Re AIC Photo, Inc.)","In the Matter of AIC PHOTO, INC., Et Al., Debtor. AMBICO, INC., Plaintiff, v. AIC PHOTO, INC., Defendant","",,,,"LU","","Patterson, Belknap, Webb & Tyler, New York City by Scott Horton, for debtors., Windels, Marx, Davies & Ives, New York City by Christopher T. Ragucci, for Ambi-co., Zalkin, Rodin & Goodman, New York City by Andrew D. Gottfried, for Manufacturers Hanover Trust.","","","","","","","","","","","5","Published",,"f","law.free.cap.br.57/56.6097565.json","harvard_pdf/1553297.pdf","1638530","","<parties id=\"b112-20\">
    In the Matter of AIC PHOTO, INC., et al., Debtor. AMBICO, INC., Plaintiff, v. AIC PHOTO, INC., Defendant.
   </parties><docketnumber id=\"Ag36\">
    Bankruptcy No. 185-50388-21.
   </docketnumber><docketnumber id=\"Aif\">
    Adv. No. 185-0055.
   </docketnumber><court id=\"ABf\">
    United States Bankruptcy Court, E.D. New York.
   </court><decisiondate id=\"AE5\">
    Oct. 29, 1985.
   </decisiondate><br><attorneys id=\"b113-8\">
<span citation-index=\"1\" class=\"star-pagination\" label=\"57\">
     *57
     </span>
    Patterson, Belknap, Webb &amp; Tyler, New York City by Scott Horton, for debtors.
   </attorneys><br><attorneys id=\"b113-9\">
    Windels, Marx, Davies &amp; Ives, New York City by Christopher T. Ragucci, for Ambi-co.
   </attorneys><br><attorneys id=\"b113-10\">
    Zalkin, Rodin &amp; Goodman, New York City by Andrew D. Gottfried, for Manufacturers Hanover Trust.
   </attorneys>"
"10154826","2024-10-22 16:55:01.985413+00","2024-10-22 16:55:02.172053+00","","2008-01-10","f","state-v-woods","Woods","State v. Woods","","",,,,"C","","","","","","","","","","","","","0","Unpublished","2024-10-22","t","","","69293379","",""
"10154827","2024-10-22 16:55:02.320424+00","2024-10-22 16:55:02.457429+00","","2008-01-10","f","state-v-weathersbee","Weathersbee","State v. Weathersbee","","",,,,"C","","","","","","","","","","","","","0","Unpublished","2024-10-22","t","","","69293380","",""
"10154828","2024-10-22 16:55:02.684449+00","2024-10-22 16:55:02.816383+00","","2008-01-10","f","state-v-walker","Walker","State v. Walker","","",,,,"C","","","","","","","","","","","","","0","Unpublished","2024-10-22","t","","","69293381","",""

Run and get this error:

qsv select -o simple_opinions.csv id,date_created,date_modified,judges,date_filed,date_filed_is_approximate,slug,case_name_short,case_name,case_name_full,scdb_id,scdb_decision_direction,scdb_votes_majority,scdb_votes_minority,source,nature_of_suit,other_dates,citation_count,precedential_status,date_blocked,blocked,filepath_json_harvard,filepath_pdf_harvard,docket_id simple.csv
csv error: CSV error: record 2 (line: 3, byte: 1232): found record with 9 fields, but the previous record has 36 fields

When I run

qsv select 1-4 simple.csv
id,date_created,date_modified,judges
1553297,2014-10-30 18:18:48.065126+00,2024-11-02 17:57:19.548229+00,Cecelia H. Goetz
csv error: CSV error: record 2 (line: 3, byte: 1232): found record with 9 fields, but the previous record has 36 fields

qsv table simple.csv works fine (but it isn't very pretty cause there's too many columns/data that it doesn't truncate)

qsv stats simple.csv
fish: Job 1, 'qsv stats simple.csv' terminated by signal SIGSEGV (Address boundary error)

No idea what that error is

Expected behavior
It should read the file properly and output the selected columns to the file

Screenshots/Backtrace/Sample Data
If applicable, add screenshots/backtraces/sample data to help explain your problem.

Desktop (please complete the following information):

• OS: Ubuntu 24.04.2
• qsv 4.0.0-mimalloc-apply;Luau 0.663;-4-4;18.73 GiB-0 B-22.75 GiB-23.42 GiB (aarch64-unknown-linux-gnu compiled with Rust 1.86) compiled

[rickhg12hs]

I think there are two issues here.

1. Typical CSVs escape quotes by doubling them, i.e. "" rather than \". Starting your pipeline with qsv input --escape '\' simple.csv fixes all of your problematic examples.
2. Your SIGSEGV and my "program never terminates" points to some state within qsv that needs attention.

[jqnatividad]

Thanks for chiming in @rickhg12hs with the much more elegant qsv input --escape '\' simple.csv solution.

And yes, I'll have to look into the SIGSEGV with stats.

stats is really tuned for performance, so it presumes a valid CSV input file. Still, it shouldn't segfault.

[alexkreidler]

Thanks, this does fix the problem! I can now select columns and run counts on my 12GB file: qsv input --escape "\\" opinion-clusters-2025-03-31.csv | qsv count: 9895730

[jqnatividad]

Thanks for the detailed report @alexkreidler .

To handle embedded XML values like this - you need to escape them by using double quotes around the entire field and escape any double quotes within the field by doubling them:

id,date_created,date_modified,judges,date_filed,date_filed_is_approximate,slug,case_name_short,case_name,case_name_full,scdb_id,scdb_decision_direction,scdb_votes_majority,scdb_votes_minority,source,procedural_history,attorneys,nature_of_suit,posture,syllabus,headnotes,summary,disposition,history,other_dates,cross_reference,correction,citation_count,precedential_status,date_blocked,blocked,filepath_json_harvard,filepath_pdf_harvard,docket_id,arguments,headmatter
"1553297","2014-10-30 18:18:48.065126+00","2024-11-02 17:57:19.548229+00","Cecelia H. Goetz","1985-10-29","f","ambico-inc-v-aic-photo-inc-in-re-aic-photo-inc","In re AIC","Ambico, Inc. v. AIC Photo, Inc. (In Re AIC Photo, Inc.)","In the Matter of AIC PHOTO, INC., Et Al., Debtor. AMBICO, INC., Plaintiff, v. AIC PHOTO, INC., Defendant","",,,,"LU","","Patterson, Belknap, Webb & Tyler, New York City by Scott Horton, for debtors., Windels, Marx, Davies & Ives, New York City by Christopher T. Ragucci, for Ambi-co., Zalkin, Rodin & Goodman, New York City by Andrew D. Gottfried, for Manufacturers Hanover Trust.","","","","","","","","","","","5","Published",,"f","law.free.cap.br.57/56.6097565.json","harvard_pdf/1553297.pdf","1638530","","""<parties id=""b112-20"">
    In the Matter of AIC PHOTO, INC., et al., Debtor. AMBICO, INC., Plaintiff, v. AIC PHOTO, INC., Defendant.
   </parties><docketnumber id=""Ag36"">
    Bankruptcy No. 185-50388-21.
   </docketnumber><docketnumber id=""Aif"">
    Adv. No. 185-0055.
   </docketnumber><court id=""ABf"">
    United States Bankruptcy Court, E.D. New York.
   </court><decisiondate id=""AE5"">
    Oct. 29, 1985.
   </decisiondate><br><attorneys id=""b113-8"">
<span citation-index=""1"" class=""star-pagination"" label=""57"">
     *57
     </span>
    Patterson, Belknap, Webb & Tyler, New York City by Scott Horton, for debtors.
   </attorneys><br><attorneys id=""b113-9"">
    Windels, Marx, Davies & Ives, New York City by Christopher T. Ragucci, for Ambi-co.
   </attorneys><br><attorneys id=""b113-10"">
    Zalkin, Rodin & Goodman, New York City by Andrew D. Gottfried, for Manufacturers Hanover Trust.
   </attorneys>"""
"10154826","2024-10-22 16:55:01.985413+00","2024-10-22 16:55:02.172053+00","","2008-01-10","f","state-v-woods","Woods","State v. Woods","","",,,,"C","","","","","","","","","","","","","0","Unpublished","2024-10-22","t","","","69293379","",""
"10154827","2024-10-22 16:55:02.320424+00","2024-10-22 16:55:02.457429+00","","2008-01-10","f","state-v-weathersbee","Weathersbee","State v. Weathersbee","","",,,,"C","","","","","","","","","","","","","0","Unpublished","2024-10-22","t","","","69293380","",""
"10154828","2024-10-22 16:55:02.684449+00","2024-10-22 16:55:02.816383+00","","2008-01-10","f","state-v-walker","Walker","State v. Walker","","",,,,"C","","","","","","","","","","","","","0","Unpublished","2024-10-22","t","","","69293381","",""

multiline.csv

Doing so allows the other commands in your example to work.

$ qsv select -o simple_opinions.csv id,date_created,date_modified,judges,date_filed,date_filed_is_approximate,slug,case_name_short,case_name,case_name_full,scdb_id,scdb_decision_direction,scdb_votes_majority,scdb_votes_minority,source,nature_of_suit,other_dates,citation_count,precedential_status,date_blocked,blocked,filepath_json_harvard,filepath_pdf_harvard,docket_id ~/multiline.csv

$ qsv sniff simple_opinions.csv
Path: /Users/jdoe/simple_opinions.csv
Sniff Timestamp: 2025-05-05T15:41:21.989222+00:00
Last Modified: 2025-05-05T15:39:19+00:00
Delimiter: ,
Header Row: true
Preamble Rows: 0
Quote Char: "
Flexible: false
Is UTF8: true
Detected Mime Type: application/csv
Detected Kind: Other
Retrieved Size (bytes): 1,262
File Size (bytes): 1,262
Sampled Records: 4
Estimated: false
Num Records: 4
Avg Record Len (bytes): 206
Num Fields: 24
Stats Types: false
Fields:
    0:   Unsigned  id
    1:   Text      date_created
    2:   Text      date_modified
    3:   NULL      judges
    4:   Date      date_filed
    5:   Boolean   date_filed_is_approximate
    6:   Text      slug
    7:   Text      case_name_short
    8:   Text      case_name
    9:   NULL      case_name_full
    10:  NULL      scdb_id
    11:  NULL      scdb_decision_direction
    12:  NULL      scdb_votes_majority
    13:  NULL      scdb_votes_minority
    14:  Text      source
    15:  NULL      nature_of_suit
    16:  NULL      other_dates
    17:  Boolean   citation_count
    18:  Text      precedential_status
    19:  Date      date_blocked
    20:  Boolean   blocked
    21:  NULL      filepath_json_harvard
    22:  NULL      filepath_pdf_harvard
    23:  Unsigned  docket_id

$ qsv select 1-4 ~/multiline.csv 
id,date_created,date_modified,judges
1553297,2014-10-30 18:18:48.065126+00,2024-11-02 17:57:19.548229+00,Cecelia H. Goetz
10154826,2024-10-22 16:55:01.985413+00,2024-10-22 16:55:02.172053+00,
10154827,2024-10-22 16:55:02.320424+00,2024-10-22 16:55:02.457429+00,
10154828,2024-10-22 16:55:02.684449+00,2024-10-22 16:55:02.816383+00,

BTW, I'll convert this issue to a Discussion so other folks with a similar problem can find it.