daft only supports large_string dtype, but lance can use string and large_string

[caican00]

Describe the bug

If I have a lance dataset whose schema is string, but daft forces it to be converted to large_string, it may cause the append of new data to fail.
The same applies to other types.

To Reproduce

No response

Expected behavior

Using Daft, we can write Lance datasets with types such as string/binary or their large variants, large_string/large_binary.

Component(s)

Lance Integration

Additional context

No response

[caican00]

    @DataframePublicAPI
    def write_lance(
        self,
        uri: Union[str, pathlib.Path],
        mode: Literal["create", "append", "overwrite"] = "create",
        io_config: Optional[IOConfig] = None,
        schema: Optional[Union[Schema, "pyarrow.Schema"]] = None,
        **kwargs: Any,
    ) -> "DataFrame":
        """Writes the DataFrame to a Lance table.

        Args:
          uri: The URI of the Lance table to write to
          mode: The write mode. One of "create", "append", or "overwrite"
          io_config (IOConfig, optional): configurations to use when interacting with remote storage.
          schema (Schema | pyarrow.Schema, optional): Desired schema to enforce during write.
            - If omitted, Daft will use the DataFrame's current schema.
            - If a pyarrow.Schema is provided, Daft will enforce the field order, types, and nullability
              by casting the data to the provided schema prior to write. Table-level (dataset) metadata present
              on the pyarrow schema is preserved during create/overwrite.
            - If the target Lance dataset already exists, the data will be cast to the existing table schema
              to ensure compatibility unless ``mode="overwrite"``.
          **kwargs: Additional keyword arguments to pass to the Lance writer.

        Returns:
            DataFrame: A DataFrame containing metadata about the written Lance table, such as number of fragments, number of deleted rows, number of small files, and version.

It seems reasonable?

schema (Schema | pyarrow.Schema, optional): Desired schema to enforce during write.
  - If omitted, Daft will use the DataFrame's current schema.
  - If a pyarrow.Schema is provided, Daft will enforce the field order, types, and nullability
    by casting the data to the provided schema prior to write. Table-level (dataset) metadata present
    on the pyarrow schema is preserved during create/overwrite.
  - If the target Lance dataset already exists, the data will be cast to the existing table schema
    to ensure compatibility unless ``mode="overwrite"``.

[colin-ho]

Yes please try passing in a pyarrow schema

[caican00]

Yes please try passing in a pyarrow schema

I'm worried that for some wide columns or particularly complex nested types, it might be rather difficult for users to manually spell a pyarrow schema. WDYT? cc @colin-ho

[colin-ho]

Perhaps we can provide a similar schema hinting mechanism that we have for reads, where you can just override the dtype of certain fields.

However it is unlikely that we will support string type natively anytime soon. A lot of the times our users are reading in strings of more than 2GB, which large string is required. We also do not aim to be 1-1 with arrow.