[BUG][Kernel] Delta kernel read checkpoint parquet file failed with exception "list not found in optional group readerFeatures (LIST)" #3082
Labels
bug
Something isn't working
Comments
|
@Youngwb Thanks for reporting. Currently, the default Parquet reader is not handling the legacy list physical format. Will post a PR shortly. |
vkorukanti
added a commit
that referenced
this issue
May 10, 2024
…repeated types (#3083) ## Description When legacy mode is enabled in Spark, array physical types are stored slightly different from the standard format. Standard mode (default): ``` optional group readerFeatures (LIST) { repeated group list { optional binary element (STRING); } } ``` When write legacy mode is enabled (`spark.sql.parquet.writeLegacyFormat = true`): ``` optional group readerFeatures (LIST) { repeated group bag { optional binary array (STRING); } } ``` TODO: We need to handle the 2-level lists. Will post a separate PR. The challenge is with generating or finding the Parquet files with 2-level lists. ## How was this patch tested? Added tests Fixes #3082
vkorukanti
added a commit
that referenced
this issue
May 10, 2024
…repeated types (#3083) ## Description When legacy mode is enabled in Spark, array physical types are stored slightly different from the standard format. Standard mode (default): ``` optional group readerFeatures (LIST) { repeated group list { optional binary element (STRING); } } ``` When write legacy mode is enabled (`spark.sql.parquet.writeLegacyFormat = true`): ``` optional group readerFeatures (LIST) { repeated group bag { optional binary array (STRING); } } ``` TODO: We need to handle the 2-level lists. Will post a separate PR. The challenge is with generating or finding the Parquet files with 2-level lists. ## How was this patch tested? Added tests Fixes #3082
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Bug
Which Delta project/connector is this regarding?
Describe the problem
When use delta kernel to read delta lake table encountered such an error
and the schema of parquet file is
And I think delta kernel currently does not supports reading this legacy array type.
Steps to reproduce
Observed results
Expected results
Further details
Environment information
Willingness to contribute
The Delta Lake Community encourages bug fix contributions. Would you or another member of your organization be willing to contribute a fix for this bug to the Delta Lake code base?
The text was updated successfully, but these errors were encountered: