當我嘗試加載一個多分區的鑲木地板文件時,由于缺少用空值填充架構的數據,某些架構被無效推斷。我認為在 pyarrow.parquet.ParquetDataset 中指定架構可以解決這個問題,但我不知道如何構建正確的 pyarrow.parquet.Schema 類型的架構。一些示例代碼:import pyarrow as paimport pa.parquet as pq test_schema = pa.schema([pa.field('field1', pa.string()), pa.field('field2', pa.float64())])paths = ['test_root/partition1/file1.parquet', 'test_root/partition2/file2.parquet']dataset = pq.ParquetDataset(paths, schema=schema)和錯誤:AttributeError: 'pyarrow.lib.Schema' object has no attribute 'to_arrow_schema'但我找不到任何關于如何構建 pyarrow.parquet.Schema 架構的文檔(如文檔(https://arrow.apache.org/docs/python/generated/pyarrow.parquet.ParquetDataset.html),并且有只做了一個 pyarrow.lib.Schema 給出了上述錯誤。
添加回答
舉報
0/150
提交
取消