Description
The ParquetIntegrationUnit generates a Parquet file from the Target tree. Nested structures such as objects, lists, and key-value maps are preserved and are not collapsed. For large data sets, the output is written to a temporary file rather than being held entirely in main memory.
Prerequisites
In Phase 5, the option Use IntegrationUnit & postexecution is enabled.
NOTE:
The DuckDB library is required to process Parquet. It must be included in the installation; however, you do not need to create a separate database connection for this.
Parameter
The parameter values are entered as free text. A value must be specified for all parameters.
Parameter | Permitted values | Default value | Description |
|---|---|---|---|
| — | — | The node in the target structure from which the Parquet file is generated. |
| snappy, gzip, lzo, brotli, zstd, lz4_raw | snappy | Compression method for the generated Parquet file. |
| v1, v2 | v1 | Parquet version to be used. |
Error handling and logging
If the export cannot be completed, an error is displayed with a descriptive message and logged with sufficient details. Silent errors do not occur.
Procedure
Create the target structure.
Select ParquetIntegrationUnit as the Integration Unit and set the Start at node parameter. Adjust the compression and Parquet version as needed.
In a Response type "File", set the "Content" to IU output.