Is there a parquet file size limit for loading files using Azure

Hi Laurent Delaquis,

It looks like you're encountering a problem when trying to load a parquet file into a database using Azure Data Factory (ADF). The error message indicates that there are more columns in the data than defined, which is usually tied to how your data is structured or how the pipeline is configured. Here’s a step-by-step approach you can try:

Check the Column Count: Since you're receiving an error regarding more columns than expected, double-check the schema and ensure that the number of columns in the source parquet file matches the defined schema in ADF.

File Inspection: You mentioned that opening the CSV version in Excel shows no extra columns. It's worth inspecting the actual parquet file as well—use tools like Azure Data Lake Storage or Azure Synapse Analytics to directly inspect the parquet file's schema.

ADF Pipeline Configuration:

• Ensure that your dataset schema in ADF matches exactly with that of the parquet file.
  • Look for any changes in the incoming files; since you mentioned new sites were added, make sure any new attributes are accounted for in the schema.
  Review Data in Context: The error occurred at row number 24,702. Check if that specific row contains any unexpected data or formatting issues that could cause misalignment. Adjust ADF Limits: If the file size (16,387 KB) is nearing any limits, consider reviewing Azure Data Factory Limits to ensure you're within acceptable thresholds. Break Down the Pipeline: If you continue to see issues, consider breaking your pipeline into smaller chunks. This might help isolate the problem and avoid hitting any memory limits, as sometimes larger files can lead to complications. Try a Different Runtime: If you're running into resource constraints, using a self-hosted integration runtime with more memory could alleviate the issues you're encountering. Ensure you have proper configuration as described in the documentation about Self-hosted IR settings.

Here's a set of follow-up questions that might help clarify the situation:

• Have you modified the schema in the target database recently?
• Are there any specific transformations applied within the ADF pipeline that could affect the column count?
• Is there a specific pattern to the rows that return errors, or do they seem random?

I hope this helps! Feel free to reach out with any more details, and we can dig deeper into the issue together. Good luck!

References

• Azure Data Factory Limits
• Self-hosted Integration Runtime
• Data Factory Copy Activity Overview