Section I Understanding Parquet
What is Parquet and Arrow?
Infino extends the Parquet format to store data. So its important to understand Parquet first before we see how Infino stores its data. To start, here's the official definition "Apache Parquet is an open source, column-oriented data file format designed for efficient data storage and retrieval. It provides high performance compression and encoding schemes to handle complex data in bulk and is supported in many programming languages and analytics tools.". This means that it is an "open source" - anyone can use/extend/modify it, "column-oriented" - data is laid out in columns instead of rows ( more on this when we see the format ). Its openness has made it very popular, with a lot of users contributing back to it. This, along with its design makes it efficient and performant while allowing it to support a wide variety of data.
How does Parquet Store Data?
A parquet file starts with a magic header "PAR1" identifying it and ends with the same magic number at the end( similar to how PDF files have %PDF and %%EOF ). This number is of 4 bytes, and the 4 bytes before it contain the footer length ( Yes. You start reading the file in reverse ). The footer ends immediately after the footer length, and we subtract the footer length from the footer end address to get the footer start. The rest of the file, ( after the magic header at the start to the start of the footer ) contains the actual data, divided into row groups. Each row group stores data within itself in columnar format. ( Columns are further divided into pages, but we'll ignore that for now ). The footer contains the metadata to help find, prune & decode these row groups and columns. It also contains the file version, schema and other metadata fields. For reference, here's how a parquet file looks
[ IMG ]