Multiple Hive Clusters

You can have as many catalogs as you need, so if you have additional Hive clusters, simply add another properties file to /etc/presto/catalog with a different name (making sure it ends in .properties). For example, if you name the property file sales.properties, Presto will create a catalog named sales using the configured connector.

HDFS Configuration

For basic setups, Presto configures the HDFS client automatically and does not require any configuration files. In some cases, such as when using federated HDFS or NameNode high availability, it is necessary to specify additional HDFS client options in order to access your HDFS cluster. To do so, add the hive.config.resources property to reference your HDFS config files:

hive.config.resources=/etc/hadoop/conf/core-site.xml,/etc/hadoop/conf/hdfs-site.xml

Only specify additional configuration files if necessary for your setup. We also recommend reducing the configuration files to have the minimum set of required properties, as additional properties may cause problems.

The configuration files must exist on all Presto nodes. If you are referencing existing Hadoop config files, make sure to copy them to any Presto nodes that are not running Hadoop.

HDFS Permissions

Before running any CREATE TABLE or CREATE TABLE ... AS statements for Hive tables in Presto, you need to check that the operating system user running the Presto server has access to the Hive warehouse directory on HDFS. The Hive warehouse directory is specified by the configuration variable hive.metastore.warehouse.dir in hive-site.xml, and the default value is /user/hive/warehouse. If that is not the case, either add the following to jvm.config on all of the nodes: -DHADOOP_USER_NAME=USER, where USER is an operating system user that has proper permissions for the Hive warehouse directory, or start the Presto server as a user with similar permissions. The hive user generally works as USER, since Hive is often started with the hive user. If you run into HDFS permissions problems on CREATE TABLE ... AS, remove /tmp/presto-* on HDFS, fix the user as described above, then restart all of the Presto servers.

Amazon S3 Configuration

The Hive connector also allows querying data stored in Amazon S3.

To access tables stored in S3, you must specify the AWS credential properties hive.s3.aws-access-key and hive.s3.aws-secret-key. Alternatively, you can use hive.s3.use-instance-credentials which if set to true, enables retrieving temporary instance profile AWS credentials.

hive.assume-canonical-partition-keys

• Type: Boolean
• Default value: false
• Description: Disable optimized metastore partition fetching for non-string partition keys. Setting this property allows to avoid ignoring data with non-canonical partition values.

hive.domain-compaction-threshold

• Type: Integer (at least 1)
• Default value: 100
• Description: Maximum number of ranges allowed in a tuple domain without compacting it. Higher value will cause more data fragmentation but allows to use row skipping feature when reading ORC data. Setting this value higher may have large impact on IN and OR clauses performance in scenarios making use of row skipping.

hive.force-local-scheduling

• Type: Boolean
• Default value: false
• Description: Force splits to be scheduled on the same node (ignoring normal node selection procedures) as the Hadoop DataNode process serving the split data. This is useful for installations where Presto is collocated with every DataNode and may increase queries time significantly. The drawback may be that if some data are accessed more often, the utilization of some nodes may be low even if the whole system is heavy loaded. See also node-scheduler.network-topology if less strict constrain is preferred - especially if some nodes are overloaded and other are not fully utilized.

hive.max-initial-split-size

• Type: String (data size)
• Default value: hive.max-split-size / 2 (32 MB)
• Description: This property describes max size of each of initially created splits for a single query. The logic of initial splits is described in hive.max-initial-splits property. Changing this value changes what is considered small query. Higher value causes smaller parallelism for small queries. Lower value increases concurrency for them. This is max size, as the real size may be lower when end of blocks in single DataNode is reached.

hive.max-initial-splits

• Type: Integer
• Default value: 200
• Description: This property describes how many splits may be initially created for a single query. The initial splits are created to allow better concurrency for small queries. Hive connector will create first hive.max-initial-splits splits with size of hive.max-initial-split-size instead of hive.max-split-size. Having this value higher will force more splits to have smaller size effectively increasing definition of what is considered small query in database.

hive.max-outstanding-splits

• Type: Integer (at least 1)
• Default value: 1000
• Description: Limit of number of splits waiting to be served by split source. After reaching this limit writers will stop writing new splits to split source until some of them are used by workers. Higher value will increase memory usage, but will allow to concentrate all IO at one time which may be much faster and increase resources utilization.

hive.max-partitions-per-writers

• Type: Integer (at least 1)
• Default value: 100
• Description: Maximum number of partitions per writer. If higher number of partitions per writer will be required to complete query, the query will fail. By manipulating this value one may change how large queries are meant to be dropped from DB which may help with error detection.

hive.max-split-iterator-threads

• Type: Integer (at least 1)
• Default value: 1000
• Description: This property describes how many threads may be used to iterate through splits when loading them to the worker nodes. Higher value may increase parallelism, but high concurrency may cause time being wasted on context switching.

hive.max-split-size

• Type: String (data size)
• Default value: 64 MB
• Description: This value describes max size of split that is created after using all hive-max-initial-split-size of initial splits. The logic of initial splits is described in hive.max-initial-splits. Having this value higher causes smaller parallelism which may be desirable when queries are very large and cluster is stable allowing to process data locally more efficiently without wasting time for context switching, synchronization and data collecting. The optimal value should be aligned with average query size in system.

hive.metastore.partition-batch-size.max

• Type: Integer (at least 1)
• Default value: 100
• Description: This together with hive.metastore.partition-batch-size.min defines range of partition sizes read from Hive. First partition is always of size hive.metastore.partition-batch-size.min and each following partition is two times bigger then previous up to hive.mestastore.partition-batch-size.max (the formula for n partition size is min(hive.metastore.partition-batch-size.max, (2``^``n) * hive.metastore.partition-batch-size.min)). This algorithm allows to adjust partition size live to what is required. If size of queries in system differs siginificantly, then this range should be extended to better adjust to processed case. In case of cluster working with queries with about the same size, both values may be same for maximal attunement giving slight edge in processing time.

hive.metastore.partition-batch-size.min

• Type: Integer (at least 1)
• Default value: 10
• Description: See hive.metastore.partition-batch-size.max.

hive.optimized-reader.enabled

• Type: Boolean
• Default value: false
• Description: Deprecated Enables number of reader improvements introduced by alternative ORC implementation. The new reader supports vectorized reads, lazy loading, and predicate push down, all of which make the reader more efficient and typically reduces wall clock time for a query. However as the code has changed significantly it may or may not introduce some minor issues, so it can be disabled if some problems with environment are noticed.

hive.orc.max-buffer-size

• Type: String (data size)
• Default value: 8 MB
• Description: Serves as default value for orc_max_buffer_size and orc_stream_buffer_size session properties defining max size of ORC read or streaming operators. Higher value will allow bigger chunks to be processed but will decrease concurrency level.

hive.orc.max-merge-distance

• Type: String (data size)
• Default value: 1 MB
• Description: Serves as default value for orc_max_merge_distance session property. Defines maximum size of gap between two reads to merge into a single read. The reads may be merged if distance between requested data ranges in data source is smaller or equal to this value.

hive.orc.stream-buffer-size

• Type: String (data size)
• Default value: 8 MB
• Description: Unused

hive.parquet-optimized-reader.enabled

• Type: Boolean
• Default value: false
• Description: Deprecated Serves as default value for parquet_optimized_reader_enabled session property. Enables number of reader improvements introduced by alternative parquet implementation. The new reader supports vectorized reads, lazy loading, and predicate push down, all of which make the reader more efficient and typically reduces wall clock time for a query. However as the code has changed significantly it may or may not introduce some minor issues, so it can be disabled if some problems with environment are noticed. This property enables/disables all optimizations except of predicate pushdown as it is managed by hive.parquet-predicate-pushdown.enabled property.

hive.parquet-predicate-pushdown.enabled

• Type: Boolean
• Default value: false
• Description: Deprecated Serves as default value for parquet_predicate_pushdown_enabled sesssion property. See hive.parquet-optimized-reader.enabled.

hive.parquet.use-column-names

• Type: Boolean
• Default value: false
• Description: Access Parquet columns using names from the file. By default, columns in Parquet files are accessed by their ordinal position in the Hive table definition. Setting this property allows to use columns names recorded in the Parquet file instead.

hive.s3.max-connections

• Type: Integer (at least 1)
• Default value: 500
• Description: This value the maximum number of connections to S3. How many connection to S3 cluster may be open at the same time by the S3 driver. Higher value may increase network utilization when cluster is used on high speed network. However higher value relies more on S3 servers being well configured for high parallelism.

hive.s3.multipart.min-file-size

• Type: String (data size, at least 16 MB)
• Default value: 16 MB
• Description: Minimum file size for an S3 multipart upload. This property describes how big file must be to be uploaded to S3 cluster using multipart feature. Amazon recommendation is to use 100 MB value here, however lower value may allow to increase upload parallelism and can decrease data lost/data sent ratio in unstable network conditions.

hive.s3.multipart.min-part-size

• Type: String (data size, at least 5 MB)
• Default value: 5 MB
• Description: Defines the minimum part size for upload parts. Decreasing the minimum part size causes multipart uploads to be split into a larger number of smaller parts. Setting this value too low has a negative effect on transfer speeds, causing extra latency and network communication for each part.