To modify the parameter, navigate to the Hive Configs tab and find the Data per Reducer parameter on the Settings page. This parameter is based on your particular data requirements, compression settings, and other environmental factors. Tuning it too low could also produce too many reducers, potentially adversely affecting performance. Tuning this value down increases parallelism and may improve performance. The .per.reducer parameter specifies the number of bytes processed per reducer. With the default settings, this example is four reducers. Hive estimates the number of reducers needed as: (number of bytes input to mappers / .per.reducer). That data in ORC format with Snappy compression is 1 GB. However, Hive may have too few reducers by default, causing bottlenecks.įor example, say you have an input data size of 50 GB. Tune reducersĪpache ORC and Snappy both offer high performance. To get an optimal result, choose appropriate parameter values. These changes affect all Tez jobs across the server. Set both parameters to 33,554,432 bytes (32 MB). Expand the General panel, and locate the -size and -size parameters. To modify the limit parameters, navigate to the Configs tab of the Tez service. -size: Upper limit on the size of a grouped split, with a default value of 1 GB (1,073,741,824 bytes).Īs a performance guideline, lower both of these parameters to improve latency, increase for more throughput.įor example, to set four mapper tasks for a data size of 128 MB, you would set both parameters to 32 MB each (33,554,432 bytes).-size: Lower limit on the size of a grouped split, with a default value of 16 MB (16,777,216 bytes).The following two configuration parameters drive the number of splits for the Tez execution engine: The number of mappers depends on the number of splits. Hadoop tries to split ( map) a single file into multiple files and process the resulting files in parallel. The Optimization property's default value is Tez.
In the Hive Configs tab, type execution engine in the filter box. HDInsight Linux clusters have Tez as the default execution engine. Hive provides two execution engines: Apache Hadoop MapReduce and Apache TEZ. To modify Hive configuration parameters, select Hive from the Services sidebar.The following sections describe configuration options for optimizing overall Apache Hive performance. For an introduction to Ambari Web UI, see Manage HDInsight clusters by using the Apache Ambari Web UI. Apache Ambari is a web interface to manage and monitor HDInsight clusters.
0 kommentar(er)
