============================================== Automated Installation on a YARN-Based Cluster ============================================== If you are planning to use HDP distribution, you can use Ambari and `Apache Slider`_. to perform automated Presto installation and integration on a YARN-based cluster. During installation, both the Apache Slider package and Presto are installed. .. contents:: Installing and Integrating Presto with YARN Deploying Presto on a YARN-Based Cluster ======================================== The installation procedures assume that you have a basic knowledge of Presto and the configuration files and properties it uses. .. note:: All example files referred to are from: https://github.com/prestodb/presto-yarn/ ----- Pre-Requisites -------------- - A cluster with HDP 2.2+ or CDH5.4+ installed - Apache Slider 0.80.0 (download from https://slider.incubator.apache.org/) - JDK 1.8 - Zookeeper - openssl >= 1.0.1e-16 - Ambari 2.1 .. _Package: https:www.teradata.com/presto .. _Apache slider: https://slider.incubator.apache.org/ .. BELOW CONTENT IS GENERATED BY PANDOC FROM PRESTO-YARN README.md file, except - added pre-requisities section - inner links got fixed - links section updates - added note where example files are stored ----- Presto Installation Directory Structure --------------------------------------- When you use Ambari Slider View to install Presto on a YARN-based cluster, the Presto installation directory structure differs from the standard structure. For more information, see :doc:`Presto Installation Directory Structure for YARN-Based Clusters `. ----- Presto Installation Configuration Options ----------------------------------------- During installation, Ambari Slider View allows you to select configuration options required for running Presto. For more information, see :doc:`Presto Configuration Options for YARN-Based Clusters `. ----- Using Ambari Slider View to Install Presto on a YARN-Based Cluster ================================================================== Ambari supports deploying Slider application packages using Slider View and provides Slider integration. Slider View for Ambari allows you to deploy and manage Slider apps from Ambari Web. Use Ambari Slider View and the following steps to deploy Presto on YARN: 1. Install the Ambari server. See: http://docs.hortonworks.com/HDPDocuments/Ambari-2.1.0.0/bk_Installing_HDP_AMB/content/ch_Installing_Ambari.html. 2. Download the Apache Slider package. See: https://slider.incubator.apache.org/ 3. Copy the Presto app package ``presto-yarn-package--.zip`` to ``/var/lib/ambari-server/resources/apps/`` directory on your Ambari server node. 4. Restart ambari-server. 5. Log on to Apache Ambari, ``http://ambariserver_ip:8080`` #username-admin password-admin 6. Name your cluster, provide the configuration of the cluster, and follow the steps on the WebUI. 7. Customize/configure the services and install them. A minimum of HDFS, YARN, Zookeeper is required for Slider to work. You must also select Slider to be installed. 8. For the Slider client installed, you need to update its configuration if you are not using the default installation paths for Hadoop and Zookeeper. Thus ``slider-env.sh`` should point to your JAVA\_HOME and HADOOP\_CONF\_DIR :: export JAVA_HOME=/usr/lib/jvm/java export HADOOP_CONF_DIR=/etc/hadoop/conf 9. For zookeeper, if you are using a different installation directory from the default one at ``/usr/lib/zookeeper``: * Add a custom property to the ``slider-client`` section in Slider configuration with the key: ``zk.home`` and value: ``path_to_your_zookeeper``. * If using a different port from the default ``2181``, then add the key ``slider.zookeeper.quorum`` and value: ``master:5181``, where ``master`` is the node and ``5181`` is the port. 10. Once you have all the services up and running on the cluster, you can configure Slider in Ambari to create and manage your application by creating a "View". a. Go to ``admin`` (top right corner) -> ``Manage Ambari``. b. From the left pane, select ``Views``. c. Create a Slider View by populating all the necessary fields with a preferred instance name (for example, Slider). ``ambari.server.url`` can be of the format ``http://:8080/api/v1/clusters/``, where ```` is what you have named your Ambari cluster. d. Select the "Views" control icon in the upper right. e. Select the instance you created in the previous step (for example, "Slider"). f. Click ``Create App`` to create a new Presto YARN application. 11. Provide details of the Presto service. By default, the UI will be populated with the values you have in the ``*-default.json`` files in your ``presto-yarn-package-*.zip``. 12. The app name should be of lower case. For example: presto1. 13. You can set the configuration property fields required for your cluster. For example, if you want to set a connector for Presto, you can update the ``global.catalog`` property. See the following for an explanation of each configuration property. | :doc:`Presto Configuration Options for YARN-Based Clusters ` 14. Prepare HDFS for Slider. The user directory you create here should be for the same user you set in the ``global.app_user`` field. If the ``app_user`` is going to be ``yarn`` then do the following: :: su hdfs hdfs dfs -mkdir -p /user/yarn su hdfs hdfs dfs -chown yarn:yarn /user/yarn 15. Change the ``global.presto_server_port`` from 8080 to another unused port, for example, 8089, since Ambari by default uses 8080. 16. Pre-create the data directory in the UI (added in ``appConfig-default.json`` eg: ``/var/lib/presto/``) on all nodes. The directory must be owned by ``global.app_user``, otherwise Slider will fail to start Presto due to permission errors. :: mkdir -p /var/lib/presto/data chown -R yarn:hadoop /var/lib/presto/data 18. If you want to add any additional custom properties, use the Custom property section. Additional properties currently supported are: * ``site.global.plugin`` * ``site.global.additional_config_properties`` * ``site.global.additional_node_properties``. For the requirements and format of these properties, see: | :doc:`Presto Configuration Options for YARN-Based Clusters ` 19. Click Finish. This is the equivalent of ``package --install`` and ``create`` performed with the bin/slider script. If successfully deployed, you will see the YARN application started for Presto. You can do the following: * Click ``app launched`` and monitor the status from Slider view. * Click``Quick Links``, which should take you to the YARN WebUI. If your application is running successfully, it should always be available in the YARN resource manager as a "RUNNING" application. 20. If the job fails, check the job history’s logs and the logs on the node’s disk. See :doc:`Debugging and Logging for YARN-Based Clusters `. 21. You can manage the application lifecycle (for example: start, stop, flex, and destroy) from the View UI. ----- Additional Configuration Options ================================ After you install Presto and Slider, you can reconfigure Presto or perform additional configuration. Reconfiguring Presto in Slider View ----------------------------------- After you launch Presto you can update its configuration. For example, you can add a new connector. 1. On the Slider View instance screen, go to ``Actions.`` 2. Stop the running Presto application. 3. Click `Destroy`` to remove the existing Presto instance running in Slider. 4. Click the ``Create App`` button to re-create a new Presto instance in Slider and make configuration updates. Advanced Configuration Options ------------------------------ The following advanced configuration options are available: + Configuring memory, CPU, and YARN CGroups + Failure policy + YARN label For more information, see :doc:`Advanced Configuration Options for YARN-Based Clusters ` ----- Debugging and Logging ===================== For more information, see :doc:`Debugging and Logging for YARN-Based Clusters `. ----- Links ===== - https://github.com/prestodb/presto-yarn/blob/master/README.md - http://slider.incubator.apache.org/docs/getting\_started.html - http://docs.hortonworks.com/HDPDocuments/Ambari-2.0.1.0/bk\_Installing\_HDP\_AMB/content/ch\_Installing\_Ambari.html