Browse code Initial commit master
0 parent commit ed99d5177a4ec5515f9f23b138260f01c9685c49 Johan Dahlberg authored on 4 Mar 2015

Showing 10 changed files

• README.md
• ansible.cfg
• create_spark_cloud_playbook.yml
• deploy_spark_playbook.yml
• localhost_inventory
• openstack_inventory.py
• templates/config.j2
• templates/hosts.j2
• templates/slaves.j2
• templates/spark-env.sh.j2

Ignore Space Show notes View README.md 0 → 100644

Setting up a stand-alone spark cluster on OpenStack =================================================== This describes how start a stand alone [Spark](http://spark.apache.org/) cluster on open stack, using two [ansible](http://www.ansible.com) playbooks. This has been tested on the [Uppmax](http://www.uppmax.uu.se/) private cloud smog. The open stack dymamic inventory code presented here is adapted from: https://github.com/lukaspustina/dynamic-inventory-for-ansible-with-openstack How to start it? ----------------- - Create a host from which to run ansible in your OpenStack dashboard and associate a floating IP to is so that you can `ssh` in to it. - `ssh` to the machine you just created. - Clone this repository: ``` git clone https://github.com/johandahlberg/ansible_spark_openstack.git ``` - Create a dir called `files` in the repo root dir and copy you ssh-keys (these cannot have a password) there. This is used to enable password-less ssh access between the nodes: - Download you OpenStack RC file from the OpenStack dashboard (it's available under "Access & Security -> API Access") - Source your OpenStack RC file: `source <path to rc file>`, and fill in your OpenStack password. This will load information about you OpenStack Setup into your environment. - Create the security group for spark. Since spark will start some services on random ports this will allow all tcp traffic within the security group: ``` nova secgroup-create spark "internal security group for spark" nova secgroup-add-group-rule spark spark tcp 1 65535 ``` - Setup the name of your network. `export OS_NETWORK_NAME="<name of your network>"` If you like you can add this to your OpenStack RC file, or set it in your `bash_rc`. (You can find the name of your network in your OpenStack dashboard) - First run the playbook which creates your nodes. Open `create_spark_cloud_playbook.yml` and edit variables to set your ssh-key and how many workers you want to create, then run: ``` ansible-playbook -i localhost_inventory --private-key=<your_ssh_key> create_spark_cloud_playbook.yml ``` - Open: `deploy_spark_playbook.yml` and set the `ssh_keys_to_use` variable to your ssh-key. Then install spark on the nodes (I've noticed that sometimes it takes a while for the ssh-server on the nodes to start, so if you get an initial ssh-error, wait a few minutes and try again). ``` ansible-playbook -i openstack_inventory.py --private-key=<your_ssh_key> deploy_spark_playbook.yml ``` - Once this has finished successfully your spark cluster should be up and running! `ssh` into the spark-master node and try your new Spark cluster it by kicking of a shell: ``` ./opt/spark-1.2.1-bin-hadoop2.4/bin/spark-shell --master spark://spark-master:7077 --executor-memory 6G ``` Tips ---- If you don't want to open the web-facing ports you can use ssh-forwarding to reach the web-interfaces, e.g ``` ssh -L 8080:spark-master:8080 -i <your key> ubuntu@<spark-master-ip> ``` Acknowledements --------------- - Mikael Huss for sharing his insights on Spark and collaborating with me on this - Zeeshan Ali Shah(@zeeshanali) for helping me get going with OpenStack

Ignore Space Show notes View ansible.cfg 0 → 100644
[ssh_connection] ssh_args = -o StrictHostKeyChecking=no

Ignore Space Show notes View create_spark_cloud_playbook.yml 0 → 100644

--- - hosts: localhost connection: local vars: state: present # Set to absent to take down machines nbr_of_slaves: 3 key_name: cloud_key tasks: - name: Create a new spark master instance nova_compute: state: "{{ state }}" name: spark-master image_id: 9bf957ba-a0ce-4513-ba8c-e80d42ea9faf key_name: "{{ key_name }}" wait_for: 200 flavor_id: 4 nics: - net-id: b045a7de-5aa4-43a2-86ff-049fd1ba1884 security_groups: spark,default meta: hostname: spark-master ansible_host_groups: spark_masters,default register: spark_master - name: Create a new spark slaves nova_compute: state: "{{ state }}" name: "{{ item }}" image_id: 9bf957ba-a0ce-4513-ba8c-e80d42ea9faf key_name: "{{ key_name }}" wait_for: 200 flavor_id: 4 nics: - net-id: b045a7de-5aa4-43a2-86ff-049fd1ba1884 security_groups: spark,default meta: hostname: "{{ item }}" ansible_host_groups: spark_slaves,default register: spark_slaves with_sequence: start=0 end="{{ nbr_of_slaves }}" format=spark-slave%02x

Ignore Space Show notes View deploy_spark_playbook.yml 0 → 100644

--- # ------------------------ # Deploy the general stuff # ------------------------ - hosts: all sudo: yes vars: ssh_keys_to_use: files/cloud.key user: ubuntu pre_tasks: - name: Update APT cache apt: update_cache=yes tasks: - name: install java apt: name=openjdk-7-jre state=present update_cache=yes - name: disable net.ipv6.conf.all.disable_ipv6 sysctl: name=net.ipv6.conf.all.disable_ipv6 value=1 state=present - name: disable net.ipv6.conf.default.disable_ipv6 sysctl: name=net.ipv6.conf.default.disable_ipv6 value=1 state=present - name: disable net.ipv6.conf.lo.disable_ipv6 sysctl: name=net.ipv6.conf.lo.disable_ipv6 value=1 state=present - name: distribute host file template: src=templates/hosts.j2 dest=/etc/hosts - name: deploy ssh-keys copy: src={{ssh_keys_to_use}} dest=/home/{{ user }}/.ssh/ register: ssh_key - name: distribute ssh config template: src=templates/config.j2 dest=/home/{{ user }}/.ssh/config register: ssh_debug - name: download spark get_url: url=http://d3kbcqa49mib13.cloudfront.net/spark-1.2.1-bin-hadoop2.4.tgz dest=/opt/ sha256sum=8e618cf67b3090acf87119a96e5e2e20e51f6266c44468844c185122b492b454 - name: unzip spark unarchive: copy=no src=/opt/spark-1.2.1-bin-hadoop2.4.tgz dest=/opt - name: deploy slaves configuration template: src=templates/slaves.j2 dest=/opt/spark-1.2.1-bin-hadoop2.4/conf/slaves # -------------------------------------------------- # Kick of spark (making the master start the slaves) # -------------------------------------------------- - hosts: spark_masters tasks: - name: stop spark master (if running) command: /opt/spark-1.2.1-bin-hadoop2.4/sbin/stop-master.sh - name: start spark master shell: SPARK_MASTER_IP="{{ ansible_hostname }}" /opt/spark-1.2.1-bin-hadoop2.4/sbin/start-master.sh - name: stop the slaves (if running) shell: /opt/spark-1.2.1-bin-hadoop2.4/sbin/stop-slaves.sh - name: start the slaves shell: /opt/spark-1.2.1-bin-hadoop2.4/sbin/start-slaves.sh

Ignore Space Show notes View localhost_inventory 0 → 100644
[localhost] 127.0.0.1

Ignore Space Show notes View openstack_inventory.py 0 → 100755

#!/usr/bin/env python ################################################################################ # Dynamic inventory generation for Ansible # In orignal version found here: https://github.com/lukaspustina/dynamic-inventory-for-ansible-with-openstack # Author lukas.pustina@codecentric.de # With additions from Johan Dahlberg - johan.dahlberg@medsci.uu.se # # This Python script generates a dynamic inventory based on OpenStack instances. # # The script is passed via -i <script name> to ansible-playbook. Ansible # recognizes the execute bit of the file and executes the script. It then # queries nova via the novaclient module and credentials passed via environment # variables -- see below. # # The script iterates over all instances of the given tenant and checks if the # instances' metadata have set keys OS_METADATA_KEY -- see below. These keys shall # contain all Ansible host groups comma separated an instance shall be part of, # e.g., u'ansible_host_groups': u'admin_v_infrastructure,apt_repos'. # It is also possible to set Ansible host variables, e.g., # u'ansible_host_vars': u'dns_server_for_domains->domain1,domain2;key2->value2' # Values with a comma will be transformed into a list. # # Metadata of an instance may be set during boot, e.g., # > nova boot --meta <key=value> # , or to a running instance, e.g., # nova meta <instance name> set <key=value> # # *** Requirements *** # * Python: novaclient module be installed which is part of the nova ubuntu # package. # * The environment variables OS_USERNAME, OS_PASSWORD, OS_TENANT_NAME, # OS_AUTH_URL and OS_NETWORK_NAME must be set according to nova. # ################################################################################ from __future__ import print_function from novaclient.v2 import client import os, sys, json OS_METADATA_KEY = { 'host_groups': 'ansible_host_groups', 'host_vars': 'ansible_host_vars' } try: OS_NETWORK_NAME = os.environ['OS_NETWORK_NAME'] except KeyError as e: print("ERROR: environment variable %s is not defined" % e, file=sys.stderr) sys.exit(-1) def main(args): credentials = getOsCredentialsFromEnvironment() nt = client.Client(credentials['USERNAME'], credentials['PASSWORD'], credentials['TENANT_NAME'], credentials['AUTH_URL'], service_type="compute") inventory = {} inventory['_meta'] = { 'hostvars': {} } for server in nt.servers.list(): floatingIp = getFloatingIpFromServerForNetwork(server, OS_NETWORK_NAME) if floatingIp: for group in getAnsibleHostGroupsFromServer(nt, server.id): addServerToHostGroup(group, floatingIp, inventory) host_vars = getAnsibleHostVarsFromServer(nt, server.id) if host_vars: addServerHostVarsToHostVars(host_vars, floatingIp, inventory) dumpInventoryAsJson(inventory) def getOsCredentialsFromEnvironment(): credentials = {} try: credentials['USERNAME'] = os.environ['OS_USERNAME'] credentials['PASSWORD'] = os.environ['OS_PASSWORD'] credentials['TENANT_NAME'] = os.environ['OS_TENANT_NAME'] credentials['AUTH_URL'] = os.environ['OS_AUTH_URL'] except KeyError as e: print("ERROR: environment variable %s is not defined" % e, file=sys.stderr) sys.exit(-1) return credentials def getAnsibleHostGroupsFromServer(novaClient, serverId): metadata = getMetaDataFromServer(novaClient, serverId, OS_METADATA_KEY['host_groups']) if metadata: return metadata.split(',') else: return [] def getMetaDataFromServer(novaClient, serverId, key): return novaClient.servers.get(serverId).metadata.get(key, None) def getAnsibleHostVarsFromServer(novaClient, serverId): metadata = getMetaDataFromServer(novaClient, serverId, OS_METADATA_KEY['host_vars']) if metadata: host_vars = {} for kv in metadata.split(';'): key, values = kv.split('->') values = values.split(',') host_vars[key] = values return host_vars else: return None def getFloatingIpFromServerForNetwork(server, network): for addr in server.addresses.get(network): if addr.get('OS-EXT-IPS:type') == 'floating': return addr['addr'] if addr.get('OS-EXT-IPS:type') == 'fixed': return addr['addr'] return None def addServerToHostGroup(group, floatingIp, inventory): host_group = inventory.get(group, {}) hosts = host_group.get('hosts', []) hosts.append(floatingIp) host_group['hosts'] = hosts inventory[group] = host_group def addServerHostVarsToHostVars(host_vars, floatingIp, inventory): inventory_host_vars = inventory['_meta']['hostvars'].get(floatingIp, {}) inventory_host_vars.update(host_vars) inventory['_meta']['hostvars'][floatingIp] = inventory_host_vars def dumpInventoryAsJson(inventory): print(json.dumps(inventory, indent=4)) if __name__ == "__main__": main(sys.argv)

Ignore Space Show notes View templates/config.j2 0 → 100644
{% for host in groups['all'] %} Host {{ hostvars[host].ansible_hostname }} User ubuntu IdentityFile {{ ssh_key.dest }} StrictHostKeyChecking no {% endfor %}

Ignore Space Show notes View templates/hosts.j2 0 → 100644

127.0.0.1 localhost {{ ansible_eth0.ipv4.address }} {{ ansible_hostname }} {% for host in groups['all'] %} {{ hostvars[host].ansible_eth0.ipv4.address }} {{ hostvars[host].ansible_hostname }} {% endfor %} # The following lines are desirable for IPv6 capable hosts ::1 ip6-localhost ip6-loopback fe00::0 ip6-localnet ff00::0 ip6-mcastprefix ff02::1 ip6-allnodes ff02::2 ip6-allrouters ff02::3 ip6-allhosts

Ignore Space Show notes View templates/slaves.j2 0 → 100644
# A Spark Worker will be started on each of the machines listed below. {% for host in groups['spark_slaves'] %} {{ hostvars[host].ansible_hostname }} {% endfor %}

Ignore Space Show notes View templates/spark-env.sh.j2 0 → 100755

#!/usr/bin/env bash # This file is sourced when running various Spark programs. # Copy it as spark-env.sh and edit that to configure Spark for your site. # Options read when launching programs locally with # ./bin/run-example or ./bin/spark-submit # - HADOOP_CONF_DIR, to point Spark towards Hadoop configuration files # - SPARK_LOCAL_IP, to set the IP address Spark binds to on this node # - SPARK_PUBLIC_DNS, to set the public dns name of the driver program # - SPARK_CLASSPATH, default classpath entries to append # Options read by executors and drivers running inside the cluster # - SPARK_LOCAL_IP, to set the IP address Spark binds to on this node # - SPARK_PUBLIC_DNS, to set the public DNS name of the driver program # - SPARK_CLASSPATH, default classpath entries to append # - SPARK_LOCAL_DIRS, storage directories to use on this node for shuffle and RDD data # - MESOS_NATIVE_LIBRARY, to point to your libmesos.so if you use Mesos # Options read in YARN client mode # - HADOOP_CONF_DIR, to point Spark towards Hadoop configuration files # - SPARK_EXECUTOR_INSTANCES, Number of workers to start (Default: 2) # - SPARK_EXECUTOR_CORES, Number of cores for the workers (Default: 1). # - SPARK_EXECUTOR_MEMORY, Memory per Worker (e.g. 1000M, 2G) (Default: 1G) # - SPARK_DRIVER_MEMORY, Memory for Master (e.g. 1000M, 2G) (Default: 512 Mb) # - SPARK_YARN_APP_NAME, The name of your application (Default: Spark) # - SPARK_YARN_QUEUE, The hadoop queue to use for allocation requests (Default: ‘default’) # - SPARK_YARN_DIST_FILES, Comma separated list of files to be distributed with the job. # - SPARK_YARN_DIST_ARCHIVES, Comma separated list of archives to be distributed with the job. # Options for the daemons used in the standalone deploy mode # - SPARK_MASTER_IP, to bind the master to a different IP address or hostname #SPARK_MASTER_IP= groups['sparkmasters'](0) # - SPARK_MASTER_PORT / SPARK_MASTER_WEBUI_PORT, to use non-default ports for the master # - SPARK_MASTER_OPTS, to set config properties only for the master (e.g. "-Dx=y") # - SPARK_WORKER_CORES, to set the number of cores to use on this machine # - SPARK_WORKER_MEMORY, to set how much total memory workers have to give executors (e.g. 1000m, 2g) # - SPARK_WORKER_PORT / SPARK_WORKER_WEBUI_PORT, to use non-default ports for the worker SPARK_WORKER_PORT=9999 # - SPARK_WORKER_INSTANCES, to set the number of worker processes per node # - SPARK_WORKER_DIR, to set the working directory of worker processes # - SPARK_WORKER_OPTS, to set config properties only for the worker (e.g. "-Dx=y") # - SPARK_HISTORY_OPTS, to set config properties only for the history server (e.g. "-Dx=y") # - SPARK_DAEMON_JAVA_OPTS, to set config properties for all daemons (e.g. "-Dx=y") # - SPARK_PUBLIC_DNS, to set the public dns name of the master or workers # Generic options for the daemons used in the standalone deploy mode # - SPARK_CONF_DIR Alternate conf dir. (Default: ${SPARK_HOME}/conf) # - SPARK_LOG_DIR Where log files are stored. (Default: ${SPARK_HOME}/logs) # - SPARK_PID_DIR Where the pid file is stored. (Default: /tmp) # - SPARK_IDENT_STRING A string representing this instance of spark. (Default: $USER) # - SPARK_NICENESS The scheduling priority for daemons. (Default: 0)

Show line notes below