Install hdfs and small fixes to spark

stephen.cranefield / ansible_spark_openstack

Browse code Install hdfs and small fixes to spark master
1 parent e3fc183 commit 6225c79bccb7fde94b26f9aa57938e2e9855d995 Johan Dahlberg authored on 5 Mar 2015

Patch

Showing 9 changed files

Ignore Space Show notes View README.md
Setting up a stand-alone spark cluster on OpenStack =================================================== This describes how start a standalone [Spark](http://spark.apache.org/) cluster on open stack, using two [ansible](http://www.ansible.com) playbooks. This has been tested on the [Uppmax](http://www.uppmax.uu.se/) private cloud smog. It will install spark and hdfs, and start the required services on the nodes. Please note that this is a proof-of-concept implementation, and that is is not ready for use in a production setting. Any pull requests to improve upon this to bring it closer to a production ready state are very much appreciated. The open stack dymamic inventory code presented here is adapted from: https://github.com/lukaspustina/dynamic-inventory-for-ansible-with-openstack How to start it? ----------------- - Create a host from which to run ansible in your OpenStack dashboard and associate a floating IP to is so that you can `ssh` in to it. - `ssh` to the machine you just created. - Install the pre-requisites: ``` sudo apt-get install python-pip python-dev sudo pip install ansible sudo pip install python-novaclient ``` - Clone this repository: ``` git clone https://github.com/johandahlberg/ansible_spark_openstack.git ``` - Create a dir called `files` in the repo root dir and copy you ssh-keys (these cannot have a password) there. This is used to enable password-less ssh access between the nodes: - Download you OpenStack RC file from the OpenStack dashboard (it's available under "Access & Security -> API Access") - Source your OpenStack RC file: `source <path to rc file>`, and fill in your OpenStack password. This will load information about you OpenStack Setup into your environment. - Create the security group for spark. Since spark will start some services on random ports this will allow all tcp traffic within the security group: ``` nova secgroup-create spark "internal security group for spark" nova secgroup-add-group-rule spark spark tcp 1 65535 ``` - Setup the name of your network. `export OS_NETWORK_NAME="<name of your network>"` If you like you can add this to your OpenStack RC file, or set it in your `bash_rc`. (You can find the name of your network in your OpenStack dashboard) - Edit the setup variables to fit your setup. Open `vars/main.yml` and setup the variables as explained there. - One all the variables are in place you should now be able to create your instances: ``` ansible-playbook -i localhost_inventory --private-key=<your_ssh_key> create_spark_cloud_playbook.yml ``` - Then install spark on the nodes (I've noticed that sometimes it takes a while for the ssh-server on the nodes to start, so if you get an initial ssh-error, wait a few minutes and try again). ``` ansible-playbook -i openstack_inventory.py --private-key=<your_ssh_key> deploy_spark_playbook.yml ``` - Once this has finished successfully your spark cluster should be up and running! `ssh` into the spark-master node and try your new Spark cluster it by kicking of a shell. Now you're ready to enter into the Spark world. Have fun! ``` spark-shell --master spark://spark-master:7077 --executor-memory 6G ``` Tips ---- If you don't want to open the web-facing ports you can use ssh-forwarding to reach the web-interfaces, e.g ``` ssh -L 8080:spark-master:8080 -i <your key> ubuntu@<spark-master-ip> ``` Acknowledements --------------- - Mikael Huss (@hussius) for sharing his insights on Spark and collaborating with me on this - Zeeshan Ali Shah (@zeeshanali) for helping me get going with OpenStack Setting up a stand-alone spark cluster on OpenStack =================================================== This describes how start a stand alone [Spark](http://spark.apache.org/) cluster on open stack, using two [ansible](http://www.ansible.com) playbooks. This has been tested on the [Uppmax](http://www.uppmax.uu.se/) private cloud smog. The open stack dymamic inventory code presented here is adapted from: https://github.com/lukaspustina/dynamic-inventory-for-ansible-with-openstack How to start it? ----------------- - Create a host from which to run ansible in your OpenStack dashboard and associate a floating IP to is so that you can `ssh` in to it. - `ssh` to the machine you just created. - Install the pre-requisites: ``` sudo apt-get install python-pip python-dev sudo pip install ansible sudo pip install python-novaclient ``` - Clone this repository: ``` git clone https://github.com/johandahlberg/ansible_spark_openstack.git ``` - Create a dir called `files` in the repo root dir and copy you ssh-keys (these cannot have a password) there. This is used to enable password-less ssh access between the nodes: - Download you OpenStack RC file from the OpenStack dashboard (it's available under "Access & Security -> API Access") - Source your OpenStack RC file: `source <path to rc file>`, and fill in your OpenStack password. This will load information about you OpenStack Setup into your environment. - Create the security group for spark. Since spark will start some services on random ports this will allow all tcp traffic within the security group: ``` nova secgroup-create spark "internal security group for spark" nova secgroup-add-group-rule spark spark tcp 1 65535 ``` - Setup the name of your network. `export OS_NETWORK_NAME="<name of your network>"` If you like you can add this to your OpenStack RC file, or set it in your `bash_rc`. (You can find the name of your network in your OpenStack dashboard) - Edit the setup variables to fit your setup. Open `vars/main.yml` and setup the variables as explained there. - One all the variables are in place you should now be able to create your instances: ``` ansible-playbook -i localhost_inventory --private-key=<your_ssh_key> create_spark_cloud_playbook.yml ``` - Then install spark on the nodes (I've noticed that sometimes it takes a while for the ssh-server on the nodes to start, so if you get an initial ssh-error, wait a few minutes and try again). ``` ansible-playbook -i openstack_inventory.py --private-key=<your_ssh_key> deploy_spark_playbook.yml ``` - Once this has finished successfully your spark cluster should be up and running! `ssh` into the spark-master node and try your new Spark cluster it by kicking of a shell: ``` /opt/spark-1.2.1-bin-hadoop2.4/bin/spark-shell --master spark://spark-master:7077 --executor-memory 6G ``` Tips ---- If you don't want to open the web-facing ports you can use ssh-forwarding to reach the web-interfaces, e.g ``` ssh -L 8080:spark-master:8080 -i <your key> ubuntu@<spark-master-ip> ``` Acknowledements --------------- - Mikael Huss (@hussius) for sharing his insights on Spark and collaborating with me on this - Zeeshan Ali Shah (@zeeshanali) for helping me get going with OpenStack

Ignore Space Show notes View

README.md

Setting up a stand-alone spark cluster on OpenStack
===================================================

This describes how start a standalone [Spark](http://spark.apache.org/) cluster on open stack, using two [ansible](http://www.ansible.com) playbooks. This has been tested on the [Uppmax](http://www.uppmax.uu.se/) private cloud smog.

It will install spark and hdfs, and start the required services on the nodes. Please note that this is a proof-of-concept implementation, and that is is not ready for use in a production setting. Any pull requests to improve upon this to bring it closer to a production ready state are very much appreciated.

The open stack dymamic inventory code presented here is adapted from: https://github.com/lukaspustina/dynamic-inventory-for-ansible-with-openstack

How to start it?
-----------------
- Create a host from which to run ansible in your OpenStack dashboard and associate a floating IP to is so that you can `ssh` in to it.
- `ssh` to the machine you just created.
- Install the pre-requisites:
```
sudo apt-get install python-pip python-dev
sudo pip install ansible
sudo pip install python-novaclient
```
- Clone this repository:
```
git clone https://github.com/johandahlberg/ansible_spark_openstack.git
```
- Create a dir called `files` in the repo root dir and copy you ssh-keys (these cannot have a password) there. This is used to enable password-less ssh access between the nodes:
- Download you OpenStack RC file from the OpenStack dashboard (it's available under "Access & Security -> API Access") 
- Source your OpenStack RC file: `source <path to rc file>`, and fill in your OpenStack password. This will load information about you OpenStack Setup into your environment.
- Create the security group for spark. Since spark will start some services on random ports this will allow all tcp traffic within the security group:
```
nova secgroup-create spark "internal security group for spark"
nova secgroup-add-group-rule spark spark tcp 1 65535
```
- Setup the name of your network. `export OS_NETWORK_NAME="<name of your network>"` If you like you can add this to your OpenStack RC file, or set it in your `bash_rc`. (You can find the name of your network in your OpenStack dashboard)

- Edit the setup variables to fit your setup. Open `vars/main.yml` and setup the variables as explained there.
- One all the variables are in place you should now be able to create your instances:
```
ansible-playbook -i localhost_inventory --private-key=<your_ssh_key> create_spark_cloud_playbook.yml
```
- Then install spark on the nodes (I've noticed that sometimes it takes a while for the ssh-server on the nodes to start, so if you get an initial ssh-error, wait a few minutes and try again).
```
ansible-playbook -i openstack_inventory.py --private-key=<your_ssh_key> deploy_spark_playbook.yml
```
- Once this has finished successfully your spark cluster should be up and running! `ssh` into the spark-master node and try your new Spark cluster it by kicking of a shell. Now you're ready to enter into the Spark world. Have fun!
```
spark-shell --master spark://spark-master:7077 --executor-memory 6G
```

Tips
----
If you don't want to open the web-facing ports you can use ssh-forwarding to reach the web-interfaces, e.g

```
ssh -L 8080:spark-master:8080 -i <your key> ubuntu@<spark-master-ip>
```

Acknowledements
---------------
- Mikael Huss (@hussius) for sharing his insights on Spark and collaborating with me on this
- Zeeshan Ali Shah (@zeeshanali) for helping me get going with OpenStack

Setting up a stand-alone spark cluster on OpenStack
===================================================

This describes how start a stand alone [Spark](http://spark.apache.org/) cluster on open stack, using two [ansible](http://www.ansible.com) playbooks. This has been tested on the [Uppmax](http://www.uppmax.uu.se/) private cloud smog.

The open stack dymamic inventory code presented here is adapted from: https://github.com/lukaspustina/dynamic-inventory-for-ansible-with-openstack

- Edit the setup variables to fit your setup. Open `vars/main.yml` and setup the variables as explained there.
- One all the variables are in place you should now be able to create your instances:
```
ansible-playbook -i localhost_inventory --private-key=<your_ssh_key> create_spark_cloud_playbook.yml
```
- Then install spark on the nodes (I've noticed that sometimes it takes a while for the ssh-server on the nodes to start, so if you get an initial ssh-error, wait a few minutes and try again).
```
ansible-playbook -i openstack_inventory.py --private-key=<your_ssh_key> deploy_spark_playbook.yml
```
- Once this has finished successfully your spark cluster should be up and running! `ssh` into the spark-master node and try your new Spark cluster it by kicking of a shell:
```
/opt/spark-1.2.1-bin-hadoop2.4/bin/spark-shell --master spark://spark-master:7077 --executor-memory 6G
```

Tips
----
If you don't want to open the web-facing ports you can use ssh-forwarding to reach the web-interfaces, e.g

```
ssh -L 8080:spark-master:8080 -i <your key> ubuntu@<spark-master-ip>
```

Acknowledements
---------------
- Mikael Huss (@hussius) for sharing his insights on Spark and collaborating with me on this
- Zeeshan Ali Shah (@zeeshanali) for helping me get going with OpenStack

Ignore Space Show notes View deploy_spark_playbook.yml
--- # ------------------------ # Deploy the general stuff # ------------------------ - hosts: all sudo: yes vars: ssh_public_key_content: "{{ lookup('file', ssh_public_key ) }}" vars_files: - vars/main.yml pre_tasks: - name: Update APT cache apt: update_cache=yes tasks: # General tasks - name: install java apt: name=openjdk-7-jre state=present update_cache=yes - name: disable net.ipv6.conf.all.disable_ipv6 sysctl: name=net.ipv6.conf.all.disable_ipv6 value=1 state=present - name: disable net.ipv6.conf.default.disable_ipv6 sysctl: name=net.ipv6.conf.default.disable_ipv6 value=1 state=present - name: disable net.ipv6.conf.lo.disable_ipv6 sysctl: name=net.ipv6.conf.lo.disable_ipv6 value=1 state=present - name: distribute host file template: src=templates/hosts.j2 dest=/etc/hosts # Install hadoop - name: create hadoop group group: name=hadoop state=present - name: create hadoop user user: name={{ hadoop_user }} comment="Hadoop user" group=hadoop shell=/bin/bash - name: Get hadoop get_url: url=http://apache.mirrors.spacedump.net/hadoop/common/stable/hadoop-2.6.0.tar.gz dest=/usr/local/ sha256sum=7a2ef6e7f468afcae95d0f7214816033c7e5c7982454061ccb117896d58d279f - name: unzip hadoop unarchive: copy=no src=/usr/local/hadoop-2.6.0.tar.gz dest=/usr/local/ owner={{ hadoop_user }} group=hadoop - name: set user and priviliges on hadoop file: path=/usr/local/hadoop-2.6.0 owner={{ hadoop_user }} group=hadoop recurse=yes - name: distribute hadoop conf template: src=templates/hadoop-env.sh.j2 dest=/usr/local/hadoop-2.6.0/etc/hadoop/hadoop-env.sh - name: distribute hadoop core-site.xml template: src=templates/core-site.xml.j2 dest=/usr/local/hadoop-2.6.0/etc/hadoop/core-site.xml - name: distribute hadoop hdfs-site.xml template: src=templates/hdfs-site.xml.j2 dest=/usr/local/hadoop-2.6.0/etc/hadoop/hdfs-site.xml # Setup ssh-setup for normal user and hadoop user - name: deploy authorized keys authorized_key: user={{ item }} key="{{ ssh_public_key_content }}" with_items: - "{{ user }}" - "{{ hadoop_user }}" - name: deploy ssh-keys copy: src={{ssh_keys_to_use}} dest=/home/{{ item }}/.ssh/ with_items: - "{{ user }}" - "{{ hadoop_user }}" - name: distribute ssh config template: src=templates/config.j2 dest=/home/{{ item }}/.ssh/config with_items: - "{{ user }}" - "{{ hadoop_user }}" # Spark stuff - name: download spark get_url: url=http://d3kbcqa49mib13.cloudfront.net/spark-1.2.1-bin-hadoop2.4.tgz dest=/opt/ sha256sum=8e618cf67b3090acf87119a96e5e2e20e51f6266c44468844c185122b492b454 - name: unzip spark unarchive: copy=no src=/opt/spark-1.2.1-bin-hadoop2.4.tgz dest=/opt - name: deploy slaves configuration template: src=templates/slaves.j2 dest=/opt/spark-1.2.1-bin-hadoop2.4/conf/slaves - name: deploy spark-env.sh configuration template: src=templates/spark-env.sh.j2 dest=/opt/spark-1.2.1-bin-hadoop2.4/conf/spark-env.sh # -------------------------------------------------- # Start the hadoop master # -------------------------------------------------- - hosts: spark_masters vars_files: - vars/main.yml tasks: - name: distribute hadoop slaves file template: src=templates/masters.j2 dest=/usr/local/hadoop-2.6.0/etc/hadoop/masters - name: distribute hadoop slaves file template: src=templates/slaves.j2 dest=/usr/local/hadoop-2.6.0/etc/hadoop/slaves - name: format hdfs (unless it's already been done) command: /usr/local/hadoop-2.6.0/bin/hadoop namenode -format creates=/usr/local/hadoop-2.6.0/ansible-format-hdfs - name: touch hfds formatted file (indicates if hdfs has been formatted) file: state=touch path=/usr/local/hadoop-2.6.0/ansible-format-hdfs - name: add hadoop and spark binaries to path for hadoop user lineinfile: dest=/home/{{ hadoop_user }}/.bashrc state=present insertafter=EOF line="export PATH=$PATH:/usr/local/hadoop-2.6.0/bin/:/opt/spark-1.2.1-bin-hadoop2.4/bin/" create=true tags: - current - name: stop hadoop (if running) command: /usr/local/hadoop-2.6.0/sbin/stop-dfs.sh - name: start hadoop command: /usr/local/hadoop-2.6.0/sbin/start-dfs.sh # -------------------------------------------------- # Kick of spark (making the master start the slaves) # and configure spark-master as hadoop master # -------------------------------------------------- - hosts: spark_masters tasks: - name: stop spark master (if running) command: /opt/spark-1.2.1-bin-hadoop2.4/sbin/stop-master.sh - name: start spark master shell: SPARK_MASTER_IP="{{ ansible_hostname }}" /opt/spark-1.2.1-bin-hadoop2.4/sbin/start-master.sh - name: stop the slaves (if running) shell: /opt/spark-1.2.1-bin-hadoop2.4/sbin/stop-slaves.sh - name: start the slaves shell: /opt/spark-1.2.1-bin-hadoop2.4/sbin/start-slaves.sh --- # ------------------------ # Deploy the general stuff # ------------------------ - hosts: all sudo: yes vars_files: - vars/main.yml pre_tasks: - name: Update APT cache apt: update_cache=yes tasks: - name: install java apt: name=openjdk-7-jre state=present update_cache=yes - name: disable net.ipv6.conf.all.disable_ipv6 sysctl: name=net.ipv6.conf.all.disable_ipv6 value=1 state=present - name: disable net.ipv6.conf.default.disable_ipv6 sysctl: name=net.ipv6.conf.default.disable_ipv6 value=1 state=present - name: disable net.ipv6.conf.lo.disable_ipv6 sysctl: name=net.ipv6.conf.lo.disable_ipv6 value=1 state=present - name: distribute host file template: src=templates/hosts.j2 dest=/etc/hosts - name: deploy ssh-keys copy: src={{ssh_keys_to_use}} dest=/home/{{ user }}/.ssh/ register: ssh_key - name: distribute ssh config template: src=templates/config.j2 dest=/home/{{ user }}/.ssh/config register: ssh_debug - name: download spark get_url: url=http://d3kbcqa49mib13.cloudfront.net/spark-1.2.1-bin-hadoop2.4.tgz dest=/opt/ sha256sum=8e618cf67b3090acf87119a96e5e2e20e51f6266c44468844c185122b492b454 - name: unzip spark unarchive: copy=no src=/opt/spark-1.2.1-bin-hadoop2.4.tgz dest=/opt - name: deploy slaves configuration template: src=templates/slaves.j2 dest=/opt/spark-1.2.1-bin-hadoop2.4/conf/slaves # -------------------------------------------------- # Kick of spark (making the master start the slaves) # -------------------------------------------------- - hosts: spark_masters tasks: - name: stop spark master (if running) command: /opt/spark-1.2.1-bin-hadoop2.4/sbin/stop-master.sh - name: start spark master shell: SPARK_MASTER_IP="{{ ansible_hostname }}" /opt/spark-1.2.1-bin-hadoop2.4/sbin/start-master.sh - name: stop the slaves (if running) shell: /opt/spark-1.2.1-bin-hadoop2.4/sbin/stop-slaves.sh - name: start the slaves shell: /opt/spark-1.2.1-bin-hadoop2.4/sbin/start-slaves.sh

Ignore Space Show notes View

deploy_spark_playbook.yml

---
# ------------------------
# Deploy the general stuff
# ------------------------
- hosts: all
  sudo: yes

vars:
    ssh_public_key_content: "{{ lookup('file', ssh_public_key ) }}"
  vars_files:
    - vars/main.yml
 
  pre_tasks:
    - name: Update APT cache
      apt: update_cache=yes

tasks:

# General tasks
    - name: install java
      apt: name=openjdk-7-jre state=present update_cache=yes

- name: disable net.ipv6.conf.all.disable_ipv6
      sysctl: name=net.ipv6.conf.all.disable_ipv6 value=1 state=present

- name: disable net.ipv6.conf.default.disable_ipv6
      sysctl: name=net.ipv6.conf.default.disable_ipv6 value=1 state=present
 
    - name: disable net.ipv6.conf.lo.disable_ipv6
      sysctl: name=net.ipv6.conf.lo.disable_ipv6 value=1 state=present

- name: distribute host file
      template: src=templates/hosts.j2 dest=/etc/hosts

# Install hadoop
    - name: create hadoop group
      group: name=hadoop state=present

- name: create hadoop user
      user: name={{ hadoop_user }} comment="Hadoop user" group=hadoop shell=/bin/bash

- name: Get hadoop 
      get_url:
        url=http://apache.mirrors.spacedump.net/hadoop/common/stable/hadoop-2.6.0.tar.gz 
        dest=/usr/local/ 
        sha256sum=7a2ef6e7f468afcae95d0f7214816033c7e5c7982454061ccb117896d58d279f

- name: unzip hadoop
      unarchive: copy=no src=/usr/local/hadoop-2.6.0.tar.gz dest=/usr/local/ owner={{ hadoop_user }} group=hadoop

- name: set user and priviliges on hadoop
      file: path=/usr/local/hadoop-2.6.0 owner={{ hadoop_user }} group=hadoop recurse=yes

- name: distribute hadoop conf
      template: src=templates/hadoop-env.sh.j2 dest=/usr/local/hadoop-2.6.0/etc/hadoop/hadoop-env.sh

- name: distribute hadoop core-site.xml
      template: src=templates/core-site.xml.j2 dest=/usr/local/hadoop-2.6.0/etc/hadoop/core-site.xml

- name: distribute hadoop hdfs-site.xml
      template: src=templates/hdfs-site.xml.j2 dest=/usr/local/hadoop-2.6.0/etc/hadoop/hdfs-site.xml

# Setup ssh-setup for normal user and hadoop user    
    - name: deploy authorized keys
      authorized_key: user={{ item }} key="{{ ssh_public_key_content }}"
      with_items:
        - "{{ user }}"
        - "{{ hadoop_user }}"

- name: deploy ssh-keys
      copy: src={{ssh_keys_to_use}} dest=/home/{{ item }}/.ssh/
      with_items:
        - "{{ user }}"
        - "{{ hadoop_user }}"

- name: distribute ssh config
      template: src=templates/config.j2 dest=/home/{{ item }}/.ssh/config
      with_items:
        - "{{ user }}"
        - "{{ hadoop_user }}"

# Spark stuff      
    - name: download spark
      get_url:
        url=http://d3kbcqa49mib13.cloudfront.net/spark-1.2.1-bin-hadoop2.4.tgz 
        dest=/opt/ 
        sha256sum=8e618cf67b3090acf87119a96e5e2e20e51f6266c44468844c185122b492b454
 
    - name: unzip spark
      unarchive: copy=no src=/opt/spark-1.2.1-bin-hadoop2.4.tgz dest=/opt

- name: deploy slaves configuration
      template: src=templates/slaves.j2 dest=/opt/spark-1.2.1-bin-hadoop2.4/conf/slaves

- name: deploy spark-env.sh configuration
      template: src=templates/spark-env.sh.j2 dest=/opt/spark-1.2.1-bin-hadoop2.4/conf/spark-env.sh

# --------------------------------------------------
# Start the hadoop master
# --------------------------------------------------
- hosts: spark_masters
  vars_files:
    - vars/main.yml

tasks:
    - name: distribute hadoop slaves file
      template: src=templates/masters.j2 dest=/usr/local/hadoop-2.6.0/etc/hadoop/masters

- name: distribute hadoop slaves file
      template: src=templates/slaves.j2 dest=/usr/local/hadoop-2.6.0/etc/hadoop/slaves

- name: format hdfs (unless it's already been done)
      command: /usr/local/hadoop-2.6.0/bin/hadoop namenode -format creates=/usr/local/hadoop-2.6.0/ansible-format-hdfs
   
    - name: touch hfds formatted file (indicates if hdfs has been formatted)
      file: state=touch path=/usr/local/hadoop-2.6.0/ansible-format-hdfs

- name: add hadoop and spark binaries to path for hadoop user
      lineinfile:
        dest=/home/{{ hadoop_user }}/.bashrc 
        state=present insertafter=EOF 
        line="export PATH=$PATH:/usr/local/hadoop-2.6.0/bin/:/opt/spark-1.2.1-bin-hadoop2.4/bin/"
        create=true
      tags:
        - current

- name: stop hadoop (if running)
      command: /usr/local/hadoop-2.6.0/sbin/stop-dfs.sh

- name: start hadoop
      command: /usr/local/hadoop-2.6.0/sbin/start-dfs.sh

# --------------------------------------------------
# Kick of spark (making the master start the slaves)
# and configure spark-master as hadoop master
# --------------------------------------------------
- hosts: spark_masters

tasks:

- name: stop spark master (if running)
      command: /opt/spark-1.2.1-bin-hadoop2.4/sbin/stop-master.sh

- name: start spark master
      shell: SPARK_MASTER_IP="{{ ansible_hostname }}" /opt/spark-1.2.1-bin-hadoop2.4/sbin/start-master.sh

- name: stop the slaves (if running)
      shell: /opt/spark-1.2.1-bin-hadoop2.4/sbin/stop-slaves.sh

- name: start the slaves
      shell: /opt/spark-1.2.1-bin-hadoop2.4/sbin/start-slaves.sh

---
# ------------------------
# Deploy the general stuff
# ------------------------
- hosts: all
  sudo: yes

vars_files:
    - vars/main.yml
 
  pre_tasks:
    - name: Update APT cache
      apt: update_cache=yes

tasks:
    - name: install java
      apt: name=openjdk-7-jre state=present update_cache=yes

- name: disable net.ipv6.conf.all.disable_ipv6
      sysctl: name=net.ipv6.conf.all.disable_ipv6 value=1 state=present

- name: distribute host file
      template: src=templates/hosts.j2 dest=/etc/hosts

- name: deploy ssh-keys
      copy: src={{ssh_keys_to_use}} dest=/home/{{ user }}/.ssh/
      register: ssh_key

- name: distribute ssh config
      template: src=templates/config.j2 dest=/home/{{ user }}/.ssh/config
      register: ssh_debug

- name: download spark
      get_url:
        url=http://d3kbcqa49mib13.cloudfront.net/spark-1.2.1-bin-hadoop2.4.tgz 
        dest=/opt/ 
        sha256sum=8e618cf67b3090acf87119a96e5e2e20e51f6266c44468844c185122b492b454
 
    - name: unzip spark
      unarchive: copy=no src=/opt/spark-1.2.1-bin-hadoop2.4.tgz  dest=/opt

- name: deploy slaves configuration
      template: src=templates/slaves.j2 dest=/opt/spark-1.2.1-bin-hadoop2.4/conf/slaves
       
# --------------------------------------------------
# Kick of spark (making the master start the slaves)
# --------------------------------------------------
- hosts: spark_masters

tasks:

- name: stop spark master (if running)
      command: /opt/spark-1.2.1-bin-hadoop2.4/sbin/stop-master.sh

- name: start spark master
      shell: SPARK_MASTER_IP="{{ ansible_hostname }}" /opt/spark-1.2.1-bin-hadoop2.4/sbin/start-master.sh

- name: stop the slaves (if running)
      shell: /opt/spark-1.2.1-bin-hadoop2.4/sbin/stop-slaves.sh

- name: start the slaves
      shell: /opt/spark-1.2.1-bin-hadoop2.4/sbin/start-slaves.sh

Ignore Space Show notes View templates/config.j2
Host 0.0.0.0 IdentityFile ~/.ssh/{{ name_of_ssh_key }} StrictHostKeyChecking no Host localhost IdentityFile ~/.ssh/{{ name_of_ssh_key }} StrictHostKeyChecking no {% for host in groups['all'] %} Host {{ hostvars[host].ansible_hostname }} #User ubuntu IdentityFile ~/.ssh/{{ name_of_ssh_key }} StrictHostKeyChecking no {% endfor %} {% for host in groups['all'] %} Host {{ hostvars[host].ansible_hostname }} User ubuntu IdentityFile {{ ssh_key.dest }} StrictHostKeyChecking no {% endfor %}

Ignore Space Show notes View

templates/config.j2

Ignore Space Show notes View templates/core-site.xml.j2 0 → 100644
<?xml version="1.0" encoding="UTF-8"?> <?xml-stylesheet type="text/xsl" href="configuration.xsl"?> <!-- Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. See accompanying LICENSE file. --> <!-- Put site-specific property overrides in this file. --> <configuration> <property> <name>fs.defaultFS</name> <value>hdfs://spark-master:9000</value> </property> <property> <name>fs.default.name</name> <value>hdfs://spark-master:54310</value> <description>The name of the default file system. A URI whose scheme and authority determine the FileSystem implementation. The uri's scheme determines the config property (fs.SCHEME.impl) naming the FileSystem implementation class. The uri's authority is used to determine the host, port, etc. for a filesystem.</description> </property> </configuration>

Ignore Space Show notes View

templates/core-site.xml.j2 0 → 100644

<?xml version="1.0" encoding="UTF-8"?>
<?xml-stylesheet type="text/xsl" href="configuration.xsl"?>
<!--
  Licensed under the Apache License, Version 2.0 (the "License");
  you may not use this file except in compliance with the License.
  You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software
  distributed under the License is distributed on an "AS IS" BASIS,
  WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
  See the License for the specific language governing permissions and
  limitations under the License. See accompanying LICENSE file.
-->

<configuration>
  <property>
    <name>fs.defaultFS</name>
    <value>hdfs://spark-master:9000</value>
  </property>

<property>
    <name>fs.default.name</name>
    <value>hdfs://spark-master:54310</value>
    <description>The name of the default file system.  A URI whose
    scheme and authority determine the FileSystem implementation.  The
    uri's scheme determines the config property (fs.SCHEME.impl) naming
    the FileSystem implementation class.  The uri's authority is used to
    determine the host, port, etc. for a filesystem.</description>
  </property>
</configuration>

Ignore Space Show notes View templates/hadoop-env.sh.j2 0 → 100644
# Licensed to the Apache Software Foundation (ASF) under one # or more contributor license agreements. See the NOTICE file # distributed with this work for additional information # regarding copyright ownership. The ASF licenses this file # to you under the Apache License, Version 2.0 (the # "License"); you may not use this file except in compliance # with the License. You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # Set Hadoop-specific environment variables here. # The only required environment variable is JAVA_HOME. All others are # optional. When running a distributed configuration it is best to # set JAVA_HOME in this file, so that it is correctly defined on # remote nodes. # The java implementation to use. export JAVA_HOME=/usr/lib/jvm/java-7-openjdk-amd64/ # The jsvc implementation to use. Jsvc is required to run secure datanodes # that bind to privileged ports to provide authentication of data transfer # protocol. Jsvc is not required if SASL is configured for authentication of # data transfer protocol using non-privileged ports. #export JSVC_HOME=${JSVC_HOME} export HADOOP_CONF_DIR=${HADOOP_CONF_DIR:-"/etc/hadoop"} # Extra Java CLASSPATH elements. Automatically insert capacity-scheduler. for f in $HADOOP_HOME/contrib/capacity-scheduler/.jar; do if [ "$HADOOP_CLASSPATH" ]; then export HADOOP_CLASSPATH=$HADOOP_CLASSPATH:$f else export HADOOP_CLASSPATH=$f fi done # The maximum amount of heap to use, in MB. Default is 1000. #export HADOOP_HEAPSIZE= #export HADOOP_NAMENODE_INIT_HEAPSIZE="" # Extra Java runtime options. Empty by default. export HADOOP_OPTS="$HADOOP_OPTS -Djava.net.preferIPv4Stack=true" # Command specific options appended to HADOOP_OPTS when specified export HADOOP_NAMENODE_OPTS="-Dhadoop.security.logger=${HADOOP_SECURITY_LOGGER:-INFO,RFAS} -Dhdfs.audit.logger=${HDFS_AUDIT_LOGGER:-INFO,NullAppender} $HADOOP_NAMENODE_OPTS" export HADOOP_DATANODE_OPTS="-Dhadoop.security.logger=ERROR,RFAS $HADOOP_DATANODE_OPTS" export HADOOP_SECONDARYNAMENODE_OPTS="-Dhadoop.security.logger=${HADOOP_SECURITY_LOGGER:-INFO,RFAS} -Dhdfs.audit.logger=${HDFS_AUDIT_LOGGER:-INFO,NullAppender} $HADOOP_SECONDARYNAMENODE_OPTS" export HADOOP_NFS3_OPTS="$HADOOP_NFS3_OPTS" export HADOOP_PORTMAP_OPTS="-Xmx512m $HADOOP_PORTMAP_OPTS" # The following applies to multiple commands (fs, dfs, fsck, distcp etc) export HADOOP_CLIENT_OPTS="-Xmx512m $HADOOP_CLIENT_OPTS" #HADOOP_JAVA_PLATFORM_OPTS="-XX:-UsePerfData $HADOOP_JAVA_PLATFORM_OPTS" # On secure datanodes, user to run the datanode as after dropping privileges. # This MUST* be uncommented to enable secure HDFS if using privileged ports # to provide authentication of data transfer protocol. This MUST NOT be # defined if SASL is configured for authentication of data transfer protocol # using non-privileged ports. export HADOOP_SECURE_DN_USER=${HADOOP_SECURE_DN_USER} # Where log files are stored. $HADOOP_HOME/logs by default. #export HADOOP_LOG_DIR=${HADOOP_LOG_DIR}/$USER # Where log files are stored in the secure data environment. export HADOOP_SECURE_DN_LOG_DIR=${HADOOP_LOG_DIR}/${HADOOP_HDFS_USER} ### # HDFS Mover specific parameters ### # Specify the JVM options to be used when starting the HDFS Mover. # These options will be appended to the options specified as HADOOP_OPTS # and therefore may override any similar flags set in HADOOP_OPTS # # export HADOOP_MOVER_OPTS="" ### # Advanced Users Only! ### # The directory where pid files are stored. /tmp by default. # NOTE: this should be set to a directory that can only be written to by # the user that will run the hadoop daemons. Otherwise there is the # potential for a symlink attack. export HADOOP_PID_DIR=${HADOOP_PID_DIR} export HADOOP_SECURE_DN_PID_DIR=${HADOOP_PID_DIR} # A string representing this instance of hadoop. $USER by default. export HADOOP_IDENT_STRING=$USER

Ignore Space Show notes View

templates/hadoop-env.sh.j2 0 → 100644

# Licensed to the Apache Software Foundation (ASF) under one
# or more contributor license agreements.  See the NOTICE file
# distributed with this work for additional information
# regarding copyright ownership.  The ASF licenses this file
# to you under the Apache License, Version 2.0 (the
# "License"); you may not use this file except in compliance
# with the License.  You may obtain a copy of the License at
#
#     http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

# Set Hadoop-specific environment variables here.

# The only required environment variable is JAVA_HOME.  All others are
# optional.  When running a distributed configuration it is best to
# set JAVA_HOME in this file, so that it is correctly defined on
# remote nodes.

# The java implementation to use.
export JAVA_HOME=/usr/lib/jvm/java-7-openjdk-amd64/

# The jsvc implementation to use. Jsvc is required to run secure datanodes
# that bind to privileged ports to provide authentication of data transfer
# protocol.  Jsvc is not required if SASL is configured for authentication of
# data transfer protocol using non-privileged ports.
#export JSVC_HOME=${JSVC_HOME}

export HADOOP_CONF_DIR=${HADOOP_CONF_DIR:-"/etc/hadoop"}

# Extra Java CLASSPATH elements.  Automatically insert capacity-scheduler.
for f in $HADOOP_HOME/contrib/capacity-scheduler/*.jar; do
  if [ "$HADOOP_CLASSPATH" ]; then
    export HADOOP_CLASSPATH=$HADOOP_CLASSPATH:$f
  else
    export HADOOP_CLASSPATH=$f
  fi
done

# The maximum amount of heap to use, in MB. Default is 1000.
#export HADOOP_HEAPSIZE=
#export HADOOP_NAMENODE_INIT_HEAPSIZE=""

# Extra Java runtime options.  Empty by default.
export HADOOP_OPTS="$HADOOP_OPTS -Djava.net.preferIPv4Stack=true"

# Command specific options appended to HADOOP_OPTS when specified
export HADOOP_NAMENODE_OPTS="-Dhadoop.security.logger=${HADOOP_SECURITY_LOGGER:-INFO,RFAS} -Dhdfs.audit.logger=${HDFS_AUDIT_LOGGER:-INFO,NullAppender} $HADOOP_NAMENODE_OPTS"
export HADOOP_DATANODE_OPTS="-Dhadoop.security.logger=ERROR,RFAS $HADOOP_DATANODE_OPTS"

export HADOOP_SECONDARYNAMENODE_OPTS="-Dhadoop.security.logger=${HADOOP_SECURITY_LOGGER:-INFO,RFAS} -Dhdfs.audit.logger=${HDFS_AUDIT_LOGGER:-INFO,NullAppender} $HADOOP_SECONDARYNAMENODE_OPTS"

export HADOOP_NFS3_OPTS="$HADOOP_NFS3_OPTS"
export HADOOP_PORTMAP_OPTS="-Xmx512m $HADOOP_PORTMAP_OPTS"

# The following applies to multiple commands (fs, dfs, fsck, distcp etc)
export HADOOP_CLIENT_OPTS="-Xmx512m $HADOOP_CLIENT_OPTS"
#HADOOP_JAVA_PLATFORM_OPTS="-XX:-UsePerfData $HADOOP_JAVA_PLATFORM_OPTS"

# On secure datanodes, user to run the datanode as after dropping privileges.
# This **MUST** be uncommented to enable secure HDFS if using privileged ports
# to provide authentication of data transfer protocol.  This **MUST NOT** be
# defined if SASL is configured for authentication of data transfer protocol
# using non-privileged ports.
export HADOOP_SECURE_DN_USER=${HADOOP_SECURE_DN_USER}

# Where log files are stored.  $HADOOP_HOME/logs by default.
#export HADOOP_LOG_DIR=${HADOOP_LOG_DIR}/$USER

# Where log files are stored in the secure data environment.
export HADOOP_SECURE_DN_LOG_DIR=${HADOOP_LOG_DIR}/${HADOOP_HDFS_USER}

###
# HDFS Mover specific parameters
###
# Specify the JVM options to be used when starting the HDFS Mover.
# These options will be appended to the options specified as HADOOP_OPTS
# and therefore may override any similar flags set in HADOOP_OPTS
#
# export HADOOP_MOVER_OPTS=""

###
# Advanced Users Only!
###

# The directory where pid files are stored. /tmp by default.
# NOTE: this should be set to a directory that can only be written to by 
#       the user that will run the hadoop daemons.  Otherwise there is the
#       potential for a symlink attack.
export HADOOP_PID_DIR=${HADOOP_PID_DIR}
export HADOOP_SECURE_DN_PID_DIR=${HADOOP_PID_DIR}

# A string representing this instance of hadoop. $USER by default.
export HADOOP_IDENT_STRING=$USER

Ignore Space Show notes View templates/hdfs-site.xml.j2 0 → 100644
<?xml version="1.0" encoding="UTF-8"?> <?xml-stylesheet type="text/xsl" href="configuration.xsl"?> <!-- Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. See accompanying LICENSE file. --> <!-- Put site-specific property overrides in this file. --> <configuration> <property> <name>dfs.replication</name> <value>3</value> </property> </configuration>

Ignore Space Show notes View

templates/hdfs-site.xml.j2 0 → 100644

Ignore Space Show notes View templates/masters.j2 0 → 100644
spark-master

Ignore Space Show notes View templates/spark-env.sh.j2
#!/usr/bin/env bash # This file is sourced when running various Spark programs. # Copy it as spark-env.sh and edit that to configure Spark for your site. # Options read when launching programs locally with # ./bin/run-example or ./bin/spark-submit # - HADOOP_CONF_DIR, to point Spark towards Hadoop configuration files # - SPARK_LOCAL_IP, to set the IP address Spark binds to on this node # - SPARK_PUBLIC_DNS, to set the public dns name of the driver program # - SPARK_CLASSPATH, default classpath entries to append # Options read by executors and drivers running inside the cluster # - SPARK_LOCAL_IP, to set the IP address Spark binds to on this node # - SPARK_PUBLIC_DNS, to set the public DNS name of the driver program # - SPARK_CLASSPATH, default classpath entries to append # - SPARK_LOCAL_DIRS, storage directories to use on this node for shuffle and RDD data # - MESOS_NATIVE_LIBRARY, to point to your libmesos.so if you use Mesos # Options read in YARN client mode # - HADOOP_CONF_DIR, to point Spark towards Hadoop configuration files export HADOOP_CONF_DIR="/usr/local/hadoop-2.6.0/etc/hadoop/" # - SPARK_EXECUTOR_INSTANCES, Number of workers to start (Default: 2) # - SPARK_EXECUTOR_CORES, Number of cores for the workers (Default: 1). # - SPARK_EXECUTOR_MEMORY, Memory per Worker (e.g. 1000M, 2G) (Default: 1G) # - SPARK_DRIVER_MEMORY, Memory for Master (e.g. 1000M, 2G) (Default: 512 Mb) # - SPARK_YARN_APP_NAME, The name of your application (Default: Spark) # - SPARK_YARN_QUEUE, The hadoop queue to use for allocation requests (Default: ‘default’) # - SPARK_YARN_DIST_FILES, Comma separated list of files to be distributed with the job. # - SPARK_YARN_DIST_ARCHIVES, Comma separated list of archives to be distributed with the job. # Options for the daemons used in the standalone deploy mode # - SPARK_MASTER_IP, to bind the master to a different IP address or hostname #SPARK_MASTER_IP= groups['sparkmasters'](0) # - SPARK_MASTER_PORT / SPARK_MASTER_WEBUI_PORT, to use non-default ports for the master # - SPARK_MASTER_OPTS, to set config properties only for the master (e.g. "-Dx=y") # - SPARK_WORKER_CORES, to set the number of cores to use on this machine # - SPARK_WORKER_MEMORY, to set how much total memory workers have to give executors (e.g. 1000m, 2g) # - SPARK_WORKER_PORT / SPARK_WORKER_WEBUI_PORT, to use non-default ports for the worker # - SPARK_WORKER_INSTANCES, to set the number of worker processes per node # - SPARK_WORKER_DIR, to set the working directory of worker processes # - SPARK_WORKER_OPTS, to set config properties only for the worker (e.g. "-Dx=y") # - SPARK_HISTORY_OPTS, to set config properties only for the history server (e.g. "-Dx=y") # - SPARK_DAEMON_JAVA_OPTS, to set config properties for all daemons (e.g. "-Dx=y") # - SPARK_PUBLIC_DNS, to set the public dns name of the master or workers # Generic options for the daemons used in the standalone deploy mode # - SPARK_CONF_DIR Alternate conf dir. (Default: ${SPARK_HOME}/conf) # - SPARK_LOG_DIR Where log files are stored. (Default: ${SPARK_HOME}/logs) # - SPARK_PID_DIR Where the pid file is stored. (Default: /tmp) # - SPARK_IDENT_STRING A string representing this instance of spark. (Default: $USER) # - SPARK_NICENESS The scheduling priority for daemons. (Default: 0) #!/usr/bin/env bash # This file is sourced when running various Spark programs. # Copy it as spark-env.sh and edit that to configure Spark for your site. # Options read when launching programs locally with # ./bin/run-example or ./bin/spark-submit # - HADOOP_CONF_DIR, to point Spark towards Hadoop configuration files # - SPARK_LOCAL_IP, to set the IP address Spark binds to on this node # - SPARK_PUBLIC_DNS, to set the public dns name of the driver program # - SPARK_CLASSPATH, default classpath entries to append # Options read by executors and drivers running inside the cluster # - SPARK_LOCAL_IP, to set the IP address Spark binds to on this node # - SPARK_PUBLIC_DNS, to set the public DNS name of the driver program # - SPARK_CLASSPATH, default classpath entries to append # - SPARK_LOCAL_DIRS, storage directories to use on this node for shuffle and RDD data # - MESOS_NATIVE_LIBRARY, to point to your libmesos.so if you use Mesos # Options read in YARN client mode # - HADOOP_CONF_DIR, to point Spark towards Hadoop configuration files # - SPARK_EXECUTOR_INSTANCES, Number of workers to start (Default: 2) # - SPARK_EXECUTOR_CORES, Number of cores for the workers (Default: 1). # - SPARK_EXECUTOR_MEMORY, Memory per Worker (e.g. 1000M, 2G) (Default: 1G) # - SPARK_DRIVER_MEMORY, Memory for Master (e.g. 1000M, 2G) (Default: 512 Mb) # - SPARK_YARN_APP_NAME, The name of your application (Default: Spark) # - SPARK_YARN_QUEUE, The hadoop queue to use for allocation requests (Default: ‘default’) # - SPARK_YARN_DIST_FILES, Comma separated list of files to be distributed with the job. # - SPARK_YARN_DIST_ARCHIVES, Comma separated list of archives to be distributed with the job. # Options for the daemons used in the standalone deploy mode # - SPARK_MASTER_IP, to bind the master to a different IP address or hostname #SPARK_MASTER_IP= groups['sparkmasters'](0) # - SPARK_MASTER_PORT / SPARK_MASTER_WEBUI_PORT, to use non-default ports for the master # - SPARK_MASTER_OPTS, to set config properties only for the master (e.g. "-Dx=y") # - SPARK_WORKER_CORES, to set the number of cores to use on this machine # - SPARK_WORKER_MEMORY, to set how much total memory workers have to give executors (e.g. 1000m, 2g) # - SPARK_WORKER_PORT / SPARK_WORKER_WEBUI_PORT, to use non-default ports for the worker SPARK_WORKER_PORT=9999 # - SPARK_WORKER_INSTANCES, to set the number of worker processes per node # - SPARK_WORKER_DIR, to set the working directory of worker processes # - SPARK_WORKER_OPTS, to set config properties only for the worker (e.g. "-Dx=y") # - SPARK_HISTORY_OPTS, to set config properties only for the history server (e.g. "-Dx=y") # - SPARK_DAEMON_JAVA_OPTS, to set config properties for all daemons (e.g. "-Dx=y") # - SPARK_PUBLIC_DNS, to set the public dns name of the master or workers # Generic options for the daemons used in the standalone deploy mode # - SPARK_CONF_DIR Alternate conf dir. (Default: ${SPARK_HOME}/conf) # - SPARK_LOG_DIR Where log files are stored. (Default: ${SPARK_HOME}/logs) # - SPARK_PID_DIR Where the pid file is stored. (Default: /tmp) # - SPARK_IDENT_STRING A string representing this instance of spark. (Default: $USER) # - SPARK_NICENESS The scheduling priority for daemons. (Default: 0)

Ignore Space Show notes View

templates/spark-env.sh.j2

#!/usr/bin/env bash

# This file is sourced when running various Spark programs.
# Copy it as spark-env.sh and edit that to configure Spark for your site.

# Options read when launching programs locally with 
# ./bin/run-example or ./bin/spark-submit
# - HADOOP_CONF_DIR, to point Spark towards Hadoop configuration files
# - SPARK_LOCAL_IP, to set the IP address Spark binds to on this node
# - SPARK_PUBLIC_DNS, to set the public dns name of the driver program
# - SPARK_CLASSPATH, default classpath entries to append

# Options read by executors and drivers running inside the cluster
# - SPARK_LOCAL_IP, to set the IP address Spark binds to on this node
# - SPARK_PUBLIC_DNS, to set the public DNS name of the driver program
# - SPARK_CLASSPATH, default classpath entries to append
# - SPARK_LOCAL_DIRS, storage directories to use on this node for shuffle and RDD data
# - MESOS_NATIVE_LIBRARY, to point to your libmesos.so if you use Mesos

# Options read in YARN client mode
# - HADOOP_CONF_DIR, to point Spark towards Hadoop configuration files
export HADOOP_CONF_DIR="/usr/local/hadoop-2.6.0/etc/hadoop/"
# - SPARK_EXECUTOR_INSTANCES, Number of workers to start (Default: 2)
# - SPARK_EXECUTOR_CORES, Number of cores for the workers (Default: 1).
# - SPARK_EXECUTOR_MEMORY, Memory per Worker (e.g. 1000M, 2G) (Default: 1G)
# - SPARK_DRIVER_MEMORY, Memory for Master (e.g. 1000M, 2G) (Default: 512 Mb)
# - SPARK_YARN_APP_NAME, The name of your application (Default: Spark)
# - SPARK_YARN_QUEUE, The hadoop queue to use for allocation requests (Default: ‘default’)
# - SPARK_YARN_DIST_FILES, Comma separated list of files to be distributed with the job.
# - SPARK_YARN_DIST_ARCHIVES, Comma separated list of archives to be distributed with the job.

# Options for the daemons used in the standalone deploy mode
# - SPARK_MASTER_IP, to bind the master to a different IP address or hostname
#SPARK_MASTER_IP= groups['sparkmasters'](0)
# - SPARK_MASTER_PORT / SPARK_MASTER_WEBUI_PORT, to use non-default ports for the master
# - SPARK_MASTER_OPTS, to set config properties only for the master (e.g. "-Dx=y")
# - SPARK_WORKER_CORES, to set the number of cores to use on this machine
# - SPARK_WORKER_MEMORY, to set how much total memory workers have to give executors (e.g. 1000m, 2g)
# - SPARK_WORKER_PORT / SPARK_WORKER_WEBUI_PORT, to use non-default ports for the worker
# - SPARK_WORKER_INSTANCES, to set the number of worker processes per node
# - SPARK_WORKER_DIR, to set the working directory of worker processes
# - SPARK_WORKER_OPTS, to set config properties only for the worker (e.g. "-Dx=y")
# - SPARK_HISTORY_OPTS, to set config properties only for the history server (e.g. "-Dx=y")
# - SPARK_DAEMON_JAVA_OPTS, to set config properties for all daemons (e.g. "-Dx=y")
# - SPARK_PUBLIC_DNS, to set the public dns name of the master or workers

# Generic options for the daemons used in the standalone deploy mode
# - SPARK_CONF_DIR      Alternate conf dir. (Default: ${SPARK_HOME}/conf)
# - SPARK_LOG_DIR       Where log files are stored.  (Default: ${SPARK_HOME}/logs)
# - SPARK_PID_DIR       Where the pid file is stored. (Default: /tmp)
# - SPARK_IDENT_STRING  A string representing this instance of spark. (Default: $USER)
# - SPARK_NICENESS      The scheduling priority for daemons. (Default: 0)

#!/usr/bin/env bash

# This file is sourced when running various Spark programs.
# Copy it as spark-env.sh and edit that to configure Spark for your site.

# Options read in YARN client mode
# - HADOOP_CONF_DIR, to point Spark towards Hadoop configuration files
# - SPARK_EXECUTOR_INSTANCES, Number of workers to start (Default: 2)
# - SPARK_EXECUTOR_CORES, Number of cores for the workers (Default: 1).
# - SPARK_EXECUTOR_MEMORY, Memory per Worker (e.g. 1000M, 2G) (Default: 1G)
# - SPARK_DRIVER_MEMORY, Memory for Master (e.g. 1000M, 2G) (Default: 512 Mb)
# - SPARK_YARN_APP_NAME, The name of your application (Default: Spark)
# - SPARK_YARN_QUEUE, The hadoop queue to use for allocation requests (Default: ‘default’)
# - SPARK_YARN_DIST_FILES, Comma separated list of files to be distributed with the job.
# - SPARK_YARN_DIST_ARCHIVES, Comma separated list of archives to be distributed with the job.

# Options for the daemons used in the standalone deploy mode
# - SPARK_MASTER_IP, to bind the master to a different IP address or hostname
#SPARK_MASTER_IP= groups['sparkmasters'](0)
# - SPARK_MASTER_PORT / SPARK_MASTER_WEBUI_PORT, to use non-default ports for the master
# - SPARK_MASTER_OPTS, to set config properties only for the master (e.g. "-Dx=y")
# - SPARK_WORKER_CORES, to set the number of cores to use on this machine
# - SPARK_WORKER_MEMORY, to set how much total memory workers have to give executors (e.g. 1000m, 2g)
# - SPARK_WORKER_PORT / SPARK_WORKER_WEBUI_PORT, to use non-default ports for the worker
SPARK_WORKER_PORT=9999
# - SPARK_WORKER_INSTANCES, to set the number of worker processes per node
# - SPARK_WORKER_DIR, to set the working directory of worker processes
# - SPARK_WORKER_OPTS, to set config properties only for the worker (e.g. "-Dx=y")
# - SPARK_HISTORY_OPTS, to set config properties only for the history server (e.g. "-Dx=y")
# - SPARK_DAEMON_JAVA_OPTS, to set config properties for all daemons (e.g. "-Dx=y")
# - SPARK_PUBLIC_DNS, to set the public dns name of the master or workers

Ignore Space Show notes View vars/main.yml
--- # Set to absent to take down machines instance_state: present # Number of workers to spawn in the cluster nbr_of_slaves: 3 # The name of the key pair you will use to log in # as set in OpenStack (see the OpenStack security dashboard) key_name: cloud_key # Id of the network to run in (can be found in the OpenStack # dashboard) network_id: b045a7de-5aa4-43a2-86ff-049fd1ba1884 # This is the ssh-key which will be distritbuted # across the cluster. It' important that this key # does NOT use password protection ssh_keys_to_use: files/cloud.key ssh_public_key: files/cloud.key.pub # This is a ugly hack - but you have to give the # file name of the ssh-key your referencing above name_of_ssh_key: cloud.key # Name of the user used to install everything # on the remote systems user: ubuntu # Name of hadoop user hadoop_user: ubuntu --- # Set to absent to take down machines instance_state: present # Number of workers to spawn in the cluster nbr_of_slaves: 3 # The name of the key pair you will use to log in # as set in OpenStack (see the OpenStack security dashboard) key_name: cloud_key # Id of the network to run in (can be found in the OpenStack # dashboard) network_id: b045a7de-5aa4-43a2-86ff-049fd1ba1884 # This is the ssh-key which will be distritbuted # across the cluster. It' important that this key # does NOT use password protection ssh_keys_to_use: files/cloud.key # Name of the user used to install everything # on the remote systems user: ubuntu

Ignore Space Show notes View

vars/main.yml

Show line notes below