Added merged Spark tools image

nigel.stanger / docker-analytics

Browse code Added merged Spark tools image master spark3
1 parent 810de14 commit 0526903157ed01f5a809ee7fdca794812d582119 Nigel Stanger authored on 21 Sep 2021

Patch

Showing 7 changed files

Ignore Space Show notes View spark-pyspark-kafka/Dockerfile 0 → 100644
# Merging all the Spark-related tools into one image saves a LOT of space # over having separate images for each, especially given they all share # the same underlying infrastructure. It’s still just as easy to set up # and run the various services via docker-compose. FROM python:3.6-slim-buster # I originally used python:3.6-alpine but dropped it after getting tied # in knots with Python dependencies around numpy. Pip under Alpine builds # all modules from source. You can try to be clever and install the bigger # modules like numpy via apk, but the problem with numpy is that the Python # version most likely will install a different version of numpy from what # the base Alpine image supplies pre-built, so you end up building numpy # from source regardless, which takes forever (well, 10 minutes ;). I tried # to figure out a combination that worked without success. # # Going to a later version of Python would probably resolve this, but that # would require moving to Spark 3 (2.4 doesn't support 3.8 or later), and # we don’t have time to test that at the moment. Which is why this is still # using Python 3.6: if it ain't broke... ENV \ # download mirrors SPARK_MIRROR="https://dlcdn.apache.org" \ GRAPHFRAMES_MIRROR="https://repos.spark-packages.org" \ KAFKA_MIRROR="https://archive.apache.org/dist" \ # Spark SPARK_VERSION="2.4.8" \ HADOOP_VERSION="2.7" \ SPARK_INSTALL="/usr/local" \ SPARK_HOSTNAME="localhost" \ SPARK_MASTER_PORT="7077" \ SPARK_MASTER_WEBUI_PORT="8080" \ # graphframes GRAPHFRAMES_VERSION="0.8.1-spark2.4-s_2.11" \ # Kafka KAFKA_VERSION="2.2.2" \ SCALA_VERSION="2.11" \ # PySpark PYSPARK_KERNEL_NAME="PySpark" \ PYSPARK_PYTHON="/usr/local/bin/python" \ KERNELS_TEMPLATE_PATH="/tmp" \ KERNELS_DIR_PATH="/root/.local/share/jupyter/kernels" \ # Miscellaneous BUILD_PACKAGES="wget gnupg software-properties-common" \ PURGE_PACKAGES="$BUILD_PACKAGES readline-common libreadline7 netbase libgdbm6" # These environment variables need to be defined separately because you # can't reference prior environment variables in the same ENV block. ENV \ # Spark SPARK_HOME="$SPARK_INSTALL/spark" \ SPARK_MASTER="spark://$SPARK_HOSTNAME:$SPARK_MASTER_PORT" \ # Kafka KAFKA_HOME="$KAFKA_INSTALL/kafka" \ # PySpark PYSPARK_DRIVER_PYTHON="$PYSPARK_PYTHON" # More PySpark stuff ENV PYTHONPATH="$SPARK_HOME/python:$SPARK_HOME/python/lib/py4j-0.10.7-src.zip" \ PYSPARK_SUBMIT_ARGS="--master $SPARK_MASTER pyspark-shell --py-files $SPARK_HOME/jars/graphframes-$GRAPHFRAMES_VERSION.jar" \ PYTHONSTARTUP="$SPARK_HOME/python/pyspark/shell.py" # Coalescing the RUNs saves quite a significant amount of space # (about 10% just for Spark alone). # <https://www.dajobe.org/blog/2015/04/18/making-debian-docker-images-smaller/> RUN apt-get update && \ # Miscellaneous infrastructure apt-get install -y --no-install-recommends tini $BUILD_PACKAGES ; \ # Install AdoptOpenJDK 8 # <https://stackoverflow.com/a/59436618> wget -qO - https://adoptopenjdk.jfrog.io/adoptopenjdk/api/gpg/key/public \| apt-key add - && \ add-apt-repository --yes https://adoptopenjdk.jfrog.io/adoptopenjdk/deb/ && \ apt-get update && apt-get install -y adoptopenjdk-8-hotspot ; \ # Download, install, and symlink spark cd $SPARK_INSTALL && \ wget -q --show-progress --progress=bar:force:noscroll $SPARK_MIRROR/spark/spark-$SPARK_VERSION/spark-$SPARK_VERSION-bin-hadoop$HADOOP_VERSION.tgz 2>&1 && \ tar xzf spark-$SPARK_VERSION-bin-hadoop$HADOOP_VERSION.tgz && \ ln -s spark-$SPARK_VERSION-bin-hadoop$HADOOP_VERSION spark && \ rm -f spark-$SPARK_VERSION-bin-hadoop$HADOOP_VERSION.tgz ; \ # Download and install graphframes cd $SPARK_INSTALL/spark/jars && \ wget -q --show-progress --progress=bar:force:noscroll $GRAPHFRAMES_MIRROR/graphframes/graphframes/$GRAPHFRAMES_VERSION/graphframes-$GRAPHFRAMES_VERSION.jar ; \ # Download, install, and symlink Kafka cd $SPARK_INSTALL && \ wget -q --show-progress --progress=bar:force:noscroll $KAFKA_MIRROR/kafka/$KAFKA_VERSION/kafka_$SCALA_VERSION-$KAFKA_VERSION.tgz 2>&1 && \ tar xzf kafka_$SCALA_VERSION-$KAFKA_VERSION.tgz && \ ln -s kafka_$SCALA_VERSION-$KAFKA_VERSION kafka && \ rm -f kafka_$SCALA_VERSION-$KAFKA_VERSION.tgz ; \ # Install required modules for PySpark and install kernel pip install --upgrade pip && \ pip install jupyter kafka-python graphframes && \ # Tornado 6 breaks sparkmonitor # tornado==5.1 \ # sparkmonitor \ ipython kernel install --name "$PYSPARK_KERNEL_NAME" ; \ # Clear out all the unnecessary crap apt-get remove -y --purge $PURGE_PACKAGES && \ apt -y autoremove && \ apt-get clean -y && \ rm -rf /var/lib/apt/lists/* && \ rm -rf /tmp/* && \ rm -rf /var/cache/* && \ rm -rf /root/.cache # Startup scripts COPY start-master.sh start-worker.sh start-kafka.sh /usr/local/bin/ # Configuration files COPY spark-defaults.conf $SPARK_HOME/conf COPY server.properties $KAFKA_HOME/config/ # PySpark kernel, based on the template from # <https://github.com/Anchormen/pyspark-jupyter-kernels>. COPY kernel.json $KERNELS_DIR_PATH/$PYSPARK_KERNEL_NAME/ # Spark doesn't seem to respond directly to SIGTERM as the exit status is # for SIGKILL (137), after a pause. Presumably docker-compose down times out. # Using tini gives immediate exit with status 143 (SIGTERM). ENTRYPOINT ["/usr/bin/tini", "--"] # Override this in your docker-compose.yml or at the command line. # Spark master (default): /usr/local/bin/start-master.sh # Spark worker: /usr/local/bin/start-worker.sh # PySpark: /usr/local/bin/jupyter notebook --ip=0.0.0.0 --port=8888 --allow-root # Kafka: /usr/local/bin/start-kafka.sh CMD ["/usr/local/bin/start-master.sh"] # debugging # CMD ["bash"]

Ignore Space Show notes View

spark-pyspark-kafka/Dockerfile 0 → 100644

# Merging all the Spark-related tools into one image saves a LOT of space
# over having separate images for each, especially given they all share
# the same underlying infrastructure. It’s still just as easy to set up
# and run the various services via docker-compose.

FROM python:3.6-slim-buster
# I originally used python:3.6-alpine but dropped it after getting tied
# in knots with Python dependencies around numpy. Pip under Alpine builds
# all modules from source. You can try to be clever and install the bigger
# modules like numpy via apk, but the problem with numpy is that the Python
# version most likely will install a different version of numpy from what
# the base Alpine image supplies pre-built, so you end up building numpy
# from source regardless, which takes forever (well, 10 minutes ;). I tried
# to figure out a combination that worked without success.
#
# Going to a later version of Python would probably resolve this, but that
# would require moving to Spark 3 (2.4 doesn't support 3.8 or later), and
# we don’t have time to test that at the moment. Which is why this is still
# using Python 3.6: if it ain't broke...

ENV \
    # download mirrors
    SPARK_MIRROR="https://dlcdn.apache.org" \
    GRAPHFRAMES_MIRROR="https://repos.spark-packages.org" \
    KAFKA_MIRROR="https://archive.apache.org/dist" \
    # Spark
    SPARK_VERSION="2.4.8" \
    HADOOP_VERSION="2.7" \
    SPARK_INSTALL="/usr/local" \
    SPARK_HOSTNAME="localhost" \
    SPARK_MASTER_PORT="7077" \
    SPARK_MASTER_WEBUI_PORT="8080" \
    # graphframes
    GRAPHFRAMES_VERSION="0.8.1-spark2.4-s_2.11" \
    # Kafka
    KAFKA_VERSION="2.2.2" \
    SCALA_VERSION="2.11" \
    # PySpark
    PYSPARK_KERNEL_NAME="PySpark" \
    PYSPARK_PYTHON="/usr/local/bin/python" \
    KERNELS_TEMPLATE_PATH="/tmp" \
    KERNELS_DIR_PATH="/root/.local/share/jupyter/kernels" \
    # Miscellaneous
    BUILD_PACKAGES="wget gnupg software-properties-common" \
    PURGE_PACKAGES="$BUILD_PACKAGES readline-common libreadline7 netbase libgdbm6"

# These environment variables need to be defined separately because you
# can't reference prior environment variables in the same ENV block.
ENV \
    # Spark
    SPARK_HOME="$SPARK_INSTALL/spark" \
    SPARK_MASTER="spark://$SPARK_HOSTNAME:$SPARK_MASTER_PORT" \
    # Kafka
    KAFKA_HOME="$KAFKA_INSTALL/kafka" \
    # PySpark
    PYSPARK_DRIVER_PYTHON="$PYSPARK_PYTHON"

# More PySpark stuff
ENV PYTHONPATH="$SPARK_HOME/python:$SPARK_HOME/python/lib/py4j-0.10.7-src.zip" \
    PYSPARK_SUBMIT_ARGS="--master $SPARK_MASTER pyspark-shell  --py-files $SPARK_HOME/jars/graphframes-$GRAPHFRAMES_VERSION.jar" \
    PYTHONSTARTUP="$SPARK_HOME/python/pyspark/shell.py"

# Coalescing the RUNs saves quite a significant amount of space
# (about 10% just for Spark alone).
# <https://www.dajobe.org/blog/2015/04/18/making-debian-docker-images-smaller/>
RUN apt-get update && \
    # Miscellaneous infrastructure
    apt-get install -y --no-install-recommends tini $BUILD_PACKAGES ; \
    # Install AdoptOpenJDK 8
    # <https://stackoverflow.com/a/59436618>
    wget -qO - https://adoptopenjdk.jfrog.io/adoptopenjdk/api/gpg/key/public | apt-key add - && \
    add-apt-repository --yes https://adoptopenjdk.jfrog.io/adoptopenjdk/deb/ && \
    apt-get update && apt-get install -y adoptopenjdk-8-hotspot ; \
    # Download, install, and symlink spark
    cd $SPARK_INSTALL && \
    wget -q --show-progress --progress=bar:force:noscroll $SPARK_MIRROR/spark/spark-$SPARK_VERSION/spark-$SPARK_VERSION-bin-hadoop$HADOOP_VERSION.tgz 2>&1 && \
    tar xzf spark-$SPARK_VERSION-bin-hadoop$HADOOP_VERSION.tgz && \
    ln -s spark-$SPARK_VERSION-bin-hadoop$HADOOP_VERSION spark && \
    rm -f spark-$SPARK_VERSION-bin-hadoop$HADOOP_VERSION.tgz ; \
    # Download and install graphframes
    cd $SPARK_INSTALL/spark/jars && \
    wget -q --show-progress --progress=bar:force:noscroll $GRAPHFRAMES_MIRROR/graphframes/graphframes/$GRAPHFRAMES_VERSION/graphframes-$GRAPHFRAMES_VERSION.jar ; \
    # Download, install, and symlink Kafka
    cd $SPARK_INSTALL && \
    wget -q --show-progress --progress=bar:force:noscroll $KAFKA_MIRROR/kafka/$KAFKA_VERSION/kafka_$SCALA_VERSION-$KAFKA_VERSION.tgz 2>&1 && \
    tar xzf kafka_$SCALA_VERSION-$KAFKA_VERSION.tgz && \
    ln -s kafka_$SCALA_VERSION-$KAFKA_VERSION kafka && \
    rm -f kafka_$SCALA_VERSION-$KAFKA_VERSION.tgz ; \
    # Install required modules for PySpark and install kernel
    pip install --upgrade pip && \
    pip install jupyter kafka-python graphframes && \
        # Tornado 6 breaks sparkmonitor
        # tornado==5.1 \
        # sparkmonitor \
    ipython kernel install --name "$PYSPARK_KERNEL_NAME" ; \
    # Clear out all the unnecessary crap
    apt-get remove -y --purge $PURGE_PACKAGES && \
    apt -y autoremove && \
    apt-get clean -y && \
    rm -rf /var/lib/apt/lists/* && \
    rm -rf /tmp/* && \
    rm -rf /var/cache/* && \
    rm -rf /root/.cache

# Startup scripts
COPY start-master.sh start-worker.sh start-kafka.sh /usr/local/bin/

# Configuration files
COPY spark-defaults.conf $SPARK_HOME/conf
COPY server.properties $KAFKA_HOME/config/

# PySpark kernel, based on the template from
# <https://github.com/Anchormen/pyspark-jupyter-kernels>.
COPY kernel.json $KERNELS_DIR_PATH/$PYSPARK_KERNEL_NAME/

# Spark doesn't seem to respond directly to SIGTERM as the exit status is
# for SIGKILL (137), after a pause. Presumably docker-compose down times out.
# Using tini gives immediate exit with status 143 (SIGTERM).
ENTRYPOINT ["/usr/bin/tini", "--"]

# Override this in your docker-compose.yml or at the command line.
# Spark master (default): /usr/local/bin/start-master.sh
# Spark worker: /usr/local/bin/start-worker.sh
# PySpark: /usr/local/bin/jupyter notebook --ip=0.0.0.0 --port=8888 --allow-root
# Kafka: /usr/local/bin/start-kafka.sh
CMD ["/usr/local/bin/start-master.sh"]
# debugging
# CMD ["bash"]

Ignore Space Show notes View spark-pyspark-kafka/kernel.json 0 → 100644
{ "display_name": "PySpark", "language": "python", "argv": [ "/usr/local/bin/python", "-m", "ipykernel", "-f", "{connection_file}" ] }

Ignore Space Show notes View spark-pyspark-kafka/server.properties 0 → 100644
# Licensed to the Apache Software Foundation (ASF) under one or more # contributor license agreements. See the NOTICE file distributed with # this work for additional information regarding copyright ownership. # The ASF licenses this file to You under the Apache License, Version 2.0 # (the "License"); you may not use this file except in compliance with # the License. You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # see kafka.server.KafkaConfig for additional details and defaults ############################# Server Basics ############################# # The id of the broker. This must be set to a unique integer for each broker. broker.id=0 ############################# Socket Server Settings ############################# # The address the socket server listens on. It will get the value returned from # java.net.InetAddress.getCanonicalHostName() if not configured. # FORMAT: # listeners = listener_name://host_name:port # EXAMPLE: # listeners = PLAINTEXT://your.host.name:9092 listeners=PLAINTEXT://kafka:9092 # Hostname and port the broker will advertise to producers and consumers. If not set, # it uses the value for "listeners" if configured. Otherwise, it will use the value # returned from java.net.InetAddress.getCanonicalHostName(). #advertised.listeners=PLAINTEXT://your.host.name:9092 # Maps listener names to security protocols, the default is for them to be the same. See the config documentation for more details #listener.security.protocol.map=PLAINTEXT:PLAINTEXT,SSL:SSL,SASL_PLAINTEXT:SASL_PLAINTEXT,SASL_SSL:SASL_SSL # The number of threads that the server uses for receiving requests from the network and sending responses to the network num.network.threads=3 # The number of threads that the server uses for processing requests, which may include disk I/O num.io.threads=8 # The send buffer (SO_SNDBUF) used by the socket server socket.send.buffer.bytes=102400 # The receive buffer (SO_RCVBUF) used by the socket server socket.receive.buffer.bytes=102400 # The maximum size of a request that the socket server will accept (protection against OOM) socket.request.max.bytes=104857600 ############################# Log Basics ############################# # A comma separated list of directories under which to store log files log.dirs=/tmp/kafka-logs # The default number of log partitions per topic. More partitions allow greater # parallelism for consumption, but this will also result in more files across # the brokers. num.partitions=1 # The number of threads per data directory to be used for log recovery at startup and flushing at shutdown. # This value is recommended to be increased for installations with data dirs located in RAID array. num.recovery.threads.per.data.dir=1 ############################# Internal Topic Settings ############################# # The replication factor for the group metadata internal topics "__consumer_offsets" and "__transaction_state" # For anything other than development testing, a value greater than 1 is recommended for to ensure availability such as 3. offsets.topic.replication.factor=1 transaction.state.log.replication.factor=1 transaction.state.log.min.isr=1 ############################# Log Flush Policy ############################# # Messages are immediately written to the filesystem but by default we only fsync() to sync # the OS cache lazily. The following configurations control the flush of data to disk. # There are a few important trade-offs here: # 1. Durability: Unflushed data may be lost if you are not using replication. # 2. Latency: Very large flush intervals may lead to latency spikes when the flush does occur as there will be a lot of data to flush. # 3. Throughput: The flush is generally the most expensive operation, and a small flush interval may lead to excessive seeks. # The settings below allow one to configure the flush policy to flush data after a period of time or # every N messages (or both). This can be done globally and overridden on a per-topic basis. # The number of messages to accept before forcing a flush of data to disk #log.flush.interval.messages=10000 # The maximum amount of time a message can sit in a log before we force a flush #log.flush.interval.ms=1000 ############################# Log Retention Policy ############################# # The following configurations control the disposal of log segments. The policy can # be set to delete segments after a period of time, or after a given size has accumulated. # A segment will be deleted whenever either of these criteria are met. Deletion always happens # from the end of the log. # The minimum age of a log file to be eligible for deletion due to age log.retention.hours=168 # A size-based retention policy for logs. Segments are pruned from the log unless the remaining # segments drop below log.retention.bytes. Functions independently of log.retention.hours. #log.retention.bytes=1073741824 # The maximum size of a log segment file. When this size is reached a new log segment will be created. log.segment.bytes=1073741824 # The interval at which log segments are checked to see if they can be deleted according # to the retention policies log.retention.check.interval.ms=300000 ############################# Zookeeper ############################# # Zookeeper connection string (see zookeeper docs for details). # This is a comma separated host:port pairs, each corresponding to a zk # server. e.g. "127.0.0.1:3000,127.0.0.1:3001,127.0.0.1:3002". # You can also append an optional chroot string to the urls to specify the # root directory for all kafka znodes. zookeeper.connect=localhost:2181 # Timeout in ms for connecting to zookeeper zookeeper.connection.timeout.ms=6000 ############################# Group Coordinator Settings ############################# # The following configuration specifies the time, in milliseconds, that the GroupCoordinator will delay the initial consumer rebalance. # The rebalance will be further delayed by the value of group.initial.rebalance.delay.ms as new members join the group, up to a maximum of max.poll.interval.ms. # The default value for this is 3 seconds. # We override this to 0 here as it makes for a better out-of-the-box experience for development and testing. # However, in production environments the default value of 3 seconds is more suitable as this will help to avoid unnecessary, and potentially expensive, rebalances during application startup. group.initial.rebalance.delay.ms=0

Ignore Space Show notes View

spark-pyspark-kafka/server.properties 0 → 100644

# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements.  See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License.  You may obtain a copy of the License at
#
#    http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

# see kafka.server.KafkaConfig for additional details and defaults

############################# Server Basics #############################

# The id of the broker. This must be set to a unique integer for each broker.
broker.id=0

############################# Socket Server Settings #############################

# The address the socket server listens on. It will get the value returned from 
# java.net.InetAddress.getCanonicalHostName() if not configured.
#   FORMAT:
#     listeners = listener_name://host_name:port
#   EXAMPLE:
#     listeners = PLAINTEXT://your.host.name:9092
listeners=PLAINTEXT://kafka:9092

# Hostname and port the broker will advertise to producers and consumers. If not set, 
# it uses the value for "listeners" if configured.  Otherwise, it will use the value
# returned from java.net.InetAddress.getCanonicalHostName().
#advertised.listeners=PLAINTEXT://your.host.name:9092

# Maps listener names to security protocols, the default is for them to be the same. See the config documentation for more details
#listener.security.protocol.map=PLAINTEXT:PLAINTEXT,SSL:SSL,SASL_PLAINTEXT:SASL_PLAINTEXT,SASL_SSL:SASL_SSL

# The number of threads that the server uses for receiving requests from the network and sending responses to the network
num.network.threads=3

# The number of threads that the server uses for processing requests, which may include disk I/O
num.io.threads=8

# The send buffer (SO_SNDBUF) used by the socket server
socket.send.buffer.bytes=102400

# The receive buffer (SO_RCVBUF) used by the socket server
socket.receive.buffer.bytes=102400

# The maximum size of a request that the socket server will accept (protection against OOM)
socket.request.max.bytes=104857600

############################# Log Basics #############################

# A comma separated list of directories under which to store log files
log.dirs=/tmp/kafka-logs

# The default number of log partitions per topic. More partitions allow greater
# parallelism for consumption, but this will also result in more files across
# the brokers.
num.partitions=1

# The number of threads per data directory to be used for log recovery at startup and flushing at shutdown.
# This value is recommended to be increased for installations with data dirs located in RAID array.
num.recovery.threads.per.data.dir=1

############################# Internal Topic Settings  #############################
# The replication factor for the group metadata internal topics "__consumer_offsets" and "__transaction_state"
# For anything other than development testing, a value greater than 1 is recommended for to ensure availability such as 3.
offsets.topic.replication.factor=1
transaction.state.log.replication.factor=1
transaction.state.log.min.isr=1

############################# Log Flush Policy #############################

# Messages are immediately written to the filesystem but by default we only fsync() to sync
# the OS cache lazily. The following configurations control the flush of data to disk.
# There are a few important trade-offs here:
#    1. Durability: Unflushed data may be lost if you are not using replication.
#    2. Latency: Very large flush intervals may lead to latency spikes when the flush does occur as there will be a lot of data to flush.
#    3. Throughput: The flush is generally the most expensive operation, and a small flush interval may lead to excessive seeks.
# The settings below allow one to configure the flush policy to flush data after a period of time or
# every N messages (or both). This can be done globally and overridden on a per-topic basis.

# The number of messages to accept before forcing a flush of data to disk
#log.flush.interval.messages=10000

# The maximum amount of time a message can sit in a log before we force a flush
#log.flush.interval.ms=1000

############################# Log Retention Policy #############################

# The following configurations control the disposal of log segments. The policy can
# be set to delete segments after a period of time, or after a given size has accumulated.
# A segment will be deleted whenever *either* of these criteria are met. Deletion always happens
# from the end of the log.

# The minimum age of a log file to be eligible for deletion due to age
log.retention.hours=168

# A size-based retention policy for logs. Segments are pruned from the log unless the remaining
# segments drop below log.retention.bytes. Functions independently of log.retention.hours.
#log.retention.bytes=1073741824

# The maximum size of a log segment file. When this size is reached a new log segment will be created.
log.segment.bytes=1073741824

# The interval at which log segments are checked to see if they can be deleted according
# to the retention policies
log.retention.check.interval.ms=300000

############################# Zookeeper #############################

# Zookeeper connection string (see zookeeper docs for details).
# This is a comma separated host:port pairs, each corresponding to a zk
# server. e.g. "127.0.0.1:3000,127.0.0.1:3001,127.0.0.1:3002".
# You can also append an optional chroot string to the urls to specify the
# root directory for all kafka znodes.
zookeeper.connect=localhost:2181

# Timeout in ms for connecting to zookeeper
zookeeper.connection.timeout.ms=6000

############################# Group Coordinator Settings #############################

# The following configuration specifies the time, in milliseconds, that the GroupCoordinator will delay the initial consumer rebalance.
# The rebalance will be further delayed by the value of group.initial.rebalance.delay.ms as new members join the group, up to a maximum of max.poll.interval.ms.
# The default value for this is 3 seconds.
# We override this to 0 here as it makes for a better out-of-the-box experience for development and testing.
# However, in production environments the default value of 3 seconds is more suitable as this will help to avoid unnecessary, and potentially expensive, rebalances during application startup.
group.initial.rebalance.delay.ms=0

Ignore Space Show notes View spark-pyspark-kafka/spark-defaults.conf 0 → 100644
# # Licensed to the Apache Software Foundation (ASF) under one or more # contributor license agreements. See the NOTICE file distributed with # this work for additional information regarding copyright ownership. # The ASF licenses this file to You under the Apache License, Version 2.0 # (the "License"); you may not use this file except in compliance with # the License. You may obtain a copy of the License at # # http://www.apache.org/licenses/LICENSE-2.0 # # Unless required by applicable law or agreed to in writing, software # distributed under the License is distributed on an "AS IS" BASIS, # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. # See the License for the specific language governing permissions and # limitations under the License. # # Default system properties included when running spark-submit. # This is useful for setting default environmental settings. # Example: # spark.master spark://master:7077 # spark.eventLog.enabled true # spark.eventLog.dir hdfs://namenode:8021/directory # spark.serializer org.apache.spark.serializer.KryoSerializer # spark.driver.memory 5g # spark.executor.extraJavaOptions -XX:+PrintGCDetails -Dkey=value -Dnumbers="one two three" spark.deploy.defaultCores 2 spark.cores.max 2 # spark.executor.memory 2g # spark.driver.memory 4g

Ignore Space Show notes View

spark-pyspark-kafka/spark-defaults.conf 0 → 100644

#
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements.  See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License.  You may obtain a copy of the License at
#
#    http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#

# Default system properties included when running spark-submit.
# This is useful for setting default environmental settings.

# Example:
# spark.master                     spark://master:7077
# spark.eventLog.enabled           true
# spark.eventLog.dir               hdfs://namenode:8021/directory
# spark.serializer                 org.apache.spark.serializer.KryoSerializer
# spark.driver.memory              5g
# spark.executor.extraJavaOptions  -XX:+PrintGCDetails -Dkey=value -Dnumbers="one two three"
spark.deploy.defaultCores   2
spark.cores.max             2
# spark.executor.memory       2g
# spark.driver.memory         4g

Ignore Space Show notes View spark-pyspark-kafka/start-kafka.sh 0 → 100755
#!/bin/sh $KAFKA_HOME/bin/zookeeper-server-start.sh $KAFKA_HOME/config/zookeeper.properties & $KAFKA_HOME/bin/kafka-server-start.sh $KAFKA_HOME/config/server.properties

Ignore Space Show notes View spark-pyspark-kafka/start-master.sh 0 → 100755
#!/bin/sh $SPARK_HOME/bin/spark-class org.apache.spark.deploy.master.Master \ --host $SPARK_HOSTNAME \ --port $SPARK_MASTER_PORT \ --webui-port $SPARK_MASTER_WEBUI_PORT \ --properties-file $SPARK_HOME/conf/spark-defaults.conf

Ignore Space Show notes View spark-pyspark-kafka/start-worker.sh 0 → 100755
#!/bin/sh $SPARK_HOME/bin/spark-class org.apache.spark.deploy.worker.Worker \ --webui-port $SPARK_WORKER_WEBUI_PORT \ --properties-file $SPARK_HOME/conf/spark-defaults.conf \ $SPARK_MASTER

Show line notes below