提交运行 spark 任务到 kubernetes

零、spark on k8s 示意图在这里插入图片描述

一、目标

运行 spark 任务到 kubernetes,spark on kubernetes。
参考 https://spark.apache.org/docs/3.2.0/running-on-kubernetes.html

二、步骤

0.前置步骤

1. 部署 docker 环境。
2. 部署 k8s 集群。
3. 准备 spark,spark-3.2.0-bin-hadoop2.7.tgz。

1.制作镜像

cd /root/spark-3.2.0-bin-hadoop2.7/bin/
ll docker-image-tool.sh
ll /root/spark-3.2.0-bin-hadoop2.7/kubernetes/dockerfiles/spark/Dockerfile
ll /root/spark-3.2.0-bin-hadoop2.7/kubernetes/dockerfiles/spark/entrypoint.sh
spark 自带脚本,制作镜像,网络超时
./docker-image-tool.sh -r harbor.dev-XXX.com/k8s/sre -t v4.0.0 build

更换原始镜像,重新制作镜像
docker pull harbor.dev-XXX.com/library/centos7-oracle-jdk1.8:1.0
cd /root/spark-3.2.0-bin-hadoop2.7/kubernetes/dockerfiles/spark/
docker build --network host --no-cache -t harbor.dev-XXX.com/k8s/sre/spark:v4.0.0 -f ./Dockerfile .
docker image ls

推送到镜像仓库
docker push harbor.dev-XXX.com/k8s/sre/spark:v4.0.0

2.提交 spark 任务到 k8s

cd /root/spark-3.2.0-bin-hadoop2.7/bin/; ls -lrt
./spark-submit 
    --master k8s://https://K8S_API_IP:6443 
    --deploy-mode cluster 
    --name spark-pi 
    --class org.apache.spark.examples.SparkPi 
    --conf spark.executor.instances=5 
    --conf spark.kubernetes.container.image=harbor.dev-XXX.com/k8s/sre/spark:v4.0.0 
    --conf spark.kubernetes.authenticate.driver.serviceAccountName=spark 
    local:///opt/spark/examples/jars/spark-examples_2.12-3.2.0.jar

kubectl get pods -A
kubectl logs -f spark-pi-f0ae047d619e134a-driver

查看任务状态
./spark-submit --status spark:spark-pi-f0ae047d619e134a-driver --master k8s://https://K8S_API_IP:6443

三、问题及解决

3.1 问题1

## 问题
Exception in thread "main" org.apache.spark.SparkException: Could not load KUBERNETES classes. This copy of Spark may not have been compiled with KUBERNETES support.

https://issues.apache.org/jira/browse/SPARK-33340

## 解决
--conf spark.kubernetes.container.image 指定正确的 spark 镜像。

3.2 问题2

## 问题
尝试不自己 build spark 的镜像源,尝试使用 hub.docker 上的镜像源。
https://hub.docker.com/r/newfrontdocker/spark
https://hub.docker.com/r/bitnami/spark

harbor.dev-XXX.com/k8s/sre/spark:v1.0.0 -- bitnami/spark:latest
harbor.dev-XXX.com/k8s/sre/spark:v2.0.0 -- newfrontdocker/spark:v3.0.1-j14

1. bitnami/spark 运行日志
    /opt/bitnami/scripts/spark/entrypoint.sh: line 32: exec: driver: not found
    镜像中没有 build 对应的程序。这个镜像不能用。

2. newfrontdocker/spark 运行日志
WARN DependencyUtils: Local jar /opt/spark/examples/jars/*.jar does not exist, skipping
Error: Failed to load class org.apache.spark.examples.SparkPi.

从 https://hub.docker.com/r/bitnami/spark 上的镜像说明中,没找到 examples 下 spark 的版本号,而 spark-submit 提交命令时,需要指定 spark 的版本号。这个镜像不能用。

## 解决
不使用 hub.docker 上的镜像源,自己 build spark 的镜像包。

3.3 问题3

## 问题
Caused by: io.fabric8.kubernetes.client.KubernetesClientException: Failure executing: GET at: https://kubernetes.default.svc/api/v1/namespaces/default/pods/spark-pi-a275a77d6134ec82-driver. Message: Forbidden!Configured service account doesn't have access. Service account may have been revoked. pods "spark-pi-a275a77d6134ec82-driver" is forbidden: User "system:serviceaccount:default:default" cannot get resource "pods" in API group "" in the namespace "default".

## 解决
kubectl create serviceaccount spark
kubectl create clusterrolebinding spark-role --clusterrole=edit --serviceaccount=default:spark --namespace=default

spark-submit 提交任务时,增加 --conf spark.kubernetes.authenticate.driver.serviceAccountName=spark

四、其他记录

4.1 社区链接
https://github.com/bitnami/bitnami-docker-spark/issues/56
https://issues.apache.org/jira/browse/SPARK-33340

4.2 修改 spark 自带的 Dockerfile,如下截图
在这里插入图片描述
4.3 spark on k8s 涉及到的文件
在这里插入图片描述
4.4 spark 的其他运行方式

## local
./spark-submit 
  --class org.apache.spark.examples.SparkPi 
  --master local[4] 
  local:///home/biadmin/XXXhadoop_client/spark-3.2.0-bin-hadoop2.7/examples/jars/spark-examples_2.12-3.2.0.jar 100

## yarn
./spark-submit 
  --class org.apache.spark.examples.SparkPi 
    --master yarn 
    --deploy-mode cluster 
    --driver-memory 4g 
    --executor-memory 2g 
    --executor-cores 1 
    --queue lx_etl 
    --jars /home/biadmin/XXXhadoop_client/spark-3.2.0-bin-hadoop2.7/examples/jars/spark-examples_2.12-3.2.0.jar 
    local:///home/biadmin/XXXhadoop_client/spark-3.2.0-bin-hadoop2.7/examples/jars/spark-examples_2.12-3.2.0.jar 100

4.5 编译 spark

cd /var/lib/docker/bigdata/workspace/spark
export MAVEN_OPTS="-Xss1g -Xmx12g -XX:ReservedCodeCacheSize=4g"

for i in {1..3}; do date; sleep 1; done
for i in {1..10}; do ./dev/make-distribution.sh --name custom-spark --pip --r --tgz -Psparkr -Phive -Phive-thriftserver -Pmesos -Pyarn -Pkubernetes; sleep 1; done
for i in {1..10}; do date; ./dev/make-distribution.sh --name custom-spark --pip --tgz -Phive -Phive-thriftserver -Pmesos -Pyarn -Pkubernetes; done

下载超时,手动上传的 jar
/your/local/.m2/repository/org/scalanlp/breeze_2.12/1.2
/your/local/.m2/repository/org/apache/parquet/parquet-jackson/1.12.2
/your/local/.m2/repository/org/apache/hive/hive-llap-common/2.3.9

4.6 执行 spark-submit 的日志

21/11/28 09:09:42 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
21/11/28 09:09:43 INFO SparkKubernetesClientFactory: Auto-configuring K8S client using current context from users K8S config file
21/11/28 09:09:44 INFO KerberosConfDriverFeatureStep: You have not specified a krb5.conf file locally or via a ConfigMap. Make sure that you have the krb5.conf locally on the driver image.
21/11/28 09:09:45 INFO LoggingPodStatusWatcherImpl: State changed, new state: 
	 pod name: spark-pi-cf8ea77d6417381c-driver
	 namespace: default
	 labels: spark-app-selector -> spark-35a1624f76584815ac052dc9138cfbbf, spark-role -> driver
	 pod uid: 872ecd0c-355a-42d8-8bf2-9464abba84e6
	 creation time: 2021-11-28T01:09:44Z
	 service account name: spark
	 volumes: spark-local-dir-1, spark-conf-volume-driver, kube-api-access-66bjs
	 node name: k8s-node1
	 start time: 2021-11-28T01:09:44Z
	 phase: Pending
	 container status: 
		 container name: spark-kubernetes-driver
		 container image: harbor.dev-XXX.com/k8s/sre/spark:v4.0.0
		 container state: waiting
		 pending reason: ContainerCreating
21/11/28 09:09:45 INFO LoggingPodStatusWatcherImpl: State changed, new state: 
	 pod name: spark-pi-cf8ea77d6417381c-driver
	 namespace: default
	 labels: spark-app-selector -> spark-35a1624f76584815ac052dc9138cfbbf, spark-role -> driver
	 pod uid: 872ecd0c-355a-42d8-8bf2-9464abba84e6
	 creation time: 2021-11-28T01:09:44Z
	 service account name: spark
	 volumes: spark-local-dir-1, spark-conf-volume-driver, kube-api-access-66bjs
	 node name: k8s-node1
	 start time: 2021-11-28T01:09:44Z
	 phase: Pending
	 container status: 
		 container name: spark-kubernetes-driver
		 container image: harbor.dev-XXX.com/k8s/sre/spark:v4.0.0
		 container state: waiting
		 pending reason: ContainerCreating
21/11/28 09:09:45 INFO LoggingPodStatusWatcherImpl: Waiting for application spark-pi with submission ID default:spark-pi-cf8ea77d6417381c-driver to finish...
21/11/28 09:09:46 INFO LoggingPodStatusWatcherImpl: Application status for spark-35a1624f76584815ac052dc9138cfbbf (phase: Pending)
21/11/28 09:09:46 INFO LoggingPodStatusWatcherImpl: State changed, new state: 
	 pod name: spark-pi-cf8ea77d6417381c-driver
	 namespace: default
	 labels: spark-app-selector -> spark-35a1624f76584815ac052dc9138cfbbf, spark-role -> driver
	 pod uid: 872ecd0c-355a-42d8-8bf2-9464abba84e6
	 creation time: 2021-11-28T01:09:44Z
	 service account name: spark
	 volumes: spark-local-dir-1, spark-conf-volume-driver, kube-api-access-66bjs
	 node name: k8s-node1
	 start time: 2021-11-28T01:09:44Z
	 phase: Pending
	 container status: 
		 container name: spark-kubernetes-driver
		 container image: harbor.dev-XXX.com/k8s/sre/spark:v4.0.0
		 container state: waiting
		 pending reason: ContainerCreating
21/11/28 09:09:47 INFO LoggingPodStatusWatcherImpl: Application status for spark-35a1624f76584815ac052dc9138cfbbf (phase: Pending)
21/11/28 09:09:47 INFO LoggingPodStatusWatcherImpl: State changed, new state: 
	 pod name: spark-pi-cf8ea77d6417381c-driver
	 namespace: default
	 labels: spark-app-selector -> spark-35a1624f76584815ac052dc9138cfbbf, spark-role -> driver
	 pod uid: 872ecd0c-355a-42d8-8bf2-9464abba84e6
	 creation time: 2021-11-28T01:09:44Z
	 service account name: spark
	 volumes: spark-local-dir-1, spark-conf-volume-driver, kube-api-access-66bjs
	 node name: k8s-node1
	 start time: 2021-11-28T01:09:44Z
	 phase: Running
	 container status: 
		 container name: spark-kubernetes-driver
		 container image: harbor.dev-XXX.com/k8s/sre/spark:v4.0.0
		 container state: running
		 container started at: 2021-11-28T01:09:46Z
21/11/28 09:09:48 INFO LoggingPodStatusWatcherImpl: Application status for spark-35a1624f76584815ac052dc9138cfbbf (phase: Running)
21/11/28 09:09:49 INFO LoggingPodStatusWatcherImpl: Application status for spark-35a1624f76584815ac052dc9138cfbbf (phase: Running)
21/11/28 09:09:50 INFO LoggingPodStatusWatcherImpl: Application status for spark-35a1624f76584815ac052dc9138cfbbf (phase: Running)
21/11/28 09:09:51 INFO LoggingPodStatusWatcherImpl: Application status for spark-35a1624f76584815ac052dc9138cfbbf (phase: Running)
21/11/28 09:09:52 INFO LoggingPodStatusWatcherImpl: Application status for spark-35a1624f76584815ac052dc9138cfbbf (phase: Running)
21/11/28 09:09:53 INFO LoggingPodStatusWatcherImpl: Application status for spark-35a1624f76584815ac052dc9138cfbbf (phase: Running)
21/11/28 09:09:54 INFO LoggingPodStatusWatcherImpl: Application status for spark-35a1624f76584815ac052dc9138cfbbf (phase: Running)
21/11/28 09:09:55 INFO LoggingPodStatusWatcherImpl: Application status for spark-35a1624f76584815ac052dc9138cfbbf (phase: Running)
21/11/28 09:09:56 INFO LoggingPodStatusWatcherImpl: Application status for spark-35a1624f76584815ac052dc9138cfbbf (phase: Running)
21/11/28 09:09:57 INFO LoggingPodStatusWatcherImpl: Application status for spark-35a1624f76584815ac052dc9138cfbbf (phase: Running)
21/11/28 09:09:58 INFO LoggingPodStatusWatcherImpl: Application status for spark-35a1624f76584815ac052dc9138cfbbf (phase: Running)
21/11/28 09:09:59 INFO LoggingPodStatusWatcherImpl: Application status for spark-35a1624f76584815ac052dc9138cfbbf (phase: Running)
21/11/28 09:10:00 INFO LoggingPodStatusWatcherImpl: Application status for spark-35a1624f76584815ac052dc9138cfbbf (phase: Running)
21/11/28 09:10:00 INFO LoggingPodStatusWatcherImpl: State changed, new state: 
	 pod name: spark-pi-cf8ea77d6417381c-driver
	 namespace: default
	 labels: spark-app-selector -> spark-35a1624f76584815ac052dc9138cfbbf, spark-role -> driver
	 pod uid: 872ecd0c-355a-42d8-8bf2-9464abba84e6
	 creation time: 2021-11-28T01:09:44Z
	 service account name: spark
	 volumes: spark-local-dir-1, spark-conf-volume-driver, kube-api-access-66bjs
	 node name: k8s-node1
	 start time: 2021-11-28T01:09:44Z
	 phase: Succeeded
	 container status: 
		 container name: spark-kubernetes-driver
		 container image: harbor.dev-XXX.com/k8s/sre/spark:v4.0.0
		 container state: terminated
		 container started at: 2021-11-28T01:09:46Z
		 container finished at: 2021-11-28T01:09:59Z
		 exit code: 0
		 termination reason: Completed
21/11/28 09:10:00 INFO LoggingPodStatusWatcherImpl: Application status for spark-35a1624f76584815ac052dc9138cfbbf (phase: Succeeded)
21/11/28 09:10:00 INFO LoggingPodStatusWatcherImpl: Container final statuses:


	 container name: spark-kubernetes-driver
	 container image: harbor.dev-XXX.com/k8s/sre/spark:v4.0.0
	 container state: terminated
	 container started at: 2021-11-28T01:09:46Z
	 container finished at: 2021-11-28T01:09:59Z
	 exit code: 0
	 termination reason: Completed
21/11/28 09:10:00 INFO LoggingPodStatusWatcherImpl: Application spark-pi with submission ID default:spark-pi-cf8ea77d6417381c-driver finished
21/11/28 09:10:00 INFO ShutdownHookManager: Shutdown hook called
21/11/28 09:10:00 INFO ShutdownHookManager: Deleting directory /tmp/spark-fc84bdad-8fca-4773-8a9d-c81d7e260eea

4.7 执行 kubectl logs -f spark-pi-cf8ea77d6417381c-driver 的日志

++ id -u
+ myuid=185
++ id -g
+ mygid=0
+ set +e
++ getent passwd 185
+ uidentry=
+ set -e
+ '[' -z '' ']'
+ '[' -w /etc/passwd ']'
+ echo '185:x:185:0:anonymous uid:/opt/spark:/bin/false'
+ SPARK_CLASSPATH=':/opt/spark/jars/*'
+ env
+ grep SPARK_JAVA_OPT_
+ sort -t_ -k4 -n
+ sed 's/[^=]*=(.*)/1/g'
+ readarray -t SPARK_EXECUTOR_JAVA_OPTS
+ '[' -n '' ']'
+ '[' -z ']'
+ '[' -z ']'
+ '[' -n '' ']'
+ '[' -z ']'
+ '[' -z x ']'
+ SPARK_CLASSPATH='/opt/spark/conf::/opt/spark/jars/*'
+ case "$1" in
+ shift 1
+ CMD=("$SPARK_HOME/bin/spark-submit" --conf "spark.driver.bindAddress=$SPARK_DRIVER_BIND_ADDRESS" --deploy-mode client "$@")
+ exec /usr/bin/tini -s -- /opt/spark/bin/spark-submit --conf spark.driver.bindAddress=10.244.36.108 --deploy-mode client --properties-file /opt/spark/conf/spark.properties --class org.apache.spark.examples.SparkPi local:///opt/spark/examples/jars/spark-examples_2.12-3.2.0.jar
21/11/28 09:09:48 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
Using Spark's default log4j profile: org/apache/spark/log4j-defaults.properties
21/11/28 09:09:48 INFO SparkContext: Running Spark version 3.2.0
21/11/28 09:09:48 INFO ResourceUtils: ==============================================================
21/11/28 09:09:48 INFO ResourceUtils: No custom resources configured for spark.driver.
21/11/28 09:09:48 INFO ResourceUtils: ==============================================================
21/11/28 09:09:48 INFO SparkContext: Submitted application: Spark Pi
21/11/28 09:09:49 INFO ResourceProfile: Default ResourceProfile created, executor resources: Map(cores -> name: cores, amount: 1, script: , vendor: , memory -> name: memory, amount: 1024, script: , vendor: , offHeap -> name: offHeap, amount: 0, script: , vendor: ), task resources: Map(cpus -> name: cpus, amount: 1.0)
21/11/28 09:09:49 INFO ResourceProfile: Limiting resource is cpus at 1 tasks per executor
21/11/28 09:09:49 INFO ResourceProfileManager: Added ResourceProfile id: 0
21/11/28 09:09:49 INFO SecurityManager: Changing view acls to: 185,root
21/11/28 09:09:49 INFO SecurityManager: Changing modify acls to: 185,root
21/11/28 09:09:49 INFO SecurityManager: Changing view acls groups to: 
21/11/28 09:09:49 INFO SecurityManager: Changing modify acls groups to: 
21/11/28 09:09:49 INFO SecurityManager: SecurityManager: authentication disabled; ui acls disabled; users  with view permissions: Set(185, root); groups with view permissions: Set(); users  with modify permissions: Set(185, root); groups with modify permissions: Set()
21/11/28 09:09:49 INFO Utils: Successfully started service 'sparkDriver' on port 7078.
21/11/28 09:09:49 INFO SparkEnv: Registering MapOutputTracker
21/11/28 09:09:49 INFO SparkEnv: Registering BlockManagerMaster
21/11/28 09:09:49 INFO BlockManagerMasterEndpoint: Using org.apache.spark.storage.DefaultTopologyMapper for getting topology information
21/11/28 09:09:49 INFO BlockManagerMasterEndpoint: BlockManagerMasterEndpoint up
21/11/28 09:09:49 INFO SparkEnv: Registering BlockManagerMasterHeartbeat
21/11/28 09:09:49 INFO DiskBlockManager: Created local directory at /var/data/spark-6ecc5286-05a5-478b-ad0c-143361509542/blockmgr-77aa2d40-7383-486c-9345-0c67d595f390
21/11/28 09:09:49 INFO MemoryStore: MemoryStore started with capacity 366.3 MiB
21/11/28 09:09:49 INFO SparkEnv: Registering OutputCommitCoordinator
21/11/28 09:09:49 INFO Utils: Successfully started service 'SparkUI' on port 4040.
21/11/28 09:09:50 INFO SparkUI: Bound SparkUI to 0.0.0.0, and started at http://spark-pi-cf8ea77d6417381c-driver-svc.default.svc:4040
21/11/28 09:09:50 INFO SparkContext: Added JAR local:///opt/spark/examples/jars/spark-examples_2.12-3.2.0.jar at file:/opt/spark/examples/jars/spark-examples_2.12-3.2.0.jar with timestamp 1638061788937
21/11/28 09:09:50 INFO SparkContext: The JAR local:///opt/spark/examples/jars/spark-examples_2.12-3.2.0.jar at file:/opt/spark/examples/jars/spark-examples_2.12-3.2.0.jar has been added already. Overwriting of added jar is not supported in the current version.
21/11/28 09:09:50 INFO SparkKubernetesClientFactory: Auto-configuring K8S client using current context from users K8S config file
21/11/28 09:09:51 INFO ExecutorPodsAllocator: Going to request 5 executors from Kubernetes for ResourceProfile Id: 0, target: 5, known: 0, sharedSlotFromPendingPods: 2147483647.
21/11/28 09:09:51 INFO BasicExecutorFeatureStep: Decommissioning not enabled, skipping shutdown script
21/11/28 09:09:51 INFO Utils: Successfully started service 'org.apache.spark.network.netty.NettyBlockTransferService' on port 7079.
21/11/28 09:09:51 INFO NettyBlockTransferService: Server created on spark-pi-cf8ea77d6417381c-driver-svc.default.svc:7079
21/11/28 09:09:51 INFO BlockManager: Using org.apache.spark.storage.RandomBlockReplicationPolicy for block replication policy
21/11/28 09:09:51 INFO BlockManagerMaster: Registering BlockManager BlockManagerId(driver, spark-pi-cf8ea77d6417381c-driver-svc.default.svc, 7079, None)
21/11/28 09:09:51 INFO BlockManagerMasterEndpoint: Registering block manager spark-pi-cf8ea77d6417381c-driver-svc.default.svc:7079 with 366.3 MiB RAM, BlockManagerId(driver, spark-pi-cf8ea77d6417381c-driver-svc.default.svc, 7079, None)
21/11/28 09:09:51 INFO BlockManagerMaster: Registered BlockManager BlockManagerId(driver, spark-pi-cf8ea77d6417381c-driver-svc.default.svc, 7079, None)
21/11/28 09:09:51 INFO BlockManager: Initialized BlockManager: BlockManagerId(driver, spark-pi-cf8ea77d6417381c-driver-svc.default.svc, 7079, None)
21/11/28 09:09:51 INFO BasicExecutorFeatureStep: Decommissioning not enabled, skipping shutdown script
21/11/28 09:09:51 INFO BasicExecutorFeatureStep: Decommissioning not enabled, skipping shutdown script
21/11/28 09:09:51 INFO BasicExecutorFeatureStep: Decommissioning not enabled, skipping shutdown script
21/11/28 09:09:51 INFO BasicExecutorFeatureStep: Decommissioning not enabled, skipping shutdown script
21/11/28 09:09:56 INFO KubernetesClusterSchedulerBackend$KubernetesDriverEndpoint: Registered executor NettyRpcEndpointRef(spark-client://Executor) (10.244.36.124:37132) with ID 3,  ResourceProfileId 0
21/11/28 09:09:56 INFO KubernetesClusterSchedulerBackend$KubernetesDriverEndpoint: Registered executor NettyRpcEndpointRef(spark-client://Executor) (10.244.36.122:58850) with ID 1,  ResourceProfileId 0
21/11/28 09:09:56 INFO KubernetesClusterSchedulerBackend$KubernetesDriverEndpoint: Registered executor NettyRpcEndpointRef(spark-client://Executor) (10.244.36.118:44710) with ID 4,  ResourceProfileId 0
21/11/28 09:09:56 INFO BlockManagerMasterEndpoint: Registering block manager 10.244.36.124:45414 with 408.9 MiB RAM, BlockManagerId(3, 10.244.36.124, 45414, None)
21/11/28 09:09:56 INFO BlockManagerMasterEndpoint: Registering block manager 10.244.36.122:44299 with 408.9 MiB RAM, BlockManagerId(1, 10.244.36.122, 44299, None)
21/11/28 09:09:56 INFO BlockManagerMasterEndpoint: Registering block manager 10.244.36.118:41553 with 408.9 MiB RAM, BlockManagerId(4, 10.244.36.118, 41553, None)
21/11/28 09:09:56 INFO KubernetesClusterSchedulerBackend$KubernetesDriverEndpoint: Registered executor NettyRpcEndpointRef(spark-client://Executor) (10.244.36.117:35256) with ID 2,  ResourceProfileId 0
21/11/28 09:09:56 INFO KubernetesClusterSchedulerBackend: SchedulerBackend is ready for scheduling beginning after reached minRegisteredResourcesRatio: 0.8
21/11/28 09:09:56 INFO BlockManagerMasterEndpoint: Registering block manager 10.244.36.117:45961 with 408.9 MiB RAM, BlockManagerId(2, 10.244.36.117, 45961, None)
21/11/28 09:09:57 INFO KubernetesClusterSchedulerBackend$KubernetesDriverEndpoint: Registered executor NettyRpcEndpointRef(spark-client://Executor) (10.244.36.125:52560) with ID 5,  ResourceProfileId 0
21/11/28 09:09:57 INFO BlockManagerMasterEndpoint: Registering block manager 10.244.36.125:33322 with 408.9 MiB RAM, BlockManagerId(5, 10.244.36.125, 33322, None)
21/11/28 09:09:57 INFO SparkContext: Starting job: reduce at SparkPi.scala:38
21/11/28 09:09:57 INFO DAGScheduler: Got job 0 (reduce at SparkPi.scala:38) with 2 output partitions
21/11/28 09:09:57 INFO DAGScheduler: Final stage: ResultStage 0 (reduce at SparkPi.scala:38)
21/11/28 09:09:57 INFO DAGScheduler: Parents of final stage: List()
21/11/28 09:09:57 INFO DAGScheduler: Missing parents: List()
21/11/28 09:09:57 INFO DAGScheduler: Submitting ResultStage 0 (MapPartitionsRDD[1] at map at SparkPi.scala:34), which has no missing parents
21/11/28 09:09:57 INFO MemoryStore: Block broadcast_0 stored as values in memory (estimated size 4.0 KiB, free 366.3 MiB)
21/11/28 09:09:57 INFO MemoryStore: Block broadcast_0_piece0 stored as bytes in memory (estimated size 2.3 KiB, free 366.3 MiB)
21/11/28 09:09:57 INFO BlockManagerInfo: Added broadcast_0_piece0 in memory on spark-pi-cf8ea77d6417381c-driver-svc.default.svc:7079 (size: 2.3 KiB, free: 366.3 MiB)
21/11/28 09:09:57 INFO SparkContext: Created broadcast 0 from broadcast at DAGScheduler.scala:1427
21/11/28 09:09:57 INFO DAGScheduler: Submitting 2 missing tasks from ResultStage 0 (MapPartitionsRDD[1] at map at SparkPi.scala:34) (first 15 tasks are for partitions Vector(0, 1))
21/11/28 09:09:57 INFO TaskSchedulerImpl: Adding task set 0.0 with 2 tasks resource profile 0
21/11/28 09:09:57 INFO TaskSetManager: Starting task 0.0 in stage 0.0 (TID 0) (10.244.36.124, executor 3, partition 0, PROCESS_LOCAL, 4597 bytes) taskResourceAssignments Map()
21/11/28 09:09:57 INFO TaskSetManager: Starting task 1.0 in stage 0.0 (TID 1) (10.244.36.125, executor 5, partition 1, PROCESS_LOCAL, 4597 bytes) taskResourceAssignments Map()
21/11/28 09:09:58 INFO BlockManagerInfo: Added broadcast_0_piece0 in memory on 10.244.36.125:33322 (size: 2.3 KiB, free: 408.9 MiB)
21/11/28 09:09:58 INFO BlockManagerInfo: Added broadcast_0_piece0 in memory on 10.244.36.124:45414 (size: 2.3 KiB, free: 408.9 MiB)
21/11/28 09:09:58 INFO TaskSetManager: Finished task 0.0 in stage 0.0 (TID 0) in 966 ms on 10.244.36.124 (executor 3) (1/2)
21/11/28 09:09:58 INFO TaskSetManager: Finished task 1.0 in stage 0.0 (TID 1) in 945 ms on 10.244.36.125 (executor 5) (2/2)
21/11/28 09:09:58 INFO TaskSchedulerImpl: Removed TaskSet 0.0, whose tasks have all completed, from pool 
21/11/28 09:09:58 INFO DAGScheduler: ResultStage 0 (reduce at SparkPi.scala:38) finished in 1.188 s
21/11/28 09:09:58 INFO DAGScheduler: Job 0 is finished. Cancelling potential speculative or zombie tasks for this job
21/11/28 09:09:58 INFO TaskSchedulerImpl: Killing all running tasks in stage 0: Stage finished
21/11/28 09:09:58 INFO DAGScheduler: Job 0 finished: reduce at SparkPi.scala:38, took 1.243353 s
Pi is roughly 3.1410157050785252
21/11/28 09:09:58 INFO SparkUI: Stopped Spark web UI at http://spark-pi-cf8ea77d6417381c-driver-svc.default.svc:4040
21/11/28 09:09:58 INFO KubernetesClusterSchedulerBackend: Shutting down all executors
21/11/28 09:09:58 INFO KubernetesClusterSchedulerBackend$KubernetesDriverEndpoint: Asking each executor to shut down
21/11/28 09:09:58 WARN ExecutorPodsWatchSnapshotSource: Kubernetes client has been closed.
21/11/28 09:09:59 INFO MapOutputTrackerMasterEndpoint: MapOutputTrackerMasterEndpoint stopped!
21/11/28 09:09:59 INFO MemoryStore: MemoryStore cleared
21/11/28 09:09:59 INFO BlockManager: BlockManager stopped
21/11/28 09:09:59 INFO BlockManagerMaster: BlockManagerMaster stopped
21/11/28 09:09:59 INFO OutputCommitCoordinator$OutputCommitCoordinatorEndpoint: OutputCommitCoordinator stopped!
21/11/28 09:09:59 INFO SparkContext: Successfully stopped SparkContext
21/11/28 09:09:59 INFO ShutdownHookManager: Shutdown hook called
21/11/28 09:09:59 INFO ShutdownHookManager: Deleting directory /tmp/spark-f2b506ed-ab8f-41b8-bd36-eb90576f93ce
21/11/28 09:09:59 INFO ShutdownHookManager: Deleting directory /var/data/spark-6ecc5286-05a5-478b-ad0c-143361509542/spark-a5cd7f8a-883c-4943-843e-0a18d912e15a

本图文内容来源于网友网络收集整理提供,作为学习参考使用,版权属于原作者。
THE END
分享
二维码
< <上一篇
 
下一篇>>