Flink on kubernetes
零、flink on k8s 示意图
https://nightlies.apache.org/flink/flink-docs-release-1.11/ops/deployment/native_kubernetes.html
一、目标
运行 flink 任务到 kubernetes，flink on kubernetes。
参考 https://nightlies.apache.org/flink/flink-docs-release-1.14/docs/deployment/resource-providers/native_kubernetes/
二、步骤
0.前置步骤
1. 部署 docker 环境。
2. 部署 k8s 集群。
3. 准备 flink，flink-1.14.0-bin-scala_2.12.tgz。
1.制作镜像
Flink 的镜像有专门维护，更新非常实时。
地址，https://hub.docker.com/_/flink
docker pull flink:latest
docker tag flink:latest harbor.dev-XXX.com/k8s/sre/bigdata/flink:v2.0.0
docker push harbor.dev-XXX.com/k8s/sre/bigdata/flink:v2.0.0
2.提交 flink 任务到 k8s
# Flink On Kubernetes Native Mode部署
## 0. 新增账号
kubectl create serviceaccount flink
kubectl create clusterrolebinding flink-role --clusterrole=edit --serviceaccount=default:flink --namespace=default
## 1. 执行 kubernetes-session.sh，启动 flink 集群
cd /root/workspace/flink-1.14.0/
./bin/kubernetes-session.sh
-Dkubernetes.cluster-id=flink-k8s-native-session-cluster
-Dkubernetes.container.image=flink:latest
-Djobmanager.heap.size=1024m
-Dtaskmanager.memory.process.size=1035m
-Dtaskmanager.numberOfTaskSlots=1
-Dkubernetes.jobmanager.cpu=1
-Dkubernetes.taskmanager.cpu=1
-Dkubernetes.namespace=default
-Dkubernetes.jobmanager.service-account=flink
## 2. 提交 job 到指定 Session 集群
./bin/flink run -d -t kubernetes-session -Dkubernetes.cluster-id=flink-k8s-native-session-cluster examples/streaming/WindowJoin.jar
./bin/flink run -d -m ${K8S_NODE_1}:30649 -Dkubernetes.cluster-id=flink-k8s-native-session-cluster examples/streaming/WindowJoin.jar
启动的 Flink 集群和提交任务的运行，如下图:
三、问题及解决
3.1 问题1
## 问题
sed: couldn't open temporary file /opt/flink/conf/sedomqry9: Read-only file system
sed: couldn't open temporary file /opt/flink/conf/sedK84eTA: Read-only file system
/docker-entrypoint.sh: line 73: /opt/flink/conf/flink-conf.yaml: Read-only file system
/docker-entrypoint.sh: line 89: /opt/flink/conf/flink-conf.yaml.tmp: Read-only file system
Starting kubernetes-taskmanager as a console application on host flink-k8s-native-session-cluster-taskmanager-1-8.
2021-12-04 08:39:53,132 INFO org.apache.flink.kubernetes.taskmanager.KubernetesTaskExecutorRunner [] - --------------------------------------------------------------------------------
2021-12-04 08:39:53,135 INFO org.apache.flink.kubernetes.taskmanager.KubernetesTaskExecutorRunner [] - Starting Kubernetes TaskExecutor runner (Version: 1.14.0, Scala: 2.12, Rev:460b386, Date:2021-09-22T08:39:40+02:00)
2021-12-04 08:39:53,136 INFO org.apache.flink.kubernetes.taskmanager.KubernetesTaskExecutorRunner [] - OS current user: flink
2021-12-04 08:39:53,136 INFO org.apache.flink.kubernetes.taskmanager.KubernetesTaskExecutorRunner [] - Current Hadoop/Kerberos user: <no hadoop dependency found>
2021-12-04 08:39:53,136 INFO org.apache.flink.kubernetes.taskmanager.KubernetesTaskExecutorRunner [] - JVM: OpenJDK 64-Bit Server VM - Oracle Corporation - 1.8/25.312-b07
2021-12-04 08:39:53,136 INFO org.apache.flink.kubernetes.taskmanager.KubernetesTaskExecutorRunner [] - Maximum heap size: 154 MiBytes
2021-12-04 08:39:53,136 INFO org.apache.flink.kubernetes.taskmanager.KubernetesTaskExecutorRunner [] - JAVA_HOME: /usr/local/openjdk-8
2021-12-04 08:39:53,136 INFO org.apache.flink.kubernetes.taskmanager.KubernetesTaskExecutorRunner [] - No Hadoop Dependency available
2021-12-04 08:39:53,137 INFO org.apache.flink.kubernetes.taskmanager.KubernetesTaskExecutorRunner [] - JVM Options:
2021-12-04 08:39:53,137 INFO org.apache.flink.kubernetes.taskmanager.KubernetesTaskExecutorRunner [] - -XX:+UseG1GC
2021-12-04 08:39:53,137 INFO org.apache.flink.kubernetes.taskmanager.KubernetesTaskExecutorRunner [] - -Xmx161061270
2021-12-04 08:39:53,137 INFO org.apache.flink.kubernetes.taskmanager.KubernetesTaskExecutorRunner [] - -Xms161061270
2021-12-04 08:39:53,137 INFO org.apache.flink.kubernetes.taskmanager.KubernetesTaskExecutorRunner [] - -XX:MaxDirectMemorySize=201326592
2021-12-04 08:39:53,137 INFO org.apache.flink.kubernetes.taskmanager.KubernetesTaskExecutorRunner [] - -XX:MaxMetaspaceSize=268435456
2021-12-04 08:39:53,137 INFO org.apache.flink.kubernetes.taskmanager.KubernetesTaskExecutorRunner [] - -Dlog.file=/opt/flink/log/flink--kubernetes-taskmanager-0-flink-k8s-native-session-cluster-taskmanager-1-8.log
2021-12-04 08:39:53,137 INFO org.apache.flink.kubernetes.taskmanager.KubernetesTaskExecutorRunner [] - -Dlog4j.configuration=file:/opt/flink/conf/log4j-console.properties
2021-12-04 08:39:53,137 INFO org.apache.flink.kubernetes.taskmanager.KubernetesTaskExecutorRunner [] - -Dlog4j.configurationFile=file:/opt/flink/conf/log4j-console.properties
2021-12-04 08:39:53,137 INFO org.apache.flink.kubernetes.taskmanager.KubernetesTaskExecutorRunner [] - -Dlogback.configurationFile=file:/opt/flink/conf/logback-console.xml
2021-12-04 08:39:53,137 INFO org.apache.flink.kubernetes.taskmanager.KubernetesTaskExecutorRunner [] - Program Arguments:
2021-12-04 08:39:53,140 INFO org.apache.flink.kubernetes.taskmanager.KubernetesTaskExecutorRunner [] - --configDir
2021-12-04 08:39:53,140 INFO org.apache.flink.kubernetes.taskmanager.KubernetesTaskExecutorRunner [] - /opt/flink/conf
2021-12-04 08:39:53,140 INFO org.apache.flink.kubernetes.taskmanager.KubernetesTaskExecutorRunner [] - -Djobmanager.memory.jvm-overhead.min=201326592b
2021-12-04 08:39:53,140 INFO org.apache.flink.kubernetes.taskmanager.KubernetesTaskExecutorRunner [] - -Dtaskmanager.resource-id=flink-k8s-native-session-cluster-taskmanager-1-8
2021-12-04 08:39:53,140 INFO org.apache.flink.kubernetes.taskmanager.KubernetesTaskExecutorRunner [] - -Djobmanager.memory.off-heap.size=134217728b
2021-12-04 08:39:53,140 INFO org.apache.flink.kubernetes.taskmanager.KubernetesTaskExecutorRunner [] - -Dweb.tmpdir=/tmp/flink-web-34afcb64-7a46-43cd-b101-2e7d61f0ef68
2021-12-04 08:39:53,140 INFO org.apache.flink.kubernetes.taskmanager.KubernetesTaskExecutorRunner [] - -Djobmanager.memory.jvm-metaspace.size=268435456b
2021-12-04 08:39:53,141 INFO org.apache.flink.kubernetes.taskmanager.KubernetesTaskExecutorRunner [] - -Djobmanager.memory.heap.size=1073741824b
2021-12-04 08:39:53,141 INFO org.apache.flink.kubernetes.taskmanager.KubernetesTaskExecutorRunner [] - -Djobmanager.memory.jvm-overhead.max=201326592b
2021-12-04 08:39:53,141 INFO org.apache.flink.kubernetes.taskmanager.KubernetesTaskExecutorRunner [] - -D
2021-12-04 08:39:53,141 INFO org.apache.flink.kubernetes.taskmanager.KubernetesTaskExecutorRunner [] - taskmanager.memory.network.min=67108864b
2021-12-04 08:39:53,141 INFO org.apache.flink.kubernetes.taskmanager.KubernetesTaskExecutorRunner [] - -D
2021-12-04 08:39:53,141 INFO org.apache.flink.kubernetes.taskmanager.KubernetesTaskExecutorRunner [] - taskmanager.cpu.cores=1.0
2021-12-04 08:39:53,141 INFO org.apache.flink.kubernetes.taskmanager.KubernetesTaskExecutorRunner [] - -D
2021-12-04 08:39:53,141 INFO org.apache.flink.kubernetes.taskmanager.KubernetesTaskExecutorRunner [] - taskmanager.memory.task.off-heap.size=0b
2021-12-04 08:39:53,141 INFO org.apache.flink.kubernetes.taskmanager.KubernetesTaskExecutorRunner [] - -D
2021-12-04 08:39:53,141 INFO org.apache.flink.kubernetes.taskmanager.KubernetesTaskExecutorRunner [] - taskmanager.memory.jvm-metaspace.size=268435456b
2021-12-04 08:39:53,141 INFO org.apache.flink.kubernetes.taskmanager.KubernetesTaskExecutorRunner [] - -D
2021-12-04 08:39:53,142 INFO org.apache.flink.kubernetes.taskmanager.KubernetesTaskExecutorRunner [] - external-resources=none
2021-12-04 08:39:53,142 INFO org.apache.flink.kubernetes.taskmanager.KubernetesTaskExecutorRunner [] - -D
2021-12-04 08:39:53,142 INFO org.apache.flink.kubernetes.taskmanager.KubernetesTaskExecutorRunner [] - taskmanager.memory.jvm-overhead.min=201326592b
2021-12-04 08:39:53,142 INFO org.apache.flink.kubernetes.taskmanager.KubernetesTaskExecutorRunner [] - -D
2021-12-04 08:39:53,142 INFO org.apache.flink.kubernetes.taskmanager.KubernetesTaskExecutorRunner [] - taskmanager.memory.framework.off-heap.size=134217728b
2021-12-04 08:39:53,142 INFO org.apache.flink.kubernetes.taskmanager.KubernetesTaskExecutorRunner [] - -D
2021-12-04 08:39:53,142 INFO org.apache.flink.kubernetes.taskmanager.KubernetesTaskExecutorRunner [] - taskmanager.memory.network.max=67108864b
2021-12-04 08:39:53,142 INFO org.apache.flink.kubernetes.taskmanager.KubernetesTaskExecutorRunner [] - -D
2021-12-04 08:39:53,142 INFO org.apache.flink.kubernetes.taskmanager.KubernetesTaskExecutorRunner [] - taskmanager.memory.framework.heap.size=134217728b
2021-12-04 08:39:53,142 INFO org.apache.flink.kubernetes.taskmanager.KubernetesTaskExecutorRunner [] - -D
2021-12-04 08:39:53,142 INFO org.apache.flink.kubernetes.taskmanager.KubernetesTaskExecutorRunner [] - taskmanager.memory.managed.size=241591914b
2021-12-04 08:39:53,143 INFO org.apache.flink.kubernetes.taskmanager.KubernetesTaskExecutorRunner [] - -D
2021-12-04 08:39:53,143 INFO org.apache.flink.kubernetes.taskmanager.KubernetesTaskExecutorRunner [] - taskmanager.memory.task.heap.size=26843542b
2021-12-04 08:39:53,143 INFO org.apache.flink.kubernetes.taskmanager.KubernetesTaskExecutorRunner [] - -D
2021-12-04 08:39:53,143 INFO org.apache.flink.kubernetes.taskmanager.KubernetesTaskExecutorRunner [] - taskmanager.numberOfTaskSlots=1
2021-12-04 08:39:53,143 INFO org.apache.flink.kubernetes.taskmanager.KubernetesTaskExecutorRunner [] - -D
2021-12-04 08:39:53,143 INFO org.apache.flink.kubernetes.taskmanager.KubernetesTaskExecutorRunner [] - taskmanager.memory.jvm-overhead.max=201326592b
2021-12-04 08:39:53,143 INFO org.apache.flink.kubernetes.taskmanager.KubernetesTaskExecutorRunner [] - Classpath: /opt/flink/lib/flink-csv-1.14.0.jar:/opt/flink/lib/flink-json-1.14.0.jar:/opt/flink/lib/flink-shaded-zookeeper-3.4.14.jar:/opt/flink/lib/flink-table_2.12-1.14.0.jar:/opt/flink/lib/log4j-1.2-api-2.14.1.jar:/opt/flink/lib/log4j-api-2.14.1.jar:/opt/flink/lib/log4j-core-2.14.1.jar:/opt/flink/lib/log4j-slf4j-impl-2.14.1.jar:/opt/flink/lib/flink-dist_2.12-1.14.0.jar::/opt/hadoop/conf:
2021-12-04 08:39:53,143 INFO org.apache.flink.kubernetes.taskmanager.KubernetesTaskExecutorRunner [] - --------------------------------------------------------------------------------
2021-12-04 08:39:53,144 INFO org.apache.flink.kubernetes.taskmanager.KubernetesTaskExecutorRunner [] - Registered UNIX signal handlers for [TERM, HUP, INT]
2021-12-04 08:39:53,219 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: blob.server.port, 6124
2021-12-04 08:39:53,219 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: taskmanager.memory.process.size, 1024m
2021-12-04 08:39:53,220 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: kubernetes.internal.jobmanager.entrypoint.class, org.apache.flink.kubernetes.entrypoint.KubernetesSessionClusterEntrypoint
2021-12-04 08:39:53,220 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: jobmanager.execution.failover-strategy, region
2021-12-04 08:39:53,220 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: jobmanager.rpc.address, flink-k8s-native-session-cluster.default
2021-12-04 08:39:53,220 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: execution.target, kubernetes-session
2021-12-04 08:39:53,221 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: jobmanager.memory.process.size, 1600m
2021-12-04 08:39:53,221 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: kubernetes.taskmanager.cpu, 1
2021-12-04 08:39:53,221 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: jobmanager.rpc.port, 6123
2021-12-04 08:39:53,221 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: kubernetes.cluster-id, flink-k8s-native-session-cluster
2021-12-04 08:39:53,222 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: taskmanager.rpc.port, 6122
2021-12-04 08:39:53,222 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: internal.cluster.execution-mode, NORMAL
2021-12-04 08:39:53,222 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: kubernetes.container.image, flink:latest
2021-12-04 08:39:53,222 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: parallelism.default, 1
2021-12-04 08:39:53,222 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: kubernetes.namespace, default
2021-12-04 08:39:53,223 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: kubernetes.jobmanager.cpu, 1
2021-12-04 08:39:53,223 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: taskmanager.numberOfTaskSlots, 1
2021-12-04 08:39:53,223 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: kubernetes.jobmanager.service-account, flink
2021-12-04 08:39:53,223 INFO org.apache.flink.configuration.GlobalConfiguration [] - Loading configuration property: jobmanager.heap.size, 1024m
2021-12-04 08:39:53,417 INFO org.apache.flink.core.fs.FileSystem [] - Hadoop is not in the classpath/dependencies. The extended set of supported File Systems via Hadoop is not available.
2021-12-04 08:39:53,511 INFO org.apache.flink.runtime.state.changelog.StateChangelogStorageLoader [] - StateChangelogStorageLoader initialized with shortcut names {memory}.
2021-12-04 08:39:53,516 INFO org.apache.flink.runtime.state.changelog.StateChangelogStorageLoader [] - StateChangelogStorageLoader initialized with shortcut names {memory}.
2021-12-04 08:39:53,530 INFO org.apache.flink.runtime.security.modules.HadoopModuleFactory [] - Cannot create Hadoop Security Module because Hadoop cannot be found in the Classpath.
2021-12-04 08:39:53,543 INFO org.apache.flink.runtime.security.modules.JaasModule [] - Jaas file will be created as /tmp/jaas-4120144799066343914.conf.
2021-12-04 08:39:53,601 INFO org.apache.flink.runtime.security.contexts.HadoopSecurityContextFactory [] - Cannot install HadoopSecurityContext because Hadoop cannot be found in the Classpath.
2021-12-04 08:39:54,426 INFO org.apache.flink.configuration.Configuration [] - Config uses fallback configuration key 'jobmanager.rpc.address' instead of key 'rest.address'
2021-12-04 08:39:54,510 INFO org.apache.flink.runtime.util.LeaderRetrievalUtils [] - Trying to select the network interface and address to use by connecting to the leading JobManager.
2021-12-04 08:39:54,510 INFO org.apache.flink.runtime.util.LeaderRetrievalUtils [] - TaskManager will try to connect for PT10S before falling back to heuristics
2021-12-04 08:39:55,212 INFO org.apache.flink.runtime.net.ConnectionUtils [] - Trying to connect to address flink-k8s-native-session-cluster.default:6123
2021-12-04 08:39:55,214 INFO org.apache.flink.runtime.net.ConnectionUtils [] - Failed to connect to [flink-k8s-native-session-cluster.default:6123] from local address [flink-k8s-native-session-cluster-taskmanager-1-8/10.244.36.111] with timeout [200] due to: flink-k8s-native-session-cluster.default
2021-12-04 08:39:55,315 INFO org.apache.flink.runtime.net.ConnectionUtils [] - Trying to connect to address flink-k8s-native-session-cluster.default:6123
2021-12-04 08:39:55,317 INFO org.apache.flink.runtime.net.ConnectionUtils [] - Failed to connect to [flink-k8s-native-session-cluster.default:6123] from local address [flink-k8s-native-session-cluster-taskmanager-1-8/10.244.36.111] with timeout [200] due to: flink-k8s-native-session-cluster.default
2021-12-04 08:39:55,518 INFO org.apache.flink.runtime.net.ConnectionUtils [] - Trying to connect to address flink-k8s-native-session-cluster.default:6123
2021-12-04 08:39:55,519 INFO org.apache.flink.runtime.net.ConnectionUtils [] - Failed to connect to [flink-k8s-native-session-cluster.default:6123] from local address [flink-k8s-native-session-cluster-taskmanager-1-8/10.244.36.111] with timeout [200] due to: flink-k8s-native-session-cluster.default
2021-12-04 08:39:55,920 INFO org.apache.flink.runtime.net.ConnectionUtils [] - Trying to connect to address flink-k8s-native-session-cluster.default:6123
2021-12-04 08:39:55,922 INFO org.apache.flink.runtime.net.ConnectionUtils [] - Failed to connect to [flink-k8s-native-session-cluster.default:6123] from local address [flink-k8s-native-session-cluster-taskmanager-1-8/10.244.36.111] with timeout [200] due to: flink-k8s-native-session-cluster.default
2021-12-04 08:39:56,723 INFO org.apache.flink.runtime.net.ConnectionUtils [] - Trying to connect to address flink-k8s-native-session-cluster.default:6123
2021-12-04 08:39:56,724 INFO org.apache.flink.runtime.net.ConnectionUtils [] - Failed to connect to [flink-k8s-native-session-cluster.default:6123] from local address [flink-k8s-native-session-cluster-taskmanager-1-8/10.244.36.111] with timeout [200] due to: flink-k8s-native-session-cluster.default
2021-12-04 08:39:58,325 INFO org.apache.flink.runtime.net.ConnectionUtils [] - Trying to connect to address flink-k8s-native-session-cluster.default:6123
2021-12-04 08:39:58,327 INFO org.apache.flink.runtime.net.ConnectionUtils [] - Failed to connect to [flink-k8s-native-session-cluster.default:6123] from local address [flink-k8s-native-session-cluster-taskmanager-1-8/10.244.36.111] with timeout [200] due to: flink-k8s-native-session-cluster.default
2021-12-04 08:40:01,528 INFO org.apache.flink.runtime.net.ConnectionUtils [] - Trying to connect to address flink-k8s-native-session-cluster.default:6123
2021-12-04 08:40:01,530 INFO org.apache.flink.runtime.net.ConnectionUtils [] - Failed to connect to [flink-k8s-native-session-cluster.default:6123] from local address [flink-k8s-native-session-cluster-taskmanager-1-8/10.244.36.111] with timeout [200] due to: flink-k8s-native-session-cluster.default
2021-12-04 08:40:04,511 WARN org.apache.flink.runtime.net.ConnectionUtils [] - Could not connect to flink-k8s-native-session-cluster.default:6123. Selecting a local address using heuristics.
2021-12-04 08:40:04,512 WARN org.apache.flink.runtime.net.ConnectionUtils [] - Could not find any IPv4 address that is not loopback or link-local. Using localhost address.
2021-12-04 08:40:04,513 INFO org.apache.flink.runtime.taskexecutor.TaskManagerRunner [] - TaskManager will use hostname/address 'flink-k8s-native-session-cluster-taskmanager-1-8' (10.244.36.111) for communication.
2021-12-04 08:40:04,575 INFO org.apache.flink.runtime.rpc.akka.AkkaRpcServiceUtils [] - Trying to start actor system, external address 10.244.36.111:6122, bind address 0.0.0.0:6122.
2021-12-04 08:40:06,138 INFO akka.event.slf4j.Slf4jLogger [] - Slf4jLogger started
2021-12-04 08:40:06,312 INFO akka.remote.RemoteActorRefProvider [] - Akka Cluster not in use - enabling unsafe features anyway because `akka.remote.use-unsafe-remote-features-outside-cluster` has been enabled.
2021-12-04 08:40:06,313 INFO akka.remote.Remoting [] - Starting remoting
2021-12-04 08:40:06,799 INFO akka.remote.Remoting [] - Remoting started; listening on addresses :[akka.tcp://[email protected]:6122]
2021-12-04 08:40:07,136 INFO org.apache.flink.runtime.rpc.akka.AkkaRpcServiceUtils [] - Actor system started at akka.tcp://[email protected]:6122
2021-12-04 08:40:07,228 INFO org.apache.flink.runtime.metrics.MetricRegistryImpl [] - No metrics reporter configured, no metrics will be exposed/reported.
2021-12-04 08:40:07,232 INFO org.apache.flink.runtime.rpc.akka.AkkaRpcServiceUtils [] - Trying to start actor system, external address 10.244.36.111:0, bind address 0.0.0.0:0.
2021-12-04 08:40:07,310 INFO akka.event.slf4j.Slf4jLogger [] - Slf4jLogger started
2021-12-04 08:40:07,325 INFO akka.remote.RemoteActorRefProvider [] - Akka Cluster not in use - enabling unsafe features anyway because `akka.remote.use-unsafe-remote-features-outside-cluster` has been enabled.
2021-12-04 08:40:07,325 INFO akka.remote.Remoting [] - Starting remoting
2021-12-04 08:40:07,413 INFO akka.remote.Remoting [] - Remoting started; listening on addresses :[akka.tcp://[email protected]:40391]
2021-12-04 08:40:07,433 INFO org.apache.flink.runtime.rpc.akka.AkkaRpcServiceUtils [] - Actor system started at akka.tcp://[email protected]:40391
2021-12-04 08:40:07,523 INFO org.apache.flink.runtime.rpc.akka.AkkaRpcService [] - Starting RPC endpoint for org.apache.flink.runtime.metrics.dump.MetricQueryService at akka://flink-metrics/user/rpc/MetricQueryService_flink-k8s-native-session-cluster-taskmanager-1-8 .
2021-12-04 08:40:07,612 INFO org.apache.flink.runtime.blob.PermanentBlobCache [] - Created BLOB cache storage directory /tmp/blobStore-a4cbbe8c-bbaa-48cb-854b-75dd30eb6e93
2021-12-04 08:40:07,617 INFO org.apache.flink.runtime.blob.TransientBlobCache [] - Created BLOB cache storage directory /tmp/blobStore-3e14485a-7eb2-40e6-b06c-8d98f5eefa47
2021-12-04 08:40:07,622 INFO org.apache.flink.runtime.externalresource.ExternalResourceUtils [] - Enabled external resources: []
2021-12-04 08:40:07,623 INFO org.apache.flink.runtime.taskexecutor.TaskManagerRunner [] - Starting TaskManager with ResourceID: flink-k8s-native-session-cluster-taskmanager-1-8
2021-12-04 08:40:07,724 INFO org.apache.flink.runtime.taskexecutor.TaskManagerServices [] - Temporary file directory '/tmp': total 25 GB, usable 11 GB (44.00% usable)
2021-12-04 08:40:07,728 INFO org.apache.flink.runtime.io.disk.iomanager.IOManager [] - Created a new FileChannelManager for spilling of task related data to disk (joins, sorting, ...). Used directories:
/tmp/flink-io-c7f2fcef-fbb8-453b-bc02-728443b35a87
2021-12-04 08:40:07,802 INFO org.apache.flink.runtime.io.network.netty.NettyConfig [] - NettyConfig [server address: /0.0.0.0, server port: 0, ssl enabled: false, memory segment size (bytes): 32768, transport type: AUTO, number of server threads: 1 (manual), number of client threads: 1 (manual), server connect backlog: 0 (use Netty's default), client connect timeout (sec): 120, send/receive buffer size (bytes): 0 (use Netty's default)]
2021-12-04 08:40:07,808 INFO org.apache.flink.runtime.io.network.NettyShuffleServiceFactory [] - Created a new FileChannelManager for storing result partitions of BLOCKING shuffles. Used directories:
/tmp/flink-netty-shuffle-c3d9cb0d-52eb-499b-bc33-c0a9812428bc
2021-12-04 08:40:08,400 INFO org.apache.flink.runtime.io.network.buffer.NetworkBufferPool [] - Allocated 64 MB for network buffer pool (number of memory segments: 2048, bytes per segment: 32768).
2021-12-04 08:40:08,422 INFO org.apache.flink.runtime.io.network.NettyShuffleEnvironment [] - Starting the network environment and its components.
2021-12-04 08:40:08,736 INFO org.apache.flink.runtime.io.network.netty.NettyClient [] - Transport type 'auto': using EPOLL.
2021-12-04 08:40:08,738 INFO org.apache.flink.runtime.io.network.netty.NettyClient [] - Successful initialization (took 315 ms).
2021-12-04 08:40:08,743 INFO org.apache.flink.runtime.io.network.netty.NettyServer [] - Transport type 'auto': using EPOLL.
2021-12-04 08:40:08,822 INFO org.apache.flink.runtime.io.network.netty.NettyServer [] - Successful initialization (took 82 ms). Listening on SocketAddress /0.0.0.0:40923.
2021-12-04 08:40:08,825 INFO org.apache.flink.runtime.taskexecutor.KvStateService [] - Starting the kvState service and its components.
2021-12-04 08:40:08,933 INFO org.apache.flink.runtime.rpc.akka.AkkaRpcService [] - Starting RPC endpoint for org.apache.flink.runtime.taskexecutor.TaskExecutor at akka://flink/user/rpc/taskmanager_0 .
2021-12-04 08:40:09,027 INFO org.apache.flink.runtime.taskexecutor.DefaultJobLeaderService [] - Start job leader service.
2021-12-04 08:40:09,029 INFO org.apache.flink.runtime.filecache.FileCache [] - User file cache uses directory /tmp/flink-dist-cache-c2966dd1-d57c-4a8e-a286-aec5398c6399
2021-12-04 08:40:09,031 INFO org.apache.flink.runtime.taskexecutor.TaskExecutor [] - Connecting to ResourceManager akka.tcp://[email protected]:6123/user/rpc/resourcemanager_*(00000000000000000000000000000000).
2021-12-04 08:40:09,408 WARN akka.remote.ReliableDeliverySupervisor [] - Association with remote system [akka.tcp://[email protected]:6123] has failed, address is now gated for [50] ms. Reason: [Association failed with [akka.tcp://[email protected]:6123]] Caused by: [java.net.UnknownHostException: flink-k8s-native-session-cluster.default: Temporary failure in name resolution]
2021-12-04 08:40:09,498 INFO org.apache.flink.runtime.taskexecutor.TaskExecutor [] - Could not resolve ResourceManager address akka.tcp://[email protected]:6123/user/rpc/resourcemanager_*, retrying in 10000 ms: Could not connect to rpc endpoint under address akka.tcp://[email protected]:6123/user/rpc/resourcemanager_*.
2021-12-04 08:40:19,558 WARN akka.remote.ReliableDeliverySupervisor [] - Association with remote system [akka.tcp://[email protected]:6123] has failed, address is now gated for [50] ms. Reason: [Association failed with [akka.tcp://[email protected]:6123]] Caused by: [java.net.UnknownHostException: flink-k8s-native-session-cluster.default: Temporary failure in name resolution]
2021-12-04 08:40:19,560 INFO org.apache.flink.runtime.taskexecutor.TaskExecutor [] - Could not resolve ResourceManager address akka.tcp://[email protected]:6123/user/rpc/resourcemanager_*, retrying in 10000 ms: Could not connect to rpc endpoint under address akka.tcp://[email protected]:6123/user/rpc/resourcemanager_*.
2021-12-04 08:40:29,611 WARN akka.remote.ReliableDeliverySupervisor [] - Association with remote system [akka.tcp://[email protected]:6123] has failed, address is now gated for [50] ms. Reason: [Association failed with [akka.tcp://[email protected]:6123]] Caused by: [java.net.UnknownHostException: flink-k8s-native-session-cluster.default: Temporary failure in name resolution]
2021-12-04 08:40:29,613 INFO org.apache.flink.runtime.taskexecutor.TaskExecutor [] - Could not resolve ResourceManager address akka.tcp://[email protected]:6123/user/rpc/resourcemanager_*, retrying in 10000 ms: Could not connect to rpc endpoint under address akka.tcp://[email protected]:6123/user/rpc/resourcemanager_*.
2021-12-04 08:40:39,661 WARN akka.remote.ReliableDeliverySupervisor [] - Association with remote system [akka.tcp://[email protected]:6123] has failed, address is now gated for [50] ms. Reason: [Association failed with [akka.tcp://[email protected]:6123]] Caused by: [java.net.UnknownHostException: flink-k8s-native-session-cluster.default: Temporary failure in name resolution]
2021-12-04 08:40:39,663 INFO org.apache.flink.runtime.taskexecutor.TaskExecutor [] - Could not resolve ResourceManager address akka.tcp://[email protected]:6123/user/rpc/resourcemanager_*, retrying in 10000 ms: Could not connect to rpc endpoint under address akka.tcp://[email protected]:6123/user/rpc/resourcemanager_*.
2021-12-04 08:40:49,711 WARN akka.remote.ReliableDeliverySupervisor [] - Association with remote system [akka.tcp://[email protected]:6123] has failed, address is now gated for [50] ms. Reason: [Association failed with [akka.tcp://[email protected]:6123]] Caused by: [java.net.UnknownHostException: flink-k8s-native-session-cluster.default: Temporary failure in name resolution]
2021-12-04 08:40:49,714 INFO org.apache.flink.runtime.taskexecutor.TaskExecutor [] - Could not resolve ResourceManager address akka.tcp://[email protected]:6123/user/rpc/resourcemanager_*, retrying in 10000 ms: Could not connect to rpc endpoint under address akka.tcp://[email protected]:6123/user/rpc/resourcemanager_*.
2021-12-04 08:40:59,764 WARN akka.remote.ReliableDeliverySupervisor [] - Association with remote system [akka.tcp://[email protected]:6123] has failed, address is now gated for [50] ms. Reason: [Association failed with [akka.tcp://[email protected]:6123]] Caused by: [java.net.UnknownHostException: flink-k8s-native-session-cluster.default: Temporary failure in name resolution]
2021-12-04 08:40:59,767 INFO org.apache.flink.runtime.taskexecutor.TaskExecutor [] - Could not resolve ResourceManager address akka.tcp://[email protected]:6123/user/rpc/resourcemanager_*, retrying in 10000 ms: Could not connect to rpc endpoint under address akka.tcp://[email protected]:6123/user/rpc/resourcemanager_*.
2021-12-04 08:41:09,808 WARN akka.remote.ReliableDeliverySupervisor [] - Association with remote system [akka.tcp://[email protected]:6123] has failed, address is now gated for [50] ms. Reason: [Association failed with [akka.tcp://[email protected]:6123]] Caused by: [java.net.UnknownHostException: flink-k8s-native-session-cluster.default: Temporary failure in name resolution]
2021-12-04 08:41:09,812 INFO org.apache.flink.runtime.taskexecutor.TaskExecutor [] - Could not resolve ResourceManager address akka.tcp://[email protected]:6123/user/rpc/resourcemanager_*, retrying in 10000 ms: Could not connect to rpc endpoint under address akka.tcp://[email protected]:6123/user/rpc/resourcemanager_*.
2021-12-04 08:41:19,854 WARN akka.remote.ReliableDeliverySupervisor [] - Association with remote system [akka.tcp://[email protected]:6123] has failed, address is now gated for [50] ms. Reason: [Association failed with [akka.tcp://[email protected]:6123]] Caused by: [java.net.UnknownHostException: flink-k8s-native-session-cluster.default: Temporary failure in name resolution]
2021-12-04 08:41:19,857 INFO org.apache.flink.runtime.taskexecutor.TaskExecutor [] - Could not resolve ResourceManager address akka.tcp://[email protected]:6123/user/rpc/resourcemanager_*, retrying in 10000 ms: Could not connect to rpc endpoint under address akka.tcp://[email protected]:6123/user/rpc/resourcemanager_*.
2021-12-04 08:41:29,901 WARN akka.remote.ReliableDeliverySupervisor [] - Association with remote system [akka.tcp://[email protected]:6123] has failed, address is now gated for [50] ms. Reason: [Association failed with [akka.tcp://[email protected]:6123]] Caused by: [java.net.UnknownHostException: flink-k8s-native-session-cluster.default: Temporary failure in name resolution]
2021-12-04 08:41:29,903 INFO org.apache.flink.runtime.taskexecutor.TaskExecutor [] - Could not resolve ResourceManager address akka.tcp://[email protected]:6123/user/rpc/resourcemanager_*, retrying in 10000 ms: Could not connect to rpc endpoint under address akka.tcp://[email protected]:6123/user/rpc/resourcemanager_*.
2021-12-04 08:41:39,946 WARN akka.remote.ReliableDeliverySupervisor [] - Association with remote system [akka.tcp://[email protected]:6123] has failed, address is now gated for [50] ms. Reason: [Association failed with [akka.tcp://[email protected]:6123]] Caused by: [java.net.UnknownHostException: flink-k8s-native-session-cluster.default: Temporary failure in name resolution]
2021-12-04 08:41:39,949 INFO org.apache.flink.runtime.taskexecutor.TaskExecutor [] - Could not resolve ResourceManager address akka.tcp://[email protected]:6123/user/rpc/resourcemanager_*, retrying in 10000 ms: Could not connect to rpc endpoint under address akka.tcp://[email protected]:6123/user/rpc/resourcemanager_*.
2021-12-04 08:41:49,993 WARN akka.remote.ReliableDeliverySupervisor [] - Association with remote system [akka.tcp://[email protected]:6123] has failed, address is now gated for [50] ms. Reason: [Association failed with [akka.tcp://[email protected]:6123]] Caused by: [java.net.UnknownHostException: flink-k8s-native-session-cluster.default: Temporary failure in name resolution]
2021-12-04 08:41:49,996 INFO org.apache.flink.runtime.taskexecutor.TaskExecutor [] - Could not resolve ResourceManager address akka.tcp://[email protected]:6123/user/rpc/resourcemanager_*, retrying in 10000 ms: Could not connect to rpc endpoint under address akka.tcp://[email protected]:6123/user/rpc/resourcemanager_*.
2021-12-04 08:42:00,035 WARN akka.remote.ReliableDeliverySupervisor [] - Association with remote system [akka.tcp://[email protected]:6123] has failed, address is now gated for [50] ms. Reason: [Association failed with [akka.tcp://[email protected]:6123]] Caused by: [java.net.UnknownHostException: flink-k8s-native-session-cluster.default: Temporary failure in name resolution]
2021-12-04 08:42:00,038 INFO org.apache.flink.runtime.taskexecutor.TaskExecutor [] - Could not resolve ResourceManager address akka.tcp://[email protected]:6123/user/rpc/resourcemanager_*, retrying in 10000 ms: Could not connect to rpc endpoint under address akka.tcp://[email protected]:6123/user/rpc/resourcemanager_*.
2021-12-04 08:42:10,084 WARN akka.remote.ReliableDeliverySupervisor [] - Association with remote system [akka.tcp://[email protected]:6123] has failed, address is now gated for [50] ms. Reason: [Association failed with [akka.tcp://[email protected]:6123]] Caused by: [java.net.UnknownHostException: flink-k8s-native-session-cluster.default: Temporary failure in name resolution]
2021-12-04 08:42:10,086 INFO org.apache.flink.runtime.taskexecutor.TaskExecutor [] - Could not resolve ResourceManager address akka.tcp://[email protected]:6123/user/rpc/resourcemanager_*, retrying in 10000 ms: Could not connect to rpc endpoint under address akka.tcp://[email protected]:6123/user/rpc/resourcemanager_*.
2021-12-04 08:42:20,127 WARN akka.remote.ReliableDeliverySupervisor [] - Association with remote system [akka.tcp://[email protected]:6123] has failed, address is now gated for [50] ms. Reason: [Association failed with [akka.tcp://[email protected]:6123]] Caused by: [java.net.UnknownHostException: flink-k8s-native-session-cluster.default: Temporary failure in name resolution]
2021-12-04 08:42:20,129 INFO org.apache.flink.runtime.taskexecutor.TaskExecutor [] - Could not resolve ResourceManager address akka.tcp://[email protected]:6123/user/rpc/resourcemanager_*, retrying in 10000 ms: Could not connect to rpc endpoint under address akka.tcp://[email protected]:6123/user/rpc/resourcemanager_*.
2021-12-04 08:42:30,175 WARN akka.remote.ReliableDeliverySupervisor [] - Association with remote system [akka.tcp://[email protected]:6123] has failed, address is now gated for [50] ms. Reason: [Association failed with [akka.tcp://[email protected]:6123]] Caused by: [java.net.UnknownHostException: flink-k8s-native-session-cluster.default: Temporary failure in name resolution]
2021-12-04 08:42:30,178 INFO org.apache.flink.runtime.taskexecutor.TaskExecutor [] - Could not resolve ResourceManager address akka.tcp://[email protected]:6123/user/rpc/resourcemanager_*, retrying in 10000 ms: Could not connect to rpc endpoint under address akka.tcp://[email protected]:6123/user/rpc/resourcemanager_*.
2021-12-04 08:42:40,225 WARN akka.remote.ReliableDeliverySupervisor [] - Association with remote system [akka.tcp://[email protected]:6123] has failed, address is now gated for [50] ms. Reason: [Association failed with [akka.tcp://[email protected]:6123]] Caused by: [java.net.UnknownHostException: flink-k8s-native-session-cluster.default: Temporary failure in name resolution]
2021-12-04 08:42:40,228 INFO org.apache.flink.runtime.taskexecutor.TaskExecutor [] - Could not resolve ResourceManager address akka.tcp://[email protected]:6123/user/rpc/resourcemanager_*, retrying in 10000 ms: Could not connect to rpc endpoint under address akka.tcp://[email protected]:6123/user/rpc/resourcemanager_*.
2021-12-04 08:42:50,269 WARN akka.remote.ReliableDeliverySupervisor [] - Association with remote system [akka.tcp://[email protected]:6123] has failed, address is now gated for [50] ms. Reason: [Association failed with [akka.tcp://[email protected]:6123]] Caused by: [java.net.UnknownHostException: flink-k8s-native-session-cluster.default: Temporary failure in name resolution]
2021-12-04 08:42:50,272 INFO org.apache.flink.runtime.taskexecutor.TaskExecutor [] - Could not resolve ResourceManager address akka.tcp://[email protected]:6123/user/rpc/resourcemanager_*, retrying in 10000 ms: Could not connect to rpc endpoint under address akka.tcp://[email protected]:6123/user/rpc/resourcemanager_*.
2021-12-04 08:43:00,319 WARN akka.remote.ReliableDeliverySupervisor [] - Association with remote system [akka.tcp://[email protected]:6123] has failed, address is now gated for [50] ms. Reason: [Association failed with [akka.tcp://[email protected]:6123]] Caused by: [java.net.UnknownHostException: flink-k8s-native-session-cluster.default: Temporary failure in name resolution]
2021-12-04 08:43:00,322 INFO org.apache.flink.runtime.taskexecutor.TaskExecutor [] - Could not resolve ResourceManager address akka.tcp://[email protected]:6123/user/rpc/resourcemanager_*, retrying in 10000 ms: Could not connect to rpc endpoint under address akka.tcp://[email protected]:6123/user/rpc/resourcemanager_*.
2021-12-04 08:43:10,370 WARN akka.remote.ReliableDeliverySupervisor [] - Association with remote system [akka.tcp://[email protected]:6123] has failed, address is now gated for [50] ms. Reason: [Association failed with [akka.tcp://[email protected]:6123]] Caused by: [java.net.UnknownHostException: flink-k8s-native-session-cluster.default: Temporary failure in name resolution]
2021-12-04 08:43:10,372 INFO org.apache.flink.runtime.taskexecutor.TaskExecutor [] - Could not resolve ResourceManager address akka.tcp://[email protected]:6123/user/rpc/resourcemanager_*, retrying in 10000 ms: Could not connect to rpc endpoint under address akka.tcp://[email protected]:6123/user/rpc/resourcemanager_*.
2021-12-04 08:43:20,420 WARN akka.remote.ReliableDeliverySupervisor [] - Association with remote system [akka.tcp://[email protected]:6123] has failed, address is now gated for [50] ms. Reason: [Association failed with [akka.tcp://[email protected]:6123]] Caused by: [java.net.UnknownHostException: flink-k8s-native-session-cluster.default: Temporary failure in name resolution]
2021-12-04 08:43:20,422 INFO org.apache.flink.runtime.taskexecutor.TaskExecutor [] - Could not resolve ResourceManager address akka.tcp://[email protected]:6123/user/rpc/resourcemanager_*, retrying in 10000 ms: Could not connect to rpc endpoint under address akka.tcp://[email protected]:6123/user/rpc/resourcemanager_*.
2021-12-04 08:43:30,463 WARN akka.remote.ReliableDeliverySupervisor [] - Association with remote system [akka.tcp://[email protected]:6123] has failed, address is now gated for [50] ms. Reason: [Association failed with [akka.tcp://[email protected]:6123]] Caused by: [java.net.UnknownHostException: flink-k8s-native-session-cluster.default: Temporary failure in name resolution]
2021-12-04 08:43:30,466 INFO org.apache.flink.runtime.taskexecutor.TaskExecutor [] - Could not resolve ResourceManager address akka.tcp://[email protected]:6123/user/rpc/resourcemanager_*, retrying in 10000 ms: Could not connect to rpc endpoint under address akka.tcp://[email protected]:6123/user/rpc/resourcemanager_*.
2021-12-04 08:43:40,504 WARN akka.remote.ReliableDeliverySupervisor [] - Association with remote system [akka.tcp://[email protected]:6123] has failed, address is now gated for [50] ms. Reason: [Association failed with [akka.tcp://[email protected]:6123]] Caused by: [java.net.UnknownHostException: flink-k8s-native-session-cluster.default: Temporary failure in name resolution]
2021-12-04 08:43:40,507 INFO org.apache.flink.runtime.taskexecutor.TaskExecutor [] - Could not resolve ResourceManager address akka.tcp://[email protected]:6123/user/rpc/resourcemanager_*, retrying in 10000 ms: Could not connect to rpc endpoint under address akka.tcp://[email protected]:6123/user/rpc/resourcemanager_*.
2021-12-04 08:43:50,551 WARN akka.remote.ReliableDeliverySupervisor [] - Association with remote system [akka.tcp://[email protected]:6123] has failed, address is now gated for [50] ms. Reason: [Association failed with [akka.tcp://[email protected]:6123]] Caused by: [java.net.UnknownHostException: flink-k8s-native-session-cluster.default: Temporary failure in name resolution]
2021-12-04 08:43:50,553 INFO org.apache.flink.runtime.taskexecutor.TaskExecutor [] - Could not resolve ResourceManager address akka.tcp://[email protected]:6123/user/rpc/resourcemanager_*, retrying in 10000 ms: Could not connect to rpc endpoint under address akka.tcp://[email protected]:6123/user/rpc/resourcemanager_*.
## 解决
本图文内容来源于网友网络收集整理提供，作为学习参考使用，版权属于原作者。
THE END
二维码