| | 指定driver和executor的命名空间 |
| | 用于指定 Spark 应用程序的容器镜像。必填，除非为每种不同的容器类型提供了显式图像。（见下方两个位置） |
| | driver镜像 |
| | executor镜像 |
| | 在 Kubernetes 中拉取镜像时使用的容器镜像拉取策略。 |
| | 用于从私有映像仓库中提取镜像的 Kubernetes 密钥。 |
| | 每轮执行程序 Pod 分配中一次启动的 Pod 数量。 |
| | 每轮执行程序 Pod 分配之间等待的时间。指定小于 1 秒的值可能会导致 Spark driver的 CPU 使用率过高。 |
| (none) | Path to the CA cert file for connecting to the Kubernetes API server over TLS when starting the driver. This file must be located on the submitting machine’s disk. Specify this as a path as opposed to a URI (i.e. do not provide a scheme). In client mode, use instead. |
| (none) | Path to the client key file for authenticating against the Kubernetes API server when starting the driver. This file must be located on the submitting machine’s disk. Specify this as a path as opposed to a URI (i.e. do not provide a scheme). In client mode, use instead. |
| (none) | Path to the client cert file for authenticating against the Kubernetes API server when starting the driver. This file must be located on the submitting machine’s disk. Specify this as a path as opposed to a URI (i.e. do not provide a scheme). In client mode, use instead. |
| (none) | OAuth token to use when authenticating against the Kubernetes API server when starting the driver. Note that unlike the other authentication options, this is expected to be the exact string value of the token to use for the authentication. In client mode, use instead. |
| (none) | Path to the OAuth token file containing the token to use when authenticating against the Kubernetes API server when starting the driver. This file must be located on the submitting machine’s disk. Specify this as a path as opposed to a URI (i.e. do not provide a scheme). In client mode, use instead. |
| (none) | Path to the CA cert file for connecting to the Kubernetes API server over TLS from the driver pod when requesting executors. This file must be located on the submitting machine’s disk, and will be uploaded to the driver pod. Specify this as a path as opposed to a URI (i.e. do not provide a scheme). In client mode, use instead. |
| (none) | Path to the client key file for authenticating against the Kubernetes API server from the driver pod when requesting executors. This file must be located on the submitting machine’s disk, and will be uploaded to the driver pod as a Kubernetes secret. Specify this as a path as opposed to a URI (i.e. do not provide a scheme). In client mode, use instead. |
| (none) | Path to the client cert file for authenticating against the Kubernetes API server from the driver pod when requesting executors. This file must be located on the submitting machine’s disk, and will be uploaded to the driver pod as a Kubernetes secret. Specify this as a path as opposed to a URI (i.e. do not provide a scheme). In client mode, use instead. |
| (none) | OAuth token to use when authenticating against the Kubernetes API server from the driver pod when requesting executors. Note that unlike the other authentication options, this must be the exact string value of the token to use for the authentication. This token value is uploaded to the driver pod as a Kubernetes secret. In client mode, use instead. |
| (none) | Path to the OAuth token file containing the token to use when authenticating against the Kubernetes API server from the driver pod when requesting executors. Note that unlike the other authentication options, this file must contain the exact string value of the token to use for the authentication. This token value is uploaded to the driver pod as a secret. In client mode, use instead. |
| (none) | Path to the CA cert file for connecting to the Kubernetes API server over TLS from the driver pod when requesting executors. This path must be accessible from the driver pod. Specify this as a path as opposed to a URI (i.e. do not provide a scheme). In client mode, use instead. |
| (none) | Path to the client key file for authenticating against the Kubernetes API server from the driver pod when requesting executors. This path must be accessible from the driver pod. Specify this as a path as opposed to a URI (i.e. do not provide a scheme). In client mode, use instead. |
| (none) | Path to the client cert file for authenticating against the Kubernetes API server from the driver pod when requesting executors. This path must be accessible from the driver pod. Specify this as a path as opposed to a URI (i.e. do not provide a scheme). In client mode, use instead. |
| (none) | Path to the file containing the OAuth token to use when authenticating against the Kubernetes API server from the driver pod when requesting executors. This path must be accessible from the driver pod. Note that unlike the other authentication options, this file must contain the exact string value of the token to use for the authentication. In client mode, use instead. |
| | Service account that is used when running the driver pod. The driver pod uses this service account when requesting executor pods from the API server. Note that this cannot be specified alongside a CA cert file, client key file, client cert file, and/or OAuth token. In client mode, use instead. |
| (none) | In client mode, path to the CA cert file for connecting to the Kubernetes API server over TLS when requesting executors. Specify this as a path as opposed to a URI (i.e. do not provide a scheme). |
| (none) | In client mode, path to the client key file for authenticating against the Kubernetes API server when requesting executors. Specify this as a path as opposed to a URI (i.e. do not provide a scheme). |
| (none) | In client mode, path to the client cert file for authenticating against the Kubernetes API server when requesting executors. Specify this as a path as opposed to a URI (i.e. do not provide a scheme). |
| (none) | In client mode, the OAuth token to use when authenticating against the Kubernetes API server when requesting executors. Note that unlike the other authentication options, this must be the exact string value of the token to use for the authentication. |
| (none) | In client mode, path to the file containing the OAuth token to use when authenticating against the Kubernetes API server when requesting executors. |
| (none) | Add the label specified by to the driver pod. For example, . Note that Spark also adds its own labels to the driver pod for bookkeeping purposes. |
| (none) | Add the annotation specified by to the driver pod. For example, . |
| (none) | Add the label specified by to the executor pods. For example, . Note that Spark also adds its own labels to the driver pod for bookkeeping purposes. |
| (none) | Add the annotation specified by to the executor pods. For example, . |
| (none) | 驱动程序 Pod 的名称。在集群模式下，如果未设置此选项，则驱动程序 pod 名称将设置为“spark.app.name”并添加当前时间戳后缀，以避免名称冲突。在客户端模式下，如果您的应用程序在 pod 内运行，强烈建议将其设置为您的驱动程序运行所在的 pod 的名称。在客户端模式下设置此值允许驱动程序成为其执行程序 pod 的所有者，这反过来又允许集群对执行器 Pod 进行垃圾收集。 |
| | Number of times that the driver will try to ascertain the loss reason for a specific executor. The loss reason is used to ascertain whether the executor failure is due to a framework or an application error which in turn decides whether the executor is removed and replaced, or placed into a failed state for debugging. |
| | 在集群模式下，是否等待应用程序完成后再退出启动器进程。When changed to false, the launcher has a “fire-and-forget” behavior when launching the Spark job. |
| | Interval between reports of the current Spark job status in cluster mode. |
| (none) | 指定driver pod 的CPU限制。 |
| (none) | 指定每个执行器pod的CPU请求。其值符合Kubernetes的规范。示例值包括0.1、500m、1.5、5等，CPU单位的定义详见CPU单位文档。这与spark.executor.cores不同：它仅在设置了执行器pod的CPU请求时使用，并优先于spark.executor.cores。这不影响任务并行性，例如，执行器可以同时运行的任务数量不受此影响。 |
| (none) | 为executor pod 指定CPU限制。 |
| (none) | Adds to the node selector of the driver pod and executor pods, with key and the value as the configuration’s value. For example, setting to will result in the driver pod and executors having a node selector with key and value . Multiple node selector keys can be added by setting multiple configurations with this prefix. |
| (none) | Add the environment variable specified by to the Driver process. The user can specify multiple of these to set multiple environment variables. |
| (none) | Add the Kubernetes Secret named to the driver pod on the path specified in the value. For example, . |
| (none) | Add the Kubernetes Secret named to the executor pod on the path specified in the value. For example, . |
| (none) | Add as an environment variable to the driver container with name EnvName (case sensitive), the value referenced by key in the data of the referenced Kubernetes Secret. For example, . |
| (none) | Add as an environment variable to the executor container with name EnvName (case sensitive), the value referenced by key in the data of the referenced Kubernetes Secret. For example, . |
| (none) | Add the Kubernetes Volume named of the type to the driver pod on the path specified in the value. For example, . |
| (none) | Specify if the mounted volume is read only or not. For example, . |
| (none) | Configure Kubernetes Volume options passed to the Kubernetes with as key having specified value, must conform with Kubernetes option format. For example, . |
| (none) | Add the Kubernetes Volume named of the type to the executor pod on the path specified in the value. For example, . |
| false | Specify if the mounted volume is read only or not. For example, . |
| (none) | Configure Kubernetes Volume options passed to the Kubernetes with as key having specified value. For example, . |
| | This sets the Memory Overhead Factor that will allocate memory to non-JVM memory, which includes off-heap memory allocations, non-JVM tasks, and various systems processes. For JVM-based jobs this value will default to 0.10 and 0.40 for non-JVM jobs. This is done as non-JVM tasks need more non-JVM heap space and such tasks commonly fail with “Memory Overhead Exceeded” errors. This prempts this error with a higher default. |
| | Python版本，如果使用python的话 |
日期：2024-10-31
移动：http://kaire.xrbh.cn/quote/6536.html