To avoid connection storms, a randomization factor of 0.2 will be applied to the timeout resulting in a random range between 20% below and 20% above the computed value. Currently applies only to OAUTHBEARER. OAuth 2.0 client authentication flows, 4.10.5.1. Examples: RACK1, us-east-1d. The window of time a metrics sample is computed over. This is similar to the producer request timeout. The SO_SNDBUF buffer of the socket server sockets. MBeans matching kafka.connect:type=task-error-metrics,connector=*,task=*, 16.9.1. JAAS login context parameters for SASL connections in the format used by JAAS configuration files. Leave this config undefined or empty for Zookeeper clusters.
Type: classDefault: nullImportance: mediumDynamic update: read-only. MBeans matching kafka.streams:type=stream-[store-scope]-metrics,client-id=*,task-id=*,[store-scope]-id=*, 16.9.5. If this minimum cannot be met, then the producer will raise an exception (either NotEnoughReplicas or NotEnoughReplicasAfterAppend). Default value 1 day. This is typically bumped after all brokers were upgraded to a new version. 'TLS', 'TLSv1.1', 'SSL', 'SSLv2' and 'SSLv3' may be supported in older JVMs, but their usage is discouraged due to known security vulnerabilities. Type: intDefault: 9000 (9 seconds)Importance: mediumDynamic update: read-only. Login uses an exponential backoff algorithm with an initial wait based on the sasl.login.retry.backoff.ms setting and will double in wait length between attempts up to a maximum wait length specified by the sasl.login.retry.backoff.max.ms setting. Configuring OAuth 2.0 authentication", Collapse section "4.10.6. Also unlike listeners, there can be duplicated ports in this property, so that one listener can be configured to advertise another listeners address. Enabling Client-to-server authentication using DIGEST-MD5, 4.8.2. Specify hostname as 0.0.0.0 to bind to all interfaces. The controller would trigger a leader balance if it goes above this value per broker. Listener names and port numbers must be unique. Using MirrorMaker 2.0 in legacy mode, 9.1. Type: longDefault: 0Importance: mediumDynamic update: read-only. Simple ACL authorizer", Collapse section "4.7.1. Using AMQ Streams with MirrorMaker 2.0", Collapse section "8. The largest record batch size allowed by Kafka (after compression if compression is enabled). Configuring OPA authorization support, 4.13.1. The maximum time before a new log segment is rolled out (in hours), secondary to log.roll.ms property. Overrides any explicit value set via the zookeeper.ssl.keyStore.type system property (note the camelCase). Enable automatic broker id generation on the server. The frequency with which the high watermark is saved out to disk. Default is GSSAPI. For standalone consumers (using manual assignment), offsets will be expired after the time of last commit plus this retention period. This is used by the broker to find the preferred read replica. Data storage considerations", Expand section "3. Type: longDefault: 20971520Valid Values: [1,]Importance: highDynamic update: read-only. If the config for the listener name is not set, the config will fallback to the generic config (i.e. With the default value for Java 11, clients and servers will prefer TLSv1.3 if both support it and fallback to TLSv1.2 otherwise (assuming both support at least TLSv1.2). Legal values are between 0 and 900 (15 minutes); a default value of 60 (1 minute) is used if no value is specified. Valid policies are: "delete" and "compact". The store password for the key store file. The maximum jitter to subtract from logRollTimeMillis (in milliseconds). If the listener name is not a security protocol, listener.security.protocol.map must also be set. Truststore password when using TLS connectivity to ZooKeeper. This configuration is ignored if log.message.timestamp.type=LogAppendTime.The maximum timestamp difference allowed should be no greater than log.retention.ms to avoid unnecessarily frequent log rolling. Type: intDefault: 2000 (2 seconds)Importance: lowDynamic update: read-only. Maximum time in milliseconds before starting new elections. Delete topic through the admin tool will have no effect if this config is turned off. The maximum number of consumers that a single consumer group can accommodate. Using Kerberos (GSSAPI) authentication, 11.1. For example: 1@localhost:9092,2@localhost:9093,3@localhost:9094. Specifying a maximum number of tasks, 8.7.
When initially registering with the controller quorum, the number of milliseconds to wait before declaring failure and exiting the broker process. Type: stringDefault: nullImportance: mediumDynamic update: per-broker. The maximum size in bytes of the offset index. Type: intDefault: 25Importance: mediumDynamic update: read-only. A comma-separated list of the names of the listeners used by the controller. becomes empty) its offsets will be kept for this retention period before getting discarded. Different security (SSL and SASL) settings can be configured for each listener by adding a normalised prefix (the listener name is lowercased) to the config name. Type: stringDefault: HTTPSImportance: lowDynamic update: read-only. A list of classes to use as metrics reporters. Important Kafka broker metrics", Expand section "16.8. Distributed tracing", Expand section "13.2. Type: intDefault: 10Valid Values: [1,]Importance: lowDynamic update: read-only.
MBeans matching kafka.consumer:type=consumer-metrics,client-id=*, 16.7.2. Tuning Kafka configuration", Collapse section "6.1. If an authentication request is received for a JWT that includes a "kid" header claim value that isnt yet in the cache, the JWKS endpoint will be queried again on demand. The max time that the client waits to establish a connection to zookeeper. Synchronizing data between Kafka clusters using MirrorMaker 2.0, 8.8. The interval at which to rollback transactions that have timed out. Enabling SASL SCRAM authentication, 4.10. Type: stringDefault: PLAINTEXT://:9092Importance: highDynamic update: per-broker. The number of bytes of messages to attempt to fetch for each partition. The socket timeout for controller-to-broker channels. This can be useful in some cases where external load balancers are used.
A list of rules for mapping from distinguished name from the client certificate to short name. The maximum size for a metadata entry associated with an offset commit. Enabling TLS client authentication, 4.9.6. Compression codec for the offsets topic - compression may be used to achieve "atomic" commits. Type: longDefault: 15000 (15 seconds)Valid Values: [0,]Importance: mediumDynamic update: cluster-wide. ZooKeeper authentication", Collapse section "4.6. Increasing bandwidth for high latency connections, 6.1.1.6. Overridden min.insync.replicas config for the transaction topic. The name of the security provider used for SSL connections. The JWT will be inspected for the standard OAuth "aud" claim and if this value is set, the broker will match the value from JWTs "aud" claim to see if there is an exact match. Configuring Kafka Java clients to use OAuth 2.0, 4.11. Login uses an exponential backoff algorithm with an initial wait based on the sasl.login.retry.backoff.ms setting and will double in wait length between attempts up to a maximum wait length specified by the sasl.login.retry.backoff.max.ms setting. Fast local JWT token validation configuration, 4.10.2.4. Overrides any explicit value set via the zookeeper.ssl.trustStore.password system property (note the camelCase). Type: classDefault: org.apache.kafka.common.security.authenticator.DefaultKafkaPrincipalBuilderImportance: mediumDynamic update: per-broker. The least recently used connection on another listener will be closed in this case. The maximum difference allowed between the timestamp when a broker receives a message and the timestamp specified in the message. The fully qualified name of a SASL client callback handler class that implements the AuthenticateCallbackHandler interface. The rules are evaluated in order and the first rule that matches a principal name is used to map it to a short name.
The (optional) value in seconds to allow for differences between the time of the OAuth/OIDC identity provider and the broker. Overrides any explicit value set via the zookeeper.ssl.enabledProtocols system property (note the camelCase). Cruise Control for cluster rebalancing, 12.2. Type: stringDefault: PKIXImportance: mediumDynamic update: per-broker. Private key in the format specified by 'ssl.keystore.type'. Internal topic creation will fail until the cluster size meets this replication factor requirement. This determines the number of retries when such failure happens. Examples of legal listener lists: PLAINTEXT://myhost:9092,SSL://:9091 CLIENT://0.0.0.0:9092,REPLICATION://localhost:9093. Only applicable for logs that are being compacted. The configuration controls the maximum amount of time the client will wait for the response of a request. Overrides any explicit value set via the zookeeper.ssl.keyStore.location system property (note the camelCase). Topic configuration synchronization, 8.2.6.
Type: listDefault: DEFAULTImportance: mediumDynamic update: per-broker. New connections are blocked if either the listener or broker limit is reached. Encryption and authentication", Expand section "4.10. Max number that can be used for a broker.id. Currently applies only to OAUTHBEARER. Using OAuth 2.0 token-based authorization, 4.11.1. Number of fetcher threads used to replicate messages from a source broker. Must be at least 1024. In general, the default (-1) should not be overridden. Enable the log cleaner process to run on the server. Type: intDefault: 10485760 (10 mebibytes)Valid Values: [4,]Importance: mediumDynamic update: cluster-wide. Overrides any explicit value set via the zookeeper.ssl.trustStore.type system property (note the camelCase). Kafka consumer configuration tuning", Expand section "6.1.3.5. New connections from the ip address are dropped if the limit is reached. Using OAuth 2.0 token-based authorization", Collapse section "4.11. Dynamically change logging levels for Kafka broker loggers, 6.1.1.2. The (optional) value in milliseconds for the initial wait between login attempts to the external authentication provider. The value should be either CreateTime or LogAppendTime. Managing logs with data retention policies, 6.1.1.7. Scan interval to remove expired delegation tokens. The maximum number of unacknowledged requests the client will send to Zookeeper before blocking.
Overview of AMQ Streams", Expand section "2.4. Currently applies only to OAUTHBEARER. The default value of null means the enabled protocol will be the value of the zookeeper.ssl.protocol configuration property. Type: stringDefault: nullImportance: lowDynamic update: per-broker. If this is not set, the value for listeners will be used.
Note that this configuration is ignored if an extension of KafkaPrincipalBuilder is provided by the principal.builder.class configuration.
Type: intDefault: 11Valid Values: [1,]Importance: lowDynamic update: read-only. The log cleaner will be throttled so that the sum of its read and write i/o will be less than this value on average. This is required only when the secret is updated. Type: intDefault: 524288Valid Values: [0,]Importance: mediumDynamic update: cluster-wide. Using AMQ Streams with MirrorMaker 2.0", Expand section "8.2. Type: stringDefault: 3.0-IV1Valid Values: [0.8.0, 0.8.1, 0.8.2, 0.9.0, 0.10.0-IV0, 0.10.0-IV1, 0.10.1-IV0, 0.10.1-IV1, 0.10.1-IV2, 0.10.2-IV0, 0.11.0-IV0, 0.11.0-IV1, 0.11.0-IV2, 1.0-IV0, 1.1-IV0, 2.0-IV0, 2.0-IV1, 2.1-IV0, 2.1-IV1, 2.1-IV2, 2.2-IV0, 2.2-IV1, 2.3-IV0, 2.3-IV1, 2.4-IV0, 2.4-IV1, 2.5-IV0, 2.6-IV0, 2.7-IV0, 2.7-IV1, 2.7-IV2, 2.8-IV0, 2.8-IV1, 3.0-IV0, 3.0-IV1, 3.1-IV0]Importance: mediumDynamic update: read-only.
The number of samples to retain in memory for client quotas. A typical scenario would be to create a topic with a replication factor of 3, set min.insync.replicas to 2, and produce with acks of "all". The metrics polling interval (in seconds) which can be used in kafka.metrics.reporters implementations. Defaults to false if neither is set; when true, zookeeper.clientCnxnSocket must be set (typically to org.apache.zookeeper.ClientCnxnSocketNetty); other values to set may include zookeeper.ssl.cipher.suites, zookeeper.ssl.crl.enable, zookeeper.ssl.enabled.protocols, zookeeper.ssl.endpoint.identification.algorithm, zookeeper.ssl.keystore.location, zookeeper.ssl.keystore.password, zookeeper.ssl.keystore.type, zookeeper.ssl.ocsp.enable, zookeeper.ssl.protocol, zookeeper.ssl.truststore.location, zookeeper.ssl.truststore.password, zookeeper.ssl.truststore.type. ZooKeeper authentication", Expand section "4.7.1. The amount of time to sleep when fetch partition error occurs. The number of milliseconds to keep a metadata log file or snapshot before deleting it. The amount of time the group coordinator will wait for more consumers to join a new group before performing the first rebalance. The number of minutes to keep a log file before deleting it (in minutes), secondary to log.retention.ms property. Overrides any explicit value set via the zookeeper.ssl.ciphersuites system property (note the single word "ciphersuites").
The directories in which the log data is kept. The connection setup timeout will increase exponentially for each consecutive connection failure up to this maximum. Idle connections timeout: the server socket processor threads close the connections that idle more than this.
How far a ZK follower can be behind a ZK leader. All then-current keys will be cached on the broker for incoming requests. Type: listDefault: ""Valid Values: non-empty listImportance: highDynamic update: read-only. Type: booleanDefault: trueImportance: mediumDynamic update: read-only. Setting up AMQ Streams to use Kerberos (GSSAPI) authentication, 12. OAuth 2.0 introspection endpoint configuration, 4.10.3. Should be enabled if using any topics with a cleanup.policy=compact including the internal offsets topic. For brokers, the config must be prefixed with listener prefix and SASL mechanism name in lower-case. Type: longDefault: -1Importance: highDynamic update: cluster-wide. Type: intDefault: 500Valid Values: [1,]Importance: highDynamic update: read-only. Setting up tracing for Kafka clients", Collapse section "13.2. The socket receive buffer for network requests. Type: intDefault: 1048576 (1 mebibyte)Valid Values: [0,]Importance: mediumDynamic update: read-only. Type: intDefault: 1073741824 (1 gibibyte)Valid Values: [12,]Importance: highDynamic update: read-only. Encryption and authentication", Collapse section "4.9.
Currently applies only to OAUTHBEARER. The amount of buffer time before credential expiration to maintain when refreshing a credential, in seconds. Specifies the enabled protocol(s) in ZooKeeper TLS negotiation (csv).
Note that producer ids may expire sooner if the last write from the producer id is deleted due to the topics retention settings. To allow connecting through other ZooKeeper nodes when that ZooKeeper machine is down you can also specify multiple hosts in the form hostname1:port1,hostname2:port2,hostname3:port3. Kafka Connect in distributed mode", Expand section "8. Legal values are between 0 and 0.25 (25%) inclusive; a default value of 0.05 (5%) is used if no value is specified. In IaaS environments, this may need to be different from the interface to which the broker binds. Minimum bytes expected for each fetch response. For brokers, login callback handler config must be prefixed with listener prefix and SASL mechanism name in lower-case. The create topic policy class that should be used for validation. Type: intDefault: 2147483647Valid Values: [1,]Importance: mediumDynamic update: read-only. Enable controlled shutdown of the server. Configuring OAuth 2.0 support for Kafka brokers, 4.10.6.3. The required acks before the commit can be accepted. For example, listener.name.sasl_ssl.scram-sha-256.sasl.login.class=com.example.CustomScramLogin. Adding the Kafka Streams API as a dependency to your Maven project, 11. Configuring OAuth 2.0 authorization support, 4.12. The URL can be HTTP(S)-based or file-based. Configuring Red Hat Single Sign-On as an OAuth 2.0 authorization server, 4.10.6.2. Increase visibility into IT operations to detect and resolve technical issues before they impact your business. For more details on the format please see security authorization and acls. The URL for the OAuth/OIDC identity provider. OAuth 2.0 Kafka broker configuration", Expand section "4.10.5. Specify the message format version the broker will use to append messages to the logs. Type: listDefault: ""Valid Values: [broker, controller]Importance: highDynamic update: read-only. Type: intDefault: 2000 (2 seconds)Importance: highDynamic update: read-only. Type: intDefault: 2Valid Values: [1,]Importance: lowDynamic update: read-only. Concretely, the user could define listeners with names INTERNAL and EXTERNAL and this property as: INTERNAL:SSL,EXTERNAL:SSL. The maximum record batch size accepted by the broker is defined via message.max.bytes (broker config) or max.message.bytes (topic config). Setting this value incorrectly will cause consumers with older versions to break as they will receive messages with a format that they dont understand. Type: intDefault: 1000Valid Values: [0,]Importance: mediumDynamic update: read-only. Indicates whether to enable replicas not in the ISR set to be elected as leader as a last resort, even though doing so may result in data loss. The fully qualified class name that implements ReplicaSelector. If the key is encrypted, key password must be specified using 'ssl.key.password'. Overrides any explicit value set via the same-named zookeeper.clientCnxnSocket system property. Enables auto leader balancing. Type: stringDefault: nullImportance: mediumDynamic update: read-only. Running Kafka Connect in standalone mode, 7.2.1. The maximum connection creation rate we allow in the broker at any time. Keystore location when using a client-side certificate with TLS connectivity to ZooKeeper. The list may contain any mechanism for which a security provider is available. Type: longDefault: -1Importance: highDynamic update: read-only. Kafka consumer configuration tuning, 6.1.3.2. Type: longDefault: 86400000 (1 day)Valid Values: [1,]Importance: mediumDynamic update: read-only. Batch size for reading from the transaction log segments when loading producer ids and transactions into the cache (soft-limit, overridden if records are too large). Connection close delay on failed authentication: this is the time (in milliseconds) by which connection close will be delayed on authentication failure. The minimum time a message will remain uncompacted in the log. The maximum number of bytes in a socket request.
In the event that the JWT includes a "kid" header value that isnt in the JWKS file, the broker will reject the JWT and authentication will fail. The default replication factors for automatically created topics. Maximum time without a successful fetch from the current leader before becoming a candidate and triggering a election for voters; Maximum time without receiving fetch from a majority of the quorum before asking around to see if theres a new epoch for leader. Using OAuth 2.0 token-based authentication", Expand section "4.10.1. Type: listDefault: nullImportance: lowDynamic update: read-only.