Security
ZooKeeper's security model — authentication schemes, per-znode ACLs, TLS encryption, and network security best practices for production deployments.
Table of Contents
Authentication
By default, ZooKeeper has NO authentication — any client that can reach port 2181 can read and write any znode. In production, you must enable authentication to control who can access the ensemble. ZooKeeper supports multiple authentication schemes.
| Scheme | How It Works | Security Level | Use Case |
|---|---|---|---|
| world | No auth — anyone can access | ❌ None | Development only |
| digest | Username:password (SHA1 hash) | ⚠️ Medium | Simple production setups |
| SASL/Kerberos | Kerberos tickets via SASL | ✅ High | Enterprise environments |
| x509 | Client TLS certificates | ✅ High | Certificate-based infrastructure |
| ip | Source IP address | ⚠️ Low (spoofable) | Network-level restriction |
# Digest Authentication (username:password) # ───────────────────────────────────────────────────────── # Client-side: add auth info to session zk.addAuthInfo("digest", "admin:secretpass".getBytes()); # The password is SHA1-hashed before transmission. # ZooKeeper stores: admin:BASE64(SHA1(admin:secretpass)) # Server-side: no config needed for digest (built-in) # ACLs reference the digest scheme: # digest:admin:hashedpass → rwcda # ───────────────────────────────────────────────────────── # SASL/Kerberos Authentication # ───────────────────────────────────────────────────────── # Server zoo.cfg: authProvider.1=org.apache.zookeeper.server.auth.SASLAuthenticationProvider requireClientAuthScheme=sasl # Server JAAS config (java.security.auth.login.config): Server { com.sun.security.auth.module.Krb5LoginModule required useKeyTab=true keyTab="/etc/security/keytabs/zookeeper.keytab" storeKey=true useTicketCache=false principal="zookeeper/zk1.prod@REALM.COM"; }; # Client JAAS config: Client { com.sun.security.auth.module.Krb5LoginModule required useKeyTab=true keyTab="/etc/security/keytabs/client.keytab" storeKey=true useTicketCache=false principal="kafka/broker1.prod@REALM.COM"; }; # ───────────────────────────────────────────────────────── # x509 Certificate Authentication # ───────────────────────────────────────────────────────── # Uses the client's TLS certificate CN as the identity # Requires TLS to be enabled (see Encryption section) # ACLs reference: x509:CN=kafka-broker-1
SASL for Production
For production environments, SASL with Kerberos provides the strongest authentication. It integrates with existing enterprise identity systems, doesn't transmit passwords, and supports mutual authentication (server proves identity to client too). Digest auth is acceptable for simpler setups where Kerberos infrastructure doesn't exist.
ACLs
ACLs (Access Control Lists) are ZooKeeper's authorization mechanism. Each znode has its own ACL that specifies which authenticated identities can perform which operations. ACLs are NOT inherited — each znode must have its own ACL set explicitly.
ACL Structure: scheme:id → permissions Schemes: world:anyone → applies to all clients (no auth needed) auth: → applies to any authenticated client digest:user:hash → specific user with password hash ip:10.0.1.0/24 → IP address or CIDR range sasl:principal → Kerberos principal Permissions (5 bits): r = READ → getData, getChildren w = WRITE → setData c = CREATE → create children d = DELETE → delete children a = ADMIN → setACL (change permissions) Examples: world:anyone → r # Everyone can read, no one can write digest:admin:hash → rwcda # Admin has full access ip:10.0.1.0/24 → rw # Internal network can read/write sasl:kafka → rwcd # Kafka service can do everything except setACL Setting ACLs: // At creation time zk.create("/secrets/db-password", data, [new ACL(Perms.ALL, new Id("digest", "admin:hash"))], CreateMode.PERSISTENT); // After creation zk.setACL("/secrets/db-password", [new ACL(Perms.READ, new Id("digest", "app:hash")), new ACL(Perms.ALL, new Id("digest", "admin:hash"))], aclVersion);
ACL Best Practices
- ✅Never use world:anyone in production — always require authentication
- ✅Use the principle of least privilege — give only the permissions each client needs
- ✅Separate admin ACLs from application ACLs — apps shouldn't be able to change permissions
- ✅ACLs are per-znode, NOT inherited — you must set them on every node explicitly
- ✅Use 'auth' scheme for convenience — applies to whoever is currently authenticated
- ✅Remember: ACLs control access to the znode, not its children (children have their own ACLs)
ACLs Are Not Inherited
Unlike filesystem permissions, ZooKeeper ACLs do NOT cascade to children. If /app has ACL [admin:rwcda], a child /app/config created without explicit ACLs gets the default (world:anyone:rwcda). You must set ACLs on every znode individually. Use helper functions to apply consistent ACLs across your namespace.
Encryption
ZooKeeper supports TLS encryption for both client-to-server and server-to-server communication. Without TLS, all data (including authentication credentials) is transmitted in plaintext.
| Connection Type | Port | TLS Config | Purpose |
|---|---|---|---|
| Client → Server | 2181 (or secureClientPort) | ssl.keyStore, ssl.trustStore | Encrypt client data and auth |
| Server → Server (follower) | 2888 | sslQuorum=true | Encrypt replication traffic |
| Server → Server (election) | 3888 | sslQuorum=true | Encrypt election messages |
# zoo.cfg — TLS Configuration # Client-to-server TLS secureClientPort=2281 # TLS-only client port ssl.keyStore.location=/etc/zk/keystore.jks ssl.keyStore.password=changeit ssl.trustStore.location=/etc/zk/truststore.jks ssl.trustStore.password=changeit # Optional: require client certificates (mutual TLS) ssl.clientAuth=need # "need" = required, "want" = optional # Server-to-server TLS (quorum) sslQuorum=true ssl.quorum.keyStore.location=/etc/zk/keystore.jks ssl.quorum.keyStore.password=changeit ssl.quorum.trustStore.location=/etc/zk/truststore.jks ssl.quorum.trustStore.password=changeit # Disable non-TLS client port (force encryption) # Comment out: clientPort=2181 # Only secureClientPort=2281 is available # TLS protocol and cipher configuration ssl.protocol=TLSv1.2 # Minimum TLS version ssl.enabledProtocols=TLSv1.2,TLSv1.3 ssl.ciphersuites=TLS_AES_256_GCM_SHA384,TLS_AES_128_GCM_SHA256
Always Enable TLS in Production
Without TLS, digest authentication passwords are sent as plaintext (only hashed, not encrypted). Anyone sniffing the network can capture credentials and replay them. Enable TLS for both client and quorum connections. Use mutual TLS (client certificates) for the strongest security posture.
Network Security
Beyond authentication and encryption, network-level security controls are essential for protecting ZooKeeper in production. Defense in depth means multiple layers of protection.
Network Security Checklist
- ✅Firewall port 2181 — only allow known application server IPs
- ✅Firewall ports 2888/3888 — only allow ensemble member IPs
- ✅Disable JMX remote access or restrict to monitoring network only
- ✅Use private network interfaces for inter-server communication
- ✅Disable four letter words you don't need (4lw.commands.whitelist)
- ✅Place ZooKeeper in a dedicated network segment (VLAN or subnet)
- ✅Use network policies (Kubernetes) or security groups (AWS) for access control
- ✅Monitor for unexpected connection sources (cons command)
# Firewall rules (iptables example) # Allow client connections only from app servers iptables -A INPUT -p tcp --dport 2181 -s 10.0.1.0/24 -j ACCEPT iptables -A INPUT -p tcp --dport 2181 -j DROP # Allow quorum traffic only between ZK nodes iptables -A INPUT -p tcp --dport 2888 -s 10.0.0.1 -j ACCEPT iptables -A INPUT -p tcp --dport 2888 -s 10.0.0.2 -j ACCEPT iptables -A INPUT -p tcp --dport 2888 -s 10.0.0.3 -j ACCEPT iptables -A INPUT -p tcp --dport 2888 -j DROP # Same for election port iptables -A INPUT -p tcp --dport 3888 -s 10.0.0.1 -j ACCEPT iptables -A INPUT -p tcp --dport 3888 -s 10.0.0.2 -j ACCEPT iptables -A INPUT -p tcp --dport 3888 -s 10.0.0.3 -j ACCEPT iptables -A INPUT -p tcp --dport 3888 -j DROP # Restrict four letter words (zoo.cfg) 4lw.commands.whitelist=ruok,mntr,srvr # Only enable what monitoring needs — disable dump, cons, etc. # JMX security (if enabled) -Dcom.sun.management.jmxremote.authenticate=true -Dcom.sun.management.jmxremote.ssl=true -Dcom.sun.management.jmxremote.access.file=/etc/zk/jmxremote.access
Defense in Depth
Don't rely on a single security layer. Combine: (1) Network firewalls (who can connect), (2) TLS encryption (data in transit), (3) Authentication (who is this client), (4) ACLs (what can they do). Even if one layer is compromised, the others provide protection.
Interview Questions
Q:How does ZooKeeper's ACL system work? How is it different from filesystem permissions?
A: ZooKeeper ACLs are per-znode access control lists with scheme:id → permissions format. Permissions are: read (r), write (w), create children (c), delete children (d), admin/setACL (a). Key difference from filesystems: ACLs are NOT inherited. Each znode has its own independent ACL — a parent's ACL doesn't apply to children. Schemes include: world (no auth), digest (user:password), sasl (Kerberos), ip (source address), x509 (certificate CN). You must explicitly set ACLs on every znode you want to protect.
Q:What authentication options does ZooKeeper support?
A: (1) None/world — default, no authentication (development only). (2) Digest — username:password, SHA1 hashed. Simple but passwords sent in cleartext without TLS. (3) SASL/Kerberos — enterprise-grade, ticket-based, no passwords on wire. Integrates with Active Directory/LDAP. (4) x509 — client TLS certificates, uses certificate CN as identity. Requires mutual TLS. (5) IP-based — restricts by source IP/CIDR. Weak (spoofable) but useful as additional layer. For production: SASL/Kerberos or x509 with TLS enabled.
Q:Why is TLS important for ZooKeeper and what does it protect?
A: Without TLS: (1) Digest passwords are sent as plaintext hashes — sniffable and replayable. (2) All znode data (potentially sensitive config, secrets) is visible on the network. (3) Inter-server replication traffic is unencrypted — an attacker could inject fake proposals. TLS protects: client-to-server (port 2181/2281), follower-to-leader (2888), and election (3888). Mutual TLS (client certificates) provides both encryption and authentication in one mechanism. Always enable TLS in production — it's the foundation that makes other security mechanisms meaningful.
Common Mistakes
Running without authentication in production
Leaving the default 'world:anyone' ACL on all znodes. Any client that can reach port 2181 has full read/write access to all coordination data.
✅Enable SASL or digest authentication. Set restrictive ACLs on all znodes. Use requireClientAuthScheme=sasl in zoo.cfg to reject unauthenticated connections entirely.
Using digest auth without TLS
Digest authentication hashes the password but sends it in cleartext. Anyone sniffing the network can capture the hash and replay it to authenticate.
✅Always enable TLS when using digest authentication. Better yet, use SASL/Kerberos which doesn't transmit passwords at all, or x509 which uses certificates.
Assuming ACLs are inherited
Setting strict ACLs on /app and assuming /app/config and /app/secrets are also protected. They're not — each znode has independent ACLs.
✅Explicitly set ACLs on every znode at creation time. Write helper functions that apply your standard ACL policy. Audit existing znodes for overly permissive ACLs.
Exposing ZooKeeper to the public internet
Not firewalling port 2181, allowing anyone on the internet to connect. Combined with no authentication, this gives full access to your coordination data.
✅Firewall all ZK ports (2181, 2888, 3888) to only allow known IPs. Place ZK in a private subnet. Use security groups or network policies. Never expose ZK to the public internet.