Kerberos Authentication Setup

Kerberos is the foundation of ODP security. Every service principal, every HDFS block access, and every JDBC connection runs through Kerberos tickets. This guide covers the end-to-end setup from infrastructure prerequisites to verifying a working realm.

Prerequisites

DNS — mandatory, no exceptions

Kerberos relies on fully-qualified domain names for both forward and reverse resolution. Before touching any KDC or Ambari wizard:

• Every node must resolve every other node by FQDN.
• Reverse PTR records must match the forward A records.
• Hostnames must be consistent: hostname -f on each node must return the FQDN used in the DNS zone.

Test from every node before proceeding:

# Forward
dig +short master01.dev01.hadoop.clemlab.com
# Reverse
dig +short -x 192.168.10.11

If either lookup fails or returns a different name, stop and fix DNS first.

NTP — clock skew kills Kerberos tickets

Kerberos tickets carry validity windows. By default, a clock difference of more than 5 minutes between client and KDC causes authentication to fail with KRB5KRB_AP_ERR_SKEW.

# RHEL 9 / Rocky 9 — chrony is the default NTP daemon
timedatectl status
chronyc tracking

All nodes must synchronise to the same NTP source. Configure /etc/chrony.conf if needed and restart chronyd.

---

Choosing a KDC: MIT KDC vs FreeIPA

Criterion	MIT KDC	FreeIPA
Scope	KDC only	KDC + LDAP + DNS + CA + Web UI
ODP recommendation	Minimal deployments	Recommended for ODP
User directory	External LDAP required for Ranger sync	Built-in 389-DS LDAP
Certificate authority	Manual	Dogtag CA (integrated)
DNS	Manual	Integrated BIND
Ambari wizard support	Yes	Yes

ODP recommends FreeIPA because it eliminates the need to deploy and integrate a separate LDAP server (required by Ranger for user/group sync), a separate CA (required for TLS certificates), and a separate DNS server — all in a single, well-tested package.

---

Installing FreeIPA Server

A dedicated FreeIPA server (not co-located with Hadoop services) is the recommended topology.

# RHEL 9 / Rocky 9
dnf install -y freeipa-server freeipa-server-dns

# Run the installer — adjust realm, domain, and IPs to your environment
ipa-server-install \
  --realm=DEV01.HADOOP.CLEMLAB.COM \
  --domain=dev01.hadoop.clemlab.com \
  --ds-password=<directory-manager-password> \
  --admin-password=<ipa-admin-password> \
  --hostname=ipa01.dev01.hadoop.clemlab.com \
  --ip-address=192.168.10.5 \
  --setup-dns \
  --forwarder=8.8.8.8 \
  --no-ntp \
  --mkhomedir \
  --unattended

After installation, enroll every Hadoop node as an IPA client:

# On each Hadoop node
dnf install -y freeipa-client

ipa-client-install \
  --server=ipa01.dev01.hadoop.clemlab.com \
  --domain=dev01.hadoop.clemlab.com \
  --realm=DEV01.HADOOP.CLEMLAB.COM \
  --principal=admin \
  --password=<ipa-admin-password> \
  --mkhomedir \
  --unattended

---

Creating the Hadoop Realm

With FreeIPA, the realm is created automatically during ipa-server-install. The realm name must be the uppercase version of the domain:

• Domain: dev01.hadoop.clemlab.com
• Realm: DEV01.HADOOP.CLEMLAB.COM

Verify Kerberos configuration on each node:

cat /etc/krb5.conf
# Should show [libdefaults] default_realm = DEV01.HADOOP.CLEMLAB.COM

---

Ambari Kerberos Wizard (Automated Principal Creation)

Ambari includes a Kerberos Wizard that automates the creation of all service principals and keytabs. This is the recommended method for ODP clusters.

Steps

1. In the Ambari UI (https://<ambari-host>:8442), navigate to Admin → Kerberos.
2. Click Enable Kerberos.
3. Select Existing IPA (or Existing MIT KDC if applicable).
4. Enter KDC host, realm, and admin credentials.
5. Ambari verifies connectivity, then generates all principals and distributes keytabs automatically.
6. Review the principal list and click Next through the wizard.
7. Ambari restarts all services after keytab distribution.

Ambari creates principals following the pattern <service>/<fqdn>@REALM and stores keytabs under /etc/security/keytabs/ on each node.

---

Manual Principal Creation (kadmin)

If you need to create principals outside the wizard (e.g., for application service accounts):

# Connect to kadmin — from an IPA admin session or MIT KDC admin
kadmin -p admin@DEV01.HADOOP.CLEMLAB.COM

# Within kadmin prompt:
addprinc -randkey hdfs/master01.dev01.hadoop.clemlab.com@DEV01.HADOOP.CLEMLAB.COM
addprinc -randkey HTTP/master01.dev01.hadoop.clemlab.com@DEV01.HADOOP.CLEMLAB.COM
addprinc -randkey yarn/master01.dev01.hadoop.clemlab.com@DEV01.HADOOP.CLEMLAB.COM

# Export keytab
ktadd -k /etc/security/keytabs/hdfs.headless.keytab \
  hdfs/master01.dev01.hadoop.clemlab.com@DEV01.HADOOP.CLEMLAB.COM

quit

With FreeIPA, use ipa commands for bulk operations:

ipa service-add hdfs/master01.dev01.hadoop.clemlab.com
ipa-getkeytab -s ipa01.dev01.hadoop.clemlab.com \
  -p hdfs/master01.dev01.hadoop.clemlab.com@DEV01.HADOOP.CLEMLAB.COM \
  -k /etc/security/keytabs/hdfs.headless.keytab

---

Keytab Distribution

Keytabs are sensitive binary files that act as password equivalents. Handle them carefully:

# Set correct ownership and permissions after distribution
chown hdfs:hadoop /etc/security/keytabs/hdfs.headless.keytab
chmod 400 /etc/security/keytabs/hdfs.headless.keytab

# Verify keytab content
klist -kt /etc/security/keytabs/hdfs.headless.keytab

Ambari manages keytab permissions automatically when using the Kerberos Wizard. For manually created keytabs, ensure the file is owned by the service OS user with mode 0400.

---

Verifying Kerberos with kinit

After setup, verify that authentication works end-to-end:

# Test with a user principal
kinit user@DEV01.HADOOP.CLEMLAB.COM
klist

# Test with a service keytab
kinit -kt /etc/security/keytabs/hdfs.headless.keytab \
  hdfs/master01.dev01.hadoop.clemlab.com@DEV01.HADOOP.CLEMLAB.COM
klist

# Verify HDFS access under Kerberos
hdfs dfs -ls /

For HDFS NameNode principal verification:

hdfs dfsadmin -report

---

Troubleshooting

Clock skew error

KrbException: KDC has no support for encryption type (14) - PREAUTH_FAILED
# or
KRB5KRB_AP_ERR_SKEW Clock skew too great

Fix: Synchronise clocks immediately on all nodes and the KDC.

chronyc makestep   # Force immediate NTP sync
timedatectl status

DNS mismatch

KrbException: Cannot locate KDC for requested realm

Fix: Verify /etc/resolv.conf points to the FreeIPA/DNS server. Verify that hostname -f on each node returns an FQDN that resolves forward and reverse.

Keytab principal mismatch

KrbException: Client not found in Kerberos database

Fix: Run klist -kt <keytab> to confirm the principal name. Ensure it matches exactly what is in the Ambari service configuration.

Ticket expired

GSSException: No valid credentials provided (Mechanism level: Fail to create credential)

Fix: Renew or re-acquire the ticket:

kinit -R   # Renew within renewal window
# or
kinit -kt /etc/security/keytabs/hdfs.headless.keytab \
  hdfs/<fqdn>@REALM

Ambari's kinit_cmd cron runs automatically for service keytabs when the cluster is managed by Ambari.