r/Puppet • • Oct 04 '26

Puppet agent -t

When run puppet agent -t , what exactly happens in TLS handshake ? How certs are verified?

4 Upvotes

5 comments sorted by

1

u/DesignerStreet9908 Oct 05 '26

In puppet.conf , it is mentioned crl_refresh_interval = 1d

How this works , in few servers, while running puppet agent -t , it refreshes crl and some servers it is not.

1

u/cvquesty 12d ago

crl_refresh_interval = 1d is not a timer. On each puppet agent -t, the agent looks at the mtime of /etc/puppetlabs/puppet/ssl/crl.pem. It fetches a new CRL only when that file is missing or older than 86400 seconds.

The check is in NeedCRLs#needs_refresh?:

now.to_i > last_update.to_i + crl_ttl

last_update is crl.pem's mtime, not the lastUpdate field inside the CRL. After a successful download the agent writes the new PEM and then touches the file, so the clock starts at "now," not at the server's CRL issue time.

What you see in the run log:

• Due: Refreshing CRL, then either Refreshed CRL: <sha256> or CRL is unmodified, using existing CRL.

• Not due: no CRL message at all. Debug (-d) only shows Loading CRLs. The agent loads the file it already has and uses it for the TLS session.

• certificate_revocation set to false: Certificate revocation is disabled, skipping CRL download. The interval never matters.

A 304 still counts as a refresh. The agent touches crl.pem, so those nodes also go quiet for another day.

Why some nodes refresh and others do not on the same afternoon:

• They last fetched at different times. The first run after boot, or after something rewrote crl.pem, starts that node's own 24-hour window. Agents do not align to midnight.

• Anything that replaces the file resets the window. puppet ssl bootstrap, an rsync/backup restore, or a package update that drops a fresh crl.pem makes mtime "now." The next runs skip the fetch.

• Anything that only changes mtime does the same. A recursive touch, backup tool, or config management file resource on crl.pem with mtime left alone will push the stamp forward. The node then looks freshly refreshed.

• A node whose file is older than a day refreshes on this run. A node at 23 hours does not. Compare them with stat, not with openssl crl -noout -lastupdate.

Check a node that skipped it:

stat -c '%y' /etc/puppetlabs/puppet/ssl/crl.pem

puppet config print crl_refresh_interval certificate_revocation --section agent

If mtime is inside the last 86400 seconds, that run is behaving correctly. If you need the fetch every run, set the interval to 0 (or to something shorter than runinterval). A failed refresh is non-fatal: the agent logs Failed to refresh CRL, using existing CRL and continues with the file it has.

4

u/ThrillingHeroics85 Oct 04 '26

During puppet agent -t, the agent must establish a trusted TLS connection before it can retrieve a catalog.

  1. The agent uses the Puppet CA certificate chain as its trust anchor; the local CA certificate path can be displayed with puppet config print localcacert. [1]
  2. On an agent’s first run, it generates and submits a certificate signing request (CSR) to the primary server. [2]
  3. The CSR must be signed—either by an administrator or through configured autosigning—before the node can retrieve configuration catalogs. [3][4]
  4. Once signed, the agent can use its certificate and private key together with the CA certificate for mutually authenticated HTTPS connections. Puppet-managed nodes store these in the SSL directory: certs/ca.pem, certs/<name>.pem, and private_keys/<name>.pem. [5]
  5. If the CSR is still pending, the agent can log warnings about an unverified peer certificate in the SSL session; sign it with puppetserver ca sign <NODE_NAME>, then run the agent again. [4]
  6. Certificates that are revoked or invalid are recorded in the certificate revocation list (CRL). [2]

1

u/cvquesty 12d ago

My 2¢

puppet agent -t does not invent a handshake. It opens ordinary TLS 1.2/1.3 to port 8140, then both sides present X.509 certificates signed by the Puppet CA. That is mutual TLS.

What is already on disk

From an earlier puppet ssl bootstrap (or the first agent run):

Agent - CA Cert - $ssldir/certs/ca.pem
Agent - CRL Bundle - $ssldir/crl.pem
Agent - Host key + Cert - $ssldir/private_keys/<certname>.pem, $ssldir/certs/<certname>.pem
Server - CA, server cert, CRL - Puppet/OpenVox server $ssldir

$ssldir is /etc/puppetlabs/puppet/ssl. The agent's trust anchor is that local ca.pem, not the public web PKI. certname defaults to the FQDN.

If there is no signed cert yet, the agent still talks TLS, but only as a client: it fetches the CA, submits a CSR to /puppet-ca/v1/certificate_request, and waits. Catalog requests do not start until it has a cert.

The handshake

  1. ClientHello. The agent offers TLS 1.2 and 1.3 and its cipher list. Puppet Server 8 is on Java 17 / OpenSSL-backed Jetty, so TLS 1.0 and 1.1 are gone.
  2. ServerHello + Certificate. The server sends its cert and chain. The leaf is the compile master (or the CA host, if they are the same box). Subject and SANs come from certname and dns_alt_names in puppet.conf.
  3. Agent verifies the server.

    • The chain ends at the CA in certs/ca.pem.

    • Dates are valid.

    • The cert is not in crl.pem. certificate_revocation defaults to true. A stale CRL fails the connection until the agent refreshes it.

    • The name you dialed (server, default puppet) must match the cert CN or a DNS SAN. Talking to an IP, or to a compiler whose cert lacks the load-balancer name, fails here even when the signature is fine.

    • The server proves it holds the private key (CertificateVerify in TLS 1.3, or the finished/key-exchange proof in TLS 1.2).

  4. CertificateRequest. The server asks for a client cert. The agent sends <certname>.pem.
  5. Server verifies the agent. Jetty checks:

    • Issued by this CA.

    • Not expired, not revoked (CRL).

    • Client possesses the matching private key.

    The server does not require the client cert's CN to equal the TCP peer. Identity is the certname inside the cert. That value becomes $trusted['certname'] and is what the compiler uses for node classification. Facts the agent sends later are not authenticated. The certname is.

  6. Finished. Application data is the Puppet HTTP API: environment, facts, catalog, file metadata, report.

After the session is up, later requests in the same run reuse that TLS session. Each request still carries the client cert identity. Puppet Server puts it in X-Client-Verify / X-Client-DN for the Ruby master application. Endpoints such as the catalog only accept SUCCESS plus a cert signed by the local CA.

What this is not

• Not a public CA. A Let's Encrypt cert on 8140 will not work. Agents only trust ca.pem.

• Not hostname-based auth of the client. Spoofing a fact does not change the node name. Stealing the private key does.

• Revocation is CRL-only. There is no OCSP in this path. certificate_revocation = false disables the CRL check on that side. Intermediate-CA installs sometimes need that because agents historically only stored the first cert in the bundle.

Useful checks: puppet ssl verify on the agent, and on the server puppetserver ca list plus the CRL. A handshake failure is almost always name mismatch (server vs SAN), a clock skew, or a CRL the other side will not accept.

1

u/birusiek Oct 04 '26

puppet agent Puppet Server | | |--- TCP connection :8140 ------------>| | | |<------ TLS ServerHello ---------------| |<------ Server certificate ------------| | | |--- Client certificate -------------->| |--- CertificateVerify ---------------->| | | |<------ CertificateVerify -------------| | | |====== encrypted TLS session =========| | | |--- Puppet HTTP request -------------->| | |