Certificate Info in Go

2023-07-14

Sometimes I write a utility just for the sake of learning something new. This particular code was about wait groups in Golang

We have a tool that inspects the TLS/SSL Certificate on a web server, I wanted to write my own clean room implementation that used wait groups.

Lets take a look at using the crypto/tls module to spit out some details I want:

package main

import (
  "crypto/tls"
  "crypto/x509/pkix"
  "fmt"
  "net"
)

 TLSHost struct {
        FQDN       string    `json:"fqdn"`
        Port       int       `json:"port"`
        DNSNames   []string  `json:"dns_names,omitempty"`
        SNI        string    `json:"sni,omitempty"`
        Version    string    `json:"version"`
        Issuer     pkix.Name `json:"issuer,omitempty"`
        Serial     string    `json:"serial"`
        Expiration time.Time `json:"expiration,omitempty"`
        Err        bool      `json:"err"`
        Message    string    `json:"message"`
}

func main() {
  h := "mywushublog.com"
  p := 443

  conn, err := tls.DialWithDialer(&net.Dialer{Timeout: 5 * time.Second}, "tcp", fmt.Sprintf("%s:%d", h, p), &tls.Config{})
  if err != nil {
     host := TLSHost{
             FQDN:    h,
             Port:    p,
             Err:     true,
             Message: fmt.Sprintf("%s", err),
     }

     *r = append(*r, host)
     continue
  }

  verifyFQDN(conn, h)
  host := TLSHost{
          FQDN:       h,
          Port:       p,
          DNSNames:   conn.ConnectionState().PeerCertificates[0].DNSNames,
          SNI:        getSNI(conn),
          Issuer:     getIssuer(conn),
          Serial:     getSerial(conn),
          Expiration: getExpiration(conn),
          Version:    getTlsVersion(conn),
          Err:        false,
          Message:    "Okay",
  }

  js, _ := json.Marshal(host)
  fmt.Printf("%s\n", js)
}

First, a couple of things that I discovered while writing this and testing it against good and bad hosts:

  1. Fail as fast as you can
  2. Verify the hostname is valid (exists) up front, this will save many seconds per host
  3. The net dialer timeout is nothing, it’s up to the system! I set it to 5. I honestly think in this day in age it could be 1 second, MAYBE 2.

Alright, moving on.

I made some helper methods:

  • verifyFQDN: If a host isn’t resolvable, set Err to true and move on.
  • getSNI: its kind of useful to know what the endpoint thinks its Server Name Indicator is
  • getTlsVersion: using a map, return a string of the version. The tls module returns a uint16 which I don’t feel is operator friendly enough
  • getIssuer: Who issued the cert
  • getExpiration: sort of the whole reason for this. Certs expire often and its good info to have up front
  • getSignature: I used this to to REALLY verify if a particular cert is rolled out or not
func verifyFQDN(conn *tls.Conn, h string) {
        err := conn.VerifyHostname(h)
        if err != nil {
                panic("Hostname doesn't match with certificate: " + err.Error())
        }
        fmt.Printf("Hostname %s is valid\n", h)
}

func getSNI(c *tls.Conn) string {
        sni := c.ConnectionState().ServerName
        return sni
}

func getTlsVersion(c *tls.Conn) string {
        versions := map[uint16]string{
                tls.VersionSSL30: "SSL",
                tls.VersionTLS10: "TLS 1.0",
                tls.VersionTLS11: "TLS 1.1",
                tls.VersionTLS12: "TLS 1.2",
                tls.VersionTLS13: "TLS 1.3",
        }

        return versions[c.ConnectionState().Version]
}

func getIssuer(c *tls.Conn) pkix.Name {
        return c.ConnectionState().PeerCertificates[0].Issuer
}

func getExpiration(c *tls.Conn) time.Time {
        return c.ConnectionState().PeerCertificates[0].NotAfter
}

func getSignature(c *tls.Conn) string {
        sig := c.ConnectionState().PeerCertificates[0].Signature
        var buf bytes.Buffer
        for i, val := range sig {
                if (i % 18) == 0 {
                        buf.WriteString(fmt.Sprintf("\n%9s", ""))
                }
                buf.WriteString(fmt.Sprintf("%02x", val))
                if i != len(sig)-1 {
                        buf.WriteString(":")
                }
        }
        return buf.String()
}

So lets run this and see what we’re working with: 

go run main.go | jq
2023/07/12 12:16:02 Hostname mywushublog.com is valid
[
  {
    "fqdn": "mywushublog.com",
    "port": 443,
    "dns_names": [
      "mywushublog.com",
      "www.mywushublog.com"
    ],
    "sni": "mywushublog.com",
    "version": "TLS 1.3",
    "issuer": {
      "Country": [
        "US"
      ],
      "Organization": [
        "Amazon"
      ],
      "OrganizationalUnit": null,
      "Locality": null,
      "Province": null,
      "StreetAddress": null,
      "PostalCode": null,
      "SerialNumber": "",
      "CommonName": "Amazon RSA 2048 M01",
      "Names": [
        {
          "Type": [
            2,
            5,
            4,
            6
          ],
          "Value": "US"
        },
        {
          "Type": [
            2,
            5,
            4,
            10
          ],
          "Value": "Amazon"
        },
        {
          "Type": [
            2,
            5,
            4,
            3
          ],
          "Value": "Amazon RSA 2048 M01"
        }
      ],
      "ExtraNames": null
    },
    "serial": "CE39325A91F1B2874E4D1270ED71B5E",
    "expiration": "2024-05-10T23:59:59Z",
    "err": false,
    "message": "Okay"
  }
]

One host is fine, how about 2? How about 20?

One design pattern I kind of like is taking input from STDIN:

   scanner := bufio.NewScanner(os.Stdin)
   for scanner.Scan() {
       h := scanner.Text()
       Hosts = append(Hosts, h)
   }

This buffered I/O scanner lets me do something like: 

$ echo << EOF > hosts
www.mywushublog.com
www.google.com
www.blizzard.com
www.llnl.gov
EOF

$ cat hosts | tls-example

and our little go app will process all of those individual hostnames as a host

cat hosts | go run main.go | jq '.[] | { fqdn, dns_names, expiration}'
2023/07/12 16:01:02 Hostname www.mywushublog.com is valid
2023/07/12 16:01:02 Hostname www.google.com is valid
2023/07/12 16:01:02 Hostname www.blizzard.com is valid
2023/07/12 16:01:02 Hostname www.llnl.gov is valid
{
  "fqdn": "www.mywushublog.com",
  "dns_names": [
    "mywushublog.com",
    "www.mywushublog.com"
  ],
  "expiration": "2024-05-10T23:59:59Z"
}
{
  "fqdn": "www.google.com",
  "dns_names": [
    "www.google.com"
  ],
  "expiration": "2023-09-11T08:21:19Z"
}
{
  "fqdn": "www.blizzard.com",
  "dns_names": [
    "blizzard.com",
    "*.blizzard.com"
  ],
  "expiration": "2024-07-19T23:59:59Z"
}
{
  "fqdn": "www.llnl.gov",
  "dns_names": [
    "*.llnl.gov"
  ],
  "expiration": "2024-01-30T23:59:59Z"
}

I’d like to point out again why I like to just emit json as an output, and let jq do the heavy lifting of selecting data.

running this against 4 hosts isn’t bad, if I slap a time in there, it reports:

0.31s user 0.08s system 69% cpu 0.556 total

which isn’t bad.

But, what if I use a large list. Like, the top 500 websites:

curl -Ls https://moz.com/top-500/download/\?table\=top500Domains | awk -F, 'gsub("\"","") {print $2}' | tail -n +2 | time go run main.go

By the way, tail -n +2 will remove the CSV header, this is better for our program that doesn’t parse the CSV but just a list of names.

1.11s user 0.22s system 1% cpu 1:43.05 total

Using the built in shell command time is okay, but lets keep track of it in our code:

package main

imports (
  ...
  "time"
  ...
)

func main() {
  ...
  start := time.Now()

  // the rest of the code...

  t := time.Now()
  elapsed := t.Sub(start)

  fmt.Printf("Finished %v hosts in %v\n", len(Hosts), elapsed)
}

With that, now we get a nicer elapsed time:

curl -Ls https://moz.com/top-500/download/\?table\=top500Domains | awk -F, 'gsub("\"","") {print $2}' | tail -n +2 | go run main.go
Finished 500 hosts in 1m34.135739699s

Wow, 1 minute and 34 seconds! For 500 hosts? Thats not very webscale.

Now lets get into sync/WaitGroup’s, and I’m totally going to cheat here.

First, read this because I don’t really know what I’m talking about.

If I wanted to, I could update my main function to basically do this: 

import (
  ...
  "sync"
  ...
)

func main() {
  var wg sync.WaitGroup

  for _, h := range Hosts {
          wg.Add(1)
          go func() {
              defer wg.Done()
              InspectCertificate(h, Port, &results)
          }
  }

  // wait until all groups are done
  wg.Wait()
}

func InspectCertificate(h string, p int, r *TLSHosts) {
    //do stuff

}

Here, for every host in the array of TLSHosts, we’re adding on to the wait group. This is fine for small array sizes, but when we have a list of 500+ thats a very large waigt group. I still like to think of waitgroups and go channels as a 1:1 match for cores/threads on my system. I don’t yet have 500+ threads.

What I did find though, was a very useful wrapper library that lets you define your wait group size.

So, We’ll add this module:

"github.com/pieterclaerhout/go-waitgroup"

We’ll move the bulk of the work out of main() and into its own function:

func InspectCertificate(h string, p int, r *TLSHosts, wg *waitgroup.WaitGroup) {
  // wg.Done is used to signal back to the wait group that the task is complete.
  defer wg.Done()

  conn, err := tls.DialWithDialer(&net.Dialer{Timeout: 5 * time.Second}, "tcp", fmt.Sprintf("%s:%d", h, p), &tls.Config{})
  if err != nil {
          log.Printf("%s has errors: %s\n", h, err)
          host := TLSHost{
                  FQDN:    h,
                  Port:    p,
                  Err:     true,
                  Message: fmt.Sprintf("%s", err),
          }

          *r = append(*r, host)
          // use return instead of continue, as this is now a separate function
          return
  }

  verifyFQDN(conn, h)
  host := TLSHost{
          FQDN:       h,
          Port:       p,
          DNSNames:   conn.ConnectionState().PeerCertificates[0].DNSNames,
          SNI:        getSNI(conn),
          Issuer:     getIssuer(conn),
          Serial:     getSerial(conn),
          Expiration: getExpiration(conn),
          Version:    getTlsVersion(conn),
          Err:        false,
          Message:    "Okay",
  }
  *r = append(*r, host)
}

and main can be updated to:

func main() {                                                                                                                                                                                                                                      
    var Hosts []string
    var results TLSHosts

    start := time.Now()
    // use 16 groups
    wg := waitgroup.NewWaitGroup(16)

    scanner := bufio.NewScanner(os.Stdin)
    fmt.Println("Reading hosts from STDIN")

    for scanner.Scan() {
            h := scanner.Text()
            Hosts = append(Hosts, h)
    }

    // same this, we iterate through a list, but add a wg block and inspect the cert.
    for _, h := range Hosts {
            wg.BlockAdd()
            go InspectCertificate(h, Port, &results, wg)
    }                                                                                                                                                                                                                                          
   
    wg.Wait()
    t := time.Now()
    elapsed := t.Sub(start)

    fmt.Printf("Finished %v hosts in %v", len(Hosts), elapsed)

    js, _ := json.Marshal(results)
    fmt.Printf("%s\n", js)
}

So putting that all together, our total runtime is:

Finished 500 hosts in 9.903309839s

9.9 seconds is better.

But wait(hehe)! We’re not done just yet.

if you mix your prints with your json, tools like jq will error out.

Convert all of those fmt.Print statements to log.Print

Final:

package main

import (
        "bufio"
        "bytes"
        "crypto/tls"
        "crypto/x509/pkix"
        "encoding/json"
        "flag"
        "fmt"
        "github.com/pieterclaerhout/go-waitgroup"
        "log"
        "net"
        "os"
        "time"
)

type TLSHost struct {
        FQDN       string    `json:"fqdn"`
        Port       int       `json:"port"`
        DNSNames   []string  `json:"dns_names,omitempty"`
        Issuer     pkix.Name `json:"issuer,omitempty"`
        Serial     string    `json:"serial"`
        Expiration time.Time `json:"expiration,omitempty"`
        Err        bool      `json:"err"`
        Message    string    `json:"message"`
}

type TLSHosts []TLSHost

func main() {
        Port := 443
        flag.Parse()

        var Hosts []string
        var results TLSHosts

        start := time.Now()

        wg := waitgroup.NewWaitGroup(16)

        file, err := os.OpenFile("tlsinfo.log", os.O_APPEND|os.O_CREATE|os.O_WRONLY, 0666)
        if err != nil {
                fmt.Println(err)
        }

        log.SetOutput(file)

        scanner := bufio.NewScanner(os.Stdin)
        log.Println("Reading hosts from STDIN")

        for scanner.Scan() {
                h := scanner.Text()
                Hosts = append(Hosts, h)
        }

        for _, h := range Hosts {
                wg.BlockAdd()
                go InspectCertificate(h, Port, &results, wg)
        }

        wg.Wait()
        t := time.Now()
        elapsed := t.Sub(start)

        log.Printf("Finished %v hosts in %v", len(Hosts), elapsed)

        js, _ := json.Marshal(results)
        fmt.Printf("%s\n", js)
}

func InspectCertificate(h string, p int, r *TLSHosts, wg *waitgroup.WaitGroup) {
        defer wg.Done()
        log.Printf("checking %s", h)

        conn, err := tls.DialWithDialer(&net.Dialer{Timeout: 5 * time.Second}, "tcp", fmt.Sprintf("%s:%d", h, p), &tls.Config{})
        if err != nil {
                log.Printf("%s has errors: %s\n", h, err)
                host := TLSHost{
                        FQDN:    h,
                        Port:    p,
                        Err:     true,
                        Message: fmt.Sprintf("%s", err),
                }

                *r = append(*r, host)
                return
        }

        verifyFQDN(conn, h)
        host := TLSHost{
                FQDN:       h,
                Port:       p,
                DNSNames:   conn.ConnectionState().PeerCertificates[0].DNSNames,
                Issuer:     getIssuer(conn),
                Serial:     getSerial(conn),
                Expiration: getExpiration(conn),
                Err:        false,
                Message:    "Okay",
        }
        *r = append(*r, host)
}

func verifyFQDN(conn *tls.Conn, h string) {
        err := conn.VerifyHostname(h)
        if err != nil {
                panic("Hostname doesn't match with certificate: " + err.Error())
        }
        log.Printf("Hostname %s is valid", h)
}

func getIssuer(c *tls.Conn) pkix.Name {
        return c.ConnectionState().PeerCertificates[0].Issuer
}

func getExpiration(c *tls.Conn) time.Time {
        return c.ConnectionState().PeerCertificates[0].NotAfter
}

func getSignature(c *tls.Conn) string {
        sig := c.ConnectionState().PeerCertificates[0].Signature
        var buf bytes.Buffer
        for i, val := range sig {
                if (i % 18) == 0 {
                        buf.WriteString(fmt.Sprintf("\n%9s", ""))
                }
                buf.WriteString(fmt.Sprintf("%02x", val))
                if i != len(sig)-1 {
                        buf.WriteString(":")
                }
        }
        return buf.String()
}

func getSerial(c *tls.Conn) string {
        return fmt.Sprintf("%02X", c.ConnectionState().PeerCertificates[0].SerialNumber)
}

Now when you run your code, you get a clean json output:

//cat hosts | go run main.go | jq '.[] | { fqdn, dns_names, expiration}'
{
  "fqdn": "badname.xyz",
  "dns_names": null,
  "expiration": "0001-01-01T00:00:00Z"
}
{
  "fqdn": "www.google.com",
  "dns_names": [
    "www.google.com"
  ],
  "expiration": "2023-09-11T08:21:19Z"
}
{
  "fqdn": "www.llnl.gov",
  "dns_names": [
    "*.llnl.gov"
  ],
  "expiration": "2024-01-30T23:59:59Z"
}
{
  "fqdn": "www.mywushublog.com",
  "dns_names": [
    "mywushublog.com",
    "www.mywushublog.com"
  ],
  "expiration": "2024-05-10T23:59:59Z"
}
{
  "fqdn": "www.blizzard.com",
  "dns_names": [
    "blizzard.com",
    "*.blizzard.com"
  ],
  "expiration": "2024-07-19T23:59:59Z"
}

but you also get a log file:

cat tlsinfo.log
2023/07/13 10:44:48 Reading hosts from STDIN
2023/07/13 10:44:48 checking www.llnl.gov
2023/07/13 10:44:48 checking www.mywushublog.com
2023/07/13 10:44:48 checking www.blizzard.com
2023/07/13 10:44:48 checking www.google.com
2023/07/13 10:44:48 checking badname.xyz
2023/07/13 10:44:48 badname.xyz has errors: dial tcp: lookup badname.xyz on 192.168.1.2:53: no such host
2023/07/13 10:44:48 Hostname www.google.com is valid
2023/07/13 10:44:48 Hostname www.mywushublog.com is valid
2023/07/13 10:44:48 Hostname www.llnl.gov is valid
2023/07/13 10:44:49 Hostname www.blizzard.com is valid

I threw in a bogus hostname, just to show what it looks like

Alright, hope this is informative for anyone out there.

#golang #software development