GRPC Gateway & Honeycomb

So at work lately I’ve been looking into GRPC. It’s a really awesome way to define RPC APIs. Up until now, everything I’ve been building has been using Gin to create HTTP REST APIs. Which has been mostly fine, but something about it has always bugged me.

What’s bugged me is JSON. When you’re working with a browser, JSON is mostly fine. It’s easy to send JSON using any of the billions of JavaScript libraries out there. The problem is that validating it and turning it into native code is kind of a pain. In some frameworks ( ie, Ruby on Rails ) there’s a ton of magic behind the scenes that parses JSON and turns it into hashes or objects or whatever you want. In other frameworks or languages you end up with a lot of boilerplate any time you want to deal with JSON. Not ideal.

Enter GRPC! Define your API in a proto file, auto-generate code that deals with all the wire-format-to-native-object transcoding.

Plus, you can also auto-generate some other awesome stuff.

Want to auto-generate OpenAPI docs? Sure, you can do that.

Want to generate a gateway that automatically transcodes JSON to GRPC? That already exists too.

All-in-all, GRPC is pretty awesome.

As part of getting these microservices set up, I’ve been looking into ways to get metrics and whatnot. I’m a huge believer in observability. Being able to see memory stats, what code paths are getting executed, and how long things are taking is a huge boon to anybody creating backend services. I was looking at Prometheus for the longest time, but the fact that it’s scraping and not an endpoint always bugged me for reasons I could never really articulate properly.

Recently, I discovered Honeycomb.io – and it was like the heavens opened up and angels started singing.

Easy to implement? Great search dashboard? Super awesome tracing?

Honeycomb.io is awesome.

However, when I started to get Honeycomb implemented in the first microservice I ran into a problem.

See, Honeycomb has a fantastic Go library that wraps around common Go libraries to automatically add tracing and basic metrics.

For example, if you’re using the standard net/http library, you can set up Honeycomb pretty simply:

router := http.NewServeMux()

router.HandleFunc("/example", func(w http.ResponseWriter, r *http.Request){
    fmt.Fprintf(w, "Hello world!")
})

wrapped := hnynethttp.WrapHandler(router)

http.ListenAndServe(":8000", wrapped)

Bam! Now each request that comes in will automatically send events to Honeycomb each time a request is handled. Pretty awesome, right?

Definitely.

Once I discovered Honeycomb, I wanted to make sure I had it all set up so that I could get awesome traces of requests coming in, hitting the GRPC Gateway, then getting handled by the GRPC handler.

However, when I used the context to try and get an event, it didn’t have the tracing information:

func Example(ctx context.Context, in *example.AutoGeneratedInput ) (*example.AutoGeneratedOutput, error ) {
    e := beeline.ContextEvent(ctx)
    spew.Dump(e) // no trace_id!
    
    return nil, fmt.Errorf("no honeycomb :(")
}

Oh no!

Turns out, I had to do a few extra things before the tracing information would be passed from the GRPC Gateway down to the GRPC service.

There’s two parts to this: getting the GRPC Gateway to set some metadata on the context, and parsing that metadata in the GRPC service.

First part: sending the metadata!

Not much to it. First, create a metadata annotator. The GPRC Gateway uses this function that you define to add metadata to the context that it uses to make a request to the GRPC service.

Without any further ado, here’s the annotator:

import (
  "context"

  beeline "github.com/honeycombio/beeline-go"
  "google.golang.org/grpc/metadata"
)

func metadataAnnotator(ctx context.Context, r *http.Request) metadata.MD {
    ev := beeline.ContextEvent(r.Context())
    
    meta := map[string]string{}
    data := ev.Fields()
    
    setMeta(data, meta, "trace.trace_id")
    setMeta(data, meta, "trace.span_id")
    
    return metadata.New(meta)
}

func setMeta(in map[string]interface{}, out map[string]string, name string) {
	tmp, ok := in[name]
	if ok {
		id, ok := tmp.(string)
		if ok {
			out[name] = id
		}
	}
}

So now you’ve got the GRPC Gateway setting metadata on the context before it uses it to send a request to the GRPC service. Now we’ve got to pull that metadata out when a request happens. To do that, we need to write two interceptors: one for unary handlers, and one for streaming handlers.

First, a function to pull the trace_id & span_id out from the incoming context and use that to set up our new context.

func CreateEvent(ctx context.Context) *libhoney.Event {
    ev := libhoney.NewEvent()
    newID := uuid.NewV4()
    
    md, ok := metadata.FromIncomingContext(ctx)

    // check to see if this request is already part of a trace
    if ok {
        tmp, ok := md["trace.trace_id"]
        if ok {
           if len(tmp) > 0 {
             // it's from the gateway
             ev.AddField("trace.trace_id", tmp[0])
           } else {
             ev.AddField(newID)
           }
        }
        
        tmp, ok = md["trace.span_id"]
        if ok {
           if len(tmp) > 0 {
               // we've got a span_id, so set that as the parent_id of this event
               e.AddField("trace.parent_id", tmp[0])
           }
        }
    } else {
        ev.AddField("trace.trace_id", newID)
    }
    
    return &ev
}

Okay, so now we’ve got to set up our interceptors.

func GetUnaryInterceptor() grpc.UnaryServerInterceptor {
    return func(ctx context.Context, req interface{}, info *grpc.UnaryServerInfo, handler grpc.UnaryHandler) (interface{}, error) {
        ev := CreateEvent(ctx)
        start := time.Now()
        
        // add fields to identify this event
        ev.AddField("name", info.FullMethod)
        ev.AddField("grpc.input", req)
        
        o, err := handler(ev.Context(), req)
        
        ev.AddField("duration_ms",float64(time.Since(start))/float64(time.Millisecond))
        if err != nil {
          ev.AddField("grpc.error", err)
        }
        _ = ev.Send()
        return o, err_
    }
}

type _stream struct {
    grpc.ServerStream
    ctx context.Context
}_

func GetStreamInterceptor() grpc.StreamServerInterceptor {
    return func(srv interface{}, stream grpc.ServerStream, info *grpc.StreamServerInfo, handler grpc.StreamHandler) error {
       ctx := stream.Context()
       ev := CreateEvent(ctx)
       
       start := time.Now()
       
       s := _stream{stream, ev.Context()}_
       
       err := handler(srv, s)
       ev.AddField("duration_ms",float64(time.Since(start))/float64(time.Millisecond))
       if err != nil {
          ev.AddField("grpc.error", err)
       }
       
       return err
    }
}

Full disclosure: I haven’t done any streaming stuff with GRPC yet, so I can’t 100% vouch for the streaming interceptor working.

Okay! Now that you’ve got those functions, time to setup your server to use them. Wherever you’re setting up your GRPC server, do the following:

srvOpts := []grpc.ServerOption{
    grpc_middleware.WithUnaryServerChain(
      GetUnaryInterceptor(),
    ),
    grpc_middleware.WithStreamSererChain(
      GetStreamInterceptor(),
    )
}

srv := grpc.NewServer(srvOpts...)

And blamo, you’ve got your server setup so that it can add more spans to your Honeycomb traces!