---
title: "Unify HTTP requests and GRPC calls on a single domain for more flexible configuration: example with Woodpecker"
date: !!timestamp '2023-10-27 09:35:00'
image: /post/woodpecker-ci-mixing-http-grpc-on-one-domain-nginx/og.webp
tags:
- network
- hosting
- continuous integration
---

I installed the continuous integration service [Woodpecker](https://woodpecker-ci.org/), to replace [DroneCI](https://drone.io), which [the company that bought it decided to bury](https://github.com/harness/gitness#where-is-drone).
As Woodpecker is a fork of the latest free version of Drone, its use is broadly similar.

However, the teams have taken different directions on certain aspects, and communication with agents/*runners*, which used to be via websockets, is now carried out in Woodpecker using the GRPC protocol.

The solution proposed by the [Woodpecker documentation](https://woodpecker-ci.org/docs/administration/proxy#caddy) is to use 2 domains: one will be used for the web interface and the REST API, the second will be used for GRPC.
Is this really necessary?

<!-- more -->

## `nginx` configuration using 2 domains

Woodpecker exposes the web interface and GRPC on two different ports: 8000 and 9000 respectively.

The simplest approach to exposing these two services on the Internet is to use two separate domains.
Here's what our reverse-proxy configuration might look like:

```
server {
    listen 80;
    server_name woodpecker.example.com;

    location / {
        proxy_set_header X-Forwarded-For $remote_addr;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_set_header Host $http_host;

        proxy_pass http://127.0.0.1:8000;
        proxy_redirect off;
        proxy_http_version 1.1;
        proxy_buffering off;

        chunked_transfer_encoding off;
    }
}

server {
    listen 80 http2;
    server_name woodpeckeragent.example.com;

    location / {
        grpc_pass grpc://127.0.0.1:9000;
    }
}
```

Here we're using `nginx`'s [GRPC module](https://nginx.org/en/docs/http/ngx_http_grpc_module.html), and in particular the [`grpc_pass`](https://nginx.org/en/docs/http/ngx_http_grpc_module.html#grpc_pass) directive.
This directive is similar to the [`proxy_pass`](https://nginx.org/en/docs/http/ngx_http_proxy_module.html#proxy_pass) directive: it will forward to port 9000 on the local machine all GRPC packets arriving on the `woodpeckeragent.example.com`.

On the other hand, I prefer to avoid declaring 2 domains, obtaining 2 certificates, etc. for 1 and the same service, with a domain that you tend to forget and neglect.
So let's see if we can't do better.


## The GRPC protocol

Without going into too much detail, GRPC is a [protocol close to HTTP/2](https://grpc.io/docs/what-is-grpc/faq/#why-is-grpc-betterworse-than-rest).
As such, many reverse-proxies are capable of transmitting GRPC requests.
This is the case of [Caddy](https://woodpecker-ci.org/docs/administration/proxy#caddy), a brief example of which is given, [Traefik](https://woodpecker-ci.org/docs/administration/proxy#traefik).
But `nginx` also [supports the transmission of GRPC requests](https://www.nginx.com/blog/nginx-1-13-10-grpc/).

When a GRPC request is received by the reverse-proxy, it sees an HTTP/2 request similar to:

```
POST /pkg.Service/Function HTTP/2.0
Host: grpc.example.com
User-Agent: grpc-go/1.21.0
[...]
```

At first glance, there's nothing confusing for a web server.
There must be something clever we can do to use these similarities to our advantage.


## Combine HTTP requests and GRPC calls on the same domain

The path for GPRC requests is fixed, and depends on the protobuf file describing calls and structures.
Each *Service* is declared within a package (*pkg*), then *Functions* complete the path.
All calls are `POSTs`.

So we need to extract the various HTTP routes used by each protobuf service.
Let's take a look at the following file for Woodpecker:

<https://github.com/woodpecker-ci/woodpecker/blob/main/pipeline/rpc/proto/woodpecker.proto>

```proto
[...]
package proto;
[...]
service Woodpecker {
  rpc Version       (Empty)                returns (VersionResponse) {}
  rpc Next          (NextRequest)          returns (NextResponse) {}
  rpc Init          (InitRequest)          returns (Empty) {}
  rpc Wait          (WaitRequest)          returns (Empty) {}
  rpc Done          (DoneRequest)          returns (Empty) {}
  rpc Extend        (ExtendRequest)        returns (Empty) {}
  rpc Update        (UpdateRequest)        returns (Empty) {}
  rpc Log           (LogRequest)           returns (Empty) {}
  rpc RegisterAgent (RegisterAgentRequest) returns (RegisterAgentResponse) {}
  rpc ReportHealth  (ReportHealthRequest)  returns (Empty) {}
}
[...]
service WoodpeckerAuth {
  rpc Auth          (AuthRequest)          returns (AuthResponse) {}
}
[...]
```

From this descriptive file, we have now determined all the routes that can be used by all clients using this version of the service.

The `proto` package contains 2 services: `Woodpecker` and `WoodpeckerAuth`.
This gives us 2 root routes:

- `/proto.Woodpecker/`
- `/proto.WoodpeckerAuth/`

Behind each, we find the functions described by each `rpc` line.
For example:

- `/proto.Woodpecker/Version`
- `/proto.Woodpecker/Done`
- `/proto.Woodpecker/Log`
- `/proto.WoodpeckerAuth/Auth`
- ...


## `nginx` configuration

So here's what our `nginx` configuration might look like, using a single domain:

```
server {
    listen 80;
    http2 on;
    server_name woodpecker.example.com;

    location / {
        proxy_set_header X-Forwarded-For $remote_addr;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_set_header Host $http_host;

        proxy_pass http://127.0.0.1:8000;
        proxy_redirect off;
        proxy_http_version 1.1;
        proxy_buffering off;

        chunked_transfer_encoding off;
    }

    location /proto.Woodpecker/ {
        grpc_pass grpc://127.0.0.1:9000;
    }

    location /proto.WoodpeckerAuth/ {
        grpc_pass grpc://127.0.0.1:9000;
    }
}
```

Note that, of course, this only works if HTTP/2 is enabled by means of [the `http2` directive](https://nginx.org/en/docs/http/ngx_http_v2_module.html#http2).

There's no need to specify functions individually, as `nginx` will perform the routing using only the package and service name.


## Timeout on the `Next` function

Woodpecker declares an `Next` function that waits for information about the next job to be performed.
This waiting time is generally quite variable, and can be quite long if your IC is not constantly in use.

`nginx` will take the initiative, by default, to stop any request that has been inactive for more than 60 seconds.
This causes a large number of reconnections for the Woodpecker agent, visible in the agent logs:

    {"level":"error","error":"rpc error: code = Unavailable desc = unexpected HTTP status code received from server: 504 (Gateway Timeout); transport: received unexpected content-type \"text/html\"","time":"2023-10-27T11:28:51Z","message":"grpc error: wait(): code: Unavailable: rpc error: code = Unavailable desc = unexpected HTTP status code received from server: 504 (Gateway Timeout); transport: received unexpected content-type \"text/html\""}

The agent will reconnect itself, but we can reduce the occurrence of this reconnection (whose main benefit is to do nothing until the server wakes us up), by adding this line to our configuration `nginx`:

```
    location /proto.Woodpecker/Next {
        grpc_pass grpc://127.0.0.1:9008;
        grpc_read_timeout 3600s;
    }
```

These few lines tell `nginx` not to disconnect inactive connections before 1 hour of inactivity, only for the `Next` function of the protocol.


## Filter IPs that can access the GRPC service

In addition to the [`grpc_pass`](https://nginx.org/en/docs/http/ngx_http_grpc_module.html#grpc_pass) directive, you can of course use all the other directives you're used to.

For example, we might want to filter the IPs that have access to GRPC.
We could do this with [the `allow` and `deny` directives](https://nginx.org/en/docs/http/ngx_http_access_module.html):

```
    location /proto.Woodpecker/ {
        allow 192.168.0.0/24;
        deny all;

        grpc_pass grpc://127.0.0.1:9000;
    }
```

Since only agents use the GRPC protocol, it would be perfectly legitimate to implement such a rule.

---

Unifying HTTP requests and GRPC calls under a single domain offers many advantages.
Not only does this approach simplify reverse-proxy configuration, it also enables more centralized, streamlined management of the service.

By the way, GRPC and protobuf are an excellent solution for transmitting structured data over the network, so don't hesitate to [take a look](https://protobuf.dev/overview/).
