{"id":3860,"date":"2026-04-16T16:12:08","date_gmt":"2026-04-16T16:12:08","guid":{"rendered":"https:\/\/rssfeedtelegrambot.bnaya.co.il\/index.php\/2026\/04\/16\/how-github-uses-ebpf-to-improve-deployment-safety\/"},"modified":"2026-04-16T16:12:08","modified_gmt":"2026-04-16T16:12:08","slug":"how-github-uses-ebpf-to-improve-deployment-safety","status":"publish","type":"post","link":"https:\/\/rssfeedtelegrambot.bnaya.co.il\/index.php\/2026\/04\/16\/how-github-uses-ebpf-to-improve-deployment-safety\/","title":{"rendered":"How GitHub uses eBPF to improve deployment safety"},"content":{"rendered":"

Did you know that, at GitHub, we host all of our own source code on github.com<\/a>? We do this because we\u2019re our own biggest customer\u2014testing out changes internally before they go to users. However, there\u2019s one downside: If github.com were ever to go down, we wouldn\u2019t be able to access our own source code.<\/p>\n

This is what you\u2019d call a very simple circular dependency: to deploy GitHub, we needed GitHub. If GitHub is down, then we wouldn\u2019t be able to deploy something to fix it. We mitigate this by maintaining a mirror of our code for fixing forward and built assets for rolling back.<\/p>\n

So we\u2019re done, right? Problem solved? Nope, there are more circular dependencies to consider. For example, how do you stop a deployment script introducing a circular dependency of its own on an internal service or downloading a binary from GitHub?<\/p>\n

When we started to design our new host-based deployment system, we evaluated some new approaches to prevent deployment code from creating circular dependencies. We found that using eBPF, we could selectively monitor and block those calls. In this blog post, we\u2019ll take you through our findings and show how you can get started writing your own eBPF programs.<\/p>\n

Types of circular dependencies<\/h2>\n

Let\u2019s start by looking at the types of circular dependencies through a hypothetical scenario.<\/p>\n

Suppose a MySQL outage occurs, which causes GitHub to be unable to serve release<\/code> data from repositories. To resolve the incident, we need to roll out a configuration change to the stateful MySQL nodes that are impacted. This configuration change is applied by executing a deploy script on each node.<\/p>\n

Now, let\u2019s look at the different types of circular dependencies that could impact GitHub during this scenario.<\/p>\n

    \n
  1. Direct dependency<\/strong>: The MySQL deploy script attempts to pull the latest release of an\u00a0open source\u00a0tool from GitHub. Since GitHub\u00a0can\u2019t\u00a0serve the release data (due to the outage), the script\u00a0can\u2019t\u00a0complete.\u00a0\u00a0<\/li>\n<\/ol>\n
    \"Diagram<\/figure>\n
      \n
    1. Hidden dependencies<\/strong>: The MySQL deploy script uses a servicing tool that is already present on the machine\u2019s disk. However, when the tool runs, it checks GitHub to see if an update is available. If it\u2019s unable to contact GitHub (due to the outage), the script may fail or hang, depending on how the tool handles the error when checking for updates.<\/li>\n<\/ol>\n
      \"Diagram<\/figure>\n
        \n
      1. Transient dependencies<\/strong>: The MySQL deploy script calls, via an API, another internal service (for example, a migrations service), which in turn attempts to fetch the latest release of an open source tool from GitHub to use the new binary. The failure propagates back to the deploy script.<\/li>\n<\/ol>\n
        \"Diagram<\/figure>\n

        How do you solve these circular dependencies?<\/h2>\n

        Until recently, the onus has been on every team who that owns stateful hosts to review their deployment scripts and identify circular dependencies.<\/p>\n

        In practice, however, many dependencies aren\u2019t identified until an incident occurs, which can delay recovery.<\/p>\n

        The obvious route would be to block access to github.com from the machines to validate that the system can deploy without it. But these hosts are stateful and serve customer traffic even during rolling deploys, drains, or restarts. Blocking github.com entirely would impact their ability to handle production requests.<\/p>\n

        This is where we started to look at eBPF, which lets you load custom programs into the Linux kernel and hook into core system primitives like networking.<\/p>\n

        We were particularly interested in the BPF_PROG_TYPE_CGROUP_SKB<\/code> program type<\/a> because it lets you hook network egress from a particular cGroup.<\/p>\n

        A cGroup<\/a> is a Linux primitive (used heavily by Docker but not limited to it) that enforces resource limits and isolation for sets of processes. You can create a cGroup, configure it, and move processes into it\u2014no Docker required.<\/p>\n

        This started to look very promising. Could we create a cGroup, place only the deployment script inside it, and then limit the outbound network access of only that script? It certainly looked possible, so we started to build a proof of concept.<\/p>\n

        Building out per-process conditional network filtering with eBPF<\/h2>\n

        We started on a proof of concept in go<\/code> that used the cilium\/ebpf<\/a><\/code> library.<\/p>\n

        ebpf-go is a pure-Go library to read, modify, and load eBPF programs and attach them to various hooks in the Linux kernel.<\/p>\n

        It massively simplifies the process of authoring, building, and running programs that use eBPF. For example, to hook the BPF_PROG_TYPE_CGROUP_SKB<\/code> program type<\/a>, we can do this as follows: \ud83d\udc47<\/p>\n

        \n
        \/\/go:generate go tool bpf2go -tags linux bpf cgroup_skb.c -- -I..\/headers \n\n \n\nfunc main() { \n\n   \/\/ Load pre-compiled programs and maps into the kernel. \n\n   objs := bpfObjects{} \n\n   if err := loadBpfObjects(&objs, nil); err != nil { \n\n       log.Fatalf(\"loading objects: %v\", err) \n\n   } \n\n   defer objs.Close() \n\n \n\n   \/\/ Link the count_egress_packets program to the cgroup. \n\n   l, err := link.AttachCgroup(link.CgroupOptions{ \n\n       Path:    \"\/sys\/fs\/cgroup\/system.slice\", \n\n       Attach:  ebpf.AttachCGroupInetEgress, \n\n       Program: objs.CountEgressPackets, \n\n   }) \n\n   if err != nil { \n\n       log.Fatal(err) \n\n   } \n\n   defer l.Close() \n\n \n\n   log.Println(\"Counting packets...\") \n\n \n\n   \/\/ Read loop reporting the total amount of times the kernel \n\n   \/\/ function was entered, once per second. \n\n   ticker := time.NewTicker(1 * time.Second) \n\n   defer ticker.Stop() \n\n \n\n   for range ticker.C { \n\n       var value uint64 \n\n       if err := objs.PktCount.Lookup(uint32(0), &value); err != nil { \n\n           log.Fatalf(\"reading map: %v\", err) \n\n       } \n\n       log.Printf(\"number of packets: %dn\", value) \n\n   } \n\n} <\/code><\/pre>\n<\/div>\n
        With the eBPF program:<\/p>\n
        \n
        \/\/go:build ignore \n\n \n\n#include \"common.h\" \n\n \n\nchar __license[] SEC(\"license\") = \"Dual MIT\/GPL\"; \n\n \n\nstruct { \n\n   __uint(type, BPF_MAP_TYPE_ARRAY); \n\n   __type(key, u32); \n\n   __type(value, u64); \n\n   __uint(max_entries, 1); \n\n} pkt_count SEC(\".maps\"); \n\n \n\nSEC(\"cgroup_skb\/egress\") \n\nint count_egress_packets(struct __sk_buff *skb) { \n\n   u32 key      = 0; \n\n   u64 init_val = 1; \n\n \n\n   u64 *count = bpf_map_lookup_elem(&pkt_count, &key); \n\n   if (!count) { \n\n       bpf_map_update_elem(&pkt_count, &key, &init_val, BPF_ANY); \n\n       return 1; \n\n   } \n\n   __sync_fetch_and_add(count, 1); \n\n \n\n   return 1; \n\n} <\/code><\/pre>\n<\/div>\n
        The \/\/go:generate<\/code> line handles compiling the eBPF C code and auto-generating the bpfObjects<\/code> struct, which allows us to attach and interact with the program. This means a simple go build<\/code> is all you need. \ud83e\udd73<\/p>\n
        (cilium\/ebpf<\/code> has a great set of examples to get started. Review the full code from above<\/a>).<\/p>\n
        There was still a missing piece though: CGROUP_SKB<\/code> operates on IP addresses. Given the breadth of GitHub\u2019s systems and rate of change, keeping an up-to-date block IP list would be very hard.<\/p>\n
        Could we use more eBPF to create a DNS-based blocked list? Yes, it turns out we could.<\/p>\n
        An eBPF program type of BPF_PROG_TYPE_CGROUP_SOCK_ADDR<\/code><\/a> allows you to hook syscalls to create sockets and change the destination IP<\/strong>.<\/p>\n
        Here is a simplified example where we rewrite any connect4<\/code> syscall targeting DNS (Port 53) to localhost:53<\/code>.<\/p>\n
        \n
        cgroupLink, err := link.AttachCgroup(link.CgroupOptions{ \n\n       Path:    cgroup.Name(), \n\n       Attach:  ebpf.AttachCGroupInet4Connect, \n\n       Program: obj.Connect4, \n\n   }) \n\n   if err != nil { \n\n       return nil, fmt.Errorf(\"attaching eBPF program Connect4 to cgroup: %w\", err) \n\n   } <\/code><\/pre>\n<\/div>\n
        \n
        \/* This is the hexadecimal representation of 127.0.0.1 address *\/ \n\nconst __u32 ADDRESS_LOCALHOST_NETBYTEORDER = bpf_htonl(0x7f000001); \n\n \n\nSEC(\"cgroup\/connect4\") \n\nint connect4(struct bpf_sock_addr *ctx) { \n\n __be32 original_ip = ctx->user_ip4; \n\n __u16 original_port = bpf_ntohs(ctx->user_port); \n\n \n\n if (ctx->user_port == bpf_htons(53)) { \n\n   \/* For DNS Query (*:53) rewire service to backend \n\n    * 127.0.0.1:const_dns_proxy_port *\/ \n\n   ctx->user_ip4 = const_mitm_proxy_address; \n\n   ctx->user_port = bpf_htons(const_dns_proxy_port); \n\n } \n\n \n\n return 1; \n\n} <\/code><\/pre>\n<\/div>\n
        We used this to intercept DNS queries from the cGroup and forward them to a userspace DNS proxy we run.<\/p>\n
        Now, any DNS queries initiated by the deployment script are routed through our DNS proxy. Our proxy evaluates each requested domain against our block list and uses eBPF Maps<\/a> to communicate with the CGROUP_SKB<\/code> program, allowing or denying the request accordingly.<\/p>\n
        If you\u2019d like to dig into the code, here\u2019s an early proof of concept<\/a> we put together. Our current implementation has progressed since then, but this should serve as a good intro.<\/p>\n
        Like any fun project, the deeper we got, the more we realized we could do.<\/p>\n
        For example, could we correlate blocked DNS requests back to the specific command or process that triggered them, so teams could more easily debug and fix issues? Yes, we can!<\/p>\n
        Inside the BPF_PROG_TYPE_CGROUP_SKB<\/code> program type<\/a>, we have the skb_buff<\/code><\/a> from which we can pull the DNS transaction ID<\/a> and also capture the Process ID<\/a> (PID) that initiated the request. We place this information into another eBPF Map tracking DNS Transaction ID -> Process ID<\/code>.<\/p>\n
        Here is a simplified version of the eBPF code (see this PoC code<\/a> for full example):<\/p>\n
        \n
          __u32 pid = bpf_get_current_pid_tgid() >> 32; \n\n     __u16 skb_read_offset = sizeof(struct iphdr) + sizeof(struct udphdr); \n\n     __u16 dns_transaction_id = \n\n         get_transaction_id_from_dns_header(skb, skb_read_offset); \n\n \n\n     if (pid && dns_transaction_id != 0) { \n\n       bpf_map_update_elem(&dns_transaction_id_to_pid, &dns_transaction_id, \n\n                           pid, BPF_ANY); \n\n     } <\/code><\/pre>\n<\/div>\n
        As we\u2019re redirecting all DNS calls to our userspace DNS proxy, we can look at the transaction ID of each request, find the domain being resolved, and lookup in the eBPF Map to see which process made the request. By reading \/proc\/{PID}\/cmdline<\/code>, we can even extract the full command line that triggered the request.<\/p>\n
        Then we can output a log line with all the information:<\/p>\n
        \n
        > WARN DNS BLOCKED reason=FromDNSRequest blocked=true blockedAt=dns domain=github.com. pid=266767 cmd=\"curl github.com \" firewallMethod=blocklist<\/code><\/pre>\n<\/div>\n
        With that, we\u2019re done.<\/p>\n
        We can now:<\/p>\n