{"id":3552,"date":"2026-03-03T18:46:20","date_gmt":"2026-03-03T18:46:20","guid":{"rendered":"https:\/\/rssfeedtelegrambot.bnaya.co.il\/index.php\/2026\/03\/03\/how-we-rebuilt-the-search-architecture-for-high-availability-in-github-enterprise-server\/"},"modified":"2026-03-03T18:46:20","modified_gmt":"2026-03-03T18:46:20","slug":"how-we-rebuilt-the-search-architecture-for-high-availability-in-github-enterprise-server","status":"publish","type":"post","link":"https:\/\/rssfeedtelegrambot.bnaya.co.il\/index.php\/2026\/03\/03\/how-we-rebuilt-the-search-architecture-for-high-availability-in-github-enterprise-server\/","title":{"rendered":"How we rebuilt the search architecture for high availability in GitHub Enterprise Server"},"content":{"rendered":"

So much of what you interact with on GitHub depends on search\u2014obviously the search bars and filtering experiences like the GitHub Issues page, but it is also the core of the releases page, projects page, the counts for issues and pull requests, and more. Given that search is such a core part of the GitHub platform, we\u2019ve spent the last year making it even more durable. That means, less time spent managing GitHub Enterprise Server, and more time working on what your customers care most about.\u00a0<\/p>\n

In recent years, GitHub Enterprise Server administrators had to be especially careful with search indexes, the special database tables optimized for searching. If they didn\u2019t follow maintenance or upgrade steps in exactly the right order, search indexes could become damaged and need repair, or they might get locked and cause problems during upgrades. Quick context if you\u2019re not running High Availability (HA) setups, they\u2019re designed to keep GitHub Enterprise Server running smoothly even if part of the system fails. You have a primary node that handles all the writes and traffic, and replica nodes that stay in sync and can take over if needed.<\/p>\n

\"Diagram<\/figure>\n

Much of this difficulty comes from how previous versions of Elasticsearch, our search database of choice, were integrated. HA GitHub Enterprise Server installations use a leader\/follower pattern. The leader (primary server) receives all the writes, updates, and traffic. Followers (replicas) are designed to be read-only. This pattern is deeply ingrained into all of the operations of GitHub Enterprise Server.<\/p>\n

This is where Elasticsearch started running into issues. Since it couldn\u2019t support having a primary node and a replica node, GitHub engineering had to create an Elasticsearch cluster across the primary and replica nodes. This made replicating data straightforward and additionally gave some performance benefits, since each node could locally handle search requests.\u00a0<\/p>\n

\"Diagram<\/figure>\n

Unfortunately, the problems of clustering across servers eventually began to outweigh the benefits. For example, at any point Elasticsearch could move a primary shard (responsible for receiving\/validating writes) to a replica. If that replica was then taken down for maintenance, GitHub Enterprise Server could end up in a locked state. The replica would wait for Elasticsearch to be healthy before starting up, but Elasticsearch couldn\u2019t become healthy until the replica rejoined.<\/p>\n

For a number of GitHub Enterprise Server releases, engineers at GitHub tried to make this mode more stable. We implemented checks to ensure Elasticsearch was in a healthy state, as well as other processes to try and correct drifting states. We went as far as attempting to build a \u201csearch mirroring\u201d system that would allow us to move away from the clustered mode. But database replication is incredibly challenging and these efforts needed consistency.<\/p>\n

What changed?<\/h2>\n

After years of work, we\u2019re now able to use Elasticsearch\u2019s Cross Cluster Replication (CCR) feature to support HA GitHub Enterprise.\u00a0<\/p>\n

\u201cBut David,\u201d you say, \u201cThat\u2019s replication between clusters. How does that help here?\u201d\u00a0<\/p>\n

I\u2019m so glad you asked. With this mode, we\u2019re moving to use several, \u201csingle-node\u201d Elasticsearch clusters. Now each Enterprise server instance will operate as independent single node Elasticsearch clusters.<\/p>\n

\"Diagram<\/figure>\n

CCR lets us share the index data between nodes in a way that is carefully controlled and natively supported by Elasticsearch. It copies data once it\u2019s been persisted to the Lucene segments (Elasticsearch\u2019s underlying data store). This ensures we\u2019re replicating data that has been durably persisted within the Elasticsearch cluster.<\/p>\n

In other words, now that Elasticsearch supports a leader\/follower pattern, GitHub Enterprise Server administrators will no longer be left in a state where critical data winds up on read-only nodes.<\/p>\n

Under the hood<\/h2>\n

Elasticsearch has an auto-follow API, but it only applies to indexes created after<\/em> the policy exists. GitHub Enterprise Server HA installations already have a long-lived set of indexes, so we need a bootstrap step that attaches followers to existing indexes, then enables auto-follow for anything created in the future.<\/p>\n

Here\u2019s a sample of what that workflow looks like:<\/p>\n

function bootstrap_ccr(primary, replica):\n  # Fetch the current indexes on each \n  primary_indexes = list_indexes(primary)\n  replica_indexes = list_indexes(replica)\n\n  # Filter out the system indexes\n  managed = filter(primary_indexes, is_managed_ghe_index)\n  \n  # For indexes without follower patterns we need to\n  #   initialize that contract\n  for index in managed:\n    if index not in replica_indexes:\n      ensure_follower_index(replica, leader=primary, index=index)\n    else:\n      ensure_following(replica, leader=primary, index=index)\n\n  # Finally we will setup auto-follower patterns \n  #   so new indexes are automatically followed\n  ensure_auto_follow_policy(\n    replica,\n    leader=primary,\n    patterns=[managed_index_patterns],\n    exclude=[system_index_patterns]\n  )<\/code><\/pre>\n
This is just one of the new workflows we\u2019ve created to enable CCR in GitHub Enterprise Server. We\u2019ve needed to engineer custom workflows for failover, index deletion, and upgrades. Elasticsearch only handles the document replication, and we\u2019re responsible for the rest of the index\u2019s lifecycle.\u00a0<\/p>\n
How to get started with CCR mode\u00a0<\/h2>\n
To get started using the new CCR mode, reach out to support@github.com<\/a> and let them know you\u2019d like to use the new HA mode for GitHub Enterprise Server. They\u2019ll set up your organization so that you can download the required license.<\/p>\n
Once you\u2019ve downloaded your new license, you\u2019ll need to set `ghe-config app.elasticsearch.ccr true`. With that finished, administrators can run a `config-apply` or an upgrade on your cluster to move to 3.19.1, which is the first release to support this new architecture.\u00a0\u00a0<\/p>\n
When your GitHub Enterprise Server restarts, Elasticsearch will migrate your installation to use the new replication method. This will consolidate all the data onto the primary nodes, break clustering across nodes, and restart replication using CCR. This update may take some time depending on the size of your GitHub Enterprise Server instance.<\/p>\n
While the new HA method is optional for now, we\u2019ll be making it our default over the next two years. We want to ensure there\u2019s ample time for GitHub Enterprise administrators to get their feedback in, so now is the time to try it out.\u00a0<\/p>\n
We\u2019re excited for you to start using the new HA mode for a more seamless experience managing GitHub Enterprise Server.\u00a0<\/p>\n
\n
Want to get the most out of search on your High Availability GitHub Enterprise Server deployment? Reach out to support<\/a> to get set up with our new search architecture!<\/p>\n<\/div>\n
The post How we rebuilt the search architecture for high availability in GitHub Enterprise Server<\/a> appeared first on The GitHub Blog<\/a>.<\/p>","protected":false},"excerpt":{"rendered":"
So much of what you interact with on GitHub depends on search\u2014obviously the search bars and filtering experiences like the […]<\/p>\n","protected":false},"author":1,"featured_media":3553,"comment_status":"","ping_status":"","sticky":false,"template":"","format":"standard","meta":{"site-sidebar-layout":"default","site-content-layout":"","ast-site-content-layout":"default","site-content-style":"default","site-sidebar-style":"default","ast-global-header-display":"","ast-banner-title-visibility":"","ast-main-header-display":"","ast-hfb-above-header-display":"","ast-hfb-below-header-display":"","ast-hfb-mobile-header-display":"","site-post-title":"","ast-breadcrumbs-content":"","ast-featured-img":"","footer-sml-layout":"","ast-disable-related-posts":"","theme-transparent-header-meta":"","adv-header-id-meta":"","stick-header-meta":"","header-above-stick-meta":"","header-main-stick-meta":"","header-below-stick-meta":"","astra-migrate-meta-layouts":"default","ast-page-background-enabled":"default","ast-page-background-meta":{"desktop":{"background-color":"var(--ast-global-color-4)","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""},"tablet":{"background-color":"","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""},"mobile":{"background-color":"","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""}},"ast-content-background-meta":{"desktop":{"background-color":"var(--ast-global-color-5)","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""},"tablet":{"background-color":"var(--ast-global-color-5)","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""},"mobile":{"background-color":"var(--ast-global-color-5)","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""}},"footnotes":""},"categories":[8],"tags":[],"class_list":["post-3552","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-github-engineering"],"_links":{"self":[{"href":"https:\/\/rssfeedtelegrambot.bnaya.co.il\/index.php\/wp-json\/wp\/v2\/posts\/3552","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/rssfeedtelegrambot.bnaya.co.il\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/rssfeedtelegrambot.bnaya.co.il\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/rssfeedtelegrambot.bnaya.co.il\/index.php\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/rssfeedtelegrambot.bnaya.co.il\/index.php\/wp-json\/wp\/v2\/comments?post=3552"}],"version-history":[{"count":0,"href":"https:\/\/rssfeedtelegrambot.bnaya.co.il\/index.php\/wp-json\/wp\/v2\/posts\/3552\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/rssfeedtelegrambot.bnaya.co.il\/index.php\/wp-json\/wp\/v2\/media\/3553"}],"wp:attachment":[{"href":"https:\/\/rssfeedtelegrambot.bnaya.co.il\/index.php\/wp-json\/wp\/v2\/media?parent=3552"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/rssfeedtelegrambot.bnaya.co.il\/index.php\/wp-json\/wp\/v2\/categories?post=3552"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/rssfeedtelegrambot.bnaya.co.il\/index.php\/wp-json\/wp\/v2\/tags?post=3552"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}