{"id":5283,"date":"2026-10-14T22:12:44","date_gmt":"2026-10-14T22:12:44","guid":{"rendered":"https:\/\/rssfeedtelegrambot.bnaya.co.il\/index.php\/2026\/10\/14\/harbor-docker-sandboxes-run-safe-reproducible-agent-evals\/"},"modified":"2026-10-14T22:12:44","modified_gmt":"2026-10-14T22:12:44","slug":"harbor-docker-sandboxes-run-safe-reproducible-agent-evals","status":"publish","type":"post","link":"https:\/\/rssfeedtelegrambot.bnaya.co.il\/index.php\/2026\/10\/14\/harbor-docker-sandboxes-run-safe-reproducible-agent-evals\/","title":{"rendered":"Harbor + Docker Sandboxes: Run Safe, Reproducible Agent Evals"},"content":{"rendered":"<div class=\"wp-block-ponyo-zeta organism toc-exclude\">\n<blockquote class=\"container\">\n<h2 class=\"wp-block-ponyo-heading text-lg\">\n        Key takeaways<br \/>\n    <\/h2>\n<ul class=\"wp-block-list\">\n<li><strong>Harbor can now run every eval task in Docker Sandboxes<\/strong><\/li>\n<li>Docker Sandboxes are MicroVM environments to run agents like Claude Code, Codex, Cursor, and more safely<\/li>\n<li>With Sandboxes, define custom network policies for fair, reproducible evals<\/li>\n<li>Run your evals locally or in the cloud: starting for free on Docker Sandboxes on local surfaces and scaling to Cloud Sandboxes<\/li>\n<\/ul>\n<\/blockquote>\n<\/div>\n<p class=\"wp-block-paragraph\">To evaluate an agent, you give it a real task, let it work, and check the result. Along the way, the agent installs packages, edits files, and runs commands. That\u2019s untrusted, AI-written code, and a benchmark runs it thousands of times.<\/p>\n<p class=\"wp-block-paragraph\">And agents don\u2019t always play fair. An agent can sometimes earn a passing score on an eval by gaming the check instead of doing the work, for example by tampering with tests or overwriting the file that records its score. This is called reward hacking, and it\u2019s a growing concern across the eval community.<\/p>\n<p class=\"wp-block-paragraph\">So running agent evals needs two things:<\/p>\n<ol class=\"wp-block-list\">\n<li><strong>An eval framework<\/strong> that runs each task, lets the agent work, checks the result, and collects the scores and logs.<\/li>\n<li><strong>An environment for every task<\/strong> that\u2019s clean and isolated, so evaluating agents is safe and consistent.<\/li>\n<\/ol>\n<p class=\"wp-block-paragraph\"><strong><a href=\"https:\/\/www.harborframework.com\/\" rel=\"nofollow noopener\" target=\"_blank\">Harbor<\/a> gives you the first.<\/strong> It\u2019s an open-source eval framework. Give it a benchmark like Terminal-Bench, and it simplifies the process of running evals.\u00a0<\/p>\n<p class=\"wp-block-paragraph\"><strong>Now, <a href=\"https:\/\/www.docker.com\/products\/docker-sandboxes\/\">Docker Sandboxes<\/a> gives you the second:<\/strong> a safe way to run those eval tasks, on your laptop or in the cloud.<\/p>\n<h2 class=\"wp-block-heading\">Here\u2019s what this means for you<\/h2>\n<p class=\"wp-block-paragraph\"><strong>Strong isolation by default.<\/strong> Docker Sandboxes were <a href=\"https:\/\/www.docker.com\/blog\/why-microvms-the-architecture-behind-docker-sandboxes\/\">designed from the ground up<\/a> for deeper isolation. This means you can run every task in a microVM, not a shared container. A microVM has its own kernel, so whatever the agent does stays inside its sandbox. It can\u2019t break out to reach your machine, other tasks, or anything else outside its environment.<\/p>\n<p class=\"wp-block-paragraph\"><strong>Fair, reproducible evals.<\/strong> Configurable network policies let, for example, a task download what it needs during setup, then cut the agent off from the internet while it works. This way, every agent can be tested under the same conditions.\u00a0<\/p>\n<p class=\"wp-block-paragraph\"><strong>Parallel runs, no machines to manage.<\/strong> With Docker Cloud Sandboxes, teams can run many tasks at once without needing to manually provision anything.\u00a0Run dozens of concurrent workers simultaneously.*<\/p>\n<p class=\"wp-block-paragraph\"><strong>Develop locally, scale in the cloud.<\/strong> The same tasks run locally or in Docker Cloud Sandboxes. Build on your laptop, then run your tasks in the background, and at scale, without changing anything.<\/p>\n<h2 class=\"wp-block-heading\">Try it<\/h2>\n<p class=\"wp-block-paragraph\">Install Harbor with Docker Sandboxes.<\/p>\n<p class=\"wp-block-paragraph\"><code>uv tool install \"harbor[sbx]\"<\/code><\/p>\n<p class=\"wp-block-paragraph\"><a href=\"https:\/\/docs.docker.com\/agentic-platform\/signup\/\" rel=\"nofollow noopener\" target=\"_blank\">Activate Cloud access<\/a> for your Docker account and <a href=\"https:\/\/docs.docker.com\/security\/access-tokens\/personal-access-tokens\/#create-a-personal-access-token\" rel=\"nofollow noopener\" target=\"_blank\">create a personal access token<\/a>. Select <strong>Cloud Sandboxes<\/strong> under access permissions. Copy and save your token. It\u2019s shown only once.<\/p>\n<ol class=\"wp-block-list\">\n<li>Set your Docker ID (your Docker login username) and PAT (personal access token), then run a task in Cloud Sandboxes.<\/li>\n<li><code>export SBX_DOCKER_USERNAME=\"your-docker-id\"<\/code><\/li>\n<li><code>export SBX_DOCKER_PAT=\"your-personal-access-token\"<\/code><\/li>\n<li><code>harbor run -t hello-world\/hello-world -a oracle -e sbx<\/code><\/li>\n<\/ol>\n<p class=\"wp-block-paragraph\">To run locally, install the <a href=\"https:\/\/docs.docker.com\/ai\/sandboxes\/\" rel=\"nofollow noopener\" target=\"_blank\">Docker Sandboxes CLI<\/a>, run sbx login, and add <code>--ek target=local<\/code> to the command above.<\/p>\n<p class=\"wp-block-paragraph\">Today\u2019s integration supports single-container tasks that run on CPUs. GPU and multi-container support are coming soon.<\/p>\n<p class=\"wp-block-paragraph\">To learn more, see the <a href=\"https:\/\/docs.docker.com\/ai\/sandboxes\/\" rel=\"nofollow noopener\" target=\"_blank\">Docker Sandboxes docs<\/a> and the <a href=\"https:\/\/docs.harborframework.com\/\" rel=\"nofollow noopener\" target=\"_blank\">Harbor docs<\/a>.<\/p>\n<p class=\"wp-block-paragraph\"><em>*We\u2019ve tested Harbor with 64 concurrent workers on Docker Cloud Sandboxes. <\/em><\/p>","protected":false},"excerpt":{"rendered":"<p>Key takeaways Harbor can now run every eval task in Docker Sandboxes Docker Sandboxes are MicroVM environments to run agents [&hellip;]<\/p>\n","protected":false},"author":1,"featured_media":94,"comment_status":"","ping_status":"","sticky":false,"template":"","format":"standard","meta":{"site-sidebar-layout":"default","site-content-layout":"","ast-site-content-layout":"default","site-content-style":"default","site-sidebar-style":"default","ast-global-header-display":"","ast-banner-title-visibility":"","ast-main-header-display":"","ast-hfb-above-header-display":"","ast-hfb-below-header-display":"","ast-hfb-mobile-header-display":"","site-post-title":"","ast-breadcrumbs-content":"","ast-featured-img":"","footer-sml-layout":"","ast-disable-related-posts":"","theme-transparent-header-meta":"","adv-header-id-meta":"","stick-header-meta":"","header-above-stick-meta":"","header-main-stick-meta":"","header-below-stick-meta":"","astra-migrate-meta-layouts":"default","ast-page-background-enabled":"default","ast-page-background-meta":{"desktop":{"background-color":"var(--ast-global-color-4)","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""},"tablet":{"background-color":"","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""},"mobile":{"background-color":"","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""}},"ast-content-background-meta":{"desktop":{"background-color":"var(--ast-global-color-5)","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""},"tablet":{"background-color":"var(--ast-global-color-5)","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""},"mobile":{"background-color":"var(--ast-global-color-5)","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""}},"footnotes":""},"categories":[4],"tags":[],"class_list":["post-5283","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-docker"],"_links":{"self":[{"href":"https:\/\/rssfeedtelegrambot.bnaya.co.il\/index.php\/wp-json\/wp\/v2\/posts\/5283","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/rssfeedtelegrambot.bnaya.co.il\/index.php\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/rssfeedtelegrambot.bnaya.co.il\/index.php\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/rssfeedtelegrambot.bnaya.co.il\/index.php\/wp-json\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/rssfeedtelegrambot.bnaya.co.il\/index.php\/wp-json\/wp\/v2\/comments?post=5283"}],"version-history":[{"count":0,"href":"https:\/\/rssfeedtelegrambot.bnaya.co.il\/index.php\/wp-json\/wp\/v2\/posts\/5283\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/rssfeedtelegrambot.bnaya.co.il\/index.php\/wp-json\/wp\/v2\/media\/94"}],"wp:attachment":[{"href":"https:\/\/rssfeedtelegrambot.bnaya.co.il\/index.php\/wp-json\/wp\/v2\/media?parent=5283"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/rssfeedtelegrambot.bnaya.co.il\/index.php\/wp-json\/wp\/v2\/categories?post=5283"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/rssfeedtelegrambot.bnaya.co.il\/index.php\/wp-json\/wp\/v2\/tags?post=5283"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}