Borg Cluster Management Overview

To make efficient use of Google's fleet of machines.

By spreading tasks of a job across failure domains like machines and racks.

It can result in savings of millions of dollars.

<p>The original size and combined workload of each cell.</p> Signup and view all the answers

<p>It would require more machines.</p> Signup and view all the answers

<p>The additional machines needed when workloads are separated into different cells.</p> Signup and view all the answers

<p>To maintain operational efficiency and minimize downtime.</p> Signup and view all the answers

<p>They show the percentage of extra machines required for segregated workloads.</p> Signup and view all the answers

<p>Priorities determine the order in which jobs receive resources, with higher priority tasks able to obtain resources at the expense of lower priority ones.</p> Signup and view all the answers

<p>The BNS provides a stable name for each task, allowing clients and systems to locate tasks even after they are relocated.</p> Signup and view all the answers

<p>Borg uses priority and quota systems to manage excess work, ensuring that resources are allocated based on task priority.</p> Signup and view all the answers

<p>Preemption can lead to a cascading effect where a higher priority task bumps a lower priority task, causing further preemptions among lower-ranked tasks.</p> Signup and view all the answers

<p>Borg defines non-overlapping priority bands for monitoring, production, batch, and best effort tasks.</p> Signup and view all the answers

<p>This restriction prevents instability and ensures that critical production tasks are not prematurely terminated by other tasks.</p> Signup and view all the answers

<p>Borg writes job size and task health information into Chubby, ensuring that load balancers have updated data for routing requests.</p> Signup and view all the answers

<p>The task's DNS name follows the format 'task_number.job_name.user_name.cell.borg.google.com'.</p> Signup and view all the answers

<p>The resource limit is used to determine if the user has enough quota to admit the job and if a machine has sufficient free resources to schedule the task.</p> Signup and view all the answers

<p>Borg typically kills tasks that try to use more RAM or disk space than requested and throttles CPU usage to the specified limit.</p> Signup and view all the answers

<p>Users who request excessive resources may not utilize them efficiently, leading to waste and potential scheduling delays.</p> Signup and view all the answers

<p>Tasks may need to use all their resources during peak times or when responding to events like denial-of-service attacks.</p> Signup and view all the answers

<p>The 1ms threshold indicates how often a runnable thread must wait longer than 1ms to access a CPU.</p> Signup and view all the answers

<p>Threads almost never waited longer than 5ms to access a CPU.</p> Signup and view all the answers

<p>Latency-sensitive tasks generally require quicker access to CPU resources compared to batch tasks, impacting scheduling delays.</p> Signup and view all the answers

<p>The error bars represent day-to-day variance in scheduling delays over the month of December 2013.</p> Signup and view all the answers

<p>The Borglet uses post-hoc usage checking of memory, disk space, and CPU cycles.</p> Signup and view all the answers

<p>Users inflate their resource requests to reduce the number of tasks that Borg can co-schedule, aiming to increase their task utilization.</p> Signup and view all the answers

<p>The standard Linux CPU scheduler, known as Completely Fair Scheduler (CFS), requires substantial tuning.</p> Signup and view all the answers

<p>The use of a thread-per-request model in many applications helps mitigate persistent load imbalances.</p> Signup and view all the answers

<p>The Borglet selectively terminates tasks that use excessive memory or disk resources.</p> Signup and view all the answers

<p>There is ongoing work to improve thread placement, CPU management that is NUMA-, hyperthreading-, and power-aware.</p> Signup and view all the answers

<p>Cpusets are sparingly used to allocate CPU cores to applications with particularly tight latency requirements.</p> Signup and view all the answers

<p>Occasional low-level resource interference, such as memory bandwidth or L3 cache pollution, can still occur.</p> Signup and view all the answers

<p>Borg surfaces debugging information to all users rather than hiding it.</p> Signup and view all the answers

<p>Borg finds it harder to deprecate features because users rely on them.</p> Signup and view all the answers

<p>Borg provides several levels of UI and debugging tools to identify anomalous events.</p> Signup and view all the answers

<p>Kubernetes aims to replicate many of Borg's introspection techniques.</p> Signup and view all the answers

<p>Kubernetes uses a unified mechanism for all components to record events.</p> Signup and view all the answers

<p>The initial Borgmaster was primarily designed and implemented by Jeremy Dion and Mark Vandevoorde.</p> Signup and view all the answers

<p>The master is the kernel of a distributed system in Borg.</p> Signup and view all the answers

<p>These tools are utilized for log aggregation in Kubernetes.</p> Signup and view all the answers

<p>Apollo's opportunistic execution boosts utilization by allowing lower-priority background tasks to run, which can lead to multi-day queueing delays.</p> Signup and view all the answers

<p>Apache Mesos uses a central resource manager with an offer-based mechanism, while Borg employs a central scheduler with a request-based mechanism.</p> Signup and view all the answers

<p>YARN is Hadoop-centric and includes a manager that negotiates resource requests specifically for Hadoop applications.</p> Signup and view all the answers

<p>The studies highlight challenges of scale and heterogeneity within modern datacenters and workloads.</p> Signup and view all the answers

<p>Alibaba's Fuxi has been running since 2009, supporting data-analysis workloads.</p> Signup and view all the answers

<p>Apollo nodes offer a prediction matrix for starting times of tasks based on two resource dimensions and estimates of startup costs.</p> Signup and view all the answers

<p>DRF focuses on high-availability and long-running applications with priority and admission quotas, while Borg uses a priority-based allocation model.</p> Signup and view all the answers

<p>The Mesos developers aim to extend their system to include speculative resource assignment and reclamation.</p> Signup and view all the answers

<p>Users initiate job operations by issuing remote procedure calls (RPCs) to Borg, commonly using a command-line tool.</p> Signup and view all the answers

<p>Updates to a running job configuration are lightweight and non-atomic transactions that can easily be undone until closed.</p> Signup and view all the answers

<p>BCL includes declarative keywords and lambda functions that allow dynamic calculations and environmental adjustments.</p> Signup and view all the answers

<p>Rolling updates allow changes to be applied progressively, limiting task disruptions and ensuring ongoing job reliability.</p> Signup and view all the answers

<p>Tasks in Borg can transition through states such as Pending, Running, and Dead, controlled by user-triggered actions.</p> Signup and view all the answers

<p>A cluster is defined by the high-performance datacenter-scale network fabric connecting machines, and it typically resides within a single datacenter building.</p> Signup and view all the answers

<p>A Borg job includes properties such as its name, owner, and number of tasks, and can impose constraints on where tasks run based on machine attributes.</p> Signup and view all the answers

<p>Borg isolates users from machine heterogeneity by managing task allocation, resources, and dependencies, ensuring efficient operation despite differences.</p> Signup and view all the answers

<p>Hard constraints must be met for a task to run on a machine, while soft constraints serve as preferences that can be relaxed if necessary.</p> Signup and view all the answers

<p>Defining task resource requirements independently allows fine-grained control over resource allocation, enabling efficient utilization of CPU, RAM, and more.</p> Signup and view all the answers

<p>Borg minimizes overhead by avoiding the use of virtual machines for the majority of its workloads, opting instead for processes running directly in containers.</p> Signup and view all the answers

<p>Task monitoring is crucial in Borg for ensuring task health, enabling automatic restarts if they fail and maintaining overall system reliability.</p> Signup and view all the answers

<p>Borg prefers statically linked programs to reduce dependencies on the runtime environment, enhancing portability and reliability.</p> Signup and view all the answers

<p>Preemption cascades can occur, where a high-priority task bumps a lower-priority one, creating a chain reaction of preemptions.</p> Signup and view all the answers

<p>Quota ensures that jobs are admitted for scheduling based on resource availability and prevents jobs from exceeding their resource allocations.</p> Signup and view all the answers

<p>Tasks use the Unix SIGTERM signal.</p> Signup and view all the answers

<p>Borg uses fine-grained priorities to differentiate between task importance, allowing master tasks to run at a higher priority than their worker tasks.</p> Signup and view all the answers

<p>Jobs that lack sufficient quota are immediately rejected upon submission.</p> Signup and view all the answers

<p>Borg assigns a small positive integer as priority, allowing higher-priority tasks to obtain resources at the expense of lower-priority ones.</p> Signup and view all the answers

<p>The Borg name service assigns stable names to tasks, enabling clients and other systems to reliably locate tasks even after relocation.</p> Signup and view all the answers

<p>The tasks associated with the alloc are rescheduled alongside it.</p> Signup and view all the answers

<p>An alloc set reserves resources on multiple machines for one or more jobs.</p> Signup and view all the answers

<p>Users often overbuy quota to guard against future shortages as their application’s demand grows.</p> Signup and view all the answers

<p>Chubby provides a consistent and highly-available store for task information, such as task health and job size, which is used by other systems for monitoring.</p> Signup and view all the answers

<p>When workload exceeds available resources, tasks may be preempted or denied entry based on defined priorities and quotas.</p> Signup and view all the answers

<p>Tasks in these bands are given precedence and can preempt lower-priority tasks.</p> Signup and view all the answers

<p>Production-priority jobs are given access to resources limited to what is available in the cell, ensuring they can run as expected if within quota.</p> Signup and view all the answers

<p>SIGKILL is used to forcibly terminate tasks that do not respond to a SIGTERM notice.</p> Signup and view all the answers

<p>The stable naming convention allows consistent tracking and monitoring of tasks, essential for providing reliable service to clients.</p> Signup and view all the answers

<p>Borg's capability system allows certain users, like administrators, to perform privileged actions such as modifying any job or accessing kernel features.</p> Signup and view all the answers

<p>Multiple tasks share the resources assigned to an alloc, enabling efficient utilization.</p> Signup and view all the answers