Ast waste-compute recovery project

Supercomputing

This codebase turns deliberately enrolled, underused machines into a heterogeneous execution mesh. Its center is BagManager.js: a coordinator that discovers workers, measures capacity, dispatches redundant copies of work, validates the first returned result, and releases the slower copies.

Runnable architecture

One task, several eligible workers, one accepted result

Submit HTTP routes accept individual or batch jobs and workload-assistance requests.
Qualify Workers advertise capacity; dispatch excludes devices above CPU or memory pressure limits.
Replicate A task slice is sent to a configurable number of independent responders, three by default.
Resolve The first valid result wins; late responders are discarded and capacity returns to the mesh.
Protocol inventory

JavaScript, Python, and native workers

NodeAgent.js implements the Node worker path. CommunicationProtocol.py supplies framed TCP for Python. MicroDistribute C and C++ sources test smaller native-worker contracts. wire_protocol.js normalizes messages at the boundary rather than pretending every runtime has the same object model.

all-stack-runnable.js is the integration launcher for exercising the coordinator and available language workers together. The browser control plane is a view over the mesh; the compute protocol remains independent of the display.

Repository defaults
Setting Value
Worker/coordinator TCP 4000
HTTP control plane 4001
Multicast discovery 239.255.0.1:45454
Dispatch redundancy 3 responders
Idle CPU threshold Below 40 percent
Critical pressure CPU 95 percent; memory 98.5 percent
Control plane

What the implementation exposes

Route family Use
/health, /capacity, /agents Coordinator, participant, and available-compute observations
/tasks, /jobs, /jobs/batch Inspectable task submission and lifecycle
/mesh, /mesh/probe Participating endpoints and configured network discovery
/memory/lease Bounded temporary worker-memory allocation
/assist/jobs External workload assistance under the same capacity rules
Why a bag instead of a cluster

Most available computation is irregular, temporary, and attached to something else.

A conventional cluster begins with machines purchased to be a cluster. The Bag model begins with machines that already exist for other reasons: a workstation between interactive jobs, a server below its normal load, a small native process, or a computer that can contribute briefly before its owner needs it again. Their processors, memory, runtimes, and connection quality differ. The coordinator's job is therefore not to make them look identical. It is to describe work narrowly enough that different participants can attempt it, and to treat availability as a changing measurement rather than a permanent entitlement.

Redundant first-responder dispatch follows from that premise. Predicting which mixed worker will finish first can cost more than sending the same bounded slice to several eligible workers. The default three-way dispatch turns variation into a race: the first valid result is accepted and late work is abandoned. This spends extra aggregate cycles to reduce tail latency and tolerate disappearance. It is useful only when slices are independent, result validation is cheaper than execution, and the redundant load does not exhaust the very idle capacity the system is trying to recover.

The mesh is not a smaller cloud. It is a scheduler for computation that would otherwise expire unused.

Capacity is a lease, not a possession

A worker advertises what it can offer now. CPU and memory thresholds protect the primary purpose of the machine, while memory leases make temporary allocation explicit. If local demand rises, contribution must contract. This is why cancellation and clean release are part of useful computation rather than administrative extras. Work that cannot survive a worker leaving is poorly shaped for this mesh.

Heterogeneity belongs at the protocol edge

The Node, Python, C, and C++ paths do not share a runtime, package graph, or object representation. They share framed messages and a small work contract. Serialization, task identity, deadlines, result validation, and capability descriptions cross the wire; language-specific machinery stays inside the worker. This permits a native experiment and a Node process to participate without requiring one to embed the other.

The control API makes the system inspectable from outside the scheduler. Capacity, agents, jobs, tasks, discovery, workloads, and leases can be observed separately. That separation matters during failure: a healthy coordinator with no eligible capacity is different from a dead coordinator; a submitted job with no responder is different from an invalid result; a discovered endpoint is different from an enrolled worker.

The hard problem is useful decomposition

No dispatch strategy can parallelize work whose pieces constantly mutate shared state. The productive workloads are those that can be divided into opaque chunks, checked independently, retried safely, and merged deterministically: search partitions, simulations, rendering tiles, batch transforms, test matrices, candidate evaluation, and other bounded maps. The research value of Supercomputing lies as much in discovering those decompositions as in discovering machines.

Implemented boundary

Opt-in workers are real; frictionless recruitment is a roadmap

The repository contains runnable coordinator, worker, API, discovery, and validation paths. It also discusses SSH, WinRM, browser workers, PXE, and broader recruitment. Those are roadmap mechanisms, not equivalent to the current opt-in worker implementation. A reachable device is not an authorized device.

The mesh handles opaque work chunks and results. It does not need a worker's personal files to contribute computation. Enrollment, visibility, capacity limits, cancellation, and clean termination are part of the compute contract.