Predicting for Sizing: Teaching Supercomputers Their Own Execution Footprint

A continuous-learning predictive model that estimates the execution time, memory, and energy of scientific applications on supercomputers — so researchers can size their allocations instead of guessing.

A conceptual illustration contrasting the initial phase of "guessing" with the clarity offered by a continuous learning model.

Research note · September 23, 2026 · 5 min read

The Sizing Problem

Every scientific campaign on a supercomputer begins with a bet. How many nodes? How much memory? How long will this run take? For the DDF Pipeline — the self-calibration and imaging software behind the LOFAR radio telescope — the stakes are enormous: the latest LoTSS (LOFAR Two-metre Sky Survey) data release alone consumed 16 million core-hours. And behind every one of those hours hides a sizing decision made, more often than not, in the dark.

The astronomers who use DDF Pipeline are scientists, not HPC experts. They allocate their project’s precious computing hours through SLURM, a scheduler that asks them to state their needs up front (--time, --cpus-per-task, --mem) and then enforces a fairness policy that rewards projects which consume less than they were granted. Over-allocate and you waste resources, lose priority, and wait in longer queues; under-allocate and your job simply fails. Translating scientific parameters into correct sizing is a guessing game with real costs on both sides.

The DDF Pipeline workflow — three steps for which we would like to know the most suitable site for executing them.

Why Guessing Is So Expensive

The difficulty is not a lack of experience — it is a lack of predictability. DDF Pipeline runs vary along several dimensions at once, and a single change can move the resource footprint by an order of magnitude:

Add a multi-site dimension and the problem compounds. In cross-facility workflows, resource sizing drives task placement decisions: disk space, queue wait times, and where a job can go all depend on the footprint you predict. Hardware constraints differ from site to site — 768 GB per node on Adastra, 192 GB of memory per node on Jean Zay, 228 GB per node on Joliot-Curie — so a configuration that fits one machine may be hopeless on another.

The configuration space is simply too large to test exhaustively. This is not a problem you can solve once: it demands a model that keeps learning as executions accumulate.

What We Need to Succeed

Building a continuous-learning predictor is a team effort across five pieces, each developed by a different specialist and designed to fit together:

  1. Structured knowledge map — a description of the supercomputer-specific runtime environment, mapped onto SLURM metrics.
  2. Job Performance Indicators Set, JoPInS — the structure containing the measurements of the formal descriptor applications jobs performance indicator relative to their executions.
  3. Monitoring plugin — the collector that captures evidence from real jobs.
  4. Database — the central store where all executions accumulate in the JoPInS format.
  5. Machine learning exposed as HAPS — a model trained by application version and by runtime environment, with access to HPC resources and applications for its continuous learning.

Step one — A structured knowledge map of the runtime environment

Before we can predict, we must be able to describe. The structured knowledge map (or ontology) formalizes the supercomputer-specific runtime environment in which jobs execute. Its value lies in the mapping: aligning the job structured knowledge map with the raw SLURM metrics so that a conceptual description — “this job needed a large-memory node” — becomes a measurable, comparable quantity.

The job structured knowledge map — the bridge between how researchers think about their runs and how SLURM measures them. Credit: Gaëlle Richet.

Step two — Describing every execution with JoPInS

Every execution is captured by a Job Performance Indicators Set, JoPInS. The descriptor defines the input parameters — application settings and system choices — and the output metrics — time, memory, energy — for each job. Each indicator is represented as a triple: JoPInS = (value, type of value, description), a structure simple enough to collect at scale and rich enough to train on.

Step three — Collecting evidence from the field

A pair of plugins turns real jobs into training data. The command-line client, cli-monitoring-haps, collects information about jobs from a specific application, exposed as HAPS, in the context of a data release campaign. The server-side collector, server-monitoring-haps, reads system metrics from SLURM (sacct -j <JOB_ID>), pulls application-specific metrics from the HAPS database. Each execution is then formalized into JoPInS and sent to the database.

Step four — Centralizing in Database

All of those JoPInS records land in the database — the single source of truth for how applications actually behave on real machines. It is the memory of the system, and the fuel for the model.

Step five — Connecting the machine learning to the system

The loop is closed around HAPS (HPC Application Services), in a setup that remains compliant with the ZRR security requirements imposed on French supercomputers, which are considered national strategic assets. Step 1: campaign execution of scientific applications on supercomputers. Step 2: data collection through the monitoring plugin. Step 3: model training and testing — an machine learning model is trained on a specific dataset campaign, tested, its accuracy measured, and only then exposed as HAPS itself.

From Execution to Prediction

Once the loop is running, the pieces produce something new: a machine-learning model exposed as a service. Train an expert model on a campaign dataset, measure its accuracy, and the model becomes an oracle that predicts, for any new configuration, the execution time, memory usage, and energy consumption of the application. The goal is practical: given the variability of application parameters and input data, identify the allocation configurations that offer the best trade-off between execution time and energy consumption — before a single core-hour is spent.

From Guessing to Knowing

A scheduler asks you to predict your own future — and traditionally, that prediction has been a judgment call. With a continuously learning model fed by real executions, the conversation changes. The supercomputer stops asking its users to guess and starts telling them what they need. Sizing stops being a bet — and scientists can spend their core-hours on science, not on hunches.

Keep reading