Changelog
Source:NEWS.md
EpiModelHPC 2.10.0
NEW FEATURES
- Restart pools: a directory of single simulation restart points (
1.rds,2.rds, …) with apool_index.csvcan now be passed aspath_to_xtonetsim_scenarios(),step_tmpl_netsim_scenarios(),netsim_swfcalib_output()andstep_tmpl_netsim_swfcalib_output(). Simulationkof a scenario (numbered over all its batches) starts from pool element(k - 1) %% N + 1, the same for every scenario, so scenarios can be compared simulation by simulation. -
make_restart_pool()makes such a pool from the batch files of a previous run, withEpiModel::make_restart_point(). Pool elementiis made from rowiofkeep_sims, and apool_index.csvfile records the batch, simulation and EpiModel version of each element. -
validate_restart_pool()checks a pool before use: a file for each element ofpool_index.csv, each a single simulation restart point made with the installed EpiModel major.minor version, all with the same number of time steps. It returns a table of the elements (number of simulations, number of time steps kept). -
netsim_path_wrapper()runs onenetsim()call per simulation, from a pool or from a single file, and is now what every batch ofnetsim_scenarios()andstep_tmpl_netsim_scenarios()runs. Batches are then merged withmerge(). Before running from a pool, it checks frompool_index.csvthat the elements it uses were made with the installed EpiModel major.minor version. -
merge_netsim_scenarios()andstep_tmpl_merge_netsim_scenarios()gain thekeep.run,keep.cumulative.edgelistandkeep.attr.historyarguments ofmerge.netsim().
BREAKING CHANGES
- Requires EpiModel 2.7.0: restart points holding a single simulation (
make_restart_point(sim_num = )), thekeep.run,keep.cumulative.edgelistandkeep.attr.historyarguments ofmerge.netsim(), and merges of batches in sequence. - Each simulation of a batch is now its own
netsim()call (one perfutureworker). Restarting from a single file holding several simulations (e.g. a merged restart object) restarts every simulation from the first one; make a pool withmake_restart_pool()instead. Runs from a fitted network model read it once per simulation. - The scenario runners reject
control$raw.output = TRUEat setup. - With checkpointing, each simulation checkpoints in
<.checkpoint.dir>/sim__<scenario>__<batch>/sim_<k>. -
step_tmpl_netsim_swfcalib_output()now runs its batches withfuture::plan("multicore", workers = n_cores)by default, likestep_tmpl_netsim_scenarios(). It previously used themultisessiondefault ofEpiModel::netsim(), whose PSOCK workers can stall at startup and outlive a cancelled job on the HPC. The localnetsim_scenarios()andnetsim_swfcalib_output()keepmultisession, which also runs in parallel from RStudio and on Windows.control$future.use.plan <- future::tweak(<your plan>)still overrides the default.
BUG FIXES
-
get_scenarios_batches_infos()sorts the batch files by scenario, then batch number. They were listed in alphabetical order of their file names (batch10before batch2), somerge_netsim_scenarios()merged the batches of a scenario with 10 or more batches out of order, and simulationkof the merged object was not simulationkof the scenario. -
step_tmpl_merge_netsim_scenarios()passedkeep.diss.statsandparam.errorin swapped order to its inner function, and swapped them back when callingmerge_netsim_scenarios(). Arguments are now passed by name.
EpiModelHPC 2.9.3
BREAKING CHANGES
-
swf_configs_rsph()targets the current RSPH partitions.partitiondefaults to"epimodel", the lab’s dedicated nodes, instead of"preemptable", which no longer exists on the cluster and now gives an error saying so. It acceptsepimodel,CIDMATH,short-cpu,day-long-cpu,week-long-cpu,month-long-cpu,interactive-cpuandlargemem, as a vector or a comma-separated string, so a step can start on whichever listed partition has room first. -
swf_configs_rsph()defaultsr_versionto 4.6.1, the current spack build. Projects whose lockfile records R 4.5 passr_version = "4.5.1".
NEW FEATURES
-
swf_configs_rsph()returnsmax_cores, the largestcpus-per-taska single task can request on every listed partition: 32, or 16 wheninteractive-cpuis listed and 31 whenlargememis. -
swf_configs_rsph()warns whenr_versionis not a full version.spack load r@4.3fails because it matches three builds. -
?swf_configs_rsphdocuments the time and core limits of each partition, the 5 GB per CPU memory default, and the array size ceiling.
OTHER
- The restore step from
swf_configs_rsph()gets 180 minutes instead of 120, as in the EpiModelHIV-Template. A first restore under a new R version builds every package from source. - The
epimodelhiv-slurmworkflowvignette is rewritten for the current EpiModelHIV-Template:make_em_workflow()inR/hpc_configs.R,renv.lock.hpcrestored on compute nodes by the first step,data/run/, the current partitions andmax_array_size = 1000. It no longer passessave_pattern(removed in 2.5.0) or reserves whole nodes for simulation steps, and it points to the Template chapter READMEs and the wiki for the step-by-step instructions.
EpiModelHPC 2.9.2
NEW FEATURES
-
degen_watch.shannounces a task that exhaustsMAX_RESTARTSinstead of retiring it quietly. The cap is a terminal state: the task is confirmed pathological, the doctor stops intervening, and the task then holds its slot until walltime producing nothing. It previously said so with one lowercaserestarts exhausted; leaving itin the middle of a verbose sweep, which is the same shape of silent ending as a TIME_LIMIT kill mailed to nobody. The line is now a!! EXHAUSTEDtoken carrying the task, classification, node and restart count; the sweep summary gains anexhausted=counter; and settingMAIL_TOsends one message per newly exhausted task. Alerts are emitted once per task rather than once per sweep, deduplicated through the campaign-scoped ledger already inSTATE_FILE, since the doctor re-probes every ten minutes and an exhausted task stays exhausted.Observed on a
swfcalibcampaign that ran the pre-2.8.3 classifier: task41767827_48wedged in PSOCK worker startup three times, getting 3, 3 and 1 of its 8 workers through package loading before each requeue, and took about 4.5 hours to deliver one 95-minute batch. Nothing surfaced that except reading the sweep log by hand.
EpiModelHPC 2.9.1
BUG FIXES
-
degen_watch.shno longer judges the job it is running inside. The doctor’s own job name necessarily matchesPATTERN, since that is how it is scoped to a campaign, so it sits inside its own task set; and once the process scan widened pastpgrep -x Rin 2.9.0, its ownsleepandsshprocesses became visible too. That reads as a task at 0% CPU with nothing in D-state, which is exactly thehungsignature, so the doctor confirmed itself hung once per sweep and would have requeued itself. Caught on the first campaign to run 2.9.0: ten consecutive confirmations of the doctor’s own job id, spared only because its restart counter already sat atMAX_RESTARTS. Any task whosejobidequalsSLURM_JOB_IDis now skipped before classification.
EpiModelHPC 2.9.0
NEW FEATURES
- The doctor no longer assumes a task’s work happens in R. Task membership is decided by
SLURM_JOB_IDfrom/proc/<pid>/environrather than by process name, so a compiled child such as anrstan/cmdstanrmodel_<hash>binary is attributed to the task that launched it.pgrep -x Rsaw only the idle R wrapper and reportednproc=1 med_cpu=0for a task consuming four cores. The batch script wrapper is excluded by name sonprocstill reflects the workload and a starting task keeps itsMIN_PROCgrace period. -
probe_node_cpu.shreportstop_cpu=, the busiest single process in a task, anddegen_watch.shdeclines to judge any task whosetop_cpuexceedsMULTICORE_CPU(default 150), reporting it asthreaded. The whole detector rests on one simulation per core at about 100%, which is what makes a median a starvation signal; a threaded binary at several hundred percent is outside that model, and a healthy 4-thread chain cannot be told from a contended 8-thread one from outside the process. Under-flagging is the correct failure, since the alternative is requeuing a healthy Bayesian fit. -
probe_node_cpu.shreportsnodeinfo=1 cores= load1=per node, printed for any node that produced a suspect. The probe only sees the invoking user’s processes, so another account’s job can starve a task with nothing in the per-task view to show for it. Load against core count separates a suspect on a node at 30 of 32, where contention is real and excluding the node may be justified, from the same suspect on a node at 4 of 32, where it is not.
BUG FIXES
-
degen_watch.shwarns when the probe reached its nodes but attributed no process to any running task. The existing guard only covered a probe that returned nothing at all, which the new per-node line makes impossible. -
probe_node_cpu.shno longer leaks “Permission denied” to stderr when scanning processes it cannot read.2>/dev/nullafter an input redirect is set up only after the failing open, so it never caught the shell’s own message; it now precedes the redirect. Only reachable since the scan widened beyond the user’s own R processes.
EpiModelHPC 2.8.3
BUG FIXES
-
degen_watch.shno longer reports an idle task as CPU-starved, and no longer excludes the node it was running on.dstate == 0defaulted tocpustarvat any CPU level, but losing a share of the cores to a competitor reads near 50% and a task with no competitor and no I/O wait reads near zero. On a 64-taskswfcalibarray, 100 of 127 confirmed events read 0-3% against 23 at the genuine 51% signature; the nodes blamed were idle and process-free minutes later, and the requeued tasks had never begun work, all stopping inside package loading.dstate == 0with CPU at or belowHUNG_CPUis now classifiedhung: still requeued promptly, but the node is not implicated, matching what the classifier already did for the same condition when one process happened to be in D-state. - The confirmation line reports the classification (
CONFIRMED HUNG,CONFIRMED CPU-STARVED,CONFIRMED IO-STALLED) instead of a blanketCONFIRMED STARVED. The label is what sends an operator looking for a competitor that may never have been there. TheCONFIRMEDtoken is unchanged, so existing log greps still match.
EpiModelHPC 2.8.2
OTHER
-
make_calibrated_scenario()callsswfcalib::load_calib_object()andswfcalib::get_default_proposal()with::now that both are exported (EpiModel/swfcalib#34), clearing the:::NOTE fromR CMD check. Requires a swfcalib built frommainat or after that merge; the version there is unchanged at 0.0.0.9000, soDESCRIPTIONcannot state the requirement.
EpiModelHPC 2.8.1
BUG FIXES
-
deploy_doctor.shlocates its sibling scripts correctly when a project wrapper submits itself and then runs the installed doctor by full path. 2.6.3 madescontrol show joboutrank$BASH_SOURCE, which is right forsbatch deploy_doctor.shbut wrong here:scontrolreports the wrapper, not the doctor. The startup guard caught it, so the job failed loudly rather than sweeping nothing, but it failed. Each candidate is now tested for the siblings instead of being ranked blind, so both launch styles work with noROOTset.
EpiModelHPC 2.8.0
BREAKING CHANGES
- Removed the pre-
slurmworkflowHPC path. It was driven by environment variables exported from a hand-written master sbatch script, and nothing in the package or in any current project still reached it. Replacements:-
netsim_hpc(): useEpiModel::netsim()directly, withncoresincontrol.net()for parallelism. -
check_cp(),initialize_cp(),save_cpdata(): use EpiModel’s.checkpoint.dirand.checkpoint.stepsincontrol.net(). -
savesim(),merge_simfiles(),process_simfiles(): usemerge_netsim_scenarios()ormerge_netsim_scenarios_tibble(). -
sbatch_master(): use theslurmworkflowstep templates,step_tmpl_netsim_scenarios()and friends. -
pull_env_vars(): handled internally byslurmworkflow. -
verbose.hpc.net(): no replacement. It printed progress fornetsim_hpc()only.
-
- Dropped dependencies, all unused once the above went:
doParallel,foreach,ergm,tergm,tidyr, andnetworkfrom Suggests.
OTHER
-
CLAUDE.mdand_pkgdown.ymlare excluded from the built package. - Removed
test-netsimpar.R, which exercisedEpiModel::netsim()rather than anything in this package. - Refreshed stale metadata: the
Descriptionfield no longer advertises PBS check-pointing, the package documentation no longer carries a hand-maintained version table, and three dead URLs are fixed.
EpiModelHPC 2.7.0
NEW FEATURES
The deploy doctor reports pending-side barriers.
slurmworkflowsubmits netsim in slices and the next slice only goes in when the last task of the current one finishes, so one queued straggler stops all progress. There is no process to probe, so the CPU detector was blind to it. Sweeps with tasks pending and none running are now counted and, pastBARRIER_WARN, reported with each pending task’s SLURM reason. Observability only; slice progression is untouched.The deploy doctor exits promptly when a campaign really is finished. It reads the same watch list
add_doctor_teardown_step()maintains, passed asWATCH_DIR. With a campaign still registered an empty queue is treated as a lull between steps and it waits the fullIDLE_EXIT; with none registered it exits after the shorterIDLE_FAST. The watch list can only make it more patient, so a stale marker cannot hold it open to its walltime.
EpiModelHPC 2.6.4
BREAKING CHANGES
-
swf_configs_rsph()no longer takesgit_versionand no longer loads git. Git is the system binary on the RSPH compute nodes, verified against the cluster.swf_configs_hyak()is unchanged.
OTHER
-
swf_configs_rsph()defaultsr_versionto 4.5.1, up from 4.2.1.
EpiModelHPC 2.6.0
NEW FEATURES
swf_cleanup_r_workers()returns bash lines trappingTERMandEXITto reap a job’s own R workers. RSPH runsproctrack/linuxprocwith no cgroup containment, so PSOCK workers reparent to init and survivescancel, staying pinned to cores SLURM has already re-allocated. One cancelled calibration array left 163 orphans on 15 nodes for about 2.5 hours.swf_configs_rsph()appends the trap tor_loaderby default; setcleanup_workers = FALSEto opt out.hpc_doctor_script()locates four bundled shell tools that detect and recover degenerate SLURM tasks by measuring CPU utilisation rather than elapsed runtime. Seeinst/hpc_doctor/README.md.add_doctor_register_step()andadd_doctor_teardown_step()maintain a watch list so a deploy doctor shared across concurrent campaigns is stopped only when the last one finishes.