Running mcasopt on Spartan
In this example, we will demonstrate how to fit the n+12C
model.
Initial setup
Connect to Spartan via SSH and navigate to a directory that contains the MCAS binary and the AMDC-masses data file.
In this example, we will call this directory mcas-experiments
:
$ cd /data/projects/punim0038/rgm/mcas-experiments
$ ls
AMDC-masses.dat mcas5.4
Install the development version of mcasopt into a Python virtual environment:
$ module load foss
$ module load SciPy-bundle
$ git clone https://bitbucket.org/robmoss/mcasopt.git
$ python3 -m venv mcas-venv
$ . ./mcas-venv/bin/activate
$ pip install --upgrade pip
$ pip install -e ./mcasopt/
Warning
If the Spartan modules fail to load, you can make these modules available by running:
$ source /usr/local/module/spartan_new.sh
You will need to ensure that this command runs automatically when you log in, so that mcasopt can load the modules required to run MCAS simulations.
Create an experiment directory
We will create a new sub-directory for this experiment, called n+12C
:
$ mkdir n+12C
$ cd n+12C
Create the required experiment files
Initial model
Define the initial model input in a file called n+12C.inp
:
&fileinfo
runname = 'n+12C'
options = 'sturm,rotation,res,fast'
&end
&system
projectile_mass = 1
projectile_charge = 0d0
target_mass = 12
target_charge = 6
&end
&integration
emin = -6.001
emax = 4d0
estep = 0.2
Rcoul = 2.8
acoul = 0.6
B = 1
rho = 7.0d0
kpmax = 80
&end
&model
name = 'C12+n'
a = 0.65d0
bigR = 3.090728
v0 = -49.0, 0.0 , -47.1
vll = 4.5548, 0.0 , 0.6098
vls = 7.3836, 0.0 , 9.176
vss = -4.77, 0.0 , -0.052
ibmax = 10
nshells = 2
nsturm = 30
V_radius = 10d0
ntargex = 3
targex(1)%energy = 0.0
targex(1)%spin = 0.0d0
targex(1)%parity = 1
targex(1)%alla = 0.0 , 1.0d+06, 1.0d+06
targex(2)%energy = 4.4389
targex(2)%spin = 2.0d0
targex(2)%parity = 1
targex(2)%alla = 0.0 , 1.0d+06, 1.0d+06
targex(3)%energy = 7.654
targex(3)%spin = 0.0d0
targex(3)%parity = 1
targex(3)%alla = 0.0 , 1.0d+06, 1.0d+06
nbeta = 1
beta(1)%nucl = -0.52
beta(1)%coul = -0.52
beta(1)%lval = 2
&end
Model input function
Define the mapping from parameter vectors to model inputs in a Python file called n+12C.py
:
# Define a function that returns the MCAS input for parameter vector ``x``.
def input_fun(mcas_input, x):
# Adjust the MCAS input according to the values in ``x``.
mcas_input.model.v0 = [x[0], 0.0, x[1]]
return mcas_input
Note
The model inputs must be accessed using valid Python operations.
Note that Python uses zero-based indexing, so the first item in a list is at index 0
, the second item is at index 1
, etc.
Here are some examples that demonstrate how to modify different model inputs:
# Modify values in the first `targex` block.
mcas_input.model.targex[0].spin = 1.0
mcas_input.model.targex[0].parity = 1
mcas_input.model.targex[0].alla = [0.0, 100000.0, 1000000.0]
# Modify values in the first `beta` block.
mcas_input.model.beta[0].lval = 2
Experiment settings
Define all of the mcasopt settings and the compound states for this experiment in a TOML file called n+12C.toml
:
[experiment]
name = "n+12C"
input_file = "n+12C.inp"
mcas_path = "../mcas5.4"
amdc_masses_path = "../AMDC-masses.dat"
input_function = "n+12C.input_fun"
x_0 = [-49, -47.1]
runner = "slurm"
backup_results = true
[solver]
method = "Nelder-Mead"
sleep_duration = 10
cache_file = "n+12C.cache"
log_level = "INFO"
epsilon = 1e-3
[slurm]
shebang = "#!/bin/bash"
header.partition = "cascade"
header.time = "30-00:00:00"
header.ntasks = "1"
header.job-name = "n+12C-job"
header.output = "n+12C-job.log"
modules = ["imkl"]
# First resonance
[[compound_state]]
weight = 1.0
energy = 0.1295
spin = ["5/2-"]
weight_half_width = 1.0
half_width = 3.2125e-10
# Second resonance
[[compound_state]]
weight = 1.0
energy = 2.0006
spin = ["5/2+"]
weight_half_width = 1.0
half_width = 1.0484e-2
# Third resonance
[[compound_state]]
weight = 1.0
energy = 2.6612
spin = ["7/2+"]
weight_half_width = 1.0
half_width = 9.4961e-7
Warning
Ensure that the mcas_path
and amdc_masses_path
are correct.
Note
The Slurm job settings are defined in the [slurm]
settings table (highlighted lines).
Experiment job script
We need to load required HPC modules and activate the mcasopt virtual environment before we can start the solver.
We will do this using a short shell script, which we will call n+12C.sh
:
#!/bin/bash
#SBATCH --partition=cascade
#SBATCH --time=30-00:00:00
#SBATCH --ntasks=1
#SBATCH --job-name=n+12C
#SBATCH --output=n+12C.log
module load foss
module load SciPy-bundle
source ../mcas-venv/bin/activate
python3 -m mcasopt n+12C.toml
Start the solver
Once all of the required experiment files have been created, we can start the solver:
$ cd /data/projects/punim0038/rgm/mcas-experiments
$ cd n+12C
$ ls
n+12C.inp n+12C.py n+12C.sh n+12C.toml
$ chmod u+x n+12C.sh
$ sbatch n+12C.sh
Monitor the solver
You can check on the solver’s progress by inspecting the log file n+12C.log
.
For example, here is what the end of the log looks like shortly after starting the solver:
$ cd /data/projects/punim0038/rgm/mcas-experiments
$ cd n+12C
$ tail n+12C.log
15:08:04:INFO:mcasopt.cache:Attempting to read cache file: /data/gpfs/projects/punim0038/rgm/mcas-experiments/n+12C/n+12C.cache
15:08:04:INFO:mcasopt.cache:Cache file does not exist
15:08:04:INFO:mcasopt.cache:Starting with an empty cache
15:08:04:INFO:mcasopt.solver:Evaluating at x = [-49. -47.1]
15:09:04:INFO:mcasopt.solver:f(x) = 9.9999999999584e-12
15:09:04:INFO:mcasopt.solver:Evaluating at x = [-49. -47.05]
15:10:05:INFO:mcasopt.solver:f(x) = 0.04051003024000999
15:10:05:INFO:mcasopt.solver:Evaluating at x = [-49. -47.0992686]
Warning
The solver script will run for a maximum of 30 days, at which point it will be terminated. If this happens, you can resume the solver by starting it again.
You can list your scheduled Slurm jobs with:
$ squeue --me
JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON)
43450855 cascade n+12C-ma rgm R 16:10 1 spartan-bm011
43452921 cascade mcas-exp rgm R 0:01 1 spartan-bm008
There should be \(2\) active jobs:
The main solver; and
Evaluating the goodness of fit \(g(x^k)\).
Successful completion
If the solver completes successfully, the log file n+12C.log
will contain the final parameter vector:
$ cd /data/projects/punim0038/rgm/mcas-experiments
$ cd n+12C
$ tail n+12C.log
15:34:13:INFO:mcasopt.solver:Evaluating at x = [-49. -47.09987578]
15:35:14:INFO:mcasopt.solver:f(x) = 8.000999999998482e-11
15:35:14:INFO:mcasopt.solver:df/dx = [9.99997611e-12 8.00000000e-08]
15:35:14:INFO:mcasopt.solver:Evaluating at x = [-49. -47.09995056]
15:36:14:INFO:mcasopt.solver:f(x) = 4.000999999993946e-11
15:36:14:INFO:mcasopt.solver:df/dx = [1.49999913e-11 3.50000000e-08]
15:36:14:INFO:mcasopt.solver:Evaluating at x = [-49. -47.09997822]
15:37:14:INFO:mcasopt.solver:f(x) = 2.0009999999916782e-11
15:37:14:INFO:mcasopt.solver:df/dx = [1.49999913e-11 2.00000000e-08]
Solution: [-49. -47.1]
In this case, the solution [-49, -47.1]
is identical to the initial parameter vector x0
(this was intentional).
Running from multiple starting points
We can run the same experiment from a different starting point (in this case, [-48, -46]
) by writing a second job script, which we will call n+12C-A
:
#!/bin/bash
#SBATCH --partition=cascade
#SBATCH --time=30-00:00:00
#SBATCH --ntasks=1
#SBATCH --job-name=n+12C-A
#SBATCH --output=n+12C-A.log
module load foss
module load SciPy-bundle
source ../mcas-venv/bin/activate
python3 -m mcasopt --name=n+12CA --x0=-48,-46 n+12C.toml
Note
The starting point must be provided as a comma-separated list of numbers.
If the solver completes successfully, the log file n+12C-A.log
will contain the final parameter vector:
$ cd /data/projects/punim0038/rgm/mcas-experiments
$ cd n+12C
$ tail n+12C-A.log
23:05:07:INFO:mcasopt.solver:f(x) = 0.0010010002855401945
23:05:07:INFO:mcasopt.solver:Evaluating at x = [-49.00085899 -47.09820795]
23:05:47:INFO:mcasopt.solver:f(x) = 0.0018040010836103018
23:05:47:INFO:mcasopt.solver:Evaluating at x = [-49.00081102 -47.10020242]
23:06:27:INFO:mcasopt.solver:f(x) = 0.0006010001234099904
Optimization terminated successfully.
Current function value: 0.000501
Iterations: 32
Function evaluations: 61
Solution: [-49.00033228 -47.09953082]
Running your own experiments
In adapting this example to your own experiments, you will probably want to change the following details:
The model parameters in the Initial model file.
The mapping from parameter vectors to model inputs in the Model input function.
The solution criteria in the Experiment settings file.
The initial parameter vector
x_0
in the Experiment settings.
Note
You can use the mcasopt.template
module to create the Model input function, the Experiment settings, and the Experiment job script:
$ source ../mcas-venv/bin/activate
$ python3 -m mcasopt.template
This will ask you to enter several details, create each of the above files, and then tell you which files need further editing.
- Path to mcas binary
This should be something like
../mcas5.4
.- Path to AMDC-masses.dat
This should be something like
../AMDC-masses.dat
.- Path to mcasopt virtual environment
This should be something like
../mcas-venv
.- Experiment name
This should be something like
n+12C
.- Create a new directory [y/n]
Enter
n
to create the files in the working directory.Enter
y
to create the files in a sub-directory with the same name as the experiment name that you entered above (e.g.,n+12C
).