Running mcasopt on Spartan

In this example, we will demonstrate how to fit the n+12C model.

Initial setup

Connect to Spartan via SSH and navigate to a directory that contains the MCAS binary and the AMDC-masses data file. In this example, we will call this directory mcas-experiments:

$ cd /data/projects/punim0038/rgm/mcas-experiments
$ ls
AMDC-masses.dat  mcas5.4

Install the development version of mcasopt into a Python virtual environment:

$ module load foss
$ module load SciPy-bundle
$ git clone https://bitbucket.org/robmoss/mcasopt.git
$ python3 -m venv mcas-venv
$ . ./mcas-venv/bin/activate
$ pip install --upgrade pip
$ pip install -e ./mcasopt/

Warning

If the Spartan modules fail to load, you can make these modules available by running:

$ source /usr/local/module/spartan_new.sh

You will need to ensure that this command runs automatically when you log in, so that mcasopt can load the modules required to run MCAS simulations.

Create an experiment directory

We will create a new sub-directory for this experiment, called n+12C:

$ mkdir n+12C
$ cd n+12C

Create the required experiment files

Initial model

Define the initial model input in a file called n+12C.inp:

Listing 4 The contents of n+12C.inp.
&fileinfo
 runname = 'n+12C'
 options = 'sturm,rotation,res,fast'
&end

&system
 projectile_mass   = 1
 projectile_charge   = 0d0
 target_mass   = 12
 target_charge = 6
&end

&integration
 emin =  -6.001
 emax  = 4d0
 estep = 0.2
 
 Rcoul = 2.8
 acoul = 0.6
 B = 1
 rho = 7.0d0
 kpmax = 80
&end

&model
 name = 'C12+n'
 a = 0.65d0
 bigR = 3.090728
 v0 = -49.0, 0.0 , -47.1
 vll =  4.5548, 0.0 ,  0.6098
 vls =  7.3836, 0.0 ,  9.176
 vss = -4.77, 0.0 , -0.052
 ibmax = 10
 nshells = 2
 nsturm = 30
 V_radius = 10d0

 ntargex = 3

 targex(1)%energy = 0.0
 targex(1)%spin = 0.0d0
 targex(1)%parity = 1
 targex(1)%alla = 0.0 , 1.0d+06, 1.0d+06

 targex(2)%energy = 4.4389
 targex(2)%spin = 2.0d0
 targex(2)%parity = 1
 targex(2)%alla = 0.0 , 1.0d+06, 1.0d+06

 targex(3)%energy = 7.654
 targex(3)%spin = 0.0d0
 targex(3)%parity = 1
 targex(3)%alla = 0.0 , 1.0d+06, 1.0d+06

 nbeta = 1

 beta(1)%nucl = -0.52
 beta(1)%coul  = -0.52
 beta(1)%lval = 2

&end

Model input function

Define the mapping from parameter vectors to model inputs in a Python file called n+12C.py:

Listing 5 The contents of n+12C.py.
# Define a function that returns the MCAS input for parameter vector ``x``.
def input_fun(mcas_input, x):
    # Adjust the MCAS input according to the values in ``x``.
    mcas_input.model.v0 = [x[0], 0.0, x[1]]
    return mcas_input

Note

The model inputs must be accessed using valid Python operations. Note that Python uses zero-based indexing, so the first item in a list is at index 0, the second item is at index 1, etc.

Here are some examples that demonstrate how to modify different model inputs:

Listing 6 Examples of modifying MCAS model inputs.
# Modify values in the first `targex` block.
mcas_input.model.targex[0].spin = 1.0
mcas_input.model.targex[0].parity = 1
mcas_input.model.targex[0].alla = [0.0, 100000.0, 1000000.0]

# Modify values in the first `beta` block.
mcas_input.model.beta[0].lval = 2

Experiment settings

Define all of the mcasopt settings and the compound states for this experiment in a TOML file called n+12C.toml:

Listing 7 The contents of n+12C.toml.
[experiment]
name = "n+12C"
input_file = "n+12C.inp"
mcas_path = "../mcas5.4"
amdc_masses_path = "../AMDC-masses.dat"
input_function = "n+12C.input_fun"
x_0 = [-49, -47.1]
runner = "slurm"
backup_results = true

[solver]
method = "Nelder-Mead"
sleep_duration = 10
cache_file = "n+12C.cache"
log_level = "INFO"
epsilon = 1e-3

[slurm]
shebang = "#!/bin/bash"
header.partition = "cascade"
header.time = "30-00:00:00"
header.ntasks = "1"
header.job-name = "n+12C-job"
header.output = "n+12C-job.log"
modules = ["imkl"]

# First resonance
[[compound_state]]
weight = 1.0
energy = 0.1295
spin = ["5/2-"]
weight_half_width = 1.0
half_width = 3.2125e-10

# Second resonance
[[compound_state]]
weight = 1.0
energy = 2.0006
spin = ["5/2+"]
weight_half_width = 1.0
half_width = 1.0484e-2

# Third resonance
[[compound_state]]
weight = 1.0
energy = 2.6612
spin = ["7/2+"]
weight_half_width = 1.0
half_width = 9.4961e-7

Warning

Ensure that the mcas_path and amdc_masses_path are correct.

Note

The Slurm job settings are defined in the [slurm] settings table (highlighted lines).

Experiment job script

We need to load required HPC modules and activate the mcasopt virtual environment before we can start the solver. We will do this using a short shell script, which we will call n+12C.sh:

Listing 8 The contents of n+12C.sh.
#!/bin/bash
#SBATCH --partition=cascade
#SBATCH --time=30-00:00:00
#SBATCH --ntasks=1
#SBATCH --job-name=n+12C
#SBATCH --output=n+12C.log
module load foss
module load SciPy-bundle
source ../mcas-venv/bin/activate
python3 -m mcasopt n+12C.toml

Start the solver

Once all of the required experiment files have been created, we can start the solver:

$ cd /data/projects/punim0038/rgm/mcas-experiments
$ cd n+12C
$ ls
n+12C.inp  n+12C.py  n+12C.sh  n+12C.toml
$ chmod u+x n+12C.sh
$ sbatch n+12C.sh

Monitor the solver

You can check on the solver’s progress by inspecting the log file n+12C.log. For example, here is what the end of the log looks like shortly after starting the solver:

$ cd /data/projects/punim0038/rgm/mcas-experiments
$ cd n+12C
$ tail n+12C.log
15:08:04:INFO:mcasopt.cache:Attempting to read cache file: /data/gpfs/projects/punim0038/rgm/mcas-experiments/n+12C/n+12C.cache
15:08:04:INFO:mcasopt.cache:Cache file does not exist
15:08:04:INFO:mcasopt.cache:Starting with an empty cache
15:08:04:INFO:mcasopt.solver:Evaluating at x = [-49.  -47.1]
15:09:04:INFO:mcasopt.solver:f(x) = 9.9999999999584e-12
15:09:04:INFO:mcasopt.solver:Evaluating at x = [-49.   -47.05]
15:10:05:INFO:mcasopt.solver:f(x) = 0.04051003024000999
15:10:05:INFO:mcasopt.solver:Evaluating at x = [-49.        -47.0992686]

Warning

The solver script will run for a maximum of 30 days, at which point it will be terminated. If this happens, you can resume the solver by starting it again.

You can list your scheduled Slurm jobs with:

$ squeue --me
   JOBID PARTITION     NAME     USER ST       TIME  NODES NODELIST(REASON)
43450855  cascade n+12C-ma      rgm  R      16:10      1 spartan-bm011
43452921  cascade mcas-exp      rgm  R       0:01      1 spartan-bm008

There should be \(2\) active jobs:

  • The main solver; and

  • Evaluating the goodness of fit \(g(x^k)\).

Successful completion

If the solver completes successfully, the log file n+12C.log will contain the final parameter vector:

$ cd /data/projects/punim0038/rgm/mcas-experiments
$ cd n+12C
$ tail n+12C.log
15:34:13:INFO:mcasopt.solver:Evaluating at x = [-49.         -47.09987578]
15:35:14:INFO:mcasopt.solver:f(x) = 8.000999999998482e-11
15:35:14:INFO:mcasopt.solver:df/dx = [9.99997611e-12 8.00000000e-08]
15:35:14:INFO:mcasopt.solver:Evaluating at x = [-49.         -47.09995056]
15:36:14:INFO:mcasopt.solver:f(x) = 4.000999999993946e-11
15:36:14:INFO:mcasopt.solver:df/dx = [1.49999913e-11 3.50000000e-08]
15:36:14:INFO:mcasopt.solver:Evaluating at x = [-49.         -47.09997822]
15:37:14:INFO:mcasopt.solver:f(x) = 2.0009999999916782e-11
15:37:14:INFO:mcasopt.solver:df/dx = [1.49999913e-11 2.00000000e-08]
Solution: [-49.  -47.1]

In this case, the solution [-49, -47.1] is identical to the initial parameter vector x0 (this was intentional).

Running from multiple starting points

We can run the same experiment from a different starting point (in this case, [-48, -46]) by writing a second job script, which we will call n+12C-A:

Listing 9 The contents of n+12C-A.sh.
#!/bin/bash
#SBATCH --partition=cascade
#SBATCH --time=30-00:00:00
#SBATCH --ntasks=1
#SBATCH --job-name=n+12C-A
#SBATCH --output=n+12C-A.log
module load foss
module load SciPy-bundle
source ../mcas-venv/bin/activate
python3 -m mcasopt --name=n+12CA --x0=-48,-46 n+12C.toml

Note

The starting point must be provided as a comma-separated list of numbers.

If the solver completes successfully, the log file n+12C-A.log will contain the final parameter vector:

$ cd /data/projects/punim0038/rgm/mcas-experiments
$ cd n+12C
$ tail n+12C-A.log
23:05:07:INFO:mcasopt.solver:f(x) = 0.0010010002855401945
23:05:07:INFO:mcasopt.solver:Evaluating at x = [-49.00085899 -47.09820795]
23:05:47:INFO:mcasopt.solver:f(x) = 0.0018040010836103018
23:05:47:INFO:mcasopt.solver:Evaluating at x = [-49.00081102 -47.10020242]
23:06:27:INFO:mcasopt.solver:f(x) = 0.0006010001234099904
Optimization terminated successfully.
         Current function value: 0.000501
         Iterations: 32
         Function evaluations: 61
Solution: [-49.00033228 -47.09953082]

Running your own experiments

In adapting this example to your own experiments, you will probably want to change the following details:

Note

You can use the mcasopt.template module to create the Model input function, the Experiment settings, and the Experiment job script:

$ source ../mcas-venv/bin/activate
$ python3 -m mcasopt.template

This will ask you to enter several details, create each of the above files, and then tell you which files need further editing.

Path to mcas binary

This should be something like ../mcas5.4.

Path to AMDC-masses.dat

This should be something like ../AMDC-masses.dat.

Path to mcasopt virtual environment

This should be something like ../mcas-venv.

Experiment name

This should be something like n+12C.

Create a new directory [y/n]

Enter n to create the files in the working directory.

Enter y to create the files in a sub-directory with the same name as the experiment name that you entered above (e.g., n+12C).