.. _mcasopt-spartan: Running mcasopt on Spartan ========================== In this example, we will demonstrate how to fit the ``n+12C`` model. Initial setup ------------- Connect to Spartan via SSH and navigate to a directory that contains the MCAS binary and the AMDC-masses data file. In this example, we will call this directory ``mcas-experiments``: .. code-block:: console $ cd /data/projects/punim0038/rgm/mcas-experiments $ ls AMDC-masses.dat mcas5.4 Install the development version of mcasopt_ into a Python virtual environment: .. code-block:: console $ module load foss $ module load SciPy-bundle $ git clone https://bitbucket.org/robmoss/mcasopt.git $ python3 -m venv mcas-venv $ . ./mcas-venv/bin/activate $ pip install --upgrade pip $ pip install -e ./mcasopt/ .. warning:: If the Spartan modules fail to load, you can make these modules available by running: .. code-block:: console $ source /usr/local/module/spartan_new.sh You will need to ensure that this command **runs automatically when you log in**, so that mcasopt_ can load the modules required to run MCAS simulations. Create an experiment directory ------------------------------ We will create a new sub-directory for this experiment, called ``n+12C``: .. code-block:: console $ mkdir n+12C $ cd n+12C Create the required experiment files ------------------------------------ .. _scripting-model: Initial model ^^^^^^^^^^^^^ Define the initial model input in a file called ``n+12C.inp``: .. literalinclude:: ../../../src/mcasopt/example/n+12C.inp :caption: The contents of ``n+12C.inp``. .. _scripting-input: Model input function ^^^^^^^^^^^^^^^^^^^^ Define the mapping from parameter vectors to model inputs in a Python file called ``n+12C.py``: .. code-block:: python :caption: The contents of ``n+12C.py``. # Define a function that returns the MCAS input for parameter vector ``x``. def input_fun(mcas_input, x): # Adjust the MCAS input according to the values in ``x``. mcas_input.model.v0 = [x[0], 0.0, x[1]] return mcas_input .. note:: The model inputs must be accessed using valid Python operations. Note that Python uses **zero-based indexing**, so the first item in a list is at index ``0``, the second item is at index ``1``, etc. Here are some examples that demonstrate how to modify different model inputs: .. code-block:: python :caption: Examples of modifying MCAS model inputs. # Modify values in the first `targex` block. mcas_input.model.targex[0].spin = 1.0 mcas_input.model.targex[0].parity = 1 mcas_input.model.targex[0].alla = [0.0, 100000.0, 1000000.0] # Modify values in the first `beta` block. mcas_input.model.beta[0].lval = 2 .. _scripting-experiment: Experiment settings ^^^^^^^^^^^^^^^^^^^ Define all of the mcasopt_ settings and the **compound states** for this experiment in a TOML file called ``n+12C.toml``: .. code-block:: toml :caption: The contents of ``n+12C.toml``. :emphasize-lines: 18-25 [experiment] name = "n+12C" input_file = "n+12C.inp" mcas_path = "../mcas5.4" amdc_masses_path = "../AMDC-masses.dat" input_function = "n+12C.input_fun" x_0 = [-49, -47.1] runner = "slurm" backup_results = true [solver] method = "Nelder-Mead" sleep_duration = 10 cache_file = "n+12C.cache" log_level = "INFO" epsilon = 1e-3 [slurm] shebang = "#!/bin/bash" header.partition = "cascade" header.time = "30-00:00:00" header.ntasks = "1" header.job-name = "n+12C-job" header.output = "n+12C-job.log" modules = ["imkl"] # First resonance [[compound_state]] weight = 1.0 energy = 0.1295 spin = ["5/2-"] weight_half_width = 1.0 half_width = 3.2125e-10 # Second resonance [[compound_state]] weight = 1.0 energy = 2.0006 spin = ["5/2+"] weight_half_width = 1.0 half_width = 1.0484e-2 # Third resonance [[compound_state]] weight = 1.0 energy = 2.6612 spin = ["7/2+"] weight_half_width = 1.0 half_width = 9.4961e-7 .. warning:: Ensure that the ``mcas_path`` and ``amdc_masses_path`` are correct. .. note:: The Slurm job settings are defined in the ``[slurm]`` settings table (highlighted lines). .. _scripting-job: Experiment job script ^^^^^^^^^^^^^^^^^^^^^ We need to load required HPC modules and activate the mcasopt_ virtual environment before we can start the solver. We will do this using a short shell script, which we will call ``n+12C.sh``: .. code-block:: shell :caption: The contents of ``n+12C.sh``. #!/bin/bash #SBATCH --partition=cascade #SBATCH --time=30-00:00:00 #SBATCH --ntasks=1 #SBATCH --job-name=n+12C #SBATCH --output=n+12C.log module load foss module load SciPy-bundle source ../mcas-venv/bin/activate python3 -m mcasopt n+12C.toml .. _start_solver: Start the solver ------------------- Once all of the required experiment files have been created, we can start the solver: .. code-block:: console $ cd /data/projects/punim0038/rgm/mcas-experiments $ cd n+12C $ ls n+12C.inp n+12C.py n+12C.sh n+12C.toml $ chmod u+x n+12C.sh $ sbatch n+12C.sh Monitor the solver ------------------ You can check on the solver's progress by inspecting the log file ``n+12C.log``. For example, here is what the end of the log looks like shortly after starting the solver: .. code-block:: console $ cd /data/projects/punim0038/rgm/mcas-experiments $ cd n+12C $ tail n+12C.log 15:08:04:INFO:mcasopt.cache:Attempting to read cache file: /data/gpfs/projects/punim0038/rgm/mcas-experiments/n+12C/n+12C.cache 15:08:04:INFO:mcasopt.cache:Cache file does not exist 15:08:04:INFO:mcasopt.cache:Starting with an empty cache 15:08:04:INFO:mcasopt.solver:Evaluating at x = [-49. -47.1] 15:09:04:INFO:mcasopt.solver:f(x) = 9.9999999999584e-12 15:09:04:INFO:mcasopt.solver:Evaluating at x = [-49. -47.05] 15:10:05:INFO:mcasopt.solver:f(x) = 0.04051003024000999 15:10:05:INFO:mcasopt.solver:Evaluating at x = [-49. -47.0992686] .. warning:: The solver script will run for a maximum of **30 days**, at which point it will be terminated. If this happens, you can resume the solver by :ref:`starting it again `. You can list your scheduled Slurm jobs with: .. code-block:: console $ squeue --me JOBID PARTITION NAME USER ST TIME NODES NODELIST(REASON) 43450855 cascade n+12C-ma rgm R 16:10 1 spartan-bm011 43452921 cascade mcas-exp rgm R 0:01 1 spartan-bm008 There should be :math:`2` active jobs: * The main solver; and * Evaluating the goodness of fit :math:`g(x^k)`. Successful completion --------------------- If the solver completes successfully, the log file ``n+12C.log`` will contain the final parameter vector: .. code-block:: console $ cd /data/projects/punim0038/rgm/mcas-experiments $ cd n+12C $ tail n+12C.log 15:34:13:INFO:mcasopt.solver:Evaluating at x = [-49. -47.09987578] 15:35:14:INFO:mcasopt.solver:f(x) = 8.000999999998482e-11 15:35:14:INFO:mcasopt.solver:df/dx = [9.99997611e-12 8.00000000e-08] 15:35:14:INFO:mcasopt.solver:Evaluating at x = [-49. -47.09995056] 15:36:14:INFO:mcasopt.solver:f(x) = 4.000999999993946e-11 15:36:14:INFO:mcasopt.solver:df/dx = [1.49999913e-11 3.50000000e-08] 15:36:14:INFO:mcasopt.solver:Evaluating at x = [-49. -47.09997822] 15:37:14:INFO:mcasopt.solver:f(x) = 2.0009999999916782e-11 15:37:14:INFO:mcasopt.solver:df/dx = [1.49999913e-11 2.00000000e-08] Solution: [-49. -47.1] In this case, the solution ``[-49, -47.1]`` is identical to the initial parameter vector ``x0`` (this was intentional). Running from multiple starting points ------------------------------------- We can run the same experiment from a different starting point (in this case, ``[-48, -46]``) by writing a second job script, which we will call ``n+12C-A``: .. code-block:: shell :caption: The contents of ``n+12C-A.sh``. #!/bin/bash #SBATCH --partition=cascade #SBATCH --time=30-00:00:00 #SBATCH --ntasks=1 #SBATCH --job-name=n+12C-A #SBATCH --output=n+12C-A.log module load foss module load SciPy-bundle source ../mcas-venv/bin/activate python3 -m mcasopt --name=n+12CA --x0=-48,-46 n+12C.toml .. note:: The starting point must be provided as a comma-separated list of numbers. If the solver completes successfully, the log file ``n+12C-A.log`` will contain the final parameter vector: .. code-block:: console $ cd /data/projects/punim0038/rgm/mcas-experiments $ cd n+12C $ tail n+12C-A.log 23:05:07:INFO:mcasopt.solver:f(x) = 0.0010010002855401945 23:05:07:INFO:mcasopt.solver:Evaluating at x = [-49.00085899 -47.09820795] 23:05:47:INFO:mcasopt.solver:f(x) = 0.0018040010836103018 23:05:47:INFO:mcasopt.solver:Evaluating at x = [-49.00081102 -47.10020242] 23:06:27:INFO:mcasopt.solver:f(x) = 0.0006010001234099904 Optimization terminated successfully. Current function value: 0.000501 Iterations: 32 Function evaluations: 61 Solution: [-49.00033228 -47.09953082] Running your own experiments ---------------------------- In adapting this example to your own experiments, you will probably want to change the following details: + The :ref:`model parameters ` in the :ref:`scripting-model` file. + The mapping from parameter vectors to model inputs in the :ref:`scripting-input`. + The :ref:`solution criteria ` in the :ref:`scripting-experiment` file. + The initial parameter vector ``x_0`` in the :ref:`scripting-experiment`. .. note:: You can use the ``mcasopt.template`` module to create the :ref:`scripting-input`, the :ref:`scripting-experiment`, and the :ref:`scripting-job`: .. code-block:: console $ source ../mcas-venv/bin/activate $ python3 -m mcasopt.template This will ask you to enter several details, create each of the above files, and then tell you which files need further editing. Path to mcas binary This should be something like ``../mcas5.4``. Path to AMDC-masses.dat This should be something like ``../AMDC-masses.dat``. Path to mcasopt virtual environment This should be something like ``../mcas-venv``. Experiment name This should be something like ``n+12C``. Create a new directory [y/n] Enter ``n`` to create the files in the working directory. Enter ``y`` to create the files in a sub-directory with the same name as the experiment name that you entered above (e.g., ``n+12C``).