We can now examine the results of the MSE, and determine which of the 5 MPs in our example had the best performance.

To interpret the MSE results it is important that a clear set of performance metrics have been defined. Fisheries managers often have broadly defined policy goals. These conceptual objectives must be translated to quantitative operational objectives so that the MSE results can be used to evaluate performance against the specified management objectives.

For example, suppose that the fishery managers had stated broad goals to maximize yield from the fishery while minimizing the risk of the stock collapsing to unacceptably low levels. In order to use MSE to determine which MPs are most likely to meet these objectives it is necessary to be more specific:

- What are
*unacceptable low stock levels*? Some fraction of unfished biomass? The lowest observed historical biomass? - What is an
*acceptable level of risk*? What chance are we willing to tolerate that the stock will fall below that limit? - How much yield are we willing to give up in order to increase the probability of the stock staying above unacceptably low limit?

It is important to recognize that performance metrics can vary considerably between different fisheries and management structures, but are a crucial component of the MSE and must be carefully defined before the analysis is carried out. The Calculating Performance section section discusses this topic in more detail.

`openMSE`

includes a number of commonly used performance metrics and a series of functions to summarize MP performance. The MSE results can be examined either graphically in a plot or summarized in a table. Advanced users can also develop their own plotting and summary functions (see the Custom Performance Metrics section for more details).

Here we briefly demonstrate some of the plotting and summary functions in `openMSE`

. The Calculating Performance section section describe the process of evaluating MSE results in more detail.

#### Summary Table

The `summary`

function can be used to generate a table of MP performance with respect to a set of performance metrics:

`summary(MSE)`

`## Calculating Performance Metrics`

```
## Performance.Metrics
## 1 Probability of not overfishing (F<FMSY)
## 2 Spawning Biomass relative to SBMSY
## 3 Average Annual Variability in Yield (Years 1-50)
## 4 Average Yield relative to Reference Yield (Years 41-50)
##
## 1 Prob. F < FMSY (Years 1 - 50)
## 2 Prob. SB > 0.5 SBMSY (Years 1 - 50)
## 3 Prob. AAVY < 20% (Years 1-50)
## 4 Prob. Yield > 0.5 Ref. Yield (Years 41-50)
##
##
## Performance Statistics:
## MP PNOF P50 AAVY LTY
## 1 AvC 0.74 0.80 1.00 0.69
## 2 BK 0.54 0.64 0.67 0.56
## 3 CC1 0.41 0.53 0.96 0.48
## 4 DCAC 0.69 0.81 1.00 0.68
## 5 Islope1 0.50 0.58 0.98 0.47
```

By default the `summary`

function includes four performance metrics, and displays the probability that:

- fishing mortality \(\left(F\right)\) is below \(F_\text{MSY}\), i.e
*Not Overfishing*(`PNOF`

); - spawning biomass \(\left(\text{SB}\right)\) is above half of biomass at maximum sustainable yield \(\left(\text{SB}_{\text{MSY}}\right)\) (
`P50`

); - average inter-annual variability in yield is less than 20% (
`AAVY`

); - long-term yield (last 10 years of projection period) is above half of the maximum yield obtainable at a constant fishing rate (
`LTY`

) (see reference yield for more details).

In this example we can see that probability of \(\text{SB} > 0.5\text{SB}_\text{MSY}\) for AvC is 0.8.

The performance metrics have been defined in such a way that a higher number is always better (e.g, probability of *Not Overfishing* rather than *Overfishing* where a lower probability would be more desirable).

These performance metrics are functions of class `PM`

that operate on objects of class `MSE`

, for example:

`P50(MSE) # probability SB > 0.5SBMSY`

```
## Spawning Biomass relative to SBMSY
## Prob. SB > 0.5 SBMSY (Years 1 - 50)
## AvC BK CC1 DCAC Islope1
## 1 1 1 0.92 1 1
## 2 1 0.98 1 1 1
## 3 0.98 0.2 0.06 0.96 1
## 4 1 1 1 1 1
## 5 0 0 0 0.46 0
## 6 1 0.06 1 1 1
## 7 0.16 1 0.12 0.26 0.08
## 8 1 0.9 1 1 1
## 9 1 0.04 0.18 0.8 0.14
## 10 0.98 0.06 0.08 0.94 1
## 11 . . . . .
## 12 . . . . .
## 13 . . . . .
## 48 1 0.04 0.04 0.58 0.02
##
## Mean
## [1] 0.80 0.64 0.53 0.81 0.58
```

Help documentation for the performance metric functions can be found in the usual way, for example:

`?PNOF`

The performance metrics in the `summary`

function are completely customizable. See the Custom Performance Metrics section for more details.

#### Plotting MSE Results

`openMSE`

includes several functions for plotting the MSE results. Here we demonstrate a few of the plotting functions for the MSE results. See the Calculating Performance section for more plotting and summary functions, and information on how to create your own.

##### Trade-Off Plots

The `Tplot`

function creates four plots that show the trade-off between the probability that the long-term expected yield is greater than half of the highest obtainable yield at a fixed *F* (reference yield) against the probability of:

- Not overfishing in all projection years (\(F/F_\text{MSY} < 1\))
- Spawning biomass (\(\text{SB}\)) above \(\text{SB}_\text{MSY}\) in all projection years (\(\text{SB} > \text{SB}_\text{MSY}\))
- Spawning biomass above \(0.5 \text{SB}_\text{MSY}\) (\(\text{SB} > 0.5 \text{SB}_\text{MSY}\))
- Spawning biomass above \(0.1 \text{SB}_\text{MSY}\) (\(\text{SB} > 0.1 \text{SB}_\text{MSY}\))

The `Tplot`

function includes minimum acceptable risk thresholds indicated by the horizontal and vertical gray shading. These thresholds can be adjusted be the `Lims`

argument to the `Tplot`

function. See `?Tplot`

for more information on adjusting the risk thresholds.

MPs that fail to meet one or both of the risk thresholds for each axis are shown in *italics* text. The `Tplot`

function returns a data frame showing the performance of each MP with respect to the 5 performance metrics, and whether the MP is *Satisificed*, i.e., if it meets the minimum performance criteria for **all** performance metrics.

`Tplot(MSE)`

```
## MP PNOF LTY P100 P50 P10 Satisificed
## 1 AvC 0.74 0.69 0.72 0.80 0.85 TRUE
## 2 BK 0.54 0.56 0.48 0.64 0.86 FALSE
## 3 CC1 0.41 0.48 0.43 0.53 0.67 FALSE
## 4 DCAC 0.69 0.68 0.67 0.81 0.88 TRUE
## 5 Islope1 0.50 0.47 0.49 0.58 0.69 FALSE
```

The `Tplot2`

function shows the trade-off between long-term and short-term yield, and the trade-off between biomass being above \(0.1B_{MSY}\) and the expected variability in the yield:

`Tplot2(MSE)`

```
## MP STY LTY P10 AAVY Satisificed
## 1 AvC 0.89 0.69 0.85 1.00 TRUE
## 2 BK 0.74 0.56 0.86 0.67 FALSE
## 3 CC1 0.93 0.48 0.67 0.96 FALSE
## 4 DCAC 0.86 0.68 0.88 1.00 TRUE
## 5 Islope1 0.92 0.47 0.69 0.98 FALSE
```

The `Tplot`

and `Tplot2`

functions are part of a family of plotting functions that are fully customizable, and designed to work with all Performance Metrics objects. See `?Tplot`

and the Calculating Performance section section for more information.

##### Wormplot

The `wormplot`

function plots the likelihood of meeting biomass targets in future years:

`wormplot(MSE)`

The arguments to the `wormplot`

function allow you to choose the reference level for the biomass relative to \(B_{MSY}\), as well as the upper and lower bounds of the colored bands.

##### Projection Plots

The `Pplot`

function plots the trajectories of biomass, fishing mortality, and relative yield for the Management Procedures.

By default, the `Pplot`

function shows the individual trajectories of \(B/B_{MSY}\) and \(F/F_{MSY}\) for each simulation:

`Pplot(MSE)`

The `Pplot2`

function has several additional arguments. The `YVar`

argument can be used to specify additional variables of interest. For example, here we have included the projections of yield relative to the long-term optimum yield:

`Pplot2(MSE, YVar=c("B_BMSY", "F_FMSY", "Yield"))`

The `traj`

argument can be used to summarize the projections into quantiles. Here we show the 20th and 80th percentiles of the distributions (the median (50th percentile) is included by default):

`Pplot2(MSE, traj="quant", quants=c(0.2, 0.8))`

Details on additional controls for the `Pplot`

and `Pplot2`

functions can be found in the help documentation associated with this function.

#### Kobe-style Plots

Kobe-style plots are often used in stock assessment and MSE to examine the proportion of time the stock spends in different states. A Kobe-style plot of the MSE results can be produced with the `Kplot`

function:

`Kplot(MSE)`

### Compare to Current Conditions

The `Cplot`

shows a scatter plot of the median biomass and median yield over the last five years of the projection relative to the current conditions (the last year in the historical period):

`Cplot(MSE)`