ANOVA (Analysis Of Variance) is a technique used to estimate the error variance and determine the relative importance of various factors.

In HyperStudy, an ANOVA can only be performed on Least Squares Regression Approximations. ANOVA is used to identify which variables are explaining the variance in the data. This is done by examining the resulting increase in the unexplained error when variables are removed.

The following definitions are useful in describing ANOVA concepts. For a given set of input values, denoted as , the Fit predictions at the same points are denoted as . The mean of the input values is expressed . For a Least Squares Regression, p is the number of unknown coefficients in the regression.

The following three values are defined as follows:

In the ANOVA tab, the following quantities are computed in the process of performing the ANOVA.

Degrees of Freedom

Number of terms in the regression associated with the variable. All degrees of freedom not associated with a variable are retained in the Error assessment. More degrees of freedom associated with the error increases the statistical certainty of the results: the p-values. Higher order terms have more degrees of freedom; for example a second order polynomial will have two degrees of freedom for a variable: one for both the linear and quadratic terms.

Sum of Squares

For each Variable rows, the quantity shown is the increase in unexplained variance if the variable were to be removed from the regression. A variable which has a small value is less critical in explaining the data variance than a variable which has a larger value.

The Error row entry represents the variance not explained by the model, which is .

The entry in the Total row, which is , will generally not equal to the sum of the others rows.

Mean Squares

The ratio between unexplained error increase and degrees of freedom, computed as the Sum of Squares divided by the associated degrees of freedom.

Mean Squares Percent

This quantity is interpreted as the relative contribution of the variables to the Fit quality, computed as the ratio of the Mean Square to the summed total of the Mean Squares. A variable with a higher percentage is more critical to explaining the variance in the given data than a variable with a lower percentage.

F-value

The quotient of the mean squares from the variable to the mean squares from the error. This is a relative measure of the variable’s explanatory variance to overall unexplained variance.

p-value

The result of an F-test on the corresponding F-value. The p-value indicates the statistical probability that the same pattern of relative variable importance could have resulted from a random sample and that the variable actually has no effect at all (the null hypothesis). A low value, typically less than 0.05, leads to a rejection of the null-hypothesis, meaning the variable is statistically significant.

Diagnostics assist in assessing the accuracy of a Fit.

In the Diagnostic tab, quantities are displayed in three columns.

The following definitions are useful in describing diagnostic concepts. For a given set of input values, denoted as , the Fit predictions at the same points are denoted as . The mean of the input values is expressed . For a Least Squares Regression, p is the number of unknown coefficients in the regression.

The following three values are defined as follows:

R-Square

R-Square, commonly called the coefficient of determination, is a measure of how well the Fit can reproduce known data points. Graphically, this can be visualized by scatter plotting the known values versus the predicted values. If the model perfectly predicts the known values, R-Square will have its maximum possible value of 1.0, and the scatter points will lie on a perfect diagonal line, as shown in the image below.

More typically, the Fit introduces modeling error, and the scatter points will deviate from the straight diagonal line, as shown in the image below.

The value of R-Square decreases as errors increase and the scatter plot deviates more from a straight line. The main interpretation of R-Square is that it represents the proportion of variance within the data which is explained by the Fit. For example, if R-Square = 0.84, then 84% of the variance in the data is predictable by the Fit. The higher the value of R-Square, the better the quality of the Fit. In practice, a value above 0.92 is often very good and a value lower than 0.7 necessitates investigation using other metrics. If R-Square is 1.0, you should be skeptical of this result unless the data was expected to be perfectly predicted by the Fit. There are some cases in which R-Square can be negative. A negative R-Square value indicates that using the raw mean would be a better predictor than the Fit itself; the Fit is very poor quality.

In the work area, these numbers are presented with a spark line to indicate the relative value of the number (values typically vary between 0 and 1). Values are color coded based on the following:

R-Square is defined as:

R-Square Adjusted (Least Squares Only)

Due to its formulation, adding a variable to the model will always increase R-Square. R-Square Adjusted is a modification of R-Square that adjusts for the explanatory terms in the model. Unlike R-Square, R-Square Adjusted increases only if the new term improves the model more than would be expected by chance. The adjusted R-Square can be negative, and will always be less than or equal to R-Square. If R-Square and R-Square Adjusted differ dramatically, it indicates that non-significant terms may have been included in the model.

R-Sqaure Adjusted is defined as:

In the work area, these numbers are presented with a spark line to indicate the relative value of the number (values typically vary between 0 and 1). Values are color coded based on the following:

Multiple R (Least Squares Only)

Multiple R is the multiple correlation coefficient between actual and predicted values, and in most cases it is the square root of R-Square. It is an indication of the relationship between two variables.

Regression Equation (Least Squares Only)

The complete formula for the predictive model as a function of the input variables.

Relative Average Absolute Error

A comparison of the average predicted error and the spread in the data itself. A low ratio is more desirable as it indicates that variance in the Fit’s predicted values are dominated by the actual variance in the data and not by error.

Relative Average Absolute Error is defined as:

Maximum Absolute Error

The maximum difference, in absolute value, between the observed and predicted values. For the input and validation matrices, this value can also be observed in the Residuals tab.

Maximum Absolute Error is defined as:

Root Mean Square Error

A measure of weighted average error. A higher quality Fit will have a lower value.

Root Mean Square Error is defined as:

Number of Samples

The number of data points used in the diagnostic computations.

Confidence Intervals (Least Squares Only)

Confidence intervals consist of an upper and lower bound on the coefficients of the regression equation. Bounds represent the confidence that the true value of the coefficient lies within the bounds, based on the given sample.

Enter a specific confidence value in the field that displays when you click , as shown in the image below.

A higher confidence value will result in wider bounds; a 95% confidence interval is typically used. T-value is defined as:

where βj is the corresponding regression coefficient (the Values column) and SE is the standard error. The standard error is defined as:

SE =

and:

where Cjj is the diagonal coefficient of the information matrix used during the regression calculation.

P-values are computed using the standard error and t-value to perform a student’s t-test. The p-value indicates the statistical probability that the quantity in the Value column could have resulted from a random sample and that the real value of the coefficient is actually zero (the null hypothesis). A low value, typically less than 0.05, leads to a rejection of the null-hypothesis, meaning the term is statistically significant.

Residuals help to identify errors for each design.

In the Residuals tab, the error (and percentage) between the original output response and the approximation is listed for each run of the input, cross-validation, or validation matrices. 

Use the Find and Sort options in the right-click context menu to search for specific cases. Switch between the input and validation matrix residuals from the menu that displays when you click (located above the Channel selector).

View the Percent Error column with caution when the values of the output response approach zero. In this situation, the Percent Error can be very high and potentially misleading.

Use the interactive response surface tools in the Trade-Off tab to perform "what if" scenarios.

In the Inputs pane, modify the values of input variables to see their effect on the output response approximations in the Output pane.

Use the Channel selector to select the desired output responses to display. Input variable controls are located in the in the Inputs pane. Change each input variable by moving the slider in the first Value column, or by entering a value into the second Value column. Set input variables to their initial, minimum, or maximum values by moving the slider in the upper right-hand corner of the Inputs frame.

For the given values of the input variables, the output responses’ predictions are calculated by the Fit, and displayed in the Output pane. Table shading is used to indicate the output response’s value between the minimum and maximum values contained in the input design matrix. When shading extends into either the Sample Min or Sample Max column, this indicates that the predicted value is beyond the bounds contained in the input matrix. If the shading extends significantly into these regions, it is suggested that you asses the validity of this value based on experience and knowledge of the modeled problem.

The Quality column is used as a normalized measure to assess the trust in the Fit at a specific point in the design space. The values in the Quality column run between 0 and 1. A value of 1 indicates that the predicted point lies in a portion of the variable space bounded by the known points, and seems trustworthy. As this number decreases, the prediction at this point becomes less reliable, partially due to the values increasing based on an extrapolation of the data.

In the Trade-Off tab you can also plot the effect of an input variable on output response approximations. Select an input variable to plot by enabling its corresponding X Axis and/or Y Axis checkbox in the Inputs pane. Use the Channel selector to select output responses to plot.

The values for the input variables which are not plotted can be modified in the Inputs pane. Move the sliders in the Value column to modify the other input variables, while studying the output response throughout the design space.

2D Trade-Off

Enabling the X Axis checkbox creates a 2D trade-off in the Outputs pane.

In a 2D trade-off the metrics shown in the Quality column can be plotted alongside the output response curve by selecting Fit Quality from the menu that displays when you click (located in the Outputs pane).

3D Trade-Off

Enabling both the X Axis and Y Axis checkboxes creates a 3D trade-off in the Outputs pane.