Project:Sandbox

1 Implementation Verification

Test ID: TS: Implementation Verification (#11) – Ankit – 2 stars

Test Name: Independent replication of model forecast and estimation


 * Objective: Independently replicate the coefficients to assess the implementation of the submodels.
 * Method:
 * Use the data detailed in the Whitepaper to derive coefficients for all submodels independently.
 * Align and contrast these derived coefficients against those provided by the developers.

2 Performance Testing and Results

2.1 Calibration / Estimation

Test ID: TS: Coefficients Intuition (#3) – Brady – TBD stars

Test Name: Intuition of Coefficients sign/magnitude


 * Objective: Evaluate intuition of sign of coefficients in models.
 * Method:
 * Divide the dataset into rolling time windows using intervals of 5 years or apply a chow test.
 * Conduct estimations on the AUD, CAD, and BRL currency models within each time window.
 * Assess how coefficients, like VIX or spread, alter across different time windows.
 * Implement a Markov-switching or similar regime-switching model for AUD, CAD, and BRL currency models.
 * Contrast the predictive power and accuracy of the regime-switching model with the traditional linear model.

Test ID: TS: Machine learning (#9) – TBD – 3 stars

Test Name: Machine learning Alternatives


 * Objective: Assess if the linear structure remains the most optimal choice.
 * Method:
 * Implement machine learning methods such as gradient boost, neural networks, and random forest to gauge potential fit enhancements.
 * Measure these enhanced results against the existing linear process to identify both strengths and potential areas of weakness.

Test ID: TS: OVERFIT (#1) – TBD – 3 stars

Test Name: Overfitting


 * Objective: Identify potential overfitting through out-of-period performance metrics.
 * Method:
 * Apply a rolling out-of-period analysis to identify potential model overfitting.
 * Concentrate the evaluation primarily on periods just before and during recessions.

Test ID: TS: COVID OVER (#10) – James – 3 stars

Test Name: Correct Covid Period Overprediction


 * Objective: Amend BPV10Y10Y and BPV10Y2Y models to rectify overprediction during the Covid period.
 * Method:
 * Incorporate a distinct identifier or flag for the Covid recession period.
 * Introduce an interaction variable that merges the recession flag with unemployment metrics.
 * Embed the 2/10 variable with lags spanning 6, 12, 18, and 24 months.
 * Analyze the modified model's results to discern any reduction in overprediction during the Covid timeframe.

Test ID: Alternative Configurations (#12) – TBD – 2 stars

Test Name: International Volatility Index


 * Objective: Introduce and evaluate new variables within the Equity Volatility Index Model.
 * Method:
 * Source a recognized international volatility index.
 * Incorporate and test this international volatility index within the existing Equity Volatility Index Model.
 * Conduct a PvA (Predicted vs. Actual) analysis to evaluate the model's performance.

Test ID: TS: Alternative Configurations (#13) (merge to #12) – TBD – 2 stars

Test Name: Additional variables to the Derived Currency Indices Model


 * Objective: Augment the Derived Currency Indices Model with novel variables.
 * Method:
 * Incorporate new variables: CPI, short-term spread, and the US volatility index into the Derived Currency Indices Model.
 * Execute a PvA (Predicted vs. Actual) evaluation to measure model performance.

Test ID: TS: MULTICO (#14) – TBD – 1 star

Test Name: Variance Inflation Factor (VIF) test


 * Reference to Testing Playbook: Linear Regression Models – Multicollinearity Tests
 * Objective: Calculate and articulate the VIF for every individual variable.
 * Method:
 * Apply the standard VIF test on the dataset.
 * Identify and flag any VIFs exceeding a value of 5 as they might indicate multicollinearity concerns.

Test ID: TS: YIELDCURVE (#19) (merged to #10)

Test Name: Yield Curve to Pick up Recessions


 * Reference to Testing Playbook: N/A
 * Objective: Analyze the effectiveness of adding the 2/10 year yield curve to BPV10Y2Y and BPV10Y10Y models in detecting recession peaks.
 * Method:
 * Integrate the 2/10 year yield curve data lagged by intervals of 6, 12, 18, and 24 months into the BPV10Y2Y and BPV10Y10Y models to monitor recession indications.
 * Conduct a PvA (Predicted vs. Actual) analysis to evaluate the accuracy and reliability of the model adjustments.

Test ID: TS: Reliance on brent oil (#5)

Test Name: Reliance on the Brent Oil Price for SPGSCITR Projections:


 * Objective: Assess the appropriateness of the existing SPGSCITR model that majorly depends on Brent oil price.
 * Method:
 * Break down SPGSCITR into its main sectors, including Energy, Agriculture, Livestock, Industrial Metals, and Precious Metals.
 * For each sector, independently apply regression using specific influential factors to produce projections.
 * Aggregate the individual sector projections to form the comprehensive SPGSCITR and contrast the output with the original model to determine the relative performance.

Test ID: TS: Alternative energy sources (#6) – 3 stars

Test Name: Evaluate the enhancement in the performance of the brent model when alternative energy source is used as one of the risk drivers in the model


 * Objective: Investigate how the availability of alternative energy sources might influence oil demand and its subsequent pricing.
 * Method:
 * Integrate an electricity demand/usage metric into the existing model specification.
 * Apply regression to the model with this new specification and determine variable coefficients.
 * Contrast the performance of the modified model against the standard model to discern any noticeable improvements or discrepancies.

Test ID: TS: Derivation of Stressed scenarios (#15) – 3 stars

Test Name: Derivation of stressed scenarios


 * Objective: Explore the outcomes when applying relative adjustments instead of absolute adjustments to stressed scenarios.
 * Method:
 * In the designated code file, apply modifications to the adjustments related to the stressed scenarios.
 * Produce revised stressed scenario curves based on the modified adjustments.
 * Align and compare these new curves with the original curves generated by the model developers to identify any significant differences.

2.2 Sensitivity Analysis

Test ID: TS: Sensitivity by launch points (#16)

Test Name: Assess the sensitivity of the NASDAQ scenarios to various launch points:


 * Objective: Investigate the stability of stress scenarios across diverse launch points.
 * Method:
 * Produce varied stress scenario projections, each initiated from a different launch point.
 * Align and compare the scenario projections derived from differing launch points.
 * Measure the fluctuation in scenario projections attributable to launch point variations by contrasting identical period data from distinct launch points.

3.4.1 Inputs

Test ID: TS: Brent model sensitivity (#17)

Test Name: Sensitivity of the Brent model


 * Objective: Analyze the sensitivity of the Brent model, especially given its recent limited range coverage.
 * Method:
 * Apply a relative adjustment to the most recent data set.
 * Using this newly adjusted data, recalibrate the Brent model.
 * Analyze the resultant changes in coefficients and contrast this recalibrated model with the original for performance differences.

2.3 Test of Model Data

Test ID: TS: Brent crude inputs (#7) (can be merged with #3) – 2 stars

Test Name: Brent crude oil input variables show significant volatility and indicate regime shift:


 * Objective: Assess the appropriateness of data employed in the brent crude model and discern potential regime shifts.
 * Method:
 * Examine the volatility present within the model's input data.
 * Using both parametric (like the Chow test or Markov switching model) and non-parametric methods, investigate signs of regime shifts in either the inputs or the dependent variable.
 * Under various identified regimes, scrutinize coefficient stability to validate model consistency.

Test ID: TS: Brent crude inputs prediction (#8) - 3 stars

Test Name: Assess the adequacy of the inputs used in the brent oil projections


 * Objective: Investigate the robustness of input data in the brent crude model, assess input-specific prediction errors, and understand their overall impact.
 * Method:
 * Determine prediction error for every input within the Brent model.
 * Evaluate the ripple effect of these errors on the broader Brent model's accuracy.
 * Employ error attribution techniques to pinpoint the input most responsible for overall prediction errors.
 * Amend historical input errors and regenerate Brent crude oil projections using accurate inputs.
 * Assess Brent projection errors arising primarily from modeling discrepancies.

3 Alternative Assumptions Testing

Test ID: TS: Autocorrelation (#18)

Test Name: Assess if any of models (VSTOXX, Nikkei VI, IVUKX30) can leverage autocorrelative factors as needed.


 * Objective: Explore potential model improvements by integrating autocorrelative factors.
 * Method:
 * Identify which models, among VSTOXX, Nikkei VI, and IVUKX30, demonstrate signs of autocorrelation.
 * Pinpoint autocorrelative elements that might enhance models displaying autocorrelation.
 * Evaluate the enhanced models' performance in comparison to their original specifications.