As a result of an internal audit we've been asked to develop some controls and governance around financially significant models used by the organization. (i.e. revenue models, capital models,
Financial Model Risk Management
Answers
There are really two main types of homegrown models -
Model version #1 - Models where an action is triggered automatically if certain events occur. This type of model requires heavy documentation regarding the model calculations/maintenance, back-up, security (fraud prevention), variables used... You really can go crazy. The Fed document is perfect.
Model version #2 - Models where a result is provided to senior managers, for something they are watching, yet the ultimate decision for action belongs to the Manager. This type of model requires lite documentation as the model is only providing you information. This documentation should discuss calculations, inputs, scenarios. But stay away from the decision process.
Please keep in mind that during your next audit, your Internal Auditors will ask for this documentation to validate the policies and procedures are being followed. What you create in the way of documentation must be maintained or you will suffer in next year's audit.
I just posted an internal audit blog on this site earlier in the week. Please take a look, as it may help.
Hope the info helps.
Where would you place brand reputation on your top 20 risk items to manage and why?
Shawn
I would be happy to share with you directly our policies and standards for Model validation, these are based on current best practices from the financial services regulators. We belive that the OCC guidance is probably the definiative guidance currently. SO far these have passed regulatory review!
Please drop me an email at [email protected]
Tom
I'm trying to collect policies, procedures, etc. that are in place for spreadsheet
At this point I haven't collected enough policies/procedures to provide you with any generalizations, but will post to Proformative when the article is complete.
All that being said, there is software to help manage controls and governance of spreadsheets/models - Prodiance (purchased by Microsoft a year ago), ClusterSeven, Finsbury Solutions to name a few.
There is also software to help audit spreadsheets - beyond the simplistic error checking
However no auditing software can do the analysis required to prove that a model is accurate from a business perspective. That's a human-only task at the moment.
For model building best-practices there are a few resources available - ssrb.org (Spreadsheet Standards Review Board) and fast-standard.org (FAST Modelling Standards - associated with a model dev company),
Tom Scott
http://www.jerts.com - @jertsconsulting
I am also interested in obtaining information on controls and governance relating to models that play a significant role in a company's financial risk
There are essentially three parts to establishing governance/controls of your key models (increasingly called 'end user computing' or EUC):
1. The first is all about 'what's out there'. This is all about establishing the current state of the organization i.e. how many models are important, what quality they are etc. This is all about discovery/transparency. It is important at this stage to use business relevant approaches to avoid just landing up with a list of several million files - which is probably what could be found on your servers. The discovery process will allow you to triage which models should receive closer attention/control. This may include decisions to migrate models into other more robust applications or to rewrite some spreadsheets with higher quality structure.
2. The second step is about control of the model calculation itself e.g. the formulas and macros that are the 'application' part of the spreadsheet. This will be a balance of preventive and detective controls that fit with the flexibility requirements of the business process under examination. i.e. if you inject too many preventive elements you may stall the business process and cause more damage to the reliability of business outputs than by doing nothing.
3. The third step is about control of the running of the model. For example it is all very well if the model is perfect and the formulas all protected but if the user forgets to refresh the data or uses incorrect data then the model will still give the wrong results. You may note that the recent Fed/OCC guidance extends model governance to the whole process (i.e. including data) to address these sorts of issues.
Of course the key issue for sustainable control is how you embed these principle into the business. One way to improve this is to look at the opportunity to save the time/effort of people who conduct manual checks on models.....you would be astonished (or perhaps not) to see how many hours are spent checking models in many organizations....but it is this expensive time that saves many businesses from suffering the public problems of financial/reputation loss.
Very good advice from Ralph Baxter! I was working on a simple Excel model this morning and the posts above prompted to add one quick thought: one significant obstacle to the maintenance of complex Excel based models is that as people move on to different responsibilities / jobs / companies and the subsequent owners of the model may not necessarily necessarily have the same level of modeling expertise as the previous owners. Multiply that over a few short years and you have the makings of significant problems in the model itself which can be compounded when one spends more time trying to figure out how the work works and is supposed to work rather than on what counts: sensitizing business strategies and the oft evolving tactics. This model risk must be managed and regulators focus much attention on the issue.
Thanks Eugene, your comment on the degradation of knowledge of models over time is right on the button.
One of our clients calls it the 'half life' of a model i.e. the time taken for a good model to begin to corrupt. It is this problem that stimulated the creation of ClusterSeven's software several years ago.
I know this is a little late to the thread, but thought I would share some quick insights.
When looking at the effective challenge of models (who can perform critical analysis, drive change, and document model assumptions and limitations), the first line of defense should be the owner of the model. The model owner is responsible for the models design, ensuring the model works as intended, and is the primary point of accountability for monitoring the model’s performance. The owner must monitor market events, uncover policy and regulation changes, and minimize operational risks.
After instituting this accountability at the model-level, take a look at the organizational structure of your company to determine effective model management at the enterprise level.
At a model risk management event in the summer of 2012, it was pretty clear that most organizations maintain a centralized approach to model management, and view models from the corporate enterprise perspective. This allows for central oversight and validation with clear guidelines on management and execution.
However, there is rationale for why a decentralized approach may work for your organization. Model risks may be assessed by committees, model uses may be confined to the same department that the model resides, or validation and audit activities are done at the business-unit level.
Either approach can and does work.
After your organizational structure and governance scope is defined, the challenges of documentation, inventories, performance monitoring, model use guidelines, and independent validation can be addressed.
Filed Under:
Risk Management