I am building an internal rating model for commercial loans relying on expert-based scorecards. The ultimate goal of the exercise is to develop the model so that it maps with credit rating agencies' ratings. I am not an expert at all on that so i don't know how to calibrate the model i am building so that an S&P AAA-rated loan is given the highest score in my model. Do you think i would need to develop a regression to determine the weights of each criterion in my scorecard or is it more complex? Sorry to appear uninformed but i am learning as i am doing.
Assume your outcome/dependant variable is the rating agencies rating category, say 10 to 20 rating categories, you can use ordinal logistic regression which is more natural for this kinda problem. So the model will predict the rating category.
If your dependant variable is the internal default flag then you can have your model predict the default rate and then you can map the outcome of the model to the agencies' categories based on default rates. The EBA technical standards document on the subject discusses some of the potential issues/considerations: https://eur-lex.europa.eu/legal-content/EN/TXT/PDF/?uri=CELEX:32016R1799&from=EN
Hope this helps.