2 of 82
Federated Learning- Hope and Scope
Lhamu Sherpa and Nandan Banerji

Abstract at IgMin Research

Our mission is to foster interdisciplinary dialogue and accelerate the advancement of knowledge across a wide spectrum of scientific domains.

Reliability Evaluation of Professional Assessments



    14 Wood Street., San Francisco, CA, 94118 USA; 361 South Palouse Street, Walla Walla, WA 99362, USA


All professional technical assessment processes are fraught with uncertainty. If a decision is premised upon the result, the decision maker must understand the reliability of the performed assessment. A causal theory application is developed utilizing distinct (linguistic, ordered) terms and continuous (numerical) variables. It uncouples the methods from the result of the assessment obtained and focuses on those aspects that are important to the reliability assessment of the conclusion, not the answer itself. Matrices provide a means of characterizing the uncertainty of the methods and information available for each principal issue impacting the reliability. These matrices are determined as paired qualitative assessments of the Quality of the Measures Used and the Quality of Implementation of component description measures. Each is qualified by two to five grades, allowing three, five, seven, or nine quality distinctions for the assessed element. Uncertainty β values are determined for each component of the assessment combined by either an RMS procedure or a weighted average and converting a numerical value back to a consistent linguistic term. This procedure yields a basis for using good judgment while being sensible and reasonably cautious by independently determining the reliability using a carefully considered approach. California State University has assessed seismic retrofit priorities for 56 buildings using this method and has committed to its continuing use as its retrofit priority evaluation tool.



    1. Thiel CC, Zsutty TC, Lee YJ. Reliability of Seismic Performance Assessments for Individual Buildings and Portfolios. Risks. 2021; 9: 199.
    2. 2018. Society of Risk Assessment Glossary, 2018 indicated in the file name as Final,
    3. Aven T. Risk Assessment and Risk Management: Review of Recent Advances on their Foundation. European Journal of Operations Research. 2016; 1- 13,
    4. Pearl J. Causality: Models, Reasoning, and Inference, Second Edition, Cambridge University Press, Cambridge, UK. 2009.
    5. Poincaré H. The Foundations of Science, 1913 authorized translation by George Bruce Halsted, the Science Press, New York, NY. 1908; 1913: 375.
    6. Heilmann C. A New Interpretation of the Representational Theory of Measurement”, Philosophy of Science. 2015;82(5):787-797. DOI 10.1086/683280.
    7. Koller D, Friedman N. Probabilistic Graphic Models, Principals, and Techniques, the MIT Press, Cambridge, Massachusetts. 2009.
    8. Pearl J, Glymour M, Jewell NP. Casual Inference in Statistics, A Primer, John Wiley and Sons, New York. 2016.
    9. Halpern JY. Actual Causality, MIT Press, Cambridge, Massachusetts, Conn. 2016.
    10. Ghosal S, van der Vaart A. Fundamental of Nonparametric Bayesian Inference, Cambridge, Cambridge, UK. 2017.
    11. Pearl J, Mackenzie D. The Book of Why: The New Science of Cause and Effect, Basic Books, New York, NY. 2018.
    12. Fenton NF, Neil M. Risk Analysis and Decision Analysis with Bayesian Network, Second Edition, CRC Press, Boca Raton, Florida. 2019.
    13. Thiel CC, Zsutty TC. Determining the Reliability of a Seismically Assessed Building Conclusion Using a Modified FEMA P-154 Procedure. Civil Eng Res J. 2022; 13(2): 555858. DOI: 10.19080/CERJ.2021.13.555858
    14. Deierlein GG, Liel AB, Haselton CB,  Kircher CA. ATC 63 Methodology for Evaluating Seismic Collapse Safety of Archetype Building, Proceedings. ASCE Structures Congress 2008.
    15. Luce RD, Krantz DH, Suppes P, Tversky A. Foundations of Measurement, Volume III, Representation, Axiomatization, and Invariance, Dover Publications, Mineola, New York. 1990.
    16. Halpern JY. Reasoning about Uncertainty, Second Edition, MIP Press, Cambridge, Massachusetts. 2017.
    17. Thiel CC, Zsutty TC. Setting the Priority for Seismic Retrofit of Buildings Using a Modified FEMA P-154 Procedure”, Civil Eng. Res J. 2022; 13(2): 555857. DOI: 10.19080/CERJ.2021.13.555857
    18. Krantz DH, Luce RD, Suppes P, Tversky A. Foundations of Measurement, Volume I, Additive and Polynomial Representations, Dover Publications, Mineola, New York. 1971.
    19. Suppes P, Krantz DH, Luce RD, Tversky A. Foundations of Measurement, Volume II, Geometrical, Threshold, and Probabilistic Representation, Dover Publications, Garden City, New York. 1989.
    20. Michell J. An Introduction to the Logic of Psychological Measurement. Hillsdale NJ: Erlbaum. See also Michell. 1990; 1995.
    21. Decoene S, Onghena P, Janssen R. Representationalism under Attack. Review of An Introduction to the Logic of Psychological Measurement, by J Michell and Philosophical and foundational issues in measurement theory. by Wade Savage C, Ehrlich P. Journal of Mathematical Psychology. 1995; 39(2): 234–242.
    22. Michell J. Further Thoughts on Realism, Representationalism, and the Foundations of Measurement Theory, Author’s Response to Review by Decoene et al. of An Introduction to the Logic of Psychological Measurement. Journal of Mathematical Psychology. 1995; 39(2): 243-247.
    23. Boumans M. Measurement. In Durlauf SN, Blume LE. Editors, the New Palgrave Dictionary of Economics. Palgrave Macmillan, Basingstoke. 2008.
    24. Reiss J. Error in Economics: Towards a More Evidence-based Methodology. Routledge. 2008.
    25. Stevens SS. On the Theory of Scales of Measurement. Science. 1946; 103(2684): 677–680.
    26. Krantz DH, Luce RD, Suppes P, Tversky A. Foundations of Measurement, Volume I; Representations, Axiomatization, and Invariance, Dover Publications, Garden City, Wiley, New York. 1989.
    27. Davidson D, McKinsey JCC, Suppes P. Outlines of a formal theory of value. I. Philosophy of Science. 1955; 22(2): 140–160.
    28. Luce RD. Seimiorders and a theory of utility discrimination. Econometrica. 1956; 24: 178-191.
    29. Luce RD, Raiffa H. Games and decisions, Introduction and critical survey, Wiley, New York, New York. 1957.
    30. Efron B, Tibshirani RJ. An Introduction to the Bootstrap. New York: Chapman and Hall/CRC. 1993.
    31. FEMA P-154. 2015. Rapid Visual Screening of Building for Potential Seismic Hazards: A Handbook, (3rd ed.), Federal Emergency Management Agency USA. 2015.
    32. ASTM E2026-16a. Standard Guide for Seismic Risk Assessment of Buildings, ASTM International, Conshohocken, PA, June 2017.
    33. ASCE 41, 2017. Seismic Evaluation and Retrofit of Existing Buildings. The Structural Engineering Institute of the American Society of Civil Engineers. Reston, Virginia, USA.
    34. FEMA P-695. Quantification of Building Seismic Performance Factors. Federal Emergency Management Agency. Washington DC. 2009. (Also sometimes referred to as ATC-63)
    35. ASCE-7. 2022. Minimum Design Loads and Associated Criteria for Buildings and Other Structures, American Society of Civil Engineers, Reston, Virginia, ASCE/SEI Standard 41-22, 2021.
    36. Kahneman DK, Sibony O, Sunstein CR. Noise: A Flaw in Human Judgement, Little Brown Spark, New York, New York. 2021.
    37. Taleb NN. Statistical Consequences of Fat Tails: Real World Preasymtotics, Epistemology and Applications Papers and Commentary. STEM Academic Press. 2020.
    38. Dawes R. The Robust Beauty of Improper Linear Models in Decision Making. Chapter 28 of Psychological Bulletin. 1979; 1974, 81: 95-106. Reprinted in Kahneman et al., 1982.
    39. Kahneman D, Slovic P, Tversky A. Judgment under Uncertainty: Heuristics and Biases, Cambridge University PO\Press, Cambridge, UK. 1982.
    40. Gilovich T, Griffin D, Kahneman D. Heuristics and Biases, the Psychology of Intuitive Judgement Cambridge University Press, Cambridge. 2002.
    41. ASTM E2557-16. Standard Practice for Probable Maximum Loss (PML) Evaluations for Earthquake Due-Diligence Assessments, ASTM International, Conshohocken, Pennsylvania, June 2016.
    42. Wasserman L. All Nonparametric Statistics, Springer, New York, New York. 2006.
    43. 2022. California Building Standards Code, California Code Regulations, Title 24, California Building Standards Commission, Sacramento, California. Current Edition. This includes both the California Building Code requirements for new buildings (Part 2), and the California Existing Building Code for Existing Buildings (Part 10).
    44. Hagen BW. Problem, Risk and Opportunity: Enterprise Management, Probabilistic Publishing, Sugar Land, Texas. 2018.