Assessment of reliability

Several criteria must be met to achieve high reliability in perceptual assessments. The speech material should be collected in a standardized manner (Lohmander & Olsson, 2004; Sell, 2009), and assessments should be made from audio or audio-video recordings made with high-quality equipment (Lohmander et al., 2009). Depending on the research question, appropriate speech material should be used (e.g., word naming, sentence repetition, conversational speech) and be clearly described. In cross-linguistic studies, it is crucial to use phonetically identical speech samples, and it is essential to be aware that the linguistic background may affect the listener's perception of speech (Hutters & Henningsson, 2004). As listeners may have different internal standards (Kreiman et al., 1993), they must calibrate their listening to become coherent (Chapman et al., 2016; Sell, 2005; Sell et al., 2009). It is recommended to have at least two impartial listeners conducting assessments, as treating SLPs/SLTs may exhibit bias. Additionally, the calculation method for reliability should be disclosed. If these criteria are not met, the reliability, and thus the validity, of the speech outcome must be questioned. 



*Chapman KL, Baylis A, Trost-Cardamone J, Cordero KN, Dixon A, Dobbelsteyn C, Thurmes A, Wilson K, Harding-Bell A, Sweeney T, Stoddard G, Sell D. The Americleft speech project: a training and reliability study. Cleft Palate-Craniofacial Journal, 2016; 53:93-108.                  

*Hutters B, Henningsson G. Speech outcome following treatment in cross-linguistic cleft palate studies: methodological implications. Cleft Palate- Craniofacial Journal, 2004;41:544-549.          

*Kreiman J, Gerratt BR, Kempster GB, Erman A, Berke GS. (1993). Perceptual evaluation of voice quality: review, tutorial, and a framework for future research. Journal of Speech and Hearing Research, 1993;36:21-40.

*Lohmander A, Olsson M. Methodology for perceptual assessment of speech in patients with cleft palate: a critical review of the literature. Cleft Palate- Craniofacial Journal, 2004;41:64-70.          

*Lohmander A, Willadsen E, Persson C, Henningsson G, Bowden M, Hutters B. Methodology for speech assessment in the Scandcleft project--an international randomized clinical trial on palatal surgery: experiences from a pilot study. Cleft Palate-Craniofacial Journal, 2009;46:347-362.              

*Sell D. Issues in perceptual speech analysis in cleft palate and related disorders: a review. International Journal of Language & Communication Disordorders, 2005;40:103-121.                

*Sell D, John A, Harding-Bell A, Sweeney T, Hegarty F, Freeman J. (2009). Cleft audit protocol for speech (CAPS-A): a comprehensive training package for speech analysis. International Journal of Language & Communication Disorders, 2009;44:529-548.