Directions for Use of Collaboration Forms

Updated: October 12, 2015

 

 

DIRECTIONS FOR USE OF COLLABORATION FORMS

NOTE:   A blank template can be found for each form described here in the ‘Forms for Collaboration’ section of the Dashboard.

 

There are 5 forms that can be used to review prosody:

Form #1. Analysis of single subject research

Form #2. Analysis of group research designs

Form #3. Analysis of secondary research such as meta analyses, systematic reviews, and narrative reviews.

Form #4. Critique of unsupported procedural descriptions. [These are intervention descriptions for which no supporting evidence is provided (i.e., expert opinion.)]

Form #5: Analysis of comparison studies. [Usually, these are neither intervention nor diagnostic studies. Rather, they provide information relevant to clinical intervention or learning, describe prosody disorders, or the prosodic characteristics of clinical conditions.]

To submit a review:

  • identify the appropriate form for your reading,
  • copy the appropriate analysis form (a blank template for each of the forms can be found in the ‘Forms for Collaboration’ section of the Dashboard),
  • if necessary, refer to the corresponding directions provided on this page
  • complete the form, and
  • submit the completed form, using the pdf format, if possible, to patricia.hargrove@mnsu.edu
  • Be sure to include your name and email address in the submission.

Directions for the Completion of Analysis Forms

 

 

——————————————————————————————————

 

#1 Directions for Single Subject Designs

 

NOTE: • Directions for sections of the form are in brackets [ ].

  • A blank template can be found in the ‘Forms for Collaboration’ section of the Dashboard.

 

 

EBP THERAPY ANALYSIS for

Single Subject Designs

 

 

Key:

C = Clinician

EBP = evidence-based practice

NA = not applicable

P = Patient or Participant

pmh = Patricia Hargrove, blog developer

SLP = speech–language pathologist

 

 

SOURCE: [Cite the source using APA style]

 

REVIEWER(S): [List the reviewer(s)’ name(s), affiliation(s), and email address(es). (The email address is for correspondence with the administrator and other reviewers. It will not be posted on the blog.)]

 

DATE: [List the date the review was completed using the Name of Month, Day of the Month, Year convention (e.g., September 25, 2015.)]

ASSIGNED OVERALL GRADE:

[• Provide your overall letter grade for the article in question.

  • Reviewers should consider the level of evidence, the strength and weaknesses of the methods, the data analysis, and interpretation of the results.
  • This is a subjective measure.
  • See “Letter Grades” in “Definitions and Terms” (on the Dashboard) for a guide to selection of a grade.]

 

TAKE AWAY: [Provide one or two sentences about the application of the prosodic intervention.]       

                                                                                                           

 

  1. What was the focus of the research?

[ – Place an “X” after the appropriate focus.

– The preferred research for these reviews is “Clinical Research.”]

  • Clinical Research

[ – This is research that directly pertains to the effectiveness of an intervention.

– For example, researchers may compare two different interventions to determine if one of them results in more change than the other or researchers may explore dosage issues such as the frequency of sessions.]

  • Clinically Related

[ – This is research that is concerned with intervention issues but does not examine whether or not a specific intervention is effective.

– For example, researchers may expose participants who are within normal limits and those who are prosodically impaired to a treatment on one occasion to see if there are immediate effects on the prosody.

– Thus, a researcher might play loud music into earphones to determine if the increased loudness level could induce the participants to speak more loudly.]

  • Essential Research:

[ – This is basic research in which the researcher explores the nature of prosody in typical learners.

– For example, a researcher might investigate developmental changes in the ability to produce sarcastic comment made by adolescents.]                                                                                   

 

 

  1. What type of single-subject evidence was identified?

                                                                                                          

2a What type of single subject design was used?

[ – Reviewers should place an xxx after the appropriate design.

– NOTE: It is possible that an article contain several case studies. Be careful not to confuse multiple single-subject studies with group studies.]

  • Case Studies:

[Case studies explore the effectiveness of an intervention but do not provide sufficient experimental controls to allow for claims that the intervention caused the changes in participants’ improvement.]

– Program Descriptions with Case Illustrations:

[ § In this type of case study, the purpose of the manuscript (i.e., article, book, webpage) is to describe an intervention.

  • A case or several cases are presented to explain the procedures, although the descriptions of results may be accompanied by claims of improvement.]

– Description of One Event: [ In this case study, the authors present the results of the intervention but do not provide sufficient pre-intervention data to allow for comparison of before and after intervention performance.]

– Description with Pre and Post Test Results: [The authors present data about the outcomes from before and after intervention to allow for comparison.]

  • Single Subject Experimental Design with Specific Client:

[These designs, in which participants serve as their own controls, can provide sufficient evidence to claim that an intervention has been successful with that specific participant. They do not, however, provide support for generalization to others.]

– Multiple Baseline:

[The researchers identify several objectives for a participant and sequentially addresses them. Progress should not be observed until the treatment begins on a specific objective.]

– ABAB (withdrawal/reversal):

[ § “A” represents no treatment on an objective and “B” represents treatment.

  • To prove effectiveness, progress should stop or regress during the second administration of “A.”]

– Alternating Treatment: [The researchers provide intervention for a specific objective using (at least) two different techniques that are administered separately.]

– Other: [There are many types of single subject experimental designs. If the design described in the research is not one of the three listed above, describe it here.]

                                                                                                           

2b What was the level of support associated with the type of evidence?

Level = ______ [See “ Levels of Evidence” in “Definitions and Terms” (on the dashboard).]           

 

                                                                                                           

  1. Was phase of treatment concealed? (answer Yes or No to each of the questions)

3a from participants? Yes ______   No _______   Unclear _______

3b from clinicians? Yes ______   No _______   Unclear _______

3c from data analyzers? Yes ______   No _______   Unclear _______

[ • Ideally assignment to treatment and control treatment should be concealed from participants, clinicians, and those who are evaluating the participants.

  • However, unlike medicine, when a “sugar pill” can be provided in place of a medication, it is difficult to conceal speech-language therapy from those who are receiving it.]
  • It also is more difficult to conceal group membership from speech-language pathologists (SLPs) who are providing treatment than from physicians who may be prescribing a drug treatment.
  • On the positive side, it is possible to conceal group membership from those who are evaluating baseline, pre- and post- treatment, or alternating treatment outcome measures.
  • This is can be accomplished by not providing evaluators with information about intervention status.
  • In some cases, this may result in preventing the SLP who administered the intervention from also serving in a dual role as evaluator.]

                                                                          

 

  1. Were the participants adequately described?

Yes ___   No ___   Unclear_______

[ • This is an overall evaluation of the quality of the researchers’ descriptions of the participants.

  • Because this is subjective, there may be considerable variation among reviewers. ]

 

4a How many participants (Ps) were involved in the study? [List total number here: ]

           

4b What P characteristics were controlled or described?

– List characteristics CONTROLLED :

[ – Using the list of characteristics below, list those characteristics that were inclusion/exclusion criteria for the research.

– For each characteristic, list the data appropriate for the characteristic (e.g., Age: preschooler or 4 years 0 months.)

– Add other controlled characteristics that are not provided in the list.]

  • age:
  • gender:
  • cognitive skills
  • expressive language:
  • receptive language:
  • MLU:
  • SES:
  • educational level of participant:
  • educational level of parents:
  • Other (list):

– List characteristics DESCRIBED:

[ – Using the list of characteristics below, list those characteristics that the researchers listed in the article.

– For each characteristic, list the data appropriate for that characteristic (e.g., Gender: 2m, 1f—this would indicate 3 case studies in which 2Ps were male and one female.)

– Add other controlled characteristics that are not provided in the list.]

  • age:
  • gender:
  • cognitive skills
  • expressive language:
  • receptive language:
  • MLU:
  • SES:
  • educational level of participant:
  • educational level of parents:
  • Other (list):

                                                 

4c Were the communication problems adequately described?

Yes___   No ___ Unclear/Variable __________

[ This is a subjective evaluation of the researchers’ description of the communicative strengths and limitations of the participants.]

 

  • List the disorder type(s): [Provide diagnosis/diagnoses here.]
  • List other aspects of communication that were described: [Provide scores on test and descriptors of participants’ communication skills.]

                                                                                                                       

  1. Was membership in treatment maintained throughout the study? Yes _____       no _____     unclear ____   not applicable (NA) _____

[ • This is the overview portion of the question.

  • Consider your responses to both 5a and 5b when responding.
  • If there is only one case, respond “not applicable.”]

                

5a If there was more than one participant, did at least 80% of the participants remain in the study? Yes ___       No _____

[ • The importance of continued involvement of all participants in intervention research cannot be understated.

  • Dawes (1999) reported that the standard used in medical research is that at least 80% of the initial group of participants must remain for a study to maintain validity.]

 

5b Were any data removed from the study? Yes ___       No _____

Explain: [Describe the data that were removed.]

[ • Research reports that excludes data from “outliers” should be viewed as adjusting group membership.

  • The ramifications of such a change should be carefully considered by the investigators and the reviewers.]

 

 

  1. Did the design include appropriate controls?

Yes _____           No _____         Unclear ____

[ • If the research involved case studies rather than single subject experimental design, the answer will be “no.”

  • If the research is single subject experimental design, answer this subjective overview question considering items 6a through 6f.]

                                                                      

6a Were baseline/preintervention data collected on outcomes?

Yes___     No ___ Data were Provided for Only Some Outcomes ___

6b Did probes/intervention data include untrained stimuli?

Yes___     No ___ No Data Provided ______

6c Did probes/intervention data include trained stimuli?

Yes___     No ___ No Data Provided ______

6d Was the data collection continuous? Yes___     No ___

6e Were different treatment counterbalanced or randomized?

Yes____     No ____     NA ____

6f If “6e” was yes, describe how it was matched (e.g., counterbalanced, randomized, not matched.) __________________________________

 

 

  1. Were the outcomes measures appropriate and meaningful?

Yes _____       No _____       Unclear/Variable ____

[Reviewers should review all dependent measures. This will help them determine their answer to this overall subjective. Consider if they might be reasonable and appropriate to clients.]

7a List the outcome(s)/dependent variable(s) with a number:

  • OUTCOME #1:
  • OUTCOME #2:
  • OUTCOME #3:

(continue numbering as needed)

7b List numbers of the outcomes that are subjective: [Use the outcome numbers from 7a.]

7c List numbers of the outcomes that are objective: [Use the outcome numbers from 7a.]

7d List the number of the outcome measures that are associated with reliability data, the type of reliability data (e.g., interobserver reliability, intraobserver reliability), and the actual data:

[ • Be sure to use the Outcome numbers from 7a.

  • Not all Outcomes will be associated with reliability data.
  • For example: Outcome #3; interobserver reliability r = 0.72; intraobserver reliability = 0.85.)]
  • Reviewers should not list outcomes that do not have reliability data.]
  • OUTCOME #1:
  • OUTCOME #2:
  • OUTCOME #3:

(continue numbering as needed)

 

  1. Results:

 

8a Did the target behavior(s) improve when treated?

Yes, for the most part___       No, for the most part ____ Variable_____

[ • This is a subjective judgment.

  • Reviewers should consider the general trend if there are multiple outcomes.]

 

8b For each of the outcomes, FROM YOUR POINT OF VIEW list the overall quality of improvement as strong, moderate, limited, ineffective, contraindicated: (The outcomes numbers should match the numbers in item 7a.)

[ • These are subjective judgments.

  • Reviewers should consider the information in Questions 7, 9, 10, 11, 12, 13 in making this judgment.]
  • OUTCOME #1:
  • OUTCOME #2:
  • OUTCOME #3:

(continue numbering as needed)

  1. Description of Baseline Data

9a Were baseline data provided? Yes _____     No _____ Variable ________

                                               

— If yes, continue through item #9 and list the number of data points for each dependent variable/outcome.

If no, proceed to item #10.

— If variable, place xxx next to the Outcomes that are associated with baseline data and only be concerned with those outcomes in items 9b through 9d. (The outcome numbers should match the numbers in item 7a.)

  • OUTCOME #1:
  • OUTCOME #2:
  • OUTCOME #3:

(continue numbering as needed)

 

9b Was baseline low (or high, as appropriate) and stable? (The outcome numbers should match the numbers in item 7a.)

  • OUTCOME #1:
  • OUTCOME #2:
  • OUTCOME #3:

(continue numbering as needed)

                                                       

9c Was the percentage of nonoverlapping data (PND) provided?

Yes _____ No ________ Only for the following Outcomes ___________

— Proceed to item 10, if the answer to item 9c is NO.

 

9d What was the PND and what level of effectiveness does it suggest?

(The outcome numbers should match the numbers in item 7a. The suggested interpretation of PND from Schlosser & Wendt, 2008, is

PND ≥ 90% = highly effective

PND = 70%-90% = fairly effective

PND – 50%-70% = questionable effectiveness

PND > 50% = unreliable/ineffective

 

  • OUTCOME #1:
  • OUTCOME #2:
  • OUTCOME #3:

(continue numbering as needed)

[A sample response to item 9d is— Outcome #70: PND = 95%; highly effective.]

  1. What was the magnitude of the treatment effect?

— If the magnitude is not provided, respond “NA ” next to the item 10 question.

 

If the magnitude is provided, list

  • the magnitude of treatment effect,
  • the measure used (e.g., Cohen’s d, number need to treat (NNT), eta, etc.),
  • the interpretation for each outcome using outcome numbers from item 7a.

 

(The interpretation can be the standard interpretation in the literature or the interpretation provided in the investigation.)

Outcome #1:

  • magnitude of effect:
  • measure calculated:
  • interpretation:

Outcome #2:

  • magnitude of effect:
  • measure calculated:
  • interpretation:

Outcome #3:

  • magnitude of effect:
  • measure calculated:
  • interpretation:

(continue as needed)

 

 

  1. Was information about treatment fidelity adequate?
  • Yes ___       No ___     Not Provided _____
  • Describe support:

 

 

 

  1. Were maintenance data reported? Yes ____ No ______

If yes, summarize findings:

 

 

 

  1. Were generalization data reported? Yes ____ No ______

If yes, summarize findings:

 

 

  1. Brief description of the design:

[ • Briefly describe the case study or single subject experimental design associated with this investigation.

  • Do not provide treatment procedures here.]

 

 

 

 

ASSIGNED OVERALL RATING OF THE QUALITY OF SUPPORT FOR THE INTERVENTION/ASSIGNED OVERALL GRADE: _________

 

[ • This should match the “Assigned Overall Grade” at the beginning of the form.

  • When Reviewers provide the rating/grade for the article in question, they should consider the level of evidence, the strength and weaknesses of the methods, the data analysis, and interpretation of the results.
  • See “Letter Grades” in “Definitions and Terms” (on the Dashboard) for an additional guide to determining a grade/rating.
  • This is a subjective measure.]

 

 

SUMMARY OF INTERVENTION

 

PURPOSE: [Paraphrase the purpose of the investigation here.]

POPULATION: [List the clinical groups described in this research.]

 

MODALITY TARGETED: [Note if the intervention is concerned with receptive or expressive prosody. ]

 

ELEMENTS/FUNCTIONS OF PROSODY TARGETED (do not list the specific dependent variables here):

[For example, note if the researchers targeted lexical stress (not sW or Ws words)]

 

ELEMENTS OF PROSODY USED AS INTERVENTION (part of independent variable; list only if prosody is being used as a treatment technique with a nonprosodic outcome):

[ • Only use this section if the intervention strategy involved prosody based treatment procedures.

  • For example, research concerned with Melodic Intonation Therapy with the outcome of improved articulation could be noted here.]

 

OTHER ASPECTS OF LANGUAGE/COMMUNICATION TARGETED (Dependent variable):

[• List nonprosodic communication outcomes here. For example, a researcher might be measuring intelligibility while using prosody treatment procedures. Intelligibility would be listed here.

  • The example “Intelligibility” could also be listed here if there were other prosodic outcomes.
  • The example “Intelligibility” would NOT be listed here if there were no prosodic outcomes and there were no prosody treatment procedures.]

 

OTHER TARGETS:

[- List here any outcomes that were not concerned directly with communication such as P’s happiness with the intervention

  • while using prosody treatment procedures or
  • when other outcomes included one or more aspects of prosody.]

DOSAGE: [Note the length of sessions, the number of sessions per week, the total number of sessions, the length of time the P was involved in intervention, whether the intervention involved group or individual therapy.]

 

ADMINISTRATOR:

[ • List whomever provided the intervention.

  • The administrator usually is a speech-language pathologist but it can be a family member, a paraprofessional, another professional, etc.]

 

STIMULI:

[ – Note

  • the types of stimuli that the administrator uses to provide the intervention (e.g., auditory, visual, kinesthetic, etc.) or
  • list all the stimuli in used in the treatment protocol.]

 

MAJOR COMPONENTS:

[ • Provide a summary of the intervention procedures described in the publication.

  • If possible, use bullets to highlight the steps or parts of the intervention(s).
  • If multiple treatments are described, be sure to list them all – even if they are just control or foil procedures.
  • If there is insufficient information in the article to describe the intervention, feel free to note this.]

 

 

 

========================================================

 

#2 Directions for Group Research Designs

 

NOTES: • Directions for sections of the form are in brackets [ ].

  • A blank template can be found in the ‘Forms for Collaboration’ section of the Dashboard.

 

Key:

C = Clinician

EBP = evidence-based practice

NA = not applicable

P = Patient or Participant

SLP = speech–language pathologist

 

 

 

SOURCE: [Cite the source using APA style]

 

REVIEWER(S):

[ • List the reviewer(s)’ name, affiliation, and email address.

  • The email address is for correspondence with the administrator and other reviewers. It will not be posted on the blog.)]

 

DATE: [List the date the review was completed using the Name of Month, Day of the Month, Year convention (e.g., September 25, 2015.)]

ASSIGNED OVERALL GRADE:

[ • Provide an overall letter grade for the article in question.

  • Reviewers should consider the level of evidence, the strength and weaknesses of the methods, the data analysis, and interpretation of the results.   This is a subjective measure.
  • See “Letter Grades” in “Definitions and Terms” (on the Dashboard) for a guide to selection of a grade.]

 

TAKE AWAY: [Provide one or two sentences about the application of the prosodic intervention.]       

                                                                                                           

 

  1. What was the focus of the research?

[ • Reviewers should highlight the by placing xxx after the appropriate focus.

  • The preferred research focus for the intervention reviews is “Clinical Research.”]
  • Clinical Research

[ – Research that directly pertains to the effectiveness of an intervention.

– For example, researchers may compare two different interventions to determine if one of them results in more change than the other or researchers may explore dosage issues (frequency of sessions, related to a specific intervention). ]

  • Clinically Related

[ – Research that is concerned with intervention issues but does not examine whether or not a specific intervention is effective.

– For example, researchers may expose participants (Ps) who are within normal limits and/or those with impaired prosody to a treatment on just ONE occasion to see if there are immediate effects on the prosody.

– Thus, a researcher might play loud music into earphones to determine if the increased loudness level could induce the Ps to speak more loudly.]

  • Essential Research:

[ – This is basic research in which the researcher explores the nature of prosody in typical learners.

– For example, a researcher might investigate developmental changes in the ability to produce sarcastic comment made by adolescents.]                                                                                   

 

 

  1. What type of evidence was identified?

                                                                                                           

1a What was the type of evidence? [place xxx next to the appropriate design]:

  • Systematic Review
  • Prospective, Randomized Group Design with Controls
  • Single Subject Experimental Design with Specific Client
  • Prospective, Nonrandomized Group Design with Controls?
  • Evidence-Based Clinical Practice Guidelines
  • Retrospective, Nonrandomized Group Design with Controls?
  • Prospective, Single Group with Pre- and Post-Testing
  • Retrospective, Single Group with Pre- and Post-Testing
  • Descriptive Research
  • Essential Research
  • Other: (List)

1b What was the level of support associated with the type of evidence?

Level = ____ [See “ Levels of Evidence” in “Definitions and Terms” (on the dashboard.)]

                                                                                                           

  1. Group membership determination:

                                                                                                           

2a If there was more than one group, were participants (Ps) randomly assigned to groups?

Yes ____   No ____     Unclear ____       Not Applicable (NA) ______

[ • The ideal research design for assessing treatment outcomes involves the random assignment of subjects to treatment and control groups.

  • Random assignment is most likely to lead to the equal distribution of known and unknown confounding variables in the treatment and control groups.
  • Unequal distribution can result in differences in the make-up of the two groups sufficient to cause the finding of a difference when there is, in fact, not one OR not finding differences when, in reality, there are differences. ]

2b If there were groups Ps were not randomly assigned to groups, were members of groups carefully matched?

Yes ____     No ___     Unclear ____     NA _______

[ • Careful matching of participants in the different groups allows for controlling of variables that are known or suspected to influence outcomes.

  • Matching may be preferred to randomization when the number of participants in each group is small.]

                                                                    

2c If the answer to 2a and 2b is ‘no’ or ‘unclear,’ describe the assignment strategy: [Note the strategy here.]

  1. Was administration of intervention concealed?

                                                                                                           

3a from participants?        Yes ____     No _____ Unclear ____

                                                                    

3b from clinicians?            Yes ____       No _____       Unclear ____

                                                                    

3c from analyzers?             Yes _____     No _____     unclear ____

[ • Ideally assignment to treatment and control groups should be concealed from participants, clinicians, and those who are evaluating the participants.

  • However, unlike medicine, when a “sugar pill” can be provided in place of a medication, it is difficult to conceal speech-language therapy from those who are receiving it.
  • Some concealment can be accomplished by comparing a target intervention to a second intervention or to a foil intervention.
  • It also is difficult to conceal group membership from speech-language pathologists (SLPs) who are providing treatment.
  • It is possible to conceal group membership from those who are evaluating pre- and post- treatment outcome measures by not providing evaluators with information about group membership.
  • The attempt to conceal group membership from evaluators, may result in preventing the SLP who provided the intervention from serving in a dual role as therapist and evaluator.
  • That is, the SLP who provided intervention should not transcribe audiotapes because it is likely that the SLP would recognize a client’s voice.
  • Guyatte and Rennie (2002) note that the more subjective the outcome measure, the more important it is that the evaluators are blind to group membership.]

                                                                    

 

  1. Were the groups adequately described?

Yes ___     No ___     Unclear___     Variable _____

[ • This is a subjective judgment.

  • Reviewers should take into consideration the information provided in items 4a-4d.
  • Sometimes the overall description is good but the description of the communication status (item 4d) is incomplete.
  • Reviewers have 2 options in that case: mark “Variable” or mark “Yes” and then add a note to the line stating that communication status was not clear.
  • Clear description of groups is particularly important when there is not random assignment to groups.
  • Guyatt and Rennie (2002) caution that with small sample size, randomization is less likely to achieve an equal distribution of known and unknown confounding variables.
  • Since speech-language pathology studies often involve a relatively small number of Ps and they often do not involve randomization, attention to the similarity of the treatment and control groups is important.
  • Research reports should provide a thorough description of the groups prior to administration of the intervention with particular attention to variables that are known or believed to affect the outcome measure (e.g., an expressive language intervention might control for variables as age, mean length of utterance, receptive language skills, etc.)]

 

4a How many Ps were involved in the study?

  • total # of Ps:  
  • # of groups:
  • # of participants in each group paired with names of the groups:

[For example, Treatment group = 15; No Treatment group = 15; Foil Group = 14]

[ Information about the number of Ps is important because the number of Ps can influence the type of statistics that should be used and can influence judgments about external validity.

 

4b List the P characteristics that are controlled (i.e., inclusion/exclusion criteria) or described. [Provide data for each characteristic. If there are multiple groups provide the data for each group.]

CONTROLLED

  • age: [Treatment group mean = 25.7 years; No Treatment group mean = 26.2 years; Foil group mean = 24.9 years]
  • gender: [Treatment group = 8m, 7f; No Treatment group = 7m; 8f; Foil Group =7m, 7f]
  • cognitive skills:
  • expressive language:
  • receptive language:
  • MLU:
  • Social Economic Status:
  • educational level of clients:
  • educational level of parents:
  • age at referral:
  • Other (list):

DESCRIBED:

  • age:
  • gender:
  • cognitive skills:
  • expressive language:
  • receptive language:
  • MLU:
  • Social-Economic Status:
  • educational level of clients:
  • educational level of parents:
  • age at referral:
  • Other (list):

[ • A list of the variables that the authors described or attempted to control can help reviewers make judgments about internal validity and freedom error.

  • In group studies, it is sufficient to provide means/averages for each of the characteristics.]

4c   Were the groups similar before intervention began?

Yes ___     No ___     Unclear ____ NA _______________

[ • Reviewers should be aware if there are grave imbalances in the groups prior to the intervention.

  • If there are imbalances, reviewers should determine if investigators made statistical adjustments as a correction in the analysis portion of the report.]

                                                         

4d Were the communication problems adequately described?

Yes ___     No ___     Unclear ____

 

  • disorder type: (List)

[ • Provide the diagnosis.

  • Additional descriptive information about the communication skills not provided in 4b should be listed here.]
  • functional level: [If this information is provided in the article, summarize the Ps’ level of functioning.]
  • other (list): [Provide any additional information about communication skills that were not noted in 4b or in the other portions of 4d.]

 

 

  1. Was membership in groups maintained throughout the study?

[ • The importance of continued involvement of all participants in intervention research cannot be understated.

  • Dawes (1999) reported that the standard used in medical research is that at least 80% of the initial group of participants must remain for a study to maintain validity.]

                                                                                                             

5a Did each of the groups maintain at least 80% of their original members?

Yes ___     No ___     Unclear ____

[ • It is important to attempt to retain as many Ps as possible in the groups.

  • Failing to maintain group membership can result in the systematic exclusion of Ps from the database which in turn can produce biased results.
  • For example, because there may be systematic reasons for research Ps failing to attend treatment sessions, it is important not to exclude Ps with erratic or poor attendance in therapy from the data analysis.
  • Erratic or poor attendance may be associated with traits or qualities that are normally distributed in the population.
  • To remove the Ps with that trait from the treatment group but not from the control group could influence the outcome.]

5b Were data from outliers removed from the study?              

Yes ____    No _____     Unclear ____

[ • Research reports that excluding data from “outliers” should be viewed as adjusting group membership.

  • The ramification of such a change in group membership should be carefully considered.]

 

 

  1. Were the groups controlled acceptably?

Yes _____       No _____       Unclear ____     NA_____

[ • This is a subjective judgment.

  • Reviewers should at least consider responses to items 4, 5, and 6 in their decisions.]

NOTE: If there was only one group, go to #7.

                                                                                                             

6a Was there a no intervention group?

Yes ___     No ___     Unclear ___    

[ • To prove that an intervention works, a treatment group should be compared to a control group.

  • One form of control group is a group in which there is no provision of services.]

                                   

6b Was there a foil intervention group?

Yes ___     No ___     Unclear ____

[ • To prove that an intervention works, the treatment group should be compared to a control group.

  • Another form of control is a group in which there is foil or placebo intervention.
  • Foil intervention provides some services but changes are not expected in the targeted outcomes.
  • For example, a foil intervention for Melodic Intonation Therapy with patients with aphasia may involve listening to classical music.]

                                   

6c Was there a comparison group?

Yes ___     No ___     Unclear ____

[ • To prove that an intervention works, the treatment group should be compared to a control group.

  • The final form of control group is a group in which there is services are provided using an existing intervention that currently has acceptance within the profession.
  • For the comparison group, Ps would be expected to improve on the targeted outcomes following the comparison intervention. ]

6d Was the time involved in the foil/comparison and the target groups constant? Yes ___     No ___     Unclear ____

[ • Guyatt and Rennie (2002) recommend that cointerventions, such as foil and comparison interventions, should receive the same amount of intervention as the treatment group.

  • However, if the question is “Does the amount of intervention result in increased performance on outcome measures?”, the time Ps are involved in the intervention should be different.]

 

 

  1. Were the outcomes measure appropriate and meaningful?

Yes _____       No _____       Unclear ____ Variable _______

[ • Reviewers should examine all the dependent/outcome measures.

  • This will help them determine if they are reasonable for the type of clients in question and the outcomes are concerned with behaviors that likely add to the client’s functioning.
  • List the targeted outcomes below; Reviewers may add outcome numbers as appropriate.
  • These numbers should be kept constant throughout the remaining items on the questionnaire.]

                                                                                                             

7a List outcomes (dependent variables):

  • OUTCOME #1:
  • OUTCOME #2:
  • OUTCOME #3:

(add additional outcomes as appropriate)

7b List the outcome measures (using outcome numbers from item 7a) that are subjective: [Subjective measures are those that require judgment on the part of the rater.]

 

 

7c List the outcome measures (using outcome numbers form item 7a) that are objective:

[ • Objective measures involve minimal judgment on the part of the rater.

  • Physiological measurements are examples of objective measures.]

 

 

  1. Were reliability measures provided?

[Reliability measures are important for subjective measures but they are appropriate for objective as well.]

                                                                                                            

8a Interobserver reliability for analyzers?

Yes ___     No ___   Unclear ____   Variable _____

[ • Interobserver reliability involves the comparing of scores for an outcome measure by at least two raters.

  • The data may be generated by the investigators or be reported from pre-existing sources.]

If yes or variable, list interobserver reliability of each outcome/variable:

 

  • OUTCOME #1:
  • OUTCOME #2:
  • OUTCOME #3:

(add additional outcomes as appropriate)

 

8b Intraobserver for analyzers?

Yes ___     No ___     Unclear ____ Variable _________

[ • Intraobserver reliability involves the comparing of scores for an outcome measure by a single rater on, at least, two separate occasions.

  • The data may be generated by the investigators or be reported from pre-existing sources.]
  • OUTCOME #1:
  • OUTCOME #2:
  • OUTCOME #3:

(add additional outcomes as appropriate)

 

8c Treatment fidelity for clinicians? Yes ___     No ___     Unclear ____

Describe treatment fidelity claims:

[ • Treatment fidelity is concerned with evidence describing how closely treatment procedures were adhered to during the investigation.

  • Often this achieved by reviewing videos of randomly selected sessions and noting if the intervention administrator included certain important components in the therapy session.]

 

 

  1. What were the results of the statistical (inferential) testing and/or descriptions of the results?

9a Summary Of Important Results

 

— What level of significance was required to claim significance? p = _______

[ • The most common criterion is p ≤ 0.05.

  • If only descriptive statistics are use, write “only descriptive statistics were provided” as a response to this query.]

Directions:

  • If the investigators compared ANY groups (e.g., treatment vs no treatment, treatment vs foil, treatment vs comparison) report the findings in the section labeled: Treatment And Foil/Comparison/No Treatment Group Analyses. Use this section to report results even if the investigators ALSO report pre and post results.
  • If the investigators ONLY report pre and post test analyses (or only post test analyses), report the finding in the section labeled: Pre And Post Treatment Only Analyses.
  • If the investigators ONLY use descriptive statistics, in response to the question about significance level (What level of significance will be required to claim significance?) write the following: “only descriptive statistics were provided.” Then, follow the 2 bullets above for directions about where to list findings.

[ • Sometimes researchers compare one intervention to another and in other cases researchers compare pre- and post- intervention scores for a specific intervention.

  • It is important to identify the compared data sets/values to develop a clear picture of the findings.
  • When there are significant differences, it is important to identify which data set was superior.]

TREATMENT AND FOIL/COMPARISON/NO TREATMENT

GROUP ANALYSES

For each outcome,

– list any significant difference among comparisons or

– in the case of the use of descriptive statistics only, list any notable differences or trends.

OUTCOME #1:

OUTCOME #2:

OUTCOME #3:

 

(Continue for all outcomes; the outcome numbers should match the outcomes in item #7.)

PRE AND POST TREATMENT ONLY ANALYSES

For each outcome,

– list any significant difference among comparisons or

– in the case of the use of descriptive statistics only, list any notable differences or trends.

OUTCOME #1:

OUTCOME #2:

OUTCOME #3:

 

(Continue for all outcomes; the outcome numbers should match the outcomes in item #7a.)

 

9b  What was the statistical test used to determine significance?  Place xxx after any test that was used in the investigation to determine significance. Write NA if only descriptive statistics were reported.

  • t-test:
  • ANOVA:
  • Mann-Whitney U:
  • Wilcoxan:
  • Other:  (List the test and the associated outcomes)

[ • Reviewers should note the name of the statistical test(s) used by the researchers.

  • This will allow for judging whether or not the test was appropriate.]

9c Was the confidence interval (CI) provided?  Yes ___     No _____

[ • Although the communication sciences and disorders literature frequently references confidence intervals (CIs) in the context of testing, CIs are not frequently addressed in our intervention literature.

  • Recall that CI signifies the certainty that a value represents the true value.
  • Thus, the CI of 95% reflects a range of values within which the true value lies in 95 out of 100 cases.
  • CIs have potential for use in evidence-based practice (EBP) treatment when estimating the strength of the difference between a treatment and a control group or between two treatment groups.
  • The upper and lower boundaries of the CI are called confidence limits and they can be used to determine important insights such as

– the adequacy of the number of subjects in the study,

– if a study in which the experimental treatment has been judged to be superior to the control treatment (positive study) needs to be viewed cautiously, and

– if a study in which the experimental treatment has NOT been judged to be superior to the control treatment (negative study) has any potential clinical utility.]

 9d  If the answer to 9c was yes, list the outcome number (from item #7a) associated with the reported confidence interval (CI):

  • 98% CI: 
  • 95% CI:
  • 90%
  • Other (List CI%):
  1. 10.  What is the clinical significance? 

[• The clinical significance or magnitude of the treatment effect can help clinicians determine the importance of the difference between the compared groups (e.g., treatment vs no treatment or pre-and post- intervention).

  • Clinical significance allows clinicians to predict the likelihood that their clients will benefit from a treatment and, in some cases it can be used to determine if one intervention is more effective than another.
  • Researchers can use a variety of EBP measures to determine clinical significance.]

10a  Place xxx next to the EBP measure(s) provided:

  • Standardized Mean Difference: 
  • Effect Size Correlation: 
  • Number Needed to Treat:
  • ETA: 
  • r2: 
  • other (List ):

10b Results of EBP testing and the interpretation:

DIRECTION: Next to the appropriate Outcome(s) list the name of the EBP measure, the result of the testing, and the interpretation (e.g., Large/Strong, Moderate, Small/Negligible, No Effect, Negative) using either the investigators’ interpretations or standard guidelines to EBP measures.

  • OUTCOME #1:
  • OUTCOME #2:
  • OUTCOME #3:

 

(add additional outcomes as appropriate)

 

  1. Were maintenance data reported? Yes ____ No ______

If yes, summarize findings:

 

 

  1. Were generalization data reported?

Yes ____     No _____     Not clear _____

If yes, summarize findings

  1. Briefly describe the experimental design of the investigation:

 

ASSIGNED OVERALL GRADE FOR OVERALL QUALITY OF EVIDENCE: _____

 

[ • Provide an overall letter grade for the publication in question.

  • Reviewers should consider the level of evidence, the strength and weaknesses of the methods, the data analysis, and interpretation of the results.
  • This is a subjective measure.
  • See “Letter Grades” in “Definitions and Terms” (on the Dashboard) for a guide to selection of a grade.]

 

 

SUMMARY OF INTERVENTION

 

PURPOSE: [Paraphrase the purpose of the investigation here.]

POPULATION: [List the clinical groups described in this research.]

 

MODALITY TARGETED: [Note if the intervention is concerned with receptive or expressive prosody. ]

 

ELEMENTS/FUNCTIONS OF PROSODY TARGETED (do not list the specific dependent variables here):

[For example, note if the researchers targeted lexical stress (not sW or Ws words); specific targets are listed at the end of this form.]

 

ELEMENTS OF PROSODY USED AS INTERVENTION (part of independent variable; list only if prosody is being used as a treatment technique with a nonprosodic outcome):

[ • Only use this section if the intervention strategy involved prosody based treatment procedures.

  • For example, research concerned with Melodic Intonation Therapy with the outcome of improved articulation could be noted here.]

 

OTHER ASPECTS OF LANGUAGE/COMMUNICATION TARGETED (Dependent variable):

[• List nonprosodic communication outcomes here. For example, a researcher might be measuring intelligibility while using prosody treatment procedures. Intelligibility would be listed here.

  • The example “Intelligibility” could also be listed here if there were other prosodic outcomes.
  • The example “Intelligibility” would NOT be listed here if there were no prosodic outcomes and there were not prosody treatment procedures.]

 

OTHER TARGETS:

[List here any outcomes that were not concerned directly with communication such as P’s happiness with the intervention

  • while using prosody treatment procedures or
  • when other outcomes included one or more aspects of prosody.]

DOSAGE: [Note the length of sessions, the number of sessions per week, the total number of sessions, the length of time the P was involved in intervention, whether the intervention involved group or individual therapy.]

 

ADMINISTRATOR:

[ • List whomever provided the intervention.

  • It usually is a speech-language pathologist but can be a family member, a paraprofessional, or another professional.]

 

STIMULI:

[Note

  • the types of stimuli that the administrator uses to provide the intervention (e.g., auditory, visual, kinesthetic, etc.) or
  • list all the stimuli in used in the treatment protocol.]

 

MAJOR COMPONENTS:

[ • Provide a summary of the intervention procedures described in the article.

  • If possible, use bullets to highlight the steps or parts of the intervention(s).
  • If multiple treatments are described, be sure to list them all – even if they are just control or foil procedures.
  • If there is insufficient information in the article to describe the intervention, feel free to note this.]

 

 

=========================================================

Form #3

NOTE: • Directions for sections of the form are in brackets [ ].

  • A blank template can be found in the ‘Forms for Collaboration’ section of the Dashboard.

SECONDARY REVIEW (SNR) CRITIQUE

KEY:

C = clinician

NA = not applicable

P = patient or participant

SLP = speech-language pathologist

SR = Systematic Review

Source: [Use APA format to cite the source.]

 

Reviewer(s): [List the name of the person performing the review.]

Date: [Write out the month day, year that the review was completed (e.g. October 6, 2015.)]

Overall Assigned Grade for Quality of Support:

[• Provide an overall letter grade for the article in question.

  • Reviewers should consider the level of evidence, the strength and weaknesses of the methods, the data analysis, and interpretation of the results.   This is a subjective measure.
  • See “Letter Grades” in “Definitions and Terms” (on the Dashboard) for a guide to selection of a grade.]

 

Level of Evidence: [See “ Levels of Evidence” in “Definitions and Terms” (on the dashboard).

What type of secondary review? (place xxx next to the type of secondary review)

  • Narrative Review:

[ This is the traditional review of the literature in which an author surveys a topic but does not provide evidence of a priori criteria for literature selection and analysis.]

  • Classic Systematic Review:

[ – This is a review of the literature in which authors of Systematic Reviews (SRs) identify the a priori criteria for literature selection and analysis of the selected literature.

– The authors summarize and analyze their findings qualitatively and do not rework the data.

– The standard for the Classic SR is the Cochrane Review in which a large team of reviewers collaborate to issue a relatively brief report on their findings which are published in media devoted to SRs or to evidence-based practice. ]

  • Narrative Systematic Review:

[ – The Narrative SR is similar to the Classic SR.

– The Narrative SR, however, involves a smaller number of authors and is published in media that does not solely focus on SRs or evidence-based practice.

– The Narrative SR tends to be longer that the Classic SR.]

 

  • Meta Analysis: [ This is a SR (Classic SR or Narrative SR) in which the authors rework to data of the reviewed articles and provide quantitative.]
  • Other:

[ • A form of secondary research not described above.

  • Reviewers should list the name of the type of secondary research.]

 

 

  1. Were the results valid? Yes ___ No ___   Unclear/Variable ___

[ – This is a subjective judgment.

– Often, it is difficult to answer this question with a simple “yes” or “no” because of the numerous factors (see 1a-1q) that enter into decisions about validity.

– The answer may be influenced by issues such as the reviewers’ expectations and their knowledge of the existing literature.]

 

1a Was the review based on a clinically sound clinical question?

Yes ___     No ___     Unclear/variable ____

[ – The purpose of the review should be reasonable given the knowledge base of the profession.

– The state of the literature will influence the focus of the question.

– If the knowledge base is strong, the question can be quite narrow (e.g., concerned with optimal therapeutic dosage).

– If little has been documented about an intervention, the question may be broader (e.g., concerned with whether or not the intervention results in positive change.)

– Some forms of secondary research, such as SRs and meta analyses, have stylized formats for asking clinical questions (Oxman et al., 2002; Sackett et al., 2000).]

 

1b Did the authors clearly describe reasonable criteria for inclusion and exclusion of literature in the review (i.e., sources)?

Yes ___     No ___   Unclear/Variable ___

[ – By including or excluding certain references, authors can construct arguments to support differing findings.

– The description of the rationale for including literature in secondary research allows reviewers to decide if they the authors of the secondary research have included a sufficient and unbiased sampling of the literature.

– Reviewers need to consider the soundness of the inclusions and exclusions.]

 

1c The authors of the secondary research noted that they reviewed the following resources: (place xxx next to appropriate resources)

  • abstracts from conferences
  • conference proceedings
  • experts                            
  • funding agencies
  • hand searches
  • internet based databases  
  • personal files
  • references from identified literature
  • registries
  • theses/dissertations
  • traditional databases
  • other (list):
  • Mark with X if the authors did not describe the search strategy _______

 

[ – Authors of secondary research may list the types of resources consulted in the search.

– A thorough review will include a variety of resources.

– Research with negative as well as positive outcomes is acceptable.]

 

  1. Did the sources involve only English language publications?

Yes ___     No ___     Unclear/Variable ___

[ • In the US, there is tendency to limit reviews to English because of the language skills of the authors.

  • Such a limitation can restrict the generalizability of the findings.]

 

  1. Did the sources include unpublished studies?

Yes ___    No ___     Unclear/Variable ___

[The inclusion of unpublished data is encouraged to balance journals’ tendency to publish only research with positive outcomes (Sackett et al., 2000.)]

 

  1. Was the time frame for the publication of the potential sources sufficient?

Yes ___     No ___     Unclear/Variable ___

[ • Reviewers can only make this judgment if they are familiar with the literature on a given topic.

  • If a reviewer is not familiar with the literature, he/she should check “Unclear/Variable.”]

 

1g Did the authors of the secondary research identify the level of evidence of the primary sources?

Yes ___     No ___     Unclear/Variable ___

[ – In their critiques of individual sources, it is helpful if authors of secondary research identify the research design of the source they are summarizing/reviewing.

– While the highest quality SRs contain randomized clinical trials, other types of reviews are not as restrictive.

– There are several systems for ranking levels of evidence, but no one strategy has achieved total acceptance (Law & Philip, 2002; Robey, 2004).

– The system used for ranking the level of evidence for the Clinical Prosody reviews can be found in “Levels of Evidence” on the dashboard.

 

1h Did the authors of the secondary research describe procedures used to evaluate the validity of each of the sources?

Yes ___     No ___     Unclear/Variable ___

[ – When authors of secondary research describe the strategy they used to analyze the quality of sources, they allow reviewers to make their own judgments about the source(s).

– In some types of secondary research (i.e., narrative reviews), authors generally do not do this.

– Reviewers should be alert to secondary research in which authors evaluate some of the sources but not all of them.]

 

1i Was there evidence that a specific, predetermined strategy was used to evaluate the sources?

Yes ___     No ___     Unclear/Variable

[ The most unbiased type of review involves the identification a strategy for analyzing sources that was established before authors started their tasks.]

1j Did the authors of the secondary research or review teams rate the studies independently?

Yes ___     No ___     Unclear/Variable  ___

[- Reviewers should note whether the authors of the secondary research (or review teams) critiqued the sources without the knowledge of other evaluations.]

1k Were interrater reliability data provided?

Yes ___     No ___     Unclear/Variable  ___

[ – Note if data were provided describing the agreement between/among authors of the secondary research or review teams.]

1l If the authors of the secondary research provided interrater reliability data, list the data here: _____________________________________________________

 

1m If there were no interrater reliability data, was an alternate means to insure reliability described?

Yes ___     No ___     Unclear/Variable  ___ Not Applicable (NA) ______

[ – There are alternatives to describing the agreement between/among (independent) reviews.

– For example, at least two authors/research teams may critique the sources independently and meet to compare critiques and resolve disagreements.]

1n Were assessments of sources sufficiently reliable?

Yes ___     No ___     Unclear/Variable ________   NA __________

[ – Even with clear instructions, critiques of individual sources are subjective and there can be differences of opinion.

– If reviews are undertaken using specific criteria, authors of secondary research or review teams should provide some type of reliability data or describe how they controlled for differing evaluations, source identification, and/or ranking the level of evidence.

– Reviewers should provide their own opinion about the quality of measures taken to insure reliability from items #1i through 1m.]

1o Was the information provided sufficient to allow a reader to undertake a replication

Yes ___     No ___     Unclear/Variable  ___    NA _______

[ If there are instructions, reviewers should judge whether there is sufficient information for an independent (and accurate) critique of the sources.]

 

1p Did the sources that were evaluated involve a sufficient number of participants?

Yes ___     No ___     Unclear/Variable  ___

[ – Ideally, authors of secondary research should inform readers of the number of participants in each source because investigations with small numbers are vulnerable to bias problems associated with the increased likelihood of unrepresentative sampling of the population.

– Moreover, investigations employing small numbers generally use less powerful statistical tests and are less likely to yield statistically significant differences.

– Reviewers should consider the number of participants while keeping in mind the difficulty of achieving homogeneity in groups with large number of participants with communication disorders.]

 

1q. Were there a sufficient number of sources?

Yes ___     No ___     Unclear/Variable  ___

[ When authors of secondary research use a small number of sources, the review findings should be considered preliminary in nature, particularly if the levels of evidence are relatively low.]

  1. Description of outcome measures:

List the outcome measure(s) of interest to the authors.

  • Outcome #1:
  • Outcome #2:
  • Outcome #3:
  • Outcome #4:

 

(add additional numbers if necessary):

 

[ – Secondary research may include outcome measures that are not concerned with prosody, treating prosody, or using prosody to treat other targets.

– In such cases, exclude outcomes that are not concerned with prosody.

– Outcome measures that should be included may be concerned with

  • prosody of typical or impaired speakers (e.g., use of stress by speakers following left hemisphere stroke)
  • treating prosody (e.g., decreased rate or accurate stress placement in multisyllabic words),
  • using prosody to treat other targets such as using rhythm to treat other aspects of language/communication (e.g., increased mean length of utterance, improved intelligibility, increased rate of communication), or increased function/ quality of life (e.g. satisfaction, willingness to interact with strangers).]

 

 

  1. Description of results:

3a What measures were used to represent the magnitude of the treatment effect/effect size? (Place xxx next to all that apply)

  • standardized mean difference (d)
  • number needed to treat (NNT)
  • effect size correlation (rxy)
  • relative success rate (RSR)
  • odds ratio (R2)
  • eta
  • other (list)
  • No metrics representing magnitude of treatment effect were reported

[ – Measures that represent the magnitude of the treatment effect are of interest because they quantify the difference between groups.

– The different measures listed above represent different aspects of the magnitude of the treatment effect or effect size.

– It is clearest, if all of the sources are reported using the same measure.]

3b Summarize overall findings of the secondary research:

[ • Note the findings of the secondary research here.

  • The results of inferential and correlational analyses also can be included here.]

 

3c Were the results precise?

Yes ___     No ___     Unclear/Variable ___

[ – Confidence intervals report the upper and lower boundaries within which the “true” score should be located.

– A narrow confidence interval represents precise results.

– Confidence intervals are usually accompanied by a probability level which signifies the likelihood that the “true” score is within the range of values (i.e., confidence interval).

– Oxman et al. (2002) recommend 95% probability level be the target as it minimizes error.

– A narrow confidence interval with a 95% probability level is difficult to achieve in the communication disorders literature because our studies often involve a small number of participants and/or the heterogeneous nature of communication disorders.

– Confidence intervals can be calculated from the data provided in the sources, if sufficient information is provided in the original source.]

3d Did the reviewers consider whether evaluations would have varied if the “true” value of metrics were at the upper or lower boundary of the confidence interval? Yes ___     No ___    Unclear/Variable ___   NA ___

[ – If confidence intervals are available, the authors of secondary research should have inspected the data to make sure that the upper limit of either the groups does not overlap with the lower limit of the other group(s).

– If there is overlap, then the differences between the groups should not be considered to be clinically significant.]

3e Were the results of individual studies clearly displayed/presented?

Yes ___     No ___     Unclear/Variable ___

[ – This item is concerned with the clarity of the reviews.

– Authors of secondary researchers should consistently describe the outcomes of the sources.

– This description can be presented in writing or graphically. Graphic representations are helpful when large number of sources are reviewed.]

3f For the most part, were the results similar from source to source?

Yes ___     No ___     Unclear/Variable ___

[ – Ideally, the secondary research will have yielded consistent results although this is not always the case.

– If the sources differ in degree of difference between groups AND the direction differs, “No” should be checked here.]

3g Were the results in the same direction?

Yes ___     No ___     Unclear/Variable ___

[ – This item is concerned only with the agreement among sources with respect to the consistency of findings relative to the direction of differences among sources.

– Sackett et al. (2000) have noted that consistency of direction is more important than consistency in the degree of difference between treatment and control groups.]

 

3h Did a forest plot indicate homogeneity?

Yes ___     No ___    Unclear/Variable ___     NA ___

[ – When authors of secondary research analyze multiple sources that have included EBP metrics/effect sizes, they can (but are not required to) present the results using a forest plot.

– In forest plots, authors graphically display the EBP measure (e.g., d) and confidence interval for each source.

– From this display, readers can identify the sources that were the most effective, least variable, or unlikely to achieve statistical significance. (The final characteristic can be surmised by noting if d crosses zero on the forest plot.)

– Overall, forest plots can provide information about the precision, the strength, direction, and consistency of the finding.

– Forest plots that exhibit agreement in direction and to a lesser degree strength are considered to be homogeneous.]

 

3i Was heterogeneity of results explored?

Yes ___     No ___     Unclear/Variable ___

[ – If the direction and, to a lesser degree, strength varies among the sources, the results are heterogeneous or inconsistent.

– Ideally, authors of secondary research will identify why these differences exist (e.g., differences in participants, dosages of the intervention, outcome measures, or experimental designs).]

  1. Were the findings reasonable in view of the current literature?

Yes ___     No ___     Unclear/Variable ___

[ – This query involves a judgment on the part of the Reviewer.

– This item requires that the Reviewer have sufficient knowledge of the literature to make this decision.

– Reviewers who are concerned that their knowledge of the literature is insufficient to make a decision should select “Unclear/Variable.”]

  1. Were negative outcomes noted?

Yes ___     No ___     Unclear/Variable ___

[ – The communication disorders intervention literature contains few reports in which there are negative outcomes.

– Reviewers should inspect the reviews to determine if negative outcomes have been reported. ]

  1. Were maintenance data reported? Yes ____ No ______

If yes, summarize findings:

 

 

  1. Were generalization data reported? Yes ____ No ___x___

If yes, summarize findings:

SUMMARY OF INTERVENTION

NOTE:   Reviewers should only complete this section if sufficient information is provided in the review to describe treatment procedure(s).

 

For each procedure detailed in the review, provide the following information:

Population: [Paraphrase the purpose of the investigation here.]

 

Prosodic Targets: [List which aspect of prosody that were the focus of the research. For example, note if the targets included lexical stress (not sW or Ws words)]

Nonprosodic Targets:

[ – List nonprosodic communication outcomes here.

– For example, a researcher might be measuring intelligibility while using prosody treatment procedures. Intelligibility would be listed here.

– The example “Intelligibility” could also be listed here if there were other prosodic outcomes.

– The example “Intelligibility” would NOT be listed here if there were no prosodic outcomes and there were not prosody treatment procedures.]

Aspects of Prosody Used in Treatment of Nonprosodic Targets:

[ – Only use this section if an intervention strategy involved prosody based treatment procedures.

– For example, research concerned with Melodic Intonation Therapy with the outcome of improved articulation could be noted here.]

 

Description of Procedure #1—(provide title)

[number each procedure separately

  • provide a title for the procedure.
  • if the authors of the secondary research do not provided a title, provide your own title
  • briefly describe the procedure]

 

 

 

Evidence Supporting Procedure #1—(provide title)

[use the number and name associated with the description of the above procedure.

  • briefly describe the evidence in favor of the procedure]

 

 

Evidence Contraindicating Procedure #1—(provide title)

[use the number and name associated with the description of the above procedure.

  • briefly describe the evidence in indicating that the procedure is not effective.]

 

————————————————————————————————————

FORM #4

CRITIQUE OF UNSUPPORTED PROCEDURAL DESCRIPTIONS

(also know as Expert Opinion)

NOTE:

  • A blank template for this form can be found in the ‘Forms for Collaboration’ section of the Dashboard.

KEY
C = clinician

NA = not applicable

P = patient or participant

pmh = Patricia Hargrove, blog developer

SLP = speech-language pathologist

Source: [Cite the source using APA style]

 

Reviewer(s):

[ • List the Reviewer(s)’ name, affiliation, and email address.

  • The email address is for correspondence with the administrator and other reviewers. It will not be posted on the blog.]

 

Date: [List the date the review was completed using the Name of Month, Day of the Month, Year convention (e.g., September 25, 2015.)]

 

Overall Assigned Grade (because there are no supporting data, the highest grade will be F)   ________________

[ • Provide your overall letter grade for the article/reading in question.

  • Reviewers should consider, at least, the level of evidence, the strength and weaknesses of the support for the authors’ claims, and clarity of the writing.
  • This is a subjective measure.
  • See “Letter Grades” in “Definitions and Terms” (on the Dashboard) for a guide to selection of a grade.]

 

Level of Evidence: F = Expert Opinion, no supporting evidence for the effectiveness of the intervention although the author may provide secondary evidence supporting components of the intervention/interpretations. [Reviewers do not have to identify the level of evidence because F is level for this type of publication.]

 

Take Away: [Provide one or two sentences about the application of the prosodic intervention.]

 

 

  1. Was there a review of the literature supporting components of the intervention? Yes ______   No ________

– If there was a review of the literature, place xxx next to the type of literature review: (hint, in most cases it will be a Narrative Review)

  • Narrative Review:

[ – This is the traditional review of the literature in which an author surveys a topic but does not provide evidence of a priori criteria for literature selection and analysis.]

  • Classic Systematic Review:

[ – This is a review of the literature in which authors of Systematic Reviews (SRs) identify the a priori criteria for literature selection and analysis of the selected literature.

– The authors summarize and analyze their findings qualitatively and do not rework the data.

– The standard for the Classic SR is the Cochrane Review in which a large team of reviewers collaborate to issue a relatively brief report on their findings which are published in media devoted to systemic reviews or to evidence-based practice. ]

  • Narrative Systematic Review:

[ – The Narrative SR is similar to the Classic SR.

– The Narrative Systematic Review, however, involves a smaller number of authors and is published in media that does not solely focus on SRs or evidence-based practice.

– The Narrative SR tends to be longer that the Classic Systematic Review.]

 

  • Meta Analysis:

[ This is a SR (Classic Systematic Review or Narrative Systematic Review) in which the authors rework to data of the reviewed articles and provide quantitative analyses.]

  • Other:

[ – A form of secondary research not described above.

– Reviewers should list the name of the type of secondary research.]

 

 

 

  1. Were the specific procedures/components of the intervention tied to the reviewed literature? Yes ___ No ___   Unclear/Variable ___

[ Even though the authors of the publication do not provide original research for their intervention plan/recommendations, they may support the intervention plan/recommendations using the existing literature.]

 

  1. Was the intervention based on clinically sound clinical procedures?

Yes ___     No ___     Unclear/variable ____

[Even though the authors of the publication do not provide original research for their intervention plan/recommendations, they may support the intervention plan/recommendations using currently accepted clinical procedures.]

  1. Did the author(s) provide a rationale for components of the intervention? Yes ___     No ___     Unclear/variable ____

[The authors of the publication describe their reasons for including the various steps in the intervention/plan.]

Optional Comments (for item #4):

 

  1. Description of outcome measures:

 

5a Are outcome measures suggested/presented?

Yes ___     No ___     Unclear/variable ____

 

5b If there are outcome measures, list them here.

  • Outcome #1:
  • Outcome #2:
  • Outcome #3:
  • Outcome #4:

 

(add additional outcomes numbers,{ if necessary):

 

  1. Was generalization addressed? Yes ____ No ______

If yes, describe:

 

 

  1. Was maintenance addressed? Yes ____ No ______

If yes, describe:

 

 

SUMMARY OF INTERVENTION

(NOTE: Only complete this portion of the form if the publication was about intervention/therapy)

 

 

PURPOSE: [Paraphrase the purpose of the publication here.]

POPULATION: [List the clinical groups described in this publication.]

 

 

MODALITY TARGETED: [Note if the intervention is concerned with receptive or expressive prosody.]

 

 

ELEMENTS/FUNCTIONS OF PROSODY TARGETED (do not list the specific outcomes here):

[ For example, note if the authors recommend targeting lexical stress (not sW or Ws words)]

 

 

ELEMENTS OF PROSODY USED AS INTERVENTION (List only if prosody is being used as a treatment technique with a nonprosodic outcome.):

[ – Only use this section if the intervention strategy involved prosody based treatment procedures.

– For example, treatments using Melodic Intonation Therapy with the outcome of improved articulation could be noted here.]

 

 

OTHER ASPECTS OF LANGUAGE/COMMUNICATION TARGETED (Outcomes):

[ • List nonprosodic communication outcomes here.

  • For example, an expert may suggest that clinicians use prosody treatment procedure to improve. Intelligibility would be listed in this section.
  • The example “Intelligibility” could also be listed here if there were other prosodic outcomes BUT…..
  • The example “Intelligibility” would NOT be listed here if there were no prosodic outcomes and there were no prosody treatment procedures.]

 

 

OTHER TARGETS:

[ – List here any outcomes that were not concerned directly with communication such as P’s happiness with the intervention

  • while using prosody treatment procedures or
  • when other outcomes included one or more aspects of prosody.]

DOSAGE: [Note the length of sessions, the number of sessions per week, the total number of sessions, the length of time the P should be involved in intervention, and whether the intervention involved group or individual therapy.]

 

 

ADMINISTRATOR:

[ – List who provides the intervention.

– The provider usually is a speech-language pathologist (SLP) but it can be a family member, a paraprofessional, or another professional.]

 

STIMULI:

[ – List

  • the types of stimuli that the administrator uses to provide the intervention (e.g., auditory, visual, kinesthetic, etc.) OR
  • all the stimuli in used in the treatment protocol.]

 

MAJOR COMPONENTS:

[ • Provide a summary of the intervention procedures described in the publication.

  • If possible, use bullets to highlight the steps or parts of the intervention(s).
  • If multiple treatments are described, be sure to list them all.
  • If there is insufficient information in the publication to describe the intervention, feel free to note this.]

————————————————————————————————————

FORM #5

ANALYSIS GUIDELINES

Comparison/Disorders Research

 

NOTES:

  • A blank template for this form can be found in the ‘Forms for Collaboration’ section of the Dashboard.
  • Use this form when the focus of the intervention is not on intervention. Accordingly, no summary of intervention is included in the review.

 

KEY:

 

MLU = mean length of utterance

NA = Not Applicable

P = participant or patient

pmh = Patricia Hargrove, blog developer

SLP = speech-language pathologist

 

 

SOURCE: [Cite the source using APA style]

 

REVIEWER(S):

[ – List the reviewer(s)’ name, affiliation, and email address.

– The email address is for correspondence with the administrator and other reviewers. It will not be posted on the blog.)]

DATE: [ List the date the review was completed using the Name of Month, Day of the Month, Year convention (e.g., September 25, 2015.)]

 

ASSIGNED GRADE FOR OVERALL QUALITY:

[ – Provide an overall letter grade for the article in question.

– Reviewers should consider the level of evidence, the strength and weaknesses of the methods, the data analysis, and interpretation of the results.  – See “Letter Grades” in “Definitions and Terms” (on the Dashboard) for a guide to selection of a grade.]

 

TAKE AWAY: [ Provide one or two sentences about the application of the prosodic intervention.]

 

 

  1. What type of evidence was identified?

                                                                                                           

1a What was the type of design?

DIRECTION: place xxx after appropriate design type(s)

  • Comparison Research
  • Correlational Research
  • Descriptive Research
  • Developmental Research
  • Evidence-Based Clinical Practice Guidelines
  • Prospective, Nonrandomized Group Design with Controls
  • Prospective, Randomized Group Design with Controls
  • Prospective, Single Group with Pre- and Post-Testing
  • Retrospective, Nonrandomized Group Design with Controls
  • Retrospective, Single Group with Pre- and Post-Testing
  • Single Subject Experimental Design with Specific Participant
  • Systematic Review
  • Other: (List)

1b What was the focus of the research?

DIRECTION: Place xxx after the appropriate focus

  • Clinical Research
  • Clinically Related
  • Essential Research
  • Clinical Research

[ – Research that directly pertains to the effectiveness of an intervention.

– For example, researchers may compare two different interventions to determine if one of them results in more change than the other or researchers may explore dosage issues (frequency of sessions, related to a specific intervention.]

  • Clinically Related

[ – Research that is concerned with intervention issues but does not examine whether or not a specific intervention is effective.

– For example, researchers may expose participants (Ps) who are within normal limits and those with impaired prosody to a condition on one occasion to determine if there are immediate effects on the prosody.

– Thus, a researcher might play loud music into earphones to determine if the increased loudness level could induce Ps to speak more loudly.]

  • Essential Research:

[ – This is basic research in which the researcher explores the nature of prosody in typical learners.

– For example, a researcher might investigate developmental changes in the ability to produce sarcastic comment made by adolescents.]                                                                                   

                                                                                                          

1c What was the level of support associated with the type of evidence? Level = ____

[See “ Levels of Evidence” in “Definitions and Terms” (on the dashboard).]

 

 

                                                                                                           

  1. Group membership determination:

                                                                                                           

2a If there were groups, were participants randomly assigned to groups?          

Yes ____   No ____     Unclear ____       Not Applicable (NA) ______.

[ – The ideal research design involves the random assignment of Ps to groups.

– Random assignment, however, is not feasible when one is comparing the speech-language of impaired and typically speaker.]

 

2b If there were groups and the Ps were not randomly assigned to groups, were members of groups carefully matched?

Yes ____     No ___     Unclear ____     NA _______

[ – Careful matching of Ps in the different groups allows for controlling of variables that are known or suspected to influence outcomes.

– Matching may be preferred to randomization when the number of Ps in each group is small.]

  1. Were experimental conditions concealed?

                                                                                                           

3a from participants?

Yes ____     No _____ Unclear ____

                                                                    

3b from administrators of experimental conditions?

Yes ____       No _____       Unclear ____

                                                                    

3c from analyzers/judges?          

   Yes _____     No _____     Unclear ____

 

[• Ideally assignment to groups should be concealed from Ps, investigators, and those who are evaluating the Ps.

  • Unfortunately for researchers, it can be difficult to conceal speech-language impairments from those who have been diagnosed with speech-language impairments.
  • Likewise, usually is difficult to conceal group membership of speakers with speech-language impairment from administrators of experimental tasks.
  • It is possible in some cases, however, to conceal group membership from those who are evaluating outcome measures by not providing evaluators with information about group membership and randomly ordering results of testing.
  • Guyatte and Rennie (2002) note that the more subjective the outcome measure, the more important it is that the evaluators are blind to group membership.]

                                                                    

 

  1. Were the groups adequately described?

Yes ___     No ___     Unclear___   Variable _______

[ • This is a judgment.

  • Reviewers should take into consideration the information provided in items 4a-4d.
  • Sometimes the overall description is good but the description of the communication status (item 4d) is incomplete.
  • Reviewers have 2 options in that case: (1) mark “Variable” or (2) mark “Yes” and then add a note to the line stating that communication status was not clear.
  • Clear description of groups is particularly important when there is not random assignment to groups.
  • Guyatt and Rennie (2002) caution that with small sample size, randomization is less likely to achieve an equal distribution of known and unknown confounding variables.
  • Since communication disorders studies often involve a relatively small number of subjects and they often do not involve randomization, attention to the similarity of groups is important.
  • Research reports should provide a thorough description of the groups with particular attention to variables that are known or believed to affect the outcome measure (e.g., age, mean length of utterance, receptive language skills, etc.) ]

 

4a       How many participants were involved in the study?

  • total # of Ps:
  • # of groups:
  • List names of groups and the # of participants in each group:

 

[• For example, Stuttering Diagnosis= 15; Nonstuttering Peers= 15

  • Information about the number of Ps is important because the number of Ps can influence the type of statistics that should be used as well as judgments about external validity.]

 

  • Did all groups maintain membership?  

   Yes ____   No   _____   Unclear _________

[ • It is important to attempt to retain as many Ps as possible in the groups.

  • Failing to maintain group membership can result in the systematic exclusion of Ps from the data which in turn can produce biased results.
  • A commonly accepted standard for group maintenance is 80%.]

 

                                                                                

4b List the controlled (i.e., inclusion/exclusion criteria) or described participant characteristics. Provide raw or summary data for each of the characteristics noted in the investigation.

[ • A list of the variables that the authors of the investigation described or attempted to control can help reviewers make judgments about internal validity and freedom error.

  • In group studies or in case studies with several Ps, it is sufficient to provide means/averages for each of the characteristics.]

CONTROLLED CHARACTERISTICS                                                     

  • age: [For example: Stuttering group mean = 25.7 years; NonStuttering group mean = 26.2 years]
  • gender:
  • cognitive skills:
  • expressive language:
  • receptive language:
  • MLU:
  • SES:
  • educational level of clients:
  • educational level of parents:
  • age at referral:
  • Physical, motor, emotional skill:
  • Hearing:
  • Vision:
  • Other (List):

DESCRIBED                       

  • age:
  • gender:
  • cognitive skills:
  • expressive language:
  • receptive language:
  • MLU:
  • SES:
  • educational level of clients:
  • educational level of parents:
  • age at referral:
  • Physical, motor, emotional skill:
  • Hearing:
  • Vision:
  • Other (List):

 

4c   Were the groups similar before intervention began?

Yes ___     No ___     Unclear ____   Not Applicable ______.

 

[ • Reviewers should note if there are grave imbalances in the groups’ characteristics.

  • If there are imbalances, reviewers should determine if investigators made statistical adjustments as a correction in the analysis portion of the report.]

                                                         

4d Were the communication problems adequately described?

Yes ___     No ___     Unclear ____

  • disorder type: (List)

[ • Provide the diagnosis.

  • Additional descriptive information about the communication skills not provided in 4b should be listed here.]
  • functional level: [ If this information is provided in the investigation, summarize the participants’ level of functioning.]
  • other (list): [ Provide any additional information about communication skills that were not noted in 4b or in the other portions of 4d.]

 

 

  1. What were the different conditions for this research?

                                                                                                             

5a Subject (Classification) Groups?

Yes ___     No ___     Unclear ____

If yes, list:

[ • Example: Specific Language Impairment (SLI), Language Age Match to SLI Group, Chronological Age Match to SLI Group]

                                                               

5b Experimental Conditions?     

Yes ____     No _____     Unclear ____

If yes, list conditions, and in parentheses, levels within each condition:

[ • For example: Comprehension of Prosodic Affect (prosody only, prosody plus speech)]

 

5c Criterion/Descriptive Conditions?

Yes ___     No ___     Unclear ____

If yes, list:

[ • For example: accuracy, response time to task]

 

  1. Were the groups controlled acceptably?

Yes ___     No ___     Unclear ___     Not Applicable ____

[ • This is a judgment.

  • Reviewers should at least consider responses to items 4 and 5 in their decisions.]

 

 

  1. Were dependent measures appropriate and meaningful?

Yes ___     No ___     Unclear ___     Not Applicable ____

[ • Reviewers should examine all the dependent/outcome measures.

  • This will help them determine (1) if the outcome are reasonable and (2) if they are concerned with reasonably important behaviors.]

 

                                                                                                             

7a List outcome (dependent) measures:

[ • List the targeted outcomes below.

  • Reviewers should add outcome numbers as appropriate.
  • These numbers should be kept constant throughout the remaining items on the questionnaire.]

                                                                                                             

  • OUTCOME #1:
  • OUTCOME #2:
  • OUTCOME #3:

(add additional outcomes as appropriate)

7b List the dependent measures (using outcome numbers from item 7a) that are subjective:

[ • Subjective measures are those that require judgment on the part of the analyzer.

  • The perceptual judgment of accuracy of stress by a rater/analyzer is an example of a subjective measure.]

 

 

7c List the dependent/ outcome measures that are objective:

[• Objective measures involve minimal judgment on the part of the rater.

  • Physiological measurements are examples of objective measures.
  • Reviewers should use the outcome numbers from item 7a.]

 

                                         

  1. Were reliability measures provided?

[ • Reliability measures are important for subjective measures but they are appropriate for objective measures as well.]

                                                                                                            

8a Interobserver for analyzers? Yes ___     No ___     Unclear ____

  • If yes, list interobserver reliability of each outcome/variable:

 

[ • Interobserver reliability involves the comparison of scores for an outcome measure by at least two raters. The data may be generated by the investigators or be reported from pre-existing sources.]

 

  • OUTCOME #1:
  • OUTCOME #2:
  • OUTCOME #3:

(add additional outcomes as appropriate)

 

 

8b Intraobserver for analyzers?   Yes ___     No ___     Unclear ____

  • If yes, list intraobserver reliability of each outcome/variable:

 

[ • Intraobserver reliability involves the comparison of scores for an outcome measure by a single rater on, at least, two separate occasions.

  • The data may be generated by the investigators or be reported from pre-existing sources.]
  • OUTCOME #1:
  • OUTCOME #2:
  • OUTCOME #3:

 

(add additional outcomes numbers as appropriate)

 

8c Treatment or test administration fidelity for investigators?

Yes ___     No ___     Unclear ____

  • If yes, describe:

 

[ • Treatment or test administration fidelity is concerned with evidence describing how closely treatment or assessment procedures were adhered to during the investigation.

  • Often this achieved by reviewing videos of randomly selected sessions and noting if the administrator included certain important components in the therapy/testing session.]

 

 

  1. Description of design: (briefly describe)

 

 

 

  1. What were the results of the statistical (inferential) testing?

 

10a List comparisons that are significant (e.g., p ≤ 0.05). For each outcome, list any significant difference among comparisons. Note if you do not use ≤ 0.05 as your criterion.

 

  • OUTCOME #1:
  • OUTCOME #2:
  • OUTCOME #3:

(add additional outcomes as appropriate)

 

10b What was the statistical test used to determine significance? Place xxx after all inferential statistical tests used in the investigation:

  • t-test:
  • ANOVA:
  • MANOVA:
  • Tukey:
  • Mann-Whitney U:
  • Wilcoxan:
  • Other: (List the test and the associated outcomes)

10c Were effect sizes provided?

Yes ____ No____

If yes, provide data and interpretation for any outcome with an effect size.

Interpretation Hints:

Cohen’s (1988) guidelines for interpreting standardized mean difference: d = .20 small treatment effect; d = .50 moderate treatment effect; d = .80 large treatment effect

– for partial eta squared: .01 small, .06 is medium, and .14 is large.

  • OUTCOME #1:
  • OUTCOME #2:
  • OUTCOME #3:

(add additional outcomes as appropriate)

 

10d Were confidence interval (CI) provided? Yes ___         No ____

 

[• CI signifies the certainty that a value represents the true value.

  • For example, the typical CI of 95% reflects a range of values within which the true value lies in 95 out of 100 cases.
  • The upper and lower boundaries of the CI are called confidence limits and they can be used to determine important insights such as (1) the adequacy of the number of Ps in the study, (2) if a study in which one condition has been judged to be superior to another (positive study) needs to be viewed cautiously, and (3) if a study in which one has NOT been judged to be superior to another (negative study) has any potential clinical utility.]

 

10e If the answer to 10d was yes, list the dependent measure (from item 7a) associated with the reported confidence interval (CI):

  • 98% CI:
  • 95% CI:
  • 90%
  • Other (List CI%):

 

 

  1. Summary of correlational results:

For any outcomes associated with correlational statistics, list the following:

  • The name of the correlational test
  • The correlational data
  • The interpretation of the correlation

 

[ – For example: OUTCOME #23: Pearson, r = 0.96 (prosody comprehension score and severity of aprosodia, strong correlation]

 

  • OUTCOME #1:
  • OUTCOME #2:
  • OUTCOME #3:

(add other outcomes as appropriate)

 

 

 

  1. Summary of descriptive results: Qualitative research

DIRECTIONS:

– Complete this item only if the investigation was solely or primarily Qualitative in nature.

– List the finding associated with each of the outcomes listed in item 7a.

  • OUTCOME #1:
  • OUTCOME #2:
  • OUTCOME #3:

(add other outcomes as appropriate)

 

  1. Brief summary of clinically relevant results:

 

[ • Reviewers should describe findings that have application to assessment, intervention, or the nature of prosody.]

 

ASSIGNED OVERALLGRADE FOR QUALITY OF EXTERNAL EVIDENCE: _____

 

[ • Provide an overall letter grade for the publication in question.

  • Reviewers should consider the level of evidence, the strength and weaknesses of the methods, the data analysis, and interpretation of the results.
  • This is a subjective measure.
  • See “Letter Grades” in “Definitions and Terms” (on the Dashboard) for a guide to selection of a grade.]

 

 

 

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s

%d bloggers like this: