A Psychometric Evaluation of Skill Clusters and Practices Used by Highly Effective Executive Presenters

The present paper is an evaluation of those skill clusters and practices that are associated with superior ratings of executive presentation skills. It focuses on improving executive presentation skills. It explains the basics of oral presentation skills in general that apply across domains including that of business and management. Six sets practices were found to be characteristic of effective executive business presentation: the preparation, the delivery, and the questions and answers that follow the delivery. As Pappas and Hendricks (2000) and DiStanza, and Legge, 2002 found that an effective executive presentation includes the presenter's mastery and skill in technical content, organization, delivery and relating to the audience. Effective presenters have must be proficient at collecting, selecting, organizing, and illustrating their data, and have to be acutely aware of the purpose of their presentation, and the needs and interests of the audience. So, what distinguishes an executive presentation from other forms of oral communication is the context (environment), the content and the audience. The ingredients of an effective executive presentation are more or less the same as those of any other types of face-to-face presentations. Hence, although this article is focused on helping executives develop and more effectively use their oral presentation skills and practices, our framework can be of used by others who want to be effective public speakers.

view presentations as the art of persuasion. In Weissman's (2003) opinion, the goal of a presentation is to connect with the audience and win them.
Of all the books and articles mentioned above, only some (Gaulke, 1996, Witherspoon andWhite, 1997;Pfarrer, 1998;Rotondo and Rotondo, 2001;Sampson, 2003;Stevenson, 2003;and Tisdale, 2005) have been written for business people in view, and Villata's (2003) has been written with medical presenters in mind. One of the goals of the present study was to determine if there were constructs and measures of the presentation practices and skills used by highly effective managers and executives. In our review of over 500 research studies conducted between 1960 and 2011, there were numerous anecdotal and conceptual papers about the characteristics of effective executive presentations. However, we failed to identify any statistical studies that focused specifically on psychometrically valid or reliable measures of executive presentation skills and practices.

Specific Studies of Presentation Skills and Practices in Educational Settings
In general, the literature focuses on three areas: a) studies and research on basic oral presentations skills and competencies or studies that focus on understanding or evaluating specific aspect of the skill, (b) literature that is basically process-oriented, nature-oriented, or impact-oriented, and (c) literature written for general presenters especially for business executives and managers. Expectedly, all books on the topic profess that if you gain some good advice about how to make presentations, you can overcome your fears and become more effective. The following is a review of effective practices used by student presenters:

1) Getting Audience to Participate--Encouraging questions:
Asking and answering questions is an important mechanism for learning and a central orally-based activity in student teaching sessions. However, many students are reluctant to either pose or respond to questions orally in front of a whole class, preferring to ask 'private' questions after class, use discussion boards and/or question boxes. With this issue in consideration, through a research project conducted collaboratively between Brunei University and Universidade de Aveiro, Portugal, Watts (2007) sought to: a) "Explore learners' 'questioning styles' and approaches to asking questions; b) Develop 'teaching for questioning' to encourage question asking and to build this into the fabric of sessions and course provision." Watts (2007) offered explanations as to why students don't ask questions, which can be summarized as follows: • learners do have questions but often avoid asking them • they fail to be stimulated to ask questions • "conceptual avoidance (the Ostrich Principal) (the Construction of Ignorance)" The design of teaching and learning can limit and inhibit student questions • they may experience feelings of social exposure and vulnerability • the nature of the subject area can impact on question asking Based on this research project's findings Watts (2007) drew a number of conclusions: • "it is clearly possible to create a 'questioning environment' where asking questions (and receiving answers) becomes an integral part of everyday transactions between teachers and students; • different strategies, such as group based mini projects, can promote questioning during sessions and enhance, more than others, the students' 'spirit of enquiry' • teacher centered strategies, such as standard lectures, do not easily motivate students to question during the class; • students need time and encouragement to ask higher level (higher quality) questions; • it is clearly possible to assess question asking and answering against explicit criteria • students need explicit formative guidance on the quality of their questions during routine teaching sessions • students need explicit guidance and practice in the writing of questions for formative assessment • to do so is to encourage the asking of higher level, higher quality questions which reflect conceptual clarity and critical thinking" ISSN 1923-4023 E-ISSN 1923 2) Identify and state the purposelobjective(s) of the presentation: Like any academic project, before conducting oral presentations students must prepare and plan for it. Study guides and skill packs encourage students to do this. Young (1998) and Drew and Bingham (2001), for example, emphasize the need to identify the objective of the presentation, allowing reflection on its purpose, aim and eventual outcome. If working in a group, at this point it may be useful to negotiate and allocate responsibilities between group members (Drew and Bingham, 2001). Next, "the students must be sure that they are familiar with the background of the information they discuss ..." (Eisen, 1998:4). In order to ensure students are familiar with the background and any current issues concerning the topic of their presentation, they are encouraged to carry out research by using textbooks, TV/films on the subject, the internet and other media (Drew and Bingham, 2001;Eisen, 1998).
3) Read--Consider your audience: Levin and Topping (2006) suggest that consideration of the audience is vital when preparing for an oral presentation. They provide five key questions which aim to guide students on the content of their presentation, depending on their audience: 1. Who will be the audience, and why will they be there? 5. What 'protocols' will you be expected to follow? (Levin and Topping, 2006: 36) 4) Plan--Organize the structure: Once these questions have been answered, students are encouraged to structure their presentations in a clear and coherent manner (Cottrell, 1999;Bryan, 2001;McCarthy and Hatcher, 2002;Levin and Topping, 2006). Blundel (2004) introduces the concept of rhetoric and identifies the ingredients that make a presentation persuasive and convincing; ethos, logos and pathos. Here, the use of sign posting is recommended, as it should "help the audience to know 'where they are' from moment to moment, and 'where they are going"' (Levin and Topping, 2006: 60). Levin and Topping quote a popular method which simplifies the process that a successful presentation must adopt: 1. Tell them what you're going to tell them.
Then tell them.

5) Practicing--Rehearsing:
Prior to the actual 'performance' rehearsing the presentation is advised (Turk, 1985;Hatcher, 2002 andLevin andTopping, 2006). McCarthy and Hatcher (2002) describe the usefulness of recording the presentation in an audio format, correcting mistakes or amending material or behavior before video recording it in order to further critically assess one's own performance. Turk (1985), on the other hand, highlights the advantage of using a close personal friend to 'try out' the presentation, as they are more likely to be honest and correct in their evaluation than the presenter will be of themselves. Rehearsing the presentation provides a good opportunity to assess and practice timekeeping skills, effective body language, such as eye contact and hand gestures, voice projection (or the lack of) and content (Young, 1998;McCarthy and Hatcher, 2002, Murray, 2003and Levin and Topping, 2006. Levin and Topping (2006) offer techniques for developing the voice, the presenters' most valuable tool, including enunciation exercises and ideas for experimenting with different tones. Bryan (2001) offers useful tips in relation to voice control, pausing and repetition and McCarthy and Hatcher (2002) also provide their readers with questions concerning volume, emphasis, rhythm, pace, pitch, etc. which speakers could consider and reflect upon before or even during their presentation.

6) Projecting Self-Confidence--Overcoming performance anxiety:
Much of the literature and self-help books dealing with presentations have been generated by the fear of public speaking. This problem is so pervasive that it has produced hundreds of books and articles on the topic. As recognized earlier in this literature review, nervousness and stress may hinder one's ability to present an oral presentation to the best of their ability. For this reason, many writers have stressed the importance of practicing relaxation techniques, or adopting other approaches to ensure presenters are calm and focused before presenting (Turk, 1985;Bryan, 2001;2002;Levin and Topping, 2006). Bryan (2001) for example, provides guidelines for students to follow before presenting, which include visualizing yourself performing well, taking deep breaths, clearing your mind as a way of Relaxing and releasing tension through isometric exercises (Bryan, 2001(Bryan, : 1145. Levin and Topping (2006) recommend techniques which are more orientated towards realizing a positive mental attitude, focusing on self appreciation and gaining a positive outlook towards the work you have accomplished, while McCarthy and Hatcher (2002: 50142) give details of specific physical exercises to aid relaxation and breathing.

7) Using nonverbal cues and communication:
The effective use of body language during an oral presentation is central to guidance for presenters. Bryan (2001) identifies particular gestures which help convey messages of authority, confidence, and trust, such as open arm gestures and finger movements (to count point 1, 2, 3, for 11 examples). Similarly, McCarthy and Hatcher, dedicate a verbal language' (2002: 107134), discussing nonverbal communication behavior which portray confidence, such as posture. Turk also dedicates a chapter to 'Nonverbal communication ' (1985: 145166), and advises consideration of factors such as dress signals, posture, positioning, movement, maintaining eye contact with the audience and hand gestures.

8) Handling Questions and Objections:
Finally, presenters are advised to include a question and answer session at the end of their presentation to help clarify points, as well as providing the audience with a chance to test the knowledge of the speaker (Turk, 1985;Young, 1998;McCarthy and Hatcher, 2002;Levin and Topping, 2006). While Levin and Topping (2006) give their readers' advice on how best to manage the session, McCarthy and Hatcher (2002) give tips on how to handle questions when they are asked.
The Use of Self and Peer Assessments: In the present study, several methods were used to rate the presenters. These included peer, audience, expert and self-ratings of presentation skills, behaviors and effectiveness. There is a growing body of research around the benefits and validation of self and peer assessment. With regard to the assessment of oral presentations skills, Magin and Helmore (2001) found that the reliability of summative assessment could be improved by combining teacher evaluation with the average from multiple peer ratings. Hughes and Large (1993) found that individuals could make reasoned assessment of their peers independently of their own ability. Baud (1989), however, argues that student assessment is too unreliable to count for summative assessment. It is also claimed that if peer assessment is only used formatively students will not take it seriously (Swanson, 1991, cited in Magin andHelmore, 2001).Unfortunately, after reviewing over 500 research studies we were not able to find any studies that related to the use or accuracy or executive, audience, expert or self-ratings of executive presentation skills, behavior or practices.

Academic and Professional Assessment of Communication Skills
It is widely accepted that business management and business educators perceive communication skills as highly valuable to employees and organizations alike. In business organizations, numerous sources have reported that communication skills are critical to career success and a significant contributor to organizational success (Du-Babcock, 2006;Roebuck, 2001;Certo, 2000;Dilenschneider, 1992;Rushkoff, 1999). In academia, research has shown faculty and administrators perceive that communication skills are very important to students' eventual career success (National Association of Colleges and Employers, 2001;Gray, 2010). Maes, Weldy, and Icenogle (1997) found that oral communication was one of the top three competencies needed to succeed in a managerial position. Yet other studies over decades have demonstrated the unsatisfactory oral communication skills of recent graduates (Bolt-Lee & Foster, 2003;Reinsch & Shelby, 1997). Thus, it appears that preparing students' oral communication skills for the managerial workplace has not been highly successful. Several studies reveal that business needs communication skill competency and that business education must be sensitive to and understand the communication skill needs of business. Accordingly, some evidence exists, that there are significant differences between the types of communication skills business considers important and those college instructors and business educators consider important.

Study Objectives
The objective of our study was to identify those behaviors and practices that are associated with self, peer, audience and expert evaluations used by managers and executive who rated as highly effective presenters. As a result of our research, we developed a 35-item executive presentation practices that we refer to as the "Presenter Assessment Inventory", or PSI. This development of this instrument was based on a thorough review of the literature on oral communication skills, effective organizational and business communication practices and extensive interviews with senior executives and experts in management, communication, and behavioral research. We initially identified over 200 item or practices. However, we were able to reduce the number of items to the final set of 35 items that were most frequently and consistently identified as behaviors that were used by outstanding business presenters and presentations. In the present study, we provided executives giving a 90 minute presentation with two types of audience feedback about the practices that were demonstrated in their presentations: 1) the frequency or "how often" they demonstrated or used specific practices or skills and 2) the level of skill or ":effectiveness": that the presenters demonstrated during their presentations. The PSI enables the business presenters to compare their self-perceptions of the skills and practices that they use and the impressions or "impact" they make on the audience with audience assessments of their behavior.

Structure of this Report
Our research is organized into 4 sections: A review of the process and statistical methods that were used to select items that were included in the PSI; 1) A review of the statistical methods that were used to determine the reliability and internal consistency of items or presentation practices and the items associated with several presentation skill clusters; 2) Present a review of the psychometric properties of the PSI clusters including the reliability of the major constructs that serve as the basis for measuring presentation practices; 3) Finally, a set of multivariate regression analyses was used to determine the relative% of variance explained by each of the 5 presentation skill clusters and the overall ratings of the presenter's communication style and effectiveness.

Statistical Methods
We performed a descriptive analysis (frequencies and percentages) of the demographic characteristics of the participants. We conducted factor analysis develop clusters for items showing a high level of commonality. Prior to conducting the factor analysis, we tested the suitability of the presentation skill rating scale using the Kaiser-Meyer-Oklin (KMO) test and the Bartlett's Test of Sphericity. Because the some of the data was skewed toward the higher values, we re-categorized the answer choices into three groups: "poor or fair," "good," and "very good or excellent." We then conducted a series of analyses to evaluate the psychometric properties of the PSI survey questionnaire. We measured internal consistency reliability (Cronbach's alpha) to assess whether all the items were contributing to the measurement of presentation practices, we also measured inter-item gamma scores to examine the level of relationship between the items in the questionnaire and calculated correlation coefficients for items in each cluster. In addition, we calculated the Kruskai-Wallis K Independent Samples test to examine differences in total score across demographic groups. The total score was computed by tabulating the sum of all 35 items to provide a continuous measure of presentation skills. We used SPSS version 13.0 (SPSS, Inc., Chicago, Illinois) for all analyses.

Development of the PSI Research Instrument
Three methods were used to develop the 35 items and 5 clusters of the Presentation Skills Inventory (PSI) questionnaire used in our research. As mentioned previously, an extensive review of behavioral science research studies was used to identify those skills, characteristics and practices used by outstanding business presenters. This review included over 500 technical documents, research reports and evaluations published during the past 30 years. A group of 40 trained content analysis evaluators who were not familiar with the research literature or the content of the questionnaire. The content reviewers analyzed the results of these research studies and initially recorded over 150 items or practices used by effective business, government and industry presenters. After rewording, editing and consolidating these items, 52 presentation practices were used in the research instrument. This set of practices was then evaluated by a second group of reviewers consisting of 35 executives, consultants, psychologists, trainers and highly experienced public speakers. The judgments of this group were based on over 10,000 presentations that they had conducted, attended or evaluated. After each of the items 52 practices were evaluated, a set of 40 were included in the pilot study. Thirty five senior executives used these practices to evaluate the effectiveness of a 90 minute, financial services presentation. After completing their assessments, the practice items were reworded to reduce overlap and improve comprehension. The data gathered during this pilot was analyzed and practice items that either had a low completion rate or very little variance were eliminated from the practices instrument. This resulted in a set ISSN 1923-4023 E-ISSN 1923 of 35 items that we used in our study.

Ratings of Presenter Effectiveness and Use of Presentation Practices
The 35 item Presentation Practices Inventory (PSI) was completed by 250 business executives who attended a 5 hour financial services education program. Several different statistical methods were used including an ANOVA to evaluate the reliability and internal consistency of the practices and presentation clusters used in the PSI. We also evaluated the degree to which the raters using the PSI varied in their ratings of the presenters. Statistics revealed that ratings of "Frequency of Use" of presentation practices ranged from 3.2 to 4.4. Ratings of "Presenter's Skill Level" ranged of 3.7 to 4.5 on a five point Likert Scale. The standard deviation 0.71 to 1.03 indicating a significant variation in ratings of these sets of practices The audience was asked at the end of the questionnaire to rate the Overall Effectiveness of the Presenter. This set of practice ratings ranged from 3.0 to 4.4; the average rating was 3.8/5.0 with a standard deviation of .82. These statistics suggest that there was a significant amount of variation in the audience evaluation of the 35 presentation practices used by the presenters in our study.

Cluster Analyses
Cluster analysis of the ratings of 250 executives who completed the Presentation Practices Inventory revealed 10 presentation practice factors or "clusters. ANOVA was used to explore the correlations between practice ratings and presentation skill factors. There were two categories of factors that demonstrated the best-fit between practices and factors.

a) Frequency of Use Clusters
The first set of questions asked the audience (presentation evaluators) to rate each presenter in terms of how often he/she used the five types of presentation skills, practices and behaviors in his/her interaction with them. The following Frequency of Use Clusters resulted from our analysis of the data: These results (See Table#1) indicate that all 5 of the frequency of use presentation practices had moderate to high correlations between component practices and overall cluster scores. Internal consistency coefficients for Frequency of Use Clusters were high ranging from r=+.81 to .90.

b) Presenter Skill-Level Clusters
A similar pattern was found when we analyzed the 35 questions that focused on the audiences' evaluation of the presenters' skill level as demonstrated by five different presentation skill clusters. The results of our analysis of the 5 Presentation Skill Level Clusters as shown below: 1) Presenters' Skill in Establishing Credibility and Authority consisted of 7 practices were moderately correlated--r ranged from +.30 and +.57. The overall Average internal consistency of items in this cluster= the internal consistency for this cluster RSic1. =+.81 was statistically significant.
2) Maintaining Focus consisted of 6 practices, correlations for these items ranged from +.42 and +.76, The overall internal consistency of items in this cluster= RSic2.=+.82 3) Building a Positive Environment consisted of 9 practices, r ranged from +.27 and +.75, The overall-Average internal consistency of items in this cluster= RSic3.=+.89 4) Delivering a Useful/Effective Presentation consisted of 4 practices, r ranged from +.27 and +.56, The overall-Average internal consistency of items in this cluster= RSic4.=+.76 ISSN 1923-4023 E-ISSN 1923 These results (See Table#2) indicate that all 5 of the presenter skill-level practices had moderate to high correlations between component practices and overall cluster scores. Internal consistency coefficients for Frequency of Use Clusters were high ranging from +.76 to .89.

Results Summary: Frequency of Use Clusters
As Table#3 indicates there was a moderate to high degree of inter-correlation between the five Frequency of Use Clusters were high ranging from r=+.66 to .77. This indicates that there was a positive, significant (+.0001 level of significance between the 5 frequency of use clusters, Cluster 5, Promoting Retention was most highly correlated with the four other frequency clusters (r=+.60 to .77). There was a moderate relationship between Maintaining Focus and Establishing Credibility and Authority (r=+.50) and with Building a Positive Environment (r=+.48). There was also a strong relationship between ratings of Establishing Credibility and Authority and Building a Positive Environment (r=+.70). The correlation between these two clusters and ratings of overall Presenter Effectiveness was r = +.69. One interpretation of these results is that presenters who exert more authority are seen by the audience (evaluators) as Promoting a More Positive Environment and the more likely they are to retain information and/or derive benefit from the presentation. Presenters who establish their Authority and Credibility are seen as more effective presenters.

Results Summary: Presenters' Skill Level Clusters
As table#4 indicates, correlations between the five presentation skill clusters range from +.55 to .75. While, the range of correlations for skill clusters were slightly lower than the frequency of use correlations, the general pattern and direction of these correlations were found to be quite similar. Audience ratings of Promoting Retention were most highly correlated with ratings of Presenter Effectiveness (R=+.76) and skill in Creating a Positive Environment (r=+.73).There was a moderate relationship Creating a Positive Environment and Establishing Authority and Credibility (r=+.73). There was a much lower relationship between the Maintaining Focus MF Cluster and the Establishing Authority and Credibility EAC Cluster (r= +.53). There was also a somewhat lower relationship between each of these 2 clusters (MF and EAC) and overall ratings of Presenter Effectiveness(r=+.50).These findings suggest that presenters who focus on Promoting Retention are rater higher on Presenter Effectiveness as compared to those who place more emphasis on Establishing Authority or Maintaining Focus. On the other hand, presenters who Establish Authority are seen as Creating a more Positive Environment. All of these correlations were positive and significant (p=.0001) indicating a high degree of consistency in audience evaluation of the presenters skills and behaviors.

Relationships between Ratings of Presenter Effectiveness and How Frequently Practices Were Used
As table#5 indicates, as we might expect intuitively, the audience rated presenters who exhibited greater use of presentation practices to be more effective; the reverse pattern was also observed. Correlations between the 5 frequency of clusters and the ratings of the 5 effectiveness clusters were significantly inter-correlated (r=+.79 to.90). The strongest and most significant relationship were between audience ratings of the skills and frequency of practices used by presenters to Create a Positive Environment (r=+.90) and between ratings (skills and frequency) that were used by presenters to Maintain Focus (keeping the audience focused) r=.+89. There was a moderate relationship (r=+.79) in terms of Establishing Control and Authority. Perhaps this suggests that presenters are too controlling are seen as less effective by the audience. However, the overall pattern across all 5 presentation practice clusters strongly suggests that audience perceptions of the presenters' effectiveness were significantly related to how frequently they used the practices.

Relationships between Presentation Practice Ratings and Audience Evaluations of Presenter Effectiveness
Significant relationships were found between audience evaluation of Presenter Effectiveness (skill in dealing with questions, responding to challenges/disagreement and facilitating two-way discussion/interaction with the audience) and audience ratings of the Frequency of Practices across all 5 clusters P=+.001 The strongest relationships were found between this cluster and Managing Conine! (disagreement) r=+70, and Facilitating Audience Discussion-Participation (r= +.68). We also found that there was a weak relationship between Audience Evaluations of the Presenters' ability to Manage Conflict and ratings of the Maintaining Focus cluster (r=+.33). This finding might suggest that "keeping the presentation on track" by failing to address issues and/or ignoring confnct or resistance is not seen positively by the audience. This has important implications; an effective presenter has to be able to "read the audience" and deviate from his/her plan when important issues are raised by the audience. Executive presenters who demonstrated Skill in Establishing Credibility, Creating a Positive Environment and Promoting Retention received much higher evaluations of their effectiveness in Managing Conflict/Resistance. This pattern was not related to the Frequency of Use of practices for these 3 clusters.

Relationships between Audience Ratings of Skill and Frequency Clusters and Audience Evaluation of the Presenters' Overall Effectiveness
The audiences' overall ratings of the presenters' effectiveness was assessed using two different measures. The final question asked the evaluator to give each presenter an Overall Effectiveness Performance Rating (OEPR) during his/her 5-hour presentation. We also measured presenter effectiveness by calculating the weighted average of the 4 different presenter ratings of executive presenters in our study. We refer to this measure of the presenter's effectiveness as the Combined Average Effectiveness Score (CAES). As table 7 indicates, there were strong positive correlations between these two measures (OEPR and CAES) and executive presentation practices. Correlations between overall ranged from r=+.55 to +.89 for Frequency of Practice Use and from +.60 to +.89 for Presenter Effectiveness. There were strong, positive and significant relationships found between the audience's overall effectiveness ratings (OEPR) and the frequency of using the following practices: Building a Positive Environment, r=+.70; Promoting Retention, r=+,69; and Establishing Credibility/Authority, r=+.66. Maintaining focus showed the lowest correlation with Audience OEPR ratings (r-=+.55). As expected, the same trends were in general found between the Combined Audience Performance Scores (CAES). However, ratings of OEPR and Promoting Retention had the highest degree of relationship (r=+.89); all of these findings were significant at the +.001 level).
The frequency of use of practices that focused around Building a Environment were significantly related to CAES, r=+.8; p=+.001. Alternatively, presenter skill in Establishing Credibility and Authority and Building a Positive Environment correlated with CAES (r=+.+76 and r=+.74 respectively. When CAES was used as an indicator of presenter effectiveness, skills associated with Promoting Retention, (r=+.89), Establishing Authority and Building A Positive Environment were highly correlated (r=+.86) p=.0001. Skill in Promoting Retention was more highly related to CAES than OEPR.
Multiple regression analysis was used to determine how much variance in ratings of presenter performance was accounted by each of the presenter practice and frequency of use clusters. Three separate, step-wise MOVA analyses were conducted. The use of step-wi se regression takes into account both the independent contribution or weighting of each cluster score from the score that accounts for the most variance in presenter ratings to the clusters that account for little or no variance. In addition, at each 'step in the analysis, weighted coefficient express the relative or comparative amount of explained variance in ratings attributed to each of the cluster scores.