Dealing with Validity of Pedagogical and Technical Features of Educational Website

This post is intended to inspire researchers who are searching for technique in dealing with validity of study that offers internet-based educational website hosting pedagogical features for students.

The term so called validity aggreed in this post is based on the definition explaining validity as an integrated evaluative judgment of the degree to which empirical evidence and the theoretical rationales support the adequacy and appropriateness of inferences and actions based on test scores or other modes of assessment 1.

Let us discuss the definition by breaking it into keywords. There are 4 points to be noted related to validity, they are: (1) evaluative judgment, (2) empirical evidence, (3) theoretical rationales, and (4) appropriateness of inferences and actions.

Literature defines evaluative judgement as the capability to make decisions about the quality of work 2. In my personal point of view, evaluative judgment refers to decision and conclusion which provide further refinement. The next keyword is empirical evidence which is general term emphasizing the scientific certainty generated through systematic study. The third keyword is theoretical rationale which emphasizes that statements made are based on scientific resources. The last keyword is appropriateness of inferences and actions which emphasizes that any action and measurement adopted/adapted should be appropriate - meaning that it is reserved for and capable of measuring what to be measured.

Expert Judgment/Appraisal

Among various validity models, expert judgment can be opted. There are abundant examples of studies which apply expert judgment or expert appraisal to deal with validity. Research that I mostly understand is, of course, the research I conducted for the completion of my doctoral degree at Universitas Negeri Makassar. I spent 5 years in conducting the research applying (1) preliminary study; (2) prototyping; (3) assessment; and (4) systematic reflection as informed by Plomp (2013) and McKenney & Reeves (2014).

In my doctoral study, I prototyped educational artefact called the web-based academic English module which was exclusively and especially designed and developed for students majoring primary school teacher education study program at Universitas Negeri Makassar. 125 6th semester students participated in the study which lasted for 3 cycles of design-development, implementation, and evaluation.

In dealing with the validity of the product - including the website in terms of technical fuction and pedagogical model applied as instructed on the website - as well as instruments composed to obtain the research data, expert judgment was chosen. Although some journal authors doubt the strength of expert judgment as a tool that informs validity, this still has a power to be the solution for the validity.

In my personal point of view, students dealing with thesis/dissertation are required to deal with instructions addressed by academic advisors. When the proposal is regarded insufficiently meet the criteria, the proposal - which is analogized as students' product - would be rejected or sent back for revision. This process describes the role of expert in appraising a product. Therefore, in dealing with individuals regarded as expert, some points to be ensured are:

  1. Individual regarded as experts should be professional practitioner in the field - proven by legth of experience in the field or certificate;
  2. Individual considered as expert should be recognized by other professional practitioners.

Thing should be noted here is that these points are not standard criteria in selecting experts to be asked to measure any educational product.

In regards with the design and development or the prototyping of website offered as educational product generated through research and development (RnD) and/or design-based research (DBR), researcher should at least address the validity of pedagogical aspect and the technical aspect. Some points to consider are:

  1. Content, material, and activity (pedagogy);
  2. Format and presentation;
  3. Instructional language;
  4. Website technical features.

Pedagogical Features

Pedagogical features simply refer to aspects that your designed and developed educational product offers in dealing with teaching and learning. For the case of website as coursebook/module, it may discuss about conformity with:

  1. Curriculum;
  2. Syllabus;
  3. Lesson plan;
  4. etc.

In developing questionnaire statements or interview items, each of the previously mentioned points can be transformed into several questions which depend on your focus. Here are some examples of statements or items that focus on pedagogical features.

No. Statement / Item 1 2 3 4
1. Material and activity are in accordance with curriculum
2. Material and activity are in accordance with syllabus
3. Material and activity are in accordance with lesson plan
4. Material and activity are realible in achieving course objectives
6. Material and activity suit the students' level
7. Material and activity encourage students to learn

Technical Features

In addressing this point, researcher is suggested to think about the website in technical aspect. Some of the technical aspects of the website are:

  1. Font size, color, type;
  2. Color contrast;
  3. Loading speed;
  4. Clarity of picture/video/audio;
  5. Accessibility;
  6. etc.
No. Statement / Item 1 2 3 4
1. The website applies appropriate font type
2. The website applies appropriate font size
3. The website applies appropriate font color
4. The website applies good color contrast
6. The website applies tolerable loading speed
7. The website is accessible through laptop
8. The website is accessible through smartphone

In dealing with validity, 2 individuals regarded as expert in the field of pedagogy and technology were asked to assess the website as well as the content and material. Their expertise was proven by looking at their theme of publications (journals/books) and their experience in teaching. In their faculty, those individuals were recognized by their fellow lecturers. These individuals can be described as professional lecturers with over 15 years of experience in teaching English. In addition, they concerned on the application of technology in language learning.

To convince the validity, 4 peers who are experienced English lecturers with over 10 years of experience in teaching English with technology were also asked to judge the product through the perspective of (1) content, material, and activity; (2) format and presentation; (3) instructional language; (4) technical features of the website that support the teaching and learning.

Validity Test

In the design and development of the web-based academic English module, validity test was also conducted by an official agency in the university where the study took place which concerned on data validation and analysis was asked to measure the questionnaire items - both questionnaire addressed to lecturers and to participants.

This is the easiest way to deal with validity - that is to know whether measurement applied really measured what to be measured. In dealing with this, a researcher should firstly conduct small-scale field testing and apply the instruments (such as questionnaire, observation checklist, and interview questions) to the participants of small-scale field testing. After that, the data obtain are sent to the recognized agency to be quantitatively processed

  1. Messick, S. (1989). Validity. In R. L. Linn (Ed.), The American Council on Education/Macmillan series on higher education. Educational measurement (pp. 13–103). New York, NY: Macmillan Publishing Co, Inc; American Council on Education. ↩︎

  2. Tai, J., Ajjawi, R., Boud, D. et al. (2018). Developing evaluative judgement: enabling students to make decisions about the quality of work. High Educ 76, 467–481. ↩︎

Content posted on blog may contain subjective interpretation. You are highly suggested to check the reference quoted in every page of the blog to ensure the validity of information provided.

This blog may also contain affiliate links which send me some bucks if you take action on my referrals.