Conducting Assessment & Evaluation of Systems


A needs assessment and a formative and summative evaluation process are intertwined components used in one way or another in every single one of the courses I’ve taken. Basically a needs assessment is analysis of a system through a variety of data collection techniques identifying the optimal or desired performance and what the current or actual performance condition is. Through this process you determine if there is a performance gap and identify ways to address it. The formative evaluative process is reviewing and evaluating repeatedly a system during its developmental phase and once implemented you evaluate its outcome or result effectiveness via a summative evaluation. Many times I’ve found myself looking back to the resources and textbooks used in these courses as I completed my program.

Summary: My individual based needs assessment project focused on a performance problem. I evaluated the completion rates for an entire instructional period curriculum of veterinary professional students to determine the discrepancies between the number of requests and number of completed evaluations.

Project Rationale: When needing to identify a system or process to evaluate for this class, instructor and course evaluations came to mind. As administrator of the CVM academic management systems I knew evaluation participation by the professional students had been historically low resulting in insufficient response rates to adequately assess and provide statistical analysis of instructors, each individual course structure and the professional curriculum as a whole. Through interviews, observations and review of extant data a performance gap was easily confirmed. It was also clear that lengthy evaluations could be re‐designed and shortened to pertinent questions. In addition, more involvement from instructors, specifically course coordinators would be beneficial by seeking student participation through encouragement, reminders, in‐class time and point incentives. All of which were included in my recommendations section of my needs assessment report in identifying solutions to closing the performance gap.

Artifact: An Evaluation of the Evaluation System

Project Reflection: The needs assessment process was new to me. Through this course I learned about performance analysis and how to design and develop qualitative protocol instruments and collection methods to gather data for assessing optimals, actuals, causes, feelings and potential solutions towards improved performance. I also learned to identify how these purposes can be categorized by typology from attitudes, behavior, beliefs, knowledge and skills. Learning these skills helped me to gather the necessary information for my needs assessment by developing my survey questionnaire, interview and observational protocols. I enjoyed the problem analysis process, identifying and exploring how to use the learned techniques and tools making those concepts much more relevant and meaningful to me as I was learning. This course provided a foundation to help identify which tools (i.e. interviews, observations, groups, surveys) would be most beneficial in different circumstances. I’ve used these tools and strategies over and over again in several of my classes - specifically in courses under the design component where I was designing or evaluating learning environments. I also referred back to various tools and techniques introduced in this course during my Formative and Summative Evaluation course. There is definitely a benefit of taking this course early in the program.

Summary: Conducted as a group collaborative project, we evaluated the effectiveness of a world history course offered through Mizzou K-12 Online through the lens of a formative and summative evaluative process.

Project Rationale: The course content was designed around evaluating a learning system both formatively (intended to improve) and summatively (intended to estimate effectiveness and worth). We were provided with a laundry list of potential clients and after preferencing those options, we were divided into teams. This course was broken into three main sections: first we learned to understand evaluations by distinguishing between the various types, their functions, and all the pieces that make up the evaluative process; the second section focused on guiding us through the planning and designing an evaluation plan; finally the third section of the course provided us with experience implementing and reporting a formative evaluation based on the previous plan we detailed. While we didn’t have the time to implement the summative evaluation element, we did provide recommendations based on the implemented formative portion to our clients.

Artifact: Mizzou K-12 Online

Project Reflection: I’ve never participated in a structured evaluation of instruction in this manner. Throughout this collaborative effort, my teammates worked together well as we produced the entire report. We agreed upon a division of labor with each of us taking responsibility for specific areas of the report. The areas I specifically addressed were:

  • Audiences/Stakeholders - Identification of the various stakeholders having both indirect and direct interests in the evaluation results.
  • Decisions - Identification of the anticipated decisions influenced by the evaluation, considering both positive and negative impacts.
  • Questions - Composing clear and concise questions utilized through the evaluation purpose and data collection methods.
  • Logistics - Outlining responsibilities for the implementation, analysis and reporting details for the summative evaluation.
  • Timeline - Delineating the schedule for the summative implementation, analysis and reporting.
The entire group provided materials to the data instrumentation section by creating the various needed survey and interview protocols. I found the process was very eye-opening and applicably relevant to numerous professional fields. This course was the culmination of other courses and concepts I had already taken in the program. It helped demonstrate how those concepts could be holistically used in a singular comprehensive project.

Summary: The scope of my individual based project focused on providing a clinical schedule for an incoming third year class composed of 114 veterinary students. That clinical schedule helps support students through their required hospital rotations and must balance the required rotation with each clinical service need in the veterinary health center.

Project Rationale: We were to identify an authentic project and apply the phases of project management cycle: definition, planning, control, execution, and closure. The first step of definition consisted of: Why it’s being undertaken, How it fits within the organization, What will it accomplish, Boundary identification, Who's involved, and How success is measured. The next step is planning which consisted of a work breakdown structure-WBS (identifying the deliverables then listing the steps to obtain those deliverables into smaller elements) and a budget aligning with the WBS. The control step is where you review your planning and establish various early preventative warning systems, detection of possible delays or variances and create a plan of action to keep the project on schedule and within scope. The execution phase is where communication is key with the various stakeholders through various tools: email, guides, meetings, group discussions to ensure information is filtered out regarding the process, provide any clarification and updates. Closure is the final step and consists of the following checklist: gaining client acceptance, transition of deliverables, finalizing contract obligations, reviewing lessons learned, updating organizations information repository, finalize financials, closeout accounts, update resource schedules, conduct performance evaluations and marketing accomplishments.

Artifact: Class 2018 Clinical Scheduling

Project Reflection: Evaluating the clinical schedule through a formal project management lens was eye opening to me. It demonstrated I was already applying many formal project management aspects though not as thoroughly or structured. The two steps that benefited me the most were the Planning and Control steps. Though planning the WBS, which was completely new to me, the breakdown outlined each needed element and responsibility delegation for my monitoring. After planning, the control step was monitoring the WBS outline which helped identify possible delays and variances and if detected to create a plan of action in the event they occurred. This approach allows me to be proactive instead of reactive. The entire process was fascinating and incredibly instructive to the point I now have an interest in pursuing a Project Management Professional (PMP) Certification through the Project Management Institute (PMI) in the very near future.

Design


The design competency of the program revolves around designing learning environments, including systems for direct instruction, constructivist learning, collaborative work, and performance support. These courses were challenging and offered me the opportunity to learn how instruction could be improved through technology.

Summary: This group collaborative project designed and created instructional strategies for those instructors needing to transition their currently existing courses from Blackboard to Canvas. We structured our process using the Backwards Design Model by analyzing the final learning objective to determine the best way of assisting the learner to achieve the outcome.

Project Rationale: Our challenge was to design an instructional training program for adult learners. We outlined developmental plans and the needed instruments to move forward with implementation. We designed a systematic needs assessment procedure for our given learning system and audience. We identified the learner and contextual variables, described the task analysis procedures for specifying content and teaching procedural skills. We designed both formative and summative evaluation procedures by developing instruments and protocols for both. While the program was not actually implemented it was organized as if it would be necessitating the inclusion of project sample materials.

Artifact: Transitioning Strategies: Blackboard to Canvas

Project Reflection: Prior to this course, the entire Instructional System Design (ISD) process seemed to be elusive. Having each step broken up into conceptually manageable chunks or skill sets helped me develop new skill sets and abilities I can confidently use. ISD incorporates needs assessment, task/procedural analysis paired with formative and summative evaluations which are all complete and in-depth processes themselves. Having had previous course and professional experience with each of those individual arenas aided my design aspects within the ISD project. My assigned responsibilities within the project were:

  • Project Point Person - I assisted with coordinating the group ensuring our timeline stayed on track to meet our due date and collated the entire project for final submission ensuring all pieces were included.
  • Broad Goals and Big Ideas - This set the tone for the project and provided the overall goal of the project and the high-level learning purpose objectives.
  • Learning Objectives - These are the building blocks that are used to accomplish or met the broader purpose objectives.
  • Summative Evaluation - I established the evaluation protocol to follow if the plan were carried out.
  • Materials for Training Program or Learning System - I designed and developed several instructional pieces from a web-page mock-up to a video demonstration to step-by-step guides that could be implemented if this project were to go live.
After that experience I can certainly comprehend the reasons instructional designers typically work in teams, especially when large projects are involved. Working in a team to divide the components was beneficial, not only to complete work in a timely manner but also provided opportunity for different points of view, experiences and opinions adding to and enhancing the project.

Summary: My synchronous instructional session (SIS) was an independent project I designed, developed and implemented to provide direct remediation instruction targeted to undergraduate college students lacking skills using MS PowerPoint in a fictional online course.

Project Rationale: Through this course we investigated the various ways to utilize instructional strategies and technologies to support a variety of learners, methods of assessing them, how to provide inclusiveness when teaching online through communication tools and course management systems. We engaged in synchronous and asynchronous instructional sessions, explored the various uses of discussion boards and through various technologies to support collaborative group work. Two of the main projects we participated in were a synchronous instructional session and a collaborative team class wiki assignment. Both of these allowed us to model behavior by practicing skills we’d like to incorporate into our own online courses as instructors.

Artifact: Synchronous Instructional Session Part I and Part II

Project Reflection: My SIS project consisted of two parts. The first part consisted of planning and organization. This included identifying the goals of the session; planning for logistics (i.e. learner characteristics, material preparation, software needed, student requirement and time needed); providing an activity outline and procedural steps; and ways of assessing student learning and outcomes of the objectives through verification and engagement. Once the plan was implemented, the second part of the process was to document the session, provide a summary feedback from the students participating in the session, and provide a reflection of the entire process identifying successes, challenges and ways to improve. This was my first instance of creating a lesson plan and as the instructor implementing that plan for an actual audience in a synchronous session. I appreciated the opportunity to provide a session that fit well with my learning style of concrete-active learning by using it to demonstrate step-by-step use of a software program.

Summary: Through a constructivist learning process approach, I created a course site and several of the activities included using technology to teach concepts to accomplish meaningful learning outcomes. The pieces I selected were developed using the Absorb-Do-Connect method and assessment of that activity through outlining a rubric.

Project Rationale: Online Learning can be similar to face-to-face learning in some aspects. One of the immediate differences is you may never see the student you are teaching. This course helps address “How can you adequately teach someone when you don’t have any visual or audible clues from the person during discussions and or lectures?”, which is where this class steps in. Through a variety of different methods and elements you can develop online learning modules, lessons and courses. Some of the explored strategies were ways of presenting and designing lesson (i.e. enabling) objectives that meet the course learning (i.e. terminal) objectives. One of the strategies we explored was the Absorb-Do-Connect method. This method (introduced by William Horton) delineates a path where you begin with allowing the learner to acquire the content through reading, watching videos, etc. (i.e. Absorb) and then allow the learner to practice (i.e. Do) using the concepts they acquired during the Absorb phase. The Connect activity allows for a higher order or categorization associated with Bloom’s Taxonomy where the the learner will be applying those skills, strategies and techniques they acquired during the first two phases through a variety of activities. As you develop these learning strategies you need to also ensure you adequately assess those learning strategies.

Artifact: Connect Activity and Assessment Rubric

Project Reflection: I really enjoyed this class. It paired using a Learning Management System (LMS) to provide ordered, sequential information with development of tangible lessons. I created a course from the ground up that nurtured student learning through my course framework and selection of lessons that addressed the learning outcomes and objectives of the course. This course offered a look into the variety of ways to build an effective learning community. Following a path of teaching online means being cognizant of different learning styles and being able to apply various online learning strategies and techniques by providing several different assessment tools into a course through discussion boards (i.e. course and smaller groups), synchronous (i.e. Blackboard Collaborate) and asynchronous (i.e. VoiceThread) programs. Even looking outside the current LMS being used to incorporate other meaningful ways to use technologies (i.e. PBWorks wiki pages, Prezi, Google Docs/Sheets/Slides, etc.) to allow students to work collaboratively or independently providing for more engaging and meaningful experiences of using technologies to learn.

Development


The developmental competency of the program is focused on leveraging application programs to create and develop learning systems from the ground up to deliver content. Understanding programming principles and coding was beneficial and aided in the course projects I developed. I completed this competency early in my program since I seem to be proficient with it and enjoy coding detail.

Summary: This group collaborative project designed and developed a mobile based application with the purpose of providing users with an educational learning tool. Our app provides users with quick strategy concepts to determine or estimate the size of several geometric shapes easily through simple techniques. It focuses on three regular geometric shapes: Quadrilaterals, Triangles and Circles.

Project Rationale: The group project was a multi-step team activity that included:

  1. A design plan: We provided a high-level description and purpose of our project, including our target audience and a clickstream (to visually demonstrate the clickable navigational options); a flowchart diagram (to visually demonstrate the different decision and options available); and a wireframe diagram (to visually demonstrate the actual navigation, menus and submenu layout).
  2. Project Management: Our plan included the project milestones development and checkpoints for development broken down by activity, due date and individual assigned. Our roles were identified in detail with my contribution being: 1. Work with the group to develop theme/design layout (perform individual research and share with group to come to consensus on final theme); 2. Work on design of home/landing page; 3. Build shell or framework for specific sections of the app (i.e. quadrilaterals, circles); and 4. Apply the formula to each of the detail pages.
  3. First Stage Web App Development:This first draft was used for our first round of peer review and usability testings.
  4. Final Web App Development: Our app was in the final state and submitted for final usability testing and peer evaluations.
  5. Final Web App Evaluation: This process actually allowed us to address any issues discovered through this process of our final high fidelity prototype. We were able to make some suggested changes and enhancements.

Artifact: Thinking Inside the Box

Project Reflection: Working through the master’s program, this was one of the first collaborative group projects I had worked in. Even though our group was small (just three of us) we were able to work together to devise a development schedule and division of labor plan. We had weekly meetings to discuss our progress and we worked very well together in all aspects. Each group member brought different skills, ideas and perspectives to the project which provided for an engaging and fruitful experience. At this point in my program I had created websites and was familiar with advanced XHTML coding, CSS and JavaScripting, but I had never done anything specifically for mobile device development. We based our development in jQuery which for me was fun to work in and I learned a lot developing a new coding skill. I was responsible for the majority of the code development creating the structure of the pages, the style, navigation and interactivity.

Summary: My individual based flash authoring project was inspired by my toddler son. Through the facilitation of an adult caregiver, it utilizes flash based interactions which center on introducing basic developmental concepts for children ages 5 and under.

Project Rationale: The purpose of our final project was to create an interactive animation to teach a concept or skill. The key elements incorporated and how I applied them into the project were:

  • Input text - I used this to customize the entire project to reference the user by name providing personalization.
  • Basic Animation - I created a variety of visually interesting objects associated with colors, shapes, numbers, etc.
  • Advanced Interaction - Through action scripting I allowed the learners to control animations, create free play structures and play matching and counting games.
  • Audio - was incorporated through out, either via a customizable soothing background soundtrack accompaniment, via correct and incorrect notifications or animal identification sounds.
  • Interactive Design Features and Navigation - the animation program was created to allow at any point the activity focus could be changed realizing that the target age group could possibly be disinterested if the program was linear.
The project was comprised of an iterative process having design, development, usability testing with the target age group and peer evaluation to validate the project.

Artifact: Activity Zone

Project Reflection: Developing with Flash was so very specific. If you had one coding element out of place you were toast. Even though it was tedious at times I thoroughly enjoyed that type of developmental work and creating the program. My project idea was based on my son since I was focused on providing him with a variety of different stimuli through various developmental stages. The advanced scripting allowed me to deliver various concepts to reinforce and aid the learning and recognition process through user engaged interactions.

While each student worked on their own individual projects, the course was designed for students to build skills and confidence using the software through mini projects. Those mini-projects were the subjects of forum discussions and classmates learned from each other through feedback and collaborative problem solving. These learned lessons were key and applied towards the final project. The usability and peer evaluations definitely helped shape and make my end submission a better, stronger and more effective project.

Summary: My individual based web project allowed me to showcase advanced web design using advanced programming skills and web interactivity through cascading style sheets (CSS) and Java scripting. It documents the the simple approach to creating fresh and natural baby food. My informational web site provides demonstrations and step by step instructions including equipment needs.

Project Rationale: This course delved into enhancing and building upon your HTML programming skills focusing on developing CSS to improve design and provide interactivity through JavaScripting. Along with development of these elements, we explored importance of the components of visual layout and following design principles. There were four main stages to our project:

  1. Development of a wireframe mock-up. Through this step I provided a description of the functional elements, the site purpose and intended audience. I created as slide deck in MS PowerPoint to illustrate design layout and explain the interactivity on each page.
  2. Peer review with constructive feedback where classmates provided instrumental suggestions about the design, interactivity and offered solutions.
  3. Usability testing and report. I sought several individuals who were in my intended audience and observed as they investigated my site.
  4. The final product. This was a culmination of all the individual stages making my final project a much better product through storyboard wireframing design followed by an evaluative process utilizing peer and usability testing.

Artifact: CUBED

Project Reflection: To strengthen my HTML coding experience I decided to first take the introductory web course to refresh my skills. The structure of the course through the mini-projects allowed me to strengthen my basic coding foundation and allowed me to focus on newer concepts of XHTML, CSS and JavaScript developing those skills before applying them in my final website. This was also my first experience holding a usability test. I found it really interesting to observe my reviewers, especially when they didn’t understand things I thought were crystal clear and or requested additional information. This really solidified how usability testing aids in the design process and based on participant feedback you can strengthen your project leading to a substantially stronger end product. Having taken this course and developing these skill sets has aided with a variety of courses I’ve taken through this program. This is definitely a course that others in the program should not only take, but should take very early in their program.