COMPETENCIES

Download artifact (PDF)

About the process
Ideally, needs assessments should be done upon attempting any type of learning system design or development. They may be different scales, but they should still be performed. Needs assessments help one to understand how people work, what they need, what they find motivating, what they understand, and what they want to do. Without some of these very basic ideas of the problems, issues, and ideas--one could easily create a beautiful, usable system that nobody will ever use. And what good is that?

Through the class Needs Assessment for Instructional Design, taught by Dr. Wedman, our goal as learners was to "Learn to analyze learner, instructional and environmental needs using a variety of techniques. Identify the components, advantages, disadvantages, and applications of instructional systems theory and instructional design and development." I performed a needs assessment at my place of employment, Special Education Service Agency, with the permission of my administration.

We learned and carried out the process for conducting a needs assessment in order to better provide instruction. We utilized Dr. Wedman's pyramid for seeing how needs interrelate:

• Tools, Environments, and Processes
• Expectation and Feedback
• Rewards, Recognition, and Incentives
• Performance Capacity
• Motivation and Self-Concept
• Competence: Knowledge and Skills

I found that many items within this pyramid were normally not addressed or were easily looked over; to have this as a guideline helped to find the holes in a situation or practice.

Although it was a bit intimidating to carry out this Needs Assessment, performing it in a real situation with the support of my administration was highly beneficial and rewarding. My workplace asked for my needs assessment upon completion, and after having received it started implementing some of the recommendations in addition to having a professional needs assessment completed the following year.

What I might do differently
I found that it was a bit too intimidating to do a needs assessment with one's own employer. For example, I was performing a needs assessment in the very place where I worked. Although I tried to eliminate bias in my questions, it more than likely was not entirely possible.

In the future, I would try to have another outside party to participate in various aspects to ensure objectivity both in my questioning, my interviewing, as well as my reporting.

How I will use this information in the future
When I had first taken Needs Assessment, I had not entirely connected it with the learning system design and development process. However, after having gone though the program and seeing how all of the pieces fit together, I found I was using my needs assessment skills in the process of design and development. I will most likely this information and the performance pyramid when I am looking at designing performance support systems or performance support instruction and/or training.

However, those are not the only areas where needs assessments apply. Ideally one will want to perform a needs assessment (to varying degrees) before looking into implementing a technology. The reason is that a needs assessment helps one to take actions which are thoughtful, purposeful, meaningful, and useful to those who will be on the receiving end. One can make something very pretty, but if it doesn't fit what people need, it was a waste of time.

Through the class, Formative and Summative Evaluation, taught by Dr. Caplow, our goal as learners was to "use the appropriate skills and techniques to effectively utilize the formative evaluation process to revise instruction, and to plan and conduct a portion of a summative or effectiveness evaluation of an instructional program."

This group project, spanning throughout the entire semester, focused on evaluating online art curricula for the purpose of creating an online art site. This is unusual in the sense that one usually evaluates something already in existence. On one had we were evaluating something already in existence, but were were using that information plus other information in order to inform us of a way to develop a learning system that had high-rated components of that site within this new site.

One of the main things that I found interesting (and difficult) was the more organic form that evaluation can take. The process that we used in this class was highly organic in the sense that the evaluation itself was highly iterative, without any absolute one right way to conduct it. One of the best points that was made in this class was matching the question with the types and form of evaluation. Nothing will be used everywhere at every time, but there is definitely a time and place for certain types of evaluation.

I am currently taking this class now and will have the artifact up within the next few weeks.

About the process
Designing is an intentional process done primarily (and ideally) before the development phase. Not only does this save time and money from redoing development, it also allows one to "think big" or "think out of the box" due to not being as constrained by development when designing based on user needs.

Designing takes into account human-computer interaction, user interface design, and various activity theories (learning theory, how people cooperate, sociology). It is done so because when designing learning environments, the environments are inherently social. By understanding how people communicate, collaborate, work together, learn, and interact with computers we can understand how to better be able to design a useful learning system.

This mockup and development plan of a portfolio assistant was designed in Designing Performance Support Systems, taught by Dr. Laffey. This particular project was a done in a series of modules (some group work, some performed individually) with the purpose of assisting those who are in the Learning Systems Design and Development track more skillfully develop their portfolio.

Electronic performance support systems are computer-based systems which assist users in performing their job, skill, or product. Many times it is used within the work industry to reduce cognitive load, to reduce training time, to provide knowledge as needed or encountered, and to enhance user's knowledge base through an easy-to-use and accessible interface. In this instance, we were designing an assistant to help a student through the process of creating their portfolio.

View Prospectus Document

Prospectus
The prospectus is a general overview and plan about how one is going to proceed. It looks at what has been done, what has been said, expert advice and information, overall ideas for the application, and some steps for proceeding. This was done individually and when we came together as a team we were able to see each of our plans and perspectives.

As a team we probably should have looked over each other's prospectus more carefully to iron out any differences in ideas insofar as the purpose of the application; however, we did not. This did create some bumps along the way and I learned that if one is working in a team, this is an essential thing to clarify right away: the purpose of the application.

View Requirements Document (PDF)

List of Requirements: Beginning
The requirements document was the next step after the prospectus. This particular piece was done at an individual level before we came together as a group. The requirements had taken into mind best practices in performance support systems (as learned in Designing on Both Sides of the Screen), what has been done prior to this to serve as a model, and what issues I thought would be a priority to meet this particular performance issue.

This is a very basic start, but it was helpful to have different individuals come together with various requirements; some we kept, some we threw out, and with others we compromised.

View Interview Report (PDF)

Interviews
In order to get an idea of what the requirements should be as a team, we interviewed both potential users and experts of this portfolio assistant. This is an essential step in order to get perspectives from appropriate users, clients, and experts. Without it, one might be spending time on needless features while other core features might be left out entirely. It's good to do this step before the outline of the design phase.

View Scenario 1View Scenario 2

Developing Scenarios
In order to get a better idea of how users might use our assistant, we developed two scenarios. Scenarios are a way to use storytelling to plan out how users may use a system in a certain context and purpose. They are helpful in revealing some of the more subtle ways users may use systems as well as the individual's motivation and purpose--all before doing any user testing.

Develop Flow Chart of Screens and Possible Features
Below is the second step (or series of steps actually) which took each individual's ideas of what the assistant should do and did the following:

• Looked at the requirements for the LSDD Portfolio
• Discussed the Interviews and information from experts
• Took information in from what has already been done and said (interaction design, experts)
• Discussed what screens might be in the system with accompanying features

What we didn't do, however, was to make sure everyone was on the same page in regards to EXACTLY what the assistant should do and EXACTLY what actions it would be supporting. As a group we went in assuming we had the same understanding, but as we started talking about requirements (core, important, "would be nice"), we realized we had different understandings of what the specific purpose would be. However, by referring back to the interviews, we were easily able to resolve differences.

The requirements document was the next step after the prospectus. This particular piece was done at an individual level before we came together as a group. The requirements had taken into mind best practices in performance support systems (as learned in Designing on Both Sides of the Screen), what has been done prior to this to serve as a model, and what issues I thought would be a priority to meet this particular performance issue.

View Frequency of UseView Priority

Prioritizing Features by Need and Frequency of Use
Not every feature can nor should be incorporated into an application. Although many users may say "I want this and that", although it may be a very nice feature it may not be used often. By looking at both:

• How the feature serves the application's purpose (Need), and
• How often the feature would be used (Frequeny of Use)

It is then easier to prioritize what features should not only be included, but what features should be dominant, easy to get to, on the first page, or left out entirely.

View User Testing (PDF)

Usability studies are evaluations which look at how a person will interact with the system. Through this observation of a user interacting with the system problems and issues can much more easily be revealed. What may not be an error in coding may indeed be an issue with usability (ease of use). In addition, ease of use implies intuitiveness. Just because it is easy for the one who developed it to use does not mean it is easy for a user new to the system to use.

Usability studies are frequently done by videotaping or watching the user perform a set of specified actions. Directions are not given to the user as to how they are to perform, but they are often requested to use a "think aloud" approach in order to let the observer know more about their thought processes as they attempt to perform the task.

Usability studies are an important way to see how real users, not the ones who have been involved in the designing and developing process, will use and think about a system. Usability studies are able to get much more in-depth contextualized subjective information from the users as well.

In this instance, as a group we created a mockup and then did usability testing using a paper version. This is a very good way to assess the usability before doing any type of development. This can also be done using ImageReady or PowerPoint since it may allow a bit more interactivity but essentially it is a similar idea.

View artifact

This Flash presentation shows our mockups as well as our thought processes for using the portfolio assistant. It also presents our ideas and rationale behind navigation, screens, and features. One of the screens is entitled "Infobase"; this is essentially the layers behind the portfolio assistant. In an application, the most accessible features (those defined as "core" as well as "frequently used") should be at the first layer of the application. No drilling or digging down should be necessary. The second layer is the next level, and the least needed and/or least used should be at the third level. Most applications strive to have most of the features within the first two layers. Anything beyond three layers allows people to become confused and lost in your application unless it is designed EXTREMELY well...and that is very difficult to do!

Summary
The major concepts I learned from this course were the following:

• What a performance support system is
• User interface and interaction design
• The process of developing a prototype
• Working as a team in developing a system

It took me a little while to fully understand and comprehend what a performance support system is; however, once I understood it I realized they are all around us. From Turbotax to Basecamp, I realized I was using them all around me. In addition, I had discovered I had been creating performance support "systems" (more like tiny support apps in Filemaker) for a while.

Design has a large basis in interaction design. Especially when dealing with performance support systems, people should not have to focus on the system when they are just trying to "get things done". By focusing on this aspect and the process of involving interviews, scenarios, prioritization of features, and real usability tests, it's amazing how much insight into what works you can get from talking to people and refining again and again.

As I had stated earlier, when working as a group I probably would have communicated better with my team from the very beginning. We had all assumed we knew the purpose for the portfolio assistant. However, some people had wanted to have a feature build a portfolio for the student, whereas I thought that was not only unnecessary but went against the idea of a portfolio. We had discovered this when creating a feature list and ended up resolving our issues with interviews with experts. If I were to do this again, I would resolve any issues about the exact purpose of the system before moving on.

I have found out over time that I am very interested in creating performance support systems for special education teachers. This process has helped me immensely in realizing to what detail one needs to go to in planning and designing a system. I will use this as a guide in the future when attempting a design, as this process takes into account user needs and usability. And that's what performance support is about: helping the user.

Understanding By Design
The main point behind designing direct instruction was to instruct in a manner that facilitates a learner having deeper understanding of a topic, not sole regurgitation of information but one that facilitates learning questioning. By understanding and getting learner misconceptions and getting at key "big questions" and "big ideas" in the topic area, one can better provide quality direct instruction to learners.

My unit was an instructional unit on infant and toddler hearing loss, taught in the online environment "Moodle". Throughout you will see screenshots of the lesson, which was created in Moodle installed at my employer's online learning system.

The units were iterated throughout the semester, with both instructor and peers providing feedback. The very first lesson was provided the most opportunity for iteration and feedback, and thus was the most in-depth of all three lessons that we created.

View Desired Results (PDF)

State Desired Results
One doesn't merely start with what topic to start with, but starts with the desired results. What do I want people to accomplish with this unit? By being able to know the goal from the start, this prevents one from creating a unit that meanders, and prevents users having piecemeal information and not being able to accomplish something of substance upon completion.

View Acceptable Evidence (PDF)

State Acceptable Evidence
By stating the acceptable evidence in the very beginning, you are setting out a goal of what will satisfy, or prove to you, that the learners have achieved the desired results. Very often we state the desired results, but don't have a tangible way of seeing whether or not those desired results were achieved and to what extent. By providing a list of acceptable evidence we are designing the unit to not only be intentional, but also provide proof that learners indeed did learn.

View Plan (PDF)

Plan Learning Experiences
The planning of learning experiences is a way to take the idea from square one to the point where the desired results and acceptable evidence can be achieved...all in a systematic, step-by-step manner. Through the course of approximately ten lessons, the learners of this unit can be taken from where they are and scaffolded until they reach the desired results.

View Lessons 1, 2, 3 (PDF)

Create Lessons
The lessons were created using strategies taught with Understanding by Design and direct instruction best practices. The strategies used were:

• Ensure that the student understands where the unit is headed and why.
  This gives the students an underlying purpose as well as schema set for categorizing and focusing the learned material. Without it, it could make the learner confused and it will make it much easier for the learner to forget or misunderstand material.
• Design around problems and not just exercises
  Simple recall and plugging in does not allow for ill-structured problem solving, which permeates the real-world work place. Allowing for practice of real-world problem solving which is not a simple cloze exercise, allows for the opportunity of greater learning with greater transfer benefits.
• Teaching and using a learnable, functional, and usable conceptual model
  The model on which this unit is built, the essential questions guiding the unit and its structure, is easy to learn, highly functional and contextual, and usable in the real world (on the job). If the goal and model we are pursuing is not real and easy to understand nor usable, what is the point? It will never be used and the information will be lost if that is the case.
• For any real-world job or skill, teach both declarative and procedural components.
  Although a particular lesson may lean a little heavy on declarative or procedural, each needs the other partner or it’s useless. If you know something but don’t know how or when to implement it, the knowledge will never be used (and thus forgotten.) If you are told how to do something but don’t have at least some basic knowledge to support it, the problem solving and on-thejob ill-structured abilities will not be there to continue the procedures.
• Teach problem solving skills in context.
  Understanding when and how to use the skill and knowledge taught is key. Teaching it within context allows the user to practice making the decisions of when and how to implement the knowledge and skill learned, just as when OTJ.
• Integrate sources of information into one presentation
  Especially when the skill is a physical skill more easily conveyed through multimedia (sound, video/ animation) it is much more easily conveyed without misunderstanding. It also allows the learner to concentrate on how to perform the skill rather than what the author is trying to convey.
• Attach labels to steps or chunks to enhance understanding of these subgoals
  Being able to look at a diagram/set of information and create categories helps the learner not only more readily access the material at a later time, but allows for greater understanding and problem solving abilities
• Use arrows/lines in diagrams to help with indication of process
  This allows the learner to follow the appropriate process immediately upon looking at the diagram

View Assessments (PDF)

Create Assessments
The planning of learning experiences is a way to take the idea from square one to the point where the desired results and acceptable evidence can be achieved...all in a systematic, step-by-step manner. Through the course of approximately ten lessons, the learners of this unit can be taken from where they are and scaffolded until they reach the desired results.

Summary
I learned a lot from the Designing Direct Instruction class; in fact, I wondered why noone had taught me some of this information before, given that I had been in education for over a decade! Most of the schooling I had experienced had frowned upon direct instruction. "It's not how people learn best," some of my teachers would say. Others would equate it with simple lecturing and call it boring. However, from this class I learned that it didn't have to be either. One could *actually* tackle ill-structured problems with direct instruction, and help facilitate students toward higher-level thinking and the "big questions" that nobody has the answers to.

In addition, it showed me that if one wants to create a great lesson, it takes time. And iteration. And feedback. And time. And iteration. And feedback. And...well, you can see the process. However, by understanding your learners first through understanding their misconceptions, where they are, and by informing them of where the are going in the unit, that is a huge step towards understanding without having done much instructing at all.

If I were to do this over again, I would probably just request more time. More time to complete the unit...more time for everything. It took so much effort to create one lesson! However, I feel very proud of these three lessons and the potential they had as a part of a unit.

When I am designing lessons in the future, even if they are not totally direct instruction, I think many of the things we learned in this DDI class is applicable to many other types of teaching. For me, the most impacting lessons were:

• Inform learners of where they are going and why
• Understand and try to collect learner misconceptions, what they understand, and where they want to go
• For any real-world job or skill, teach both declarative and procedural components
• The overall designing process of goals first, assessment second, lessons third

I will take these lessons with me and they will serve as a guide in the future when I am designing instruction. They will serve me well.

View Proposal (PDF)

Proposal
The proposal was the very beginning step: it's starting with an idea and listing the objectives, purpose, audience, and main actions. In this way, the instructor can provide feedback before moving forward with any further designing.

View Design Plan (Storyboard, PDF)

Storyboard
The design plan is an essential component when creating a system. The reason is when design is done intentionally with the needs, users, activities, and goals in mind, there is far less chance that there will be major overhauls once you get to the feedback, user testing, and usability stages. The usability stage is not the stage where one starts to think of the user; the user's needs, activities, and goals should be thought about from the very beginning. This Design Plan is a part of that process.

The storyboard is a way to show screens and summarize actions that the user will take. Because Flash has many embedded symbols, it also makes it easier to plan out the files and file names that you will have with each screen. It also makes it easier because one needs to create symbols first before they can be placed within a movie clip and then onto the main stage.

Summary
Since this was approximately the fifth or sixth development course I had taken, I had experienced proposals and storyboards before. I have found this to be not only a helpful experience, but an essential component to better design especially when combined with peer and user feedback.

The use of designing by screens is even more critical when developing in Flash, but it is something that is useful in other design environments as well (such as when designing for performance support systems). In this manner, it is very similar in that there are features, and those features must be prioritized.

If I were to do this over again, I would probably take it in a different semester where I would have more time. Creating my idea within a few weeks was very stressful. Due to the short timeline, the development still needs some reworking and the look and feel is mediocre.

However, I am looking forward to seeing how Flash or other applications that utilize flash (like the Flex development environment) can be used with Actionscript in developing more friendly performance support systems online.

View artifact (external website)

Smart Signer is a web application developed with XHTML, CSS, PHP, and MySQL. The core of it was developed in Web Application Development I class with Chris Amelung and Tawnya Means as instructors. Users can login, choose a module, see the signs, see the sign details, take a test on the signs, and print out a certificate if they passed.

Eventually I would like to add a community section in order for users to ask questions of each other; I would like it to be a supportive place for users to learn basic sign vocabulary and support each other. However, I still need to learn how to integrate a forum within an existing web application. Part of that may consist of finding appropriate APIs for the forum I choose to implement.

This website is still within a "draft" section as it is not ready for main public view as of yet. However, I hope to continue to improve and perform some basic usability tests on it in the future in order to get it ready for beta.

• username: user
• password: enter

View artifact (external website)

This was by far one of the most challenging development tasks I had done during the program. In addition, it was done within a few weeks (since Flash was a summer class). This program customizes itself based on what the user's current sign language module is. For example, if your current Smart Signer module is Adult Signing Level 1, then the game will give you those signs. If your current Smart Signer module is "colors", then the game will give you color signs.

This is also within the draft folder as it is also not yet ready for full public view yet. There are a few things that need to be corrected in order to prevent the user from triggering some error (example: fast clicking). However, I look forward to when this can be for full public view and help families or individuals learn some basic signs in a fun way.

• username: user
• password: enter

View artifact (external admin website)

View artifact (external user website)

I had developed this performance support system while working for Special Education Service agency as a program coordinator. It originally began as an idea of wanting to get otoacoustic emission screeners and tympanometry screeners out to local infant learning programs on a loan basis. However, based on my experience with them, I knew that it could be a bit overwhelming in keeping track of who needed followup, rescreening, etc. In order to relieve the load of remembering, storing, or keeping track of this data, I developed the hearing screening tracker system. Due to confidentiality, the link I give here is the same database but on a different server with fake data.

It started out with a brief Surveymonkey questionnaire out to all infant learning programs throughout the state. Were they even interested in the idea of hearing screening for their children? Secondly, for those that were interested, I had in-depth conversations (interviews) with them both on the phone and in-person over the course of approximately 6 months while I was promoting the idea to the state and planning out how to develop the system. Most of the things users of the system wanted was:

• No more extra data to enter. They had just started a new system with the state and while they would love to track hearing screenings, they didn't want to enter any more data.
• Easy to use. They wanted something plain and simple, not complicated, get in, see what's needed, get out.
• Less words, more pictures. Many complained of the current health forms they are required to fill out....too many words, too many checkboxes, and not enough visuals to give a simple idea of where a child is at.
• Trackable, searchable by child. This came out as a desire after the first four programs used it for a few months.

We are still in the process of iterating it, but currently we have eight out of the fifteen programs currently using the system. Percentages were added on the front so that perhaps if they came across a grant where they could apply for an OAE and Tympanometry set, they would have the percentages of failed screens within their area.

Admin Interface: Link

• username: visitor
• password: visitor!

User Interface (Loaner of OAE and Tymp) Link

• username: user
• password: enter

User Interface (Owner of own OAE and Tymp) Link

• username: user2
• password: enter2

View artifact (external website)

The We Need Apples website was a collaboration between Matti Koivula and I during Web Application Development II. The most difficult part of the process was learning ASP.net 2.0 and understanding all the concepts it entails within a self-taught environment.

We Need Apples is a place to create an online grocery list, print it, and sort it by store. You can create your own stores by adding them and then adding items to the particular store.

ASP was definitely difficult for me to learn, predominantly because I wasn't aware of how the .NET framework worked. Over time I began to understand it more, although I haven't spent much time in ASP or VB since that time. I have found much greater use for PHP and MySQL in the non-profit and educational sector simply because of the platforms available.

I also had several experiences where MSSQL did not have some of the features that MySQL had; in the end, I had to change the format of how the items were stored due to this (and limited knowledge of manipulating code behind).

• username: hopper
• password: hopper!

The word "systematic" means methodical, thought out, purposeful, or planned. It is not haphazard; it is not on the fly or done "just because". Thus, systematic evaluation is not going up to a friend and saying "Hey, can you check out my website? What do you think?". It's the process of iterating a product with purposeful evaluations, mainly formative and some summative, in order to create a system that works well fro the user. My Flash-based Sign Memory game was one of the systems that underwent systematic evaluation.

View Peer Evaluations (PDF)

Peer Evaluations
These Peer evaluations and other evaluations like it provide a great first step in the evaluation process. By setting up a form and getting feedback from individual users, you can see a general sense of where the system is at, what their perceptions are, and if there are any patterns in comments regarding usability, functionality, and impressions.

Some people may consider these types of evaluations usability evaluations, but I do not. Getting a survey back from users is very different from watching them use a system. However, they both provide valuable feedback. The survey can be used on a more massive scale giving one lots of generalized information, whereas the usability evaluations will probably be one on one, not allowing it to be done on a massive scale, but one will get some very specific information about how the system is used.

View User Observation Results and Final Evaluation (PDF)

User Testing
User observation is an essential component to having a system which feels intuitive to the user. Ideally, you want the mechanisms to be invisible. In essence, they "just use it." By giving the user a task and watching their progression, one can see what is frustrating, what was confusing, what is a pattern with all users, what seems to work well.

The User Observation Results and Final Evaluation includes a summary of the observation results as well as a summary of the previously mentioned peer feedback. Combining them together it creates a list of low-to-high priority items which need to be changed in order to improve the system. This is included in the "Usability Form Ratings".

Although this game is still in the process of improvement, I feel this process and these documents to demonstrate systematic evaluation in order to improve a learning system.

Summary
Systematic evaluation is a key component in understanding the success of a system. Is it meeting its goals? Is it serving its purpose? Is it enjoyable to use? The systematic evaluation encompasses both formative and summative components, and continually tries to assess where the system needs to improve, and if it is working for its users.

One of the important things I learned from systematic evaluation is to start early and often with feedback and evaluation of a product. The formative evaluation phases are very important in making a system a good fit for its users as well as resolving usability issues.

If given more time I would have iterated the product more. There were still some usability and development issues towards the end, and some issues that left users confused about where to click, or why something didn't work when they clicked too quickly. After those were resolved, I would have also performed a summative evaluation with the product. Does it help users retain more signs after playing the game a couple of times? Do users enjoy the game and come back to the site? Do they see it as a good way to practice their signs in a fun and enjoyable manner? Those are some of the questions I have regarding the game. I did get some answers from the final evaluation of this in class, but not those surrounding learning.

I do plan on continuing to develop learning systems in the future. When I do, I will try to implement the evaluation process as we did here with the Flash component, and will also try to include a summative component as well. The process of summarizing screens with actions and rationale for those actions is something I will continue to use, along with the skills I learned in the performance support system class.