improvement

Performance Data Collection For All Professionals

/ Chris Sawyer, M.Ed.,BCBA / Leave a comment

No matter what you do, you’ll often find yourself in a position to either teach a skill or train someone in a proficiency you have. In some cases, many times. One of the most necessary parts in my line of work is data collection on human behavior and performance. I’ve met hundreds of professionals and paraprofessionals over the years who see how behavior analytic therapy and training are delivered using daily data collection and measurement and often get asked “Do you have a spare sheet I could use?”. Workshops, after school programs, camps, job training events, painting classes, apprenticeships, exam prep, clinical trainings, driving courses, and other various skill based events have all had opportunities for me to show off what data collection can be used for, and how it can be applied to any profession where one person needs to learn a new skill and their performance needs to be evaluated in a well defined and stable way. If this is something that you do, or have an interest in doing, I have just the form for you. In just 15-30 minutes of reading and reviewing the instructions below, I aim to make sure you learn and can use the following cool tools from the world of applied behavior analysis:

  • How to track data on performance for a single day and across days.
  • What a “Cold Probe” is, and how you can use it to configure and adjust your training plan.
  • What “Discrete Trials” means, and how you can use them to work on a single or multiple skills in a single training session and deliver effective feedback for performance improvement.
  • How simple and effective percentage data is for performance.
  • How to practice a trained skill repeatedly without become repetitive.
  • When to deliver reinforcement (social praise) for success, and when to deliver prompts (correction).
  • How to compare today’s performance of your client to their future or past performance and use visual analysis of the data to make better decisions.
  • What “behavior coding” is and why defining our target performance goals matters.
  • How to do an analysis of component skills and break your trained skill down into pieces.

I am attaching the link to this performance data collection tool below. You can either print it out and use it in free writing, or use it digitally if you carry a tablet or similar device. This pdf has been formatted to use text fields for typing in easily, a spot to import your logo in the heading with no fuss, and the data sections can be clearly exported into the spreadsheet software of your choice. There does exist some very advanced software out there that can do more than this. This is not the be all-end all, and if linear regressions, or reversal designs are your thing, this might not check all of your boxes. I suggest visiting other subscription software for the research level of analysis you might use in a human operant lab, but if you want something practical, with ease of use, and is completely free of charge, by all means enjoy the form below.

performance-data-collection-sheetDownload

Instructions:

Let’s talk about the top portion of the form for a moment where we have three fields:

  • Name:
  • Date:
  • Instructions:

When we are training an individual, or even a small group of individuals, we need a way to separate out performance data so that we do not get confused when it comes to evaluation and analysis of it. Each individual stays separate from one another, and each day’s performance is distinct from another. The “Name” section here applies to the individual you are training, and not the trainer. We also will need the date of the training so that we can review our data in order, and include instructions if we have multiple trainers performing the same training across different times or dates. Every profession is different and every trainee is going to require different skills, so I will not be able to describe every form of instruction you might want to use here. I would suggest something concise and to the point. Your co-trainers on the topic would likely understand the skills and only need an instructive structure in delivering the training. For example, if we had a client who we wanted to train to high proficiency in jump roping for their schoolyard double-dutch competition, we might want our trainers to know what to have ready.

Cold Probes:

In behavior analytic terminology, a “cold probe” is something that you do to test a skill without prompting or incentives to see where the client’s performance is without assistance. Simply put, at the start of your training or teaching section, you ask them to perform the skill and see how they do. Can they do it completely independently to your established level of competence? If so, you might mark a “Y” for “Yes”. If not, you might mark a “N” for “No”, and that gives you an idea of where that day’s training targets might focus on. Cold probes are useful when you have a client who has mastered something, or maybe is coming in for the first time, and you want to see if they can produce that specific target of performance on demand. Reviewing the cold probe isn’t a final answer on whether that person has or does not have a skill in their repertoire, but it can give you an example of their unaided performance for you to use your training judgement on for what they might need to be taught, practiced, or have a long term strategy for performance improvement on. Cold probes are tools, not something to make or break a training plan on. Performance can fluctuate. Use them to determine a focus for that day, but keep in mind it might only be a part of your overall goal for the client. You can also use cold probes to remove a planned part of the training that day that might not be worth giving extra time for. If our imaginary jump roping client can perform their three alternate foot step jumps without aid, perhaps we gear our training topics for the day for something a little more advanced to make the best use of our time.

A Component Skills Analysis and Discrete Trial Training (DTT):

We can use our cold probe data to figure out what skills we can target for improvement. Often, when we come across a difficulty in competency with a trainee, the skill is often made up of smaller more basic skills, or have a precursor skill that needs to be strengthened before they can move on to the original target skill. Discrete Trial Training (DTT) is a process by which a complex skill is broken down into smaller component behaviors which are taught in order to meet the original target. They are “discrete” or singular component skills which are set up in a distinct training opportunity, where we can follow up demonstration of a skill with either praise/reinforcement when performed correctly, or prompting/feedback when there are errors in need of our assistance. Each practice opportunity is a new chance to try again and build towards greater success. The number of trials you use is not set in stone, but for this training sheet I have provided five opportunities for each component skill. Let’s talk about our example jump-roper. What would happen if our trainee did not perform their alternate foot jump to our criterion of success? Take a look at the sample data below.

In this example we’ve had our trainee demonstrate the skill five times, with each component skill being performed an equal number of times. What might this data suggest? Is our trainee having difficulty in all areas? Probably not. In this case we see that they are able to lift their left foot into a jump perfectly for all tracked trials, but when it comes to the right foot, and the heels being up during jump roping, we see errors. A good part of using these types of trials is that you can compare performance in one component behavior to another. Look at the data above. You will see that the right foot lifting, and the heels up components share a trend of errors. That could lead us as trainers to suspect that there might be a relation between the two, and our training and corrective procedures can be tailored at this point for helping them improve. With this style of data collection we can pin point exactly where errors occur, which makes our training time tailor fitted to the need and increases our efficiency.

Do not forget about reinforcement in these stages. Reinforcement is what increases rates of the target behavior that it follows. We praise and reward as soon as a success, or approximation to success (improvement) is seen. By praising and rewarding what goes right, we can keep that level of performance high. We can use reinforcement following prompts to maintain a level of engagement and improvement. Do not simply focus on the errors alone. Target the successes and reinforce them. A solid training procedure is heavy on reinforcement.

Percentage Data and Analysis:

In our trial data above we use percentage data as a form of measuring performance and success. In this scenario, using five trials means that each trial counts as a distinct 20% of the final score. When we measure performance we want to make sure we have a criterion by which we consider mastery. Not all skills necessarily, or humanly, can be done with 100% every single time. In most cases, keeping to 80-90% as a goal is not a bad benchmark to have in mind. It is well above blind luck, and with proficiency at those levels, it is often easier to discover patterns of what environmental stimuli correlate to higher performance than others. Does our jump roping trainee do better during our individual training than they do in front of peer crowds on the playground? A variance of 20% or more might let us know that if we see a pattern emerge over time.

The sheet above is structured so that you can export data from the probe and trial sections into a spreadsheet, where you can use a visual analysis (graph) of your choice. I, and many professionals, enjoy line graphs which show percentage of performance by date. By combining the results of multiple daily data sheets, you can create graphs and perform a visual analysis of progress in a way that is cleaner than raw data. By comparing the date of the data sheet, with the final percentage scores of success you can see something like this.

Reviewing performance data with your client (or their caretaker) is key. Visual data presentations like the one above can be a tool in your toolbelt to make large trends easier to understand. Line graphs are easy ways to show trends and to use that visual to breakdown where their performance was, compared to where it is now. Even if you see a negative trend, this can be a great tool to discuss what might be going on outside of the training and analysis that might be a factor. You can even learn what is impacting the graph, but might be missing from the training regimen. No data is ever wasted. It is all a resource.

Behavior Coding:

The final sections of the sheet involve spots where you can do what we call in the field of behavior analysis, and research in general, behavior coding. Behavior coding is a process by which you operationally define your target performance skills in observable and measurable ways. When you are working with a team, or with multiple trainers, your success can depend on whether everyone is measuring the exact same things the exact same way. We want as much inter-observer agreement as possible. Coding makes that possible.

Let’s take an example from our jump roping client above. One of the component skills we chose was “Left Foot Up (Jump)”. That can be confusing without further explanation. It could use an operationally defined and coded skill. We can use our behavior coding section to put simple and quick definitions so that everyone measuring that skill in the future knows exactly what it looks like, and what we consider success. The better our coding, the more sensitive our data is. We want to find a middle ground of detail without being obfuscating with too much wording. There is a difference between precision, and a code that makes tracking impractical. The main goals we want are something we can observe which lets anyone watching have the same opportunity to track it exactly as we might, and measurable, meaning that our behavior coding of our target skill can fit into the data tracking format.

For example: “Left Foot Up (Jump)”- The left foot is lifted up completely from the ground during a jump with enough space for the jump-rope to clear it underneath.

You may increase the precision of your measurement to match the distinct needs of the skill, but the goal is to be sure that everyone tracking data on that skill is using the same definition. This one above is what I would consider low to medium in precision, but will do for what we need it for. Match your definitions and coded behavior to your specific profession and needs, but be sure it is not vague or subjectively unobservable (“a spirited and joyful jump” could mean just about anything to anyone). If you need to use what some would consider subjective language, try coding for that as well (“Joyful” is defined as smiling during a jump, etc.).

Keeping a Running List of Component Skills:

Component skills which become mastered or are ongoing targets for future weeks can be listed on the second page as well. This helps us distinguish how we broke down our probed larger skills into their discrete and distinct components. Keeping a list of what we have worked on, and what we have yet to work on, can give us better ideas for future trials to run in the next training opportunity, give us a log of what was mastered or completed in a previous training, and give us a section for note taking on the component skills that would fit the needs of your professional training. I would suggest if you use the component skill section to determine future training targets, less is more. Training ten skills within an hour or two makes sense, but over training tens of skills within a time frame might lead to lesser mastery across the entire list. Focus on the most important component skills that make up the larger cusp skills. You may find success in picking your particular targets for each training session, or week.

Further Training:

I hope you enjoyed the material here and the review. It would be impossible for me to include every potential usage of these sheets, and the more complex data analysis processes you might want to use them for, but if you have need of further training, consultation, or simply questions, you may reach me on this website or email at csawyer@behavioralinquiry.com. I would be happy to help you with further training on this data sheet, how to adapt and construct your own, and any further interest you might have in performance tracking or behavior analysis.

Comments? Questions? Leave them below.

Getting Back Up After Failure

/ Chris Sawyer, M.Ed.,BCBA / Leave a comment

Failure is a tough topic to bring up but a necessary one. When we are in it, it’s all we can think about. When we are past it, we often do not want any further reminders of it. Failure, behaviorally, and psychologically, is a part of everything we do as a variable, and factors in to every future strategy we use. It is a part of our past that defines how we interact with the future. In a previous writing I discussed “Overcoming the Fear of Failure”, but this one will be about what to do when it happens to us. How do we move on? How do we grow from it? How to we set our future expectancies to do better? To what do we attribute failure to? All of these and more are necessary to making each failure a stepping stone to a future success, or else we might find ourselves in a loop generating ever worse strategies. Instead, we need to learn to get back up. Let’s talk now about some of the research we have on the topic and how we might navigate failure and find motivation from it.

Mastery Orientation vs. Learned Helplessness

When it comes to deriving motivation from failures, both big and small, the strategies that we develop in childhood have a great deal of influence on our current behavior. You may have heard of the term “learned helplessness” before, which describes a pattern of behavior of low motivation and outputs after repeated failures. The individual receives so little reinforcement following their actions that they simply do not continue to try. Diener and Dweck (1978) popularized these concepts in a study on youths that they split into two groups based on patterns and strategies that they observed without being taught. They found that some children when faced with repeated challenges and varying degrees of failure would either consistently give up, and reduce responding, while others would re-assess and modify their responding based on the inputs of their failure. The researchers were very interested in the cognitive strategies that both of these groups displayed, all without any coaching, and determined that even at a very young age, there were clear distinctions on these two types by their ideas on their loci of control. A locus of control is a belief system that people use to determine whether they have control of outcomes, or if outside forces do. A person with an internal locus of control would see the results of their actions as largely based on their own actions and future control. An individual with an external locus of control would see the results of their actions as largely impacted by an outside force or their environment. Now, there is a part of this study that some consider a little unfair. No matter what answer the children gave to their respective stimuli at the start, they were told they were incorrect. How they responded afterwards largely correlated based on how they viewed their loci of control.

Mastery oriented individuals appeared to generally attribute their failures to a lack of effort or something they’d missed. Even at that age, their first reaction focused on pivoting and reassessing.

Learned helpless individuals tended to attribute the failures to the situation as largely beyond their control (in this case, without knowing it, they were technically right as far as the experiment was concerned).

So what happened?

Mastery oriented individuals kept trying, kept changing their responses based on feedback, and largely kept at the task longer than the other individuals. They showed no decline and became more sophisticated in their strategy use (which was eventually validated).

Learned helpless individuals tended to show a progressive decline in the use of good-problem solving strategies and began to include less sophisticated and poorer problem solving strategies. Ones that would be even less likely to work.

This model of attribution is still used to this day, but has a few caveats. Unlike this study, in the real world, people are not always one or the other. In many cases, and complex problems, it requires using multiple loci of control, but also understanding whether the factors we evaluate and learn from are stable (long term) or unstable (temporary). The stability of an attribution is its relative permanence as a factor. If you know you are good at jumping rope, meaning you have high ability, you have a stable factor to consider your next success with. But, if you attribute jumping rope to how much effort your legs can put out, then the source of success is unstable—effort can vary and has to be renewed on each occasion or else it disappears. We’ll talk a little more about how effort and ability works in a second. The important part is that when it comes to evaluating our part in the grand scheme, the internal locus of control tends to help us perform better.  Let’s look at some examples.

It rained today and we got all wet. We hate that. What if it rains tomorrow and we don’t want to be rained on? Would a belief system around an internal loci of control make sense if we focus purely on ourselves and ignore the sky? Not very well. No matter how many strategies we might attempt based on our own feedback, we are unlikely to change the weather. On the other hand, a person using this internal loci of control might decide to travel away from the storm as a strategy, bring an umbrella, or wear a rain coat, which has some functionality for them but the rain still happens where they once were. Internal loci of control work best when we take into account our solutions but do not ignore the immutable environmental factors.

What about using an external loci of control on task performance? Perhaps we’d like to pick up three items off of our room’s floor within ten minutes. We might begin to generate all the reasons why we cannot, and how far the floor is from our fingers, and how many other factors there are between the items and the trash can, leading to very low performance on this task within a time frame. It’s the room that’s messy. It’s been messy for days now. So messy. So much mess too. What if we just pick up one thing then go back to bed? It’s still messy. Might as well not. Then, we’ve just effectively wasted time generating non-functional thoughts (poor strategy), and nothing was done (poor outcome). That isn’t helpful either.

Generally speaking, when it comes to our own behavior, within our own repertoires of ability, it is wiser to use an internal locus of control to conceptualize our potential impact on tasks and problems. When there are larger systems and unavoidable outcomes from the outside, it does not hurt to consider what lies in an external locus of control. We, as individuals, cannot control everything. But, as we see above, when faced with continual failure feedback, utilizing an internal locus of control early on can help us come up with strategies which mitigate the external circumstances and perhaps land us in a better spot. There is no harm in generating increasingly sophisticated strategies to put ourselves into better conditions and allow the external factors outside of our control to be managed from ever increasing positions of control and strategy on our part. Sometimes when failure comes, it comes after we thought we had a great strategy focusing on our own improvement and it just did not work.

How do we do it? How do we take back some semblance of control when the waves of failures keep coming?

Consider that the concepts of a locus of control, and how our actions impact our goals are called attributions, and have an effect on our future behavior and how we respond to challenges. When we attribute too much to external causes, it can lead us to decrease our attempts. When we attribute too much to internal causes, it can sometimes lead to more sophisticated problem solving, but blind us to other factors might be outside of our control and narrow our perspective too much.

Mediating these attributions not just in the moment of the first failure we come across, but those that follow can help us create a better perspective on our situation. We can also rely on our social circle, relay our experiences, to see if others can help us see what we might have missed and help our future strategies find better success.

  • Evaluate your current attribution and locus of control of the problem.
  • What are some ways we can evaluate our own pattern of responding and improve it? (Internal Locus)
  • What are some environmental factors that impacted our failure that our behavior did not change (External Locus)
  • How do we refine our strategy so that our next attempt can put us in a better position against those environmental variables if they happen again? Can we mitigate what held us back?

Purposive Behaviorism and Re-Training our Attributions

As individuals we can create systems that help us maintain a level of reinforcement to offset failure, and as social creatures, help create an environment of positive interactions that can help us both realize our achievable goals and find strategies to access them. Thankfully, we have concepts and theories at our disposal to explain the hows and whys. Let’s talk Purposive Behaviorism and how we can re-training our Attributional Theories.

If you’ve read my other works on this site, behaviorism itself is familiar to you. Purposive Behaviorism goes beyond the more mechanistic systems of reinforcement and punishment, stimulus and response, that you see in some of the more traditional theories. Yes, reinforcement is important to keep us moving forward. Yes, punishment (failure) can knock us back. But we are human, and complex beings, and a good analysis always takes that into account. From a purposive behavior standpoint, we use goals and work hard to achieve them. That is an intrinsic part of what it is to be human. In older theories by Edward Tolman, the term cognitive map was developed to describe how we do that. Our cognitive map is how we envision our path to our goal. We all have beliefs, unspoken ones, that a specific action on our part will get us closer to an intended consequence or goal. Let’s call these expectancies. They cover both the behavior we intend to do, and the goal we intend to achieve with them. It’s a roadmap. Tolman also believed that we learn from our successes and failures largely through a latent process. There is an automaticity to reinforcement that helps us pick up what has worked and set aside what has not worked, and integrating more cognitive and conscious strategies to what we have learned latently is the best way to move forward. Keep in mind not just what you can remember and consciously recall, but also what might have been learned latently from the experience.

When we map out our actions to meet a goal, we often give ourselves a time frame (hopefully realistic) in which to reach them. By giving our goals, or conceptual map of how we achieve them, a context in time we help judge how to act and what to expect. Generally speaking, acting now is always better than acting later unless you have a more advantageous use of time further along to position towards your goal. With our expectancies in mind, we have our actions, our goals, and our time frame. As adults, we also learn to discriminate effort from ability. Effort can be defined as the amount of energy or resources we must expend to progress towards the goal, while ability may be defined by our existing proficiency or skills that can achieve it. In most situations it is a combination of both effort and ability that help us reach complex goals.

Let’s reintroduce failure here. Let’s say that we mapped out our goal, we made our attempt to the best of our effort and ability, and we find that we simply did not meet success. Perhaps we even see repeated failure. It can be easy to get disheartened, and even travel down that path of learned helplessness, but we should do everything we can to avoid it. Let’s imagine that we did our best to conceptualize our locus/loci of control, and they were as accurate as they could be, but we still missed the mark. We tried, we failed. Let’s say our expectancy, our goal and plan to reach it, is still very important and we do not want to change the goal. How do we use our time most effectively now to get back up and try again? We need to re-train ourselves, and that means re-training our attributions.

Do we have the ability to achieve this next step in our goal? What did our failure show us?

Did we apply the necessary effort to achieve the next step in our goal? What did our failure show us?

Were our attributions on stability based around factors that were stable (ability) or unstable (effort)?

The combination of evaluating our ability and effort and attribute our failure and successes along these variables is key to knowing when something can be achieved alone, if further training, resources, or additional help from others is needed, and how to adjust our plans going forward to include these more sophisticated and evaluated plans that came from the experience. Failure here is a teacher. It’s not always easy to maintain effort after a failed attempt even if the ability was there. To retrain ourselves to analyze our attributions of the failure correctly, we must take some time to evaluate the factors. Use this tool from Dweck (2000), who we saw in that earlier study too, below to take a particular situation you might have been in the past, and see where the attributions fall.

Plug some of your attributions in the grid above and see where they fall. Do you think anyone else evaluating your situation might have a different series of attributions for it?

We tend to get the best results out of ourselves and planning ahead by attributing a reasonable portion our previous successes to internal and stable causes. What went right in the situation within our ability, even if there was an ultimate failure, that we can consistently do again? Example: I might not have won the race, but this was close to my best personal time yet.

When analyzing our failures, we can go wrong in attributing things entirely to unstable and external causes. Things that we see as completely out of our control, and leaves nothing for us to work and grow on. Example: I was going to go in to work today but then the roads were so busy and you know I can’t drive on busy roads…

The take away:

  • Turning failures into successes takes analysis of what happened.
  • Sometimes we analyze the situation well and can think of some improvements for next time focusing on our internal factors.
    • “Stable Dimension” attributions help us reflect on our ability and how to improve it.
    • “Unstable Dimension” attributions help us reflect on our level of effort and if we can improve it next time.
  • If we see many attributions leaning in the unstable or external direction, maybe it could take an extra pair of eyes to help us get a new perspective.
    • Reaching out to a trusted friend, or experienced advisor on the topic.
    • Re-evaluating the attribution by considering internal factors.
  • Learned helplessness can arise from attributing too much to external factors, avoiding evaluation of internal factors, leading to poor problem solving and less sophisticated goal directed behavior.

Getting back up after failure requires analysis of our actions, re-training our attributions to avoid learned helplessness, and consistent effort going forward.

What are some attributions you’ve thought about recently? Have the behaviors you’ve used to reach those goals been effective? Have they been ineffective? How has your belief system on the locus of control impacted the process? Have you utilized others to help you with alternate perspectives?

Comments? Questions? Feedback? Leave them below.

References:

Cooper, J. O., Heron, T. E., & Heward, W. L. (1987). Applied Behavior Analysis. Merrill.

Edward Chace Tolman. (2015). Introduction to Theories of Learning, 302–326. https://doi.org/10.4324/9781315664965-16

Hoose, N. A.-V. (n.d.). Educational psychology. Lumen. Retrieved November 11, 2021, from https://courses.lumenlearning.com/edpsy/chapter/attribution-theory/.

Molden, D. C., & Dweck, C. S. (2000). Meaning and motivation. Intrinsic and Extrinsic Motivation, 131–159. https://doi.org/10.1016/b978-012619070-0/50028-3

Schunk, D. H., Meece, J. L., & Pintrich, P. R. (2014). Motivation in education: Theory, research, and applications. Pearson Education Ltd.

Tolman, E. C. (1967). Purposive behavior in animals and men. Irvington.

Image Citations:

Title image: Getty Images/iStockphoto
Attribution Grid: Christian Sawyer, M.Ed., BCBA