5 Reasons Why Your In-House HR Assessment Will Fail (and how to avoid that) (2024)

HR analytics is about using data for informed decision making. The majority of variables of interest regard the H – human – in HR. They focus on abstract phenomena and one cannot measure them directly (e.g. job engagement).

Assessments are the most used method for data collection of soft variables. We need them to provide key information to answer research questions. We also use them to predict the impact on other HR variables and on business outcomes. So, the use of the right assessment is an important part of HR Analytics.

In this article, we’ll look at the development of in-house assessments. We’ll discuss the importance of reliability and validity when it comes to HR assessments and we’ll share 5 tips to help your in-house HR assessment succeed.

Validated or in-house HR assessments?

The ability to choose or develop assessments is an essential capability to be successful in the HR analytics field (Coolen & IJsselstein, 2015). Using validated measures is the safest and more economical choice. They are more reliable and developed following a strict process. An alternative to finding such measures is the Academy of Management Measurement Chest.

However, in-house development of HR assessments is quite common. We use assessments to measure perceptions, attitudes, and opinions of employees about HR and related outcomes. As they are so widespread, sometimes it’s tempting to develop in-house solutions. Or customize existing ones to our organization.

However, problems with measures’ reliability and validity often lead to unreliable metrics. For example, usually, in-house solutions are developed without a rigorous literature review to define the phenomena and to formulate items. The time available to develop the assessment and the sample to test it is usually limited. The scarcity of key resources for such projects results in low reliability and validity. As a consequence, results might be difficult to interpret. That’s why the wisest solution is to use a validated HR assessment.

Reliability and validity

Reliability and validity are the main properties of HR assessments proposed by researchers. Reliability means that the assessment should deliver results that are stable in different moments and samples.

Reliability assures that your result is not only obtained due to your organization – your sample – characteristics. It also guarantees that if you repeat your measurement in different points in time you will still get consistent results.

Related to reliability is the internal consistency of the HR assessment. For example, let’s say you are measuring how employees experience the HR policy. You will use multiple questions on different HR domains. Each of these items measures small parts of the whole HR definition. The internal consistency indicates that they succeed in that.

The second one is validity. Validity refers to the property of an assessment to actually measure the variable of interest, in this case, HR. For example, the items should represent the phenomenon that they intend to assess.

As HR is a latent variable, items are the way we have to operationalize the assessment. If an HR assessment contains items that are not related to HR, the results will not provide high-quality information on HR. That’s why reliability and validity are so important.

Developing an in-house HR assessment: 5 tips

Validated measures are only published when they present good properties and have adequate validity and reliability. Developing an in-house HR assessment is a great challenge that risks failing for many reasons. If you still decide to pursue the development of an in-house HR assessment, here I stress five issues that are common in organizations and provide suggestions on how to avoid them.

1. Incorrect definition of HR – or another related variable of interest

When we work in a certain field, we get so used to the terminology that we rarely ask ourselves if we are talking about the same thing. The exercise of asking your coworkers in the same team how they define HR can show you that we have different perceptions of one term.

The definition is what delimits the phenomena and will inform all the following steps. Commonly, organizations skip the definition stage and rush to formulate items. That can be a big mistake. To avoid that, use evidence to back your proposition. There are plenty of studies with empirical testing that offer adequate definitions of HR and related variables.

2. Gaps in the item generation step

After reviewing the scientific literature and choosing one definition, the next step in developing an assessment is item generation. Researchers normally generate items based on rigorous screening and the comparison of different published and validated HR assessments that inform this step of the process.

In organizations, it is very common for HR teams or even people from different departments to come up with items. In general, they take into consideration personal experience and gut feelings. This practice tends to harm the quality of the items and to generate items that do not assess HR at all.

Since these items are not related to the general assessment they compromise the whole measure. So, you need operationalization of what you aim to measure, in accordance with your definition of HR. Also, base the items in literature.

3. Interferences of external stakeholders in the item generation step

Like the previous challenge, when people know the HR department is developing a new measure, it is very common that an overwhelming quantity of suggestions of items flows to the HR department. The consequences, again, are ill-written items. Depending on who formulates the items – it can sometimes also be difficult for HR to reject the suggestions they receive.

Some HR teams decide to share the HR assessment they develop with a group of stakeholders that lack the necessary knowledge and experience to contribute. Again, the suggestions might not be very helpful and hard not to include in the final version of the assessment.

For the item generation, advise the stakeholders about the consequences of inventing items and scales that are not evidence-based. It’s expensive and useless. As a suggestion, keep the project and team as small as possible to avoid interferences. If there’s no way, create a separate scale with such items in order to not impoverish the final results you intend to achieve.

4. Lack of adequate planning to validate the measure

Let’s face it. Organizations and academia have a different pace. Publishing an HR assessment in academia takes multiples studies in various samples with different analyses and related variables. It’s a lot of effort, time, and people involved. These resources can be scarce in organizations and planning is essential.

We cover two main planning issues. For the first, you need to take your time to collect enough data. Data from a small sample size results in too little statistical power to test the scale properties. That means that you might not have enough variance to evaluate the quality of your assessment.

More strict researchers (Hair, Black, Babin, & Anderson, 2010) suggest a sample size of 20 people per item. If the scale has 20 items, that results in the need for 400 respondents. One good practice is to also test your results in two samples to check if they are reliable, which would result in 800 employees participating. That would count as one study when comparing to the HR literature.

For the second, attention to scale validation is important through factor analyses. Factor analyses test the structure of the assessment and the relationship of items with the main variables. Conducting such studies in a hurry can also result in poor quality or incorrect analyses.

Running a pilot study can help with that. Moreover, advice for the possibility of getting more evidence to better inform decision making and strategic planning. For the scale development testing, understand your numbers and theory to defend your project. Evidence-based decisions and choices are stronger and more difficult to ignore.

5. Predictive analysis of outcome variables

By the very definition, HR assessments aim to identify multivariate relationships that are complex in nature. A common inclination is to settle for more simple analyses and univariate analyses. Such shortcuts bring big troubles to the interpretation of results and strategic action. For managerial applications, the results must offer a reliable effect that informs action and provides informative results. The mean and standard deviation in isolation might be very shallow.

Inferential analyses enrich the quality of the HR assessment findings. The investigations of HR and business outcomes make predictive analyses a possible design. As a final suggestion, for the scale evaluation, stress the goal of using a scale and the research question that you aim to respond to. Also, stress why testing outcomes reinforces the importance of HR assessments and point in the right direction.

Good luck!

Subscribe to our weekly newsletter to stay up-to-date with the latest HR news, trends, and resources.