In the competitive and ever-evolving world of compensation management, having reliable market intelligence is crucial. At Novo Insights, we understand the challenges Compensation professionals face in finding, evaluating, and leveraging compensation market intelligence. Here, we present a framework to help you evaluate key factors when considering what data sources to use for your compensation benchmarking analysis.
The variety of data sources for compensation benchmarking has expanded—it’s no longer just about survey data. In our market research, we’ve identified six distinct types of market data. Within those six types, there are numerous providers to consider.
So how do you pick which data source to use in your business? How do you evaluate the usefulness of any single provider of compensation intelligence?
When evaluating market intelligence data providers, apply this five-part framework:
Focus
What metric or idea are we looking to summarize and compare?
The first step in evaluating compensation market data is defining the focus of your analysis. Are you interested in base salary, total cash compensation, long-term incentives, or perhaps all three? The metric you choose to summarize and compare will shape the rest of your evaluation process.
Across the distinct types of market data, certain types are limited in their ability to answer certain questions. For instance, job posting data or analytical data will have limited insight into stock-based compensation levels and design practices, since postings and most underlying analytical sources focus primarily on base wages. On the other hand, other data categories can often provide detailed insights into equity award practices.
Understand what focal metrics your benchmarking sources can provide. In this stage of assessment, it’s also worth understanding how the provider defines and standardizes their view of that benchmark.
Sample
From whom are the underlying data points being gathered?
The sample from which data is drawn is critical for ensuring the validity and relevance of your compensation analysis. It’s not just about what percentile of the market you target – since that percentile is a measure within a defined population you view as the market. A diverse and representative sample ensures that your analysis is robust and applicable to your specific needs.
To understand the population from which data is gathered, here are key considerations and questions to ask:
Time Frame
When was the underlying data gathered? Compensation trends can shift rapidly, and outdated data may not reflect the present market conditions. Some benchmarking sources contain data points that are up to 24 months old, which are then “updated” based on other analytical methods.
Participant Demographics
It’s not just about how many companies. If the underlying companies aren’t relevant to your talent market, more companies just add more noise to the data. That said, be careful not to limit your perspective to your immediate business competitors, as you participate in a broader talent market for many roles.
When thinking about the participant demographics, think about all of the following:
- Industry
- Geographic location
- Organization size
- Ownership structure
A well-defined sample allows for more accurate comparisons and insights, reducing the risk of skewed data and erroneous conclusions.
Methods
How are data points gathered and normalized for consistent comparison?
Benchmarking is ultimately about comparison: understanding the variations of a metric and where you stand within that variation. To translate data from a broad sample of organizations to make it comparable across organizations, benchmarking providers have an underlying methodology. Understanding this methodology behind data collection and normalization is essential for ensuring the data's reliability and applicability to your organization.
Key aspects to consider include:
Job Taxonomy
In compensation management, the job is our primary unit of analysis. As a result, how a benchmark provider presents a job taxonomy – their method for organizing and standardizing job roles – is essential to building confidence in the data.
For example, compensation survey providers have well-articulated benchmark roles, organized into job families and job levels with detailed descriptions. Analytical providers might align with the U.S. Office of Management and Budget’s Standard Occupational Classification system. Smaller niche providers might rely on individual titles and role descriptions.
The job taxonomy creates the common baseline within the data: the more it reflects how you view your talent market, the more likely the dataset can be actionable for you.
Data Alignment
There are multiple methods for how data from companies in a sample ultimately align with the job taxonomy and provide underlying metrics.
The best understood is the traditional job matching and survey submission process. A compensation professional in the participating organization is given access to information about the job taxonomy and focal metric definitions, matches jobs from within their organization to the taxonomy jobs, and provides data based on the underlying focal metric definitions.
But this isn’t the only method.
Some non-survey data providers use data science to try and make this alignment process more efficient. One provider, through its integration with a participating organization’s underlying HR data, uses multiple data points (title, reporting relationships, experience, even compensation) to recommend a benchmark role and level in its taxonomy, and allows participants to refine those recommendations if needed. Providers of benchmarks based on job postings or offers can algorithmically process underlying job posting descriptions to predict and align jobs to an appropriate job level and infer compensation structures from posted ranges or the data in the offer. Note that these are examples – no two providers have the same process for data alignment.
As data science matures, the methods for job matching and aligning to job taxonomies continue to evolve. It’s critical that compensation professionals understand the methods used by their providers and the potential risks or biases in those methods.
Quality Control
Yes, there are risks and biases in the Data Alignment process. Compensation Analysts are humans, and will sometimes err in selecting a job functional match or level a role inconsistently. Algorithmic job matching is subject to bias in the underlying data set or can be fooled by skewed data in the data being processed.
As a result, it’s critical to understand how a benchmarking provider promotes quality in their data collection and processing. Some providers have compensation experts reviewing all or a subset of data. Others use templates to identify obvious errors or questionable values. Others use reinforcement techniques in their machine learning models to learn from errant data and process it more efficiently in the future.
Scrutinizing the methodology helps identify potential sources of error and ensures that the data you rely on is both accurate and comparable.
Tools
What methods can be used to support insightful interpretation?
Ultimately, you select and use a benchmark provider to better understand the market. Each provider may supply a number of different tools to help you get a clearer view of the underlying metric. The tools and techniques available for interpreting compensation data play a significant role in deriving actionable insights.
Consider the following:
- Data Cuts: Ability to segment the underlying data by various criteria such as geography, industry, and other participant or employee-level demographics
- Descriptive Statistics: This is potentially quite obvious for compensation professionals, as we often think and speak in market percentiles. It’s important to understand what descriptive statistic choices are available (or not) to understand the market composition and distribution.
- Visualizations: It need not be just about tables of percentiles and prevalence. Data visualization can help drive understanding through the thoughtful use of graphs, charts, and dashboards that help in visualizing data comparisons.
- Trends: Some providers can show and explain how benchmarks are changing, rather than just the current distribution of the statistic.
Value
What investment is required relative to the data and tools you receive?
There’s no such thing as a free lunch – but there is such thing as free market data. The level of investment required to secure compensation benchmark data runs from free to several thousand dollars. However, a low-cost benchmark source may also have notable gaps and limitations on Sample, Methods, and Tools.
There are also different levels of investment in time to provide data, whether that’s establishing integrations, matching jobs, or processing survey submission templates. Be sure to think about the investment in both the cash outlay and the time required on your side.
By thoroughly evaluating these five key factors—Focus, Sample, Methods Tools, and Value—you can ensure that the compensation market data you use is robust, relevant, and reliable for your organization.
Which Benchmarks are Best?
They say, "All models are wrong, but some are useful." We find all data sources to provide some value, and organizations should invest in data sources that reflect how they want to execute their compensation strategy. We recommend a portfolio approach to monitoring the market, understanding the composition of different data sources, and how to use them to better understand the market. Even long-loathed self-reported data must be considered, as your workforce sees this data and it informs expectations of what is competitive.
