Defining and measuring success

Kati Price is the head of digital media at the Victoria and Albert Museum, London, and Dafydd James is the head of digital at Amgueddfa Cymru (National Museum Wales). In August 2018 they published Structuring for Digital Success, a global survey of how museums and other cultural organisations resource, fund, and structure their digital teams and activity. 

The survey and paper explore many aspects of museum digital activity based on responses from 56 galleries, libraries, archives and museums sector (Glam) organisations from across the world. It examines how these organisations are changing their digital teams to define and drive success, and identifies patterns that are beginning to emerge. 

It also looks at the changing structures and relationships that digital teams have with museum colleagues, and includes key insights and practical advice to help organisations of all sizes understand how best to structure their digital teams.

Below is an extract from the paper, which is available here.

Around half the organisations surveyed had a defined digital vision or had specific digital objectives within their vision. Others saw digital as embedded in other objectives, enabling broader strategic aims around areas such as engagement, access, or learning:

Naturally, the organisations we surveyed had very different definitions of success. Many respondents defined success as increasing digital reach (building a website, social and subscriber bases), and deepening engagement with their audiences. Very few had criteria for specific commercial goals, and some only defined success and related metrics within distinct projects or products. Some also mentioned digital transformation goals that define what digital success looks like internally.

Disappointingly, less than half of all respondents agree (19%) or slightly agree (29%) with the statement: “We have a clear and agreed set of criteria that we use to prioritise our work.” And over a quarter (28%) disagree or slightly disagree that they use criteria to prioritise their workloads.

Only 10% of respondents agree with the statement: “We have well-defined success criteria for digital activity and product development.” And just over half our respondents disagree or slightly disagree with the statement (52%).

Measurement and defining targets

Though the majority (56%) are measuring performance against defined targets, it’s of concern that so many aren’t, or are unsure: almost a quarter (23%) are not measuring against their targets, and a fifth (21%) said “maybe”.

Most respondents (40%) slightly agree with the statement: “We use data to inform product and service development within my organisation.” Note the hesitation here; why aren’t the majority agreeing with this statement? It’s likely that this stems from data management/analysis being the most underrepresented skill among almost 60% of the organisations surveyed (see section 6).

In our survey responses, there were a relatively low number of well-defined visions for digital success. It appears that organisations are not always taking time to define what digital success looks like. And, if they are, they are not necessarily communicating this vision across the organisation.

The focus for many organisations is on digital engagement and increasing digital reach; for some, it is business transformation; and a few had specific commercial goals that contributed to their definition of digital success.

Alarmingly, even with a vision for digital success, most teams don’t have a shared set of criteria to measure that success and to help prioritise their work; and, even if they do have them, most aren’t measuring against objectives.

For most organisations, there is more work to be done in defining and communicating criteria to plan and prioritise digital activity, and to measure success.

Several of our respondents revealed that their key performance indicators (KPIs) are focused on so-called vanity metrics (for example, visits) and are not convinced that this is the best way to understand if they’re delivering against organisational goals.

Some, however, have KPIs that are embedded in the organisation’s strategy. More digitally mature organisations have KPIs that are much more ambitious – not just measuring website sessions or growth, but more qualitative measures around satisfaction, for example.

Digital leaders need to ensure that they and their teams are confident in understanding how to define success criteria. They need to think about how digital can help drive audience engagement, distribute knowledge and research, deliver against commercial goals, and accelerate transformation across the organisation.

They also need to understand how to measure against their targets. For this, they will need to build up data analysis skills across the team and consider having dedicated data analysis resource (a skill currently underrepresented in the teams we surveyed). In the meantime, leaders should consider bringing in freelance support. At all costs, they must avoid operating in a data void.

Teams must map priority areas that will help them achieve digital success (however they choose to define it) with KPIs for each area, to better understand how each is delivering digital impact.

In defining digital success, teams must focus on outcomes (rather than outputs) with goals that are measurable.