Kirsi Hyytinen, Faïz Gallouj, Marja Toivonen (2019). A Multi-Criteria and Multi-Actor Perspective for the Evaluation of Sustainability Services. In: Advances in The Human Side of Service Engineering. AHFE Open Access, vol. 8. https://doi.org/10.54941/ahfe100299
The purpose of this paper is to study the challenge of the evaluation in the context of the services. Because of their specific nature the traditional evaluation methods and measures are not able to capture neither the diversity of the innovations nor the multifaceted dimensions of performance. This paper aims to contribute to the need for a more diverse evaluation approach. We focus on the context of service innovation in the environmental sector, in which we develop further multi-criteria and multi-actor perspectives. The multi-criteria framework describes the impacts of sustainability services and the multi-actor framework aims at analyzing the impacts from the perspective of various actors involved. Thus, our study provides a two dimensional approach to assess the impacts of services. The focus is on understanding the dynamics of service creation in the environmental sector, and on using new evaluation methods and indicators in that sector.
Systematic evaluation is an activity that has been an essential part of strategic thinking for decades. It has been linked both to the general development of organizations and to more specific functions. It has played a particularly important role in the context of innovation and R&D. In public interventions – often carried out in the form of policy programs – evaluation has been an embedded practice whose purpose has been to support decision making (Rip, 2003). Since the early days of evaluation, impact assessment has been a typical way to implement evaluation, which has meant that evaluation has been understood in terms of performance-related steering and monitoring (Chelimsky, 1997; Rip, 2003). The primary idea has been to produce indicator based information to prove accountability and to legitimate the role and existence of individual organizations and policy instruments.
This kind of an approach to evaluation includes several problems. First, it is backward looking and does not work well as a guiding instrument in the current society that is characterized by rapid changes (Todd and Wolpin, 2010; Weijermars and Wesemann, 2013). Second, the approach is “atomistic”: it focuses on individual organizations or policies and overlooks the fact that impacts are co-produced by several actors and emerge in a broader context (Rip, 2003). Third, the indicator based assessment simplifies phenomena that emerge in cyclic, complex and long-term processes. Indicators assume a simple causal relationship between intervention and impacts, which is incompatible with the modern view about the emergence of innovations (Hansson, 2006; Van der Knaap, 2006; Cozzens and Melkers, 1997). Summarizing, the linear input-output-outcome -thinking included in the traditional evaluation does not correspond to the complex development processes and the multiple relationships between the contributing actors (Arnold, 2004; Patton, 2011).
As an answer to these problems, researchers have suggested a more versatile view: “plurality of methods” Human Side of Service Engineering (2019)
(Dyehouse et al., 2009; Williams and Imam, 2007). Systems thinking have often been applied hand in hand with this view. It means highlighting interrelationships and feedbacks between the whole, its parts and the environment they are interacting with (Cabrera et al., 2008).
Recently, the increasing “servitization” of society has put pressure to develop a more advanced approach to evaluation. Here, an additional problem linked to the traditional impact assessment has become apparent: the indicators have been one-sidedly technologically focused (Toivonen, 2010; Rubalcaba et al., 2013). These kinds of indicators do not capture the immaterial and interactive aspects that are central characteristics of services. Today services are also increasingly forming systemic wholes – especially the most urgent problems in the present society cannot be solved via the development of individual services, but service systems. The issues of sustainability are an illustrative example of this development. The current evaluation methods are unable to tackle the challenge of the systemic nature of innovation and service creation (Smith, 2000; Edqvist, 2005; Ahrweiler, 2010).
This paper aims to contribute to the above described need for a more diverse evaluation approach. We focus on the context of service innovation, in which we develop further a multi-criteria and multi-actor perspective for evaluation. Our starting points are the multi-criteria framework developed by Djellal and Gallouj (2010), and the multi-actor framework developed by Windrum and Garcia-Goñi (2008). The former strives for better visibility of the multi-faceted dimensions of service innovation and highlights the importance of quality and social value. The impacts of innovation are analyzed using the concept of performance which is also interpreted broadly. The purpose is to diminish the “double innovation and performance gap” that derives from a narrow view on these concepts and has resulted in a difference between the reality and the ability of the traditional measures to perceive it (Djellal and Gallouj, 2010, 2013). The latter framework – the multi-actor analysis – is built on the characteristics-based view on service innovation (Gallouj and Weinstein, 1997). It focuses on understanding the dynamics and impacts of service innovation from the perspective of the different actor groups participating in the service development.
In the conceptual part of our paper we open the “innovation and performance gap” in the service context and review the opportunities for new kinds of evaluation approaches. Then we discuss and concretize the multi-criteria and multi-actor frameworks by applying them in services that are emerging to support the strivings for sustainability. More specifically, we analyze service innovations and their assessment in the context of sustainable energy production. The multi-criteria framework that we develop further utilizes the categorization of different “worlds of services” – equating to different sets of justificatory criteria (Gadrey, 2005; Djellal and Gallouj, 2012). Empirical data has been collected in Finland in the context of public-private development programs in the area of environmental sustainability. The following research questions are guiding our work:
- How and by what means should the outcomes and impacts of environmental services be evaluated in order to take into account the complex and integrative nature that characterizes innovation in these services?
- What are the impacts of environmental service innovations from the perspectives of different societal fields and of different actors?
The results of our study provide a two dimensional approach to assess the impacts of services by combining the multi-criteria and multi-actor frameworks. The focus is on understanding the dynamics of service creation in the environmental sector and the use of evaluation methods and indicators in that sector. At a more detailed level, our study provides analytical material about the complementarities and contradictions between different actors and indicators in service development. Because the links between services and environmental issues have been scarcely studied, our study also contributes to narrowing the gap between these two fields – a point whose importance has been emphasized recently (Rubalcaba et al., 2012; Djellal and Gallouj, 2010; Gadrey, 2010).
The paper is divided into six sections. The second section after this introduction is based on literature and discusses the challenges of measurement and evaluation, particularly in the context of services. The third section presents the two main frameworks that we apply in our study to evaluate the outputs and impacts of services: the multi-criteria and multi-agent frameworks. In the fourth section, our case-study context and methodology are described. The fifth section analyzes the results of the application of the frameworks in the assessment of environmental services. Concluding remarks ends up the paper.
MEASUREMENT CHALLENGE IN SERVICE INNOVATION AND PERFORMANCE The importance of invisible innovation and performance
For two decades, service studies and specifically the studies on service innovation have argued that the traditional tools, indicators and measures do not capture the performance, innovativeness and impacts in services (Sundbo, 1998; Metcalfe & Miles 2000). A central background reason for the existence of the gap is the “assimilation” perspective adopted in the early service research (Coombs and Miles 2000; Gallouj, 1994). This perspective analyzed services innovation as an imitation of technological and manufacturing innovations, i.e. it did not recognize independent innovation possibilities in services. The perspective was based on the traditional definition of innovation as an invention which results from an R&D project (Howells, 2004). The linear, stage-gate model of an innovation process, which was raised to the position of a norm and marketed as a prerequisite for success (e.g. Cooper and de Brentani, 1991), increased the bias.
These arguments and perspectives have been developed in the context of service innovations. However, they are strongly rooted in general innovation theories basing on the Schumpeter’s definition of innovation that has afterwards been regenerated to “the neo-Schumpeterian” theory of innovation or “the broad view of innovation” (Lundvall, 2007). These traditions are interlinked by several common aspects that affect both the definition of innovations and evaluation of their effects and impacts. Important cornerstones are complexity and uncertainty of innovation process, intangible nature of innovation (focus in new solutions and processes) and systemic view of innovation encompassing multiple sources and actors taking part into the innovation process. (Kline and Rosenberg, 1986; Dosi,1988; Lundvall, 1992; Freeman, 1991; Nelson and Rosenberg 1993). Also from the perspective of general innovation literature conclusion has been parallel: current practices in defining innovation and evaluating them do still follow the mainstream linear innovation thinking which simplifies too much the innovation process as well as the complex dynamics between actors contributing innovation (e.g. Arnold 2004, Patton 2011). Thus, it does not take into account the systemic view of innovation (e.g. Smith, 2000; Edqvist, 2005; Ahrweiler, 2010) and does not capture the multifaceted and diversified performance of innovations (e.g. dealing with societal issues such as health, education, sustainable development) (see e.g. Freeman & Soete, 2009).
A broader view on innovation has increasingly gained ground, but the assimilationist origin of the perspective on innovation and services is still influential. In particular, the measurement and indicators of innovation and performance are still largely based on the industrialist and technologist definitions (e.g. Smith, 2000; Edqvist, 2005; Smits and Kuhlmann, 2004; Djellal & Gallouj, 2010, 2013). In addition to the narrow view on innovation, the dominating view on performance is also mechanical and narrow. It is usually linked to the concept of productivity which refers to the linear and mechanistic input-output function (e.g. Djellal and Gallouj, 2010, 2013; Kellog foundation, 2004; Patton 2011). Its’ traditional definition is unable to recognize the “hidden performance” concerning the societal aspects of services and innovations like equality, ecological sustainability and societal well- being. It also often excludes the aspect of social innovation. (e.g. Rubalcaba et al., 2012) Djellal and Gallouj (2010) have described the interaction between performance and innovation by referring to the visible vs. invisible nature of these phenomena. Whereas technology-based innovations are visible, non- technological innovations are invisible. As regards performance, the authors link the visible-invisible dichotomy to short-term vs. long-term influences. Both in scientific and the managerial discussions, short-term influences of performance are often analyzed in terms of productivity and growth. Longer-term influences are increasingly analyzed in terms of environmental or social sustainability. There are four possibilities in the relationship between innovation and performance as Figure 1 illustrates (ibid., 668). The most apparent relation is between visible innovation and visible performance, but visible innovation may also lead to invisible performance by promoting the long term ecological sustainability or societal well-being. Correspondingly, invisible innovation may be a source of visible performance, i.e. growth and productivity, or promote sustainability.
Figure 1. Innovation and performance gap in measurement of services (source Djellal and Gallouj 2010) Opening the specific characteristics of service innovation and performance
After the realization of the narrowness of the assimilation view, two alternatives have gained ground. The first is the differentiation perspective – also known as the demarcation perspective (Coombs and Miles, 2000; Gallouj 1994, 1998). It focuses on the specific characteristics of service innovation and has highlighted the difficulty of recognizing “newness” and its creator in the service context (Preissl, 2000). The second alternative is an integrative or synthesis perspective (Cooms and Miles, 2000; Gallouj 2002) which has become increasingly relevant due to the blurring lines between goods and services. It highlights the production and consumption that focus on integrated solutions and systems.
Recently, these approaches have also been applied when searching better indicators for innovation and performance in services. The peculiar characteristics of services that specifically have been pointed out in this context are intangibility and the central role of interaction; the latter refers to the central role of co-production between the provider, customers and partners. An important implication of intangibility is the difficulty of defining the “unit of output” and differentiating the product from the process. These aspects challenge the definition of innovation and quality improvements in it (ibid.). Interactivity increases the complexity of the development of services innovations (Gallouj et al. 2013). Ignoring these complex and dynamic relationships (cf. Arnold 2004) is often linked to the traditional technologic measures and the linear innovation model (Smith, 2000; Edqvist, 2005; Ahrweiler, 2010). It may lead to the oversimplification of the reality and to the biased understanding – not only of the impacts of services and innovations – but also of their drivers and dynamics (Arnold, 2004).
New aspects in the analysis are inclusion of the time factor and the social nature of services (Djellal and Gallouj, 2013). Time highlights the dynamic nature of services, focusing to their evolution over time. Considering time as a dynamic factor leads to the differentiation of short-term outputs from medium and long-term effects. The social nature of services derives from the fact that the value and benefit of services is always defined by users. This means that different actors have different values, and it is just this multiplicity of values which makes it necessary to include various criteria in evaluation. In addition to immediate users, it is important to take into account the multiple values of indirect users as well as those of different actors participating in the development, (Djellal and Gallouj, 2010, 2013). Thus, in service innovation boundary lines are blurring – not only between products and processes – but also organizations often change simultaneously (Preissl, 2000). These kinds of combinatory innovations and their broader socio-economic impacts cannot be tackled on the basis of the traditional linear and industrialist models.
The double gap has been noticed to cause significant implications to public policies, which are still very technologist oriented and do not take into account the innovation and performance gap included. Thus, the invisible innovation and performance remain invisible in the policy making, causing problems – not only in the performance measurement as such – but also in target setting, and in steering and policy planning. In order to improve the situation, both the visible and invisible aspects in innovation and performance has to be included in an integrative way (Djellal and Gallouj, 2010).
However, the analysis above does not mean that all existing innovation and performance measures and indicators should be abandoned. Instead of that, the need for a more diversified, multi-criteria framework can be identified.
The framework should also be flexible, and besides the traditional economic measures (such as productivity and performance), focus on analyzing the aspects relating to social innovation and use value linked to the performance. This kind of an approach also prefers the concept “evaluation” over the concept of “measurement” (Djellal and Gallouj, 2010, 2013).
A NEW FRAMEWORK FOR EVALUATION Multi-criteria approach to diversify the perspectives of evaluation
These above described challenges in measurement and evaluation of services intend to assess the performance and impacts on the basis of a multidimensional approach which takes into account the special characteristics of services as well as their aspects of quality and social value (Djellal and Gallouj, 2010, 2013; Rubalcaba et al., 2013). Like Djellal and Gallouj (2010, 2013) have argued one potential and diversified way of analyzing the various purposes and outcomes of services is to link to them to the idea of different “worlds”, which are the industrial and technological word, the market and financial world, the relational and domestic world, the civic world, the world of innovation and the word of reputation. Referring to the initial description of worlds (deriving for the economic tradition; Boltanski and Thévenot, 1991; Gallouj, 2002) aim is in giving an insight to different justificatory aspects in the society considering their different principles and values.
In that model the outcomes are evaluated from the perspective of different goals or target areas, encompassing both the traditional measures arising from the industrial and technical as well as market and financial world and the modern evaluation criteria taking into account the needs of knowledge society (e.g. relational aspects such as trust and aspects relating to the responsibility and societal wellbeing). In addition to the different target areas the model focuses on the long time-scale in the generation of impacts by dividing outcomes into the direct, short-term outputs and indirect, long term-outcomes.
We have made some modifications to the original model. Our changes concern especially the aspect of “the civic world” that originally was focused to social relations characterized by the ethical issues such as equal treatment and fairness. In this analysis we suggest to integrate the concept of societal value and responsibility into the idea of civic world and also rename it “responsible world”. Our aim is to better take into the account the aspects of social innovations, environmental sustainability and societal well-being (see Rubalcaba et al., 2012, Djellal and Gallouj, 2013). In the original model also the word of innovation is differentiated. Because in study focuses is on innovations the perspective of innovation world is excluded as a vertical dimension. Instead we clarify the different elements of innovation horizontally by analyzing the core aspect of innovation in the light of each world.
In our study innovations are analyzed based on the following different worlds given the specific justification criteria related to the each of the worlds: Industrial and technical world (Volumes, flows and technical operations); Market and financial world (Value and monetary and financial transactions); Relational world (Interpersonal and organizational relations, trust, quality of relationship); Responsible world (Values like sustainable development, responsibility, equal treatment and justice), Reputational world (Brand and image). The different impact criteria described in the analysis are interlinked and dynamic in their nature, thus they me reinforcing and contradicting each other.
Multi-actor perspective to analyze the impacts of multiple actors involved in services creation The interactive and subjective nature of services (and their implications to the evaluation) leads to emphasize the multi-actor nature of services. In the recent literature the multi-actor perspective has been proposed as an analytical framework to better understand the complex interaction between policy makers (society), public and private service providers as well as users of the services and to help to evaluate the benefits of the services from the different perspectives (Gallouj et al., 2013). By integrating the actors’ perspective to the evaluation we aim to emphasize the subjective nature of evaluation; evaluation criteria are always dependent on actors’ specific values and interests in a specific situation. In addition we aim to pay attention to the interaction of various actors and their values creating an evaluation situation (cf. Giddens, 1987).
Interaction may be operationalized based on the multi-agent framework that has been developed specifically to the Human Side of Service Engineering (2019)
context of public and private innovation networks to develop innovations and services (Windrum, 2013).The central idea in the framework is to study and understand the complex interactions between multiple actors in concerning service innovations. Framework is based on the former work of (Windrum and Garcia-Goñi, 2008; Gallouj and Weinstein, 1997). When the primary work focuses to analyze services based on their characteristics in the later work the core is in actors. Particularly important are the characteristics and preferences of the actors. One of the values of this dimension is that it puts particular emphasis on the public sector’s role in the innovation process. Typically innovation research has been focused on the private sector innovation without putting too much effort to understand the role of public sector in creation of innovations. However, according to Windrum (2013), both political organizations and non-governmental organizations play an important role in innovation generation and dissemination and needs to be considered. Compared to the former model the value added in the recent one is in the recognition of public sector actors and thus the interaction between economic, social and political spheres (ibid.).
In our study we are analyzing the impacts from the perspective of multiple different actors taking part to the service and innovation development. In the analysis the preferences and thus the benefits are emphasized from the viewpoint of society (represented by policy makers), public sector service providers, private sector service providers and end users.
CASE CONTEXT AND METHODOLOGY In our empirical analysis the focus is in the complex combinatory services in the area of environmental analysis and measurement. Paper studies outputs and outcomes from the viewpoint of different worlds and different actors focusing to the Environmental data platform that aims to be a comprehensive platform to support continuous data gathering and real-time environmental monitoring, analysis and reporting. Innovation has been developed in research and development program built on public-private innovation network that aim to promote systemic change in the field of environment in Finland.
We have applied the case study methodology and qualitative approach in our study. The main method in data gathering in our study was face-to-face interviews (30 in total). The main topics focused on the manifestations of new innovative solutions within the programs, factors that promote or slow down their generalization, impacts of the innovations and their evaluation. The interviews were gathered between February and June 2013. The final sample represented actors in the area of environment in a versatile way. It consisted of representatives of private companies (e.g. companies developing environmental and industrial measurement), and universities and other public research organizations.
To analyze and interpret empirical data we applied the modified multi-criteria and multi-actor frameworks. To start we analyzed what are the impacts of innovation in a short and long term from the perspective of different worlds. In the analysis we reflect impacts both from the traditional and modern perspectives. From the industrialist viewpoint typical evaluation criteria are for example technical operations and volumes. Similarly market world uses criteria like value (in financial terms), market share and business transaction. Relational and responsible worlds aim to broaden the focus to take into account the societal value as well as intangible, interactive, social and systemic nature of services. Moreover, world of reputation analyzes innovation from the perspective of brand and image effect. Finally we illustrate in very general level how the impacts vary from the perspective of different actors participating into the innovation creation. In the analysis the role of policymakers, public service providers, private service providers and end-users are taken into account.
RESEARCH RESULTS Multi-criteria perspective for the analysis of new sustainable services
Environmental data platform (table 1) is from an industrial perspective a prototype platform to gather and share environmental data. As a short term output it integrates real-time data sources (e.g. measured data of water quality and satellite data concerning environment and atmosphere) and provides visualized maps based on the data. In a long-term goal is to integrate multiple data sources in it and provide “cloud-based comprehensive solution” to
Proceedings of the 5th International Conference on Applied Human Factors and Ergonomics AHFE 2014, Kraków, Poland 19-23 July 2014 Edited by T. Ahram, W. Karwowski and T. Marek
Table 1. A multi-criteria framework to analyse the impacts of environmental data platform
7
Proceedings of the 5th International Conference on Applied Human Factors and Ergonomics AHFE 2014, Kraków, Poland 19-23 July 2014 Edited by T. Ahram, W. Karwowski and T. Marek
produce and share environmental data. Cloud-based solution would make the access to big data possible. In addition the architecture for real-time monitoring, analyzing and reporting will be created to improve the quality and reliability of environmental information, weather forecasts and warnings. New architecture helps in developing new end-user applications and thus accelerates business start-ups generation. Our respondents highlighted that services are in the central role in the development of comprehensive solution and new information architecture.
In the market world the main characteristic of the innovation is free access to the multiple data sources and especially to public sectors data. According to our interviewees free access to data is considered to be an important social goal and as a starting point for the development of data platform. In a long term goal is in opening new markets based on environmental analysis and in creation of new export possibilities.
Preferences and valuation principles in the relational world highlight the interaction between multiple actors and a role of end users is the development and use of service. Central innovative aspect here is the connectivity via one mediator to multiple data sources. As output goal trust in the public-private innovation networks is enhanced. From the performance viewpoint, that enhances actors’ connectivity and consolidates the networks. Long-term goal is to integrate citizens to the data provision, to better take into account the specific users’ needs and requirements and hence provide personalized environmental data. In a long term users will be integrated part of the environmental data generation. That development removes the clear distinction between the production and use of data (citizens are becoming data producers), which has been considered as a prerequisite for a systemic change in the sector.
From the perspective of responsible world overall sustainability is highlighted as a central value in development of new solution. The other value based aspects highlighted are transparency and the citizens’ equal rights to participate to the data provision. Innovative aspect relates to the open and equal access to public sector data. As a short term goal platform enhances transparency, availability and multifaceted us of public data. As a long-term goal respondents have considered the possibilities to apply new end-user applications for example to the game and school worlds to support the environmental education and thus increase the awareness of environmental issues from the early age. From the performance viewpoint that may increase responsibility as a value in decision making in individual, firm and policy level.
Finally in the world of reputation innovation is analyzed from the perspective of brand and image effect. From a short and long term the aim is in analyzing how the innovation impact to the attractiveness of different actors. The aspects concerning the image and brand needs to be analyzed from the perspective of specific actors and are thus analyzed in details in the following sub-chapter.
To summarize the central findings from the perspective of different type of evaluation criteria following aspects have been recognized. According to our interviews services have important role in renewing the environment sector. To solve the most urgent societal problems, like sustainability, there is need to develop technologies and services as systemic wholes instead of individual technologies or services. The emergence of comprehensive service solutions in renewing the environment and energy technologies can be perceived by traditional measures. However, the focus from the perspective of industrial and financial world is in technical characteristics of the concepts and solutions, not so much in their social and systemic nature.
To capture the interactive and social nature of service development as well as its’ interlinks to the social goals and system level changes the other criteria are needed. Like our example show, relational and responsible worlds are particularly important to make these aspects visible. Like our respondents highlighted the changing roles of data users and producers (“from user to producer”) plays a central role in the renewal and a system level change in a sector. That can be perceived only from the perspective of relational world. In addition our example shows that impacts generated in the different worlds are often interdependent and complementary to each other. Some factors in relational and responsible worlds can be seen as a prerequisite to effects generated from the viewpoint of technical and financial worlds. For example for the development of environmental data platform the transparency and open access to public sector data is seen as a precondition.
Multi-actor perspective in the analysis of new sustainable services In the following discussion and table (2) we exemplify the impacts from the perspective of multiple actors. Central actors taking part into the service innovation development are society (policy makers setting the societal goals and
Proceedings of the 5th International Conference on Applied Human Factors and Ergonomics AHFE 2014, Kraków, Poland 19-23 July 2014 Edited by T. Ahram, W. Karwowski and T. Marek
Table 2. A multi-actor framework to analyse the impacts from the perspective of different actors
9
Proceedings of the 5th International Conference on Applied Human Factors and Ergonomics AHFE 2014, Kraków, Poland 19-23 July 2014 Edited by T. Ahram, W. Karwowski and T. Marek
preferences for the development), public service providers, private service providers and end-users. By integrating the actors’ perspective to the evaluation we aim to emphasize the subjective nature of evaluation; evaluation criteria are always dependent on actors’ specific values and preferences in a specific situation. In the analysis we zoom environmental data platform from the perspective of longer term outcome to exemplify the diversity of the impacts based on the values and preferences of different actors.
Based on the preferences and values of the private service providers it is natural the benefits of the innovation relates to the knowledge transfer, the development of new commercial applications and thus in the creation of new business opportunities. From the financial perspective, that brings new revenues and opens up the access to the international markets. New evaluation criteria enrich the analysis by highlighting the importance of mutual trust between multiple partners, which has been highlighted as a core aspect in development of new innovations and services. In addition it relational world makes visible the opportunity for the new and changing roles and responsibilities when developing and offering new services. Moreover sustainability as a value and as a competitive advantage is has also been highlighted from the perspective of private service providers: a reputation as a responsible company is a competitive advantage for the contemporary companies which may boost their success in international market.
A long term outcome for the public service providers relates primarily to the access and new opening opportunities based on big data. That creates opportunity for the profound and multifaceted knowledge production which enhances more quality environmental data production and based on that creation of new public services. From the perspective of public sector service providers the opening the new markets and research opportunities were highlighted. And like the private sector companies they also found that knowing new actors and creating the mutual trust between the multiple partners for deepening the collaboration are in the part of the main benefits from the process. In the other words, trust can be characterized as an enabling factor in the co-creation of innovation. From the responsibility viewpoint provision of predictive information of weather and environmental changes helps to manage potential environmental risks. Because of the specific role of public sector service providers this perspective is highlighted here. Capacity to provide environmental (e.g. safety environment) and societal impact (e.g. citizens health and good living conditions) belongs to the roles and functions of public service providers.
From the viewpoint of end-users receiving personalized real-time environmental data is the most self-evident benefit of the innovation. The data personalized to your own needs and received via you own mobile phone based on your location may – for example real-time pollen reports for the allergic and asthmatic person – helps people to predict the pollen risks and thus have important health impacts. In a longer term the citizens’ ability to provide data and act as an active and equal data provider and empowered citizen was examined as an important impact. From the relational viewpoint the change in end-users role was recognized as an important impact from individual but also societal point of view: their active and equal role as an environmental data provider creates preconditions for the new innovative and service oriented culture as well as demonstrates the potential for the systemic change in the sector.
Society is serving the multiple actors thus the impacts described here is combination of everything we have represented above. In enabling the creation of new business ecosystems society’s preference is to enhance the emergence of start-ups, help them to internationalize (export share of GDP) and to create the jobs. From the financial viewpoint that brings new revenues for the society and helps in developing societal services (which in turn causes impact in health and living conditions). In addition society has a role to promote and consider more soft and social aspects in the society. For example create new ‘programs’ (meaning devices and educational programs) for the environmental education and thus raising the awareness and responsibility of citizens are the most important impacts from the viewpoint of responsible world. In addition transparency of the society relating to the openness of public data is a self-evident goal for the society.
To summarize, like our analysis show experience of impacts and benefits are strongly related to the preferences and values of actors. Both their goal and motivation to take part into co-creation is different and thus are the experienced impacts. From that perspective it is evident that there are contradictions regarding the benefits of different actors. However, different goals and preferences are in the most cases seen as complementaries which enhances the co- creation. When considering the impacts crosswise from the perspective of multiple actors and different worlds we create more diversified and analytical picture of the generated or potential impacts. In addition analysis makes
Proceedings of the 5th International Conference on Applied Human Factors and Ergonomics AHFE 2014, Kraków, Poland 19-23 July 2014 Edited by T. Ahram, W. Karwowski and T. Marek
visible that in many cases the impacts are strongly interlinked and are systemic in nature.
CONCLUDING REMARKS The purpose of this paper is to study the challenge of the evaluation in the context of the services. Because of their specific nature the traditional evaluation methods and measures are not able to capture neither the diversity of the innovations nor the multifaceted dimensions of performance in the sector. This paper aims to contribute to the need for a more diverse evaluation approach. We focus on the context of service innovation in the sector on environment and energy, in which we develop further multi-criteria and multi-actor perspectives. Multi-criteria framework describes the impacts of new sustainable services and multi-actor framework aims in analyzing the impacts from the perspective of multiple actors taking part into the service and innovation development.
As a result we identified that services have an important role in renewing the energy and environment sector. The emergence of comprehensive service solutions in renewing the environmental sector can be perceived partially the by traditional measures. However the focus from that angle is in technical and measurable characteristics of the concepts and solutions. From the traditional perspective their interactive nature and multifaceted performance (including the social and value based aspects) cannot be captured. To perceive and make visible the societal goals of the service innovations as well as their interactive nature relational and responsible worlds are particularly important. Moreover, our analysis also shows that impacts generated in the different worlds are often interdependent and complementary to each other. Some factors in relational and responsible worlds can be seen as a prerequisite to effects generated from the viewpoint of technical and financial worlds. In other words relational and responsible aspects are needed to generate long-term technological and economic impacts.
From the multi-actors perspective impacts and benefits are strongly related to the preferences and values of actors. Analysis make visible that there are evident (value and motivation based) contradictions between different actors. However, when there is a mutual trust these are seen as complementaries which enhances the co-creation. When considering the impacts crosswise from the perspective of multiple actors and different worlds we create more diversified and analytical picture of the generated or potential impacts. In addition analysis makes visible that in many cases the impacts are strongly interlinked and are systemic in nature.
Regarding to the further studies more studies would be useful to test the generalizability of our results. Furthermore to go deeper to the dynamic relations (systemic nature) of the different actors and impacts the application of systemic modelling methods could be the next step. In addition generating indicators based on these findings would be both interesting and useful.
REFERENCES
Ahrweiler, P. (ed.) (2010), “Innovation in Complex Social Systems”. Routledge, New York Arnold, E. (2004), “Evaluating research and innovation policy: a systems world needs systems evaluations”. Research Evaluation, Volume 13 No.1, pp. 3–17. Cabrera, D., Colosi, L. & Lobdell, C. (2008), “Systems thinking”. Evaluation and Program Planning, Volume 31 No. 2, pp. 299-
310 Chelimsky, E. (1997), “The Coming Transformation in Evaluation”. In E. Chelimsky & W. Shadish, (eds.) Evaluation for 21st century. A handbook. Thousand Oaks. Sage Publications, pp. 1-26. Coombs, R. and Miles, I. (2000), “Innovation, measurement and services: the new problematique”. In J.S. Metcalfe and I. Miles
(eds), Innovation Systems in the Service Economy: Measurement and Case Study Analysis, London: Kluwer Academic Publishing, pp.85–104. Cooper R.G. & De Brentani U. (1991), “New industrial financial services: what distinguishes winners”, Journal of Product Innovation Management,8 (2), pp. 75-90. Cozzens, S. & Melkers J. (1997), “Use and Usefulness of Performance Measurement in State Science and Technology Programs"
Policy Studies Journal, Volume 25 No. 3, pp. 425-435 Djellal, F. & Gallouj, F. (2010), “The Innovation gap and the performance gap in the service economies: a problem for public policy”, in F. Gallouj & F. Djellal, (eds.), The Handbook of Innovation in Services. A Multi-disciplinary Perspective, Edward Elgar, Cheltenham, UK, pp. 653-673 Djellal, F. & Gallouj, F. (2013), “The Productivity in services: measurement and strategic perspectives.” The Service Industries
Journal Volume 33 No 3-4 pp. 282-299 Djellal, F. & Gallouj, F. (2013b), “How public-private innovation networks in services (ServPPINs) differ from other innovation
Proceedings of the 5th International Conference on Applied Human Factors and Ergonomics AHFE 2014, Kraków, Poland 19-23 July 2014 Edited by T. Ahram, W. Karwowski and T. Marek
networks: what lesson for theory?”, in F. Gallouj, & L. Rubalcaba, P. Windrum, (eds.) Public-private innovation networks in services, Edward Elgar, Cheltenham, UK. Dosi, G. (1999), “Some notes on national systems of innovation and production, and their implications for economic analysis”, in D. Archibugi & J. Howells and J. Michie, (eds.): Innovation Policy in a Global Economy, Cambridge: Cambridge University Press. Dyehouse, M., Bennett, D., Harbor, J., Childress, A. & Dark, M. (2009), “A comparison of linear and systems thinking
approaches for program evaluation illustrated using Indiana Interdisciplinary GK-12”. Evaluation and program planning 32(2009) 187.196. Edquist, C. (2005), “Reflections on the systems of innovation approach”. Science and public policy, Volume 31 No. 6, 485-489. Freeman C. (1991), “Networks of innovators: A synthesis of research issues”, Research Policy, Volume 20 No. 5, 499-514. Freeman, C. & Soete, L. (2009), “Developing science, technologic and innovation indicators. What can we learn from past”.
Research Policy, Volume 38, pp. 583-589 Gadrey, J. (2010), “The environmental crisis and the economics of services: the need for revolution. in: The Handbook of Innovation in F. Gallouj & F. Djellal (eds.), Services. A Multi-disciplinary Perspective, Edward Elgar, Cheltenham, UK, pp. 653-673 Gallouj, F. & Weinstein, O. (1997), “Innovation in services”. Research Policy, Volume 26 No. 4/5, 537-556. Gallouj, F. (1994): “Economie de l'innovation dans les services”, Editions L'Harmattan, Logiques. Économiques, Paris. Gallouj, F. (2002),” Innovation in the service economy: The new wealth of nations”, Edward Elgar. Gallouj, F., Rubalcaba, L., Windrum, P. (2013), “Public-private innovation networks in services”. Edward Elgar. Giddens, A. (1987). “Social Theory and Modern Sociology”. Cambridge: Polity Press. Hansson, F. (2006), “Organisational use of evaluation”. Evaluation Volume 12 No. 2, pp.159-178. Howells, J. (2004), ‘Innovation, consumption and services: encapsulation and the combinatorial role of services’, The Service
Industries Journal, Volume 24 No. 1, pp. 19-36. Kellogs Foundation, 2004. “Using logic models to bring together planning, evaluation & action logic model development guide”.
Battle Creek, Michigan. http:www.wkkf.org/Pubs/Tools/Evaluation/Pub3669.pdf Kline, S.J. and Rosenberg, N. (1986), “An Overview of Innovation”, in R. Landau & N. Rosenberg (eds.): The Positive Sum Strategy – Harnessing Technology for Economic Growth, Washington D.C.: National Academy Press. Lundvall, B-Å. (1992), “National Systems of Innovation: Towards a Theory of Innovation and Interactive Learning”, London and
New York: Pinter Publishers. Metcalfe, J.S. and Miles, I. (eds.) (2000), ”Innovation Systems in the Service Economy – Measurement and Case Study Analysis”. Boston, Dordrecht and London: Kluwer Academic Publishers. Nelson, R.R. and Rosenberg, N. (1993), “Technical Innovation and National Systems”, in R.R. Nelson (ed.): National Innovation
Systems – A Comparative Analysis, New York and Oxford: Oxford University Press. Patton, M. Q. (2011). “Developmental evaluation: Applying complexity concepts to enhance innovation and use”. New York, NY: Guilford. Preissl, B. (2000): “Service Innovation: What makes it different? Empirical Evidence from Germany”, in J.S. Metcalfe, & I.
Miles (eds.): Innovation Systems in the Service Economy – Measurement and Case Study Analysis. Boston, Dordrecht and London: Kluwer Academic Publishers. Rip, A. (2003), “Societal Challenges for R&D Evaluation”, in P. Shapira and S. Kuhlmann (eds.), Learning from Science and Technology Policy Evaluation: Experiences from the United States and Europe. Cheltenham: Edward Elgar, pp. 35-59. Rubalcaba, L. & Michel, S. & Sundbo, J. & Brown, S.W. & Reynoso, J. (2012), Shaping, Organizing and Rethinking Service
innovation: A multidimensional Framework. Journal of Service Management, Volume 23, Issue 5, pp. 696 – 715, ISSN 1757-5818, Smith K.R. (2000), “Innovation as a systemic phenomenon: Rethinking the role of policy”. Enterprise and Innovation Management Studies Volume 1 No. 1, pp. 73-102. Smits, R. & Kuhlmann, S. (2004), “The rose of systemic instruments in innovation policy”. Foresight and Innovation Policy,
Volume 172 Sundbo, J. (1998), “The theory of innovation: entrepreneurs, technology and strategy”. Edward Elgar, Cheltenham, UK. Toivonen, M. (2010), “Different types of innovation processes in services and their organizational implications”, in F. Gallouj & F. Djellal (eds.), The Handbook of Innovation in Services. A Multi-disciplinary Perspective, Edward Elgar, Cheltenham, UK, pp. 653-673 Van der Knaap, P. (2006), “Performance Evaluation and Performance management. Owercoming the Dowensides of Policy
Objectives and Performance Indicators.” Evaluation Volume 12 No 3, pp. 278-293 Williams, B. & Imam, I. (eds.) (2007), “Systems Concepts in Evaluation: An Expert Anthology”. Point Reyes: EdgePress of Inverness. Windrum, P. & García-Goñi, M. (2008), A neo-Schumpeterian model of health services innovation, Research Policy, Volume 37
No. 4, pp.649-672. Windrum, P. (2013), “Multi-agent framework for understanding the success and failure of ServPPINs, in F. Gallouj & L.
Rubalcab & P. Windrum (eds.) Public-private innovation networks in services, Edward Elgar, Cheltenhamn, UK
LISTED IN THE AHFE DIGITAL BOOKSTORE CATALOGUE
Explore the complete digital volume
Advances in The Human Side of Service Engineering
Read this individual research paper free through the original publisher, or explore the full digital proceedings book.
ISBN 978-1-4951-2091-6
Explore this ISBN in the AHFE Bookstore ↗Listed in the July 2026 bookstore catalogue; search by ISBN to confirm current availability · Digital book purchase optional