DOI: 10.7341/20171343

JEL CODES: M20, O30, P35 /

Regina Lenart-Gansiniec, Ph.D., Jagiellonian University, ul. Łojasiewicza 4, 30-348 Kraków, Poland, e-mail: regina. This email address is being protected from spambots. You need JavaScript enabled to view it..

Abstract

In our times, a growing interest of organizations, including also the public ones, in crowdsourcing, can be observed. It enables to acquire knowledge located in virtual communities. However, despite many benefits, crowdsourcing initiatives very often fail. Therefore, a need for their evaluation is recognized. Nonetheless, in the subject literature, a shortfall of criteria and methods of evaluating crowdsourcing may be observed. The existing proposals do not ensure a comprehensive picture of crowdsourcing, and they do not take into account its multidimensionality. The article is intended for a presentation of the ways of evaluating crowdsourcing and an original proposal of a list of indicators, which may be used for evaluating crowdsourcing in public organizations. The article presents the original proposal of activities, by which it is possible to assess the degree of implementation of the adopted tasks and determine the level of obtained crowdsourcing results. The conducted research allowed to recognize that it is possible to measure crowdsourcing results using quantitative and qualitative indicators. A prerequisite for selecting the appropriate means is first of all to indicate the purposes for which crowdsourcing should be used.

Keywords: crowdsourcing, effectiveness, measurement, public organizations.

INTRODUCTION

Crowdsourcing is a relatively new notion, but one which is nonetheless raising more and more interest with researchers. In short, it means a selection of functions, which have until present been performed by employees, are transferred in the form of an open on-line call, to an undefined community – the crowd. For many organizations, crowdsourcing is an opportunity to achieve or increase competitive advantage (Rouse, 2010; Whitla, 2009). It is also used by public organizations in their activity. What is important, is that the existing crowdsourcing activity of public organizations in Poland enables one to ascertain that it may generate considerable interest among the citizens and serve as a source of innovations: an example is the Otwarta Warszawa (Open Warsaw) platform: 16, 600 registered users, 1,147 ideas generated by the crowd, out of which 24 have been implemented.

Regardless of the premises for making a decision about crowdsourcing, organizations must be aware of the fact that as a result it may bring some benefits, but also generate some specific losses. Taking into account the high percentage of crowdsourcing initiatives’ failure, it is worth considering measuring crowdsourcing. However, a shortage and fragmentariness in the scope of the methodology of measuring the effectiveness of crowdsourcing may be observed. In addition, organizations often make use of crowdsourcing without fully understating its effectiveness (Bayus, 2013). A lack of measurement may make achieving the goal of crowdsourcing impossible. This subject matter seems to be important – the evaluation of crowdsourcing seems to be of importance to public organizations. Especially since it is even demanded that the actions of public organizations are evaluated (Frączkiewicz-Wronka, 2013).

The aim of this article is to present the ways of evaluating crowdsourcing and an original proposal of a list of indicators, which may be used for evaluating crowdsourcing in public organizations. The article is composed of three parts. In the first one, information on the essence and notion of crowdsourcing and its importance to a public organization is presented. The second part is devoted to measuring the effectiveness of crowdsourcing in public organizations. An original proposal of measures, based on which one may carry out an evaluation of the degree of realization of assumed tasks and specify the level of the achieved crowdsourcing results, is presented in the article.

LITERATURE REVIEW

The essence and notion of crowdsourcing

The first time the notion of crowdsourcing appeared in the subject literature was in 2006 by J. Howe. He defined crowdsourcing as ”the act of a company or institution taking a function once performed by employees and outsourcing it to an undefined (and generally large) network of people in the form of an open call. This can take the form of peer-production (when the job is performed collaboratively), but is also often undertaken by sole individuals” (Howe, 2006).

With time new definitions of crowdsourcing started to appear, including the role of the Internet as a specific moderator (Quinn & Bederson, 2011; Brabham, 2013). It started to be linked with establishing cooperation and relations with virtual communities (Yang, Adamic & Ackerman, 2008), and further making use of their wisdom (Surowiecki, 2004) to solve problems (Vukovic, 2009), creating innovative solutions (Sloane, 2011), and open source software (Rouse, 2010). Selected definitions were presented in Table 1.

Table 1. Selected definitions of crowdsourcing

Date

Author/authors

Definition

2006

Reichwald, Piller

Interactive creation of values: collaboration between the organization and the users in the development of a new product

2008

Chanal, Caron-Fasan

Opening of the innovation process in the organization in order for integration through a competence network

2008

Howe

Act of a company or institution taking a function once performed by employees and outsourcing it to an undefined (and generally large) network of people in the form of an open call. This can take the form of peer-production (when the job is performed collaboratively), but is also often undertaken by sole individuals

2008

Kleeman et al.

Form of integration of users or consumer in internal processes of value creation. The essence of crowdsourcing is an intended mobilization with allocation of commercial exploration of creative ideas and other form of work performed by the consumer

2008

Yang et al.

Making use of a virtual community to transfer tasks

2009

DiPalantino, Vojnovic

Methods while using an open call to encourage communities to solve problems

2009

Poetz, Schreier

Outsourcing of the phase of generating ideas to potentially large and unknown groups of people in the form of an open call

2009

Vukovic

A new production model widespread on the Internet in which people collaborate in order to complete a task

2009

Whitla

The process of outsourcing of an organization’s activity to the virtual community. The process of organising work in which the organization offers payment for realization of tasks by the crowd members

2010

Heer, Bostok

A relatively new phenomenon in which Internet workers carry out one or more micro-tasks, often for a micro-payment ranging from $ 0.01 to $ 0.10 for the tasks

2010

Burger-Helmchen, Penin

The way in which the organization gains access to external knowledge

2010

Buecheler et al.

A specific case of collective intelligence

2010

La Vecchia, Cisternino

Tools for solving problems in the organization

2010

Ling

A new business model of innovation through the Internet

2010

Mazzola, Distefano

Purposeful mobilization through web 2.0, creation of innovative ideas, incentives for problem solving, where users coming forward voluntarily are taken into account by the organization in the process of solving internal problems

2010

Oliveira et al.

A way of outsourcing to the crowd tasks related to the creation of intellectual assets, often together in order for easier access to the necessary palette of skills and experience

2011

Alonso, Lease

Outsourcing of tasks to a large group of people rather than assigning these tasks to the employees or contractors at home

2011

Bederson, Quinn

People devote themselves to perform Internet tasks managed by organizations

2011

Doan et al.

A method of a general purpose of solving problems

2011

Grier

A way of making use of the Internet to employ a large number of dispersed workers

2011

Heymann, Garcia-Molina

Acquiring one or more Internet users to remote performance of work

2013

Brabham

A way of solving problems, as well as a model of production, in which, in order to achieve goals characteristic of an organization, collective intelligence of Internet communities is used.

Source: Lenart-Gansiniec (2017, pp. 25-34); Estelles Arolas & González-Ladrón-De-Guevara (2012, pp. 189-200).

A review of the selected definitions of crowdsourcing enables one to ascertain that it is defined and formulated in various ways in the literature. Despite the proliferation of the considerations on crowdsourcing, there is no agreement as to the definition of crowdsourcing. It is interpreted not only as a way to solve problems (Doan, Ramakrishnan & Halevy, 2011; Brabham, 2008) or a method for collecting ideas (Kleeman, Voss & Rieder, 2008), but also as a phenomenon which accompanies all expressions of the technology Web 2.0 (Andriole, 2010). Crowdsourcing is therefore, a difficult concept, often vague, capacious, and complex (Estellés-Arolas & González-Ladrón-de-Guevara, 2012). Sivula and Kantola, in their accurate formulation of the issue of defining crowdsourcing, mention that it includes the human factor. This means that defining crowdsourcing alone is a challenge for researchers (Sivula & Kantola, 2015).

A synthesis of the existing scientific output enables one to formulate a definition of crowdsourcing. Taking the above into account, based on analyses of various definitions, the following definition of crowdsourcing has been proposed: crowdsourcing is a way to engage by the organization, through an online crowdsourcing platform, a non-specified, dispersed group of people to realise various tasks, whereby each party obtains certain benefits.

Crowdsourcing is a relatively new concept, which is constantly developing – there is however a lack of comprehensive research. According to one of the most frequently quoted researchers of crowdsourcing, Zhao and Zhu (2014), during crowdsourcing measurement three perspectives should be considered, i.e. the participant, the crowdsourcing platform, and the organization. Such an approach to the measurement is also shared by Soliman (2014). Despite recommendations and indications, research is limited to one level of crowdsourcing chosen by the researchers. Not without importance are also crowdsourcing phases. Most often the following phases are pointed out: preparation, initiation, generation, evaluation, and implementation (Gassmann, Daiber & Muhdi, 2010). In the preparation phase the identification of the problem, the defining of tasks which the organization wants to hand over to the virtual community, and defining the target group, all take place. The initiation phase includes: developing a project for collaboration with the virtual community, schedule, preparing an open call to the virtual community, selecting motivators, criteria for evaluating the submitted ideas, and ways to protect intellectual property. The generation phase concerns the incoming ideas, coordination, and entering into interactions with the virtual community. In the evaluation phase verification of the received solutions and ideas according to the criteria defined earlier, selection of the best solutions, and granting awards takes place. The closing stage is the implementation phase in which the organization informs the virtual community about implementing ideas acquired within crowdsourcing, carries out the implementation, possible commercialization, and makes a decision on continuing collaboration with the virtual community.

Crowdsourcing in public organization management

Since 2008 we have been observing tendencies to incorporate crowdsourcing by public organizations into their activity. There are many various crowdsourcing initiatives (Table 2). Taking into account the existing crowdsourcing classifications an attempt was made to integrate them into four categories, types, or areas of usage: (1) Problem solving (collective intelligence, wisdom of the crowd); (2) Rating ready solutions (crowdvoting, crowdrating); (3) Raising money (crowdfunding); (4) Creating creative contents, co-creation (crowdcreation, user-generated content). This division makes reference to the results obtained by other researchers (Hudson-Smith et al., 2009; 2012; Rosen, 2011; Alonso & Mizzaro, 2012; Chandler & Kapelner, 2013; Cabiddu, Lui & Piccoli, 2013; Hossain & Kauranen, 2015).

Table 2. Selected examples of crowdsourcing initiatives realised by public organizations

Type

Examples

How does it work?

Potential Usage

Broadcast search

White House SAVE Award

The organization hands over problems to the crowd asking them to search for ideas and solutions

Identification of new solutions to problems, e.g. improvement of clerks’ work

Peer-vetted creative production

Open Data, Dear Mr. President, Challenge.gov, Change by Us, Amsterdam Opent, Medellin, Otwarta Warszawa, Dobre Pomysły, Next Stop Design, Logo for the Police in Poland, logo for Muzeum Żołnierzy Wyklętych in Ostrołęka, idea for developing a crossroads in Salt Lake City, National Defence Ministry – idea for the name of an army truck, constitution in Iceland, Share an Idea, Ministry of Environment and the Future Commission in Finland: change of regulations of the act on road traffic, Ministry of Justice in Brazil: act on cyberspace, Paris: Madame La Maire, j’ai une idée” (Madame mayor, I have an idea), Plamus, Malaysia: MyIdea (Ministry of Science, Technology and Innovation), Genovasi Challenge (National Innovation Agency), MY Innovation Tree (Malaysian Productivity Corporation), Budget2014 (Finance Ministry), Melbourne (futuremelbourne.com.au),

The organization encourages web users to generate new ideas, solve problems of an image, social, and political nature

Obtaining of ready designs of logotypes, names, plans for developing of urban space, strategies

Knowledge discovery and management

We the People, FixMyStreet, SeeClickFix, NaprawmyTo.pl, San Jose Mobile City Hall, Did You Feel It?, Ushahidi, Kidenga, POPVOX

The organization encourages the Internet community to hand over their opinions, judgments on a given subject, analyse information, notify about problems

Reporting about occurring threats, problems,

Distributed human intelligence tasking

mTurk.com

The organization gives a request to the crowd connected with carrying out of a specific task

Processing, analysing of a big quantity of data, arranging of information, creating registers

Crowdfunding

Citizinvestor, Neighborly, Spacehive

The organization directs to the crowd a request for funds for the realization of an endeavour for the inhabitants

Financing of construction designs, social infrastructure facilities

The selected examples of crowdsourcing initiatives presented above show that crowdsourcing in public organizations is becoming more and more popular. What is more, one may attempt to ascertain that, although in Poland it is in the early development phase, it is becoming almost an obligation abroad (and especially in the United States). The biggest interest is raised by encouraging the crowd to generate new ideas, test products, services, and solve various problems. It seems that crowdsourcing facilitates the process of collective designing. It is a solution which enables the realization of the demands of an open government by public organizations.

Crowdsourcing and its measuring

Effectiveness, both in the vernacular and in the subject literature, is understood and defined in various ways. In the foundation of management sciences, it is defined as an action or way of action which, “leads to an effect intended as a goal” (Kotarbiński, 1969) where the goal is understood as a state of the reality which the entity wishes to achieve through action. And therefore, it is treated as a category, which enables obtaining information about the usefulness of some action in the future. Those actions or ways of acting should be defined as effective, which enable or cause reaching a goal. It should be remembered that effectiveness is gradable and the measurement of effectiveness is the degree to which you reach all the final goals of an action. Therefore, it needs to be borne in mind that the fact of possessing a crowdsourcing platform alone does not decide about the success of the whole initiative. It is important to define the goal, criteria, and measurement indicators (Krawiec, 2014).

One may search in vain the methods related to crowdsourcing in the literature. Only a few publications about this topic may be found in the literature, however they mainly focus on the factors on which crowdsourcing effectiveness depends on. Nonetheless, a statement appears that crowdsourcing actions depend to a large extent on a thought over plan (Krawiec, 2014). In the opinion of Estellés-Arolas and González-Ladrón-de-Guevara (2012) the effectiveness of crowdsourcing requires a simultaneous existence of precisely those three key aspects. This means that crowdsourcing and the tools connected with it must be built taking into consideration concrete tasks and needs. Only such a configuration may contribute to obtaining and making use of the benefits of crowdsourcing, while at the same time eliminating potential barriers or obstacles (Louis, 2013; Cullina, Conboy & Morgan, 2015).

In line with the above, it is assumed that crowdsourcing is effective when the organization has attained the assumed goal. However, it is dependent on intermediate goals, which draw closer to the intended effect – i.e. specific decisions. Making these endeavours by the organizations is dependent on seeing the benefits which may be gained thanks to crowdsourcing, among others: access to talents, external knowledge (Burger-Helmchen & Penin, 2010), valuable information (Greengard, 2011), resources (Brabham, 2008), skills and experience (Oliveira, Ramos & Santos, 2010), mobilization (Zhao & Zhu, 2012), and competences (Chanal & Caron-Fasan, 2008). It may be used for organizational learning, openness of the organization to new external knowledge (Chesbrough & Crowther, 2006; Chesbrough, 2010; Huston & Sakkab, 2006; Feller et al., 2012; Majchrzak & Malhotra, 2013), creating open innovations (Brabham, 2008; Burger-Helmchen & Penin, 2010), building competitive advantage (Leimeister & Zogaj, 2013), improving business processes (Burger-Helmchen & Penin, 2010; Brabham, 2008; Balamurugan & Roy, 2013), optimising costs of the organization’s activity or business models (Garrigos-Simon et al., 2014). The possibility of building crowd capital is emphasised (Prpić & Shukla, 2013; Lenart-Gansiniec, 2016).

In relation to the fact that the current literature conceptualizations related to measuring crowdsourcing do not ensure a full picture of the whole phenomenon (Geiger, Rosemann & Fielt, 2011) – an own, original evaluation tool has been proposed. Considering the fact that crowdsourcing is a complex concept, a two-stage evaluation of crowdsourcing in public organizations may be proposed. The fact that the category of crowdsourcing may be presented in the form of indicators, which enable measuring the level of a given category, was taken into account. One should however bear in mind that not all features can be expressed in a quantitative way, especially when a given notion refers to a real value, which describes real phenomena (Zieleniewski, 1966). The assessment of effectiveness is conditioned by the goal’s formula itself. If the goal has measurable features then the organization has the capability of evaluating the effectiveness of its realization.

The first stage of evaluating the effectiveness of crowdsourcing is based on a “binary” way of evaluating in the sense of a “yes” or “no” answer to the question whether the goal has been attained. This does not, of course, exclude the possibility of graduating the level of realization of each particular goal. In case of the criterion it is achieving the goal alone (Table 3), while expressing crowdsourcing in a holistic way, i.e. the level of the initiator (organizational), crowd (virtual community), and technology (crowdsourcing platform). The phases of crowdsourcing have also been considered.

In the proposal simple measures were developed, which to a large part are of a quantitative and qualitative nature. What is important is that the choice of proper measures is a derivative of the goals that the organization wants to achieve by means of crowdsourcing – and that they should also cover those aspects as they are a priority to the organization. It should, however, be remembered that some indicators work out only in the case of specific subjects of crowdsourcing – the measurement should take into account their specifics. This is particularly important in the case of public organizations. Examples prove that not every crowdsourcing initiative ends with a success. Some of them do not arouse the interest of the virtual community (www.dobrepomysły.krosno.pl), whereas others receive a great deal of attention (www.otwartawarszawa.pl). Moreover, the decisions about purchasing or hiring a crowdsourcing platform by public organizations is connected with utilizing public funds – therefore, it is important to evaluate the effectiveness of the whole action.

Table 3. An original tool for evaluating crowdsourcing effectiveness – audit questions

Crowdsourcing phases

Organizational level

Technological level

Virtual community level

Preparation phase

Has the goal of crowdsourcing been defined?

Has the choice of crowdsourcing type been made?

Has the task directed to the crowd been selected?

Have expectations towards the virtual community been formulated?

Will the organization use the existing crowdsourcing platform?

Has the crowdsourcing target group been defined (sex, age, education, place of residence)?

Initiation phase

Has an open call inviting the community to collaborate been prepared?

Have the tasks and problems been defined properly?

Has a schedule of crowdsourcing actions been developed?

Has a promotional campaign for the project been planned?

Have the regulations of selecting a project for implementation been developed?

Has a system of evaluating the quality of submitted ideas been implemented?

Have prizes for the best ideas been agreed upon?

Have persons responsible for entering into interactions with the virtual community been appointed?

Have procedures related to protecting the organization’s intellectual property been implemented?

Has a way of communication between the organization and the virtual community been developed?

Is the platform easy to operate?

Is it possible to add comments?

Is the platform accessible by means of various devices and operating systems?

Has a system of motivating employees to make use of the knowledge been developed?

Generation phase

Is the idea inflow process continuously monitored?

Are the submitted ideas coordinated?

Does the organization communicated with the virtual community?

Does the organization inspire the virtual community to take action?

Does the organization verify the received solutions?

Are the submitted ideas categorised?

Has the range been measured (platform’s range, number of hits, participation of the target group, number of clicks, number of visits at the site)?

Is the virtual community encouraged do exchange opinions?

Is the virtual community encouraged to submit ideas?

Do the members of the virtual community enter into interactions with other users?

Do the members of the virtual community possess appropriate competences?

Generation phase

   

Do the members of the virtual community collaborate with each other?

Do the members of the virtual community share knowledge among themselves?

Has the measurement of the confidence towards the organization of the virtual community members been made?

Evaluation phase

Have the submitted solutions been verified?

Are the submitted ideas conforming to the assumed criteria?

Has the best solution been selected?

Has involvement been measured (number of registered users, number of entries onto the platform, number of entries/comments, number of clicks, number of added ideas, number of users, number of votes given to entries, number of themes/posts on the forum, number of displays of entries)?

Have the member of the virtual community been informed about the selection of the best solution?

Implementation phase

Has the decision on further collaboration with the virtual community been made?

Has an evaluation of the crowdsourcing platform usefulness been conducted?

Has a decision about the future of the possessed platform been made (need to modify/change the platform /continuation of work on a chosen platform)?

Has the virtual community been informed about the scope of idea implementation?

The second stage of the proposed analysis of crowdsourcing effectiveness consists of a point evaluation of the conditions of this collaboration using a scale from 1 to 7 (1 – “I absolutely do not agree”, 7 – “I absolutely agree”). Two reasons justify the introduction of a 7-point Likert’s scale. Firstly, based on a systematic literature review (Lenart-Gansiniec, 2017), it may be concluded that it is the most popular scale used for crowdsourcing measurement. Secondly, this scale enables one to increase measurement accuracy and to ensure greater transparency and reliability of the evaluation. This stage is a complement to the indicators obtained previously. Based on literature research (Buettner, 2015) the following conditions of crowdsourcing were defined, which should be evaluated qualitatively – its multidimensionality has been taken into account in this respect:

  • organizational level: innovative culture and organizational structure, a positive organizational climate, proactive leadership, openness of the organization to novelties and changes, an appropriate level of employees’ motivation, innovation strategy, coherence of the vision and strategy with the crowd’s aspirations, appropriately shaped relations with external entities, the organization’s trust towards virtual communities;
  • technological level: abilities to capture open and hidden knowledge of the virtual, compatibility and functionality of the crowdsourcing platform;
  • virtual community level: a readiness to share knowledge, the level of external and internal motivation, and an inclination to trust.

The proposed quantitative and qualitative approach in the measurement of crowdsourcing effectiveness may contribute to a comprehensive and reliable diagnosis. The quantitative and qualitative approach is recommended in Brabham’s (2014) literary works.

Nonetheless, it should be remembered that the measurement of crowdsourcing may end with a failure – taking into account the barriers and obstacles which contribute to the organization not being able to achieve the intended crowdsourcing goal. One may include in the barriers at the organizational level the following: communication problems, reluctance to acquire others’ knowledge, a bureaucratic organizational structure, reluctance to crowdsourcing, lack of trust towards virtual communities, difficulties connected with intellectual property protection, process barriers included in administrative processes, fear of changing the business model, and an organizational culture which is closed to innovation. The obstacles at the level of the virtual community are among others the following: a lack of trust towards the organization, a lack of motivation, and a lack of sufficient knowledge or experience. From the process perspective an important role is played by unreliability or an improperly selected crowdsourcing platform, i.e. inadequate and unsuitable for the contextual, relational, and situational needs of the organization (Erickson et al., 2012). Among the potential dangers, one may point to the risk of obtaining low quality ideas developed by the virtual community and reluctance of the crowd towards interactions from crowdsourcing. To minimise them, the key importance is the proper selection of the target group – this will enable the realization of the expectations of both parties, i.e. the organization will obtain useful knowledge, whereas the virtual community will get a task that is interesting to it. In addition, attention should be paid to a suitable motivation system for the crowd and employees, an effectiveness communication between the employees, agreeing a concrete goal and the benefits to be obtained by the organization, building of trust and implementing procedures for securing protection of the organization’s intellectual property.

CONCLUSIONS

The presented deliberations on the measurement of crowdsourcing enable the formulation of the following conclusions:

Measuring crowdsourcing enables making an ascertainment connected with the degree of realization or rather approaching the goal assumed by the organization. This enables the faster achievement of the benefits of crowdsourcing assumed by it. Nonetheless, it is only possible owing to a multi-level approach to crowdsourcing.

The measurement of crowdsourcing is necessary in public organizations. It results from the necessity and pressure put on public organizations, which results from the growing expectations of the citizens. And so, in order that the organization may meet the dynamically changing requirements of its surroundings, it has to evaluate the actions taken by it. It seems that it is necessary to develop a model of assessing its success and introducing mechanisms enabling its permanent monitoring and guaranteeing the expected level. In the author’s opinion an attempt should be made to create a full model evaluation of crowdsourcing undertaken by public organizations, taking into account as precisely as possible the nature and complexity of crowdsourcing and the specificity of public organizations.

The measurement of crowdsourcing causes many problems, since so far no tool has been developed that would make it possible. By the same token, it has become necessary to develop an original tool. The results of crowdsourcing may be measured by means of quantitative and qualitative indicators. The condition for selecting appropriate measures is first indicating the goals for which crowdsourcing is to be used.

Acknowledgments

This project was financed from the funds provided by the National Science Centre, Poland awarded on the basis of decision number DEC-2016/21/D/HS4/01791.

References

Alonso, O., & Mizzaro, S. (2012). Using crowdsourcing for TREC relevance assessment. Information Processing and Management, 48(6), 1053-1066.

Balamurugan, Ch., & Roy, S. (2013). Human computer interaction paradigm for business process task crowdsourcing.Proceedings of the 11th Asia Pacific Conference on Computer Human Interaction (pp. 264–273). Bangalore, India — September 24 – 27.

Bayus, B. (2013). Crowdsourcing new product ideas over time: An analysis of the Dell Ideastorm community. Management Science, 59, 226–244.

Bielski, M. (2002). Podstawy Teorii Organizacji i Zarządzania. Warszawa: C. H. Beck.

Brabham, D.C. (2008). Crowdsourcing as a model for problem solving: An introduction and cases, convergence.The International Journal of Research into New Media Technologies, 14(1), 75–90.

Brabham, D.C. (2013). Crowdsourcing: A model for leveraging online communities. In A.A. Delwiche & J.J. Henderson (Eds.), The Participatory Cultures (pp. 120–129). New York: Routledge.

Buettner, R. (2015). A systematic literature review of crowdsourcing research from a human resource management perspective. Paper presented at the 48th Hawaii International Conference on System Sciences, Kauai, HI.

Burger-Helmchen, T., & Pénin, J. (2010). Crowdsourcing d’Activités Inventives: Une Analyse Critique par les Théories de l’Entreprise. Conférence GECSO. Strasbourg.

Cabiddu, F., Lui, T.-W., & Piccoli, G. (2013). Managing value co-creation in the tourism industry. Annals of Tourism Research, 42, 86–107.

Cameron, K.S., & Whetten, D.A. (1983). Organizational: A Comparison of Multiple Models. New York: Academic Press.

Chanal, V., & Caron, M.L. (2008). How to invent a new business model based on crowdsourcing: The Crowdspirit ® case. EURAM Conference, Slovenia.

Chandler, D., & Kapelner, A. (2010). Breaking monotony with meaning: Motivation in crowdsourcing markets. Journal of Economic Behavior & Organization, 90,
123–133.

Chesbrough, H. (2010). Business model innovation: Opportunities and barriers. Long Range Planning, 43(2-3), 354–363.

Chesbrough, H., & Crowther, A.K. (2006). Beyond high tech: Early adopters of open innovation in other industries. R&D Management, 36(3), 229–236.

Cullina, E., Conboy, K., & Morgan, L. (2015). Measuring the crowd – A preliminary taxonomy of crowdsourcing metrics. Proceedings of The International Symposium on Open Collaboration, OpenSym.

Erickson, L.B., Petrick, I., & Trauth, E.M. (2012). Organizational uses of the crowd: Developing a framework for the study of enterprise-crowdsourcing. Proceedings of ACM SIGMIS Computers and People Research Conference. Milwaukee.

Estelles Arolas, E., & González-Ladrón-De-Guevara, F. (2012). Towards an integrated crowdsourcing definition. Journal of Information Science, 32(2), 1–14.

Feller, J., Finnegan, P., Hayes, J., & O’Reilly, P. (2012). Orchestrating sustainable crowdsourcing: A characterization of solver brokerages. Journal of Strategic Information Systems, 21(3), 216–232.

Frączkiewicz-Wronka, A. (Ed.). (2013). Efektywność zarządzania organizacjami publicznymi i jej pomiar. Studia Ekonomiczne. Zeszyty Naukowe Wydziałowe Uniwersytetu Ekonomicznego w Katowicach, 168, 9.

Garrigos-Simon, F.J., Narangajavana, Y., & Galdón-Salvador, J.L. (2014). Crowdsourcing as a competitive advantage for new business models. In I. Gil-Pechuán, D. Palacios-Marqués, M. Peris Peris-Ortiz, E. Vendrell & C. Ferri-Ramirez (Eds.), Strategies in E-Business. Positioning and Social Networking in Online Markets. New York: Springer.

Gassmann, O. (2012). Crowdsourcing – Innovationsmanagement mit Schwarmintelligenz. München: Verlag.

Gassmann, O., Daiber, M., & Muhdi, L. (2010). Der Crowdsourcing prozess. In Crowdsourcing – Innovationsmanagement mit Schwarmintelligenz (pp. 21.44). München: Hanser.

Geiger, D., Rosemann, M., & Fielt, E. (2011). Crowdsourcing information systems – a systems theory perspective. Proceedings of the 22nd Australasian Conference on Information Systems.

Greengard, S. (2011). Following the crowd. Communications of the ACM, 2(54), 20–22.

Hossain, M., & Kauranen, I. (2015). Crowdsourcing: A comprehensive literature review. Strategic Outsourcing: An International Journal, 1(8), 2–22.

Howe, J. (2006). The rise of crowdsourcing. Wired, 14(6), 1–4.

Hudson-Smith, A., Batty, M., Crooks, A., & Milton, R. (2009). Mapping for the masses: Accessing Web 2.0 through crowdsourcing. Social Science Computer Review, 27(4), 524–553.

Huston, L., & Sakkab, N. (2006). Connect and develop –inside P&G’s new model for innovation. Harvard Business Review, 84(3), 58–66.

Kosinski, M., Bachrach, Y., Kasneci, G., Van-Gael, J., & Graepel, T. (2012). Crowd IQ: Measuring the intelligence of crowdsourcing platforms. Proceedings of the 3rd Annual ACM Web Science Conference ACM.

Kotarbiński, T. (1969). Traktat o Dobrej Robocie. Wrocław: Zakład Narodowy im. Ossolińskich.

Krawiec, W. (2014). Crowdsourcing – czynniki motywujące tłum do działania.
Marketing i Rynek, 4, 14–21.

Krawiec, W. (2014). Skuteczność projektów crowdsourcingowych – planowanie oraz możliwość pomiaru. Marketing i Rynek, 4, 60–66.

Leimeister, J.M., Huber, M., Bretschneider, U., & Krcmar, H. (2009). Leveraging crowdsourcing: Activation-supporting components for IT-based ideas competition. Journal of Management Information Systems, 26(1), 197-224.

Leimeister, J. M., & Zogaj, S. (2013). Neue Arbeitsorganization durch Crowdsourcing: Eine Literaturstudie. Düsseldorf: Hans Böckler Stiftung.

Lenart-Gansiniec, R. (2017). Crowdsourcing – systematyczny przegląd literatury. Przegląd Organizacji, 3, 25–34.

Lenart-Gansiniec, R. (2016). Crowd capital – conceptualization attempt. International Journal of Contemporary Management, 15(2). 29–57.

Uppström, E., & Lönn, C.M. (2013). The promise of a crowd. 19th Americas Conference on Information Systems (AMCIS). Chicago.

Louis, C.A. (2013). Organizational perspectives of open innovation in government. iConference Proceedings.

Majchrzak, A., & Malhotra, A. (2013). Towards an information systems perspective and research agenda on crowdsourcing for innovation. The Journal of Strategic Information Systems, 22(4), 257–268.

Oliveira, F., Ramos, I., & Santos, L. (2010). Definition of a crowdsourcing Innovation Service for the European SMEs. In F. Daniel & F.M. Facca (Eds.), Current Trends in Web Engineering. Berlin/Heidelberg: Springer.

Pszczołowski, T. (1978). Mała Encyklopedia Prakseologii i Teorii Organizacji. Wrocław – Warszawa – Kraków – Gdańsk: Zakład Narodowy imienia Ossolińskich.

Quinn, A.J., & Bederson, B.B. (2011). Human computation: A survey and taxonomy of a growing field. Proceedings of the SIGCHI Conference on Human Factors in Computing Systems. New York, USA.

Rosen, P.A. (2011). Crowdsourcing lessons for organizations. Journal of Decision Systems, 20(3), 309–324.

Rouse, A.C. (2010). A preliminary taxonomy of crowdsourcing. Australian Conference on Information System (ACIS). Brisbane.

Prpić, J., & Shukla, P. (2013). The theory of crowd capital. Proceedings of the Hawaii International Conference on System Sciences 46.

Skrzypek, E. (Ed.) (2000). Efektywność Systemów Zarządzania. Lublin: UMCS.

Skurzyńska-Sikora, U. (2008). Poprawa efektywności organizacji przy wykorzystaniu modelu PEMM. Organizacja i Zarządzanie, 3, 513–523.

Sloane, P. (2011). A Guide to Open Innovation and Crowdsourcing: Advice from Leading Experts. UK: Kogan Page Publishers.

Surowiecki, J. (2004). The Wisdom of Crowds. New York: W. W. Norton & Company, Inc.

Vukovic, M. (2009). Crowdsourcing for Enterprises. SERVICES ‘09 Proceed of the 2009 Congress on Services – I. Los Angeles, CA.

Whitla, P. (2009). Crowdsourcing and its application in marketing activities. Contemporary Management Research, 1(5), 15–28.

Yang, J., Adamic, L.A., & Ackerman, M.S. (2008). Crowdsourcing and knowledge sharing: Strategic user behavior on tasks. Paper presented at Proceedings of the 9th ACM Conference on Electronic Commerce. Chicago.

Zhao, X., & Zhu, Q. (2012). Evaluation on crowdsourcing research: Current status and future direction. Information Systems Frontier, 16(3), 417–434.

Zieleniewski, J. (1966). Efektywność Badań Naukowych. Pracownia Ogólnych Problemów Organizacji Pracy Polskiej Akademii Nauk. Warszawa: PWN.

Biographical note

Regina Lenart-Gansiniec – Assistant Professor at Jagiellonian University, Institute of Public Affairs. An expert in open innovation, knowledge management, clusters and public management of the Ministry of Economic Development (Poland) and Ministry of Economy (Poland). Research interests include open innovation, crowdsourcing, knowledge management, and organizational learning in public organizations. An author of publications on knowledge management, crowdsourcing, open innovation who has participated in several research projects.