• Trustness.- For the assignment of tasks, requesters trust on contributor's capability to deliver solutions with good quality in terms of effectiveness2. B. The Nichesourcing method Figure 1 shows the nichesourcing method adapted for the completion of complex tasks, such as the identification of sustainability requirements dependencies, for which decomposition is non-trivial and high domain-specific knowledge is required. -PERMISSIONS TO ACCESS A WORKING ENVIRONMENT -DEPLOY LIST OF REQUIREMENTS -DEPLOY CAMPAIGN GOAL IN TASKS -SCHEDULE INTERACTIONS IMPLEMENTATON ORIENTATION -DETERMINE GOAL OF CAMPAIGN -IDENTIFY THE SET OF OBJECTS -IDENTIFY NICHE COMMUNITIES EXECUTION -PILOT CONDUCTION -RUN TASKS ONLINE EVALUATION -ASSESS QUALITY OF INPUTS Fig 1. Nichesourcing methodology, adapted from[12] It consists of four phases: Orientation, Implementation, Execution and Evaluation. All phases must be supported by a software tool. 1) Orientation: a) Determine goal of campaign. The requester first determines what he or she wants to achieve with the sustainability requirements and dependencies identified during a nichesourcing campaign. It is important to remark that the goal has to be formulated in such a way that contributors deem it worthy to invest time into, fitting with their domain interest. E.g., the goal of a campaign could be as follows: “Contribute to the definition of a software sustainability model for assessing software –intensive systems” b) Identify the set of objects to analyze. The requestor carries out this step manually. However, as a dependency 2 Degree to which something is successful in producing a desired result identification task can be time consuming and require specific domain knowledge, requesters must consider a preliminary set of (candidate) requirements to address sustainability. For instance, traditional quality requirements that were identified as relevant for each sustainability dimension through a survey research method [9] could be considered as objects to analyze. Moreover, with the purpose of getting a common understanding among contributors, definitions of sustainability dimensions and requirements must also be given. c) Identify niche communities to select potential contributors. To ensure the feasibility of a nichesourcing study, the set of objects to analyze (i.e., sustainability requirements of software-intensive systems) should match with the expertise of a niche community. It will facilitate to select potential contributors (professionals, senior researchers with background in Software Engineering and Sustainability). Examples of niche comunities in this topic are: members of the Karlskrona Manifesto for Sustainability Design [14], senior researchers from the Requirements Engineering community working on requirements for sustainability IT projects, etc. According to Dijkshoorn et al. [12], a common feature of niche communities is that they can be divided into even more specialized sub-niches. It is useful to identify such sub-niches because it will help select most knowledgeable contributors. Table 2 shows examples of sub-niches (SN) that can be identified based on who can contribute the best professional knowledge needed for achieving the campaign goal. Each sub-niche consists of a set of candidate experts, contributing to at least in two of the four sustainability dimensions defined in previous works [9], [26]: Technical (T), Economic (E), Social (S) and Environmental (Env). TABLE 2. SUB-NICHES DISTRIBUTION PER DIMENSIONS T E S Env T E S Env 0 SN1 0 SN2 SN4 0 SN3 SN5 SN6 0 2) Implementation: In this stage, as shown in Figure 2, tasks are designed and assigned to the selected contributors. This stage consists of the following steps: a) Permissions to access a working environment. Permissions are granted to all contributors who accept participating in the nichesourcing campaign. According to the expertise of the contributor, the corresponding working environment is loaded. For this, it is very important to ask an explicit confirmation for addressing the "skills matching" requirement. b) Deploy list of requirements per sustainability dimension in Tool. In this step, the tool is deployed and all relevant requirements identified in the orientation stage are loaded. All definitions of sustainability requirements and dimensions are also loaded. c) Deploy campaign goal in Tasks. During this step, the goal of the campaign is translated into a set of tasks. With the purpose of ensuring the quality of solutions/inputs, the campaign is organized in a set of sessions. For example, by applying the Delphi technique, the requester can organize the campaign in three sessions as shown in Table 3. TABLE 3: NICHESOURCING TASKS PER SESSIONS Purpose Session 1 Session 2 Session 3 Identify dependencies among each pair of requirements (x,y) that belongs respectively to the sustainability dimensions (A and B) Review your dependency matrix by paying first attention to possible missing or incorrect dependencies Reach a consensus by discussing with your peer(e.g. via chat) to solve conflicts. claim accept claim Available reject claim assign Claimed accept assignment reject assignment Assigned [Continue later] Paused Aborted [discarded] Finished [accepted] [rejected] Fig. 2. Possible states of tasks evaluation stages of nichesourcing break complete resume Activated Accepted provide input Initiated start processing Approved Failed during implementation, execution and d) Schedule interactions between contributors and requesters. To facilitate the interaction between contributors and requesters, the nichesourcing tool must support both synchronous and asynchronous communication. Scheduling synchronous communication is harder because of different time zones and the diversity of work. However, scheduling this type of communication upfront with each sub-niche is needed especially for some sessions (e.g. session 3). 3) Execution: a) Pilot conduction. To test the nichesourcing tool, the formulated tasks, and the corresponding planning, a pilot is conducted with a limited number members of the targeted niche community. b) Run tasks online. Once contributors accept a task, they can start providing their inputs in an asynchronous way. With the purpose of avoiding/reducing aborted tasks, continuous support should be provided to contributors. Moreover, as shown in Figure 2, interruptions might occur during the study. If so, the tool should enable to resume suspended tasks in case requesters decide to pursue it later. Reminders to meet the set deadlines of the campaign should be sent for the task completion. 4) Evaluation: Once a task is finished, the quality of the inputs provided in each session should be verified by the requester. This assessment can be manual or automated. An example of automated assessment is the identification of conflicts in the third session. The requester decides, based on the assessment, to approve or to reject the solution(responses). It is failed if the contributor is unable to deliver the desired output. A task is successful if most of the outcomes of a contributor are approved by the requester. Table 4 shows examples of external applications that can be used for supporting each nichesourcing stage. Moreover, based on our own experience in running a pilot study with 12 experts that implemented the method presented, averaged estimated durations are also proposed. The length of the blue bars corresponds to 8 (A, B, D), 4 (E), and 3 (C,F) working days. The lighter blue bars represent the additional time that should be considered at i) orientation stage when more contributors were needed to be included; ii) execution stage when issues detected in the pilot may require more time to be solved. TABLE 4. ESTIMATED PLANNING OF A NICHESOURCING CAMPAIGN Campaign Support Orientation (A) Implementation (B) Execution: Pilot (C) Run tasks: First session (D) Second session (E) Third session (F) Linkedin, Twiter, ResearchGate Tool, email Tool, Email, chat, V. ISSUES AND CHALLENGES Mao et al.[10] identified a series of issues regarding the application of crowdsourcing in software engineering. Nichesourcing is a natural evolution of crowdsourcing that emerged to address some of these issues shown in Table 5. TABLE 5. STRENGTHS AND CHALLENGES FOR IMPLEMENTING NICHESOURCING IN SOFTWARE ENGINEERING Crowdsourcing Issues [10] Nichesourcing Strengths (S) Task decomposition Skill barriers Motivation Careful planning S1: The assigned complex tasks are carried out in an iterative way. S2: Involvement of contributors with specific knowledge selected from niche communities. S3: Social trust and reputation are key drivers of niche communities. S4: Traditional motivation techniques from crowdsourcing like remuneration are not needed anymore. S5: A flexible planning makes feasible the completion of tasks. CH1: Tools or platforms should provide continuous communication and coordination support for making right assignments of complex tasks, and monitoring contributors during the “n” iterations of the study. One question that arise is: How many iterations would be enough for considering a task as successful? CH2: Matching skills of potential contributors with specific knowledge required for solving a task can demand a lot of effort. Platforms/tools should provide (semi-)automated support for identifying “suitable contributors”. Considering additional selection criteria can be even more challenging, such as availability, enthusiasm or motivation on the theme, that would ensure high levels of engagement. CH3: Keep the motivation of contributors in long-term studies is a well-known challenge, especially when they are locally distributed (cultural aspects should be considered). It is needed to empirically investigate the applicability of new motivators for nichesourcing. CH4: Adapting continuously the planning and schedule according to the availability of contributors could affect to other ones. So, some mechanisms to facilitate agreeing on deadlines and schedule for synchronous communication should be provided. Challenges(CH) Crowdsourcing Issues [10] Nichesourcing Strengths (S) Challenges(CH) Quality assurance S6: Higher quality individual results because nichesourcing tasks are performed by members from socially- trusted communities, CH5: Determination of minimum level of skills and knowledge to ensure an enough group of skilled contributors for each required sub-niche. CH6: Provide opportune support and positive feedback to contributors for increasing the quality of individual results. It also summarizes corresponding strengths and challenges for implementing nichesourcing studies in software engineering. The conduction of the pilot has been helpful on realizing these strengths and challenges. It is also important to remark that although our method has been applied specifically to support the identification of sustainability requirements and dependencies, we consider that the method might be applicable to any type of complex knowledge-intensive task. As a future work, we are going to analyze the collected data from our pilot and apply our method for identifying sustainability requirements of a specific healthcare software. ACKNOWLEDGMENT This work has been partially funded by the Spanish Ministry of Economy, Industry and Competitiveness with the projects: TIN2016-78011-C4-1-R, TIN2016-77158-C4-3-R; and Conselleria de Cultura, Educacion e Ordenacion Universitaria (accreditation 2016-2019, ED431G/08) and the European Regional Development Fund (ERDF). REFERENCES [1] M. Wieczorek, D. Vos, H. Bons. Systems and Software Quality. Springer, Germany. 2014 [2] C. Becker, R. Chitchyan, L. Duboc, S. Easterbrook, B. Penzenstadler,N. Sey, C. C. Venters, Sustainability design and software: The karlskronamanifesto, in: Proceedings of the 37th International Conferenceon Software Engineering - Vol 2, ICSE SEIS, IEEE Press, 2015, pp. 467-476. [3] P. Lago, S. A. Kocak, I. Crnkovic, B. Penzenstadler, Framing sustainabilityas a property of software quality, Communications of the ACM 58 (10) (2015) 70-78 [4] M. Razavian, G. Procaccianti, D. A. Tamburri, Four-dimensional sustainablee-services, in: U. V. A. W. B. R. N. G. Jorge Marx Gmez,Michael Sonnenschein (Ed.), BIS-Verlag, 2014, pp. 221-228. [5] S. Betz, C. Becker, R. Chitchyan, L. Duboc, S. M. Easterbrook, B. Penzenstadler, N. Seyff, C. C. Venters, Sustainability debt: A metaphor to support sustainability design decisions, in: RE, 2015. [6] A. Raturi, B. Penzenstadler, B. Tomlinson, D. Richardson, Developing asustainability non-functional requirements framework, in: Proceedings of the Int. Workshop on Green and Sustainable Software, GREENS 2014, ACM, New York, NY, USA, 2014, pp. 1-8. [7] H. Koziolek, Sustainability evaluation of software architectures: A systematicReview. In: Proceedings of the Joint ACM SIGSOFT Conference-QoSA and ACM SIGSOFT Symposium -ISARCS, ACM, New York, USA, 2011, pp. 3-12. [8] P. Avgeriou, M. Stal, R. Hilliard, Architecture sustainability [guest editors'introduction], Software, IEEE 30 (6) (2013) 40-44. [9] N. Condori-Fernandez, P. Lago, Characterizing the contribution of quality requirements to software sustainability, Journal of Systems and Software,Volume 137,2018, Pages 289-305. [10]Ke Mao, Licia Capra, Mark Harman, YueJia. A survey of the use of crowdsourcing in software engineering. The Journal of Systems & Software (2017) 126:57-84. [11] V. de Boer et al. (2012). Nichesourcing: Harnessing the Power of Crowds of Experts. In Knowledge Engineering and Knowledge Management. EKAW 2012. LNCS, vol 7603. Springer, Berlin. [12]C. Dijkshoorn, V. De Boer, L. Aroyo, G. Schreiber. Accurator: Nichesourcing for Cultural Heritage, Human Computation. pp. 101–131, 2014. [13]H. Zhang, J. Li, L. Zhu, R. Jeffery, Y. Liu, Q. Wang, M. Li, Investigating dependencies in software requirements for change propagation analysis, Information and Software Technology, Volume 56, Issue 1,2014. pp. 40-53. [14]Karlskrona manifesto for sustainability http://sustainabilitydesign.org/ (Last visit 30th December, 2018) [15]D. Geiger, S. Seedorf, T. Schulze, R. C. Nickerson, and M. Schader, “Managing the crowd: Towards a taxonomy of crowdsourcing processes.” in AMCIS, 2011. [16]M. Hosseine, K. Phalp, J. Taylor, R. Ali. The four Pillars of Crowdsourcing: a Reference Model. 2014 IEEE Eighth International Conference on Research Challenges in Information Science (RCIS). Marrakech, Morocco. 2014. [17]E. C. Groen, N. Seyff, R. Ali, F. Dalpiaz, J. Doerr, E. Guzman, M. Hosseini, J. Marco, M. Oriol, A. Perini, and M. Stade, “The Crowd in Requirements Engineering: The Landscape and Challenges” IEEE Softw., vol. 34, no. 2, pp. 44–52, 2017. [18]A. Kittur, B. Smus, S. Khamkar, R. E. Kraut. CrowdForge: Crowdsourcing Complex Work. 24th ACM Symposium on User Interface Software and Technology. Santa Barbara. USA. October 2011. [19]L. Lyu, M. Kantardzic. Evaluating Complex Task through Crowdsourcing: Multiple Views Approach. International Journal of Crowd Science. 1(2). 2017 [20]Q. Zheng, et al. Crowdsourcing Complex Task Automatically by workflow Technology. Management of Information, Process and Cooperation. Springer. 2017. [21]M. Hosseini, K. Phalp, J. Taylor, and R. Ali. “Towards crowdsouricing for requirements engineering”. In Procesedings of the 20th Internationational working conference on Requirements Engineering: foundation for software quality (REFSQ)-Empirical Track, 2014. [22]M. Hosseini, A. Shahri, K. Phalp, J. Taylor, R. Ali and F. Dalpiaz, "Configuring crowdsourcing for requirements elicitation," 2015 IEEE 9th International Conference on Research Challenges in Information Science (RCIS), Athens, 2015, pp. 133-138. [23]L. Layman and G. Sigurdsson, "Using Amazon's Mechanical Turk for User Studies: Eight Things You Need to Know," 2013 ACM / IEEE International Symposium on Empirical Software Engineering and Measurement(ESEM), Baltimore, MD USA, 2014, pp. 275-278. [24]Hosseini, M.; Shahri, A.; Phalp, K.; Taylor, J. & Ali, R. Crowdsourcing: A taxonomy and systematic mapping study Computer Science Review, 2015, 17, 43 – 69. [25]Hilty, L.M., Arnfalk, P., Erdmann, L., Goodman, J., Lehmann, M., and Wäger, P.A. The relevance of information and communication technologies for environmental sustainability: A prospective simulation study. Environmental Modelling & Software 21, 11 (Nov. 2006). [26]P. Lago, S. A. Kocak, I. Crnkovic, B. Penzenstadler, Framing sustainability as a property of software quality, Communications of the ACM 58 (10) (2015) 70-78. design:
ACKNOWLEDGMENT This work has been partially funded by the Spanish Ministry of Economy, Industry and Competitiveness with the projects: TIN2016-78011-C4-1-R, TIN2016-77158-C4-3-R; and Conselleria de Cultura, Educacion e Ordenacion Universitaria (accreditation 2016-2019, ED431G/08) and the European Regional Development Fund (ERDF).