Supporting co-creation with software, the idSpace platform

Innovation, in general, requires teamwork among specialist of different disciplines. The idSpace project developed ideas on how teams of collaborating innovators could best be supported. These ideas were embodied in a platform that the project developed. This idSpace platform allows its users to choose between various creativity techniques, pedagogical approaches and context-aware uses of stored information on projects, people and techniques. The platform follows a general process metaphor with specialised modules for specific parts, i.e. it starts with defining the problem to be addressed and through a sequence of steps concludes with a proposed solution. The platform was designed and developed by a multi-disciplinary team. It was evaluated through a realistic usage scenario which focused on the integral platform, from both an end-user and expert user perspective embodying a combination of qualitative and quantitative measurements on usability, general functionality and creativity aspects. This combination, as will be explained, being a powerful way to prioritise and steer the further development of the platform.


Introduction
Increasingly, society in general and industry in particular rely on innovative designs for products and services.
Although such innovations start with the creativity of individuals, these individuals collaborate in teams in which they exchange ideas, share knowledge and informally learn together (Farooq, Carroll, & Ganoe, 2005;Gutwin & Greenberg, 2000).Such teams go through explorative cycles of idea generation, elaboration, evaluation and, ultimately, acceptance of the idea for further exploration or its rejection.They need appropriate tools to generate ideas, reuse them, take them apart, criticise them, or reject them.Currently, innovation processes rely on a large variety of tools that are used mostly independently of each other (Dolog, Lin, Grube et al., 2009).As a consequence, past efforts are hardly taken into account, and little or no attention is paid to the context in which the ideas arose.Furthermore, little effort is made to safeguard and share the results of innovation processes, for future use, in other contexts, or with other people.Any means of remedying this situation, would arguably contribute to product and service innovation, not so much by increasing the number of novel solutions suggested but by improving their quality.This may be achieved by building upon an inventory of solution proposals or a historical record of tried and tested solutions; by taking into account the context in which a team operated, with respect to the team members but also the environment in which the innovation is supposed to be deployed; by replacing implicit and intuitive techniques for fostering collaboration by explicitly tested ones; and finally by acknowledging that the collaborating innovators need to learn, from each other and with each other, with the aid of appropriate pedagogical strategies.
Achieving this kind of improvement of collaborative innovation requires research into how the various elements mentioned -history, context, collaboration, learning -may be blended together in an optimal mix.But it requires above all an artefact that embodies this blend in such a way that real teams of innovators may interact with it and use it for the problems they have, allowing one to evaluate the sensibility of the research results.The artefact thus supports a design-based approach, implementing proposed solutions, testing them and feeding the evaluation results back into the design of a new version.It is this design-based approach to innovation, using theoretical insights and evaluation results to built ever better artefacts (platforms) that the EU-funded idSpace project (www.idspace-project.org)embraced in the realm of collaborative product and service innovation.
The idSpace artefact is a web-based platform; it features an integrative toolset that allows a distributed team of innovators to create new ideas, to contribute to and preserve existing ideas, and to learn about them.It employs techniques for exploring new ideas (e.g.mind mapping in story writing and brainstorming) and for refinement of ideas (e.g.morphological analysis.)The platform supports traceability among stories, mind maps, concept maps, goals, new product features, as well as company values and policies.Its platform also preserves semantic relationships among the different viewpoints for later exploration, retrieval, and navigation purposes.Topic Maps (Dolog et al., 2009;Dols & Siebers, 2009) are used to represent the ideas connected to the problem addressed, their relations, and relevant background information, including e.g.related concepts, product features and experts.
As a consequence of these ambitions, the idSpace platform is both technically and conceptually complex.The evaluation method to be used for a platform this complex needs to take three parameters into account.The first one concerns the artefact itself; obviously evaluation of a prototype needs to be done differently than the evaluation of a final tool or product.The idSpace project, at this stage, was expected to deliver a functioning proof of concept, a first prototype.
The second one the design principle used to develop the platform.A platform designed along a corporate edict model (Kuniavsky, 2003) or a waterfall method, working along clearly defined stages, will most probably conclude with a single evaluation as the last phase before implementation.The evaluation questions in these projects are often best met by involving external experts reporting -sometimes confidentially -to the client.The focus of the evaluation is on the functioning of technology and on the match between the requirements defined by the client and the end product.This approach fits very well a project design where a client, a company or a public authority decides on the requirements and the design of a technological artefact.This method is often used by public governance institutions to evaluate large technological projects.The idSpace platform, however, is developed using a user-centric, design-based approach in which users were actively involved throughout the development.This results in a unique platform because it unites cooperative creation, storage and reuse of creative ideas (Lin et al., 2009).The evaluation method chosen needs to reflect this design-based approach.
The third parameter to be considered in deciding on the evaluation design is the type of usage the platform is supposed to facilitate.The idSpace project is predicated upon cooperation in a multi-disciplinary team.This implies that evaluation is an ongoing -formal and often also informal -process which matches the complexity and the creativity of the development process.If you like, the platform has to be evaluated from the creative perspective of the project itself.This requires a compilation of different evaluation methods to have both an evaluative insight in the platform as a whole and in the individual parts.
In section 2 we will introduce the idSpace platform and contrast it with a few similar systems.Section 3 concentrates on the evaluation design.Section 4 contains a discussion of the results and, finally, section 5 reflects on how the evaluation approach taken enabled the development team's shared understanding and in this way steered and prioritised the next version and why the approach taken may be of use for similar projects.

idSpace: an integrated web platform for creativity
idSpace provides a creativity support environment, it seeks to empower its users to realise creative solutions in distributed collaboration.It features integrative tooling that allows a team to create new ideas, to contribute to existing ideas, and to integrate ideas from others.In addition to this, idSpace facilitates co-creativity via appropriate recommendations to the moderator of a team.. Unlike most creativity support tools idSpace supports the whole workflow of the project team (see Figure 1).The Idea Expander system (Wang, Cosley, & Fussell, 2010), for example, takes into account the socio-cognitive processes that influence brainstorming, but fails to consider creativity as a process in which problem identification, idea generation and evaluation play a role next to idea generation.Also, Idea Storming Cube (Huang, Li, Wang, & Chang, 2007) does not consider the pre-and post-idea generation processes to be parts of the team creativity process.Idea Storming Cube supports creativity through a series (game) of creative utterances assisted by an artificial agent.Again, creative utterances are considered to be of value only in the creative process.In idSpace, the users are supported by appropriate collaboration facilities and context-aware recommendations across multiple phases.For instance, ideas from previous projects will be taken into account and could be recommended by the context-aware recommendation system.Also, recommendations of candidate project members are provided (Sie, Bitter-Rijpkema & Sloep, 2010).The idSpace platform suggests but does not impose a particular workflow.It suggests to start with the project definition and goal definition; these then are followed by idea generation, finding common ground, the selection and evaluation of ideas and finally, the proposed solution.The details of the workflow furthermore depend on the type of pedagogical strategy or creativity technique it recommends.The pedagogical strategies that are provided are Think-Pair-Share (TPS), the Pyramid technique, Progressive Inquiry, and 5W1H (cf.Grube & Schmid, 2008).At present, only 5W1H (Who is it about?, What happened?, When did it take place?,Where did it take place?, Why did it happen?, How did it happen?)one is included.
Figure 1.Workflow visualization idSpace: below the main menu bar each of the main steps (overview to solution) of the workflow can be accessed directly; alternatively, the user can go through the steps one by one.
As an environment for creativity, "ideas" form the primitive building blocks of the system.They may come in various forms and may comprise conventional ideas or resources (links to text, images, video, etc.).Also, an idea may be the answer to a question generated by a creativity technique.For instance, 5W1H may help you expand your set of ideas by asking you the question 'Why?'.Answering this question creates a new idea, linked to another idea by the link named 'Why?triggered by' (Figure 2).
Tooling enables expression of ideas and visualizes them in relation to other ideas via mind mapping functionalities.Ideas can be linked via semantic relationships and one can merge other or previously built conceptual networks of ideas with each other, as can be seen from Figure 2 (Shneiderman, 2007;Dolog, Lin, Grube, & Schmid, 2009).In this way one can link new ideas to existing idea networks and preserve the relationships among different viewpoints.Ideas arise in a context.Therefore, idSpace features a context awareness mechanism (depicted in Figure 3).It provides relevant suggestions to the users.Based on available information on earlier and other projects, the team moderator receives suggestions for relevant resources or people, one can ask for advice or add to the team, because of their expertise for the project under investigation.More detailed, such a recommendation is based on what the system knows about the current state of the project, the information it holds about people, creativity techniques and strategies (Sielis, Tzanavari, & Papadopoulos, 2009).Creativity thrives in multidisciplinary teams, we already established.This, however, implies that common grounding is an inevitable part of any collaborative creative process.In turn, this requires that the various participants to the creative process fill in the gaps in their current understanding, in short, learn.Learning, that is acquiring existing but not necessarily assimilated knowledge, is also needed to venture into novel territory when looking for creative solutions to problems.For these reasons, idSpace also issues pedagogy-based recommendations.Aimed at a team's learning and knowledge building for optimal creative problem solving we refer to this as KS4CC (knowledge support strategies for co-creativity) (Bitter-Rijpkema et al., 2011).They offer tailored advice to the moderator, suggesting actions to take to enhance integrated collaboration, knowledge building and creative processes.Thanks to the context-awareness functionalities, which keep track of the actual state of affairs, idSpace selects suggestions for strategies that help organise the collaborative learning processes.This includes the selection and deployment of appropriate creativity strategy (i.e.matching the actual team situation and the problem at stake).In this way the system makes the team and the moderator aware of the relevant creativity techniques available for their situation.A specific quality of idSpace is that the system not only provides support to -for example -the idea generation phase, by visualising the relation of ideas generated in mind-map format, but it also provides dedicated advice for the selection and actual implementation of the most appropriate creativity technique (Grube & Schmid, 2008).Problem's characteristics (see figure 1) contribute to suggest a certain recommendation.The Jigsaw method (Clarke, 1994;Dimitriadis, McAndrew, Conole, & Makriyannis, 2009) with its suggestion to first investigate sub-problems in sub-groups, brings together team members with the same role in a temporary "expert group" before the final overall integration to solve the whole problem.It requires that the problem is divisible into subproblems.Also, a team's experience with the Jigsaw method influences the recommendation of this method.If the team members have no prior experience with this method, then Jigsaw is unlikely to show up amongst the top recommendations of techniques.

Evaluating the idSpace prototype
Taking into account the user-centric design approach and the platform's focus on creativity, the evaluation should be able to assess both the usability and the usefulness of the platform with regard to creativity support and bring its results as input into the design and development loop.The approach chosen started from a set of evaluation goals, each further elaborated in a set of measurements and measured (Basili, Caldiera & Rombach, 1994;Jordanous, 2012) in three different settings.

Evaluation goals
The evaluation aimed to: 1. evaluate the effectiveness of the idSpace platform as a tool for supporting actively and in a context-aware manner the creation of new ideas.platform.Computers in Human Behavior.Vol. 37, pp 413-422. Available online at: DOI 10.1016/j.chb.2014.04.036 2. evaluate the effectiveness of the idSpace platform as a tool for representation, storage, and management of ideas.
3. assess the usability of the idSpace platform's interfaces.4. assess the user's experience with the interaction.5. identify any specific problems with the idSpace platform.
The first two goals primarily focus on the functionality offered, i.e. to what extent did the platform achieve its objective to support actively and in a context-aware manner the creation of new ideas.The latter three mainly concentrate on usability aspects and user acceptance, given the widely accepted notion that technology acceptance is determined by the combination of perceived usefulness and perceived ease of use of the technology offered (Davis, 1993;Yi & Hwang, 2003).We opted to carry out an expert and a user evaluation concurrently.In this way, we could obtain their possibly different viewpoints at the same time.In our perspective the advantage of having a complete picture of strengths and weaknesses of the platform was more important than the risk to trouble the user unnecessarily with system flaws.It would allow us to select and prioritise changes based on an integrated instead of a waterfall-like perspective.The expert evaluation was split in two parts: a heuristic evaluation concentrating on usability aspects, both general and creativity and collaboration specific, and a test claim validation focussing on the functionality offered.The user-centred evaluation focussed on both aspects.In summary the evaluation methods selected were as follows: 1.A heuristic evaluation comparing the platform with commonly accepted, usability heuristics.

A test claim validation comparing the platform with its original design.
3. A set of case studies to study the users' perception both on perceived usefulness and ease of use.
Having defined our evaluation goals and our evaluation methods (for a complete description see idSpace Deliverable D5.4 & D5.5, 2010), we proceeded to elaborate the details for each of the methods selected, utilizing, in line with the project's phase, a mainly qualitative research design.As stipulated earlier, the leading principle of evaluation was to generate data that can be used in a summative way, enhancing a shared understanding by the team members for the future directions to be taken.

Method and data
In line with the design-based, user-centred approach followed, the ideas about the design of the evaluation framework developed during the project.Most important was the insight that the collaboration of technical experts and scientists in the context of the user-centred design would lead to a practice of formal and informal evaluative feedback.This resulted in a large interest of participants in both the evaluation method and the results on the one hand.On the other hand, it created a strong consensus on the outcomes of the project.We assumed that artefacts such as for example the storyboard suggested a common view about the outcomes of the project.However, at the time of the delivery of the project this common view turned out not to be that obvious or simple to use.For the evaluation perspective this lead to the decision to use a combined method, using an expert-based, i.e. a heuristic evaluation and a test claim validation, and a user evaluation at the same time, to gather information allowing a view as complete as possible.This would allow advising on a next version of the prototype from a complete, formal and empirically sound perspective.

Heuristic evaluation
The main goal of the heuristic evaluation was to examine the interface and to judge its compliance with recognized usability heuristics in order to find usability problems in the design so that they could be addressed as part of the iterative design and development process.The heuristic evaluation was carried out by three experts in human computer interaction (HCI), who examined the interface and judged its compliance with recognized usability principles (guidelines, heuristics, checklists).The experts had a background in HCI design and development, usability testing and creative collaborative processes,.They had not been involved in the development of idSpace.The experts evaluated the idSpace platform based on compliance with the following set of 19 heuristics or guidelines: • Usability Heuristics (Nielsen, 1994).They resulted from studies of practical applications in various contexts and give ten rules of thumb, e.g."the system should always keep users informed about what is going on through appropriate feedback within reasonable time".
• Heuristics for Creativity Support Tools.They (3) came from theoretical work and practical studies on how to support creativity in design (Warr, 2007).An example is: "Does the tool support the creation of new externalisations -divergent thinking?" • Design Heuristics for Computer Supported Collaborative Creativity.Herrmann (2009)

formulated five
Computer Supported Collaborative Work (CSCW) -oriented design heuristics for collaborative creativity.These domain-independent heuristics focus on collaboration support in heterogeneous teams and are used to facilitate creative collaboration, to improve CSCW features, and to compare different settings of creativity support, for instance smooth transitions between different modes of creative collaboration.
• Interference Protection.Concurrent activity is common in a shared workspace such as idSpace; however, it also introduces the potential for conflict.Baker, Greenberg, & Gutwin (2001) argued that users should be protected from inadvertently interfering with work that others are doing, or altering or destroying work that others have done.
Three factors of severity were considered for each usability problem and combined into a single rating (a 5 point rating scale starting with 0 = "no problem" to 4 = "imperative to fix") as an overall assessment of each problem: • Frequency with which the problem occurred -was it common or rare.
• Impact of the problem -was it difficult or easy for the users to overcome.
• Persistence of the problem -was it a one-time problem that users could overcome or would users repeatedly be bothered by it.

Test Claim Validation
The main objective of the test claim validation was to validate to what extent the platform complied with the original set of core functions that had been identified in the idSpace user requirement analysis (idSpace Deliverable 5. 1, 2008;Bitter-Rijpkema, Pannekeet, & Rutjens, 2009;Dolog et al., 2009;idSpace Deliverable D4.3, 2009) as compared to what was expected and agreed for this version.The user requirement baseline identified a set of essential functions that the platform had to cover in order to support actively and in a contextaware manner the creation of new ideas.For validating the compliance of the idSpace platform with those test claims a compliance matrix (Table 1) was available to track and score the compliance.For each requirement applicable the status (compliant, partially compliant, not compliant, compliance cannot be assed so far), the method used (Inspection, Analysis or Test) and a comment for each score for further explanation, can be given.
Table 2 gives a sample of the most characteristic user requirements, derived from the matrix, to allow an impression of the idSpace platform.

C1
The idSpace platform allows users to formulate and clarify the problem to solve: The platform shall have an assistant, template, or form for the easy definition of the problem.C The system shall implement techniques like the Prime Objective technique to achieve goal clarity.C

C2
The idSpace platform supports creation of a user team: During the invitation of participants to an ideation session, the system shall be able to propose experts with a profile relevant to the specific session.NC

C3
The idSpace platform provides context and additional material for the ideation session: The system shall be able to suggest on demand ideation techniques during the ideation session.NC During the ideation session the system shall provide contextually relevant associations.PC

C6
The idSpace platform offers appropriate creativity techniques: The system shall offer a number of appropriate creativity techniques.PC

C8
The idSpace platform supports the participants in sharing ideas: The system shall support the commenting and critique of ideas.C C10 The idSpace platform allows to post-process the outcome of the session: It shall be possible to integrate past ideation outcomes with future ones.PC C11 The idSpace platform supports evaluation of ideas: For the evaluation of ideas, the system shall support the rating and ranking of ideas.PC C12 The idSpace platform is accessible via web browser: The system shall be accessible using a web browser.C C15 The idSpace platform integrates creativity techniques: The system shall provide implementation of different creativity techniques C C16 The idSpace platform allows collaborative learning: The system shall allow collaborative learning.PC C20 An exhaustive user manual is available.
The system shall offer step by step guidelines for the available tools.C C22 The idSpace platform provides contextual help and FAQ: The platform shall be able to provide contextual help and FAQs on each step and throughout all possible situations the user can get into.C

User Evaluation
The main objective of the user trials was to study the users' perception both on the idSpace platform's perceived usefulness and ease of use, focusing on the five aforementioned evaluation goals (cf.Section 3.1).All partners were involved in the organisation and execution of one or more case studies either as observer or as evaluation leader.In order to make the goals operational and measurable, evaluation criteria and test claims were formulated focusing on different key usability, functional and operational aspects of the platform.Both qualitative and quantitative measures of effectiveness, efficiency, learnability, flexibility and user satisfaction were used in the evaluation.To prepare the necessary metrics, we used an approach inspired by the "The Goal Question Metric Method" of Basili, Caldiera, and Rombach (1994), which offers a systematic, stepwise transformation of evaluation goals into metrics.This resulted in fifteen questions with in total 40 metrics for the five evaluation goals (Table 3).following data were collected: • A background questionnaire.This questionnaire included general questions about e.g.age, study, work and gender and creativity specific questions, e.g.experience with creativity techniques and tools.
• A post-test questionnaire.It was organised around three issues, i.e. user interface (e.g."I always knew how to use the creativity techniques"), functionality (e.g."the context-sensitive support -i.e.recommendationsis crucial to ideation") and pedagogical strategies (e.g."Over time, my partner(s) and I came to share more and more ideas") and contained 55 rating scale questions and 7 open questions (e.g."how would you estimate the impact using idSpace in terms of creativity or creative thinking").
• Observer notes.Each session was monitored by a dedicated observer who took notes on the progress made with the scenarios and the way the participants accomplished their tasks.
• System (session) data.The results of the session were available for inspection and analysis.The session data gave access to measurements such as 'ideas generated' and 'ideas accepted' which are used in the evaluation of related systems such as the Idea Storming Cube (Huang, Yeh, Li, & Chang, 2010).
Each of the user studies followed a similar, fixed setup: • User received, in advance, a briefing package with background information on idSpace, the evaluation, and the background questionnaire; • User received, as part of the evaluation, an introduction to idSpace and its system; • User followed a scenario with six pre-designed tasks covering the main functionality of the system in a realistic set-up focusing on a group specific problem statement.up and administrating an ideation session; creating new ideas; exchanging ideas and sharing knowledge; elaboration of ideas (storing ideas, reusing and reworking them) and finally, evaluating and selecting ideas.
Table 3.Some examples of the questions connected to the evaluation goals and their metrics (for the complete set see idSpace Deliverable D5.4 & D5.5 -pp19-21, 2010).

Goal G1 Purpose Issue Object Viewpoint
To evaluate the effectiveness of the idSpace system for supporting in a context-aware manner the creation of new ideas from the viewpoint of the end-user.

Heuristic evaluation
The outcome of the heuristic evaluation concentrated on identifying and prioritising issues connected with the usability of the platform.The heuristics selected, focused the evaluator's attention as they worked their way through the system, executing a set of predefined user tasks, using their expertise to role-play the behaviour of the users.The experts exchanged and discussed their findings in a debriefing session.All results were aggregated and compiled in one final table.In total 19 heuristics were taken into account, the evaluation resulted in "imperative to fix" issues for each of the four kinds of heuristics: • For usability (10 heuristics) a total of 61 issues, 27 of severity 4, were identified that violated the general usability heuristics indicating that the interface of the system needs a careful redesign both with regard to the usage of terminology, the use of more common interface elements, as with regard to the overall screen and portlet design.
PREPRINT Van Rosmalen, P., Boon, J., Bitter-Rijpkema, M., Sie, R. & Sloep, P. ( 2014) Supporting co-creation with software, the idSpace platform.Computers in Human Behavior.Vol.37, pp 413-422.Available online at: DOI 10.1016/j.chb.2014.04.036 • For creativity support (3 heuristics), ten issues, 3 of severity 4, have been identified that violated the heuristics for creativity support tools.They include errors while saving the results of a session and to move data between parts of the system.
• For Computer Supported Collaborative Creativity (5 design heuristics), 9 issues, 1 of severity 4, were identified.The idea associations in the ideas diagram are (at least visually) lost.Also, it is not possible to add illustrative background materials.
• Finally, 2 issues, each of severity 4, have been identified that violated the heuristic for data protection.There is no protection at the system level for loss of data due to co-editing of shared objects.Therefore users will need to fall back on social appointments for editing on data in shared objects.

Test Claim Validation
The objective of the test claim validation was to evaluate the functionality offered by the system, i.e. to what extent the available release of idSpace was compliant with the user requirements baseline (see idSpace Deliverable 5.1, 2008).An important part of the idSpace user requirement baseline should be covered by the validated release.For validating the compliance of the idSpace platform with those test claims, a system walkthrough was performed by two evaluators.The evaluators were familiar with the idSpace requirements.
A total number of 23 test claims covering 50 different user requirements were investigated by the two experts.
Of those, 46% (23) were considered compliant, while 24% (12) only partially complied and 30% (15) did not comply at all.Table 2 gives an overview of a selection of the claims assessed and the resulting status.
Compliancy to the user requirements was mainly found in the areas of sharing ideas, supporting evaluation and for some technical aspects of the system.The validation exercise and obtained results showed that the system, even though it was compliant with 23 requirements, was partially or non compliant with a significant number of requirements including requirements that were considered core functionality including the provision of contextual awareness, integration of appropriate creativity techniques, post-processing of ideation outcomes and supporting effective communication and exchange between participants.In addition, the validation indicated that part of the core functionality may have to be reconsidered.Some of the requirements, e.g.related to communication may be unnecessarily ambitious because of existing, widely available, good alternatives such as forum, email or Skype.

User evaluation
The system was trialled in six user studies with in total thirty users, representative of the idSpace end-user community.All users had completed an academic study; the majority claimed to have experience in at least one creativity technique and to have at least some experience in distributed sessions.Two of the six groups consisted of members of different organizations, not knowing each other in advance, so working on different locations.
The other four teams consisted of members of one organization.The pedagogical strategies selected ranged from TPS, Pyramid to Progressive Inquiry.From the creativity techniques only 5W1H was selected.All sessions made use of Skype -in addition to the idSpace system -to support the communication.Finally, a variety of problem statements was addressed.For each group, in advance, a realistic, challenging problem statement was defined related to the groups' background.This resulted in three business and three research oriented ideation sessions with the following problem statements to explore: • New ideas for potential products or services of Company 1.
• Diversification strategies for the turnover of Company 2.
• New features for a software product owned and developed by Company 3.
• Themes and topics for the forthcoming TMRA (Topic Maps Research and Applications) conference.
• New mobile applications for learning of working professionals.
The main objective of the user trials was to study the user performance and appreciation of the idSpace platform focusing on the five evaluation goals.Overall, the following aspects were identified as strong points of the platform: • The access to the platform via the web -without installation requirements, which was judged easy and straightforward.
• The structured, distributed and collaborative approach to ideation.
However, many small and big issues were identified in the user sessions that had a negative impact on the overall appreciation of the platform.The Post-Test Questionnaire centred around four major indicators of user experience, i.e., effectiveness, efficiency, learnability and user satisfaction indicated that on average -in line with the task observations -the score was negative for all criteria.Three sessions yielded clear negative results, two were moderately negative, whereas the outcomes of one session were neutral.The impact of idSpace on creativity and collaboration was undecided.Three sessions were positive and three negative about the impact of idSpace.As a result, overall, the evaluation goals were assessed as follows: • The effectiveness of the idSpace platform as a tool for supporting actively and in a context-aware manner the creation of new ideas.The overall performance of the users pointed out that the idSpace platform was not (yet) effective.The users were able to create ideas and, on average, indicated that they appreciated the structured approach to ideation.Moreover, three of the sessions indicated a positive impact on the creativity and on collaboration.However, all users clearly suffered from errors and interface problems, which kept them from completing the ideation process.
• The effectiveness of the idSpace platform as a tool for representation, storage, and management of ideas.
The representation, storage and management of ideas were only partly working.The users had to jump backward and forward to keep an overview.Sharing was difficult and there was no reporting of the achievements.
• The usability of the idSpace platform's interfaces.As discussed above the user interface did not suffice, parts of the functionality were not suitably organised within the platform and terminology was not clear, moreover, the support provided by idSpace was too limited.
• To identify any specific problems with the idSpace platform.System errors occurred and/or parts of the platform did not respond (in particular the contextual recommendation portlets).
• Finally, the user's experience with the interaction and the overall outcome of the user validation can probably be best summarised by two user quotes: -"I can feel that there is a potential for a useful tool to be developed to help ideation" -"The prototype nicely presents the core idea of the project.But there are many usability issues in the software and some logical problems in the ideation and solution screens" So, despite the many errors and usability issues, the platform was perceived as potentially useful.

Discussion and conclusions
The evaluation results of idSpace indicated that the users appreciated the access to the platform via the webwithout installation requirements -and the structured, distributed and collaborative approach to ideation.
However, many small and big issues were identified in the six user sessions, the heuristic evaluation and the test claim validation.Collectively, they had a strong negative impact on the overall appreciation of this release.The results indicated that each of the four major indicators of user experience, i.e., effectiveness, efficiency, learnability and user satisfaction scored negative.Whereas the first design iteration was successfully concluded with detailed shared storyboards description of the platform, this release of idSpace, typically, suffered from being the very first, fully integrated software version of the platform with as a result various loose ends, bugs and inconsistencies.Apparently, the complexity of the idSpace platform did not allow for a one step smooth integration.
Obviously, this lack of usability experienced may easily affect the evaluation of the purported creativity enhancing aspects of the platform Nevertheless the impact of idSpace on creativity and collaboration was perceived in half of the case studies as positive.The positive opinions were given by users who did not know each other in advance and, given their geographical distribution, only would be able to work at a distance.This might indicate that they, having no face-to-face alternatives, were more open to new ways of working and looked at the potential of the evaluated platform and therefore judged its clear flaws less stringently.The evaluation consisted of an expert evaluation (2 experts) following the design of the case studies and two targeted evaluation studies (11 and 20 users respectively).The expert evaluation reported that both effectiveness and usability of the platform had been clearly improved.Many issues had been repaired and important, missing functionality had been added.The targeted evaluation studies focused on an important feature of the platform i.e.
the automatic recommendation e.g. to identify new ideas, show solutions to related problems and show suggestions for additional team members.In an earlier evaluation round, due to a combination of technical problems and a lack of data to reason upon, no useful suggestions had been proposed.In the present round, which build upon previously entered data, the recommendations were received positively indicating that recommendations are an important part of the platform's functionality (Schmid & El-Sharkawy, 2011).
Overall, the results of the evaluation indicate that the idSpace platform performed at a level ready for a final large scale evaluation prior to deployment.A strong point of idSpace is that it offers an integrated toolset.
Nevertheless, we have to take into account that being creative is a cognitively demanding task and having to move to the unfamiliar settings of a new platform therefore may detract from the ability to be creative, let alone learn to be creative (Kirschner, 2002).Therefore in future work we will also explore if it is possible to add the functionality offered in apps or widgets which extend the existing platforms of choice of the users.As stated in the introduction, the evaluation approach followed was clearly successful in contributing to the iterative design and development process of a complex project such as idSpace.Its success based on a combination of simultaneously testing the platform's usability, general functionality and creativity support on pre-set objectives, a set of connected measurements and measuring (Basili, Caldiera & Rombach, 1994;Jordanous, 2012) with both users and experts.The storyboard created a common view about the system to be developed.Not surprisingly, the transition from storyboard to integrated system was still a complex step.As a next step, in line with the user-centered design followed, we combined a distant expert observer perspective (usability and test claims) with a highly participatory case study method in our evaluation approach.
Participatory in the sense that all core platform designers and developers were involved in the case studies as observer or evaluation session leader.Together, this resulted in a balanced and, very important, a shared judgement.The researchers and developers involved did share the analysis and agreed with the list of identified issues, independent of their contribution to the platform.As a result, instead of a complex discussion on the evaluation outcomes on the one hand and the required achievements on the other hand, the project team could concentrate on its next cycle and relatively quickly (less than 2 months) iterate successfully to a new (accepted) version.Apparently the combination of objective criteria and the direct involvement of all researchers and developers did create the conditions of a shared perspective on the achievements and shortcomings of the idSpace platform.Therewith it enabled a smooth transition to the next R&D cycle.Typically, the evaluation approach of this kind of projects only applies one of the evaluation methods followed at a time.The advantage, obviously, is that users are as little as possible confronted with system flaws.The disadvantage is, however, that users and experts may differ in perception.In our case this was salient in the perception on the possible impact of the platform.Part of the users (see above) had a different opinion on the impact of the platform as compared to the other part and the experts.By putting the designers and developers themselves in a central user-alike position and accompanying their findings with two independent expert-based analyses did ease their understanding of the current platform state and acceptance for the work to be done.In this case, the sum of the individual three evaluation approaches was more than its individual parts.It well aligned with the first, less formal design iteration based on storyboards.Again, all project staff did operate and act from one shared, this time functioning, platform perspective to conceive the next iteration.Further research has to show to what extent the evaluation approach is applicable and successful in similar projects and, additionally, to what extent the disadvantages of our approach can be accommodated.

Acknowledgement
We would like to thank our idSpace peers for our collective work and discussions that inspired this paper and all people that contributed to the evaluations, in particular Lynn De Proft and Rani Pinchuk who had a crucial input both to the preparation and the execution of the evaluation.The present work was carried out as part of the idSpace project on Tooling and Training for collaborative product innovation (http://idspace-project.org).This Community is not responsible for any use that might be made of its content.

Figure 2 .
Figure 2. The 'Ideation' screen in which ideas can be generated, linked and expanded.Mind mapping features are offered, and an overview of current active team members is shown.

Figure 4
Figure 4 provides an example of such a recommendation.Note that the recommendation of a creativity technique, collaboration strategy or a learning strategy has to take several context dimensions into account.

Figure 4 .
Figure 4. Example of a recommendation for a specific strategy (in this case the use of the 5W1H strategy) project is partially supported by the European Community under the Information and Communication Technologies (ICT) theme of the 7th Framework Programme for R&D (FP7-IST-2007-1-41, project number 216799) This document does not represent the opinion of the European Community, and the European