Connecting Evaluation Methods to Artifacts in Design Science Research

Abstract

Design Science (DS) research has emerged as an important research paradigm in the discipline of Information Systems. It is widely used by many IS researchers and practitioners to create and evaluate innovative artifacts to solve identified organizational issues. Artifacts in Design Science can be broadly classified into four categories - constructs, models, methods and instantiations. They are considered as the heart of Design Science Research. As quality continues to be one of the central concerns in Information Systems, the quality of designed artifacts in the Information Systems discipline needs to be rigorously assessed. Evaluation, which is recognised as a crucial element in many research areas, has also been applied to measure the quality and effectiveness of artifacts in Design Science. Since there exist various types of artifacts in Design Science research, methods proven to be effective in evaluating one type of designed artifact may not be appropriate in assessing other types of artifacts. The objectives of this research are to develop a conceptual framework which describes appropriate evaluation methods for various types of designed artifacts and to evaluate the validity and applicability of the conceptual framework developed in this study. Both primary and secondary case studies are conducted in this research. Interviews are conducted with four participants to collect empirical evidence about the artifact evaluation process of a payroll project. In addition to the primary case study, data is collected from four secondary case studies to demonstrate how designed artifacts are developed and evaluated. A matrix representation is created in this study which summarises the design evaluation methods applied in evaluating the artifacts constructed in the selected case studies. Furthermore, results presented in the matrix representation are used to evaluate the validity and applicability of the conceptual framework constructed in this research. One of the primary outcomes of this research is the development of the conceptual framework which connects design evaluation methods to artifacts in Design Science research. The framework is evaluated based on the research findings and five pre-defined requirements. The evaluation shows that the framework has largely fulfilled the identified requirements. The proposed framework is therefore believed to be able to assist DS researchers in selecting the appropriate evaluation methods for different types of designed artifacts and therefore improving the artifact evaluation process in Design Science research.