Algorithmic Intelligence in Research: Prevalent topic modeling practices and implications for rigor in IS and management research

Algorithms and tools originating from the field of computer science offer a great utility in enabling automated data analyses to practitioners as well as academic scholars. Researchers in the disciplines of management, organization, and information systems have increasingly started relying on such algorithms and tools. However, this increasing reliance on algorithmic intelligence has also called for caution from senior scholars in highlighting challenges for researchers, reviewers, and editors in knowledge creation. Despite this caution, the information systems scholarship so far has provided very limited guidance for maintaining high standards of reliability, validity, and generalizability while relying on algorithmic intelligence in research. In this study, we propose a framework to help scholars in mindfully employing algorithmic intelligence in research by alleviating the threats to academic rigor. Our framework is based on the insights from a systematic methodological review of articles relying on topic modeling, which uncovers some problematic practices prevalent in the scholarship that could potentially threaten the academic values. Our paper contributes to the emerging interdisciplinary scholarship on algorithmic intelligence at the intersection of management, organization, and information systems disciplines.