This paper deals with the inverse problem for the class of orthogonal functions that for the scalar case was introduced by Ellis and Gohberg (J Funct Anal 109:155–198, 1992). The problem is reduced to a linear equation with a special right hand side. This reduction allows one to solve the inverse problem for square matrix functions under conditions that are natural generalizations of those appearing in the scalar case. These conditions lead to a unique solution. Special attention is paid to the polynomial case. A number of partial results are obtained for the non-square case. Various examples are given to illustrate the main results and some open problems are presented.