Combining Two Models of Successful Information System Measurement

This paper purposes is to measure successful of Academic Advisory information system by combining two models of information system measurement. DeLone & McLean IS Success Model use to measure the successful of system while COBIT framework is to measure system maturity level. Result of this research showed that the successful of Academic Advisory IS affected by User Satifaction, Quality of Service, Quality of System while Maturity level at 3.7. The result also showed there’s a relation between level of maturity system with the success of system

 ISSN: 1693-6930 Irmawati [13], Diana and Michel [14], Azhari and Melia [15] to evaluate information system maturity level. On the other studies Haryanto and Sarno [17] conduct a research to propose the use of COBIT Maturity Model (CMM) and Structural Equation Model (SEM) to measure the alignment between the University Academic Regulations and Information Technology Goals where the results of this study proved that the alignment measurement using CMM and SEM gave relatively the same results, which described the same priority list of maturity levels of the IT processes.
Based on the exposure of the previous studies above, through this research we combine two measurements of information systems to measure the academic advisory information system. D & M model to measure the success of academic advisory information systems and CMM to measure the maturity level of academic advisory information system also to analyze the relation between the success of academic advisory information system with the system maturity level to find the affected factors of successful information system.

Research Method
According to DeLone and McLean [5], where have been revised [6], the implementation of information systems is said to be successful if organizations get the net benefits of information systems, while the net benefits gained due to user satisfaction in using the system. In this case, user satisfaction in using the system is influenced by the information quality, service quality and system quality.  On the other hand according to The IT Governance Institute [12] the advantage of a maturity model approach is that it is relatively easy for management to place itself on the scale and appreciate what is involved if improved performance is needed.

Measurement and Indicators Development
In this study data collection is done through questionnaire sheet. The distribution of questionnaires was conducted to users of the Academic Advisory Information System. Of the 200 questionnaires that spread as many as 150 were returned with details to measure the success of information systems with a total of 140 users of system users. Meanwhile, to measure the maturity level of the system as much as 10 respondents.
Measurement variables for D & M model used Likert scale from strongly disagree to strongly agree. The scale is indicated by the following criteria: number 1 means strongly disagree (STS), 2 means disagree (TS), 3 means sufficient (C), 4 means agree (S), 5 means strongly agree [18]. Indicators for D & M model shown in Table 1.
On the other measurement, COBIT framework has defined information technology activities in four domain that is Plan and Organize, Acquire and Implement, Deliver and Support, Monitor and Evaluate. Maturity levels in COBIT framework are designed as profiles of IT processes that an enterprise would recognise as descriptions of possible current and future states. The maturity levels scale are 0-non existent, 1-Initial/Ad-hoc, 2-Repeatable but Intuitive, 3-Defined Process, 4-Manage and Measureable, 5-Optimised [12]. The questionnaires to asses maturity level of information system was taken from the statement in each COBIT Maturity level [12].  [20 ] User Satisfaction (USatisfy) Y1 = Easy to use system Y2 = Happy to use system Y3 = informatin availability Y4 = Grows motivation Y5 = System flexibility [19 ] Net Benefits (NetB) Y6 = Performance improvement Y7 = Accelerate the task Y8 = Productivity improvement Y9 = Effectiveness improvement Y10 = Easier the task Y11 = Usefull [19 ]

Proposed Model
In research conducted by Livari [9] provide empirical evidence that the Quality of the System and the Quality of Information does not have a significant effect on the ntensity of Use, but has significant effect on User Satisfaction. This is because the object of research using a mandatory system. Other research conducted by McGill [10] find that Quality of the System and Quality of the Information was a significant predictor to User Satisfaction, but not a significant predictor for System of Use. Academic information system is a mandatory system. Based on exposure above the developed model for this research dropped System of Use variable, as shown in Figure 2.

Results and Analysis
Since this study purpose is to analyze relationship between variables the researchers use Structural Equation Modelling (SEM) to analyze the proposed research model. SEM is a multivariate statistical technique that is a combination of factor analysis and regression analysis, which aims to examine the relationships among variables that exist in a model [21].

DM Model
In SEM, Confirmatory Factor Analysis (CFA) measurement intended to confirm that indicator are valid constructor to its latent variable. The result of CFA measurement showed that all indicators estimation above 0.5 which is fulfilled validity criteria (> 0.5) as shown in Table 2.  .884 valid Y10 .822 valid Y11 .854 valid After finding in confirmatory factor analysis that all indicators are valid to its variable, the next step is to analyze the structural model. At this stage we analyzed the overall model conformity test and the significance of the causality relationship buit into the model. Based on AMOS software calculation we found that Quality of Information (QI) have P=0.51 (see Table 3) which is above the cut off of 0.05 and negative value in relation with User Satisfaction (USatisfy) (see Table 4). Quality of Service (Qserv) and Quality of System (Qsys) have a relation to User Satisfaction (Usatisfy) 0.64 and 0.35 respectively. Furthermore User Satisfaction (Usatisfy) have a relation with Net Benefits (NetB) as big as 0.83 (see Table 4). Because P=0.51 as shown in Table 3 does not meet the requirement and negative impact from Quality of Information (QI) to User Satisfaction (USatisfy) as shown in Table 4 we modify the model as the last model by dropping Quality of Informatin (QI) variable. After dropping Quality of Information (QI) variable the next step is to re-calculate the estimation. The result of the modification model calculation shows that Quality of Service (Qserv) and Quality of System (Qsys) have a relation to User Satisfaction(Usatisfy) 0.61 and 0.33 respectively. Furthermore User Satisfaction (Usatisfy) have a relation to Net Benefits (NetB) 0.83 as shown in Table 5. Overall, the result can be described as follow: Quality of Service and Quality of System have a relation to User Satisfaction although Quality of Service have more strongest relation to User Satisfaction. User Satisfaction have a strong relation to Net Benefits.
Comparing this study to other papers, the results is support previous researches conducted by Livari [9] and McGill [10] that use DeLone and McLean [6] Information System Success Model for measuring successful of information system with results that were only partially proven.

Maturity Level
The data collection in this study was carried out by spreading questionnaires to respondent had meet the criterria of RACI chart. COBIT defines RACI chart as the duties, which are Responsible, Accountable, Consulted, and Informed. The questions of questionnaires is taken from control objectives of Monitor and Evaluate (ME) domain [12]. The result of maturity level based on questionaires on Monitor and Evaluate domain shown by Table 6. There is a gap when we compare between the result of existing maturity level and the expected maturity level. We can see the gap as the Figure 3 shown.

Maturity Level Relation with Information System Success
After we calculate DM model measurement and get the result of existing maturity level of BINUS University Academic Advisory we combine the model to find out if there's a relation between ME domain of Cobit maturity level and Academic Advisory success model. As shown of the Table 7 above we can see there's an impact from Cobit maturity level ME domain to Net Benefits variable which is the impact of successful information system. The result shown that there's a relation between Cobit maturity level to information system success of 0.59 . The result about this study supports the research conducted by Johan and Angelia [8] that there's a relation between the maturity level of system and successful of information system. Table 7. Standardized Regression Weights COBIT ME and DM IS Success Model Estimate NetB <---ME .590

Conclusion
The result of this paper provides affected factors to the successful of Academic Advisory information system. The finding prove that Quality of Information (QI) is not the affected factor to the successful of BINUS University Academic Advisory information system. The success of the Academic Advisory Information System is affected by Quality of System (Qsys), Quality of Service (Qserv), User Satifaction (Usatisfy) and Net Benefits (NetB). Where Quality of System has an impact of 0.33 to User Satisfaction and Quality of Service has an impact of 0.61 on User satisfaction and User Satisfaction has an impact of 0.83 against Net Benefits. In this research variable Quality of Information (QI) has a negative impact of -0.04 on User Satisfaction.
Academic Advisory system maturity level is at level 3.7 where the gap with level 4 is quite small (0.3). However, recommendations are given for improvements to all sub-processes in the ME domain accordance with the COBIT framework documentation [8], especially in subprocesses that have a low enough value (ME1.2.3, ME1.5.1, ME2.2.1, ME3.5.1, ME4.2.1). The result of this research also shows that there is a relationship between system maturity level and the success of information system. In other word maturity level is the affected factor to the successful of information system. However, the relationship between the maturity level of the system and the success of the information system is not very strong relation. The next research will be done by adding more data collection and modification of relevant indicator.