Your browser version is outdated. We recommend that you update your browser to the latest version.

Evaluating the knowledge of experts in the maritime regulatory field


“This is an Accepted Manuscript of an article published by Taylor & Francis in Maritime
Policy & Management on 2017, available online at the Taylor & Francis Ltd web site: .

Regulations are introduced by the International Maritime Organization (IMO) into the
maritime industry as the result of safety accidents and/or pollution incidents. When there
is lack of historical data, then the IMO appoints experts in order to collect information
regarding the costs and benefits generated to a stakeholder of the maritime industry once
implementing a maritime regulation. Therefore, the role of experts in providing
qualitative and quantitative information is crucial with respect to the quality of the
maritime regulatory process within the IMO or other regulatory authorities. In this article,
a methodology is proposed involving common criteria in determining the expertise of an
individual in the maritime regulatory field. As essential part of the research methodology,
analytic hierarchy process (AHP) is utilized to determine the expertise of an individual
based on his/her own judgements. The regulatory authorities and other stakeholders of
the maritime industry may use this method when selecting experts for decision-making.
In this article, a simulation is carried showing the potentials of the AHP methodology in
expertise evaluation followed by a case study.
Expert knowledge; analytic hierarchy process; maritime regulations
1. Introduction
In January 2013, the total volume of goods loaded worldwide was 9.6 billion tons by a
world fleet reaching more than 1.6 billion deadweight tons (UNCTAD 2014). These
figures show how important is the sea trade for the wealth of the nations. The carriage of
goods by sea has lately been affected by the rising influence of developing countries, and
the containerization (Zhao 2016). With respect to global trade, the development of an
appropriate framework of regulating safety at sea is governed by a variety of international
conventions introduced by the International Maritime Organization (IMO). The IMO
succeeded in producing a high number of conventions, codes, and circulars, which are
referred in this article as the ‘Maritime Regulations’. However, the implementation of the
maritime regulations in many geographical regions is slow and inadequate (Lambrou et
al. 2008; Sampson and Bloor 2007; McMahon 2007; Karahalios et al. 2011). A main
result of this inadequate implementation is the existence of a multiform regulatory



The IMO recognizing the need for uniform implementation of maritime regulations
promoted the Formal Safety Assessment (FSA) as a part of the regulatory process (IMO
2000). The FSA method was proposed to the IMO by the UK’s Maritime and Coastguard
Agency, which was accepted as an essential tool to evaluate maritime regulations (Ruuda
and Mikkelsen 2008). However, an outcome of the FSA method is very likely to depend
on the different data that will be selected and evaluated (Rosqvist and Tuominen 2004).
Regulations are usually introduced into the maritime industry as the result of accidents
and/or pollution incidents. The historical records of those incidents are often used to carry
out risk analysis studies. However, the validity of recognized risk analysis methods
depends on experienced analysts (Arslan and Er 2008). Furthermore, in global relevance,
there is no certain consensus on the statistical distribution on the causes of maritime
accidents due to the different viewpoints of accident analysis and investigation
approaches (Celik, Lavasani, and Wang 2010). Consequently, the safety standards of a
ship are usually determined by the agreement among experts (Lee 2006). An example to
illustrate this issue is the differences between the FSA conclusions made by Greece and
the UK in respect of double-skin bulk carriers’ efficiency (IMO 2004). Therefore, the
role of experts in providing qualitative and quantitative information is crucial with
respect to the quality of an FSA (Rosqvist and Tuominen 2004). Consequently, there is a
need to identify individuals with expertise in maritime regulatory field. Those individuals
would participate in FSA studies carried out by the IMO or other maritime stakeholders
involved in maritime regulatory enforcement.
The use of experts to collect information is broadly used in the private organizations of
maritime industry. The introduction of a maritime regulation affects many areas in the
maritime industry and consequently a group of experts may have different academic or
industrial background (Karahalios et al. 2011). For instance, ship operators dedicate risk
managers as system experts to read accident reports or use algorithms to identify the
causal networks of the accidents (Hyttinen, Mazaheri, and Kujala 2014). Furthermore, at
a minor level, ship operators need to recruit such individuals that would assist in
compliance with maritime regulations and train their co-workers. As in any other firm,
the know-how is achieved by the use of in-house, or the combination of in-house and
external expertise (Monteiro, Neto, and Noronha 2014). Alternative quality of services
may be significantly improved by having an expert readily available to users when
human experts are in short supply (Liu et al. 2014). A continuous advance of quality
services is a dimension that affects the performance of a ship operator (Pantouvakis and
Karakasnaki 2016). The constraints that affect the future size of large container ships
have also been examined with the assistance of experts (Gomez et al. 2015).
The aim of this article is to develop a methodology suitable of measuring the expertise of
an individual with respect to his/her knowledge on maritime regulations. Section 1
presents a discussion on the significance of choosing experts in the maritime industry
regarding maritime regulations. A critical review regarding the measurement of expertise
is presented in Section 2, and Section 3 introduces a proposed methodology in measuring
the expertise of individuals. The applicability of the proposed methodology is
demonstrated as a simulation, which is shown in Section 4. Eventually in Section 5, a
case study is carried out, and in Section 6, conclusions are presented.


2. Literature review on expertise
The choice of an expert in the maritime regulations field is a crucial issue for safety at
sea. Fenstad, Dahl, and Kongsvik (2016) mentioned that there has been considerable
discussion on how well ‘subjective’ safety evaluations made by lay persons correspond to
‘objective’ safety estimated by risk analyses experts. However, to date, there is not an
acceptable standard to evaluate the knowledge of an expert for a specific topic. Relevant
literature is presented in Section 2.1. Nevertheless, there is significant research with
respect to measurement of the expertise of an individual, which is discussed in Section
2.2, that could be applicable in maritime industry as well.
2.1. Criteria for determining an expert
One of the most common criteria in determining the expertise of an individual is the
years of his/her working experience in a field. Experts are usually determined based on
the length of their experience and reputation (Rassafiani et al. 2008). Supporting evidence
for working experience as a main criterion of an expert can be mainly found in a variety
of industries. For instance, initial experiments of Malhotra, Lee, and Khurana (2007) in
distinguishing experts have been encouraged, even though results are only indicative
given a small sample of 20 participants. Malhotra, Lee, and Khurana (2007) in their
research indicate that decision quality practicing managers in the oil and gas industry
substantially depend upon the decision-makers’ level of expertise. Their analysis shows
expertise to be highly correlated with the breadth of experience and poorly with
experience in the same working environment. Another method for determining expertise
is to look at how individuals make decisions in their area of expertise (Rassafiani et al.
2008). On the other hand, the experience of an individual may not be a sufficient factor to
identify this person as an expert. A variety of studies have shown that increasing
experience is not always associated with better judgements. For instance, Witteman,
Weiss, and Metzmacher’s (2012) results showed that novice counsellors performed
almost at the same level as very experienced counsellors.
Certification is another mean of measuring expertise of an individual. Proof of
knowledge, education, qualifications, and training could be presented in the way of
transcripts, diplomas, or certificates of achievement. Clearly, the objective expertise
items are evidence based (Germain and Tejeda 2012). However, Pauley, O’Hare, and
Wiggins (2009) in their attempt to measure aeronautical
experience found that the degree of expertise was not significantly correlated with age,
level of certification, or holding an instrument or instructor rating.
An individual in order to be qualified as an expert should be consensus with other
experts. Consensus appears to be the most important criterion in deterring expert
judgement although the differences in judgement should be retained (Martin et al. 2012).
Holm et al. (2014) proposed that consensus is a necessary condition for expertise. The
experts in a given field should agree with each other if they do not, then it suggests that at
least some of the candidate experts are not really, what they claim to be (Weiss 2003; 

Holm et al. 2014). Consensus reliability (that was found between experts) can be used to
identify or confirm experts; however, this can be confounded by artificial consensus such
as group thinking in which a group of experts may agree on a course of action without
this necessarily being the most appropriate (Rassafiani et al. 2008). Consensual answers
have often been proposed as surrogates for correct answers, although the logic of doing
so has been criticized (Weiss and Shanteau 2003). An approach that is useful for
determining the degree of consensus within a scientific community, and for exploring
collective views on ranges of uncertainties, is to conduct a structured expert elicitation
with formalized pooling of opinions (Bamber and Aspinall 2013). The gist of the
criticism is simply that people may agree on poor answers (Weiss et al. 2009). However,
in a variety of studies, consensus is considered of paramount importance. Wang et al.
(2009) used the fact that their team members reached a consensus as acceptable due to
the difficulty in precisely assessing risk factors and their relative importance weights in
their study. In a similar way, the judgements of the experts on a topic are determined after
long-run discussions to ensure the different points of view as final group decisions in a

consensus (Celik, Deha, and Ozok 2009; Yang, Bonsall, and Wang 2011). In brain-
storming methods, such as the Delphi method, typically a forecast is generated for many

events, without providing any direct information as to the respondents’ views as to
whether questioned events are interrelated. Thus, it is possible that the outcomes of a
Delphi implementation could produce a forecast of events mutually reinforcing or
exclusive of each other, and that an artificial consensus may be reached (Scapolo and
Miles 2006).
Discrimination refers to the person’s differential evaluation of similar stimuli, and
consistency refers to the person’s assessment of identical stimuli over time (Rassafiani et
al. 2008). Experts must be able to discriminate between different stimuli, and they must
be able to make these discriminations reliably or consistently (Lee et al. 2012). Protocols
for measuring expertise in terms of these two properties are well developed and have
been applied in settings. However, because these protocols need to assess discriminability
and consistency, they have two features that will not work in all applied settings. Some
researchers have reported substantial work in the area of expertise, arguing that only
people who are able to (a) differentiate between similar, but not identical, cases and (b)
repeat their judgement with consistency qualify as ‘experts’ (Malhotra, Lee, and Khurana

2007). However, they acknowledge in their methodology it is possible (even for non-
experts) to obtain inappropriate high scores using a consistent, but incorrect rule.

An expert must make consistent decisions when repeatedly faced with the same or similar
cases (Witteman, Weiss, and Metzmacher 2012). Many researchers suggest that the
consistency of an expert’s answers is an indication of his expertise (Weiss 2003;
Shanteau et al. 2002). Weiss (2003) suggested that a key element to determine the
expertise of an individual in a certain area is his ability to be consistent at his judgements.
Therefore, a further criterion that should be considered for an expert is the consistency of
his judgements over the time.


2.2. Measurement of expertise

In applications that generate numerical data, discrimination and inconsistency have been
operationalized using terms familiar from analysis of variance (Weiss et al. 2009). An
experimental design suitable for analysis may be as simple as the presentation of each of
several stimuli more than once (Weiss et al. 2009). For instance, Witteman, Weiss, and
Metzmacher (2012) applied the Cochran–Weiss–Shanteau (CWS) index, which assesses
the ability to consistently discriminate. The team of researchers named the assessment
tool after their respective initials, CWS (Cochran, Weiss, and Shanteau). The CWS was
Cochran, Weiss, and Shanteau’s approach to assessing expertise purely from data. The
CWS index calculates expertise as the ratio between discrimination and inconsistence of
an expert. However, the index can only be interpreted relatively, not absolutely.
Consequently, the CWS can be used only to determine which of two candidate experts is
performing better (Weiss and Shanteau 2001). Germain and Tejeda (2012) as an
alternative method proposed a psychometric scale. This scale consists of 18 items that
measure the expertise of an employee. However, as a limitation, the authors noted that
the samples were relatively small and instruments can be improved only through rigorous
field testing. It was further suggested that future research should explore the proposed
measures to alternative structures that may range from a single factor to more complex


6. Conclusions
The AHP is a tool that could be used to identify the degree of an individual’s expertise in
the maritime regulatory field. AHP could be used as an alternative to CWS index to
measure and quantify the inconsistency of a candidate expert. The contribution of the
CRind is significant in this research because they pointed out the confusion among
industrial experts for regulatory issues. The inclusion of more individuals without prior
evaluation of their expertise can provide a ranking of priorities that could not necessarily
be real. The CRover value is used as a reference point to measure the knowledge distance
that an individual may has from the remaining panel as measured by CRind. The several
rounds of questionnaires give the opportunity to each candidate to comment in each
survey question at his/her convenience without any pressure that may be generated if
panel was physically formed.
In this article, it was also found that the inconsistency of a candidate may be due to
careless participation in a survey. By examining the survey material, it appears that a
candidate could be inconsistent when he is using extreme ranking while doing pairwise

comparisons. Another reason identified is that some pairwise comparisons may not be
completed with care by a candidate.
Further research is required to use the earlier method in measuring the inconsistency of
individuals in other fields of maritime sectors. The maritime industry suffers from
overregulation, and the proposed methodology could be used to measure the expertise of
regulators and industrials. The methodology could be useful for ship operators with more
complicated structures. Furthermore, maritime administrations that need accurate results
for multiple activities may find the suggested approach helpful. It may be valuable to test
if the proposed methodology can be equally applicable on the practices of other
organizations such as nuclear plants, aviation, petrochemical plants, and governments.