Decrypting the Different Exam and Certification Models
{LANG_NAVORIGIN} Certifications
By: Derek Melber, 04/21/2005
Innovative Item Exams
There are some exams that have combined the best of the conceptual exam question format with
a semi-simulation question format. These exams discriminate better than the conceptual exams.
The discrimination comes in that the exam questions pose a screen or environment that forces the
candidate to interact with the question in some manner.
There are many different types of innovative item exams, and new types are developed all the
time. In the following section, we will look at a couple of options, including the build list and
reorder and active screen.
Build List and Reorder
The build list and reorder item question is relatively new and provides the candidate with a
format that can test analysis and troubleshooting skills. The question scenario can set up almost
any situation, resulting in the candidate evaluating a list of tasks, steps, or objects. The candidate
is then responsible for selecting the essential list items and dragging them into the correct order
to answer the scenario. Figure 2.1 illustrates how a build list and reorder question type looks.
Figure 2.1: Build list and reorder questions present a scenario, then provide a list of tasks, steps, or objects that need to be ordered into a specific list.
These question types are ideal for scenarios that result in the candidate ordering the list into the
correct steps to complete a task. This task might be to configure a firewall, route table, or
security policy. Another good use of the build list and reorder question type is to order a list of
troubleshooting steps. Examples might include troubleshooting network connections or security
configurations.
Active Screen
Active screen questions are as close to simulations as you can get. These questions provide a
single screen or multiple screens that have all of the buttons, radio buttons, text boxes, and other
objects active. Some active screen questions can allow multiple screens that are in a hierarchy.
Figure 2.2 shows an active screen question example.
Figure 2.2: Active screen questions allow the candidate to configure a screen and all the objects that are
displayed on that screen.
Written Exams
The written exam can be a very powerful tool in evaluating a candidate’s knowledge. The written
exam format exposes new and unique options for what can be tested. This exam format is
typically not used standalone, but instead is used in conjunction with another exam type. This
format necessitates that the certification require more than one exam.
Written exams in the tech industry can fall into different categories. Some of the written exams
can follow a research format, which requires the candidate to develop a solution to a known
problem in the industry or research and report on a recent solution to a known problem. This
format is useful for pushing the limits of the candidate, not only for technical knowledge but also
for writing and research abilities.
Written exams can also fall into the category of design solutions. This format differs from the
research paper in that the candidate typically won’t be able to research previous solutions
because the design in the written paper would be unique for the given problem. This exam
format is for higher-level certifications dealing with technology design. This format provides for
a flexible environment, making the exam very difficult. In some cases, the final paper could be
published and used as a resume-building tool. Not as high profile as the hands-on exams, the
written exam format can be used to promote a candidate’s talents and skills.
The written exam format does tell a lot about the candidate, but there are many drawbacks to the
format. First, there is a lot of room for interpretation of the final paper written for the exam. The
person evaluating the final results must have certain milestones and points to look for in the
paper; otherwise, the same paper could get two different grades by two different evaluators.
Second, unless the paper is written in a testing center, there is too much room for someone to
cheat. Third, if the paper is written in a testing center, it is difficult to make resources available
to candidates. In these cases it is best to just have the candidate work solo, with no resources.
Finally, the grading of these papers is difficult. The people that evaluate the final grade and
passing score need to be very strict as to what they are looking for. If there are flexible rules, the
quality of the written exams can falter, making the certification less attractive and valuable.
For these reasons and because these exams are more difficult and expensive to develop, most
vendors avoid using this format. However, you can clearly see that the written exam can provide
a robust and strong exam. It is not likely that you will see these exams often, but if you do come
upon a certification that includes a written exam, thoroughly evaluate your approach to the exam.
Consider the time vs. rewards for completing the exam. Also, consider how you might leverage
the final results of the exam, for example, by publication or authoring options.
Case Study Exams
Case study exams are used to create as close to a real world environment as possible without
putting the candidate in a production environment. The case study provides an overview of the
company, network, and security requirements. The information provided in each case study is
provided as narrative, interviews, and company security policy requirements. From this
information, the candidate is required to answer questions that configure, implement, and fix
situations that are presented in the question setup. Figure 2.3 illustrates what a case study
interface might look like.
Figure 2.3: Case studies provide a lot of information categorized into areas that are accessible through
buttons or tabs.
The case study questions can be a combination of conceptual item questions or innovative item
questions. You might stumble upon a few exams that also throw in simulation item types, but
this format is rare.
The complexity of the case study exams are that they set up an entire environment, including
security requirements, problems, and existing conditions. The challenge for the candidate is to
compile all of this information to develop a solution. There are typically many questions related
to a single case study item on the exam. For those exams that use this format, you will typically
see four to five case studies on a single exam.
It is most common to see the case study exam format on security or enterprise-level exams.
These topic areas that require laying out the current infrastructure and desired infrastructure
lends itself well to this format.
When you take an exam that uses the case study format, you will quickly know it because the
interface and format is so unique compared with other exams. As you can see from Figure 2.3,
the interface is unique, with the buttons on the left pane and the questions on the right. It is also
common for the buttons to be tabs, allowing you to jump back and forth between different parts
of the exam quickly.
Simulation Exams
Simulation exams are a notch above the innovative item exams, in that the examinee is forced to
understand the entire OS or tool (instead of just a small portion of it). The scenarios around the
question setup will be the same as with innovative item exams; however, in simulation exams,
the candidate will need to find the correct path in the different interfaces to find the
configuration(s) that will solve the scenario. Figure 2.4 illustrates a sample simulation scenario.
Figure 2.4: Simulations still use a scenario to introduce the situation and ask the question.
As you can see in Figure 2.4, after the scenario has been detailed for the candidate, the answer
must be completed within the simulation of the OS or tool. In this example, the candidate would
select the Simulation button to move to the simulation, which Figure 2.5 shows.
Figure 2.5: The simulation provides the candidate with a portion of the OS or tool.
In this simulation, all of the nodes would be active within this interface. If the candidate double-
clicked My Computer, another window would open with the contents of My Computer. The
candidate could also right-click nodes and some of the menu options would be active. In most
cases, the menu options would be inactive because they do not pertain to the goals dictated in the
scenario. This case illustrates where the simulation falls short of the real computing environment,
which would have all menu options and nodes active. In essence, the candidate is only given a
short leash to make mistakes and dilly-dally in the interface.
This exam format obviously tests more than the conceptual or innovate item exams. The
simulation does not need to be limited to a single screen or interface. Many times the simulation
encompasses an entire aspect of the OS or tool. It is clear that an exam using simulations would
discriminate between the candidate that knows how to configure the interface and one that is not
familiar with the question or interface.
To create an exam that uses simulations takes a lot of time and effort. A programmer needs to
develop an environment that duplicates the look and behavior of the real OS or tool. For an exam
that covers only a small tool or a portion of the OS, this task might not be very difficult. When it
comes to testing multiple aspects of a large OS such as UNIX or Windows, the task becomes
much more difficult.
OSs and tools are so complex that it is difficult to simulate even a simple interface. In some
cases, one click or configuration makes many changes, forcing the developers of the simulation
to catch all possible areas that the single change affects. If the candidate can’t work in an
environment that duplicates what he or she is familiar with, the exam becomes unfair.
When developed successfully, the simulation exam provides a great platform for testing the
knowledge of a candidate. The simulation is typically a standalone program that can be run on
any computer. This format provides a flexible environment for delivering the exam in a variety
of places.
E-Mail Link
Your IP address will be sent with this e-mail
