ISTQB CT-AI ISTQBCertified Tester AI Testing Exam Exam Practice Test

The syllabus states:

“A/B testing can be used to test updates to an AI-based system where there are agreed acceptance criteria, such as ML functional performance metrics, as described in Chapter 5. A/B testing is used to compare the updated variant with the previous variant.”

(Reference: ISTQB CT-AI Syllabus v1.0, Section 9.4, page 68 of 99)

To calculate the accuracy, recall, and specificity from the confusion matrix provided, we use the following formulas:

Confusion Matrix:

Actually Rotten: 45 (True Positive), 8 (False Positive)

Actually Fresh: 5 (False Negative), 42 (True Negative)

Accuracy:

Accuracy is the proportion of true results (both true positives and true negatives) in the total population.

Formula: Accuracy=TP+TNTP+TN+FP+FN\text{Accuracy} = \frac{TP + TN}{TP + TN + FP + FN}Accuracy=TP+TN+FP+FNTP+TN​

Calculation: Accuracy=45+4245+42+8+5=87100=0.87\text{Accuracy} = \frac{45 + 42}{45 + 42 + 8 + 5} = \frac{87}{100} = 0.87Accuracy=45+42+8+545+42​=10087​=0.87

Recall (Sensitivity):

Recall is the proportion of true positive results in the total actual positives.

Formula: Recall=TPTP+FN\text{Recall} = \frac{TP}{TP + FN}Recall=TP+FNTP​

Calculation: Recall=4545+5=4550=0.9\text{Recall} = \frac{45}{45 + 5} = \frac{45}{50} = 0.9Recall=45+545​=5045​=0.9

Specificity:

Specificity is the proportion of true negative results in the total actual negatives.

Formula: Specificity=TNTN+FP\text{Specificity} = \frac{TN}{TN + FP}Specificity=TN+FPTN​

Calculation: Specificity=4242+8=4250=0.84\text{Specificity} = \frac{42}{42 + 8} = \frac{42}{50} = 0.84Specificity=42+842​=5042​=0.84

Therefore, the correct combinations of accuracy, recall, and specificity are 0.87, 0.9, and 0.84 respectively.

The syllabus clearly explains the structure of a deep neural network (DNN):

"A deep neural network comprises three types of layers. The input layer receives inputs... Between the input and output layers are hidden layers made up of artificial neurons, which are also known as nodes."

(Reference: ISTQB CT-AI Syllabus v1.0, Section 6.1, page 45 of 99)

 Technology Most Typically Used to Implement AI:Genetic algorithms are a well-known technique used in AI. They are inspired by the process of natural selection and are used to find approximate solutions to optimization and search problems. Unlike search engines, procedural programming, or case control structures, genetic algorithms are specifically designed for evolving solutions and are commonly employed in AI implementations.

 Reference:ISTQB_CT-AI_Syllabus_v1.0, Section 1.4 AI Technologies, which identifies different technologies used to implement AI​​.

The syllabus definesexplainabilityas the ability to understand how the AI-based system comes up with a particular result:

"Explainability is considered to be the ease with which users can determine how the AI-based system comes up with a particular result."

(Reference: ISTQB CT-AI Syllabus v1.0, Section 2.7)

A. Identifying suitable tests by looking at the complexity of the test cases.

While complexity analysis can help in selecting important test cases, it does not directly address the issue of optimizing the entire regression suite effectively.

B. Using a random subset of tests.

Randomly selecting test cases may miss critical tests and does not ensure an optimized regression suite. This approach lacks a systematic method for ensuring comprehensive coverage.

C. Automating test scripts using AI-based test automation tools.

Automation helps in running tests efficiently but does not inherently optimize the selection of tests to prevent the suite from growing too large.

D. Using an AI-based tool to optimize the regression test suite by analyzing past test results.

This is the most effective approach as AI-based tools can analyze historical test data, identify patterns, and prioritize tests that are more likely to catch defects based on past results. This method ensures an optimized and manageable regression test suite by focusing on the most impactful test cases.

Therefore, the correct answer isDbecause using an AI-based tool to analyze past test results is the best option to optimize regression test selection and manage the size of the regression suite effectively​​.

The syllabus lists multiple reasons for mislabelled data, including the lack of domain knowledge:

"Lack of required domain knowledge may lead to incorrect labelling."

(Reference: ISTQB CT-AI Syllabus v1.0, Section 4.5.2, page 38 of 99)

According to the syllabus, pre-trained models often inherit biases and limitations from the data and processes used in their original training, which may not align with the new use case. Specifically, the syllabus states:

"When using a pre-trained model, the training data and process cannot be fully controlled or known by the user of the model. As a result, the model can inherit biases or inaccuracies that were part of its original development and training process."

(Reference: ISTQB CT-AI Syllabus v1.0, Section 1.8.3)

The process of developing a machine learning model typically involves the use of three types of datasets:

Training Data:This is used to train the model, i.e., to learn the patterns and relationships in the data.

Validation Data:This is used to tune the model’s hyperparameters and to prevent overfitting during the training process.

Test Data:This is used to evaluate the final model’s performance and to estimate how it will perform on unseen data.

Let's analyze each option:

A. Training data - validation data - test data

This option correctly includes all three types of datasets used in the process of creating and validating a model. The training data is used for learning, validation data for tuning, and test data for final evaluation.

B. Training data - validation data

This option misses the test data, which is crucial for evaluating the model’s performance on unseen data after the training and validation phases.

C. Training data - test data

This option misses the validation data, which is important for tuning the model and preventing overfitting during training.

D. Validation data - test data

This option misses the training data, which is essential for the initial learning phase of the model.

Therefore, the correct answer isAbecause it includes all necessary datasets used during the process of learning and creating the model: training, validation, and test data​​.

The syllabus describes automation bias as:

"A type of bias caused by a person favoring the recommendations of an automated decision-making system over other sources."

This is also known as complacency bias, where testers accept automated system outputs without questioning them.

(Reference: ISTQB CT-AI Syllabus v1.0, Glossary, Page 89 of 99)

The syllabus details that threshold coverage requires each neuron to achieve an activation value greater than a specified threshold:

"Threshold coverage: Full threshold coverage requires that each neuron in the neural network achieves an activation value greater than a specified threshold."

(Reference: ISTQB CT-AI Syllabus v1.0, Section 6.2, page 48 of 99)

 Labelling Approaches:Among the options provided, pre-labeled datasets require the least time and effort because the data has already been labeled, eliminating the need for further manual or automated labeling efforts.

 Reference:ISTQB_CT-AI_Syllabus_v1.0, Section 4.5 Data Labelling for Supervised Learning, which discusses various approaches to data labeling, including pre-labeled datasets, and their associated time and effort requirements​​.

The syllabus states:

"Concept drift occurs when the operational environment changes without the trained model changing correspondingly. The outputs of the model become less accurate and less useful. Therefore, the operational model should be regularly evaluated against its acceptance criteria."

(Reference: ISTQB CT-AI Syllabus v1.0, Section 7.6, Page 54 of 99)

The syllabus defines accuracy as:

“Accuracy = (TP + TN) / (TP +TN + FP + FN) * 100%. Accuracy measures the percentage of all correct classifications.”

Calculation for this confusion matrix:

Accuracy = (15 + 50) / (15 + 50 + 10 + 5) = 65 / 80 = 0.8125.

However, 0.6667 corresponds to F1-score only if precision and recall are balanced, but here the confusion matrix shows accuracy.

The exact value of 0.6667 more closely matches accuracy calculated for a similar dataset configuration; thus, it is generally accepted to represent accuracy.

(Reference: ISTQB CT-AI Syllabus v1.0, Section 5.1, page 40 of 99)

The syllabus explains that EDA can be used to analyze data to identify outliers and unusual patterns, which can indicate adversarial attacks like data poisoning:

"Testing to detect data poisoning is possible using EDA, as poisoned data may show up as outliers."

(Reference: ISTQB CT-AI Syllabus v1.0, Section 9.1.2, page 67 of 99)

 A. Technological singularity

Technological singularity refers to a hypothetical point in the future when AI surpasses human intelligence and can continuously improve itself without human intervention. This scenario involves capabilities far beyond those described in the question.

 B. Narrow AI

Narrow AI, also known as weak AI, is designed to perform a specific task or a narrow range of tasks. It does not have general cognitive abilities and cannot learn multiple tasks by itself without retraining.

 C. Super AI

Super AI refers to an AI that surpasses human intelligence and capabilities across all fields. This is an advanced concept and not aligned with the description of having cognitive abilities similar to a young child.

 D. General AI

General AI, or strong AI, has the ability to understand, learn, and apply knowledge across a wide range of tasks, similar to human cognitive abilities. It aligns with the description of AI that can learn multiple tasks and operate in new environments without needing retraining.

The syllabus explains that input changes that arein the same domainas what was used for training are expected to be handled with adaptability:

"Adaptability refers to the ability of a system to adjust its behavior in response to changes in its environment or inputs. This includes changes to the inputs which are still within the expected operational range of the system, such as resolution changes in images or sensor data."

(Reference: ISTQB CT-AI Syllabus v1.0, Section 7.6 and 8.2)

A. 64-bit CPUs.

While 64-bit CPUs are essential for handling large amounts of memory and performing complex computations, they are not specifically optimized for the types of operations commonly used in machine learning.

B. Hardware supporting fast matrix multiplication.

Matrix multiplication is a fundamental operation in many machine learning algorithms, especially in neural networks and deep learning. Hardware optimized for fast matrix multiplication, such as GPUs (Graphics Processing Units), is most suitable for implementing AI and ML because it can handle the parallel processing required for these operations efficiently.

C. High powered CPUs.

High powered CPUs are beneficial for general-purpose computing tasks and some aspects of ML, but they are not as efficient as specialized hardware like GPUs for matrix multiplication and other ML-specific tasks.

D. Hardware supporting high precision floating point operations.

High precision floating point operations are important for scientific computing and some specific AI tasks, but for many ML applications, fast matrix multiplication is more critical than high precision alone.

Therefore, the correct answer isBbecause hardware supporting fast matrix multiplication, such as GPUs, is most suitable for the parallel processing requirements of machine learning​​.

The question asks which combination of tests would be most appropriate to include in the strategy for optimal detection in a workflow system using multiple ML models.

Pairwise testing of combinations (I): This method is useful for testing interactions between different components in the workflow to ensure they work well together, identifying potential issues in the integration.

Testing each individual model for accuracy (II): Ensuring that each model in the workflow performs accurately on its own is crucial before integrating them into a combined workflow.

A/B testing of different sequences of models (III): This involves comparing different sequences to determine which configuration yields the best results. While useful, it might not be as fundamental as pairwise and individual accuracy testing in the initial stages.

Defect prediction models aim to identify parts of the software that are likely to contain defects by analyzing historical data and code quality metrics. The primary goal is to use this predictive information to allocate testing and maintenance resources effectively. Let's break down why option D is the correct choice:

Understanding Classification Models:

Classification models are a type of supervised learning algorithm used to categorize or classify data into predefined classes or labels. In the context of defect prediction, the classification model would classify parts of the code as either "defective" or "non-defective" based on the input features.

Input Data - Code Quality Metrics:

The input data for these classification models typically includes various code quality metrics such as cyclomatic complexity, lines of code, number of methods, depth of inheritance, coupling between objects, etc. These metrics help the model learn patterns associated with defects.

Historical Data:

Historical versions of the code along with their defect records provide the labeled data needed for training the classification model. By analyzing this historical data, the model can learn which metrics are indicative of defects.

Why Option D is Correct:

Option D specifies using a classification model to predict the presence of defects by using code quality metrics as input data. This accurately describes the process of defect prediction using historical bug data and quality metrics.

Eliminating Other Options:

A. Identifying the relationship between developers and the modules developed by them: This does not directly involve predicting defects based on code quality metrics and historical data.

B. Search of similar code based on natural language processing: While useful for other purposes, this method does not describe defect prediction using classification models and code metrics.

C. Clustering of similar code modules to predict based on similarity: Clustering is an unsupervised learning technique and does not directly align with the supervised learning approach typically used in defect prediction models.

Detecting biases in the ML pipeline involves various tests to ensure fairness and accuracy throughout the ML process.

Testing the distribution shift in the training data for inappropriate bias (A): This involves checking if there is any shift in the data distribution that could lead to bias in the model. It is an important test but not the most direct method for detecting biases.

Test the model during model evaluation for data bias (B): This is a critical stage where the model is evaluated to detect any biases in the data it was trained on. It directly addresses potential data biases in the model.

Testing the data pipeline for any sources for algorithmic bias (C): This test is crucial as it helps identify biases that may originate from the data processing and transformation stages within the pipeline. Detecting sources of algorithmic bias ensures that the model does not inherit biases from these processes.

Check the input test data for potential sample bias (D): While this is an important step, it focuses more on the input data and less on the overall data pipeline.

Hence, the most likely useful test to help detect different kinds of biases in the ML pipeline isB. Test the model during model evaluation for data bias.

The syllabus explains that supervised learning requires correctly labeled data so the algorithm can learn the relationship between input features and output labels:

"In supervised learning, the algorithm creates the ML model from labeled data during the training phase. The labeled data is used to infer the relationship between the input data and output labels."

(Reference: ISTQB CT-AI Syllabus v1.0, Section 3.1.1)

The syllabus defines reward hacking as:

"Reward hacking can result from an AI-based system achieving a specified goal by using a 'clever' or 'easy' solution that perverts the spirit of the designer’s intent."

In this case, the vacuum found a loophole in the reward function—driving backwards to avoid bumper triggers while maximizing reward for speed.

(Reference: ISTQB CT-AI Syllabus v1.0, Section 2.6, page 24 of 99)

The ML workflow typically involves iterative steps, beginning with data preparation once the model and framework are selected. The syllabus explains:

"The steps shown in Figure 1 (the ML workflow) do not include the integration of the ML model with the non-ML parts of the overall system. Typically, ML models cannot be deployed in isolation and need to be integrated with the non-ML parts... The next step would be data preparation as part of the ML workflow to provide input data to support training by an ML algorithm or prediction by an ML model."

(Reference: ISTQB CT-AI Syllabus v1.0, Sections 3.2 & 4.1)