iSQI CT-GenAI ISTQB Certified Tester Testing with Generative AI (CT-GenAI) v1.0 Exam Practice Test

Embeddingsare a fundamental concept in modern Natural Language Processing (NLP) and LLMs. They are high-dimensional numerical vectors—essentially lists of numbers—that represent the meaning (semantics) of a piece of text (a word, sentence, or document). Unlike traditional keyword matching, which looks for identical strings of characters, embeddings allow the model to understand the "closeness" of concepts. For example, in a vector space, the word "bug" would be mathematically closer to "defect" or "error" than to "feature" or "requirement." This captures the semantic relationship between terms. This technology is the backbone of Retrieval-Augmented Generation (RAG) used in testing: when a tester queries a documentation set, the system converts the query into an embedding and looks for other chunks of text with similar vector values. This allows the AI to retrieve relevant context even if the exact keywords do not match. It is not about logical rules (Option C) or groups of tokens (Option A), but rather a mathematical representation of language that enables machines to process human meaning.

In the context of LLM inference,Temperatureis a hyperparameter that controls the randomness or "creativity" of the model's output. When the temperature is set high, the model's probability distribution is "flattened," meaning it is more likely to select less-probable tokens, leading to more diverse and sometimes unpredictable text. For software testing, where precision and repeatability are paramount,lowering the temperature(Option C) is the standard practice. A temperature of 0.0 makes the model "deterministic," meaning it will consistently choose the token with the highest probability. This narrows the sampling distribution and significantly reduces variability between runs. While a larger context window (Option D) allows the model to process more information, it does not directly control the randomness of token selection. Similarly, the "learning rate" (Option B) is a parameter used during thetrainingorfine-tuningphase, not during inference. For generating test cases or scripts that must follow strict logic, a lower temperature ensures that the model remains focused and produces consistent results.

A hallucination refers to a phenomenon where a Large Language Model generates text that is grammatically correct and seemingly plausible but is factually incorrect or unsupported by the provided context or real-world data. In the context of software testing, this is a critical limitation. For example, an LLM might generate a test case for a software feature that does not exist or cite a non-existent API parameter. These errors occur because LLMs are probabilistic engines designed to predict the "most likely" next token rather than "reasoning" from a set of verified facts. They do not have a built-in "truth" mechanism. While a logical mistake (Option B) is a failure in reasoning and a systematic preference (Option D) describes bias, a hallucination is specifically about the fabrication of information. Testers must be particularly vigilant regarding hallucinations, as they can lead to "false confidence" in test coverage or the creation of invalid bug reports. Mitigations include grounding the model with Retrieval-Augmented Generation (RAG) and implementing rigorous "human-in-the-loop" verification of all AI-generated test artifacts.

In the context of structured Generative AI capability building, a prompt pattern is a formalized method of interaction that ensures repeatability and reliability. Much like software design patterns, prompt patterns provide a reusable and structured template designed to guide Large Language Models (LLMs) toward producing specific, high-quality, and consistent outputs. Without these patterns, testers often rely on "zero-shot" or ad hoc prompting, which frequently leads to non-deterministic results that are difficult to validate in a professional testing lifecycle. By adopting prompt patterns, organizations can standardize how requirements are translated into test cases or how code is analyzed for defects. This standardization is critical for scaling GenAI across a team, as it allows for the creation of a "prompt library" where successful structures—such as Persona-based, Few-shot, or Chain-of-Thought patterns—are documented and reused. This approach moves the use of GenAI from a trial-and-error activity to a disciplined engineering practice, ensuring that the model understands the specific context, constraints, and expected output formats required for rigorous software testing tasks.

The primary differentiator between an LLM-powered test infrastructure and a traditional chatbot is the move from "deterministic" to "probabilistic" logic. Traditional chatbots (Option D) rely on "if-then" logic, decision trees, and predefined scripts. They can only respond to queries that match specific keywords or patterns mapped in their database. In contrast, an LLM-powered infrastructure utilizes the generative capabilities of Large Language Models to synthesize and create new content based on context. This allows it todynamically generate test insights(Option A)—such as predicting potential regression risks based on unstructured code diffs or drafting test cases for a brand-new feature described in natural language. While traditional bots provide fixed, scripted responses (Option B), LLMs can "reason" through multi-step testing problems and provide nuanced explanations. This contextual awareness is powered by the model's training on vast amounts of technical documentation, enabling it to assist in exploratory testing and complex analysis that traditional, rule-based systems simply cannot handle.

Prompt Chainingis a technique where a complex task is decomposed into several smaller, sequential steps, where the output of one step serves as the context or input for the next. This is far more reliable than a "one-shot" approach (Option A) because it reduces the cognitive load on the LLM and allows for intermediate verification. In the scenario of test analysis, the most logical and effective chain begins by extracting discretetest conditionsfrom the raw requirements. Once these conditions are established, the next "link" in the chain is toprioritize them based on risk(impact and likelihood), which requires the model to reason specifically about the importance of each condition. The final step is tomap these prioritized conditions back to the original requirementsto identify any "coverage gaps." This systematic flow (Option B) mirrors the professional test analysis process defined in the ISTQB/CT-GenAI standards. By following this sequence, the tester ensures that the AI-generated output is logically derived and thorough, providing a clear "audit trail" from the initial requirement to the final prioritized test suite.

In structured prompt engineering, theRolecomponent (also known as a Persona) is used to set the perspective, expertise, and tone of the LLM’s response. By assigning the role of a "senior test manager," the tester instructs the model to adopt the specific domain knowledge, vocabulary, and professional standards associated with that position. This technique is highly effective because LLMs are trained on vast datasets containing diverse professional documents; invoking a specific persona helps the model narrow its "latent space" to retrieve information relevant to that specific field. For instance, a senior test manager persona will prioritize risk management, resource allocation, and high-level strategy, whereas a "junior developer" persona might focus more on syntax and local unit tests. WhileContext(Option B) provides the background of the project andInstruction(Option A) defines the specific task to be performed, theRoleserves as the foundation for how those instructions are interpreted. This ensures the generated testware aligns with the expected professional seniority and organizational maturity required for high-stakes environments like a payments platform.

It is important to distinguish between a general-purposeChatbotand a specializedLLM applicationwithin a testing framework. A chatbot is primarily designed for multi-turn, conversational interactions where the user can ask questions and receive answers in a natural language format. While useful for general queries, it often lacks the specialized workflow integration needed for rigorous testing. Conversely,LLM applications(or "LLM-powered tools") are built with a specific "scope" in mind, such as automated test generation, code analysis, or requirement mapping. These applications often use the LLM as an underlying engine but surround it with specific UI components, data connectors (like RAG), and fixed task-oriented prompts to achieve a defined testing outcome. While chatbots are "free-form," LLM apps are "capability-driven." This distinction is key for organizations defining a GenAI strategy; simply providing a chatbot to testers is rarely sufficient. Instead, organizations should develop or adopt LLM applications that integrate directly into the CI/CD pipeline and provide structured, actionable test artifacts that support defined quality engineering tasks.

This scenario describes a specialized form ofData Exfiltration(specifically targeting the model's internal "weights" or training memory). While data exfiltration usually refers to stealing data from a database, in the context of LLMs, it can also refer to techniques that force the model to "reveal" sensitive information it was trained on or data that exists within its current context window. By using long, repetitive, or specifically "crafted" prompts to overwhelm the model's normal attention mechanisms or safety filters, an attacker may cause the model to output verbatim snippets of proprietary information, PII, or internal documentation that should have remained confidential. This is different fromRequest Manipulation(Option D), which aims to change the model's behavior, orData Poisoning(Option A), which happens during training. In testing, this risk is high when models are fine-tuned on private company repositories. Testers must be aware that if a model is accessible to unauthorized users, those users might use adversarial prompting techniques to extract sensitive code or business logic through these types of data leakage attacks.

A successful Generative AI strategy for testing is heavily dependent on the quality of the data used for grounding (RAG) and prompting. The principle of "Garbage In, Garbage Out" is magnified with LLMs; therefore, a key strategic pillar is the prioritization of accurate, relevant, and high-quality input data. This involves establishing defined quality procedures to ensure that the requirements, codebases, and historical defect logs fed into the model are "clean" and representative of the current system state. Strategy must avoid the "unfiltered" approach (Option C), as including contradictory or obsolete data can lead to hallucinations or irrelevant test cases. While synthetic data (Option D) is a powerful tool for privacy, it cannot entirely replace the nuanced reality found in secured enterprise data. Furthermore, legacy data (Option A) often contains valuable insights for regression testing. Consequently, the strategy should focus on building a robust data pipeline that ensures only verified, contextually appropriate information is utilized, thereby increasing the reliability of AI-generated testware and ensuring it aligns with the organization's quality standards.

A Retrieval-Augmented Generation (RAG) workflow is designed to "ground" an LLM's output in specific, verifiable data. The logical flow begins with an initial input or "focused query" (Step iv) that defines the tester's goal—in this case, generating cases for a new payments rule. The system then uses this query to perform a semantic search in avector database(Step i) to find the most relevant "chunks" of the new API specification. Following this, the system retrieves complementary data from therelational database(Step iii), such as historical test cases that might provide structural patterns or regression context. Finally, all the retrieved information—the new specs and the historical context—is bundled together and "fed" into the LLM as part of an augmented prompt (Step ii). This ensures the LLM doesn't hallucinate rules but instead synthesizes the new requirements with established organizational testing standards. Following the order in Option B ensures that the model is provided with the most relevant and logically organized context prior to generating the final testware.

In professional Generative AI applications, thesystem prompt(sometimes called the system message) is the foundational set of instructions that defines the AI's persona, boundaries, and overall behavior. In a testing workflow, this is typically defined by atester or test engineerwho is configuring the AI assistant for a specific project. Unlike the user prompt, which changes with every interaction, the system prompt remains relatively static and acts as a "guardrail" to ensure the model stays in its role (e.g., "You are an expert in ISO 26262 automotive testing standards"). By defining the system prompt, the tester ensures that the model consistently uses specific terminology, adheres to data privacy constraints, and formats its output according to the team's requirements. While end users (Option B) provide the task-specific input, they do not usually have the permissions or technical need to alter the underlying system-level instructions. Similarly, while CI servers (Option C) might trigger the prompt, they do not "define" the human-centric logic contained within it. Properly crafting the system prompt is a core part of setting up an AI-augmented test environment.