Databricks Databricks-Machine-Learning-Professional Databricks Certified Machine Learning Professional Exam Practice Test

MLflow Autologging is a feature that automatically logs parameters, metrics, and artifacts for machine learning frameworks, such as Hyperopt, without requiring any code changes. When using MLflow Autologging with Hyperopt, MLflow will automatically create a parent run for the hyperparameter tuning process and a child run for each unique combination of hyperparameter values. This way, the user can easily compare and visualize the results of different hyperparameter settings in the MLflow UI. MLflow Autologging also supports nested runs, which means that if the user starts a manual run before calling fmin, the parent run will be nested under the manual run. This allows the user to organize and group related runs together. References: MLflow Autologging, Hyperparameter Tuning with MLflow, Apache Spark MLlib and Hyperopt, Using MLFlow with Hyperopt for Automated Machine Learning, Parallelize hyperparameter tuning with scikit-learn and MLflow, [Optimizing Model Performance with MLflow and Hyperopt: From Hyperparameter Tuning to Serving]

To compute predictions for spark_df when the missing feature values can be found in the Feature Store by searching for features by customer_id, you can use the following code block:

Python

# Get the missing features from the Feature Store using the model URI and the customer_id column

df = fs.get_missing_features(spark_df, model_uri, lookup_key="customer_id")

# Score the DataFrame using the model URI and the Feature Store Client

fs.score_batch(model_uri, df)

AI-generated code. Review and use carefully. More info on FAQ.

The fs.get_missing_features method takes a Spark DataFrame, a model URI, and a lookup key as arguments. It returns a new Spark DataFrame that contains the originalcolumns plus the missing features that are required by the model. The missing features are retrieved from the Feature Store by joining the DataFrame with the feature tables using the lookup key. The lookup key must match the primary key of the feature tables. The model URI must point to a registered model that was trained using features from the Feature Store1.

The fs.score_batch method takes a model URI and a Spark DataFrame as arguments. It applies the model to the DataFrame and returns a new Spark DataFrame that contains the original columns plus a prediction column. The model URI must point to a registered model that was trained using features from the Feature Store2.

The other options are incorrect because:

• Option A: fs.score_model is not a valid method name, as it is missing an underscore. The correct method name is fs.score_batch2.
• Option B: fs.score_model without getting the missing features will not work, as the model expects the DataFrame to have all the features that were used for training. The correct way is to use fs.get_missing_features before fs.score_batch12.
• Option D: fs.score_batch without getting the missing features will not work, as the model expects the DataFrame to have all the features that were used for training. The correct way is to use fs.get_missing_features before fs.score_batch12.
• Option E: fs.score_batch without specifying the lookup key will not work, as the fs.get_missing_features method requires a lookup key to join the DataFrame with the feature tables. The correct way is to use fs.get_missing_features with the lookup key “customer_id” before fs.score_batch12. References: Get missing features, Score batch

To obtain run-level results for an experiment ID in a Spark DataFrame, you can use the mlflow.search_runs(exp_id) function. This function takes a filter string, which acts as a filter to the query, and returns a pandas.DataFrame of runs, where each metric, parameter, and tag are expanded into their own columns named metrics., params., and tags.* respectively1. You can then convert the pandas.DataFrame to a Spark DataFrame using the spark.createDataFrame() method2. The other options are incorrect because:

• Option A: client.list_run_infos(exp_id) will return a list of RunInfo objects, not a DataFrame3.
• Option B: spark.read.format(“delta”).load(exp_id) will not work for MLflow experiments, as it expects a Delta table path, not an experiment ID4.
• Option C: There is a way to programmatically return row-level results from an MLflow experiment using the mlflow.search_runs(exp_id) function1.
• Option E: spark.read.format(“mlflow-experiment”).load(exp_id) is not a valid format for reading MLflow experiments, as there is no such format registered in Spark5. References: Search Runs, Create a DataFrame from an RDD, a list or a pandas.DataFrame, MlflowClient.list_run_infos, Delta Lake Quickstart, spark.read.format

To replace all data in a feature table with newly computed data, you can use the fs.write_table method with the mode argument set to “overwrite”. This will overwrite the existing data in the feature table with the data in the provided DataFrame. The other options are incorrect because:

• Option A: fs.create_table will create a new feature table, not replace the data in an existing one. If the feature table already exists, it will raise an exception.
• Option B: fs.write_table without specifying the mode argument will use the default mode of “merge”, which will merge the new data with the existing data, not replace it.
• Option C: fs.write table is not a valid method name, as it is missing an underscore. Also, the mode argument of “merge” will not replace the data, but merge it with the existing data.
• Option E: fs.create_table with the mode argument of “merge” is not a valid combination, as the mode argument is only applicable for fs.write_table. Also, the mode argument of “merge” will not replace the data, but merge it with the existing data. References: Databricks FeatureStoreClient, Work with features in Workspace Feature Store

Concept drift is a change in the relationship between input and output data over time, which can affect the performance and accuracy of the model. To monitor concept drift, one needs to compare the predicted values from the model with the observed (actual) label values from the data source. A common way to do this is to compute an evaluation metric, such as accuracy, precision, recall, F1-score, etc., using the observed and predicted values. This metric can indicate how well the model is performing on the current data, and how it differs from the baseline performance on the training data. If the metric shows a significant decline over time, it may suggest that concept drift has occurred and the model needs to be updated123 References:

• Monitoring Concept Drift - Databricks
• Data drift, concept drift, and how to monitor for them - Mona Labs
• 8 Concept Drift Detection Methods To Use With ML Models - Aporia

The Databricks REST APIs allow users to programmatically access Databricks features and services, such as clusters, jobs, workspaces, libraries, secrets, and more. The Jobs API enables users to create, edit, delete, run, and list jobs, as well as manage job runs. Users can use the Jobs API to programmatically create a new Databricks job whose schedule depends on the result of some automated tests in a machine learning pipeline. For example, users can use the create endpoint to create a new job with a cron_schedule parameter that specifies the cron expression for the job schedule. Users can also use the run_now endpoint to run an existing job with a run_name parameter that specifies the name of the run. Users can use the runs/get endpoint to get the run result state of a job run, and use conditional logic to trigger another job run based on the result. Users can also use the runs/list endpoint to list all the runs of a job, and use the runs/cancel endpoint to cancel a job run if needed. References:

• Databricks REST API — Databricks Documentation, p. 1
• Jobs API — Databricks Documentation, p. 1-2
• Create and run Databricks Jobs | Databricks on AWS, Create and run jobs using the CLI, API, or notebooks, p. 1

Streaming with Spark as a model deployment strategy means applying a machine learning model to data streams that are processed incrementally and continuously by Spark Structured Streaming. Spark Structured Streaming is a scalable and fault-tolerant stream processing engine that enables complex analytics on live data streams using the Dataset/DataFrame API1. Spark Structured Streaming supports various sources and sinks for streaming data, such as Kafka, Kinesis, TCP sockets, Delta tables, etc2. Spark Structured Streaming also supports various types of operations on streaming data, such as aggregations, windowing, joins, and stateful transformations3. To deploy a machine learning model on streaming data, you can use the MLflow model registry to managethe model lifecycle and versioning4. You can also use the MLflow model serving feature to serve the model as a REST API endpoint that can be invoked by Spark Structured Streaming5. Alternatively, you can use the UDF (user-defined function) feature to apply the model to streaming data within Spark Structured Streaming6.

The inference of incrementally processed records as soon as trigger is hit describes the streaming with Spark as a model deployment strategy. A trigger defines when the results of a streaming query should be written to the output sink. A trigger can be based on a processing time interval, a data size limit, or a continuous mode that writes the results as soon as they are available. The trigger ensures that the streaming query is executed incrementally and continuously, and the model inference is applied to the latest available data. The other options are incorrect because:

• Option A: The inference of batch processed records as soon as a trigger is hit does not describe streaming with Spark, but rather batch processing with Spark. Batch processing means applying a machine learning model to a finite set of data that is processed as a single job. Batch processing does not require a trigger, as the results are written to the output sink when the job is completed.
• Option B: The inference of all types of records in real-time does not describe streaming with Spark, but rather a generic definition of real-time processing. Real-time processing means applying a machine learning model to data streams that are processed as soon as they arrive, with minimal latency. Real-time processing does not necessarily use Spark Structured Streaming, as there are other frameworks and tools that can support it, such as Apache Flink, Apache Storm, etc.
• Option C: The inference of batch processed records as soon as a Spark job is run does not describe streaming with Spark, but rather batch processing with Spark. Batch processing means applying a machine learning model to a finite set of data that is processed as a single job. Batch processing does not depend on a Spark job, as the model inference can be done outside of Spark, such as using a REST API endpoint, a command-line tool, etc.
• Option E: The inference of incrementally processed records as soon as a Spark job is run does not describe streaming with Spark, but rather a contradiction. Incrementally processed records imply streaming processing, while a Spark job implies batch processing. Streaming processing and batch processing are different paradigms of data processing, and cannot be mixed in this way. References: Structured Streaming Programming Guide, Input Sources and Output Sinks, Operations on streaming DataFrames/Datasets, MLflow Model Registry, MLflow Model Serving, Apply machine learning models, [Triggers], [Trigger Types], [Batch Processing], [Real-time Processing], [Real-time Data Processing Frameworks], [Deploy machine learning models], [Batch vs Streaming Processing]

To restore the original scikit-learn model object, the correct method is mlflow.sklearn.load_model, which takes the model URI as an argument and returns a scikit-learn model. This method can load models with the sklearn flavor, which preserves the original attributes and methods of the model. The mlflow.load_model method is a generic method that can load models with any flavor, but it returns a pyfunc model, which does not have the feature_importances_ attribute. The client.list_artifacts method is used to list theartifacts associated with a run, not to load a model. The client.pyfunc.load_model method is similar to mlflow.load_model, but it takes a local file path instead of a model URI. The MLflow Experiments UI can show the logged parameters, metrics, and artifacts of a run, but it cannot show the model object itself. References:

• MLflow Models
• MLflow Python API

Structured Streaming is a scalable and fault-tolerant stream processing engine built on the Spark SQL engine. It allows users to express streaming computations the same way as batch computations on static data, using DataFrame and Dataset APIs. Structured Streaming can handle both unbounded and bounded data sources, and can process data in micro-batches or continuously. Structured Streaming can be used to provide continuous processing of data for machine learning applications, such as data ingestion, preprocessing, feature engineering, and model inference. StructuredStreaming can also integrate with MLflow and Delta Lake to enable end-to-end machine learning pipelines with tracking, reproducibility, and governance123 References:

• Structured Streaming Programming Guide - Spark 3.2.0 Documentation
• Structured Streaming + Kafka Integration Guide (Kafka broker version 0.10.0 or higher) - Spark 3.2.0 Documentation
• Machine Learning with Structured Streaming - Databricks
• [Continuous Machine Learning with Structured Streaming and MLflow - Databricks]

The mlflow.sklearn module provides an API for logging and loading scikit-learn models. To load a model as an sklearn object for batch deployment, the mlflow.sklearn.load_model function can be used. This function takes a model_uri argument that specifies the location of the model to load. The model_uri can be a local file path, a remote file path, or a Databricks or MLflow Model Registry URI. The function returns an sklearn model object that can be used for inference or further training. The function also preserves the model signature and the input example, if they were logged with the model. References:

• mlflow.sklearn Documentation, p. 1-2
• MLflow Models Documentation, Model Signature and Input Example, p. 6-7

Jensen-Shannon (JS) distance is a symmetric measure of the distance between two probability distributions. It is based on the Kullback-Leibler (KL) divergence, but unlike KL, it does not depend on which distribution is considered the reference and which is considered the comparison. JS distance is also bounded between 0 and 1, which makes it easier to interpret and compare across different features.

Kolmogorov-Smirnov (KS) test is a non-parametric test that compares the cumulative distribution functions (CDFs) of two samples. It calculates the maximum difference between the CDFs and compares it to a critical value to determine if the samples are from the same distribution. KS test is sensitive to the shape, location, and scale of the distributions, but it can also be too sensitive for large datasets, where small changes can result in significant differences.

JS distance is more robust than KS test for numeric feature drift detection because it can handle large datasets without being too sensitive to minor fluctuations. It can also capture changes in the shape and spread of the distributions, not just the location. JS distance does not require any manual threshold or cutoff determinations, unlike KS test, which requires choosing a significance level and a critical value. JS distance can also be generalized to multivariate cases, while KS test is limited to univariate cases. References:

• Which test is the best? We compared 5 methods to detect data drift on large datasets
• The Jensen-Shannon Divergence: A Measure of Distance Between Probability Distributions
• Kullback–Leibler vs Kolmogorov-Smirnov distance

The MLflow Model Registry provides a centralized model store, set of APIs, and UI, to collaboratively manage the full lifecycle of an MLflow Model. It provides model lineage, model versioning, model aliasing, model tagging, and annotations. The Model Registry defines four model stages that represent the lifecycle of a model: None, Staging, Production, and Archived. Each model stage has a different meaning and purpose:

• None: This is the default stage when a new model version is registered. It indicates that the model version is not ready for any downstream use cases and needs further development or testing.
• Staging: This stage indicates that the model version is ready for testing or validation in a pre-production environment. It can be used for quality assurance, user acceptance testing, or performance evaluation before moving to production.
• Production: This stage indicates that the model version is ready for deployment in a production environment. It can be used for serving predictions, scoring data, or supporting business decisions. Only one model version per registered model can be in the production stage at a time.
• Archived: This stage indicates that the model version is no longer in active use and can be safely deleted or moved to a long-term storage. It can be used for historical or audit purposes, or for restoring a previous model version if needed.

References:

• MLflow Model Registry Documentation, p. 1-2
• Introducing the MLflow Model Registry - Databricks, p. 1-2
• Manage MLflow Models — Databricks Documentation, p. 1-2

MLflow Model flavors are a key concept that makes MLflow Models powerful: they are a convention that deployment tools can use to understand the model, which makes it possible to write tools that work with models from any ML library without having to integrate each tool with each library. MLflow defines a set of built-in model flavors for popular ML frameworks, such as scikit-learn, TensorFlow, PyTorch, etc. Each flavor defines a standard format for model data and a common interface for model inference. For example, the python_function flavor allows any model to be loaded as a Pythonfunction that can be applied to input data. MLflow also supports custom model flavors for user-defined models or community-developed flavors. MLflow Model flavors enable model portability and interoperability across different platforms and environments12 References:

• MLflow Models — MLflow 2.9.1 documentation
• Models, Flavors, and PyFuncs in MLflow

The correct code block is the one that matches the event type and the model name specified in the webhook. The event type is MODEL_VERSION_TRANSITIONED_TO_STAGING, which means the webhook will be triggered when a model version is transitioned to the staging stage. The model name is model, which means the webhook will only apply to the registered model with that name. Therefore, the code block that satisfies both conditions is:

client.transition_model_version_stage (name=model, version=model_version, stage=“Staging”)

This code block will transition the model version to the staging stage and invoke the webhook to run the associated job.

References: MLflow Model Registry Webhooks on Databricks, MLflow Python API

A model signature is a specification of the input and output schema of a model, which defines the names, data types, and shapes of the model inputs and outputs. Logging a model signature with an MLflow model has several benefits, such as:

• The signature can be used to document what type of data is expected as input and output by the model, which can help users and developers understand how to use the model correctly.
• The signature can be used to enforce schema validation when serving models via REST API, which can help prevent errors and ensure data quality. For example, Azure Machine Learning enforces compliance with the model signature for MLflow models that contain a signature1.
• The signature can be used to enable automatic conversion of input data to the appropriate format for the model, such as pandas DataFrame, numpy array, or PyTorch tensor. This can simplify the data preprocessing and inference steps and reduce the code complexity.

References:

• MLflow Models Documentation, Model Signature and Input Example, p. 6-7
• mlflow.models.signature Documentation, p. 1-2
• Guidelines for deploying MLflow models - Azure Machine Learning, p. 1

The context parameter is an instance of mlflow.pyfunc.model.ModelContext, which is a wrapper around the model metadata and artifacts. The context parameter allows the user to provide the model access to objects like preprocessing models or custom configuration files that are needed for the model inference. For example, the context parameter can be used to load a scikit-learn preprocessing model that was logged as an artifact with the MLflow model, and then apply it to the input data before passing it to the main model. The context parameter can also be used to access the model signature and the input example, if they were logged with the model. The context parameter is passed to the load_context and predict methods of the PythonModel class, which is asubclass of mlflow.pyfunc.PythonModel that defines the custom logic for loading and scoring the model. References:

• mlflow.pyfunc Documentation, p. 1-2
• mlflow.pyfunc.model Documentation, p. 1-2
• Understanding PyFunc in MLflow, Part 2: Pyfunc Components, p. 1-2
• python - How to load model from mlflow with custom predict without local file?, p. 1-2

Feature drift is a change in the distribution of the input data over time, which can affect the performance and accuracy of the model. Monitoring feature drift is important to ensure that the model is still valid and reliable for the current data. One simple and low-cost method of monitoring numeric feature drift is to track the summary statistics trends of the input features, such as mean, standard deviation, minimum, maximum, etc. These statistics can be computed easily and efficiently, and can provide a quick overview of the changes in the data distribution. If the summary statistics deviate significantly from the baseline values, it may indicate a feature drift. However, summary statistics trends may not capture all the nuances of the data distribution, such as outliers, multimodality, skewness, etc. Therefore, other methods, such as statistical tests or visualizations, may be needed to complement the summary statistics trends and provide a more comprehensive analysis of the feature drift123 References:

• Monitoring Feature Drift - Databricks
• Monitor feature attribution skew and drift | Vertex AI | Google Cloud
• Monitoring Model Drift - 6 Different Methodologies - Qualdo

To view all of the active MLflow Model Registry Webhooks for a specific model, the machine learning engineer needs to replace POST with GET in the call to http request. This is because the HTTP method for listing webhooks is GET, not POST. The GET method is used to retrieve information from the server, while the POST method is used to send data to the server1. The syntax of the http request function is as follows:

Python

http_request(host_creds, endpoint, method, json=None, headers=None)

AI-generated code. Review and use carefully. More info on FAQ.

The other options are incorrect because:

• Option A: There are no necessary changes is incorrect, as the code block will not work with the POST method.
• Option B: Replace list with view in the endpoint URL is incorrect, as the endpoint URL for listing webhooks is “/api/2.0/mlflow/registry/webhooks/list?model_name=model”, not “/api/2.0/mlflow/registry/webhooks/view?model_name=model”. There is no view endpoint for webhooks2.
• Option D: Replace list with webhooks in the endpoint URL is incorrect, as the endpoint URL for listing webhooks is “/api/2.0/mlflow/registry/webhooks/list?model_name=model”, not “/api/2.0/mlflow/registry/webhooks/webhooks?model_name=model”. The list keyword is required to specify the list operation2.
• Option E: Replace POST with PUT in the call to http request is incorrect, as the HTTP method for listing webhooks is GET, not PUT. The PUT method is used to update data on the server, not to retrieve information1. References: HTTP request methods, List webhooks