Frequently asked questions

This document provides answers to frequently asked questions about the Health AI Foundation models.

General

This section general questions about the models.

What are models and embeddings?

Models are mathematical functions that process input data in a useful way to produce an output.

For example, a model to determine if an image is of a cat or dog takes the image pixels as input, and uses mathematical functions to extract useful features. Intuitively, the features can be thought of as numerical descriptions of the image, such as the shape of the ears or eyes. This numerical vector is also called an embedding. You can further process these embeddings to predict a specific outcome, for example the likelihood the image is a cat or a dog. Models that only output embeddings and not the final likelihood score for a specific task are called embeddings models.

All of HAI-DEF's models are focused on "understanding" healthcare-related data, such as chest X-rays, pathology slides, or sounds of coughs. These embedding models can "teach" another model how to understand the data that the encoder was trained on, using much less data and compute than if the second model was trained directly with that data. In the earlier example, instead of needing to train a "cat versus dog" model from scratch, just the final part of the model that interprets the features (or embeddings), needs development. This is much easier.

To better understand what each of our models do, visit their overview page.

Why do you offer open-weight models? What is your goal?

The purpose of the HAI-DEF models is to provide the developer community and businesses with the basis and tools to accelerate development of AI within healthcare. Using HAI-DEF assets relevant to your applications, you can build AI applications with less data and less compute than traditional methods.

Terms, Licensing, and Regulation

This section answers questions about terms, licensing, and regulations related to the use of the HAI-DEF models.

Can I use the HAI-DEF models for making commercial applications and products?

Yes, you can. Refer to HAI-DEF's Terms of use and Prohibited use for more details.

This is a key difference from the research endpoints, which can't be used for commercial applications or products.

What regulatory approvals, such as for medical applications and devices, should developers who want to use these models be aware of?

It's the responsibility of the model developer to ensure they use HAI-DEF models in compliance with relevant regulations. Refer to HAI-DEF's Terms of use and Prohibited use for more details. Reach out to [e-mail] if you have any questions.

Are HAI-DEF models open source?

These models are "open weight", not "open source". While the "weights" (parameters for the neural network computation) are openly released, not all of the supporting code that was used to produce those weights is open. With open weight models, you as the developer can download and use the model, subject to HAI-DEF's Terms of use and Prohibited use.

Any accompanying assets in each model's GitHub repository are open source under an Apache 2.0 license.

What else should I be aware of regarding terms and compliance requirements?

Software in each respective GitHub repository may depend on other open source software with their own license terms.

For compliance when distributing pre built Docker images of our model service, each GitHub repository has a THIRD_PARTY_LICENSES file that aggregates the licenses of all direct and transitive dependencies involved in the service implementation. The source code for those dependencies is pulled into the image as part of the Docker image building process.

What do I need to know before contributing to a HAI-DEF project?

To contribute to HAI-DEF, refer to the Community Guidelines' Contributing section for more details. In particular, a contributor needs to sign a Contributor License Agreement (CLA).

Model Characteristics

This section answers questions about the characteristics of the HAI-DEF models.

What AI acceleration hardware can these models take advantage of?

HAI-DEF uses a Hugging Face repository naming convention where the repository suffix denotes the accelerator type it has been tested on. Most models can run on any accelerator type because AI frameworks such as TensorFlow handle the abstraction. Our suffix convention is that a missing suffix represents CPU, while a -gpu or -tpu suffix represents a GPU or TPU accelerator, respectively.

What are the limitations of the HAI-DEF models?

There is a unique answer to this question per model. Consult each model card's "Use and limitations" section to learn more (example).

In what types of situations do the models perform best?

The models perform differently for different tasks, just like humans. The publications will give you a sense for which tasks it will likely do better on, but more data often improves performance.

Consult each model card's "Performance and validation" section to learn more (example).

Model Serving

This section answers questions about serving the HAI-DEF models.

Can these models be used on different systems and infrastructure? Can I use them locally?

As open-weight models, you can download the weights from Hugging Face to use locally or to package into a custom service for use with any cloud provider. We also have pre-made Google Cloud Vertex AI-compatible services you can deploy, as well as Research Endpoints available to try out instead of deploying your own service. Consult each model's "Get Started" page for details (example).

What are the advantages of using the models with Google Cloud?

Using Google Cloud to host and serve the HAI-DEF models for your application development and deployment has these advantages:

• HAI-DEF models are integrated into the Vertex AI platform, providing an end-to-end and integrated MLOps user experience. Whether you are a researcher, a clinician, or an application developer, Vertex AI takes away the burden of infrastructure management and operations with convenient point-and-click graphical user interface.
• HAI-DEF models are containerized as REST APIs and can be hosted on any API management system. However, they are specially designed to run on Vertex prediction with >=99.5% uptime for online predictions and >=99.9% for batch predictions (see Vertex AI's SLA). HAI-DEF APIs are designed for low latency, and thanks to Vertex prediction endpoints, can autoscale with your traffic.
• HAI-DEF containers integrate with Google Cloud Storage —and where applicable, Google DICOM Store —and can process your data securely, efficiently and at scale. You control how they access your data and for how long. The containers process your data in memory without persisting the source or derivative data, and the implementation is open source and available to you (example). Refer to the general container architecture for more details.
• When using HAI-DEF models on Google Cloud, you take advantage of full stack security, both at rest and in transit.
• All the managed services that we have used to make your HAI-DEF experience possible on Google Cloud are among HIPAA covered products, enabling you to build real products.
• Systems integrators who are familiar with both HAI-DEF models and Google Cloud can support your development.
• Recipes in Colab Enterprise show you how to train models using the HAI-DEF embeddings.

Refer to the Google Cloud website for more details. HAI-DEF models are also directly available from Google Cloud's Model Garden for straightforward Google Cloud integration and deployment.

If I am unable to use Google Cloud, can I use the model on other platforms?

Yes, you can. Since the HAI-DEF models are open weight, you can download the weights from Hugging Face and use them in a custom service deployed to your chosen platform.

Do the model serving services collect or log the data they work with?

The services don't log or collect any of the data they operate on. Because the services you can deploy yourself are open source, you may also audit and modify them to suit your needs.

Other model use cases

This section answers questions about additional use cases for the HAI-DEF models.

How do I improve the models, for example using fine tuning?

Refer to each model's notebooks/ directory in GitHub for examples of how to do this (example).