Foundation Models

A simple machine learning model is used to solve one problem. This results in a lot of reused work when doing a similar but different problem that the original model is used in. If instead, the problem tasked to the model is to construct a "grammar" or "high level ideas" of the dataset, then we can reuse these "foundation" models for many downstream tasks. An important example of a foundation model is GPT, on which, ChatGPT is created.

One application can be found in the following paper: https://arxiv.org/abs/2402.08875

Wherein we use a masked autoencoder (MAE) trained on videos from TikTok to function as a foundation model. It's primary downstream task is to perform action recognition.