qchem-lmm

Let’s build a Large Multimodal Model (LMM)—an architecture that extends a Large Language Model's (LLM) reasoning power and knowledge to any data type.  You can use a LLM to do perhaps unexpected tasks like prediction the properties of molecules. 

Task:  Create a LMM and model molecule properties.  Machine Learning Operations (MLOps).  Development Operations  (DevOps).  

Ingredients list: Machine Learning Engine (MLE), Graph Neural Network (GNN) model, LLM,  molecule dataset, Graphics Processing Unit (GPU)

Constraint:  Single GPU.

For the MLE I use cosmosis.  Its in pytorch.  For the GNN and molecule dataset I use qchem which uses cosmosis.  For the LLM I am going to use Llama-3.1-8B which fits nicely on my single modest GPU.  You can substitute a different but similarly formatted LLM.  In order to get the model to fit on a single GPU and fine tune it we use Low-Rank Adaptation (LoRA).  

Ok lets draw a quick sketch of what we want to make.  

# create the repo in the github gui then locally...

mkdir qchem-lmm

cd qchem-lmm

git init

git branch -m main

git remote add origin https://github.com/icanswim/qchem-lmm.git

git pull origin main --allow-unrelated-histories

git submodule add https://github.com/icanswim/qchem.git qchem

git submodule update --init --recursive

git submodule update --remote --recursive

git config submodule.recurse true

git add .

git commit -m "initial commit. registered nested submodules."

git push -u origin main

git switch -c dev

git push -u origin dev