sagan

https://app.wylderhayes.com is running sagan.  

sagan is a tool for serving data science projects.  

This instance is a google kubernetes engine running a streamlit frontend  with a  fastapi backend and the cosmosis machine learning engine running gpt -- a Shakespeare chatbot.  

gpt is a  decoder only generative pre-trained transformer created with my cosmosis machine learning framework.  

Its sagan running gpt running cosmosis.  

Engineering task:  What is the smallest gke cluster needed to train and deploy our Shakespeare chatbot, ensuring it can scale and support future ML use cases?

This is a toy app but the machine learning engine and architecture are real.  This is not the best solution to the problem of creating a  Shakespeare chat bot.  I've limited it to a single cluster for cost but also because it would turn this into a  distributed training problem which would require some more  engineering and when the best solution by far is to just use a gpu.  Again I chose this architecture because is was the cheapest that can demonstrate the most.  Its a dedicated e2-standard-2 machine which holds the kube-dns and streamlit frontend pods and a spot e2-standard-2 machine which holds the backend fastapi inference and training pod.  Using a spot instance saves perhaps 50 percent of the cost of a dedicated machine but long training jobs get canceled and the training lost.   Since I don't want to turn this into a distributed training problem it needs to be limited to one training job at a time and since I want to keep the backend to one machine I also need to limit to one inference task at a time.  I put an asynchronous lock on the inference model and use a callback to post the inference engine output.  I reduce the compute and bandwidth burden between the front and back ends by creating an asynchronous non-blocking monitoring loop in the backend which updates a cache rather than having the api ping the underlying functions directly.  In this way the backend can handle many more api requests.  The monitoring loop also finds training jobs which have silently failed due to preemption or other causes and maintains the training job database.