The right way to Construct a Calorie Advisor App Utilizing GenAI?

Introduction

Synthetic Intelligence has many use circumstances, and a few of the greatest ones are within the Well being Trade. It could actually actually assist individuals keep a more healthy life. With the rising growth in generative AI, sure functions are made as of late with much less complexity. One very helpful utility that may be constructed is the Calorie Advisor App. On this article, we’ll solely take a look at this, impressed by caring for our well being. We shall be constructing a easy Calorie Advisor App the place we will enter the pictures of the meals, and the app will assist us calculate the energy of every merchandise current within the meals. This challenge is part of NutriGen, specializing in well being by AI.

Studying Goal

• The App we shall be creating on this article shall be primarily based on fundamental Immediate engineering and picture processing strategies.
• We shall be utilizing Google Gemini Professional Imaginative and prescient API for our use case.
• Then, we are going to create the code’s construction, the place we are going to carry out Picture Processing and Immediate Engineering. Lastly, we are going to work on the Person Interface utilizing Streamlit.
• After that, we are going to deploy our app to the Hugging Face Platform for Free.
• We can even see a few of the issues we are going to face within the output the place Gemini fails to depict a meals merchandise and offers the improper calorie rely for that meals.  We can even focus on completely different options for this drawback.

Pre-Requisites

Let’s begin with implementing our challenge, however earlier than that, please guarantee you could have a fundamental understanding of generative AI and LLMs. It’s okay if you understand little or no as a result of, on this article, we shall be implementing issues from scratch.

For Important Python Immediate Engineering, a fundamental understanding of Generative AI and familiarity with Google Gemini is required. Moreover, fundamental information of Streamlit, Github, and Hugging Face libraries is critical. Familiarity with libraries corresponding to PIL for picture preprocessing functions can be useful.

This text was revealed as part of the Knowledge Science Blogathon.

Undertaking Pipeline

On this article, we shall be engaged on constructing an AI assistant who assists nutritionists and people in making knowledgeable selections about their meals decisions and sustaining a wholesome life-style.

The circulate shall be like this: enter picture -> picture processing -> immediate engineering -> ultimate perform calling to get the output of the enter picture of the meals. This can be a temporary overview of how we are going to method this drawback assertion.

Overview of Gemini Professional Imaginative and prescient

Gemini Professional is a multimodal LLM constructed by Google. It was educated to be multimodal from the bottom up. It could actually carry out properly on numerous duties, together with picture captioning, classification, summarisation, question-answering, and so forth. One of many fascinating information about it’s that it makes use of our well-known Transformer Decoder Structure. It was educated on a number of sorts of information, lowering the complexity of fixing multimodal inputs and offering high quality outputs.

Step1: Creating the Digital Atmosphere

Making a digital surroundings is an efficient follow to isolate our challenge and its dependencies such that they don’t coincide with others, and we will at all times have completely different variations of libraries we’d like in several digital environments. So, we are going to create a digital surroundings for the challenge now. To do that, observe the talked about steps under:

• Create an Empty folder on the desktop for the challenge.
• Open this folder in VS Code.
• Open the terminal.

Write the next command:

pip set up virtualenv
python -m venv genai_project

You should use the next command if you happen to’re getting sa et execution coverage error:

Set-ExecutionPolicy RemoteSigned -Scope Course of

Now we have to activate our digital surroundings, for that use the next command:

.genai_projectScriptsactivate

We now have efficiently created our digital surroundings.

Step Create Digital Atmosphere in Google Colab

We are able to additionally create our Digital Atmosphere in Google Colab; right here’s the step-by-step process to try this:

• Create a New Colab Pocket book
• Use the under instructions step-by-step

!which python
!python --version
#to verify if python is put in or not

%env PYTHONPATH=
# setting python path surroundings variable in empty worth guaranteeing that python
# will not seek for modules and packages in further listing. It helps
# in avoiding conflicts or unintended module loading.

!pip set up virtualenv 

# create digital surroundings 
!virtualenv genai_project

!wget https://repo.anaconda.com/miniconda/Miniconda3-latest-Linux-x86_64.sh

#It will assist obtain the miniconda installer script which is used to create
# and handle digital environments in python

!chmod +x Miniconda3-latest-Linux-x86_64.sh
# this command is making our mini conda installer script executable inside
# the colab surroundings. 

!./Miniconda3-latest-Linux-x86_64.sh -b -f -p /usr/native
# that is used to run miniconda installer script and 
# specify the trail the place miniconda needs to be put in

!conda set up -q -y --prefix /usr/native python=3.8 ujson
#it will assist set up ujson and python 3.8 set up in our venv.

import sys
sys.path.append('/usr/native/lib/python3.8/site-packages/')
#it should permit python to find and import modules from a venv listing

import os
os.environ['CONDA_PREFIX'] = '/usr/native/envs/myenv'

# used to activate miniconda enviornment 

!python --version
#checks the model of python throughout the activated miniconda surroundings

Therefore, we additionally created our digital surroundings in Google Colab. Now, let’s verify and see how we will make a fundamental .py file there.

!supply myenv/bin/activate
#activating the digital surroundings

!echo "print('Good day, world!')" >> my_script.py
# writing code utilizing echo and saving this code in my_script.py file

!python my_script.py
#working my_script.py file

It will print Good day World for us within the output. So, that’s it. That was all about working with Digital Environments in Google Colab. Now, let’s proceed with the challenge.

Step2: Importing Crucial Libraries

import streamlit as st
import google.generativeaias genai 
import os 
from dotenv import load_dotenv
load_dotenv()
from PIL import Picture

If you’re having bother importing any of the above libraries, you may at all times use the command “pip set up library_name” to put in it.

We’re utilizing the Streamlit library to create the essential person interface. The person will be capable of add a picture and get the outputs primarily based on that picture.

We use Google Generative to get the LLM and analyze the picture to get the calorie rely item-wise in our meals.

Picture is getting used to carry out some fundamental picture preprocessing.

Step3: Establishing the API Key

Create a brand new .env file in the identical listing and retailer your API key. You will get the Google Gemini API key from Google MakerSuite.

Step4: Response Generator Perform

Right here, we are going to create a response generator perform. Let’s break it down step-by-step:

Firstly, we used genes. Configure to configure the API we created from the Google MakerSuite Web site. Then, we made the perform get_gemini_response, which takes in 2 enter parameters: the enter immediate and the picture. That is the first perform that may return the output in textual content.

genai.configure(api_key=os.getenv("GOOGLE_API_KEY"))

def get_gemini_response(input_prompt, picture):

    mannequin = genai.GenerativeModel('gemini-pro-vision')

    response = mannequin.generate_content([input_prompt, image[0]])

    return response

Right here, we’re utilizing the ‘Gemini-pro-vision’ mannequin as a result of it’s multimodal. After calling our mannequin from the genie.GenerativeModel dependency, we’re simply passing in our immediate and the picture knowledge to the mannequin. Lastly, primarily based on the directions offered within the immediate and the picture knowledge we fed, the mannequin will return the output within the type of textual content that represents the calorie rely of various meals objects current within the picture.

Step5: Picture Preprocessing

This perform checks if the uploaded_file parameter is None, that means the person has uploaded a file. If a file has been uploaded, the code proceeds to learn the file content material into bytes utilizing the getvalue() technique of the uploaded_file object. It will return the uploaded file’s uncooked bytes.

The bytes knowledge obtained from the uploaded file is saved in a dictionary format beneath the key-value pair “mime_type” and “knowledge.” The “mime_type” key shops the uploaded file’s MIME sort, which signifies the kind of content material (e.g., picture/jpeg, picture/png). The “knowledge” key shops the uploaded file’s uncooked bytes.

The picture knowledge is then saved in an inventory named image_parts, which incorporates a dictionary with the uploaded file’s MIME sort and knowledge.

def input_image_setup(uploaded_file):
    if uploaded_file isnotNone:
        #Learn the file into bytes
        bytes_data = uploaded_file.getvalue()
        image_parts = [
            {
                "mime_type":uploaded_file.type, 
                "data":bytes_data
            }
        ]
        return image_parts
    else:
        increase FileNotFoundError("No file uploaded")

Step6: Creating the UI

So, lastly, it’s time to create the person interface for our challenge. As talked about earlier than, we shall be utilizing the Streamlit library to write down the code for the entrance finish.

## initialising the streamlit app
st.set_page_config(page_title="Energy Advisor App")
st.header("Energy Advisor App")
uploaded_file = st.file_uploader("Select a picture...", sort=["jpg", "jpeg", "png"])
picture = ""
if uploaded_file isnotNone:
    picture = Picture.open(uploaded_file)
    st.picture(picture, caption="Uploaded Picture", use_column_width=True)
submit = st.button("Inform me in regards to the complete energy")

Initially, we arrange the web page configuration utilizing set_page_config and gave the app a title. Then, we created a header and added a file uploader field the place customers can add photos. St. Picture exhibits the picture that the person uploaded to the UI. Finally, there’s a submit button, after which we are going to get the outputs from our giant language mannequin, Gemini Professional Imaginative and prescient.

Step7: Writing the System Immediate

Now’s the time to be artistic. Right here, we are going to create our enter immediate, asking the mannequin to behave as an professional nutritionist. It’s not crucial to make use of the immediate under; you can even present your customized immediate. We’re asking our mannequin to behave a sure method for now. Based mostly on the enter picture of the meals offered, we’re asking our mannequin to learn that picture knowledge and generate the output, which is able to give us the calorie rely of the meals objects current within the picture and supply a judgment of whether or not the meals is wholesome or unhealthy. If the meals is dangerous, we ask it to offer extra nutritious alternate options to the meals objects in our picture. You possibly can customise it extra in response to your wants and get a superb technique to preserve monitor of your well being.

Typically it may not in a position to learn the picture knowledge correctly, we are going to focus on options concerning this additionally on the finish of this text.

input_prompt = """

You might be an professional nutritionist the place you might want to see the meals objects from the 
picture and calculate the full energy, additionally give the small print of all 
the meals objects with their respective calorie rely within the under fomat.

        1. Merchandise 1 - no of energy

        2. Merchandise 2 - no of energy

        ----

        ----

Lastly you can even point out whether or not the meals is wholesome or not and likewise point out 
the proportion break up ratio of carbohydrates, fat, fibers, sugar, protein and 
different essential issues required in our weight-reduction plan. When you discover that meals shouldn't be wholesome 
then you could present some various wholesome meals objects that person can have 
in weight-reduction plan.

"""
if submit:

    image_data = input_image_setup(uploaded_file)

    response = get_gemini_response(input_prompt, image_data)

    st.header("The Response is: ")

    st.write(response)

Lastly, we’re checking that if the person clicks the Submit button, we are going to get the picture knowledge from the

input_image_setup perform we created earlier. Then, we move our enter immediate and this picture knowledge to the get_gemini_response perform we created earlier.  We name all of the features we created earlier to get the ultimate output saved in response.

Step8: Deploying the App on Hugging Face

Now’s the time for deployment. Let’s start.

Will clarify the only technique to deploy this app that we created. There are two choices that we will look into if we need to deploy our app: one is Streamlit Share, and the opposite one is Hugging Face. Right here, we are going to use Hugging Face for the deployment; you may attempt exploring deployment on Streamlit Share iFaceu if you would like. Right here’s the reference hyperlink for that – Deployment on Streamlit Share

First, let’s shortly create the necessities.txt file we’d like for the deployment.

Open the terminal and run the under command to create a necessities.txt file.

pip freeze > necessities.txt1plainText

It will create a brand new textual content file named necessities. All of the challenge dependencies shall be out there there. If this causes an error, it’s okay. You possibly can at all times create a brand new textual content file in your working listing and duplicate and paste the necessities.txt file from the GitHub hyperlink I’ll present subsequent.

Now, just remember to have these information useful (as a result of that’s what we’d like for the deployment):

• app.py
• .env (for the API credentials)
• necessities.txt

When you don’t have one, take all these information and create an account on the cuddling face. Then, create a brand new area and add the information there. That’s all. Your app shall be routinely deployed this fashion. Additionally, you will be capable of see how the deployment is happening in real-time. If some error happens, you may at all times determine it out with the easy interface and, after all, the cuddling face neighborhood, which has plenty of content material on resolving some frequent bugs throughout deployment.

After a while, it is possible for you to to see the app working. Woo hoo! We now have lastly created and deployed our calorie predictor app. Congratulations!!, You possibly can share the working hyperlink of the app with the family and friends you simply constructed.

Right here’s the working hyperlink to the app that we simply created – The Alorcalorieisor App

Let’s take a look at our app by offering an enter picture to it:

Earlier than:

After:

Full Undertaking GitHub Hyperlink

Right here’s the entire github repository hyperlink that features supply code and different useful info concerning the challenge.

You possibly can clone the repository and customise it in response to your necessities. Attempt to be extra artistic and clear in your immediate, as it will give your mannequin extra energy to generate appropriate and correct outputs.

Scope of Enchancment

Issues that may happen within the outputs generated by the mannequin and their options:

Typically, there might be conditions the place you’ll not get the right output from the mannequin. This will occur as a result of the mannequin was not in a position to predict the picture accurately. For instance, if you happen to give enter photos of your meals and your meals merchandise incorporates pickles, then our mannequin would possibly think about it one thing else. That is the first concern right here.

• One technique to sort out that is by efficient immediate engineering strategies, like few-shot immediate engineering, the place you may feed the mannequin with examples, after which it should generate the outputs primarily based on the learnings from these examples and the immediate you offered.
• One other answer that may be thought of right here is creating our customized knowledge and fine-tuning it. We are able to create knowledge containing a picture of the meals merchandise in a single column and an outline of the meals objects current within the different column. It will assist our mannequin be taught the underlying patterns and predict the objects accurately within the picture offered. Thus, getting extra appropriate outputs of the calorie rely for the images of the meals is crucial.
• We are able to take it additional by asking the person about his/her vitamin targets and asking the mannequin to generate outputs primarily based on that. (This fashion, we can tailor the outputs generated by the mannequin and provides extra user-specific outputs.)

Conclusion

We’ve delved into the sensible utility of Generative AI in healthcare, specializing in the creation of the Calorie Advisor App. This challenge showcases the potential of AI to help people in making knowledgeable selections about their meals decisions and sustaining a wholesome life-style. From organising the environment to implementing picture processing and immediate engineering strategies, we’ve lined the important steps. The app’s deployment on Hugging Face demonstrates its accessibility to a wider viewers. Challenges like picture recognition inaccuracies had been addressed with options corresponding to efficient immediate engineering. As we conclude, the Calorie Advisor App stands as a testomony to the transformative energy of Generative AI in selling well-being.

Key Takeaways

• We now have mentioned rather a lot thus far, Beginning with the challenge pipeline after which a fundamental introduction to the big language mannequin Gemini Professional Imaginative and prescient.
• Then, we began with the hands-on implementation. We created our digital surroundings and API key from Google MakerSuite.
• Then, we carried out all our coding within the created digital surroundings. Additional, we mentioned the best way to deploy the app on a number of platforms, corresponding to Hugging Face and Streamlit Share.
• Other than that, we thought of the doable issues that may happen, and mentioned soluFaces to these issues.
•  Therefore, it was enjoyable engaged on this challenge. Thanks for staying until the tip of this text; I hope you bought to be taught one thing new.

Often Requested Questions

The media proven on this article shouldn’t be owned by Analytics Vidhya and is used on the Writer’s discretion.