1. Defining the problem

Analyzing qualitative student feedback can be time consuming, especially with large classes and diverse responses: direct, exuberant, cagey, passive aggressive, ambiguous, etc. If we want to get a snapshot of students' opinions about a class, one way would be to rate their response based on sentiment. In other words, we could rate the overall positivity or negativity of each student's experience. In this tutorial, we'll use Gemini to rate the positivity of student feedback on a scale of 1-5 and create a structured dataset. This will allow us to quickly identify trends and focus on specific areas for improvement.

2. Installing Gemini 1.5 Flash

Gemini 1.5 Flash is Google’s free, lightweight LLM that can handle 10,000+ words in a prompt. To get started we'll need to register for a Gemini API key and install Google's Generative AI Python library.

1. Follow the directions to generate your API key. Once generated, copy and save the key in a new file on your computer called: gemini-api-key.txt Do not share this key.

2. Then pip install Google's Generative AI Python library:

pip install google-generativeai

3. Your first function with Gemini

With everything installed, we’re ready to take Gemini for a test drive. Our first project will be an AI tool that creates an anagram of a person's name.

Let's get our files organized. Create a new folder called my-ai-tool and put the gemini-api-key.txt file inside the folder. Create a new Python file in the folder and call it ai-tool.py Your file structure should look like this:

    —— my-ai-tool
            — gemini-api-key.txt
            — anagram.py

Inside the IDE of your choice, open anagram.py and read your API key into memory. By printing the contents of gemini-api-key.txt to the terminal, you can test that everything is in order:

with open('gemini-api-key.txt') as f:
    genai_api_key = f.read()

print(genai_api_key)

Next, remove the print function, import Google's Generative AI library, and configure the language model. In just two short lines of code, you'll establish a connection to the Gemini API and create an object called "model" that can generate text with the gemini-1.5-flash LLM:

import google.generativeai as genai

#Read Gemini API key into memory.
with open('gemini-api-key.txt') as f:
    genai_api_key = f.read()
    
#Configure API and select the Gemini 1.5 Flash language model.
genai.configure(api_key=genai_api_key)
model = genai.GenerativeModel('gemini-1.5-flash')

With Gemini configured, let's create a function that'll create anagrams using Gemini. We’ll start with a simple function called create_anagram(name) that takes a person's name as input. Inside the function, we'll use model.generate_content() to send a prompt to Gemini, asking it to generate an anagram based on the given name. I’m using a Python feature called an f-string (formatted string literals) to add our function’s name argument at the end of the prompt: Create one anagram with the letters in this person’s name: {name}.

To retrieve the generated anagram as text, we'll use response.text. If we omit the .text attribute, the function will return metadata about the generated text.

Let's try running the function with and without .text to see the difference. What anagrams can we create for Albert Einstein?

def create_anagram(name):
    """Create one anagram of the input name"""
    response = model.generate_content(f"Create one anagram with the letters in this person's name: {name}")
    #Experiment running the function with .text and without .text.
    return response.text

print(create_anagram("albert einstein"))

4. Rating student reviews with Gemini

Now that we understand the basics of the Gemini API, let's build a new AI tool that analyzes student reviews for their sentiment.

In the my-ai-tool folder, create a new Python file called rate-sentiment.py. Then download student-review-1.txt and add it to the folder. Your file structure should look like this:

    —— my-ai-tool
            — gemini-api-key.txt
            — anagram.py
            — student-review-1.txt
            — rate-sentiment.py

Read this student review into memory and print it to the terminal as we did earlier with your API key:

#Read student review into memory.
with open('student-review-1.txt') as f:
    student_review = f.read()

print(student_review)

When you run the above code, you should see this student review in the terminal:

Stats class was definitely challenging, but I learned 
a lot! The teacher was helpful, and class discussions 
were interesting. However, I struggled with the
assignments. The instructions were often confusing,
and I spent way too much time trying to figure out
what was expected. I wish the assignments were more
clear-cut.

On an individual basis, student reviews are nuanced and provide some meaningful feedback. But multiply this out 50 or 100 times, and it becomes harder to understand whether students, in general, found a course useful.

We can use Gemini to rate (quantify) the sentiment of student reviews. A simple way to rate sentiment is with a Likert scale of 1-5, with 1 representing Very Negative and 5 representing Very Positive sentiment.

Getting Gemini to rate sentiment effectively takes us into an area called prompt engineering, which is the process of crafting instructions to guide AI models towards desired outputs. Prompt engineering is an iterative process, so by refining our instructions we'll discover how to write a prompt that produces the most desirable output.

Let's start by creating a function that will accept student feedback as a paragraph. This should look familiar from earlier:

import google.generativeai as genai

#Read Gemini API key into memory.
with open('gemini-api-key.txt') as f:
    genai_api_key = f.read()

#Read student review into memory.
with open('student-review-1.txt') as f:
    student_review = f.read()

#Configure API and select the Gemini 1.5 Flash language model.
genai.configure(api_key=genai_api_key)
model = genai.GenerativeModel('gemini-1.5-flash')

def rate_sentiment(text):
    """Rates the positivity of a text on scale of 1-5: 
    5 = Very Positive. 1 = Very Negative.
    """    
    response = model.generate_content() 
    return response.text

#Apply the function to our text file and see Gemini's response.
print(rate_sentiment(student_review))

We can use this prompt: Rate the positivity of this student's review of a class on a scale of 1-5, with 1 representing Very Negative and 5 representing Very Positive. We can then use an f-string again like this:

def rate_sentiment(text):
    """Rates the positivity of a text on scale of 1-5: 
    5 = Very Positive. 1 = Very Negative.
    """    
    response = model.generate_content(f"Rate the positivity of this student's review of a class on a scale of 1-5, with 1 representing Very Negative and 5 representing Very Positive: {text}") 
    return response.text

print(rate_sentiment(student_review))

Run the code and you should get a response similar to this in the terminal:

I would rate this review a **3/5**. 

Here's why:

* **Positive:** The student acknowledges learning a lot, appreciates the professor's helpfulness, and finds the class discussions engaging.
* **Neutral:**  The student's experience with the assignments is mixed. While they struggle, the student doesn't express outright dissatisfaction with the professor or the class.
* **Negative:**  The student explicitly expresses frustration with the assignment instructions and wishes they were clearer. 

Overall, the review is balanced, highlighting both positive and negative aspects. It's not overly enthusiastic but also not overtly critical.

The good news is that Gemini understands the prompt and is giving us a meaningful response. However, for our purposes, we just need the Likert score and not the full justification. What we learn from Gemini's justified response is that we need to refine our prompt to be more specific about what is our desired output. This is prompt engineering. Also, if require just a single digit, we should apply the Python .strip() method to remove any incidental white space from our output. Let's try this:

def rate_sentiment(text):
    """Rates the positivity of a text on scale of 1-5: 
    5 = Very Positive. 1 = Very Negative.
    """    
    response = model.generate_content(f"Rate the positivity of this student's review of a class on a scale of 1-5, with 1 representing Very Negative and 5 representing Very Positive. Your output should be in the form of a single digit representing your positivity score: {text}") 
    # .text returns just text from Gemini and .strip() removes whitespace.
    return response.text.strip()

print(rate_sentiment(student_review))

Let's recap: we’ve successfully passed our data to Gemini, had it analyze that data for sentiment positivity, and refined the format of its output through prompt engineering. Next we’ll apply our rate_sentiment() function to an entire set of student feedback.

5. Iterating over a large dataset

In this section we will learn how to apply our Gemini-infused function to analyze a large dataset of student reviews.

I've prepared a collection of 10 student reviews that we'll use to practice iterating over a dataset. Download student-reviews-dataset.csv and add it to the my-ai-tool folder. Your file structure should look like this:

    —— my-ai-tool
            — gemini-api-key.txt
            — anagram.py
            — student-review-1.txt
            — rate-sentiment.py
            — student-reviews-dataset.csv

To read the CSV dataset into memory and apply our rate_sentiment() function to an entire set of reviews, we'll use pandas, which is a very popular Python library for data manipulation and analysis. If you are not familiar with pandas already, it is well worth your time to become familiar with it through this tutorial. You can use pip to install pandas:

pip install pandas

Once installed, create a dataframe from student-reviews-dataset.csv and print the resulting dataframe to take a look at our data:

import google.generativeai as genai
import pandas as pd

[...]

# Read student review data into memory.
df = pd.read_csv('student-reviews-dataset.csv')
print(df)

You should see that the dataframe has two columns, student_name and review and that there are reviews from 10 students.

One powerful function in the pandas library is .apply(), which will allow us to create a new column in our dataset that is the result of iterating over and manipulating the review column data with our rate_sentiment() function:

import google.generativeai as genai
import pandas as pd

#Read Gemini API key into memory.
with open('gemini-api-key.txt') as f:
    genai_api_key = f.read()

#Configure API and select the Gemini 1.5 Flash language model.
genai.configure(api_key=genai_api_key)
model = genai.GenerativeModel('gemini-1.5-flash')

def rate_sentiment(text):
    """Rates the positivity of a text on scale of 1-5: 
    5 = Very Positive. 1 = Very Negative.
    """    
    response = model.generate_content(f"Rate the positivity of this student's review of a class on a scale of 1-5, with 1 representing Very Negative and 5 representing Very Positive. Your output should be in the form of a single digit representing your positivity score: {text}") 
    # .text returns just text from Gemini and .strip() removes whitespace.
    return response.text.strip()

# Read student review data into memory.
df = pd.read_csv('student-reviews-dataset.csv')

#Iterate over student reviews with Gemini
df['sentiment_rating'] = df['review'].apply(rate_sentiment)

To do any kind of statistical analysis of our rated reviews, we need to be sure that they are integers. Use pandas' .dtypes attribute to see what kind of data we have in each column of our dataframe:

output = df.dtypes
print(output)

In your terminal you'll see that in fact the data we have in our sentiment_rating column are not integers:

student_name        object
review              object
sentiment_rating    object
dtype: object

We can easily rectify this by using pandas to convert sentiment_rating data to integers (int):

# Read student review data into memory.
df = pd.read_csv('student-reviews-dataset.csv')

#Iterate over student reviews with Gemini
df['sentiment_rating'] = df['review'].apply(rate_sentiment)

# Convert sentiment_rating to integers
df['sentiment_rating'] = df['sentiment_rating'].astype(int)

result = df.dtypes
print(result)

Run this code again, and you'll see that sentiment_rating data are now integers (int64). At this point, we can use pandas to get a statistical summary of our data using the .describe() function. Here is what your full code should look like:

import google.generativeai as genai
import pandas as pd

#Read Gemini API key into memory.
with open('gemini-api-key.txt') as f:
    genai_api_key = f.read()

#Configure API and select the Gemini 1.5 Flash language model.
genai.configure(api_key=genai_api_key)
model = genai.GenerativeModel('gemini-1.5-flash')

def rate_sentiment(text):
    """Rates the positivity of a text on scale of 1-5: 
    5 = Very Positive. 1 = Very Negative.
    """    
    response = model.generate_content(f"Rate the positivity of this student's review of a class on a scale of 1-5, with 1 representing Very Negative and 5 representing Very Positive. Your output should be in the form of a single digit representing your positivity score: {text}") 
    # .text returns just text from Gemini and .strip() removes whitespace.
    return response.text.strip()

# Read student review data into memory.
df = pd.read_csv('student-reviews-dataset.csv')

#Iterate over student reviews with Gemini
df['sentiment_rating'] = df['review'].apply(rate_sentiment)

# Convert sentiment_rating to integer
df['sentiment_rating'] = df['sentiment_rating'].astype(int)

# Use .dtypes to get a summary of data types
result = df.dtypes
print(result)

# Use describe() to get summary statistics of sentiment_rating
summary_statistics = df['sentiment_rating'].describe()
print(summary_statistics)

When you run this code, you'll have some useful information for assessing students' sentiments about a class. For example, the average (mean) and quartiles. You'll also get the highest rating (max) and lowest rating (min):

count    10.000000
mean      2.800000
std       1.549193
min       1.000000
25%       1.250000
50%       3.000000
75%       3.750000
max       5.000000

See the GitHub Repo containing all files for this lesson.

6. Ethics of AI and Student Feedback

As Chalkbeat reported in early 2024, some schools are already exploring how artificial intelligence can transform the way teachers receive feedback on their classroom instruction. The Urban Assembly school network invested half a million dollars in a tool that evaluates sentiment in teacher instruction, similar to the process demonstrated in this tutorial, and provides feedback based on that sentiment analysis.

In our tutorial, sentiment analysis was used to better understand student feedback and make personal adjustments to teaching methods and not make personnel or grading decisions.

The rollout at Urban Assembly schools raises ethical concerns about the use of AI in employee performance reviews. According to Harvard Business Review, a major concern is the lack of transparency regarding the data the AI uses for sentiment analysis. This uncertainty makes it difficult to assess potential biases, particularly those related to language nuances such as accent, regional dialects, non-verbal communication, and subtle forms of human interaction like humor.

However, a study by professors at the Indiana University Kelley School of Business found that employees generally believe AI to be less biased than humans in performance reviews.

Based on these articles and your experience in this tutorial, what do you think are some best practices to balance the efficiency of AI with the potential consequences of its use in performance reviews?