Getting Started with Redis in Python

by John

 

 

Using Redis with Python is incredibly useful for a range of applications. The ability to store keys and values in redis is similar to having an in-memory dictionary that can be used across processes and even machines! In this post, we will show how to connect to Redis with Python and perform some basic tasks like reading and writing keys. 

 

For this article we will assume that the reader has followed the previous post about setting up Redis through Docker

 

Establish Connection

 

First ensure you have created and activated a virtual environment, in which we need to install redis 

 

pip install redis

 

Create a script called test_redis.py with 

 

sudo nano test_redis.py

 

And copy the contents of the script below in to the file. 

 

import os

import redis

# get password from the bashrc file we made before
password = os.getenv("REDIS_PASSWORD")

if not password:
    raise EnvironmentError("❌ REDIS_PASSWORD environment variable is not set.")

redis_url = f"redis://:{password}@localhost:6379/0"

try:
    client = redis.from_url(redis_url, decode_responses=True)
    if client.ping():
        print("✅ Connected to Redis via URL!")
    else:
        print("❌ Ping failed — connection not confirmed.")
except Exception:
    raise

 

Run the script with 

python test_redis.py

 

 

Common Errors 

 

If you get the error shown below, this means that the password has not been set correctly, ensure you have followed the previous post.

 

redis.exceptions.AuthenticationError: invalid username-password pair or user is disabled.

 

The error below usually means that the docker container is not running, if you see this error, you need to ensure that your container is running

 

redis.exceptions.ConnectionError: Error 111 connecting to localhost:6379. Connection refused.

 

 

Reading  & Writing Keys to Redis with Python

 

Usually when I write keys to redis I use the hset command, first let's create some values to save to redis. We will start off very simple, by creating a list of 10 integers and saving them to redis

 

vals = [x for x in range(10)]

 

So the task will be to save this list of integers to a key and a name, and ensure we dump to json and save to redis 

 

import json 

client.hset(key="somekey:", name="someID", value=json.dumps(vals))

 

And if we want to read the values back, we can do the following

 

vals_ = client.hget(key="somekey:", name="someID")

print(vals_)

 

The command above results in the output shown below 

 

 

However, it is important to note that the vals_ (values we read back from redis are still in json format, so we need to convert them back with the json loads command shown below. 

 

 

 

 

Get all Data by Hash in Redis 

 

It often occurs that we want to save multiple keys in our Redis instance, it can get quite hard to remember , so a useful command to remember is the hkeys() command, which will return all values based on the given hash.  

 

For this example we will use US cities and generate fake weather data for each of the cities and store them by a key called weather:USA

 

Let's generate some fake data and save with the hget comamnd in order to show how this command can be useful. 

 

import redis
import json
import random
import os


hash_key = "weather:USA"

# Fake weather data
cities = [
    "New_York", "Los_Angeles", "Chicago", "Houston", "Phoenix",
    "Philadelphia", "San_Antonio", "San_Diego", "Dallas", "San_Jose"
]

for city in cities:
    temps = [str(random.randint(20, 35)) for _ in range(7)]  # 7 days of temps
    client.hset(name=hash_key, key=city, value=json.dumps(temps))

 

Then we can retrieive it with 

 

all_cities = client.hkeys(hash_key)
for city in all_cities:
    print(f" - {city}")

 

 

 

 

So note that this command, does not actually load all the values, it simply loads in the keys that are present for the specified name. This is a useful command to keep a track of what keys are actually inside the database. 

 

 

Delete Keys in Redis 

 

Let's delete all the weather keys in the database. We can do this with the hdel command , there are a few ways to do this, first we can do multiple cities at once 

 

client.hdel("weather:USA", "Dallas", "Houston", "Phoenix")

# verify that it is gone

print(client.hget(name="weather:USA", key="Dallas"))

 

 

 

Or we can delete them one by one in a loop

 

 

 

 

Write Multiple Values at once with Redis Pipeline

 

While we can take advantage of a command called mset in redis, to set multiple keys at the same time, I personally find it very annoying and not easy to use. Instead it is much better to use a pipeline to set multiple values at the same time, let's take an example of that by setting the same keys for city weather. 

 

pipeline = client.pipeline()

hash_key = "weather:USA"

for city in cities:
    temps = [str(random.randint(20, 35)) for _ in range(7)]
    pipeline.hset(name=hash_key, key=city, value=json.dumps(temps))

pipeline.execute()

 

This code, is looping over the cities as before, but the pipeline.hset() part is not actually writing to redis until we call the execute() method on the pipline object. This is a very neat way to write many keys at once in an efficient way. 

 

 

 

Read and Write a Pandas DataFrame 

 

Now that we have touched the basics, it is maybe worth doing something a bit more realistic in terms of how we can leverage redis in our code. When working with data in Python we are almost always using Pandas library. So it is probably a good idea to learn how to save a dataframe to redis and load it back in again. 

 

Create a random dataframe with the code below. 

 

import pandas as pd 
import numpy as np
import string
import random 


df = pd.DataFrame()

df['nums'] = np.random.normal(size=1000)
df['strings'] = [
    ''.join(random.choices(string.ascii_letters + string.digits, k=20))
    for _ in range(1000)
]

 

 

 

 

Now we have our dataframe, we need to convert it to a dictionary and save it to redis. This is actually very simple. We can take advantage of Pandas built in to_dict method, for this I have a preference for using orient='records' which gives us a list of dictionaries back. 

 

 

 

Write Pandas Dataframe to Redis

 

values = df.to_dict(orient='records')

client.hset(name='pandas', key='fake_df', value=json.dumps(values))

 

And we can read in back in to a dataframe with 

 

values_ = client.hget(name='pandas', key='fake_df')

df_ = pd.DataFrame(json.loads(values_))

df_

 

 

 

That is a little bit easier than it looks, quite often we are using timestamps in dataframes, and if we try to save a timestamp (pd.Timestamp) values we will get something like the error below 

 

File ~/.pyenv/versions/3.11.0/lib/python3.11/json/encoder.py:200, in JSONEncoder.encode(self, o)
    196         return encode_basestring(o)
    197 # This doesn't pass the iterator directly to ''.join() because the
    198 # exceptions aren't as detailed.  The list call should be roughly
    199 # equivalent to the PySequence_Fast that ''.join() would do.
--> 200 chunks = self.iterencode(o, _one_shot=True)
    201 if not isinstance(chunks, (list, tuple)):
    202     chunks = list(chunks)
...
    179     """
--> 180     raise TypeError(f'Object of type {o.__class__.__name__} '
    181                     f'is not JSON serializable')

TypeError: Object of type Timestamp is not JSON serializable

 

 

There are a whole range of different types of values in Pandas and Numpy that will raise this error, therefore it may be advisable to do the following, which converts everything to a string and then we just have to process the values when we are reading them back in. 

 

df_safe = df.convert_dtypes().astype(str)
values = df_safe.to_dict(orient='records')
client.hset(name='pandas', key='fake_df', value=json.dumps(values))

 

 

 


Previous Next