A Simple Postgres Logger for OpenAI Endpoints

Open Source Python Package

Being able to log LLM traces (i.e. prompts and responses) underpins:

1. Evaluating the performance of LLM products

2. Creating synthetic data to fine-tune and improve LLMs

There are many tools that can be used for logging (Braintrust, Humanloop, Langfuse etc.), but here’s a really simple one:

• pip install trelis-openai-logger

• from trelis-openai-logger import OpenAI

• create the same OpenAI client you always use, but pass in a postgres URL.

It’s a two line swap and all of your queries are logged to a Postgres database that you can easily run locally or run cheaply in the cloud (e.g. a Digital Ocean Droplet).

Cheers, Ronan

P.S. 🛠️ (NEW) Trelis Fine-tuning and Evals Seminars - learn more here.

---

Trelis Links:

🤝 Are you a talented developer? Work for Trelis

💡 Need Technical or Market Assistance? Book a Consult Here

💸 Starting a New Project/Venture? Apply for a Trelis Grant

---

Setting Up LLM Response Logging with Trelis OpenAI Logger

The trelis-openai-ai logger library provides a straightforward way to log Large Language Model (LLM) interactions to a Postgres database. This tool helps track prompts, responses, latency, and token usage for production deployments.

Core Functionality

1. Logs all prompts and responses to Postgres

2. Records model name, latency, token counts, and metadata

3. Works with OpenAI-style API endpoints

4. Supports both local and remote database deployments

Installation & Basic Usage

from Trelis_openai_logger import OpenAI
client = OpenAI(
   postgres_url="postgresql://localhost:5432/llm_logs",
   api_key="your-api-key" # Optional
)

Database Setup Options

1. Local Postgres

1. Install Postgres (brew install postgresql@15 on Mac)

2. Create database: createdb llm_logs

3. Run migrations using dbmate

4. Connection string: postgresql://localhost:5432/llm_logs

2. DigitalOcean Droplet

1. Minimum 2GB RAM recommended

2. Uses cloud-init for automated setup

3. Runs Postgres directly on virtual machine

4. Approximate cost: $12/month

3. DigitalOcean Managed Database ~$15/month

1. Fully managed Postgres instance

2. Built-in backups and monitoring

3. Higher reliability for production use

4. Connection string provided by DigitalOcean

Data Structure

The logger stores:

1. Model name

2. Endpoint used

3. Input messages

4. Raw response

5. Latency

6. Token counts (prompt, completion, total)

7. Additional metadata

Use Cases

1. Performance monitoring

2. Quality evaluation

3. Data collection for fine-tuning

4. Usage tracking and optimization

The logger is available as an open-source package on GitHub and PyPI, installable via pip install trelis-openai-logger .