The pgvector extension allows PostgreSQL to support vector searches. This ability is useful for implementing RAG solutions. The link has all the background information on the technology. This post will not duplicate the info.
This post is a runbook on setting up:
a Docker environment with PostgreSQL and the pgvector extension.
a Django application with the client libraries ready to use the vector extension of the PostgreSQL DB.
Setup (Docker)
The first step is to create a “service” (docker-compose concept) for the DB:
services:
...
postgres:
# Check for latest version as necessary/desired
image: pgvector/pgvector:pg17
environment:
# remove before pushing up to source control
- POSTGRES_PASSWORD=mysecretpassword
ports:
- "5432:5432"
expose:
- "5432"
volumes:
# Optional; use a host volume
- ./data/psql:/var/lib/postgresql/data
Then bring up the service once to initialize the DB:
> docker-compose up postgres
...
postgres-1 | 2025-02-13 03:05:22.288 UTC [1] LOG: database system is ready to accept connections
The image pgvector/pgvector:pg17 is a Docker image for PostgreSQL 17 with the pgvector extension bundled.
Create a DB (if necessary)
Create a DB to be used if necessary.
> docker-compose run --rm postgres psql -h postgres -U postgres
Password for user postgres: .....
...
> create database mydb;
> \q
The POSTGRES_PASSWORD environment entry can now be removed from docker-compose.yml.
Django Connection to PostgreSQL
Add the psycopg dependency so we can use PostgreSQL.
This is a useful but difficult-to-remember command for PSQL:
\pset pager off
Or, if the invocation of psql is within control, add the -P pager=off argument:
psql -P pager=off ...
This turns off a “more” type of feature when displaying long results to allow paging up and down the results.
Unlike when using text-base terminals, most terminals nowadays are in a GUI shell with scroll buffers. We can scroll back and forward just fine, thank you very much.
With the basics down, the next step in this little adventure w/ Javascript is to make it work in a React.js environment (Next.js to be more precise).
There were some iterations. What emerged was (using a Next.js framework):
app/models.ts — the model types (Card) I’m working with.
app/cards/page.tsx — the component displaying the card “carousel” and a scroller.
app/cards/ui/CardCarousel.tsx — component displaying the card “carousel.”
app/cards/ui/cardcarousel.module.css — styling for the CardCarousel.
app/cards/ui/Scroller.tsx — component displaying the scroller (I will omit this in the post since it’s not really relevant).
app/lib/ui/renderers.tsx — a renderer function that renders a Card.
The hierarchy of files will differ from time to time and from person to person. What I consider the main trick to get this working with React.js is in the app/cards/ui/CardCarousel.tsx::Carousel component:
The Trick
To avoid the component immediately rendering the center card, I instead render the card passed into the component immediately in its left and right cards that are initially hidden, leaving the center card blank (or whatever it was before). This means that the component should render the center card using the model stored in a useRef value which starts out being undefined.
Then, in a useEffect, call a function that adds a slide CSS class to the left (or right, depending on the direction I want to slide) card. This should kick off the translation animation. At the same time, start a timer (just like the raw JS example) matching the animation duration to:
update the useRef.current to now point to the card passed in.
remove the slide CSS class so that the cards snap back into place.
trigger a refresh so that the center card gets rendered using the updated useRef.current (which should now be the card passed into the carousel).
'use client'
...
import { renderCard } from "@/app/lib/ui/renderers";
import { Carousel, Direction } from "../ui/CardCarousel"
...
export default function Page() {
...
// Tracks which Card (index into an array) is being displayed and how it
// should slide into the view
const [cardDisplayState, dispatch] = useReducer(
cardDisplayStateReducer, {index: 0, slideDirection: Direction.LEFT}
)
...
const { cards } = stack
return (
<div className="...">
...
<Carousel
t={cards[cardDisplayState.index]}
slideDirection={cardDisplayState.slideDirection}
renderer={renderCard}
/>
<Scroller ... />
</div>
)
}
app/cards/ui/CardCarousel.tsx
import styles from "@/app/cards/ui/cardcarousel.module.css"
...
export enum Direction {
LEFT,
RIGHT
}
const SLIDE_CSS_CLASSNAME: Record<Direction, string> = {
[Direction.LEFT]: "slide-left",
[Direction.RIGHT]: "slide-right"
}
/**
* Parameters to the Carousel component.
*
* @param t the item model being rendered into a "card" to be displayed
* @param slideDirection the direction the new card will slide in to replace the center card
* @param renderer a function that will render the card for the item model
*/
type Parameters<T> = {
t?: T
slideDirection: Direction
renderer: (c: T|undefined) => JSX.Element
}
export function Carousel<T>({ t, slideDirection, renderer }: Parameters<T>) {
const currentTRef = useRef<T|undefined>(undefined)
// This is just to allow us to force an update (got this trick from a search)
const [, updateState] = useState<{}>();
const forceUpdate = useCallback(() => updateState({}), []);
const slideWithCssClassName = useCallback((slideCssClassName: string) => {
const elems = Array.from(document.getElementsByClassName("card"))
// Adding the class will trigger the animation
elems.forEach(elem => {
elem.classList.add(styles[slideCssClassName])
})
// Set up a timer to reset the elems (hopefully immediately after the animation completes)
setTimeout(() => {
// By this time, the slide animation should be complete. We can snap the elems back
// to the original positions by removing the class.
// currentTRef is not sync'ed, so this should not trigger a refresh on its own. We
// don't want it to in order to not interfere with the slide.
currentTRef.current = t
elems.forEach((elem) => {
elem.classList.remove(styles[slideCssClassName])
})
// Since currentTRef is not sync'ed, force an update manually now.
forceUpdate()
}, 300)
}, [t, currentTRef.current])
useEffect(
() => {
setTimeout(
() => slideWithCssClassName(SLIDE_CSS_CLASSNAME[slideDirection]),
150
)
},
[t, slideDirection]
)
// Truncated most of the Tailwind CSS classes for brevity
return (
<div className="...">
<div className="...">
<div className="card ... left-[-600px]" id="previous-card">
{renderer(t)}
</div>
<div className="card ..." id="current-card">
{renderer(currentTRef.current)}
</div>
<div className="card ... left-[600px] " id="next-card">
{renderer(t)}
</div>
</div>
</div>
)
app/lib/ui/renderers.tsx
export function renderCard(card: Card|undefined): JSX.Element {
// Nothing special; just render the title and copy of the card
const cardDisplay = card ? (
<div className="...">
<div className="flex-none font-bold">{card!.title}</div>
<div className="grow">{card.copy}</div>
</div>
) : (
<></>
)
return <div className="size-full">{cardDisplay}</div>
}
renderer === Card component?
That renderCard function in renders.tsx looks like just a Card component. Why not just write the component and call it from Carousel?
That’s what I started with. However, doing this will couple the Carousel component to the Card. I thought I’d try to have the Carousel be more reusable for other models/components.
The result is that a Carousel<T> that is bound to a model T and accepts a renderer function that knows how to render a “card” for that T. The Carousel<T> will just delegate the rendering of T to that renderer when needed.
So far I don’t know if there is a better way to accomplish that decoupling. An obvious approach is to have Carousel<T> accept a children as other composite components do. However, I think in that case children would be the rendered “current” card, and there is no way to hold on to the “previous” card without pushing up the logic of maintaining a “previous” vs. “current” card to the parent of the Carousel, and that seems to me like a leakage of the implementation concerns of the Carousel to its parent.
For now, therefore, the Carousel<T> will require a function that renders a “card” for the T. That function will be called with either the current or previous T depending on where in the flow the Carousel is.
And as can be seen with the Page.tsx, the parent (Page.tsx in this case) doesn’t need to worry about which is the previous Card and which is the current (or current vs. next) Card; the parent just passes the T (in this case Card) it wants to slide in, and the Carousel<T> component takes care of maintaining that previous/current state. The cost is that the parent does need to pass in a rendering function for the Card / T.
Do I get points taken off for using a rendering function instead of a “component”? 🤔
Result
I’ve increased the dimensions of the card (from 200×100 in Part 1) to 600×350. Again, I have skipped the whole Scroller component above since there’s nothing special there. However, it’s displaying those two chevrons at the bottom used to scroll between 3 cards.
Part of a project I’m working on involves some animation of “cards” sliding in and out of existence. This trinity of HTML/CSS/JS serves as a basis for that functionality.
The illusion is that there is an endless (at least until integer overflow anyway) number of rectangles that will scroll either forward or backward.
The truth is that there are only 3 such rectangles. Theoretically only 2 are needed, but that comes with the cost of more complexity in the code. When a scroll is happening, there is one rectangle that is scrolling out, then there is another one scrolling in to take its place. Doing it once is easy with CSS transform. However, to continue to do it over and over again seamlessly requires some trickery.
The trick is that, once the transform animation to replace the center rectangle with the new rectangle completes, we need to immediately shift the rectangles back to be ready for the next scroll. This is done by the reset() function. The sequence looks like this:
Render the new rectangle (off screen).
Scroll the center rectangle out.
Scroll the new rectangle in.
Render the center rectangle (off screen) to match the new rectangle.
Reset the position of the rectangles so that the center rectangle is again at the center. Because the content of the rerendered center rectangle now matches that of the new rectangle, this should look seamless.
This is done by the functions slideRightWithCount and slideLeftWithCount. Adding the slideRight or slideLeft class to the rectangles will animate the sliding for 0.3s. At the same time, set a timer for 300ms (0.3s) to then call reset() which positions the rectangles to be ready for another scroll. There is some 🤞 going on assuming that the transform animation is done by the time the timer expires and reset() is called (preferrably at the same time). If for some reason the timer expires earlier than the full transform is complete, there can be some flickering (easily tested by either increasing the transition duration and/or decreasing the timer expiry).
function flipImage() {
const container = document.querySelector('.flip-container');
container.classList.toggle('flipped');
}
I plugged in two JPGs I had lying around, and it did work quite well.
With that, I tweaked it to fit more what I had in mind.
HTML:
<html lang="en">
<head>
<meta charset="UTF-8">
<meta name="viewport" content="width=device-width, initial-scale=1.0">
<title>Image Flip Animation</title>
<link rel="stylesheet" href="styles.css">
</head>
<body>
<div class="flip-container">
<div class="flip-element">
<div class="textcard-container front">
<div class="title">FRONT</div>
<div class="body">This is the front card</div>
</div>
<div class="textcard-container back">
<div class="title">BACK</div>
<div class="body">
Lorem ipsum dolor sit amet, consectetur adipiscing elit. Pellentesque imperdiet libero eu neque facilisis, ac pretium nisi dignissim. Integer nec odio. Praesent libero. Sed cursus ante dapibus diam. Sed nisi. Nulla quis sem at nibh elementum imperdiet. Duis sagittis ipsum. Praesent mauris. Fusce nec tellus sed augue semper porta. Mauris massa. Vestibulum lacinia arcu eget nulla.
</div>
</div>
</div>
</div>
<div id="flipper">
<button onclick="flipImage()">Flip</button>
</div>
<script src="script.js"></script>
</body>
</html>
Looking at the various pieces, here are the notes:
Stacking Two Elements
The first thing is stacking the two cards with the “front” card on top of the “back” card. This is done by:
Use the same dimensions for two DIVs (cards) inside a parent DIV (container).
Add transform-style: preserve-3d for the container so that whatever animation we apply will stay.
Add a transition: transform 0.3s (or however long desired for the animation sequence) to the container as well.
Use absolute positioning for the cards so that they overlap.
Give the front card a z-index of 2 (anything higher than that of the back card).
Set the backface-visibility property to hidden. This ensures that cards flipped will be hidden.
Rotate the back card by 180 degrees. This makes the back card invisible for now. It’s strictly not necessary since it’s obscured by the front card. However, this will be important when we do the flip.
Flip Animation
Set a CSS rule “flipped” that, when applied to the container, will rotate both child cards 180 degrees. This will:
Flip the front card 180 degrees, and because we have backface-visibility: hidden, the front card will now be invisible (even though its z-index is still higher than that of the back card).
Flip the back card 180 degrees. Since the back card was initially flipped 180 degrees, another 180 degrees now flips it to be front-facing. And since it’s now no longer flipped, it will be visible!
In summary, when we flip both cards, we make the front card invisible and the back card visible.
If we didn’t need to animate, then theoretically we could leave the back card alone (visible) and just flip the front card.
However, because we ARE animating the flip, leaving the back card stationary will have it show through as the front card rotates in and out of visibility. Therefore, flipping both cards will give the smooth illusion that we are flipping a two-faced card with the front and back cards’ contents. Sneaky, isn’t it?
Now it just relies on toggling the CSS class “flipped” on the container. And that’s where the JavaScript comes in.
I didn’t know that JavaScript now can toggle a class for an element easily using the {element}.classList.toggle({class name}) function. I used to have to split the class list into an array and then iterate through the array to either add or skip and then setting the class list to the resulting list. This was a nice surprise.
Perspective
Setting perspective: 1000px on the container adds a nice touch that, when flipping, there’s a small 3-D effect. Though it’s something I’d miss if I blinked.
The following notes apply when using Celery with the RabbitMQ broker.
Celery Settings
task_acks_late
The Celery setting task_acks_late (by default disabled), if set, will defer message ACK with RabbitMQ until the task completes successfully.
If it is enabled, and the Celery task takes too long (see consumer_timeout below), the message will be requeued and redelivered to the next worker. This will cause the message to be processed multiple times.
If it is disabled, then Celery will ACK the message to RabbitMQ as soon as the task starts. This tells RabbitMQ that the message was “delivered and processed,” and RabbitMQ will delete this message from the queue. This will cause the message to be lost if the Celery task was interrupted before it finishes.
task_acks_on_failure_or_timeout
Then there is task_acks_on_failure_or_timeout (by default enabled). According to the doc, this will ACK the message even if the task failed or timed out. This may or may not be the correct choice depending on the task.
RabbitMQ Settings
consumer_timeout
The RabbitMQ consumer_timeout configuration (by default 30 minutes) gives a task a maximum timeout before requeuing the message. If a client (a Celery task in this case) does not ACK the message before this timeout expires, the message will be requeued and redelivered.
TTL / message TTL
Furthermore, the TTL / messageTTL configuration (by default ??) determines how long a message will stay in RabbitMQ before being discarded. If a message stays in a queue for longer than this time, then it is removed from the queue.
Implications
To increase the chance that tasks are not aborted/lost due to restarts (e.g. deployments):
Design each task to finish as soon as possible. Spawning additional smaller tasks if necessary.
The entire task is designed to survive a partial completion state and is able to completely restart without ending up in a bad state (e.g. duplicate records created, abandoned/orphaned/partial updates).
Enable task_acks_late so that RabbitMQ ACKs are not sent until tasks are finished.
Extend the TTL / message TTL for RabbitMQ so that we have enough time to consume messages (in case the Celery task needs to be paused for some time to fix bugs).
There is no support to retry failed tasks. In fact, unless there is a Result Backend set up, there won’t even be a good way to audit which tasks failed (other than application logs).