Advanced Filtering in Neo4j with GraphQL and Vector Embeddings

Neo4j, a popular graph database, offers powerful filtering capabilities that can be enhanced when combined with GraphQL and vector embeddings. This guide will explore how to implement advanced filtering techniques in Neo4j, particularly in the context of HybridRAG (Hybrid Retrieval Augmented Generation) systems.

Understanding HybridRAG

HybridRAG is an approach that combines symbolic methods (like database queries) with vector embeddings-based methods (like semantic search). This hybrid approach is particularly useful for improving results when working with large volumes of unstructured data.

Filtering in Neo4j

Neo4j’s graph structure allows for complex filtering based on node properties and relationships. When combined with vector embeddings and GraphQL, it becomes a powerful tool for advanced information retrieval.

Basic Filtering in Neo4j

Let’s start with a simple example of filtering movies in Neo4j:

MATCH (m:Movie)
WHERE m.genre = 'Sci-Fi' AND m.rating >= 8.0
RETURN m.title, m.rating

This Cypher query filters movies by genre and minimum rating.

Integrating Vector Embeddings

To enhance our filtering capabilities, we can add vector embeddings to our Neo4j nodes. These embeddings can represent semantic information about the movies, such as plot summaries or themes.

Adding Embeddings to Neo4j

First, let’s add vector embeddings to our movie nodes:

MATCH (m:Movie {title: 'Inception'})
SET m.embedding = [0.1, 0.2, ..., 0.9]

Filtering with Embeddings

Now we can combine traditional filtering with similarity search using these embeddings:

MATCH (m:Movie)
WHERE m.genre = 'Sci-Fi' AND m.rating >= 8.0
WITH m, gds.alpha.similarity.cosine(m.embedding, $query_vector) AS similarity
WHERE similarity > 0.7
RETURN m.title, m.rating, similarity
ORDER BY similarity DESC
LIMIT 5

This query filters movies by genre and rating, then ranks them by similarity to a given query vector.

Using GraphQL for Advanced Filtering

GraphQL provides a flexible interface for querying Neo4j, allowing clients to specify exactly what data they need. Here’s an example of how to set up a GraphQL schema and resolver for our movie database:

const typeDefs = `
  type Movie {
    id: ID!
    title: String!
    genre: String!
    rating: Float!
  }

  type Query {
    movies(genre: String, minRating: Float, vector: [Float!]): [Movie!]!
  }
`;

const resolvers = {
  Query: {
    movies: async (_, { genre, minRating, vector }, { driver }) => {
      const session = driver.session();
      try {
        const result = await session.run(
          `
          MATCH (m:Movie)
          WHERE m.genre = $genre AND m.rating >= $minRating
          WITH m, gds.alpha.similarity.cosine(m.embedding, $vector) AS similarity
          RETURN m.id AS id, m.title AS title, m.genre AS genre, m.rating AS rating
          ORDER BY similarity DESC
          LIMIT 5
          `,
          { genre, minRating, vector }
        );
        return result.records.map(record => record.toObject());
      } finally {
        await session.close();
      }
    },
  },
};

Implementing HybridRAG with Neo4j and Vector Search

To create a full HybridRAG system, we can combine Neo4j’s graph capabilities with a specialized vector search engine like Qdrant. Here’s a Python example that demonstrates this hybrid approach:

from neo4j import GraphDatabase
from qdrant_client import QdrantClient

neo4j_driver = GraphDatabase.driver("bolt://localhost:7687", auth=("neo4j", "password"))
qdrant_client = QdrantClient("localhost", port=6333)

def hybrid_search(query_text, genre, min_rating):
    # Step 1: Get vector embedding for the query text
    query_vector = get_embedding(query_text)  # Implement this function to get embeddings

    # Step 2: Perform vector search in Qdrant
    vector_results = qdrant_client.search(
        collection_name="movies",
        query_vector=query_vector,
        limit=20
    )

    # Step 3: Use Neo4j for filtering and additional information
    with neo4j_driver.session() as session:
        result = session.run("""
        MATCH (m:Movie)
        WHERE m.id IN $ids AND m.genre = $genre AND m.rating >= $min_rating
        RETURN m.id AS id, m.title AS title, m.rating AS rating
        """, ids=[r.id for r in vector_results], genre=genre, min_rating=min_rating)
        
        return [record.data() for record in result]

# Example usage
results = hybrid_search("science fiction time travel", genre="Sci-Fi", min_rating=8.0)
for movie in results:
    print(movie)

This example demonstrates how to:

1. Perform a vector search in Qdrant based on a query.
2. Use the results to filter and fetch additional data from Neo4j.
3. Apply additional filters (genre and rating) within Neo4j.

Best Practices for Filtering in Neo4j

1. Index Creation: Create indexes on properties used frequently in filters to improve query performance.
2. Query Optimization: Use EXPLAIN and PROFILE to analyze and optimize your Cypher queries.
3. Batching: When dealing with large result sets, use batching to avoid overwhelming the database or the client.
4. Caching: Implement caching strategies for frequently accessed data to reduce database load.
5. Regular Database Maintenance: Regularly update statistics and perform database maintenance to ensure optimal performance.

Conclusion

Combining Neo4j’s graph capabilities with vector embeddings and GraphQL provides a powerful framework for implementing advanced filtering and search functionalities. By leveraging these technologies together, you can create sophisticated HybridRAG systems that offer both the flexibility of graph databases and the semantic understanding of vector embeddings.

Remember to optimize your queries, create appropriate indexes, and consider the scale of your data when implementing these solutions. With proper implementation, you can create highly efficient and accurate information retrieval systems that cater to complex query needs.