Keyboard shortcuts

Press ← or → to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

Needle Banner

We have found the Needle in haystack! 🪡🔍

Needle is an open-source image retrieval database with high accuracy that can easily handle complex queries in natural language. It is Fast, Efficient, and Precise, outperforming state-of-the-art methods. Born from high-end research, Needle is designed to be accessible to everyone while delivering top-notch performance. Whether you’re a researcher, developer, or an enthusiast, Needle opens up innovative ways to explore your image datasets. ✨

See Needle in Action

*Watch as Needle transforms natural language queries into precise image retrieval results in real time.*

🎨 Try the Interactive Demo

Experience Needle’s full capabilities with our interactive demo! Test different queries, explore the interface, and see how Needle works with real data.

The demo showcases sample queries, image generation, and similarity search with pre-processed datasets.

Comparison to State-of-the-Art Methods

Curious how Needle measures up against other cutting-edge approaches? Here, you’ll soon find performance plots that compare Needle with OPEN-AI CLIP image retrieval method for LVIS, Caltech256 and BDD100k.

Mean Average Precision Charts

Mean Average Precision Across Datasets (All Queries)

Mean Average Precision Across Datasets (Hard Queries)

User Study Preferences

Get Started Today!

Ready to revolutionize your image retrieval process? 🚀
Install and test Needle now to experience the future of multimodal search!

Tip: For detailed installation instructions, check out the Getting Started section.

Cite us

For a deep dive into Needle’s theoretical guarantees and performance insights, please refer to our research paper.

If you find Needle beneficial for your work, we kindly ask that you cite our work to help support continued innovation.

@article{erfanian2024needle,
  title={Needle: A Generative-AI Powered Monte Carlo Method for Answering Complex Natural Language Queries on Multi-modal Data},
  author={Erfanian, Mahdi and Dehghankar, Mohsen and Asudeh, Abolfazl},
  journal={arXiv preprint arXiv:2412.00639},
  year={2024}
}

Getting Started

Prerequisites

Before installing Needle, ensure that you have the following prerequisites installed:

  • Docker: Needle relies on Docker to containerize its infrastructure services.
    Install Docker

  • Docker Compose: This tool is required to orchestrate the multi-container setup.
    Install Docker Compose

  • Python 3.8+: Required for the backend and image generator services.
    Install Python

  • Git: Required for cloning the repository and managing updates.
    Install Git

Warning: Make sure your user account is added to the Docker group so you can run Docker commands (e.g., docker ps) without needing root privileges.

Note: Currently, Needle is supported on Linux and macOS.

Installation

To install Needle, run the following one-liner in your terminal:

curl -fsSL https://raw.githubusercontent.com/UIC-InDeXLab/Needle/main/scripts/install.sh -o install.sh && bash install.sh && rm install.sh

Manual Installation

If you prefer to install manually or need more control over the process:

  1. Clone the repository:

    git clone https://github.com/UIC-IndexLab/Needle.git
cd Needle

  2. Run the installation script:

    ./scripts/install.sh

Configuration Options

During the installation process, you will be prompted to choose the database mode. The available options are:

  • Fast: Offers quick responses and indexing with lower accuracy.
  • Balanced: Provides a compromise between speed and accuracy.
  • Accurate: Prioritizes high accuracy, which may come at the cost of slower performance.

Warning: Once the database mode is set, it cannot be changed without uninstalling and reinstalling Needle, which will result in data loss.

Note: Needle automatically checks for GPU accessibility and will use the GPU if available to optimize performance.

What Gets Installed

The installation process sets up:

  • Virtual Environments: Separate Python environments for backend and image generator services
  • Docker Infrastructure: PostgreSQL, Milvus, and Redis services via Docker Compose
  • Configuration Files: Performance-optimized settings based on your chosen mode
  • needlectl CLI: Command-line interface for managing Needle
  • Web UI: Optional web interface for easier interaction

After the installation completes, please close and reopen your terminal session (or source your shell configuration file) to ensure that all environment variables are correctly set.

Starting the Needle Service

Once installed, start the Needle service by running:

needlectl service start

This command will download the required model weights and initiate all the necessary services.

To verify that everything is running as expected, you can check the service status with:

needlectl service status

And confirm the installed version using:

needlectl --version

About needlectl

The needlectl command-line tool is the primary interface for interacting with Needle. It will be discussed in detail in the subsequent sections, where you’ll learn how to leverage its full capabilities.

Demo

Experience Needle’s capabilities with our interactive demo! The demo showcases the complete Needle workflow: text-to-image generation and similarity search.

🎯 What You’ll See

The demo demonstrates Needle’s core functionality:

  1. Text Queries: Click on sample queries to see how Needle works
  2. Image Generation: AI-generated images based on your text prompts
  3. Similarity Search: Find similar images from a curated dataset
  4. Real-time Results: See how Needle processes queries and returns results

🚀 Try the Demo

📋 Demo Features

Search Interface

  • Sample Queries: Pre-defined queries to test different scenarios
  • Generated Images: See AI-generated images for each query
  • Similarity Results: Browse through similar images from the dataset
  • Performance Metrics: View timing information for each search

Directory Management

  • Dataset Overview: See indexed directories and their status
  • Indexing Progress: Watch real-time indexing progress
  • Directory Details: View file counts, sizes, and metadata

Generator Configuration

  • Available Generators: Browse different image generation engines
  • Configuration Options: See how generators can be configured
  • Service Status: Monitor generator service health

System Status

  • Health Monitoring: Check overall system status
  • Service Overview: View all running services
  • Performance Metrics: See system performance indicators

🔧 Technical Details

The demo is built with:

  • Frontend: React with Tailwind CSS
  • Data: Pre-processed sample queries and images
  • API: Mock API simulating the full Needle backend
  • Images: Curated dataset of 1000+ images with embeddings

📁 Demo Structure

demo/
├── src/
│   ├── sample-queries.json    # Pre-defined queries and results
│   └── services/
│       └── mockApi.js         # Mock API responses
├── public/
│   └── demo-images/           # Sample images
└── scripts/
    └── build-sample-queries.py # Script to generate demo data

🎨 Customizing the Demo

You can customize the demo with your own queries and images:

  1. Add Images: Place your images in public/demo-images/
  2. Update Queries: Edit src/sample-queries.json
  3. Rebuild: Run the build script to update the demo

For detailed customization instructions, see the Demo Customization Guide.

🚀 Getting Started with Needle

After trying the demo, you can:

  1. Install Needle: Follow our Getting Started guide
  2. Use Needlectl: Learn about our command-line interface
  3. Configure Services: Set up your own image generation and search pipeline

The demo is updated automatically with each release. For the latest features, make sure you’re viewing the most recent version.

needlectl Command Line Tool

needlectl is the primary command-line interface that connects to the Needle backend service and performs a variety of tasks to manage and interact with Needle. It provides both general options and component-specific commands.

General Options

The following options are available with needlectl:

  • --api-url: Specifies the backend URL to connect to.
    Default: http://127.0.0.1:8000

  • --output: Determines the format of the output.
    Supported formats: human, json, yaml.
    Default: human.

  • --version or -v: Prints the version information for both the Needle backend and the needlectl tool.

  • --home or -H: Specifies the Needle installation directory.
    Default: Auto-detected from installation

  • --profile: Selects the runtime profile (prod or dev).
    Default: prod

  • --config-dir: Overrides the configuration directory path.

Note: --api-url and --output are accessible globally and in all commands, for example, in order to get the outputs of a query in JSON format you can use following command:

needlectl --output json query run "a wolf howling"

Profiles & Configuration

You can use the --home/-H flag to point needlectl at a custom Needle installation or local checkout, and select a runtime profile (prod or dev) that auto-configures the compose files and config directory.

Development Profile

needlectl --home $(pwd) --profile dev service start

This uses:

  • NEEDLE_CONFIG_DIR=$NEEDLE_HOME/configs/fast
  • Infrastructure: docker/docker-compose.infrastructure.yaml
  • Backend: Virtual environment
  • Image Generator: Virtual environment

Production Profile

needlectl --home /opt/needle --profile prod service start

This uses:

  • NEEDLE_CONFIG_DIR=$NEEDLE_HOME/configs/balanced (or your chosen mode)
  • Infrastructure: docker/docker-compose.infrastructure.yaml
  • Backend: Virtual environment
  • Image Generator: Virtual environment

Components Overview

needlectl is organized into several components, each dedicated to specific functions within the Needle system. For more detailed information, please refer to the corresponding pages for each component.

Service

Manages Needle’s core service operations, including:

  • Starting the service
  • Restarting the service
  • Stopping the service
  • Viewing logs
  • Updating components
  • Managing configuration

Directory

Handles image directory management tasks, such as:

  • Adding directories
  • Removing directories
  • Listing directories
  • Modifying directory settings
  • Viewing directory details

Query

Executes natural language queries against the image database, providing flexible and powerful search capabilities:

  • Running search queries
  • Viewing query logs
  • Managing query configuration

Generator

Manages image generation operations, allowing you to:

  • List available generators
  • Configure generator settings
  • Manage image generation parameters

UI

Controls the web-based user interface:

  • Starting the web UI
  • Stopping the web UI
  • Managing UI configuration

Service Component

The service component of needlectl manages the core operations of Needle. It connects to the Needle backend service and uses Docker Compose to start, stop, restart, check status, view logs, update components, and configure the service environment.

Note: Most commands rely on the --api-url option (default: http://127.0.0.1:8000) to connect to the backend API. You can set this globally if needed.

Commands Overview

start

Starts the Needle services by launching the required Docker containers and waiting for the backend API to become available.

  • What it does:

    • Uses the Docker Compose Manager to start containers.
    • Calls the backend API and waits until it is accessible.

  • Usage Example:

    needlectl service start

  • Output: Displays messages indicating that the services are starting, and if the backend API becomes available, it prints “Services started.” If not, an error is shown.

stop

Stops the Needle services by shutting down the Docker containers.

  • What it does:

    • Uses the Docker Compose Manager to stop all running containers.

  • Usage Example:

     needlectl service stop

  • Output: A confirmation message “Services stopped.” is printed once the containers are shut down.

restart

Restarts the Needle services by first stopping and then starting the Docker containers again. It also waits for the backend API to be available after the restart.

  • Usage Example: 
    needlectl service restart

stop

Stops the Needle services by shutting down the Docker containers.

  • What it does:

    • Uses the Docker Compose Manager to restart containers.
    • Waits for the backend API to respond after restarting.

  • Output: Displays messages for restarting and confirms when the services have been restarted successfully.

  • Usage Example:

    needlectl service stop

status

Retrieves and displays the current status of Needle services using the backend API.

  • Usage Example: 
    needlectl service status

log

Displays the logs from the Needle backend service.

  • Usage Example: 
    needlectl service log

update

Updates Needle components to the latest versions.

  • Options:

    • --force or -f: Force update even if already up to date
    • --component or -c: Update specific component (needlectl, backend, ui, or all)

  • What it does:

    • Checks for the latest release information
    • Shows current versions of all components
    • Updates the specified components
    • Provides status updates during the process

  • Usage Examples:

    # Update all components
needlectl service update

# Update only the backend
needlectl service update --component backend

# Force update even if up to date
needlectl service update --force

  • Output: Displays current and latest versions, then updates the specified components with progress indicators.

config

Manages the configuration for the Needle service environment.

  • Usage Example: 
    needlectl service config

Warning: Changes made via this command can affect service operations. Ensure you understand the impact before making modifications.

Directory Component

The directory component of needlectl is responsible for managing image directories in the Needle system. It allows you to add directories for image indexing, remove directories, list existing directories, modify directory configurations, and view detailed information about a directory.

Commands Overview

add

This command adds a new image directory to Needle’s indexing system.

  • Options:

    • path (string): The path to the directory to be added.
    • --show-progress (boolean, optional): If enabled, displays a progress bar showing the indexing progress.

  • What It Does:

    1. Connects to the backend API and adds the directory.
    2. If successful, returns a directory ID.
    3. If the --show-progress flag is set, a progress bar (powered by the tqdm library) is displayed. This progress bar periodically updates based on the indexing ratio reported by the backend and shows an estimated time for completion.

  • Usage Examples:

    # Add a directory without displaying indexing progress
needlectl directory add /path/to/my/images

# Add a directory and display the indexing progress
needlectl directory add /path/to/my/images --show-progress

remove

This command removes an image directory from Needle.

  • Options:

    • path (string): The path to the directory to be removed.

  • Usage Examples:

    needlectl directory remove /path/to/my/images

list

This command lists all the directories currently registered in Needle.

  • What it does:

    • Connects to the backend API
    • Retrieves the list of all registered directories
    • Displays directory information including status, file counts, and indexing progress

  • Usage Examples:

    # List all directories
needlectl directory list

# List directories with JSON output
needlectl --output json directory list

  • Output: Shows directory ID, path, name, status, file count, indexing progress, and other relevant information.

modify

This command allows you to modify the configuration for your added directories, you can enable/disable them for searching. Needle will not search in disabled directories.

  • Usage Examples: 
    needlectl directory modify

describe

This command provides detailed information about a specific directory by its directory ID.

  • Options:

    • did (integer): The unique identifier of the directory.

  • Usage Examples:

    needlectl directory describe 1

config

This command manages the environmental configuration for the directory component.

  • Usage Examples: 
    needlectl directory config

Query Component

The Query Component of needlectl allows you to interact with Needle’s image retrieval capabilities. It lets you execute natural language queries, view query logs, and configure query-related settings.

Commands Overview

run

This command executes a search query against the Needle image retrieval database.

  • Options:

    • prompt (string): The search prompt in natural language.
    • -n (integer, optional): Specifies the number of images to retrieve.
    • --num-images-to-generate (integer, optional): Specifies the number of images to generate.
    • --num-engines (integer, optional): Specifies the number of engines to use.
    • --image-size (string, optional): Sets the desired image size for image generation (e.g., “512”, “1024”).
    • --include-base-images (boolean, optional): Indicates whether to include base images in the results preview.
    • --use-fallback (boolean, optional): Indicates whether to use fallback mode if the primary engines set fails.

  • What It Does:

    1. Retrieves the enabled and activated generator configurations using the Generator Config Manager.
    2. If no generator configuration is found, it displays an error message suggesting you use needlectl generator config to configure a generator.
    3. Executes the search query with the provided parameters via the backend API.
    4. Displays the search results in the specified output format (human-readable, JSON, or YAML).

  • Usage Examples:

    # Run a basic query with a prompt
needlectl query run "red cars"

# Run a query with additional options
needlectl query run "red cars" -n 5 --num-images-to-generate 2 --num-engines 3 --image-size 512 --include-base-images true --use-fallback false

# Run a query with JSON output
needlectl --output json query run "mountain landscape" -n 10

log

This command retrieves and displays the logs of search queries executed by Needle.

  • Usage Examples: 
    needlectl query log

config

This command manages the environmental configuration for the query component.

  • Usage Examples: 
    needlectl query config

Generator Component

The Generator Component of needlectl is used to manage image generators for Needle. It allows you to list all available generators from the backend and configure generator settings that are used during image queries.

Commands Overview

list

This command lists all available image generators as retrieved from the backend.

  • What It Does:

    1. Connects to the backend API.
    2. Retrieves the list of available generators.
    3. Displays the result in the specified output format (human-readable, JSON, or YAML).

  • Usage Example:

    needlectl generator list

config

This command manages the configuration for image generators used by Needle.

  • Usage Example: 
    needlectl generator config

UI Component

The UI component of needlectl manages the web-based user interface for Needle. It allows you to start and stop the web UI, which provides a graphical interface for interacting with Needle’s features.

Commands Overview

start

Starts the Needle web UI on a specified port.

  • Options:

    • --port or -p: Port to run the UI on (default: 3000)

  • What it does:

    • Starts the web UI server
    • Serves the React-based interface
    • Provides access to all Needle features through a web browser

  • Usage Examples:

    # Start UI on default port (3000)
needlectl ui start

# Start UI on custom port
needlectl ui start --port 8080

  • Output: Displays the URL where the UI is accessible (e.g., http://localhost:3000)

stop

Stops the running Needle web UI.

  • What it does:

    • Stops the web UI server
    • Frees up the port for other uses

  • Usage Example:

    needlectl ui stop

  • Output: Confirms that the UI has been stopped

Web UI Features

The web interface provides:

  • Search Interface: Interactive query interface with sample queries
  • Directory Management: Visual management of image directories
  • Generator Configuration: Easy configuration of image generators
  • System Status: Real-time monitoring of service health
  • Results Visualization: Rich display of search results and generated images

Accessing the UI

Once started, you can access the web UI by opening your browser and navigating to:

  • Default URL: http://localhost:3000
  • Custom Port: http://localhost:[PORT] (where [PORT] is your chosen port)

The UI will automatically connect to your running Needle backend service.

Uninstallation

If you decide to remove Needle from your system, you can do so quickly by running the following one-liner command in your terminal:

curl -fsSL https://raw.githubusercontent.com/UIC-InDeXLab/Needle/main/scripts/uninstall.sh | bash

Note: This uninstallation command will remove Needle’s services, data and configurations. However, it will not remove any cached Docker images. If you need to free up additional disk space, please consider manually removing those images.