Shreder: Building a Distributed Cache System

Welcome to Shreder, a high-performance distributed cache system built in Go. This project will guide you through understanding, building, and using a production-ready distributed cache that implements consistent hashing and peer-to-peer replication.

What You'll Learn

By the end of this project, you'll understand:

• How distributed caching works
• Consistent hashing algorithms
• Peer-to-peer replication strategies
• Building scalable Go applications
• RESTful API design for distributed systems

Architecture Overview

Shreder implements a distributed cache using three core components:

┌─────────────┐    ┌─────────────┐    ┌─────────────┐
│   Node 1    │◄──►│   Node 2    │◄──►│   Node 3    │
│  :8060      │    │  :8061      │    │  :8062      │
└─────────────┘    └─────────────┘    └─────────────┘
       │                   │                   │
       └───────────────────┼───────────────────┘
                           │
                    Hash Ring
                 (Consistent Hashing)

Key Features

• Distributed Architecture: Data is automatically distributed across multiple nodes
• Consistent Hashing: Ensures even distribution and minimal data movement when nodes join/leave
• Automatic Replication: Data is replicated across peer nodes for fault tolerance
• RESTful API: Simple HTTP interface for cache operations
• High Availability: System continues operating even when individual nodes fail

Project Structure

This project is organized into the following topics:

1. Hash Ring Implementation - The consistent hashing algorithm
2. In-Memory Cache with LRU Eviction - Thread-safe cache with TTL support
3. HTTP Server and Request Routing - Request handling and forwarding
4. Peer-to-Peer Replication - Data replication for fault tolerance
5. Running the Cluster - Setting up and testing a multi-node cluster
6. Production Deployment - Security, configuration, and Docker deployment

Getting Started

Prerequisites

• Go 1.23.1 or later
• Basic understanding of HTTP and JSON

Installation

Clone and build the project:

git clone https://github.com/UnplugCharger/shreder.git
cd shreder
go mod tidy
go build -o bin/shreder main.go

Navigate to the next topic to start learning about the Hash Ring implementation.