Skip to main content

Storm - Real-Time Water Management System

Welcome to the Storm documentation! This guide will help you understand, develop, and deploy Storm, a real-time data processing system built for water management and IoT monitoring.

About Storm​

Storm is a distributed real-time computation system built on Apache Storm that processes streaming data from IoT sensors in water management infrastructure. It ingests sensor data from Azure Event Hubs, processes it through a series of computational bolts, generates intelligent alerts based on configurable thresholds, and persists results to Azure Cosmos DB.

The system is designed to handle high-throughput, low-latency data processing for monitoring water quality, detecting anomalies, tracking energy consumption, and managing flow rates across multiple industrial units and locations.

info

Storm processes data in real-time, ensuring immediate detection and response to critical water quality issues, leaks, and operational anomalies.

Key Features and Capabilities​

Real-Time Data Processing​

  • High-throughput ingestion from Azure Event Hubs
  • Sub-second latency for critical alert detection
  • Parallel processing across distributed Storm workers
  • Stateful computation for tracking historical trends and patterns

Comprehensive Alert System​

  • Quality Alerts: Monitor water quality parameters (TDS, pH, turbidity, etc.)
  • Energy Alerts: Track power consumption and detect anomalies
  • Level Alerts: Monitor tank levels with threshold-based triggers
  • Flow Alerts: Detect flow rate changes, stable flow conditions, and leakage
  • Configurable thresholds per unit and industry
  • Multi-channel notifications (Email, SMS, WhatsApp, FCM push)
  • Permission-based alert routing with user groups and custom roles

Intelligent Data Processing​

  • Junk Data Handling: Filters out invalid, malformed, or duplicate sensor readings
  • Error Recovery: Built-in retry mechanisms and dead-letter queue handling
  • Data Validation: Schema validation and type checking before processing
  • State Management: Maintains previous states for delta calculations and trend analysis
  • Reference Object Pattern: Efficient state tracking for time-series data

Metrics and Analytics​

  • Instantaneous Metrics: Real-time sensor readings
  • Aggregated Metrics: Hourly, daily, and custom time-window aggregations
  • Historical Data: Long-term storage in Cosmos DB for trend analysis
  • Dashboard Integration: Metrics available for visualization dashboards

Scalability and Reliability​

  • Horizontal scaling with Storm's distributed architecture
  • Fault tolerance through Storm's acking mechanism
  • Guaranteed processing with configurable retry policies
  • Monitoring and observability with Sentry integration

Technology Stack​

Storm is built with modern, enterprise-grade technologies:

Core Framework​

Apache Storm 2.7.0
  • Distributed real-time computation system
  • Guaranteed message processing
  • Horizontal scalability
  • Fault-tolerant architecture

Programming Language​

Kotlin 1.9.22 (JVM 17)
  • Modern, concise syntax
  • Full Java interoperability
  • Null safety
  • Coroutines for async operations

Database​

Azure Cosmos DB 4.65.0
  • Globally distributed NoSQL database
  • Multi-model support (Document, Key-Value)
  • Automatic indexing
  • Configurable consistency levels

Message Queue​

Azure Event Hubs 2.5.0
  • High-throughput data streaming
  • Kafka-compatible protocol
  • Built-in partitioning
  • Message retention policies

Additional Technologies that​

Sentry 5.3.0         // Error tracking and monitoring
Jackson 2.18.1       // JSON serialization
OkHttp 4.12.0        // HTTP client for webhooks
Gradle 7.x           // Build automation

System Architecture Overview​