Performance Optimization

info

AquaGen API is designed for high throughput with large IoT data volumes. This document covers the key performance strategies implemented throughout the system.

Cosmos DB: Partition Key Strategy

tip

The partition key choice critically affects query performance and cost.

Container	Partition Key	Rationale
industries	industryId	All industry queries filter by ID
users	industryId	Users always queried within industry context
devices_data	date	Date is the primary filter for device readings
processed_data	Industry-based	Optimized for monthly aggregation queries

note

Cross-partition queries are used only when necessary (e.g., admin queries across all industries). Most production queries use partition-key-aligned access to minimize RU consumption.

Parallel Queries with ThreadPoolExecutor

For report generation covering multiple dates or units, DatabaseSupporter uses concurrent.futures.ThreadPoolExecutor to parallelize Cosmos DB queries:

import concurrent.futures

with concurrent.futures.ThreadPoolExecutor() as executor:
    futures = [
        executor.submit(DatabaseSupporter.get_device_data_by_date, date, industry_id, unit)
        for date in date_range
    ]
    results = [f.result() for f in concurrent.futures.as_completed(futures)]

info

This reduces report generation time from O(n) sequential to near-O(1) parallel, especially significant for monthly reports spanning 30 days.

CachedData: In-Memory Cache

app/cachedData/cachedData.py — CachedData

Holds frequently-accessed, rarely-changing data in memory to avoid repeated Cosmos DB reads:

• Standard categories (standardCategoriesMap): All entries from standard_categories_master container, loaded at app startup via CachedData.initData().
• JWT Manager: Flask-JWT-Extended JWTManager instance.

class CachedData:
    standardCategoriesMap = {}
    jwt = None

    @classmethod
    def initData(cls):
        categories = DatabaseSupporter.get_all_standard_categories()
        cls.standardCategoriesMap = {c['standardCategoryId']: c for c in categories}

note

User context caching: Industry data, unit mappings, and alert configs are loaded once per request via load_user() and stored in Flask's current_user proxy. All services access this cached data rather than re-querying the database.

pandas for Data Aggregation

Raw device readings are loaded from Cosmos DB into pandas DataFrames for efficient aggregation:

dataframe = pd.DataFrame(raw_results)
dataframe = dataframe.replace({np.nan: None})
# Aggregations run in pandas (in-memory) rather than Cosmos DB queries
daily_totals = dataframe.groupby('date')['value'].sum()

tip

This avoids complex Cosmos DB aggregation queries (which consume more RUs) by doing aggregation in application memory using pandas' highly optimized operations.