System Architecture#

┌─────────────────┐    ┌──────────────────┐    ┌─────────────────┐
│   Price Data    │───▶│  Neural Network  │───▶│ Price Prediction│
│   (Nord Pool)   │    │   (18 inputs)    │    │   (24h ahead)   │
└─────────────────┘    └──────────────────┘    └─────────────────┘
                                                         │
                                                         ▼
┌─────────────────┐    ┌──────────────────┐    ┌─────────────────┐
│  Device Agents  │◀───│      HEMS        │◀───│  Q-Learning     │
│ (EV, HVAC, etc.)│    │   Coordinator    │    │   Optimizer     │
└─────────────────┘    └──────────────────┘    └─────────────────┘

Core Components#

1. Price Prediction Module (ANN)

   • 18 input features (time, weather, historical prices)
   • Sigmoid activation function
   • Levenberg-Marquardt training algorithm

2. Multi-Agent Decision System (MARL)

   • Independent Q-learning agents for each device
   • ε-greedy exploration strategy
   • Distributed optimization approach

3. Battery Management Models (4 implementations)

   • Non-Shiftable, Shiftable, Naïf, and Controllable approaches

1. Non-Shiftable Battery (NSL_Battery)#

Baseline model - Charges immediately when plugged in

• ✅ Simple implementation
• ❌ No cost optimization
• 🎯 Reference point for comparisons

2. Shiftable Battery (SL_Battery)#

Time-shifting model - Delays charging to optimal windows

• ✅ Significant cost savings (up to 1.99% improvement)
• ✅ Reduces peak demand
• ❌ Must charge continuously once started
• 🔧 Parameters: k (inconvenience cost), Tne (charging duration)

3. Naïf Battery (Naif_Battery)#

Greedy optimization - Charges during cheapest available hours

• ✅ Best overall performance (up to 16.89% improvement)
• ✅ Can split charging across multiple periods
• ✅ Simple yet effective algorithm
• 🔧 Parameters: deficit (charge level target)

4. Controllable Battery (CL_Battery)#

Continuous optimization - Adjusts charging power dynamically

• ✅ Maximum flexibility (up to 3.43% improvement)
• ✅ Real-time adaptation
• ❌ Most complex implementation
• 🔧 Parameters: β (satisfaction cost), action_number (power levels)

Scenario Analysis#

The system was evaluated across three scenarios with different user priorities:

Key Findings#

• Naïf Battery models consistently outperformed complex RL approaches
• Flexibility beats optimization - ability to split charging sessions was crucial
• Context matters - optimal strategy depends on user priorities
• Diminishing returns - simple heuristics often sufficient

Device Classes#

Each battery model inherits from base classes with specific behaviors:

class SL_Battery(Shiftable_load):
    def function(self):
        # Q-learning training loop
        for episode in range(loops):
            # Simulate charging scenarios
            # Update Q-table based on rewards
        
        # Execute optimal action
        return energy_consumed, utility_cost

Q-Learning Implementation#

# Q-value update rule
Q[state][action] = Q[state][action] + learning_rate * (
    reward + discount_factor * max(Q[next_state]) - Q[state][action]
)

Reward Function#

Balances cost optimization with user satisfaction:

def get_reward(self, price_index, start_time, energy_consumed):
    cost_component = (1-p) * price[price_index] * energy_consumed
    satisfaction_component = p * (k * (start_time - preferred_time))
    return 1 / (cost_component + satisfaction_component + epsilon)

Adding New Battery Models#

1. Create a new class inheriting from appropriate base class
2. Implement the function() method with your optimization logic
3. Add initialization in insert_devices() function
4. Configure parameters in the device list

Modifying Reward Functions#

Customize the reward calculation to reflect different optimization objectives:

def custom_reward(self, price, energy, time_delay, battery_health):
    return weighted_sum([
        price_component(price, energy),
        comfort_component(time_delay), 
        longevity_component(battery_health)
    ])