BSc
Bachelor thesis, scheduling EV charging with reinforcement learning
My BSc thesis at Sapienza: four ways to model an electric car inside a home energy management system, simulated over a year, and the finding that the simplest of them beat both reinforcement-learning models.
Electricity is cheaper at some hours than others, and an electric car is a large, flexible load that mostly does not care when it charges as long as it is full by morning. The thesis reproduces a published home-energy-management algorithm and extends it with a car, modelled four different ways, to find out which framing actually pays.
- Institution
- Sapienza University of Rome, 2021
- Supervisor
- Prof. Igor Melatti
- Simulation
- One year, 8,760 hourly steps
- Compared
- 4 battery models × 3 cost-versus-comfort weightings
- Repository
- VincenzoImp/bachelor-thesis
The problem
Shifting a load to a cheaper hour is only free if nobody notices. Charging the car at 3am saves money; leaving it half full at 8am does not, and the two effects have to be traded against each other explicitly rather than assumed away. The thesis does that with a weighting between cost and comfort, evaluated at three settings so the answer is not tuned to one preference.
The other constraint is knowledge. Two of the four models need to know when the car will be unplugged; one does not. That turns out to matter more than the learning algorithm.
What I built
A simulator in which each household device runs as its own thread with its own independent Q-learner – no shared table – stepping hour by hour through a year of real price and consumption data. Four models of the car sit inside it: a do-nothing baseline, a shiftable battery that postpones a single contiguous charge, a controllable battery that modulates charging power across twenty-five levels, and a deliberately naive one that simply charges whenever it is cheap.
Hard parts
The simplest model won. The naive battery beat both reinforcement-learning models in every scenario, because it can split charging across several disjoint cheap intervals while the shiftable model can only move one block. Its best result cuts cost by 10.8% while charging exactly as many kilowatt-hours; a variant that leaves the battery 3% short reaches 16.5%, at the cost of 5.8% less energy delivered, and loses outright once comfort is weighted heavily.
Three deviations from the reference paper, each with a negative result recorded. The paper’s state definition was unclear, so both a two-dimensional and a three-dimensional Q-table were tried: the flatter one won for the shiftable battery, the richer one for the controllable battery, whose utility genuinely depends on state of charge. The paper’s convergence criterion never converged – the car’s bursty load keeps the environment too dynamic – so a fixed episode count replaced it. And carrying a Q-table across hours, rather than resetting it, degraded results as the table accumulated bad tendencies.