Found a solution to my problem : it had to do with the way I created the x and y rotation while handling drag events.
I replaced the creation of x and y rotation as to not use rotation Vectors (example below)
var x_rotation = event.relative.y * 0.05
var y_rotation = event.relative.x * 0.05
rotateCamera(-y_rotation, -x_rotation)
My guess is that rotation.y kept growing exponentially and was stuck between two values.