Posted on by Eric

Linear Quadratic Regulators

No, they’re not named after the three kids on my highschool football team that thought they were super cool and referred to themselves as the regulators.

Building off of what we talked about in the case of the Kalman filter, we can derive a state space regulator that is also optimal in some sense. This sense is to drive some state x to the smallest possible value as soon as possible, but without spending too much control effort to achieve that goal. Given a continuous-time linear time invariant (LTI) system of the form

\begin{aligned}  1) \displaystyle \qquad \dot{x} &= Ax + Bu, \quad x \in \mathbb{R}^n, u \in \mathbb{R}^k, \\  y &= Cx, \qquad y \in \mathbb{R}^m, \\  z &= Gx + Hu, \quad z \in \mathbb{R}^l \\  \end{aligned}

The above dynamic system looks more complicated that it needs to be because we have a measured output y, and a controlled output z. The measured output is the signal that we have available for measurement, whereas the controlled output is the signal that we would like to make as small as possible in the shortest time possible. It’s possible that z and y are the same, or z can be comprised of y and it’s derivative, etc.

The LQR problem is then to find a control input, u, that minimizes the following cost function:

\begin{aligned}  2) \displaystyle \qquad J_{LQR} = \int_0^\infty \| z(t) \|^2 + \rho \|u(t) \|^2 dt  \end{aligned}

\rho is simply a tuning parameter used to weigh which part of our minimization problem we want to place more importance on. Is it more important for us to minimize the energy of the controlled output, z? Then select \rho to be very small. Is it more important for us to minimize the energy of the control signal, u? Then select \rho to be very large. One of the things that makes the LQR a bit annoying to some is that the LQR problem involves minimizing both of these energies.

Equation 2 is useful because it illustrates more compactly what we are trying to achieve, but the LQR problem is usually written more generally in textbooks as

\begin{aligned}  3) \displaystyle \qquad J_{LQR} = \int_0^\infty \left ( z^T \bar{Q} z + \rho \, u^T \bar{R} u \right ) dt  \end{aligned}

Where \displaystyle \bar{Q} \in \mathbb{R}^{lxl} and \displaystyle \bar{R} \in \mathbb{R}^{mxm} are symmetric positive-definite matrices.

Or we can represent things in a bit more complicated way as:

\begin{aligned}  4) \displaystyle \qquad J_{LQR} = \int_0^\infty \left ( x^T Q x + u^T R u + 2x^T N u \right ) dt  \end{aligned}

Without getting into the math yet (algebraic Riccati equations, yay!) the feedback control law that minimizes the value of the above cost function is 

\begin{aligned}  5) \displaystyle \qquad u = -Kx  \end{aligned}

Where the control gain, K is given by:

\begin{aligned}  6) \displaystyle \qquad K = R^{-1} \left ( B^T P +N^T \right )  \end{aligned}

Where P is the solution to the, in this case, continuous time algebraic Ricatti equation (ARE):

\begin{aligned}  7) \displaystyle \qquad A^TP + PA + Q - \left (PB + N) \right ) ^{-1} \left (B^T P +N^T \right )=0  \end{aligned}

Note: another thing I’ve noticed is that algebraic Riccati equation is a bit of a buzz word with a lot of students not being able to define what one is and profs just throwing it around like “ofcourse, if you remember back to 3rd grade, just after learning your multiplication tables you’ll recall the ever present algebraic Ricatti equation.” -_-
Now, a Riccati equation is just any first-order ordinary differential equation that is quadratic in the unknown function. In our case, P is the unknown function, and if we carry out all of the matrix multiplications we can easily see that the equation in question is quadratic in P.

Now, this is about the point where people start to get frustrated with LQRs… not only do we have these stupid matrices Q and R that we have to figure out what to do with, but we also have to figure out how to solve the ARE above. Oh, and there’s now this nuisance of a cross term, N, that most books decide, in a super frustrating fashion, to just ignore talking about.

To arrive back at the more simple representation in equation 2 we use the following relations:

\begin{aligned}  8) \displaystyle \qquad Q &= G^T G\\  R &= H^T H + \rho I \\  N &= G^T H  \end{aligned}

And to arrive back at the slightly more general equation 3, use the following relations:

\begin{aligned}   9) \displaystyle \qquad Q &= G^T \bar{Q} G \\   R &= H^T \bar{Q} H + \rho \bar{R} \\   N &= G^T \bar{Q} H   \end{aligned}

If you’re using Matlab, as it seems like controls theoriticians are absolutely in love with (move to python, please), you can use the command:

[K, P, E] = lqr(A,B,Q,R,N)

to solve the ARE and compute the optimal state feedback gain matrix, K.

Now we are left with a couple things to scratch our head about.

First, what the hell do we do with those \displaystyle \bar{Q} and \displaystyle \bar{R} matrices? And second, how the hell do we actually solve that algebraic Riccati equation?

The first one is a bit easier to tackle. The simplest thing to do is set \displaystyle Q = I, \ R = \rho I . From here we just vary the single parameter \displaystyle \rho until we get something that has a decent response.

Many books suggest using Bryson’s Rule to begin your tuning of the LQR [1]

\begin{aligned}   10) \displaystyle \qquad \bar{Q}_{ii} &= \frac{1}{maximum \ acceptable \ value \ of \ z_i^2}, \qquad i \in \left (1,2,\dotsb,l \right ),\\   \bar{R}_{jj} &= \frac{1}{maximum \ acceptable \ value \ of \ u_j^2}, \qquad j \in \left (1,2, \dotsb, k \right )   \end{aligned}

which then corresponds to the following cost function

\begin{aligned}  11) \displaystyle \qquad J_{LQR} = \int_0^\infty \left ( \sum_{i=1}^l \bar{Q}_{ii} z_i(t)^2 + \rho \sum_{j=1}^m \bar{R}_{jj} u(t)^2 \right ) dt  \end{aligned}

This effectively sets the maximum acceptable value for each term to be 1. This generally gives semi-decent results, but really only acts as a starting point for us to start twiddling our control engineer knobs.

For example, if for our first state we want the maximum possible error to be 1 m and the second state to have a maximum possible error of 1/60 radians, we can use the following

\begin{aligned}  \displaystyle \qquad q_{11} &= \left ( \frac{1}{1} \right ) ^2 \qquad q_{11}x_1^2 = 1 \ when \ x_1 = 1m \\  q_{22} &= (60)^2 \qquad q_{22}x_2^2 = 1 \ when \ x_2 = \frac{1}{60} \ rad  \end{aligned}

The second question can be easy or hard… If you just want to use an already available library, like the python control library, you can just use the lqr method of the control class.

OR, you can write your own solver piggybacking on the scipy linear algebra library using the following code snippet[2]:

import numpy as np
import scipy.linalg
 
def lqr(A,B,Q,R):
    """Solve the continuous time lqr controller.
     
    dx/dt = A x + B u
     
    cost = integral x.T*Q*x + u.T*R*u
    """
    #ref Bertsekas, p.151
 
    #first, try to solve the ricatti equation
    X = np.matrix(scipy.linalg.solve_continuous_are(A, B, Q, R))
     
    #compute the LQR gain
    K = np.matrix(scipy.linalg.inv(R)*(B.T*X))
     
    eigVals, eigVecs = scipy.linalg.eig(A-B*K)
     
    return K, X, eigVals
 
def dlqr(A,B,Q,R):
    """Solve the discrete time lqr controller.
     
     
    x[k+1] = A x[k] + B u[k]
     
    cost = sum x[k].T*Q*x[k] + u[k].T*R*u[k]
    """
    #ref Bertsekas, p.151
 
    #first, try to solve the ricatti equation
    X = np.matrix(scipy.linalg.solve_discrete_are(A, B, Q, R))
     
    #compute the LQR gain
    K = np.matrix(scipy.linalg.inv(B.T*X*B+R)*(B.T*X*A))
     
    eigVals, eigVecs = scipy.linalg.eig(A-B*K)
     
    return K, X, eigVals

If you’re interested in writing your own solver, what you can do is solve the ARE via the eigenvalues and eigenvectors of the Hamiltonian matrix.

I’ll expand on this Hamiltonian matrix solution more later, but for now I’ll leave off by saying that we first construct the Hamiltonian matrix, find the eigenvalues and eigenvectors of it, selected the eigenvectors associated with stable eigenvalues, then compute the steady state solution P from these.

References

  1. Hespahna, J. (2009). Linear Systems Theory. Princeton University Press
  2. http://www.kostasalexis.com/lqr-control.html

Leave a Reply

Your email address will not be published. Required fields are marked *