Creating Vectors and Matrices in Numpy
Take the following column vector as an example
\(x = \begin{pmatrix} 1 \\2 \\3\\4\\5\end{pmatrix}\)
To write this vector in a way that python will understand we can create an array object in numpy and insert the values separated by commas
import numpy as np
x = np.array([1, 2, 3, 4, 5])
If we wanted to create a row vector we simply add an additional set of square brackets
\(x = \begin{pmatrix} 1 &2&3&4&5\end{pmatrix}\)
x = np.array([[1, 2, 3, 4, 5]])
Examples of creating matrices in numpy
\(A = \begin{pmatrix} 1.0 & 1.0 \\ 2.0 & 1.0 \\ 1.0 & -1.0 \end{pmatrix} \)
A = np.array([[1,1],
[2,1],
[1,-1]])
\(\begin{pmatrix} 10.1 & -4.0 & 7.0 \\ 8.0 & 5.6 & 2.0 \\ 4.0 & 2.5 & 2.0 \end{pmatrix} \)
X = np.array([[10.1, -4, 7],
[8, 5.6, 2],
[4, 2.5, 2]])
\(C = \begin{pmatrix} 2.0 & 3.0 & 4.0 & 5.0 & 6.0 \\ 1.0 & 3.0 & 5.0 & 2.0 & 7.0 \\ 4.0 & 5.0 & 2.0 & 6.0 & 3.0 \end{pmatrix} \)
C = np.array([[2, 3, 4, 5, 6],
[1, 3, 5, 2, 7],
[4, 5, 2, 6, 3]])
Transpose a Matrix
The transpose of a matrix is just flipping the rows and columns. So, if you have a matrix with 3 rows and 2 columns, the transpose will have 2 rows and 3 columns.
The transpose of a matrix is denoted by a superscript T. So, if the original matrix is A, the transpose is A^T.
\(\begin{pmatrix} 1.0 & 1.0 \\ 2.0 & 1.0 \\ 1.0 & -1.0 \end{pmatrix} ^T = \begin{pmatrix} 1.0 & 2.0 & 1.0 \\ 1.0 & 1.0 & -1.0 \end{pmatrix} \)
A = np.array([[1,1],
[2,1],
[1,-1]])
print(A.T)
'''
[[ 1 2 1]
[ 1 1 -1]]
'''
\(\begin{pmatrix} 10.1 & -4.0 & 7.0 \\ 8.0 & 5.6 & 2.0 \\ 4.0 & 2.5 & 2.0 \end{pmatrix} ^T = \begin{pmatrix} 10.1 & 8.0 & 4.0 \\ -4.0 & 5.6 & 2.5 \\ 7.0 & 2.0 & 2.0 \end{pmatrix} \)
X = np.array([[10.1,-4,7],
[8,5.6,2],
[4,2.5,2]])
print(X.T)
'''
[[10.1 8. 4. ]
[-4. 5.6 2.5]
[ 7. 2. 2. ]]
'''
\(\begin{pmatrix} 2.0 & 3.0 & 4.0 & 5.0 & 6.0 \\ 1.0 & 3.0 & 5.0 & 2.0 & 7.0 \\ 4.0 & 5.0 & 2.0 & 6.0 & 3.0 \end{pmatrix} ^T = \begin{pmatrix} 2.0 & 1.0 & 4.0 \\ 3.0 & 3.0 & 5.0 \\ 4.0 & 5.0 & 2.0 \\ 5.0 & 2.0 & 6.0 \\ 6.0 & 7.0 & 3.0 \end{pmatrix} \)
C = np.array([[2, 3, 4, 5, 6],
[1, 3, 5, 2, 7],
[4, 5, 2, 6, 3]])
print(C.T)
'''
[[2 1 4]
[3 3 5]
[4 5 2]
[5 2 6]
[6 7 3]]
'''
Scalar Matrix Multiplication
Take the following matrix A and say we want to multiply it by 5
\(5\times A = \begin{pmatrix} 1.0 \times 5& 1.0\times 5 \\ 2.0\times 5 & 1.0\times 5 \\ 1.0 \times 5& -1.0\times 5 \end{pmatrix} = \begin{pmatrix} 5 &5\\ 10 &5 \\ 5 & -5 \end{pmatrix} \)
A = np.array([[1,1],
[2,1],
[1,-1]])
print(5*A)
'''
[[ 5 5]
[10 5]
[ 5 -5]]
'''
Element Wise Multiplication
\(A=\begin{pmatrix} 1.0 & 2.0 \\ 3.0 & 4.0 \end{pmatrix} , B = \begin{pmatrix} 10.0 & 20.0 \\ 30.0 & 40.0 \end{pmatrix} \)
\(A\odot B \space \text{or} \space A*B = \begin{pmatrix} 1 \times10.0 & 2 \times20\\ 3 \times 30 & 4 \times 40 \end{pmatrix} = \begin{pmatrix} 10.0 & 40.0 \\ 90.0 & 160.0 \end{pmatrix} \)
A = np.array([[1,2],
[3,4]])
B = np.array([[10,20],
[30,40]])
print(A*B)
'''
[[ 10 40]
[ 90 160]]
'''
Dot Product / Matrix Multiplication in Numpy
To take the dot product of two vectors
\(x = \begin{pmatrix} 1 \\2 \\3\end{pmatrix} , y = \begin{pmatrix} 10 \\20 \\30\end{pmatrix}\)
\(xy = 1\times 10 + 2\times 20+ 3\times30 =140\)
x = np.array([1,2,3])
y = np.array([10,20,30])
print(np.dot(x ,y))
print(x @ y)
'''
140
140
'''
Take the two matrices below
\(A =\begin{pmatrix} 1.0 & 2.0 \\ 3.0 & 4.0 \end{pmatrix} , B = \begin{pmatrix} 10.0 & 20.0 \\ 30.0 & 40.0 \end{pmatrix} \)
\(AB = \begin{pmatrix} (1 \times 10+2\times30)&(1 \times 20 + 2 \times 40) \\ (3 \times 10 + 4\times 30)& (2\times30 + 4\times40) \end{pmatrix} = \begin{pmatrix} 70.0 & 100.0 \\ 150.0 & 220.0 \end{pmatrix} \)
A = np.array([[1,2], [3,4]])
B = np.array([[10,20], [30,40]])
print(A@B)
'''
[[ 70 100]
[150 220]]
'''
print(np.dot(A,B))
'''
[[ 70 100]
[150 220]]
'''
In the event that the dimensions are not aligned in that columns A do not equal rows B , with A being the matrix on the left of the multiplication, python will raise the following error
ValueError: matmul: Input operand 1 has a mismatch in its core dimension 0,
with gufunc signature (n?,k),(k,m?)->(n?,m?) (size 3 is different from 2)
Simultaneous Equations
Single Unknown
First lets take a few simple analogies that we will use to build intuition on what exactly the point of linear regression is. Take the very simply example below in which we have 2 times some quantity is equal to 5.
\(\begin{equation} 2x = 5 \end{equation}\)
We can easily solve for x by
\(\begin{equation} \begin{split} 2x & = 5 \\ x& = \frac{5}{2}\\ x & = 2.5 \end{split} \end{equation}\)
Note that we could also have used the notation \(x = 2^{-1}5 = 2.5\) i.e. taking the reciprocal of 2 to isolate x. In this example we have 1 unknown and only one equation, which is commonly taught to students at a very young age.
Multiple Unknowns
Now we take another example given below in which we have two unknown values.
\(\begin{equation} \begin{split} x + y&=10 \\ x -y& = 2\ \end{split} \end{equation}\)
Again, we will assume most readers know that solving this simply involves adding the 2 equations together as follows.
\(\begin{equation} \begin{split} x+x +y-y & = 10 + 2 \\ 2x& = 12\\ x & = 6 \end{split} \end{equation}\)
Note we can also complete this by setting the problem up in matrix form , taking the inverse and then taking the dot product. To set this up in matrix form we simply have to note how many times x and y appears in each equation.
\(\begin{equation} \begin{split} x + y&=10 \\ x -y& = 2\ \end{split} \end{equation}\)
So we see there is 1 x and 1y in the first equation and 1 x and -1 y in the second, we can then create a matrix we will call A with these values
\(A =\begin{pmatrix} 1 & 1 \\ 1 & -1 \end{pmatrix} \)
Then we can take the values on the left of the equality 10, 2 and place them in a column vector which we will call Y
\(Y =\begin{pmatrix} 10 \\ 2 \end{pmatrix} \)
Now we can restate the problem as
\(\begin{pmatrix} 1 & 1 \\ 1 & -1 \end{pmatrix} \begin{pmatrix} x \\ y \end{pmatrix} = \begin{pmatrix} 10 \\ 2 \end{pmatrix} \)
In a similar way when we took the reciprocal for a single equation, here we take the matrix inverse of A which is similar to what we have done before. Note we set \(\begin{pmatrix} x \\y\end{pmatrix} = b\) below.
\(\begin{equation} \begin{split} Ab&=Y \\ b& = A^{-1}Y \end{split} \end{equation}\)
Note that \(A^{-1}\) is being matrix multiplied by \(Y\) in the equation above.
The python script below is an example of how we can do this in code. Note that the @ operator is for matrix multiplication of numpy arrays and the np.linalg.inv() method is for taking the matrix inverse,
import numpy as np
A = np.array([[1,1],
[1,-1]])
Y = np.array([10,2])
b = np.linalg.inv(A) @ Y
print(b)
'''
[6. 4.]
'''
The inverse method shown above will work as long as
a) The matrix is not singular, think of a singular matrix as trying to divide \(\frac{x}{0}\) in the context of the single equation and single unknown example
b) There are as many equations as there are unknowns
However, for multiple regression we want to solve problems that take the following form.
\(\begin{pmatrix} 2 \\3 \\ \vdots \\5\end{pmatrix} = \begin{pmatrix} 1 & 1 & 2\\1 & 1 &2 \\ \vdots & \vdots & \vdots\\1 & 1 &1\end{pmatrix} \begin{pmatrix} x \\ y \\z\end{pmatrix} \)
Note in the problem above, we have more equations than we do unknowns. In the next article we will cover how equations of this form can be solved analytically.