Unit - 2

Matrices

2.1 Linear dependence of vectors

Linear dependence-

A set of n – vectors. x1, x2, …….., xr is said to be linearly dependent if there exist scalars. k1, k2, …….,kr not all zero such that

k1 + x2k2 + …………….. + xrkr = 0     … (1)

Important notes-

• Equation (1) is known as a vector equation.
• If the vector equation has non – zero solution i.e. k1, k2, ….,kr not all zero. Then the vector x1, x2, ……….xr are said to be linearly dependent.
• If the vector equation has only trivial solution i.e.

k1 = k2 = …….=kr = 0. Then the vector x1, x2, ……,xr are said to linearly independent.

Linear combination-

A vector x can be written in the form.

x = x1 k1 + x2 k2 + ……….+xrkr

Where k1, k2, ………….., kr are scalars, then X is called linear combination of x1, x2, ……, xr.

Results:

1. A set of two or more vectors are said to be linearly dependent if at least one vector can be written as a linear combination of the other vectors.
2. A set of two or more vector are said to be linearly independent then no vector can be expressed as linear combination of the other vectors.

2.2 Eigen values and Eigen vectors, Reduction to diagonal form

First we will go through some important definitions before studying Eigen values and Eigen vectors.

1. Vector-

An ordered n – touple of numbers is called an n – vector. Thus the ‘n’ numbers x1, x2, …………xn taken in order denote the vector x. i.e. x = (x1, x2, ……., xn).

Where the numbers x1, x2, ………..,xn are called component or co – ordinates of a vector x. A vector may be written as row vector or a column vector.

If A be an mxn matrix then each row will be an n – vector & each column will be an m – vector.

2. Linear dependence-

A set of n – vectors. x1, x2, …….., xr is said to be linearly dependent if there exist scalars. k1, k2, …….,kr not all zero such that

k1 + x2k2 + …………….. + xrkr = 0… (1)

3. Linear independence-

A set of r vectors x1, x2, ………….,xr is said to be linearly independent if there exist scalars k1, k2, …………, kr all zero such that

x1 k1 + x2 k2 + …….. + xrkr = 0

Important notes-

1. Equation (1) is known as a vector equation.
2. If the vector equation has non – zero solution i.e. k1, k2, ….,kr not all zero. Then the vector x1, x2, ……….xr are said to be linearly dependent.
3. If the vector equation has only trivial solution i.e.

k1 = k2 = …….=kr = 0. Then the vector x1, x2, ……,xr are said to linearly independent.

4. Linear combination-

A vector x can be written in the form.

x = x1 k1 + x2 k2 + ……….+xrkr

Where k1, k2, ………….., kr are scalars, then X is called linear combination of x1, x2, ……, xr.

Results:

1. A set of two or more vectors are said to be linearly dependent if at least one vector can be written as a linear combination of the other vectors.
2. A set of two or more vector are said to be linearly independent then no vector can be expressed as linear combination of the other vectors.

Example 1

Are the vectors x1 = [1, 3, 4, 2], x2 = [1, 3, 4, 2], x3 = [3, -5, 2, 6], linearly dependent. If so, express x1 as a linear combination of the others.

Solution:

Consider a vector equation,

x1k1 + x2k2 + x3k3 = 0

i.e., [1, -3, 4, 2]k1 + [3, -5, 2,6]k2 + [2, -1, 3, 4]k3 = [0, 0, 0, 0]

∴ k­1 +3k2 + 2k3 = 0

+3k1 + 5k2 + k3 = 0

4k1 + 2k2 + 3k3 = 0

2k1 + 6k2 + 4k3 =0

Which can be written in matrix form as,

R2 + 3R1, R3 – 4R1, R4 – 2R1

1/14  R2

R3 + 10R2

Here ρ(A) = 2 & no. Of unknown 3. Hence the system has infinite solutions. Now rewrite the questions as,

k1 + 3k2 + 2k3 =0

k2 + ½ k3 =0

Put k3 =t

k2 = -1/2 t and

k1 = -3k2 – 2k3

= - 3 (- ½ t) -2t

= 3/2 t – 2t

= - t/2

Thus k1 = - t/2, k2 = -1/2 t – k3 = t ∀ t ∈ R

i.e., - t/2 × 1 + ( - t/2) x2 + t × 3 = 0

i.e.,- x1/2 – (-x2/2) + x3 = 0

- x1 – x2 + 2 x3 = 0

x1 = - x2 + 2 x3

Since F11k2, k3 not all zero. Hence x1, x2, x3 are linearly dependent.

Example 2

Examine whether the following vectors are linearly independent or not.

x1 = [3, 1, 1], x2 = [2, 0, -1] and x3 = [4, 2, 1]

Solution:

Consider the vector equation,

x1k1 + x2k2 + x3k3 = 0

i.e., [3, 1, 1] k1 + [2, 0, -1]k2 + [4, 2, 1]k3 = 0              ...(1)

3k1 + 2k2 + 4k3 = 0

k1 + 2k3 = 0

k1 – k2 + k3 = 0

Which can be written in matrix form as,

R12

R2 – 3R1, R3 – R1

+ ½ R2

R3 + R2

Here Rank of coefficient matrix is equal to the no. Of unknowns. i.e. r = n = 3.

Hence the system has unique trivial solution.

i.e. k1 = k2 = k3 = 0

i.e. vector equation (1) has only trivial solution. Hence the given vectors x1, x2, x3 are linearly independent.

Example 3

At what value of P the following vectors are linearly independent.

[1, 3, 2], [2, P+7, 4], [1, 3, P+3]

Solution:

Consider the vector equation.

[1, 3, 2]k1 +  [2, P+7, 4]k2 + [1, 3, P+3]k3 = 0

i.e.

k1 + 2k2 + k3 = 0

3k1 + (p + 7)k2 + 3k3 = 0

2k1 + 4k2 + (P +3)k3 = 0

This is a homogeneous system of three equations in 3 unknowns and has a unique trivial solution.

If and only if Determinant of coefficient matrix is non zero.

consider .

i.e.,

1[(P + λ)(P+3) – 12] – 2[3(P+3) – 6] + 1[12 – (2P + 14)] 0

P2 + 10P + 21 – 12 -6P – 6 – 2P – 2

P2 + 2P + 1

(P + 1)2

(P + 1)

P

Thus for the system has only trivial solution and Hence the vectors are linearly independent.

Note:-

If the rank of the coefficient matrix is r, it contains r linearly independent variables & the remaining vectors (if any) can be expressed as linear combination of these vectors.

Characteristic equation:

Let A he a square matrix, λ be any scalar then |A – λI| = 0 is called characteristic equation of a matrix A.

Note:

Let a be a square matrix and ‘λ’ be any scalar then,

1)     [A – λI] is called characteristic matrix

2)     | A – λI| is called characteristic polynomial.

The roots of a characteristic equations are known as characteristic root or latent roots, Eigen values or proper values of a matrix A.

Eigen vector:

Suppose λ1 be an Eigen value of a matrix A. Then ∃ a non – zero vector x1 such that.

[A – λI] x1 = 0                                                                             ...(1)

Such a vector ‘x1’ is called as Eigen vector corresponding to the Eigen value λ1

Properties of Eigen values:

1. Then sum of the Eigen values of a matrix A is equal to sum of the diagonal elements of a matrix A.
2. The product of all Eigen values of a matrix A is equal to the value of the determinant.
3. If λ1, λ2, λ3, ..., λn  are n Eigen values of square matrix A then 1/λ1, 1/λ2, 1/λ3, ..., 1/λn     are m Eigen values of a matrix A-1.
4. The Eigen values of a symmetric matrix are all real.
5. If all Eigen values are non –zero then A-1 exist and conversely.
6. The Eigen values of A and A’ are same.

Properties of Eigen vector:

1. Eigen vector corresponding to distinct Eigen values are linearly independent.
2. If two are more Eigen values are identical then the corresponding Eigen vectors may or may not be linearly independent.
3. The Eigen vectors corresponding to distinct Eigen values of a real symmetric matrix are orthogonal.

Example1: Find the sum and the product of the Eigen values of ?

Sol.   The sum of Eigen values =  the sum of the diagonal elements

   =1+(-1)=0

The product of the Eigen values is the determinant of the matrix

On solving above equations we get

Example2: Find out the Eigen values and Eigen vectors of ?

Sol.  The Characteristics equation is given by

Or

Hence the Eigen values are 0,0 and 3.

The Eigen vector corresponding to Eigen value is

Where X is the column matrix of order 3 i.e.

This implies that

Here number of unknowns are 3 and number of equation is 1.

Hence we have (3-1) = 2 linearly independent solutions.

Let

Thus the Eigen vectors corresponding to the Eigen value are (-1,1,0) and (-2,1,1).

The Eigen vector corresponding to Eigen value is

Where X is the column matrix of order 3 i.e.

This implies that

Taking last two equations we get

Or

Thus the Eigen vectors corresponding to the Eigen value are (3,3,3).

Hence the three Eigen vectors obtained are  (-1,1,0), (-2,1,1) and (3,3,3).

Example3: Find out the Eigen values and Eigen vectors of

Sol.  Let A =

The characteristics equation of A is .

                                           Or 

                                                Or 

                                                      Or

                                                       Or

The Eigen vector corresponding to Eigen value is

Where X is the column matrix of order 3 i.e.

         Or

On solving we get 

Thus the Eigen vectors corresponding to the Eigen value is (1,1,1).

The Eigen vector corresponding to Eigen value is

Where X is the column matrix of order 3 i.e.

Or    

On solving or .

Thus the Eigen vectors corresponding to the Eigen value is (0,0,2).

The Eigen vector corresponding to Eigen value is

Where X is the column matrix of order 3 i.e.

   Or    

On solving we get  or .

Thus the Eigen vectors corresponding to the Eigen value is (2,2,2).

Hence three Eigen vectors are (1,1,1), (0,0,2) and (2,2,2).

Example-4:

Determine the Eigen values of Eigen vector of the matrix.

Solution:

Consider the characteristic equation as, |A – λI| = 0

i.e., (8 – λ) (21 - 10 λ + λ2 – 16) + 6(-18 + 6 λ + 8) + 2(24 – 14 + 2 λ) =0

(8- λ)( λ2 -10 λ + 5) + 6(6 λ – 10) + 2(2 λ + 10) = 0

8 λ2 - 80 λ + 40 – λ3 + 10 λ2 - 5 λ + 36 λ – 60

- λ3 + 18 λ - 45 λ = 0

i.e., λ3 – 18 λ + 45 λ  = 0

Which is the required characteristic equation.

λ(λ2 - 18 λ + 45) = 0

λ(λ – 15)( λ – 3) = 0

λ = 0, 15, 3 are the required Eigen values.

Now consider the equation

[A – λI] x = 0                                           ...................(1)

Case I:

If λ = 0                          Equation (1) becomes

R1 + R2

R2 + 3R1, R3 – R1

-1/1 R2

R3 + 5R2

Thus ρ[A – λI] = 2 & n = 3

3 – 2 = 1 independent variable.

Now rewrite equation as,

2x1 + x2 – 2x3 = 0

x2 – x3 = 0

Put x3 = t

x2 = t &

2x1 = - x2 + 2x3

2x1 = -t + 2t

x1 = t/2

Thus x1 =

Is the eigen vector corresponding to λ  = t.

Case II:

If λ  = 3 equation (1) becomes,

R1 – 2R3

R2 + 6R1 , R3 – 2R1

½ R2

R3+R2

Here ρ[A – λI] = 2, n = 3

3-2 = 1 Independent variables

Now rewrite the equations as,

x1 + 2x2 + 2x3 = 0

8x2 + 4x3 = 0

Put x3 = t

x2 = ½ t &

x1 = -2x2 – 2x3

= - 2 ( ½ ) t – 2 (t)

= - 3t

x2 =

Is the eigen vector corresponding to λ = 3.

Case III:

If λ = 15 equation (1) becomes,

R1 + 4R3

½ R3

R13

R2 + 6R1 , R3 – R1

R3 – R2

Here rank of [ A – λ I] = 2 & n = 3

3 – 2 = 1 independent variable

Now rewrite the equations as,

x1 – 2x2 – 6x3 = 0

-20x2 – 40x3 =0

Put x3 = t

x2 = - 2t & x1 = 2t

Thus x3 =

Is the eigen vector for λ = 13.

Example-5:

Find the Eigen values of Eigen vector for the matrix.

Solution:

Consider the characteristic equation as |A – λ I| = 0

i.e.,

(8 – (-3 + 3λ – λ + λ2 – 8) + 8(4 - 4 λ + 6) – 2(-16 + 9 + 3 λ) = 0

(8 – λ)( λ2 + 2 λ – 1) + 8(0 - 4 λ) – 2(3 λ – 7) = 0

8 λ2 + 16 λ – 88 – λ3 - 2 λ2 + 11 λ + 80 - 32 λ - 6 λ + 14 = 0

i.e., - λ3 + 6 λ2 - 11 λ + 6 = 0

λ3 - 6 λ2 + 11 λ + 6 = 0

(λ – 1)( λ – 2)( λ – 3) = 0

λ = 1, 2, 3 are the required eigen values

Now consider the equation

[A – λI] x = 0                                             ..............(1)

Case I:

λ  = 1 Equation (1) becomes,

R1 – 2R3

R2 – 4R1, R3 – 3R1

R3 – R2

Thus ρ[A – λI] = 2 and n = 3

3 – 2 = 1 independent variables

Now rewrite the equations as,

x1 – 2x3 = 0

-4x2 + 6x3 = 0

Put x3= t

x2 = 3/2 t , x1 = 2b

x1 =

I.e.

The Eigen vector for

Case II:

If λ = 2 equation (1) becomes,

1/6 R1

R2 – 4R1, R3 – 3 R1

Thus ρ(A – λI) = 2 & n = 3

3 – 2 = 1

Independent variables.

Now rewrite the equations as,

x1 – 4/3 x2 – 1/3 x3 = 0

1/3 x2 + 2/3 x3 = 0

Put x3 = t

1/3 x2 = - 2/3 t

x2 = -2t

x1 = - 4/3 x2 + 1/3 x3

= (-4)/3 (-2t) + 1/3 t

= 3t

X2 =

Is the Eigen vector for λ = 2

Now

Case III:

If λ = 3 equation (1) gives,

R1 – R2

R2 – 4R1, R3 – 3R1

R3 – R2

Thus ρ[A – λI] = 2 & n = 3

3 – 2 = 1

Independent variables

Now x1 – 2x2 = 0

2x2 – 2x3 = 0

Put x3 = t, x2 = t, x1 = 25

Thus

x3 =

Is the Eigen vector for λ = 3

Rayleigh's power method-

Suppose x be an eigen vector of A. Then the eigen value corresponding to this eigen vector is given as-

This quotient is called Rayleigh’s quotient.

Proof:

Here x is an eigen vector of A,

As we know that ,

We can write-

Diagonalization of a square matrix of order two

Two square matrix and A of same order n are said to be similar if and only if

for some non singular matrix P.

Such transformation of the matrix A into with the help of non singular matrix P is known as similarity transformation.

Similar matrices have the same Eigen values.

If X is an Eigen vector of matrix A then is Eigen vector of the matrix

Reduction to Diagonal Form:

Let A be a square matrix of order n has n linearly independent Eigen vectors Which form the matrix P such that

Where P is called the modal matrix and D is known as spectral matrix.

Procedure: let A be a square   matrix of order 3.

Let three Eigen vectors of A are  Corresponding to Eigen values

Let     

  {by characteristics equation of A}

Or

Or 

Note: The method of diagonalization is helpful in calculating power of a matrix.

  .Then for an integer n we have  

We are using the example of 1.6*

Example1:  Diagonalise the matrix

Let A=

The three Eigen vectors obtained are  (-1,1,0), (-1,0,1) and (3,3,3) corresponding to Eigen values .

Then    and

Also we know that    

Example2: Diagonalise the matrix

Let A =

The Eigen vectors are (4,1),(1,-1) corresponding to Eigen values .

Then    and also 

Also we know that    

2.3 Singular value decomposition, Sylvester’s theorem (Statement only)

Singular value decomposition:

The singular value decomposition of a matrix is usually referred to as the SVD. This is the final and best factorization of a matrix

Where U is orthogonal, Σ is diagonal, and V is orthogonal.

In the decomoposition

A can be any matrix. We know that if A is symmetric positive definite its eigenvectors are orthogonal and we can write A = QΛ. This is a special case of a SVD, with U = V = Q. For more general A, the SVD requires two different matrices U and V. We’ve also learned how to write A = SΛ, where S is the matrix of n distinct eigenvectors of A. However, S may not be orthogonal; the matrices U and V in the SVD will be

Working of SVD:

We can think of A as a linear transformation taking a vector v1 in its row space to a vector u1 = Av1 in its column space. The SVD arises from finding an orthogonal basis for the row space that gets transformed into an orthogonal basis for the column space: Avi = σiui. It’s not hard to find an orthogonal basis for the row space – the Gram Schmidt process gives us one right away. But in general, there’s no reason to expect A to transform that basis to another orthogonal basis. You may be wondering about the vectors in the null spaces of A and . These are no problem – zeros on the diagonal of Σ will take care of them.

Matrix form:

The heart of the problem is to find an orthonormal basis v1, v2, ...vr for the row space of A for which

A[ v1 v2 ... vr] = [ σ1u1  σ2u2  . . .   σrur] = [ u1  u2  . . .   ur]

With u1, u2, ...ur an orthonormal basis for the column space of A. Once we add in the null spaces, this equation will become AV = UΣ. (We can complete the orthonormal bases v1, ...vr and u1, ...ur to orthonormal bases for the entire space any way we want. Since , ... will be in the null space of A, the diagonal entries , ... will be 0.) The columns of U and V are bases for the row and column spaces, respectively. Usually U is not equal to V, but if A is positive definite we can use the same basis for its row and column space.

Sylvester theorem: Let A be the matrix of a symmetric bilinear form, of a finite dimenasional vector space, with respect to an orthogonal basis. Then the number of 1’s and -1’s and 0’s is independent of the choice of orthogonal basis.

Proof:

Let {e1, . . . , en} be an orthogonal basis of V such that the diagonal matrix A representing the form has only 1, -1 and 0 on the diagonal. By permuting the basis we can furthermore assume that the first p basis vectors are such that hei , eii = 1, the m next vectors give -1, and the final s = n − (p + m) vectors produce the zeros. It follows that s = n − (p + m) is the dimension of the null-space AX = 0. Therefore the number s = n − (p + m) of zeros on the diagonal is unique. Assume that we have found another orthogonal basis of V such that the diagonal matrix representing the form has first p 0 diagonal elements equal to 1, next m0 diagonal elements equal to -1, and the remaining s = n−(p 0 +m0 ) diagonal elements are zero. Let {f1, . . . , fn} be such a basis. We then claim that the p + m0 vectors e1, . . . , ep, fp 0+1, . . . , fp 0+m0 are linearly independent. Assume the converse, that there is a linear relation between these vectors.

Such a relation we can split as

             ..........................(1)

The vector occurring on the left hand side we call X. We have that

The vector X also occurs on the right hand side of (1), and gives that

These two expressions for the number hX, Xi implies that b1 = · · · = bp = 0 and that cp 0+1 = · · · = cp 0+m0 = 0. In other words we have proven tha

The p+ m0 vectors e1, . . . , ep, fp 0+1, . . . , fp 0+m0 are linearly independent. Their total number can not exceed s which gives p+m0 ≤ s. Thus p ≤ p 0 . However, the argument presented is symmetric, and we can interchange the role of p 0 and p, from where it follows that p = p 0 . Thus the number of 1’s is unique. It then follows that also the number of -1’s is unique.

The pair (p, m) where p is the number of 1’s and m is the number of -1’s is called the signature of the form. To a given signature (p, m) we have the diagonal (n × n)-matrix Ip,m whose first p diagonal elements are 1, next m diagonal elements are -1, and the remaining diagonal (n − p − m) elements are 0.

Examples:

1. Let V be the vector space of real (2 × 2)-matrices. (a) Show that the form hA, Bi = det(A + B) − det(A) − det(B) is symmetric and bilinear. (b) Compute the matrix A representing the above form with respect to the standard basis {E11, E12, E21, E22}, and determine the signature of the form. (c) Let W be the sub-vector space of V consisting of trace zero matrices. Compute the signature of the form discribed above, restricted to W.
2. 2. Determine the signature of the form hA, Bi = Trace(AtrB) on the space of real (n × n)-matrices.

2.4 Largest Eigen value and its corresponding Eigen vector by iteration method

Rayleigh's power method-

Suppose x be an eigen vector of A. Then the eigen value corresponding to this eigen vector is given as-

This quotient is called Rayleigh’s quotient.

Proof:

Here x is an eigen vector of A,

As we know that ,

We can write-

Gauss-Seidel iteration method-

Step by step method to solve the system of linear equation by using Gauss Seidal Iteration method-

Suppose,

This system can be written as after dividing it by suitable constants,

Step-1 Here put y = 0 and z = 0  and x = in first equation

Then in second equation we put this value of x that we get the value of y.

In the third eq. We get z by using the values of x and y

Step-2: we repeat the same procedure

Example: solve the following system of linear equations by using Guass seidel method-

6x + y + z = 105

4x + 8y + 3z = 155

5x + 4y - 10z = 65

Sol. The above equations can be written as,

   ………………(1)

………………………(2)

   ………………………..(3)

Now put z = y = 0 in first eq.

We get

x = 35/2

Put x = 35/2 and z = 0 in eq. (2)

We have,

Put the values of x and y in eq. 3

Again start from eq.(1)

By putting the values of y and z

y = 85/8 and z = 13/2

We get

The process can be showed in the table format as below

At the fourth iteration, we get the values of x = 14.98, y = 9.98 , z = 4.98

Which are approximately equal to the actual values,

As x = 15, y = 10 and  y = 5 ( which are the actual values)

Example: Solve the following system of linear equations by using Gauss-seidel method-

5x + 2y + z = 12

x + 4y + 2z = 15

x + 2y + 5z = 20

Sol. These equations can be written as,

   ………………(1)

………………………(2)

   ………………………..(3)

Put y and z equals to 0 in eq. 1

We get,

x = 2.4

Put x = 2.4, and z = 0 in eq. 2 , we get

Put x = 2.4 and y = 3.15 in eq.(3) , we get

Again start from eq.(1), put the values of y and z , we get

= 0.688

We repeat the process again and again,

The following table can be obtained –

Here we see that the values are approx. Equal to exact values.

Exact values are, x = 1, y = 2,  z = 3.

References:

1. E. Kreyszig, “Advanced Engineering Mathematics”, John Wiley & Sons, 2006.
2. P. G. Hoel, S. C. Port And C. J. Stone, “Introduction To Probability Theory”, Universal Book Stall, 2003.
3. S. Ross, “A First Course in Probability”, Pearson Education India, 2002.
4. W. Feller, “An Introduction To Probability Theory and Its Applications”, Vol. 1, Wiley, 1968.
5. N.P. Bali and M. Goyal, “A Text Book of Engineering Mathematics”, Laxmi Publications, 2010.
6. B.S. Grewal, “Higher Engineering Mathematics”, Khanna Publishers, 2000.