a) If you perform matrix multiplication on the system they give you, you end up with two equations:
u1*x - u3 = 0, which I'm guessing is equivalent to u1*x -> u3
and
u2*y - 6*u3 = 0, which should be u2*y -> 6*u3
Looking at our original equation, we can conclude that the first equation is referring to the amounts of sulfur reacted, and the second equation is referring to the amounts of fluorine reacted.
u1*S8 + u2*F2 -> u3*S*F6
Thus, x = 8 and y = 2.
b) We need to perform matrix reduction on our newly found matrix
[8 0 -1]
[0 2 -6]
R1/8 -> R1
[1 0 -1/8]
[0 2 -6]
R2/2 -> R2
[1 0 -1/8]
[0 1 -3]
Thus a = -1/8 and b = -3.
Not sure if I explained this that clearly. The main things you need to know for these types of problems are matrix multiplication and row reduction.
http://tutorial.math.lamar.edu/Classes/DE/LA_Matrix.aspx and
http://tutorial.math.lamar.edu/Classes/Alg/AugmentedMatrix.aspx might be useful for those respectively.
Also, you can use
www.wolframalpha.com to row reduce matrices for you and not run the risk of simple algebra errors. For example, you can type in "row reduce {{8,0,1},{0,2,-6}}" to reduce the above matrix (
http://www.wolframalpha.com/input/?i=row+reduce+{{8,0,-1},{0,2,-6}}) and obtain the answer much more easily.
