A class of correlation functions of half-BPS composite operators are computed exactly (at finite $N$) in the zero coupling limit of N=4 SYM theory. These have a simple dependence on the four-dimensional spacetime coordinates and are related to correlators in a one-dimensional Matrix Model with complex Matrices obtained by dimensional reduction of N=4 SYM on a three-sphere. A key technical tool is Frobenius-Schur duality between symmetric and Unitary groups and the results are expressed simply in terms of U(N) group integrals or equivalently in terms of Littlewood-Richardson coefficients. These correlation functions are used to understand the existence/properties of giant gravitons and related solutions in the string theory dual on $ AdS_5 \times S^5$. Some of their properties hint at integrability in N=4 SYM.