In fact, these characteristics of Brahms led to problems right from the beginning. His first opportunity to enter the composing world was through Franz Liszt, who was impressed with the young man's talent and who was a frequent benefactor of young artists. However, Liszt's artistic sensibilities was not very well-suited to that of Brahms.
[Liszt's] new school... tended towards a certain loosening of the fetters of tonality, and demanded, in particular, that the musical form be dictated by the content of poetic ideas. Brahms, however, did not find it possible to become a disciple... [He] firmly rejected an artistic conception that was in some degree derived from an extra-musical point of view.It is instructive to understand the context of the musical world at the time: Beethoven was a huge influence and overshadowed the entire musical landscape. Wagner and Liszt responded by exploding the old forms and crafting new ones upon the Romantic model; the former by embracing operatic and dramatic themes, and the latter by embracing virtuosity and poetic forms. Brahms responded by recalling and recasting the older forms, synthesizing them with the Romanticism that Beethoven championed. Perhaps it was inevitable, in this idealistic period, that the musical community began lining up behind one school or the other, and using only the most intense rhetoric.
But this "synthesizing" of Brahms gives some insight into his musical sensibilities. Though he was schooled in the Romanticism of his day, he was a classicist at heart. He had a large personal collection of music manuscripts and edited the published works of such composers as Handel, Mozart, Schubert, and Schumann. He was particularly interested and versed in the contrapunctual art of Bach. Moreover, he was quite reserved and shunned dramatic outward expressions of emotion; he expressed himself openly with only his closest few friends, among whom were Robert and Clara Schumann. Thus the strong emotions characteristic of the Romantic period (and characteristic of Brahms himself) were bound and interpreted through the rigorous structure of music. At times the emotion will "break through" in his compositions but never in a way that threatens musical integrity.
Overall, then, the music of Brahms tends to be intense, introspective, complex, and very serious. It's not easy to listen to Brahms with "half an ear" or as background music. His music is also tightly edited: there's very little fluff or fat to his compositions, and demands much of the listener. After hearing a movement from his 4th symphony, a critic quipped: "I felt I was being thrashed by two very clever men."
However, this doesn't relegate his works to music theorists alone. The music of Brahms will open up to the interested ear, and will richly reward those who take the time to listen. It's not rare for me to listen to one of his compositions a couple of times in succession, in order to grasp it better. This is not to say that all of his works are uniformly serious; certainly his Hungarian Dances were and still remain extremely popular. They show as well as anything the combination of his Romantic expressionism with his desire to preserve those old rhythms in a more structured form. I had the opportunity to perform one of his choral works with a choir a couple of years ago, and I found singing the piece both intellectually stimulating and deeply emotional. Even if his symphonies leave you cold, don't let that deter you from enjoying Johannes Brahms.