The origins of the Library can be traced back to 1926 with the establishment of the Junior College of the Lower Rio Grande Valley which later was renamed as Brownsville Junior College in 1931. The library of these early Brownsville institutions of higher learning was located in the Brownsville High School on Palm Boulevard, and was part of the high school library. In 1948 the Library was relocated to a wing of Gorgas Hall on what was previously the site of Fort Brown. Fort Brown was de-activated in 1947 and purchased by the City of Brownsville and the Brownsville Independent School District resulting in the relocation of Brownsville Junior College from Brownsville High School to the grounds of old Fort Brown.
By 1954 the Library, now named the Zachary Taylor Library, was moved into Stokely Hall within the Jacob Brown Civic Center. The Library served the needs of both Texas Southmost College (the new institutional name for Brownsville Junior College as of 1949) and the City of Brownsville. During the 1960's an additional building served as a “reserve” library. This building was a surviving Fort Brown building (no longer standing) which was located in front of Gorgas Hall and briefly served to supplement the library facility at the Jacob Brown Civic Center by providing students with materials placed on reserve by college instructors and as a storage area for the library's journal collection. In 1966 the Library again relocated, moving from Stockely Hall, to the beginnings of the present day structure. In order to facilitate the growth of the library’s book collection, a second floor was added in 1975. The Library was renamed as the Arnulfo L. Oliveira Memorial Library in 1980 upon the untimely death of Dr. Oliveira, formerly a president of Texas Southmost College.
By 1983 the Library collection was converted from the Dewey Decimal Classification System to the Library of Congress Classification System. During the late 1980’s the Library retired its card catalog and implemented its first online catalog.
The partnership between Texas Southmost College and the University of Texas at Brownsville was enacted by the Texas Legislature in 1991 and the Library then became a part of the University of Texas System. During the period of 1994-95, the City of Brownsville withdrew its material from the Library in order to relocate its collection to the present day Brownsville Public Library and the Library at this point, became solely an academic library.
The University Boulevard Library was opened in 2009 and much of the highly circulated material was moved to this new facility. After the separation of UTB and TSC was announced in late 2010, the University Boulevard Library was renamed "University Library" and the majority of the remaining library services, staff, and collections were moved to this structure. The Arnulfo L. Oliveira Memorial Library was renamed by TSC as the Oliveira Student Center in 2013. Only Special Collections & Archives now remain in the Oliveira Student Center.