In many ways, the recently refurbished Graves Art Gallery in Sheffield is still the good old, same old Graves it has been for the past 70 or so years.

It sits sedately in its suite of rectangular rooms on top of the rather engagingly austere 1930s City Library behind the Town Hall and now, rather poignantly, bang opposite the brash new kid on the block, the Millennium Galleries.

As we learn from the exemplary introductory text, JG Graves, whose collection lies at the heart of the city's art holdings, considered that "a well-chosen collection may interest and influence a wider public, and be readily accessible to the art student and scholars of the city, as well as to businessmen and women and workers generally".

It also happens to have one of the best civic collections in the country. It can't compare in size with the great municipal collections in Birmingham, Manchester, Liverpool and, for 20th-century art, Leeds, but, by dint of the original gift and some careful acquisitions, particularly in the 1950s and the 1980s, it offers a surprisingly representative sweep of European art.

The refurbishment has been going on room by room, so it has not been necessary to close the whole gallery. Indeed, the changes have been so incremental that occasional visitors have failed to register the full extent of what's actually been achieved.

Once the new partition walls that subdivide some of the long galleries had registered, I couldn't at first decide if they actually did improve things. Visually, they tend to disrupt the stately enfilade of the original galleries, but they have also provided valuable additional wall space and allowed the curators to create subsections that work to the collection's advantage.

The entry section is inspired. Once up a flight of stairs, you arrive in the still rather drab but helpfully-staffed shop/reception. The permanent displays are thematic and at the outset are rooted firmly in Sheffield itself.

"Sheffield, dingy hole"

Then there's the masterstroke. Instead of plunging straight into the thematic narrative, we're faced with John Singer Sargent's triple portrait of the Vickers sisters and his comments on receiving the commission: " I am to paint… three ugly women at Sheffield, dingy hole."

And there we have it neatly summed up; the new money, the brazenness of trade and industry and its pride in the face of the dismissive attitudes of the cultural establishment, which were the spur to all the great northern collections.

The Sargent portrait also acts as the entrée to Reflections: portraits from the 16th to the 18th centuries, with particularly engaging works by Lely, Lawrence and Romney.

The introductory panel to Devotions had dropped to the floor, but that didn't lessen the impact of the works, particularly the Spanish paintings, among which Ribera's Head of an Apostle is an absolute stunner.

Here again the text was carefully judged and helpful, as it was in Telling Tales, where we're taken on an iconographic treasure hunt through the work of, among others, Watts and Burne-Jones.

In the Great Outdoors, 18th- and 19th-century landscape painting is represented by the work of artists such as Linnell, Turner, Sisley, Cézanne, Pissaro, Boudin and Corot - an impressive roll call by any standards.

Breaking Tradition introduces us to 20th-century British art in the Sheffield City collection, where, within the constraints of a limited purchase budget, there's been some intelligent collecting. Nevinson's Ypres after the First Bombardment leaps out at you, as do Spencer's Helterskelter and Auerbach's Looking Towards Mornington Crescent.

In Abstractions, for the first time, women are well represented, with works by Bridget Riley and Helen Chadwick and one of Sam Taylor-Wood's suspended self-portraits.

The temporary exhibition areas housed photographs from the V&A and a nicely judged biographical tribute to local lads Derrick Greaves and George Fullard, both significant figures in the kitchen sink school.

Given the richness of the Sheffield collection in the area, it would have been good to have had a stronger representation of the galleries own holdings of photography, but perhaps that's to come. As, I hope, is a decent publication on a collection that deserves to be better known, visited and cited.

But despite the collections, the new paint and some incisive curatorship, the Graves remains, as it always has been, somehow lost and sad.

Although a few people were trickling in and out, the galleries and the cafe were empty, while across the road the Millennium Galleries' shop was busy and there were even plenty of people in the travesty that is the Ruskin Gallery.

You can't escape the feeling that, in all the regenerative excitement, the Graves got the leftovers, in terms not only of cash, but of respect.

But it is only two-thirds of the way into the first phase of the Graves' redevelopment, and £200,000 will be spent on phase two, which will be completed in 2009. This will include new furniture, audio, more in-depth interpretation and a refurbished learning centre, and should make a real difference.

Nichola Johnson is the director of the Sainsbury Centre for Visual Arts at the University of East Anglia

Project data

Cost: Phase 1 (2007-08) £100,000; Phase 2 (2008-09) approx £200,000

Main funders: Department for Culture, Media and Sport, Wolfson Foundation

Curators: Sian Brown, Liz Haine Louisa Briggs: Sheffield Galleries & Museums Trust

Project coordinator: Catherine Mailhac: Sheffield Galleries & Museums Trust

Graphic design: CafeteriaCost Phase 1 (2007-08) £100,000; Phase 2 (2008-09) approx £200,000

Main funders: Department for Culture, Media and Sport, Wolfson Foundation

Curators: Sian Brown, Liz Haine Louisa Briggs: Sheffield Galleries & Museums Trust

Project coordinator: Catherine Mailhac: Sheffield Galleries & Museums Trust

Graphic design: Cafeteria