The Gilbert Collection, which received the largest Heritage Lottery Fund (HLF) grant ever, could be moving across London from its £30.7m premises at Somerset House to the Victoria and Albert Museum (V&A).

A spokeswoman for the Gilbert Collection said: "The two institutions are in negotiation, but nothing has been decided." But she confirmed that moving the entire collection, which comprises 800 objects worth more than £100m in total, to the V&A was an option being considered.

"It is very difficult when a small museum is trying to raise funds," she added. "The first problem is that it is not free." She said there had been successful temporary exhibitions at Somerset House, such as the Tiffany exhibition, but the shows were "very expensive to stage".

The Gilbert Collection opened in May 2000 after receiving £20.7m in capital funding from the HLF. It also received £10m in an HLF endowment, which will go with the collection if it moves.

Visitor figures have fallen from 129,000 when it opened to 37,800 in 2005, the Gilbert Collection's worst year. There have been only 22,900 visitors in the first nine months of 2007. In 2006-07, turnover was £2.2m.

Carole Souter, director of the HLF, said the Gilbert Collection opened up Somerset House to the public. On the possible move to the V&A, she said: "This move to a world-class museum would ensure the collection is enjoyed by an exceptionally wide range of visitors. The gallery space vacated by the collection, one of the capital's best exhibition venues, would enable the Somerset House Trust to host a new public programme."

Museums Journal has also learned that the Hermitage Rooms at Somerset House, which opened in November 2000 and have objects on loan from the State Hermitage Museum in St Petersburg, is to close as an exhibition space this month.

The Hermitage Rooms programme is co-organised by the Courtauld Institute of Art and St Petersburg. The emphasis will shift from exhibitions to scholarship, as a result of rising costs and poor funding opportunities .

A statement said the UK Friends of the Hermitage is to open an office to help facilitate collaborations with UK institutions.

Felicity Heywood