We are all concerned about public art – witness the massive interest created recently by Tower Hamlets Council’s controversial decision to sell a Henry Moore sculpture that it owns.

We all have a view on public art, which, more than any other genre, inspires great communal anxiety. There’s no doubt some outdoor sculptures can crudely interrupt what was once an acceptable vista.

Clearly, public art is not easy to get right. I’ve been fortunate enough to present works by the late, great Franz West on the roof of the ICA, whose brightly coloured sausage-shaped sculptures overlooking The Mall couldn’t have looked better.

But that’s my opinion. Conversely, I’ve been held to account, if not so much for getting it wrong (I’d like to think), then for leaving the door open to a negative response.

One such project took place in the Southwood Garden at St James’s Church in Piccadilly, where – with the help of Martin Creed’s Work No. 700 – I learned that a 40-foot-long stack of i-beams was not to everyone’s liking. Having had their lunchtime oasis disrupted, local office workers would no longer eat their sandwiches in the garden.

Whether it is inside or outside the gallery, a harmonious relationship between the public and public art can never be guaranteed. But it’s this sense of risk and experimentation that makes the arena so stimulating. No amount of planning can second guess a public reaction.

There are many different ways in which we encounter public art. London is filled with a heroic mishmash of monuments, forming a cacophony of outmoded iconography. Many such sights have slipped from our minds as meaningful or in the least bit artful, which they clearly are.

Monuments, epitaphs, graffiti, video projections, even modern footbridges might be said to operate in the realm of urban artworks. Yet these everyday gestures, resting somewhere between architecture and design, remain unchallenged when compared with the intense debate surrounding public art.

Successes are thin on the ground. Moore springs to mind – his wonderful arch, for example (thankfully restored to Kensington Gardens), or a more recent manifestation, such as Antony Gormley’s Angel of the North, which is firmly rooted in public affection.

And then, of course, there’s the Fourth Plinth in Trafalgar Square, which forms the focus of an exhibition at the ICA (until 20 January). When we think of the Fourth Plinth we recall it for various reasons, be it Marc Quinn’s Alison Lapper, Rachel Whiteread’s inverted pedestal or Mark Wallinger’s depiction of Christ.

Despite the difficulties surrounding public art, which often stem from inexperienced commissioners who allow amateurism to flourish, artists and the public should celebrate the positive. Public art can work and when it does it is truly revered.

When I worked at Tate we acquired a queue of people for an instructional performance by the Slovakian artist Roman Ondak. It is an impressive work, in which a number of hired hands arrive out of nowhere to form an utterly pointless queue.

Tino Sehgal’s recent work, commissioned by Tate Modern for its Turbine Hall, involved groups of people directly engaging the public through chanting, storytelling and moving about the space as though playing a game.

Works such as these lead me to wonder whether public art in this country should be more daring. Should we allow it to make interventions in more sophisticated or unexpected ways?

For me, the answer is yes. We should be bolder and even more contemporary than we presently are.

Gregor Muir is the executive director of the ICA, London