You need to investigate the issue and get the facts first. It is perfectly reasonable to ask your neighbour how they arrived at this conclusion and request to see their survey plan. They may be basing their contention on some un-authoritative (non-survey) source and lack proof that the fence encroaches onto their property. But don’t ignore your neighbour; their claim may be valid and the problem won’t go away. You don’t want it to grow into a dispute that will cause emotional and financial distress.
The law does not compel you to move the fence on demand, and your neighbour cannot simply rip down the fence and move it. If the fence has been there for a long time, it may represent evidence of a possessory claim, sometimes called “squatter's rights” (i.e., acquiring title to land when the true owner has neglected to assert their own rights for a specified period of time).
Start by getting a survey to assess your neighbour’s claim. Commission a new survey or buy an existing survey plan from PYB (if you don’t have one) or come to an agreement with the neighbour to have a surveyor mark the boundary on the ground so you can both see the actual location of the boundary. If the dispute progresses, you may need legal advice. Ultimately only a court of law can make the final determination.
Existing survey plans are a valuable resource to homeowners. Please remember, though, that an older survey plan is a snapshot of the property at that time and may not be an accurate reflection of what exists on the property today. If you are unsure, consider consulting a surveyor or commissioning a new survey.