One; the property on your side of the fence is yours and you can say who comes and goes on it.

Two; unless the area around the property line is designated on the official plat as an easement, which I will bet it is not, your neighbor has no right to trespass on it. Check your property plat to see if it is either a public easement, in which case it must lead to some place that the public is entitled to go, or if it is a utility easement, in which case, like an alley, it will have power line, gas or water lines on it.

Three; in Texas the neighbor can not claim the property by adverse possession unless he is paying taxes on it, has fenced it, and has treated it as his own. Check with your state attorney general, your county DA, your city attorney, or even your city code officer about local laws regarding adverse possession.