It would really depend on the case. What is their relationship with you? (If they are peaceful, then they might just be having a stay in there, if not then i suggest not just removing those row and shared map) Are they in the centre point of your province (if they are, it may become a change of possession and cause a war. If not, you should be fine.)? If you shut off share map and right of war, I believe that you can start a war but I do not know if that scenario is changed since I have left.

Post a pic and I'll be able to tell. Make sure the file is in JPG for when you upload it.

The rule is: if your army passes an empty neutral province centre or a neutral unit on its own territory it inevitably provokes war with the owner of the army/province. This means setting peace back would trap them there, unless there is still a way to get out without passing your province centres or your armies.
Setting peace will cancel any orders that would result in provoking war, it will stop the army from moving in this case. I recommend negotiating it out with Egypt, however 2 infantry don't seem like a threat to me, leave an infantry and an AA gun at home and if they attack, put your army in their way and don't give any orders during the fight. You would likely leave some units at home anyway.