In particular, tptacek's comments about PerfumeBay are relevant to your situation.
Short version: yes, you should be worried. Even if you are in the clear, legally, you may have to deal with the legal system itself. Being legally vindicated can still be expensive.
You should read this entire thread about "TeachBook," which was sued by Facebook: http://news.ycombinator.com/item?id=1633106
In particular, tptacek's comments about PerfumeBay are relevant to your situation.
Short version: yes, you should be worried. Even if you are in the clear, legally, you may have to deal with the legal system itself. Being legally vindicated can still be expensive.