From the sound of the problem, it seems like the clutch pack for the reverse gear and the associated band are shot. The reason that one can draw this conclusion is that you say your car is performing poorly when you shift and that the vehicle manages to move in reverse but there is a strong vibration. You also note that you have to hit the accelerator hard to make the Toyota move.
Put in this light, it seems pretty obvious that the clutch/band combination is the culprit. The reason being this: if the clutch pack/band was working correctly, it would allow your transmission to remain properly disengaged until the engine and transmission had reaching the proper speed. When it had reached the proper speed, the disengaged clutch pack would engage smoothly, assisted by the band which tightens things up a bit more, moving the car ahead smartly with no other problems.
The fly in the ointment here is that since the clutch pack is slipping the engine-transmission linkage slips all of the time, when it should be a good tight bond. As your engine speeds up, the transmission keeps trying to engage but slips out of gear. Since the engine and tranny revs don’t match, primary and secondary vibrations can easily set up, noise can occur and the performance will be extraordinarily lousy.
Fixing this problem involves obtaining the proper clutch pack kit and proper band replacement kit, which will cost about $300 for just the parts. Accessing the clutch pack/band requires a fairly extensive transmission teardown which will cost about $800. Putting it together, this repair will cost you about $1,100.
And, while the transmission is open, you should have the transmission take a look at the major subsystems and see any other systems that need replacement. Since the clutch is shot and since your transmission had had the wrong ATF installed, the chances are very good here that there are other systems that need work. It’s best to do it now before the transmission suffers further damage and increases the cost even more.