Matt,
Have you read the Database Quickstart located here: http://www.myeclipseide.com/images/tutorials/quickstarts/dbexplorer/
Once you have configured this, you need to add a copy of that same mysql driver to the WEB-INF/lib directory of your web project. Your appserver is respondisble for putting everything that is in the WEB-INF/lib directory into the classpath on a deployed webapplication. So to ensure that your web application can access your driver, just copy the driver into this folder.
Then you can add this jar to your project’s build path of your web project. Note, this doesn’t have anything to do with your appserver finding this file, this is just for your project locally.