I would suggest keeping two ideas separate:
(1) Among all Markov Processes, the Diffusion Processes have certain smoothness properties as described on Page 16. The Brownian motion is a classic example. There are also MP's whose statistical properties are not smooth, i.e. do not satisfy the Page 16 properties; a major category are the Jump Processes, of which the classic example is the Poisson Counting Process.
(2) Stochastic Differential Equations SDE's are widely used to generate and study specific examples of diffusion. Which allows us to look at many other types of diffusion beyond BM. If the SDE has a solution, then the solution is always a diffusion. In more advanced books like Oksendal Page 66 there are specific conditions on $a(X,t)$ and $b(X,t)$ that are required for the solution to exist and be unique; roughly speaking these require that A and B do not increase too fast as $X$ increases, or else the stochastic process is going to diverge to $\pm \infty$. But to repeat: if the solution of the SDE exists, it is a diffusion.