√(a + b√11) = c/(√11 - 3)²

(a + b√11) = c²/(20 - 6√11)²

a + b√11 = c²/(400 - 240√11 + 36(11))

a + b√11 = c²/(796 - 240√11)

(a + b√11)(796 - 240√11) = c²

796a + (796b - 240a)√11 - 240b(11) = c²

(796a - 2640b) + (796b - 240a)√11 = c²

Since a,b and c are rational numbers but √11 is irrational,

Then coefficient of √11 = 0

796b - 240a = 0
796b = 240a
a = 796b/240
a = 199/60 b

Let b = 60. Then a = 199

796a - 2640b = c²

796(199) - 2640(60) = c²

158404 - 158400 = c²

c² = 4

c = √4 or c = -√4

c = 2 or

c = -2
(reject as √(a + b√11) ≥ 0 for real values of a and b, and (√11 - 3)² > 0 . Since √(a + b√11) = c/(√11 - 3)² , the numerator c cannot be negative)

So a possible set of values would be

a = 199, b = 60 , c = 2

a + b√11 = c²/(796 - 240√11)

At this step it is better to rationalise the right hand side.

a + b√11 = c² (796 - 240√11) / 16
a + b√11 = 49.75 c² - 15c² √11

We find values of a, b and c such that
a = 49.75c²
b = -15c²

If c = 1. for example, then a = 49.75 and b = -15.

It should be a + b√11 = c²(796 + 240√11)/16 instead

And, a,b,c cannot be negative.