Pardon the handwriting. You were in the right track, and your thinking is almost there. So my working will be a continuation from yours, so hopefully you can understand.

I added some symbols to the diagram, which is the sides of the shaded rectangles. A1 and B1 are the shorter sides of the rectangles, while A2 and B2 are the longer sides. By doing this labelling, you can see that the side of the bigger square is the sum of one short side of a rectangle and one long side of the other. This can be represented as the side of the square is A1+B2 or A2+B1.

Now that you have your different combinations of multiplication to get your 2 areas, you just have to see which combination suites the sum of A1+B2 and A2+B1, making sure that it is consistent that you add up one short length with one long length of both rectangles.

The 4 tables I made is used to represent this trial and error. I first numbered the multiplications of both A and B from smallest to biggest. Since there are 2 multiplications for rectangle A and 3 for rectangle B, you have a total of 6 combinations for A+B, which I numbered at the side of the table.

For example, if I want to build the table for A1+B2, I will number the sides (1,1) for the first factor of the smaller side of rectangle A, which is 3, and add it the the first factor of the longer side of rectangle B, which is 12. I will continue this for (1,2), (1,3), (2,1) and so on. I will then repeat this for A2+B1.
I will eventually get all of the additions of the factors of both rectangles.

If you can keep up from this point on then that is great! Just a bit more. Now you must take a look at both sides of the tables, both A1+B2 and B1+A2. To find the answer, the two tables must share the same combination, aka (1,1), (1,2) etc, and have the same sum. This is because when building the rectangles, the sides have to be consistent, to get the same areas throughout.

So keep comparing this way. The answer will be the combination that has the same sum. For example looking at (1,2), A1+B2 has the sum of 11, but B1+A2 has the sum of 15. This means the larger figure is not a square, as the sides are different lengths. The only one with the same lengths are (2,2), which is 12cm.

Looking at the bottom of the picture, we can test that at (2,2), the combination works, as the product of the factors and the sum adds up. Which means rectangle A must be 4cmx9cm and rectangle B is 3cmx4cm. With these dimensions, the larger figure is a square.

I hope you can understand this method. The way you represent can be different, but in the end it is still a guess and check method. As long as you are able to show that the dimensions of both A and B are those, then should be fine.

Hope it helps!