I'd look at this problem this way - your goal is to get as many nonfiction books as possible, right? If that's the case, and 2/3 of hardcovers are nonfiction as opposed to only 1/4 of softcovers, you're going to want to maximize the number of hardcover books.
But there's a catch...in order to have exactly 1/4 of the softcovers as nonfiction, we need to have at least 4 softcover books. But to only have 4 would mean that there are 104 hardcovers, and because we can't divide 104 into 2/3, we need to find a multiple of softcovers that will leave a multiple of 3 hardcovers. The common multiple of 3 and 4 is 12, and since we want to minimize the number of softcovers, we only want to use 12 of them.
Therefore, as nonfiction books we have:
1/4 of the 12 softcovers = 3
2/3 of the 96 remaining hardcovers = 64
Sum them for a total of 67 nonfiction books.
In the equation they give in the solution, they get to 40 by taking 5/12 of the number of hardcover books, and as we did above noting that we can't have 108 as that number so they're taking 5/12 of 96 (96/12 = 8, which multiplied by 5 is 40). That's another interpretation of how to solve, but if the pure algebra is annoying you can do this one by more logically reasoning out the steps with numbers, too.